A banned Korean word can still pass.
Try the forbidden-word checker used in a pinned version of IFEval-Ko, a Korean instruction-following benchmark.
01 / TRY A RESPONSE
Do not include 열쇠 — Korean for “key”.02 / CHECK THE RESULT
The response passes this check and still contains 열쇠.
\b열쇠\bWhat changed?
The checker looks for a word boundary on both sides of 열쇠. Korean particles attach directly to a noun, so 열쇠는 has no boundary after 열쇠. For 열쇠는, the check accepts the response even though it still contains 열쇠.
Literal string presence is shown as a diagnostic, not a universal replacement checker. A benchmark still needs to define how it treats particles, compounds and quoted words.
What this tells us
This is a checker reproduction, not a live AI model. In a separate audit of 100 public Mistral-large-2512 responses, the nine forbidden-word items had no changed verdicts from this issue. That run’s score change was 0 percentage points. This demo does not establish an error rate or a leaderboard change.
Source: Allganize IFEval-Ko, commit 54199e3. Browser behavior for the displayed examples was compared with the pinned Python method. The downloadable Python script fetches and verifies that source before running the examples.