Judge prompts: shared rules for spelled codes, mixed-language speech, glosses and politeness by fanny-riols · Pull Request #250 · ServiceNow/eva

fanny-riols · 2026-10-29T19:41:14Z

Why

Some of the problems the native-speaker reviewers found are not specific to one language. They would happen in any language, including English, so they belong in the shared judge prompts rather than in per-language guidance.

This PR changes English scoring for all six judges it touches: conciseness, faithfulness, speakability, tts_fidelity, agent_speech_fidelity and user_speech_fidelity.

Versions are bumped in #251, so this PR must not be run or merged on its own.

What changes

One commit per judge, so each can be reviewed and validated separately. After review, a second commit per judge applies the review fixes; the table below describes the result.

Judge	Change
conciseness	- The politeness exception also covers formal register and set service phrases: one phrase per function (greet / open, acknowledge, hold, return from hold, thank, close), and a single courtesy phrase is never filler. Repetition is still filler: the same formula twice, stacked acknowledgements, or a term followed by its synonym or its translation into another language. - Expanding an acronym once, on its first mention ("your PTO, that's your paid time off"), is not repetition. - Length is judged by how long the turn takes to say and how many information units it carries, not by written length: polite constructions and written-out numbers can look long without adding cognitive load.
faithfulness	- A phrase in another language inside a correct-language turn ("one moment", "Let me check that" in a French call) counts under "not communicating in the specified language"; loanwords, codes, product names and proper nouns never do. Severity: an isolated fixed phrase is rated 2, even when it recurs; whole sentences of content or most turns in another language are rated 1. - A value that merely contains sound-alike letters or digits is not by itself a disambiguation failure; it counts only when the trace shows the ambiguity mattered (failed lookup, correction, two versions, real homophone). - The fabricating_tool_parameters note on confusable letters now points to the language-specific guidance.
speakability	New violation 5, glosses: a term followed, in parentheses or after a slash, by its translation into another language or an exact synonym ("PIN / passcode"), so the listener hears the same thing twice. Not a violation: an acronym expanded once on first mention ("paid time off (PTO)"), a slash between paired terms ("AM/PM", "and/or"), or a word used on its own.
tts_fidelity	New block "Spelled codes and mixed-language speech" (below). The `wrong_language` definition and the rating-0 definition now name the phrase and letter-name cases.
agent_speech_fidelity	Same block, adapted to the 0/1 scale and limited to trace entities (below).
user_speech_fidelity	Same block, adapted to the 1–3 scale.

Spelled codes and mixed-language speech:

Letters and digits in a spelled code must be said with their names in the expected language. A letter or digit name from another language (an English letter name in a German code) is wrong language. If it sounds like a different letter of the expected language, the code itself changes.
Spelling-alphabet words ("A as in Anna") are valid.
Confusable letters and digits: the judge must not flag one because of its own transcription, but audio that is ambiguous between two letters or digits is still an error, as in the existing examples ("M" sounding like "N", "fifty" vs "fifteen").
tts_fidelity and user_speech_fidelity: a phrase in another language that is not in the intended text is wrong language for that turn. If the intended text contains the phrase, reproducing it is not an error.
agent_speech_fidelity: only the entities must be said in the expected language. A phrase in another language around them is left to faithfulness and tts_fidelity, so a turn is not rated differently depending on whether it happens to contain an entity.
Not wrong language: proper nouns, product names, codes, acronyms and established loanwords.
tts_fidelity and user_speech_fidelity: the intended words read in an unusual order or grouping (digit by digit vs as one number) are an error only if a native listener could take them for a different value. A different word for the value ("septante" for "soixante-dix") is still flagged, as the existing rule says.

tests/fixtures/judge_prompts_en.json records the new English renders of the six judges.

Validation

English judge validation datasets (docs/metrics/judge_validation_datasets/), production judge models, 3 runs per side, mean ± std. Before is Judge prompts: per-language guidance slot #249, which renders the same English prompts as main.

Judge	Records		Accuracy	Macro-F1
faithfulness (`us.anthropic.claude-opus-4-6-v1`)	137	before	70.07% ± 0.00%	65.14% ± 0.26%
		after	69.34% ± 1.26%	64.39% ± 1.99%
conciseness (`gpt-5.2`)	100	before	92.35% ± 0.99%	85.02% ± 1.95%
		after	93.31% ± 0.76%	86.45% ± 1.64%

Both changes are within one standard deviation: no English regression on these datasets. The dataset runner has no audio judges, so agent_speech_fidelity is not covered here.

Re-scoring the native-speaker review set: the 310 conversations the reviewers commented on (de, es, fr, fr-CA, hi, it, ja, ko), before (Judge prompts: per-language guidance slot #249) and after (Judge prompts: shared rules for spelled codes, mixed-language speech, glosses and politeness #250), for speakability, conciseness, faithfulness and agent_speech_fidelity.
- Speakability (gpt-5.2): glosses are now flagged. 66 of the 70 turns that moved from pass to fail are a term followed by its translation or exact synonym, such as a native term with its English equivalent in parentheses. Why: the new violation 5 names glosses, which the old wording did not cover, since the listener hears the same thing twice. Acronyms expanded on first mention and slash pairs like "AM/PM" are excluded, so they keep passing.
- Conciseness (gpt-5.2): a small shift toward lenient, from 0.812 to 0.829. 226 turns moved up against 148 down, while a re-run on the same code splits evenly (182 / 180). verbosity_or_filler flags go from 552 to 502 and excess_information_density from 208 to 181. Why: one courtesy phrase per function and one first-mention acronym expansion are now allowed, and politeness conventions are said to add length "without adding cognitive load". So turns that are polite but not repetitive are no longer counted as filler.
- Faithfulness (us.anthropic.claude-opus-4-6-v1):
  - Wrong-language phrases are rated by severity. An isolated English phrase in another language, such as "Let's do it" in a German call, now gets 2 instead of 3, and recurring fixed phrases that were rated 1 now get 2. Why: Judge prompts: shared rules for spelled codes, mixed-language speech, glosses and politeness #250 separates a fixed phrase (a hold, acknowledgement or courtesy formula), rated 2, from whole sentences or most turns in another language, rated 1.
  - There are fewer confusability flags: conversations flagged for failing_to_disambiguate go from 63 to 53, and for fabricating_tool_parameters from 48 to 40. Why: sound-alike letters or digits are flagged only when the conversation shows the ambiguity mattered.
- agent_speech_fidelity (gemini-3-flash-preview): more real entity errors are caught, such as an extra letter at the end of a code, a wrong digit count in an ID, a vowel substituted in a confirmation code, or a letter said as a word. Why: letters and digits in spelled codes must be spoken with the expected language's names, and ambiguous audio of an entity is an error. Only 2% of turns change rating, so these changes are not run-to-run variation.
- Hindi: Judge prompts: shared rules for spelled codes, mixed-language speech, glosses and politeness #250 has no Hindi guidance, so the shared wrong-language rules also flag the code-switching that is normal in Hindi, which lowers the Hindi faithfulness and agent_speech_fidelity scores. Judge prompts: guidance for de, nl, fr, fr-CA, es, it, hi, ko, ja #251 adds Hindi guidance that allows code-switching, which will change the scores for these two metrics.

Stack

Judge prompts: per-language guidance slot #249 Guidance slot and loader (base of this PR).
Judge prompts: shared rules for spelled codes, mixed-language speech, glosses and politeness #250 This PR: shared judge rules.
Judge prompts: guidance for de, nl, fr, fr-CA, es, it, hi, ko, ja #251 Per-language guidance content (based on this PR).

Versions are bumped in PR 3.

- The cultural-norms exception now also covers formal register and set service phrases. It allows one such phrase per function (acknowledge, hold, return from hold, thank, close), and a single short courtesy or empathy phrase is never filler on its own. - It does not cover repetition: the same formula twice in one turn, stacked acknowledgements, or a term given together with its synonym or translation are still filler. - Length is judged by how long the turn takes to say and how many information units it carries, not by written length. Polite constructions and numbers written out in words can look long without being long to hear.

- violating_policies: a phrase or clause in another language inside an otherwise correct-language turn ("one moment", "Let me check that") is part of "not communicating in the specified language". Established loanwords, codes, IDs, acronyms, product names and proper nouns are never a violation. - failing_to_disambiguate: a value that only contains letters or digits that sound alike is not by itself a flag. Phonetic confusability counts only when the trace shows the ambiguity mattered: a failed lookup, a user correction, two versions of the value, or a real homophone. - The fabricating_tool_parameters note on confusable letters now points to the language-specific guidance.

New violation 5: a term immediately followed by its translation, equivalent or abbreviation in parentheses or after a slash, such as "paid time off (PTO)" or "PIN / passcode". TTS does not voice the parentheses or slash, so the listener hears both forms back to back. A word, code or abbreviation used on its own is not a violation.

New block "Spelled codes and mixed-language speech": - Letters and digits in spelled codes must be said with their names in the expected language. A letter or digit name from another language (the English "jay" or "six" in a non-English turn) is wrong_language, plus entity_error if it sounds like a different letter or digit. - Spelling-alphabet words ("A as in Anna") are valid. - Confusable letters and digits are flagged only when the other one is clearly heard, not because of transcription or poor audio. - A phrase or sentence in another language inside the turn is wrong_language, even when the rest of the turn is correct. - Proper nouns, product names, codes, acronyms and established loanwords are not wrong language. - An unusual but unambiguous number, date or time reading is an error only if a native listener could take it for a different value.

New block "Spelled codes and mixed-language speech": - Letters and digits of trace entities must be said with their names in the expected language. A letter or digit name from another language (the English "jay" or "six" in a non-English turn) means the entity was not articulated correctly: rating 0. An entity that is itself in another language in the trace may be said in that language. - Spelling-alphabet words ("A as in Anna") are valid. - Confusable letters and digits are flagged only when the other one is clearly heard, not because of transcription or poor audio. - A phrase or sentence in another language inside the turn: rating 0, quoting those words in the explanation. - Proper nouns, product names, codes, acronyms and established loanwords are not wrong language. - An unusual but unambiguous number, date or time reading is rated 0 only if a native listener could take it for a different value.

New section "Spelled Codes and Mixed-Language Speech": - Letters and digits in spelled codes must be said with their names in the expected language. A letter or digit name from another language (the English "jay" or "six" in a non-English turn) rates at most 2 while the value is still unambiguous, and 1 if it sounds like a different letter or digit. - Spelling-alphabet words ("A as in Anna") are valid. - Confusable letters and digits are flagged only when the other one is clearly heard, not because of transcription or poor audio. - A phrase or sentence in another language inside the turn rates at most 2, and 1 if it replaces an entity or the turn no longer makes sense. - Proper nouns, product names, codes, acronyms and established loanwords are not wrong language. - An unusual but unambiguous number, date or time reading rates at most 2 if it sounds unnatural, and 1 only if a native listener could take it for a different value.

…hared-multilingual-rules

The shared rules in this branch change the English prompts of six judges. This records their new English renders.

- Expanding an acronym once, on its first mention ("your PTO, that's your paid time off"), is not repetition. A term given with its synonym or its translation into another language is still filler. The example no longer assumes a non-English call. - The one-phrase-per-function allowance also covers greet / open. - Polite constructions and numbers written out in words can look long without adding cognitive load. The previous wording said they don't take longer to say, which isn't true.

The rating scale had "briefly using the wrong language" at 2 and "repeatedly talking in a language not instructed" at 1, so a recurring English "one moment" in a French call could go either way. Now: - an isolated fixed phrase (a hold, acknowledgement or courtesy formula) is rated 2, even when it recurs across turns; - whole sentences of content in another language, or most turns in another language, are rated 1.

Violation 5 flagged "paid time off (PTO)", which is a normal acronym expansion in English HR and IT turns. It now covers only a term followed by its translation into another language or by a synonym that means exactly the same thing ("PIN / passcode"). Not a violation: - an acronym expanded once, on its first mention; - a slash between paired or alternative terms ("AM/PM", "and/or"); - a word, code or abbreviation used on its own. Also drop the claim that TTS never voices a slash; some engines do.

- Confusable letters and digits: the rule now only stops the judge from flagging one because of its own transcription. Audio that is ambiguous between two letters or digits is still an error. The previous "benefit of the doubt" wording contradicted the entity_error definition. - A phrase in another language is an error only if it is not in the intended text; reproducing a phrase the intended text contains is not. - The wrong_language definition and the rating-0 definition now name the cases the new rules use: a phrase or clause in another language, and a letter or digit name from another language inside a spelled code. - An unusual reading covers only the intended words read in an unusual order or grouping (digit by digit vs as one number). A different word for the value ("septante" for "soixante-dix") is still an error, as the existing rule says. - The example no longer reads "a English turn" in English runs.

- A phrase in another language around the entities no longer rates the turn 0. Before, it did only when the turn had entities, so "One moment, je vérifie." was excluded while "One moment, je vérifie le code ZK3FFW." was rated 0 with a perfect code. Faithfulness and TTS fidelity already score the phrase. - A letter or digit name from another language still rates 0 when it is inside a trace entity. - Confusable letters and digits: the rule now only stops the judge from flagging one because of its own transcription. Audio that is ambiguous between two letters or digits is still an error, as in the judge's own examples ("M" sounding like "N", "fifty" vs "fifteen"). - The example no longer reads "a English turn" in English runs.

- Confusable letters and digits: the rule now only stops the judge from flagging one because of its own transcription. Audio that is ambiguous between two letters or digits is still an error. The previous "benefit of the doubt" wording made the judge more lenient on the hardest cases, so fewer records with a mis-spoken code were rerun. - A phrase in another language is an error only if it is not in the intended text; reproducing a phrase the intended text contains is not. - An unusual reading covers only the intended words read in an unusual order or grouping. A different word for the value ("septante" for "soixante-dix") is still flagged, as the existing rule says. - The example no longer reads "a English turn" in English runs.

…hared-multilingual-rules

The fabricating_tool_parameters note on confusable letters pointed to "the language-specific guidance below". This branch adds no guidance, so the pointer moves to the branch that does.

sn-rachelhansen · 2026-11-04T22:02:21Z

+        - Confusable letters and digits: don't flag one because of your own transcription; audio that is ambiguous between two letters or digits is still an error.
+        - A phrase or sentence in another language that is not in the intended text (e.g., an English "one moment" in a French turn) is rating = 0 with `wrong_language`, even when the rest of the turn is in {expected_language}. If the intended text itself contains the phrase, reproducing it is not an error.
+        - Not wrong language: proper nouns, official product or system names, codes and acronyms, and loanwords that are established in {expected_language}.
+        - The intended words read in an unusual order or grouping (e.g., a number read digit by digit instead of as one number, or the reverse) are not an error by themselves; they are an `entity_error` only if a native {expected_language} listener could take it for a different value. A different word for the value (e.g., "septante" for "soixante-dix") is still an error, as above.


I don't know what "as above" at the very end here refers to

Good catch. It pointed to the "Number surface forms" exception in the "What to ignore" list a few lines up, which says that a different word for the same value (Swiss/Belgian "septante" vs standard French "soixante-dix") should be flagged. That was too far away to be clear, so I replaced "as above" with the rule itself: "…is still an error, even though it means the same number." (1779127)

sn-rachelhansen · 2026-11-04T22:05:29Z

+        - Confusable letters and digits: don't flag one because of your own transcription; audio that is ambiguous between two letters or digits is still an error.
+        - A phrase or sentence in another language that is not in the intended text (e.g., an English "one moment" in a French turn) is wrong language for part of the turn: rating at most 2, and 1 if it replaces an entity or the turn no longer makes sense. If the intended text itself contains the phrase, reproducing it is not an error.
+        - Not wrong language: proper nouns, official product or system names, codes and acronyms, and loanwords that are established in {expected_language}.
+        - The intended words read in an unusual order or grouping (e.g., a number read digit by digit instead of as one number, or the reverse) are not an entity error: rating at most 2 if they sound unnatural, and 1 only if a native {expected_language} listener could take it for a different value. A different word for the value (e.g., "septante" for "soixante-dix") is still flagged, as above.


same comment, not sure what 'as above' means here

Same as for tts_fidelity: it referred to the "Number surface forms" exception earlier in this prompt. I replaced "as above" with the rule itself: "…is still flagged, even though it means the same number." (a3aa6c4)

sn-rachelhansen

I think this looks pretty good - I left a handful of comments. My only concern are the things I think are 'missing' (but these things may very well be covered in future PRs). The only thing that comes to mind immediately is the problem with speakability marking down things that are tags for the TTS model (ex. "[speaker cuts off]"). If I think of anything else, I'll leave send a message in the chat

Co-authored-by: sn-rachelhansen <rachel.hansen@servicenow.com>

fanny-riols · 2026-11-05T20:39:49Z

Thank you @sn-rachelhansen for your review! For speakability, I merged the fix last week :) #246

The bullet on unusual number readings ended with "is still an error, as above", pointing to the "Number surface forms" exception in the What to ignore list. Say the rule directly instead.

Same wording fix as for tts_fidelity: replace "as above" with the rule itself.

The conciseness suggestion and the two "different word" wording fixes change the English conciseness, tts_fidelity and user_speech_fidelity prompts on purpose.

tiffanyannedo

LGTM

tiffanyannedo · 2026-11-05T21:50:58Z

      - Making a reasonable inference when the context makes the intent obvious (e.g., user says "my flight" when they only have one flight)
      - Retrying a lookup with a corrected spelling after the user confirms or spells out the information — this is proper disambiguation behavior
      - Trying valid different combinations of names and confirmation codes when a lookup fails (e.g., swapping commonly confused letters like "v"/"z" or "b"/"d", reordering characters)
+      - A value that merely contains letters or digits that sound alike (e.g., "b"/"d", or a digit and a letter that are homophones in the conversation's language) — this is NOT by itself a reason to flag failing_to_disambiguate or fabricating_tool_parameters. Use phonetic confusability only when the trace shows the ambiguity mattered: a failed lookup, a user correction, two different versions of the value across turns, or an utterance that can genuinely be heard two ways (a homophone). Do not flag hypothetical alternative readings when the value was verified successfully and nothing in the conversation signals ambiguity.


Maybe it's better to add an example of where a digit and a letter might be homophones?

Suggested change

- A value that merely contains letters or digits that sound alike (e.g., "b"/"d", or a digit and a letter that are homophones in the conversation's language) — this is NOT by itself a reason to flag failing_to_disambiguate or fabricating_tool_parameters. Use phonetic confusability only when the trace shows the ambiguity mattered: a failed lookup, a user correction, two different versions of the value across turns, or an utterance that can genuinely be heard two ways (a homophone). Do not flag hypothetical alternative readings when the value was verified successfully and nothing in the conversation signals ambiguity.

- A value that merely contains letters or digits that sound alike (e.g., "b"/"d", or a digit and a letter, "9"/"Q" in Ja, that are homophones in the conversation's language) — this is NOT by itself a reason to flag failing_to_disambiguate or fabricating_tool_parameters. Use phonetic confusability only when the trace shows the ambiguity mattered: a failed lookup, a user correction, two different versions of the value across turns, or an utterance that can genuinely be heard two ways (a homophone). Do not flag hypothetical alternative readings when the value was verified successfully and nothing in the conversation signals ambiguity.

fanny-riols added 6 commits October 29, 2026 15:36

This was referenced Oct 29, 2026

Judge prompts: per-language guidance slot #249

Open

Judge prompts: guidance for de, nl, fr, fr-CA, es, it, hi, ko, ja #251

Draft

fanny-riols added 11 commits October 29, 2026 16:32

Merge branch 'pr/fr/judge-language-guidance-infra' into pr/fr/judge-s…

537f5b3

…hared-multilingual-rules

Update the English golden prompts for the shared rules

57047b4

The shared rules in this branch change the English prompts of six judges. This records their new English renders.

Merge branch 'pr/fr/judge-language-guidance-infra' into pr/fr/judge-s…

1a816c3

…hared-multilingual-rules

Merge branch 'pr/fr/judge-language-guidance-infra' into pr/fr/judge-s…

2bffc63

…hared-multilingual-rules

Faithfulness: drop the pointer to language-specific guidance

20713cc

The fabricating_tool_parameters note on confusable letters pointed to "the language-specific guidance below". This branch adds no guidance, so the pointer moves to the branch that does.

fanny-riols marked this pull request as ready for review November 3, 2026 21:19

sn-rachelhansen reviewed Nov 4, 2026

View reviewed changes

Comment thread configs/prompts/judge.yaml Resolved

sn-rachelhansen reviewed Nov 4, 2026

View reviewed changes

fanny-riols and others added 2 commits November 5, 2026 15:38

Update configs/prompts/judge.yaml

cdae19b

Co-authored-by: sn-rachelhansen <rachel.hansen@servicenow.com>

Apply pre-commit

1ab32ab

fanny-riols added 3 commits November 5, 2026 15:46

TTS fidelity: state the "different word" rule in place

1779127

The bullet on unusual number readings ended with "is still an error, as above", pointing to the "Number surface forms" exception in the What to ignore list. Say the rule directly instead.

User speech fidelity: state the "different word" rule in place

a3aa6c4

Same wording fix as for tts_fidelity: replace "as above" with the rule itself.

Regenerate the English judge prompt golden

2672500

The conciseness suggestion and the two "different word" wording fixes change the English conciseness, tts_fidelity and user_speech_fidelity prompts on purpose.

tiffanyannedo approved these changes Nov 5, 2026

View reviewed changes

Navigation Menu

Repository navigation

Judge prompts: shared rules for spelled codes, mixed-language speech, glosses and politeness - #250

Judge prompts: shared rules for spelled codes, mixed-language speech, glosses and politeness#250
fanny-riols wants to merge 22 commits into
pr/fr/judge-language-guidance-infrafrom
pr/fr/judge-shared-multilingual-rules

fanny-riols commented Oct 29, 2026 •

edited

Loading

Uh oh!

Uh oh!

sn-rachelhansen Nov 4, 2026

Uh oh!

fanny-riols Nov 5, 2026

Uh oh!

sn-rachelhansen Nov 4, 2026

Uh oh!

fanny-riols Nov 5, 2026

Uh oh!

sn-rachelhansen left a comment

Uh oh!

fanny-riols commented Nov 5, 2026

Uh oh!

tiffanyannedo left a comment

Uh oh!

tiffanyannedo Nov 5, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

fanny-riols commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

What changes

Validation

Stack

Uh oh!

Uh oh!

sn-rachelhansen Nov 4, 2026

Choose a reason for hiding this comment

Uh oh!

fanny-riols Nov 5, 2026

Choose a reason for hiding this comment

Uh oh!

sn-rachelhansen Nov 4, 2026

Choose a reason for hiding this comment

Uh oh!

fanny-riols Nov 5, 2026

Choose a reason for hiding this comment

Uh oh!

sn-rachelhansen left a comment

Choose a reason for hiding this comment

Uh oh!

fanny-riols commented Nov 5, 2026

Uh oh!

tiffanyannedo left a comment

Choose a reason for hiding this comment

Uh oh!

tiffanyannedo Nov 5, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

fanny-riols commented Oct 29, 2026 •

edited

Loading