Regex Scorer

The Regex scorer checks whether the agent’s output matches a regular expression pattern. The pattern is defined in the scenario’s metadata.regex_pattern field and evaluated using Python’s re.search, which looks for a match anywhere in the output. The score is 1.0 if the pattern matches, and 0.0 otherwise.

How It Works

1. The scorer reads the regex_pattern key from the scenario’s metadata dictionary.
2. It compiles the pattern using Python’s re module.
3. It runs re.search(pattern, result.output) against the agent’s output.
4. If a match is found, the score is 1.0. If not, 0.0.

If metadata does not contain a regex_pattern key, the scorer fails with an explanation. If the pattern is an invalid regular expression, the scorer also fails with an error message describing the problem.

Setting Up the Regex Pattern

The pattern is specified in the metadata field of a scenario:

suite:
  name: output-format-tests
  adapter: my_project.adapters.MyAdapter
  scorer: regex
 
scenarios:
  - name: returns-a-number
    input: "How many items are in stock?"
    metadata:
      regex_pattern: "\\d+"

The metadata field is a free-form dictionary. The Regex scorer specifically looks for the regex_pattern key. All other metadata keys are ignored by this scorer (but can be used by custom scorers or your adapter).

Example Scenarios

Check that output contains a number

- name: numeric-response
  input: "What is the current price?"
  metadata:
    regex_pattern: "\\$\\d+\\.\\d{2}"

Matches output like "The price is $29.99" or "$100.00 per unit".

Validate email address in output

- name: email-extraction
  input: "What is the support email?"
  metadata:
    regex_pattern: "[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}"

Matches standard email addresses anywhere in the output.

Check for a date format

- name: date-in-response
  input: "When was the order placed?"
  metadata:
    regex_pattern: "\\d{4}-\\d{2}-\\d{2}"

Matches ISO-style dates like 2025-03-15.

Anchored full-match

- name: exact-label
  input: "Classify this ticket"
  metadata:
    regex_pattern: "^(bug|feature|question)$"

The ^ and $ anchors ensure the entire output is exactly one of the listed labels, with nothing else.

Case-insensitive matching

- name: greeting-check
  input: "Greet the user"
  metadata:
    regex_pattern: "(?i)hello|hi|hey"

The (?i) inline flag makes the pattern case-insensitive. Matches "Hello", "HI", "hey there", etc.

Multiple required phrases

- name: contains-both
  input: "Summarize the policy"
  metadata:
    regex_pattern: "(?s)(?=.*refund)(?=.*30 days)"

Uses lookaheads to require that both "refund" and "30 days" appear somewhere in the output. The (?s) flag makes . match newlines, so this works across multiline output.

Common Patterns Reference

Regex Syntax Tips

The Regex scorer uses Python’s re module. Here are the most relevant syntax elements:

• . matches any character (except newline by default).
• * matches zero or more of the preceding element.
• + matches one or more of the preceding element.
• ? makes the preceding element optional.
• ^ matches the start of the string.
• $ matches the end of the string.
• \d matches a digit, \w matches a word character, \s matches whitespace.
• [abc] matches any one of the characters inside the brackets.
• (a|b) matches a or b.
• (?i) inline flag for case-insensitive matching.
• (?s) inline flag to make . match newlines.
• (?=...) lookahead — asserts that the pattern matches ahead without consuming characters.

For a complete reference, see the Python re module documentation .

YAML Escaping Gotchas

YAML and regex both use backslashes, which creates a common source of errors. Here are the rules to follow:

Always double your backslashes in YAML. A single backslash in YAML is an escape character. To get a literal backslash in the resulting string, you must write \\.

# WRONG -- YAML interprets \d as an escape sequence
metadata:
  regex_pattern: "\d+"
 
# CORRECT -- double backslash produces literal \d in the string
metadata:
  regex_pattern: "\\d+"

Or use a YAML literal block scalar. The | style preserves backslashes as-is, so you do not need to double them:

metadata:
  regex_pattern: |
    \d{4}-\d{2}-\d{2}

However, note that the | style adds a trailing newline to the string. Use |- to strip it:

metadata:
  regex_pattern: |-
    \d{4}-\d{2}-\d{2}

Single-quoted strings also preserve backslashes:

metadata:
  regex_pattern: '\d+'

In YAML, single-quoted strings do not process backslash escapes. This is often the simplest option for regex patterns.

Summary of quoting strategies:

When in doubt, use single quotes for simple patterns and block scalars for complex ones.

Error Handling

Missing regex_pattern

If the scenario does not include regex_pattern in its metadata, the scorer returns a score of 0.0 with details:

Regex: FAIL. No regex_pattern defined in scenario metadata.

Invalid regex

If the pattern cannot be compiled, the scorer returns 0.0 with details that include the error message from Python’s re module:

Regex: FAIL. Invalid regex pattern: missing ), unterminated subpattern at position 5.

This makes it straightforward to diagnose typos or syntax errors in your patterns.

The Details Field

On success, the details include the pattern and the matched text:

Regex: PASS. Pattern "\\d{4}-\\d{2}-\\d{2}" matched: "2025-03-15".

On failure (no match):

Regex: FAIL. Pattern "\\d{4}-\\d{2}-\\d{2}" did not match output.

When to Use Regex vs. Other Scorers

The Regex scorer occupies the middle ground between PassFail’s simple substring check and Exact Match’s strict equality. It is ideal when you care about the structure or format of the output without requiring an exact string.

See Also

• PassFail Scorer — the default scorer with substring checking
• Exact Match Scorer — strict string equality
• Similarity Scorer — semantic comparison
• Custom Scorers — building your own
• Python re documentation  — full regex syntax reference