agenticassure validate

The validate command checks your YAML scenario files for correctness without running any scenarios. It performs both JSON Schema validation and semantic checks to catch errors early, before you spend time and API credits on a full test run.

agenticassure validate <PATH>

Arguments

Options

What It Validates

The validate command runs two layers of checks on each YAML file:

JSON Schema Validation

Every scenario file is validated against the AgenticAssure JSON Schema. This catches structural errors such as:

• Missing required fields (e.g., suite.name, scenarios).
• Incorrect field types (e.g., timeout set to a string instead of a number).
• Unknown or misspelled top-level keys.
• Invalid nested structures.

Semantic Checks

Beyond schema compliance, the validator checks for logical correctness:

• Required fields have non-empty values where applicable.
• Scorer names reference valid, registered scorers.
• Types are correct in context (e.g., expected_tools is a list of strings).

Path Resolution

• Single file: Validates only that file.
• Directory: Recursively finds all .yml and .yaml files and validates each one.

Only files with .yml or .yaml extensions are considered. Other files in the directory are ignored.

Output

Each file is reported as either OK or FAIL:

agenticassure validate scenarios/

OK scenarios/search_tests.yaml
OK scenarios/regression_tests.yaml
FAIL scenarios/broken_test.yaml
    Missing required field: 'suite.name'
    Scenario 'test_1': 'scorers' must be a list

Validation failed

If all files pass:

OK scenarios/search_tests.yaml
OK scenarios/regression_tests.yaml

All 2 file(s) valid

For failing files, each issue is listed with an indented description below the file path.

Exit Codes

When to Use validate

During Development

Run validate after editing scenario files to catch typos and structural errors immediately, without waiting for a full test run:

agenticassure validate scenarios/new_tests.yaml

In CI/CD Pipelines

Add a validation step before the test run in your CI pipeline. This ensures that broken scenario files fail the build early:

# GitHub Actions example
- name: Validate scenarios
  run: agenticassure validate scenarios/
 
- name: Run tests
  run: agenticassure run scenarios/ --adapter my_agent.MyAgent

Since validate exits with code 1 on failure, it will cause the CI step to fail and prevent the more expensive run step from executing.

As a Pre-Commit Hook

You can use validate as a git pre-commit hook to prevent invalid scenario files from being committed:

#!/bin/sh
# .git/hooks/pre-commit
agenticassure validate scenarios/

Or with a pre-commit framework like pre-commit , add a local hook:

# .pre-commit-config.yaml
repos:
  - repo: local
    hooks:
      - id: validate-scenarios
        name: Validate AgenticAssure scenarios
        entry: agenticassure validate scenarios/
        language: system
        pass_filenames: false

Before Sharing Scenario Files

If you distribute scenario files to teammates or across repositories, validate them first to ensure they conform to the expected schema.

Examples

Validate a single file:

agenticassure validate scenarios/search_tests.yaml

Validate an entire directory:

agenticassure validate scenarios/

Validate from a different working directory:

agenticassure validate /path/to/project/scenarios/

What’s Next

• agenticassure run — Run validated scenarios against your agent.
• agenticassure list — List scenarios and their metadata.
• Scenarios & Suites — YAML format reference.