Loader API Reference

The loader module handles reading, parsing, and validating YAML scenario files. It converts YAML definitions into typed Pydantic models and validates them against a built-in JSON Schema.

Module: agenticassure.loader

from agenticassure.loader import (
    load_scenarios,
    load_scenarios_from_dir,
    validate_scenario_file,
    validate_with_schema,
    SUITE_SCHEMA,
)

Functions

load_scenarios

def load_scenarios(path: str | Path) -> Suite

Load a test suite from a single YAML file. The file is parsed, validated against the JSON Schema, and converted into a Suite object containing typed Scenario instances.

Parameters:

Returns: Suite — A fully populated suite with all scenarios.

Raises:

Behavior details:

• If the YAML file contains a suite block with a name field, that name is used for the Suite.name.
• If no suite block is present, the filename stem (without extension) is used as the suite name.
• Suite-level config values (e.g., default_timeout, retries) are parsed into a SuiteConfig object.
• Each entry in the scenarios list is converted into a Scenario model.

Example:

from agenticassure.loader import load_scenarios
 
# Load from a YAML file
suite = load_scenarios("scenarios/agent_tests.yaml")
 
print(f"Suite: {suite.name}")
print(f"Scenarios: {len(suite.scenarios)}")
 
for scenario in suite.scenarios:
    print(f"  - {scenario.name}: {scenario.input[:50]}...")

load_scenarios_from_dir

def load_scenarios_from_dir(directory: str | Path) -> list[Suite]

Recursively load all YAML scenario files from a directory. Each .yml or .yaml file is loaded as a separate Suite.

Parameters:

Returns: list[Suite] — A list of suites, one per YAML file found. Files are processed in sorted order.

Raises:

Example:

from agenticassure.loader import load_scenarios_from_dir
 
# Load all YAML files from a directory tree
suites = load_scenarios_from_dir("scenarios/")
 
for suite in suites:
    print(f"Suite '{suite.name}': {len(suite.scenarios)} scenarios")

validate_scenario_file

def validate_scenario_file(path: str | Path) -> list[str]

Validate a YAML scenario file and return a list of human-readable issue descriptions. An empty list means the file is valid.

This function performs two levels of validation:

1. JSON Schema validation — Checks structural correctness (required fields, types, no unknown properties).
2. Semantic validation — Additional checks that the JSON Schema cannot express, such as verifying required fields on individual scenarios.

Unlike load_scenarios, this function does not raise exceptions on validation failures. It collects all issues and returns them, making it suitable for linting and CI workflows.

Parameters:

Returns: list[str] — A list of issue descriptions. Empty if the file is valid.

Possible issues returned:

• "File not found: <path>" — The file does not exist.
• "Not a YAML file: <path>" — The file extension is not .yml or .yaml.
• "YAML parse error: <details>" — The file contains invalid YAML syntax.
• "Empty YAML file" — The file is empty or contains only comments.
• "Root element must be a mapping" — The top-level YAML element is not a dictionary.
• "Schema: <path>: <message>" — A JSON Schema validation error at a specific path.
• "No scenarios defined" — The scenarios list is missing or empty.
• "Scenario [N]: missing required field '<field>'" — A scenario is missing name or input.
• "Scenario [N]: 'scorers' must be a list" — Scorers field has the wrong type.

Example:

from agenticassure.loader import validate_scenario_file
 
issues = validate_scenario_file("scenarios/test.yaml")
 
if issues:
    print("Validation failed:")
    for issue in issues:
        print(f"  - {issue}")
else:
    print("File is valid")

validate_with_schema

def validate_with_schema(data: dict) -> list[str]

Validate a parsed YAML dictionary against the SUITE_SCHEMA JSON Schema. Returns a list of human-readable error messages. An empty list means the data is valid.

This is a lower-level function useful when you have already parsed the YAML yourself and want to check it against the schema programmatically.

Parameters:

Returns: list[str] — A list of error messages in the format "Schema: <json_path>: <message>". Empty if valid.

Example:

from agenticassure.loader import validate_with_schema
 
data = {
    "scenarios": [
        {"name": "test1", "input": "Hello"},
        {"name": "test2"},  # missing required "input" field
    ]
}
 
errors = validate_with_schema(data)
for error in errors:
    print(error)
# Schema: scenarios.1: 'input' is a required property

JSON Schema (SUITE_SCHEMA)

The SUITE_SCHEMA constant is a Python dictionary defining the JSON Schema (Draft 2020-12) used to validate scenario YAML files. It enforces the following structure:

Root Object

The root must be an object with the following properties (no additional properties allowed):

suite Object

suite.config Object

scenarios Array Items

Each scenario must be an object (no additional properties allowed):

Important schema behaviors

• additionalProperties: false is set at the root, suite, suite.config, and each scenario item. Any unrecognized property will cause a validation error like "Additional properties are not allowed ('X' was unexpected)".
• minItems: 1 is set on the scenarios array. An empty scenarios list will fail schema validation.
• exclusiveMinimum: 0 is set on timeout_seconds. Zero or negative values will fail validation.

Using the schema directly

from agenticassure.loader import SUITE_SCHEMA
import json
 
# Export the schema as JSON for use with external tools
print(json.dumps(SUITE_SCHEMA, indent=2))

YAML File Format

A complete example of a valid YAML scenario file:

suite:
  name: customer-support-tests
  description: Tests for the customer support chatbot
  config:
    default_timeout: 45
    retries: 1
    default_scorers:
      - passfail
    fail_fast: false
 
scenarios:
  - name: greeting
    input: "Hello, I need help with my order"
    expected_output: "help"
    scorers:
      - passfail
    tags:
      - smoke
      - greeting
 
  - name: order_lookup
    input: "What is the status of order #12345?"
    expected_tools:
      - lookup_order
    expected_tool_args:
      lookup_order:
        order_id: "12345"
    scorers:
      - passfail
    tags:
      - tools
    timeout_seconds: 60
 
  - name: regex_validation
    input: "Generate a confirmation code"
    metadata:
      regex_pattern: "^[A-Z]{3}-\\d{4}$"
    scorers:
      - regex

A minimal valid file requires only the scenarios list:

scenarios:
  - name: simple_test
    input: "Say hello"

When no suite block is provided, the suite name defaults to the filename stem.