Configuration

AgenticAssure can be configured through config files, CLI flags, environment variables, and suite-level settings. This page documents all configuration options and how they interact.

Configuration Precedence

When the same setting is specified in multiple places, the following precedence order applies (highest priority first):

CLI flags — Command-line arguments like --timeout, --retry, --adapter.
Suite-level config — The config block inside a suite’s YAML file.
Config file — Global settings in agenticassure.yaml or agenticassure.toml.
Defaults — Built-in default values.

For example, if retries: 3 is set in the YAML suite config and --retry 1 is passed on the command line, the CLI value (1) takes effect.

Config File

AgenticAssure looks for a config file in the current working directory when the CLI is invoked. Two formats are supported:

YAML Format (`agenticassure.yaml`)


adapter: mymodule.MyAgent

TOML Format (`agenticassure.toml`)


adapter = "mymodule.MyAgent"

If both files exist, the YAML file is checked first. The TOML file is used only if no YAML file is found.

Config File Options

Option	Type	Description
`adapter`	`string`	Python dotted path to your `AgentAdapter` class (e.g., `"myproject.agent.MyAgent"`). This is the only option currently supported in the config file.

Config File Purpose

The config file primarily eliminates the need to pass --adapter on every CLI invocation. Instead of:


agenticassure run scenarios/ --adapter myproject.agent.MyAgent

You can create agenticassure.yaml:


adapter: myproject.agent.MyAgent

And then simply run:


agenticassure run scenarios/

The --adapter CLI flag overrides the config file value when both are present.

CLI Flags

The run command supports the following flags:

Flag	Short	Type	Default	Description
`--adapter`	`-a`	string	None	Python dotted path to an `AgentAdapter` class. Overrides the config file.
`--output`	`-o`	choice	`cli`	Output format: `cli`, `json`, or `html`.
`--timeout`		float	`30.0`	Default timeout in seconds for scenario execution.
`--retry`		int	`0`	Number of retry attempts per scenario on failure.
`--suite`	`-s`	string	None	Run only the named suite (matches against `suite.name` in YAML).
`--tag`	`-t`	string (multiple)	None	Filter scenarios by tag. Can be specified multiple times.
`--dry-run`		flag	`false`	Validate and list scenarios without executing them.

Usage Examples


# Basic run with adapter
agenticassure run scenarios/ --adapter mymodule.MyAgent
 
# Specify output format and retry count
agenticassure run scenarios/ -a mymodule.MyAgent -o html --retry 2
 
# Filter by tag
agenticassure run scenarios/ -a mymodule.MyAgent -t smoke -t regression
 
# Run a specific suite
agenticassure run scenarios/ -a mymodule.MyAgent --suite "api-tests"
 
# Dry run to validate scenarios
agenticassure run scenarios/ --dry-run
 
# Run a single file
agenticassure run scenarios/test.yaml -a mymodule.MyAgent

Suite-Level Configuration

Each YAML scenario file can include a config block inside the suite section. These settings override the runner defaults for that specific suite.


suite:
  name: integration-tests
  description: Full integration test suite
  config:
    default_timeout: 60
    retries: 2
    default_scorers:
      - passfail
      - exact
    fail_fast: true
 
scenarios:
  - name: test1
    input: "Hello"
  # ...

Suite Config Options

Option	Type	Default	Description
`default_timeout`	`float`	`30.0`	Default timeout in seconds for scenarios in this suite.
`retries`	`int`	`0`	Number of retry attempts for each scenario. If set, overrides the runner’s retry count for this suite.
`default_scorers`	`list[str]`	`["passfail"]`	Default scorers applied to scenarios that do not specify their own `scorers` list.
`fail_fast`	`bool`	`false`	If `true`, stop executing remaining scenarios after the first failure.

How Suite Config Interacts with the Runner

When using Runner.run_suite():

retries from the suite config overrides the runner’s constructor retries value (if the suite config value is non-zero).
fail_fast from the suite config overrides the runner’s constructor fail_fast value (if the suite config value is True).
default_timeout and default_scorers are set on the SuiteConfig model but are available for adapter and scorer implementations to read.

When using Runner.run_scenario() directly, suite-level config does not apply. Only the runner’s constructor values are used.

Scenario-Level Configuration

Individual scenarios can override certain settings:


scenarios:
  - name: slow_test
    input: "Complex multi-step task"
    timeout_seconds: 120       # Override default timeout
    scorers:                   # Override default scorers
      - passfail
      - similarity
    metadata:                  # Scorer-specific configuration
      similarity_threshold: 0.85
      regex_pattern: "\\d+"
      exact_normalize: false

Scenario-Level Options

Option	Location	Description
`timeout_seconds`	Scenario field	Timeout for this specific scenario.
`scorers`	Scenario field	Scorers to use for this scenario. Overrides `default_scorers` from suite config.
`similarity_threshold`	`metadata`	Override the similarity scorer threshold (default: `0.7`).
`regex_pattern`	`metadata`	Regex pattern for the regex scorer.
`exact_normalize`	`metadata`	Whether the exact scorer normalizes strings (default: `true`).

Environment Variables

Variable	Description
`HF_TOKEN`	Hugging Face API token. Used by the `similarity` scorer to authenticate with the Hugging Face Hub when downloading sentence-transformer models. Setting this avoids rate limit warnings and errors.
`OPENAI_API_KEY`	Required when using the built-in `OpenAIAdapter`. The OpenAI client reads this automatically.
`PYTHONPATH`	Add directories to Python’s module search path. Needed when your adapter module is not in an installed package.

Setting Environment Variables


# Linux/macOS (bash/zsh)
export HF_TOKEN=hf_your_token_here
export OPENAI_API_KEY=sk-your_key_here
 
# Windows (PowerShell)
$env:HF_TOKEN = "hf_your_token_here"
$env:OPENAI_API_KEY = "sk-your_key_here"
 
# Windows (CMD)
set HF_TOKEN=hf_your_token_here
set OPENAI_API_KEY=sk-your_key_here

Example Configurations

Minimal Setup

A single YAML file with no config file needed:


# scenarios/test.yaml
scenarios:
  - name: basic
    input: "Hello"
    expected_output: "hello"


agenticassure run scenarios/test.yaml --adapter mymodule.MyAgent

Standard Project Setup

agenticassure.yaml (in project root):


adapter: my_project.agent.ProductionAgent

scenarios/smoke.yaml:


suite:
  name: smoke-tests
  config:
    retries: 1
    fail_fast: true
 
scenarios:
  - name: greeting
    input: "Hello"
    expected_output: "hello"
    tags: [smoke]
 
  - name: farewell
    input: "Goodbye"
    expected_output: "goodbye"
    tags: [smoke]

scenarios/integration.yaml:


suite:
  name: integration-tests
  config:
    default_timeout: 60
    retries: 2
 
scenarios:
  - name: tool_usage
    input: "Look up order #123"
    expected_tools: [lookup_order]
    tags: [integration, tools]
    timeout_seconds: 90
 
  - name: semantic_quality
    input: "Explain our refund policy"
    expected_output: "Customers can request a refund within 30 days"
    scorers: [similarity]
    tags: [integration, quality]
    metadata:
      similarity_threshold: 0.75

Running:


# Run everything
agenticassure run scenarios/
 
# Run only smoke tests
agenticassure run scenarios/ --tag smoke
 
# Run with different adapter (overrides config file)
agenticassure run scenarios/ --adapter my_project.agent.StagingAgent
 
# Validate without running
agenticassure run scenarios/ --dry-run

CI Pipeline Setup


# Install with optional dependencies
pip install agenticassure[similarity]
 
# Validate scenario files first
agenticassure validate scenarios/
 
# Run tests with JSON output for artifact collection
agenticassure run scenarios/ --adapter my_project.agent.CIAgent --output json --retry 1
 
# Run with HTML report
agenticassure run scenarios/ --adapter my_project.agent.CIAgent --output html

The run command exits with code 1 if any scenario fails, making it suitable for CI pipelines.

Programmatic Configuration

When using the Python API, all configuration is passed directly:


from agenticassure.runner import Runner
from agenticassure.loader import load_scenarios
 
suite = load_scenarios("scenarios/tests.yaml")
 
runner = Runner(
    adapter=my_adapter,
    default_timeout=60.0,
    retries=2,
    fail_fast=True,
)
 
result = runner.run_suite(
    suite,
    tags=["smoke"],
    context={"environment": "staging"},
)

In programmatic mode, config files are not read. All settings are explicit in the code.