Project Setup

This guide covers how to structure an AgenticAssure project, configure defaults, manage environment variables, and organize test scenarios at scale.

Using agenticassure init

The init command scaffolds a new project with a scenarios/ directory and an example scenario file:

agenticassure init .

You can also initialize into a specific directory:

agenticassure init tests/agent-tests

This creates:

tests/agent-tests/
  scenarios/
    example_scenarios.yaml

The generated example_scenarios.yaml includes two starter scenarios (a greeting test and a tool usage test) that demonstrate the YAML format. Edit or replace this file with your own scenarios.

Recommended Project Structure

For a typical project, we recommend keeping your agent adapter and test scenarios together in a dedicated test directory:

my-project/
  src/
    my_app/
      agent.py                 # Your AI agent
  tests/
    agent/
      adapters/
        my_adapter.py          # AgenticAssure adapter wrapping your agent
      scenarios/
        smoke_tests.yaml       # Fast, basic sanity checks
        tool_tests.yaml        # Tool usage verification
        regression.yaml        # Regression tests for known issues
      agenticassure.yaml       # Config file (optional)
  pyproject.toml

For smaller projects, a flat structure works fine:

my-project/
  my_agent.py
  scenarios/
    tests.yaml

The key requirement is that your adapter module must be importable from the directory where you run agenticassure. If your adapter is at tests/agent/adapters/my_adapter.py, you would either:

• Run agenticassure from the repository root with PYTHONPATH=. set, or
• Install your project in editable mode (pip install -e .) so the module is on the Python path.

Config File

Instead of passing --adapter on every CLI invocation, you can create a config file that sets the default adapter. AgenticAssure looks for a config file in the current working directory.

agenticassure.yaml

adapter: my_adapter.MyAgent

agenticassure.toml

adapter = "my_adapter.MyAgent"

AgenticAssure checks for these files in order:

1. agenticassure.yaml in the current working directory
2. agenticassure.toml in the current working directory

The first file found is used. The --adapter CLI flag always takes precedence over the config file.

With a config file in place, you can run tests without specifying the adapter:

# These are equivalent:
agenticassure run scenarios/ --adapter my_adapter.MyAgent
agenticassure run scenarios/   # Uses adapter from agenticassure.yaml

Config File Fields

Currently, the config file supports a single field:

The adapter path follows the format module.ClassName. For nested packages, use the full dotted path: my_package.adapters.openai_adapter.OpenAIAgent.

Environment Variables

AgenticAssure itself does not require any environment variables, but your adapter and certain optional features may need them.

Common Environment Variables

Using a .env File

For local development, it is common to store environment variables in a .env file at the project root:

# .env
OPENAI_API_KEY=sk-...
HF_TOKEN=hf_...

AgenticAssure does not load .env files automatically. You have several options:

Option 1: Load in your adapter. Use python-dotenv in your adapter module:

# my_adapter.py
from dotenv import load_dotenv
load_dotenv()
 
import os
from openai import OpenAI
 
class MyOpenAIAgent:
    def __init__(self):
        self.client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
 
    def run(self, input, context=None):
        # ...

Option 2: Export before running. Source the file in your shell:

export $(cat .env | xargs)
agenticassure run scenarios/

Option 3: Use direnv or similar tooling. Tools like direnv automatically load .env files when you enter the project directory.

Remember to add .env to your .gitignore to avoid committing secrets.

Working with Multiple Scenario Files

AgenticAssure can load scenarios from a single file or recursively from an entire directory tree. This lets you organize tests however makes sense for your project.

Running a Single File

agenticassure run scenarios/smoke_tests.yaml

Running a Directory

agenticassure run scenarios/

This recursively finds all .yaml and .yml files under scenarios/ and loads every suite.

Organizing by Concern

A common pattern is to split scenarios into files by area of concern:

scenarios/
  smoke/
    basic_io.yaml              # Basic input/output tests
    error_handling.yaml        # Edge cases and error inputs
  tools/
    search_tool.yaml           # Tests for search tool usage
    calculator_tool.yaml       # Tests for calculator tool usage
  regression/
    issue_42.yaml              # Regression test for a specific bug
    prompt_v2_migration.yaml   # Tests after a prompt change

Running agenticassure run scenarios/ loads and executes all of them. Each YAML file defines its own suite with its own name, so results are reported per-suite.

Filtering with Tags

Tags let you run a subset of scenarios across all files. Define tags in your YAML:

scenarios:
  - name: fast_check
    input: "Hi"
    tags:
      - smoke
      - fast
 
  - name: slow_integration
    input: "Analyze this dataset..."
    tags:
      - integration
      - slow

Then filter at runtime:

# Run only smoke tests
agenticassure run scenarios/ --tag smoke
 
# Run only tool-related tests
agenticassure run scenarios/ --tag tools

Multiple --tag flags are combined with OR logic: a scenario is included if it has any of the specified tags.

Filtering by Suite Name

If you want to run a single suite from a directory that contains multiple files:

agenticassure run scenarios/ --suite search-agent-tests

This loads all files but only executes the suite with the matching name.

Listing Scenarios

To see what scenarios are defined without running them:

# Rich table output
agenticassure list scenarios/
 
# Filter by tag
agenticassure list scenarios/ --tag smoke
 
# JSON output (useful for scripting)
agenticassure list scenarios/ --json-output

Validating Scenario Files

Before running tests (especially in CI), validate that all YAML files are structurally correct:

agenticassure validate scenarios/

This checks every .yaml and .yml file against the JSON Schema and reports any structural or semantic issues:

OK scenarios/smoke/basic_io.yaml
OK scenarios/tools/search_tool.yaml
FAIL scenarios/regression/broken.yaml
    Schema: scenarios.0: 'input' is a required property

1 file(s) failed validation

The validate command exits with code 1 if any file is invalid, making it suitable for CI checks.

PYTHONPATH and Module Resolution

The --adapter flag (and the adapter config field) takes a dotted Python import path like my_module.MyClass. For this to work, Python must be able to import my_module.

If your adapter is in the current directory, this works out of the box. For other layouts, set PYTHONPATH:

# Adapter is at src/adapters/my_adapter.py
PYTHONPATH=src agenticassure run scenarios/ --adapter adapters.my_adapter.MyAgent

Or, if your project uses a pyproject.toml with a proper package layout, install it in editable mode:

pip install -e .

This makes all your project’s modules importable regardless of the current directory.

CI/CD Integration

AgenticAssure is designed to work in automated pipelines. A minimal GitHub Actions example:

# .github/workflows/agent-tests.yml
name: Agent Tests
on: [push, pull_request]
 
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
 
      - name: Install dependencies
        run: |
          pip install agenticassure
          pip install -e .  # Install your project
 
      - name: Validate scenarios
        run: agenticassure validate scenarios/
 
      - name: Run agent tests
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: agenticassure run scenarios/ --output json
 
      - name: Upload results
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: agent-test-results
          path: results_*.json

Key points for CI:

• agenticassure run exits with code 1 on any failure, so the CI step will fail appropriately.
• Use --output json to produce machine-readable artifacts.
• Use agenticassure validate as an early check before running the full suite.
• Store API keys as CI secrets and pass them as environment variables.

Next Steps

• Quickstart — If you haven’t already, follow the end-to-end walkthrough.
• Scorers — Learn about the available scoring strategies.
• Adapters — Write adapters for different agent frameworks.
• CLI Reference — Full documentation of all CLI commands and flags.