How to Test AI Agents: Unit, Integration and Chaos Testing

You've spent weeks building an autonomous agent that negotiates SaaS contracts via email. In your local notebook, it's brilliant. It drafts perfect counteroffers and knows when to walk away. But yesterday, in production, it approved a 40% discount because the underlying model updated its tokenization logic, confusing the percentage extraction tool.

This is the reality of building agentic systems in 2026. The stack has matured-context windows are massive, inference is cheap-but the engineering surface area has exploded. You aren't just testing code; you're testing probabilistic reasoning, external tool reliability, and state management across long-running loops.

If you treat AI agents like standard microservices, you will fail. If you treat them like pure prompts, you will burn cash. You need a hybrid testing strategy that respects the non-deterministic nature of LLMs while enforcing deterministic boundaries around your business logic. Here is the framework I use to ship reliable agents.

Unit Testing the Deterministic Boundaries

The biggest mistake I see teams make is trying to unit test the LLM itself. Don't do it. The model is a third-party service you don't control, and its output is inherently non-deterministic. Instead, unit test everything around the model.

Your agent is likely composed of tool definitions, prompt templates, and output parsers. These are deterministic code paths that deserve 100% coverage. Specifically, focus on input validation for your tools. If your agent calls a create_invoice tool, the schema validation must happen before the API call is ever made.

In my pipelines, I use Pydantic models to strictly define tool arguments. I then write unit tests that feed edge cases into these validators, ensuring the agent cannot construct invalid tool calls even if the model hallucinates.

# tools.py

from pydantic import BaseModel, Field, validator



class CreateInvoiceArgs(BaseModel):


    amount: float = Field(gt=0)


    currency: str


    client_id: str



    @validator('currency')


    def check_currency(cls, v):


        if v not in ['USD', 'EUR', 'GBP']:


            raise ValueError('Unsupported currency')


        return v



test_tools.py


import pytest


from tools import CreateInvoiceArgs



def test_invalid_currency_rejected():


    with pytest.raises(ValueError):


        CreateInvoiceArgs(amount=100.0, currency='BTC', client_id='c_123')



def test_negative_amount_rejected():


    with pytest.raises(ValueError):


        CreateInvoiceArgs(amount=-50.0, currency='USD', client_id='c_123')

Beyond schemas, unit test your prompt rendering logic. If you use f-strings or template engines to inject context, verify that special characters don't break the structure. I've seen agents fail because a user's name contained a curly brace that broke the Jinja2 template. Treat prompt templates as code, not text. Mock the data injection and assert the final string structure matches expectations before it ever hits the inference API.

Integration Testing the Agent Loop

Once your boundaries are secure, you need to test how the agent orchestrates them. This is where integration testing comes in. The goal here is to verify the flow: Does the agent recognize when to use a tool? Does it handle the tool's response correctly? Does it know when to stop?

To make these tests deterministic, you must mock the LLM. You cannot rely on model.generate() during CI/CD. Instead, record specific model responses that trigger specific branching logic in your agent's state machine.

If you are using a framework like LangGraph or AutoGen, you can inject a fake model provider that returns predefined messages. This allows you to simulate a multi-turn conversation without spending credits or waiting on latency.

# test_agent_flow.py

from unittest.mock import Mock


from agent_framework import Agent, State



def test_agent_refunds_after_verification():


    # Mock the LLM to return a tool call followed by a final answer


    mock_llm = Mock()


    mock_llm.generate.side_effect = [


        # Turn 1: Agent decides to check user tier


        {"tool_calls": [{"name": "get_user_tier", "args": {"user_id": "u_1"}}]},


        # Turn 2: Agent decides to process refund based on tool output


        {"content": "Refund processed successfully."}


    ]


    


    agent = Agent(llm=mock_llm)


    state = State(user_input="Please refund my last order", user_id="u_1")


    


    result = agent.run(state)


    


    # Assert the agent called the right tool


    assert mock_llm.generate.call_count == 2


    assert state.tools_called[0].name == "get_user_tier"


    assert result.final_response == "Refund processed successfully."

This approach tests the glue. It verifies that your agent parses the tool call, executes the function, feeds the result back into the context, and generates a final response. It also helps you catch infinite loops. In the test above, if the agent entered a loop trying to verify the user tier repeatedly, the call_count assertion would fail, or you would hit a hardcoded recursion limit in your test harness.

Integration tests should also cover the "happy path" and the "expected failure path." What happens when the tool returns a 404? Does the agent apologize and ask for clarification, or does it crash? Simulate tool errors in your mocks to ensure your error handling logic is robust.

Chaos Testing for Non-Determinism and Drift

Unit and integration tests cover your code, but they don't cover the model's behavior in the wild. Models drift. APIs latency spikes. Tokens get rate-limited. This is where chaos engineering for AI agents becomes critical.

Chaos testing for agents involves intentionally injecting noise into your system to observe how it degrades. In 2026, model providers are reliable, but they aren't perfect. You need to know what happens when the model returns malformed JSON, ignores your system prompt, or takes 30 seconds to respond.

I implement chaos decorators around my LLM clients. These decorators can randomly corrupt outputs, introduce latency, or raise exceptions during local staging runs.

# chaos_llm.py

import random


import time


from functools import wraps



def chaos_injection(probability=0.1):


    def decorator(func):


        @wraps(func)


        def wrapper(args, *kwargs):


            if random.random() < probability:


                # Scenario 1: Latency spike


                time.sleep(5) 


            if random.random() < probability:


                # Scenario 2: Malformed JSON response


                return {"content": "{ invalid_json: true }"}


            if random.random() < probability:


                # Scenario 3: Rate limit simulation


                raise Exception("429 Too Many Requests")


            return func(args, *kwargs)


        return wrapper


    return decorator



Apply to your production LLM client in staging


@chaos_injection(probability=0.2)


def generate_completion(prompt, **kwargs):


    return real_llm_client.generate(prompt, **kwargs)

Run your integration test suite against this chaotic client. Does your agent retry on 429 errors? Does it have a fallback mechanism when JSON parsing fails? Does the latency spike trigger a timeout in your upstream API gateway?

Another crucial aspect of chaos testing is semantic drift. When a provider releases a new model version (e.g., gpt-5.1 vs gpt-5.0), behavior can shift subtly. I maintain a "golden dataset" of 50-100 complex prompts with expected outcomes. Before promoting a new model version to production, I run this dataset through the new model. If the semantic similarity score drops below a threshold (e.g., 0.85 cosine similarity on embeddings), the deployment is blocked. This prevents silent regressions where the agent is still "working" but performing significantly worse.

Building the Evaluation Pipeline

Testing isn't a one-off task; it's a pipeline. In traditional DevOps, you have linting, unit tests, and deployment. For AI agents, you need an Evaluation Gate.

Your CI/CD pipeline should look like this:

  • Code Change: Trigger standard linting and unit tests (Pydantic schemas, tool logic).
  • Prompt/Model Change: Trigger evaluation against the golden dataset.
  • Integration Check: Run mocked agent flows to ensure orchestration logic holds.
  • Cost Check: Estimate the token usage of the new flow. If the average cost per task increases by >10%, flag for review.

    • I use tools like Arize Phoenix or LangSmith to track these evals, but you can start simple with a script that scores outputs. The key metric here is Pass@K. Run your agent on a test task K times (e.g., 5 times). If it succeeds 3 out of 5 times, is that acceptable? For a code generation agent, maybe. For a financial transaction agent, absolutely not.

    Define your success criteria per use case.

    Automate this. If a developer changes a system prompt to be "more friendly," but the evaluation pipeline shows a 15% drop in tool-calling accuracy, the merge request should fail automatically. This shifts the culture from "let's try it and see" to "let's verify it before shipping."

    Key Takeaways

    Published on agentic.dev.