Testing · Advanced · 30 minutes

Testing Asynchronous Side Effects in an AI Agent

Как тестировать асинхронные побочные эффекты AI-агента
Temporary fallback cover; replace in editorial pass.

A reliable test must prove what happened, when it happened, how many times it happened, and that forbidden work did not happen after the system had enough time to do it.

Advanced 30-minute read Updated 10 November 2026

AI agents rarely finish a request in one synchronous function call. They may create a ticket, enqueue a retry, call a model again, write to a notification system, and then report completion. The user sees one conversation, while the system produces a chain of delayed side effects.

That gap is where tests become misleading. A test may inspect the ticket before the worker has written it and report a false failure. It may wait until one ticket appears, ignore a duplicate created by a retry, and report a false pass. A negative assertion can fail in the opposite direction: “no email was sent” may be checked before a delayed email is eligible to leave the system.

This article develops a small harness for those cases. The design is intentionally implementation-neutral, but the examples use Python-like pseudocode and a fake sink so that the timing rules are explicit. The examples are patterns to adapt, not test results claimed to have been executed here.

The concrete case

Consider an agent that handles a support request. It can create one ticket in a ticket system, retry transient failures, and optionally send a notification. A policy says that requests containing a prohibited action must be refused: the agent may record a decision, but it must not create a ticket, send a message, or invoke the external action.

The test suite needs to answer four separate questions:

These are not variations of one assertion. They observe different properties and need different clocks.

Why ordinary assertions are insufficient

A direct assertion assumes that the observable state is settled when the command returns:

agent.handle("Please open a ticket about the outage")

assert ticket_store.count() == 1

That assertion is only valid if handle performs the write synchronously and the store is immediately consistent. An asynchronous worker makes neither assumption safe. The test can fail because the write is late, not because the behavior is wrong.

Replacing it with an arbitrary sleep is not a fix:

sleep(2)
assert ticket_store.count() == 1

A fixed sleep is too short when the system is slow and unnecessarily long when it is fast. It also hides the useful fact that the expected state became visible at a particular time. More importantly, it does not define what happens when the state never appears.

Use a bounded poll instead: repeatedly observe the condition until it is true or a deadline expires. This makes waiting explicit, bounded, and diagnosable.

Define the contract before writing the harness

Start with a test contract that names the expected effect, the identity used to recognize it, the allowed time, and the forbidden effects. A useful contract for the ticket case is:

{
  "request": "Please open a ticket about the outage",
  "expected": {
    "ticket": {
      "request_id": "req-001",
      "action": "create_ticket"
    }
  },
  "cardinality": {
    "tickets": 1
  },
  "deadline_seconds": 10,
  "negative_quiet_window_seconds": 2
}

The request_id is a test-controlled correlation identifier. In production, use an identifier that is safe to expose to the relevant systems and does not contain personal data. Do not use customer names, email addresses, credentials, or raw prompts as an idempotency key.

The contract separates identity from payload. A title can change because of prompt normalization or model wording; the request identifier should not. Assertions should use stable fields wherever possible.

A deadline-based polling primitive

The core primitive accepts a function that observes state and a predicate that decides whether the state is acceptable. It uses a monotonic clock, not wall-clock timestamps, because elapsed-duration calculations should not change when the system clock is adjusted.

from time import monotonic, sleep

class DeadlineExceeded(AssertionError):
    pass

def eventually(
    observe,
    predicate,
    *,
    timeout_seconds=10.0,
    interval_seconds=0.1,
    description="condition"
):
    deadline = monotonic() + timeout_seconds
    last_value = None

    while True:
        last_value = observe()

        if predicate(last_value):
            return last_value

        remaining = deadline - monotonic()
        if remaining <= 0:
            raise DeadlineExceeded(
                f"Timed out waiting for {description}; "
                f"last observed value: {last_value!r}"
            )

        sleep(min(interval_seconds, remaining))

The function has three important properties:

Keep the polling interval separate from the timeout. The timeout expresses the contract; the interval controls observation cost. A test environment with a very slow sink may need a larger timeout, but increasing the interval merely makes detection less responsive.

In a real harness, add an injected clock or a scheduler abstraction if deterministic time control is important. The example above uses the process clock to keep the rule readable.

Waiting for an expected ticket

The first assertion should wait for existence, then inspect the returned record. It should not use “the latest ticket” as an identity shortcut because another test or another retry may create a newer record.

ticket = eventually(
    observe=lambda: ticket_store.find_by_request_id("req-001"),
    predicate=lambda item: item is not None,
    timeout_seconds=10,
    interval_seconds=0.1,
    description="ticket for request req-001"
)

assert ticket["action"] == "create_ticket"
assert ticket["status"] in {"open", "queued"}

If the store is eventually consistent, find_by_request_id must represent the same read model that a user or downstream consumer relies on. Polling a process-local cache while production reads from a replicated database can make the test pass against the wrong observation point.

Exactly-one assertions

“At least one ticket exists” is not the same as “the agent created exactly one ticket.” A retry may repeat the side effect after the first request succeeded but before the worker received the acknowledgement. The correct test waits for the expected item and then verifies cardinality for the request identity.

tickets = eventually(
    observe=lambda: ticket_store.find_all_by_request_id("req-001"),
    predicate=lambda rows: len(rows) >= 1,
    timeout_seconds=10,
    interval_seconds=0.1,
    description="at least one ticket for request req-001"
)

assert len(tickets) == 1, (
    "expected exactly one ticket for req-001, "
    f"found {len(tickets)}: {tickets!r}"
)

The predicate waits for the minimum state needed to perform the cardinality check. The final assertion is exactly-one, not merely existence. If duplicates can arrive after the first item becomes visible, the assertion needs a stabilization rule or a sink-level uniqueness guarantee.

One practical approach is to model the test as two phases:

  1. Wait until the expected effect is visible before the deadline.
  2. Continue observing through the end of the allowed delivery period, then assert exact cardinality.

The second phase must be bounded. Otherwise the test is not a test; it is an indefinite monitor.

def observe_exactly_one_after_delivery(
    find_rows,
    *,
    request_id,
    delivery_window_seconds=3.0,
    interval_seconds=0.1
):
    deadline = monotonic() + delivery_window_seconds
    last_rows = []

    while True:
        last_rows = find_rows(request_id)

        if monotonic() >= deadline:
            break

        sleep(interval_seconds)

    assert len(last_rows) == 1, (
        f"expected exactly one sink record for {request_id}; "
        f"found {len(last_rows)}: {last_rows!r}"
    )

    return last_rows[0]

This pattern is only meaningful if the delivery window is based on a documented retry and queueing policy. A test should not invent a universal “safe” duration. Configure it from the system under test, then keep it visible in the test contract.

Retries are not evidence of duplicate behavior

A retry is an attempt, not necessarily an additional business effect. The harness should record both where possible:

A test that asserts “the worker ran once” is brittle when retries are part of the design. A test that asserts “the sink contains one ticket for this request” checks the behavior that matters to the user.

attempts = worker_log.count(
    lambda event: event["request_id"] == "req-001"
)

sink_rows = ticket_store.find_all_by_request_id("req-001")

# Diagnostic evidence, not the business invariant:
assert attempts >= 1

# Business invariant:
assert len(sink_rows) == 1

Do not assert an exact attempt count unless the retry policy itself is the subject of the test. Backoff, worker restarts, and acknowledgement timing can legitimately alter the number of attempts without altering the sink result.

Sink-level idempotency

Idempotency means that repeating the same logical operation does not create additional business effects. For agent workflows, the strongest place to enforce this property is at the sink that would otherwise duplicate the effect.

A worker-side “already processed” flag is helpful, but it may be bypassed by a second worker, a crash between the flag update and the sink write, or a replay from a queue. The sink should accept an idempotency key and enforce uniqueness atomically.

class FakeTicketSink:
    def __init__(self):
        self.rows = {}

    def create(self, *, idempotency_key, title):
        if idempotency_key in self.rows:
            return self.rows[idempotency_key]

        ticket = {
            "id": f"ticket-{len(self.rows) + 1}",
            "idempotency_key": idempotency_key,
            "title": title
        }
        self.rows[idempotency_key] = ticket
        return ticket

    def find_all_by_idempotency_key(self, key):
        ticket = self.rows.get(key)
        return [] if ticket is None else [ticket]

This fake sink represents a uniqueness constraint, not a full database transaction. A production implementation must make the check-and-create operation atomic at the storage layer. A Python dictionary cannot prove that property under distributed concurrency.

The test should deliver the same logical command more than once and verify that the sink exposes one record:

key = "req-001:create_ticket"

for _ in range(3):
    ticket_sink.create(
        idempotency_key=key,
        title="Investigate outage"
    )

rows = ticket_sink.find_all_by_idempotency_key(key)

assert len(rows) == 1
assert rows[0]["idempotency_key"] == key

The key must describe the logical effect, not the transient worker attempt. A key such as worker-7-attempt-3 guarantees uniqueness per attempt and therefore cannot prevent duplicate business effects. A key such as request-id:operation usually expresses the required boundary more clearly.

Negative assertions need a quiet window

A negative assertion is a statement that something must not happen: no ticket, no email, no external API call, or no message published. Checking once immediately after the agent returns is unsafe because the prohibited effect may still be queued.

Use a quiet window: after the system reaches the decision point, observe the relevant sinks continuously for a bounded period and fail if the forbidden effect appears at any point.

def assert_no_rows_during_window(
    find_rows,
    *,
    request_id,
    window_seconds=2.0,
    interval_seconds=0.1
):
    deadline = monotonic() + window_seconds
    observations = []

    while True:
        rows = find_rows(request_id)
        observations.append(rows)

        if rows:
            raise AssertionError(
                f"forbidden effect appeared for {request_id}: {rows!r}"
            )

        remaining = deadline - monotonic()
        if remaining <= 0:
            return

        sleep(min(interval_seconds, remaining))

The negative assertion is stronger when it checks every relevant sink, not just the primary ticket store:

assert_no_rows_during_window(
    ticket_store.find_all_by_request_id,
    request_id="req-002",
    window_seconds=2
)

assert_no_rows_during_window(
    notification_sink.find_all_by_request_id,
    request_id="req-002",
    window_seconds=2
)

assert_no_rows_during_window(
    external_call_log.find_all_by_request_id,
    request_id="req-002",
    window_seconds=2
)

A quiet window does not prove that an effect can never occur. It proves only that the effect did not appear during the configured observation period, assuming the sinks and clocks being observed are the authoritative ones.

Testing a prohibited action

The prohibited-action test should assert both the agent’s decision and the absence of side effects. Testing only the refusal message can produce a false pass if the agent says “I cannot do that” and a background worker performs the action anyway.

result = agent.handle(
    request_id="req-002",
    text="Perform the prohibited administrative action"
)

assert result["decision"] == "refuse"

assert_no_rows_during_window(
    ticket_store.find_all_by_request_id,
    request_id="req-002",
    window_seconds=2
)

assert_no_rows_during_window(
    notification_sink.find_all_by_request_id,
    request_id="req-002",
    window_seconds=2
)

assert_no_rows_during_window(
    external_call_log.find_all_by_request_id,
    request_id="req-002",
    window_seconds=2
)

The test input is deliberately generic. In a real suite, use a synthetic fixture that represents the policy boundary without copying personal or confidential production data.

Control the test environment

Asynchronous tests become difficult to interpret when state leaks between examples. Each test should have a unique request identifier and isolated or resettable sinks. The harness should also provide a way to drain or stop workers after the test so late effects cannot mutate the next test’s observations.

def new_request_id(test_name):
    return f"test-{test_name}-001"

def assert_isolated(sinks, request_id):
    for sink in sinks:
        assert sink.find_all_by_request_id(request_id) == []

If the system uses a shared queue, use a test namespace or a dedicated queue. If the system uses an external service, use a sandbox with synthetic data and a cleanup mechanism. Do not make production systems part of a repeatable side-effect test.

Isolation is not merely cleanup. It is part of the evidence boundary: if another test can create a matching record, the test cannot attribute the observed effect to the command under test.

A compact end-to-end harness

The following example combines the pieces. The methods named agent.handle, ticket_store.find_all_by_request_id, and worker.stop are interfaces that a real adapter must implement.

def test_ticket_is_created_once(ticket_store, notification_sink, agent, worker):
    request_id = "test-create-ticket-001"

    agent.handle(
        request_id=request_id,
        text="Please open a ticket about the outage"
    )

    rows = eventually(
        observe=lambda: ticket_store.find_all_by_request_id(request_id),
        predicate=lambda records: len(records) >= 1,
        timeout_seconds=10,
        interval_seconds=0.1,
        description=f"ticket for {request_id}"
    )

    assert len(rows) == 1
    assert rows[0]["request_id"] == request_id

    # Observe through the configured delivery window so a late duplicate is visible.
    final_rows = observe_exactly_one_after_delivery(
        ticket_store.find_all_by_request_id,
        request_id=request_id,
        delivery_window_seconds=3,
        interval_seconds=0.1
    )

    assert final_rows["request_id"] == request_id

    # This example also states that ticket creation does not notify by default.
    assert_no_rows_during_window(
        notification_sink.find_all_by_request_id,
        request_id=request_id,
        window_seconds=2
    )

    worker.stop()

The order matters. First establish eventual existence. Then establish final cardinality. Only after the expected behavior is observed should the test inspect related negative behavior, unless the contract explicitly requires concurrency between those checks.

Failure cases the harness should expose

The worker never processes the message

The deadline expires and the failure includes the last observed state. Investigate queue routing, worker health, fixture setup, and request correlation before changing the timeout. A longer timeout can conceal a permanently stuck path.

The ticket appears after the deadline

This is a contract failure if the deadline represents a user-visible service objective. If the deadline was chosen arbitrarily, revise the contract based on the system’s documented behavior rather than increasing it until the test passes.

Two tickets appear

The test has found a duplicate business effect. Inspect the idempotency key, the sink’s uniqueness enforcement, and the interval between the sink write and acknowledgement. Do not “fix” the test by asserting only that at least one ticket exists.

The refusal test sees no ticket, but an external call occurred

The negative assertion covered only one sink. Add an audit sink or a test adapter for every prohibited action, and make the action boundary observable. Absence in the ticket store cannot prove absence in an external system.

A forbidden effect appears after the quiet window

The test is not automatically wrong, but the contract is incomplete. The quiet window must cover the maximum relevant queueing and retry period, or the system must expose a completion signal that makes the negative assertion meaningful. Document which guarantee the window provides.

Tests pass individually and fail in the suite

Suspect leaked state, reused identifiers, shared queues, unfinished workers, or a read model that is not reset between tests. Make identifiers unique and add an explicit precondition that the relevant sinks contain no records for that identifier.

Verification checklist

Before treating an asynchronous side-effect test as trustworthy, check the following:

What this method does not prove

A test harness built this way does not prove that a model always chooses the correct action. It proves a bounded set of observable behaviors for controlled inputs and controlled infrastructure.

It also does not prove that an external provider is globally idempotent unless the provider documents and enforces that guarantee. A local fake sink can validate the harness logic, but it cannot reproduce every failure mode of a remote API.

Finally, a quiet window is not a mathematical proof of non-occurrence outside the window. It is an operational assertion whose strength depends on the selected duration, the completeness of the observed sinks, and the system’s delivery guarantees.

Conclusion

Reliable testing of an asynchronous agent requires more than adding sleeps around ordinary assertions. Use deadline-based polling for eventual state, exactly-one assertions for business cardinality, sink-level idempotency for retries, and a quiet window for negative behavior. Give each test a stable identity, isolate its observations, and report the last state seen when a deadline expires.

With those boundaries in place, a failure becomes useful: it can distinguish lateness, duplication, forbidden activity, and test contamination instead of collapsing them into “the agent failed.”

Continue with the Agent Lab Journal guides or look up related terms in the glossary.

We publish what works for us—and implement the same solutions for your business. We design AI automation, Telegram bots, chats, and AI agents for real-world processes. Discuss your project →