Compaction Verifier

The ag2.extensions.compaction_verifier module measures whether a compaction strategy makes an agent's next actions worse. It answers one question about a CompactStrategy: when the agent's raw history is replaced by what the strategy keeps, do its next tool calls fail or repeat work more often?

A strategy that summarizes well and one that silently drops the session token look the same from the outside — both shrink the context. The difference only shows in what the agent does next, so that is what the verifier measures, from real execution.

This is the TRACE method (Trajectory-Relative Agent Context comprEssion), expressed in AG2's events.

Installation#

Note

The extension ships with AG2 and uses only the standard library. No additional package or API key is required beyond what your agent itself needs.

from ag2.extensions.compaction_verifier import (
    CompactionVerifier,
    IdentityCompact,
    Recording,
    ReplayEnvironment,
)

How it works#

The verifier takes runs you have already recorded and picks boundaries in them: points right after a tool result, where the agent is about to decide its next action. At each boundary it builds two contexts for the same execution state:

PRE — the raw history up to the boundary (the control);
POST — what the strategy returns for that history (the treatment).

For each context it resumes the agent with Agent.resume several times, in an environment rebuilt to the state at the boundary, and lets it run until horizon tool calls have completed or it finishes. Every completed call counts, including one that failed and one to a tool that does not exist, and the model answers at most horizon times, so a rollout always ends. A model call that raises is not counted, so a provider error that the agent's own middleware retries (RetryMiddleware, say) does not end a rollout early. Calls the model issues in one response all execute, so the last response can take a rollout past horizon; only the first horizon calls are scored. Nothing is teacher-forced: what the recorded run did next is never shown to the agent.

Each tool call is then read for two regressions:

	An action is…	Read from
blocked	a call whose execution failed	a `ToolErrorEvent` (or `ToolNotFoundEvent`)
refetch	a call repeating one already executed, before the boundary or earlier in the same rollout	the tool name plus its arguments as canonical JSON

An action that is either is wasted, and counts once. A strategy's score at a boundary is POST minus PRE: the mean wasted actions over the first k calls after compaction, minus the same without it. The agent is unstable on its own, so a rollout merely behaving differently is not evidence of harm; subtracting PRE differences that variance out rather than assuming it away. A positive delta means compaction added burden.

Burden only counts actions the agent took, so an agent that gives up wastes nothing. A separate signal covers that: a rollout is stopped when the agent ended its turn before k calls, and its delta is reported the same way. A strategy that drops the original task request can make the agent stop — harmless on burden, but it finished nothing — so always read stopped next to wasted.

Quick start#

import asyncio

from ag2 import Agent, MemoryStream, tool
from ag2.compact import SummarizeCompact, TailWindowCompact
from ag2.config import OpenAIConfig
from ag2.extensions.compaction_verifier import (
    CompactionVerifier,
    IdentityCompact,
    Recording,
    ReplayEnvironment,
)

config = OpenAIConfig(model="gpt-4.1-mini", temperature=0.7)

class Shop:
    """Deterministic state: the same calls always leave it the same way."""

    def __init__(self) -> None:
        self.pages = [["widget", "gadget"], ["gizmo", "sprocket"], ["flange", "bolt"], ["hinge", "valve"]]
        self.stock = {
            "widget": 3, "gadget": 0, "gizmo": 0, "sprocket": 5,
            "flange": 0, "bolt": 2, "hinge": 1, "valve": 0,
        }
        self.session = ""

    def login(self) -> str:
        """Start a session. Returns the session id every other tool needs."""
        self.session = "s-4821"
        return self.session

    def list_items(self, session: str, page: int) -> str:
        """List one page of items; pages start at 1."""
        self._authorize(session)
        return f"page {page} of {len(self.pages)}: {', '.join(self.pages[page - 1])}"

    def check(self, session: str, item: str) -> str:
        """Units of an item in stock."""
        self._authorize(session)
        return str(self.stock[item])

    def reorder(self, session: str, item: str, quantity: int) -> str:
        """Order more units of an item."""
        self._authorize(session)
        self.stock[item] += quantity
        return "ok"

    def _authorize(self, session: str) -> None:
        if not self.session or session != self.session:
            raise PermissionError("invalid session; call login first")

def fresh_tools() -> dict:
    shop = Shop()
    return {"login": shop.login, "list_items": shop.list_items, "check": shop.check, "reorder": shop.reorder}

async def main() -> None:
    # 1. Record a few runs, uncompacted.
    agent = Agent("stock-keeper", "Keep every item in stock.", config=config)
    task = "Check every item on every page and reorder what is out of stock."
    recordings = []
    for _ in range(3):
        stream = MemoryStream()
        tools = [tool(f, name=n) for n, f in fresh_tools().items()]
        await agent.ask(task, stream=stream, tools=tools)
        events = list(await stream.history.get_events())
        recordings.append(Recording(events, ReplayEnvironment(fresh_tools)))

    # 2. Verify strategies against them.
    verifier = CompactionVerifier(agent, horizon=5, samples=3)
    report = await verifier.verify(
        recordings,
        {
            "control": IdentityCompact(),
            "tail": TailWindowCompact(target=15),
            "summary": SummarizeCompact(target=15, config=config),
        },
        every=3,
        min_prefix=3,
    )
    print(report.summary())

asyncio.run(main())

With gpt-4.1-mini, each recorded run logs in, lists the four pages one at a time, and checks and restocks the items on each: 17 tool calls in 13 rounds. every=3, min_prefix=3 takes four boundaries from each run, 12 in all, and the script finishes in about three minutes. One run printed:

compaction verifier: POST minus PRE over the first 5 actions (3 samples per arm; * = 95% CI excludes 0)
PRE rollouts failed: 0 of 36 (shared by every strategy)
12 boundaries from 3 recordings; intervals and p treat boundaries as independent, but boundaries from one recording are correlated, so both are optimistic
strategy                  n                  wasted                 blocked                 refetch                 stopped   kept  noop     failed       p
control                  12  +0.000 [+0.000,+0.000]   +0.000 [+0.000,+0.000]   +0.000 [+0.000,+0.000]   +0.000 [+0.000,+0.000]    100%    12       0/36   1.000
tail                     12  +1.139 [+0.333,+2.056]*  +0.000 [+0.000,+0.000]   +1.139 [+0.333,+2.056]*  +0.028 [+0.000,+0.083]     31%     0       0/36   0.062
summary                  12  +0.000 [+0.000,+0.000]   +0.000 [+0.000,+0.000]   +0.000 [+0.000,+0.000]   +0.056 [+0.000,+0.167]     40%     0       0/36   1.000
n = boundaries scored / tested; stopped = share of rollouts that ended before the horizon; kept = median share of context tokens kept; noop = boundaries left unchanged; failed = POST rollouts that failed, not scored; p = exact sign-flip permutation test on per-boundary wasted delta

The control scored exactly zero on every channel. tail kept 31% of the context and cost about one repeated call in five: resumed from a window that no longer showed which pages were done, the agent started again from page 1. Its interval excludes zero, but the harm came from 5 of the 12 boundaries, and with five boundaries showing any difference the exact test cannot go below p = 0.06; record more runs before relying on it. summary kept 40% at no measurable cost. Your numbers will differ from run to run.

report.summary() prints one row per strategy: the boundaries scored out of those tested, the mean POST-minus-PRE delta at the full horizon for wasted, blocked and refetch actions and for stopped rollouts, each with a 95% bootstrap interval (* when it excludes zero), the median share of context tokens the strategy kept, how many boundaries it left unchanged, how many POST rollouts failed, and an exact sign-flip permutation p-value on the wasted delta. The p-value is computed from each arm's counts and number of rollouts, so it is exact whatever the arm sizes. It reads n/a, with a note under the table, only when an exact test would take too long: many boundaries whose arms kept different numbers of rollouts at a large samples. Everything else in the report is unaffected. A line above the table gives the PRE failures, which every strategy shares. With fewer than 10 boundaries no interval is marked: a percentile bootstrap over so few units is not reliable.

Build the agent without the environment's tools: each rollout passes tools bound to its own rebuilt environment.

Reading the report#

Always include IdentityCompact. It returns the history unchanged, so PRE and POST see the same context and its delta is the agent's own run-to-run noise at your sample size. A strategy's delta is evidence only when it clears the control's.
kept is how much of the context a strategy kept, and noop how many boundaries it left untouched. A strategy that keeps 100% of the context is not compacting at all, and its delta is noise. kept can also exceed 100%: a summary can come out longer than the history it replaced, and the summary then says at how many boundaries that happened. A TailWindowCompact window narrower than one tool round (five conversational events) keeps everything, because the window's start would split a call from its result.
TailWindowCompact drops the original request along with everything else outside the window. Between turns the next ask() brings a new request; resumed mid-task, an agent can lose track of what it was asked to do and stop, which shows up in stopped.
Check failed before reading a delta. A rollout that raises (a provider error, say) is recorded but not scored, and a boundary where every rollout of one arm failed is left out of the averages, which n shows as boundaries scored / tested. A ! marks a strategy whose arms failed at different rates, which biases its delta. More failures after compaction, for instance a provider rejecting the compacted context, leave only the POST samples that ran and can understate the harm. More failures without compaction, for instance the raw history overflowing the context window, can understate what compaction saves. The most common error is printed under the table, and every error is in to_dict().
Failures never stop a verification. A strategy that raises instead of returning a context (a summarization call that is rate-limited, say) is recorded for that boundary: no POST rollout runs there, the boundary is left out of that strategy's averages, and the summary marks the strategy with ! and prints the error. The other strategies and boundaries carry on, and so does everything already finished.
Intervals and p-values treat boundaries as independent. Each boundary is one unit in the bootstrap interval and in the sign and permutation tests. Boundaries cut from the same recording share one agent run and are correlated. So with several boundaries per recording (a small every on long runs), the intervals come out narrower and the p-values smaller than they should. The summary's header says how many recordings the boundaries come from. Spread boundaries over more recordings rather than packing many into a few, and treat a borderline result with caution.
Every horizon is available. report.strategies[name].at(k) gives the summary at k = 1 … horizon; report.to_dict() holds every boundary and every rollout's actions for your own analysis.
Sampling. Variance only averages out if the agent's model samples with a non-zero temperature; with temperature 0 use samples=1. Some models accept only their default temperature (OpenAI's gpt-5 family rejects temperature=0.7, for example). For those, leave temperature unset: their default of 1 is non-zero, which is what the verifier needs.

Recording a run#

A Recording holds the history the agent's own stream kept. Run the agent on a MemoryStream and pass what it holds, as the quick start does: await agent.ask(task, stream=stream, tools=tools), then list(await stream.history.get_events()). That history has the user request, every model response with the calls it issued, and the ToolResultsEvent that closes each round of calls, which is where a boundary can be cut and the agent resumed.

An ag2.eval Trace cannot be used. The traces run_agent and evaluate_traces hand to scorers are rebuilt from telemetry spans. They keep each tool call and its result (cut at 8,192 characters by default), but not the user request, the calls on each model response, or the ToolResultsEvent that closes each round, so there is no point the agent could resume from. Recording rejects a Trace with a TypeError. verify() rejects a history that has tool results but no ToolResultsEvent with a ValueError, so passing a trace's events fails with the reason instead of finding no boundary. To verify an agent you also evaluate, record its runs separately on a stream.

Environments#

Both arms of a boundary must start from the state the environment was in at that point, or a difference between them measures the environment, not the context. AG2 cannot snapshot what arbitrary tools did to the world, so each Recording carries an Environment: an object with one method, async def restore(prefix), that returns tools bound to a fresh environment into which the recorded calls prefix have been applied.

ReplayEnvironment(factory) does this by re-executing the recorded calls. factory returns a fresh {tool name: function} mapping on every call; each function is wrapped as a tool and each recorded call replayed through it, with the same argument validation and coercion the agent's executor applied. This is correct when the functions are deterministic given their arguments and their state. Replay checks that as far as it can: if a call that succeeded when recorded fails on replay, or the reverse, it raises ReplayMismatchError. The verifier records that as a failed rollout at that boundary, so no rollout starts from the wrong state, and the error appears in the summary. For tools with external side effects, implement Environment yourself, for example by restoring a snapshot.

Choosing boundaries#

By default a boundary is taken after the first min_prefix actions and then every every actions (verify(..., every=5, min_prefix=3)). Boundaries fall between tool rounds, and only where the recorded run went on to call another tool. Calls the model issues in one response are one round, so a run that makes all its calls at once has no boundary, whatever every and min_prefix are. When no recording has a boundary, verify() raises a ValueError that says how many actions and tool rounds each recording has. Pass Recording(events, environment, cuts=[...]) to choose them yourself; resumable_cuts(events) lists every valid point.

AG2's compaction middleware fires between turns, so a long single-turn tool loop is not compacted in production. The verifier does not depend on where a strategy happens to fire: it asks what the strategy would cost if the history were compacted at each boundary.

Cost#

Each boundary costs samples × (1 + number of strategies) rollouts of up to horizon answered model calls (plus any retries the agent's own middleware makes), plus one compaction per strategy (an LLM call for SummarizeCompact). PRE rollouts are shared by every strategy compared at a boundary. concurrency= bounds how many run at once.

Scoring recorded rollouts#

The arithmetic is available on its own for rollouts you collected elsewhere: actions_from_events reads executed calls off any event sequence, score_boundary(pre, post, history, horizons) returns the POST-minus-PRE deltas, and bootstrap_mean, exact_sign_test and exact_permutation_test aggregate them across boundaries. Pass exact_permutation_test each delta as a Fraction (BoundaryDelta.exact_wasted gives one) to keep it exact for any arm sizes; a float is accepted when it is a difference of means over arms of up to 1,000 rollouts. When no exact computation fits, it raises PermutationTestTooLargeError.