Network

Where Agent Telemetry traces what happens inside one agent, network telemetry traces what happens between them — the hub's envelope dispatch, channel lifecycles, agent registration, and tasks. It is emitted by the hub itself, as OpenTelemetry spans on the same TracerProvider your agents use, so the two stitch together into one picture.

Like agent telemetry it is opt-in: nothing is traced unless you hand the hub a TracerProvider.

Installation#

pip install "ag2[openai,tracing]"

Quick Start#

Two opt-ins wire up the full picture: pass a tracer_provider to Hub.open (the hub owns envelope spans), and register a HubTelemetryListener (it owns channel, agent, and task spans). Give each agent a TelemetryMiddleware so its turn spans join the same provider.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57

import asyncio

from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor, ConsoleSpanExporter

from ag2 import Agent
from ag2.config import OpenAIConfig
from ag2.knowledge import DiskKnowledgeStore
from ag2.middleware.builtin import TelemetryMiddleware
from ag2.network import Hub, HubClient, LocalLink, Passport, Resume
from ag2.network.hub import HubTelemetryListener

async def main() -> None:
    # 1. One TracerProvider, shared by the hub and every agent. The resource
    #    attributes identify this hub instance across all of its traces.
    resource = Resource.create({
        "service.name": "ag2-network",
        "service.instance.id": "hub-001",
        "ag2.hub.id": "hub-001",
    })
    provider = TracerProvider(resource=resource)
    provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))
    trace.set_tracer_provider(provider)

    store = DiskKnowledgeStore("./hub_store")

    # 2. The hub owns `network.envelope` spans — pass it the provider.
    hub = await Hub.open(store, tracer_provider=provider)

    # 3. The listener owns `network.channel`, `agent.lifetime`, and
    #    `network.task` spans (plus channel-attached signal events).
    hub.register_listener(HubTelemetryListener(store, tracer_provider=provider))

    link = LocalLink(hub)

    # 4. Each agent's TelemetryMiddleware shares the provider, so its turn
    #    spans link back to the envelope that triggered them.
    alice = Agent(
        "alice",
        config=OpenAIConfig(model="gpt-4o-mini"),
        middleware=[TelemetryMiddleware(tracer_provider=provider, agent_name="alice")],
    )

    hc = HubClient(link, hub=hub)
    await hc.register(alice, Passport(name="alice"), Resume())

    # ... open channels and run as usual; spans are emitted automatically ...

    # 5. shutdown() unregisters agents so long-lived `agent.lifetime` spans
    #    end (and export). See "Getting a complete trace set" below.
    await hc.shutdown()
    await hub.close()
    provider.shutdown()

asyncio.run(main())

What Gets Traced#

The hub and listener emit five span types. Each carries an ag2.span.type attribute, mirroring the agent-side convention.

Agents instrumented with TelemetryMiddleware add their own agent / llm / tool / human_input spans (see Agent Telemetry), linked to the triggering envelope.

Checkpoint events#

Task checkpoints are written straight to the hub's store, so they bypass the envelope path and are not their own spans. Instead, HubBackedCheckpointStore pins them as span-events on whatever span is active when a checkpoint is saved or loaded — typically the owning agent's turn span:

These are no-ops when no tracer is configured, so checkpointing carries no tracing cost unless you opt in.

Bounded Traces, Joined by Links#

Rather than forcing everything onto one enormous hub-wide trace, each entity is its own bounded trace — an envelope, a channel, an agent's lifetime, and a task each root a separate trace. This matches how OTLP collectors and trace viewers expect to ingest data (a long-lived channel that stays open for hours does not hold one trace open indefinitely).

The causal relationships between these traces are expressed two ways:

• SpanLinks connect related spans across traces:

  Link	From → To	Meaning
  in_channel	envelope → channel	the envelope was dispatched in this channel
  triggered_by	envelope → caller's span	the envelope was posted by this work
  in_channel	task → channel	the task ran in this channel

• Resource attributes (service.instance.id, ag2.hub.id) stamp every span the hub emits, so all of a hub's traces are joinable in your backend without a shared parent span.

Connecting agent turns to envelopes#

You do not wire this up yourself — adding TelemetryMiddleware to your agents is enough. Under the hood, the hub injects the network.envelope span's W3C traceparent into the envelope before it is written to the WAL (so the write-ahead log is the source of truth), and the receiving agent's middleware extracts it to parent the invoke_agent span. The result: a continuous trace from dispatch into the recipient's work.

What a Single Exchange Produces#

To make the model concrete, here is the exact output of one consulting exchange — a 1-question / 1-reply channel where alice asks bob, both agents instrumented with TelemetryMiddleware. It yields five bounded traces (indentation = parent → child within one trace):

network.channel consulting                      # the channel: open → auto-close
                                                 #   ← linked from each envelope below

network.envelope ag2.msg.text   (alice → bob)   # dispatch of alice's question
└─ invoke_agent bob                              #   bob's turn, triggered by the question
   └─ chat                                       #   bob's LLM call
                                                 #   → in_channel link to network.channel

network.envelope ag2.msg.text   (bob → alice)   # dispatch of bob's reply
                                                 #   → in_channel link to network.channel

agent.lifetime alice                            # alice: registered → unregistered
agent.lifetime bob                              # bob:   registered → unregistered

What to read off this:

• Each entity is its own trace. The channel, each envelope, and each agent's lifetime are separate bounded traces — not one giant tree — joined by SpanLinks and the shared ag2.hub.id resource attribute.
• The recipient's work nests under the triggering envelope. invoke_agent bob (and its chat, plus any execute_tool) are children of the question envelope, because the traceparent rides that envelope into bob's turn. One trace follows the message from dispatch into the recipient's LLM call.
• The reply is its own envelope trace (bob → alice) — the hub starts a fresh network.envelope root for every dispatched message.
• Lifetime spans appear only once the agent is unregistered — they stay open from registered until shutdown() (see Getting a complete trace set).

It scales predictably: N messages on a channel → N network.envelope traces; each agent turn adds an invoke_agent (with its chat / execute_tool children) under the envelope that triggered it; one network.channel per channel; one agent.lifetime per registered agent.

Two Sinks: Collector and Disk#

Every span the hub and listener emit goes to two places:

1. The OTLP collector — via the TracerProvider you configured, exactly like agent telemetry. Any OTLP-compatible backend works.
2. Disk — each ended span is also mirrored as one JSON object per line to /telemetry/spans.jsonl inside the hub's KnowledgeStore. This is hub-private storage (agents cannot read it), it survives restarts, and it can be queried offline without a running collector.

The disk records use the same hex trace/span IDs as the OTLP export, so a span on disk and the same span in your backend cross-reference one another.

Span Attributes#

Beyond the resource attributes, spans carry ag2.network.* and ag2.agent.* attributes you can filter and aggregate on:

Channel spans also record diagnostic span events — dispatch_failed (carrying ag2.network.recipient_id + ag2.error.type / ag2.error.message), turn_failed, envelope_rejected, and expectation violations — and agent.lifetime spans record inbox_pressure (ag2.inbox.pending / ag2.inbox.cap), so failures and back-pressure surface in the trace without their own spans.

The network.envelope span reports ERROR when delivery fails: per-recipient delivery failures set ag2.network.dispatch_failures, and an exception that escapes dispatch (for example, a custom HubArbiter error) is recorded on the span as an exception event.

Owner code that checkpoints a task adds checkpoint.write / checkpoint.read span events on the active span — see Checkpoint events above.

Custom Span Attributes#

HubTelemetryListener accepts a span_attributes dict, stamped onto every channel / agent / task span it emits — the same pattern as the agent middleware, useful for tenant or environment labels.

1
2
3
4
5
6
7

hub.register_listener(
    HubTelemetryListener(
        store,
        tracer_provider=provider,
        span_attributes={"deployment": "production", "ag2.org.id": "org-abc123"},
    )
)

Getting a Complete Trace Set#

Long-lived spans only export when they end. agent.lifetime spans end when an agent is unregistered, and network.channel spans end when a channel closes.

• HubClient.shutdown() unregisters the client's agents — ending their agent.lifetime spans.
• HubClient.close() only drops the connection; registrations (and their lifetime spans) stay open.

So for a clean, complete trace set at the end of a run, call shutdown() (not just close()) on your hub clients, then provider.shutdown() to flush the exporter.

Configuration#

HubTelemetryListener accepts:

To enable hub-owned network.envelope spans, pass the same tracer_provider to Hub.open(store, tracer_provider=...).

Backend Integration#

Network telemetry emits standard OpenTelemetry spans, so any OTLP-compatible backend works — Grafana Tempo, Jaeger, Datadog, Honeycomb, Langfuse, and others. Swap the ConsoleSpanExporter for an OTLPSpanExporter pointed at your backend or a local OpenTelemetry Collector; the hub and agents share the one provider, so all their spans flow to the same place.