> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mutagent.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Tracing Overview

> Understand how MutagenT captures and visualizes AI application behavior

# Tracing Overview

Tracing gives you full visibility into what your AI application is doing at every step. Each request flows through multiple stages -- LLM calls, tool invocations, retrieval lookups, orchestration logic -- and tracing captures all of them as a structured timeline you can inspect, debug, and optimize.

## Why Tracing Matters for AI Applications

Traditional application monitoring tracks HTTP requests and database queries. AI applications introduce new challenges:

* **Non-deterministic outputs** -- The same prompt can produce different results across runs
* **Multi-step orchestration** -- Agents chain multiple LLM calls, tool uses, and retrieval steps together
* **Cost and latency visibility** -- Token usage and model latency vary per request and need tracking
* **Debugging complex chains** -- When an agent fails on step 7 of 12, you need to see the full execution path

MutagenT tracing captures this entire execution graph as a **trace** -- a tree of **spans** that represent each operation in your AI pipeline.

## Core Concepts

### Traces

A **trace** represents a single end-to-end request through your application. It is identified by a unique `traceId` and contains one or more spans arranged in a parent-child hierarchy.

### Spans

A **span** represents a single operation within a trace. Each span captures:

* **Name** -- What this operation does (e.g., "generate-response", "search-docs")
* **Kind** -- The category of operation (see SpanKind below)
* **Input / Output** -- Structured data flowing through the operation
* **Metrics** -- Token counts, cost, latency, model information
* **Status** -- Whether the operation succeeded (`ok`), failed (`error`), or is in progress (`unset`)
* **Duration** -- How long the operation took
* **Attributes** -- Custom key-value metadata

### Parent-Child Relationships

Spans form a tree. When a span is created while another span is active, the new span automatically becomes a **child** of the active span. This gives you a hierarchical view of your application's execution:

<Mermaid>
  flowchart TD
  A\["agent\n(root span)"] --> B\["llm.chat\n(planning step)"]
  A --> C\["tool\n(web search)"]
  A --> D\["retrieval\n(vector lookup)"]
  D --> E\["llm.embedding\n(embed query)"]
  A --> F\["llm.chat\n(final response)"]

  style A fill:#7C3AED,color:#fff
  style B fill:#06B6D4,color:#fff
  style C fill:#334155,color:#fff
  style D fill:#334155,color:#fff
  style E fill:#06B6D4,color:#fff
  style F fill:#06B6D4,color:#fff
</Mermaid>

Context propagation is handled automatically using `AsyncLocalStorage` (TypeScript) or `contextvars` (Python), so child spans are linked to their parents without any manual wiring.

## SpanKind Taxonomy

Every span has a `kind` that classifies the type of operation it represents. MutagenT defines 15 span kinds, aligned with the [OpenTelemetry Gen AI semantic conventions](https://opentelemetry.io/docs/specs/semconv/) (v1.37+):

### Generation

| Kind             | Description          | Example Use                           |
| ---------------- | -------------------- | ------------------------------------- |
| `llm.chat`       | Chat completion call | OpenAI `chat.completions.create()`    |
| `llm.completion` | Text completion call | Legacy completion APIs                |
| `llm.embedding`  | Embedding generation | Creating vector embeddings for search |

### Orchestration

| Kind         | Description                    | Example Use                         |
| ------------ | ------------------------------ | ----------------------------------- |
| `chain`      | Sequential pipeline of steps   | LangChain chain, RAG pipeline       |
| `agent`      | Autonomous agent execution     | ReAct agent, tool-using agent       |
| `graph`      | Graph-based execution          | LangGraph, state machine workflows  |
| `node`       | Single node in a graph         | Individual graph node execution     |
| `edge`       | Transition between graph nodes | Conditional routing logic           |
| `workflow`   | Multi-step workflow            | Mastra workflow, orchestration flow |
| `middleware` | Request/response middleware    | Input validation, output formatting |

### Tools

| Kind   | Description              | Example Use                                 |
| ------ | ------------------------ | ------------------------------------------- |
| `tool` | External tool invocation | Function calling, API calls, code execution |

### Retrieval

| Kind        | Description             | Example Use                               |
| ----------- | ----------------------- | ----------------------------------------- |
| `retrieval` | Document/data retrieval | Vector store query, database lookup       |
| `rerank`    | Result re-ranking       | Cross-encoder reranking of search results |

### Safety

| Kind        | Description            | Example Use                       |
| ----------- | ---------------------- | --------------------------------- |
| `guardrail` | Safety or policy check | Content moderation, PII detection |

### Other

| Kind     | Description                     | Example Use           |
| -------- | ------------------------------- | --------------------- |
| `custom` | Any operation not covered above | Custom business logic |

## How Tracing Works

### Automatic Tracing with Integration Packages

If you use one of MutagenT's integration packages (`@mutagent/langchain`, `@mutagent/langgraph`, `@mutagent/openai`, `@mutagent/vercel-ai`, `@mutagent/mastra`), tracing happens **automatically**. The integration intercepts framework callbacks and creates properly structured spans without any manual instrumentation.

```typescript theme={null}
import { initTracing } from "@mutagent/sdk/tracing";
import { MutagentCallbackHandler } from "@mutagent/langchain";

initTracing({ apiKey: process.env.MUTAGENT_API_KEY! });

const handler = new MutagentCallbackHandler();

// Traces are created automatically for every chain/agent/LLM call
const result = await chain.invoke({ query: "..." }, { callbacks: [handler] });
```

<Tip>
  If `MUTAGENT_API_KEY` is set in your environment, you can skip the `initTracing()` call entirely. The SDK will lazily initialize tracing on the first `startSpan()` call. This is especially useful for quick scripts and development environments.
</Tip>

### Manual Tracing with the SDK

When you need fine-grained control, or when building custom orchestration logic, use the SDK's tracing API directly. MutagenT provides three levels of abstraction:

<AccordionGroup>
  <Accordion title="Decorators -- Class method instrumentation">
    Use the `@trace()` decorator to automatically wrap class methods with span creation, input/output capture, and error handling.

    ```typescript theme={null}
    import { trace } from "@mutagent/sdk/tracing";

    class MyAgent {
      @trace({ kind: "agent", name: "my-agent" })
      async run(query: string): Promise<string> {
        // Span is created automatically
        return await this.generate(query);
      }
    }
    ```
  </Accordion>

  <Accordion title="Wrappers -- Function-level instrumentation">
    Use `withTrace()` to wrap any function with a span and get a `SpanHandle` for manual enrichment.

    ```typescript theme={null}
    import { withTrace } from "@mutagent/sdk/tracing";

    const result = await withTrace(
      { kind: "chain", name: "rag-pipeline" },
      async (span) => {
        span.setMetrics({ model: "gpt-5.4", provider: "openai" });
        const answer = await generateAnswer(query);
        span.setOutput({ text: answer });
        return answer;
      }
    );
    ```
  </Accordion>

  <Accordion title="Low-level API -- Full manual control">
    Use `startSpan()` and `endSpan()` for maximum flexibility, typically in integration adapters.

    ```typescript theme={null}
    import { startSpan, endSpan, runInSpanContext } from "@mutagent/sdk/tracing";

    const span = startSpan({ kind: "tool", name: "web-search" });
    if (span) {
      const result = await runInSpanContext(span, async () => {
        return await searchWeb(query);
      });
      endSpan(span, { status: "ok", output: { raw: result } });
    }
    ```
  </Accordion>
</AccordionGroup>

## Architecture

The tracing pipeline follows a simple, non-blocking flow from your application to the MutagenT API:

<Mermaid>
  flowchart TD
  A\["Your Application"] --> B{"Integration Package\nor Manual SDK Calls"}
  B --> C\["SDK Tracing Module"]
  C --> D\["Span Lifecycle\n(AsyncLocalStorage context propagation)"]
  D --> E\["Batch Buffer\n(buffers spans, flushes every 5s or 10 spans)"]
  E --> F\["TraceHTTPClient\n(POST /api/traces with x-api-key)"]
  F --> G\["MutagenT API"]
  G --> H\["Persistent Store"]
  G --> I\["MutagenT Dashboard\n(trace visualization, analytics)"]

  style A fill:#06B6D4,color:#fff
  style C fill:#7C3AED,color:#fff
  style G fill:#06B6D4,color:#fff
  style H fill:#334155,color:#fff
</Mermaid>

The API also accepts traces from OpenTelemetry-compatible exporters via the OTLP bridge endpoint (`POST /api/traces/otlp`), which normalizes `ExportTraceServiceRequest` payloads into the MutagenT format before storing them.

Key design decisions:

* **Non-blocking** -- Span collection never blocks your application. Spans are buffered in memory and flushed asynchronously.
* **Batch transport** -- Spans are grouped by trace ID and sent in batches to minimize HTTP overhead. Default: flush every 5 seconds or when 10 spans accumulate.
* **Retry with backoff** -- Failed flushes are retried with exponential backoff (up to 3 retries, base delay 1s doubling each retry).
* **Graceful shutdown** -- `shutdownTracing()` flushes all remaining spans before the process exits.
* **Lazy initialization** -- If `MUTAGENT_API_KEY` is set in the environment, tracing initializes automatically on the first `startSpan()` call without requiring an explicit `initTracing()`.
* **Hybrid OTel strategy** -- MutagenT uses a lean native schema with `gen_ai.*` OTel semantic conventions (v1.37+) plus `mutagent.*` extensions, and provides an OTLP bridge for any OTel-compatible exporter.
* **Idempotent ingestion** -- Traces and spans use upsert semantics, so re-sending the same trace ID updates rather than duplicates.

## Claude Code Hook Trace Shape

When tracing Claude Code sessions via `mutagent hooks install`, the hook events map directly to spans in the trace tree. Each Claude Code session produces a structured trace:

<Mermaid>
  flowchart TD
  A\["SessionStart\nroot agent span"] --> B\["UserPromptSubmit\nturn chain span"]
  B --> C\["PreToolUse\ntool span"]
  C --> D1\["PostToolUse → status: ok"]
  C --> D2\["PostToolUseFailure → status: error"]
  B --> E\["SubagentStart\nchild agent span"]
  E --> F\["SubagentStop\nchild agent span closed"]
  B --> G\["PreCompact\ncustom event"]
  B --> H\["PostCompact\ncustom event"]
  A --> I\["Stop\ngraceful stop event"]
  A --> J\["SessionEnd\nroot span closed"]

  style A fill:#7C3AED,color:#fff
  style B fill:#06B6D4,color:#fff
  style C fill:#334155,color:#fff
  style E fill:#7C3AED,color:#fff
  style D2 fill:#dc2626,color:#fff
</Mermaid>

Each turn (user prompt → assistant response) produces a `chain` span. Tool calls nest under the turn span as `tool` spans. Sub-agents spawn their own `agent` spans as children. Compaction events appear as `custom` events on the active span rather than creating new spans.

See the [CLI Hooks reference](/cli/commands#hooks) for the full event taxonomy and configuration.

## When to Use What

| Scenario                                                 | Approach                                                          |
| -------------------------------------------------------- | ----------------------------------------------------------------- |
| Using LangChain, LangGraph, Vercel AI, OpenAI, or Mastra | Use the integration package -- tracing is automatic               |
| Custom orchestration logic                               | Use `withTrace()` for function-level spans                        |
| Class-based agent architecture                           | Use `@trace()` decorator on methods                               |
| Building a framework integration                         | Use the low-level `startSpan()` / `endSpan()` API                 |
| Simple scripts or one-off calls                          | Use `withTrace()` for quick instrumentation                       |
| OTel-compatible exporter already in use                  | Send traces to the OTLP bridge endpoint (`POST /api/traces/otlp`) |

## Next Steps

<CardGroup cols={2}>
  <Card title="Tracing Setup" href="/tracing/setup">
    Configure tracing in your application with initTracing() and environment variables.
  </Card>

  <Card title="Tracing API Reference" href="/tracing/api">
    Full reference for decorators, wrappers, and low-level span primitives.
  </Card>
</CardGroup>