> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mutagent.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Tracing

> SDK tracing module for custom instrumentation

# Tracing

The `@mutagent/sdk/tracing` module provides custom instrumentation primitives for tracing LLM calls, agent workflows, tool usage, and more. Use it when the [Integration packages](/integrations/overview) do not cover your framework, or when you need fine-grained control over span creation.

<Note>
  For concepts, span kinds, and the overall tracing architecture, see the [Tracing Overview](/tracing/overview). For the full server-side API reference, see the [Tracing API](/tracing/api).
</Note>

## Installation

The tracing module is included in `@mutagent/sdk`. No additional packages are required.

```bash theme={null}
npm install @mutagent/sdk
```

## Quick Start

```typescript theme={null}
import { initTracing, shutdownTracing } from '@mutagent/sdk/tracing';

// Initialize once at application startup
initTracing({
  apiKey: process.env.MUTAGENT_API_KEY!,
  environment: 'production',
  debug: false,
});

// ... your application code using @trace, withTrace, or startSpan ...

// Shutdown gracefully (flushes remaining spans)
await shutdownTracing();
```

### Lazy Initialization

If you do not call `initTracing()` explicitly, the module automatically initializes from the `MUTAGENT_API_KEY` environment variable on the first `startSpan()` call. This is a one-time attempt -- if the variable is not set, tracing remains disabled for the rest of the process lifecycle.

```typescript theme={null}
// No initTracing() call needed if MUTAGENT_API_KEY is set
import { startSpan, endSpan } from '@mutagent/sdk/tracing';

// First call triggers lazy init from env var
const span = startSpan({ kind: 'llm.chat', name: 'openai-chat' });
```

## Architecture

<Mermaid>
  flowchart TD
  A\["@trace() / withTrace() / startSpan()"] --> B\[Span Lifecycle]
  B --> C\[AsyncLocalStorage Context]
  B --> D\[Batch Buffer]
  D -->|"Every 5s or 10 spans"| E\[TraceHTTPClient]
  E -->|"POST /api/traces"| F\[MutagenT API]
  D -->|"Retry with backoff"| E
</Mermaid>

The tracing module uses a singleton architecture:

* **Span lifecycle manager** -- Creates and manages spans with `AsyncLocalStorage` for parent-child propagation
* **Batch buffer** -- Buffers finished spans in memory and flushes when either the batch size (default 10) or the flush interval (default 5s) is reached. Retries with exponential backoff on failure (up to 3 attempts).
* **TraceHTTPClient** -- Sends grouped span payloads to `POST /api/traces`

## Instrumentation Approaches

The SDK provides three levels of instrumentation, from highest to lowest abstraction.

### 1. `withTrace()` Wrapper

The primary instrumentation API. Wraps any function with a tracing span and provides a `SpanHandle` for manual enrichment during execution.

```typescript theme={null}
import { withTrace } from '@mutagent/sdk/tracing';

const result = await withTrace(
  { kind: 'chain', name: 'rag-pipeline' },
  async (span) => {
    span.setAttributes({ 'rag.corpus': 'knowledge-base-v2' });

    const docs = await retrieve(query);
    span.addEvent('retrieval-complete', { documentCount: docs.length });

    const answer = await generate(query, docs);
    span.setOutput({ text: answer });
    span.setMetrics({
      model: 'gpt-5.4',
      provider: 'openai',
      inputTokens: 150,
      outputTokens: 300,
      totalTokens: 450,
    });

    return answer;
  }
);
```

When tracing is not initialized, `withTrace` runs the function directly with a no-op `SpanHandle` -- your code executes normally without any overhead.

#### Wrapper Options

```typescript theme={null}
interface WithTraceOptions {
  kind: SpanKind;                        // Required: span kind
  name?: string;                         // Span name
  input?: SpanIO;                        // Initial structured input
  attributes?: Record<string, unknown>;  // Custom attributes
}
```

#### SpanHandle Methods

| Method                   | Description                                                     |
| ------------------------ | --------------------------------------------------------------- |
| `setAttributes(attrs)`   | Add custom key-value attributes to the span                     |
| `addEvent(name, attrs?)` | Record a lifecycle event with timestamp and optional attributes |
| `setOutput(output)`      | Set structured output (`SpanIO`)                                |
| `setInput(input)`        | Set or override structured input (`SpanIO`)                     |
| `setMetrics(metrics)`    | Set token counts, cost, latency, and model info                 |
| `spanId`                 | Read-only span ID                                               |
| `traceId`                | Read-only trace ID                                              |

### 2. `@trace()` Decorator

For class-based code. Automatically captures method arguments as input and the return value as output. Requires `experimentalDecorators: true` in `tsconfig.json`.

```typescript theme={null}
import { trace } from '@mutagent/sdk/tracing';

class MyAgent {
  @trace({ kind: 'agent', name: 'research-agent' })
  async run(query: string): Promise<string> {
    // Your agent logic here
    return result;
  }

  @trace({ kind: 'tool', name: 'web-search' })
  async search(query: string): Promise<string[]> {
    // Tool implementation
    return results;
  }
}
```

The decorator:

* Creates a span with the specified `kind` and `name`
* If `name` is omitted, falls back to the method name, then the `kind`
* Captures arguments as `input.raw` and the return value as `output.raw`
* Sets `status: 'ok'` on success, `status: 'error'` on exceptions (with message)
* Propagates parent-child span relationships via `AsyncLocalStorage` context
* Works with both sync and async methods, preserving `this` context
* No-ops gracefully when tracing is not initialized

#### Decorator Options

```typescript theme={null}
interface TraceDecoratorOptions {
  kind: SpanKind;                        // Required: span kind
  name?: string;                         // Span name (defaults to method name)
  attributes?: Record<string, unknown>;  // Custom attributes
}
```

### 3. `startSpan()` / `endSpan()` Manual API

The lowest-level API for integration adapters or scenarios where automatic span lifecycle does not fit.

```typescript theme={null}
import {
  startSpan,
  endSpan,
  runInSpanContext,
  getCurrentSpan,
  getCurrentTraceId,
} from '@mutagent/sdk/tracing';

const span = startSpan({
  kind: 'llm.chat',
  name: 'openai-chat',
  input: {
    messages: [
      { role: 'user', content: 'Hello!' },
    ],
  },
});

if (span) {
  // Run nested code within span context (enables parent-child propagation)
  const result = await runInSpanContext(span, async () => {
    // getCurrentSpan() returns this span inside the callback
    // getCurrentTraceId() returns the active trace ID
    const response = await callLLM();

    endSpan(span, {
      status: 'ok',
      output: {
        messages: [
          { role: 'assistant', content: response.text },
        ],
      },
      metrics: {
        model: 'gpt-5.4',
        provider: 'openai',
        inputTokens: response.usage.prompt_tokens,
        outputTokens: response.usage.completion_tokens,
        totalTokens: response.usage.total_tokens,
      },
    });

    return response;
  });
}
```

#### Low-Level API Functions

| Function                     | Description                                                           |
| ---------------------------- | --------------------------------------------------------------------- |
| `startSpan(options)`         | Create a new span. Returns `MutagentSpan \| undefined`                |
| `endSpan(span, endOptions?)` | End a span, compute duration, deliver to batch collector              |
| `runInSpanContext(span, fn)` | Run `fn` within the span's async context for parent-child propagation |
| `getCurrentSpan()`           | Get the active span from async context                                |
| `getCurrentTraceId()`        | Get the active trace ID from async context                            |

#### SpanOptions

```typescript theme={null}
interface SpanOptions {
  kind: SpanKind;                        // Required: span kind
  name?: string;                         // Span name
  input?: SpanIO;                        // Structured input
  attributes?: Record<string, unknown>;  // Custom attributes
  parentSpanId?: string;                 // Explicit parent span (auto-inferred from context if omitted)
}
```

#### SpanEndOptions

```typescript theme={null}
interface SpanEndOptions {
  status?: SpanStatus;                   // 'ok' | 'error' | 'unset'
  statusMessage?: string;                // Error message or status detail
  output?: SpanIO;                       // Structured output
  metrics?: SpanMetrics;                 // Token counts, cost, latency
  attributes?: Record<string, unknown>;  // Additional attributes
  events?: SpanEvent[];                  // Lifecycle events
}
```

## Configuration Reference

The `initTracing()` function accepts a `TracingConfig` object:

| Property          | Type      | Default                   | Description                                      |
| ----------------- | --------- | ------------------------- | ------------------------------------------------ |
| `apiKey`          | `string`  | -- (required)             | MutagenT API key for authentication              |
| `endpoint`        | `string`  | `https://api.mutagent.io` | API endpoint URL                                 |
| `environment`     | `string`  | --                        | Environment name (e.g., `production`, `staging`) |
| `batchSize`       | `number`  | `10`                      | Number of spans to buffer before flushing        |
| `flushIntervalMs` | `number`  | `5000`                    | Flush interval in milliseconds                   |
| `debug`           | `boolean` | `false`                   | Log span start/end events to console             |
| `source`          | `string`  | `sdk`                     | Source identifier for traces                     |

### Lifecycle Functions

| Function                 | Description                                                                                                             |
| ------------------------ | ----------------------------------------------------------------------------------------------------------------------- |
| `initTracing(config)`    | Initialize tracing. Idempotent: calling again replaces the previous configuration (shuts down the old collector first). |
| `shutdownTracing()`      | Flush remaining spans, clear timers, and reset singleton state. Returns a `Promise<void>`.                              |
| `isTracingInitialized()` | Check whether tracing has been initialized. Returns `boolean`.                                                          |

## Type Reference

### SpanKind

Taxonomy aligned with OTel Gen AI semantic conventions (v1.37+):

| Category      | Values                                                              |
| ------------- | ------------------------------------------------------------------- |
| Generation    | `llm.chat`, `llm.completion`, `llm.embedding`                       |
| Orchestration | `chain`, `agent`, `graph`, `node`, `edge`, `workflow`, `middleware` |
| Tools         | `tool`                                                              |
| Retrieval     | `retrieval`, `rerank`                                               |
| Safety        | `guardrail`                                                         |
| Other         | `custom`                                                            |

### SpanIO

Structured input/output for spans:

| Field       | Type                                           | Description                                                                             |
| ----------- | ---------------------------------------------- | --------------------------------------------------------------------------------------- |
| `messages`  | `Array<{ role, content, name?, toolCallId? }>` | Chat messages (most common for LLM spans). Roles: `system`, `user`, `assistant`, `tool` |
| `toolCalls` | `Array<{ id, name, arguments, result? }>`      | Tool calls for LLM spans requesting tool use                                            |
| `documents` | `Array<{ content, metadata?, score? }>`        | Retrieved documents (for retrieval spans)                                               |
| `text`      | `string`                                       | Raw text (for completions, generic spans)                                               |
| `raw`       | `unknown`                                      | Framework-specific data (escape hatch)                                                  |

### SpanStatus

`'ok'` | `'error'` | `'unset'`

### SpanMetrics

| Field          | Type     | Description                                             |
| -------------- | -------- | ------------------------------------------------------- |
| `model`        | `string` | Model identifier (e.g., `gpt-5.4`, `claude-sonnet-4-6`) |
| `provider`     | `string` | Provider name (e.g., `openai`, `anthropic`)             |
| `inputTokens`  | `number` | Input token count                                       |
| `outputTokens` | `number` | Output token count                                      |
| `totalTokens`  | `number` | Total token count                                       |
| `costUsd`      | `number` | Estimated cost in USD                                   |
| `latencyMs`    | `number` | Total latency in milliseconds                           |
| `ttftMs`       | `number` | Time to first token in milliseconds                     |

### SpanEvent

| Field        | Type                      | Description                                      |
| ------------ | ------------------------- | ------------------------------------------------ |
| `name`       | `string`                  | Event name                                       |
| `timestamp`  | `Date`                    | When the event occurred (auto-set by `addEvent`) |
| `attributes` | `Record<string, unknown>` | Optional event attributes                        |

### MutagentSpan

Internal mutable span representation (returned by `startSpan`):

| Field           | Type                      | Description                   |
| --------------- | ------------------------- | ----------------------------- |
| `id`            | `string`                  | Unique span ID                |
| `traceId`       | `string`                  | Trace ID this span belongs to |
| `parentSpanId`  | `string \| undefined`     | Parent span ID (if nested)    |
| `kind`          | `SpanKind`                | Span kind                     |
| `name`          | `string`                  | Span name                     |
| `startedAt`     | `Date`                    | When the span started         |
| `input`         | `SpanIO \| undefined`     | Structured input              |
| `output`        | `SpanIO \| undefined`     | Structured output             |
| `status`        | `SpanStatus`              | Current status                |
| `statusMessage` | `string \| undefined`     | Error/status message          |
| `metrics`       | `SpanMetrics`             | Token counts, cost, etc.      |
| `attributes`    | `Record<string, unknown>` | Custom attributes             |
| `events`        | `SpanEvent[]`             | Lifecycle events              |

### FinishedSpan

Immutable span ready for serialization and transport:

| Field           | Type                  | Description              |
| --------------- | --------------------- | ------------------------ |
| `spanId`        | `string`              | Unique span ID           |
| `traceId`       | `string`              | Trace ID                 |
| `parentSpanId`  | `string \| undefined` | Parent span ID           |
| `name`          | `string`              | Span name                |
| `kind`          | `SpanKind`            | Span kind                |
| `startTime`     | `string`              | ISO 8601 start time      |
| `endTime`       | `string`              | ISO 8601 end time        |
| `durationMs`    | `number`              | Duration in milliseconds |
| `status`        | `SpanStatus`          | Final status             |
| `statusMessage` | `string \| undefined` | Error/status message     |
| `model`         | `string \| undefined` | Model used               |
| `provider`      | `string \| undefined` | Provider name            |
| `inputTokens`   | `number \| undefined` | Input token count        |
| `outputTokens`  | `number \| undefined` | Output token count       |
| `totalTokens`   | `number \| undefined` | Total token count        |
| `costUsd`       | `number \| undefined` | Estimated cost           |

## Exports Summary

All exports from `@mutagent/sdk/tracing`:

```typescript theme={null}
// Lifecycle
import {
  initTracing,
  shutdownTracing,
  isTracingInitialized,
} from '@mutagent/sdk/tracing';

// Decorator
import { trace } from '@mutagent/sdk/tracing';
import type { TraceDecoratorOptions } from '@mutagent/sdk/tracing';

// Wrapper
import { withTrace } from '@mutagent/sdk/tracing';
import type { WithTraceOptions } from '@mutagent/sdk/tracing';

// Low-level API
import {
  startSpan,
  endSpan,
  getCurrentSpan,
  getCurrentTraceId,
  runInSpanContext,
} from '@mutagent/sdk/tracing';

// Types
import type {
  SpanKind,
  SpanIO,
  SpanStatus,
  SpanEvent,
  SpanMetrics,
  SpanOptions,
  SpanEndOptions,
  SpanHandle,
  MutagentSpan,
  FinishedSpan,
  TracingConfig,
  TracePayload,
  TraceHTTPResponse,
} from '@mutagent/sdk/tracing';
```

## Next Steps

<CardGroup cols={2}>
  <Card title="Tracing Overview" icon="chart-tree-map" href="/tracing/overview">
    Concepts, span hierarchy, and tracing architecture
  </Card>

  <Card title="Tracing API Reference" icon="book" href="/tracing/api">
    Full server-side API reference for ingesting and querying traces
  </Card>

  <Card title="Integrations" icon="puzzle-piece" href="/integrations/overview">
    Framework adapters with automatic tracing (recommended for most users)
  </Card>

  <Card title="TypeScript SDK Setup" icon="js" href="/sdk/typescript/installation">
    Install and configure the TypeScript SDK
  </Card>
</CardGroup>