Skip to main content

Observability & Telemetry

Observability is what lets you answer "what is the agent doing, how fast, and why did it fail?" without attaching a debugger. AION Agents V3 integrates a complete native observability stack based on OpenTelemetry (OTel, the open standard for traces, metrics and logs), designed for monitoring LLM (large language model) agents in production. This page is for developers and IT/operations staff who need to wire AION into a monitoring backend (Jaeger, Tempo, Prometheus, Grafana, Opik) or troubleshoot a deployment.

OTLP below stands for OpenTelemetry Protocol, the wire format used to push data to an OTel collector.

Architecture Overview​

AION's observability is structured around three pillars — Traces, Metrics, Logs — all based on a push-based model towards an OTel collector (AION_OTEL_ENDPOINT). In addition, there is a second telemetry layer dedicated solely to the LLM component: Opik (Comet ML), a self-hosted system for prompt library management, LLM call tracing and session feedback scores.

┌─────────────────────────────────────────────────────────────┐
│ AION Agents Runtime │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Traces │ │ Metrics │ │ Logs │ │
│ │ (OTel) │ │ (Dual) │ │(struct) │ │
│ └────┬─────┘ └────┬─────┘ └────┬─────┘ │
│ │ │ │ │
│ ┌────▼─────┐ ┌────▼─────┐ ┌────▼─────┐ │
│ │ OpenLit │ │ Prometheus│ │ OTLP Log │ │
│ │ (auto) │ │ + OTel │ │ Exporter │ │
│ │ │ │ Push │ │ │ │
│ └────┬─────┘ └────┬─────┘ └────┬─────┘ │
└───────┼───────────────┼─────────────┼──────────────────────┘
│ │ │
▼ ▼ ▼
┌───────────────┐ ┌───────────┐ ┌──────────────┐
│ OTel Collector│ │ /metrics │ │ OTel Collector│
│ (gRPC/HTTP) │ │ (local) │ │ (gRPC/HTTP) │
└───────┬───────┘ └───────────┘ └──────┬───────┘
│ │
▼ ▼
┌──────────────────┐ ┌────────────────┐
│ Phoenix / │ │ Opik (Comet │
│ Jaeger / etc. │ │ ML) LLM Trace │
└──────────────────┘ └────────────────┘

Main Components​

PillarTechnologiesModeEndpoint / Destination
TracesOpenLit → OTel SDKOTLP PushAION_OTEL_ENDPOINT
MetricsPrometheus + OTelDual-export (Push + Pull)/metrics (local) + AION_OTEL_ENDPOINT
Logsstructlog + OTelOTLP PushAION_OTEL_ENDPOINT
LLM TelemetryOpik (Comet ML)HTTP PushOPIK_URL_OVERRIDE

Tracing (OpenTelemetry)​

AION emits hierarchical spans for each critical processing phase. Traces use aion.* attributes (not gen_ai.*) and are compatible with any backend that accepts OTLP (Jaeger, Tempo, etc.).

OpenLit vs Standard OTel SDK​

AION first attempts to initialize OpenLit (full auto-instrumentation: FastAPI, SQLAlchemy, HTTPX). If OpenLit is unavailable or fails, it falls back to the standard OTel SDK fallback.

The init flow is located in src/observability/otel_setup.py:

  1. If AION_OTEL_ENABLED=1 and AION_OPENLIT_ACTIVE=1 → uses OpenLit (already initialized in aion_env.py)
  2. Otherwise if AION_OTEL_ENABLED=1 → fallback to standard OTel SDK with BatchSpanProcessor
  3. If AION_OTEL_ENABLED=0 → local dummy tracer provider (no export)

Span Flow​

Each conversational turn generates a tree similar to the following:

conversation.turn (root)
├── hooks.pre_llm_call
├── llm.call (TTFT, Total Latency)
├── tool.execute (e.g. prometheus_query)
│ ├── mcp.request (cross-process propagation)
└── hooks.post_llm_call

Span Attributes​

Spans use the aion.* prefix for custom metadata:

AttributeDescription
aion.profileName of the active agent profile
aion.session_idChat session ID
aion.user_idUser ID
aion.tenant_idTenant ID
tool.nameName of the executed tool
tool.mcp_serverName of the MCP server
tool.outputTool output (if brief)
tool.errorError message

Trace Correlation​

The trace_id generated by OpenTelemetry is automatically propagated to the persistence layer and saved within the unified DB in the following tables:

  • messages
  • steps (tool calls)
  • audit_log

This allows starting from an application log or an audit and pivoting to the trace visualization in a compatible OTel backend (Jaeger, Tempo, Phoenix, etc.).

Tracing Configuration​

VariableDefaultDescription
AION_OTEL_ENABLED0Enables OTel tracing (0 = disabled, local tracer only)
AION_OTEL_ENDPOINThttp://localhost:4317OTLP collector endpoint
AION_OTEL_PROTOCOLgrpcOTLP protocol: grpc or http
AION_OTEL_SERVICE_NAMEaion-agentService name in traces
AION_OTEL_TRACE_CONTENT1Captures LLM content in spans (prompt/response)

Metrics​

AION uses a dual-export pattern: each metric is simultaneously backed by Prometheus (prometheus_client) and OTel. The wrappers in src/observability/metrics.py (OTelPrometheusCounter, OTelPrometheusHistogram, OTelPrometheusGauge) ensure that every inc(), observe() or set() operation writes to both backends.

Prometheus Exposure​

Metrics are exposed locally on the /metrics endpoint (default) via prometheus_fastapi_instrumentator, enabled by AION_METRICS_ENABLED=1. This allows Prometheus/Grafana to scrape directly without additional configuration.

OTel Metrics Push​

When AION_OTEL_ENABLED=1, metrics are also periodically pushed to AION_OTEL_ENDPOINT via PeriodicExportingMetricReader with a configurable interval via AION_OTEL_METRIC_EXPORT_INTERVAL (default 5000ms).

Wrapper Classes​

Each metric is a wrapper that maintains a native Prometheus metric and an OTel instrument:

WrapperPrometheusOTel
OTelPrometheusCounterCounterCounter
OTelPrometheusHistogramHistogramHistogram
OTelPrometheusGaugeGaugeGauge (or UpDownCounter as fallback)

_init_otel() is called after the MeterProvider has been configured (in otel_setup.py), linking the OTel instruments to the wrappers.

Metrics Inventory​

AION defines 11 metrics divided into 3 categories:

Conversational Metrics​

MetricTypeLabelsDescription
aion_messages_totalCountertenant_id, profile, role, finish_reasonTotal messages processed (user or assistant)
aion_tool_calls_totalCountertenant_id, profile, tool_name, mcp_server, statusTotal tool invocations (status: ok, error, blocked)

Performance Metrics​

MetricTypeLabelsDescription
aion_turn_duration_secondsHistogramtenant_id, profileEnd-to-end turn duration (buckets: 0.5–120s)
aion_tool_call_duration_secondsHistogramtool_name, mcp_serverSingle tool execution duration (buckets: 0.05–30s)
aion_session_cache_size_bytesGaugetenant_idSize of the session's local filesystem

LLM Metrics (self-reported)​

MetricTypeLabelsDescription
aion_llm_tokens_totalCountertenant_id, profile, model, token_typeCumulative tokens (token_type: prompt, completion, reasoning)
aion_llm_turn_tokensGaugetenant_id, profile, model, token_typeTokens of the last turn
aion_llm_turn_callsGaugetenant_id, profileNumber of LLM calls in the last turn
aion_agent_failures_totalCountertenant_id, profile, error_typeAgent turn failures (error_type: timeout, error, cancelled, etc.)
aion_mcp_server_healthyGaugemcp_serverMCP server health status (1=healthy, 0=unhealthy)

Note: LLM metrics (aion_llm_*) are self-reported from the LLM response metadata. AION does not integrate native vLLM metrics — token counts come from the usage field of the API response.

Emission via Hook​

Metrics are emitted at runtime via hooks registered in src/observability/hooks_emitter.py:

HookEmitted Metrics
on_user_messageaion_messages_total (role=user)
pre_tool_useTimestamp registration for duration calculation
post_tool_useaion_tool_calls_total, aion_tool_call_duration_seconds
post_turnaion_turn_duration_seconds, aion_messages_total (role=assistant), aion_llm_tokens_total, aion_llm_turn_tokens, aion_llm_turn_calls, aion_agent_failures_total, aion_session_cache_size_bytes

Evaluation & Metrics Architecture (/admin/metrics)​

AION Agents includes a dedicated telemetry dashboard (/admin/metrics) and backend aggregator (/admin/metrics/overview) designed to operate seamlessly both with a remote Prometheus server and in standalone/local environments without Prometheus.

┌───────────────────────────────────────────────┐
│ Admin UI: Evaluation & Metrics Page │
│ (KPI Cards, Sparklines, Profiles & Users) │
└───────────────────────┬───────────────────────┘
│
▼ GET /admin/metrics/overview
┌───────────────────────────────────────────────────────────────────────────────────────────┐
│ FastAPI Backend (`src/api/metrics_api.py`) │
│ │
│ 1. Check Prometheus Connectivity (GET {AION_PROMETHEUS_URL}/api/v1/query?query=up) │
│ │
│ ┌───────────────────────────────────────────┐ ┌───────────────────────────────────┐ │
│ │ WITH PROMETHEUS │ │ WITHOUT PROMETHEUS │ │
│ │ (Prometheus Online) │ │ (Local Fallback) │ │
│ ├───────────────────────────────────────────┤ ├───────────────────────────────────┤ │
│ │ • Range PromQL queries: │ │ • Local `prometheus_client` │ │
│ │ `increase(aion_llm_tokens_total[step])` │ │ metric objects in RAM │ │
│ │ • `histogram_quantile(0.95, ...)` latency │ │ • In-memory rolling ring buffer │ │
│ │ • Dynamic step intervals (1m → 1d) │ │ `_time_series_ring_buffer` │ │
│ │ • Cross-tenant and profile label filters │ │ • Exact per-minute bucketized pts │ │
│ │ • Tool error aggregations │ │ • Rolling `_recent_mcp_errors` │ │
│ └─────────────────────┬─────────────────────┘ └─────────────────┬─────────────────┘ │
│ │ │ │
│ └─────────────────────┬─────────────────────┘ │
│ ▼ │
│ Standardized ISO 8601 UTC Timestamps (`dt.isoformat()`) │
└───────────────────────────────────────────────┬───────────────────────────────────────────┘
│
▼ JSON Response
┌───────────────────────────────────────────────────────────────────────────────────────────┐
│ Client-side Dynamic Timezone Rendering (`admin-ui/app/metrics/page.tsx`) │
│ • Converts ISO 8601 UTC → User Browser Local Timezone (e.g. UTC+2 / CEST) │
│ • Near real-time auto-refresh (every 3 seconds) displaying last-turn activity immediately │
└───────────────────────────────────────────────────────────────────────────────────────────┘

1. Scope & Target Audience​

AION partitions observability between Grafana and the Admin UI:

Target DashboardSupported Telemetry & Visualizations
Both (Grafana + Admin UI)LLM Tokens (Input, Output, Reasoning), Turn Durations (Average, Last Turn, P95 Latency), Total Turns & Failures, Total Tool Invocations & Success Rate %.
Grafana ExclusiveRaw tool execution latency histograms, Session workspace disk cache footprint (aion_session_cache_size_bytes), MCP server binary health gauges (aion_mcp_server_healthy), worker pool sizes.
Admin UI ExclusivePer-agent profile metric breakdown table, Per-user expandable profile usage & tool distribution, Last turn tool invocation snapshot, Searchable MCP tool metrics, MCP Call Error Diagnostics list with full stack traces, and local timezone trendline sparklines.

2. Operation WITH Prometheus​

When AION_PROMETHEUS_URL is reachable:

  • The backend queries Prometheus via standard REST endpoints (/api/v1/query and /api/v1/query_range).
  • Dynamic Step Calculation: Step intervals automatically adapt to the selected time range to generate optimal chart resolution (12 to 60 data points):
    • 1h range → 1m step (60 points)
    • 6h range → 10m step (36 points)
    • 24h range → 30m step (48 points)
    • 7d range → 6h step (28 points)
    • 30d / all → 1d step (30 points)
  • Aggregations: Executes vector operations like sum(increase(aion_llm_tokens_total[step])) and histogram_quantile(0.95, sum(rate(aion_turn_duration_seconds_bucket[step])) by (le)).

3. Operation WITHOUT Prometheus (Standalone Fallback)​

When Prometheus is unavailable or offline:

  • The backend automatically switches to zero-dependency in-process fallback mode without failing API calls.
  • Cumulative counters, gauges, and histograms are directly extracted from local Python memory structures in src/observability/metrics.py.
  • Time-series charts for tokens, turn durations, turns, and tool calls are generated from _time_series_ring_buffer in src/observability/hooks_emitter.py, which records every conversational turn and tool invocation with epoch timestamps.
  • MCP execution errors are captured in _recent_mcp_errors with full error payloads.

4. Timezone & Real-Time Synchronization​

  • ISO 8601 UTC Emission: All server-side time-series data points and error records emit standard ISO 8601 timestamps (e.g. 2026-09-03T12:51:06+00:00).
  • Client-Side Localization: The Admin UI parses these timestamps using the browser's native JavaScript Date engine, displaying accurate local time (e.g. 14:51 in UTC+2/CEST) in chart hover tooltips and diagnostic tables.
  • Near Real-Time Telemetry: With auto-refresh enabled (3s interval), requests made in the chat or via API immediately reflect on the rightmost bucket of the trendlines.

Structured Logging​

All core infrastructure logs use structlog to produce structured logs (JSON or human-readable text). The configuration is managed by src/observability/logging.py.

Formatting​

The format is selectable via AION_LOG_FORMAT:

ValueFormatUse
json (default)JSON on stdoutProduction, ingestion in Loki/Elastic
textConsoleRenderer (human-readable)Development

OTel Context Injection​

Each log automatically includes the current trace_id and span_id, obtained from the active OTel span. Span attributes (aion.profile, aion.session_id, tool.name, etc.) are also injected into the logs.

The add_otel_context processor in logging.py reads the current span and adds the fields:

{
"event": "tool execution complete",
"trace_id": "0000000000000000abcdef1234567890",
"span_id": "0000000012345678",
"aion.profile": "tecnico-senior",
"aion.session_id": "sess-abc123",
"level": "info"
}

OTLP Log Export​

When AION_OTEL_ENABLED=1, logs are also exported via OTLP using BatchLogRecordProcessor + OTLPLogExporter. This allows correlating application logs with traces in the OTel backend.

Logging Configuration​

VariableDefaultDescription
AION_LOG_LEVELINFOLogging level (DEBUG, INFO, WARNING, ERROR)
AION_LOG_FORMATjsonFormat: json or text

Logs go only to stdout (no file logging). Uvicorn and FastAPI loggers are propagated to the root logger to maintain format consistency.

Opik — Self-Hosted LLM Telemetry​

Opik (Comet ML) is a telemetry layer separate from the OTel pipeline, dedicated specifically to LLM tracking, prompt library and session feedback scores. It is designed to be self-hosted and communicate with an Opik instance on an internal network.

Opik Architecture​

┌─────────────────────────────────┐
│ Agent Runtime │
│ │
│ ┌──────────────┐ ┌──────────┐ │
│ │ opik_setup.py│ │opik_telemetry.py│
│ │ (SDK init, │ │ (@track │ │
│ │ Prompt Lib) │ │ decorators)│ │
│ └──────┬───────┘ └────┬─────┘ │
│ │ │ │
│ └───────┬───────┘ │
│ │ │
└─────────────────┼─────────────────┘
│ HTTP
▼
┌─────────────────────────────────┐
│ Opik (Comet ML) │
│ │
│ • Prompt Library │
│ • Traces (LLM, tool, skill) │
│ • Threads (multi-turn) │
│ • Feedback Scores │
└─────────────────────────────────┘

Setup (src/observability/opik_setup.py)​

The opik_setup.py module manages the initialization of the Opik SDK:

  1. Checks if AION_OPIK_ENABLED is set to 1 (or true / yes / on). If not enabled, it completely bypasses Opik client initialization and sets OPIK_AVAILABLE to False.
  2. Reads OPIK_URL_OVERRIDE and OPIK_PROJECT_NAME from environment variables.
  3. Automatically sets a placeholder for OPIK_API_KEY if not configured (required for client validation in self-hosted mode).
  4. Initializes the Opik client with Opik().
  5. Exports OPIK_AVAILABLE (bool) for graceful degradation.

If AION_OPIK_ENABLED=0, the SDK is not installed, or the server is unreachable, OPIK_AVAILABLE becomes False and no requests fail — the application continues normally without attempting connections.

Prompt Library​

get_or_create_prompt() manages the versioning of prompt templates:

  1. Searches for the prompt by name in the Opik Prompt Library
  2. If it exists and the template has changed → creates a new version (commit)
  3. If it exists and the template is unchanged → returns the existing prompt
  4. If it does not exist → creates a new prompt with the provided template

The template uses Jinja-style placeholders (e.g. {{profilo}}, {{user_input}}) that are dynamically formatted at runtime via prompt_obj.format().

Tracking Decorators (src/observability/opik_telemetry.py)​

The module provides @track decorators to mark functions as Opik spans of different types:

DecoratorTypeFunctionDescription
@track(type="llm")call_llm_api()Real LLM call with token tracking, temperature, endpoint
@track(type="tool")execute_tool_ricerca()Tool execution with input/output in the span
@track(type="general")skill_ricerca_db(), skill_calcolo_metrico()Agent skills as nested spans
@track(type="general")run_agent()Main root trace of the agent

Agent Integration​

Opik wrappers are applied at runtime in:

  • src/main.py:1169-1261: Opik wrappers applied to chat_generator and MCP tools
  • src/agent_pipeline.py:987-1015: Opik trace initialization in the agent pipeline

Session Feedback​

chiudi_sessione_e_valuta() sends session-level feedback scores to Opik at the end of the interaction. Currently it sends a fixed score of 1.0 for the risoluzione_problema metric, but the design supports future extensions for real feedback (e.g. user thumbs up/down).

OTel vs Opik Distinction​

AspectOTelOpik
PurposeGeneric infrastructureLLM-specific
TracesGeneric spans (HTTP, DB, tool)LLM, prompt, skill spans
MetricsPrometheus + OTel pushNo metrics
Prompt LibraryNoneYes, with versioning
Feedback ScoreNoneYes, at session level
BackendPhoenix, Jaeger, Tempo, etc.Opik (Comet ML)

Opik Configuration​

VariableDefaultDescription
OPIK_API_KEYlocal-self-hosted-placeholderOpik API key (placeholder if not set)
OPIK_PROJECT_NAMEAION-AgentOpik project name
OPIK_URL_OVERRIDEhttp://localhost:5173/apiAPI endpoint of the self-hosted Opik instance

Complete Configuration​

OTel / Infrastructure Variables​

VariableDefaultDescription
AION_OTEL_ENABLED0Enables OTel tracing (push to collector)
AION_OTEL_ENDPOINThttp://localhost:4317OTLP collector endpoint (gRPC)
AION_OTEL_PROTOCOLgrpcOTLP protocol: grpc or http
AION_OTEL_SERVICE_NAMEaion-agentService name in traces/metrics/logs
AION_OTEL_TRACE_CONTENT1Captures LLM content in spans (prompt/response)
AION_OTEL_METRIC_EXPORT_INTERVAL5000OTel metrics push interval (ms)
AION_METRICS_ENABLED1Enables Prometheus exposure on /metrics
AION_METRICS_PATH/metricsPrometheus endpoint path
AION_LOG_LEVELINFOLogging level
AION_LOG_FORMATjsonLog format: json or text
AION_ENVdevDeployment environment (dev, staging, prod)

Opik Variables (LLM Telemetry)​

VariableDefaultDescription
AION_OPIK_ENABLED0Enables/disables Opik (Comet ML) LLM telemetry tracing (0 = disabled, 1 = enabled)
OPIK_API_KEYlocal-self-hosted-placeholderOpik API key (placeholder for self-hosted client validation)
OPIK_PROJECT_NAMEAION-AgentOpik project name
OPIK_URL_OVERRIDEhttp://localhost:5173/apiAPI endpoint of the self-hosted Opik instance

Per-step LLM call audit (local JSON)​

Separate from Opik/OTel: when AION_LLM_CALL_AUDIT=1, src/runtime/llm_call_audit.py writes full request/response payloads for each agent LLM step to data/diagnostics/llm_calls/ (override with AION_LLM_CALL_AUDIT_DIR). CLI: python scripts/audit_llm_calls.py. Intended for dev/staging regression on tool-call JSON and vLLM thinking behavior — not a replacement for production trace export.

Code Reference Files​

FileResponsibility
src/observability/otel_setup.pyOTel init (OpenLit + fallback, traces, dual-export metrics)
src/observability/metrics.py11 dual-export metrics, OTel/Prometheus wrapper classes
src/observability/logging.pyStructured logging, OTLP log export, OTel context injection
src/observability/hooks_emitter.pyHook-based metrics emission (4 hooks)
src/observability/opik_setup.pyOpik SDK init, Prompt Library, session feedback
src/observability/opik_telemetry.py@track decorators for LLM, tool, skill tracing
src/aion_env.pyEarly OTel + Opik env var setup (first import)
src/main.pyOpik wrappers on chat_generator and tools
src/agent_pipeline.pyOpik trace initialization in the pipeline