Observability & Telemetry
Observability is what lets you answer "what is the agent doing, how fast, and why did it fail?" without attaching a debugger. AION Agents V3 integrates a complete native observability stack based on OpenTelemetry (OTel, the open standard for traces, metrics and logs), designed for monitoring LLM (large language model) agents in production. This page is for developers and IT/operations staff who need to wire AION into a monitoring backend (Jaeger, Tempo, Prometheus, Grafana, Opik) or troubleshoot a deployment.
OTLP below stands for OpenTelemetry Protocol, the wire format used to push data to an OTel collector.
Architecture Overview
AION's observability is structured around three pillars — Traces, Metrics, Logs — all based on a push-based model towards an OTel collector (AION_OTEL_ENDPOINT). In addition, there is a second telemetry layer dedicated solely to the LLM component: Opik (Comet ML), a self-hosted system for prompt library management, LLM call tracing and session feedback scores.
┌─────────────────────────────────────────────────────────────┐
│ AION Agents Runtime │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Traces │ │ Metrics │ │ Logs │ │
│ │ (OTel) │ │ (Dual) │ │(struct) │ │
│ └────┬─────┘ └────┬─────┘ └────┬─────┘ │
│ │ │ │ │
│ ┌────▼─────┐ ┌────▼─────┐ ┌────▼─────┐ │
│ │ OpenLit │ │ Prometheus│ │ OTLP Log │ │
│ │ (auto) │ │ + OTel │ │ Exporter │ │
│ │ │ │ Push │ │ │ │
│ └────┬─────┘ └────┬─────┘ └────┬─────┘ │
└───────┼───────────────┼─────────────┼──────────────────────┘
│ │ │
▼ ▼ ▼
┌───────────────┐ ┌───────────┐ ┌──────────────┐
│ OTel Collector│ │ /metrics │ │ OTel Collector│
│ (gRPC/HTTP) │ │ (local) │ │ (gRPC/HTTP) │
└───────┬───────┘ └───────────┘ └──────┬───────┘
│ │
▼ ▼
┌──────────────────┐ ┌────────────────┐
│ Phoenix / │ │ Opik (Comet │
│ Jaeger / etc. │ │ ML) LLM Trace │
└──────────────────┘ └────────────────┘
Main Components
| Pillar | Technologies | Mode | Endpoint / Destination |
|---|---|---|---|
| Traces | OpenLit → OTel SDK | OTLP Push | AION_OTEL_ENDPOINT |
| Metrics | Prometheus + OTel | Dual-export (Push + Pull) | /metrics (local) + AION_OTEL_ENDPOINT |
| Logs | structlog + OTel | OTLP Push | AION_OTEL_ENDPOINT |
| LLM Telemetry | Opik (Comet ML) | HTTP Push | OPIK_URL_OVERRIDE |
Tracing (OpenTelemetry)
AION emits hierarchical spans for each critical processing phase. Traces use aion.* attributes (not gen_ai.*) and are compatible with any backend that accepts OTLP (Jaeger, Tempo, etc.).
OpenLit vs Standard OTel SDK
AION first attempts to initialize OpenLit (full auto-instrumentation: FastAPI, SQLAlchemy, HTTPX). If OpenLit is unavailable or fails, it falls back to the standard OTel SDK fallback.
The init flow is located in src/observability/otel_setup.py:
- If
AION_OTEL_ENABLED=1andAION_OPENLIT_ACTIVE=1→ uses OpenLit (already initialized inaion_env.py) - Otherwise if
AION_OTEL_ENABLED=1→ fallback to standard OTel SDK withBatchSpanProcessor - If
AION_OTEL_ENABLED=0→ local dummy tracer provider (no export)
Span Flow
Each conversational turn generates a tree similar to the following:
conversation.turn (root)
├── hooks.pre_llm_call
├── llm.call (TTFT, Total Latency)
├── tool.execute (e.g. prometheus_query)
│ ├── mcp.request (cross-process propagation)
└── hooks.post_llm_call
Span Attributes
Spans use the aion.* prefix for custom metadata:
| Attribute | Description |
|---|---|
aion.profile | Name of the active agent profile |
aion.session_id | Chat session ID |
aion.user_id | User ID |
aion.tenant_id | Tenant ID |
tool.name | Name of the executed tool |
tool.mcp_server | Name of the MCP server |
tool.output | Tool output (if brief) |
tool.error | Error message |
Trace Correlation
The trace_id generated by OpenTelemetry is automatically propagated to the persistence layer and saved within the unified DB in the following tables:
messagessteps(tool calls)audit_log
This allows starting from an application log or an audit and pivoting to the trace visualization in a compatible OTel backend (Jaeger, Tempo, Phoenix, etc.).
Tracing Configuration
| Variable | Default | Description |
|---|---|---|
AION_OTEL_ENABLED | 0 | Enables OTel tracing (0 = disabled, local tracer only) |
AION_OTEL_ENDPOINT | http://localhost:4317 | OTLP collector endpoint |
AION_OTEL_PROTOCOL | grpc | OTLP protocol: grpc or http |
AION_OTEL_SERVICE_NAME | aion-agent | Service name in traces |
AION_OTEL_TRACE_CONTENT | 1 | Captures LLM content in spans (prompt/response) |
Metrics
AION uses a dual-export pattern: each metric is simultaneously backed by Prometheus (prometheus_client) and OTel. The wrappers in src/observability/metrics.py (OTelPrometheusCounter, OTelPrometheusHistogram, OTelPrometheusGauge) ensure that every inc(), observe() or set() operation writes to both backends.
Prometheus Exposure
Metrics are exposed locally on the /metrics endpoint (default) via prometheus_fastapi_instrumentator, enabled by AION_METRICS_ENABLED=1. This allows Prometheus/Grafana to scrape directly without additional configuration.
OTel Metrics Push
When AION_OTEL_ENABLED=1, metrics are also periodically pushed to AION_OTEL_ENDPOINT via PeriodicExportingMetricReader with a configurable interval via AION_OTEL_METRIC_EXPORT_INTERVAL (default 5000ms).
Wrapper Classes
Each metric is a wrapper that maintains a native Prometheus metric and an OTel instrument:
| Wrapper | Prometheus | OTel |
|---|---|---|
OTelPrometheusCounter | Counter | Counter |
OTelPrometheusHistogram | Histogram | Histogram |
OTelPrometheusGauge | Gauge | Gauge (or UpDownCounter as fallback) |
_init_otel() is called after the MeterProvider has been configured (in otel_setup.py), linking the OTel instruments to the wrappers.
Metrics Inventory
AION defines 11 metrics divided into 3 categories:
Conversational Metrics
| Metric | Type | Labels | Description |
|---|---|---|---|
aion_messages_total | Counter | tenant_id, profile, role, finish_reason | Total messages processed (user or assistant) |
aion_tool_calls_total | Counter | tenant_id, profile, tool_name, mcp_server, status | Total tool invocations (status: ok, error, blocked) |
Performance Metrics
| Metric | Type | Labels | Description |
|---|---|---|---|
aion_turn_duration_seconds | Histogram | tenant_id, profile | End-to-end turn duration (buckets: 0.5–120s) |
aion_tool_call_duration_seconds | Histogram | tool_name, mcp_server | Single tool execution duration (buckets: 0.05–30s) |
aion_session_cache_size_bytes | Gauge | tenant_id | Size of the session's local filesystem |
LLM Metrics (self-reported)
| Metric | Type | Labels | Description |
|---|---|---|---|
aion_llm_tokens_total | Counter | tenant_id, profile, model, token_type | Cumulative tokens (token_type: prompt, completion, reasoning) |
aion_llm_turn_tokens | Gauge | tenant_id, profile, model, token_type | Tokens of the last turn |
aion_llm_turn_calls | Gauge | tenant_id, profile | Number of LLM calls in the last turn |
aion_agent_failures_total | Counter | tenant_id, profile, error_type | Agent turn failures (error_type: timeout, error, cancelled, etc.) |
aion_mcp_server_healthy | Gauge | mcp_server | MCP server health status (1=healthy, 0=unhealthy) |
Note: LLM metrics (aion_llm_*) are self-reported from the LLM response metadata. AION does not integrate native vLLM metrics — token counts come from the usage field of the API response.
Emission via Hook
Metrics are emitted at runtime via hooks registered in src/observability/hooks_emitter.py:
| Hook | Emitted Metrics |
|---|---|
on_user_message | aion_messages_total (role=user) |
pre_tool_use | Timestamp registration for duration calculation |
post_tool_use | aion_tool_calls_total, aion_tool_call_duration_seconds |
post_turn | aion_turn_duration_seconds, aion_messages_total (role=assistant), aion_llm_tokens_total, aion_llm_turn_tokens, aion_llm_turn_calls, aion_agent_failures_total, aion_session_cache_size_bytes |
Evaluation & Metrics Architecture (/admin/metrics)
AION Agents includes a dedicated telemetry dashboard (/admin/metrics) and backend aggregator (/admin/metrics/overview) designed to operate seamlessly both with a remote Prometheus server and in standalone/local environments without Prometheus.
┌───────────────────────────────────────────────┐
│ Admin UI: Evaluation & Metrics Page │
│ (KPI Cards, Sparklines, Profiles & Users) │
└───────────────────────┬───────────────────────┘
│
▼ GET /admin/metrics/overview
┌───────────────────────────────────────────────────────────────────────────────────────────┐
│ FastAPI Backend (`src/api/metrics_api.py`) │
│ │
│ 1. Check Prometheus Connectivity (GET {AION_PROMETHEUS_URL}/api/v1/query?query=up) │
│ │
│ ┌───────────────────────────────────────────┐ ┌───────────────────────────────────┐ │
│ │ WITH PROMETHEUS │ │ WITHOUT PROMETHEUS │ │
│ │ (Prometheus Online) │ │ (Local Fallback) │ │
│ ├───────────────────────────────────────────┤ ├───────────────────────────────────┤ │
│ │ • Range PromQL queries: │ │ • Local `prometheus_client` │ │
│ │ `increase(aion_llm_tokens_total[step])` │ │ metric objects in RAM │ │
│ │ • `histogram_quantile(0.95, ...)` latency │ │ • In-memory rolling ring buffer │ │
│ │ • Dynamic step intervals (1m → 1d) │ │ `_time_series_ring_buffer` │ │
│ │ • Cross-tenant and profile label filters │ │ • Exact per-minute bucketized pts │ │
│ │ • Tool error aggregations │ │ • Rolling `_recent_mcp_errors` │ │
│ └─────────────────────┬─────────────────────┘ └─────────────────┬─────────────────┘ │
│ │ │ │
│ └─────────────────────┬─────────────────────┘ │
│ ▼ │
│ Standardized ISO 8601 UTC Timestamps (`dt.isoformat()`) │
└───────────────────────────────────────────────┬───────────────────────────────────────────┘
│
▼ JSON Response
┌───────────────────────────────────────────────────────────────────────────────────────────┐
│ Client-side Dynamic Timezone Rendering (`admin-ui/app/metrics/page.tsx`) │
│ • Converts ISO 8601 UTC → User Browser Local Timezone (e.g. UTC+2 / CEST) │
│ • Near real-time auto-refresh (every 3 seconds) displaying last-turn activity immediately │
└───────────────────────────────────────────────────────────────────────────────────────────┘
1. Scope & Target Audience
AION partitions observability between Grafana and the Admin UI:
| Target Dashboard | Supported Telemetry & Visualizations |
|---|---|
| Both (Grafana + Admin UI) | LLM Tokens (Input, Output, Reasoning), Turn Durations (Average, Last Turn, P95 Latency), Total Turns & Failures, Total Tool Invocations & Success Rate %. |
| Grafana Exclusive | Raw tool execution latency histograms, Session workspace disk cache footprint (aion_session_cache_size_bytes), MCP server binary health gauges (aion_mcp_server_healthy), worker pool sizes. |
| Admin UI Exclusive | Per-agent profile metric breakdown table, Per-user expandable profile usage & tool distribution, Last turn tool invocation snapshot, Searchable MCP tool metrics, MCP Call Error Diagnostics list with full stack traces, and local timezone trendline sparklines. |
2. Operation WITH Prometheus
When AION_PROMETHEUS_URL is reachable:
- The backend queries Prometheus via standard REST endpoints (
/api/v1/queryand/api/v1/query_range). - Dynamic Step Calculation: Step intervals automatically adapt to the selected time range to generate optimal chart resolution (12 to 60 data points):
1hrange →1mstep (60 points)6hrange →10mstep (36 points)24hrange →30mstep (48 points)7drange →6hstep (28 points)30d/all→1dstep (30 points)
- Aggregations: Executes vector operations like
sum(increase(aion_llm_tokens_total[step]))andhistogram_quantile(0.95, sum(rate(aion_turn_duration_seconds_bucket[step])) by (le)).
3. Operation WITHOUT Prometheus (Standalone Fallback)
When Prometheus is unavailable or offline:
- The backend automatically switches to zero-dependency in-process fallback mode without failing API calls.
- Cumulative counters, gauges, and histograms are directly extracted from local Python memory structures in
src/observability/metrics.py. - Time-series charts for tokens, turn durations, turns, and tool calls are generated from
_time_series_ring_bufferinsrc/observability/hooks_emitter.py, which records every conversational turn and tool invocation with epoch timestamps. - MCP execution errors are captured in
_recent_mcp_errorswith full error payloads.
4. Timezone & Real-Time Synchronization
- ISO 8601 UTC Emission: All server-side time-series data points and error records emit standard ISO 8601 timestamps (e.g.
2026-09-03T12:51:06+00:00). - Client-Side Localization: The Admin UI parses these timestamps using the browser's native JavaScript
Dateengine, displaying accurate local time (e.g.14:51in UTC+2/CEST) in chart hover tooltips and diagnostic tables. - Near Real-Time Telemetry: With auto-refresh enabled (3s interval), requests made in the chat or via API immediately reflect on the rightmost bucket of the trendlines.
Structured Logging
All core infrastructure logs use structlog to produce structured logs (JSON or human-readable text). The configuration is managed by src/observability/logging.py.
Formatting
The format is selectable via AION_LOG_FORMAT:
| Value | Format | Use |
|---|---|---|
json (default) | JSON on stdout | Production, ingestion in Loki/Elastic |
text | ConsoleRenderer (human-readable) | Development |
OTel Context Injection
Each log automatically includes the current trace_id and span_id, obtained from the active OTel span. Span attributes (aion.profile, aion.session_id, tool.name, etc.) are also injected into the logs.
The add_otel_context processor in logging.py reads the current span and adds the fields:
{
"event": "tool execution complete",
"trace_id": "0000000000000000abcdef1234567890",
"span_id": "0000000012345678",
"aion.profile": "tecnico-senior",
"aion.session_id": "sess-abc123",
"level": "info"
}
OTLP Log Export
When AION_OTEL_ENABLED=1, logs are also exported via OTLP using BatchLogRecordProcessor + OTLPLogExporter. This allows correlating application logs with traces in the OTel backend.
Logging Configuration
| Variable | Default | Description |
|---|---|---|
AION_LOG_LEVEL | INFO | Logging level (DEBUG, INFO, WARNING, ERROR) |
AION_LOG_FORMAT | json | Format: json or text |
Logs go only to stdout (no file logging). Uvicorn and FastAPI loggers are propagated to the root logger to maintain format consistency.
Opik — Self-Hosted LLM Telemetry
Opik (Comet ML) is a telemetry layer separate from the OTel pipeline, dedicated specifically to LLM tracking, prompt library and session feedback scores. It is designed to be self-hosted and communicate with an Opik instance on an internal network.
Opik Architecture
┌─────────────────────────────────┐
│ Agent Runtime │
│ │
│ ┌──────────────┐ ┌──────────┐ │
│ │ opik_setup.py│ │opik_telemetry.py│
│ │ (SDK init, │ │ (@track │ │
│ │ Prompt Lib) │ │ decorators)│ │
│ └──────┬───────┘ └────┬─────┘ │
│ │ │ │
│ └───────┬───────┘ │
│ │ │
└─────────────────┼─────────────────┘
│ HTTP
▼
┌─────────────────────────────────┐
│ Opik (Comet ML) │
│ │
│ • Prompt Library │
│ • Traces (LLM, tool, skill) │
│ • Threads (multi-turn) │
│ • Feedback Scores │
└─────────────────────────────────┘
Setup (src/observability/opik_setup.py)
The opik_setup.py module manages the initialization of the Opik SDK:
- Checks if
AION_OPIK_ENABLEDis set to1(ortrue/yes/on). If not enabled, it completely bypasses Opik client initialization and setsOPIK_AVAILABLEtoFalse. - Reads
OPIK_URL_OVERRIDEandOPIK_PROJECT_NAMEfrom environment variables. - Automatically sets a placeholder for
OPIK_API_KEYif not configured (required for client validation in self-hosted mode). - Initializes the Opik client with
Opik(). - Exports
OPIK_AVAILABLE(bool) for graceful degradation.
If AION_OPIK_ENABLED=0, the SDK is not installed, or the server is unreachable, OPIK_AVAILABLE becomes False and no requests fail — the application continues normally without attempting connections.
Prompt Library
get_or_create_prompt() manages the versioning of prompt templates:
- Searches for the prompt by name in the Opik Prompt Library
- If it exists and the template has changed → creates a new version (commit)
- If it exists and the template is unchanged → returns the existing prompt
- If it does not exist → creates a new prompt with the provided template
The template uses Jinja-style placeholders (e.g. {{profilo}}, {{user_input}}) that are dynamically formatted at runtime via prompt_obj.format().
Tracking Decorators (src/observability/opik_telemetry.py)
The module provides @track decorators to mark functions as Opik spans of different types:
| Decorator | Type | Function | Description |
|---|---|---|---|
@track(type="llm") | call_llm_api() | Real LLM call with token tracking, temperature, endpoint | |
@track(type="tool") | execute_tool_ricerca() | Tool execution with input/output in the span | |
@track(type="general") | skill_ricerca_db(), skill_calcolo_metrico() | Agent skills as nested spans | |
@track(type="general") | run_agent() | Main root trace of the agent |
Agent Integration
Opik wrappers are applied at runtime in:
src/main.py:1169-1261: Opik wrappers applied tochat_generatorand MCP toolssrc/agent_pipeline.py:987-1015: Opik trace initialization in the agent pipeline
Session Feedback
chiudi_sessione_e_valuta() sends session-level feedback scores to Opik at the end of the interaction. Currently it sends a fixed score of 1.0 for the risoluzione_problema metric, but the design supports future extensions for real feedback (e.g. user thumbs up/down).
OTel vs Opik Distinction
| Aspect | OTel | Opik |
|---|---|---|
| Purpose | Generic infrastructure | LLM-specific |
| Traces | Generic spans (HTTP, DB, tool) | LLM, prompt, skill spans |
| Metrics | Prometheus + OTel push | No metrics |
| Prompt Library | None | Yes, with versioning |
| Feedback Score | None | Yes, at session level |
| Backend | Phoenix, Jaeger, Tempo, etc. | Opik (Comet ML) |
Opik Configuration
| Variable | Default | Description |
|---|---|---|
OPIK_API_KEY | local-self-hosted-placeholder | Opik API key (placeholder if not set) |
OPIK_PROJECT_NAME | AION-Agent | Opik project name |
OPIK_URL_OVERRIDE | http://localhost:5173/api | API endpoint of the self-hosted Opik instance |
Complete Configuration
OTel / Infrastructure Variables
| Variable | Default | Description |
|---|---|---|
AION_OTEL_ENABLED | 0 | Enables OTel tracing (push to collector) |
AION_OTEL_ENDPOINT | http://localhost:4317 | OTLP collector endpoint (gRPC) |
AION_OTEL_PROTOCOL | grpc | OTLP protocol: grpc or http |
AION_OTEL_SERVICE_NAME | aion-agent | Service name in traces/metrics/logs |
AION_OTEL_TRACE_CONTENT | 1 | Captures LLM content in spans (prompt/response) |
AION_OTEL_METRIC_EXPORT_INTERVAL | 5000 | OTel metrics push interval (ms) |
AION_METRICS_ENABLED | 1 | Enables Prometheus exposure on /metrics |
AION_METRICS_PATH | /metrics | Prometheus endpoint path |
AION_LOG_LEVEL | INFO | Logging level |
AION_LOG_FORMAT | json | Log format: json or text |
AION_ENV | dev | Deployment environment (dev, staging, prod) |
Opik Variables (LLM Telemetry)
| Variable | Default | Description |
|---|---|---|
AION_OPIK_ENABLED | 0 | Enables/disables Opik (Comet ML) LLM telemetry tracing (0 = disabled, 1 = enabled) |
OPIK_API_KEY | local-self-hosted-placeholder | Opik API key (placeholder for self-hosted client validation) |
OPIK_PROJECT_NAME | AION-Agent | Opik project name |
OPIK_URL_OVERRIDE | http://localhost:5173/api | API endpoint of the self-hosted Opik instance |
Per-step LLM call audit (local JSON)
Separate from Opik/OTel: when AION_LLM_CALL_AUDIT=1, src/runtime/llm_call_audit.py writes full request/response payloads for each agent LLM step to data/diagnostics/llm_calls/ (override with AION_LLM_CALL_AUDIT_DIR). CLI: python scripts/audit_llm_calls.py. Intended for dev/staging regression on tool-call JSON and vLLM thinking behavior — not a replacement for production trace export.
Code Reference Files
| File | Responsibility |
|---|---|
src/observability/otel_setup.py | OTel init (OpenLit + fallback, traces, dual-export metrics) |
src/observability/metrics.py | 11 dual-export metrics, OTel/Prometheus wrapper classes |
src/observability/logging.py | Structured logging, OTLP log export, OTel context injection |
src/observability/hooks_emitter.py | Hook-based metrics emission (4 hooks) |
src/observability/opik_setup.py | Opik SDK init, Prompt Library, session feedback |
src/observability/opik_telemetry.py | @track decorators for LLM, tool, skill tracing |
src/aion_env.py | Early OTel + Opik env var setup (first import) |
src/main.py | Opik wrappers on chat_generator and tools |
src/agent_pipeline.py | Opik trace initialization in the pipeline |