Benchmarks & Evaluation
Benchmarks are repeatable tests that score how well a part of AION performs, so a change can be checked against a known baseline before it ships. This page lists the available benchmarks, where their datasets live and how to run them from the command line; it is for developers working on memory, retrieval or the agent loop.
Benchmarks are developer tools only — they validate memory retrieval, agent regressions, and integration paths during development. They are not exposed in the admin UI or customer-facing surfaces.
Layout
| Path | Purpose |
|---|---|
src/benchmarks/ | CLI harness, runners, metrics |
config_std/benchmarks/ | Committed micro datasets |
data/benchmarks/ | Downloaded datasets and run artifacts (gitignored) |
docs/benchmarks/ | Documentation (this folder) |
Available benchmarks
| ID | Purpose | Typical runtime |
|---|---|---|
mnemos_bench | Mnemos recall (smoke: 6, full: 81, adversarial: 53) | seconds–minutes |
general_agent | Agent pipeline smoke via JSON cases | seconds |
agent_smoke_tests | End-to-end multi-scenario agent diagnostics (CLI & Admin UI) | seconds |
longmemeval_v2_small | Integration stress test (Mnemos-only ingest + agent) | minutes–hours |
See agent-smoke-tests.md for the end-to-end Agent Smoke Tests suite and diagnostics dashboard. See mnemos-bench.md for the primary Mnemos validation suite.
The full dataset is a regression guard (expected 100%). The adversarial dataset is a scorecard for memory quality: after hardening it reaches ~76% FTS-only and ~87% hybrid (53 cases). See mnemos-bench.md and Mnemos Architectural Hardening.
LongMemEval-V2 is documented in longmemeval-v2.md — note it measures Mnemos project-scope retrieval, not the full AION memory stack (SQL QM, KHUB RAG, Agent DB).
CLI
# Mnemos recall micro-benchmark (recommended for CI)
python -m src.benchmarks.cli run \
--benchmark mnemos_bench \
--dataset config_std/benchmarks/mnemos_recall.json
# Mnemos full evaluation (81 cases, accuracy_by_category in metrics.json)
python -m src.benchmarks.cli run \
--benchmark mnemos_bench \
--dataset config_std/benchmarks/mnemos_recall_full.json
# Mnemos adversarial scorecard (53 cases — see mnemos-bench.md)
python -m src.benchmarks.cli run \
--benchmark mnemos_bench \
--dataset config_std/benchmarks/mnemos_recall_adversarial.json
# Hybrid recall is on by default (AION_MNEMOS_EMBEDDING_RECALL=1); requires embedding service
python -m src.benchmarks.cli run \
--benchmark mnemos_bench \
--dataset config_std/benchmarks/mnemos_recall.json
# General agent smoke
python -m src.benchmarks.cli run \
--benchmark general_agent \
--dataset config_std/benchmarks/smoke_agent.json \
--profile aion_std
# LongMemEval-V2 (integration / stress test)
python -m src.benchmarks.cli prepare-lme
python -m src.benchmarks.cli run \
--benchmark longmemeval_v2_small \
--profile benchmark_memory \
--config-json '{"max_questions":8}'
Optional flags:
--run-id <id>— fixed run id (otherwise auto-generated)--config-json '{"max_cases":3}'— limit cases (mnemos_bench) or questions (LME)
Artifacts
Each run writes under data/benchmarks/runs/<run_id>/:
| File | Contents |
|---|---|
run.log | Phase lines (ingest / query / done) |
per_case.jsonl | Per-case scores |
metrics.json | Aggregates (accuracy_overall, latency, etc.) |
debug.jsonl | Verbose trace (LME runs) |
REPORT.md | Human summary (LME runs) |
Official LME results can be exported to docs/benchmarks/results/.
Mnemos hybrid recall
When AION_MNEMOS_EMBEDDING_RECALL=1 (default since hardening), memory_recall runs FTS + embedding search and merges results with RRF, then applies recency and importance boosts. If the embedding service is unreachable or unconfigured, recall falls back to FTS-only automatically.
Reuses AION_EMBEDDING_URL, AION_EMBEDDING_MODEL, AION_EMBEDDINGS_PROVIDER (same as SQL Query Memory). upgrade-aion migrates legacy installs from EMBEDDING_RECALL=0 to 1.
| Variable | Default | Description |
|---|---|---|
AION_MNEMOS_EMBEDDING_RECALL | 1 | Enable hybrid FTS+embedding recall |
AION_MNEMOS_EMBED_ON_BULK | 1 | Embed notes during insert_notes_bulk (set 0 for huge ingests) |
AION_MNEMOS_EMBEDDING_MIN_SCORE | 0.25 | Min cosine similarity for embedding candidates |
AION_MNEMOS_EMBEDDING_SCAN_LIMIT | 300 | Max notes scanned per scope for embedding search |
AION_MNEMOS_HYBRID_CANDIDATE_MULT | 3 | Candidate pool multiplier before RRF |
AION_MNEMOS_RANK_HALF_LIFE_DAYS | 90 | Recency decay half-life in final ranking |
AION_MNEMOS_RANK_W_RECENCY | 0.3 | Recency boost weight |
AION_MNEMOS_RANK_W_IMPORTANCE | 0.2 | Importance boost weight |
See Mnemos Architectural Hardening for dream cycle, entity index, and provenance variables.
Optuna tuning
Hyperparameter search remains CLI-only:
python -m src.optimizer.cli --trials 20 --study-name aion-v3-tuning
Legacy eval CLI
python -m src.eval.cli --dataset config_std/benchmarks/smoke_agent.json --threshold 0.8