Skip to main content

Benchmarks & Evaluation

Benchmarks are repeatable tests that score how well a part of AION performs, so a change can be checked against a known baseline before it ships. This page lists the available benchmarks, where their datasets live and how to run them from the command line; it is for developers working on memory, retrieval or the agent loop.

Benchmarks are developer tools only — they validate memory retrieval, agent regressions, and integration paths during development. They are not exposed in the admin UI or customer-facing surfaces.

Layout​

PathPurpose
src/benchmarks/CLI harness, runners, metrics
config_std/benchmarks/Committed micro datasets
data/benchmarks/Downloaded datasets and run artifacts (gitignored)
docs/benchmarks/Documentation (this folder)

Available benchmarks​

IDPurposeTypical runtime
mnemos_benchMnemos recall (smoke: 6, full: 81, adversarial: 53)seconds–minutes
general_agentAgent pipeline smoke via JSON casesseconds
agent_smoke_testsEnd-to-end multi-scenario agent diagnostics (CLI & Admin UI)seconds
longmemeval_v2_smallIntegration stress test (Mnemos-only ingest + agent)minutes–hours

See agent-smoke-tests.md for the end-to-end Agent Smoke Tests suite and diagnostics dashboard. See mnemos-bench.md for the primary Mnemos validation suite.

The full dataset is a regression guard (expected 100%). The adversarial dataset is a scorecard for memory quality: after hardening it reaches ~76% FTS-only and ~87% hybrid (53 cases). See mnemos-bench.md and Mnemos Architectural Hardening.

LongMemEval-V2 is documented in longmemeval-v2.md — note it measures Mnemos project-scope retrieval, not the full AION memory stack (SQL QM, KHUB RAG, Agent DB).

CLI​

# Mnemos recall micro-benchmark (recommended for CI)
python -m src.benchmarks.cli run \
--benchmark mnemos_bench \
--dataset config_std/benchmarks/mnemos_recall.json

# Mnemos full evaluation (81 cases, accuracy_by_category in metrics.json)
python -m src.benchmarks.cli run \
--benchmark mnemos_bench \
--dataset config_std/benchmarks/mnemos_recall_full.json

# Mnemos adversarial scorecard (53 cases — see mnemos-bench.md)
python -m src.benchmarks.cli run \
--benchmark mnemos_bench \
--dataset config_std/benchmarks/mnemos_recall_adversarial.json

# Hybrid recall is on by default (AION_MNEMOS_EMBEDDING_RECALL=1); requires embedding service
python -m src.benchmarks.cli run \
--benchmark mnemos_bench \
--dataset config_std/benchmarks/mnemos_recall.json

# General agent smoke
python -m src.benchmarks.cli run \
--benchmark general_agent \
--dataset config_std/benchmarks/smoke_agent.json \
--profile aion_std

# LongMemEval-V2 (integration / stress test)
python -m src.benchmarks.cli prepare-lme
python -m src.benchmarks.cli run \
--benchmark longmemeval_v2_small \
--profile benchmark_memory \
--config-json '{"max_questions":8}'

Optional flags:

  • --run-id <id> — fixed run id (otherwise auto-generated)
  • --config-json '{"max_cases":3}' — limit cases (mnemos_bench) or questions (LME)

Artifacts​

Each run writes under data/benchmarks/runs/<run_id>/:

FileContents
run.logPhase lines (ingest / query / done)
per_case.jsonlPer-case scores
metrics.jsonAggregates (accuracy_overall, latency, etc.)
debug.jsonlVerbose trace (LME runs)
REPORT.mdHuman summary (LME runs)

Official LME results can be exported to docs/benchmarks/results/.

Mnemos hybrid recall​

When AION_MNEMOS_EMBEDDING_RECALL=1 (default since hardening), memory_recall runs FTS + embedding search and merges results with RRF, then applies recency and importance boosts. If the embedding service is unreachable or unconfigured, recall falls back to FTS-only automatically.

Reuses AION_EMBEDDING_URL, AION_EMBEDDING_MODEL, AION_EMBEDDINGS_PROVIDER (same as SQL Query Memory). upgrade-aion migrates legacy installs from EMBEDDING_RECALL=0 to 1.

VariableDefaultDescription
AION_MNEMOS_EMBEDDING_RECALL1Enable hybrid FTS+embedding recall
AION_MNEMOS_EMBED_ON_BULK1Embed notes during insert_notes_bulk (set 0 for huge ingests)
AION_MNEMOS_EMBEDDING_MIN_SCORE0.25Min cosine similarity for embedding candidates
AION_MNEMOS_EMBEDDING_SCAN_LIMIT300Max notes scanned per scope for embedding search
AION_MNEMOS_HYBRID_CANDIDATE_MULT3Candidate pool multiplier before RRF
AION_MNEMOS_RANK_HALF_LIFE_DAYS90Recency decay half-life in final ranking
AION_MNEMOS_RANK_W_RECENCY0.3Recency boost weight
AION_MNEMOS_RANK_W_IMPORTANCE0.2Importance boost weight

See Mnemos Architectural Hardening for dream cycle, entity index, and provenance variables.

Optuna tuning​

Hyperparameter search remains CLI-only:

python -m src.optimizer.cli --trials 20 --study-name aion-v3-tuning

Legacy eval CLI​

python -m src.eval.cli --dataset config_std/benchmarks/smoke_agent.json --threshold 0.8