Skip to main content

Mnemos Bench

Mnemos is AION's long-term memory (LTM); Mnemos Bench is the test suite that checks whether it returns the right notes for a query. It runs fixed datasets of notes and questions through keyword (FTS, full-text search) and hybrid (keyword + embeddings) recall and reports a score, so retrieval changes can be compared against a baseline. Read it if you change Mnemos retrieval or tune its settings.

Mnemos Bench validates what Mnemos is designed for: short post-it notes, scoped recall (user / project), and hybrid retrieval — not full episodic memory over 100k trajectory chunks.

Use this benchmark in CI and during FTS/embedding tuning. For integration stress tests over large corpora, see longmemeval-v2.md.

Architectural context and phase-by-phase improvements: Mnemos Architectural Hardening.

What it measures​

AxisCases
FTS keyword hitDiscriminative terms in note content
Phrase queryQuoted phrases + portal vocabulary
Scope isolationUser scope does not leak into project scope
Semantic paraphraseHybrid recall (prefer_hybrid: true)
Short tokensClass, Price, State not dropped by token ranking
Noise rejectionBoilerplate terms do not dominate results

Dataset​

FileCasesUse
config_std/benchmarks/mnemos_recall.json6CI smoke / quick regression
config_std/benchmarks/mnemos_recall_full.json81Regression guard, expected 100%
config_std/benchmarks/mnemos_recall_adversarial.json53Weakness tracker + post-hardening scorecard

Regenerate after editing generators:

python scripts/build_mnemos_recall_full.py
python scripts/build_mnemos_recall_adversarial.py

Categories (full dataset)​

CategoryCasesWhat it tests
fts_keyword10Discriminative keyword retrieval
fts_phrase8Quoted phrases and multi-word entities
scope_isolation8User vs project scope must not leak
semantic_paraphrase12Hybrid recall (prefer_hybrid: true)
short_token6Short discriminative tokens (State, MFA, v2)
noise_rejection6Boilerplate must not dominate results
disambiguation8Target note among 6–8 similar notes
numeric_id6Tickets, ports, IBAN, build numbers
url_context6traj/url/action patterns from agent memory
dense_corpus6Target buried in 20+ distractor notes
multi_hit5Multiple expected phrases in top-k

Each case:

  1. Inserts setup_notes into an isolated scope (tenant=mnemos_bench)
  2. Calls recall(query, limit=recall_limit)
  3. Scores 1.0 if expected_substrings appear in top results (and optional forbidden_substrings absent)

Adversarial suite​

The adversarial dataset targets capabilities that are hard to exercise with keyword-only cases. After the hardening programme it serves as a scorecard: rising scores mean memory quality is improving.

Categories​

CategoryCasesWhat it tests
alias_coref8Target identified by alias (k8s → Kubernetes), with sibling distractors
true_paraphrase8Query shares zero content tokens with the target (assertion-enforced)
temporal_validity5Superseded values must not surface as current
as_of_query3recall(as_of=…) must return facts valid at a past date
deletion_completeness2Hard-deleted notes must not appear in recall or wake
contradiction5Two active contradictory notes — ranking must prefer the right one
recency_rank4Only created_at differs between matching notes
importance_rank4importance 5 must outrank importance 1
cross_scope5Project note must survive a crowded user scope
scale_recall4Target under 400–600 filler notes
precision_noise5Filler sharing only stopwords must not pollute results

Current scores (post-hardening)​

ModeOverallKey categories
FTS-only (AION_MNEMOS_EMBEDDING_RECALL=0)40/53 (75.5%)precision_noise 5/5, cross_scope 5/5, recency_rank 4/4, true_paraphrase 0/8
Hybrid (default, embeddings on)46/53 (86.8%)true_paraphrase 8/8, alias_coref 7/8, precision_noise 0/5

Hybrid mode requires AION_EMBEDDING_URL + AION_EMBEDDING_MODEL configured. Use AION_MNEMOS_EMBED_ON_BULK=1 when running the adversarial suite with hybrid so filler notes receive vectors.

Trade-off: hybrid recall improves paraphrase and alias resolution but can surface semantically similar filler notes (precision_noise regression). See known trade-offs.

Regression guards​

Fixed defects are covered by normal (non-xfail) tests in src/test/test_mnemos_known_gaps.py:

  1. Hard delete purges covering digests and wake skips stale digest blocks.
  2. Cross-scope recall merges all scopes with global ranking.
  3. FTS queries exclude English/Italian stopwords from the default path.

Extra dataset fields​

Backward compatible — the 81-case full dataset runs unchanged.

{
"id": "recency_budget",
"category": "recency_rank",
"scope_type": "project",
"setup_notes": [
{ "content": "Budget approved at 120000 EUR", "age_days": 540 },
{ "content": "Budget approved at 180000 EUR", "importance": 5 }
],
"query": "budget approved",
"expected_substrings": ["180000"],
"expect_top_k": 1
}
FieldEffect
setup_notes[].contentNote body (plain string still accepted)
setup_notes[].category / .importanceWritten to the note row
setup_notes[].age_daysBackdates created_at for recency tests
setup_notes[].supersedesIndex of earlier note in the same list to supersede
setup_notes[].valid_from / .valid_toBi-temporal validity (ISO-8601)
as_ofPassed to recall(as_of=…)
expect_top_kExpected substrings must appear in top N; else rank_miss
filler{count, position, template} — bulk noise before/after setup
extra_scope_type / extra_scope_notesSecond scope for cross-scope cases
recall_scope: "across"Uses recall_across_scopes
hard_delete_indexHard-delete a setup note before recall/wake check
wake_forbidden_substringsFail if substring appears in wake output
min_hits: 0Pass on absence of forbidden hits only (deletion cases)

Forbidden substrings are checked across all returned rows; expected substrings inside the expect_top_k window.

Run​

# List available benchmarks
python -m src.benchmarks.cli list

# Smoke (CI)
python -m src.benchmarks.cli run \
--benchmark mnemos_bench \
--dataset config_std/benchmarks/mnemos_recall.json

# Full regression (81 cases, must stay 100%)
python -m src.benchmarks.cli run \
--benchmark mnemos_bench \
--dataset config_std/benchmarks/mnemos_recall_full.json

# Adversarial scorecard (53 cases)
python -m src.benchmarks.cli run \
--benchmark mnemos_bench \
--dataset config_std/benchmarks/mnemos_recall_adversarial.json

# FTS-only baseline (explicit)
AION_MNEMOS_EMBEDDING_RECALL=0 \
python -m src.benchmarks.cli run \
--benchmark mnemos_bench \
--dataset config_std/benchmarks/mnemos_recall_adversarial.json

# Hybrid (default in .env.example; requires AION_EMBEDDING_*)
AION_MNEMOS_EMBEDDING_RECALL=1 AION_MNEMOS_EMBED_ON_BULK=1 \
python -m src.benchmarks.cli run \
--benchmark mnemos_bench \
--dataset config_std/benchmarks/mnemos_recall_adversarial.json

# Subset for quick iteration
python -m src.benchmarks.cli run \
--benchmark mnemos_bench \
--config-json '{"max_cases":2}'

Expected output​

[cli] run_id=bench_c35507bb90
[banner] recall: hybrid FTS+embedding
[result] PASS fts_keyword_hit score=1.0 reason=ok 0.26s hits=['PostgreSQL']
[done] 81/81 passed (100.0%) total 6.9s
[done] by category:
[done] true_paraphrase: 8/8 (100%)
[cli] summary: 46/53 passed (86.8%)

Results: data/benchmarks/runs/<run_id>/ — REPORT.md, metrics.json, per_case.jsonl.

Adding cases​

{
"id": "my_case",
"scope_type": "user",
"setup_notes": ["Note text stored in Mnemos"],
"query": "search terms",
"expected_substrings": ["Note"],
"recall_limit": 5,
"min_hits": 1,
"prefer_hybrid": false
}

Optional scope-leak check:

{
"negative_scope_type": "project",
"negative_notes": ["Unrelated project note"],
"negative_query": "secret token"
}

Relation to production memory​

Mnemos is one layer in AION's memory stack:

STM (chat window)
Mnemos notes (this benchmark)
SQL Query Memory (embedding + FTS on SQL cache)
Project RAG / KHUB (long documents — future native hook)
Database navigation notes (Mnemos project scope; formerly Agent DB / MemPalace)

Mnemos Bench does not replace end-to-end eval of the full stack.