Skip to main content

LongMemEval-V2 + Mnemos

This page describes how AION runs the public LongMemEval-V2 (LME) benchmark against Mnemos, its long-term memory (LTM) backend. It is a stress test that loads a large volume of recorded agent sessions into memory and checks whether the right facts can still be recalled; it is for developers who want to see how Mnemos behaves at scale. For everyday regression checks use Mnemos Bench instead.

LongMemEval-V2 evaluates whether a memory system can internalize long web-agent trajectories and answer environment-specific questions.

Scope of this benchmark

This runner tests Mnemos project-scope FTS/hybrid recall only — not the full AION memory stack (SQL Query Memory, KHUB RAG, Agent DB). LME trajectories are a poor fit for Mnemos post-it notes; treat this as an integration stress test, not a production Mnemos quality score. For Mnemos-native validation use mnemos-bench.md.

CLI only

Benchmarks are not exposed in the admin UI. Run via python -m src.benchmarks.cli (see overview.md).

AION adaptation (v1)​

AspectChoice
TierLME-V2-Small (~100 shared trajectories, ~25M text tokens)
ModalityText observations only (screenshots deferred)
Memory backendMnemos (ltm_notes + FTS wake/recall)
Isolationtenant=benchmark, user=lme_v2_<run_id>, project=lme_<run_id> (per-run scope)

Pipeline​

  1. Prepare — download questions.jsonl, trajectories.jsonl, haystacks/lme_v2_small.json from HuggingFace
  2. Ingest — denoise accessibility trees, detect cross-run UI chrome boilerplate, chunk at line boundaries (~70k notes for 100 trajectories), bulk-insert into Mnemos (insert_notes_bulk)
  3. Query — per question: Haystack agent with profile benchmark_memory calls memory_recall iteratively (scope=project), answers in \boxed{}
  4. Score — the dataset's own eval_function per question; aggregates by ability, eval_function family, and latency p50/p95

Five memory abilities​

  • static — static state recall
  • dynamic — dynamic state tracking
  • workflow — workflow knowledge
  • gotchas — environment gotchas
  • premise — premise awareness

Scoring: the dataset's eval_function​

Every row in questions.jsonl declares how it must be graded, e.g. norm_phrase_set_match|lower=true|normalize_hyphen=true|strip_punct=true|separators=,;|require_non_empty=true. src/benchmarks/longmemeval_v2/scoring.py implements those semantics; ad-hoc substring matching is not used, because it produces both false positives (a partial phrase set scoring 1.0) and false negatives (abstention answers scoring 0.0 regardless of quality).

FamilyRows (of 451)Semantics
norm_phrase_set_match200Split on the declared separators, normalize each phrase, compare as a set — a partial answer fails
llm_abstention_checker128The question carries a false premise; an LLM grader checks the model rejects it. Answering UNKNOWN is incorrect
mc_choice_match68Single letter A–H; 11 of these rows use true/false instead and fall back to literal comparison
llm_gotchas_checker28LLM grader for semantic equivalence of free-form advice
norm_phrase_set_match_ordered26As above but order-sensitive
mc_choice_set_match1Set of letters (e.g. A,B,F)

Normalization applies NFKC, unifies Unicode dashes, folds hyphens to spaces, drops punctuation, collapses whitespace, and lowercases — symmetrically on both the reference and the prediction.

Two properties are asserted over all 451 gold answers in src/test/test_lme_v2_official_scoring.py: every gold scores 1.0 against itself, and a nonsense answer never scores 1.0.

Multimodal rows

29 of the 451 questions ship an image. This runner is text-only; with AION_LME_V2_SKIP_IMAGE_QUESTIONS=1 (default) they are reported as skipped_image and excluded from accuracy denominators.

Agent profile​

Use benchmark_memory (in config_std/profiles/benchmark_memory.yaml): Mnemos native tools only (mcp_servers: []), readonly recall during eval (AION_MNEMOS_READONLY_TOOLS=1). Scope binding per run:

  • AION_DEFAULT_TENANT_ID=benchmark
  • user_id=lme_v2_<run_id> (via eval_scope)
  • project_slug=lme_<run_id> (via sql_query_project on AgentPipeline.run_stream)

apply_benchmark_isolation_env() disables LTM extraction, wake injection, skill distillation, and STM pollution for benchmark turns.

Environment variables​

VariableDefaultDescription
AION_BENCHMARK_DATA_DIRdata/benchmarksRoot for datasets and runs
AION_BENCHMARK_VERBOSE1Echo detailed benchmark logs to terminal + debug.jsonl
AION_LME_V2_TIERsmallsmall (medium = future)
AION_LME_V2_TEXT_ONLY1Skip screenshot archives
AION_LME_V2_JUDGE_PROFILEaion_stdProfile for final QA step
AION_LME_V2_QA_DISABLE_REASONING1Set enable_thinking=false for Qwen3/vLLM judge calls
AION_LME_V2_SKIP_WAKE1Skip Mnemos wake block during QA (reduces unrelated trajectory noise)
AION_LME_V2_UI_LABEL_LIMIT80Max non-menuitem labels per state (menuitems always kept)
AION_LME_V2_NOTE_MAX_CHARS480Cap each ingested note (Mnemos CONTENT_MAX_CHARS is 500)
AION_LME_V2_RECALL_LIMIT20FTS notes merged per question (boosted query + full text)
AION_LME_V2_LLM_JUDGE1Grade llm_abstention_checker / llm_gotchas_checker rows with an LLM; 0 scores them 0.0 offline
AION_LME_V2_JUDGE_MAX_TOKENS16The grader only emits CORRECT / INCORRECT
AION_LME_V2_JUDGE_TIMEOUT60Per-verdict timeout (seconds)
AION_LME_V2_MAX_TRAJECTORIES100Haystack trajectories to ingest
AION_LME_V2_MAX_STATES_PER_TRAJ00 = all states (full retention)
AION_LME_V2_MAX_TREE_CHARS00 = no per-state tree cap
AION_LME_V2_MAX_CHUNKS_PER_TRAJ00 = unlimited notes per trajectory
AION_LME_V2_BOILERPLATE_THRESHOLD0.6Fraction of states for UI chrome detection
AION_LME_V2_INGEST_BATCH_SIZE500Notes per insert_notes_bulk transaction
AION_LME_V2_COMPRESS_SCOPE0Skip digest compression after ingest (recommended)
AION_LME_V2_AGENT_PROFILEbenchmark_memoryHaystack profile for query phase
AION_LME_V2_SKIP_IMAGE_QUESTIONS1Skip multimodal rows (report skipped_image)
AION_MNEMOS_READONLY_TOOLS1Expose only memory_recall during benchmark

Mnemos knobs are snapshotted in each run's config.json (AION_LTM_WAKE_MAX_ROWS, AION_MNEMOS_RECALL_LIMIT, etc.).

CLI​

# Prepare dataset
python -m src.benchmarks.longmemeval_v2.prepare

# Run benchmark (agent + full ingest defaults applied at runtime)
python -m src.benchmarks.cli run \
--benchmark longmemeval_v2_small \
--profile benchmark_memory \
--config-json '{"max_questions":8,"max_trajectories":100}'

Staged validation runs​

Compare accuracy_by_eval_function in metrics.json after each milestone:

  1. max_questions=8 — smoke after ingest + FTS + agent wiring
  2. max_questions=50 — mid-scale
  3. Full 451 questions — omit max_questions in config JSON

Limitations​

  • Text-only ingest (29 image questions skipped by default)
  • Orchestration sidebar tools are still merged by build_all_tools even with mcp_servers: []
  • Full Small tier ingest ~8 min + agent query cost scales with question count

Results​

Each run writes under data/benchmarks/runs/<run_id>/:

FileContents
run.logAll [phase] lines (ingest + query + done)
debug.jsonlFull per-case trace: prompts, raw LLM, evidence, score_reason
per_case.jsonlCompact results (+ score_reason)
config.jsonLLM endpoint/model, scope, limits
metrics.jsonAggregates + accuracy_by_eval_function + image_cases + score_reasons map

score_reason tells you why a case failed: phrase_set_under_answered (missed a phrase), phrase_set_over_answered (invented one), phrase_list_mismatch (right phrases, wrong order), mc_mismatch, no_choice_letter_found, llm_answered_unknown, llm_judge_incorrect, llm_judge_error. The per-case score_debug in debug.jsonl also carries missing_phrases / extra_phrases and, for reference, the old heuristic verdict under heuristic.

Document official runs under docs/benchmarks/results/ (JSON + REPORT.md exported from data/benchmarks/runs/<id>/).