Skip to main content

Context Compaction

When a conversation grows long enough to approach the LLM's token window limit, AION automatically compacts it: old messages are summarized into a single compact block and the originals are deleted from the database, while the most recent messages are preserved intact.

Without compaction, a long conversation would eventually exceed the model's context window and fail or be truncated. This page explains when compaction triggers, what is kept and what is summarized; it is for developers and administrators who tune the related AION_CONTEXT_COMPRESS_* settings or debug a conversation that "forgot" something.

This mechanism has two independent levels that can both activate in the same turn.


Overview​


Level 1 — Pre-Turn Compaction​

Runs before every turn, as part of build_turn_context() in turn_context.py.

Activation threshold​

Env varDefaultMeaning
AION_MODEL_MAX_CONTEXT131072Model total context window (tokens)
AION_CONTEXT_COMPRESS_THRESHOLD0.5Trigger fraction of window (e.g. 65,536 tokens)
AION_CONTEXT_COMPRESS_KEEP_LAST6Messages to preserve intact at the tail
AION_CHAT_MAX_TOKENS8192Output budget subtracted from max prompt
AION_CONTEXT_COMPRESS_ENABLED1Enable/disable pre-turn compaction
AION_CONTEXT_COMPRESS_PERSIST1Persist the compacted block to the DB
AION_CONTEXT_COMPRESS_MAX_ROUNDS3Max in-memory compression rounds

Compaction activates if:

total_tokens >= trigger OR total_tokens >= max_prompt

It also activates unconditionally when force_compact=True (set by the /compact slash command via Redis).

Step-by-step flow​

  1. Token estimation — estimate_full_prompt_tokens() calculates total tokens (messages + system overhead).
  2. Threshold check — compares against trigger and max_prompt or the Redis force-compact flag.
  3. SSE notification — emits context_compacting { active: true } to the client so the UI shows a loading indicator.
  4. DB-first path (preferred):
    • fetch_messages_for_compaction() retrieves all rows from the DB except the last keep_last_n, enriched with timeline_json, reasoning, and tool_name.
    • precompact_flush() writes extracted facts to LTM (MemPalace) before deletion so information is not lost.
    • The LLM generates a summary via summarize_transcript().
    • persist_stm_compaction() executes:
      • DELETE of old messages
      • INSERT of a single compacted block at seq = min_seq_tail - 1
    • _reload_stm_window() fetches the refreshed window from the DB.
  5. In-memory fallback (if DB is empty): compress_until_fits() loops up to max_rounds, replacing the head with a summary and keeping the tail.
  6. SSE notification — emits context_compacting { active: false }.

What is preserved — precompact_flush and LTM​

Before deletion, ltm_orchestrator.precompact_flush() runs extract_and_persist() in batch mode. This:

  • Sends the transcript to the LLM for structured fact extraction (entities, decisions, errors, file names).
  • Writes the extracted facts permanently into MemPalace (the long-term memory store).
  • Optionally calls the mempalace_precompact_flush MCP tool for custom server-side logic.

Individual messages may no longer exist in the messages table after compaction, but the agent can still access extracted information via its LTM retrieval tools.


Level 2 — Mid-Turn Compaction​

Runs inside the agent loop, between an LLM step and the next, triggered by callbacks in agent_pipeline.py.

Triggers​

HookWhenWhat it does
maybe_compact_after_tool()After every tool callTruncates tool output + calls compact_agent_messages_in_place()
maybe_compact_after_reasoning()After every reasoning chunk (if AION_CONTEXT_COMPRESS_MID_TURN_REASONING=1)Updates token estimate + calls compact_agent_messages_in_place()

By default reasoning-triggered mid-turn compaction is off (AION_CONTEXT_COMPRESS_MID_TURN_REASONING=0) so reasoning streams do not block the agent thread.

compact_agent_messages_in_place() — how it works​

Operates directly on the Haystack agent State object (accessed via ContextVar):

  1. Debounce — enforces a minimum interval between consecutive mid-turn compactions (AION_CONTEXT_COMPRESS_MID_TURN_MIN_SEC, default 15 s).
  2. Token check — calculates total = msg_tokens + overhead.
  3. Mid-turn threshold — mid_trigger = max_prompt × threshold_ratio (default 92%).
  4. Guard — if total < mid_trigger AND total < compress_trigger → skip.
  5. Sync guard — if AION_CONTEXT_COMPRESS_MID_TURN_SYNC=0 (default), skip the blocking LLM summarization on the agent thread.
  6. Summarization — when sync is enabled, synchronous LLM call (complete_text_sync(), timeout 90 s) produces a summary of the head.
  7. In-place replacement — data["messages"] = system_msgs + [summary_msg] + tail (summary uses <summary> XML via format_compaction_block()).
  8. Async DB persist — schedules _schedule_db_persist() via asyncio.run_coroutine_threadsafe.
  9. SSE event — emits context_compacting on the turn queue.

With AION_HARNESS_V2_COMPACTION=1, head/tail splits use find_valid_cut_index() so tool call/result pairs are never split.

Tool output truncation​

Before considering mid-turn compaction, maybe_compact_after_tool() always truncates tool output:

cap = AION_TOOL_RESULT_MAX_CHARS (default: 24,000 chars)
head = text[:cap//2]
tail = text[-(cap//4):]
note = "[AION: output {tool} troncato — N characters omitted…]"

Exception: mempalace_* tools and outputs smaller than 800 chars do not trigger compaction (only truncation).

Mid-turn env vars​

Env varDefaultMeaning
AION_CONTEXT_COMPRESS_MID_TURN1Enable mid-turn compaction
AION_CONTEXT_COMPRESS_MID_TURN_RATIO0.92Threshold as fraction of max_prompt
AION_CONTEXT_COMPRESS_MID_TURN_MIN_SEC15Minimum seconds between compactions (debounce)
AION_CONTEXT_COMPRESS_MID_TURN_SYNC0Blocking LLM summarization on agent thread
AION_CONTEXT_COMPRESS_MID_TURN_REASONING0Compact after each reasoning chunk
AION_CONTEXT_COMPRESS_MID_TURN_TIMEOUT90LLM call timeout for mid-turn summary
AION_TOOL_RESULT_MAX_CHARS24000Max chars per tool result before truncation
AION_TOOL_WEB_FETCH_MAX_CHARS6000Lower cap for web_fetch_page tool result in agent context
AION_TOOL_WEB_SEARCH_MAX_CHARS4000Lower cap for web_search tool result in agent context
AION_MECHANICAL_COMPACT_KEEP_TOOLS3Full tool outputs kept during mechanical mid-turn shrink
AION_CONTEXT_RECOVERY1Auto compact + retry on context overflow
AION_CONTEXT_RECOVERY_MAX2Max automatic retries per turn

Manual Trigger — /compact​

The slash command /compact (slash.py):

  1. Writes force_compact:<conversation_id> to Redis with TTL 600 s.
  2. On the next user message, redis_consume_force_compact() reads and deletes the flag.
  3. force=True is passed to _apply_context_compression(), making compaction run unconditionally regardless of token count.

The Compaction Block​

The block inserted into the DB and displayed in the UI has this structure:

[AION COMPACTION — contesto precedente sintetizzato] (N turni precedenti)
- The user asked to implement feature X...
- A bug was found in module Y and resolved...
- File utils.py was created...

The summary language is resolved by resolve_compaction_language(user_id, db_lang) and follows the user's conversation language (Italian, English, Spanish, French, German).

The COMPACTION_MARKER sentinel is used to:

  • Skip already-compacted messages in fetch_messages_for_compaction() (prevents double compaction).
  • Detect if the tail already contains a compacted block in persist_stm_compaction().

How to Verify Compaction Occurred​

1. Backend logs (stdout)​

Look for these lines in sequence in the server terminal:

# Compaction triggered (e.g. by /compact)
CONTEXT ENHANCEMENT DEBUG - force_compact: True
>>> [CONTEXT] session=a846c38a total=19026 msg=370 overhead=18656 trigger=65536 max_prompt=114688 compact=YES

# DB rows read for summarization
context_compress db_start session=a846c38a rows=8 transcript_chars=1447

# After compaction — msg tokens reduced from 370 → 152
>>> [CONTEXT] session=a846c38a total=18808 msg=152 overhead=18656 trigger=65536 max_prompt=114688 compact=no
context_compress db_done session=a846c38a messages=5 total 19026→18808

For mid-turn compaction specifically:

mid_turn_compact session=a846c38a messages 15→8 tokens 85000→12000

2. Chat UI (real-time)​

During compaction the UI receives a context_compacting SSE event and shows a brief "Compacting context…" shimmer on the agent status indicator.

After the turn completes, scroll to the top of the chat: old individual messages are replaced by a single summary block starting with [AION COMPACTION — contesto precedente sintetizzato].

3. SQLite database​

Query data/aion.db directly:

SELECT role, content
FROM messages
WHERE conversation_id = '<your-session-id>'
ORDER BY seq ASC;

You should see the compaction block (role = 'user', content starts with [AION COMPACTION) at a low seq value, followed only by the preserved tail messages.


LTM Extraction and llm_extract.py​

Both levels of compaction (and regular post-turn LTM extraction) call the helpers in llm_extract.py:

FunctionPurposeCaller
complete_json_sync()Synchronous LLM call → returns parsed JSONcomplete_json_async(), mid-turn sync path
complete_json_async()Async wrapper around complete_json_syncltm_orchestrator.extract_and_persist()
complete_text_sync()Synchronous LLM call → returns plain textcontext_compressor.summarize_transcript(), mid-turn compact_agent_messages_in_place()

All three functions use LiteLLMChatGeneratorWrapper exclusively — direct requests.post HTTP calls were removed in favour of the unified LiteLLM adapter to ensure consistent provider routing.


Key Source Files​

FileRole
src/runtime/turn/turn_context.pyPre-turn orchestration, threshold check, SSE
src/agent_pipeline.py_apply_context_compression(), pre-compact flush
src/runtime/turn_compaction.pyMid-turn compaction, tool truncation, ContextVar runtime
src/memory/context_compressor.pyContextCompressor: budgets, thresholds, LLM summary, format_compaction_block
src/data/history_bridge.pyfetch_messages_for_compaction(), persist_stm_compaction()
src/memory/ltm_orchestrator.pyprecompact_flush() → LTM batch save before pruning
src/memory/llm_extract.pycomplete_json_sync/async, complete_text_sync — unified LiteLLM helpers
src/runtime/slash.py/compact command, Redis flag
src/runtime/redis_client.pyredis_set_force_compact / redis_consume_force_compact