Batbox Knowledge Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Cache-aware compaction ======

**Source snapshot:** 2026-10-25 18:22 UTC, commit ''bc8adfe6a032429c9a3726d81e912e91ceedb6e3''. Live availability follows the pinned rollout receipt.

==== The unchanged budgets ====

The main exact compaction trigger remains **120000 tokens**, main maximum output **50000 tokens**, and context window **200000 tokens**. This feature changes the construction of an eligible summary request; it does not lower those settings. Summary work keeps its existing **90000-token input cap** and **20000-token output allowance**.

==== When a summary can reuse a prefix ====

The conversation retains evidence from its last successfully completed main request. An eligible fork keeps the same model/endpoint/template identity, system text, ordered tool schemas, request options and unchanged history prefix, including raw reasoning fields. The head ends at a complete tool-group boundary. One summary instruction follows that prefix.

The endpoint must advertise supported vLLM prefix caching, and serving-tokenizer IDs must prove the unchanged head is a prefix of both parent and summary request. Equal-looking JSON or a token count alone is insufficient. Tokenization itself does not warm a generation cache.

For example, an edited prompt or refreshed tool catalog can invalidate the previous snapshot. A resumed session begins without warm evidence until a new main request finishes. Unsupported metrics, missing token IDs, changed prefixes or an oversized head use the existing bounded standalone summary path.

==== What the receipts mean ====

Observations include ''eligible'', ''fork_sent'', ''common_prefix_tokens'', a fallback reason and ''cache_hit_verified=false''. An eligible sent fork is an opportunity for reuse, not proof of a cache hit: eviction, backend restart and scheduling can still require prefill. No GPU KV restoration or free-input guarantee follows from these fields.

A summary must be nonempty, visibly complete, have closed thinking and finish with ''stop''. Tool intent, length stops, cancellation and errors do not replace source history or create a successful checkpoint. The utility has no tool dispatcher. Once a fork has generated and failed, its error is surfaced rather than hidden behind another automatic generation.

==== Sources and memory ====

Source: ''docs/cache-aware-compaction.md'', ''src/conversation/Compactor.cpp'', ''src/conversation/Conversation.cpp'', and ''tests/integration/test_compactor.cpp'' at ''bc8adfe6a032429c9a3726d81e912e91ceedb6e3''.

A memory or wiki note can record the checkpoint/receipt, source commit, observation time and fallback reason. Keep “eligible prefix” separate from measured cache reuse or latency. Preserve cited source history and its original dates when summarizing; later summary creation is a different timestamp. See [[memory:provenance-time|source and time]].

**Installed code observation:** 2026-10-25T18:36:38.457052+00:00, native release ''d6949e909ffa08af7951''. This page describes the verified repair rollout; live activity and later changes need fresh observations.

[[https://github.com/Xananthium/batbox/blob/bc8adfe6a032429c9a3726d81e912e91ceedb6e3/docs/wiki/seed/runtime/cache-aware-compaction.txt|Pinned source snapshot]]