EPU-HEKV investigates hierarchical external memory for frozen large language models (LLMs) through two complementary augmentation stages. The first stage, referred to as the span/KV lane, separates historical key-value (KV) states from the compute device. During streaming prefill, KV states are organized into spans and offloaded to RAM/NVMe storage. A real attention scorer with rotary positional embeddings (RoPE) is then used to select a fixed-size working set, while the active state is strictly bounded to 1,664 tokens.
The second stage, referred to as the entity–relation–state (ERS) lane, changes the management unit from spans to entities enriched with relational and temporal-state information. Selected spans are decoded back into text, filtered to remove noise, and re-encoded into a clean prompt before subsequent processing.
On Qwen2.5-0.5B-Instruct in FP16 precision, the span/KV lane maintains a peak decode memory footprint of approximately 55 MB as the context length increases from 4K to 53K tokens, whereas the full-KV approach increases from 117.3 MB to 1,454.6 MB. The entity + re-encode + ERS pipeline achieves 90% first-pass accuracy, with 10% hallucination cases caused by insufficient evidence, using 416 context tokens, 124.22 MB of peak additional VRAM, and 0.03 MB of resident state on a 10-sample benchmark.
Overall, this work clarifies the distinction between active state, external storage, and retrieval cost. The bounded-memory guarantee applies only to the active/resident state and does not imply bounded total storage, bandwidth consumption, or retrieval computation

External memory, KV cache, large language models (LLMs), entity–relation–state (ERS), long-context, retrieval