Abstract: A computation-aware tiered eviction system is proposed for managing key-value (KV) caches during large-language model inference. A major difficulty lies in the fact that standard eviction policies, such as the least-recent-used algorithm, do not consider the differing computational expenses of recalculating distinct KV blocks after their transfer to slower memory. Our method introduces a Computation-Aware Eviction Controller that assigns a score to each KV block according to its attention variance, depth of dependency within the transformer, and temporal locality. A secondary cache backed by an SSD is used for lower-scoring blocks to keep them out of high-bandwidth memory, whereas high-scoring blocks are kept in high-bandwidth memory. A lightweight runtime profiler updates these scores with minimal overhead. Furthermore, a predictive prefetch mechanism retrieves blocks from the SSD that are likely to be needed before they become necessary. The system achieved a 1.8x improvement in sustained throughput for sequences exceeding 32K tokens compared to a baseline LRU policy, without degrading generation quality. This work shows that embedding computational expense into cache handling is a feasible and efficient method to resolve memory constraints in extended-context inference.

Key words: computation-aware KV cache management, long-context LLM inference, hybrid HBM-SSD memory architecture, attention variance-based eviction, predictive KV cache prefetching, transformer memory optimization