ResearchMedium importance
TripleKV, three-layer cache for AI agents, cuts wait time for the first token by up to 49%
An article on arXiv introduces TripleKV, a system that stores the KV cache for AI agents across three layers: GPU, DRAM memory, and SSD. A predictor estimates when each tool call will finish and prefetches blocks from the SSD to DRAM, without reserving scarce GPU memory or interfering with other inference requests. In data, web, and coding agents, the fraction of requests that need to read from the SSD on the critical path dropped from 45% to 76% to under 10%. The time to first token at the 95th percentile decreased by up to 49% compared to vLLM with DRAM and SSD layers, and was 1.5 to 7 times shorter than recent systems designed for agents.
Published in edition no. 41 of Noticiário do Joãozinho ·
Translated by Joãozinho AI from the original Brazilian Portuguese edition. Facts, numbers and sources are the same as in the original; quotes, video titles and source names stay in their original language. Read the original in Portuguese
Original source
https://arxiv.org/abs/2611.25112