Open SourceMedium importance
Valkey 9.2 Brings Distributed Caching for Inference, Promising to Cut Spending by Up to 40% and First-Token Latency by Up to 80%
Valkey, an open-source in-memory database, released version 9.2 on 11/19 featuring distributed key-value caching for AI inference: inference engines reuse tensors from already-processed prompts instead of recalculating them. According to the project's announcement, the shared cache can reduce inference costs by up to 40% and first-token latency by up to 80% on GPUs the company already owns. Multiverse Computing handled 50% more requests on the same GPUs. In the project's synthetic test, with 100 simultaneous conversations ranging from 12,000 to 28,000 context tokens, median first-token latency dropped by 23% in normal traffic and 95% at peak, while caching added roughly 15% to the GPU bill.
Published in edition no. 40 of Noticiário do Joãozinho ·
Translated by Joãozinho AI from the original Brazilian Portuguese edition. Facts, numbers and sources are the same as in the original; quotes, video titles and source names stay in their original language. Read the original in Portuguese
Original source
https://www.unite.ai/valkey-9-2-launches-with-distributed-kv-caching-for-ai-inference/