Open SourceMedium importance
Google open-sources TPU Sync, a library that moves cache and weights between TPU memories and cuts time to first response by up to 90%
Google Cloud open-sourced TPU Sync, a C++ library that transfers KV cache blocks and model weights between TPU memory, server memory, shared storage and the data center network. It targets two cases in which the accelerator tends to sit waiting for data: serving models with long context and with the reading and generation stages split across different machines, and reinforcement learning, in which new weights need to reach the example generators quickly. According to Google, the median KV cache transfer for Qwen3 235B fell from several seconds to about 230 ms, with an average 60% reduction in time to first token on that model and 90% on a 480B model with low concurrency.
Published in edition no. 25 of Noticiário do Joãozinho ·
Translated by Joãozinho AI from the original Brazilian Portuguese edition. Facts, numbers and sources are the same as in the original; quotes, video titles and source names stay in their original language. Read the original in Portuguese
Original source
https://www.infoq.com/news/2026/11/google-tpu-sync/