TPU Sync is an open C++ library for moving KV caches and model weights on Cloud TPUs. It asynchronously transfers data between TPU HBM, host RAM and remote nodes, reducing Python GIL involvement and intermediate copies while overlapping inference computation. The repository lists validation on v7x and v6e and recommends pinning a tag or the verified commit in lkg.version because APIs change rapidly. Internal throughput figures do not imply the same latency benefit for every model, and JAX and library compatibility must also be checked.
Back to Tech
Google TPU Sync provides open KV-cache and weight-transfer infrastructure
When memory movement limits inference, pin a verified revision and benchmark the actual workload.