Migrating the Hub from Git LFS to Xet

Source: HuggingFace Date: 2025-07-15 URL: https://huggingface.co/blog/migrating-the-hub-to-xet

Summary

Infrastructure update: HF completes migration of 20PB, 500k+ repositories, and 1M+ users from Git LFS to Xet with backward compatibility via a Git LFS Bridge. Largest individual migrations: RichardErkhov (1.7PB, 25k repos), mradermacher (6.1PB, 42k repos). Peak throughput during migrations: ~300 Gb/s vs 40 Gb/s everyday load. Background migration system triggered by webhooks batches files by size. Rolling out Xet to all users July 2025; open-sourcing Xet protocol planned.

Implications

HF as open-source ML hub. Migrating 20PB without disrupting users (“quietest migration of this magnitude”) is an operational credibility signal — the Git LFS Bridge maintaining full backward compatibility while the underlying storage migrates is the engineering work that enabled zero downtime. Xet’s content-addressed chunk store is now the production storage backend for the Hub.

Open-weights ecosystem health. mradermacher at 6.1PB and 42,000 repos represents the scale of community quantization work that flows through the Hub — this is the GGUF and GPTQ quantization ecosystem that makes open-weights models accessible on consumer hardware. The Xet migration means these high-update repositories (quantized variants constantly being added) will benefit immediately from deduplication savings, since quantized models share large fractions of weights with their base models.