Rearchitecting Hugging Face Uploads and Downloads

Source: HuggingFace Date: 2024-11-26 URL: https://huggingface.co/blog/rearchitecting-uploads-and-downloads

Summary

Infrastructure architecture post: HuggingFace is redesigning uploads and downloads around a Content-Addressed Store (CAS) with chunk-level deduplication, replacing the single-bucket Git LFS approach that hits CloudFront’s 50GB file limit. The new system operates three geographic PoPs (us-east-1, eu-west-3, ap-southeast-1) for distributed writes. Oct 2024 snapshot: 130.8TB data transferred from 88 countries in a single day. Projected 12% bandwidth reduction from deduplication. Production deployment targeted end of 2024 / 2025 (this preceded the Xet rollout announced in March 2025).

Implications

HF as open-source ML hub. The 50GB CloudFront limit was a real ceiling for 70B+ model shards — redesigning around CAS removes a structural constraint that was pushing large model uploads toward workarounds. The three-PoP geographic distribution also meaningfully reduces latency for the European and Asia-Pacific communities that contribute ~50% of uploads.

Open-weights ecosystem health. Infrastructure that handles 130TB/day reliably with chunk-level dedup is the kind of platform investment that is invisible when it works but defines the ceiling for the ecosystem when it doesn’t. This is the architectural precursor to the Xet storage migration announced later.