Run AI workloads on any cloud, store on Hugging Face: zero-egress storage with SkyPilot

Source: HuggingFace Date: 2026-07-07 URL: https://huggingface.co/blog/skypilot-hf-storage

Summary

Hugging Face and SkyPilot integrated Hugging Face Storage (built on Xet content-defined chunking) as a first-class SkyPilot backend, letting workloads run on any of 20+ supported clouds, Kubernetes, Slurm, or on-prem clusters while data stays parked on Hugging Face with zero egress fees. Benchmarks cited: ~500 MB/s lazy-mounted model loads, 112-168 MB/s checkpoint writes across AWS/GCP/Lambda, and dedup that moved ~10 MB instead of ~106 MB when appending 10K rows to a 100K-row dataset. List pricing is $12-18/TB/month versus S3’s ~$23/TB plus egress.

Implications

• infra/cost — egress fees are a primary lock-in mechanism for cloud storage; a zero-egress, multi-cloud-attached storage layer erodes that lock-in and pressures hyperscaler storage margins.
• dev-tooling consolidation — Hugging Face extending from a model/dataset hub into compute-orchestration plumbing (via SkyPilot) is part of the broader pattern of ML platforms absorbing adjacent infrastructure layers rather than staying single-purpose.