Hugging Face launches Storage Buckets
hugging face introduced storage buckets, an s3-like object storage solution on the hub backed by xet. this feature offers mutable storage, deduplication for ml artifacts, and pre-warming capabilities for aws and gcp. users can manage buckets via the hf cli or python.
Hugging Face released Storage Buckets, a mutable object storage solution built on the Xet backend. Unlike Git-based repos, Buckets handle non-versioned data for ML workloads like training checkpoints and agent traces, utilizing chunk-based deduplication to reduce bandwidth and storage costs. Enterprise billing is based on deduplicated footprint, and the feature includes pre-warming for data locality on AWS and GCP.