CONNECT WITH US
AI & Deeptech

AI & Deeptech

Run AI workloads on any cloud, store on Hugging Face: zero-egress storage with SkyPilot

Hugging Face logo

Published on

Add as a preferred source on Google
Run AI workloads on any cloud, store on Hugging Face: zero-egress storage with SkyPilot

Together with Hugging Face, we've joined the two halves: your models and datasets stay on the Hub, and SkyPilot runs the compute (dev, training, or serving) on whatever cluster has the GPUs. Mount a Hugging Face Bucket or any Hub repo into a SkyPilot job with one hf:// URL and the HF_TOKEN you already have, then launch it wherever capacity is. Hugging Face charges no egress, so reading your data onto those GPUs costs nothing, on any cloud.

SkyPilot tasks already read and write cloud object stores (S3, GCS, Azure, R2, and many more) by mounting them at a local path. Hugging Face Storage now joins that list as store: hf, reached through the hf:// scheme:

That one hf:// scheme covers the whole lifecycle: read the model and dataset from their repos, write checkpoints to a Bucket while you train, publish the finished model back to a repo, and pull it onto inference servers when you serve. Most teams already keep their models and datasets on the Hub, so there is no migration step and no new storage account to create.

MOUNT uses Hugging Face's hf-mount FUSE backend, so a bucket or repo shows up as a local path next to SkyPilot's other FUSE mounts ( gcsfuse , blobfuse2 , rclone , goofys ). The fetching happens at the filesystem layer: when your code issues a read() , the driver pulls just those bytes from the Xet backend, so only the data you actually touch crosses the network, and hf-mount keeps an on-disk cache so repeat reads stay local.


Source link
Related Topics:AI & Deeptech

Disclaimer

We strive to uphold the highest ethical standards in all of our reporting and coverage. We TheMorningPulse.fyi want to be transparent with our readers about any potential conflicts of interest that may arise in our work. It's possible that some of the investors we feature may have connections to other businesses, including competitors or companies we write about. However, we want to assure our readers that this will not have any impact on the integrity or impartiality of our reporting. We are committed to delivering accurate, unbiased news and information to our audience, and we will continue to uphold our ethics and principles in all of our work. Thank you for your trust and support.