Apple internally uses a system known as “Minions,” a “virtual GPU cloud” made up of idle Macs, to help manage workloads, as described in a recent arXiv paper published by its researchers. The system helps take stalled AI jobs, some that once had 10-plus-hour company GPU queues, and distributes them to Macs on the network, with resulting wait times down to as little as minutes. A request from a user on the company’s Slack can get the ball rolling, although the researchers observed a common concern about teams’ data running on devices outside their group.

The paper, also shared on Apple’s Machine Learning Research site, describes the system as having a “Minion Server,” a “Minion Executor,” and a “Minion Watchdog.” The system relies specifically on Apple Silicon for AI inference, the only type of work it does. There are two kinds of jobs: embeddings with open-source Hugging Face transformer models, and text generation from large language models through Ollama, which is installed on each participating Mac.

The Minion Server works by splitting submitted tasks into subtasks, stored in a PostgreSQL database. Then, on each Mac, a Minion Executor is run to pull the next subtask over REST APIs; there is no central scheduler in this “pull-based execution model.” The Mac reads inputs from Amazon S3, grabs the model if needed, runs the subtask on its own GPU, then returns the result to S3.

Failures are caught by the Minion Watchdog, which tracks the heartbeats that Macs running work send every five minutes.

Source link