CONNECT WITH US
AI & Deeptech

AI & Deeptech

180B Parameters, 32GB of RAM: Pushing MoE Inference to the Limit with Transformers

Hugging Face logo

Published on

Add as a preferred source on Google
180B Parameters, 32GB of RAM: Pushing MoE Inference to the Limit with Transformers

Streaming MoE experts directly from SSD gets within 6% of Strata on token generation.

I've been trying to rely less on online models and run bigger ones locally. At home I have a PC with two RTX 4090s (let's call it Orthrus - the two-headed hound of Greek mythology), and for work I have a MacBook M5 with 32G of unified memory.

Orthrus runs on Windows 11, because my kids play games all the time. I had a dual boot with Ubuntu, but their games ended up taking most of the two terabytes of disk.

I am also a huge fan of RMD (rooms moving development): one hour in the kitchen sipping my coffee, two hours at my office desk, then the sofa. A fixed desktop tower is not for me.

My first experiment was to run Qwen 3.8 27B with llama.app on Orthrus and use it from my laptop through pi. That model does the coding tasks, an online model like Opus does the orchestration, and the second RTX runs a FIM model for auto-completion in Neovim.

When I talked about this on social networks, I got very strong feedback: use Strata with Qwen3.8-Flash-Next.

Strata is an inference engine for MoE models like Qwen. Only the hot experts stay in VRAM, the rest goes to RAM, or even to disk. Experts are 90%+ of a MoE model's weights, so a 180B model like this one can run on a GPU with much less memory. Like my RTX 4090!


Source link

Disclaimer

We strive to uphold the highest ethical standards in all of our reporting and coverage. We TheMorningPulse.fyi want to be transparent with our readers about any potential conflicts of interest that may arise in our work. It's possible that some of the investors we feature may have connections to other businesses, including competitors or companies we write about. However, we want to assure our readers that this will not have any impact on the integrity or impartiality of our reporting. We are committed to delivering accurate, unbiased news and information to our audience, and we will continue to uphold our ethics and principles in all of our work. Thank you for your trust and support.