CONNECT WITH US
Tech

Tech

Homelabber’s second-gen Optane build uses two RTX 3090s to run 2.8-trillion-parameter Kimi AI model at 3.11 tokens per second

Tom's Hardware logo

Published on

Add as a preferred source on Google
Homelabber’s second-gen Optane build uses two RTX 3090s to run 2.8-trillion-parameter Kimi AI model at 3.11 tokens per second

A homelabber detailed improvements to the second-gen Optane AI build in a follow-up post on Reddit, showing the addition of two RTX 3090s to run the 2.8-trillion-parameter Moonshot Kimi K3 model at 3.11 tokens per second. Transferring almost half the model onto a RAM drive added a significant 41% to generation speed and 26% to prompt evaluation. The extra fans are still needed, though, as the persistent Optane memory gets very hot.

That speed is not the most impressive, but it comes with 16.8 tokens per second for prompt evaluation, with 45 of 94 shards, or 414 GB out of 865 GB, living on a RAM drive. Such an arrangement is not new to enthusiasts who would set aside system memory for extra-fast performance in a pseudo-non-volatile manner — it’s still volatile, but appears like a disk — although not everyone has that much RAM sitting around free these days. With both GPUs but no RAM drive, the build ran at 2.20 tokens per second for generation and 13.3 for prompt evaluation.

45 of 94 shards (414 GB of 865 GB) in a tmpfs RAM drive, the rest in Optane

5% of the 865 GB model file. That said, the model is a mixture-of-experts (MoE) type, where its weights are split into multiple “experts” with only a few run per token. Any given token will have “routed” experts selected for it, while “shared” experts run on every token.



Source link

Disclaimer

We strive to uphold the highest ethical standards in all of our reporting and coverage. We TheMorningPulse.fyi want to be transparent with our readers about any potential conflicts of interest that may arise in our work. It's possible that some of the investors we feature may have connections to other businesses, including competitors or companies we write about. However, we want to assure our readers that this will not have any impact on the integrity or impartiality of our reporting. We are committed to delivering accurate, unbiased news and information to our audience, and we will continue to uphold our ethics and principles in all of our work. Thank you for your trust and support.