CONNECT WITH US
AI & Deeptech

AI & Deeptech

DeepSeek open-sources the inference engine behind its API. And Huawei gets equal billing with Nvidia.

TheNewStack (AI) - ArtificialIntelligence logo

Published on

Add as a preferred source on Google
DeepSeek open-sources the inference engine behind its API. And Huawei gets equal billing with Nvidia.

DeepSeek has open-sourced the inference engine it uses to serve its models in production, a step it said last year wasn’t yet practical.

Dubbed DeepInfer, the framework is designed to let developers run DeepSeek’s V4.1 Flash model on their own infrastructure, with many of the same optimizations the company uses behind its API. Support for future DeepSeek models is also planned.

DeepSeek has, of course, long made its model weights available for developers to download and run using third-party software such as vLLM. But running a model efficiently is a different matter altogether. The software that manages requests, distributes work across chips, and generates responses can have a significant impact on speed, throughput and computing costs. And so DeepInfer essentially gives developers access to DeepSeek’s own approach to those challenges.

“We’re sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more affordable.”

Taking to X on Tuesday, DeepSeek said it was making its internal inference technology available to outside developers in an effort to bring down the expense of running large AI models.

“We’re sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more affordable,” the company writes.

🚀 Introducing DeepInfer, our in-house inference framework, now fully open source!

Powering DeepSeek-V4.1-Flash at scale with high throughput and low latency. We’re sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more…

0 license , DeepInfer includes optimized computing kernels and communication libraries, including DeepSeek’s previously open-sourced FlashMLA , DeepGEMM , and DeepEP , alongside mechanisms for distributing inference across multiple chips.


Source link
Related Topics:AI & Deeptech

Disclaimer

We strive to uphold the highest ethical standards in all of our reporting and coverage. We TheMorningPulse.fyi want to be transparent with our readers about any potential conflicts of interest that may arise in our work. It's possible that some of the investors we feature may have connections to other businesses, including competitors or companies we write about. However, we want to assure our readers that this will not have any impact on the integrity or impartiality of our reporting. We are committed to delivering accurate, unbiased news and information to our audience, and we will continue to uphold our ethics and principles in all of our work. Thank you for your trust and support.