CONNECT WITH US
AI & Deeptech

AI & Deeptech

Variable-Length Prompts on Chimera GPNPU

Semiconductor Engineering logo

Published on

Add as a preferred source on Google
Variable-Length Prompts on Chimera GPNPU

If you are sizing a language model for an edge product, the prompts it will see span a wide range. A spoken command is a dozen tokens, a pasted document is several thousand, and the same device serves both. Somewhere in the software stack, a decision is made about the gap between the longest prompt you provisioned for and the prompt that actually arrives. That decision sets how many binaries you build and qualify. It also sets how much of every inference is spent computing padding.

Most compiled inference stacks do not make that decision at all. A graph compiled for 1024 tokens processes 1024 positions every time it runs. If you send it 50 tokens, it computes the other 974 as padding. To serve short prompts efficiently as well as long ones, you have to build a separate binary for each length range and qualify all of them. Quadric Chimera GPNPU does not work this way. It gives you one guarantee and one optimization.

Fig.1: One graph serves every length up to the configured capacity. Source: Quadric



Source link
Related Topics:AI & Deeptech

Disclaimer

We strive to uphold the highest ethical standards in all of our reporting and coverage. We TheMorningPulse.fyi want to be transparent with our readers about any potential conflicts of interest that may arise in our work. It's possible that some of the investors we feature may have connections to other businesses, including competitors or companies we write about. However, we want to assure our readers that this will not have any impact on the integrity or impartiality of our reporting. We are committed to delivering accurate, unbiased news and information to our audience, and we will continue to uphold our ethics and principles in all of our work. Thank you for your trust and support.