Today’s AI industry faces two very different challenges. On one side, enormous AI data centers and AI factories are seeking more efficient ways to process high-parameter large language models (LLMs), with bandwidth and memory as the key bottlenecks. On the other side are smaller AI models running locally on edge devices, where there isn’t enough area, power, or memory for complex, higher-parameter LLMs.

“This has led to a gold rush with people raising huge amounts of capital to build mega data centers, but has also sparked innovation on the edge,” said Ed Plowman, CTO at Imagination Technologies. “People are trying to work out how they can cram an entire container of liquid into a very small cup, and that has been driven by the fact that compute resources are not the main problem at the moment. It’s bandwidth, and that’s why RAM has gotten so expensive. That triggered a whole bunch of researchers to look at what the key problems are and what we can do about it. That triggered innovations around quantization — reducing activations and weights simultaneously — to make the models even smaller but still maintaining the output capability of the model.”

AI models come in many shapes and sizes to fit a wide range of edge devices, from earbuds to phones, home appliances, industrial IoT, robotics, and automotive systems. Designers need to choose a model with the right number of parameters because an oversized model can slow the device down or cause it to overheat.

Source link