The shift from AI training in data centers to AI inferencing at the edge will require a fundamental rethinking of where and how data is processed and stored, how it is prioritized, and what needs to be traded off in terms of power and performance to stay within a limited power budget.

At the center of these efforts is a recognition that the explosion of data — especially in physical AI where data is being live-streamed from sensors — requires much of that data to be processed as close to memory as possible. This has been a topic of discussion since the inception of the Internet of Things, but it really began gaining traction with AI due to the flood of data and concerns about the amount of power required to move and process it. In battery-electric vehicles and drones, for example, the less energy spent on processing data, the greater the range per charge and the more energy that can be allocated to critical and not-so-critical functions.

The challenge is understanding what data needs to be kept, what can be discarded, and what needs to be acted upon. Those are very different priorities. Making those choices requires advanced processing, and the quickest path to results using the least amount of energy is to minimize the distance data needs to travel between processing and memory.

“The realities of these LLMs getting into physical devices are causing everyone to rethink how they architect their memory,” said Frank Ferro, group director for product management at Cadence .

Source link