Impressive benchmark results should make it easy for engineers to select the right open model for the outcome they are tasked with developing. However, that is not the case. 

Benchmarks indicate an open model’s optimal performance, but in production, a model must work repeatedly. The challenge is that in production, other factors matter, such as the information provided to the model and its accuracy. So benchmarks can’t tell an engineer whether the AI they are developing will complete the task accurately, or even at all. This is especially important for long-horizon AI tasks.

Ryan Hanrui Wang, SVP of Applied AI at AI cloud provider Nebius, tells The New Stack it is important to measure an open model by the industry benchmarks in conjunction with internal benchmarks and harnesses to ensure you get the inference outcomes the organization requires, something he says the Amsterdam, Netherlands-based company always does in partnership with its clients. 

Wang joined Nebius when it acquired Eigen AI, the inference-optimization startup he co-founded and led as CEO, in a roughly $643 million deal announced in May; Nebius sells inference through its Token Factory platform. 

“Customers often have their own benchmarks, so we can use their agent workflow and its traces to see the level of success we can work with. We always work closely with them to define the benchmarks.”

” He says this can misalign agent tasks with the true cost of inference calls. “The workflow is not tested by the benchmark design.

Source link