The big trend in robots is handing the keys over to a generative AI model, but that brings with it a problem: that architecture isn’t predictable the way traditional algorithms are. How can you be sure your brand new humanoid will be safe?

Dr. Ding Zhao, who directs the Safe AI lab at Carnegie Melon University, has been working on this problem for almost his entire career. Now, along with veteran start-up executive Kyle Wong and machine learning engineer Simo Rachidi, he’s founded a company, Safeworld, intended to solve it.

“The safety challenge that we’re talking about is a combination of, one, really advanced generative AI probabilistic evals — how do you underwrite the risk of a probabilistic system?” Zhao says. “The second part that’s really hard is the trust part, and you need both to deploy a robot.”

Safeworld is emerging from stealth today with a seed round of more than $12 million, led by Shine Capital and a16z Speedrun, with additional investment from Box Group, Carnegie Mellon University Endowment, Innovation Endeavors and SV Angel.

“The time to build an industry safety standard is now while robots are being designed and deployed,” a16z Speedrun partner Jonathan Lai told TechCurnch. “By the time you have robots in households colliding with kids and causing safety incidents, that’s way too late.”

Safeworld’s speciality is evaluating a robotic control system in simulations that are populated with realistic human models. It’s akin to the challenge faced by companies like Tesla or Wayve, who must ensure that their vehicles respond appropriately to a variety of surprising incidents they may encounter on the road.

Source link