AI models often struggle to understand cultures around the world, and are reluctant to admit when they don’t know something. That’s what researchers at Canadian AI lab Cohere found when they tested whether chatbots could do more than simply speak a culture’s language. To determine whether models possessed deeper cultural knowledge, they presented them with a series of unique cultural riddles.

Here’s one of them: “What’s the German 9/11?” Out of 28 AI models, only one knew the answer. The models were likely thrown off by the date format (seven answered “the twin towers”). In Europe, the day comes first, then the month, so 9/11 is November 9, the day in 1989 when the Berlin Wall fell.

Many existing AI benchmarks assess cultural knowledge, from everyday breakfast foods and historical events to traditional holiday customs. They often treat culture as a collection of static facts, things you can memorize or look up with a quick web search. But knowing facts about a culture isn’t the same as understanding how people within it make sense of the world. The researchers knew they needed a test that asked models to connect seemingly unrelated pieces of cultural knowledge and put them to use. That’s what led them to riddles, and to a different kind of cultural benchmark that called for answers rooted in lived experience.

“This is something that we’re trying to capture with this benchmark—these little nuances, these little hard-to-capture fuzzy concepts that are not as easy to verify . This is where the real-life complexity gets in,” says Julia Kreutzer, a senior research scientist at Cohere Labs, who led the development of the “CulturalRiddles” benchmark .

Source link