Much AI advice is about what to automate. But what happens when the model in your product hands a customer the wrong number to act on?

As many as 77% of small and midsize businesses in the U.S. use AI regularly, and customer service is one of the top three uses.

The trouble starts when that answer is a number the customer acts on, because a large language model (LLM) predicts numbers rather than working them out.

When a model drafts an email or ad copy, there’s usually no single correct answer because it’s a creative task. But when a product returns a number the customer treats as a settled fact, that’s where the hazard lies. Examples of such numbers include:

A wrong number can read like a right one, which can erode trust in the company when discovered. A Toronto BMW dealership learned this lesson the hard way.

A man who wanted to sell his car sent an inquiry to the BMW dealership he’d bought it from, and got a text back from “Quinn” offering 27,162.79 Canadian dollars, about $19,000, to buy it back. Quinn was an AI chatbot, and that figure was not a valuation, but the balance the man still owed on his loan, handed to the bot by mistake and passed along as the price. A salesperson later called to revoke the offer, and the dealership only reinstated it after CBC News asked for comment.

Before we build anything at Omni Calculator, where the model produces a value our users act on, we ask ourselves if the output must be reproducible.

Source link