Imagine an artificial intelligence system that has performed reliably for months. Its recommendations are almost always right, and the employee reviewing them has learned that close scrutiny rarely changes the outcome. Then something unusual happens. The AI makes an error or can’t proceed and needs human intervention. The employee is expected to step in and remains responsible for the outcome.

I’m a professor of management information systems and the academic director of Drexel University’s Center for Applied AI & Business Analytics. In my research on human-AI relationships, I call this latent risk: the possibility that people may lose capabilities their organizations still depend on them to exercise.

AI can continue to perform well while the people overseeing it become less practiced at exercising independent judgment and less prepared to intervene. The risk becomes latent when the organization still depends on that human capability to step in but has little cause to notice that that capability may be weakening.

The concern is not entirely new. The idea of latent risk builds on decades of research into how people respond to automation. For example, studies of automation bias have shown that people can miss problems or follow automated recommendations even when other evidence suggests they should not.

What is different today is the role AI is beginning to play in organizations. It is beginning to assume larger parts of the analysis and decision-making that people previously performed.

The better AI becomes, the more the nature of human work changes. Work that once required an employee to analyze a problem and reach a decision may instead be performed by AI, leaving the employee to review its recommendation.

Source link