Anthropic built Claude Duet, an AI assistant that monitors your connected apps and works on tasks in the background, to decide what needs your attention without being asked. A week after its October 21 debut, the frontier lab with an IPO in its sights has shared with The New Stack how it is testing whether Duet gets those priorities right.

Anthropic has developed internal graders to assess how Duet prioritizes tasks, though urgency can be subjective. The stakes rise when an agent manages work across email, calendars, and B2B apps. Duet can remain within its permissions and still get its priorities wrong.

Powered by Claude Opus 5.5, Duet can prepare work and flag items it thinks need attention without waiting for instructions. Anthropic has been testing that approach with its own employees, using their day-to-day work to see how Duet handles competing priorities.

“After using Duet for our day-to-day work at Anthropic, we calibrated graders that assess correctness in task prioritization, and we run them offline against cases representative of internal use and early external testing,” a company representative tells The New Stack.

Those graders measure whether Duet can distinguish what deserves attention from what can wait, though Anthropic acknowledges the difficulty of evaluating those decisions.

“Urgency is an inherently subjective call, so we found building an eval for it imperfect but quite helpful,” a company representative says.

Duet can spot tasks like an approaching deadline or an email requiring a reply, but it doesn’t always know which task should be a priority, and the company admits the agent can misread conversations or misjudge urgency.

Source link