NVIDIA's NeMo Compass and Eval Author streamline AI agent optimization by identifying failures, creating tests, and improving harness instructions.

NVIDIA has introduced a comprehensive blueprint for optimizing AI agent harnesses using its NeMo platform. The workflow, detailed in a November 3, 2026 tutorial, focuses on improving agent reliability by addressing failures in prompts, tools, routing, and other harness components without altering model weights.

The optimization process begins with NeMo Compass, which analyzes stored traces to detect recurring issues and prioritize improvements. In a fictional example, the Hermes enterprise assistant incorrectly called the "send_message" tool without user confirmation after being asked to draft a project-update message. NeMo Compass flagged this failure by comparing the agent's actions against predefined behavior rules in an ETHOS.md file.

Compass findings are structured into a backlog of issues, supported by trace IDs linking errors to specific runs. NVIDIA reports that Compass identified anomalies in thousands of traces during internal benchmarks, outperforming coding agents tasked with similar evaluations.

After identifying issues, NeMo Eval Author helps create executable tests and verifiers to reproduce and evaluate failures. In the Hermes example, Eval Author generated four Harbor evaluations to test whether the agent improperly sent messages without user consent. These evaluations were based on Compass findings, raw traces, and the agent's ETHOS.md file.

Eval Author also supports creating custom evaluations before production data is available, allowing developers to expand test suites as real-world failures emerge.

To address the identified failure, NVIDIA updated the Hermes agent's harness instructions, specifying that requests to draft messages should authorize preparation only, without automatic execution.

Source link