On BrowseComp-Plus, GPT-4.1 answered 93.49% of the questions correctly when it was given the supporting documents directly. When an agent used the same model but relied on BM25 to search for evidence to pass to the model, accuracy fell to 14.58%.

That difference matters because a system that gets the answer right 93% of the time is far more useful than one that gets it right only 15% of the time.

An agent can look excellent in a controlled demo. The corpus is clean, the task is familiar, and the relevant material is easy to find. Production is messier. Documents may be duplicated, outdated drafts can remain searchable, and teams may use different terminology for the same thing. Several versions of a policy can coexist, while the task itself may unfold in ways the evaluation set never covered.

“More model intelligence does not compensate for evidence the agent never retrieved.”

Under those conditions, the agent may retrieve incomplete, stale, or simply wrong material. More model intelligence does not compensate for evidence the agent never retrieved.

This is where retrieval starts to determine what the model can actually do.

BrowseComp-Plus provides a useful way to separate these effects because it evaluates agents against a fixed corpus rather than a changing web-search service. The benchmark includes 100,195 documents and 830 research questions, each with human-verified supporting documents.

One evaluation gives the model all labeled positive documents directly. The paper calls this the oracle setting. Another requires the agent to search the corpus and gather the material it needs.

Source link