CONNECT WITH US
AI & Deeptech

AI & Deeptech

Gaia2 and ARE: Empowering the community to study agents

Hugging Face logo

Published on

Add as a preferred source on Google
Gaia2 and ARE: Empowering the community to study agents

That’s why we’re very happy to introduce Gaia2, the follow-up to the agentic benchmark GAIA, allowing analysis of considerably more complex behaviors. Gaia2 is released with the open Meta Agents Research Environments (ARE) framework to run, debug and evaluate agents. ARE simulates complex real world-like conditions and can be customized to further study agents behaviors. Gaia2 dataset is released under CC by 4.0 license, and ARE under MIT license.

GAIA is an agentic benchmark published in 2023, with 3 levels of information retrieval questions requiring tools, web browsing, and reasoning to solve. In 2 years, the easiest levels have become too easy for models, and the community is coming close to solving the hardest questions, so it was time for an entirely new and harder agent benchmark!

Here comes Gaia2, a follow up to GAIA, going way beyond it in terms of capabilities studied!

Where GAIA was read-only, Gaia2 is now a read-and-write benchmark, focusing on interactive behavior and complexity management. Agents are now evaluated not only on search and retrieval, but also on instruction following over ambiguous or time-sensitive queries, in a noisy environment with controlled failures - reflecting real-world conditions more than any other simulated environment. We want to test how agents manage tools or APIs that sometimes do not work, plan successions of actions with very specific time frames, and adapt to new events - a whole new range of complexity!

To do this, we use the following task groups (thanks to 1000 brand new human-created scenarios):



Source link
Related Topics:AI & Deeptech

Disclaimer

We strive to uphold the highest ethical standards in all of our reporting and coverage. We TheMorningPulse.fyi want to be transparent with our readers about any potential conflicts of interest that may arise in our work. It's possible that some of the investors we feature may have connections to other businesses, including competitors or companies we write about. However, we want to assure our readers that this will not have any impact on the integrity or impartiality of our reporting. We are committed to delivering accurate, unbiased news and information to our audience, and we will continue to uphold our ethics and principles in all of our work. Thank you for your trust and support.