Google and Artificial Analysis introduce the Multimodal Knowledge Agent benchmark to evaluate AI performance on visually complex tasks.

Google, in collaboration with Artificial Analysis, has launched the Multimodal Knowledge Agent (MKA) benchmark to evaluate how AI agents handle visually complex tasks in professional workflows. The benchmark aims to address the limitations of text-only benchmarks by focusing on tasks that require reasoning across multimodal inputs, such as images, page layouts, and spreadsheets.

The MKA benchmark was developed with input from domain experts across 10 industries and 83 occupations, including architects, biologists, underwriters, and data scientists. The evaluation set consists of 200 tasks that simulate real-world workflows, each requiring three to eight hours of effort from a seasoned professional. Tasks are paired with multimodal input files, realistic prompts, and an expert-written rubric comprising approximately 70 yes/no checks. The rubric weights factual accuracy at 75% and visual design at 25%.

To ensure fairness and consistency, all models are evaluated using the open-source agent harness Stirrup, which provides a standardized set of tools, a sandbox environment, and a fixed turn budget. Deliverables are graded by an agentic autorater built on a non-Google model, which achieved a 96% agreement rate with human domain experts during testing. To prevent data leakage, the 200-task evaluation set is kept private, while a separate set of 10 public tasks is available on Hugging Face for broader experimentation.

Google's technical report highlights significant variability in model performance on the MKA benchmark. Frontier models demonstrated differences in efficiency, requiring between 30 and 130 turns on average to complete tasks.

Source link