CONNECT WITH US
Web3 & Blockchain

Web3 & Blockchain

AI models agreed with each other more than with reality in a new test

CryptoSlate logo

Published on

Add as a preferred source on Google
AI models agreed with each other more than with reality in a new test

Science Prediction Universe found that pooling rankings improved average results, while shared scientific errors survived.

AI models agreed more closely with one another than with laboratory measurements in a new test of AI predictions. Pooling their rankings improved results on average, but several systems could still share the same mistake.

The Science Prediction Universe study, led by UniPat AI and submitted to arXiv on Nov. 11, tested 13 agent configurations spanning 12 models on 100 public tasks and 16 private tasks. Each configuration combined a model, reasoning setting and agent software. The study asked them to rank specified experiments across chemistry, materials science, biology and medicine.

Science Prediction Universe researchers reported that, on the public tasks, average agreement between scored configurations reached a Spearman rank correlation of 0.52, compared with 0.33 against measured outcomes. Agreement between agents exceeded their correlation with measurements on 81 of the 100 tasks. These averages use available scored predictions, with coverage varying by configuration.

Those figures describe how closely orderings match, rather than percentages of answers correct. Agents might rank reaction conditions by yield or materials by conductivity without having to predict numerical values. A perfect ordering without measurement ties scores 1; random rankings average 0.

The researchers found a benefit to pooling predictions. Averaging each experiment's position across configurations produced a consensus ranking that scored 0.08 points above the average submission, improving results on 73 of the 100 public tasks. The reported 95% confidence interval for the gain was 0.05 to 0.11.


Source link

Disclaimer

We strive to uphold the highest ethical standards in all of our reporting and coverage. We TheMorningPulse.fyi want to be transparent with our readers about any potential conflicts of interest that may arise in our work. It's possible that some of the investors we feature may have connections to other businesses, including competitors or companies we write about. However, we want to assure our readers that this will not have any impact on the integrity or impartiality of our reporting. We are committed to delivering accurate, unbiased news and information to our audience, and we will continue to uphold our ethics and principles in all of our work. Thank you for your trust and support.