Crowdsourced AI benchmarks have serious flaws, some experts say

AI labs are more and more counting on crowdsourced benchmarking platforms similar to Chatbot Arena to probe the strengths and weaknesses of their newest fashions. However some specialists say that there are severe issues with this method from an ethical and academic perspective.

Over the previous few years, labs together with OpenAI, Google, and Meta have turned to platforms that recruit customers to assist consider upcoming fashions’ capabilities. When a mannequin scores favorably, the lab behind it is going to typically tout that rating as proof of a significant enchancment.

It’s a flawed method, nonetheless, in line with Emily Bender, a College of Washington linguistics professor and co-author of the guide “The AI Con.” Bender takes specific problem with Chatbot Enviornment, which duties volunteers with prompting two nameless fashions and deciding on the response they like.

“To be legitimate, a benchmark must measure one thing particular, and it must have assemble validity — that’s, there must be proof that the assemble of curiosity is well-defined and that the measurements really relate to the assemble,” Bender stated. “Chatbot Enviornment hasn’t proven that voting for one output over one other really correlates with preferences, nonetheless they might be outlined.”

Asmelash Teka Hadgu, the co-founder of AI agency Lesan and a fellow on the Distributed AI Analysis Institute, stated that he thinks benchmarks like Chatbot Enviornment are being “co-opted” by AI labs to “promote exaggerated claims.” Hadgu pointed to a latest controversy involving Meta’s Llama 4 Maverick mannequin. Meta fine-tuned a version of Maverick to score well on Chatbot Arena, solely to withhold that mannequin in favor of releasing a worse-performing version.

“Benchmarks needs to be dynamic slightly than static knowledge units,” Hadgu stated, “distributed throughout a number of unbiased entities, similar to organizations or universities, and tailor-made particularly to distinct use instances, like training, healthcare, and different fields executed by working towards professionals who use these [models] for work.”

Hadgu and Kristine Gloria, who previously led the Aspen Institute’s Emergent and Clever Applied sciences Initiative, additionally made the case that mannequin evaluators needs to be compensated for his or her work. Gloria stated that AI labs ought to be taught from the errors of the information labeling business, which is notorious for its exploitative practices. (Some labs have been accused of the identical.)

“Typically, the crowdsourced benchmarking course of is effective and jogs my memory of citizen science initiatives,” Gloria stated. “Ideally, it helps herald extra views to supply some depth in each the analysis and fine-tuning of knowledge. However benchmarks ought to by no means be the one metric for analysis. With the business and the innovation shifting shortly, benchmarks can quickly turn into unreliable.”

Matt Frederikson, the CEO of Grey Swan AI, which runs crowdsourced pink teaming campaigns for fashions, stated that volunteers are drawn to Grey Swan’s platform for a variety of causes, together with “studying and working towards new expertise.” (Grey Swan additionally awards money prizes for some assessments.) Nonetheless, he acknowledged that public benchmarks “aren’t a substitute” for “paid non-public” evaluations.

“[D]evelopers additionally must depend on inside benchmarks, algorithmic pink groups, and contracted pink teamers who can take a extra open-ended method or deliver particular area experience,” Frederikson stated. “It’s necessary for each mannequin builders and benchmark creators, crowdsourced or in any other case, to speak outcomes clearly to those that observe, and be responsive when they’re referred to as into query.”

Alex Atallah, the CEO of mannequin market OpenRouter, which lately partnered with OpenAI to grant customers early entry to OpenAI’s GPT-4.1 models, stated open testing and benchmarking of fashions alone “isn’t enough.” So did Wei-Lin Chiang, an AI doctoral scholar at UC Berkeley and one of many founders of LMArena, which maintains Chatbot Enviornment.

“We actually assist using different assessments,” Chiang stated. “Our objective is to create a reliable, open area that measures our neighborhood’s preferences about totally different AI fashions.”

Chiang stated that incidents such because the Maverick benchmark discrepancy aren’t the results of a flaw in Chatbot Enviornment’s design, however slightly labs misinterpreting its coverage. LM Enviornment has taken steps to stop future discrepancies from occurring, Chiang stated, together with updating its insurance policies to “reinforce our dedication to honest, reproducible evaluations.”

“Our neighborhood isn’t right here as volunteers or mannequin testers,” Chiang stated. “Individuals use LM Enviornment as a result of we give them an open, clear place to have interaction with AI and provides collective suggestions. So long as the leaderboard faithfully displays the neighborhood’s voice, we welcome it being shared.”