alt Hacker NewsEvidence of inconsistencies in evaluation process and selection of winners
Comments
I don't know about this exact competition but overall fair hackathons have been killed by AI.
It all seems fine from the outside but all the code is generated in all the projects and judging happens via AI, I have seen projects win because they prompt inject that they are the winners.
It used to be about human skill, now it's about ideas and of course insiders are the main winners.
Hi all, I'm Nick, Product Manager for Kaggle Benchmarks and one of the co-organizers and judges for this AGI hackathon.
First off, I want to set some context on the AGI hackathon. This was co-organized by Kaggle and Google DeepMind, and we had ~20 judges from both organizations. The hackathon concluded on Apr 16 and we had initially anticipated a judging period of 1.5 months (till May 31). However, we ended up extending the judging period by another 1.5 months (to Jul 13) because we wanted to do right by participants.
Second, I want to emphasize and unequivocally clarify that every single winning submission went through at least 2 human judges, and in some cases, up to 3-4 human judges. These judges reviewed and scored the submissions independently based on the rubric we highlighted on the hackathon page.
Thirdly, I acknowledge that there is always an element of human subjectivity to reviewing qualitative submissions in hackathons. As best we can, we have put in place processes that ensure rigorous human review against objective standards and to reduce the possibility of bias by having multiple independent judges. We understand there may be valid disagreement over outcomes, but hopefully the above context clarifies this was not carelessly outsourced to LLM judges.
Thanks, Nick
AI submissions and AI judges a match made in (AI) heaven.
People have been using brute force methods to win Kaggle competitions since the beginning (and other people have been complaining about it just as long).
At its core, ML is all about computer generated models (automated feature selection, hyper parameter tuning). Many (most?) of the models produced in Kaggle are already black boxes and have been for a long time. The model that won the Netflix prize was never used in production for that reason.
Using an LLM to generate code to generate a black box is pretty much par for the course.
I thought Kaggle was a website where you download dubious CSV files of annualized bean consumption in Bolivia, or whatever.
Was Kaggle ever a reputable source of original research, or a source of anything with any provenance at all? That would be news to me. The fact that 25 grand was involved this time is unique, I guess.
I think this is a good meta-lesson for Kaggle. When you have objective metrics to hill-climb towards, AI can do quite well. When you just phone it in and rely on LLM as a Judge, the results are not so great.
It’s a shame that Arvix (and once thoughtful places like Kaggle) are used for self-promotion.
I get people want to work at an AI lab but slopping it in public in this manner is counterproductive to the original intended purpose of these places.
Went through the comments here and there and one thing to note is that there was a question about who do you think should have won instead. This is a good question because it is possible that all submissions were like this or there were ones that looked just worse. It would be quite useful to know who came close as well in this case. If you knew which submissions were good you could have a process to revoke the prize and give it to someone else in case of fraud or negligence or similar.
Having said that it is also possible that the mistakes and claims were a human error, sure a lot gets ai generated these days but there is a chance in which case the accusation does not look so severe anymore.
<obama medal meme>
"I think you just need to accept the results of the competition. The winning submissions clearly provide value and had a lot of effort invested in them. I'm not really worried about a few inconsistencies or mistakes if the value is still there. Did you think another submission deserved to win over these?"
That comment is gold. Yeah, I'm not worried about hallucinated slop, just accept it was the winner folks.
This is why I'm not wasting my time on these things (either in participating or in paying attention to the results). The noise is just too high. I'd be furious if I'd spent a bunch of time on doing actual work for something like this only to have slop win. Hopefully, the other contestants worked on stuff that is transferable to other purposes.
Best question - how many tokens in dollars were spent to win the comp?
> "Finding 1: Scale Buys Evaluation, Not Control"
The attached paper's (https://arxiv.org/pdf/2604.16009) title is "MEDLEY-BENCH: Scale Buys Evaluation but Not Control in AI Metacognition"
This is the most blatant Claude line, or as Claude would put it, the smoking gun.
I think there is a saying about having a million monkeys with typewriters... or something of that sort.
What's up with all the AI generated responses on that page?
Sadly, the major ML/AI/NLP conferences are being inundated with AI slop papers. That will arguably have a bigger impact on the quality of research moving forward.
Gross
overall, the quality of products has been going downhill.
AI is not there yet, instead of working hard, everyone is choosing the easy way out.
AI slop wins prize, I wonder if Ai slop read it also. would not be surprised. however not to judge anyone, I think we are seeing slop everywhere, hope some things still require hard blocks for low quality.
its difficult to justify lack of attention and details
deepmind using AI to evaluate submissions?
Please note this Post was just renamed without my involvement from:
Blatant AI slop just won a 25K USD Deepmind Kaggle Grand Prize
into
"Evidence of inconsistencies in evaluate process and selection of winners"
The real story here is the judging potentially being AI slop.
It was probably scored by AI too. Same reason why slop-filled resumes apparently work better these days.
I think the heyday of Kaggle is in the past. I would not lose sleep over this site.
Computers and software on the whole has been net negative allocation of resources. We are a major industry that burns time and money while spinning our wheels in the mud. AI is just the biggest example of this in history. It simply doesn't matter if the solution is correct because we aren't solving problems either way.
[dead]
[dead]
[dead]
[dead]
[dead]
AI is 95% useless. Not quite worth the trillion dollar market cap lol.
* The AI bots are downvoting me * hooray
LLM-as-a-judge?
Given that LLMs are trained with RL && LLM-as-a-judge, is it really cheating if real competitions use the same?
Maybe the real alignment is the slop we decoded along the way
What a whiner!
The problem with removing bullying from the upbringing process is you get insufferable twats like this who can't take "No" for an answer and who can't take a loss.
His mom told him "Everything you do is art!"
All AI companies have slop press sites that hype them up. What we saw since 2020 is the largest industrial propaganda campaign in history. It started with Lex Fridman planted interviews to make AI researchers appear human and ends with AI awarding AI prizes.
Mainstream journalists didn't know any better and thought they were reading secret inside information and parroted it - until now when the house of cards is collapsing.
Notice that the defense in the comment section is the Silicon Valley platitude that "it provides value". No sane person believes that any longer, only the financially invested and some SciFi trash addicts.
I always find it interesting when I see posts here around "LLMs are just fancy autocompletion machines" and there are 100 comments below it.
I think that a lot of software engineers are using LLMs and a lot of very popular tools are developed by, or are assisted by, LLMs. Is this not just going to be a thing going forward?
This feels akin to traditional artists getting angry at digital art winning competitions when that was a new concept.
We're simply in the early stages of a paradigm shift, no?
AI is useful. But the amount of people that are simply offloading all of their thinking to AI and blindly accepting the answer is absurd. Kaggle is most likely using ai to assess the submissions and are not using any common sense by blindly accepting the results.