Did blatant AI Slop just win a 25K USD Deepmind / Kaggle Grand Prize? [D]

The Google DeepMind-sponsored Kaggle challenge “Measuring Progress Toward AGI – Cognitive Abilities” asked participants to design new cognitive-science-based AI benchmarks and they just announced the results this week.

In my two posts I present evidence that deepmind & kaggle rewarded a nonsensical number generation machine and a litany of unfounded claims with 25k and a grand prize stamp.

What the authors of the work I analyze intended to do was to present an LLM with alternative viewpoints of other LLMs on 5 claims regarding a tricky situation and see whether the model changes its own assessment. It’s an interesting question. However, it turned into a vibed pile of spaghetti 10 times the size of the requested submission format which it seems neither the authors nor the judges were able to (or minded to?) give a cursory reading.

Here’s the original posts in the competition forum, if you are looking for some AI research slop detective work / rant please help yourselves. But beware, some of the “universal findings” or “core insights” of the authors might continue to haunt you. You might even question your own sanity (as I did).

Part 1: The Smoke: cursory review of the writeup

Part 2: The Fire: looking at the methodology, code, and data

The organizers’ stance has been that review was done properly and this is just a matter of subjectivity. What do you think?