That's fair, my immediate concern would be that there would be very few comments comparing any two models, so the data would be very anecdotal.

The context would be really nice to have, but reading the comments myself, it often just isn't very clear what exactly users are building or which programming language they are using.

I think analyzing more comments is promising. If you get enough data, you can generalize across use cases and get more meaningful ratings. The obvious lever is including more posts, although it might hit diminishing returns. I'll play around with it.

For the context, I want to try giving Gemini a "scratch pad", where it can note down strengths and weaknesses per model that it finds in the comments. Something like "some users say that model x is good for writing tests". Then on each run, I let it update the scratch pad and publish the results as more of a qualitative analysis.

For the wording, I'd like to keep a certain amount of click bait, sorry ;)