I'm suprised folks are having such great coding experiences.
Using Gemma-4 on a moderately complex code base, it utterly flailed and gave a half baked implementation.
I mean OP had them code something called "parse_csv_summary", which sounds like something a junior dev can whip up in an hour or so, since CSV manipulation in Python is one of the most basic tasks you can encounter in the problem space.
Banking, scientific data analysis, sales, etc. Everything uses and manipulates csvs.
So it was firmly in the bottom quartile of difficulty - and there LLMs actually do quite well.
I mean OP had them code something called "parse_csv_summary", which sounds like something a junior dev can whip up in an hour or so, since CSV manipulation in Python is one of the most basic tasks you can encounter in the problem space.
Banking, scientific data analysis, sales, etc. Everything uses and manipulates csvs.
So it was firmly in the bottom quartile of difficulty - and there LLMs actually do quite well.