It's pretty safe to say that AI will be used on the battlefield making real life and death decisions before it will be able to render a decent pelican on a bike in SVG.

> the bicycle frame is the correct shape

No, the handlebar is wrong. The handle bar is rotating the frame instead of rotating the front wheel. The handle bar should be mounted on the same line as the front wheel is.

Hopefully 4.9 will read my comments :)

I bet someone shares this link every time you post about bicycles, but since I didn't see anyone share it yet in this thread, I'll take the opportunity to do so:

https://www.gianlucagimini.it/portfolio-item/velocipedia/

Turns out even humans can be pretty bad at drawing bicycles :)

Glad to see that the "high thinking" level adds a helmet. Always a smart choice.

Sadly I think the correlation between this benchmark and performance is starting to break down imo. Still a legendary idea will be remembered and ingrained in the models forever haha

Here's pelicans in all of the thinking levels - low, medium, high, xhigh, max

https://tools.simonwillison.net/markdown-svg-renderer#url=ht...

You've peed in the pool Simon, this has to be a part of the internal evals by now! You got to try something new - maybe a panda in a canoe?

I really like that thinking level high gave the pelican a helmet.

Hey simonw I love your test, do you think using thinking level "max" makes sense for this test? I would love to see the results about it.

It's funny that we've reached the level where LLMs draw more correct bikes than any random person

Simon, is your pelican test really captures differences among models or should you at least try like 10 times or something to average the random effects

The vast majority (if not all) of these make it impossible to turn, among other fun things. Only out of curiosity, have you tried prompting further with how a bike must operate to see if it does the right thing?

I actually like the 4.7 the most, interestingly enough. Not like you can "objectively" weight artistic output like this.

I find the most miraculous thing about 4.7 to be that the pelican is facing left, wonder why the right facing everything is so ubiquitous in these images.

I don't see how a frame without a headtube can be "the correct shape".

thanks for always providing this very much on time. I'm wondering what the next, harder challenge could be? Maybe some animated svg?

That little red hat on hard mode is sending me. 4.8 has whimsy

Is the "opossum riding an e-scooter" benchmark in the works for Opus 4.8? ;)

For comparison, what's GPT-5.5 producing today?

Am I allowed to say that pelican's little helmet is adorable? I can't provide a strong computational proof, or even a shred of anecdata...

...but that pelican's little helmet is adorable.

Eventually the frontier model folks are going to pick up on your pelican on a bike test and bake-in flawless results for that particular request.

4.7 reigns supreme IMO.