Time to dump this test. Probably not a coincidence every version has the same rolling green hills, gradient blue sky, sun in the corner, etc.

I think the 'pelican test' is becoming useless. It's been around long enough that now I'm sure good examples are in the training data, and hell they might even do some hand tuning to make it do a decent job since they know people will ask about it.

But either way, with no real way to visualize the result of the text it starts with - it will always be stabbing in the dark. It can't understand conceptually what any of it should look like and then refine the SVG to improve it gradually. It just throws darts at a wall and hopes it comes out alright.

What's strange with this is the prompt "Photorealistic photograph of a pelican riding a bicycle down a coastal boardwalk, wings gripping the handlebars, webbed feet on the pedals, large orange bill, detailed feather texture, golden hour lighting, shallow depth of field, shot on a DSLR with 85mm lens, natural motion blur on the wheels" produced, well, exactly what I asked it for. I wonder if I tell it then to make it SVG ...

https://chatgpt.com/share/6a5009de-fff8-83ea-98ff-0da17d1d04...

Cool. I still find these a useful visualization of some the qualities of llms. Even if they did train for [animal] on [vehicle] svg, it's still nice to see at a glance how the different models and reasoning levels perform. Lunar misses part of the frame, except on max reasoning. While most of the others have a mostly correct bike at all reasoning levels.

I once used something like karpathy's auto-scientist to mutate the prompts and rank them with a vison model. Some of the winners where pretty neat. I think they have a lot more style than the gpt-5.6 ones. https://xcancel.com/xundecidability/status/20449185674144196...

people are saying this is benchmark is saturated but all of these have occlusion issues, even sol max.

A skilled human artist wouldn't have both legs in front of the bike, or a single straight line representing both leg's crank arms.

Is the direction of the pelicans encoded in your prompt? Curious why they are all left to right with the exception of terra xhigh.

At what stages will models start to internally reflect the drawn SVG and automatically fix their own mistakes?

I assume multimodal models can do it already do it today if constantly asked "make it better"

The quality of sol on effort=none makes me think this test is saturated or they are benchmarkmaxxing this exercise.

Ok, I'll never use max effort again on OAI models..

I'm waiting for the day that the "generate a Pelican" test comes back with a SVG-art like illustration of a Pelican equipment case, like a model 1620 or similar.

https://www.google.com/search?client=firefox-b-d&q=pelican+1...

They said in the AI community, a pelican riding a bicycle is a good test to measure effectiveness of the model, wondering if they were referring to you, or is it really a standard in the AI community ?

Also would be good to have a tool where users can select models and instantly see each model's generated pelicans. That will make it easy to compare the output of different models.

Surely "how to draw a SVG pelican on a bike" has made it into the training data by now ...

Thank you Simon! Luna is surprisingly decent across all reasoning levels.

something is wrong with Terra model series, most pelicans, except Max, looks bad

gpt-5.6-sol Max pelican didn’t skip neck day

gpt-5.6-sol x XHIGH is my favourite

Apparently plus users do not have access to Sol, so I'm really worried about the ugly Terra Pelican.

somehow Terra really struggles here even compare to Luna.

max effort sol clearly over-engineered

terra is just weird. in this nothingburger test, time nor higher costs seem to not strongly correlate with the aesthetics.

AI really sucks at bicycles...

I like Terra High the best. That pelican is utterly yoked.

[dead]

[flagged]

this looks like the same shit from 4 years ago. give it up.