alt Hacker NewsThere’s something about geospatial work that seems to break LLMs. I’ve experienced it a lot with latest versions of both Fable and Opus, and it’s the classic nightmare of confidently-spoken-but-wrong.
Yesterday, I had a relatively trivial bit of work in play using distance-weighting to measure stuff, and Claude did really well / but right at the end, despite having a success metric and lots of great results matching it, Claude insisted my maths was wrong and needed changing.
When i asked why, and pointed out that the suggested replacement would cancel out the distance-weighting entirely, it immediately agreed with me and asked to write its fifth project memory about it.
The other big ones it tends to mess up on without strict guardrails are coordinate systems; and measurements, often converting back and forth between a given nation grid and wgs84 latitude/longitude in ways that lose good amounts of precision; and then custom-coding haversine-equivalents to measure a distance that’s otherwise be a simple subtraction via the national grid coordinates. It’s baffling.
Replies
I had similar issue with 3d code in threejs. Models used to be quite bad over a year ago. got ok around year ago. the latest are quite good at it.
so i made a simple ab testing eval on fixed prompts: https://threejseval.com
There is a 7 degree o freedom robot arm that is really telling of the improvements in coordinates reasoning:
This is my experience too, it has got a bit better but nowhere near as quickly as other domains I deal with. I can usually guide the LLM to a decent solution but there's a lot of hand holding and I'm often left wondering if it wouldn't have been quicker for me to just get stuck in and do it the old fashioned way. Maybe because geospatial stuff is pretty niche so is under-represented in the training data relative to yr standard CRUD apps and React front ends which are ten-a-penny?
They're notoriously, hilariously bad at chess, even today. So yes, agree, their shortcomings come much more forcefully and immediately to light when they attempt spatial tasks.
I mean, it's obvious, no? LLMs operate exclusively on language, and language is an especially poor way to represent geometry, let alone geography to any useful resolution.
I’ve found it to be surprisingly effective lately.
I’ve been renovating a spatial/topology validation system for an OpenLayers driven map editor for robot fleets. I set up a clean interface for spatial indexes and predicates as well as your basic geotoolbox functions.
I was most surprised by a moment where Opus detected that using a union approach to ensure that robot footprints always sat within valid polygons (possibly straddling multiple valid polygons) was slow in testing. So it decided to find the difference between footprint and each polygon within the AABB of it. And if the remaining polygon is zero size, it’s valid. If not, the remaining shape is what is invalid.
I was also surprised just how good it was at recognizing, “in the test data there’s a lot of this kind of feature so we should explore what’s more common: hits or misses?
Before robotics I spent a lot of time building spatial models for data processing and analysis. I find that I can usually just explain the spatial question in natural language and it will figure out the relevant inputs from the dataset and what the process can be. I’m not obsolete quite yet though because maybe 10-20% of the time it goes on a pretty fantastical journey through unnecessary intermediate data products.