> One thing that surprised us was that JSON key order changed the answer. Sending the same content with keys in a different order moved the output probabilities by up to 0.19.
That's like... really basic LLM knowledge? Like how e.g. this article measures how often answers change based on order https://zachlandes.com/trolley-problem/#appendix-a:~:text=Ch...
Or is this the Claudish "One thing that suprised us" that just means "and here's another thing"?
I did not know that json order can change the answer. Doesn't that make LLM's unreliable? Do we know why this happens?