I think this is similar to how small changes in phrasing can completely change the answer you get from an LLM. This all goes to confirm that LLMs are, at their core, still text completion engines, they (frequently / always?) don't have a firm semantic grip on the topic at hand.
Here's an interesting reference on this that gives a bit more background information:
-- 8< --
Because LLMs (and many other ML systems) are chaotic, it is possible to manipulate them into doing something unexpected through a small, apparently innocuous change to their input. These changes can be illegible to human observers, which makes them harder to detect and prevent.
For example, flipping a single pixel in an image can make computer vision systems misclassify images. You can replace words with synonyms to make LLMs give the wrong answer, or introduce misspellings or homoglyphs. You can provide strings that are tokenized differently, causing the LLM to do something malicious. You can publish poisoned web pages and wait for an LLM maker to use them for training. Or sneak invisible Unicode characters into open-source repositories or social media profiles.
alt Hacker News
I think this is similar to how small changes in phrasing can completely change the answer you get from an LLM. This all goes to confirm that LLMs are, at their core, still text completion engines, they (frequently / always?) don't have a firm semantic grip on the topic at hand.
Here's an interesting reference on this that gives a bit more background information:
-- 8< --
Because LLMs (and many other ML systems) are chaotic, it is possible to manipulate them into doing something unexpected through a small, apparently innocuous change to their input. These changes can be illegible to human observers, which makes them harder to detect and prevent.
For example, flipping a single pixel in an image can make computer vision systems misclassify images. You can replace words with synonyms to make LLMs give the wrong answer, or introduce misspellings or homoglyphs. You can provide strings that are tokenized differently, causing the LLM to do something malicious. You can publish poisoned web pages and wait for an LLM maker to use them for training. Or sneak invisible Unicode characters into open-source repositories or social media profiles.
-- >8 --
Source: https://aphyr.com/posts/412-the-future-of-everything-is-lies...