alt Hacker NewsThere's a difference between how language effects LLM output and how it affects humans.
If you mention something in a system prompt, an LLM will be more likely to mention it because it's in context unless it's explicitly forbidden in strong language and, even then, they can still work the opposite way.
Yeah, but in this case I'd expect the result of the language is "because of company's actions, the LLM fails to inform user of their legal rights, which the company was legally obliged to communicate to the user"
And I feel like if you are legally obliged to communicate something, sabotaging that communications process is, at a minimum, obviously unethical? And ideally illegal?