alt Hacker NewsNo, that's a misconception that led to decades of vulnerabilities.
You see, you can (usually) easily tell what a particular escaping transformation does. That does not tell you neither how it will be interpreted down the line, nor what should be done.
Arguably the most common problem is double escaping. This typically manifests as various double escaping bugs.
If your hand rolled implementation just chains `.replaceAll("<sepcial>", input)` and `.replaceAll("<escape>", input)` the escaping result depends on evaluation order, at least on already pre-escaped inputs if your instruction sequences are single-element. Even if you get it right and don't reduce escaped sequences to double escape + unescaped, you are still dealing with stray escape sequences: "John o\'Doe".
You have to meticulously (all the way through your call tree and even through persistent storage cycle!) track if a value has already been escaped and whether it needs escaping. As paradoxical as it may sound, meticulous and defensive escaping produces tons of bugs. If the project decides to escape raw inputs right when passed in, escaping them once more before passing out (to storage, another process) is a bug that you cannot easily statically test against.
On top of that, various modules that you interact with (storage, libraries, modules pulled from another team) will have different escaping semantics: some will apply escaping on their own, some will expect input to be "sanitized" and treat it raw. The semantics can even be different on different paths: write to storage module accepts input as is, but retrieval method "helpfully" runs escaping.
Furthermore, in different contexts the escaping rules are going to be different. In a web world, what's safe to write directly to html, pass to js `alert`, and pass to sql query are entirely different things. Input safe to dump into html is not necessarily safe to pass to string-interpolating SQL DTO layer, and vice versa. Then the DBA changes config to allow variables in queries and your escaping _semantics_ are now entirely different.
It's a minefield with essentially unbounded surface. You will trip up. People have tried to solve the problem for decades. Very smart people have tried. They all have failed.
Replies
You can solve this with refinement types. It's entirely tractable in many systems in principle - APIs can be written to be safe, you can track data as separate from code, etc. None of this exists in an agent - you can't separate data from code, it's intractable because it's impossible in principle.
We have come to a consensus. That is, A, you use the raw input right up till the very last second that you can and B you use the technology specific escaping APIs.
In the case of writing to the DOM, that means you use .innerText rather than .innerHTML. In the case of writing to a database, that means you use the driver and insert variables rather than directly into the string. Both of these are technology-specific escaping APIs. It's just that the browser is much better at making sure HTML is passed safely than you are.
> They all have failed.
That's not true. Most have failed, but those who used the right tool for the job - a rich, static type system in a functional language - did succeed. It's just that such type systems are rare, and even if nominally a type system is good enough, the required boilerplate might be uneconomical to maintain. Scala and F# are probably the only two languages that are mainstream-adjacent, at least, and have type systems expressive enough that using them to track escaping is not an absolute hassle. And they're both tiny in terms of the number of users.
In the end, we did settle on APIs that hide the escaping process (prepared statements, innerText vs. innerHTML, etc.) just because it's a) good enough; and b) possible to implement more or less uniformly across the TIOBE Top 20. That's good, but if you happen to use a language that's powerful enough, you probably also should track the escaped/unescaped status in the type system - it can be a cheap, additional safety net.