> It's hard to argue that it isn't an optimization, because it doesn't affect the semantics of the program.

Depends on the semantics of the programming language itself. For some languages, it is truly an optimization, for some, it is required, and does meaningfully change observed semantics.