alt Hacker NewsA lot of criticism of python often mentions the whitespace as lexical scope tokens, and that criticism is usually posited by users of the language.
As implementer of an interpreter, did you feel that whitespace for lexical scoping made the job of writing the lexer significantly more complex?
Replies
> did you feel that whitespace for lexical scoping made the job of writing the lexer significantly more complex?
Significant indentation requires a more complex lexer because it means the lexical grammar is no longer regular. The lexer can't just be a finite state machine, instead it has to maintain a stack of previous indentation levels.
But I don't think many modern languages have a regular lexical grammar anyway. Without significant indentation, some other features still require the lexer to maintain a stack - e.g. string interpolation (Python's f-strings).
> that criticism is usually posited by users of the language.
Uhhh, no. Sure, it's posited by people who feel they are are forced to use it, but it's basically unlearning other syntax.
Here's a study about people with no experience. They do better with python:
https://www.researchgate.net/publication/262256894_An_Empiri...
When the scala language made whitespace optional, it was very divisive, but now it's extremely well accepted.
And, there are multiple white space symbols!
<space><space><tab><space>
is different than
<space><tab><space><space>
So you also have to track the actual sequence of counts of white space used for each level, rather than just a simple count.