1) am I allowed to scan the entire corpus in advance before I populate the dictionary? Is this a stream, or is their an EOF marker I will know in advance?

2) if 1) then "prediction" isn't the word I'm looking at.