I think of the context window as a pot of soup that you add ingredients to between meals. If you have a relatively focused recipe and you are able to add only the ingredients you want, the soup stays good. If you or the agent add an ingredient that isn't fresh, it is going to be difficult to salvage and it is better to start over with a new pot.

It is not that agents can't function with a large context window, they can if that information generally has a desirable signal (like a large initial document or a well-focused session). Mistakes and the confusing signals that come out of fixing mistakes are why performance degrades. I start to trust the context window less not as a matter of size but the amount of friction we run into. The friction can be random but it is more often an issue with the path that I have us on.