The problem to me seems to be that we are trying to map everyday language onto the mathematics. Even though we have a symbol for infinity, infinity is not necessarily a "thing" that the symbol points to.

In analysis, when we write "the limit as x goes to infinity" this translates into a logical statement like "for all x, there exists some y > x such that ..." I don't really see anything conceptually difficult or contradictory here.