"Timescales that makes sense" may be a human reasoning but not necessarily the reasoning of inconceivably advanced timeless civilizations. Sure, that planet of fish may be harmless now, but what about in a quick three billion years when they have FTL and AGI and Von Neuman probes and Dyson spheres and antimatter bombs? Easier to click the delete button now to save the trouble later.
The laws of physics apply to the civilization of alien fish too, it's not a human specific timescale. They have nothing to fight over with us, no reason to spend effort developing weapons to reach us, if they do have weapons they'd have a much higher probability of wiping themselves up first, and no way for their weapons to reach us within a time boundary that makes sense for any sentient race.
Dark Forest seems to be based on a scifi/fiction need to have conflict with "the other", which is thrilling but doesn't necessarily reflect the real cosmos.