At most points in time resources are scarce and not all life can be supported. At those times beings with a tendency to say “them” will give way to beings who most strongly assert “us”.
The scarcity view of life is so deeply ingrained in some of us that its become an entire world view. Some folks cant even imagine a post scarcity society.
Try though. Space is mind blowingly large and anything with the ability to traverse interstellar space alteady has access to unlimited resources and energy, likely closer than us. We live in a tiny backwater of the galaxy.
We literally have nothing of value to such a civilization. Gold and diamonds a just more rocks to any serious space faring civilizations. Energy is also unlimited and free.
Unless they find us delicious our only value would likely be an exchange of knowledge.
Comments like yours make me think that sci-fi shows (and a lot of literary sci-fi) like Star Trek did a grave disservice to our conception of advanced spacefaring civilizations. They were always focused on alien planets, assuming that once a race was able to travel the stars that they would still chain themselves to a gravity well. It's like imagining an industrial society where everyone still lives on a farm like their agricultural predecessors. Nothing makes sense if you start with that assumption, living space becomes scarce, utilities are hard to provide, travel is cumbersome and a lot of things simply aren't possible. The same way that industrial societies urbanized and concentrated in cities, spacefaring civilizations will move to orbital structures that allow them to design their environments. These may be located near planets the same way cities are located near important geographical features but there will also be many that are only in orbit around their host star. Habitable worlds will become the equivalent of bucolic countryside estates or national parks. Most industry will be located around lesser gravity wells such as moons or minor planets where mined materials are easy to transport to orbit and there aren't biospheres or atmospheres to worry about. It would take a long time and a lot of expansion to exhaust the materials in a solar system.