Sure you can, via e.g. Termux. Have done so.
Also, I don't want to run Xcode, it is arguably the worst development environment and hasn't gotten any better over the years.
I have to use Xcode regardless, if I want to develop for iOS or macOS. I don't have to use Android Studio, I got plenty of alternatives for that.
You're welcome to write code however you want and invoke xcodebuild via the command line, granted so long as it's on a Mac. But look at all the vibe coded apps we're seeing these days - plenty of folks are developing without opening Xcode.