From the comments:

The notion of "believing in" axioms is absurd ... as absurd as believing in the rules of chess and disbelieving the rules of checkers. Each set of rules or axioms forms a system (possibly degenerate if the axioms are inconsistent). The rules, axioms, and systems aren't "true" or "false" -- that's a category mistake. Studying the systems resulting from the Peano axioms or ZFC is a worthwhile endeavor. Studying the systems resulting from finitist axioms may well be too, but the nonexistence of infinities in the latter doesn't mean that they don't exist in the former--that's crackpottery. Mathematics has room for both sorts of systems.

Which axiomatic systems best model the world is a different matter. Now we're in an empirical realm, where there are observations, evidence, facts. And observational reports are necessarily finite, so even if there are "real" infinities they can't be demonstrated. But "all models are wrong", so both infinite and finitist axiom systems might serve as good approximations.

Likewise with computer systems--all actual computer systems are finite state machines, but it's convenient and useful to model them as Turing Machines that allow for both infinite non-halting systems and finite halting systems.

And since both this medium and I are finite, I will stop there.