You can download an app that does that. You don’t have to use Siri, and it can be turned off entirely, or decoupled from AI entirely, and the wake word can be disabled entirely. A lot of options are available.

Make a Shortcut that will open your local voice assistant app of choice when you press the action button.