Almost as crazy as running random arbitrary code from unknown third parties automatically. Yet that's the unquestioned status quo for the web and if you don't do it you're considered a weirdo. This is whataboutism, of course, and does not invalidate your claims. But really the only risk comes from the harness that does tool calling, not inference. Just running inference without it is not at all risky.