An accountant does our taxes now, but up until a decade or so ago I was always impressed that I could file taxes and all necessary forms, for myself and my small business with dozens of employees, all using the $25 version of TurboTax.
It's been popular to dump on TurboTax for several years now, but I had a great experience using it.
I personally don't trust AI enough to say "here's my data, please do my taxes for me" yet, but you're right, I suppose that is just a matter of time...
I wouldn't trust AI out of the box either.
That said, there's basically nothing stopping someone from creating fee software equivalent or better than TurboTax from scratch using AI. They can even decompile TurboTax source code as an input