We were getting routine audits from security companies as well as public universities around the world on our OSS healthcare project. I would say it did give us in some way a false sense of security as we had the belief that a lot of the major bugs were addressed. The last 6 months have been very eye opening as the slew of reports have been difficult due to how many there have been, even with all of the false positives there's been a lot of legitimate vulnerabilities. This has been taxing for us to go through (and humbling as well).

That said, if we were a closed source project, I don't know that our own tooling would have discovered all the issues using LLMs as we are seeing more and more creative ways of people exposing exploits that I never would have thought to put into the LLM prompts.