I've done essentially the same thing before (using my existing Go2RTC and Frigate setup) but the challenge I ran into is that the microphones on my cameras just aren't very good.

You can adjust which detections are accepted based on confidence level, so it might be that the settings are very strict.