Have you tried call screening? I know for sure apple phones can do this but I believe some android phones can too?

For me, all unknown calls are screened. Which means if someone who’s not in my contacts calls me, they get some automatic voice saying something like “this is apple screen calling, please let me know who you are and why you are calling” The person calling then either hangs up or says something, if they say something I can see the live transcription and answer the call immediately if it’s important.

It’s been incredibly effective. I’ve been using it for a couple of years now with no complaints from anyone for not being able to reach me, and not once have I answered a scam call during that period.

I’ve had a few situations similar to what you describe, some service I’m using like a doctor, realestate agent etc called me and I immediately recognised who they are and why they are calling and answered.