I tried something similar by putting together a "global ledger" of speaker identities. This was more happening during the call than having a predefined one but I just couldn't get it to work properly. The issue being that as soon as some speech gets assigned to a new label or an incorrect one, everything ends up getting misaligned and gets messy quickly.

I might take another look into doing it in a different way that gradually builds up from successful calls, I just need to think of how to do this in a simple(ish) way for non-technical users and a way that still works well enough on low to mid tier laptops.