I wonder if there’s a simpler and less interesting answer? That it’s just picking up on voice and style, not anything that would apply to the average non-writer?
This person is a skilled writer. Part of that skill is developing a unique voice and style. The AI can identify that - and while that’s certainly impressive because it can identify even relatively niche authors, it has nothing to do with a wider capability to deanonymize people based on arbitrary written text (ex Facebook or text messages).
If you are a professional musician, it’s not difficult to identify a well known musician / recording after listening to only a few seconds - whether they’re playing Bach or Rachmaninov, the style is just “them” - this is the same thing. But you couldn’t take some anonymous high school musician and guess who they were, even if they were your student - the median quickly regresses towards a homogenous, non-distinct style / voice.
It appears to largely be able to identify people who are prolific public writers. I just asked it to identify a whole bunch of comments I've made on private Discord servers and it said it couldn't for all of them, even when they had details that would identify me uniquely to anyone who knows me well enough (work locations, city I live in, wife's employer, my employer).
All the people it seems to be identifying are bloggers, journalists, and/or published authors.
Some tens of years ago I used to hang around on an online forum related punk, hc, heavy metal etc. music, and it had a recurring problem of quite unsavoury individuals coming there to spout racism, nazi-ideology etc. They of course got banned, but returned with a new account trying to ” lay low” and be more indirect in their rhetoric. However even this did not work because the admin of the forum had unbelievable nack to recognise people based on their style of writing.
Web has never been as anonymous as people think and this writer seems to have a clear confusion what it really means to be anonymous and hide your identity. Really, having a distinct writerly voice and being a published writer is pretty much the same as leaving your finger prints on the axe.