I agree with the gist of your sentiment, but I struggle with how to reconcile it with my use of these tools. My use LLMs is precisely because I can never hope to express myself clearly. I have many fuzzy, disparate thoughts and it was a struggle to express them at all before this time in which I can do a long rambling voice transcription (absent of social pressures) and continue doing the same throughout the exploration and drafting processes. In my view, I give them a seed or a morbidly obese skeleton and LLMs take that into something a lot more coherent for myself and for others. I was passive before in my life, now I’m engaging. As two concrete examples: I never would have sent an email to my city regarding snow removal last winter when their failure to remove snow from the only sidewalk going over the interstate. I never would have pulled together my previous coursework and e-mails to send to my current professors in a petition for transfer-credit equivalency when it became clear that my deteriorating health would not allow me to finish the semester as-is. I never would have remembered all the concerns that I want to bring up at medical appointments. That is not to say these things were ever impossible, there are coping strategies available, and whatnot, but I’m in my late 30’s; I know what I did and not get done in the last 10 years, and “AI”-tooling/environments have helped me express my dreams and passions and concerns, etc., etc., etc. over the past years in a way that was simply not practically available to me, with a new capacity that I simply could not previously hope to match due to limited and fragmented time,attention, and energy. While I was hand-typing this on my keyboard, I could have instead been talking, reviewing, redrafting in spurts as I go about my morning, instead of a solid 45 minutes at max effort on my own part, as I did here (and I don’t feel that I conveyed myself half-as-well here as I could have achieved at much less cost to my energy, etc. by using an LLM). I do take offense when people do not review and vouch for things written on their behalf by LLMs.

How do we make room for their use as a cognitive-prosthesis (if you will allow such framing) without losing humane-ness?

Wow, you must be one very unique and special person if you converse the way you write. By definition, a carefully written and edited piece will be much more refined and cogent than an extemporaneous conversation.

Just watch an interview with a famous author. Or compare a legal brief to what court actually sounds like.

I agree with the sentiment with a small caveat, since IMO good writers don't write the way that they talk. The very simple reason is that you can revise your writing but not your talking. But yeah, I completely agree that using an LLM to write your words is intellectual catfishing, and that's a great way to phrase it.

I'm in the process of writing a pretty long essay that has taken a few months. I've written the first draft, and I'm almost done with the second draft, which incurred substantial revisions. At the end of the process, maybe no sentence will be something I would actually utter in person, but the writing is still something I created, and as such, it is a representation of who I am as a person. Had I used an LLM to do any of the writing, that would no longer be the case, and the writing would at best represent me as a person when the LLM is at my side, and at worst (and most likely) not really represent me at all.

That's privacy.