I found [1] which appears to offer state-of-the-art performance of ~1000s latency for a FHE GPT-2 transformer block, equating I think to 3.33h inter-token latency (0.00008 token/s) for GPT-2(small) which has 12 transformer blocks. This result is using optimised packed arithmetic operations on a GPU as well--so seemingly is unlikely to have much performance upside from further optimisation.

I'm not sure I've interpreted [1] correctly though, and would appreciate correction if necessary.

[1] https://arxiv.org/pdf/2604.04783 -- GPU Acceleration of TFHE-Based High-Precision Nonlinear Layers for Encrypted LLM Inference -- Guoci Chen, Xiurui Pan, Qiao Li, Bo Mao, Congming Gao, Chengying Huan, Mingzhe Zhang, Jie Zhang -- Apr 2026