(Co-)author here :)

I'd absolutely still use Highway, and do. My experience is that even two separate implementations diverge over time and I'd have low confidence in bringing the same updates and improvements to all, even with LLM assistance.

Our programming model is 1) an agent+human to generate the algorithmic approach, 2) a C++ library (Highway) to translate to intrinsics, while filling in gaps + allowing customization, 3) a compiler to generate the actual code with some optimizations.

Asking the compiler to do #1 is a pipe dream: compiler friends tell me they are not going to devise new shuffles/data layouts (like what VQSort does). Conflating #2 and #3 means a custom compiler/IR which has high engineering costs (ABI boundaries, hard to debug/profile/sanitize). And doing #3 at runtime (JIT), or moving fusions into #3 (MLIR), vastly complicates the compiler. We can still get runtime adaptability thanks to Highway's multi-target support. Fusion has been much easier to implement manually for LLMs than to construct a general fusion infrastructure. Templates hide most data type differences and we see 2-5x speedup vs llama.cpp for 128k prefill+batch decode on Zen5.

Instead of requiring a compiler to do heroic transforms at runtime, and get it right every time, we can do all kinds of agentic exploration, then verify the result/approach, check in the source code, then we 'just' have a C++ compiler afterwards. And if/when something breaks, it's easy to update centrally, in code we can modify directly, rather than indirectly via updating a compiler.