alt Hacker NewsAnd it's only using AVX-2 and not AVX-512, AMX or ACE. Or built-in GPUs and NPUs (the M series doesn't emphasize matrix multiplication on the CPU side because it already has matrix multiplication units on the GPU, which is always attached).
Replies
ranger_danger • yesterday at 6:33 PM
But do processors actually offload any CPU opcodes to their GPU? That could be quite useful if it can be used to improve execution speed.
➕ show 1 reply
bigyabai • yesterday at 4:54 PM
Before the M5, there was no dedicated matrix multiplication hardware on the Apple Silicon GPU. Their solution was generally using the NPU and AMX coprocessors for tensor and matrix workloads.
saidnooneever • yesterday at 5:49 PM
[dead]
The NPU in previous AMD generation has unfortunately it's own dedicated RAM that is far too small for an LLM.
But even those should comfortably run an SLM of around 100-400M parameters at crazy high speeds and with minimal power usage.