tedd4u • yesterday at 5:28 PM

Agreed, seems like the M5 has already made steps in that direction, with 4x prompt processing / prefill performance vs. M4. [1] And token generation also got a 10% boost. The data below is only for Pro & Max but I think the base M5 got the same relative boosts vs. M4 base.

    Chip         BW (GB/s)   GPU Cores   Q4_0 Prompt   Q4_0 Gen
    M4 Pro (20c)    273         20          439.78        50.74
    M4 Max (40c)    546         40          885.68        83.06
    M5 Pro (20c)    307         20      ~1500 to 1700    ~56
    M5 Max (40c)    614         40      ~3000 to 3500    ~92

[1] https://www.hardware-corner.net/m5-pro-m5-max-local-llm-4x-f...

aurareturn • today at 1:05 AM

  Apple's biggest bottleneck for real-world inference is prefill processing.

Much less true since the M5 generation. Prefill, aka prompt processing, got a 4x increase.