it's the widening (for example: fp16 * fp16 = fp32) instructions, which are useful on some loads.
I assumed the comment above is referring to operand dtype and not accumulator dtype since they mentioned ops. In general even for nvfp4 you use fp32 accumulators because you are usually reducing over large Ks which mean large values.
alt Hacker News
I assumed the comment above is referring to operand dtype and not accumulator dtype since they mentioned ops. In general even for nvfp4 you use fp32 accumulators because you are usually reducing over large Ks which mean large values.