You can save steps when diffusing from a pre-drafted canvas instead of seeding with random noise! If you have a drafter that is good enough, steps drop by ~50% or more, and tok/s can definitely increase.

I'd be curious to see your CUDA approach - happy to combine efforts as well.