alt Hacker NewsIsn't that some kind of gambling if you offload random experts onto the CPU?
Or is it only layers but that would affect all Experts?
Replies
namibj • last Friday at 2:55 PM
Speculative decoding is already gambling.
Isn't that some kind of gambling if you offload random experts onto the CPU?
Or is it only layers but that would affect all Experts?
Speculative decoding is already gambling.
Pretty sure all partial offload systems I’ve seen work by layers, but there might be something else out there.