alt Hacker Newsquanting is actually cheap and you can compress a model that does not fit on a GPU. You can process layer by layer, this is what the sequential processor in llm-compressor does.
quanting is actually cheap and you can compress a model that does not fit on a GPU. You can process layer by layer, this is what the sequential processor in llm-compressor does.