What’s the context size?
128k, 256k is also possible, but my tok/s drops off and the performance isn’t good.
128k, 256k is also possible, but my tok/s drops off and the performance isn’t good.