alt Hacker NewsThis is exactly what I came to say. The price of Flash is so cheap that trying to run it locally or with your own hardware is pointless. I was using it about a month ago to program some stuff and ran it for 4 days non-stop and it cost me about $2.
Replies
> trying to run it locally or with your own hardware is pointless.
Serving local models has advantages other than price. If you work in restricted industries, or have a strong need to protect your IP, or if you just value privacy more than cost, you now have options.
If you don't do any attention steering, custom decoding or meddle with the weights maybe. Services are worthless unless all you do is write positive prompts.
As others have mentioned, there's the privacy factor as well.
With the cost of electricity, hardware depreciation and tok/s it rarely makes sense to run locally.
It's expensive to run it locally at full quality, but at least on my setup, its about 5 times faster than any API, and is completely private.