Running an LLM locally without regrets: when it pays off, the hardware math, and the failure signals

Local LLM deployment is attractive because the token price is zero, but the real cost is hardware and maintenance. The decision should start from one question: how many tokens do you actually run per month? If the answer is below roughly the cost of a mid-range GPU card over two years, buying a card for local inference is usually more expensive than API usage.

The hardware math is concrete: a model needs roughly its parameter count in gigabytes of VRAM for a 4-bit quantized build, and about twice that for 16-bit weights. A 7B model fits in 8 GB, a 13B model wants 16 GB, and a 70B model needs 48 GB or more. Throughput also matters: consumer cards deliver far fewer tokens per second than API endpoints, so batch or streaming workloads feel different locally.

The failure signals that mean the local setup is losing you more than it saves: the model lags behind the hosted version you actually need; the GPU idles most of the day because the workload is bursty; or your team spends more time managing the runtime than writing code. Each of these is a reason to stay on a hosted API and rent the capability instead of owning it.

When local does pay off: privacy-constrained data that cannot leave the machine, fully offline environments, and steady high-volume inference where the card is busy every day. The method is to re-run the token count and hardware math every quarter, because both model sizes and card prices move.