Local AI, With the Meter Running

Talk is cheap. Show me the numbers.

Every week somebody tells me local AI is the future, and every week somebody else tells me it’s a toy and you should just pay for the API. Neither of them has ever measured anything. They have opinions. Opinions are free, and that’s about what they’re worth.

So here are numbers.

A client of mine works with stuff that should not leave the building. Customer data, contracts, internal documents. Some of that was still going through Claude, because Claude is good and the team trusts it. Fine. But a cloud API is outside the building. That’s not a technicality, that’s the whole definition. And for a German company it immediately turns into the usual dance about data processing agreements, third-country transfers, and what exactly happens to your data when something goes wrong. Nobody enjoys that dance.

The brief: make the local models good enough that switching isn’t a downgrade. Then find out what it actually costs. Not “estimate”. Measure.

The setup

Two machines. Same desk, same UPS.

One is a perfectly normal workstation: Ryzen 9 9950X, 64 GB RAM, RTX PRO 4500 Blackwell with 32 GB VRAM. It runs Qwen3.8-27B, dense, 4-bit quant, with a small MTP head for speculative decoding. 131k context over four slots. That’s 29.9 of 32.6 GB used. Yes, I could squeeze in a bigger quant if I cut the context. No, I’m not going to. Context matters more than the last few percent of weight quality for this kind of work, and anybody who tells you otherwise hasn’t fed a model a real contract.

The other is a DGX Spark, 128 GB unified memory, running Qwen3.8-Flash-Next. 125B mixture-of-experts, about 6B active, 3-bit quant.

Both run llama.cpp, built from source, every release into its own directory. Why? Because a new build should never, ever touch the server that’s currently running. That’s not paranoia. That’s just how you do it.

Neither server is reachable from the outside. In front of them sits a LiteLLM router. One key, one OpenAI-compatible endpoint, two model names. That’s all a client ever sees:

client ──► LiteLLM :4000
             ├─ qwen3.8-27b        ──► llama-server (workstation)
             └─ qwen3.8-flash-next ──► llama-server (Spark)
                                       (Spark down → falls back to the 27B)

The fallback is the important part, and people always skip it. Spark busy, rebooting, network gone? The small model answers instead. The request does not fail. The agent on the other end never notices. That’s the difference between a demo and something people can actually use.

And because it speaks the OpenAI API, switching existing tools over is changing a URL. That’s it. No migration project.

The boring parts ARE the work

Getting a model to answer a curl request: one afternoon. Getting it to the point where a team can depend on it: the rest of the week. And none of that week had anything to do with AI.

The driver. NVIDIA’s CUDA repository ships a pin that ranks it above Ubuntu’s own packages. And it contains driver libraries. So some random future apt upgrade will happily replace Ubuntu’s signed driver with NVIDIA’s, the libraries no longer match the kernel module, and with Secure Boot the module simply doesn’t load anymore. Your expensive GPU is now a space heater. The fix is changing one number in one file. Finding out that you need to change it is the expensive part. That’s always the expensive part.

The router. LiteLLM needs its database client installed even when you don’t use a database. Without it, every error, including a completely ordinary 401 for a missing key, comes back as a 500 complaining about a missing Python module. An authentication failure that reports itself as an import error. Think about that for a second. Error messages are user interface. This one lies.

Monitoring. Telegraf on both boxes, InfluxDB and Grafana on the Spark, alerts into a Google Chat space. GPU power, CPU package power from the RAPL counter, VRAM per process, tokens per second straight out of llama-server. And since the alerting lives on the Spark, the workstation runs a small watchdog that checks the Spark every minute. Because otherwise the one failure you most need to hear about is exactly the one that can’t tell you. Monitoring that dies together with the thing it monitors isn’t monitoring.

Power. The UPS talks USB to the workstation. NUT reads it and decides by itself when the battery is low: 30 % or five minutes left. Then it takes the Spark down first, waits for it, and then itself. Every step goes to chat, sent directly from the workstation. Not from Grafana. Grafana is on the machine that just shut down.

That’s what “self-hosted” means once you’re past the demo. A cloud model is a price per token. A model under your desk is a small operations problem that happens to also generate text. If you’re not willing to own that problem, don’t do it. Seriously.

Speed

One hour of benchmark this morning. Every request streamed through the router, every watt logged.

Two things.

Four parallel streams give the 27B almost three times the throughput of one. So if your batch jobs and sub-agents are sitting in a queue, you’re doing it wrong. Run them in parallel.

And running both models flat out at the same time costs neither of them anything. The router is not the bottleneck. The GPUs are. As they should be.

So the 27B is the workhorse. Sub-agents, bulk work, anything where waiting is the cost. The Spark gets the hard problems, where a slower answer from a bigger model is actually worth waiting for.

Watts

The UPS reports its load, so I measured at the wall. Both machines, everything included:

Look at rows two and three. The RTX PRO 4500 hits its 200 W limit with a single stream. Three more streams: more tokens, same watts. Batching is basically free. If you’re not batching, you’re throwing that away.

Money

At 0.35 €/kWh, every watt at the wall counted, idle included. No creative accounting. Per million output tokens:

For the 27B, electricity is seven to thirteen times cheaper than renting the exact same model. As long as you keep it busy. Input tokens, which is where agents burn most of their volume, it’s even more lopsided: about two cents per million locally, three to 38 cents in the cloud.

Now the Spark. And here’s where I’m going to disappoint the local-AI crowd: the cheapest cloud offer for Flash-Next costs about the same as the Spark’s electricity. With a single stream, the cloud is cheaper. The Spark does not win on price per token. Period. It wins because the data never leaves the room. That’s a perfectly good reason. Just don’t pretend it’s about price.

And now the line everybody conveniently leaves out of their blog posts:

Idle costs about €376 a year. Before a single token. Ask a few questions a day and that number is basically your entire cost. Run the 27B saturated around the clock and it makes about 17.6 million tokens a day and earns back a year of idle in about ten days. Your real usage is somewhere in between. Figure out where before you buy hardware, not after.

And no, the hardware isn’t in these numbers. Purchase price and depreciation decide the real break-even, and that depends on what you already own and what you’d have bought anyway. I’m not going to make up a number for you.

You don’t need me for that part anyway. GPU and Spark prices are one search away, and everybody has a calculator. Take the purchase price, spread it over the years you’ll actually run the thing, add the idle line from above, and put that next to your current cloud bill. Do the math before you order hardware. It takes ten minutes.

The engine is not the car

Car analogies are usually lazy. But I spent years in automotive, so I’ve earned one.

The model is the engine. The harness is everything else: steering, wheels, brakes, dashboard, fuel system. Nobody buys an engine. You buy a car. And the exact same engine goes into a sports car or a delivery truck, and you get two completely different vehicles.

Most of the “local models are useless” takes I read are somebody bolting an engine to a pallet and complaining that it doesn’t drive. They paste a prompt into a chat window, get a mediocre answer, compare it to a frontier model with an entire product built around it, and call it a day. Of course it loses. They compared an engine to a car.

The harness is what turns a model into something that does work. The tools it can call. The context it gets, and the context it doesn’t get. The instructions. The files it can read. The checks that tell it it got something wrong, and the loop that lets it try again. And yes, the boring plumbing from above: the router, the fallback, the monitoring.

That’s where our results came from. Not from a bigger model. From putting a medium-sized model into a proper car. The 27B on its own is decent. The 27B in the right harness does real work.

Which harness? Less important than people think. Claude Code, OpenCode, omp, LangChain, or something you build yourself for exactly one job. The router speaks the OpenAI API, and through LiteLLM the Anthropic one too, so pointing any of them at the local models is one URL and one key. What matters is not the brand. What matters is that the harness gives the model tools, the right context, and a way to check its own work. Pick the one that fits the job, not the one with the most GitHub stars.

Which also means: when you evaluate a local model, test the car, not the engine. The whole setup, on your tasks. Anything else measures the wrong thing.

So should you?

Local makes sense when three things are true at the same time:

• Your data is confidential. Customer data, contracts, engineering documents that can’t leave the building. Then the question isn’t “is the cloud cheaper”. The question is “is the cloud even allowed”.
• You have volume. Agents, batch jobs, documents by the hundred. The per-token cost only falls off a cliff when the GPU actually has work.
• Somebody owns the operations. Drivers, updates, monitoring, the UPS. None of it is hard. All of it is work. And it never stops.

If you ask a chatbot five questions a day, don’t bother. You’ll be paying for a very expensive idle loop.

And let’s be honest about one more thing: a 27B model is not a frontier model. A good harness closes a lot of that gap. Not all of it. For a lot of everyday work it’s more than good enough. For some it isn’t. Which is which for your work isn’t on any data sheet, and the vendor won’t tell you. The only way to know is to test it on your own tasks, with real examples and somebody who can actually judge the output. Anything else is guessing.

The end

Local AI isn’t free and it isn’t magic. It’s a box that pulls 123 watts doing nothing, a pile of small operational decisions, and a per-token cost that beats the cloud by an order of magnitude when you keep it busy and loses to it when you don’t.

What you get is control. The data stays where it is. The model is the one you measured, not whatever the provider decided to ship this week. And when it breaks, you can see the request, the response, and the watts. And you can fix it.

That was the brief. The meter says it holds.