Failure Notes: How to Run an LLM on a User Laptop

In a recent R&D project we were told to process documents with our own AI model and never send them to a server. Our enterprise users work with resumes, interview notes, meeting minutes, and contracts, so they refuse external transfer in the first place. We decided to run the model on the user PC, and in return we lost control of the hardware. A large share of that hardware is office laptops with no GPU, which is awkward. But what else is there - if AI is going to be democratized, the hardware constraint has to be pushed through.

Work that finishes in seconds on a dev machine takes 23 minutes on an office laptop. This post is the record of the hypothesis we formed to cut those 23 minutes, and of the places we got stuck while implementing it, in order.

With no GPU, it runs on the CPU

The first constraint is the missing graphics card, which puts all token generation on the CPU. On CPU, decode is dominated by memory bandwidth. Every token has to pull the full weight set from memory again, so a laptop at roughly 40GB/s with a 4-bit 4B model at about 2.5GB has a theoretical ceiling of 16 tokens per second, and measured throughput is less than half of that.

The second constraint is RAM. A large share of the target laptops have 8GB, and the OS, our app, and the model all have to live inside it, so even at 4-bit quantization the size we can load is decided before anything else. On device, architecture and RAM and quantization cut the candidate list first, and quality comparison only happens among whatever survives.

Architecture gets caught in the same place. The recent generation, for example, is 75% linear attention by layer, which lowers memory cost on long documents, but the design replaces a growing cache with a fixed-size recurrent state, so under single-request short-input conditions it is more memory-bound than the standard path. Our workload is exactly that condition. A design that pays off on a server does not carry that payoff onto a user laptop.

Growing and shrinking are blocked together

Grow the model and quality can improve, but the ceiling above falls in proportion to size, and if it does not fit in 8GB it will not run at all. Shrink it and it runs, but the speed itself is bad. The 23 minutes came from an already shrunk state.

Bigger does not always mean better quality either. The benchmark items where the size gap opens up clearly are the ones that consume a lot of context, and RAM keeps us out of that regime in the first place. What we actually run is a single short input, so the regime where a large model shows its strength does not overlap with the regime we use.

Hypothesis: small model, no JSON, grammar, thinking off

So the hypothesis was four lines. Use a small model, remove JSON from the output, enforce format with a GBNF grammar, and turn thinking mode off. All four have the same reason, which is that a token is expensive on CPU, so we cut wasted tokens and remove the room for the remaining ones to be wrong.

Why we removed JSON

We used to make the model emit {"name": "...", "email": "..."} itself. A large share of the output is punctuation, a one-character error invalidates the whole object, and on empty slots the model cannot say "I don't know," so it invents values. The third feeds hiring decisions. On top of that the prompt never said what each field meant, so the model was matching the JSON shape without knowing what content belonged where.

My first fix was to take free text and parse it ourselves, and that was wrong too. The fragility just moved into the parser, and failures went quiet there.

(we write into context)   Name:
(model decodes)           Seo-yeon Park
(we write)                Phone:
(model decodes)           N/A

We write the labels into the cache ourselves, so the model cannot mistype a label. The document is processed once and that state stays in cache, so each field only appends a few label tokens. Absence tokens are legal, so the model can honestly say nothing is there, one failed field leaves the rest alive, and values appear on screen as they fill.

We applied the same structure to extract, evaluate, generate, and template paths, and deleted the JSON generation function.

What a grammar does

GBNF is a rule that limits which tokens the model may emit next. Asking for "digits only" in a prompt only tilts probability that way, while grammar sits in the sampler layer and pushes illegal token probabilities to zero, which makes the violation itself impossible.

flowchart TD
  A["Model forward pass<br/>logits over full vocabulary"] --> B["Grammar mask<br/>illegal token probs to 0"]
  B --> C["Sample"]
  C --> D["Advance grammar state<br/>on chosen token"]
  D --> A

So if grammar allows a token but the model assigns it no probability, it will not appear, and if grammar blocks it, the model cannot emit it even if it wants to. One of the places we got stuck below is exactly that second case.

(Aside: one etymology I learned while writing this. GBNF is GGML BNF and BNF is Backus-Naur Form, though the original name was Backus Normal Form. In 1964 Donald Knuth wrote to the journal that this was not a mathematical normal form and that Naur's contribution belonged in the name, and the suggestion stuck. In 1967 P.Z. Ingerman proposed crediting Pāṇini, who formalized Sanskrit grammar, and calling it Pāṇini-Backus Form, and Wikipedia still lists the alias. The GG in GGML is Georgi Gerganov.)

We got stuck six times implementing it

The hypothesis only looked good on paper, so we had to check it against the documents our product actually handles. We built a setup that takes resumes and interview notes, fills slots like name, contact, and years of experience, and then scores them, and we turned all four decisions on at once. We also put a recent-generation small model on the candidate list, since the license was Apache 2.0, a 4-bit build was already published so we could take it as it came, and one model handling both text and images meant we could stop shipping a separate vision model.

We got stuck six times, and four of them are worth keeping.

The first was one newline. Turning thinking mode off means handing the model a "skip the thinking and answer directly" signal in a fixed shape, and the signal we wrote differed from the shape in the distribution's docs by a single line break. The model never recognized it as a signal, and every score slot came back empty. The cause was our own template.

The second was that the model had no way to say "I am done." A grammar restricts which tokens may come next, and the rules we wrote had no path to the token that ends an answer. So the model finished what it had to say and still could not stop, printing whitespace until one document burned through the entire token budget at 1393s. The 23 minutes at the top of this post is that number.

The third was a scoring table that lied to us. We put no digit limit on integer slots, so when a value was missing from the document the prior generation spewed an endless number, that answer was filtered as a format error, and the tally counted it as "correct." The new generation produced one plausible number and the tally counted it as a "hallucination." Same failure, and the table made only the new model look bad.

The fourth was speed. Every time we picked a token we were pulling down a probability table over the whole vocabulary, and all we used was the probability of the token we picked. Deleting one option cut evaluation on CPU from 4+ minutes to 82s and extraction on GPU from 242s to 113s. Much of the gap I had written down as "the new generation is 2.1x slower" was that bug.

All four causes sat in our own code. After also breaking our own rules while trying to tighten awkward fields like email and phone in the grammar, we settled on one rule. The grammar only decides which kinds of tokens may come next, and when to stop, where to cut, and how to align belong in our code.

We moved judgment into a harness

After getting stuck four times, what became clear is that reading results by eye keeps fooling us. So we built a tool that feeds a document in and compares the answers against ground truth we prepared in advance, and added one more check every time we hit a new failure. Apache 2.0: redrob-labs/redrob-eval.

With that tool we measured a small model and the next size up in the same family, across both the prior and the recent generation. The numbers are how many field answers matched.

We expected that within one family bigger scores higher, and on short documents the recent generation's small model did not lose to the mid one, while in the prior generation the mid one led by six fields. Had we measured one size and stopped, we would have written the generation comparison backwards.

We still did not adopt the recent generation. When months-of-experience was missing from the document it returned 0, and going up a size did not fix that, and Korean worst-case tokens per character rose from 2 to 3. More tokens means more time to read the same document, and we were fighting on the side where each token is expensive.

What we ship now

So we stayed on Qwen3 and pick the size by the user's machine.

The office laptop that took 23 minutes at the top of this post now takes 88s. That is too long to watch a screen for and short enough to leave running, so evaluation happens in the background while the user does something else and notifies them when it finishes. A machine with a GPU answers in 7s, close enough to just wait for. That 12.6x became the line in the product between what a user can wait on and what they cannot, and chat cannot absorb 88s on either machine, so it does not sit on this configuration.

What I hope you do not copy

• Before reading a comparison table, look for paths where a failure gets scored as a success.
• Do not judge a model from one size; measure two sizes in the same family.
• Use grammar to enforce format only, and decide when to stop in code.
• Do not budget tokens by character count. English-only checks pass; Korean and Indic break.
• Do not let a failure quietly fall through to another path. Only the checks that ran at startup caught anything properly.

In one line: when the result says "the model is bad," suspect your own code first.

We should contribute upstream

The engine is llama.cpp. Some of the blockers were bugs in our own rules and some are still on the engine side. For the latter, filing issues and PRs is the right move rather than hiding them in a fork only we use, and that is how we are working through them.

What is still unfinished

We only measured single short inputs, not larger scale. That is where large models are said to win, so without that run we cannot generalize "small is enough."

Accuracy used English synthetic documents only. India is the home market and Korean is next, and we measured three tokens per Hangul character without ever scoring extraction on Korean documents. Scores were also single-shot, so we do not know the variance.

I expect the seventh wrong call to land here.

Further reading

• Backus Normal Form vs. Backus Naur Form (CACM 7:12, 1964) - Knuth's letter. How BNF got its name.
• Backus-Naur form - Including the Pāṇini-Backus Form alias.
• ggml-org/llama.cpp - The inference engine we use. History and format via secondary sources.
• llama.cpp issue #19894 - Performance regression and CPU fallback reports for that architecture.
• unsloth/Qwen3.5-2B-GGUF / run docs - Distribution and thinking-mode defaults used in verification.
• arXiv:2603.05931 - Analysis that linear-attention decode is memory-bound at batch 1.
• redrob-labs/redrob-eval - Text routing benchmark harness. Apache 2.0.

External facts were checked against the sources above, and own numbers are measured; the sample is small and mostly single-shot.