What You'll Learn in This Post
This is a practical guide to KV cache optimization — the technique that lets you squeeze more tokens out of limited GPU memory. We're covering how reducing "thinking" traces (reasoning steps) affects inference throughput and why this matters for local models like Qwen3.6-27B.
The real numbers come from Prague startup BottleCap's work fine-tuning Qwen, where they achieved a 46% average reduction in reasoning tokens across 12 benchmarks while actually improving answer quality on GSM8K (93.3% → 96.5%). We'll explain how that translates to your own hardware.
KV Cache Basics
Every transformer model maintains a key-value cache during inference — one entry per token generated. The cache grows with every token the model produces, consuming GPU memory at a rate proportional to context length squared. This is why longer contexts kill throughput on consumer GPUs.
The optimization comes from recognizing not all tokens are equal. Some tokens represent pure reasoning traces — internal monologue that doesn't contribute to the answer itself. Others carry actual content: final answers, explanations, and direct responses to prompts. If you can halve the former without affecting the latter, your effective context window doubles for the same memory budget.
Thinking Tokens vs Content Tokens
BottleCap's approach treats these categories separately during fine-tuning. They created "ThinkingCap" models that learn to distinguish between reasoning steps and content tokens at a structural level — not by ignoring reasoning entirely, but by making it more concise while preserving answer quality.
The result: 74% fewer thinking tokens on GSM8K evaluation (the math word problem benchmark), with accuracy improving from 93.3% to 96.5%. That's counterintuitive at first — reducing the model's "thinking" made it smarter. The insight is that excessive reasoning traces often represent exploration paths the model eventually discards anyway. Compressing those paths into more direct routes saves memory without losing reach.
The Throughput Multiplier Effect
Here's where the math matters for local inference:
| Scenario | Context Budget (effective) | Tokens per Second |
|----------|---------------------------|-------------------|
| Baseline — no optimization | 8K tokens | ~20 tps (Qwen3.6-27B on RTX 4090, FP8) |
| Halving thinking traces | 16K tokens effective | ~40+ tps same hardware |
The throughput improvement comes from two sources:
1. Reduced cache growth rate. If you cut reasoning steps by half while maintaining the same answer quality, your token generation slows down — but that's intentional. You're trading raw speed for more usable context window. The 46% average reduction means you can fit roughly twice as many useful tokens into the same memory budget.
2. Better attention utilization. Models don't use all their attention capacity on reasoning traces during inference — those are often internal steps that don't contribute to cross-attention patterns with user content. When you compress reasoning, the remaining tokens (actual answers) get more of the model's focus per unit time. The effective "thinking speed" increases because each generated token carries more information density toward the final answer.
What This Means for Your Hardware
On a consumer GPU like an RTX 4090 with 24GB VRAM, Qwen3.6-27B in FP8 format occupies roughly 15–18GB at maximum context — leaving only ~6GB headroom for system overhead and future tokens. That's why you get about 20 tokens per second on average generation: the model is filling its memory budget with every token, most of which are reasoning traces that don't move the needle toward completion.
Halving thinking traces means you can now fit 16K–32K effective context into the same 24GB envelope. You get twice as many useful tokens before hitting OOM, or you finish prompts faster because each generated token carries more informational weight toward the answer. The model still "thinks" — it just thinks more efficiently.
How to Apply This Locally
You can't directly fine-tune a thinking-cap model yourself without the same training data and compute budget BottleCap used (they're not sharing their exact recipe, just publishing results). But you have three practical levers:
1. Use ThinkingCap where available. The Qwen3.6-27B ThinkingCap variant is published on Hugging Face in GGUF and FP8 formats. If your hardware supports it — a GPU with at least 16GB VRAM for the base model — you can run it as a drop-in replacement. It's designed to be interchangeable; Qwen3.6-27B → ThinkingCap-Qwen3.6-27B is a direct swap without changing your code or prompts.
2. Quantization with awareness. FP8 quantization (supported on Ada Lovelace GPUs like the 4090, RTX 50xx) already compresses the model weights — but it doesn't compress reasoning traces in generation. The ThinkingCap models come pre-optimized for this: they produce fewer tokens per answer while maintaining quality. That's the key difference from standard quantization alone.
3. System-level context management. Even with thinking-cap models, you control how much context gets passed to the model via your application layer. If you're running an agent that chains multiple turns in a single session, consider splitting sessions or summarizing intermediate steps into system prompts instead of passing raw history. The optimization works best when you pair reduced reasoning traces with intentional context pruning — not because you're throwing away information, but because you're moving it out of the KV cache and into structured memory where retrieval cost is zero.
What This Doesn't Do
This isn't about making models think slower or less thoroughly. It's about compressing the representation of thinking during inference without affecting the model's reasoning ability at test time. The fine-tuning teaches the model to produce more direct paths to answers — it doesn't disable its reasoning capacity, just how much explicit work it shows for each token generated.
Think of it like a writer who drafts an outline before writing full sentences: you still get the same story, but the draft itself is shorter and faster to read. The ThinkingCap models are doing something similar at the transformer level — they've learned that certain reasoning paths can be compressed into fewer tokens without losing answer quality.
Bottom Line
If you're running local Qwen3.6-27B on limited hardware (RTX 4090 or similar), halving thinking traces via ThinkingCap models gives you roughly a 2× effective context window boost — from ~8K usable tokens to ~16–32K — while improving answer quality on benchmarks like GSM8K. The throughput gain comes not from faster decoding, but from more information per token and reduced KV cache growth rate.
The model still thinks; it just thinks in fewer steps. And for local inference where memory is the bottleneck, that's exactly what you want.