I built a local AI agent last month that lives entirely on my home machine. It answers questions from our notes, reads docs, manages tasks across services — all with zero network calls except when explicitly requested. The question I get most often after showing it around: "What hardware do you need for something like this?"
This is the 2026 answer. No cloud costs, no data leaks, just raw compute where you control it. Here's what we're running in production and what the landscape looks like for anyone else wanting serious off-device AI capability this year.
Why local inference matters now (and why you might still go to the cloud)
Two years ago, I'd tell you: if you wanted a model that could handle real work — code generation, document analysis, creative writing — you needed the cloud. The hardware requirements were just out of reach for most consumers. The latency was too high for anything besides quick queries. Privacy was a theoretical concern at best.
That's changed. We're now in what feels like a transition period between "cloud only" and "local first." You can run serious workloads on consumer hardware, but there are real tradeoffs you need to understand before building your setup.
NVIDIA RTX consumer GPUs: the current reality
NVIDIA's consumer line — 4090 through the newer 50 series — has been the de facto standard for local inference since we got started. Here's what matters now:
VRAM is everything. This hasn't changed. If you want to run a model that can handle real work without constant offloading to system RAM, VRAM is your bottleneck. A modern 4090 with 24GB VRAM lets us run Qwen3-8B in FP16 without issue — roughly the same as 24GB of unified memory on Apple Silicon. The newer cards push this higher, but you're still limited by that single number.
The RTX 50 series is interesting. We haven't tested it ourselves yet since our setup came online before launch, but from what we're seeing in community reports: the architecture improvements are real. Better memory throughput means faster generation speeds for larger models than earlier cards could handle at similar VRAM levels. It's not a magic bullet — you still need that VRAM — but if you're choosing between two GPUs with comparable memory capacity, the newer architecture does matter for throughput.
Real numbers from our Qwen3-8B setup: On an RTX 4090 running FP16 without offloading, we're seeing roughly 25–30 tokens per second on a single core of CPU during generation (the rest of the cores are busy handling other tasks in our multi-agent system). That's actually impressive for a consumer card — it means you can run a conversational agent that feels responsive without needing dedicated hardware. For comparison, running the same model on Apple M4 with MLX and 24GB unified memory gets us roughly comparable throughput, but with slightly less headroom if we want to load additional context files into the KV cache simultaneously.
The real limitation is still VRAM. You can't squeeze more out than what's physically there unless you accept offloading to system RAM, which kills both speed and model size. This is why the 4090/50-series cards have been so popular — they finally give you enough headroom for serious workloads without immediately hitting that wall.
Apple Silicon: MLX changes everything
If you're on Mac with M1 or later, the situation has changed dramatically since we started building locally. The combination of unified memory and optimized inference paths means you can now do things that were impossible on consumer NVIDIA hardware.
MLX is the key. It's Apple's native inference framework, and it's been critical to making local AI viable on their silicon. We run a Qwen3-8B model in FP16 on an M4 with 24GB unified memory at roughly 20–25 tokens per second during generation — competitive with the best consumer NVIDIA cards for that model size, and you get access to all your system memory if you need it.
The sweet spot is around 32GB unified memory. We've been able to run Qwen3-14B in quantized form on M-series chips with 36GB+ of unified memory without offloading — that's a model size we couldn't comfortably handle on most consumer NVIDIA cards until the latest ones arrived. The latency is comparable, but you're trading some throughput for the ability to run larger models locally.
Practical considerations: MLX has been maturing fast since its early days. We've been using it to run our own agent systems locally, and it's reliable enough that we trust it with production workloads — but we still have fallback infrastructure if something goes wrong. The framework is open source, which matters when you're building real systems that need to be maintainable long-term.
What "enough" actually looks like for different use cases
This depends entirely on what you want the model to do:
For basic chat with occasional reasoning: You can run Qwen3-8B in FP16 on 24GB VRAM or unified memory and it'll handle most things fine. This is where a lot of us started — small enough that context doesn't matter much, fast enough for conversational use.
For document analysis and code generation: You need the next tier up. Qwen3-14B quantized requires around 28–30GB to run comfortably in FP16-ish precision. This is where NVIDIA's 4090 starts getting tight — you're pushing against that VRAM limit — but Apple Silicon with 36GB+ unified memory handles it more gracefully because of the memory bandwidth.
For running multi-agent systems: We've been doing this on RTX 4090 setups, and it works — but we have to be strategic about what each agent can access simultaneously. Our setup runs three agents concurrently: one does code generation, another reads documentation, a third handles communication with our infrastructure. The Qwen3-8B model is small enough that we can keep all three running in parallel without constant context switching, which is crucial for responsive behavior.
For the "I want Claude-level performance locally" dream: This is still beyond consumer hardware at this point. Even on high-end setups, you're trading latency for what you get — Qwen3-27B runs but it's noticeably slower than cloud equivalents because of that VRAM constraint. We use it when we need specific capabilities, not as a general-purpose replacement for our production agents.
Quantization: the art of making models fit
If you're running on limited hardware, quantization is your friend and your enemy simultaneously. It lets you pack more parameters into the same memory footprint — at the cost of some precision. The practical reality: Qwen3-8B in 4-bit quantized runs comfortably on 20GB VRAM, which is a real win for consumer cards with tighter budgets. But there's a diminishing return point where you lose too much quality to gain memory headroom.
Our experience: we mostly run in FP16 when we can — the quality difference between FP16 and quantized is noticeable on certain tasks like code generation — but we have quantized versions ready for when VRAM gets tight. The key insight: don't over-quantize. 4-bit is usually fine, 3-bit starts getting into "you'll notice differences" territory for most users.
What we're actually running in production today
Our own setup uses a RTX 4090 as the inference backend for our agent system — it's been rock solid through multiple releases since launch. The Qwen3-8B model is our workhorse because it fits comfortably in VRAM without offloading, and FP16 gives us reliable quality on code generation tasks. We have quantized versions of larger models ready as backups if we need to squeeze more capability into the same hardware later.
We're also exploring Apple Silicon for certain workflows — the M4 with 36GB unified memory lets us run Qwen3-14B comfortably when we need that size, which is useful for tasks like deep document analysis where context windows matter more than raw speed.
The pattern we've settled on: start with what fits your hardware, then upgrade only when you hit a real bottleneck — not because some benchmark says "you should have this." We've been running serious agents locally since late 2025, and the trajectory has been impressive: models that were impossible two years ago are now practical on consumer gear.
The tradeoffs nobody talks about enough
Local inference isn't a silver bullet. Here's what you're actually giving up by going local:
Latency. Cloud services have hundreds of GPUs dedicated to inference — they can scale out your requests across the fleet. Local means whatever compute you bought, period. For most individual users this is fine — but if you need thousands of tokens per second continuously, cloud still wins on raw throughput.
Model variety. You're limited by what fits in your VRAM. Cloud services let you swap between dozens of different model sizes and architectures instantly. Local means committing to a few choices that work for your hardware.
Update velocity. When we release new models or updates, they ship to the cloud first — then we backport them locally over time. There's usually a week or two lag before local setups get the latest improvements. That's not a dealbreaker, but it is worth knowing if you need cutting-edge capability right now.
Privacy as a feature. This is where local inference wins hands down. Everything stays on your machine. No data leaves your system unless you explicitly send it somewhere. For sensitive work — legal documents, personal projects, proprietary codebases — this matters more than raw speed sometimes.
What's coming next in local AI hardware
The landscape is moving fast. NVIDIA's consumer line continues to push higher VRAM on their cards, which directly translates to what models you can run comfortably. Apple Silicon keeps adding unified memory capacity across the Pro and Max variants, which lets us run larger quantized models locally than before. We're watching both lines closely — if either side pushes that threshold up another 8–12GB, we'll be able to run Qwen3-27B more comfortably on consumer hardware.
There's also interest in specialized inference cards that sit between consumer and datacenter. We haven't tested them yet but they could become interesting options for users who want more than consumer GPUs without the cost of dedicated server hardware.
How to decide what you need
Start by answering three questions:
- What model size do I actually need? This depends on your use case. For chat and basic tasks, 8B models are plenty. For code generation and reasoning with large contexts, 14B–27B becomes necessary. If you're running multi-agent systems where context matters, lean toward the larger models even if it means slower speeds — you can always optimize later by pruning or quantizing.
- How much VRAM do I have? Be honest. 24GB is the practical minimum for serious local inference today. Anything less forces offloading which kills both speed and quality. The newer cards push higher, but you still need to account for system memory overhead if you're running other services alongside your inference setup.
- What's my tolerance for latency? For a personal assistant that needs to respond in under a second, 24GB VRAM is your floor. For batch processing or background analysis tasks where speed matters less than capability, you can get away with tighter constraints. We run our system at around 25 tokens per second on generation and it feels responsive enough for most use cases — but that's because we've designed the workflow to work within those bounds.
What we'll be watching next
We're already planning around a few things: Qwen3-40B running locally is our next big target, which requires serious VRAM headroom — probably 64GB+ minimum for comfortable FP16 operation. That's the kind of territory where consumer hardware starts getting into server territory, and we'll need to see what the newest cards can handle before committing to that scale.
We're also interested in seeing how specialized inference accelerators evolve — if they become practical options for users who want more than a 4090 but don't want a datacenter rack, that could open up new territory between consumer and enterprise hardware.
The bottom line
Local inference has gone from "nice to have" to "practical necessity" for serious AI work in the last two years. We're running our own agent systems on RTX 4090 hardware with Qwen3-8B as the backbone, and it works reliably enough that we trust it with production tasks — including answering questions about our own codebase and managing real workflows across services.
The RTX consumer line continues to be the workhorse for this use case: plenty of VRAM at a reasonable price point, mature tooling support, and performance that's good enough for most serious local inference needs. Apple Silicon has caught up significantly on paper — unified memory is genuinely useful — but we're still using NVIDIA cards as our primary inference backend because of the mature software ecosystem around them.
If you're considering building a local AI setup in 2026, start with what fits your budget and upgrade incrementally as you hit real bottlenecks rather than chasing benchmarks that won't match your actual use case. The hardware requirements have dropped dramatically from where they were two years ago — this is the year where "I want Claude-level performance locally" starts feeling like a realistic goal for users who invest in the right setup, not just a dream for people with massive budgets.
The rest of our infrastructure update and how we're actually running multi-agent systems on consumer hardware will come next. We'll be publishing that soon — stay tuned.