A $10K Private AI Is Starting to Look Reasonable

$10K for a private AI system still sounds expensive.

It also sounds much less absurd than it did a few years ago.

Two NVIDIA DGX Spark systems provide 256 GB of combined unified memory. NVIDIA documents support for models up to 200B parameters on one unit, or up to 405B in a dual-unit configuration. That does not mean every model in that range will run quickly, cheaply, or well. It does mean that serious local inference is no longer automatically a hyperscaler problem.

For a small or medium-sized company, the possibility of running a large open-weight model on-premise, keeping sensitive data inside its own environment, and reducing dependence on OpenAI, Anthropic, or another provider changes the equation.

The number is still a lot of money, especially in Latin America. But we are starting to talk about two desktop machines instead of a giant data center.

That is the part of local AI I find interesting.

Not the ability to open a chat without an internet connection. The more interesting shift is that inference is becoming infrastructure.

The Local Model Story Is Too Small

Most conversations about local models begin with a simple question: can I run a chatbot on my laptop?

That is useful, but it is a narrow way to think about the problem. A chat application keeps the model, the interface, and the workflow in one place. It is convenient, but the model remains trapped inside the application that started it.

I wanted to think about the next layer up: can my development machine become an inference server that different tools can consume?

I recently built a talk around that idea using a MacBook Pro with 48 GB of memory, MLX, OpenCode, and Tailscale. The complete setup lives in the MLX tools in my developer-tools repository.

The architecture is simple:

Laptop
   -> inference server
   -> OpenAI-compatible API
   -> OpenCode
   -> Tailscale
   -> any authorized device

The point is not the number of components. The point is the boundary between them.

A MacBook Can Be an Inference Server

The setup uses mlx-lm and a small launcher that applies machine-specific defaults. On the 48 GB M4 profile, I can start a local server with:

run-mlx-server --m4-48gb \
  --model mlx-community/Qwen3.6-35B-A3B-4bit-DWQ

The launcher starts mlx_lm.server on port 8080. The server exposes HTTP endpoints such as /v1/chat/completions and /v1/models, with an interface intended to be similar to the OpenAI chat API. It does not open a browser chat window. It starts a process that other software can call.

That distinction changes the conversation.

I did not open a local chatbot. I started a server.

Apple Silicon helps here because the CPU and GPU share the same memory pool. MLX is designed around that unified memory architecture, so arrays do not need to be copied between separate system RAM and GPU VRAM pools in the traditional way.

My 48 GB are not 48 GB of dedicated VRAM, and they are not entirely available to model weights. macOS, the runtime, the KV cache, and the context all need memory too. But unified memory gives the GPU access to a larger shared pool than many discrete laptop GPUs provide.

That makes a laptop a surprisingly capable inference machine.

The Agent Does Not Need to Know What Is Behind the Endpoint

Once the model is running, OpenCode can point to it through a provider configuration:

OpenCode
    -> http://127.0.0.1:8080/v1
    -> MLX server
    -> Qwen

OpenCode does not need to know that the runtime is MLX. It needs a model identifier, a base URL, and an API that supports the features it uses.

That is the useful abstraction. The harness and the model become replaceable components.

The same shape can connect OpenCode to llama.cpp, LM Studio, or a larger distributed serving runtime. The provider and base URL change. The idea does not.

There is an important terminology caveat here. “OpenAI-compatible” is not an official standard with an absolute compatibility guarantee. It is a common de facto interface. Different runtimes expose different subsets of the API, and tool calling, structured output, streaming, and model-specific parameters can behave differently.

Still, a common HTTP contract is enough to decouple a large part of the system.

Harness
   -> OpenAI-compatible HTTP API
   -> inference runtime
   -> model

The important thing is not OpenCode, and it is not MLX. The important thing is the contract between them.

The Demo Is About Real Work, Not a Chat Window

To test the setup, I would not ask the model to write a small function. That proves very little.

I would ask it to work through a real repository without changing anything:

Analyze this repository without modifying anything. Explain its architecture, identify its entry points, and tell me which five files I should read first to understand it.

Then I would make the request more demanding:

Find a concrete technical risk or inconsistency in this repository. Show me where it is, explain why it is a problem, and propose the minimum change needed to fix it. Do not modify it yet.

These prompts test context handling, repository navigation, tool use, reasoning, and judgment. They also create a useful approval boundary. The model can inspect and explain before it is allowed to change anything.

That is closer to how I want to use a local model. Not as a smaller chatbot, but as one component in an engineering workflow.

35B Parameters Is Not the Same as 35B Parameters Working

The Qwen model in this setup is Qwen3.6-35B-A3B-4bit-DWQ. The name contains several details that matter.

It is a Mixture-of-Experts model. It has 35B total parameters, with approximately 3B activated for a given token. A router chooses which experts participate in each step.

That is different from a 27B dense model, where practically the complete network participates for each token.

The useful distinction is:

35B total parameters  -> capacity stored in the model
~3B active parameters -> capacity used for one token
memory footprint      -> what the machine must actually keep available

These numbers are related, but they are not interchangeable. The inactive experts still exist in the model weights. A 35B-A3B model is not a 3B model in memory.

The model also uses 4-bit quantization. Fewer bits per weight reduce memory use and the amount of data that needs to move through the system, at the cost of some possible quality loss.

DWQ means Distilled Weight Quantization. It describes a method for obtaining better low-precision weights. It does not mean that the model architecture was reduced through distillation.

The MLX learned quantization documentation explains the distinction. This is a small terminology detail, but it matters when discussing what the machine is actually running.

total parameters != active parameters != memory footprint

The same distinction becomes even more important when the hardware gets larger.

Local Inference Does Not Make the Whole Workflow Local

So far the setup is local:

OpenCode -> localhost:8080 -> MLX

That is convenient when the client and server are on the same machine. It is less useful when I want to leave the Mac running and use the model from another laptop.

Tailscale Serve can proxy a local port to other devices in the same tailnet. A simplified version looks like this:

tailscale serve 8080

Tailscale provides an HTTPS URL under the tailnet domain, while the service continues running locally on the Mac:

OpenCode on another device
    -> https://my-mac.my-tailnet.ts.net/v1
    -> Tailscale Serve
    -> http://127.0.0.1:8080
    -> MLX
    -> Qwen

From the client’s point of view, the model is no longer “on my laptop.” It is on a private node in my network. Nothing moved, but the inference service became reachable from any authorized device.

This is where local inference starts looking like a small private provider.

It is also where the privacy caveat becomes important. The model weights, context, and generated tokens can remain on the inference server. But an agent that uses web search, GitHub, external MCP servers, SaaS tools, or remote APIs will still send parts of the workflow outside the machine.

Local inference and a local agent workflow are not necessarily the same thing.

Tailscale Serve also does not turn a basic model server into a production security design. Its access controls still need to be configured, and the MLX-LM server documentation warns that the server only implements basic security checks and is not recommended for production. Private network reachability is useful, but it is not the same as authentication, authorization, auditing, and operational hardening.

The $10K Question Starts With Memory

The MacBook demonstrates the architecture, but its memory still limits the models it can load.

That is where desktop hardware such as DGX Spark changes the range. NVIDIA lists 128 GB of unified system memory for one DGX Spark and documents model support up to 200B parameters on one unit, or 405B with two units.

Those figures should be read as capacity ranges, not promises that every model will run at a useful speed. Model weights are only one part of the memory budget. The runtime, KV cache, context length, quantization format, parallelism, and communication between nodes all affect the result.

DeepSeek-V4-Flash-0731 is the specific checkpoint I have in mind for the two-DGX-Spark discussion. Its model card describes it as the official release that supersedes the preview, with enhanced agentic capabilities and local deployment instructions for runtimes such as vLLM and SGLang. The broader DeepSeek-V4 release describes the Flash architecture as a 284B-parameter MoE model with 13B active parameters and a context length of up to 1 million tokens.

The dated model card also reports benchmark results across coding and agentic tasks. Those results are useful context, but they are still vendor-reported benchmarks, not a guarantee of performance on two DGX Spark systems.

That does not justify saying that two DGX Spark systems can run a model equivalent to every frontier model. Vendor benchmarks need to be treated as vendor benchmarks, and a model that fits in memory still needs the right runtime, quantization, parallelism, and workload configuration.

The more accurate claim is narrower and more interesting: this hardware makes it plausible for smaller organizations to operate open-weight models in a class that used to require specialized data-center infrastructure.

Entering memory is not the same as running fast and well.

Private AI Is a Control Decision, Not Just a Cost Decision

The strongest argument for this infrastructure is not that it will always be cheaper than an API. It will not.

On-premise inference has hardware costs, power costs, cooling, networking, maintenance, model updates, monitoring, security work, and somebody responsible when the server stops behaving. If usage is bursty, a cloud provider may still be the more rational choice.

The argument is control.

A private inference server can change the boundary around sensitive source code, customer data, internal documents, and unfinished work. It can reduce exposure to provider pricing, rate limits, outages, policy changes, and model deprecations. It can also make it possible to route routine or privacy-sensitive tasks locally while sending only the hardest work to a remote model.

That does not mean everything should be local. The useful architecture is probably mixed:

• local inference for private, repetitive, or latency-sensitive work
• remote frontier models for tasks that justify the cost and external boundary
• a harness that can choose between them
• an API contract that keeps the workflow from depending on one runtime

The business case is not “we bought a computer, so AI is free now.” The business case is that some organizations may prefer to own the inference path for important workloads.

The Architecture Scales Further Than the Laptop

On the MacBook, the stack looks like this:

OpenCode
   -> OpenAI-compatible API
   -> MLX
   -> Qwen 35B-A3B

On larger infrastructure, it could look like this:

OpenCode / another agent / another harness
               -> OpenAI-compatible API
               -> distributed runtime
               -> very large model
               -> two or more GPU nodes

The internal complexity changes dramatically. The conceptual separation does not.

The agent can be OpenCode today and something else tomorrow. The runtime can be MLX, llama.cpp, vLLM, SGLang, or a distributed system. The model can change too.

The API boundary is what lets those parts move independently.

That is why the MacBook demo matters. It is not a claim that a laptop replaces a data center. It is a small, understandable version of the same architecture.

The model is only one part of the system. The harness and the inference runtime can be decoupled.

Local models are becoming infrastructure, not just local chatbots.

The question is no longer only whether I can run a model without the internet. It is which workloads I want to own the inference path for, and where that boundary should live.