We wanted to explore the opposite design: the operator-owned model should be the default, while every external model remains an explicit bring-your-own-key choice.

That experiment became OpenMake LLM, an MIT-licensed, self-hosted AI workspace. This article is not a benchmark or a claim that local inference is always cheaper or better. It is a walkthrough of the architecture we actually run, the boundaries we chose, and the tradeoffs we still have to improve.

The running chat workspace. Conversation titles and account details in the public capture are blurred.

The physical topology

Our development and live-test setup uses two machines with deliberately different responsibilities:

Browser
   |
   v
Mac mini
  - OpenMake API and web app under PM2
  - PostgreSQL and isolated Docker workloads
  - LiteLLM bound to loopback as the inference gateway
   |
   | least-privilege Tailscale path
   v
NVIDIA DGX Spark (GB10 Grace Blackwell Superchip)
  - private vLLM endpoints
  - Qwen generation models
  - BGE embeddings
  - FLUX image generation

The Mac mini owns the application and policy plane. The DGX Spark owns the GPU-heavy inference plane. vLLM is not exposed directly to the public internet, and LiteLLM remains the single gateway the application talks to.

This hardware is our real setup, not a requirement. OpenMake talks to OpenAI-compatible endpoints, so another vLLM machine, an Ollama endpoint, or an intentionally enabled external provider can take the DGX Spark’s place. The important part is the boundary: the application should not need to know whether a model is in the same process, elsewhere on a private network, or behind an operator-approved service.

Local-first is a policy decision, not just an endpoint

A local endpoint alone does not answer the operational questions:

• Who may enable an external provider?
• Which key is used, and how is it stored?
• Do local and external models get the same tools?
• What happens when a prompt exceeds the model’s context window?
• Which actions require human approval?
• Does “multi-agent” run for every prompt, or only when the user asks for it?

OpenMake sends local and external models through the same provider-gated message pipeline and MCP tool loop. External providers are opt-in and use the operator’s or user’s own key. The local model remains available as the default path rather than becoming a fallback after a cloud model.

The simplified request path looks like this:

request
  -> authentication and provider policy
  -> language, memory, style, and tool assembly
  -> selected mode or normal chat
  -> one OpenAI-compatible inference gateway
  -> bounded tool loop
  -> streamed response

That uniform path has been more valuable than adding many provider-specific features. Tool permissions, audit behavior, and context handling can be reasoned about once instead of being rebuilt for every model vendor.

Normal chat should stay normal

One design choice is intentionally unexciting: an ordinary message resolves one model and runs one chat/tool loop.

It does not automatically start a committee of agents. Multi-agent orchestration can improve difficult answers, but invoking it for every prompt increases latency, token use, failure modes, and the difficulty of explaining where an answer came from.

OpenMake therefore makes Discussion an explicit mode. When selected, it can:

1. choose relevant expert roles;
2. run a bounded number of opinions in parallel;
3. cross-review those opinions;
4. reuse collected evidence; and
5. synthesize one final answer.

The distinction matters operationally. “Chat” is a predictable single-model interaction. “Discussion” is a more expensive orchestration workflow that the user deliberately requests.

Model roles without hiding the routing

Different workloads benefit from different models. A fast summarizer, a tool-using agent, a judge, and a research model do not necessarily need the same latency or reasoning profile.

OpenMake supports role-based model assignment for roles such as agent, judge, research, parallel sub-agents, review, and thinking-summary. Resolution follows visible configuration and falls back to the local default when a role-specific model is unavailable.

The goal is not to build a mysterious router that asks another model where every prompt should go. It is to let the operator make stable assignments that can be inspected, tested, and changed.

Agents need an execution boundary

Once a model can run shell commands, Python, browser actions, or file operations, inference is no longer the main security question. Execution is.

Agent tasks in OpenMake run in persistent Docker sandboxes. A task can keep checkpoints, survive an application restart, and be resumed manually. Approval boundaries stop the workflow before actions that should not be silently automated.

Persistence and approval may sound contradictory, but they solve different problems:

• persistence prevents useful work from disappearing when a process restarts;
• approval prevents persistence from becoming uncontrolled autonomy;
• checkpoints make it possible to inspect what happened before resuming;
• container isolation limits the environment available to the task.

The same principle applies to external MCP servers. They run in isolated containers with restricted capabilities and tool allowlists, so connecting a large server does not mean exposing every tool schema to every conversation.

What remains difficult

This is not a one-command lightweight application. The current release requires Node.js 24, Docker, PostgreSQL, and an OpenAI-compatible model endpoint. Running a useful local model also requires compatible GPU memory and an operator who understands the deployment.

The split topology creates additional work:

• private-network reachability must be measured rather than assumed;
• model names and context limits must match what the inference server actually exposes;
• streaming, tool calls, embeddings, and image generation need separate validation;
• gateway telemetry must never block inference;
• secrets and model endpoints need narrower access than the public application;
• external BYOK models may create charges even when the application itself is open source.

We also do not call the current system a finished “AgentOS.” Durable execution, scoped memory, policy, verification, and operator-facing auditability are directions that still need engineering work. Naming the destination is useful; pretending to have arrived is not.

What we learned

The strongest lesson is that local-first architecture is mostly about ownership and boundaries:

1. Keep one inference gateway between the application and every model endpoint.
2. Treat external providers as explicit policy choices, not invisible fallbacks.
3. Keep ordinary chat separate from expensive multi-agent orchestration.
4. Give tool-using agents durable state, but stop them at human approval boundaries.
5. Make role-based routing inspectable instead of adding another opaque LLM decision.
6. Test the live path: streaming completion, tool execution, rendered sources, and observable errors—not only HTTP 200 responses.

If you are building a self-hosted AI workspace, I would be interested in three practical comparisons:

• Which GPU and open-weight model combinations are reliable for tool-using agents rather than chat alone?
• Which actions do you always place behind human approval?
• Would you keep the application and inference gateway on one host, or separate them as we did?

The source, setup guide, and additional real screenshots are available in the OpenMake LLM repository. There is also a live demo and an English project overview.

OpenMake is not a company. We are a small team that enjoys open-source software, open hardware, and running services on machines we control.

Disclosure: this article was prepared with AI writing assistance and reviewed against the OpenMake LLM source tree, documentation, and running deployment by the maintainers. We are responsible for the technical claims and any mistakes.