Skip to main content
Sean Brill
Sean Brill
Email meGitHub (opens in a new tab)LinkedIn (opens in a new tab)

Local AI tooling

2026

In development

LLM Factory

Bake open-weight models into portable, OpenAI-compatible containers.

Source on GitHub (opens in a new tab)
$ docker load -i images/support-bot.tarLoaded image: support-bot:latest$ docker run -d -p 8080:8080 support-botllmgate init prompt: baked in, as a defaultserver listening on :8080 (OpenAI-compatible)$ curl localhost:8080/v1/chat/completions \ -d '{"messages":[{"role":"user","content":"Hi"}]}'{"choices":[{"message":{"role":"assistant", "content":"Hi! How can I help with your order?"}}]}

Overview

A local web app that downloads open-weight models and bakes each one into a self-contained llama.cpp server image for Docker or Podman, on CPU, CUDA, or Vulkan and Metal. Every image saves as a single .tar that serves an OpenAI-compatible API: private, offline AI you can ship as one file.

My role

Creator. Go backend, Svelte UI, runtimes and container builds.

Built with

  • Go
  • Svelte 5
  • TypeScript
  • Vite
  • llama.cpp
  • whisper.cpp
  • stable-diffusion.cpp
  • Docker
  • Podman
  • CUDA
  • Vulkan

In numbers

per model, OpenAI-compatible
1 .tar
third-party Go dependencies
0
kinds of model, chat to speech
6

How it's built.

  • Download, build, ship

    Builds run on the server and survive a page refresh. Multi-gigabyte models are hardlinked into a minimal build context instead of copied, and llama.cpp is statically linked.

  • A GPU inside containers on a Mac

    Podman with libkrun passes the Apple GPU through (Vulkan to Metal) at roughly 70 to 80% of native speed, something Docker Desktop cannot do.

  • The system prompt llama.cpp lacks

    llmgate, a static Go shim that runs as PID 1 in every image, injects a baked-in initialization prompt into chat completions, as a default or always enforced.

  • Beyond chat

    Images for vision, embeddings, image generation, speech-to-text and text-to-speech, with video and multi-model ensembles routed within a VRAM budget in testing.

  • No dependencies

    The backend is standard-library Go only, and the Svelte 5 UI is embedded in the binary.