Harbor at github.com/av/harbor is an open-source CLI and companion app that assembles local AI services with Docker Compose. A basic harbor up can start Ollama and Open WebUI already connected; adding service handles can wire search, speech, image generation, gateways, observability and other components into the same stack. The project is Apache-2.0 licensed.
This is not the CNCF Harbor container registry at goharbor/harbor. It is also not an LLM, a security boundary or a managed production platform. It is a fast integration layer around many independently maintained images. Harbor reduces configuration work; the operator still owns image provenance, exposed ports, authentication, GPU capacity, model licenses, persistent data, upgrades and every service’s vulnerabilities.
A local AI stack is a network, not one application
browser / coding client
|
v
[Open WebUI / API gateway]
|
.------+---------+--------------.
v v v
[Ollama] [llama.cpp] [vLLM] model inference
| | |
'----------------+--------------'
|
.---------+----------.
v v
[SearXNG] [Speaches] tools / speech
|
Internet
Docker network ≠ trusted network
local host ≠ automatically private
Harbor selects Compose fragments and cross-service configuration so consumers can find providers by internal service name and expected API path. That convenience expands the trust graph: a web UI can reach a model backend, search service and speech engine; plugins or agents may call still more endpoints. Inventory data flows before adding services.
Service roles and the decision each introduces
| Layer | Examples documented by Harbor | Operator decision |
|---|---|---|
| Inference backend | Ollama, llama.cpp, vLLM, SGLang and others | Model, quantization, GPU, context, license and API exposure |
| Frontend | Open WebUI and alternative chat interfaces | Authentication, users, history, uploads and tools |
| Gateway/router | LiteLLM, Bifrost, Harbor Boost | Keys, fallbacks, logs, quotas and provider routing |
| Retrieval/search | SearXNG, research and RAG services | Outbound traffic, index privacy, citations and prompt injection |
| Media | Speaches, ComfyUI and voice/design services | Model rights, GPU contention and generated-media review |
| Agent/workflow | Dify, n8n, coding/agent services | Tool authority, credentials, sandboxing and side effects |
| Proxy/access | Traefik and built-in tunnel | TLS, hostnames, authentication and internet exposure |
Install with a reviewable supply chain
Official materials show installation scripts and Docker-based operation. A command such as curl ... | bash is convenient but executes mutable network content immediately. For controlled use, download the script, inspect it, pin a commit/tag and verify repository ownership before running. Review Dockerfiles and Compose definitions for the selected services.
Container tags such as latest are not reproducible. Pin image digests for a stable environment, record Harbor’s version and export the effective Compose configuration. Scan images and software bills of materials under your policy. An official image can still contain a vulnerable base layer.
| Artifact | Record | Why |
|---|---|---|
| Harbor CLI/app | Version, commit and installer hash | Reproduce orchestration behavior |
| Compose config | Rendered/ejected file and environment names | Know what actually ran |
| Container image | Registry, repository and digest | Prevent silent tag drift |
| Model | Source, revision, hash, quantization and license | Quality, security and legal provenance |
| Profile | Reviewed local file and checksum | Profiles can change endpoints and secrets |
GPU and memory planning
Harbor can detect capabilities and pass GPU resources to supported containers, but it cannot make incompatible workloads fit. Estimate model weights, KV cache, runtime overhead and concurrent requests. A model that loads successfully may still fail at the requested context or when ComfyUI and speech services compete for VRAM.
| Workload | Capacity driver | Test |
|---|---|---|
| Single-user chat | Model weights + one KV cache | Time to first token and tokens/sec at target context |
| Long-context RAG | KV cache and prompt preprocessing | Peak VRAM/RAM and latency at realistic document size |
| Multiple users | Batching, queue and cache fragmentation | P50/P95 latency and rejected requests under load |
| Image generation | Diffusion model and resolution | Contention while LLM backend is active |
| Speech pipeline | Audio duration, model and real-time factor | End-to-end capture-to-response latency |
Set memory, CPU, GPU and disk quotas where practical. Monitor temperature and power on a workstation. Avoid assuming that “local” means free: include hardware depreciation, electricity, storage, engineering and idle capacity in cost per accepted task.
Ports and authentication
Bind development services to loopback unless another host must reach them. Docker-published ports can bypass assumptions about a host firewall. List listeners after startup and test from another device. Many local LLM endpoints accept dummy or no API keys because they expect a trusted machine; do not expose them to a LAN or internet unchanged.
The project explicitly warns that tunneling a service to the internet is unsafe without authentication. Treat harbor tunnel as an exposure event: require strong identity, TLS, rate limits, request-size limits, logs and a narrow target. Never tunnel a raw vLLM/Ollama endpoint or admin UI with default credentials.
| Exposure | Minimum control | Residual risk |
|---|---|---|
| Loopback only | Host account security | Malicious local process/browser request |
| Trusted LAN | Firewall, auth and segmented subnet | Compromised peer and weak Wi-Fi |
| Reverse proxy/VPN | SSO/MFA, TLS and access policy | Proxy misconfiguration and stolen session |
| Public tunnel | All above plus abuse protection and monitoring | Internet scanning, cost/DoS and model extraction |
Secrets and cross-service trust
Gateways and UIs may need API keys for hosted models, search or storage. Put secrets in a managed store or protected environment file that is excluded from Git. Prefer service-specific, low-scope keys and rotate them. Inspect effective container environment and mounted files; a UI should not receive a backend key merely because both share a Compose project.
OpenAI-compatible APIs create interface compatibility, not identical authentication or semantics. A dummy key that works locally must not be reused as a real access control. Verify streaming, tool calls, structured output, model names, limits and error behavior for each client/backend pair.
Persistent data, backups and deletion
Map each volume: model cache, chat database, uploaded documents, vector indexes, generated images, logs and gateway telemetry. Decide what survives down, upgrades and profile changes. Encrypt disks and backups. Test restore with the same pinned services.
- Separate replaceable model caches from irreplaceable user data.
- Do not back up API keys inside application databases when avoidable.
- Set retention for chat and uploaded documents; deletion should cover vector indexes and thumbnails.
- Before removing a service, export data in a documented format and confirm volume cleanup.
Profiles are executable configuration
Harbor profiles save configuration for scenarios and can be imported from a URL. A profile can alter image versions, endpoints, arguments and potentially credentials. Never apply an untrusted remote profile directly. Download, inspect, pin and strip secrets; treat profile review like a code change.
Maintain profiles such as offline-private, gpu-chat and rag-test with explicit service lists and budgets. Avoid one ever-growing default profile that silently starts dozens of containers and expands attack surface.
Web RAG adds an untrusted-content channel
A Harbor stack combining Open WebUI, SearXNG and an LLM can search the web and use results as context. Web text can contain prompt injections and false claims. Label retrieved content as untrusted data, restrict tools, preserve URLs and require citations. Do not allow page instructions to access secrets or invoke administrative endpoints.
Search queries and fetched pages also leave the machine even when the LLM is local. Document this exception in any privacy claim. Proxy and DNS logs may expose user intent.
Upgrade one layer at a time
| Change | Canary evidence | Rollback asset |
|---|---|---|
| Harbor update | Rendered Compose diff and smoke tests | Prior CLI/version and config |
| Service image | Schema, auth, UI and integration checks | Previous digest and volume snapshot |
| Model | Quality, tool-call and latency benchmark | Previous model hash |
| GPU driver/runtime | All backends load and sustain target workload | Known-good host image/driver |
| Profile | Port, secret and data-flow diff | Versioned prior profile |
Back up state, stop writes and test compatibility before a migration. “Container starts” is not enough: send known prompts, perform RAG retrieval, test tool calls and verify user/history persistence.
Use eject as a graduation path
harbor eject can produce a standalone Compose representation of selected services and variables. This is valuable when an experimental combination becomes important: review the output, replace floating tags, externalize secrets, define health checks, backups, monitoring and ownership, then manage it as ordinary infrastructure.
Ejection is not automatic production hardening. Generated Compose reflects the experiment’s assumptions. Production may require Kubernetes or another scheduler, external databases, high availability, autoscaling, audit logging and tested disaster recovery.
Evaluation scorecard
| Dimension | Measure | Pass question |
|---|---|---|
| Setup value | Time from clean host to working scenario | Does Harbor materially reduce repeatable setup time? |
| Reliability | Cold start, restarts and failed dependency recovery | Can another operator recover it? |
| Quality | Task benchmark on pinned model/settings | Are accepted results sufficient? |
| Performance | Latency, throughput, VRAM and queue | Does it meet real concurrent load? |
| Security | Ports, auth, secrets, scans and data flows | Is each service’s authority justified? |
| Portability | Eject and restore on a clean host | Can the stack leave Harbor without lost data? |
Alternatives
| Option | Best fit | Tradeoff versus Harbor |
|---|---|---|
| Harbor (av) | Exploring many pre-wired local AI services | Large fast-changing service catalog |
| Ollama + Open WebUI manually | Simple stable local chat | Less abstraction and easier audit, more manual wiring |
| LM Studio | Desktop-first model download and chat/API | Less multi-service Compose orchestration |
| Docker Compose built in-house | Small production-owned stack | More initial engineering, clearer minimal surface |
| Kubernetes/Helm | Multi-node production platform | Much higher operational complexity |
| Managed model/API platform | Teams avoiding GPU infrastructure | Usage cost, uploads and provider dependency |
Frequently asked questions
Is this the Harbor container registry?
No. This listing points to av/harbor, a local AI-stack orchestrator. The registry project is goharbor/harbor.
Is Harbor an LLM?
No. It downloads/runs and connects model servers, UIs and supporting tools.
Is it free?
The project is Apache-2.0 licensed. Models, hosted APIs, electricity, hardware and third-party services can cost money.
Does local mean private?
Only if all selected services, models, tools and storage remain local and securely configured. Search, cloud APIs and tunnels create external flows.
Can I expose it remotely?
The project includes tunneling and proxy options but warns about internet exposure. Use authenticated VPN/reverse-proxy controls and never expose raw unauthenticated services.
Is it production-ready?
It is an excellent exploration/integration layer. Production readiness depends on the selected services plus your pinning, security, monitoring, backups and support.
How can I migrate away?
Use the eject workflow as a starting Compose file, then harden and operate it independently.
Primary sources
- Official Harbor repository, service overview and license
- Official installation guide
- Official CLI reference
- Official backend guide
- Official local Web RAG guide
- Docker Engine security documentation
- Docker Compose environment-variable practices
- OWASP prompt-injection guidance
- CNCF Harbor registry for disambiguation
Last reviewed July 26, 2026. Harbor and its service catalog move quickly. Pin the orchestrator, rendered Compose, images, models and profiles and re-run security and compatibility checks after changes.
