Docker gives AI developers a repeatable boundary around three things that otherwise drift independently: application code, model-serving infrastructure, and native runtime dependencies. It will not make inference fast or an agent safe by itself. It does make the environment explicit enough to reproduce, inspect, limit, and deploy.

This guide keeps the Docker fundamentals, then applies them to model APIs, local inference, retrieval services, and tool-using agents.

Where Docker fits in an AI stack

Three patterns account for most practical use:

1. Containerize the application, call a hosted model. Your API, retrieval worker, database, and observability stack run in containers; model inference stays with a provider.
2. Run the application and local model as separate services. Ollama, vLLM, or another runtime exposes an HTTP API on a private Docker network. See the tested Ollama Docker setup and vLLM serving guide.
3. Give an agent an isolated execution environment. A container limits filesystem and process access, but it is not a complete hostile-code sandbox. Strong multi-tenant isolation can require rootless containers, restrictive capabilities, seccomp, or a microVM boundary.

Docker now also ships AI-specific tooling. Its official AI overview covers Model Runner, MCP tooling, agent runtimes, and sandboxes. Docker Model Runner can pull and serve models behind OpenAI- and Ollama-compatible APIs, while Compose supports a top-level models declaration.

Images, containers, and model data

An image is a read-only template containing your runtime, libraries, and application. A container is a running instance with a thin writable layer. Keep these concerns separate:

• Put application dependencies in the image and pin them.
• Mount model caches and generated artifacts on volumes instead of baking large, frequently changing weights into an application image.
• Keep API keys outside the image and inject them at runtime.
• Treat prompts, evaluation fixtures, and model configuration as versioned inputs.

If you want the kernel-level explanation, how Docker containers actually work covers namespaces, cgroups, layers, and the OCI runtime chain.

A small AI service with Compose

The most portable pattern is an app container calling a model service over an internal network:

services:
  api:
    build: .
    environment:
      MODEL_BASE_URL: http://model:11434/v1
    depends_on:
      model:
        condition: service_healthy

  model:
    image: ollama/ollama:latest
    volumes:
      - model-cache:/root/.ollama
    healthcheck:
      test: ["CMD", "ollama", "list"]
      interval: 10s
      timeout: 5s
      retries: 12

volumes:
  model-cache:

Pin a tested image tag or digest in production. The unpinned tag above keeps the example readable, not deployment-safe. The app should have timeouts and retries because “container is running” does not mean “model is loaded.” For a larger generated starting point, see the AI Docker Compose generator tutorial.

GPU access is explicit

On supported hosts, Docker exposes accelerators only when requested. The official GPU access documentation uses --gpus after the vendor driver and NVIDIA Container Toolkit are installed:

docker run --rm --gpus all nvidia/cuda:12.6.0-base-ubuntu22.04 nvidia-smi

Production checks should cover driver/runtime compatibility, GPU visibility, memory headroom, and a real inference request. A successful nvidia-smi proves device access, not that your model fits or performs well. If CUDA memory is exhausted, use the vLLM CUDA OOM guide rather than raising limits blindly.

Build images that are reproducible and smaller

Use multi-stage builds, copy dependency manifests before source code to preserve cache hits, run as a non-root user, and exclude model caches and secrets with .dockerignore.

FROM python:3.12-slim AS runtime
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY src/ ./src/
RUN useradd --create-home appuser
USER appuser
CMD ["python", "-m", "src.api"]

Build errors often come from oversized context or architecture mismatches. The build-context guide and multi-platform guide cover those cases.

Resource limits matter more for AI

Model servers can consume all available RAM or GPU memory, while embedding and retrieval workers can saturate CPU. Set limits deliberately, observe actual peaks, and leave headroom for the host. Docker uses cgroups for CPU and memory accounting; the practical consequences are explained in the container internals guide.

Also separate readiness from liveness:

• Liveness: the process is alive.
• Readiness: the model is loaded and can answer.
• Dependency health: vector store, model provider, and queue are reachable.

Restarting an alive service during a slow model load can create an endless restart loop.

Security boundaries for agents

Do not mount the Docker socket into a tool-using agent: control of that socket is effectively control of the host daemon. Avoid privileged containers, drop unnecessary Linux capabilities, use read-only filesystems where possible, and allowlist outbound destinations and mounted directories.

Docker documents both the Engine security model and rootless mode. For a broader production threat model, use the agent sandboxing guide.

Docker versus Kubernetes for AI

Docker Compose is usually enough for local development, evaluation, and a single inference host. Kubernetes becomes defensible when you need multiple GPU node types, automated placement, rolling model deployments, cluster autoscaling, or a shared serving platform. The Kubernetes inference guide covers that boundary.

Do not adopt Kubernetes merely because a model runs in a container. A hosted model API plus a small containerized application is often cheaper and easier to operate.

Production checklist

• Pin images and dependencies; scan them before release.
• Keep secrets and model data outside the application image.
• Set CPU, memory, process, and GPU boundaries.
• Add model-aware readiness checks and graceful shutdown.
• Log model name/version, latency, token usage, failures, and request IDs without logging sensitive prompts by default.
• Persist only the state that must survive container replacement.
• Test cold model load, restart, provider failure, and disk exhaustion.

Docker is valuable to AI engineering because it makes boundaries visible. Use it to separate services, reproduce runtimes, control resources, and reduce deployment drift—not as a claim that the workload is automatically secure, scalable, or production-ready.