Learn how to deploy AI models to production using Kubernetes and Docker. Complete guide covering containerization, orchestration, scaling, and monitoring.

AI Model Deployment with Kubernetes and Docker: Production Guide 🚀

TL;DR – This comprehensive guide teaches you how to deploy AI models to production using Kubernetes and Docker. We'll cover containerization, orchestration, auto-scaling, monitoring, and best practices for running AI workloads at scale.

Why Kubernetes for AI Deployment
Prerequisites and Setup
Containerizing Your AI Model
Creating Kubernetes Manifests
Setting Up Model Serving
Implementing Auto-scaling
Monitoring and Logging
CI/CD Pipeline Integration
Security Best Practices
Troubleshooting Common Issues

Why Kubernetes for AI Deployment

Kubernetes has become the de facto standard for deploying AI models in production environments. Here's why:

Key Benefits

Scalability: Automatically scale based on demand
Resource Management: Efficient GPU and CPU allocation
High Availability: Built-in fault tolerance and recovery
Version Management: Easy model updates and rollbacks
Multi-tenancy: Run multiple models on shared infrastructure

Real-World Use Cases

Netflix: Serves millions of ML predictions daily
Uber: Powers real-time pricing and routing
Spotify: Delivers personalized recommendations
Airbnb: Dynamic pricing and fraud detection

Prerequisites and Setup

Required Tools

1# Install Docker
2curl -fsSL https://get.docker.com -o get-docker.sh
3sh get-docker.sh
4
5# Install kubectl
6curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
7chmod +x kubectl
8sudo mv kubectl /usr/local/bin/
9
10# Install Helm
11curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
12
13# Install kind (for local testing)
14go install sigs.k8s.io/kind@v0.20.0

Local Kubernetes Cluster

1# Create local cluster with GPU support
2cat <<EOF > kind-config.yaml
3kind: Cluster
4apiVersion: kind.x-k8s.io/v1alpha4
5nodes:
6- role: control-plane
7  extraMounts:
8  - hostPath: /dev/nvidia0
9    containerPath: /dev/nvidia0
10  - hostPath: /dev/nvidiactl
11    containerPath: /dev/nvidiactl
12  - hostPath: /dev/nvidia-uvm
13    containerPath: /dev/nvidia-uvm
14EOF
15
16kind create cluster --config kind-config.yaml --name ai-cluster

Containerizing Your AI Model

Step 1: Create Model Server

1# app.py - FastAPI model server
2from fastapi import FastAPI, HTTPException
3from pydantic import BaseModel
4import torch
5import numpy as np
6import logging
7from typing import List, Dict, Any
8import uvicorn
9import os
10
11# Configure logging
12logging.basicConfig(level=logging.INFO)
13logger = logging.getLogger(__name__)
14
15app = FastAPI(title="AI Model API", version="1.0.0")
16
17class PredictionRequest(BaseModel):
18    data: List[List[float]]
19    
20class PredictionResponse(BaseModel):
21    predictions: List[float]
22    model_version: str
23    processing_time: float
24
25class ModelServer:
26    def __init__(self):
27        self.model = None
28        self.model_version = os.getenv("MODEL_VERSION", "1.0.0")
29        self.load_model()
30    
31    def load_model(self):
32        """Load the trained model"""
33        try:
34            model_path = os.getenv("MODEL_PATH", "/app/models/model.pt")
35            self.model = torch.load(model_path, map_location="cpu")
36            self.model.eval()
37            logger.info(f"Model loaded successfully from {model_path}")
38        except Exception as e:
39            logger.error(f"Failed to load model: {e}")
40            raise
41    
42    def predict(self, data: np.ndarray) -> np.ndarray:
43        """Make predictions"""
44        with torch.no_grad():
45            tensor_data = torch.FloatTensor(data)
46            predictions = self.model(tensor_data)
47            return predictions.numpy()
48
49# Global model instance
50model_server = ModelServer()
51
52@app.get("/health")
53async def health_check():
54    """Health check endpoint"""
55    return {"status": "healthy", "model_version": model_server.model_version}
56
57@app.get("/ready")
58async def readiness_check():
59    """Readiness check endpoint"""
60    if model_server.model is None:
61        raise HTTPException(status_code=503, detail="Model not loaded")
62    return {"status": "ready", "model_version": model_server.model_version}
63
64@app.post("/predict", response_model=PredictionResponse)
65async def predict(request: PredictionRequest):
66    """Make predictions"""
67    import time
68    start_time = time.time()
69    
70    try:
71        data = np.array(request.data)
72        predictions = model_server.predict(data)
73        processing_time = time.time() - start_time
74        
75        return PredictionResponse(
76            predictions=predictions.tolist(),
77            model_version=model_server.model_version,
78            processing_time=processing_time
79        )
80    except Exception as e:
81        logger.error(f"Prediction error: {e}")
82        raise HTTPException(status_code=500, detail=str(e))
83
84if __name__ == "__main__":
85    uvicorn.run(app, host="0.0.0.0", port=8000)

Step 2: Create Dockerfile

1# Dockerfile
2FROM python:3.10-slim
3
4# Install system dependencies
5RUN apt-get update && apt-get install -y \\
6    gcc \\
7    g++ \\
8    && rm -rf /var/lib/apt/lists/*
9
10# Set working directory
11WORKDIR /app
12
13# Copy requirements
14COPY requirements.txt .
15
16# Install Python dependencies
17RUN pip install --no-cache-dir -r requirements.txt
18
19# Copy application code
20COPY app.py .
21COPY models/ ./models/
22
23# Create non-root user
24RUN useradd -m -u 1000 appuser && chown -R appuser:appuser /app
25USER appuser
26
27# Expose port
28EXPOSE 8000
29
30# Health check
31HEALTHCHECK --interval=30s --timeout=30s --start-period=5s --retries=3 \\
32    CMD curl -f http://localhost:8000/health || exit 1
33
34# Run application
35CMD ["python", "app.py"]

Step 3: Requirements File

1# requirements.txt
2fastapi==0.104.1
3uvicorn[standard]==0.24.0
4torch==2.1.0
5numpy==1.24.3
6pydantic==2.4.2
7python-multipart==0.0.6

Step 4: Build and Test

1# Build Docker image
2docker build -t ai-model:v1.0.0 .
3
4# Test locally
5docker run -p 8000:8000 ai-model:v1.0.0
6
7# Test endpoints
8curl http://localhost:8000/health
9curl -X POST http://localhost:8000/predict \\
10  -H "Content-Type: application/json" \\
11  -d '{"data": [[1.0, 2.0, 3.0]]}'

Creating Kubernetes Manifests

Step 1: Deployment Configuration

1# deployment.yaml
2apiVersion: apps/v1
3kind: Deployment
4metadata:
5  name: ai-model-deployment
6  labels:
7    app: ai-model
8    version: v1.0.0
9spec:
10  replicas: 3
11  selector:
12    matchLabels:
13      app: ai-model
14  template:
15    metadata:
16      labels:
17        app: ai-model
18        version: v1.0.0
19    spec:
20      containers:
21      - name: ai-model
22        image: ai-model:v1.0.0
23        ports:
24        - containerPort: 8000
25        env:
26        - name: MODEL_VERSION
27          value: "1.0.0"
28        - name: MODEL_PATH
29          value: "/app/models/model.pt"
30        resources:
31          requests:
32            memory: "512Mi"
33            cpu: "250m"
34          limits:
35            memory: "2Gi"
36            cpu: "1000m"
37        livenessProbe:
38          httpGet:
39            path: /health
40            port: 8000
41          initialDelaySeconds: 30
42          periodSeconds: 10
43        readinessProbe:
44          httpGet:
45            path: /ready
46            port: 8000
47          initialDelaySeconds: 5
48          periodSeconds: 5
49        volumeMounts:
50        - name: model-storage
51          mountPath: /app/models
52      volumes:
53      - name: model-storage
54        persistentVolumeClaim:
55          claimName: model-pvc
56---
57apiVersion: v1
58kind: PersistentVolumeClaim
59metadata:
60  name: model-pvc
61spec:
62  accessModes:
63    - ReadOnlyMany
64  resources:
65    requests:
66      storage: 10Gi

Step 2: Service Configuration

1# service.yaml
2apiVersion: v1
3kind: Service
4metadata:
5  name: ai-model-service
6  labels:
7    app: ai-model
8spec:
9  selector:
10    app: ai-model
11  ports:
12  - port: 80
13    targetPort: 8000
14    protocol: TCP
15  type: ClusterIP
16---
17apiVersion: networking.k8s.io/v1
18kind: Ingress
19metadata:
20  name: ai-model-ingress
21  annotations:
22    nginx.ingress.kubernetes.io/rewrite-target: /
23    nginx.ingress.kubernetes.io/ssl-redirect: "true"
24spec:
25  tls:
26  - hosts:
27    - ai-model.example.com
28    secretName: ai-model-tls
29  rules:
30  - host: ai-model.example.com
31    http:
32      paths:
33      - path: /
34        pathType: Prefix
35        backend:
36          service:
37            name: ai-model-service
38            port:
39              number: 80

Setting Up Model Serving

Step 1: Deploy to Kubernetes

1# Apply configurations
2kubectl apply -f deployment.yaml
3kubectl apply -f service.yaml
4
5# Check deployment status
6kubectl get deployments
7kubectl get pods
8kubectl get services
9
10# View logs
11kubectl logs -l app=ai-model -f

Step 2: Configure Load Balancing

1# hpa.yaml - Horizontal Pod Autoscaler
2apiVersion: autoscaling/v2
3kind: HorizontalPodAutoscaler
4metadata:
5  name: ai-model-hpa
6spec:
7  scaleTargetRef:
8    apiVersion: apps/v1
9    kind: Deployment
10    name: ai-model-deployment
11  minReplicas: 2
12  maxReplicas: 10
13  metrics:
14  - type: Resource
15    resource:
16      name: cpu
17      target:
18        type: Utilization
19        averageUtilization: 70
20  - type: Resource
21    resource:
22      name: memory
23      target:
24        type: Utilization
25        averageUtilization: 80

Step 3: GPU Support (Optional)

1# gpu-deployment.yaml
2apiVersion: apps/v1
3kind: Deployment
4metadata:
5  name: ai-model-gpu-deployment
6spec:
7  replicas: 1
8  selector:
9    matchLabels:
10      app: ai-model-gpu
11  template:
12    metadata:
13      labels:
14        app: ai-model-gpu
15    spec:
16      containers:
17      - name: ai-model
18        image: ai-model:v1.0.0-gpu
19        resources:
20          limits:
21            nvidia.com/gpu: 1
22          requests:
23            nvidia.com/gpu: 1
24        env:
25        - name: CUDA_VISIBLE_DEVICES
26          value: "0"
27      nodeSelector:
28        accelerator: nvidia-tesla-k80

Implementing Auto-scaling

Step 1: Metrics Server Setup

1# Install metrics server
2kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
3
4# Verify installation
5kubectl get deployment metrics-server -n kube-system

Step 2: Custom Metrics (Prometheus)

1# prometheus-config.yaml
2apiVersion: v1
3kind: ConfigMap
4metadata:
5  name: prometheus-config
6data:
7  prometheus.yml: |
8    global:
9      scrape_interval: 15s
10    scrape_configs:
11    - job_name: 'ai-model'
12      static_configs:
13      - targets: ['ai-model-service:80']
14      metrics_path: /metrics
15      scrape_interval: 5s

Step 3: Custom HPA with Prometheus

1# custom-hpa.yaml
2apiVersion: autoscaling/v2
3kind: HorizontalPodAutoscaler
4metadata:
5  name: ai-model-custom-hpa
6spec:
7  scaleTargetRef:
8    apiVersion: apps/v1
9    kind: Deployment
10    name: ai-model-deployment
11  minReplicas: 2
12  maxReplicas: 20
13  metrics:
14  - type: Pods
15    pods:
16      metric:
17        name: requests_per_second
18      target:
19        type: AverageValue
20        averageValue: "100"
21  behavior:
22    scaleUp:
23      stabilizationWindowSeconds: 60
24      policies:
25      - type: Percent
26        value: 100
27        periodSeconds: 15
28    scaleDown:
29      stabilizationWindowSeconds: 300
30      policies:
31      - type: Percent
32        value: 10
33        periodSeconds: 60

Monitoring and Logging

Step 1: Prometheus and Grafana

1# Install Prometheus Operator
2helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
3helm repo update
4
5helm install prometheus prometheus-community/kube-prometheus-stack \\
6  --namespace monitoring \\
7  --create-namespace \\
8  --set grafana.adminPassword=admin123

Step 2: Custom Metrics in Application

1# metrics.py
2from prometheus_client import Counter, Histogram, Gauge, generate_latest
3import time
4
5# Define metrics
6REQUEST_COUNT = Counter('ai_model_requests_total', 'Total requests', ['method', 'endpoint'])
7REQUEST_LATENCY = Histogram('ai_model_request_duration_seconds', 'Request latency')
8MODEL_PREDICTIONS = Counter('ai_model_predictions_total', 'Total predictions made')
9ACTIVE_CONNECTIONS = Gauge('ai_model_active_connections', 'Active connections')
10
11# Middleware for FastAPI
12@app.middleware("http")
13async def add_prometheus_middleware(request, call_next):
14    start_time = time.time()
15    
16    # Increment request counter
17    REQUEST_COUNT.labels(method=request.method, endpoint=request.url.path).inc()
18    
19    # Process request
20    response = await call_next(request)
21    
22    # Record latency
23    REQUEST_LATENCY.observe(time.time() - start_time)
24    
25    return response
26
27@app.get("/metrics")
28async def metrics():
29    """Prometheus metrics endpoint"""
30    return Response(generate_latest(), media_type="text/plain")

Step 3: Grafana Dashboard

1{
2  "dashboard": {
3    "title": "AI Model Monitoring",
4    "panels": [
5      {
6        "title": "Request Rate",
7        "type": "graph",
8        "targets": [
9          {
10            "expr": "rate(ai_model_requests_total[5m])",
11            "legendFormat": "{{method}} {{endpoint}}"
12          }
13        ]
14      },
15      {
16        "title": "Response Time",
17        "type": "graph",
18        "targets": [
19          {
20            "expr": "histogram_quantile(0.95, ai_model_request_duration_seconds_bucket)",
21            "legendFormat": "95th percentile"
22          }
23        ]
24      },
25      {
26        "title": "Pod Count",
27        "type": "stat",
28        "targets": [
29          {
30            "expr": "kube_deployment_status_replicas{deployment=\"ai-model-deployment\"}",
31            "legendFormat": "Replicas"
32          }
33        ]
34      }
35    ]
36  }
37}

CI/CD Pipeline Integration

Step 1: GitHub Actions Workflow

1# .github/workflows/deploy.yml
2name: Deploy AI Model
3
4on:
5  push:
6    branches: [main]
7  pull_request:
8    branches: [main]
9
10env:
11  REGISTRY: ghcr.io
12  IMAGE_NAME: ${{ github.repository }}/ai-model
13
14jobs:
15  test:
16    runs-on: ubuntu-latest
17    steps:
18    - uses: actions/checkout@v4
19    
20    - name: Set up Python
21      uses: actions/setup-python@v4
22      with:
23        python-version: '3.10'
24    
25    - name: Install dependencies
26      run: |
27        pip install -r requirements.txt
28        pip install pytest
29    
30    - name: Run tests
31      run: pytest tests/
32  
33  build:
34    needs: test
35    runs-on: ubuntu-latest
36    permissions:
37      contents: read
38      packages: write
39    
40    steps:
41    - uses: actions/checkout@v4
42    
43    - name: Log in to Container Registry
44      uses: docker/login-action@v3
45      with:
46        registry: ${{ env.REGISTRY }}
47        username: ${{ github.actor }}
48        password: ${{ secrets.GITHUB_TOKEN }}
49    
50    - name: Extract metadata
51      id: meta
52      uses: docker/metadata-action@v5
53      with:
54        images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
55        tags: |
56          type=ref,event=branch
57          type=ref,event=pr
58          type=sha
59    
60    - name: Build and push Docker image
61      uses: docker/build-push-action@v5
62      with:
63        context: .
64        push: true
65        tags: ${{ steps.meta.outputs.tags }}
66        labels: ${{ steps.meta.outputs.labels }}
67  
68  deploy:
69    needs: build
70    runs-on: ubuntu-latest
71    if: github.ref == 'refs/heads/main'
72    
73    steps:
74    - uses: actions/checkout@v4
75    
76    - name: Configure kubectl
77      run: |
78        echo "${{ secrets.KUBE_CONFIG }}" | base64 -d > kubeconfig
79        export KUBECONFIG=kubeconfig
80    
81    - name: Deploy to Kubernetes
82      run: |
83        export KUBECONFIG=kubeconfig
84        sed -i "s|ai-model:v1.0.0|${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }}|g" deployment.yaml
85        kubectl apply -f deployment.yaml
86        kubectl apply -f service.yaml
87        kubectl rollout status deployment/ai-model-deployment

Step 2: ArgoCD GitOps (Alternative)

1# argocd-application.yaml
2apiVersion: argoproj.io/v1alpha1
3kind: Application
4metadata:
5  name: ai-model
6  namespace: argocd
7spec:
8  project: default
9  source:
10    repoURL: https://github.com/your-org/ai-model-k8s
11    targetRevision: HEAD
12    path: manifests
13  destination:
14    server: https://kubernetes.default.svc
15    namespace: ai-models
16  syncPolicy:
17    automated:
18      prune: true
19      selfHeal: true
20    syncOptions:
21    - CreateNamespace=true

Security Best Practices

Step 1: Network Policies

1# network-policy.yaml
2apiVersion: networking.k8s.io/v1
3kind: NetworkPolicy
4metadata:
5  name: ai-model-network-policy
6spec:
7  podSelector:
8    matchLabels:
9      app: ai-model
10  policyTypes:
11  - Ingress
12  - Egress
13  ingress:
14  - from:
15    - namespaceSelector:
16        matchLabels:
17          name: ingress-nginx
18    ports:
19    - protocol: TCP
20      port: 8000
21  egress:
22  - to: []
23    ports:
24    - protocol: TCP
25      port: 443  # HTTPS
26    - protocol: TCP
27      port: 53   # DNS
28    - protocol: UDP
29      port: 53   # DNS

Step 2: Pod Security Standards

1# pod-security-policy.yaml
2apiVersion: v1
3kind: Pod
4metadata:
5  name: ai-model-pod
6spec:
7  securityContext:
8    runAsNonRoot: true
9    runAsUser: 1000
10    fsGroup: 1000
11    seccompProfile:
12      type: RuntimeDefault
13  containers:
14  - name: ai-model
15    image: ai-model:v1.0.0
16    securityContext:
17      allowPrivilegeEscalation: false
18      readOnlyRootFilesystem: true
19      capabilities:
20        drop:
21        - ALL
22    volumeMounts:
23    - name: tmp
24      mountPath: /tmp
25    - name: var-tmp
26      mountPath: /var/tmp
27  volumes:
28  - name: tmp
29    emptyDir: {}
30  - name: var-tmp
31    emptyDir: {}

Step 3: Secrets Management

1# secret.yaml
2apiVersion: v1
3kind: Secret
4metadata:
5  name: ai-model-secrets
6type: Opaque
7data:
8  api-key: <base64-encoded-api-key>
9  db-password: <base64-encoded-password>
10---
11apiVersion: apps/v1
12kind: Deployment
13metadata:
14  name: ai-model-deployment
15spec:
16  template:
17    spec:
18      containers:
19      - name: ai-model
20        env:
21        - name: API_KEY
22          valueFrom:
23            secretKeyRef:
24              name: ai-model-secrets
25              key: api-key

Troubleshooting Common Issues

Issue 1: Pod Startup Failures

1# Debug pod issues
2kubectl describe pod <pod-name>
3kubectl logs <pod-name> --previous
4
5# Common solutions
6# 1. Check resource limits
7kubectl top pods
8kubectl describe nodes
9
10# 2. Verify image availability
11docker pull ai-model:v1.0.0
12
13# 3. Check secrets and configmaps
14kubectl get secrets
15kubectl describe secret ai-model-secrets

Issue 2: Service Discovery Problems

1# Test service connectivity
2kubectl run debug --image=busybox -it --rm -- sh
3nslookup ai-model-service
4wget -qO- http://ai-model-service/health
5
6# Check endpoints
7kubectl get endpoints ai-model-service
8kubectl describe service ai-model-service

Issue 3: Performance Issues

1# Monitor resource usage
2kubectl top pods
3kubectl top nodes
4
5# Check HPA status
6kubectl get hpa
7kubectl describe hpa ai-model-hpa
8
9# Analyze metrics
10kubectl port-forward svc/prometheus-server 9090:80
11# Visit http://localhost:9090

Issue 4: GPU Not Available

1# Check GPU nodes
2kubectl get nodes -l accelerator=nvidia-tesla-k80
3
4# Verify GPU plugin
5kubectl get daemonset nvidia-device-plugin-daemonset -n kube-system
6
7# Check pod GPU allocation
8kubectl describe pod <gpu-pod-name>

Conclusion

Deploying AI models with Kubernetes and Docker provides a robust, scalable foundation for production AI systems. Key takeaways:

Best Practices Summary

Containerize properly with health checks and security
Use resource limits to prevent resource starvation
Implement monitoring from day one
Automate deployments with CI/CD pipelines
Plan for scaling with HPA and cluster autoscaling
Secure by default with network policies and RBAC

Next Steps

Implement A/B testing for model versions
Add distributed tracing with Jaeger
Explore service mesh with Istio
Implement chaos engineering practices
Consider multi-cluster deployments

Resources

Ready to deploy your AI models at scale? Start with a simple deployment and gradually add complexity as your needs grow. Remember: production readiness is a journey, not a destination! 🚀