From Prototype to Production

Model Selection Framework

Container-Based Production Stack

# docker-compose.yml - Full production stack
services:
  inference:
    image: vllm/vllm-openai:latest
    runtime: nvidia
    ports: ["8000:8000"]
    volumes: ["./models:/models"]
    deploy:
      resources:
        reservations:
          devices:
           - driver: nvidia
              count: all
              capabilities: [gpu]
    command: >
      --model /models/Mistral-Large-3-AWQ
      --quantization awq
      --tensor-parallel-size 2
      --gpu-memory-utilization 0.9
      --max-model-len 32768
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 10s
      retries: 3

  proxy:
    image: nginx:alpine
    ports: ["443:443", "80:80"]
    volumes:
     - ./nginx.conf:/etc/nginx/nginx.conf
     - ./certs:/etc/nginx/certs
    depends_on: [inference]

  monitoring:
    image: grafana/grafana:latest
    ports: ["3000:3000"]
    volumes: ["./grafana:/var/lib/grafana"]

Key Metrics to Monitor

• Throughput: Tokens/second (aggregate and per-request)
• Latency: P50, P95, P99 response times
• VRAM Usage: Model weights + KV cache + overhead
• Queue Depth: Pending requests (indicates capacity limits)
• Cost/Token: Hardware amortization per token generated

Security Checklist

• ✅ Reverse proxy with TLS termination
• ✅ API key authentication at proxy layer
• ✅ Rate limiting per client
• ✅ Input sanitization (prompt injection defense)
• ✅ Output filtering (PII, harmful content)
• ✅ Network isolation (no direct internet access for inference)
• ✅ Regular model updates and security patches

Open-Source SOE (Standard Operating Environment)

Open-Source AOE (AI Operations Engineering) Blueprint

• Prometheus Inference Telemetry: Scrape vllm:num_requests_running, vllm:gpu_cache_usage_factor, and vllm:time_to_first_token_seconds every 5 seconds.
• Latency & TTFT SLAs: Enforce strict TTFT (Time To First Token) < 250ms and ITL (Inter-Token Latency) < 25ms P95 bounds.
• Memory Pressure Circuit Breakers: When GPU KV cache usage exceeds 92%, trigger preemptive request shedding or redirect traffic to quantized standby workers (AWQ/FP8).
• Continuous Batching & Chunked Prefill: Configure chunked prefill (--enable-chunked-prefill) to prevent long prompts from starving conversational streams.
• Canary Rollouts & Shadow Traffic: Replay 5% of production query streams to canary model instances to validate perplexity and latency drift before full traffic cutover.