7.1. Scaling Training and Serving

When datasets and models become too large for a single GPU or machine, you split the workload across multiple nodes. There are two main strategies:

🧮 Data Parallelism

Each worker gets a different chunk of the data but the same model copy. After each batch, workers synchronize gradients to keep model weights consistent.

Example: If you have 4 GPUs and 1 million samples, each GPU trains on 250k samples per step. After gradients are averaged, the model updates once per global batch.

Tools:

• PyTorch Distributed Data Parallel (DDP)
• TensorFlow tf.distribute.MirroredStrategy
• Horovod (Uber’s distributed training library)

When to Use: When your model fits into one GPU’s memory but data is huge.

💡 Intuition: Imagine multiple chefs each cooking part of the meal with the same recipe — then combining results into one final dish.

🧩 Model Parallelism

When the model itself is too big to fit on one GPU, you split the model’s layers across devices.

Example:

• GPU 1: Layers 1–4
• GPU 2: Layers 5–8
• GPU 3: Layers 9–12

Each GPU passes forward activations to the next, like an assembly line.

Tools:

• DeepSpeed, Megatron-LM (used for large LLMs)
• Tensor Parallelism and Pipeline Parallelism

When to Use: For extremely large models (e.g., GPT, BERT, Vision Transformers).

💡 Intuition: Think of model parallelism like a car factory — each station (GPU) assembles a different part before passing it along.

Once your model is trained, the next challenge is how to deliver predictions efficiently.

🍱 Batch Inference (Offline Serving)

• Used for large-scale predictions on static datasets (e.g., scoring millions of users nightly).
• Typically triggered by schedulers like Airflow or Spark jobs.
• High throughput, no real-time constraints.

Example: Recomputing all customer credit risk scores every 24 hours.

💡 Analogy: Like delivering meals in bulk once a day — efficient but not instant.

⚡ Online Inference (Real-Time Serving)

• Used when predictions must be made in milliseconds (e.g., recommendation, fraud detection).
• Model is deployed behind an API endpoint (e.g., FastAPI, TensorFlow Serving, TorchServe).
• Prioritize low latency, auto-scaling, and caching.

Example: When a user clicks “Pay,” the fraud model predicts instantly if it’s suspicious.

💡 Analogy: Like a fast-food counter — every order is served instantly, one customer at a time.

🌊 Streaming Inference (Continuous Serving)

• Ideal for event-driven pipelines (e.g., IoT, logs, or real-time personalization).
• Models process data as it arrives — using tools like Kafka Streams, Flink, or Ray Serve.
• Requires asynchronous buffering, checkpointing, and backpressure management.

Example: Predicting energy consumption in real-time from thousands of IoT sensors.

💡 Analogy: Like a conveyor belt sushi restaurant — new items arrive continuously, and the system keeps up in real time.

Scaling isn’t just about adding more machines — it’s about adding them at the right time.

Kubernetes’ Horizontal Pod Autoscaler (HPA) automatically adjusts the number of serving pods based on demand.

🧠 How It Works:

1. Define CPU or memory thresholds.
2. When usage exceeds threshold, HPA spawns new pods.
3. When demand falls, it scales down to save cost.

Example Config (YAML):

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: model-serving
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: fraud-detector
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

Metrics You Can Use for Scaling:

• CPU/GPU utilization
• Request latency
• Queue length
• Throughput (requests per second)

💡 Intuition: Autoscaling is like adding more delivery drivers during rush hour and letting them rest when the roads are quiet.

In queueing theory, average latency is approximated by:

Where:

• $L$ = latency
• $\lambda$ = incoming request rate
• $\mu$ = service rate (capacity per server)

As $\lambda$ approaches $\mu$, latency spikes exponentially. Autoscaling increases $\mu$ dynamically, keeping $(\mu - \lambda)$ large and latency low.

• “Scaling means just adding more machines.” No — poor parallelization or data skew can still bottleneck your system.

• “Model size determines serving speed.” Not entirely — serving speed depends more on hardware, batching, and architecture.

• “Autoscaling handles everything automatically.” You still need to define thresholds, metrics, and cost controls to avoid runaway scaling.