Optimize your endpoints

Optimization involves measuring performance with benchmarking, identifying bottlenecks, and tuning your endpoint configurations.

​

Quick optimization checklist

​

Understanding delay time

Two metrics affect request response time: Delay time breaks down into:

• Initialization time: Downloading Docker image
• Cold start time: Loading model into GPU memory

​

Reduce cold starts

​

Use cached models (recommended)

For models on Hugging Face, cached models provide the fastest cold starts and lowest cost.

​

Bake models into images

For private models, embed them in your Docker image. Models load from high-speed local NVMe storage instead of downloading at runtime.

​

Maintain active workers

Set active workers > 0 to eliminate cold starts entirely. Formula: Active workers = (Requests/min × Request duration in seconds) / 60 Example: 6 requests/min × 30 seconds = 3 active workers needed.

​

Improve availability

​

Select multiple GPU types

Specify multiple GPU types in priority order. A single high-end GPU often outperforms multiple lower-tier cards for . For endpoints with five or more workers, Runpod distributes workers across your GPU priorities to reduce throttling when your primary GPU type is constrained.

​

Add headroom to max workers

Set max workers ~20% above expected concurrency to handle load spikes without throttling.

​

Tune auto-scaling

Lower the queue delay threshold to 2-3 seconds (default: 4) for faster worker provisioning.

​

Architecture considerations