Throughput

Category: Hardware & Infrastructure

Definition

Throughput measures the number of AI inference requests a system can process per unit time, typically expressed as queries per second or tokens per second.

How It Works

Throughput depends on model size, batch processing efficiency, and hardware capabilities. Systems optimize throughput through batching, parallel processing, and efficient scheduling.

Load balancing across multiple GPUs or instances increases aggregate throughput for production systems.

Why It Matters

High throughput reduces serving costs and enables AI systems to handle millions of users. It's the key metric for production AI deployments.

Improving throughput by 10x can make previously uneconomical AI applications viable at scale.

Related Terms

• Batch Inference
• Load Balancing
• Inference Serving
• Latency

← Back to Hardware & Infrastructure | All Terms