Batch sizing

Hold

Techniques

Configuring how many requests are processed together to balance latency against throughput.

Why it's here

Placed in Hold: 1 article(s) of evidence from 1 source(s), led by research-stage coverage, with 0 in the last 30 days. Confidence 24%. Low accumulated evidence, so it defaults conservatively pending more signal.

Evidence (1)

4Hacker News·9/1/2026research
Efficient frontier tradeoffs in LLM inference
The article explains how LLM inference engineers manage tradeoffs between latency, throughput, cost, and quality using the idea of an efficient frontier. It distinguishes techniques that move a deployment along the frontier, such as batch sizing and parallelism choices, from techniques that expand the frontier through gains like KV cache reuse, quantization, distillation, and pruning.