Autoregressive decoding
AssessTechniques
Token-by-token generation where each next token depends on prior tokens.
Why it's here
Placed in Assess: 2 article(s) of evidence from 2 source(s), led by research breakthroughs, with 0 in the last 30 days. Confidence 43%.
Evidence (2)
- 5The New Stack·9/3/2026researchGuide to optimizing token usage in production LLM systems
The article explains why token costs in production LLM applications should be treated as a distributed systems problem rather than a simple billing issue. It describes how stateless APIs, repeated history replay, and large prompts create quadratic growth in cost and latency, then outlines prompt-hygiene and retrieval-based fixes to reduce token usage.
- 7Hacker News·6/30/2026breakthroughMoondream Details Pipelined Decoding to Reduce GPU Idle Time
Moondream describes how its Photon inference engine reduces GPU bubbles during autoregressive decoding by overlapping CPU bookkeeping with GPU forward passes. The approach uses pipelined decoding, ping-pong buffers, and deferred sampling/cleanup to improve decode throughput, with reported near-realtime VLM inference and up to 35% higher throughput on NVIDIA B200 hardware.