Grouped Query Attention
AssessTechniques
An attention mechanism that reduces KV-cache cost by sharing key/value heads across query groups.
Why it's here
Placed in Assess: 3 article(s) of evidence from 3 source(s), led by model releases, with 0 in the last 30 days. Confidence 58%.
Evidence (3)
- 7Hugging Face Blog·8/25/2026model_releaseHow Granite 4.2 Reasoning LLMs Were Built
IBM and the Granite team detailed how Granite 4.2 was built as a new family of dense, decoder-only reasoning models in 3B, 8B, and 30B sizes. The models were pre-trained from scratch on about 15T tokens, then supervised fine-tuned and post-trained with multi-stage reinforcement learning, including agentic RL for tool use in the 8B and 30B versions. The release also adds thinking/non-thinking modes, low-effort reasoning, native tool calling, and an expanded 512K context window.
- 7Hugging Face Blog·4/29/2026model_releaseGranite 4.1 LLMs: How They Are Built
IBM’s Granite Team details how Granite 4.1 was trained, from data engineering through supervised fine-tuning and reinforcement learning. The family includes dense decoder-only models in 3B, 8B, and 30B sizes, trained on about 15 trillion tokens and extended to a 512K context window, with the 8B instruct model reportedly matching or exceeding the prior Granite 4.0-H-Small despite using fewer parameters.
- 7Hugging Face Blog·1/20/2026researchDifferential Transformer V2
Hugging Face Blog presents Differential Transformer V2, a revised attention design that doubles query heads while keeping key-value heads unchanged. The post argues this improves decoding speed, avoids custom attention kernels, and preserves standard output projection size while adding a differential subtraction step between paired heads.