SwiGLU

Assess

Techniques

A gated activation function used in transformer MLP blocks.

Why it's here

Placed in Assess: 2 article(s) of evidence from 2 source(s), led by model releases, with 0 in the last 30 days. Confidence 45%.

8Hacker News·9/1/2026research
Small Transformer Trained in 1.5 Hours Beats Many LLMs on ARC
The author reports training a small transformer from scratch in about 1.5 hours on a 5090 GPU and claims it matches or beats several larger models on ARC-style evaluation. The updated approach adds architectural and training changes such as SwiGLU, RMSNorm, fewer augmentations, and using ARC-2 non-overlapping tasks while filtering duplicates to avoid data leakage.
7Hugging Face Blog·4/29/2026model_release
Granite 4.1 LLMs: How They Are Built
IBM’s Granite Team details how Granite 4.1 was trained, from data engineering through supervised fine-tuning and reinforcement learning. The family includes dense decoder-only models in 3B, 8B, and 30B sizes, trained on about 15 trillion tokens and extended to a 512K context window, with the 8B instruct model reportedly matching or exceeding the prior Granite 4.0-H-Small despite using fewer parameters.