Rotary Positional Embedding

Evidence (5)

• 8Hacker News·9/1/2026research
  Small Transformer Trained in 1.5 Hours Beats Many LLMs on ARC

  The author reports training a small transformer from scratch in about 1.5 hours on a 5090 GPU and claims it matches or beats several larger models on ARC-style evaluation. The updated approach adds architectural and training changes such as SwiGLU, RMSNorm, fewer augmentations, and using ARC-2 non-overlapping tasks while filtering duplicates to avoid data leakage.

• 7Hugging Face Blog·8/25/2026model_release
  How Granite 4.2 Reasoning LLMs Were Built

  IBM and the Granite team detailed how Granite 4.2 was built as a new family of dense, decoder-only reasoning models in 3B, 8B, and 30B sizes. The models were pre-trained from scratch on about 15T tokens, then supervised fine-tuned and post-trained with multi-stage reinforcement learning, including agentic RL for tool use in the 8B and 30B versions. The release also adds thinking/non-thinking modes, low-effort reasoning, native tool calling, and an expanded 512K context window.

• 8Hacker News·7/28/2026model_release
  Kimi K3 Architecture Highlights LatentMoE and NoPE

  Kimi K3 is presented as a scaled-up production version of Kimi Linear, growing from 48B to 2.8T parameters and positioning itself as the largest open-weight model to date. The architecture emphasizes inference efficiency with components such as LatentMoE, Kimi Delta Attention, attention residuals, and a full switch to NoPE, while also adding native multimodal support.

• 7Hugging Face Blog·4/29/2026model_release
  Granite 4.1 LLMs: How They Are Built

  IBM’s Granite Team details how Granite 4.1 was trained, from data engineering through supervised fine-tuning and reinforcement learning. The family includes dense decoder-only models in 3B, 8B, and 30B sizes, trained on about 15 trillion tokens and extended to a 512K context window, with the 8B instruct model reportedly matching or exceeding the prior Granite 4.0-H-Small despite using fewer parameters.

• 5Hugging Face Blog·2/3/2026research
  Ablation Lessons for Training Text-to-Image Models

  Hugging Face publishes the second part of Photoroom’s series on training efficient text-to-image foundation models from scratch. The post reviews a baseline Flow Matching setup for the PRX-1.2B model and examines which training changes improve convergence, stability, and efficiency in practice.