vLLM

Evidence (17)

• 7NVIDIA GenAI·10/2/2026product_launch
  NVIDIA adds 64GB DGX Spark for local AI development

  NVIDIA is introducing a 64GB DGX Spark configuration through select partners, aimed at developers, researchers, and enthusiasts building local AI workloads. The system ships with DGX OS and NVIDIA’s AI software stack, supports up to 100-billion-parameter models on device, and can be clustered in pairs to expand memory and performance without additional setup.

• 6Hacker News·9/30/2026product_launch
  Launch HN: Magnitude, a self-optimizing local inference engine for agents

  Magnitude is a newly launched open source inference engine for local agents that tunes itself on the target hardware to improve speed and memory efficiency. The team claims it can run on Mac, Linux, and Windows and reports benchmark gains over llama.cpp for both decode and prefill on tested hardware.

• 7Hacker News·9/16/2026research
  4B Model Beats Postgres Query Plans on Join Queries

  The article describes an experiment that post-trained a small 4B open-weights model to generate Postgres query plans that outperform the database's default optimizer on join-heavy workloads. Using supervised fine-tuning and reinforcement learning, the model reportedly achieved a 44.7% latency reduction across 113 queries and handled cases the initial model could not plan for.

• 5Hacker News·9/7/2026research
  Speculative Decoding Performance in vLLM on AMD GPUs

  This article explains how speculative decoding works in vLLM and compares several draft-and-verify methods, including native MTP, Gemma 4 MTP, EAGLE-3, DFlash, and DSpark. The authors report that throughput gains varied significantly depending on the drafting approach, proposal length, model family, draft checkpoint, workload, and token acceptance behavior on AMD Instinct MI300X and MI355X GPUs.

• 8NVIDIA GenAI·9/3/2026product_launch
  NVIDIA Brings Local AI, Faster Inference, and New RTX Spark PCs to IFA 2026

  NVIDIA announced new local AI features and hardware partnerships at IFA 2026, including simplified setup for agent apps on NVIDIA GPUs, faster inference for popular local stacks, and a new personal AI routing tool. The company also said compact RTX Spark Windows PCs from Lenovo and Acer will arrive in October, alongside support for several newly launched models and video-generation tools optimized for NVIDIA hardware.

• 8Hacker News·8/28/2026open_source
  GLM-5.3 released as open weights

  GLM-5.3 is now available as an open-weights model, built on the same base model as GLM-5.2 with improvements coming entirely from post-training. The release highlights stronger coding and long-horizon task performance, along with notable gains in cyber-related benchmarks and local deployment support across several inference frameworks.

• 7Hacker News·8/21/2026research
  Qwen3-TTS implementation reaches sub-50 ms voice latency

  The authors report a Qwen3-TTS 1.7B CustomVoice implementation that achieves sub-50 ms p95 time-to-first-audio and about 10 requests per second on a single NVIDIA H100 SXM while maintaining real-time playback. They benchmarked multiple serving engines, tuned them for low-latency streaming, and released the implementation and benchmark as open source.

• 8Hacker News·8/12/2026model_release
  Qwen3.8 Open-Model Release

  Qwen has released Qwen3.8-2.4T-A95B, a post-trained open model available in Hugging Face Transformers format and compatible with tools such as vLLM and SGLang. The model is positioned as the most capable Qwen open-model generation to date, with improvements in coding, agentic task execution, long-context handling, and flexible reasoning control.

• 8Hugging Face Blog·8/10/2026model_release
  Meta releases Muse Glimmer, a local multimodal open-source model

  Meta has released Muse Glimmer, a 30B-parameter multimodal model distilled for local, agentic use cases and licensed under Apache 2.0. Hugging Face highlights day-one support across transformers, llama.cpp, vLLM, and Inference Endpoints, along with benchmarks and architecture details showing its focus on privacy-aware coding, document analysis, and personal assistant workflows.

• 8Hugging Face Blog·8/4/2026model_release
  Hugging Face introduces LFM2.5-2.6B for on-device agents

  Hugging Face announced LFM2.5-2.6B, a 2.6B-parameter model designed to run capable agents entirely on-device with tool calling and multi-step workflows. The model is positioned for laptops and phones, emphasizing privacy, low memory use, and strong benchmark performance relative to much larger models.

• 6InfoQ·7/27/2026framework_update
  Netflix Explains Its In-House LLM Serving Platform

  Netflix outlined the production lessons behind its internal LLM inference serving platform, focusing on how it supports models of different sizes and hardware requirements. The company also highlighted the need to adapt quickly as inference engines continue to evolve, with Triton and vLLM among the technologies involved.

• 8Hugging Face Blog·7/8/2026framework_update
  Transformers backend in vLLM reaches native speed

  Hugging Face says the transformers modeling backend in vLLM now matches or exceeds the throughput of hand-written native vLLM implementations for several Qwen3 models. The update lets model authors run Hugging Face Transformers models in vLLM with a single flag while keeping standard parallelism and serving workflows intact.

• 7Hugging Face Blog·7/7/2026product_launch
  Hugging Face models arrive on Microsoft Foundry Managed Compute

  Microsoft announced that curated Hugging Face open-weight models are now available on Foundry Managed Compute, with weekly refreshes and one-click deployment. The offering adds enterprise controls such as security scanning, governance, observability, billing, and support for managed GPU deployment of open-source and custom models.

• 5Hacker News·6/28/2026open_source
  AMD Strix Halo RDMA Cluster Setup Guide

  This GitHub guide explains how to set up an RDMA-enabled cluster using AMD Strix Halo hardware for distributed workloads. It is aimed at developers experimenting with high-performance networking and local AI or compute clustering rather than announcing a new product.

• 5Hugging Face Blog·6/26/2026product_launch
  Run a vLLM Server on HF Jobs in One Command

  Hugging Face explains how to launch a private, OpenAI-compatible vLLM endpoint on HF Jobs with a single command, using a GPU-backed container and a public jobs proxy. The post covers how to query the server with curl or the OpenAI Python client, plus how to stop the job and manage access securely.

• 7Hugging Face Blog·5/27/2026framework_update
  TRL Adds Delta Weight Sync via Hugging Face Hub Bucket

  Hugging Face describes a TRL update that sends only weight deltas, instead of full model checkpoints, during async RL training. The approach stores sparse safetensors diffs in a Hugging Face Bucket and lets vLLM fetch them, cutting per-step transfer size dramatically and enabling disaggregated training across separate machines and Spaces.

• 7Hugging Face Blog·5/6/2026framework_update
  vLLM V1 Migration: Fixing Rollout Logprobs Before RL Objective Changes

  Hugging Face describes a migration from vLLM V0 to V1 in an RL training pipeline, where mismatched rollout logprobs initially changed training behavior. The team restored parity by enabling processed logprobs, adjusting V1 runtime defaults, fixing the inflight weight-update path, and using an fp32 lm_head before considering any RL objective corrections.