sglang

Evidence (11)

• 6Hacker News·9/30/2026product_launch
  Launch HN: Magnitude, a self-optimizing local inference engine for agents

  Magnitude is a newly launched open source inference engine for local agents that tunes itself on the target hardware to improve speed and memory efficiency. The team claims it can run on Mac, Linux, and Windows and reports benchmark gains over llama.cpp for both decode and prefill on tested hardware.

• 7Hugging Face Blog·9/24/2026model_release
  LiquidAI releases DSpark draft model for LFM2.5-VL-3B

  LiquidAI released an experimental DSpark drafter for its vision-language model LFM2.5-VL-3B, adding speculative decoding to speed up inference without changing output quality. The company reports decode speedups of up to 3.13x on-device and 2.66x on H100, with end-to-end gains up to 2.62x and 2.27x, and day-one support in llama.cpp, MLX-VLM, and SGLang.

• 8Hacker News·8/28/2026open_source
  GLM-5.3 released as open weights

  GLM-5.3 is now available as an open-weights model, built on the same base model as GLM-5.2 with improvements coming entirely from post-training. The release highlights stronger coding and long-horizon task performance, along with notable gains in cyber-related benchmarks and local deployment support across several inference frameworks.

• 8The New Stack·8/26/2026model_release
  Z.ai unveils GLM-5.3-Flash, a low-cost hybrid model on Chinese chips

  Z.ai revealed that the stealth model Ox-alpha is GLM-5.3-Flash, a 320B-parameter hybrid model with 18B active parameters designed for very low-cost inference. The model is open-sourced on Hugging Face under the MIT license and is already available on platforms like OpenRouter, with Z.ai emphasizing that it was served efficiently on Chinese AI chips using its own inference stack based on SGLang.

• 7Hacker News·8/23/2026research
  Qwen 3.8 27B completes a reverse-engineering task in 30 minutes

  A hands-on test found Qwen 3.8 27B could work through a complex local reverse-engineering task on a single workstation and produce both a report and a bypass. The model relied on static analysis and Bash-based tools, and it corrected its own mistakes during the process. The article frames this as evidence of strong local model capability, while also highlighting cybersecurity concerns.

• 7Hacker News·8/21/2026research
  Qwen3-TTS implementation reaches sub-50 ms voice latency

  The authors report a Qwen3-TTS 1.7B CustomVoice implementation that achieves sub-50 ms p95 time-to-first-audio and about 10 requests per second on a single NVIDIA H100 SXM while maintaining real-time playback. They benchmarked multiple serving engines, tuned them for low-latency streaming, and released the implementation and benchmark as open source.

• 7Hugging Face Blog·8/20/2026model_release
  LFM2.5 DSpark Draft Models Deliver Faster Inference

  Hugging Face and Liquid AI released DSpark draft checkpoints for three LFM2.5 models, adding speculative decoding to improve inference speed without changing output quality. The company reports up to 3.18x higher GPU throughput, up to 2.87x on-device speedups, and a 57% average reduction in function-calling latency for LFM2.5-2.6B, with day-one support in llama.cpp and SGLang.

• 8Hacker News·8/12/2026model_release
  Qwen3.8 Open-Model Release

  Qwen has released Qwen3.8-2.4T-A95B, a post-trained open model available in Hugging Face Transformers format and compatible with tools such as vLLM and SGLang. The model is positioned as the most capable Qwen open-model generation to date, with improvements in coding, agentic task execution, long-context handling, and flexible reasoning control.

• 7Cloudflare Blog·8/3/2026research
  Cloudflare scales Kimi and GLM with KV cache quantization and weight compression

  Cloudflare says it is serving Moonshot's Kimi K-series and Z.ai's GLM more efficiently on Workers AI by combining KV cache quantization, weight compression, and cache protection techniques. The company reports that FP8 KV cache and INT4 weights cut GPU memory use and lower costs while preserving model accuracy, using SGLang as the serving framework.

• 7Hugging Face Blog·7/7/2026product_launch
  Hugging Face models arrive on Microsoft Foundry Managed Compute

  Microsoft announced that curated Hugging Face open-weight models are now available on Foundry Managed Compute, with weekly refreshes and one-click deployment. The offering adds enterprise controls such as security scanning, governance, observability, billing, and support for managed GPU deployment of open-source and custom models.

• 6Hacker News·7/3/2026research
  Wafer reports faster GLM5.2 inference on AMD MI355X

  Wafer says it ran GLM5.2 on AMD MI355X with 2626 tokens per second per node at 2.4 RPS, while claiming the setup cost is more than 2x lower than Blackwell-based alternatives. The post also describes using MXFP4 quantization with AMD Quark and serving the model with sglang on ROCm after fixing speculative decoding support.