LLM-as-a-judge
AssessTechniques
A method that uses an LLM to evaluate outputs, behavior, or policy compliance.
Why it's here
Placed in Assess: 4 article(s) of evidence from 2 source(s), led by research-stage coverage, with 1 in the last 30 days. Confidence 48%.
Evidence (4)
- 6The New Stack·8/8/2026product_launchSpeakeasy launches enterprise AI skill management
Speakeasy has introduced Skills Management, a system for centrally registering AI skills as immutable enterprise artifacts with versioning, scoped access, and usage observability. The goal is to help companies track ownership, identify the current version, and reduce duplicated or outdated skills spread across laptops, repositories, Slack, and plugins.
- 6The New Stack·7/9/2026researchAutonomous ingestion errors poison a vector database
A fintech team describes how an autonomous PDF ingestion pipeline inserted incorrect metadata into a vector database, causing a chatbot to cite outdated and wrong financial data. The post-mortem says the root cause was treating probabilistic LLM extraction as deterministic, and that a second LLM-based validation step failed due to mutual confirmation bias.
- 4InfoQ·7/7/2026researchDesigning Reliable AI Platforms
Aaron Erickson describes how NVIDIA designs and tests AI agent hierarchies for production use. The presentation emphasizes combining deterministic tools with agentic discovery, using rare context effectively, and applying LLM-as-a-judge test pyramids to improve reliability at scale.
- 5InfoQ·6/30/2026securitySecuring AI-Accelerated Development
Sriram Madapusi Vasudevan outlines security patterns for autonomous AI agents used in production development workflows. He highlights risks in the ReAct loop, including context, reasoning, and tool-execution vulnerabilities, and discusses mitigations such as defense-in-depth, LLM-as-a-judge critics, and MAESTRO threat modeling.