LLM-as-a-judge
AssessTechniques
A method that uses an LLM to evaluate outputs, behavior, or policy compliance.
Why it's here
Placed in Assess: 7 article(s) of evidence from 2 source(s), led by framework updates, with 1 in the last 30 days. Confidence 57%.
Evidence (7)
- 5InfoQ·10/5/2026framework_updateBuilding Reusable Evaluation Frameworks for Agentic AI
Susan Chang describes how Elastic moved from scattered, ad hoc agent evaluations to a unified framework designed for production use. The approach combines LLM-as-a-judge with deterministic rules, connects Python-based evaluation workflows to TypeScript production systems, and adds deep tracing to detect regressions in complex RAG and cybersecurity workloads.
- 6The New Stack·8/31/2026product_launchOpenAI Tests Outcome-Based AI Pricing for Enterprises
OpenAI is reportedly testing a pricing model that charges enterprise customers only when an AI agent successfully completes a task, rather than billing by tokens consumed. The approach raises technical and operational challenges around defining success, validating outcomes, and handling partial completions or failures caused by external systems.
- 5InfoQ·8/14/2026framework_updateContext Engineering for Reliable Coding Agents
Patrick Debois and Baruch Sadogursky explain why coding agents often fail when prompts are overloaded with noisy context. They outline practical context engineering patterns such as lazy-loaded skills, versioned context artifacts, externalized memory banks, and LLM-as-a-judge evaluations to build more reliable agent workflows.
- 6The New Stack·8/8/2026product_launchSpeakeasy launches enterprise AI skill management
Speakeasy has introduced Skills Management, a system for centrally registering AI skills as immutable enterprise artifacts with versioning, scoped access, and usage observability. The goal is to help companies track ownership, identify the current version, and reduce duplicated or outdated skills spread across laptops, repositories, Slack, and plugins.
- 6The New Stack·7/9/2026researchAutonomous ingestion errors poison a vector database
A fintech team describes how an autonomous PDF ingestion pipeline inserted incorrect metadata into a vector database, causing a chatbot to cite outdated and wrong financial data. The post-mortem says the root cause was treating probabilistic LLM extraction as deterministic, and that a second LLM-based validation step failed due to mutual confirmation bias.
- 4InfoQ·7/7/2026researchDesigning Reliable AI Platforms
Aaron Erickson describes how NVIDIA designs and tests AI agent hierarchies for production use. The presentation emphasizes combining deterministic tools with agentic discovery, using rare context effectively, and applying LLM-as-a-judge test pyramids to improve reliability at scale.
- 5InfoQ·6/30/2026securitySecuring AI-Accelerated Development
Sriram Madapusi Vasudevan outlines security patterns for autonomous AI agents used in production development workflows. He highlights risks in the ReAct loop, including context, reasoning, and tool-execution vulnerabilities, and discusses mitigations such as defense-in-depth, LLM-as-a-judge critics, and MAESTRO threat modeling.