Uncertainty-Guarded Selective Prediction for Large Language Models
Abstract
LLMs are increasingly used as verifiers or judges for benchmarking, reward modeling in reinforcement learning, and data curation. However, LLMs, specifically in their judging role, remain vulnerable to systematic bias, hallucination, and miscalibration. Such limitations reduce the reliability of LLM-based evaluation and can propagate erroneous judgments into downstream systems. This thesis develops uncertainty-guarded selective evaluation to improve the reliability of LLM-based judging. We first develop a reference-guided consensus framework that aggregates the verdicts of multiple LLM judges. This reduces judge uncertainty and improves agreement with human evaluation. We then develop a lightweight selective voting approach that uses inter-judge disagreement as an uncertainty signal. The approach invokes a third judge only when the primary judges disagree. This avoids unnecessary model calls and lowers the cost of multi-judge evaluation. Next, we develop a search-augmented evaluation framework to address knowledge uncertainty. The LLM judge retrieves and synthesizes external evidence instead of relying solely on parametric knowledge. We further develop a conformal selective evaluation framework for pairwise LLM judging. It combines a bias-aware uncertainty measure with formal risk control to bound the error rate among accepted judgments at a user-specified level. Finally, we propose a unified judge–retrieve–abstain framework that combines parametric judging, adaptive retrieval, and calibrated abstention. The framework provides finite-sample risk guarantees while preserving useful coverage. These contributions establish a principled approach to reliable and risk-controlled LLM-based evaluation.
SCOPE: Selective Conformal Optimized Pairwise LLM Judging
Abstract
Large language models (LLMs) are increasingly used as scalable judges in pairwise evaluation, but they remain prone to miscalibration and biases. We propose SCOPE (Selective Conformal Optimized Pairwise Evaluation), a framework that calibrates an acceptance threshold so that, under exchangeability, the error rate among non-abstained judgments is at most a user-specified level α. To supply SCOPE with a bias-neutral uncertainty signal, we introduce Bidirectional Preference Entropy (BPE), which queries the judge under both response positions and converts the order-averaged preference probability into an entropy-based score. Across various pairwise judging benchmarks, BPE outperforms standard confidence proxies in calibration and discrimination, while SCOPE consistently satisfies the target risk bound (empirical FDR ≈ 0.097–0.099 at α = 0.10) and retains substantial coverage. Compared to vanilla baselines, SCOPE accepts up to 2.4× more judgments under the same risk constraint, demonstrating that BPE enables reliable and high-coverage LLM-based evaluation.
SAGE: A Search-AuGmented Evaluation of Large Language Models on Free-Form QA
Abstract
As Large Language Models (LLMs) become increasingly used for question-answering (QA), relying on static, pre-annotated references for evaluation poses significant challenges in cost, scalability, and completeness. Meanwhile, using LLMs themselves as evaluators without external grounding remains unreliable for objective tasks, as they systematically over-accept incorrect answers, fabricate supporting rationales, and degrade sharply on questions that fall outside their training data. We propose Search-AuGmented Evaluation (SAGE), a framework to assess LLM outputs without fixed ground-truth answers. Unlike conventional metrics that compare to static references or depend solely on LLM-as-a-judge knowledge, SAGE acts as an agent that actively retrieves and synthesizes external evidence. It iteratively generates web queries, collects information, summarizes findings, and refines subsequent searches through reflection. By reducing dependence on static reference-driven evaluation protocols, SAGE offers a scalable and adaptive alternative for evaluating the factuality of LLMs. Experimental results on multiple free-form QA benchmarks show that SAGE achieves substantial to perfect agreement with human evaluations.
CLEV: LLM-Based Evaluation Through Lightweight Efficient Voting for Free-Form Question-Answering
Abstract
Evaluating free-form Question-Answering (QA) remains a challenge due to its diverse and open-ended nature. Traditional automatic metrics fail to capture semantic equivalence or accommodate the variability of open-ended responses. Leveraging Large Language Models (LLMs) as evaluators offers a promising alternative due to their strong language understanding and instruction-following capabilities. We propose the Consensus via Lightweight Efficient Voting (CLEV), which employs two primary LLMs as judges and engages a third judge only in cases of disagreement. This approach prioritizes evaluation reliability while reducing unnecessary computational demands. Through experiments, including human evaluation, we demonstrate CLEV's ability to provide consistent, scalable, and resource-efficient assessments, establishing it as a robust framework for evaluating LLMs on free-form QA.
Reference-Guided Verdict: LLMs-as-Judges in Automatic Evaluation of Free-Form QA
Abstract
The emergence of Large Language Models (LLMs) as chat assistants capable of generating human-like conversations has amplified the need for robust evaluation methods, particularly for open-ended tasks. Conventional metrics such as EM and F1, while useful, are inadequate for capturing the full semantics and contextual depth of such generative outputs. We propose a reference-guided verdict method that automates the evaluation process by leveraging multiple LLMs as judges. Through experiments on free-form question-answering tasks, we demonstrate that combining multiple models improves the reliability and accuracy of evaluations, especially in tasks where a single model may struggle. The results indicate a strong correlation with human evaluations, establishing the proposed method as a reliable alternative to traditional metrics.
Judge, Retrieve, or Abstain: Uncertainty-Guided LLM Judging with Provable Risk Guarantees
Abstract
Using LLMs as judges has become standard practice for evaluating model outputs at scale. This is particularly common for subjective, open-ended tasks such as assessing helpfulness or alignment, where no single reference answer exists. However, objective tasks introduce a distinct reliability challenge for reference-free LLM judging. In the absence of a reference answer, the judge evaluates factual correctness either through its parametric knowledge or tool augmentation. While parametric knowledge enables efficient evaluation, it may suffer from hallucinations or lack of evidence. Conversely, tool augmentation can provide additional evidence but introduces extra computational cost and yet requires an appropriate mechanism to determine when and how that evidence should be used reliably. More importantly, neither approach alone provides formal control over the risk of accepted verdicts, making it difficult to guarantee a desired level of reliability. We propose a risk-controlled framework that calibrates uncertainty thresholds on a held-out set so that the false discovery rate among accepted verdicts stays below a user-specified level~$\alpha$ with high probability, using finite-sample Clopper--Pearson intervals. When the parametric mode is not confident, the instance is routed to a retrieval-augmented mode, where the judge gathers web evidence and re-evaluates under a second calibrated threshold. The finite-sample guarantee carries over to this two-threshold routing without additional assumptions. Across open-domain QA benchmarks and judges of varying scale, the framework holds the target error rate while achieving substantially higher coverage than single-mode baselines.
