LLM judges are increasingly common among AI teams due their ability to automate decisions that require complex reasoning and analysis. Pairing their reasoning ability with calibrated confidence scores unlocks entirely new ways to work with AI. For one, active learning enhanced prompt optimization uses low confidence decisions to curate a golden set, allowing judges to learn human expertise with lower annotation effort. Additionally, safety classifiers for agents and chatbots can use confidence...