Design visual, review-gated agent loops for Claude Code before you run them.
-
Updated
Jul 7, 2026 - Python
#
llm-as-judge
Here are 391 public repositories matching this topic...
Pairwise LLM judges (A/B/tie): budget-aware multi-turn packing, position-bias correction, pseudo-label distillation. Generalized from the 4th-place (gold) solution to Kaggle LMSYS Chatbot Arena.
nlp kaggle-competition lora preference-learning kaggle-solution gold-medal llm rlhf reward-model llm-as-judge chatbot-arena
-
Updated
Jul 11, 2026 - Python
agentevals is a framework-agnostic evaluations solution based on OpenTelemetry traces
-
Updated
Jul 13, 2026 - Python
Academic Personal AI Infrastructure
-
Updated
Jun 16, 2026 - Python
Lumen — learner-owned AI education platform. Tell the AI what you want to learn: it builds you a private course in ~a minute, tutors you with course-scoped RAG + citations, and lets you share, clone & remix via a moderated catalog. BYOK, custom no-LangChain multi-agent orchestrator, golden evals in CI, MCP server. Live demo + public /eval.
python docker postgres typescript mcp nextjs e-learning celery observability rag fastapi llm byok pgvector evals llm-as-judge ai-tutor agentic-ai
-
Updated
Jun 7, 2026 - Python
LLM and agent evaluation for Java & Kotlin. Runs in JUnit and CI. Spring AI, LangChain4j, Koog, Embabel, and any LLM client.
kotlin java evaluation junit evaluation-metrics evaluation-framework rag llm retrieval-augmented-generation langchain4j llm-evaluation rag-evaluation spring-ai llm-as-judge agentic-ai agent-evaluation spring-ai-alibaba koog embabel spring-ai-evaluation
-
Updated
Jul 20, 2026 - Java
AI evaluators CLI for your AI apps and Agents - Dynatrace AI Observability
-
Updated
Jul 21, 2026 - TypeScript
Causal Judge Evaluation: calibrate LLM-as-judge scores against oracle labels with valid uncertainty.
calibration uncertainty-quantification causal-inference evaluation-framework semiparametric-estimation ai-evaluation llm-evaluation llm-as-judge
-
Updated
Jul 20, 2026 - Python
A dependency-free Python genetic-algorithm engine with pluggable fitness criteria — the reusable search core for an LLM/AI autoresearcher.
python open-source machine-learning ai optimization genetic-algorithm santander search-algorithm evolutionary-computation ai-agents llm llm-as-judge apache-2-0 autoresearch
-
Updated
Jul 16, 2026 - Python
HonestyMeter: An NLP-powered framework for evaluating objectivity and bias in media content, detecting manipulative techniques, and providing actionable feedback.
-
Updated
Feb 28, 2026 - TypeScript
Code and data for Koo et al's ACL 2024 paper "Benchmarking Cognitive Biases in Large Language Models as Evaluators"
nlp evaluation bias bias-detection llm llms llm-evaluation llms-benchmarking llm-as-judge llm-as-a-judge llm-as-evaluator
-
Updated
Feb 16, 2024 - Jupyter Notebook
Evidence-first multi-agent debate skill: get a second opinion by pitting Codex × Claude Code (or GLM/DeepSeek/Qwen) to independently review, red-team & judge high-stakes code and architecture decisions.
opencode multi-agent code-review red-team codex ai-agents rag architecture-review openai-codex prompt-engineering llm-as-judge claude-code claude-code-skill agent-skill openclaw hermes-agent agent-arena evidence-checking deliberative-analysis
-
Updated
Jun 23, 2026
Diagnosing policy grounding in LLM judges via causal erasure. Companion code to the AIES 2026 submission READ.
-
Updated
Jul 2, 2026 - Python
Toward Expert-Level Medical Text Validation with Language Models
-
Updated
Oct 23, 2025 - Python
Production-grade Playwright + TypeScript QA framework with AI-powered testing, LLM-as-Judge evaluation, MCP server, 7 CLI agents, security fuzzing, CI/CD pipelines, Jira sync, and Slack reporting — zero-config, plug-and-play.
typescript mutation-testing mcp test-automation ci-cd allure-report api-testing owasp-zap security-testing page-object-model stryker e2e-testing github-actions qa-automation playwright jira-integration ai-testing aws-bedrock llm-as-judge
-
Updated
Jun 23, 2026 - TypeScript
Open WebUI Logic Tree Multi-Model LLM Project - Advanced AI reasoning with Sakana AI's AB-MCTS and sophisticated multi-model collaboration
docker machine-learning ai pipeline multi-model monte-carlo-tree-search research-software sakana llm reasoning-engine open-webui llm-as-judge advanced-reasoning open-webui-tools ab-mcts
-
Updated
Jul 4, 2026 - Python
Observe an agent run and GroundEval drafts the policy and diagram for you, no hand-written policy required, then scores what it checked, what it skipped, and what it wasn't allowed to touch.
benchmarking evaluation-framework ai-agents test-agents llm-evaluation llm-as-judge llm-as-a-judge agent-evals agent-evaluation agent-harness agent-loop agent-eval debug-agents
-
Updated
Jul 17, 2026 - Python
University for AI agents. 92 courses, 4400+ scenarios, any model via OpenRouter. Auto-training loops generate per-model SKILL.md documents. Works with Claude Code, OpenClaw, Cursor, Windsurf. No fine-tuning required.
training cli typescript mcp evaluation openai knowledge-distillation ai-agents claude curriculum-learning llm ai-testing openrouter llm-as-judge model-context-protocol agent-skills claude-code agent-benchmark openclaw skill-generation
-
Updated
May 2, 2026 - TypeScript
Agentic Extensible Test Automation Framework
docker bdd selenium atdd requests ui-automation automation-framework paramiko chaos-engineering multi-layer-architecture playwright llm-as-judge
-
Updated
Jun 1, 2026 - Python
Practical LLM evaluation for teams that ship to production. Deterministic + LLM-as-judge evaluators, dataset support, CI/CD integration.
python mlops ai-evaluation prompt-engineering llmops llm-eval evals llm-evaluation rag-evaluation hallucination-detection llm-as-judge agent-evaluation
-
Updated
Jul 13, 2026 - Python
Improve this page
Add a description, image, and links to the llm-as-judge topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the llm-as-judge topic, visit your repo's landing page and select "manage topics."