# Build Reliable LLM Evaluation Systems

LLM quality reviews often drift because judges use vague criteria and exhibit systematic bias. This skill provides calibrated rubrics, comparison methods, and validation metrics.

## Install

```bash
npx skillstore add muratcankoylan/advanced-evaluation
```

## Metadata

- Status: approved
- Slug: muratcankoylan-advanced-evaluation
- Skillstore revision: r2
- Version status: missing
- Tree hash: 91c7180c92732df01e16fca1d52c15842d9bf8bd8323517b4e4a5789cc554839
- Author: muratcankoylan
- GitHub username: muratcankoylan
- License: MIT
- Repository: https://github.com/muratcankoylan/Agent-Skills-for-Context-Engineering/tree/main/skills/advanced-evaluation
- Ref: 02be9409c79ca1183f7844009c14d9df684d0cf9
- Supported tools: Claude, Codex, Claude Code
- Audit status: complete
- Agent install advisory: allowed
- Manual install advisory: allowed
- Artifact signature: available
- Audit attestation: unavailable
- Human verification: not\_verified
- Risk factors: external\_commands, network
- Quality score: 73
- Public page: https://skillstore.pages.dev/skills/muratcankoylan-advanced-evaluation
- Manifest: https://skillstore.pages.dev/api/skills/muratcankoylan-advanced-evaluation/manifest

## Capabilities

- Selects direct scoring or pairwise comparison based on objective ground truth and subjective preference.
- Creates evidence-based scoring rubrics with level boundaries, examples, edge cases, and strictness settings.
- Mitigates position, length, verbosity, authority, and self-enhancement bias in LLM judges.
- Calibrates confidence using position consistency, evidence strength, and multi-judge agreement.
- Chooses validation metrics for binary, ordinal, pairwise, and multi-label evaluations.
- Provides local Python examples for direct scoring, position swapping, and rubric generation.

## Use Cases

- Compare Model Responses: Design a position-swapped pairwise test that selects stronger responses while measuring consistency and confidence.
- Standardize Quality Reviews: Create domain-specific rubrics that help human and LLM evaluators apply the same scoring standards.
- Validate Automated Judges: Select agreement and ranking metrics that reveal systematic differences between automated scores and expert labels.

## Prompt Templates

### Choose an Evaluation Method

```
Review this evaluation goal: [goal]. Identify whether direct scoring or pairwise comparison fits best. Explain the decision and list required inputs.
```

### Create a Scoring Rubric

```
Create a [scale] rubric for [criterion] in [domain]. Define each level, observable evidence, edge cases, and balanced scoring guidance.
```

### Design a Bias-Controlled Comparison

```
Design a pairwise evaluation for [responses] using [criteria]. Include position swapping, label remapping, tie handling, confidence rules, and length-neutral instructions.
```

### Audit an Evaluation Pipeline

```
Audit this evaluation pipeline: [pipeline]. Check rubric calibration, prompt injection resistance, bias controls, human agreement, confidence calibration, and escalation thresholds.
```

## Limitations

- It does not call model APIs or provide a production evaluation service.
- Its example script uses fixed demonstration results instead of executing real judge requests.
- Automated scores still require validation against representative human judgments.
- Prompt templates need additional isolation before evaluating adversarial or untrusted responses.

## Best Practices

- Require evidence before every score so reviewers can audit each judgment.
- Swap response positions and remap labels before calculating pairwise agreement.
- Validate automated judgments against representative human labels and monitor criterion-level disagreement.

## Anti Patterns

- Do not use broad criteria that combine several unrelated quality dimensions.
- Do not trust a single pairwise pass because position bias can determine the winner.
- Do not deploy changed judge prompts without regression tests and fresh calibration.

## Security Audit

- Audited at: 2026-08-09T09:23:16.086\+00:00
- Summary: All 44 static alerts are false positives caused by Markdown formatting, ordinary collection methods, evaluation terminology, and research links. The local example script only calculates and prints demonstration results. However, evaluator prompts interpolate untrusted candidate responses without explicit prompt-injection isolation, which could permit scoring manipulation.

## Stats

- Views: 391
- Downloads: 12
- Favorites: 0
- Popularity score: 0