# Build Reliable Agent Evaluations

Agent quality is difficult to measure because runs vary and valid solutions follow different paths. This skill builds practical rubrics, test sets, and evaluation pipelines.

## Install

```bash
npx skillstore add sickn33/evaluation
```

## Metadata

- Status: approved
- Slug: sickn33-evaluation
- Skillstore revision: r2
- Version status: missing
- Tree hash: fc3af5e56bf54dfa7fc52c62b77e4c794641e72529663d9cdad7ac46ee7564bf
- Author: sickn33
- GitHub username: sickn33
- License: MIT
- Repository: https://github.com/sickn33/antigravity-awesome-skills/tree/main/skills/evaluation
- Ref: 88a8e9a07f4c54ab105c1c41b6267c287146b07b
- Supported tools: Claude, Codex, Claude Code
- Audit status: complete
- Agent install advisory: allowed
- Manual install advisory: allowed
- Artifact signature: available
- Audit attestation: unavailable
- Human verification: not\_verified
- Risk factors: external\_commands, network
- Quality score: 78
- Quality tier: bronze
- Public page: https://skillstore.pages.dev/skills/sickn33-evaluation
- Manifest: https://skillstore.pages.dev/api/skills/sickn33-evaluation/manifest

## Capabilities

- Defines multi-dimensional rubrics for accuracy, completeness, citations, source quality, and tool efficiency.
- Structures test sets across simple, medium, complex, and extended agent tasks.
- Compares context strategies using quality scores, token usage, and efficiency metrics.
- Combines automated LLM judgments with targeted human review.
- Designs continuous evaluation pipelines for regression detection and production monitoring.
- Establishes weighted scores, baselines, thresholds, and trend tracking.

## Use Cases

- Validate a New Agent: Create a representative test set and weighted quality rubric before release.
- Compare Context Strategies: Measure quality, token cost, and tool efficiency across competing context configurations.
- Monitor Production Quality: Define sampled evaluations, regression alerts, and human review checkpoints for deployed agents.

## Prompt Templates

### Create a Basic Rubric

```
Create an evaluation rubric for [agent task]. Include accuracy, completeness, and efficiency with clear scoring levels and a passing threshold.
```

### Design a Test Set

```
Design a test set for [agent system]. Cover simple through very complex tasks, realistic usage, known edge cases, and expected outcomes.
```

### Compare Agent Configurations

```
Build an experiment comparing [configuration A] and [configuration B]. Define controlled variables, quality metrics, token metrics, sample size, and decision criteria.
```

### Design Continuous Evaluation

```
Design a continuous evaluation pipeline for [production agent]. Include sampling, automated judging, human review, baselines, alerts, privacy controls, and regression reporting.
```

## Limitations

- It provides methodology and examples, but does not include an executable evaluation package.
- Rubric weights and passing thresholds require adaptation to each product and risk profile.
- LLM judges can introduce bias and require human validation.
- Production monitoring still requires environment-specific infrastructure and privacy controls.

## Best Practices

- Evaluate end outcomes while allowing multiple valid execution paths.
- Establish a baseline and use realistic test cases before comparing changes.
- Combine automated scoring with systematic human review for edge cases.

## Anti Patterns

- Do not judge quality with one metric or one successful example.
- Do not overfit tests to a specific sequence of agent actions.
- Do not deploy automated judges without checking bias and agreement with humans.

## Security Audit

- Audited at: 2026-07-23T23:59:44.577\+00:00
- Summary: All nine static findings are false positives caused by Markdown fences, descriptive prose, integration names, and a source metadata URL. The skill is documentation-only and contains no executable commands, network requests, reconnaissance behavior, or prompt injection.

## Stats

- Views: 147
- Downloads: 11
- Favorites: 0
- Popularity score: 0