llm-evaluation
Evaluate LLM Applications with Measurable Quality
LLM teams need evidence before changing prompts, models, or retrieval flows. This skill guides automated metrics, judge reviews, human labels, A/B tests, and regression checks.
Install with my Agent
Copy this request to your Agent. It includes the canonical Skill page and manifest.
Review the Skillstore skill "llm-evaluation" from https://skillstore.io/skills/wshobson-llm-evaluation.md and its manifest at https://skillstore.io/api/skills/wshobson-llm-evaluation/manifest. Verify the artifact. You may proceed after verification, subject to the environment's own policy.Your Agent should still show its plan and request any confirmation required by the security policy.
Agent-readable resources
Use these links when an AI agent, crawler, or script needs clean context instead of reading the full page.
Test it
Using "llm-evaluation". Evaluate a customer support chatbot before launch.
Expected outcome:
- A metric set covering correctness, relevance, tone, safety, and resolution quality.
- A human review plan with rating dimensions and issue labels.
- A release gate based on baseline comparison and regression thresholds.
Using "llm-evaluation". Compare two prompts for a RAG search assistant.
Expected outcome:
A pairwise plan with groundedness scoring, retrieval quality checks, judge calibration, and a regression gate.
Using "llm-evaluation". Investigate a drop in summarization quality.
Expected outcome:
- A baseline comparison across ROUGE, semantic similarity, and human clarity ratings.
- A failure analysis plan for hallucinations, missing facts, and style drift.
Security Audit
SafeThe static external-command findings are markdown Python code fences in SKILL.md, not executable Ruby or shell backtick use. The certificate/key and network reconnaissance findings are false positives from ordinary evaluation text and resource names. No prompt injection attempt, data exfiltration intent, or malicious behavior was found.
Risk Factors
⚙️ External commands (23)
Share & cite this report
Share the versioned assessment report, neutral badge, embed card, and citations. Skillstore reports evidence without deciding whether this Skill is safe.
Copy report link
https://skillstore.io/skills/wshobson-llm-evaluation/audits/7?utm_source=security_passport&utm_medium=share&utm_campaign=versioned_reportMarkdown badge
[](https://skillstore.io/skills/wshobson-llm-evaluation?utm_source=security_passport_badge)HTML badge
<a href="https://skillstore.io/skills/wshobson-llm-evaluation?utm_source=security_passport_badge"><img src="https://skillstore.io/badges/skills/wshobson-llm-evaluation/security.svg" alt="Skillstore security assessment" loading="lazy"></a>Embed card
<iframe src="https://skillstore.io/embed/skills/wshobson-llm-evaluation.html" title="Skillstore Security Assessment" sandbox="allow-popups allow-popups-to-escape-sandbox" loading="lazy" referrerpolicy="no-referrer" width="420" height="180"></iframe>Academic citations (APA · BibTeX · CFF)
APA citation
wshobson. (2026). llm-evaluation security audit report (audit version 7) [Author version unspecified]. Skillstore. https://skillstore.io/skills/wshobson-llm-evaluation/audits/7BibTeX citation
@techreport{wshobson-wshobson-llm-evaluation-2026,
author = {wshobson},
title = {llm-evaluation security audit report (audit version 7)},
institution = {Skillstore},
year = {2026},
number = {7},
url = {https://skillstore.io/skills/wshobson-llm-evaluation/audits/7},
note = {Author version unspecified}
}CITATION.cff
cff-version: 1.2.0
message: "If you use this Skill, cite its author and this versioned security audit report."
title: "llm-evaluation security audit report (audit version 7)"
version: "unspecified"
type: report
authors:
- name: "wshobson"
date-released: "2026-07-07"
url: "https://skillstore.io/skills/wshobson-llm-evaluation/audits/7"
identifiers:
- type: other
value: "skillstore:wshobson-llm-evaluation:audit:7"
description: "Skillstore immutable audit report identifier"
Compare variants
2 installable variantsEach author remains a separate installable skill. The recommended variant is ranked by Skillstore evidence.
Why this variant is first
wshobson-llm-evaluation
2026-09-09
sickn33-llm-evaluation
2026-09-09
Skillstore Score
Why this score Evidence Confidence: HighWhat You Can Build
Compare prompt changes
Measure whether a prompt revision improves accuracy, helpfulness, and clarity before release.
Validate product quality
Define launch gates that combine automated scores, human review, and baseline comparisons.
Catch model regressions
Track evaluation scores over time and identify quality drops after model, prompt, or data changes.
Try These Prompts
Help me define an evaluation plan for my LLM feature. Include metrics, test data, baselines, and human review steps.
Recommend automated and human evaluation metrics for this LLM task: [describe task]. Explain when each metric is useful and where it can fail.
Design a regression evaluation workflow for my existing LLM application. Include baseline storage, thresholds, failure triage, and release gates.
Create an advanced evaluation protocol comparing two model versions with LLM-as-judge, human calibration, A/B testing, and statistical analysis.
Best Practices
- Use representative examples that reflect production traffic and known failure modes.
- Combine automated metrics with calibrated human review for subjective quality dimensions.
- Keep baselines and thresholds versioned so prompt and model changes are comparable.
Avoid
- Relying on one metric for every task and release decision.
- Testing only easy examples or samples from training data.
- Treating LLM judge scores as objective without calibration or spot checks.
Frequently Asked Questions
Does this skill run evaluations automatically?
Can I use it with Claude, Codex, and Claude Code?
What evaluation methods does it cover?
Does it include datasets or runnable scripts?
Is LLM-as-judge enough for production decisions?
Who benefits most from this skill?
Developer Details
Author
wshobsonLicense
MIT
Skillstore revision
r1
Version notice
The author did not declare a version.
Repository
https://github.com/wshobson/agents/tree/main/plugins/llm-application-dev/skills/llm-evaluationRef
36e07d5e13068e5be64447e8f20b427cf2cbd21a
Maintenance freshness
7/18/2026
Usage
16 downloads · 297 views
File structure
📄 SKILL.md