# Detect LLM Prompt Regressions with Promptfoo

Prompt changes can silently reduce output quality. This skill builds golden datasets, runs Promptfoo evaluations, and summarizes failures and recent trends.

## Install

```bash
npx skillstore add 7alexhale5-rgb/regression-test
```

## Metadata

- Status: approved
- Slug: 7alexhale5-rgb-regression-test
- Skillstore revision: r1
- Version status: missing
- Tree hash: 243c3dd590a95db2cbb6a19990dea2e34348d3536a863981205d652da635341f
- Author: 7alexhale5-rgb
- GitHub username: 7alexhale5-rgb
- License: MIT
- Repository: https://github.com/7alexhale5-rgb/alexhale-skills/tree/e0f44c48adcc2ac8123a4a8dec2c02503d1c5945/skills/regression-test
- Ref: cabd63e41a4ad4b66103abc7e50c525e4710fb64
- Supported tools: Claude, Codex, Claude Code
- Audit status: complete
- Agent install advisory: allowed
- Manual install advisory: allowed
- Artifact signature: available
- Audit attestation: unavailable
- Human verification: not\_verified
- Risk factors: external\_commands, network, filesystem
- Quality score: 77
- Quality tier: bronze
- Public page: https://skillstore.pages.dev/skills/7alexhale5-rgb-regression-test
- Manifest: https://skillstore.pages.dev/api/skills/7alexhale5-rgb-regression-test/manifest

## Capabilities

- Initializes a Promptfoo directory, golden dataset folder, configuration file, and results ignore rule.
- Captures named test cases with system prompts, user inputs, expected behavior, and optional variables.
- Runs Promptfoo evaluations without cache and stores timestamped local result files.
- Reports total, passed, and failed cases with assertion details and duration.
- Summarizes pass rates and new failures across the five most recent evaluations.

## Use Cases

- Validate prompt changes: Compare updated prompts against recorded expectations before merging code changes.
- Check model migrations: Rerun a stable golden dataset after changing a model or provider version.
- Enforce release quality: Run prompt evaluations in continuous integration and stop releases when assertions fail.

## Prompt Templates

### Initialize a test suite

```
Initialize Promptfoo regression testing in this repository. Show the files created and explain any prerequisites that are missing.
```

### Add a golden case

```
Add a golden test named {name}. Ask for the system prompt, user input, expected behavior, and optional variables before writing it.
```

### Run regression tests

```
Run all golden prompt tests. Summarize the pass rate, duration, and each failed assertion with expected and actual behavior.
```

### Analyze release trends

```
Review the last five evaluations, identify new failures, compare pass-rate trends, and recommend quality gates for {release_workflow}.
```

## Limitations

- Requires the Promptfoo CLI, provider access, and any credentials required by the configured model.
- The generated configuration targets one Anthropic model and uses general rubric assertions.
- Trend reports cover only the five most recent local evaluation files.
- Shell examples assume a Unix environment with common command-line utilities.

## Best Practices

- Review every golden case before committing it, and exclude prompts containing secrets or sensitive data.
- Describe expected behavior precisely enough that rubric assertions produce consistent results.
- Run the suite after prompt or model changes and before every release.

## Anti Patterns

- Do not treat one current model response as a permanent definition of correct behavior.
- Do not commit API keys, private prompts, personal data, or confidential outputs in golden files.
- Do not ignore unstable rubric failures without reviewing the assertion and provider behavior.

## Security Audit

- Audited at: 2026-08-21T01:02:35.961\+00:00
- Summary: All 38 static findings are false positives caused by Markdown syntax, bounded shell examples, or a Promptfoo message template. No malicious intent, prompt injection, credential handling, or unauthorized network behavior was found.

## Stats

- Views: 0
- Downloads: 2
- Favorites: 1
- Popularity score: 0