# Design Reliable A/B Tests

Teams risk inconclusive experiments when hypotheses, metrics, and sample sizes are unclear. This skill structures A/B tests so decisions use reliable evidence.

## Install

```bash
npx skillstore add coreyhaines31/ab-test-setup
```

## Metadata

- Status: approved
- Slug: coreyhaines31-ab-test-setup
- Skillstore revision: r1
- Version status: missing
- Tree hash: 97fa36f0e7854d1456e6d354cc4730ffbf0369fbf76c65ecb304e771b01dfebf
- Author: coreyhaines31
- GitHub username: coreyhaines31
- License: MIT
- Repository: https://github.com/coreyhaines31/marketingskills/tree/main/skills/ab-test-setup
- Ref: d11af420e07cd866fe39cfa937efef3223896c06
- Supported tools: Claude, Codex, Claude Code
- Audit status: complete
- Agent install advisory: allowed
- Manual install advisory: allowed
- Artifact signature: available
- Audit attestation: unavailable
- Human verification: not\_verified
- Risk factors: external\_commands, network
- Quality score: 77
- Quality tier: bronze
- Public page: https://skillstore.pages.dev/skills/coreyhaines31-ab-test-setup
- Manifest: https://skillstore.pages.dev/api/skills/coreyhaines31-ab-test-setup/manifest

## Capabilities

- Builds hypotheses using observation, expected effect, audience, and success metrics.
- Selects A/B, A/B/n, multivariate, or split URL test formats.
- Estimates sample size and duration from baseline rate, MDE, traffic, power, and significance.
- Defines primary, secondary, and guardrail metrics for common test types.
- Creates test plans with variants, traffic allocation, implementation approach, and analysis criteria.
- Summarizes results with decisions, actions, learnings, and next tests.

## Use Cases

- Plan a Landing Page Test: Create a clear test plan for headline, CTA, layout, or social proof changes.
- Prioritize Product Experiments: Turn product ideas into measurable experiments with variants, metrics, and decision rules.
- Review Experiment Results: Assess significance, practical impact, guardrails, and next actions after a test ends.

## Prompt Templates

### Create a Basic Test Plan

```
Help me plan an A/B test for [page or flow]. Current conversion is [rate], daily traffic is [number], and the proposed change is [change].
```

### Refine a Hypothesis and Metrics

```
Review this test idea: [idea]. Turn it into a clear hypothesis and define primary, secondary, and guardrail metrics.
```

### Estimate Sample Size and Duration

```
Design the sample size approach for a test with [baseline rate], [MDE], [daily traffic], [variants], 95% confidence, and 80% power.
```

### Build an Experiment Program Brief

```
Create a complete experiment brief for [business goal]. Include hypothesis, variants, allocation, QA, analysis plan, risks, decision rules, and learning repository notes.
```

## Limitations

- Does not install analytics tracking or write production experiment code.
- Requires reliable baseline conversion rates and traffic estimates from the user.
- Sample size references are approximate and should be verified with a calculator or statistics tool.
- Cannot guarantee statistical validity if tests stop early or tracking data is incomplete.

## Best Practices

- Predefine the hypothesis, sample size, duration, and decision criteria before launch.
- Use one primary metric and guardrail metrics tied to revenue, retention, or user quality.
- Keep variants stable during the test and document external factors.

## Anti Patterns

- Stopping a test early because the current result looks significant.
- Testing multiple unrelated changes without enough traffic for reliable attribution.
- Calling a winner from segment data that was not planned before launch.

## Security Audit

- Audited at: 2026-07-06T06:55:55.23\+00:00
- Summary: The flagged external command findings are Markdown code fences used for templates, formulas, and output examples. The hardcoded URLs are public A/B testing calculator references, and the reconnaissance findings are ordinary experimentation guidance.

## Stats

- Views: 139
- Downloads: 7
- Favorites: 0
- Popularity score: 0