# Design Rigorous A/B Tests

Weak experiment design creates misleading results and wasted traffic. This skill applies hypothesis, metrics, sample size, tracking, execution, and analysis gates before decisions.

## Install

```bash
npx skillstore add sickn33/ab-test-setup
```

## Metadata

- Status: approved
- Slug: sickn33-ab-test-setup
- Skillstore revision: r2
- Version status: missing
- Tree hash: d95b780bb2f5785a7fa79d9b1c959a16bcfc7c4b89200459bbfaf5e3fe829e76
- Author: sickn33
- GitHub username: sickn33
- License: MIT
- Repository: https://github.com/sickn33/antigravity-awesome-skills/tree/main/skills/ab-test-setup
- Ref: ebdfe608f5de2b66ff37ab4af12af8ac4f5e8006
- Supported tools: Claude, Codex, Claude Code
- Audit status: complete
- Agent install advisory: allowed
- Manual install advisory: allowed
- Artifact signature: available
- Audit attestation: unavailable
- Human verification: not\_verified
- Quality score: 78
- Quality tier: bronze
- Public page: https://skillstore.pages.dev/skills/sickn33-ab-test-setup
- Manifest: https://skillstore.pages.dev/api/skills/sickn33-ab-test-setup/manifest

## Capabilities

- Checks hypotheses for evidence, one specific change, a defined audience, an expected direction, and measurable success criteria.
- Recommends A/B, A/B/n, multivariate, or split URL designs based on the proposed change and available traffic.
- Defines one primary metric, supporting secondary metrics, and safety guardrails before launch.
- Guides sample planning with baseline rate, minimum detectable effect, significance level, power, and expected duration.
- Provides checks for event delivery, variant attribution, duplicate events, randomization balance, dashboards, and alerts.
- Applies readiness, monitoring, analysis, documentation, and refusal rules across the experiment lifecycle.

## Use Cases

- Validate a Product Experiment: Turn a product idea into a locked hypothesis, measurable outcome, guardrails, and launch checklist.
- Plan Reliable Measurement: Define metrics, estimate sample needs, and verify event collection before exposing users to variants.
- Review Experiment Readiness: Identify missing assumptions, tracking risks, and stopping rules before implementation or launch approval.

## Prompt Templates

### Draft a Hypothesis

```
Help me turn this idea into a valid A/B test hypothesis: [idea]. Ask for the audience, metric, direction, baseline, and MDE.
```

### Review Hypothesis Quality

```
Review this hypothesis and stop if required details are missing: [hypothesis]. Identify assumptions, one primary metric, secondary metrics, and guardrails.
```

### Plan Sample and Duration

```
Plan an experiment using baseline [rate], MDE [effect], significance [level], power [target], and daily traffic [volume]. State assumptions and estimate sample size and duration.
```

### Run a Readiness Review

```
Audit this complete experiment plan: [plan]. Check design choice, metrics, sample assumptions, tracking, randomization, guardrails, stopping rules, analysis, and documentation. Block unresolved requirements.
```

## Limitations

- Does not connect to analytics platforms, assignment systems, dashboards, or production environments.
- Cannot validate traffic, event quality, randomization, or business assumptions without reliable user-provided evidence.
- Sample estimates depend on accurate baselines, effect sizes, significance levels, power targets, and unit definitions.
- Does not replace statistical, legal, ethical, privacy, or domain-specific review for high-impact experiments.

## Best Practices

- Lock the hypothesis, audience, primary metric, expected direction, and MDE before designing variants.
- Verify tracking and randomization with raw records before launch, then monitor technical health without peeking at outcomes.
- Document assumptions, external factors, achieved sample, results, guardrails, decisions, and follow-up learning.

## Anti Patterns

- Do not change variants, traffic sources, success criteria, or the primary metric after launch.
- Do not stop early because interim results appear favorable or unfavorable.
- Do not ship a primary-metric win when a predefined guardrail fails.

## Security Audit

- Audited at: 2026-07-23T19:57:16.315\+00:00
- Summary: All five static findings are false positives caused by ordinary experiment-design language in SKILL.md. The skill contains procedural A/B testing guidance and no reconnaissance, prompt injection, command execution, or data-exfiltration intent.

## Stats

- Views: 133
- Downloads: 10
- Favorites: 0
- Popularity score: 0