# Design a reliable email experiment

Email teams can make costly decisions from underpowered or poorly controlled tests. This skill creates a structured experiment plan and interprets results using user-provided campaign data.

## Install

```bash
npx skillstore add aaron-he-zhu/send-experiment-designer
```

## Metadata

- Status: approved
- Slug: aaron-he-zhu-send-experiment-designer
- Version: 19.0.0
- Author version: 19.0.0
- Skillstore revision: r2
- Version status: valid
- Tree hash: 00b40ce150c5e202c36aeb702d9cc20eba5994544ad864f050bc9ab15769f076
- Author: aaron-he-zhu
- GitHub username: aaron-he-zhu
- License: Apache-2.0
- Repository: https://github.com/aaron-he-zhu/aaron-marketing-skills/tree/main/email/deliver/send-experiment-designer
- Ref: 0715a6e09ea875c8e28cb705ce87cc83045e69c1
- Supported tools: Claude, Codex, Claude Code
- Audit status: complete
- Agent install advisory: allowed
- Manual install advisory: allowed
- Artifact signature: available
- Audit attestation: unavailable
- Human verification: not\_verified
- Risk factors: external\_commands, network, filesystem
- Quality score: 82
- Quality tier: silver
- Public page: https://skillstore.pages.dev/skills/aaron-he-zhu-send-experiment-designer
- Manifest: https://skillstore.pages.dev/api/skills/aaron-he-zhu-send-experiment-designer/manifest

## Capabilities

- Selects A/B, multivariate, send-time, or hold-out test modes from the stated goal.
- Creates falsifiable hypotheses and variant matrices with controls.
- Defines primary metrics, secondary metrics, and deliverability guardrails.
- Plans sample size, minimum detectable effect, power, and estimated duration.
- Interprets supplied variant counts with effect size, uncertainty, and practical-impact flags.
- Returns an undecided outcome when no owner-approved decision rule exists.

## Use Cases

- Plan a subject-line experiment: Define a controlled subject-line A/B test with sample size, guardrails, and a read date.
- Assess a completed campaign test: Interpret delivered, open, click, and conversion counts with statistical and practical effect flags.
- Measure incremental email value: Create a randomized hold-out plan to evaluate conversion or revenue lift from a send.

## Prompt Templates

### Beginner: plan an A/B test

```
Design an email A/B subject-line test. My baseline open rate is 38%, my list has 12,000 people, and I want to detect a 3-point lift.
```

### Intermediate: plan a send-time test

```
Create a send-time test for our weekly newsletter. Baseline open rate is 40%, the list has 20,000 people, and content must stay constant.
```

### Advanced: size a multivariate test

```
Plan a 2 by 2 subject and CTA experiment. Baseline click rate is 2.1%, send volume is 8,000 per week, alpha is .05, and power is .80.
```

### Advanced: read a hold-out result

```
Read this hold-out result using my decision rule. Control: 10,000 delivered and 180 conversions. Variant: 10,000 delivered and 215 conversions. Guardrail unsubscribe rates were 0.12% and 0.13%.
```

## Limitations

- It relies on campaign data, baselines, and business rules supplied by the user.
- It does not write email creative or approve a business action independently.
- It does not replace legal, privacy, deliverability, or statistical expert review.
- The optional local statistics helper must be available and reviewed before command-based calculations.

## Best Practices

- Define one primary metric, a practical-effect threshold, and guardrails before sending.
- Keep one variable isolated per cell and preserve a randomized control group.
- Set the sample size and read date before the test, then avoid unplanned early decisions.

## Anti Patterns

- Changing subject, creative, and send time in one test cell.
- Calling a winner from an underpowered test or an early unplanned look.
- Treating statistical significance as an automatic business decision without an approved rule.

## Security Audit

- Audited at: 2026-07-27T11:44:10.423\+00:00
- Summary: Most static findings are false positives from Markdown formatting, documentation links, metadata URLs, and a statistical decimal. One documented command runs a local statistics helper; its numeric inputs and referenced script should be reviewed before use. No prompt-injection language, credential collection, data-exfiltration intent, or automatic network activity was found in SKILL.md.

## Stats

- Views: 1
- Downloads: 6
- Favorites: 0
- Popularity score: 0