Use an A/B test sample size calculator as part of a complete experiment plan. Provide the current conversion rate, the smallest improvement worth detecting and expected traffic; this Skill combines a Python calculation with hypothesis, metric and stopping-rule guidance so your team can judge whether a proposed test is practical.
A test plan needs more than two page variants. This Skill makes the baseline conversion rate, minimum detectable effect, significance level and statistical power explicit, then connects the calculation to the business question and operating constraints.
Use it before launching a conversion-rate experiment. The bundled calculator follows its documented two-proportion assumptions; it is not a general engine for revenue-per-order, continuous metrics or multivariate tests. You still need a testing platform or implementation to allocate visitors and collect results.
The Python calculator exposes the assumptions behind the sample estimate, including whether the effect is entered as an absolute or relative change. This helps prevent a small wording difference in the target from producing a misunderstood testing commitment.
Hypotheses, metrics, stopping rules and business guardrails sit alongside the sample calculation. Expected traffic helps assess feasibility, while a predefined decision rule reduces the temptation to stop as soon as a favorable result appears.
Baseline conversion, absolute or relative MDE, traffic, allocation, alpha, power, and business guardrails.
Use a Skill-capable agent with file access. Python 3 is required for the bundled Python helpers; inspect their sample schemas before supplying data.
The calculator is for its documented two-proportion assumptions; it is not a general continuous-metric or multivariate test engine and does not launch tests.
Copy this request into Codex or Claude Code. The assistant downloads the original repository at the specified commit, installs the complete files and checks required dependencies.
Install experiment-designer in the app I am using and carry out the file installation, not just explain it.
Original repository: https://github.com/alirezarezvani/claude-skills.git
Commit: 6d2f02ea077eead1bee5411a31bcc6db5a11a70c
Skill folder: product-team/skills/experiment-designer
1. Download this exact source into a new directory. Read SKILL.md and its setup references before running any bundled scripts. Do not substitute another author’s package.
2. For Codex, use ~/.agents/skills/experiment-designer; for Claude Code, use ~/.claude/skills/experiment-designer. Install only for my current app. Install the complete Skill folder, including scripts, references, assets and bundled notices. Keep any root license with the installed folder. Check all destination folders first; if any already exist, report differences before replacing them.
3. Compare the source and installed file lists and hashes. Inspect documented runtimes, dependencies and required connections. Configure only the local dependencies needed for this Skill; use an isolated environment when appropriate. Ask me to complete any account login locally, without putting credentials in chat.
4. Report the source commit, actual installation path, file-check results and any remaining runtime setup. Explain how to invoke experiment-designer in this app. Check discovery separately from file copying; start a new session if needed. Do not run production tasks or claim a successful business outcome from installation alone.This is the unchanged source SKILL.md for the reviewed version. Install the complete package to retain referenced scripts, assets, and instructions. Starter prompts below are EcomAgentTools examples.
---
name: experiment-designer
description: Use when planning product experiments, writing testable hypotheses, estimating sample size, prioritizing tests, or interpreting A/B outcomes with practical statistical rigor.
---
# Experiment Designer
Design, prioritize, and evaluate product experiments with clear hypotheses and defensible decisions.
## When To Use
Use this skill for:
- A/B and multivariate experiment planning
- Hypothesis writing and success criteria definition
- Sample size and minimum detectable effect planning
- Experiment prioritization with ICE scoring
- Reading statistical output for product decisions
## Core Workflow
1. Write hypothesis in If/Then/Because format
- If we change `[intervention]`
- Then `[metric]` will change by `[expected direction/magnitude]`
- Because `[behavioral mechanism]`
2. Define metrics before running test
- Primary metric: single decision metric
- Guardrail metrics: quality/risk protection
- Secondary metrics: diagnostics only
3. Estimate sample size
- Baseline conversion or baseline mean
- Minimum detectable effect (MDE)
- Significance level (alpha) and power
Use:
```bash
python3 scripts/sample_size_calculator.py --baseline-rate 0.12 --mde 0.02 --mde-type absolute
```
4. Prioritize experiments with ICE
- Impact: potential upside
- Confidence: evidence quality
- Ease: cost/speed/complexity
ICE Score = (Impact * Confidence * Ease) / 10
5. Launch with stopping rules
- Decide fixed sample size or fixed duration in advance
- Avoid repeated peeking without proper method
- Monitor guardrails continuously
6. Interpret results
- Statistical significance is not business significance
- Compare point estimate + confidence interval to decision threshold
- Investigate novelty effects and segment heterogeneity
## Hypothesis Quality Checklist
- [ ] Contains explicit intervention and audience
- [ ] Specifies measurable metric change
- [ ] States plausible causal reason
- [ ] Includes expected minimum effect
- [ ] Defines failure condition
## Common Experiment Pitfalls
- Underpowered tests leading to false negatives
- Running too many simultaneous changes without isolation
- Changing targeting or implementation mid-test
- Stopping early on random spikes
- Ignoring sample ratio mismatch and instrumentation drift
- Declaring success from p-value without effect-size context
## Statistical Interpretation Guardrails
- p-value < alpha indicates evidence against null, not guaranteed truth.
- Confidence interval crossing zero/no-effect means uncertain directional claim.
- Wide intervals imply low precision even when significant.
- Use practical significance thresholds tied to business impact.
See:
- `references/experiment-playbook.md`
- `references/statistics-reference.md`
## Tooling
### `scripts/sample_size_calculator.py`
Computes required sample size (per variant and total) from:
- baseline rate
- MDE (absolute or relative)
- significance level (alpha)
- statistical power
Example:
```bash
python3 scripts/sample_size_calculator.py \
--baseline-rate 0.10 \
--mde 0.015 \
--mde-type absolute \
--alpha 0.05 \
--power 0.8
```
Starter prompts for the main use cases—copy and use them directly.
Size a checkout conversion experiment Use for: Size a checkout conversion experiment. Prepare: Baseline conversion, absolute or relative MDE, traffic, allocation, alpha, power, and business guardrails.
Use the installed experiment-designer Skill. Task: Size a checkout conversion experiment. Inputs: [INPUT]. Constraints: [CONSTRAINTS]. Check missing information first. Produce a reviewable result with evidence, assumptions, and next steps. The calculator is for its documented two-proportion assumptions; it is not a general continuous-metric or multivariate test engine and does not launch tests.[INPUT][CONSTRAINTS]Prioritize test ideas with explicit evidence and effort Use for: Prioritize test ideas with explicit evidence and effort. Prepare: Baseline conversion, absolute or relative MDE, traffic, allocation, alpha, power, and business guardrails.
Use the installed experiment-designer Skill. Task: Prioritize test ideas with explicit evidence and effort. Inputs: [INPUT]. Constraints: [CONSTRAINTS]. Check missing information first. Produce a reviewable result with evidence, assumptions, and next steps. The calculator is for its documented two-proportion assumptions; it is not a general continuous-metric or multivariate test engine and does not launch tests.[INPUT][CONSTRAINTS]Write decision criteria and guardrails before launch Use for: Write decision criteria and guardrails before launch. Prepare: Baseline conversion, absolute or relative MDE, traffic, allocation, alpha, power, and business guardrails.
Use the installed experiment-designer Skill. Task: Write decision criteria and guardrails before launch. Inputs: [INPUT]. Constraints: [CONSTRAINTS]. Check missing information first. Produce a reviewable result with evidence, assumptions, and next steps. The calculator is for its documented two-proportion assumptions; it is not a general continuous-metric or multivariate test engine and does not launch tests.[INPUT][CONSTRAINTS]The merchant plans to test shipping information beside the purchase button, from a 2.5% baseline with a 0.5-percentage-point minimum effect. Inputs include daily eligible traffic and simulated outcomes.
Case input files: business-inputs.zip
Use the experiment-designer Skill.
Design a controlled test of the proposed product-page change using our traffic and purchase baseline. Define the unit of randomization, seven-day purchase metric, guardrails, sample size, enrollment duration and analysis rule. Then analyze the supplied experiment dataset, including confidence intervals and costs, against the decision thresholds.
The attached business materials are:
- inputs/merchant.json
- inputs/operating-brief.json
- inputs/synthetic_outcome.json
Deliver a complete result.md report that the operations team can read, plus any finished files needed for this task. Keep supporting calculations and evidence with the deliverables. Write the report in English; use the requested target language for customer-facing copy. The business datasets are fictional examples unless the brief explicitly names a live public website.The primary file produced by this prompt is shown below in its original language.
The supplied synthetic outcome has a positive purchase-rate difference, but its 95% confidence interval does not establish the required minimum improvement of 0.5 percentage points. Under the frozen decision rule, do not label this an unconditional launch win. This is an actual analysis of a fictional dataset, not a live experiment performed on shoppers.
If the page puts the $24 price, $5 one-mug US shipping charge and hand-wash/no-microwave restrictions beside Add to cart, then seven-day purchase probability should rise from 2.5% toward at least 3.0%, because shoppers can evaluate cost and care before committing. Hold price, targeting and other page content constant.
Randomize unique eligible shoppers 50:50, persist assignment across visits and analyze by first assigned variant. The primary measure is at least one paid, non-test purchase within seven days of first exposure, divided by all assigned eligible shoppers. Count each shopper once; orders/sessions is not a substitute. Track exposure, assignment and transaction deduplication before enrollment.
Guardrails: pause on instrumentation errors; investigate checkout-error increases of at least 0.5 percentage points and 30-day returned-order increases of at least one percentage point using matched denominators. Observe stock and fulfillment changes that could invalidate the comparison. Evaluate guardrails continuously for operational safety; do not repeatedly test the primary metric for an early win.
The original Skill's calculator ran with baseline 0.025, absolute MDE 0.005, two-sided alpha 0.05 and power 0.80. It returns 16,793 per group, 33,586 total. At 600 newly eligible shoppers a day total, enrollment takes 56 days. Allow seven more days for the last entrant's purchase window: approximately 63 days for the purchase readout. A 30-day return window after the last qualifying purchase can require roughly 93 days from enrollment start. Actual calendar timing depends on exposure and purchase timestamps.
Freeze the assignment, sample threshold, attribution window and analysis before starting. Run through whole weeks and wait for maturity; do not reuse the outcome dataset's assumptions as proof that real instrumentation and return windows passed.
| Measure | Control | Variant |
|---|---|---|
| Eligible shoppers | 16,793 | 16,793 |
| Purchasers | 420 | 537 |
| Seven-day purchase rate | 2.5010% | 3.1978% |
Absolute difference: 0.6967 percentage points. Relative difference: 27.86%. Unpooled Wald 95% interval: 0.3409 to 1.0525 percentage points. Pooled two-proportion z-test, two-sided p = 0.00012448. These were calculated from counts, not copied from the supplied summary fields.
Counts are exactly balanced; that alone cannot verify assignment implementation. The fixture assumes mature observations and acceptable guardrails but provides no underlying event log or guardrail counts. Treat those as exercise assumptions.
For a proposed 12-week rollout at 600 shoppers/day, the economic scenario covers 50,400 shoppers. At the supplied $12 incremental contribution per additional order, one-time $120 implementation and $20/week operating cost, total fixed/operating cost is $360. Applying the effect interval to that scenario yields modeled net contribution of $1,701.89 to $6,005.62, with point estimate $3,853.76. This is a projection under stated contribution and traffic assumptions, not achieved revenue or profit.
Statistical evidence against zero is strong in the fixture. The lower bound (0.003409) remains below the preselected 0.005 practical threshold, so the strict rule fails. Keep the factual information improvements as a separate product-quality decision if necessary, but do not redefine the experiment's success threshold after seeing results. Investigate comprehension and design a fresh test if the merchant still requires evidence of a 0.5-point minimum effect.
The package includes the original calculator output and independently recomputed analysis.json. No live traffic was randomized and no production page was changed by this case execution.
Merchant business data is synthetic. Website observations, when present, come from the public URLs listed in the report.
The calculator is for its documented two-proportion assumptions; it is not a general continuous-metric or multivariate test engine and does not launch tests.
Source and package files reviewed: GitHub 6d2f02ea077e. This is a static review, not an installation or business-outcome test.
Use authorized inputs and keep secrets out of prompts. Review customer-facing claims, calculations, code changes, and any proposed external action before applying them.
Plan test hypotheses, metrics, stopping rules, and conversion-rate sample sizes. A Python calculator makes baseline, minimum detectable effect, significance, and power explicit.
Choose the method for your agent and install the complete package, not just the displayed SKILL.md. Baseline conversion, absolute or relative MDE, traffic, allocation, alpha, power, and business guardrails. Start a new agent session and explicitly ask to use experiment-designer. Try one use case below with a small sample and review the output.
Size a checkout conversion experiment; Prioritize test ideas with explicit evidence and effort; Write decision criteria and guardrails before launch
The calculator is for its documented two-proportion assumptions; it is not a general continuous-metric or multivariate test engine and does not launch tests.
Source and package files reviewed: GitHub 6d2f02ea077e. This is a static review, not an installation or business-outcome test.
Source and package files reviewed: GitHub 6d2f02ea077e. This is a static review, not an installation or business-outcome test.
Source and package files reviewed: GitHub 6d2f02ea077e. This is a static review, not an installation or business-outcome test.
No reviews yet.