实验设计

Product-page experiment: positive estimate, practical threshold not cleared

The supplied synthetic outcome has a positive purchase-rate difference, but its 95% confidence interval does not establish the required minimum improvement of 0.5 percentage points. Under the frozen decision rule, do not label this an unconditional launch win. This is an actual analysis of a fictional dataset, not a live experiment performed on shoppers.

Predefined design

If the page puts the $24 price, $5 one-mug US shipping charge and hand-wash/no-microwave restrictions beside Add to cart, then seven-day purchase probability should rise from 2.5% toward at least 3.0%, because shoppers can evaluate cost and care before committing. Hold price, targeting and other page content constant.

Randomize unique eligible shoppers 50:50, persist assignment across visits and analyze by first assigned variant. The primary measure is at least one paid, non-test purchase within seven days of first exposure, divided by all assigned eligible shoppers. Count each shopper once; orders/sessions is not a substitute. Track exposure, assignment and transaction deduplication before enrollment.

Guardrails: pause on instrumentation errors; investigate checkout-error increases of at least 0.5 percentage points and 30-day returned-order increases of at least one percentage point using matched denominators. Observe stock and fulfillment changes that could invalidate the comparison. Evaluate guardrails continuously for operational safety; do not repeatedly test the primary metric for an early win.

Sample and timing

The original Skill's calculator ran with baseline 0.025, absolute MDE 0.005, two-sided alpha 0.05 and power 0.80. It returns 16,793 per group, 33,586 total. At 600 newly eligible shoppers a day total, enrollment takes 56 days. Allow seven more days for the last entrant's purchase window: approximately 63 days for the purchase readout. A 30-day return window after the last qualifying purchase can require roughly 93 days from enrollment start. Actual calendar timing depends on exposure and purchase timestamps.

Freeze the assignment, sample threshold, attribution window and analysis before starting. Run through whole weeks and wait for maturity; do not reuse the outcome dataset's assumptions as proof that real instrumentation and return windows passed.

Recomputed outcome

Measure	Control	Variant
Eligible shoppers	16,793	16,793
Purchasers	420	537
Seven-day purchase rate	2.5010%	3.1978%

Absolute difference: 0.6967 percentage points. Relative difference: 27.86%. Unpooled Wald 95% interval: 0.3409 to 1.0525 percentage points. Pooled two-proportion z-test, two-sided p = 0.00012448. These were calculated from counts, not copied from the supplied summary fields.

Counts are exactly balanced; that alone cannot verify assignment implementation. The fixture assumes mature observations and acceptable guardrails but provides no underlying event log or guardrail counts. Treat those as exercise assumptions.

Costs and decision

For a proposed 12-week rollout at 600 shoppers/day, the economic scenario covers 50,400 shoppers. At the supplied $12 incremental contribution per additional order, one-time $120 implementation and $20/week operating cost, total fixed/operating cost is $360. Applying the effect interval to that scenario yields modeled net contribution of $1,701.89 to $6,005.62, with point estimate $3,853.76. This is a projection under stated contribution and traffic assumptions, not achieved revenue or profit.

Statistical evidence against zero is strong in the fixture. The lower bound (0.003409) remains below the preselected 0.005 practical threshold, so the strict rule fails. Keep the factual information improvements as a separate product-quality decision if necessary, but do not redefine the experiment's success threshold after seeing results. Investigate comprehension and design a fresh test if the merchant still requires evidence of a 0.5-point minimum effect.

The package includes the original calculator output and independently recomputed analysis.json. No live traffic was randomized and no production page was changed by this case execution.