abaudit — Experiment Validity Report

Generated 2026-05-19 22:42  ·  abaudit 0.1.0

Key Numbers

0.0003
p-value (primary)
0.89
PPV — prob. effect is real
0.95
Statistical power
0.163
Cohen's d
2,000
Total sample size
3
Metrics tested

Bias score: 15%   Low concern

Validity Checks

Check Result Status
p-value (primary) 0.0003 ✔
p-value (Bonferroni corrected) 0.0009 ✔
PPV — prob. effect is real 0.89 ✔
Statistical power 0.95 ✔
Sample Ratio Mismatch p = 0.371 ✔
Metrics tested 3 ⚠
Optional stopping eff. α = 0.226 ⚠
Effect size (Cohen's d) 0.163 ✔

⚠ Warnings

⚠ 3 metrics tested — Bonferroni-corrected p = 0.0009 (raw p = 0.0003).
⚠ Optional stopping risk: p-value checked 5 times. Effective α ≈ 0.226 (nominal: 0.05).

💡 Recommendations

💡 Use sequential testing (SPRT) or an alpha-spending function when interim looks are necessary.

Parameters Used

Prior f0.3
Alpha (α)0.05
n control980
n treatment1,020
Corrected p-value0.0009
Effective alpha0.2262