try it with real data
Experiment Lab
The browser version of abkit, my A/B testing library. Load a real test with 90,189 mobile game players or your own CSV, and it walks through the analysis step by step. Nothing gets uploaded.
00Pick the data
Cookie Cats is a puzzle game. New players got the first gate at level 30 or level 40. Did moving it later change whether they came back?
Your own file needs one row per user, a group column, and numbers or true/false for each outcome.
01Is the split healthy?
Check this first. A split that's further from 50/50 than chance allows usually means a bug, and then every result below can be off too.
02What happened?
Bars show the 95% interval. The middle line is zero.
Interval. If it crosses zero, the data can't rule out "no change".
Adjusted p. Corrected for testing several metrics at once.
Chance better. The same data as a plain chance the new version wins.
03Can we cut the noise?
CUPED uses something measured before the test to remove noise, so you need fewer users. It can't use anything the change itself could affect.
04What if you'd checked it every day?
A normal 95% interval is only right if you look once. The always valid one stays right however often you check.
The data has no timestamps, so this replays it in a shuffled order.
05Planning the next one
0 users per group
Small changes need a lot of users. Picking the size up front is what keeps a normal test honest.