Sample Ratio Mismatch (SRM)

When we run an A/B test, visitors are assigned randomly to variations. Because randomness is involved, we can’t expect the number of visitors in each variation to be exactly equal. For example, if we flip a coin 100 times, it won’t always land on 50 heads and 50 tails — but it should be close.

What matters is that the difference stays within the expected margin of random variation. We measure this using something called an SRM check (Sample Ratio Mismatch). If the split is within this margin, we know the randomization is working correctly. If it’s outside, that could mean a technical problem with how traffic is being allocated.

If you however suspect that the allocation between variations is not evenly distributed, you can check this via an Sample Ratio Mismatch (SRM) check. The allocation is randomised which will make the number of visits

Normally this should not happen. However, If you have changed the allocation during the test there could be issues with this. (It is not recommended to change allocation during a active test)

Use the link below or search for a Sample Ratio Mismatch checker to make sure you've got an even split between variations.

Lukas Vermeer's Sample Ratio Mismatch (SRM) Checker

 

 

 

Was this article helpful?
0 out of 0 found this helpful