Glossary term
Glossary term
Evaluation and Benchmarks
A statistical way of comparing two (or more) techniques—the A and the B. Typically, the A is an existing technique, and the B is a new technique. A/B testing not only determines which technique performs better but also whether the difference is statistically significant.
A/B testing usually compares a single metric on two techniques; for example, how does model accuracy compare for two techniques? However, A/B testing can also compare any finite number of metrics.
Created for this library
A SaaS product team A/B tests two onboarding flows against a seven-day activation metric before rolling the winner to all new sign-ups.
An online retailer A/B tests two product recommendation models in production with a 50/50 traffic split to measure conversion lift over four weeks.
A mobile game studio A/B tests a new monetization screen against the existing version to compare ARPDAU without harming day-seven retention.
Definition source: Google for Developers Machine Learning Glossary | Creative Commons Attribution 4.0 License