#SEQUENTIAL TESTING MODELS DRIVER#

The product manager could use our XP to glean the following metrics: the average values of the metric in both treatment and control groups, the lift (treatment effect), whether the lift is significant, and whether the sample sizes are large enough to wield high statistical power. Suppose a product manager wants to evaluate whether a new feature increases user satisfaction with Uber’s platform. Uber’s Experimentation Platform team conducts randomized experiments by leveraging A/B/N tests to determine the lift.Ī common use case for this methodology is feature release experiments. In essence, classic A/B testing enables us to randomly split users into control and treatment groups to compare the decision metrics between these groups and determine the experiment’s treatment effects. Uber applies this technique to make objective, data-driven, and scientifically rigorous product and business decisions. Randomized A/B or A/B/N tests are considered the gold standard in many quantitative scientific fields for evaluating treatment effects. In this article, we discuss how each of these statistical methods are used by Uber’s Experimentation Platform to improve our services. We also apply block bootstrap and delta methods to estimate standard errors, as well as regression-based methods to measure bias correction when calculating the probability of type I and type II errors in our statistical analyses. Broadly, we use four types of statistical methodologies: fixed horizon A/B/N tests (t-test, chi-squared, and rank-sum tests), sequential probability ratio tests (SPRT), causal inference tests (synthetic control and diff-in-diff tests), and continuous A/B/N tests using bandit algorithms (Thompson sampling, upper confidence bounds, and Bayesian optimization with contextual multi-armed-bandit tests, to name a few). There are various factors that determine which statistics methodology we should apply to a given use case. Uber’s Experimentation Platform conducts both randomized experiments and observational studies. On average, over 1,000 experiments are running on Uber’s Experimentation Platform at any given time.īelow is a chart outlining the types of experimentation methodologies that the Experimentation Platform team uses: Figure 2. The platform also lets users configure the universal holdout, used to measure the long-term effects of all experiments for a specific domain.

#SEQUENTIAL TESTING MODELS DRIVER#

For example, before Uber launched our new driver app, completely redesigned with our driver-partners in mind, it went through extensive hypothesis testings through a series of experiments conducted with our XP.Īt a high level, Uber’s XP allows engineers and data scientists to monitor treatment effects to ensure they do not cause regressions of any key metrics. There are over 1,000 experiments running on our platform at any given time. The platform supports experiments across our driver, rider, Uber Eats, and Uber Freight apps and is widely used to run A/B/N, causal inference, and multi-armed bandit (MAB)-based continuous experiments. Uber’s Experimentation Platform (XP) plays an important role in this process, enabling us to launch, debug, measure, and monitor the effects of new ideas, product features, marketing campaigns, promotions, and even machine learning models. Uber applies several experimental methodologies to use cases as diverse as testing out a new feature to enhancing our app design. Experimentation is at the core of how Uber improves the customer experience.

Sequential testing models