Treffer: Foundations of A/B Experimentation Quality in Large-Scale Distributed Systems.

A/B testing has become a key system of evidence-based decision-making in large-scale distributed systems in the technology enterprise, e-commerce platform, and service provider of digital services. Online controlled experimentation helps organizations to test their hypotheses, optimize user experiences, and reduce the risks of deploying a product by comparing several variants of the product under scientifically rigorous controlled conditions. The basis of credible experimentation programs that could provide the support of consequential business decisions is randomization integrity, reasonable selection of the units of the experiment, and robust metric design. Bootstrap resampling and the use of variance reduction methods such as CUPED are statistical tools that help to reinforce the validity of experiments by measuring the uncertainty of the dataset accurately and do not rely on classical parameter assumptions. Distributed computing environments present exceptional threats such as system-level noises, variability in infrastructures, geographic heterogeneity, and interference effects due to social network connections, which can bias treatment effect estimations in the absence of suitable controls. Integrated experimentation models standardize the design, implementation, and evaluation procedures and allow the same form of interpretation of organizational testing programs. Guardrail measures cover unwanted adverse effects, and the holdout groups maintain the measurement of long-term effects. The development of a full experimentation infrastructure allows companies to be innovative with a good degree of certainty and allow the systems to be refined in relation to user experiences under a wide variety of real world circumstances. [ABSTRACT FROM AUTHOR]