Wednesday, November 11 2026, All day Caldwell Hall Room 204 Industry Day 2026 Flyer Final.pdf (243.47 KB) Lunch Lecture: From Classical DOE to Online Experimentation at Scale: Challenges, Lessons, and Opportunities Controlled experiments are widely regarded as the gold standard for establishing causal relationships and guiding decision-making. While the core principles of Design of Experiments (DOE) remain fundamental, applying them at the scale of modern online platforms introduces new challenges and opportunities. This lunch lecture provides an overview of online experimentation and how technology companies use online controlled experiments and large-scale A/B testing to evaluate products, algorithms, and business decisions. We will discuss common use cases, compare online experimentation with classical DOE, and examine practical challenges that arise in real-world settings, including interference (violations of the SUTVA assumption), metric design, data quality, and decision-making under uncertainty. Along the way, I will share lessons learned from deploying experimentation systems at scale and discuss emerging opportunities for statisticians and data scientists in this rapidly evolving field. 4:30pm Seminar: Advanced Experimentation in Online Marketplaces and Recommender Systems Online platforms increasingly rely on controlled experiments to evaluate product changes, optimize marketplace efficiency, and improve recommender systems. However, standard A/B testing often faces fundamental challenges in these environments due to interference among experimental units, budget constraints, competition for shared opportunities, and the need to jointly evaluate impacts on multiple stakeholders. In this talk, we present two recent advances in experimentation methodology for large-scale online platforms. First, we discuss methodological and system-level advances in Budget-Split Testing (BST), a framework widely used in online advertising marketplaces to mitigate bias caused by advertiser competition and budget cannibalization effects. We examine how BST interacts with marketplace dynamics, characterize the statistical and business trade-offs associated with increasing the number of budget partitions, and identify previously overlooked sources of bias in standard BST implementations. We then introduce a Robust BST framework that systematically eliminates partition-related biases while preserving measurement accuracy. Second, we present a principled framework for producer-side experimentation in online recommender systems. Unlike consumer-side experiments, producer-side experiments require items from treatment and control groups to be ranked by different models and merged into a single recommendation list for each consumer, creating unique experimental design challenges. We discuss fundamental limitations of existing approaches, introduce the principles of consistency and monotonicity for producer-side experimentation, and present a systematic solution based on counterfactual interleaving designs that enables accurate measurement of ranking-model impacts on producers. Together, these methods illustrate how new experimentation methodologies can address interference and counterfactual evaluation challenges in modern online marketplaces and recommender systems, enabling more reliable measurement of causal effects and data-driven decision-making at scale. Bio Dr. Shan Ba is a Data Science Applied Scientist at LinkedIn. He works on online experimentation, causal inference, and statistical methodologies for large-scale decision making in AI-driven online marketplaces. He also serves as Chair of the Quality & Productivity Section of the American Statistical Association (ASA) and as an Associate Editor of Technometrics. He holds a Ph.D. in Industrial Engineering (Statistics) from the Georgia Institute of Technology and an MBA from the University of Chicago Booth School of Business.
