Article: The Surprising Power of Online Experiments: Getting the Most Out of A/B and Other Controlled Tests (Kohavi & Thomke, 2017
3 important questions on Article: The Surprising Power of Online Experiments: Getting the Most Out of A/B and Other Controlled Tests (Kohavi & Thomke, 2017
Build a large-scale capability
- The majority of new ideas fail in experiments.
- All this goes to show that companies need to kiss a lot of frogs (that is, perform a massive
number of experiments) to find a prince.
Adress the definition of success
Every business group must define a suitable evaluation metric for experiments that aligns with its strategic goals.
Identifying which short-term figures are the best predictors of long-term results is difficult. Getting it right – coming up with an overall evaluation criterion (OEC) – takes often extensive internal
debate and requires close cooperation between senior executives who understand the strategy and data analysts who understand metrics.Looking at different metrics is crucial because it helps teams discover whether an experiment has an unanticipated impact on another area.
Finally, to really understand the value of an experiment, look at the difference between the expected result and the actual result.
- If you thought something was going to happen and it did, you haven't learned much not learned much.
- If it did not happen, you learned something important (positive or negative).
By combining the power of software with the scientific rigor of controlled experiments, your company can create a learning lab. The returns you reap—in cost savings, new revenue, and improved user experience—can be huge. If you want to gain a competitive advantage, your firm should build an experimentation capability and master the science of conducting online tests
The question on the page originate from the summary of the following study material:
- A unique study and practice tool
- Never study anything twice again
- Get the grades you hope for
- 100% sure, 100% understanding
Remember faster, study better. Scientifically proven.
