Jan Bosch is a research center director, professor, consultant and angel investor in startups. You can contact him at jan@janbosch.com.


Outdated belief #8: A/B testing is only for online systems

Leestijd: 4 minuten

In the online world of software as a service (SaaS), A/B testing in some form is used continuously for virtually any type of development. For an A/B test, two alternative realizations of a feature or some functionality are developed and presented to a test group (alternative B) and a control group (alternative A). Some outcome metric is tracked and measured for both groups, eg conversion, to determine which of the alternatives performs better. There are numerous alternatives and variations available, but the basic notion is that we use data from actual users to determine the preferred alternative to roll out to all users.

When working with companies in the embedded-systems industry and the B2B space, it surprises me to meet lots of people who either are against the use of A/B testing or who believe that it doesn’t apply to them and their industry. The arguments are multiple but generally fall into two main categories: “can’t do it” and “don’t need it.”

This article is exclusively available to premium members of Bits&Chips. Already a premium member? Please log in. Not yet a premium member? Become one and enjoy all the benefits.


Related content