The experiment’s configuration, lifecycle, and runtime should be handled externally of the host application, leaving it completely unconcerned with those details. Moving experimentation-related details out of the host application’s domain achieves the complete separation of experience implementation (handled by the host application) and experiment instrumentation (handled by the experimentation server)—the holy grail of any successful experimentation framework.
The fundamental flaw of all existing commercial A/B tools is that they all started out as SaaS, which was a reasonable approach to the shallow experiments—their beachhead market,—but that architecture proved untenable when they attempted to extend down the host application’s stack. Without cogent support for full-stack experiments enterprise customers by enlarge opted to roll their own.
Although many tech-capable enterprises have been able to develop custom experimentation platforms, many tech-second companies have had to spend sweat and treasure on what are essentially stop-gap solutions. Here are the three principal reasons why implementing an A/B testing practice is a particularly gnarly problem to solve in the wild.