Unlike A/B experiments that are typically done by segmenting users, our SEO experiment framework segments pages. For example, in an experiment that has 50 percent of its pages grouped in “enabled” and the other 50 percent grouped in “control,” a page will fall into one of the groups depending on its URL:
‘enabled’ if hash(experiment_name + page_url) in enabled_group_range
‘control’ if hash(experiment_name + page_url) in control_group_range
Hashing the page URL combined with the experiment name ensures even distribution of pages among experiment groups and enables us to simultaneously run multiple experiments with varying group sizes.
Once the experiment is up and running, we can measure the performance of each group by comparing traffic.