IBM Web Team Measures Impact of an Editor on Content

James Mathewson, Frank Donatone, and Cynthia Fishel of IBM conducted an experiment to measure the impact of an editor on web content:

We took a sample of unedited pages with high traffic from across our various business units and ran them through Dave Harlan, the editing lead for the group that creates a lot of our marketing content. We then ran an A/B test, where we served the unedited versions to a random sample of users and the edited versions to the rest of the users. We then measured engagement (defined as clicks to desired links on the page) on those pages over the course of a month.

The results were astonishing. The mean difference in engagement was 30 percent across the set of pages. And the standard deviation was one percent–we got a 30 percent improvement on the desired call to action for the pages across the board. Now it was just one test and it needs to be replicated before we draw strong conclusions. Your mileage may vary depending on the quality of your editors (Dave is exceptional, by all accounts). But we can provisionally conclude that well edited pages do 30 percent better than unedited pages.