Start here: how our SEO split tests work
If you aren't familiar with the fundamentals of how we run controlled SEO experiments that form the basis of all our case studies, then you might find it useful to start by reading the explanation at the end of this article before digesting the details of the case study below. If you'd like to get a new case study by email every two weeks, just enter your email address here.
In this week’s #SPQuiz on Twitter, we asked our followers what they thought would happen when we added emojis to title tags for thematic content pages. Here’s what they thought:
The largest proportion opted for positive, but it’s fair to say that opinion was evenly split between all three options - there was no intuitive feel among the community for how this test would go. In situations like this, testing is extremely useful so that we know the true impact of a change rather than guessing.
In this case, the result was negative - just under a third of respondents got it right!
The Case Study
Attempting to make title tags more “clickable” is often one of the types of SEO split test with the highest risk and reward. We often see very large uplifts 📈, but there’s also the risk that the change we make puts searchers off and negatively impacts traffic 📉. Add in the fact that Google now picks and chooses whether or not to show the exact title tag in search results, and title tag tests focused on clickthrough rate can be especially unpredictable.
One of our customers has a group of content pages, each of which comprises a collection of short content snippets on a topic or theme. They wanted to test adding a thematically appropriate emoji to the title tag on these pages to see what impact it would have on organic traffic. For context, back in 2020, we wrote about a test adding emoji to meta descriptions, which was a negative result for the listings site that tried it. This test differs in that it was title tags instead of descriptions, and the context of the site is more informational, which may mean users are more likely to be swayed by something like emojis.
Control:
<title>Golf stories. Read short stories all about golf | Example.com</title>
Variant:
<title>Golf stories ⛳️: Read short stories all about golf | Example.com</title>
After the test was launched and the titles were updated, Google ignored the change that we made (🤦♂️). We had hoped to see this in search results:
In fact, this is what we saw:
Google was choosing not to include the emoji, and everything before it, leaving us with a shorter title than before the test, while also removing the primary keyword for the page from what was displayed in search results.
The traffic impact of this test was as follows:
This equated to a negative result with a best estimate of 6% loss of organic traffic. There are two major reasons this may have happened. It may have been due to a drop in clickthrough rate because the primary keyword was no longer at the start of the title that users saw in search results. Alternatively, there may have been a drop in ranking performance due to adding emojis to title tags, reducing emphasis on the other keywords in there
As mentioned before, CTR focused title tag tests can be high risk and high reward. In this case, it came down on the side of risk. Google was rewriting the titles to remove both the element we thought would attract more clicks (the emoji), and the main keyword.
How our SEO split tests work
The most important thing to know is that our case studies are based on controlled experiments with control and variant pages:
- By detecting changes in performance of the variant pages compared to the control, we know that the measured effect was not caused by seasonality, sitewide changes, Google algorithm updates, competitor changes, or any other external impact.
- The statistical analysis compares the actual outcome to a forecast, and comes with a confidence interval so we know how certain we are the effect is real.
- We measure the impact on organic traffic in order to capture changes to rankings and/or changes to clickthrough rate (more here).
Read more about how SEO A/B testing works or get a demo of the SearchPilot platform.