I speak to a lot of folks in and around the SEO industry, and occasionally, I meet people who are building in-house SEO testing capabilities. It struck me that exploring the ‘build vs. buy’ debate would be useful for our audience.
So, in this article, I’m going to discuss the benefits and drawbacks of both building and buying the tools that will support SEO testing and I’ll highlight some useful information that can help inform the decision.
A quick detour into the world of conversion rate optimization (CRO)
An informative place to start is by thinking about conversion rate optimization. While SEO and CRO have been around for a while, businesses’ approach and attitude towards testing them varies dramatically.
Testing has always been considered an integral part of the CRO process.
SEO, on the other hand, was around for decades before structured, rigorous testing became common. SearchPilot was among the pioneers.
For this reason, the CRO testing and tool market is much more mature than its SEO counterpart. The CRO space is extremely competitive with many companies that have raised significant venture capital or grown substantially.
For us at SearchPilot, being early in this journey means we face less competition in our market, which is both a challenge and an exciting opportunity. We are trailblazing a new path and creating something innovative that people want, which is both exhilarating and challenging. Observing the more advanced CRO market allows us to anticipate what might lie ahead for SEO testing.
If you want to dive deeper into this, I’ve been really enjoying some of the resources from VP and controlled experiment evangelist Ronny Kohavi’s training, especially this insightful series of decks and information. The depth of thought put into these materials is impressive, particularly from my perspective within the SEO testing space. It’s fascinating to compare our area of expertise with the CRO testing market and to envision what the future holds.
The case for partnering with a third-party vendor
I came across this presentation by Ryan Lucht, formerly director of growth strategy at CRO Metrics and now at Eppo, and I’m really aligned with his thoughts on this. It’s nice for me to have an example of someone who’s working on something that’s so similar to what we do but who isn’t even slightly a competitor. It just means I have no problem highlighting it.
Ryan's experience with enterprise customers typically involves them opting for third-party tools to increase the number of live experiments they can run. This aligns with my experience in SEO testing. By partnering with a company like SearchPilot, customers can run more tests and identify more winners due to the increased test cadence and improved test sensitivity. Indeed, our biggest pitches tend to revolve around this idea: running more tests leads to more winners.
Ryan emphasizes the importance of test velocity—the focus on running more tests and getting them live. The concept of ‘fat tails of results’ shows that a significant portion of wins comes from a small number of experiments. Over 90% of the positive impact from testing programs comes from the top 5% of experiments. Finding these high-impact experiments requires running many tests to uncover the one in twenty that brings most of the rewards.
I discussed this in an SEO context in my presentation SEO MoneyBall, which I delivered at MozCon. I talked about how having a high testing cadence as a key performance indicator is a hallmark of advanced SEO maturity.
In any case, I agree with Ryan that increasing the number of live tests is the not-so-secret ingredient to success. This need to run more tests often leads companies to opt for buying third-party tools rather than building their own in the name of speed, and often justifiably so.
When you should build and scale your own testing capabilities
Partnering with a third party is not always the right path for everyone, of course, and some organizations with the scale and resources most likely should build their own experimentation tools. Companies like Amazon, Netflix and Meta may buck the ‘buy is better’ advice for several reasons:
- They have CEO involvement. Their CEOs are deeply involved in experimentation and regularly talk about key terms like velocity and impact. For example, Reed Hastings at Netflix teaches Experimentation 101 to new hires.
- Their founder or leadership team created the first iteration of their testing infrastructure themselves. This is something that Mark Zuckerberg apparently did at Meta in the early days.
- The philosophy and ethos of testing are baked into the organizational structure. These topics and related projects are often brought up at board meetings and investor calls.
We at SearchPilot aspire to this level of executive engagement and foundational development. Part of our product vision is to see our data, dashboards, and work gain visibility higher in the organizations we work with, such as during earnings calls, board meetings, and public company filings. This kind of integration shows that a company is on the right track and building its own capabilities is the right move.
Build vs Buy: The final verdict
Most companies operating today don’t have the resources in place to realize this kind of experimentation culture. In this case, partnering with a third-party vendor is the most practical bet. The all-in cost of building tools internally is significant, and often underestimated. When you factor in the costs of engineers, data scientists, and other high-skill resources, these expenses can quickly exceed even substantial tool budgets.
Focusing solely on costs, which is a temptation in any business, often overlooks the revenue and growth opportunities that third-party tools can provide. Effective SEO experimentation can drive large amounts of organic search traffic and result in explosive business growth. For teams responsible for these outcomes, the most important factor is the capability to design, implement, analyze, and deploy tests end-to-end. Partnering with a specialized tool like SearchPilot can streamline this process and free up important resources for other growth initiatives.
Ryan's resources from his agency analysis provide valuable insights into when to buy versus build that might be useful to you. He covers factors such as company longevity, cost models, and the value of testing programs. I’ll link it here again so you can have a read.
The big question you need to answer
What you’re really trying to figure out is whether building or buying SEO testing tools will give you the best quality tests at the best speed and cadence.
For many organizations of a certain size and dedication, the best option is a third-party vendor because they will have greater sensitivity, analytics capabilities, and ability to roll out tests. They will also be supported by experts, professional services, and other leaders in the space.
But if your CEO built the first version of your testing platform like Zuck supposedly did at Meta, there’s a pretty good chance that particular internal tool is going to be more tightly coupled to your code base. You’re actually going to be able to go faster and it’s going to be more accurate for your specific needs. The final decision, of course, rests with you.
Learn more about building and executing successful SEO tests
If you’re looking to get started building and executing effective SEO tests we have a huge amount of resources available, including detailed guides on how testing works, instructions you can share with your team, and a searchable catalogue of test results. You can sign up for our email list to receive these insights before anyone else. If you’d like to take SearchPilot for a test flight, get in touch to schedule a demo.
