Alumni Questions: Reliable case studies, SEO, and test design

Students and alumni of the MarketingExperiments Training and Certification Program often share their questions and concerns with our analysts before, during, and after they take one of our courses. The questions below are an example of the interaction you can expect if you attend a MarketingExperiments course:

Q: Do you know any other resources, except MarketingExperiments and MarketingSherpa, that are good at providing useful insights from case studies?

Believe it or not, I’ve been in this Internet marketing field for two years (you’re probably here a lot longer) and been through a lot of misleading information until I accidently found you guys and really learned how to test things and see if they REALLY work instead of blindly believing some “gurus” who told me something like…”this is tested and it’s working 100%!” (the only thing that was truthful was that 100% thing…the problem was that almost 100% of what they were telling me didn’t work.

Surely others – for example online marketing consultancies – will offer to advise you on changing your website to improve performance and will use a third-party testing tool to measure the impact. Also, some of the large-scale testing tool vendors offer hosted and/or managed service engagements using their products.

Unfortunately, as you said, most other organizations are not research focused. MarketingExperiments is a research institute dedicated to discovering what really works in online marketing to help our Research Partners, certification program students, and Journal subscribers succeed. So there are very few resources we can point you to.

One notable exception is the award-winning Get Elastic blog written by ecommerce analyst Linda Bustos. Get Elastic provides useful insights about SEO (search engine optimization), usability, analytics, email, shopping cart abandonment, and social media. Linda is also a MarketingExperiments certified optimization professional and knows our methodology inside-out.

Q: I’ve been through your Fundamentals of Online Testing course. You teach about landing page and order page optimization. I was wondering if you have some specific advice/studies where the SEO effectiveness of a campaign was tested (SEO, not paid traffic) because I can’t find any valid way to measure the effectiveness of an SEO campaign.

Regarding testing in which the primary channel is “natural search” or SEO traffic, we do have considerable experience working with companies and organizations for whom SEO is a significant portion of their demand, and we have published some research on the topic. In fact, all of our research is readily available for free in the MarketingExperiments Research Directory.

As you’re already aware, based upon your question, there are a number of challenges associated with the dramatic differences between the key optimization factors over online marketers can control when choosing between PPC (pay per click) and SEO. Specifically, not only is there a relative dearth of information available to search marketers as compared to paid advertising, the search networks are comparatively opaque about their results-positioning algorithms and tend to change them frequently to confound SEO-gamers. Further, the rate at which changes to a site are detected by the networks and “shaken into the mix” is volatile and unpredictable.

Consequently, the MarketingExperiments approach is to evaluate the subject site/page based upon the principles of Offer/Response-Optimization – such as the Conversion Heuristic that you learn in the Landing Page Optimization course – then develop hypotheses about how to improve its performance and test those hypotheses using paid search traffic (which is designed to mirror the motivation profile of their ideal customers through SEO). This provides comparatively rapid and precise evidence about the specific factors of study. Then, those test results are used in concert with the latest SEO-algorithm information to develop the channel-specific page designs and a plan for deploying them to get the largest and most rapid performance gain.

Q: Do you have some advice/case studies about testing the effectiveness of an online service?

For example, a site like Traffic Bug submits your URL to social networks (Connotea, Propeller) automatically and claims that this increases your rankings and indexes your pages.

I want to do a test to see if what they say is true. I would take 10 very obscure pages (to be exact, profile links) that are rarely indexed by Google. I create 10 of those obscure pages on different URLs and do nothing with them. Then I create 10 pages on the same URL and submit them to Traffic Bug. I wait for seven days.

Of course, I make sure everything is satisfied in terms of validity and all that. So I wait for seven days (that’s the first milestone, I then check them again after 30 days but let’s focus on the period after seven days) and then take a look. My sample size is small but what I’m interested in is discovering if this service is highly effective in indexing pages on Google.

So if the first sample (that is not submitted) gets one out of 10 pages indexed and the second sample (which is submitted) gets nine out of 10 pages indexed, and I make sure that this is a valid test (using the MarketingExperiments validation formula from the Fundamentals of Online Testing course), can I assume an online service is very effective?

I wanted to hear your comments on this. What do you think is wrong/right with the above test and what would be some things to do for improvement? Also, do you think that a sample size of 10 is big enough for a test to discover whether an online service has a dramatic effect?

While the approach you described appears sound in principle, you will probably have difficulty actually achieving valid results based upon the circumstances you outlined. And even if the results are valid, they may not really answer your question.

In evaluation of a tool like this, a different approach may serve you better. When building an SEO campaign and links there are other things to consider:

1. Are you sure where all these links are getting posted? Some indexing tools use less-than-kosher link-building strategies that can actually get your domain in trouble with search engine providers. The appearance of link spamming and posting links on flagged sites can cause domains to suffer penalties that can affect the ranking of their sites…occasionally on a permanent basis.
2. Are these links actually driving traffic and revenue? Many indexing services cost money and you need to perform due diligence with an ROI analysis to see if the efforts are recouping their costs.
3. Does the service provide a list of links they have generated for tracking? Not only is this good for tracking but allows you to see the places your links are getting placed. Some business owners consider it (as you should too), important to see the company you are keeping on these sites.  For example, are links to, “Adam’s XXX site” right next to your link or the content on these pages? You can use tools like Yahoo Site Explorer or Google Webmaster Tools to fish out these links, but the service should do this for you.
4. It is important to note that we are not accusing Traffic Bug of doing any of these things, but with any sort of service along these lines you need to do your research first.

From your experience, how would you answer the above questions? Share your advice in the comments section.

Special thanks to Director of Sciences Bob Kemper and Research Analyst Corey Trent for their help in answering these questions.