Diagnosing Why a Site's Set of Pages May Be Ranking Poorly - Whiteboard Friday

Posted by randfish

Your rankings have dropped and you don't know why. Maybe your traffic dropped as well, or maybe just a section of your site has lost rankings. It's an important and often complex mystery to solve, and there are a number of boxes to check off while you investigate. In this Whiteboard Friday, Rand shares a detailed process to follow to diagnose what went wrong to cause your rankings drop, why it happened, and how to start the recovery process.

Click on the whiteboard image above to open a high-resolution version in a new tab!

Video Transcription

Howdy, Moz fans, and welcome to another edition of Whiteboard Friday. This week we're going to talk about diagnosing a site and specifically a section of a site's pages and why they might be performing poorly, why their traffic may have dropped, why rankings may have dropped, why both of them might have dropped. So we've got a fairly extensive process here, so let's get started.

Step 1: Uncover the problem

First off, our first step is uncovering the problem or finding whether there is actually a problem. A good way to think about this is especially if you have a larger website, if we're talking about a site that's 20 or 30 or even a couple hundred pages, this is not a big issue. But many websites that SEOs are working on these days are thousands, tens of thousands, hundreds of thousands of pages. So what I like to urge folks to do is to

A. Treat different site sections as unique segments for investigation. You should look at them individually.

A lot of times subfolders or URL structures are really helpful here. So I might say, okay, MySite.com, I'm going to look exclusively at the /news section. Did that fall in rankings? Did it fall in traffic? Or was it /posts, where my blog posts and my content is? Or was it /cities? Let's say I have a website that's dealing with data about the population of cities. So I rank for lots of those types of queries, and it seems like I'm ranking for fewer of them, and it's my cities pages that are poorly performing in comparison to where they were a few months ago or last year at this time.

B. Check traffic from search over time.

So I go to my Google Analytics or whatever analytics you're using, and you might see something like, okay, I'm going to look exclusively at the /cities section. If you can structure your URLs in this fashion, use subfolders, this is a great way to do it. Then take a look and see, oh, hang on, that's a big traffic drop. We fell off a cliff there for these particular pages.

This data can be hiding inside your analytics because it could be that the rest of your site is performing well. It's going sort of up and to the right, and so you see this slow plateauing or a little bit of a decline, but it's not nearly as sharp as it is if you look at the traffic specifically for a single subsection that might be performing poorly, like this /cities section.

From there, I'm going to next urge you to use Google Trends. Why? Why would I go to Google Trends? Because what I want you to do is I want you to look at some of your big keywords and topics in Google Trends to see if there has been a serious decline in search volume at the same time. If search demand is rising or staying stable over the course of time where you have lost traffic, it's almost certainly something you've done, not something searchers are doing. But if you see that traffic has declined, for example, maybe you were ranking really well for population data from 2015. It turns out people are now looking for population data for 2016 or '17 or '18. Maybe that is part of the problem, that search demand has fallen and your curve matches that.

C. Perform some diagnostic queries or use your rank tracking data if you have it on these types of things.

This is one of the reasons I like to rank track for even these types of queries that don't get a lot of traffic.

1. Target keywords. In this case, it might be "Denver population growth," maybe that's one of your keywords. You would see, "Do I still rank for this? How well do I rank for this? Am I ranking more poorly than I used to?"

2. Check brand name plus target keyword. So, in this case, it would be my site plus the above here plus "Denver population growth," so My Site or MySite.com Denver population growth. If you're not ranking for that, that's usually an indication of a more serious problem, potentially a penalty or some type of dampening that's happening around your brand name or around your website.

3. Look for a 10 to 20-word text string from page content without quotes. It could be shorter. It could be only six or seven words, or it could be longer, 25 words if you really need it. But essentially, I want to take a string of text that exists on the page and put it in order in Google search engine, not in quotes. I do not want to use quotes here, and I want to see how it performs. This might be several lines of text here.

4. Look for a 10 to 20-word text string with quotes. So those lines of text, but in quotes searched in Google. If I'm not ranking for this, but I am for this one ... sorry, if I'm not ranking for the one not in quotes, but I am in quotes, I might surmise this is probably not duplicate content. It's probably something to do with my content quality or maybe my link profile or Google has penalized or dampened me in some way.

5. site: urlstring/ So I would search for "site:MySite.com/cities/Denver." I would see: Wait, has Google actually indexed my page? When did they index it? Oh, it's been a month. I wonder why they haven't come back. Maybe there's some sort of crawl issue, robots.txt issue, meta robots issue, something. I'm preventing Google from potentially getting there. Or maybe they can't get there at all, and this results in zero results. That means Google hasn't even indexed the page. Now we have another type of problem.

D. Check your tools.

1. Google Search Console. I would start there, especially in the site issues section.

2. Check your rank tracker or whatever tool you're using, whether that's Moz or something else.

3. On-page and crawl monitoring. Hopefully you have something like that. It could be through Screaming Frog. Maybe you've run some crawls over time, or maybe you have a tracking system in place. Moz has a crawl system. OnPage.org has a really good one.

4. Site uptime. So I might check Pingdom or other things that alert me to, "Oh, wait a minute, my site was down for a few days last week. That obviously is why traffic has fallen," those types of things.

Step 2: Offer hypothesis for falling rankings/traffic

Okay, you've done your diagnostics. Now it's time to offer some hypotheses. So now that we understand which problem I might have, I want to understand what could be resulting in that problem. So there are basically two situations you can have. Rankings have stayed stable or gone up, but traffic has fallen.

A. If rankings are up, but traffic is down...

In those cases, these are the five things that are most typically to blame.

1. New SERP features. There's a bunch of featured snippets that have entered the population growth for cities search results, and so now number one is not what number one used to be. If you don't get that featured snippet, you're losing out to one of your competitors.

2. Lower search demand. Like we talked about in Google Trends. I'm looking at search demand, and there are just not as many people searching as there used to be.

3. Brand or reputation issues. I'm ranking just fine, but people now for some reason hate me. People who are searching this sector think my brand is evil or bad or just not as helpful as it used to be. So I have issues, and people are not clicking on my results. They're choosing someone else actively because of reputation issues.

4. Snippet problems. I'm ranking in the same place I used to be, but I'm no longer the sexiest, most click-drawing snippet in the search results, and other people are earning those clicks instead.

5. Shift in personalization or location biasing by Google. It used to be the case that everyone who searched for city name plus population growth got the same results, but now suddenly people are seeing different results based on maybe their device or things they've clicked in the past or where they're located. Location is often a big cause for this.

So for many SEOs for many years, "SEO consultant" resulted in the same search results. Then Google introduced the Maps results and pushed down a lot of those folks, and now "SEO consultant" results in different ranked results in each city and each geography that you search in. So that can often be a cause for falling traffic even though rankings remain high.

B. If rankings and traffic are down...

If you're seeing that rankings have fallen and traffic has fallen in conjunction, there's a bunch of other things that are probably going on that are not necessarily these things. A few of these could be responsible still, like snippet problems could cause your rankings and your traffic to fall, or brand and reputation issues could cause your click-through rate to fall, which would cause you to get dampened. But oftentimes it's things like this:

1. & 2. Duplicate content and low-quality or thin content. Google thinks that what you're providing just isn't good enough.

3. Change in searcher intent. People who were searching for population growth used to want what you had to offer, but now they want something different and other people in the SERP are providing that, but you are not, so Google is ranking you lower. Even though your content is still good, it's just not serving the new searcher intent.

4. Loss to competitors. So maybe you have worse links than they do now or less relevance or you're not solving the searcher's query as well. Your user interface, your UX is not as good. Your keyword targeting isn't as good as theirs. Your content quality and the unique value you provide isn't as good as theirs. If you see that one or two competitors are consistently outranking you, you might diagnose that this is the problem.

5. Technical issues. So if I saw from over here that the crawl was the problem, I wasn't getting indexed, or Google hasn't updated my pages in a long time, I might look into accessibility things, maybe speed, maybe I'm having problems like letting Googlebot in, HTTPS problems, or indexable content, maybe Google can't see the content on my page anymore because I made some change in the technology of how it's displayed, or crawlability, internal link structure problems, robots.txt problems, meta robots tag issues, that kind of stuff.

Maybe at the server level, someone on the tech ops team of my website decided, "Oh, there's this really problematic bot coming from Mountain View that's costing us a bunch of bandwidth. Let's block bots from Mountain View." No, don't do that. Bad. Those kinds of technical issues can happen.

6. Spam and penalties. We'll talk a little bit more about how to diagnose those in a second.

7. CTR, engagement, or pogo-sticking issues. There could be click-through rate issues or engagement issues, meaning pogo sticking, like people are coming to your site, but they are clicking back because they weren't satisfied by your results, maybe because their expectations have changed or market issues have changed.

Step 3: Make fixes and observe results

All right. Next and last in this process, what we're going to do is make some fixes and observe the results. Hopefully, we've been able to correctly diagnose and form some wise hypotheses about what's going wrong, and now we're going to try and resolve them.

A. On-page and technical issues should solve after a new crawl + index.

So on-page and technical issues, if we're fixing those, they should usually resolve, especially on small sections of sites, pretty fast. As soon as Google has crawled and indexed the page, you should generally see performance improve. But this can take a few weeks if we're talking about a large section on a site, many thousands of pages, because Google has to crawl and index all of them to get the new sense that things are fixed and traffic is coming in. Since it's long tail to many different pages, you're not going to see that instant traffic gain and rise as fast.

B. Link issues and spam penalty problems can take months to show results.

Look, if you have crappier links or not a good enough link profile as your competitors, growing that can take months or years even to fix. Penalty problems and spam problems, same thing. Google can take sometimes a long time. You've seen a lot of spam experts on Twitter saying, "Oh, well, all my clients who had issues over the last nine months suddenly are ranking better today," because Google made some fix in their latest index rollout or their algorithm changed, and it's sort of, okay, well we'll reward the people for all the fixes that they've made. Sometimes that's in batches that take months.

C. Fixing a small number of pages in a section that's performing poorly might not show results very quickly.

For example, let's say you go and you fix /cities/Milwaukee. You determine from your diagnostics that the problem is a content quality issue. So you go and you update these pages. They have new content. It serves the searchers much better, doing a much better job. You've tested it. People really love it. You fixed two cities, Milwaukee and Denver, to test it out. But you've left 5,000 other cities pages untouched.

Sometimes Google will sort of be like, "No, you know what? We still think your cities pages, as a whole, don't do a good job solving this query. So even though these two that you've updated do a better job, we're not necessarily going to rank them, because we sort of think of your site as this whole section and we grade it as a section or apply some grades as a section." That is a real thing that we've observed happening in Google's results.

Because of this, one of the things that I would urge you to do is if you're seeing good results from the people you're testing it with and you're pretty confident, I would roll out the changes to a significant subset, 30%, 50%, 70% of the pages rather than doing only a tiny, tiny sample.

D. Sometimes when you encounter these issues, a remove and replace strategy works better than simply upgrading old URLs.

So if Google has decided /cities, your /cities section is just awful, has all sorts of problems, not performing well on a bunch of different vectors, you might take your /cities section and actually 301 redirect them to a new URL, /location, and put the new UI and the new content that better serves the searcher and fixes a lot of these issues into that location section, such that Google now goes, "Ah, we have something new to judge. Let's see how these location pages on MySite.com perform versus the old cities pages."

So I know we've covered a ton today and there are a lot of diagnostic issues that we haven't necessarily dug deep into, but I hope this can help you if you're encountering rankings challenges with sections of your site or with your site as a whole. Certainly, I look forward to your comments and your feedback. If you have other tips for folks facing this, that would be great. We'll see you again next week for another edition of Whiteboard Friday. Take care.

Video transcription by Speechpad.com

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read!