Cleaning Up Crawl ErrorsThose of us working in the technical trenches of SEO have likely faced the dread of a long list of crawl errors when using Google Search Console for the crawl error report. While a long list of errors can be scary and more than a little overwhelming, having a plan of attack and setting priorities can save you time and headaches.
Eoghan Henn over at Rebelytics wrote a blog a few weeks ago detailing a methodical way of handling crawl errors. In our blog, we'll cover the basics of these errors, including definitions of terms and how to apply Henn's methodology for addressing crawl errors in a way that is applicable to destination marketing websites.
Start by Understanding
What is a crawl error?
Crawl errors are specific URL errors that affect a web page and not the website as a whole. They need to be addressed, because if they are left unchecked over time, they can negatively affect search rankings.
What causes crawl errors?
- A Page is Renamed
- Redesign/New Site Launch
- Change of Domain
Identifying the different types of crawl errors
There are several different types of crawl errors that will be displayed in the crawl error report, however, the most common and easiest to fix are the 404 and 500 errors.
404 errors are indexed pages that no longer exist on the server or they have been renamed. The proper way to fix these is to redirect the nonexistent page to an existing one.
If a page 404s, then Google crawlers will not be able to access it. While there may not be a huge change in page rankings, if you click on a 404 link to an internal web page, the user experience is affected and could result in the user clicking out of the website to find updated, more reliable sources of information. You could see engagement metrics take a hit if you do not address 404 errors, affecting time on-site and increases in bounce rate. Because increasing user engagement on your website is a often a major goal, addressing 404 errors should be a high priority to ensure users are not discouraged from continuing to use your site as a trusted source of pertinent, accessible information and inspiration to travel.
500 server errors happen due to certain subdomains, directories, or file extensions that cause the server to send a message to your browser saying that something has gone wrong. Sometimes this could be an issue with the server itself or something wrong with the file it is trying to display. This is more or less a generic error that something is broken, but the specifics are unclear.
This type of error can potentially cause poor ranking if not corrected in a timely manner. Contact your Account Manager, Web Hosting Provider, or Web Administrator to have the issues investigated.
How a Site Relaunch Affects the Number of Crawl ErrorsIf you're working with Simpleview on a site relaunch, you may be curious about how this relaunch affects your SEO and what we have in store to ensure the smoothest transition possible.
Relaunching a website can cause an exceptional peak in crawl errors. Because of this increase, the best thing to do first is to clear out the 404s. What our SEO team frequently sees are errors created by old static and system pages. These pages were indexed when the old site was live, but they are no longer available.
Clearing out all of these errors from the start removes any of the old "junk" issues and allows you to see the real issues post launch. The crawl errors that you need to address will show up as soon as a couple of days after a site relaunch. Checking back weekly to gather a list of errors and fixing as many as possible with redirects will keep search engines and users happy.
Then Start Cleaning House
Now that we have an understanding of the different types of crawl errors, it is time to start cleaning things up. Think of this as cleaning your room or house. It may seem overwhelming at first, but starting at a single point and working your way out seems to be the most commonly suggested method. We have put together a short and easy list to help you clean out the errors.
Step 1: Identify important content pages (i.e. meetings, things to do, etc.)
First things first. If you have an overabundance of crawl errors, start by marking them all as fixed. This will remove any irrelevant crawl errors to help narrow down your list, and these will not show up again.
Step 2: Remove listings and event pages
Be sure to sort through listings and events pages. When a user deletes or turns off a listing or an event manually from the SImpleview CMS, the page will no longer exist, causing crawl errors when the pages are not found. Events that have a expiration date will be turned off in both the CMS and CRM, removing the page from the site, also causing it to be seen as 404 when the site is crawled and that event is not found.
Step 3: Clean up the rest of the errors
Filter out and remove non-relevant items such as system files. For instance, you may see old code, system pages, and test pages that were left behind. Typically these will be indexed files that were part of old sites that are no longer needed or utilized after a relaunch. Since they are a part of the search index, they will show up even if the files are no longer on your server. Telling Google that they are fixed will make Google assume they are no longer relevant and Google will push them to the bottom of the site files to be ignored.
Step 4: Check back at a later date (1 or 2 weeks later)
Check back weekly to biweekly to see the remaining errors and redirect these accordingly.
Step 5: Repeat steps 1- 4
Repeat as necessary to ensure overall health and ranking.
Keep it Clean
A question we always encounter is "how often should I be checking for crawl errors?"
The simple answer to that is once a month, because Google's bot will scrape a site weekly for errors. Google Search Console only allows you to clear and fix up to 1000 crawl errors a day. If you have a large number of them, it might take several weeks of just clearing them out to get to a manageable number after a launch. Checking and cleaning up the site more often will help improve search rankings. Again, if errors are left unchecked over time, they can negatively affect search rankings.
Just Like Housecleaning, It Gets Easier the More You Do ItNobody wants to individually inspect URL errors, especially on the verge of a complete panic due to thousands of crawl errors showing on your crawl report. However, with experience and repetition of the steps above, you will become quicker at noticing patterns and distinctions in the errors and thus be able to decide which URLs you need to redirect and which can be safely cleared.
Want to learn more about the travel and tourism industry's leading provider of SEM service? Contact your Account Manager or send us a message at firstname.lastname@example.org.