Preparing Redirects

Process Overview

The following process is how we prepare redirect mapping files.

These are used when existing web sites are updated and their URLs change.

Having redirects in place is important for two reasons:

  • SEO - ensure that Google doesn't drop important pages from its index
  • User experience - ensure that visitors don't get 404 error pages

Overview

  • Prepare a list of the source (original) URLs
  • Prepare a list of the target (new) URLs
  • Map these in a spreadsheet ie with a row for each original URL and where it will map to as a new URL
  • Convert the mapping into a format to use on web site
  • Upload to site
  • Test
  • Check after go-live

Prepare list of source (or original) URLs

The list of source URLs is prepared by pulling from:

  • ScreamingFrog (to crawl the site)
  • Google Analytics (to check all the pages that have had visits)
  • Google Search Console (to get a list of all the pages that Google has indexed)

Process

  • Use each of the tools to export out a list of URLs, ideally with the URL and the Page Title
  • Add these all into a spreadsheet (ideally in Excel)
  • Check parameters on the URLs for ones that are unnecessary (eg any ?page= type parameters)
  • Use Excel's remove duplicates tool to clean the list into a unique set
  • Order by URL

Estimated time

  • Depends on size of the site, but typically for:
  • smaller sites (eg less than 1000 pages) the list can usually be prepared in under 2 hours
  • larger sites (eg 10K+ pages) can take a day to prepare the full source list

Tips:

  • Leave ScreamingFrog running overnight for larger sites
  • Use the Exclude Parameters config in ScreamingFrog to exclude crawling duplicate parameter driven URLs
  • Add everything into Google Sheets initially, and then export out to Excel to use the Remove Duplicates
  • Use the Advanced Search in Google Analytics for removing parameters

Prepare list of target (or new) URLs

Process

  • Use ScreamingFrom to crawl the new site (ie usually on a staging site)
  • Get any other export of pages from the CMS
  • Prepare in Excel as a unique set of URLs (similar process as the Source list)

Estimated Time

  • Depends on the CMS and access to the Staging site
  • Ideally the new site CMS has a simple export of all the URLs and this can be used as the main list
  • Even better if the CMS export also includes Page Titles in the export
  • If no export options in the CMS, then using ScreamingFrog is usually a similar process to preparing the source list
  • Assume less than 2 hours for smaller sites (less than 1000 pages) and upto a day for larger sites (10K+ pages)

Prepare the mapping file

Now that you have both the source and target lists you can prepare the mapping file.

In Excel or Google Sheets:

  • Have columns with the source URL and page title
  • Have columns with the target URL and page title
  • Go through each row in the source URL list and then match to a target URL
  • This is usually very manual and done visually by using the URL (ie if friendly URLs) or the Page title (if unclear from the URL)
  • You should be able to match 80-90% of the URLs visually
  • Some URLs (perhaps many) may have the same URL - these can be excluded later
  • For any that you aren't sure of, you will need to check with the client and confirm
  • Once all the URLs are mapped, go through and ignore any where the URL hasn't changed eg /about might be the same on both the original site and the new site
  • Export out only the mappings for the URLs that have changed, and thus need the redirects
  • Prepare the redirect mapping file in the format required (using Excel formulas for the actual lines)

Priorities:

  • In some cases you may not be able to identify a match
  • Check back in Google Analytics to see if the source URL had a lot of traffic (ie whether it is an important page or not)
  • If it had a lot of traffic, discuss with the client about a suitable page to redirect to
  • If hardly any traffic, just set a redirect to a top level page (could even be the Home page as a default)
  • As a minimum, the top 20% of source pages should all have very specific redirect targets - since these are the most important pages, and we don't want to lose Google juice for them

Estimated Time

  • For source and target lists that have friendly URLs, assume approx 1 hour per 100-200 URL mappings
  • If the source and target sites have a good architecture this can reduce the time
  • For sites with non-friendly URLs, matching purely on page titles can be time consuming, assume 60 pages per hour
  • Once the mapping rows are all in place, preparing the actual file is usually no more than 30 mins using formulas in Google Sheets or Excel

Upload the redirect mapping file onto the server or site

Once the mapping has been prepared, it needs to be exported out into a format for the client's website hosting.

This could be any of a number of options, including:

  • Prepare as a set of lines to be added to their .htaccess file (if they are running on Apache, eg this is what we'd usually do for WordPress sites on WP Engine)
  • Prepare as an XML config file for .NET based sites (see more details here)
  • Prepare as a CSV to import into a CMS (eg for adding into HubSpot or other systems)

Estimated Time

  • Depends on the client's setup
  • For smaller clients, aim to be put in touch with their IT Manager or key server manager early to organise the process for uploading the file
  • Will also need to have confirmed with them the format for the file
  • Allow at least 2 hours of liaising with them and updating the file
  • For larger clients, especially Government, allow at least a day (8 hours) for various meetings, email threads and requirements checking. You may need to liaise with multiple parties eg a Security Officer, who then connects you with the Infrastructure team. In addition to the time you spend liaising, be prepared for upto a two week turnaround for the file to actually be uploaded. ie you should have the redirect file ready at least 2 weeks before testing commences on the staging site

Test the redirects

Once the mapping file is uploaded onto the staging site, test all the URLs by:

  • Testing the source URL (but changing the domain to be the staging site) eg if the source URL is: https://myorig.com/my-page-1 and the new destination on staging is: https://mynew.com/page-1-overview, then try testing this URL: https://mynew.com/my-page-1 to check that it redirects to https://mynew.com/page-1-overview
  • You can use ScreamingFrog to prepare the list of URLs to test, and then run it against the staging site
  • Make note of any URLs that didn't redirect as a 301, and instead returned a 404
  • Check if they need to be updated in the redirect mapping file

Things to check after go-live

After go-live, keep an eye on any errors visitors encounter:

  • Check in error logs for the server
  • Check in Google Search Console for error URLs
  • Update the redirect mapping file as required

Other Notes

It is also helpful to prepare an XML sitemap on the new site, and upload this to Google Search Console.

This will help Google to crawl the new site layout and index the new pages.

As Google hits the old URLs and sees they are redirecting to new pages (that have been indexed already, thanks to the XML sitemap) it will remove the old URLs from its results and replace them with the new URLs

What to expect

If the URL mapping is all in place, you can expect to keep most of your Google juice and not suffer in rankings.

Google has indicated that redirects can have up to a 3-5% impact on reducing the Google juice, so at most you would likely see is a 3% drop in organic initially due to redirects.

However, other factors will also impact (both positively and negatively):

  • If content on a page has changed significantly, this will impact what the page ranks for (much more so than the redirect side-effect)