Sitemaps are a great way of advising Google about the important pages within your website. You may already have a sitemap – that’s great! If you don’t, follow my easy 9-steps guide to creating one.
Let’s make sure that your sitemap.xml file is a healthy one! How? Let’s audit it with Screaming Frog and see what we are dealing with!
Why Audit Your XML Sitemap?
Before start, let’s quickly chat about why this matters. If your sitemap has URLs that are broken, inaccessible, or just unnecessary, it’s like inviting Google to a party at a non-existent address. Not only are you wasting Google’s time (and let’s face it, they’re pretty busy), but you’re also risking your pages not being indexed properly. And if your pages aren’t indexed, you can’t show up in search results.
So, let’s make sure your sitemap is a healthy one.
Open Screaming Frog
I assume you already have your Screaming Frog downloaded? If not, you can download it. Just be aware that the free version (although awesome) does not include all features. Make sure to get a license or contact me and I can audit your sitemap(s) for you. Once ready – fire it up. If you’ve been using it as long as I have (over seven years, but who’s counting?), you probably feel a little thrill every time it opens up. Or maybe that’s just me! Either way, let’s get started.
Navigate to Configuration > Spider > Crawl > XML Sitemaps
Head over to the ‘Configuration’ tab, click on ‘Spider’, then on ‘Crawl’ and finally navigate to ‘XML Sitemaps.’ This is where the magic happens. You’re about to unlock the full potential of your sitemap.
Tick ‘Crawl Linked XML Sitemaps’
Make sure to tick the box that says ‘Crawl Linked XML Sitemaps.’ This tells Screaming Frog to crawl through your sitemap and only your sitemap. No distractions, no going down rabbit holes, just your sitemap(s).
Populate Your Sitemap URL in the ‘Crawl These Sitemaps’ Field
Now, pop your sitemap URL into the ‘Crawl These Sitemaps’ field. If you don’t know your sitemap URL off the top of your head, don’t worry—just ask your CMS, it’ll tell you. You can usually find it by navigating to your robots.txt file (https://example-domain.co.uk/robots.txt) and copying your sitemap URL. If your sitemap URL looks a little bit like: sitemap-index, make sure you copy individual sitemap URL locations, e.g. post-sitemap.xml, page-sitemap.xml and so on.
This step is crucial because it’s how Screaming Frog knows where to start its audit.
Crawl sitemap(s)
Paste your website into the crawler: ‘Enter URL to spider‘, start the crawl and let Screaming Frog do what it does best. While it’s working, you can sit back, relax, maybe grab that cup of coffee. In just a few moments, you’ll have a full report on the health of your sitemap.
Configure the Crawl Analysis
Let Screaming Frog analyse the crawl for you. Before you begin, make sure that the Crawl Analysis Configuration settings have Sitemaps ticked and start the analysis. If you don’t have the paid Screaming Frog version, you will need to do it manually.
Audit the crawl
Does your sitemap contain exclusively
- 200 status code URLs?
- Indexable URLs?
- Important URLs?
If your sitemap contains a lot of broken, invalid, non-indexable URLs, Google will attempt to crawl and index them and you’ll end up with indexation issues in your GSC indexation report, and you’ll be wondering how to resolve them! A little secret, they are not necessarily errors, they are more of an FYI.
But if you want to ensure that your important pages are known to Google, create a new, clean sitemap to avoid potential issues. It can be done in 9 easy steps.
Only Valid URLs Allowed!
And just like that, you’ve ensured that your sitemap contains only valid, accessible URLs. You’ve done more than just tidy up—you’ve optimised your website’s chances of showing up where it matters: in front of potential customers.
What’s Next?
Now that your sitemap is in tip-top shape, you’re not just improving your site’s visibility—you’re paving the way for more traffic, conversions, and sales. And who doesn’t want that?
So, here’s to a cleaner, more effective sitemap—and to you, for taking the time to make sure your site is the best it can be.
