Search engines discover your site by following links, one page at a time. Screaming Frog SEO Spider does the same thing on your desktop, then lays out everything it found in a spreadsheet-style interface: every URL, status code, title, canonical, redirect and link. For more than a decade it has been the first tool many technical SEOs open when they start an audit.
It is also intimidating. The interface has dozens of tabs, hundreds of filters and a configuration menu deep enough to get lost in. The good news is that beginners only need a small part of it to find the issues that matter most: broken links, redirect chains, missing or duplicate titles, accidental noindex tags and pages that nothing links to.
This guide walks you through installing Screaming Frog, configuring your first crawl sensibly, reading the key tabs, and exporting a fix list your developer or content team can act on in 2026.
Key Takeaways
- Screaming Frog crawls your site like a search engine bot and reports technical SEO data for every URL it finds.
- The free version crawls up to 500 URLs; a paid licence removes the limit and unlocks JavaScript rendering, custom extraction, API integrations and saved crawls.
- Use Spider mode for full-site audits and List mode for checking a specific set of URLs, such as a migration redirect map.
- The Issues tab and Crawl Analysis give beginners a prioritised starting point instead of raw data.
- Crawl data shows what is technically on your site; pair it with Search Console data to see what Google actually does with it.
What Screaming Frog Does
The SEO Spider is a desktop application for Windows, macOS and Linux. You give it a starting URL, it requests that page, extracts every link, follows them, and repeats until it runs out of URLs or hits a limit. Its homepage sums up the scope:
“Identify over 300 SEO issues, warnings and opportunities to improve SEO, website health and user experience.”
— Screaming Frog, SEO Spider Tool
Because it runs on your computer, you control the crawl speed, the user agent, whether robots.txt is respected and whether JavaScript is rendered. That flexibility is why it is useful for staging sites, migrations and one-off investigations that cloud tools handle less well. If you want the bigger picture of where a crawl fits, our SEO audit guide covers the full process.
Free vs paid
| Feature | Free version | Paid licence |
|---|---|---|
| URLs per crawl | Up to 500 | No fixed limit (depends on memory and storage) |
| Save and reopen crawls | No | Yes |
| JavaScript rendering | No | Yes |
| Crawl configuration options | Limited | Full |
| Custom extraction and search | No | Yes |
| Google Analytics, Search Console and PageSpeed API integrations | No | Yes |
| Scheduling and crawl comparison | No | Yes |
| AI and semantic similarity features | No | Yes |
The licence is priced per user per year (the site lists £199 in the UK; prices vary by currency, so check the current rate). For a small brochure site, the free version is often enough. For an online store with thousands of filtered URLs, you will need a licence.
Before You Crawl: Three Settings to Check
Recent versions of the SEO Spider default to database storage mode, which writes crawl data to disk rather than holding it all in RAM. The user guide puts it plainly:
“By default the SEO Spider will start in database storage mode with just 2GB of RAM allocated.”
— Screaming Frog, SEO Spider User Guide: General
- Storage mode (File > Settings > Storage Mode on Windows/Linux, or under the application menu on macOS): keep database mode, ideally on an SSD. It autosaves crawls and handles far larger sites.
- Memory allocation (File > Settings > Memory Allocation): raise this if you crawl large sites and your computer has spare RAM.
- Crawl scope: the switcher next to the URL bar lets you crawl a subdomain, a subfolder or all subdomains. Pick the one that matches your audit.
How to Run Your First Crawl: Step by Step
- Download and install the SEO Spider from screamingfrog.co.uk and, if you have one, enter your licence key under the licence menu.
- Confirm the mode is Mode > Spider. This crawls by following links from a start URL.
- Enter your homepage in the "Enter URL to spider" box, using the exact protocol and host your site uses (for example, https://www.example.com/).
- Review configuration under the Configuration menu. Beginners should leave most defaults. Two common changes: enable JavaScript rendering under Spider > Rendering if your site relies on client-side JavaScript, and tick the option to crawl linked XML sitemaps so pages only listed in sitemaps are found too.
- Connect APIs (optional) under Configuration > API Access: Google Search Console, Google Analytics and PageSpeed Insights data will be added to each URL.
- Click Start and watch the progress bar. For large sites, reduce the crawl speed so you do not overload your server.
- Run Crawl Analysis when the crawl finishes. Some filters, such as orphan pages and sitemap issues, only populate after this post-crawl step.
- Open the Issues tab to see a list of problems sorted by type and priority, with a plain-English explanation of each.
The Tabs That Matter Most for Beginners
The top window shows tabs; each has filters on the right-hand Overview panel. The lower window shows details for whichever URL you select, including Inlinks, Outlinks and the Crawl Path Report. Start with these:
| Tab | Filter to check | What it reveals | Typical fix |
|---|---|---|---|
| Response Codes | Client Error (4xx) | Broken internal links | Update the links or redirect the dead URL |
| Response Codes | Redirection (3xx) | Internal links pointing at redirects | Link directly to the final URL |
| Page Titles | Missing, Duplicate, Over X characters | Weak or repeated titles | Write unique, descriptive titles per page |
| Meta Description | Missing, Duplicate | Pages with no summary snippet text | Write unique descriptions for key pages |
| H1 | Missing, Multiple, Duplicate | Unclear page headings | One clear main heading per template |
| Canonicals | Canonicalised, Missing, Non-indexable canonical | Conflicting duplicate signals | Point canonicals to indexable, 200-status URLs |
| Directives | Noindex | Pages blocked from indexing | Remove noindex from pages that should rank |
| Images | Missing Alt Text, Over 100 KB | Accessibility and speed issues | Add alt text; compress large images |
| Content | Exact and Near Duplicates | Thin or duplicated pages | Consolidate or differentiate |
| Sitemaps | Orphan URLs, Non-indexable URLs in sitemap | Sitemap hygiene problems | Remove non-indexable URLs; link to orphans |
Two exports save a lot of time. Bulk Export > Links lets you export every inlink pointing to 4xx or 3xx URLs, so developers know exactly which source pages to edit. Reports > Redirects includes a redirect chains report; our guide to redirect chains explains why flattening them matters.
Five Beginner Audits You Can Do in an Afternoon
1. Broken links and redirects
Filter Response Codes by Client Error, then use the Inlinks tab to find where each broken URL is linked from. Fix the source links rather than relying on redirects.
2. Indexability
The Internal tab shows an "Indexability" column. Sort by Non-Indexable and check the reason (noindex, canonicalised, blocked by robots.txt, redirect). Any important page in this list is a red flag. Confirm what Google sees with the URL Inspection tool.
3. Duplicate URLs and protocols
Look for HTTP and HTTPS, www and non-www, or trailing-slash variants that all return 200. Our guide to choosing one URL version explains how to consolidate them, and duplicate content covers the wider issue.
4. Orphan pages
Connect Search Console and Analytics, tick the option to crawl sitemaps, then run Crawl Analysis. Pages found in those sources but not via internal links appear as orphans. Read more on fixing orphan pages.
5. Internal link depth
The Crawl Depth column and the Site Structure view show how many clicks each page sits from the start URL. Important pages buried four or more clicks deep usually deserve stronger internal links; our internal link audit guide walks through the process.
Spider Mode vs List Mode
- Spider mode discovers URLs by following links. Use it for full-site audits.
- List mode (Mode > List) crawls only the URLs you paste or upload. Use it to check a migration redirect map, test a sitemap, or recheck a batch of fixed URLs.
- Compare mode (paid) compares two crawls to show what changed, which is useful before and after a release or between staging and production.
Common Mistakes Beginners Make
- Crawling too fast. Aggressive crawl speeds can slow down or crash small shared-hosting servers. Lower the speed and crawl outside peak hours.
- Ignoring JavaScript. If navigation or content is injected by JavaScript, a text-only crawl will miss it. Compare a rendered crawl with a non-rendered one.
- Treating every warning as urgent. A title over the pixel limit on a low-traffic tag page is not a priority. Focus on templates and revenue pages.
- Forgetting Crawl Analysis. Several filters stay empty until you run it.
- Crawling the wrong version. Starting on http:// or a non-canonical host can produce a crawl full of redirects.
- Assuming crawl data equals Google data. A crawler reports what is on the page; it cannot tell you whether Google indexed it. Cross-check with Search Console.
Once your basics are fixed, our full SEO checklist and Lighthouse audit guide help you move on to content and performance.
Related Guides
- Search Console URL Inspection Tool: A Complete Guide
- Bing Webmaster Tools: Setup and Key Features
- Internal Link Audit: A Step-by-Step Guide
Frequently Asked Questions
Is Screaming Frog free?
There is a free version that crawls up to 500 URLs with limited configuration. A paid annual licence removes the limit and unlocks features such as JavaScript rendering, saved crawls, custom extraction and API integrations.
Can Screaming Frog crawl JavaScript websites?
Yes, with a paid licence. Enable JavaScript rendering in the Spider configuration so the tool renders pages in a headless Chrome browser before extracting links and content.
Will crawling my site with Screaming Frog harm it?
Not if you use sensible settings. On small or shared servers, reduce the crawl speed so the server is not overloaded, and avoid crawling during peak traffic.
How often should I crawl my website?
Monthly is a good baseline for most sites, plus a crawl before and after major releases, migrations or template changes. Paid users can schedule crawls automatically.
Is Screaming Frog better than Semrush Site Audit?
They serve different needs. Screaming Frog is a highly configurable desktop crawler ideal for deep investigations; cloud audit tools are easier to schedule and share. Many SEOs use both.
Conclusion
Screaming Frog rewards a methodical approach. Configure storage and scope, crawl, run Crawl Analysis, then work through broken links, indexability, duplicates, orphans and depth. Export fix lists by template and recrawl in List mode to confirm the work is done.
If you would rather hand the crawl and the fixes to specialists, our SEO services include a full technical audit with prioritised recommendations, and our technical SEO guide covers what comes next. Get a quote for an audit of your site.
