Screaming Frog SEO Spider: A Developer's Guide to Technical Crawls in 2025

Screaming Frog SEO Spider is a desktop-based website crawler that has become an essential tool for technical SEO professionals. Unlike cloud-based crawlers, SEO Spider runs locally, giving you full control over crawl configuration, data extraction, and export formats. In 2025, version 20.0 introduced support for Core Web Vitals collection at crawl time, schema extraction in JSON-LD formats, and improved JavaScript rendering via Chromium. This guide covers the configuration patterns and workflows that developers need to run efficient, technically rigorous crawls.

Screaming Frog runs on Windows, macOS, and Linux. For headless CI/CD servers, use the command-line edition (screamingfrogseospider.com, 2025, "SEO Spider System Requirements"). After installation, navigate to Configuration > Spider to set:

• User-Agent: Use Googlebot or Bingbot depending on your target search engine. Googlebot uses a Chromium-based rendering pipeline, so selecting it matches Google's crawling behavior most closely.
• Crawl Speed: Set requests per second based on your server capacity. For production sites, start at 2 requests per second and monitor server response times. Increase incrementally if no degradation is observed.
• Respect robots.txt: Enabled by default. If you need to crawl blocked sections for testing, disable this setting only on staging environments.

Modern web frameworks like React, Next.js, and Angular rely heavily on JavaScript to render content. Screaming Frog's JavaScript rendering engine, introduced in version 19.0 and refined in version 20.0, uses an embedded Chromium browser to fully render pages before analysis.

To enable JavaScript rendering, go to Configuration > Spider > Rendering and select "JavaScript." You can also configure a custom page wait time (default is 10 seconds) for sites that load content asynchronously. The crawler will wait for the specified duration before capturing the rendered HTML (Screaming Frog, 2025, "JavaScript Rendering: How to Crawl JavaScript Websites").

Key considerations for JavaScript rendering:

• Memory usage: Each concurrent thread consumes approximately 150 MB of RAM when rendering JavaScript. For a 10-thread crawl, allocate at least 2 GB of free memory.
• Authentication: Use the "Form Based Authentication" option under Configuration > Access to crawl behind login walls.

One of Screaming Frog's most powerful features is custom extraction. You can define XPath or CSS selector rules to extract any data from a page during the crawl. Examples include:

• Extracting hreflang tags for international SEO audits
• Collecting schema markup (e.g., Product, FAQPage, Article) with JSON-LD extraction
• Pulling Google Analytics or GTM tracking IDs to verify consistent deployment

To configure custom extraction, go to Configuration > Custom > Extraction. Define your rules, and the results populate a dedicated tab in the interface. Export results as CSV or SQLite for further analysis in Python or R (Screaming Frog, 2025, "Custom Extraction: How to Extract Data from HTML Elements").

The "Crawl Comparison" feature allows you to diff two crawls of the same site taken at different times. This is invaluable for regression testing after a site migration, redesign, or content management system upgrade. Load a baseline crawl and a new crawl, and SEO Spider highlights:

• New and missing pages (404s, redirects, new URLs)
• Changes in status codes, meta data, and response times
• Differences in internal linking patterns

Export the diff report to identify regressions before they impact organic traffic.

Screaming Frog includes a command-line interface that can be integrated into automated workflows. The CLI supports all crawl configurations, exports, and comparison features. Use it in CI/CD pipelines to run a crawl after every deployment and compare results against the previous baseline. If critical issues such as broken pages or missing meta data exceed a threshold, fail the deployment and notify the team (Screaming Frog, 2025, "Command Line: How to Run SEO Spider from the Command Line").

Screaming Frog SEO Spider provides granular control and deep technical analysis that cloud-based tools often cannot match. Run a full crawl after every deployment, use custom extraction to validate structured data and tracking codes, and leverage crawl comparison for regression testing. Automate baseline crawls via the command-line interface to integrate SEO checks directly into your development workflow. Consistent use of SEO Spider prevents technical issues from reaching production.

Screaming Frog. (2025). "SEO Spider System Requirements." Screaming Frog. https://www.screamingfrog.co.uk/seo-spider/system-requirements/

Screaming Frog. (2025). "JavaScript Rendering: How to Crawl JavaScript Websites." Screaming Frog. https://www.screamingfrog.co.uk/seo-spider/javascript-rendering/

Screaming Frog. (2025). "Custom Extraction: How to Extract Data from HTML Elements." Screaming Frog. https://www.screamingfrog.co.uk/seo-spider/custom-extraction/

Screaming Frog. (2025). "Command Line: How to Run SEO Spider from the Command Line." Screaming Frog. https://www.screamingfrog.co.uk/seo-spider/command-line/