SEO Spider Tutorials

How To Crawl JavaScript Websites

Render web pages using the integrated Chromium WRS to crawl dynamic, JavaScript rich websites and frameworks, such as Angular, React and Vue.js.

How To Crawl Large Websites

Crawl large websites by switching to database storage mode, increasing memory and by configuring the crawl to extract the data you need.

How To Automate Crawl Reports In Looker Studio

Learn how to set-up fully automated Google Looker Studio crawl reports to monitor site health, detect issues, & track performance.

How To Audit Core Web Vitals

Analyse which pages pass or fail Google's Core Web Vitals assessment using field data from CrUX to improve page speed.

How to Crawl with ChatGPT

Open up a world of possibilities by communicating with OpenAI’s ChatGPT during a crawl.

How To Perform A Web Accessibility Audit

Learn how to run an automated accessibility audit of a site using the integrated AXE engine for automated accessibility validation.

How To Test & Validate Structured Data

Audit and validate structured data against Schema.org specifications, and Google’s search feature requirements to enable special search result features.

Web Scraping & Custom Extraction

Scrape any data from the HTML of a page using CSS Path, XPath and regex to enhance a crawl, such as author name, comments, shares or more.

How To Audit Redirects In A Site Migration Using The SEO Spider

Learn how to crawl a list of URLs, follow redirect chains, identify the final target & identify errors & non-indexable pages.

How To Use N-Grams

Learn how to use N-grams in the SEO Spider for text analysis, on-page alignment and even finding internal linking opportunities at scale.

How To Use Custom Search

Find anything you want in the HTML or text of a website, such as old brand names, keywords for internal linking, or verifying tracking tags.

Robots.txt Testing In The SEO Spider

View URLs blocked by robots.txt, the disallow lines & use the custom robots.txt to check & validate a site's robots.txt thoroughly, and at scale.

What Is Link Score?

Find out more about our Link Score algorithm which calculates the relative importance of URLs based on the internal linking of a site.

How To Use The SEO Spider In A Site Migration

Learn how to use the SEO Spider's crawling and features in a site migration to work more efficiently, and improve chances of success.

How To Bypass Geo IP Redirection In A Crawl

Learn how to set your preferred country using web form authentication to crawl a site when it uses automatic GEO IP redirection.

How To Perform A Cookie Audit

Perform a cookie audit for compliance to identify all cookies that are being automatically set without explicit user consent at scale.

How To Analyse Link Position

Analyse the position of links found in a crawl to identify and fix links faster, and improve internal linking for users and search engines.

How To Perform A Parity Audit

Learn how to perform a JavaScript and mobile vs desktop parity audit to uncover differences and potential SEO issues.

How To Automate The URL Inspection API

Collect URL Inspection API data in bulk, work with the 2k limit, & automate URL Inspection data reporting to monitor indexing.

How To Use The SEO Spider For Broken Link Building

Scale broken link building by using the SEO Spider to crawl a list of seed URLs, and discovering broken external links on them for outreach.

How To Debug Invalid HTML Elements In The Head

Identify invalid HTML Elements in the <head>, view metadata affected, and debug what's actually causing the issue.

Configure X Virtual Framebuffer

When running headless on Linux, a display is needed if you want to use JS rendering. You can use a Virtual Frame Buffer to simulate having a display attached.