Live Web Crawler & Technical SEO Spider
Automated robots.txt & sitemap.xml (.gz included) discovery, parallel HEAD link verification, weighted Dice near-duplicate detection, E-E-A-T trust scoring, and interactive site graph visualization.
Scope Filters
Response Codes
Content Type
Live Crawl Statistics
Data Size & Telemetry
No Crawl Data Yet
Enter a domain or URL above and click Start Crawl, or load a previously saved .json crawl snapshot.
How the Enterprise Web Spider Works
Multi-layered technical SEO auditing bringing desktop spider capabilities directly to your browser with zero installation.
1. Sitemap & Robots.txt Discovery
Before crawling begins, the engine parses /robots.txt rules and recursive XML/Gzip sitemaps to queue deep and orphan URLs.
2. Parallel DOM & HEAD Verification
Extracts titles, meta descriptions, canonicals, hreflang, OpenGraph, JSON-LD, and analytics while running parallel HEAD checks on embedded images and external links.
3. Dice Duplication & E-E-A-T Scoring
Detects near-duplicate pages via weighted Dice similarity, scores E-E-A-T trust signals, and renders the full site link topology on an interactive HTML5 canvas.
Frequently Asked Questions About SEO Crawler
How does near-duplicate content detection work?
Once crawling finishes, the engine compares all internal 200 OK HTML pages using a weighted Dice similarity coefficient across tokenized titles (35%), meta descriptions (35%), H1 headings (20%), and word count ratio (10%). Pages exceeding the threshold (default 85%) are flagged.
Are decorative images with empty alt="" flagged as errors?
No. Following WCAG accessibility standards, an explicit empty alt="" attribute is recognized as a valid decorative image; only <img> tags completely missing the alt attribute are flagged.
Can I save a crawl session and reload it later?
Yes. Use the Save (.json) button in the command bar to download the complete crawl state, link graph, and issue report, and reload it anytime via Load (.json).