Website Crawling Architecture: Headless Playwright vs Cheerio for SPAs
Over 40% of modern SaaS websites and documentation portals are built as Single-Page Applications (SPAs) using React, Next.js, Vue, or Angular. When a basic HTTP scraper requests an SPA URL, it often receives an empty `<div id="root"></div>` shell because the actual content is rendered client-side via JavaScript.
If an AI chatbot crawler relies purely on static HTML parsers (like Cheerio), it indexes blank pages and fails to answer questions. Here is how SiteMind’s two-tier crawler architecture combines ultra-fast static parsing with headless browser fallback rendering to ingest 100% of website content reliably.
The SPA Scraping Dilemma: Speed vs JavaScript Execution
Crawling engineers face a core performance trade-off between HTTP request speed and client-side JavaScript execution:
| Dimension | Static Cheerio Crawler | Headless Playwright Crawler | SiteMind Two-Tier Engine |
|---|---|---|---|
| Crawl Speed | ⚡ Sub-50ms per page | 🐢 1,500–3,000ms per page | ⚡ Sub-50ms for 90% of pages |
| Server RAM & CPU | Minimal (lightweight text) | Heavy (Chromium processes) | Optimized: Headless only when needed |
| Client-Side JS Render | ❌ Returns empty `<div id="root">` | ✅ Executes JS and renders DOM | ✅ Automatically triggers on empty shells |
| Sitemap Traversal | ✅ Fast XML parsing | ⚠️ High overhead for sitemaps | ✅ Fast parallel sitemap discovery |
The SiteMind Two-Tier Heuristic Fallback Engine
Rather than running slow, expensive headless browsers on every single URL, SiteMind implements an Adaptive Two-Tier Pipeline:
- 1
Tier 1: High-Speed Cheerio Extraction
The crawler fetches the page via lightweight HTTP and parses text using Cheerio in under 50 milliseconds.
- 2
Tier 2: Content Density Heuristic Check
If the extracted meaningful body text is below a minimum threshold (indicating a client-rendered React/Vue SPA shell or hydration failure), the URL is immediately flagged for Tier 2.
- 3
Tier 3: Isolated Headless Playwright Renderer
The URL is dispatched to an isolated headless Chromium worker that waits for network idle (`networkidle`), renders the full dynamic DOM, and extracts the fully-hydrated text.
An adaptive two-tier crawler achieves the blazing speed of static scraping while guaranteeing that modern React, Vue, and Webflow animations are captured completely.
Robots.txt & Respectful Rate Limiting
A professional website crawler must never degrade the target website’s server performance. SiteMind automatically parses and obeys `robots.txt` disallow rules, enforces per-domain request concurrency limits, and applies exponential backoff on HTTP 429 rate limits.
Frequently asked questions
Related Engineering Guides
How AI Knowledge Bases Work (The Ultimate 2026 Guide to Vector RAG)
A complete technical breakdown of how AI knowledge bases turn unstructured website content into sub-second, hallucination-free customer answers.
How to train an AI chatbot on your own website content
A step-by-step, no-code guide to turning your existing website into an AI assistant that answers customers from your own content.
What is a RAG chatbot? (and why it beats fine-tuning for support)
RAG is the technique behind trustworthy AI support. What it means, how it differs from fine-tuning, and why it’s the right choice for a website chatbot.