Adding a website and how crawling works

Your workspace has a single assistant. The first website you add becomes its primary domain and carries the chat widget. Any additional domains you add feed the same shared knowledge base but don’t get a widget of their own, they’re knowledge sources, not separate bots. This keeps one assistant consistent across every property you own instead of fragmenting into several.

SiteMind fetches your pages, reads the rendered HTML, and splits the text into chunks small enough to search accurately. Each chunk gets embedded so the assistant can find the right passage for a given question later. Crawling respects your site’s pace: requests go out with a short delay between them rather than all at once.

Each plan caps how many pages a single crawl can index: 100 pages on Starter, 500 on Growth, 2,000 on Pro, 5,000 on Enterprise. If your site is larger than your limit, the crawl still completes, it just stops at the cap rather than failing outright. See Billing and plans for the full plan comparison.

You can trigger a manual recrawl from a website’s detail page any time your content changes. There’s also an optional automatic schedule: daily, weekly, or monthly. It’s off by default, since re-embedding changed pages uses AI quota, so turning it on is a deliberate choice rather than something switched on for you.

SiteMind’s crawler natively extracts standard server-rendered HTML and includes automatic headless-browser fallback (powered by Playwright). If a page returns an empty JavaScript shell (common in React, Vue, or client-rendered Next.js applications), the crawler automatically renders the full page in a headless browser before parsing text and building knowledge chunks.