Website sources

Website sources let BoundBot discover, fetch, and classify pages from your site. Product-detail pages become structured Products; informational pages become searchable FAQ-style knowledge. When you add a source, BoundBot starts discovering and scraping pages in the background automatically.

​

Add a source

Open Knowledge -> Websites and click Add Website. You can add a source in three modes:

• Crawl links: discover pages by following internal links
• Sitemap: import URLs from your sitemap
• Individual link: index one specific page

If you paste a direct sitemap.xml URL, BoundBot detects it and treats it as a sitemap source automatically.

​

Advanced options

For crawl and sitemap sources, you can also set:

• Include path prefix to limit the crawl to part of a site
• Exclude path prefix to skip sections such as /admin
• Max links to control crawl size
• Auto Recrawl if you want the source refreshed on a schedule

After you save the source, BoundBot opens the source detail page and starts the first crawl automatically. While the source is running, the page updates live with discovered links, crawl progress, failures, and stored size.

​

How pages are classified

BoundBot classifies each successfully crawled page by what the page actually contains:

• Product: a specific purchasable product or service detail page. Structured product data is preferred when the site provides it.
• FAQ: informational pages such as help articles, policies, category pages, blog posts, and general landing pages.

For product pages, BoundBot can store the canonical website URL, description, image, SKU, category, tags, regular price, sale price, currency, and real stock quantity when those values are present. A sale keeps the higher amount as the regular price and the lower amount as the discount price. Recrawling updates the linked Product or FAQ record. If the page’s content changes enough to switch classification, BoundBot removes the old linked record and creates the correct type instead. Product-classified pages count toward the product limit on your plan. If the team does not have catalog capacity, that URL fails instead of silently turning a product page into FAQ content; upgrade or free product capacity, then retry it.

​

Manage a source

Each source card shows:

• current status
• number of discovered links
• number of scraped links
• total stored size
• last crawl time
• last error, if one occurred

Open a source to inspect every URL, filter by status, search links, and run background actions:

• Fetch & Crawl to rediscover links and crawl pending pages in one run
• Fetch Links to start the same background discovery-and-crawl workflow from the source card or detail page
• Crawl Pending to scrape only links that are still pending
• Retry Failed to retry links that failed on the last pass
• Retrain Agent to mark every discovered link for recrawl and scrape the whole source again
• Crawl Selected to crawl only the links you choose from the table

Deleting a source or an individual crawled link also deletes the Product or FAQ record created from that URL.

​

Watch your limits

The top summary shows:

• website source count
• total links
• stored data size
• crawl credit cost per page

These limits vary by plan. If you hit a limit, BoundBot prompts you to upgrade before you add more sources. If a crawl runs out of credits partway through, the remaining links stay pending so you can resume later.

​

Best practices

• Start with the smallest useful section of your site.
• Exclude duplicate or low-value pages such as admin routes, login pages, and legal archives if they do not help customers.
• Use Fetch & Crawl after major navigation or sitemap changes.
• Use Retrain Agent when existing pages changed heavily and you want the full source scraped again.
• Review errors early so broken pages do not quietly degrade your answers.
• Review the Products page after crawling a storefront to confirm pricing, image, and stock are represented as expected.

​

Related pages