Skip to main content

Crawler: learn from your website

Crawler brings website content into your assistant's knowledge workflow. A submitted URL is a request to learn content, not proof that the content is ready to answer questions.

Before you crawl​

Choose public pages you own or are authorized to use. Start with service details, product information, FAQs, policies, and contact information. Exclude login pages, customer records, duplicate variants, and expired offers.

Collect and inspect content​

  1. Select the intended workspace and assistant.
  2. Add your website through the creation flow or Knowledge controls available in your account.
  3. Review the source status. Wait for processing to finish; investigate items needing attention.
  4. Check that the learned material includes the actual answer, not just navigation or a page title.
  5. Ask a source-specific question in DIRTbox.

A page that needs a login, blocks automated access, or renders its essential content only after interaction may not supply usable knowledge. Do not bypass restrictions. If you own the content, use an approved text or file source offered by your workspace instead.

Refresh after changes​

Update the original page first, then refresh the affected source with the available Knowledge controls. Test changed answers again. Do not assume a scheduled refresh or a whole-site crawl is enabled merely because the first URL was accepted.

Understand usage​

Pages learned, retained sources, and retained storage are different limits. Long documents can consume multiple page units under the upcoming catalog. Read usage and billing before estimating capacity.

Next: Organize sources in BrainVault.