Website Crawling

Instead of uploading documents, you can point BHOMY to your website and let it crawl and extract content automatically.

Adding a URL

1. Go to your agent's Knowledge Base tab.
2. Click "Add Website".
3. Enter the URL you want to crawl.
4. Configure crawl settings.
5. Click "Start Crawl".

Crawl Settings

• Single Page — Crawl only the specified URL.
• Full Site — Follow all internal links and crawl the entire website.
• Custom Depth — Crawl up to N levels deep from the starting URL.

Sitemap Auto-Detection

BHOMY automatically looks for a sitemap.xml file on your website. If found, it uses the sitemap to discover and prioritize pages for crawling, ensuring comprehensive coverage.

Include/Exclude URL Patterns

Fine-tune which pages are crawled:

• Include patterns — Only crawl URLs matching these patterns (e.g., /products/*, /support/*).
• Exclude patterns — Skip URLs matching these patterns (e.g., /admin/*, /login).

Crawl Scheduling

• One-time — Crawl once and keep the content.
• Daily — Re-crawl every day to capture changes.
• Weekly — Re-crawl once a week.

Managing Crawled Pages

View all crawled pages in your Knowledge Base. For each page you can see the URL, title, word count, and last crawl date. Remove individual pages or trigger a manual re-crawl at any time.