Website Crawling

Instead of copying the information on your website one by one, you can crawl it with HaveAI. HaveAI analyzes the text on the pages and prepares records. After reviewing and selecting these records, you save them for your assistant to use.

How It Works

You can analyze a single page, discover pages across the whole site, or enter multiple page addresses. Page discovery uses sources such as the sitemap. You select the ones you want to import from among the pages found.

When appropriate, the content is prepared as FAQ, product and service, free-form information or brand information. The records created from the crawl are saved to the relevant sections with your approval.

Crawling and Importing

Open the website crawling section

Open the website crawling section under AI Training in the panel.

Enter the address

Enter the site or page address. When you enter a site address, choose from the offered options either just that page or page discovery across the whole site. You can also start with a sitemap or a list of page addresses.

Select pages

If a page list is shown, select the pages you want to use. A single batch processes up to 50 pages. For more pages, run separate operations.

Approve and start the analysis

If consent about your right to use the content is requested, read it, complete it and start the analysis.

Review the prepared records

Review the prepared FAQ, product, free-form information and brand records. Make necessary corrections; deselect unnecessary records.

Save

Save the records you want to use. Then check the relevant sections and the AI Test answers.

Completing the crawl and saving the records are separate stages. At the end of the operation, make sure you have saved your selection on the review screen.

Check Before Saving the Content

Menus, footers, cookie notices or repeated promotional text may not be useful for the customer's question. Filter these out. Especially check prices, dates, campaign conditions and service scope.

Also review the brand information suggestions carefully. Make sure your company's name, industry and description are correct.

If a Page Is Not Accessible

Password protection, Cloudflare or another security measure may block automatic crawling. In such cases, check the error message. You can upload the content as a document or add it manually to the appropriate section.

When to Use It

If you have a current site that explains your business and working conditions.
If your product and service pages contain enough detail.
If you want to bring relevant information from your help center or blog to the assistant.

You do not have to add every page on your site. Choose the current pages that help answer customers' questions.

When Your Website Changes

A crawl takes the content of the pages you selected at the time of that operation. Do not assume that changes on the site are automatically reflected in the information you saved earlier. Update the relevant records or re-crawl the page and check the result. During re-import, be careful not to leave old and new records on the same topic together.

Redaction of Secret Information

HaveAI masks secrets it can recognize in the content processed for training, such as API keys, passwords and private keys. This control is not a guarantee of detecting every kind of secret or personal information.

Import only pages suitable to share and use in training. If a secret has been published on your website, its masking in training does not mean the information has been removed from the source web page.