Websites often contain a lot of information that may not be relevant to the AI. When you add webpages to the knowledge base, you can exclude certain sections of the page from being crawled and indexed by the AI. This can be useful for removing irrelevant information as well as preventing unnecessary characters from being counted towards your plan’s limit.

​
How to exclude sections of a webpage

  1. Go to the Knowledge Base.
  2. Click the New Data Source or Add new button.
  1. Select Website, Sitemap, or URL list as the data source type.
  2. Click the Advanced Settings button.
  1. You can exclude sections by entering the HTML classes or IDs of the elements you want to exclude. You must press Enter after each class or ID to add it to the list.
You can also select the HTML tags you want to remove, such as header or footer. You can now go ahead with entering the website, sitemap, or list of URLs that you want to add to the knowledge base. The crawler will exclude the sections you specified.