Introducing Documentation Crawler — Meilisearch Cloud

In this era of information overload, the efficiency of documentation is determined not just by its quality but also by how easily users can find what they're searching for. Enhancing your documentation with advanced search capabilities can significantly elevate user experience.

Meilisearch Cloud's Documentation crawler offers just that, bridging the gap between vast content and user-friendly searchability.

What is a crawler?

A web crawler, sometimes known as a 'spider' or 'bot', is a digital tool that roams the vast expanse of the internet, indexing and cataloging content for search engines. This ensures that the most relevant pieces of information are presented to users during their searches.

What is Meilisearch’s Documentation crawler?

Meilisearch's Documentation crawler is not just a traditional crawler; it also functions as an efficient scraper. While it extracts vital data from web pages, it simultaneously honors the inherent structure of the documentation, ensuring precise and structured indexing.

This system allows users to integrate search functionality into their documentation, ensuring that their audiences can swiftly locate the information they need.

How to use the documentation crawler?

Using the Documentation Crawler is a breeze; all it requires is a Meilisearch Cloud project and your existing documentation's URL.

Setting up your project

First of all, you need a running project in Meilisearch Cloud. Once you have created a project, navigate to your project's index list and create a new index.

Crawling your content

When prompted to import your documents, choose 'Documentation Crawler.'

Then, input or paste your documentation's URL and click on ‘Next’.

Click on 'Run' to start crawling your documentation content and push it into a Meilisearch instance. This process may take several minutes.

Post-crawling verification

Once the status updates to 'ready', you can return to your project's overview and access the 'search preview'. Click on the project’s URL or select the 'Preview search' option from the top menu.

Perform a few searches to ensure that your documentation has been indexed correctly. This step is especially useful for understanding the schema of your documents.

Adding a search bar to your documentation

Once you've successfully indexed your documentation using Meilisearch's Documentation Crawler, the next step to enhancing user experience is integrating a search bar into your documentation.

Return to your project and navigate to your index settings. From there, select the 'tools' tab and then choose 'crawler'. This will reveal the crawler settings and associated resources.

The UI starter kit helps you add a functional search bar to your documentation with just a few lines of code.

You are now fully equipped to add a seamless search experience to your documentation!

_____

Stay in the loop by subscribing to our newsletter. To learn more about Meilisearch's future and help shape it, take a look at our roadmap and come participate in our Product Discussions.

For anything else, join our developer community on Discord.