AI-powered hybrid search is in closed beta. Join the waitlist for early access!

Go to homeMeilisearch's logo
Back to articles
04 Sept 2023

Introducing Documentation Crawler — Meilisearch Cloud

Introducing Documentation Crawler to help users integrate search functionality into their documentation

Carolina Ferreira
Carolina FerreiraDeveloper Advocate @ Meilisearch@CarolainFG
Introducing Documentation Crawler — Meilisearch Cloud

In this era of information overload, the efficiency of documentation is determined not just by its quality but also by how easily users can find what they're searching for. Enhancing your documentation with advanced search capabilities can significantly elevate user experience.

Meilisearch Cloud's Documentation crawler offers just that, bridging the gap between vast content and user-friendly searchability.

What is a crawler?

A web crawler, sometimes known as a 'spider' or 'bot', is a digital tool that roams the vast expanse of the internet, indexing and cataloging content for search engines. This ensures that the most relevant pieces of information are presented to users during their searches.

What is Meilisearch’s Documentation crawler?

Meilisearch's Documentation crawler is not just a traditional crawler; it also functions as an efficient scraper. While it extracts vital data from web pages, it simultaneously honors the inherent structure of the documentation, ensuring precise and structured indexing.

This system allows users to integrate search functionality into their documentation, ensuring that their audiences can swiftly locate the information they need.

How to use the documentation crawler?

Using the Documentation Crawler is a breeze; all it requires is a Meilisearch Cloud project and your existing documentation's URL.

Setting up your project

First of all, you need a running project in Meilisearch Cloud. Once you have created a project, navigate to your project's index list and create a new index.

Step 1 of Create Index form: Enter the index name.

Crawling your content

When prompted to import your documents, choose 'Documentation Crawler.'

Step 2 of Create Index form: Import documents via crawler or file upload.

Then, input or paste your documentation's URL and click on ‘Next’.

Step 2 of Create Index form: Import documents using a crawler by pasting the website URL

Click on 'Run' to start crawling your documentation content and push it into a Meilisearch instance. This process may take several minutes.

Setup interface for the Crawler with a 'Run' button.

Post-crawling verification

Once the status updates to 'ready', you can return to your project's overview and access the 'search preview'. Click on the project’s URL or select the 'Preview search' option from the top menu.

Project overview

Perform a few searches to ensure that your documentation has been indexed correctly. This step is especially useful for understanding the schema of your documents.

Search preview displaying results for the query 'releva' with highlighted matches.

Adding a search bar to your documentation

Once you've successfully indexed your documentation using Meilisearch's Documentation Crawler, the next step to enhancing user experience is integrating a search bar into your documentation.

Return to your project and navigate to your index settings. From there, select the 'tools' tab and then choose 'crawler'. This will reveal the crawler settings and associated resources.

Setup interface for the Crawler with the 'UI starter kit: Doscsearch' option.

The UI starter kit helps you add a functional search bar to your documentation with just a few lines of code.

You are now fully equipped to add a seamless search experience to your documentation!

_____

Stay in the loop by subscribing to our newsletter. To learn more about Meilisearch's future and help shape it, take a look at our roadmap and come participate in our Product Discussions.

For anything else, join our developer community on Discord.

How to add AI-powered search to a React app

How to add AI-powered search to a React app

Build a React movie search and recommendation app with Meilisearch's AI-powered search.

Carolina Ferreira
Carolina Ferreira24 Sept 2024
Build your Next.js Shopify storefront with Blazity

Build your Next.js Shopify storefront with Blazity

Learn to build a Shopify storefront with Next.js and the Blazity commerce starter.

Laurent Cazanove
Laurent Cazanove19 Aug 2024
Meilisearch 1.8

Meilisearch 1.8

Meilisearch 1.8 brings negative keyword search, improvements in search robustness and AI search, including new embedders.

Carolina Ferreira
Carolina Ferreira07 May 2024