Patching LMDB: how we made Meilisearch’s vector store 3× faster

Search-as-a-Service explained: how it works, providers, and more

# Meilisearch updates: March 2026

Welcome to the March edition of Meilisearch updates!  We've been shipping multiple improvements to search relevance, hit a major milestone for distributed search, made exciting AI performance gains, and continued our commitment to security transparency.

## Product updates: what's new at Meilisearch

#### Cloud UI: Generate better AI prompts, right from the Cloud
A prompt generator is now available directly in the Meilisearch Cloud UI. It helps you craft effective prompts for your AI-powered search workflows without leaving the dashboard - less back-and-forth, faster iteration:

![Cloud UI: Generate better AI prompts](https://unable-actionable-car.media.strapiapp.com/Cloud_UI_Generate_better_AI_prompts_fc1c7cff51.png)

#### Cloud UI: Delete documents from the Cloud UI

You can now delete documents from your index directly in the Cloud interface. No API calls needed for quick cleanup or content management tasks:
![Cloud UI: Delete documents from the Cloud UI](https://unable-actionable-car.media.strapiapp.com/Cloud_UI_Delete_documents_from_the_Cloud_UI_64e4dfac52.png)

#### Cloud UI: Index stats at a glance

Index statistics are now visible in the Cloud UI, giving you an immediate view of your index size, document count, and field distribution - useful for understanding your data and keeping indexes healthy:
![Cloud UI: Index stats at a glance](https://unable-actionable-car.media.strapiapp.com/Cloud_UI_Index_stats_at_a_glance_c972c93433.png)


#### Cross-index document hydration to link idexes together

[v1.39](https://github.com/meilisearch/meilisearch/releases/tag/v1.39.0) introduces a new experimental 'foreignKeys' feature that lets you define cross-index relationships, allowing you to 'hydrate' documents with documents coming from other indexes. Store a reference in one index, get back the full linked document at search time. For example, a document storing actor IDs will automatically resolve to full actor objects in the search response - a first step toward JOIN-style queries in Meilisearch.
> What's next? Experimental on non-sharded instances today. Filtering on joined fields + sharded support coming next.

#### More control over your search ranking

Version v1.36 introduces two new ranking rules:

- **`attributeRank`**: Prioritizes matches found in higher-weighted searchable attributes, without factoring in where the match appears within that attribute.
- **`wordPosition`**: Prioritizes matches that appear closer to the beginning of an attribute, regardless of the attribute's weight.

Previously, the `attribute` ranking rule applied both behaviors in a fixed order. Now you can use them independently for finer control, and it's easier to slot custom ranking rules like **price or publication date** exactly where you need them in your ranking pipeline. [Learn more](https://github.com/meilisearch/meilisearch/releases/tag/v1.36.0) about the new ranking rules.

#### Search performance logs are available in the API

Want to know exactly where your search time is going? The new `showPerformanceDetails` parameter returns a breakdown of time spent at each stage of a search query, making it much easier to debug slow queries and understand search performance under the hood. More [here](https://github.com/meilisearch/meilisearch/releases/tag/v1.35.0).

#### Replication is now in the engine!

[v1.37](https://github.com/meilisearch/meilisearch/releases/tag/v1.37.0) shipped **replicated sharding** directly in the Meilisearch engine. This is the most significant step yet toward making distributed, resilient search available to all users - and a direct stepping stone to one-click replication in the Cloud UI.

> Available now under our [Enterprise edition](https://www.meilisearch.com/pricing).
> What’s next? One-click setup in the Cloud UI is on the way - stay tuned.
> 

#### Faster, more reliable AI-powered indexing

- We've **stabilized our HNSW-backed vector store** (Hannoy). The legacy `vectorStoreSetting` experimental feature has been permanently removed. During a dumpless upgrade, the engine will automatically migrate your indexes to the new backend. Newly created indexes will use the HNSW store by default, bringing improved embedding performance and consistency. More [here](https://github.com/meilisearch/meilisearch/releases/tag/v1.37.0).

**Our [v1.38](https://github.com/meilisearch/meilisearch/releases/tag/v1.38.0) release significantly improves embedding indexing performance:**

- **Drastically faster embedding updates**: We eliminated unnecessary full database scans when adding or updating embeddings. Adding embeddings to a large index is now far more efficient.
- **Reliable remote embedders**: A fix for intermittent `connection reset by peer` errors when using remote embedders, improving stability for AI-powered search workflows.
- **Improved task deletion**: A long-standing edge case with task and batch deletion has been resolved.

> _Vector search users: [our v1.38 release](https://github.com/meilisearch/meilisearch/releases/tag/v1.38.0) includes important embedding performance improvements and reliability fixes. The update is backward-compatible._ 
> What’s next? We're also working on an easier way to configuring the chat settings in Cloud UI. Stay tuned for more!
> 

📍 See our updated public roadmap [here](https://www.notion.so/Public-Meilisearch-Roadmap-2cb4b06b651f80868508c2c1409c1b10?pvs=21).

## Community & ecosystem

**Meilisearch & Rust Foundation**

It's [official](https://rustfoundation.org/media/rust-foundation-member-announcement-meilisearch-doulos/): Meilisearch is now a member of the Rust Foundation. Built on Rust from day one, this felt like a natural step, and a way to give back to the ecosystem that makes our engine fast and reliable.

### Events

**FOSDEM 2026 in Brussels.**
    - Our team had a blast at FOSDEM in Brussels this past January - it was great to connect with so many passionate developers from the open source community. If you missed our talk on language support, the [recording](https://fosdempwa.com/event/8323?year=2026) is now available.

**AWS Summit in Paris on April 1st.**
    - Coming to [AWS Summit](https://aws.amazon.com/events/summits/paris/) in Paris on April 1st? It's free to attend - so if you're around, swing by our booth for a demo and a chat with the team.

## Security & transparency

- As a security improvement, the **mini-dashboard now stores API keys in RAM** instead of `localStorage`. More [here](https://github.com/meilisearch/meilisearch/releases/tag/v1.37.0).
- A security researcher recently disclosed a vulnerability to our team through responsible disclosure. We investigated, fixed the issue immediately, and coordinated a public **CVE disclosure**. We're grateful to the community for helping us keep Meilisearch secure. Learn more [here](https://www.meilisearch.com/blog/CVE-update-Jan-2026).

Meilisearch March updates

Establishing more efficient systems and cleaner documentation is key to successful scaling. There are multiple ways to improve documentation organization, and one is to use an index file.

Here’s what this guide covers:

- The definition of an index file with examples, such as index.html or an index used in a search engine.
- How systems store references and file paths to make index files work.
- Where index files are being used, including databases, web servers, file systems, and search engines.
- The pros and cons of indexing.
- How an index file differs from a data file.

Let’s get started!

## What is an index file?

An index file is a structured file that speeds up information retrieval by storing document metadata, or references to the actual data.

It serves as the search system’s lookup layer and eliminates the need for the system to scan every file in detail until it finds the right one.

You’ll find index files in databases and web servers that load an index.html page, search engines that map terms to documents, and in file systems that track file names and paths.

## How does an index file work?

An index file maps a data key or identifier to the exact location of the data in a file. The search system can then read the index file, locate the key or pointer, and jump to the correct position, rather than scanning the entire file to find the right information.

This lookup function is a product of data structures like B-trees and hash tables.

## What are index files used for?

The purpose of index files is to speed up data retrieval, leading to improved system performance and better information organization at scale.

You’ll find index files across databases, web servers, file systems, and search engines. Index-based search is also essential for internal systems such as [intranets](https://www.meilisearch.com/blog/intranet-search-engine), where teams rely on fast, structured access to large sets of internal documents.

### 1\. Databases

Databases use index files to map keys to rows or documents. This lets queries skip full-table scans and jump directly to the right location.

[Indexes](https://meilisearch.com/docs/learn/getting_started/indexes) store pointers, file names, and metadata, making filtering, sorting, and joining much faster.

### 2\. Web servers

Web servers rely on index files such as index.html and index.php to load default webpages. When visitors open a directory path, the server automatically looks for that file name in the index and delivers the home page in the blink of an eye, which drastically improves the user experience.

### 3\. File systems

Index files are used to track file paths and referenced data blocks. Thanks to the structure they provide, the file system can locate a text file or a new file more easily without scanning the entire disk.

### 4\. Search engines

Search engines use index files to connect search terms with specific documents. Again, thanks to the pointers in the index files, retrieval is fast and accurate, even at a large scale.

Modern engines, such as Meilisearch, have developed the capability to power real-time search through structured indexes. The impact is evident in e-commerce sites, internal tools, and even web development projects.

Now, let’s discuss the pros and cons of index files.

## What are the pros and cons of index files?

Teams need to understand both the benefits and drawbacks of index files before deciding whether indexing is the right approach for them.

### Pros

- **Rapid search and retrieval:** Index files reduce or even eliminate the need for full scans.
- **Significant upgrade to the user experience:** When a web page can load a default file name such as index.html, it gives users a consistent, smooth experience. On top of this, search engines can use index structures to quickly return [on-site search](https://www.meilisearch.com/blog/onsite-search) results.
- **Scalability for large datasets:** Index files enable systems to manage growing volumes of information. They help organize text, data, and HTML files without slowing performance.

### Cons

- **Increased workload:** Indexes require storage. More storage means more maintenance, updates, and cleanup.
- **Potential for outdated indexes:** If an index file is not periodically updated, it may reference old file paths or missing content. This leads to inaccurate lookups or failed queries.
- **Security considerations:** Since indexes store information about files and directories, there’s a risk of sensitive data leaking if not properly secured.

## Examples of index files

Let’s look at some examples of technologies that use indexing to organize content and enable quick access:

- **index.html:** The default home page for many websites that is served automatically by web servers.
- **index.dat:** A Windows system file used to store browser history and related metadata.
- **index.xml:** Used for sitemaps or catalog indexes in web design and web development.
- **Database index files (.ndx, .idx):** Essential structures that store keys and pointers for fast queries.

## What is the difference between an index file and a data file?

A data file stores the actual content that the user searches for and that the search system retrieves.

An index file speeds up this retrieval by using identifiers and metadata.

![A table highlighting the key differences between an index file and a data file.](https://unable-actionable-car.media.strapiapp.com/Index_file_vs_Data_file_162cb60a11.png)

Index and data files work together to enable fast retrieval. The index file directs the systems, while the data files provide the content.

This division of roles is what makes large-scale systems and search engines so reliable.

## How do search engines use index files?

Here is how search engines use index files:

1.  They first crawl the content they have, such as webpages, text files, and data files.
2.  They parse these resources, read the metadata and keywords, and identify the structure.
3.  The engines then store all of this inside index files that map terms to the documents where they appear.

The result of this process is a structured map that the engine will use to quickly find relevant documents.

To further [optimize the retrieval](https://www.meilisearch.com/blog/document-boosting), modern engines add ranking rules and filters to the mix.

For example, Meilisearch builds lightweight yet high-performance indexes that support search-as-you-type, typo tolerance, filtering, and rapid updates. As a result, developers not only gain better relevance but also unmatched speed without needing to design custom indexing systems themselves.

## Understanding what an index file is and why it’s essential for efficient data access

Once you start using index files, data access becomes far quicker and more reliable. Thanks to the pointers, file paths, and metadata stored in index files, the system can instantly jump to the correct information without scanning the entire database.

This structured approach improves the user experience and retrieval.


What is an index file and why does it matter in modern computing?

Not all search engines work the same way. Beyond Google, there is a host of specialty search engines built for specific tasks.

Specialty search engines focus on specific domains, such as academic research, jobs, travel, shopping, or local businesses, instead of indexing the entire web.

In this article, we will cover:

- How specialty engines deliver higher search relevance.
- Examples of specialty engines in the real world (Google Scholar, JSTOR, GitHub, etc.).
- Features a specialty search engine should have, like full-text search, fuzzy search, and similarity matching.
- The types of specialty engines that exist.
- Knowing which search engine to use for your specific task, and more.

Let’s get into it!

## What are specialty search engines?

Specialty search engines are search tools designed for specific types of content, rather than indexing the entire web.

While general search engines, such as Google, search for everything, specialty engines focus on one area, such as academic papers, job listings, local businesses, medical research, and more.

They are built with features tailored to their niche, making them more efficient for specialized searches.

Examples of specialty search engines include PubMed for medical studies and Indeed for job searches.

## What are the key features of specialty search engines?

Specialty search engines have important features that distinguish them from general-purpose search engines.

Let’s look at these features:

![Key features of specialty search engines.png](https://unable-actionable-car.media.strapiapp.com/Key_features_of_specialty_search_engines_a095aa93ae.png)

### 1\. Niche focus

Specialty search engines focus on specific content types or industries, rather than attempting to index everything. For example, a legal search engine such as Lexis concentrates solely on case law and legal documents.

By limiting their scope, these engines can dig deeper rather than wider. They understand the unique terminology and context relevant to their field.

### 2\. Specialized filtering

Specialized engines offer filtering tools explicitly tailored to their content type. This makes precise searches much easier.

For instance, academic databases let users filter by peer-review status or publication type. Job sites allow users to filter by salary range, publication date, job location, experience level, and more.

These filters are built based on what people need in that field, making the search experience more seamless.

### 3\. Verified and curated sources

Specialty search engines often work with vetted and high-quality sources.

Medical search engines get information from established journals and health organizations. Legal databases include official court documents and verified legal texts. Academic engines partner with universities and publishers.

This limits misinformation and low-quality content. While the coverage may be limited, the trade-off in credibility and trustworthiness is worth it.

### 4\. Domain-specific ranking

Specialty engines use ranking systems tailored to what matters most in their particular field.

Academic engines may prioritize citation counts and journal reputations. E-commerce engines weigh customer ratings and price competitiveness. Medical databases emphasize study quality and recency.

These algorithms understand context that general search engines might miss.

## What types of specialty search engines exist?

There are various specialty search engine types, each designed for a specific purpose.

![Specialty search engine types.png](https://unable-actionable-car.media.strapiapp.com/Specialty_search_engine_types_93f08f2726.png)

### 1\. Academic search engines

These engines are built for scholars, students, and researchers who need credible information.

Examples of academic engines include:

- **Google Scholar:** Primarily used for indexing academic papers, theses, and journals across all disciplines.
- **Microsoft Academic** (now integrated into Bing)
- **PubMed:** Explicitly focused on medical and life sciences literature.
- **JSTOR:** Providing access to archived scholarly journals.

### 2\. Healthcare search engines

Healthcare search engines help patients and healthcare professionals find reliable medical information:

- **PubMed** is the gold standard for medical professionals because it provides access to millions of biomedical research papers.
- **WebMD** search tools help patients understand symptoms and conditions in plain English.
- **Healthline** aggregates health content with medical reviews.

By prioritizing evidence-based information from verified sources, these engines filter out unproven medical conjectures and focus on peer-reviewed facts.

Medical professionals also use specialized databases, such as Medscape, for clinical decision support.

### 3\. Real estate search engines

Real estate platforms specialize in listing properties and tracking housing market trends:

- **Zillow** and **Trulia** offer home valuations and neighborhood statistics.
- **Realtor.com** pulls directly from Multiple Listing Services for the most current data.
- **Redfin** combines search with agent services.

These engines let you filter by price, square footage, school districts, and dozens of other criteria that matter when house hunting. They have map views, virtual tours, and historical pricing data.

### 4\. Legal search engines

Legal databases provide access to case law and legal analysis for attorneys and researchers.

**LexisNexis** and **Westlaw** are popular options. They offer comprehensive legal databases with advanced search capabilities.

**Google Scholar** also offers a robust and free case-law search.

These platforms understand legal citation formats and can find precedents across jurisdictions.

### 5\. Local business and review search engines

These platforms help you find local services and read customer reviews before you visit:

- **Yelp** is common for reviews of restaurants and local businesses in many markets.
- **Google Maps** integrates business search, navigation, and reviews.
- **Angi** (formerly Angie's List) focuses on home services, such as contractors and plumbers.

These engines prioritize proximity and ratings, understanding that you want ‘good pizza near me,’ not just information about pizza. They add customer reviews, photos, hours, and contact information.

Now, let’s really see the comparison between a specialty search engine and a general search engine.

## What is the difference between specialty and general search engines?

Specialty search engines serve specific niches with curated content, whereas general search engines scan the entire web.

![Specialty vs. general search engine.png](https://unable-actionable-car.media.strapiapp.com/Specialty_vs_general_search_engine_39bbc0df1f.png)

The key difference boils down to depth versus breadth.

General engines like Google, Bing, or Yahoo want to be your one-stop shop for everything, which works great for everyday queries.

However, when you need something specific, such as peer-reviewed cancer research, entry-level marketing jobs in Finland, or homes in Kansas under $200k, then you should use specialty engines.

They understand the terminology, sources, and needs of their particular field in ways general engines simply cannot match.

## How do specialty search engines work?

Specialty search engines focus their crawling, indexing, and ranking on a specific niche rather than the entire web.

Unlike general-purpose engines that index everything, these vertical search engines use targeted crawlers that visit only relevant sources.

Job boards crawl company career pages, academic engines pull from scholarly databases, legal engines get information from law journals, and shopping engines focus on e-commerce sites.

Some specialty engines are built on developer-friendly backends. A good example is Meilisearch, which offers fast indexing, customizable ranking rules, and typo-tolerant search. It is perfect for creating domain-specific search experiences without building everything from scratch.

Many specialty engines also function as metasearch engines, combining results from multiple sources within their niche. For example, a flight search engine queries multiple airlines simultaneously, comparing options in real time. A full [similarity search](https://www.meilisearch.com/blog/similarity-search) enables this engine to match your preferences with comparable flights even when exact criteria are not available.

Also, [fuzzy search](https://www.meilisearch.com/blog/fuzzy-search) helps users find what they need even when they do not use the correct spelling.

## Who should use specialty search engines?

Specialty search engines are valuable for anyone looking for better [search relevance](https://www.meilisearch.com/blog/search-relevance) in specific domains.

Here are the users who benefit the most from using specialty search engines:

- **Students:** When writing a research paper or thesis, Google Scholar and JSTOR should be the go-to tools. They deliver peer-reviewed sources with proper citations. [Full-text search](https://www.meilisearch.com/blog/how-full-text-search-engines-work) also helps find specific quotes or concepts within lengthy academic papers.
- **Job seekers:** Indeed and LinkedIn Jobs combine thousands of listings with salary filters and company reviews.
- **Professionals:** Doctors use PubMed to research treatment options. Patients use WebMD to try to understand their symptoms. Lawyers use LexisNexis and Westlaw to find case precedents and statutes. Financial analysts rely on the Bloomberg Terminal for market data. HR professionals use specialized recruiting platforms to find qualified candidates faster.

## What are the advantages of using specialty search engines?

Specialty search engines offer various benefits that general search engines cannot match:

- **Greater precision:** Users get precisely what they are looking for. For example, searching for ‘Python’ on a coding site will bring up programming resources, not information about the snake.
- **Saving time:** Users don’t have to click through ten pages of results. Specialty engines cut straight to quality sources in their field.
- **Advanced filtering options:** Specialty search engines can filter by various highly specific criteria. Academic search engines can filter academic papers by publication date or citation count. Job search engines filter by salary range and remote options. A product search engine filters by price and customer ratings.
- **High-quality sources:** Content from specialty search engines is curated from the most relevant, most high-quality sources.

## What are the limitations of specialty search engines?

Here are the biggest drawbacks of using specialty search engines:

- **Limited scope:** Specialty engines only cover their specific niche, so if you need to cover different topics, you will need multiple specialty engines.
- **Smaller databases:** Even the best specialty engines cannot match Google's massive index. If something is newly published, it might not yet be in their database, especially for smaller platforms with limited crawling resources.
- **Less frequent updates:** Specialty engines tend to update their data less frequently than general search engines.
- **Learning curve:** Each search engine platform has its own interface and filters. What works on one engine fails on another. Legal databases can be particularly confusing because they have complex search operators that take time to master.

## How do you choose the right specialty search engine?

Select a specialty search engine that aligns with your specific needs by matching its focus, features, and reliability:

- **Match your niche:** Start by identifying your exact need. The engine you choose should specialize in exactly what you want your users to search for, and not just be indirectly related.
- **Evaluate database:** Assess the search engine's coverage. Does the academic engine index the journals you need? Does the job search engine cover your industry and location? Note that bigger is not always better; however, you want sufficient coverage in your area of interest.
- **Test the search features:** Experiment with the filters and search options. Can you narrow the results to what you need? Academic researchers need citation filters, job seekers need salary ranges, and shoppers need price sorting. The right engine should have filters that actually matter for the searchers.
- **Check update frequency:** How often does the platform refresh its data? Real-time updates matter more for stock prices and flight bookings than for historical research. Ensure the engine updates frequently enough to meet your needs.
- **Assess user experience:** Is the interface intuitive or frustrating to use? Can you figure out how to search without a manual? Some engines are outdated, while others feel modern and responsive.
- **Build your own solution:** If existing engines do not meet your needs, consider building a custom one. Meilisearch offers flexibility, speed, and straightforward implementation for organizations needing a niche search solution. It is developer-friendly, with customizable ranking and fast indexing, making it perfect for teams creating specialized search experiences from scratch.

## Final thoughts on what specialty search engines are

Specialty search engines understand the terminology and context of their fields in ways a general search engine never will.

They have some limitations, but for their specific purposes, they are unbeatable. The key is knowing when to use them and picking the right one for your task.

What are specialty search engines and why do they matter?

If you’ve ever wondered why specific results rank higher even when they don’t match a query exactly, you’ve encountered proximity search in action.

Modern search engines don’t just check whether terms exist in a document – they evaluate how close those terms appear, in what order, and under which constraints.

Proximity search is the mechanism that bridges the gap between simple keyword matching and semantic retrieval.

Here is what we’ll cover in our guide:

- The definition of proximity search and its role in improving search ranking.
- How it works based on distance, word order, and other specified distance rules.
- Why you can’t overlook the importance of proximity search for user satisfaction and result quality.
- The most common proximity operators and how they differ from phrase search.
- Practical application of proximity search for Google, PubMed, and Boolean searches.
- A step-by-step guide on how you can implement a proximity search algorithm.
- The most common mistakes developers must avoid, as well as the limitations of proximity search.
- How engines like Meilisearch complement proximity search to deliver better full-text results.

Let’s get into it.

## What is proximity search?

Proximity search is a technique that finds documents where the search terms appear close to each other, rather than anywhere in the text.

For example, a user is looking for documents containing ‘error handling.’

Document A contains the following sentence: ‘This guide explains **error handling** in distributed systems.’ While document B includes this one: ‘This guide explains **errors** in distributed systems and later discusses **handling** user input.’

Proximity search will prefer document A and rank it higher in the results because the words ‘error’ and ‘handling’ are closer together – right next to each other, in fact – in the text.

This helps search engines and databases return results that make more sense than just keyword matching.

## How does proximity search work?

Proximity search works based on a simple principle: how close search terms appear to each other within a document or page.

Instead of matching keywords anywhere in the full text, the search engine measures the distance between the search terms or the number of words between them. If the terms fall within a specific range, the result will rank higher.

Different systems support proximity search in various ways. They can rely on proximity operators, adjacency operators, or custom syntax such as the NEAR operator, quotation marks, or a tilde to indicate distance.

Thanks to these rules, relevance tuning becomes more predictable and consistent across various search engines or databases.

For related concepts, developers often combine proximity rules with [fuzzy search](https://www.meilisearch.com/blog/fuzzy-search) techniques for handling misspellings and variations.

Now, let’s take a look at why proximity search plays such an important role in relevance.

## Why is proximity search important?

Proximity search is so important because it improves [search relevance](https://www.meilisearch.com/blog/search-relevance). It ranks documents in which search terms appear close together rather than scattered throughout the page.

As a result, there’s a significant decrease in useless or irrelevant matches, especially where word order affects meaning.

Academic IR evaluations and health sciences search systems like PubMed and the National Library of Medicine highlighted that keyword matching is easily outperformed by proximity matching. PubMed’s [own search methodology and proximity operators](https://pubmed.ncbi.nlm.nih.gov/help/) explain how enforcing term closeness in titles and abstracts improves precision and reduces irrelevant matches in extensive biomedical collections.

Search engineering experts agree that proximity is one of the strongest signals for [semantic closeness](https://www.meilisearch.com/blog/similarity-search) without requiring full semantic models. This conclusion is supported by recent research such as [_Text Retrieval in Restricted Domains by Pairwise Term Co-occurrence_](https://csimq-journals.rtu.lv/csimq/article/view/csimq.2024-41.05), which demonstrates that incorporating proximity-based co-occurrence information significantly improves retrieval effectiveness compared to keyword-only matching across multiple text types.

Now, let’s look at the operators that enable proximity control.

## What are proximity operators?

Proximity operators are special tools used in proximity searching. They control how far apart search terms may appear within a document by defining the maximum number of words between terms.

You’ll find these operators most commonly used in advanced search platforms, such as databases, academic engines, PubMed, and so on.

The most common proximity operators include:

- **NEAR / N:** Finds terms within a specified distance; for example, _term1 NEAR/5 term2_ is a match when both appear within five words of each other.
- **AROUND():** Mostly used by Google to set the number of words that are allowed between terms.
- **~ (tilde):** For example, _climate change~3_. This means that the terms must appear within three words of each other.
- **ADJ:** An adjacency operator that requires terms to appear next to each other.
- **W / WITHIN:** This proximity operator matches terms within a defined word span in specific systems.

These different proximity operators ensure that your search is both flexible and precise.

## How does proximity search differ from phrase search?

The main difference between proximity and phrase search is that proximity search allows terms to appear within a specified distance.

Phrase search is more strict. It requires the exact phrase in the exact word order. It uses double quotes or quotation marks to enforce the strict matching.

For example, let’s say that a user is searching with the phrase ‘data consistency.’

Document A contains the sentence: ‘This paper discusses **data consistency** models in distributed databases.’

Document B includes the sentence: ‘This paper discusses the **consistency of data** across distributed databases.’

Even though both documents talk about the same idea, phrase search won’t treat document B as a relevant result, while proximity search will.

![A table showcasing the key differences between proximity search and phrase search.](https://unable-actionable-car.media.strapiapp.com/Proximity_search_vs_phrase_search_083b91e2e9.png)

This added tolerance for variation in proximity search can lead to more context-rich matches. However, in domains where accuracy is non-negotiable, you should stick to phrase search.

Let’s take a look at use cases where proximity search genuinely makes a difference.

## What are the main use cases for proximity search?

The real value of proximity search lies in use cases where context and relationships are crucial to the quality of results.

The most common use cases of proximity search are:

- **Academic research:** Using databases like PubMed, NLM, EBSCO, and ProQuest.
- **Legal search:** Proximity search helps lawyers or law students find cases in which concepts appear near each other, enabling them to improve their interpretation and citations.
- **AI and semantic search:** AI models combine keyword logic with AI ranking, so the relevance tuning requires the inclusion of proximity search.
- **Recruitment search:** It helps recruiters surface resumes where skills appear closely together rather than scattered.
- **Enterprise and government search:** Proximity search plays a huge part in ensuring accuracy in [large document repositories](https://www.meilisearch.com/blog/enterprise-search) across business and government systems.

Now, let’s move on to the most prominent search engines and databases that use proximity search. First up is Google.

## How do you use proximity search in Google?

Google supports proximity search through the AROUND() operator. It instructs the engine to find search terms within a specified distance of each other.

The syntax is: term1 AROUND(number of words) term2

This will help you retrieve tightly related concepts rather than broad keyword matches.

Here’s how it works step-by-step:

1.  First, you will enter your two terms.
2.  Then, you insert AROUND() between them.
3.  After the operator, place the number of words allowed in parentheses.
4.  You run the query and review the results where the terms appear close together.

**Examples:**

- _Batman AROUND(3) Joker_

Or, if you want a slightly wider distance:

- _Batman AROUND(5) Gotham_

These queries rank results higher when terms appear within the specified range.

Feeling like a master searcher yet? Let’s talk about proximity search in PubMed.

## How do you use proximity search in PubMed?

PubMed is fond of the ADJ operator, which finds terms within a specified distance and within the same word order.

For reference, the syntax is: term1\[Field\] ADJ\[number of words\] term2\[Field\]

This is part of PubMed’s NLM search engine and is crucial if you want to narrow results in health sciences research.

Step-by-step:

1.  First, you enter your two search terms.
2.  Next, add the ADJ plus the maximum number of words allowed between them.
3.  To move on, you will specify the field, such as \[TIAB\] for title/abstract.
4.  Run the search to filter studies where terms appear close together.

**Example:**

_coronavirus\[TIAB\] ADJ3 vaccine\[TIAB\]_

This finds articles in which the word ‘coronavirus’ appears within three words of ‘vaccine’ in the title or abstract.

Now, let’s look at proximity search for Boolean searches.

## How do you use proximity search in Boolean searches?

Proximity search fits naturally into Boolean logic by adding distance control to operators like AND, OR, and NOT. These proximity operators will define how close the terms must appear.

Many systems also use NEAR, NEAR/n, or WITHIN to combine proximity with Boolean expressions.

You’ll notice that this allows more precise search techniques than using AND alone, which matches terms anywhere in the [full text](https://www.meilisearch.com/blog/how-full-text-search-engines-work).

For example:

_batman NEAR/5 joker AND gotham_

This finds documents in which ‘Batman’ appears within five words of ‘Joker’ and ‘Gotham’ appears anywhere in the document.

Another example:

_superhero NEAR/3 origin NOT "secret identity"_

Here, proximity is combined with exclusion through NOT to refine results. We would get results with ‘superhero’ and ‘origin’ within three words of each other, provided that the document does not contain the phrase ‘secret identity’ anywhere in its text.

Now, let’s walk through the steps of implementing a proximity search algorithm.

## How do you implement a proximity search algorithm?

The outline of our implementation here includes:

1.  Cleaning the documents we will be working with (parsing and tokenizing them).
2.  Building a document index.
3.  Matching queries to the documents based on defined distances.

We will also explain how (and why) to use Meilisearch to simplify this process.

### 1\. Parse documents and record term positions

First things first, you will normalize and tokenize each document, then store the term positions.


```python
_from collections import defaultdict_

_from typing import Dict, List_

_def index_document(doc_id: str, text: str) -> List\[tuple\[str, int\]\]:_

_"""_

_Tokenize text and return (term, position) pairs._

_"""_

_tokens = text.lower().split()_

_return \[(term, pos) for pos, term in enumerate(tokens)\]_
```


### 2\. Build a positional inverted index

Next up, you will build a positional inverted index: term → doc_id → \[positions\].


```python
_PosIndex = Dict\[str, Dict\[str, List\[int\]\]\]_

_def build_positional_index(docs: Dict\[str, str\]) -> PosIndex:_

_"""_

_Build a positional inverted index from {doc_id: text}._

_"""_

_index: PosIndex = defaultdict(lambda: defaultdict(list))_

_for doc_id, text in docs.items():_

_for term, pos in index_document(doc_id, text):_

_index\[term\]\[doc_id\].append(pos)_

_return index_
```


### 3\. Match queries and compute distances

It is crucial that, at query time, you compare positions and check whether they fall within the max distance range.

```python
_def within_distance(positions_a: List\[int\],_

_positions_b: List\[int\],_

_max_dist: int) -> bool:_

_"""_

_Return True if any pair of positions is within max_dist._

_Assumes positions_a and positions_b are sorted._

_"""_

_i, j = 0, 0_

_while i < len(positions_a) and j < len(positions_b):_

_diff = positions_a\[i\] - positions_b\[j\]_

_if abs(diff) <= max_dist:_

_return True_

_if diff < 0:_

_i += 1_

_else:_

_j += 1_

_return False_
```

You would wrap this in a function that, for a given pair of query terms and a _max_dist_, scans the positional index and returns matching document IDs.

In a custom engine, you’d have to do all of this on your own: parsing, indexing, and distance logic.

With Meilisearch, you eliminate most of this complexity. In Meilisearch, proximity is a built-in ranking rule: you define _searchableAttributes_, tune _rankingRules_ (including _proximity_), and let Meilisearch handle proximity scoring under the hood, instead of manually managing each positional index.

The following are the common mistakes teams make with proximity search.

## What are common mistakes with proximity search?

These are the most common mistakes we’ve noticed when working with proximity search:

- Using the wrong proximity operators or adjacency operators. As a result, the system interprets the search terms literally rather than by a specified distance, which undermines the purpose of proximity search.
- Setting the number of words too narrowly or too broadly. This can either bring back irrelevant search results or none at all.
- Ignoring word order in systems that require a specific sequence can return no results.
- Incorrectly mixing proximity searching with Boolean operators, overriding constraints in advanced search queries.

With these issues covered, we can now look at the broader limitations of proximity search itself.

## What are the limitations of proximity search?

Different systems use different syntax, proximity operators, and adjacency operators. This creates problems when you want to use queries across platforms.

Some databases restrict the maximum number of words allowed per query. In fact, some even require controlled vocabulary or strict word order to return accurate citations.

Extensive full-text indexes can slow down performance because the engine must track positional data for every search term. In advanced workflows, these constraints can even clash with Boolean operators.

## Why proximity search matters for modern information retrieval

Proximity search is your best option for strengthening relevance by rewarding documents in which the search terms appear close together in the full text.

It reduces noise and improves search accuracy. Plus, the advanced search techniques it supports play a considerable role in domains such as health sciences, legal research, and large government or academic databases.

Proximity search: Improve relevance & deliver smarter results

## The challenge isn't connecting to an LLM. It's figuring out what people actually mean.

Last week at [MCP Connect Days](https://luma.com/bj6kgvxh) in Paris, I watched a room full of engineers nod in unison. The speaker from The Fork was describing their biggest challenge building AI-powered search: not the LLM integration, not the API calls, not even the infrastructure. The hard part? Understanding what users actually want.

When someone types _"Italian restaurant near Bastille for under 30€ with outdoor seating,"_ the system needs to decompose that into: _cuisine type_ (Italian), _location_ (Bastille), _price range_({'<'}30€), and amenity (outdoor seating). Four distinct filters, zero explicit structure. Just vibes.

This is the **intent understanding problem**. And after talking to dozens of teams building conversational search interfaces, I'm convinced it's where most implementations quietly fail.

> **What is intent understanding?** Intent understanding is the process of translating natural language queries into structured search parameters. It means extracting the right keywords, filters, and facets from conversational input so a search engine can return relevant results.
> 

## The gap between natural language and structured queries

Search engines are remarkably good at what they do. Give them a query with the right filters, and they'll return relevant results in milliseconds. The technology is mature, battle-tested, fast.

The problem is that humans don't speak in structured queries.

Coming back to our Italian restaurant search: the system correctly identified four distinct parameters from a single sentence. But that's the simple case. Now scale this complexity. Real queries include negations (_"not too touristy"_), preferences (_"preferably with a terrace"_), comparisons (_"somewhere nicer than last time"_), and context that requires memory (_"that place my colleague mentioned"_). Each of these requires a different parsing strategy.

Most teams building natural language search underestimate this translation layer. They focus on the LLM integration, assuming the hard part is getting GPT to respond. But LLMs are commoditised. The translation from natural language to structured search? That's where the differentiation lives.

## How The Fork tackles intent parsing

At [MCP Connect Days](https://luma.com/bj6kgvxh), [The Fork's](https://www.thefork.fr/) engineering team shared their approach to building a ChatGPT app for restaurant discovery. Their challenge perfectly illustrates the intent understanding problem.

A typical query: _"Where can I take my parents for their anniversary? They like French food, somewhere quiet, budget around 80€ per person."_

Breaking this down:

- **Occasion context** → anniversary (implies upscale, romantic)
- **Party composition** → parents (older demographic, accessibility matters)
- **Cuisine** → French
- **Atmosphere** → quiet
- **Budget** → ~80€/person

The Fork's system needs to translate these signals into filters their search infrastructure understands. But _"quiet"_ isn't a database field. Neither is _"anniversary-appropriate."_ They've built a sophisticated interpretation layer that maps conversational signals to searchable attributes.

Their key insight: the quality of results depends almost entirely on how well you understand the query, not how powerful your LLM is. A perfectly tuned GPT-4 with mediocre intent parsing will lose to a simpler model with excellent query understanding.

## The rise of conversational commerce

This isn't just a technical curiosity. Conversational commerce is reshaping how people shop and discover products online.

For online retailers, users now expect to interact with search the same way they'd ask a knowledgeable shop assistant. _"I need running shoes for flat feet, nothing too flashy, under 150€"_ is a reasonable query for a human sales associate. It should work for your search too. The companies getting this right are seeing real results: better query understanding means higher conversion rates, lower bounce rates, and customers who actually find what they're looking for.

For classified marketplaces like [Leboncoin](https://www.leboncoin.fr/), the challenge multiplies. _"Looking for a used iPhone, good condition, around Lyon, under 400€"_ requires parsing: _product category_ (smartphones), _brand_ (Apple), _model family_ (iPhone), _condition_ (good), _location_ (Lyon area), and _price_ ({'<'}400€). Each maps to a different filter in their search system. Get one wrong, and you've lost the sale.

The pattern repeats everywhere. Marketplaces, travel sites, content platforms. All building custom intent parsing layers. All solving the same fundamental problem.

## Why building this from scratch is painful

Every team I've talked to that's built conversational search has essentially reinvented the same wheel. They all build:

1. **Entity extraction** – pulling out product types, brands, attributes
2. **Filter mapping** – converting extracted entities to search parameters
3. **Ambiguity resolution** – handling cases where intent is unclear
4. **Validation** – ensuring the generated query actually makes sense

This is significant engineering effort. And it's duplicated across every company building conversational interfaces.

The build-vs-buy decision here is stark. You can spend months building a custom query parsing layer, tuning it for your domain, handling edge cases, maintaining it as your schema evolves. Or you can choose infrastructure that handles this natively.

## How Meilisearch solves intent understanding

This is exactly why we built the [chat route](https://www.meilisearch.com/docs/reference/api/chat-completions) directly into Meilisearch.

Instead of requiring every developer to build their own intent parsing layer, Meilisearch handles the natural language to structured query translation natively. The search engine itself understands how to interpret conversational input.

Here's what this looks like in practice. When you send a natural language query to the [chat route](https://www.meilisearch.com/solutions/rag), Meilisearch:

1. **Parses the intent** – identifies what the user is searching for versus filtering by
2. **Maps to your schema** – automatically routes extracted entities to the appropriate filterable attributes
3. **Generates the query** – produces an optimised search with the right combination of keywords and filters
4. **Returns results** – with full context about how the query was interpreted

The key insight: your search engine already knows your data schema. It knows which fields are filterable, what values exist, what combinations make sense. This context is gold for intent understanding.

When Meilisearch sees _"red derbies under 100€"_ and knows your schema has _"color"_ (with _"red"_ as a value), _"category"_ (with _"derbies"_), and _"price"_ (numeric, filterable), it can make intelligent parsing decisions. Building this at the search engine level is the right abstraction because that's where the schema knowledge lives.

This approach differs builds upon semantic search, which focuses on understanding meaning and similarity. Intent understanding goes further: it's about decomposing a query into actionable search parameters. You need both for truly conversational search, and Meilisearch combines them.

## Why this matters for the AI search ecosystem

The [MCP Connect Days](https://luma.com/bj6kgvxh) conference made one thing clear: everyone is building conversational interfaces. ChatGPT apps, Claude integrations, MCP servers, custom assistants. The race is on.

But most of these implementations will be mediocre. Not because the LLMs are bad, but because intent understanding is treated as an afterthought. Teams will ship, users will try queries that don't quite work, and the magic of AI-powered search will feel more like a gimmick than a feature.

The teams that win will be the ones who invest in robust query understanding. Either by building sophisticated custom layers (expensive, time-consuming) or by choosing infrastructure that handles this natively.

At the conference, I heard the same frustration repeatedly: _"The LLM part was easy. Making it actually understand what users want? That took months."_

It doesn't have to.

## The bottom line

Conversational search is coming to everything. The user expectation is set: if I can ask ChatGPT anything, why can't I ask your product search the same way?

But the gap between that expectation and reality is intent understanding. It's the unsexy, difficult, often-ignored layer between natural language and structured results.

You can build this yourself. Many teams do. But if you're evaluating search infrastructure for conversational use cases, ask one question: **does it understand what my users mean, or do I have to teach it?**

The search engine that answers "yes" to the first question is the one worth betting on.

---

## Frequently Asked Questions

**What is intent understanding in search?**

Intent understanding is the process of interpreting what a user actually wants from their search query. It involves extracting keywords, identifying filters (like price ranges or categories), and understanding implicit requirements. For example, turning "cheap red sneakers" into a search for sneakers + colour filter (red) + price sort (ascending).

**What's the difference between semantic search and conversational search?**

Semantic search focuses on understanding the meaning of words and finding conceptually similar results. Conversational search goes further: it parses natural language queries into structured parameters, handles follow-up questions, and maintains context. Semantic search might understand that "sneakers" and "trainers" are related; conversational search understands that "under 100€" is a price filter.

**How do you parse natural language search queries?**

There are two main approaches: building a custom NLP pipeline (entity extraction, intent classification, slot filling) or using a search engine with built-in query understanding. Custom pipelines offer control but require significant engineering investment. Native solutions like [Meilisearch's chat route](https://www.meilisearch.com/solutions/rag) handle parsing automatically using your existing schema.

**Why is intent understanding harder than LLM integration?**

LLM APIs are well-documented and relatively straightforward to integrate. Intent understanding requires domain-specific knowledge: understanding your product taxonomy, mapping natural language to your filter schema, handling ambiguous queries, and maintaining quality as your catalogue evolves. It's an ongoing engineering challenge, not a one-time integration.

---

*Thomas Payet is COO and co-founder at Meilisearch. He attended MCP Connect Days in Paris in February 2026.*

Why intent understanding is the hardest part of AI-powered search (and how to solve it)

**[Scenario](https://www.scenario.com/) is an enterprise AI platform that enables studios and brands to train custom models on their visual libraries, create workflows, and leverage agents in their creative process.**

By building a "creative memory" unique to each customer, Scenario generates unlimited on-brand assets, from images to video, 3D, and beyond, while maintaining full governance over creative IP. With over **dozens of million fully indexed assets** and **35 million monthly searches**, the platform requires search infrastructure that matches the speed and intelligence of its AI capabilities.

> _“Our enterprise customers need to find the right asset from millions of possibilities in milliseconds. Whether by keyword, visual similarity, or both, Meilisearch lets us deliver that while we focus on what makes Scenario unique: building AI that truly understands each customer's creative DNA. 
Meilisearch is really in the right spot - both because of its product and their team. They listen to us and adapt the roadmap to help us push the boundaries of what’s possible in AI powered search.”_
— Hervé Nivon, Co-Founder & CTO at Scenario.

## Overview: Scenario

Scenario’s modern platform enables businesses and creators to generate any kind of media using state-of-the-art generative AI models and streamline content creation. Scenario's community-led library provides a common language for teams to more easily access generative AI tools for content, including image, audio, video, and 3D. 

Trusted by leading studios and brands, including Unity, Tripledot Studios, InnoGames, Fastory, and Alan, the platform manages a vast and growing volume of creative assets requiring a modern Asset Data Management (ADM) system to maintain an always-on platform ready for the generative era.

## The challenge: scaling search for the AI era

Before adopting Meilisearch, Scenario relied on OpenSearch on AWS to power their asset discovery. As the platform scaled toward tens of millions of indexed assets and 35 million monthly searches, the team needed search infrastructure purpose-built for AI-native workloads:

- **Performance at scale:** Delivering instant search across millions of assets required more than tuning a general-purpose engine. Users expected real-time results across both text and visual queries.
- **Cost and operational overhead:** The AWS-managed setup demanded constant manual optimization, diverting engineering time from core AI and product development.
- **Growing user expectations:** As the asset library expanded, users needed faster, more relevant discovery, making search a product-critical capability rather than a backend utility.

## The solution: a modern hybrid search engine

Scenario’s team needed a solution that offered the speed of a dedicated vector database with the rich feature set of a traditional search engine. After evaluating Pinecone and Qdrant, Scenario chose Meilisearch based on a recommendation from the founders of Hugging Face, thanks to Meilisearch’s superior support for hybrid search.

Meilisearch provided:

- **A developer-first experience:** Clear documentation and a straightforward onboarding process allowed the Scenario team to move fast.
- **Enterprise readiness:** SOC2 compliance and seamless AWS Marketplace integration streamlined billing and ensured baseline security.
- **Hybrid search:** The ability to combine Scenario's proprietary embeddings with keyword and multi-modal (image) search in a single interface made it easier for users to navigate the vast library of assets.

## Implementation: building a scalable search infrastructure

Scenario’s implementation initially focused on creating a seamless discovery experience, leveraging multi-modal search to allow users to navigate public and private asset libraries via both text and visuals.
- **Continuous indexing:** the system handles a high-velocity environment where documents are constantly being ingested, updated, and deleted, ensuring the library remains real-time.
- **Custom embeddings:** rather than using default models, Scenario leverages its own AI capabilities to generate text and image embeddings.
- **Vector search integration:** these embeddings are then indexed in Meilisearch, powering an "Exploration" feature that lets users find similar assets by directly clicking on images.

The result: scaling volumes with minimal maintenance.
## 
The transition to Meilisearch transformed Scenario’s search into a core product advantage: 


>Enterprise teams can now discover and reuse existing assets through visual similarity search, reducing duplicate asset creation and accelerating production cycles.
_— Hervé Nivon, Co-Founder & CTO at Scenario._


- **Dramatic reduction in support load:** support requests related to search performance have virtually disappeared. Scenario’s team now focuses mainly on fine-tuning vector results.
- **Exceptional scale:** Meilisearch effortlessly handles upwards of 35 million searches per month, with 80% of those being complex vector searches.
- **Productivity gains:** by offloading search infrastructure to Meilisearch, Scenario’s engineering team has been able to focus entirely on their core generative AI and API features.
- **Increased user engagement:** the “exploration” feature, powered by visual vector search, has become a primary driver of time spent on the platform.

## Conclusion: pushing the boundaries of AI search

Scenario's partnership with Meilisearch is entering its next phase. As the platform's library of AI models grows past 200 and counting, search must go beyond asset discovery: intelligently matching each task to the most appropriate model becomes just as critical as finding the right asset.

The team is building toward AI agents that autonomously surface relevant reference assets during the creative process and recommend the best-fit model for any given task, delivering context-aware results in real time.

By choosing a partner that values adaptability and direct feedback, Scenario has built a search foundation that not only meets the demands of today’s tens of millions of documents but is ready for the next stage of scaling.


Powering the generative AI era: how Scenario scales asset discovery with Meilisearch

Designing effective typeahead search for faster, smarter UX

Document indexing enables quick searching and retrieval of digital documents. In this guide, we will show you how to build an indexing system that supports fast search and lets you access reliable information quickly.

Here’s what we’ll cover:

- What exactly document indexing is and how documents become searchable.
- The importance of indexing and why it is critical for accessing reliable information.
- The types of industries that rely on indexing.
- The workflow behind document indexing, with explanations for each step.
- The key difference between scanning and indexing, and when you should use one over the other.
- Common indexing methods, such as field-based indexing, metadata indexing, and full-text indexing.
- The unique set of challenges that you can expect to encounter in document indexing.
- How you can build a real-time indexing system with the help of Meilisearch.

Let’s start with the first step: what document indexing actually is.

## What is document indexing?

Document indexing is a structured organization of digital documents based on information such as keywords, metadata, and other relevant fields. The sole purpose is to ensure that these documents can be found quickly when needed.

Document indexing is especially important for industries such as healthcare, where specific documents (patient records, reports, contracts, etc.) need to be immediately pulled from vast amounts of data.

## Why is document indexing important?

Document indexing is important because it simplifies information retrieval. Without it, teams would waste a lot of time having to review digital documents themselves, without automation, and being hampered by manual indexing errors.

When you eliminate human error in favor of information-based indexing based on metadata, names, dates, or other elements, you improve your bottom line. Information is instantly available, and no one spends more time than necessary finding exactly what they need.

So which industries benefit most from document indexing? Let’s see that below.

## What industries benefit most from document indexing?

Many industries rely heavily on large volumes of digital files and would greatly benefit from a fast and simple means to access relevant information.

These industries include:

- **Legal:** Law firms use indexing to sort and organize contracts and case files to keep up with court records. It also helps them not lose any case-critical documents.
- **Healthcare:** Hospitals use document indexing to keep track of patient charts, lab reports, and insurance forms.
- **Finance:** Banks can’t compromise on invoices, account numbers, and other digital information, so they rely on document indexing for records management.
- **Education:** The need to manage enrollment forms, transcripts, research materials, and attendance sheets makes document indexing essential for educational facilities.
- **Government:** Agencies often index permits, licenses, and public records so the public, as well as government offices, can access them.

## How does document indexing work?

Document indexing works by identifying key information in digital documents. This information is then used to store these digital documents in an organized [indexing system](https://meilisearch.com/docs/learn/indexing/indexing_best_practices). For instance, we can use keywords, tags, and metadata to make indexed documents easy to search.

![A graph listing the steps in the inner workings of document indexing.](https://unable-actionable-car.media.strapiapp.com/How_Document_Indexing_Works_26490bc137.png)

### 1\. Document ingestion

The first step is to gather all documents you want indexed. They can be collected from websites, external databases, internal knowledge bases, etc.

### 2\. Text processing

The text is cleaned and prepared so that the search engine can understand it. This can include processes such as lowercasing letters, removing stop words such as ‘and’ and ‘the’, stemming or lemmatization (reducing words to their base form), and similar.

### 3\. Metadata extraction

Key information, or metadata, is identified in each document. Key information can be specific keywords, names, barcodes, dates, or similar.

### 4\. Index building

Finally, the search catalog is built based on the extracted metadata. When a user types in a query, the system pulls up relevant documents in the blink of an eye.

## How is document indexing different from document scanning?

Document indexing and document scanning work together, but they solve different problems in information management.

The role of scanning is to convert paper documents into digital documents. Indexing is the step that comes after scanning.

![Document Indexing vs Document Scanning.png](https://unable-actionable-car.media.strapiapp.com/Document_Indexing_vs_Document_Scanning_d84112d388.png)

In short, scanning digitizes documents, and indexing makes them findable.

## What are common methods of document indexing?

Document indexing can be done in a variety of ways, depending on the types of documents, workflow needs, and the volume of digital files being managed.

1.  **Manual indexing:** A person enters document information and organizes everything by hand. Manual indexing offers control but it takes too much time, and there’s always the possibility of human error.
2.  **Metadata-based indexing:** As the name suggests, the documents are organized using metadata elements, such as file name, authors, dates, etc.
3.  **OCR-based indexing:** Text is extracted from paper documents or scanned documents using optical character recognition. It helps turn paper into digital documents that can be indexed automatically.
4.  **Automated indexing:** Algorithms or systems apply tags or [full-text indexing](https://www.meilisearch.com/blog/how-full-text-search-engines-work) without any manual work. This works well when dealing with large volumes of document-based data.

## What types of information are used for indexing?

Using specific data points makes it easier to organize files and filter results, just as arranging your closet by color makes it easier to find items you want.

Common types of information used for document indexing include:

- **Keywords:** The phrases that indicate the main subject of the document.
- **Metadata:** Specific details of the document, such as the file name, author, title, etc.
- **Document type:** The classification labels of documents, such as invoices, reports, forms, etc.
- **Identifiers:** Values like an invoice number, account number, or customer ID.
- **Dates:** Creation, modification, or event dates for easy sorting.
- **Tags or categories:** Assigning tags or categories helps group related files together.
- **Extracted text:** The type of content that is captured through full-text indexing or OCR for deep search.

Based on this set of data points, it becomes relatively easy to retrieve accurate information from the indexing system.

Let’s look at how indexing supports information retrieval.

## What is the role of indexing in information retrieval?

Indexing is the backbone of information retrieval because it provides a structured map of digital documents. By storing keywords, different identifiers, and metadata, it groups relevant information together.

Document indexing eliminates the need for manual scanning and strengthens the search functionality across databases and document management systems. If your indexing process is clean, even extensive collections cannot hinder retrieval.

Now, let’s examine the challenges that go hand in hand with document indexing.

## What challenges are common in document indexing?

Document indexing can become overwhelming when organizations have to manage large volumes of digital files and rely on unoptimized processes. These issues affect retrieval accuracy and long-term information management.

Here are some of the most common challenges teams face:

- **Inconsistent tagging:** Different teams label documents differently. As a result, the indexing system becomes unreliable. The fix for this is a set of standardized tags, categories, and vocabulary.
- **Human error:** Manual entry leads to mistakes such as missing fields, typos, and mislabeling. The best way to fix this is to shift toward automated indexing.
- **Poor metadata quality:** Weak or incomplete metadata makes document retrieval difficult. Clear rules for required fields, such as dates or invoice numbers, significantly improve the structure.
- **Unscannable or low-quality documents:** When the quality of scanned documents is poor, even the best OCR cannot make them readable. Better scanning tools for data extraction should help here.

Let’s see how automated document indexing works with Meilisearch in real environments.

## How automated document indexing works with Meilisearch

Meilisearch eliminates the need for manual work, replacing it with fast, optimized, searchable indexes in real time. It extracts key document attributes and organizes them into an optimized indexing system.

Here are the main steps to indexing your documents using Meilisearch:

### 1\. Import and prepare your documents

The first step is to load the documents into your workflow. These documents can come from various sources, such as APIs, storage buckets, scanned documents, or even existing databases. Meilisearch accepts JSON, so there’s no problem in pushing the field titles, dates, tags, etc.

```json
_curl \\_

_\-X POST 'http://localhost:7700/indexes/documents/documents' \\_

_\-H 'Content-Type: application/json' \\_

_\--data-binary @documents.json_

```

### 2\. Extract key fields and metadata

Extraction is key for identifying attributes within each file, such as metadata, keywords, categories, and so on.

For example, if a document comes from OCR, the extracted text instantly becomes a part of full-text indexing.

Meilisearch categorizes files and enriches indexed documents to improve [relevant information retrieval](https://www.meilisearch.com/blog/latent-semantic-indexing).


```json
_{_

_"id": "doc_01",_

_"title": "Insurance Claim Form",_

_"date": "2024-01-19",_

_"account_number": "298231",_

_"content": "Full OCR text goes here..."_

_}_
```

### 3\. Index documents and enable search features

Indexing uploads the enriched documents into Meilisearch. Immediately, the engine builds indexes and applies features, enabling various search functionalities such as typo tolerance and synonyms.

```json
_curl \\_

_\-X PATCH 'http://localhost:7700/indexes/documents/settings' \\_

_\-H 'Content-Type: application/json' \\_

_\--data '{_

_"searchableAttributes": \["title", "content"\],_

_"filterableAttributes": \["date", "account_number"\]_

_}'_
```

### 4\. Apply custom ranking and optimize relevance

Now comes the step that defines ranking rules based on the priorities you set, such as dates, status, or file names. Different types of documents require different priority levels.

```json
_curl \\_

_\-X PATCH 'http://localhost:7700/indexes/documents/settings' \\_

_\--data '{_

_"rankingRules": \[_

_"words",_

_"typo",_

_"proximity",_

_"attribute",_

_"sort",_

_"exactness"_

_\]_

_}'_
```

### 5\. Keep the index updated in real time

Lack of index updates can cause bugs and inconsistencies. Meilisearch supports incremental updates so that any new or modified documents become searchable immediately. This takes away time-consuming manual updates and keeps business operations running smoothly.

Thanks to these real-time updates, document management systems always reflect the most current data.

```json
_curl \\_

_\-X POST 'http://localhost:7700/indexes/documents/documents' \\_

_\--data '\[{ "id": "doc_01", "status": "updated" }\]'_
```

## Document indexing in modern information systems

Modern systems rely heavily on document indexing because they require up-to-date information and accurate retrieval. Scaling is also a big reason automated indexing has flourished in recent years.

Through effective indexing, you can optimize all document-related processes for your business, including search accuracy, fast access across workflows, and document management.

Meilisearch latest news and company updates