If you’ve ever wondered why specific results rank higher even when they don’t match a query exactly, you’ve encountered proximity search in action.

Modern search engines don’t just check whether terms exist in a document – they evaluate how close those terms appear, in what order, and under which constraints.

Proximity search is the mechanism that bridges the gap between simple keyword matching and semantic retrieval.

Here is what we’ll cover in our guide:

- The definition of proximity search and its role in improving search ranking.
- How it works based on distance, word order, and other specified distance rules.
- Why you can’t overlook the importance of proximity search for user satisfaction and result quality.
- The most common proximity operators and how they differ from phrase search.
- Practical application of proximity search for Google, PubMed, and Boolean searches.
- A step-by-step guide on how you can implement a proximity search algorithm.
- The most common mistakes developers must avoid, as well as the limitations of proximity search.
- How engines like Meilisearch complement proximity search to deliver better full-text results.

Let’s get into it.

## What is proximity search?

Proximity search is a technique that finds documents where the search terms appear close to each other, rather than anywhere in the text.

For example, a user is looking for documents containing ‘error handling.’

Document A contains the following sentence: ‘This guide explains **error handling** in distributed systems.’ While document B includes this one: ‘This guide explains **errors** in distributed systems and later discusses **handling** user input.’

Proximity search will prefer document A and rank it higher in the results because the words ‘error’ and ‘handling’ are closer together – right next to each other, in fact – in the text.

This helps search engines and databases return results that make more sense than just keyword matching.

## How does proximity search work?

Proximity search works based on a simple principle: how close search terms appear to each other within a document or page.

Instead of matching keywords anywhere in the full text, the search engine measures the distance between the search terms or the number of words between them. If the terms fall within a specific range, the result will rank higher.

Different systems support proximity search in various ways. They can rely on proximity operators, adjacency operators, or custom syntax such as the NEAR operator, quotation marks, or a tilde to indicate distance.

Thanks to these rules, relevance tuning becomes more predictable and consistent across various search engines or databases.

For related concepts, developers often combine proximity rules with [fuzzy search](https://www.meilisearch.com/blog/fuzzy-search) techniques for handling misspellings and variations.

Now, let’s take a look at why proximity search plays such an important role in relevance.

## Why is proximity search important?

Proximity search is so important because it improves [search relevance](https://www.meilisearch.com/blog/search-relevance). It ranks documents in which search terms appear close together rather than scattered throughout the page.

As a result, there’s a significant decrease in useless or irrelevant matches, especially where word order affects meaning.

Academic IR evaluations and health sciences search systems like PubMed and the National Library of Medicine highlighted that keyword matching is easily outperformed by proximity matching. PubMed’s [own search methodology and proximity operators](https://pubmed.ncbi.nlm.nih.gov/help/) explain how enforcing term closeness in titles and abstracts improves precision and reduces irrelevant matches in extensive biomedical collections.

Search engineering experts agree that proximity is one of the strongest signals for [semantic closeness](https://www.meilisearch.com/blog/similarity-search) without requiring full semantic models. This conclusion is supported by recent research such as [_Text Retrieval in Restricted Domains by Pairwise Term Co-occurrence_](https://csimq-journals.rtu.lv/csimq/article/view/csimq.2024-41.05), which demonstrates that incorporating proximity-based co-occurrence information significantly improves retrieval effectiveness compared to keyword-only matching across multiple text types.

Now, let’s look at the operators that enable proximity control.

## What are proximity operators?

Proximity operators are special tools used in proximity searching. They control how far apart search terms may appear within a document by defining the maximum number of words between terms.

You’ll find these operators most commonly used in advanced search platforms, such as databases, academic engines, PubMed, and so on.

The most common proximity operators include:

- **NEAR / N:** Finds terms within a specified distance; for example, _term1 NEAR/5 term2_ is a match when both appear within five words of each other.
- **AROUND():** Mostly used by Google to set the number of words that are allowed between terms.
- **~ (tilde):** For example, _climate change~3_. This means that the terms must appear within three words of each other.
- **ADJ:** An adjacency operator that requires terms to appear next to each other.
- **W / WITHIN:** This proximity operator matches terms within a defined word span in specific systems.

These different proximity operators ensure that your search is both flexible and precise.

## How does proximity search differ from phrase search?

The main difference between proximity and phrase search is that proximity search allows terms to appear within a specified distance.

Phrase search is more strict. It requires the exact phrase in the exact word order. It uses double quotes or quotation marks to enforce the strict matching.

For example, let’s say that a user is searching with the phrase ‘data consistency.’

Document A contains the sentence: ‘This paper discusses **data consistency** models in distributed databases.’

Document B includes the sentence: ‘This paper discusses the **consistency of data** across distributed databases.’

Even though both documents talk about the same idea, phrase search won’t treat document B as a relevant result, while proximity search will.

![A table showcasing the key differences between proximity search and phrase search.](https://unable-actionable-car.media.strapiapp.com/Proximity_search_vs_phrase_search_083b91e2e9.png)

This added tolerance for variation in proximity search can lead to more context-rich matches. However, in domains where accuracy is non-negotiable, you should stick to phrase search.

Let’s take a look at use cases where proximity search genuinely makes a difference.

## What are the main use cases for proximity search?

The real value of proximity search lies in use cases where context and relationships are crucial to the quality of results.

The most common use cases of proximity search are:

- **Academic research:** Using databases like PubMed, NLM, EBSCO, and ProQuest.
- **Legal search:** Proximity search helps lawyers or law students find cases in which concepts appear near each other, enabling them to improve their interpretation and citations.
- **AI and semantic search:** AI models combine keyword logic with AI ranking, so the relevance tuning requires the inclusion of proximity search.
- **Recruitment search:** It helps recruiters surface resumes where skills appear closely together rather than scattered.
- **Enterprise and government search:** Proximity search plays a huge part in ensuring accuracy in [large document repositories](https://www.meilisearch.com/blog/enterprise-search) across business and government systems.

Now, let’s move on to the most prominent search engines and databases that use proximity search. First up is Google.

## How do you use proximity search in Google?

Google supports proximity search through the AROUND() operator. It instructs the engine to find search terms within a specified distance of each other.

The syntax is: term1 AROUND(number of words) term2

This will help you retrieve tightly related concepts rather than broad keyword matches.

Here’s how it works step-by-step:

1.  First, you will enter your two terms.
2.  Then, you insert AROUND() between them.
3.  After the operator, place the number of words allowed in parentheses.
4.  You run the query and review the results where the terms appear close together.

**Examples:**

- _Batman AROUND(3) Joker_

Or, if you want a slightly wider distance:

- _Batman AROUND(5) Gotham_

These queries rank results higher when terms appear within the specified range.

Feeling like a master searcher yet? Let’s talk about proximity search in PubMed.

## How do you use proximity search in PubMed?

PubMed is fond of the ADJ operator, which finds terms within a specified distance and within the same word order.

For reference, the syntax is: term1\[Field\] ADJ\[number of words\] term2\[Field\]

This is part of PubMed’s NLM search engine and is crucial if you want to narrow results in health sciences research.

Step-by-step:

1.  First, you enter your two search terms.
2.  Next, add the ADJ plus the maximum number of words allowed between them.
3.  To move on, you will specify the field, such as \[TIAB\] for title/abstract.
4.  Run the search to filter studies where terms appear close together.

**Example:**

_coronavirus\[TIAB\] ADJ3 vaccine\[TIAB\]_

This finds articles in which the word ‘coronavirus’ appears within three words of ‘vaccine’ in the title or abstract.

Now, let’s look at proximity search for Boolean searches.

## How do you use proximity search in Boolean searches?

Proximity search fits naturally into Boolean logic by adding distance control to operators like AND, OR, and NOT. These proximity operators will define how close the terms must appear.

Many systems also use NEAR, NEAR/n, or WITHIN to combine proximity with Boolean expressions.

You’ll notice that this allows more precise search techniques than using AND alone, which matches terms anywhere in the [full text](https://www.meilisearch.com/blog/how-full-text-search-engines-work).

For example:

_batman NEAR/5 joker AND gotham_

This finds documents in which ‘Batman’ appears within five words of ‘Joker’ and ‘Gotham’ appears anywhere in the document.

Another example:

_superhero NEAR/3 origin NOT "secret identity"_

Here, proximity is combined with exclusion through NOT to refine results. We would get results with ‘superhero’ and ‘origin’ within three words of each other, provided that the document does not contain the phrase ‘secret identity’ anywhere in its text.

Now, let’s walk through the steps of implementing a proximity search algorithm.

## How do you implement a proximity search algorithm?

The outline of our implementation here includes:

1.  Cleaning the documents we will be working with (parsing and tokenizing them).
2.  Building a document index.
3.  Matching queries to the documents based on defined distances.

We will also explain how (and why) to use Meilisearch to simplify this process.

### 1\. Parse documents and record term positions

First things first, you will normalize and tokenize each document, then store the term positions.


```python
_from collections import defaultdict_

_from typing import Dict, List_

_def index_document(doc_id: str, text: str) -> List\[tuple\[str, int\]\]:_

_"""_

_Tokenize text and return (term, position) pairs._

_"""_

_tokens = text.lower().split()_

_return \[(term, pos) for pos, term in enumerate(tokens)\]_
```


### 2\. Build a positional inverted index

Next up, you will build a positional inverted index: term → doc_id → \[positions\].


```python
_PosIndex = Dict\[str, Dict\[str, List\[int\]\]\]_

_def build_positional_index(docs: Dict\[str, str\]) -> PosIndex:_

_"""_

_Build a positional inverted index from {doc_id: text}._

_"""_

_index: PosIndex = defaultdict(lambda: defaultdict(list))_

_for doc_id, text in docs.items():_

_for term, pos in index_document(doc_id, text):_

_index\[term\]\[doc_id\].append(pos)_

_return index_
```


### 3\. Match queries and compute distances

It is crucial that, at query time, you compare positions and check whether they fall within the max distance range.

```python
_def within_distance(positions_a: List\[int\],_

_positions_b: List\[int\],_

_max_dist: int) -> bool:_

_"""_

_Return True if any pair of positions is within max_dist._

_Assumes positions_a and positions_b are sorted._

_"""_

_i, j = 0, 0_

_while i < len(positions_a) and j < len(positions_b):_

_diff = positions_a\[i\] - positions_b\[j\]_

_if abs(diff) <= max_dist:_

_return True_

_if diff < 0:_

_i += 1_

_else:_

_j += 1_

_return False_
```

You would wrap this in a function that, for a given pair of query terms and a _max_dist_, scans the positional index and returns matching document IDs.

In a custom engine, you’d have to do all of this on your own: parsing, indexing, and distance logic.

With Meilisearch, you eliminate most of this complexity. In Meilisearch, proximity is a built-in ranking rule: you define _searchableAttributes_, tune _rankingRules_ (including _proximity_), and let Meilisearch handle proximity scoring under the hood, instead of manually managing each positional index.

The following are the common mistakes teams make with proximity search.

## What are common mistakes with proximity search?

These are the most common mistakes we’ve noticed when working with proximity search:

- Using the wrong proximity operators or adjacency operators. As a result, the system interprets the search terms literally rather than by a specified distance, which undermines the purpose of proximity search.
- Setting the number of words too narrowly or too broadly. This can either bring back irrelevant search results or none at all.
- Ignoring word order in systems that require a specific sequence can return no results.
- Incorrectly mixing proximity searching with Boolean operators, overriding constraints in advanced search queries.

With these issues covered, we can now look at the broader limitations of proximity search itself.

## What are the limitations of proximity search?

Different systems use different syntax, proximity operators, and adjacency operators. This creates problems when you want to use queries across platforms.

Some databases restrict the maximum number of words allowed per query. In fact, some even require controlled vocabulary or strict word order to return accurate citations.

Extensive full-text indexes can slow down performance because the engine must track positional data for every search term. In advanced workflows, these constraints can even clash with Boolean operators.

## Why proximity search matters for modern information retrieval

Proximity search is your best option for strengthening relevance by rewarding documents in which the search terms appear close together in the full text.

It reduces noise and improves search accuracy. Plus, the advanced search techniques it supports play a considerable role in domains such as health sciences, legal research, and large government or academic databases.

Proximity search: Improve relevance & deliver smarter results

## The challenge isn't connecting to an LLM. It's figuring out what people actually mean.

Last week at [MCP Connect Days](https://luma.com/bj6kgvxh) in Paris, I watched a room full of engineers nod in unison. The speaker from The Fork was describing their biggest challenge building AI-powered search: not the LLM integration, not the API calls, not even the infrastructure. The hard part? Understanding what users actually want.

When someone types _"Italian restaurant near Bastille for under 30€ with outdoor seating,"_ the system needs to decompose that into: _cuisine type_ (Italian), _location_ (Bastille), _price range_({'<'}30€), and amenity (outdoor seating). Four distinct filters, zero explicit structure. Just vibes.

This is the **intent understanding problem**. And after talking to dozens of teams building conversational search interfaces, I'm convinced it's where most implementations quietly fail.

> **What is intent understanding?** Intent understanding is the process of translating natural language queries into structured search parameters. It means extracting the right keywords, filters, and facets from conversational input so a search engine can return relevant results.
> 

## The gap between natural language and structured queries

Search engines are remarkably good at what they do. Give them a query with the right filters, and they'll return relevant results in milliseconds. The technology is mature, battle-tested, fast.

The problem is that humans don't speak in structured queries.

Coming back to our Italian restaurant search: the system correctly identified four distinct parameters from a single sentence. But that's the simple case. Now scale this complexity. Real queries include negations (_"not too touristy"_), preferences (_"preferably with a terrace"_), comparisons (_"somewhere nicer than last time"_), and context that requires memory (_"that place my colleague mentioned"_). Each of these requires a different parsing strategy.

Most teams building natural language search underestimate this translation layer. They focus on the LLM integration, assuming the hard part is getting GPT to respond. But LLMs are commoditised. The translation from natural language to structured search? That's where the differentiation lives.

## How The Fork tackles intent parsing

At [MCP Connect Days](https://luma.com/bj6kgvxh), [The Fork's](https://www.thefork.fr/) engineering team shared their approach to building a ChatGPT app for restaurant discovery. Their challenge perfectly illustrates the intent understanding problem.

A typical query: _"Where can I take my parents for their anniversary? They like French food, somewhere quiet, budget around 80€ per person."_

Breaking this down:

- **Occasion context** → anniversary (implies upscale, romantic)
- **Party composition** → parents (older demographic, accessibility matters)
- **Cuisine** → French
- **Atmosphere** → quiet
- **Budget** → ~80€/person

The Fork's system needs to translate these signals into filters their search infrastructure understands. But _"quiet"_ isn't a database field. Neither is _"anniversary-appropriate."_ They've built a sophisticated interpretation layer that maps conversational signals to searchable attributes.

Their key insight: the quality of results depends almost entirely on how well you understand the query, not how powerful your LLM is. A perfectly tuned GPT-4 with mediocre intent parsing will lose to a simpler model with excellent query understanding.

## The rise of conversational commerce

This isn't just a technical curiosity. Conversational commerce is reshaping how people shop and discover products online.

For online retailers, users now expect to interact with search the same way they'd ask a knowledgeable shop assistant. _"I need running shoes for flat feet, nothing too flashy, under 150€"_ is a reasonable query for a human sales associate. It should work for your search too. The companies getting this right are seeing real results: better query understanding means higher conversion rates, lower bounce rates, and customers who actually find what they're looking for.

For classified marketplaces like [Leboncoin](https://www.leboncoin.fr/), the challenge multiplies. _"Looking for a used iPhone, good condition, around Lyon, under 400€"_ requires parsing: _product category_ (smartphones), _brand_ (Apple), _model family_ (iPhone), _condition_ (good), _location_ (Lyon area), and _price_ ({'<'}400€). Each maps to a different filter in their search system. Get one wrong, and you've lost the sale.

The pattern repeats everywhere. Marketplaces, travel sites, content platforms. All building custom intent parsing layers. All solving the same fundamental problem.

## Why building this from scratch is painful

Every team I've talked to that's built conversational search has essentially reinvented the same wheel. They all build:

1. **Entity extraction** – pulling out product types, brands, attributes
2. **Filter mapping** – converting extracted entities to search parameters
3. **Ambiguity resolution** – handling cases where intent is unclear
4. **Validation** – ensuring the generated query actually makes sense

This is significant engineering effort. And it's duplicated across every company building conversational interfaces.

The build-vs-buy decision here is stark. You can spend months building a custom query parsing layer, tuning it for your domain, handling edge cases, maintaining it as your schema evolves. Or you can choose infrastructure that handles this natively.

## How Meilisearch solves intent understanding

This is exactly why we built the [chat route](https://www.meilisearch.com/docs/reference/api/chat-completions) directly into Meilisearch.

Instead of requiring every developer to build their own intent parsing layer, Meilisearch handles the natural language to structured query translation natively. The search engine itself understands how to interpret conversational input.

Here's what this looks like in practice. When you send a natural language query to the [chat route](https://www.meilisearch.com/solutions/rag), Meilisearch:

1. **Parses the intent** – identifies what the user is searching for versus filtering by
2. **Maps to your schema** – automatically routes extracted entities to the appropriate filterable attributes
3. **Generates the query** – produces an optimised search with the right combination of keywords and filters
4. **Returns results** – with full context about how the query was interpreted

The key insight: your search engine already knows your data schema. It knows which fields are filterable, what values exist, what combinations make sense. This context is gold for intent understanding.

When Meilisearch sees _"red derbies under 100€"_ and knows your schema has _"color"_ (with _"red"_ as a value), _"category"_ (with _"derbies"_), and _"price"_ (numeric, filterable), it can make intelligent parsing decisions. Building this at the search engine level is the right abstraction because that's where the schema knowledge lives.

This approach differs builds upon semantic search, which focuses on understanding meaning and similarity. Intent understanding goes further: it's about decomposing a query into actionable search parameters. You need both for truly conversational search, and Meilisearch combines them.

## Why this matters for the AI search ecosystem

The [MCP Connect Days](https://luma.com/bj6kgvxh) conference made one thing clear: everyone is building conversational interfaces. ChatGPT apps, Claude integrations, MCP servers, custom assistants. The race is on.

But most of these implementations will be mediocre. Not because the LLMs are bad, but because intent understanding is treated as an afterthought. Teams will ship, users will try queries that don't quite work, and the magic of AI-powered search will feel more like a gimmick than a feature.

The teams that win will be the ones who invest in robust query understanding. Either by building sophisticated custom layers (expensive, time-consuming) or by choosing infrastructure that handles this natively.

At the conference, I heard the same frustration repeatedly: _"The LLM part was easy. Making it actually understand what users want? That took months."_

It doesn't have to.

## The bottom line

Conversational search is coming to everything. The user expectation is set: if I can ask ChatGPT anything, why can't I ask your product search the same way?

But the gap between that expectation and reality is intent understanding. It's the unsexy, difficult, often-ignored layer between natural language and structured results.

You can build this yourself. Many teams do. But if you're evaluating search infrastructure for conversational use cases, ask one question: **does it understand what my users mean, or do I have to teach it?**

The search engine that answers "yes" to the first question is the one worth betting on.

---

## Frequently Asked Questions

**What is intent understanding in search?**

Intent understanding is the process of interpreting what a user actually wants from their search query. It involves extracting keywords, identifying filters (like price ranges or categories), and understanding implicit requirements. For example, turning "cheap red sneakers" into a search for sneakers + colour filter (red) + price sort (ascending).

**What's the difference between semantic search and conversational search?**

Semantic search focuses on understanding the meaning of words and finding conceptually similar results. Conversational search goes further: it parses natural language queries into structured parameters, handles follow-up questions, and maintains context. Semantic search might understand that "sneakers" and "trainers" are related; conversational search understands that "under 100€" is a price filter.

**How do you parse natural language search queries?**

There are two main approaches: building a custom NLP pipeline (entity extraction, intent classification, slot filling) or using a search engine with built-in query understanding. Custom pipelines offer control but require significant engineering investment. Native solutions like [Meilisearch's chat route](https://www.meilisearch.com/solutions/rag) handle parsing automatically using your existing schema.

**Why is intent understanding harder than LLM integration?**

LLM APIs are well-documented and relatively straightforward to integrate. Intent understanding requires domain-specific knowledge: understanding your product taxonomy, mapping natural language to your filter schema, handling ambiguous queries, and maintaining quality as your catalogue evolves. It's an ongoing engineering challenge, not a one-time integration.

---

*Thomas Payet is COO and co-founder at Meilisearch. He attended MCP Connect Days in Paris in February 2026.*

Why intent understanding is the hardest part of AI-powered search (and how to solve it)

**[Scenario](https://www.scenario.com/) is an enterprise AI platform that enables studios and brands to train custom models on their visual libraries, create workflows, and leverage agents in their creative process.**

By building a "creative memory" unique to each customer, Scenario generates unlimited on-brand assets, from images to video, 3D, and beyond, while maintaining full governance over creative IP. With over **dozens of million fully indexed assets** and **35 million monthly searches**, the platform requires search infrastructure that matches the speed and intelligence of its AI capabilities.

> _“Our enterprise customers need to find the right asset from millions of possibilities in milliseconds. Whether by keyword, visual similarity, or both, Meilisearch lets us deliver that while we focus on what makes Scenario unique: building AI that truly understands each customer's creative DNA. 
Meilisearch is really in the right spot - both because of its product and their team. They listen to us and adapt the roadmap to help us push the boundaries of what’s possible in AI powered search.”_
— Hervé Nivon, Co-Founder & CTO at Scenario.

## Overview: Scenario

Scenario’s modern platform enables businesses and creators to generate any kind of media using state-of-the-art generative AI models and streamline content creation. Scenario's community-led library provides a common language for teams to more easily access generative AI tools for content, including image, audio, video, and 3D. 

Trusted by leading studios and brands, including Unity, Tripledot Studios, InnoGames, Fastory, and Alan, the platform manages a vast and growing volume of creative assets requiring a modern Asset Data Management (ADM) system to maintain an always-on platform ready for the generative era.

## The challenge: scaling search for the AI era

Before adopting Meilisearch, Scenario relied on OpenSearch on AWS to power their asset discovery. As the platform scaled toward tens of millions of indexed assets and 35 million monthly searches, the team needed search infrastructure purpose-built for AI-native workloads:

- **Performance at scale:** Delivering instant search across millions of assets required more than tuning a general-purpose engine. Users expected real-time results across both text and visual queries.
- **Cost and operational overhead:** The AWS-managed setup demanded constant manual optimization, diverting engineering time from core AI and product development.
- **Growing user expectations:** As the asset library expanded, users needed faster, more relevant discovery, making search a product-critical capability rather than a backend utility.

## The solution: a modern hybrid search engine

Scenario’s team needed a solution that offered the speed of a dedicated vector database with the rich feature set of a traditional search engine. After evaluating Pinecone and Qdrant, Scenario chose Meilisearch based on a recommendation from the founders of Hugging Face, thanks to Meilisearch’s superior support for hybrid search.

Meilisearch provided:

- **A developer-first experience:** Clear documentation and a straightforward onboarding process allowed the Scenario team to move fast.
- **Enterprise readiness:** SOC2 compliance and seamless AWS Marketplace integration streamlined billing and ensured baseline security.
- **Hybrid search:** The ability to combine Scenario's proprietary embeddings with keyword and multi-modal (image) search in a single interface made it easier for users to navigate the vast library of assets.

## Implementation: building a scalable search infrastructure

Scenario’s implementation initially focused on creating a seamless discovery experience, leveraging multi-modal search to allow users to navigate public and private asset libraries via both text and visuals.
- **Continuous indexing:** the system handles a high-velocity environment where documents are constantly being ingested, updated, and deleted, ensuring the library remains real-time.
- **Custom embeddings:** rather than using default models, Scenario leverages its own AI capabilities to generate text and image embeddings.
- **Vector search integration:** these embeddings are then indexed in Meilisearch, powering an "Exploration" feature that lets users find similar assets by directly clicking on images.

The result: scaling volumes with minimal maintenance.
## 
The transition to Meilisearch transformed Scenario’s search into a core product advantage: 


>Enterprise teams can now discover and reuse existing assets through visual similarity search, reducing duplicate asset creation and accelerating production cycles.
_— Hervé Nivon, Co-Founder & CTO at Scenario._


- **Dramatic reduction in support load:** support requests related to search performance have virtually disappeared. Scenario’s team now focuses mainly on fine-tuning vector results.
- **Exceptional scale:** Meilisearch effortlessly handles upwards of 35 million searches per month, with 80% of those being complex vector searches.
- **Productivity gains:** by offloading search infrastructure to Meilisearch, Scenario’s engineering team has been able to focus entirely on their core generative AI and API features.
- **Increased user engagement:** the “exploration” feature, powered by visual vector search, has become a primary driver of time spent on the platform.

## Conclusion: pushing the boundaries of AI search

Scenario's partnership with Meilisearch is entering its next phase. As the platform's library of AI models grows past 200 and counting, search must go beyond asset discovery: intelligently matching each task to the most appropriate model becomes just as critical as finding the right asset.

The team is building toward AI agents that autonomously surface relevant reference assets during the creative process and recommend the best-fit model for any given task, delivering context-aware results in real time.

By choosing a partner that values adaptability and direct feedback, Scenario has built a search foundation that not only meets the demands of today’s tens of millions of documents but is ready for the next stage of scaling.


Powering the generative AI era: how Scenario scales asset discovery with Meilisearch

Designing effective typeahead search for faster, smarter UX

Document indexing enables quick searching and retrieval of digital documents. In this guide, we will show you how to build an indexing system that supports fast search and lets you access reliable information quickly.

Here’s what we’ll cover:

- What exactly document indexing is and how documents become searchable.
- The importance of indexing and why it is critical for accessing reliable information.
- The types of industries that rely on indexing.
- The workflow behind document indexing, with explanations for each step.
- The key difference between scanning and indexing, and when you should use one over the other.
- Common indexing methods, such as field-based indexing, metadata indexing, and full-text indexing.
- The unique set of challenges that you can expect to encounter in document indexing.
- How you can build a real-time indexing system with the help of Meilisearch.

Let’s start with the first step: what document indexing actually is.

## What is document indexing?

Document indexing is a structured organization of digital documents based on information such as keywords, metadata, and other relevant fields. The sole purpose is to ensure that these documents can be found quickly when needed.

Document indexing is especially important for industries such as healthcare, where specific documents (patient records, reports, contracts, etc.) need to be immediately pulled from vast amounts of data.

## Why is document indexing important?

Document indexing is important because it simplifies information retrieval. Without it, teams would waste a lot of time having to review digital documents themselves, without automation, and being hampered by manual indexing errors.

When you eliminate human error in favor of information-based indexing based on metadata, names, dates, or other elements, you improve your bottom line. Information is instantly available, and no one spends more time than necessary finding exactly what they need.

So which industries benefit most from document indexing? Let’s see that below.

## What industries benefit most from document indexing?

Many industries rely heavily on large volumes of digital files and would greatly benefit from a fast and simple means to access relevant information.

These industries include:

- **Legal:** Law firms use indexing to sort and organize contracts and case files to keep up with court records. It also helps them not lose any case-critical documents.
- **Healthcare:** Hospitals use document indexing to keep track of patient charts, lab reports, and insurance forms.
- **Finance:** Banks can’t compromise on invoices, account numbers, and other digital information, so they rely on document indexing for records management.
- **Education:** The need to manage enrollment forms, transcripts, research materials, and attendance sheets makes document indexing essential for educational facilities.
- **Government:** Agencies often index permits, licenses, and public records so the public, as well as government offices, can access them.

## How does document indexing work?

Document indexing works by identifying key information in digital documents. This information is then used to store these digital documents in an organized [indexing system](https://meilisearch.com/docs/learn/indexing/indexing_best_practices). For instance, we can use keywords, tags, and metadata to make indexed documents easy to search.

![A graph listing the steps in the inner workings of document indexing.](https://unable-actionable-car.media.strapiapp.com/How_Document_Indexing_Works_26490bc137.png)

### 1\. Document ingestion

The first step is to gather all documents you want indexed. They can be collected from websites, external databases, internal knowledge bases, etc.

### 2\. Text processing

The text is cleaned and prepared so that the search engine can understand it. This can include processes such as lowercasing letters, removing stop words such as ‘and’ and ‘the’, stemming or lemmatization (reducing words to their base form), and similar.

### 3\. Metadata extraction

Key information, or metadata, is identified in each document. Key information can be specific keywords, names, barcodes, dates, or similar.

### 4\. Index building

Finally, the search catalog is built based on the extracted metadata. When a user types in a query, the system pulls up relevant documents in the blink of an eye.

## How is document indexing different from document scanning?

Document indexing and document scanning work together, but they solve different problems in information management.

The role of scanning is to convert paper documents into digital documents. Indexing is the step that comes after scanning.

![Document Indexing vs Document Scanning.png](https://unable-actionable-car.media.strapiapp.com/Document_Indexing_vs_Document_Scanning_d84112d388.png)

In short, scanning digitizes documents, and indexing makes them findable.

## What are common methods of document indexing?

Document indexing can be done in a variety of ways, depending on the types of documents, workflow needs, and the volume of digital files being managed.

1.  **Manual indexing:** A person enters document information and organizes everything by hand. Manual indexing offers control but it takes too much time, and there’s always the possibility of human error.
2.  **Metadata-based indexing:** As the name suggests, the documents are organized using metadata elements, such as file name, authors, dates, etc.
3.  **OCR-based indexing:** Text is extracted from paper documents or scanned documents using optical character recognition. It helps turn paper into digital documents that can be indexed automatically.
4.  **Automated indexing:** Algorithms or systems apply tags or [full-text indexing](https://www.meilisearch.com/blog/how-full-text-search-engines-work) without any manual work. This works well when dealing with large volumes of document-based data.

## What types of information are used for indexing?

Using specific data points makes it easier to organize files and filter results, just as arranging your closet by color makes it easier to find items you want.

Common types of information used for document indexing include:

- **Keywords:** The phrases that indicate the main subject of the document.
- **Metadata:** Specific details of the document, such as the file name, author, title, etc.
- **Document type:** The classification labels of documents, such as invoices, reports, forms, etc.
- **Identifiers:** Values like an invoice number, account number, or customer ID.
- **Dates:** Creation, modification, or event dates for easy sorting.
- **Tags or categories:** Assigning tags or categories helps group related files together.
- **Extracted text:** The type of content that is captured through full-text indexing or OCR for deep search.

Based on this set of data points, it becomes relatively easy to retrieve accurate information from the indexing system.

Let’s look at how indexing supports information retrieval.

## What is the role of indexing in information retrieval?

Indexing is the backbone of information retrieval because it provides a structured map of digital documents. By storing keywords, different identifiers, and metadata, it groups relevant information together.

Document indexing eliminates the need for manual scanning and strengthens the search functionality across databases and document management systems. If your indexing process is clean, even extensive collections cannot hinder retrieval.

Now, let’s examine the challenges that go hand in hand with document indexing.

## What challenges are common in document indexing?

Document indexing can become overwhelming when organizations have to manage large volumes of digital files and rely on unoptimized processes. These issues affect retrieval accuracy and long-term information management.

Here are some of the most common challenges teams face:

- **Inconsistent tagging:** Different teams label documents differently. As a result, the indexing system becomes unreliable. The fix for this is a set of standardized tags, categories, and vocabulary.
- **Human error:** Manual entry leads to mistakes such as missing fields, typos, and mislabeling. The best way to fix this is to shift toward automated indexing.
- **Poor metadata quality:** Weak or incomplete metadata makes document retrieval difficult. Clear rules for required fields, such as dates or invoice numbers, significantly improve the structure.
- **Unscannable or low-quality documents:** When the quality of scanned documents is poor, even the best OCR cannot make them readable. Better scanning tools for data extraction should help here.

Let’s see how automated document indexing works with Meilisearch in real environments.

## How automated document indexing works with Meilisearch

Meilisearch eliminates the need for manual work, replacing it with fast, optimized, searchable indexes in real time. It extracts key document attributes and organizes them into an optimized indexing system.

Here are the main steps to indexing your documents using Meilisearch:

### 1\. Import and prepare your documents

The first step is to load the documents into your workflow. These documents can come from various sources, such as APIs, storage buckets, scanned documents, or even existing databases. Meilisearch accepts JSON, so there’s no problem in pushing the field titles, dates, tags, etc.

```json
_curl \\_

_\-X POST 'http://localhost:7700/indexes/documents/documents' \\_

_\-H 'Content-Type: application/json' \\_

_\--data-binary @documents.json_

```

### 2\. Extract key fields and metadata

Extraction is key for identifying attributes within each file, such as metadata, keywords, categories, and so on.

For example, if a document comes from OCR, the extracted text instantly becomes a part of full-text indexing.

Meilisearch categorizes files and enriches indexed documents to improve [relevant information retrieval](https://www.meilisearch.com/blog/latent-semantic-indexing).


```json
_{_

_"id": "doc_01",_

_"title": "Insurance Claim Form",_

_"date": "2024-01-19",_

_"account_number": "298231",_

_"content": "Full OCR text goes here..."_

_}_
```

### 3\. Index documents and enable search features

Indexing uploads the enriched documents into Meilisearch. Immediately, the engine builds indexes and applies features, enabling various search functionalities such as typo tolerance and synonyms.

```json
_curl \\_

_\-X PATCH 'http://localhost:7700/indexes/documents/settings' \\_

_\-H 'Content-Type: application/json' \\_

_\--data '{_

_"searchableAttributes": \["title", "content"\],_

_"filterableAttributes": \["date", "account_number"\]_

_}'_
```

### 4\. Apply custom ranking and optimize relevance

Now comes the step that defines ranking rules based on the priorities you set, such as dates, status, or file names. Different types of documents require different priority levels.

```json
_curl \\_

_\-X PATCH 'http://localhost:7700/indexes/documents/settings' \\_

_\--data '{_

_"rankingRules": \[_

_"words",_

_"typo",_

_"proximity",_

_"attribute",_

_"sort",_

_"exactness"_

_\]_

_}'_
```

### 5\. Keep the index updated in real time

Lack of index updates can cause bugs and inconsistencies. Meilisearch supports incremental updates so that any new or modified documents become searchable immediately. This takes away time-consuming manual updates and keeps business operations running smoothly.

Thanks to these real-time updates, document management systems always reflect the most current data.

```json
_curl \\_

_\-X POST 'http://localhost:7700/indexes/documents/documents' \\_

_\--data '\[{ "id": "doc_01", "status": "updated" }\]'_
```

## Document indexing in modern information systems

Modern systems rely heavily on document indexing because they require up-to-date information and accurate retrieval. Scaling is also a big reason automated indexing has flourished in recent years.

Through effective indexing, you can optimize all document-related processes for your business, including search accuracy, fast access across workflows, and document management.

How document indexing works, methods, implementation, & more

Text clustering organizes unstructured data by grouping similar documents without any labels. It identifies underlying themes, enabling users to enhance their research.

Here’s what this guide covers:

* What text clustering is and how it helps sequester text into groups based on meaning.
* The mechanism behind text clustering, from preprocessing to algorithms.
* Popular methods for text clustering, such as K-means, DBSCAN, and hierarchical clustering.
* A brief look at the benefits and applications of text clustering and how you can take advantage of it.
* For hardcore developers, we’ll look at a step-by-step Python workflow that builds, visualizes, and evaluates clusters.
* Most common text clustering mistakes and challenges for scaling businesses.
* How to use Meilisearch to make clusters searchable and useful.

Let’s get into it!

## What is text clustering?

Text clustering is an unsupervised learning technique used by AI models to group similar text documents based on their content. Its primary purpose is to help determine themes in large datasets without pre-defined labels, especially when you want to separate text into different clusters within a broader natural language processing workflow. 

It’s most commonly used to organize and analyze customer reviews, news articles, and research documents, among other types of content.

Let’s see how text clustering works behind the scenes.

## How does text clustering work?

![How Text Clustering Works.png](https://unable-actionable-car.media.strapiapp.com/How_Text_Clustering_Works_d67d9a7362.png)

Text clustering consists of the following stages:

1. **Text preprocessing:** Raw text is cleaned by removing unnecessary whitespace, special characters, and URLs. Then it is tokenized or broken apart into smaller units (words or phrases). Stop word removal (such as ‘a’ and ‘the’) and stemming and lemmatization (reducing words to their root form) can also be part of this stage.
2. **Embedding:** Pre-processed text gets converted into numerical vectors. The techniques used here include term frequency-inverse document frequency (TF-IDF), Word2Vec, and transformer-based embeddings such as BERT. 
3. **Clustering:** In the third stage, clustering algorithms, such as K-means, DBSCAN, and hierarchical clustering, are applied. These algorithms assess the vector space created in stage two and group similar vectors into clusters.
4. **Evaluation:** To assess how well the documents fit within the clusters, use metrics such as the Adjusted Rand Index (ARI) or the silhouette score. Techniques such as principal component analysis also enable more precise visualization when assessing separation quality.

Let’s take a closer look at the clustering algorithms that are most effective for different types of text data.

## What algorithms are used in text clustering?

Every algorithm handles text data differently, based on the dataset’s size, shape, and noise. 

The most commonly used clustering methods include K-means, DBSCAN, hierarchical clustering, and newer approaches such as spectral clustering and Gaussian Mixture Models (GMMs). 

![Algorithms Used in Text Clustering.png](https://unable-actionable-car.media.strapiapp.com/Algorithms_Used_in_Text_Clustering_ee4a86486d.png)

Let’s break down the most widely used methods:


### 1. K-means

K-means is a method in which each data point is assigned to the cluster with the nearest mean (centroid) thereby dividing the data into k clusters. The centroid positions are repeatedly recalculated to be the mean of all data points in a specific cluster. The process is repeated until the centroid positions are no longer changing. 

This method performs really well on large datasets. However, it can struggle with noisy or overlapping text data.


### 2. DBSCAN

DBSCAN (Density-Based Spatial Clustering of Applications with Noise) groups data points that are closely packed and marks the distant ones as outliers. 

It works well for irregularly shaped clusters. It also doesn’t need you to predefine the number of clusters, so it works really well for grouping social media posts or product reviews.


### 3. Hierarchical clustering

Hierarchical clustering builds a tree-like structure that shows relationships among documents. You can cut this tree at different levels, forming clusters of varying granularity. 

This comes in handy when dealing with smaller datasets where the relationships between entities are more important than processing speed.


### 4. Spectral clustering and GMM 

Spectral clustering relies on similarity matrices to cluster data. It is computationally more expensive, but the payoff is its effectiveness when dealing with non-linear structures. 

Gaussian Mixture Models (GMM) models assume that the data points are generated from Gaussian distributions. As a result, they allow soft cluster assignments. This is useful when you’re dealing with documents that span multiple topics. 

## What are the benefits of text clustering?

The main function of text clustering is to help teams access large amounts of data in a structured, easily digestible manner. It eliminates manual labeling and complex arrangements.

Here are the benefits of text clustering: 

1. **Automated organization:** Text clustering eliminates the need for manual sorting and tagging.
2. **Discovery of hidden patterns:** Clustering reveals themes across datasets, such as the general sentiment in customer reviews or customer support tickets.
3. **Scalable analysis:** Clustering enables large-scale analytics without disrupting operations.
4. **Better search and recommendations:** Text clustering groups related documents to help search engines surface results that share the same theme. For example, if a user searches for ‘running shoes,’ the system can also return documents from clusters focused on durability, fit, or trail performance.
5. **Data-driven insights:** For instance, support tickets may cluster around billing issues, login problems, or feature requests, enabling teams to quickly see which issues users struggle with the most.

Now that you know how valuable it is, let’s explore where text clustering is most useful in practice.

## What are the main applications of text clustering?

Text clustering is most helpful in industries with a continual, daily influx of unstructured data. It can be used for: 

1. **Document organization:** Enterprises regularly group similar documents, emails, or reports, making extensive archives searchable and easier to manage.
2. **Easy search results retrieval:** Clustering yields better search results faster by significantly improving relevance. 
3. **Customer feedback analysis:** When customer feedback is organized efficiently, businesses can analyze it to improve the customer experience.
4. **Academic and research use:** Researchers can cluster papers, abstracts, or citations to identify overlapping topic areas and stay up to date with emerging trends.

Next up is why embeddings play a critical role in clustering quality.

## Why use embeddings for text clustering?

Embedding converts text into numerical vectors, capturing its semantic meaning. Based on this, clustering algorithms can group documents by intent and context, rather than just by shared keywords. 

AI models such as BERT and Word2Vec are built on deep learning and generate high-dimensional embeddings that capture the nuance of words and their meanings. Techniques like dimensionality reduction also simplify the underlying vector space, making the clusters much more contextually relevant and meaningful.

## How do you perform text clustering in Python?

The Python workflow we will outline here has four steps. With the help of modern libraries like spaCy and scikit-learn, you can easily move from messy text to production-ready clusters. 

Once you have them, integrating with search engines like Meilisearch is a breeze: 

### 1. Pre-processing and tokenization

First things first, we need to normalize the text we have and remove any noise. 

For this, you will convert the text to lower case, remove punctuation and stop words, and then apply stemming. Here’s what it looks like:


```python
import re, nltk, spacy
from nltk.corpus import stopwords

# Download once if needed
# nltk.download('punkt'); nltk.download('stopwords')
# python -m spacy download en_core_web_sm

nlp = spacy.load("en_core_web_sm")
stop_words = set(stopwords.words("english"))

def preprocess(text):
    text = re.sub(r"http\S+|[^a-zA-Z\s]", " ", text.lower())
    doc = nlp(text)
    tokens = [token.lemma_ for token in doc if token.text not in stop_words and len(token.text) > 2]
    return " ".join(tokens)

docs_clean = [preprocess(doc) for doc in raw_docs]
```


The text is now standardized and tokenized, ready for feature extraction and [semantic indexing](https://www.meilisearch.com/blog/latent-semantic-indexing).


### 2. Vectorization and embedding generation

You need to choose a vector representation that matches your goal and scale: 


* TF-IDF: lightweight and interpretable.
* Word2Vec / GloVe: semantic similarity via word embeddings.
* BERT / Transformer models: contextual, sentence-level embeddings for highest accuracy.



 ```python
from sklearn.feature_extraction.text import TfidfVectorizer
from sentence_transformers import SentenceTransformer

# TF-IDF representation
tfidf = TfidfVectorizer(max_features=50000, ngram_range=(1, 2))
X_tfidf = tfidf.fit_transform(docs_clean)

# Transformer-based embeddings
model = SentenceTransformer("all-MiniLM-L6-v2")
X_emb = model.encode(docs_clean, normalize_embeddings=True)
```



Another option is to store embeddings in Meilisearch to enable hybrid (keyword + vector) retrieval later.


### 3. Clustering algorithms in Python

After vectorization, you will need to select an algorithm suited to your data distribution:



* K-means for balanced, spherical clusters.
* DBSCAN for noisy or irregular data.
* Agglomerative (hierarchical) for dendrogram-based analysis.



 ```python
from sklearn.cluster import KMeans, DBSCAN, AgglomerativeClustering

# K-means
kmeans = KMeans(n_clusters=10, n_init=10, random_state=42)
labels_km = kmeans.fit_predict(X_emb)

# DBSCAN
dbscan = DBSCAN(eps=0.3, min_samples=5, metric="cosine")
labels_db = dbscan.fit_predict(X_emb)

# Hierarchical
agg = AgglomerativeClustering(n_clusters=10, linkage="ward")
labels_ag = agg.fit_predict(X_emb)
```



Remember that each algorithm has its own set of trade-offs. Before you consider scaling, try benchmarking with small subsets first.


### 4. Evaluation of clusters and integration into search-oriented workflow

You will need to evaluate the quality of your clusters using both intrinsic and extrinsic metrics:



* *Silhouette Score*, *Davies-Bouldin*, or *Calinski-Harabasz* for structure quality.
* *Adjusted Rand Index (ARI)* if ground-truth labels exist.

Visualize using PCA or t-SNE for interpretability.


```python
from sklearn.metrics import silhouette_score, adjusted_rand_score
from sklearn.decomposition import PCA
import matplotlib.pyplot as plt

sil_score = silhouette_score(X_emb, labels_km, metric="cosine")
print(f"Silhouette score: {sil_score:.3f}")

pca = PCA(n_components=2)
coords = pca.fit_transform(X_emb)
plt.scatter(coords[:,0], coords[:,1], c=labels_km, cmap="tab10", s=10)
plt.show()
```


Store each document with its `cluster_id` and index it in Meilisearch. This will provide you with all the perks of using Meilisearch, including typo-tolerance, filtering, and [hybrid-vector](https://www.meilisearch.com/blog/semantic-vs-vector-search) keyword search.

Let’s address the key challenges you’ll face when applying text clustering in real-world projects.


## What are the challenges in text clustering?

Useful as it is, text clustering comes with several practical challenges that can affect the accuracy and scalability of your projects:

1. **High dimensionality:** Text data often produces thousands of features, especially with TF-IDF embeddings. It has a very high dimensionality. You can reduce dimensions using PCA or t-SNE, but this can ultimately distort distances.
2. **Ambiguous language:** Algorithms have difficulty accurately grouping semantically similar documents when language elements vary in meaning, such as when sarcasm or slang is used.
3. **Hard to scale:** Large text datasets demand memory-efficient clustering. Algorithms like K-means scale well, but methods like hierarchical clustering can be too computationally expensive.
4. **Choosing the correct number of clusters:** Without labels, deciding how many clusters to create is subjective. Some metrics, such as silhouette score or the elbow method, can provide guidance, but this is still mainly a trial-and-error process.

Next, let’s look at common mistakes teams make when clustering text data.

## What are common mistakes in text clustering?

The following are some of the most common mistakes in text clustering that you need to be aware of, because even minor oversights can become costly very quickly:

1. **Slacking on pre-processing:** If you make it a habit to skip cleaning steps, such as stop word removal, you will end up with noisy, inconsistent clusters. Plus, irrelevant tokens will inflate your already high dimensionality, leading to poorer model performance.
2. **Choosing the wrong clustering algorithm:** For instance, if you apply the K-means algorithm to non-spherical data, there might be a split between related documents and a merge between non-related documents. 
3. **Ignoring evaluation metrics:** Without validation using silhouette or ARI scores, clusters may look good, but statistically, they will be a mess.
4. **Overlooking embeddings:** If you rely solely on TF-IDF, you might miss out on the contextual meaning. Transformer embeddings, such as BERT or MiniLM, deliver much better separation. 

## When should you use text clustering over classification?

You should use text clustering when there is a shortage of labeled data, and you want to discover its overarching themes. Text clustering is best when you want to explore new datasets or identify emerging topics with flexible clustering models. 

On the other hand, text classification comes in handy when you’re dealing with predefined labels and the data categories are already known.

## How do you choose the number of clusters?

Choosing the right number of clusters is part science, part best-guess. 

The elbow method plots the sum of squared Euclidean distances to identify a point at which adding more clusters yields suboptimal returns. 

On another note, silhouette analysis measures how well a document fits within the aligned cluster, as opposed to others within the same dimensional space. 

By combining these metrics with domain knowledge and reviewing your clustering results, you get the most meaningful outcomes.

## What is the difference between text clustering and topic modeling?

Text clustering and topic modeling both organize large text datasets. However, the goals they serve are different. 

Clustering is used to find relevant documents within a cluster. This helps in building search systems or recommendation engines.  

Topic modeling, on the other hand, uncovers latent themes within a collection. Methods such as latent Dirichlet allocation (LDA) or non-negative matrix factorization (NMF) assign multiple topics to each document, with probabilities indicating the prevalence of each topic in that document. 

This is more useful in content analysis or summarizing large corpora. 

In short, clustering focuses on grouping similar documents, while topic modeling focuses on interpreting what they’re about.

## Bringing it all together: mastering text clustering for real-world insights

Text clustering gives structure to disorganized text documents. It ensures that text can be analyzed and searched for insights, especially when you deal with document clustering at scale. 

Text clustering supports better information retrieval and workflows powered by modern language models and deep learning. All of this leads to smarter analytics and data-driven decision-making for a business.

What is text clustering? A complete guide

## Summary

Meilisearch versions v1.8 to v1.34.0 are vulnerable to an authenticated, blind Server-Side Request Forgery (SSRF) attack, and we recommend that open-source users update immediately to Meilisearch v1.34.1 or later. 

Customers of Meilisearch Cloud do not need to take any specific action, as the Cloud environment is protected against exploitation of this vulnerability. The vulnerability has not yet been assigned a CVE number; it has CVE Request 1975471 for CVE ID.

## Technical details

In Meilisearch versions v1.8 to v1.34.0, a user with an API key with *write permissions* to the configuration of the Meilisearch instance could set up Meilisearch to send POST or GET requests to domains and IPs local to the private network of the Meilisearch instance, effectively bypassing the firewall.

Meilisearch v1.34.1 fixes the vulnerability by forbidding Meilisearch from making any requests to an host resolving to a non-global IP, in the sense of the [IANA IPv4 Special-Purpose Address Registry](https://www.iana.org/assignments/iana-ipv4-special-registry/iana-ipv4-special-registry.xhtml) or the [IANA IPv6 Special-Purpose Address Registry](https://www.iana.org/assignments/iana-ipv6-special-registry/iana-ipv6-special-registry.xhtml). If you need this functionality in your Meilisearch instance, you can set private IP networks as allowed with the `--experimental-allow-ip-networks` parameter after reviewing the security implications.

We detected no signs of exploitation on [Meilisearch Cloud](https://cloud.meilisearch.com/). We recommend that any self-hosted Meilisearch instance apply [defense in depth and least privilege principles](https://cheatsheetseries.owasp.org/cheatsheets/Server_Side_Request_Forgery_Prevention_Cheat_Sheet.html#imdsv2-in-aws) and have the most limited possible access to the private network.

## Acknowledgments

Thanks to Gabriel Rodrigues (aka Texugo), for reporting this vulnerability and for helping us improve the security of Meilisearch.

Security update: Meilisearch v1.34.1 fixes an SSRF vulnerability 

# Meilisearch Monthly: January 2026

Welcome to the first 2026 edition of Meilisearch Monthly! We’re kicking off the new year with major performance and security improvements, along with a sneak peek at some highly requested features we’re working on.

---

## **Product Updates: What’s New at Meilisearch**

## Just dropped: SSO & SCIM

We are excited to share that our **Enterprise Identity and Access** features are now live. Designed for teams requiring secure, centralized authentication, these features ensure your Identity Provider (IdP) remains the single source of truth.

- **SAML-based SSO:** Users can now log in using their company accounts (Okta, Azure AD, or Google Workspace) instead of managing separate passwords.
- **Auto-Provisioning:** When a user authenticates via SSO, the system can automatically create their account, add them to your organization, and assign them to a default team.
- **SCIM-based User Lifecycle Management:** Available as a separate add-on, SCIM allows your IdP to automatically create, update, and—most importantly—de-provision users.
- **Immediate Revocation:** When a user is removed in your IdP, their access is revoked immediately in Meilisearch, ensuring no orphaned access remains.
    
    
![SSO.gif](https://unable-actionable-car.media.strapiapp.com/SSO_09d9616fd1.gif)

<aside>

👉 [Get in touch](https://meet.meilisearch.com/meetings/meilisearch/get-in-touch) with us to enable SSO for your organization 🔒 

</aside>

### Search analytics in GA - now enabled for all Cloud projects.

Here’s what that means for you:

- **Always-on insights**: Minimal setup with analytics working directly on your project URL - no more edge proxy required.
- **20% faster searches**: With the edge removed, you get the same great analytics with significantly lower latency.
- **Federated search? Fully supported.** Events from multi-search queries are now tracked accurately and visible in your dashboard.

Whether you're tracking performance or laying the groundwork for search personalization, Meilisearch’s built-in analytics are now production-ready and built into the experience by default.

👉 [Try analytics now
](https://cloud.meilisearch.com/login)

> 💡 *Using the edge.meilisearch.com URL? You don’t need to switch right away, but we recommend updating to your project URL **before February 26, 2026**.*
> 

### New Year, New Sharding Improvements

We are starting the year by making our sharding capabilities even more flexible. Meilisearch now allows you to **add new shards dynamically** as your data or traffic grows. This allows you to:

- **Scale horizontally** without rethinking your entire architecture.
- **Maintain fast speeds** via federated search across all shards.
- **Split indices** across multiple instances seamlessly.

> **What’s Next?** Replicated sharding is officially on the horizon to bring even more resilience and flexibility to your stack.
> 

📍 See our updated public roadmap [here](https://www.notion.so/Public-Meilisearch-Roadmap-2cb4b06b651f80868508c2c1409c1b10?pvs=21).

### Performance: 7x Faster Indexing

We’ve been optimizing how we handle big payloads under the hood. Our latest tests show that indexing speed has improved by **7x** when extracting large datasets.

Additionally, we are running tests to switch to AWS **Graviton** machines, which are already showing promising **20% performance gains**. 


<aside>

###  ❗️ Vector search alert:
> If you’re using vector search in production, we strongly recommend updating to **v1.32.0**. This release includes important bug fixes and boosts relevance across hybrid and pure vector queries. The update is backward-compatible, so migrating should be smooth.

</aside>


### Semantic Search with Gemini Embeddings

We have a new embeddings guide. It walks you through the process of setting up Meilisearch with Gemini embeddings to enable semantic search capabilities.

👉 Check it out [here](https://www.meilisearch.com/docs/guides/embedders/gemini)

---

## **Community & Ecosystem**

## Meilisearch at FOSDEM 2026:

Our engineering team is gearing up for FOSDEM, excited to connect with one of the most passionate open source communities in Europe!

- If you’re attending, don’t miss our talk on Language support in Meilisearch - a deep dive into how we’re building a truly multilingual search engine that developers love.
- 👉 Join the conversation ahead of the event in our [community chat room](https://fosdem.org/2026/schedule/event/GS7B8Y-meilisearch-laguage-support/)

## Meilisearch is a Rust Foundation gold sponsor:

We’re proud to support the Rust ecosystem as a Gold Sponsor of the [Rust Foundation](https://rustfoundation.org/). Rust plays a major role in how we build fast, safe, and efficient search. This partnership is a nod to our roots and a signal of our long-term commitment to open source performance at scale.

Meilisearch January updates

How to build a search engine: A complete guide for developers

The durability and potential of your retrieval-augmented generation (RAG) system depend on the database you choose for it. In RAG information retrieval, vector databases and knowledge graphs serve distinct roles: the former focuses on similarities between vector representations, while the latter focuses on relationships between entities. 

In this guide, we will cover:

* What exactly knowledge graphs and vector databases are, and how they work within an RAG AI system.
* The process through which RAG retrieves and augments information to generate user-friendly responses.
* When you should go for a knowledge graph, and when you should use vector search.
* An in-depth comparison of their data models, along with their scalability and retrieval performance. 
* A look into the advantages and limitations of both, and whether or not they can be combined for a better yield.
* How you can use real-world tools and frameworks to build RAG pipelines that are strong and scalable.

By the end, you will have a solid idea of which setup will work best for your goals, data, and production requirements. 

Let’s get into it.

## What is a knowledge graph?

A knowledge graph is, well, a graph of knowledge. It’s a structured data model that represents entities such as people, places, numbers, and other data points, along with the relationships between them. Using nodes and edges, a knowledge graph organizes information and deciphers the connections within it to answer complex or hidden queries for users. 

A graph structure enables the system to apply reason to the data and uncover the real connections needed to answer complex queries. 

In AI applications and RAG, knowledge graphs are primarily used for contextual reasoning and compliance monitoring, where the depth and explainability of answers are paramount.

## What is a vector database?

A [vector database](https://www.meilisearch.com/blog/what-is-a-vector-database) runs on vector embeddings. These are more easily understood as numerical representations of text, images, and even audio that capture meaning instead of full-fledged keywords. These vectors are indexed using similarity search algorithms, such as cosine similarity or HNSW. 

The most popular vector databases, such as Pinecone and Weaviate, power RAG apps by allowing fast retrieval across massive datasets. Most modern semantic and generative AI systems use vector retrieval as their backbone. 

## How does RAG work?

[Retrieval-augmented generation](https://www.meilisearch.com/blog/what-is-rag) is a power-up for large language models (LLMs). It grounds their outputs in external data and adds a smarter, context-aware layer. It works in three steps that are seemingly very simple:

![How RAG Works.png](https://unable-actionable-car.media.strapiapp.com/How_RAG_Works_4bbeaa79f9.png)

1. A retriever traverses the knowledge base or vector database to find documents similar to the user’s query. 
2. The retrieved documents and their context are handed off to the LLM.
3. The LLM generates an answer that’s supported by the context and addresses the user’s query.

With this externally fetched context, the likelihood of AI hallucinations or generic answers is significantly reduced.

## Do you need a vector database for RAG?

A dedicated vector database can significantly improve retrieval precision, but it isn’t always a prerequisite for a functional RAG pipeline. 

Plenty of teams use hybrid search engines that combine [full-text search](https://www.meilisearch.com/blog/how-full-text-search-engines-work), keyword extraction, and vector embeddings within a single layer. The benefit of doing this is that they don’t need to rely on multiple services or heavy infrastructure. 

For example, Meilisearch offers a simple way to set up your workflow through a single <code><em>/chat</em></code> endpoint. It can perform retrieval and generation in a single API call, unlike the traditional multi-step RAG approach. 

## When should you use a knowledge graph for RAG?

A knowledge graph is ideal for RAG systems that have structured or interconnected data. It excels in scenarios where the ‘how’ is just as important as the ‘why,’ such as in medical research, financial compliance, analytics, and business data. 

A knowledge graph enables multi-hop reasoning by modeling entities such as people, organizations, roles, and products. It understands the relationships between these entities and helps your LLM trace how the answer was retrieved, not just the information. 

## When is a vector database better for RAG?

A vector database is a good option when you’re dealing with unstructured or high-volume semantic data. This can include documents, support tickets, research papers, and so on. 

In all situations where meaning matters more than the relationships, you want to use a [vector database for RAG](https://www.meilisearch.com/blog/vector-dbs-rag). 

Since it stores the data in the form of vector embeddings, the retriever can use semantic similarity and deliver the relevant information you’re looking for. 

All of this makes vector databases the go-to option for search, summarization, and content summarization. Unlike knowledge graphs, they scale easily and return results in milliseconds.

## What are the key differences between knowledge graphs and vector databases?

If you’re looking for the main difference between [knowledge graphs and vector databases](https://www.meilisearch.com/blog/graph-rag-vs-vector-rag), you’ll find it in how they model and retrieve information. 

Let’s take a look at some specifics for both. 

![Knowledge graph vs vector database.png](https://unable-actionable-car.media.strapiapp.com/Knowledge_graph_vs_vector_database_4d6864f956.png)

Need reasoning and traceability? Knowledge graphs are your best bet. 

On the other hand, if you want semantic matching and scalability, go with vector databases. 

That said, modern RAG systems often combine the best of both worlds for quick recall and solid reasoning.

## What are the advantages of using knowledge graphs in RAG?

Knowledge graphs provide the structure and reasoning that enable RAG systems to understand how data points are connected, rather than treating each point in isolation.

The benefits of employing knowledge graphs in RAG include:

1. **Relational reasoning on point:** Knowledge graphs capture relationships between entities and are helpful in domain-specific contexts such as legal or medical research.
2. **Easy to explain:** Developers enjoy the traceability of knowledge graphs through nodes and edges because it gives results that can be audited.
3. **Upholding the context:** Data linked through knowledge graphs helps the LLM maintain context coherence across future prompts or windows.
4. **Semantic precision:** Knowledge graphs reduce noise and improve the quality of retrieval.

## What are the advantages of using vector databases in RAG?

Here’s why RAG pipelines rely heavily on vector databases:

1. **Effortless scalability:** Since vector databases can handle millions of embeddings without a hitch, you never have to worry about increased latency across large-scale datasets. 
2. **High speed and performance:** Vector databases can deliver results within milliseconds without compromising accuracy or quality.
3. **Easy handling of unstructured data:** They’re optimized to work smoothly with text, audio, and even image embeddings. This is particularly useful in chatbots.
4. **Flexible and low-cost:** Their architecture supports easy scaling without complex infrastructure.

Modern retrieval systems, such as Meilisearch, [combine vector search, full-text search](https://www.meilisearch.com/solutions/hybrid-search), and reranking in a single engine. This hybrid approach targets both speed and reasoning for retrieval.

## What are the limitations of knowledge graphs in RAG?

Even though they are helpful in specific cases, knowledge graphs have limitations that you need to keep in mind before putting all your retrieval eggs in this basket: 

1. **Hard to develop:** As effective as knowledge graphs are, building and maintaining them takes significant time. Development teams need to work on schema design, ontology mapping, and, of course, manual relationship curation.
2. **Can’t scale as easily:** More edges and nodes mean multiple hops. These traversals degrade retrieval performance. 
3. **Don’t handle unstructured data well:** When the content is text-heavy, it doesn’t fit well into node-edge structures. This limits recall breadth and might limit responses.
4. **Maintenance is expensive:** Frequent schema updates and data versioning increase overhead.

## What are the limitations of vector databases in RAG?

Vector databases aren’t without drawbacks either. Let’s look at a few:

1. **No transparency:** Since the similarity scores are based on math, it’s tough to trace why certain documents were retrieved. As such, you cannot justify outputs either. 
2. **Not for complex relationships:** Vector databases don’t do well with structured relationships because they can’t support entity hierarchies or established relationships between them.
3. **Operational complexity:** Ever heard of ‘glue code’? That’s what you get when integrating vector databases with retrievers and generators. It leads to a high infrastructure overhead that nobody wants. The best modern hybrid systems have already simplified this. For example, Meilisearch does so by bundling retrieval and generation into a single endpoint, reducing the number of orchestration layers and the LLM's call volume.

## How can you combine knowledge graphs and vector databases in RAG?

Combining a knowledge graph and vector database in a single RAG pipeline enables teams to capture both semantic similarity and structured relationships.* *As a result, you get stronger reasoning and pinpoint precision.

Let’s take the example of Meilisearch. It handles high-recall retrieval and grounding by running a hybrid search. In addition, a knowledge graph layer can be set up to trace entity relationships and perform multi-hop reasoning. 

This architecture is useful for enterprise-scale data where both speed and explainability are a must. Meilisearch’s <code><em>/chat</em></code> endpoint handles both retrieval and generation in a single call, while the knowledge graph adds a traceability layer on top.

## How does semantic search differ in knowledge graphs and vector databases?

[Semantic search](https://www.meilisearch.com/blog/semantic-search) behaves differently across these two systems because of how each one represents meaning.

In knowledge graphs, meaning revolves around entities and their relationships. The retrieval then depends on these logical connections and relationships. You’ll find it most useful in domains such as law or healthcare, where there’s no room for error. 

In vector databases, semantic search is tied to embeddings that rely on similarity algorithms to generate answers for users. It’s fast and efficient. For example, if a user searches for ‘HR policy’, it will find content by matching the query with relevant documents, such as ‘employee handbook.’ You won’t even need to define this relationship.

## What are popular vector databases used for RAG?

Let’s explore the most commonly used vector DB tools: 

* **Meilisearch:** The perfect modern hybrid blend. Meilisearch is a search engine with a built-in vector store and hybrid retrieval. It also provides the `/chat` API, which unifies understanding, reranking, and generation into a single layer. Here’s a not-so-secret fact: it often replaces a standalone vector database in production-grade RAG pipelines.
* **Pinecone:** A staple in the industry when it comes to managed vector DBs. It’s known for its low latency and high reliability.
* **Weaviate:** An open-source option that supports hybrid search and schema-based reasoning. Plus, it integrates with most major embedding models.
* **Qdrant:** Qdrant is popular because it is lightweight and developer-friendly. It’s been optimized primarily for real-time similarity search.
* **Milvus:** This one has been designed specifically for enterprise workloads with a distributed architecture and high-throughput performance.

## Examples of knowledge graph platforms for RAG

Let’s now look at some examples of knowledge graph platforms that have been popular with development teams. 

* **Neo4j:** One of the most widely used graph databases. It is the ideal option if you want to store entities and edges with Cypher queries that support semantic reasoning.
* **Amazon Neptune:** Amazon Neptune is a fully managed graph service, optimized for RDF and property graph models.
* **Ontotext GraphDB:** As the name suggests, this one was designed for ontology-driven graphs and semantic search. It is widely used in sectors such as publishing and life sciences. 
* **ArangoDB:** A multi-model database supporting both graph and document queries for flexible, hybrid RAG applications.

## How do you choose between a knowledge graph and a vector database for RAG?

The perfect foundation for your RAG system depends on your data structure, reasoning needs, and operational goals. Here are the factors you must consider before finalizing this decision:

* **Data type:** The data you’re working with is the main factor that determines the foundation you need. If your data is structured, go for a knowledge graph. If it’s unstructured, go with a vector DB. 
* **Explainability:** Choose a knowledge graph when you need traceability and reasoning paths. Vector databases excel when you only need similarity-based retrieval.
* **Scale and performance:** Vector databases handle massive datasets and low-latency retrieval better. Knowledge graphs give up retrieval speed but make up for it with deeper reasoning and richer context.
* **Maintenance and integration:** Knowledge graphs require schema definition and maintenance, whereas vector databases integrate easily into modern AI pipelines.

If you want a deeper understanding of RAG, here are the different [types of RAG](https://www.meilisearch.com/blog/rag-types) to know, along with the [RAG techniques](https://www.meilisearch.com/blog/rag-techniques) and [tools](https://www.meilisearch.com/blog/rag-tools) that drive the modern AI wave.

## Choosing the right path: knowledge graph vs. vector database for RAG

Modern RAG systems are as effective as they are because of the foundations on which they are built. Both knowledge graphs and vector databases are a part of this foundation. 

The choice between them will always depend on your needs, and most teams today lean toward a hybrid approach to better prepare for future needs.

Meilisearch latest news and company updates