We're excited to unveil Meilisearch v1.9. In this article, we’ll review the most impactful changes. For an exhaustive listing, check out the [changelog](https://github.com/meilisearch/meilisearch/releases/tag/v1.9.0) on GitHub.

<Callout>
  Meilisearch 1.9 is available on [Meilisearch](https://cloud.meilisearch.com/?utm_campaign=release-v1-9&utm_source=blog) Cloud, too—upgrade now!
</Callout>

New: ranking score threshold
----------------------------

Meilisearch 1.9 allows excluding search results with low ranking scores. When using the new `rankingScoreThreshold` option, Meilisearch will not return any documents below the defined threshold.

```bash
curl \
 -X POST 'http://localhost:7700/indexes/movies/search' \
 -H 'Content-Type: application/json' \
 --data-binary '{
    "q": "green ogre living in a swamp",
    "hybrid": { 
	    "semanticRatio": 0.9, 
	    "embedder": "default" 
	},
    "showRankingScore": true,
    "limit": 5,
    "rankingScoreThreshold": 0.2
 }'
```

Using the ranking score threshold when implementing [hybrid search](https://www.meilisearch.com/docs/reference/api/search?utm_campaign=release-v1-9&utm_source=blog#hybrid-search-experimental) removes irrelevant results and allows your [search analytics](https://www.meilisearch.com/docs/learn/analytics/analytics?utm_campaign=release-v1-9&utm_source=blog) to collect the _No search results_ metric properly.

<Callout>
  Excluded results do not count towards `estimatedTotalHits`, `totalHits`, or facet distribution.
</Callout>

New: distinct attribute at search time (group by)
-------------------------------------------------

Meilisearch 1.9 adds the ability to define a [distinct attribute](https://www.meilisearch.com/docs/learn/configuration/distinct?utm_campaign=release-v1-9&utm_source=blog) at search time. Meilisearch will only return one document with the specified attribute value when using the new `distinct` search parameter.

This feature is commonly used in ecommerce applications. Consider a `products` index containing multiple variants of the same product, e.g. `Blue iPhone 15` and `Red iPhone 15` documents that share the same `product_id`. The API call below will return _a single_ iPhone 15:

```bash
curl \
 -X POST 'http://localhost:7700/indexes/products/search' \
 -H 'Content-Type: application/json' \
 --data-binary '{
    "q": "iphone",
    "distinct": "product_id"
 }'
```
<Callout>
  When `distinct` is provided, Meilisearch ignores the index’s [distinct attribute](https://www.meilisearch.com/docs/learn/configuration/distinct?utm_campaign=release-v1-9&utm_source=blog).
</Callout>

New: `frequency` matching strategy
----------------------------------

Meilisearch 1.9 introduces a new [matching strategy](https://www.meilisearch.com/docs/reference/api/search?utm_campaign=release-v1-9&utm_source=blog#matching-strategy) to prioritize results that contain occurrences of the least frequent query terms. When using the `frequency` matching strategy, Meilisearch will deprioritize very common words.

Let’s take the example of the `"the little prince"` query. In our indexed documents, the words `"the"` and `"little"` likely have a lot of occurrences. Consequently, the matching strategy will prioritize documents containing `"prince"` instead.

Experimental: new similar documents API
---------------------------------------

Meilisearch 1.9 introduces a new [AI-powered search](https://www.meilisearch.com/docs/learn/ai-powered-search/getting_started_with_ai_search?utm_campaign=release-v1-9&utm_source=blog) feature that allows searching for documents similar to an existing document.

The following API call searches for documents similar to the document whose [primary key](https://www.meilisearch.com/docs/learn/core_concepts/primary_key?utm_campaign=release-v1-9&utm_source=blog) is `23` in the movies index:

```bash
curl \
  -X POST /indexes/movies/similar
  -H 'Content-Type: application/json' \
  --data-binary '{
    "id": "23",
    "embedder": "default",
  }'
```

Check out the [similar documents API](https://www.meilisearch.com/docs/reference/api/similar?utm_campaign=release-v1-9&utm_source=blog) for more information about additional parameters.

Experimental: avoid re-generating embeddings
--------------------------------------------

When importing a [dump](https://www.meilisearch.com/docs/learn/advanced/dumps?utm_campaign=release-v1-9&utm_source=blog) created with Meilisearch 1.9 or higher, Meilisearch will not re-generate the embeddings. This will avoid unnecessary computing when upgrading your Meilisearch database.

### New: `regenerate` parameter

Additionally, Meilisearch 1.9 introduces a new API to give more fine-grained control over [document embedding generation](https://www.meilisearch.com/docs/learn/ai-powered-search/getting_started_with_ai_search?utm_campaign=release-v1-9&utm_source=blog#generate-vector-embeddings-with-openai). Specifically, it enables embedding generation whenever a document is updated.

The document `_vectors` object now accepts objects in addition to arrays. The provided object accepts a `regenerate` boolean and an optional `embeddings` array.

Consider the example document below with user-provided embeddings:

```json
{
	"id": 42,
	"_vectors": {
		// Embeddings for the `default` embedder
		// Equivalent to `regenerate: true`
		"default": [0.1, 0.2 ],
		// Embeddings for the `text` embedder
		"text": {
			"embeddings": [[0.1, 0.2, 0.3], [0.4, 0.5, 0.6]],
			// Never regenerate embeddings
			"regenerate": false
		},
		"translation": {
			"embeddings": [0.1, 0.2, 0.3, 0.4],
			// Regenerate embeddings when document is updated
			"regenerate": true
		}
	}
}

```

While, in general, you might want to re-generate your embeddings whenever the document is updated, this facilitates migrating from user-provided embeddings to letting Meilisearch handle embedding without incurring unnecessary costs.

Experimental: hybrid search breaking changes
--------------------------------------------

As we move toward [stabilizing AI-powered search features](https://github.com/meilisearch/meilisearch/issues/4733), we introduced minor breaking changes to make the APIs less error-prone.

### Breaking: empty `embeddings` array

Following user feedback that the previous behavior was unexpected and unhelpful, providing an empty `embeddings` array will now tell Meilisearch that the document has no embeddings.

<Callout>
  Before Meilisearch 1.9, an empty `embeddings` array was interpreted as a single embedding of dimension 0.
</Callout>

### Breaking: removed `_vectors` in search results

Starting with Meilisearch 1.9, API responses to [vector search and hybrid search](https://www.meilisearch.com/docs/learn/ai-powered-search/getting_started_with_ai_search?utm_campaign=release-v1-9&utm_source=blog) requests will not include `_vectors` in the response.

However, you can now use the new `retrieveVectors` search parameter if you want API responses to include them:

```bash
curl \
  -X POST 'http://localhost:7700/indexes/movies/search' \
  -H 'Content-Type: application/json' \
  --data-binary '{
    "q": "star wars",
    "retrieveVectors": true
  }'
```

### Breaking: optimizing user-provided `embeddings`

Starting with Meilisearch 1.9, vector embeddings will no longer be stored as-is. Numbers will be cast to floats in canonicalized representation to save storage and optimize performance. Simply put, the vector `[3]` might be stored as `[3.0]`.

Contributors shout-out
----------------------

Thanks to all community members who participated in this release. Shout out to [@gh2k](https://github.com/gh2k), [@writegr](https://github.com/writegr), and [@yudrywet](https://github.com/yudrywet) for their contributions to [Meilisearch](https://github.com/meilisearch/meilisearch) and [@mosuka](https://github.com/mosuka), [@Soham1803](https://github.com/Soham1803), and [@tkhshtsh0917](https://github.com/tkhshtsh0917) for their contributions to [Charabia](https://github.com/meilisearch/charabia).

And, of course, many thanks to our SDK maintainers, thanks to whom Meilisearch is available across many languages. Special thanks to [@the-sinner](https://github.com/the-sinner) and [@norkunas](https://github.com/norkunas). 🫶

* * *

And that’s a wrap for v1.9! This release post highlights the most significant updates. For an exhaustive listing, read the [changelog on Github](https://github.com/meilisearch/meilisearch/releases/tag/v1.9.0).

Stay in the loop of everything Meilisearch by subscribing to our [monthly newsletter](https://dub.sh/meili-newsletter?utm_campaign=release-v1-9&utm_source=blog). To learn more about Meilisearch's future and help shape it, take a look at our [roadmap](https://roadmap.meilisearch.com/?utm_campaign=release-v1-0&utm_source=blog) and participate in our [Product Discussions](https://github.com/orgs/meilisearch/discussions).

For anything else, join our developers community on [Discord](https://discord.meilisearch.com/?utm_campaign=release-v1-9&utm_source=blog).

Meilisearch 1.9

On June 11, 2024, Meilisearch hosted a live event packed with product updates, sneak peeks, and roadmap discussions. If you couldn't make it, here's a quick recap of what went down.

**AI-powered search refresher and new embedders**
-------------------------------------------------

Our product team took the stage to share insights into developing our AI-powered search solution and outlined our vision for providing ready-to-use components for seamless AI-powered search integration. By adding semantic and AI capabilities to Meilisearch's core keyword and full-text product, we introduced a [hybrid search solution](https://www.meilisearch.com/solutions/hybrid-search) in early 2024.

To streamline the developer experience, we have since added [Ollama and REST embedders](__GHOST_URL__/meilisearch-1-8/) for easier integration within the v1.8 scope.

![](https://lh7-us.googleusercontent.com/docsz/AD_4nXd_QlnM1707ZYLjpOrhzxjZrWDxgdVo3TL5Dewz6dM1ZIbhZp4CcPUJdtDhV4xAA_ZZACDSv8h9JF_d2AoFSNGYNlShW_JOI_s1TrqbEtntN-7pvxW8UnTTdVMNml9oBroLhZfrVprM5dQmM0HiTP55MZPw?key=pr3OBspkNUkqXQYoKJ84OA)

You are welcome to try out the hybrid search feature using our [movie demo database](https://where2watch.meilisearch.com/). 

**New relevancy improvements and smarter search**
-------------------------------------------------

We also introduced several new features designed to _improve search relevancy_ in v1.9:

*   **Negative keyword search**: exclude unwanted results easily.
*   **New frequency matching strategy**: enhance search accuracy by removing the most common term found in the dataset.
*   **Filter by score**: set a baseline _rankingScore_ to return only results considered "good enough" in terms of relevancy.
*   **Get similar documents:** utilize AI to find similar documents within your dataset.

This new feature set is slated for launch on July 1st.

**Query suggestions sneak peek**
--------------------------------

Attendees also got a sneak peek into an upcoming release. The new _query suggestions_ feature aims to enhance search efficiency by returning real-time, relevant suggestions as users type, ordered by popularity. This feature leverages [cloud search analytics](__GHOST_URL__/search-analytics-release/) to generate and rank suggestions based on popularity and relevance. Designed for seamless integration, it requires no new endpoints or tools and is expected to be released in Q3. Ongoing improvements will be based on user feedback.

**Check out the recording of the event for more details:**
----------------------------------------------------------

* * *

Get these monthly updates straight to your inbox by subscribing to our [newsletter](https://dub.sh/meili-newsletter).

For more things Meilisearch, join our developers' community on [Discord](https://discord.meilisearch.com/?ref=blog.meilisearch.com). You can learn more about the product by checking out the [roadmap](https://roadmap.meilisearch.com/?ref=blog.meilisearch.com) and participating in [product discussions](https://github.com/meilisearch/product/discussions?ref=blog.meilisearch.com).

Until next time!

Here's what you missed in the latest Meilisearch live event: V1.8, V1.9, new embedders, query suggestions, and more.

🪼 Announcing Meilisearch 1.8
-----------------------------

[![Meilisearch 1.8 release notes](https://unable-actionable-car.media.strapiapp.com/1_8_1fa2c43277.jpeg)](__GHOST_URL__/meilisearch-1-8/?utm_campaign=newsletter&utm_source=mailchimp&utm_medium=april-24-updates)

Meilisearch 1.8 brings **negative keyword search**, improvements to **search robustness**, and **new embedders** for AI search. It’s time to upgrade. 👇

[Read release notes](__GHOST_URL__/meilisearch-1-8/?utm_campaign=newsletter&utm_source=mailchimp&utm_medium=april-24-updates)

📺 Join our June 11th webinar
-----------------------------

[![What's new in Meilisearch: v1.8, AI search updates, open source embedders, and more!](https://unable-actionable-car.media.strapiapp.com/1f114ac0_8673_4b91_a4ea_d3f4467b2e59_43e7fe7365.png)](https://us06web.zoom.us/meeting/register/tZMudO-qpz8qGdDQ_MNn3NmGV0m8WLf1uECM?utm_campaign=newsletter&utm_source=blog&utm_medium=april-24-updates#/registration)

Join our product team as we dive into what our latest released features. You will learn more about the AI-powered search and get a sneak peek of what's coming soon. 👀

[Register now](https://us06web.zoom.us/meeting/register/tZMudO-qpz8qGdDQ_MNn3NmGV0m8WLf1uECM?utm_campaign=newsletter&utm_source=blog&utm_medium=april-24-updates#/registration)

🤝 OCTO customer story
----------------------

[![Meilisearch is a partner of choice for OCTO Technology](https://unable-actionable-car.media.strapiapp.com/octo_58ab726648.png)](__GHOST_URL__/meilisearch-is-a-partner-of-choice-for-octo/?utm_campaign=newsletter&utm_source=blog&utm_medium=april-24-updates)

This customer story discusses why **OCTO Technology** chose Meilisearch to answer its client's complex needs.

[Read customer story](__GHOST_URL__/meilisearch-is-a-partner-of-choice-for-octo/?utm_campaign=newsletter&utm_source=blog&utm_medium=april-24-updates)

🖇️ Links, links, links
-----------------------

### 💬 [RFC: Query suggestions](https://github.com/orgs/meilisearch/discussions/740)

We’re looking for feedback on our query suggestions proposal.

### 🤔 [Full-text search vs vector search](__GHOST_URL__/full-text-search-vs-vector-search/?utm_campaign=newsletter&utm_source=blog&utm_medium=april-24-updates)

A comparative analysis of full-text, vector, and hybrid search.

### 💡 [Indexing best practices](__GHOST_URL__/best-practices-for-faster-indexing/?utm_campaign=newsletter&utm_source=blog&utm_medium=april-24-updates)

All you need to know to speed up Meilisearch indexing.

### 🛍️ [Adding site search to your Nuxt 3 ecommerce](__GHOST_URL__/nuxt-ecommerce-search-guide/?utm_campaign=newsletter&utm_source=blog&utm_medium=april-24-updates)

Learn to build sorting, filtering, and faceted search in Vue & Nuxt apps.

### 🍿 [Next.js hybrid search demo](https://where2watch.meilisearch.com/?utm_campaign=newsletter&utm_source=blog&utm_medium=april-24-updates)

Find where to stream movies in our hybrid search demo for Next.js.

🥰 Made with Meilisearch
------------------------

[Arvid Kahl](https://twitter.com/arvidkahl/status/1780350234410746084) is building [Podscan.fm](https://podscan.fm/?ref=meilisearch), a tool allowing users to subscribe to notifications when keywords are mentioned in podcasts. Meilisearch powers the app’s [full-text search over 2.8 million episode transcripts](https://twitter.com/arvidkahl/status/1780350234410746084).

[![Podscan.fm screenshot](https://unable-actionable-car.media.strapiapp.com/Screenshot_2024_05_14_at_17_27_59_514db7168c.png)](https://podscan.fm/?ref=meilisearch)

_We’re always looking to promote new projects from the community. Post yours in the_ [_#built-with-meilisearch_](https://discord.com/invite/hTD5hU2yWV) _Discord channel!_

🏡 Life at Meili
----------------

Some Rust libraries are like oil and water, they just don't mix. Our very own [@dureuill](https://github.com/dureuill) learnt that the hard way. Read on to learn about the "funny" bug he created while working on Meilisearch.

[![Amateur scientist creating a mess by mixing products](https://unable-actionable-car.media.strapiapp.com/dontmix_a951985054.gif)](https://blog.dureuill.net/articles/dont-mix-rayon-tokio/?ref=meilisearch)

[Read blog post](https://blog.dureuill.net/articles/dont-mix-rayon-tokio/?ref=meilisearch)

🦸 We're hiring!
----------------

See our open positions:

*   [Growth Marketing Manager](https://www.meilisearch.com/careers?utm_campaign=newsletter&utm_source=mailchimp&utm_medium=april-updates-24&utm_content=growth-marketing-manager)

* * *

Get these monthly updates straight to your inbox by subscribing to our [newsletter](https://dub.sh/meili-newsletter).

For more things Meilisearch, join our developers community on [Discord](https://discord.meilisearch.com/?ref=blog.meilisearch.com). You can learn more about the product by checking out the [roadmap](https://roadmap.meilisearch.com/?ref=blog.meilisearch.com) and participating in [product discussions](https://github.com/meilisearch/product/discussions?ref=blog.meilisearch.com).

Until next time!

Meilisearch April Updates

[OCTO Technology](https://www.octo.com/) is a consulting firm specializing in new technologies and the challenges of digital transformation. Headquartered in Paris, it’s recognized for its technological expertise and innovative approach.

> "The searchable page with Meilisearch is really fast and efficient. Meilisearch allows us to meet the customers’ needs with a turnkey solution and reactive support." - Simon Belbeoch, Tech Lead/Manager at Octo Technology

**OCTO Technology uses Meilisearch for its client, an undisclosed customer. The customer runs a platform that aims to connect young people with relevant job opportunities, allowing them to search all kinds of information about integrating into working life. The project involved implementing a search feature to look through data consolidated from various sources.**

Given the specific needs of their sector, the client strongly preferred an open-source solution from the beginning. Other decision-making factors included good performance and greater control over data, allowing users to search through a comprehensive list of jobs to find their ideal offer quickly.

Challenge
---------

The client faced search challenges that required a specific solution, not just standard configurations or a generic search engine. Their website had custom fields that required custom programming. They aimed to replace their previous search engine, [Algolia](https://www.meilisearch.com/docs/learn/what_is_meilisearch/comparison_to_alternatives#algolia), with a new Meilisearch solution, without compromising on functionality or performance

The task became more complex due to the need to aggregate data from various partners in differing formats and manage incomplete data to ensure a consistent search experience across the platform. OCTO’s customer also required more control over search results and sought integration with their CMS, Strapi, for better data control.

Why OCTO chose Meilisearch
--------------------------

Before settling for a specific search provider, OCTO Technology ran a few different scenarios on behalf of its customer and explored the following solutions:

*   Building a proprietary, in-house solution was briefly considered but ultimately ruled out because of the cost-effectiveness considerations and high-performance expectations.
*   Leveraging their prior experience with the platform, the OCTO team initially considered Algolia integration. However, they ultimately favored the open-source transparency offered by Meilisearch, which provided them with greater control over technology and customization settings.
*   Although the team at OCTO considered Elasticsearch too complex for the given client's use case, they decided to use it for logging and configuring data. Ultimately, despite having previous Elasticsearch expertise, the product's complexity prevented the team from benefitting from the Elasticsearch’s offering fully.

Ultimately, the following factors played the most in the decision-making:

### Preference for an open-source

For transparency and to align with the specific needs of the OCTO's client vertical, there was a requirement for the tech stack to be primarily open-source and publicly accessible.

The stress tests were conducted with Meilisearch, and the performance provided better search results compared to the custom search solution that was initially in place.

### Seamless CMS integration

OCTO’s client utilized [Strapi](https://www.meilisearch.com/docs/learn/cookbooks/strapi_v4), a headless CMS solution, for data management and discovered that Meilisearch supports seamless integration via the CMS. Adding data and updating the front-end were straightforward, with the Strapi connection functioning smoothly.

### Consolidation of multiple data sources

This challenge highlighted the need for precise control and robust performance.  To demonstrate Meilisearch's capabilities, the OCTO team conducted a comprehensive proof of concept, focusing on aggregating various job offers from diverse sources and formats.

OCTO conducted a series of stress tests to evaluate the search page functionality under extreme conditions, including handling high data volumes and maintaining search functionality during indexing. While measuring response speed, they also tested the system’s stability under heavy data ingestion to assess whether the search capabilities would withstand extreme condition.

* * *

Ready to elevate your search experience?

[Contact sales](https://dub.sh/meili-sales)

* * *

Implementation
--------------

The platform, developed using Next.js and hosted on Github (though not publicly linked at OCTO’s team request for client anonymity), showcases OCTO’s client's self-managed instance. 

The search engine was straightforward to implement, even without specific Meilisearch expertise, and was integrated with Strapi through a [plugin](https://www.meilisearch.com/docs/learn/cookbooks/strapi_v4). In the front-end, OCTO developed custom Strapi components to allow their client to easily manage UI edits via the CMS.

> _“At the beginning, we had to create custom components, extending from open-source, to meet our client’s specific requirements. Since then, we have consistently reused these front-end components. The early stages of the Strapi to Meilisearch plugin encountered some bugs, but thanks to the responsive collaboration with the Meilisearch team, these issues were resolved swiftly.”_ - Simon Belbeoch, Tech Lead/Manager at Octo Technology.

Further comprehensive stress tests were included in the implementation process to ensure robust system performance. These tests involved data insertion, simultaneous searches, and data insertion while searching.

Results
-------

The outcome of implementing Meilisearch was positive. The comprehensive proof of concept and stress tests conducted before going live showed that the search performed well and the page did not break. OCTO’s customer now fully and independently manages the instance, enhancing the user experience with more relevant search results that reduce user frustration. Although specific metrics were not set beforehand, the implementation successfully met the project's goals, with OCTO’s client observing that Meilisearch's search performance exceeded that of their previous custom search solution.

Vision
------

The experience gained through the completed project with Meilisearch allows OCTO to integrate Meilisearch into its discussions with clients at an earlier stage. It positioned the OCTO team to more effectively meet customer requirements, leading them to consider adopting Meilisearch for internal usage as well as for its new clients. Additionally, the client has expressed strong interest in Meilisearch, prompting considerations for its application in other products.

* * *

Want to stay in the loop with everything Meilisearch? Subscribe to our [newsletter](https://dub.sh/meili-newsletter). Help us shape Meilisearch's future by checking out our [roadmap](https://roadmap.meilisearch.com/?utm_campaign=release-v1-8&utm_source=blog) and participating in our [product discussions](https://github.com/meilisearch/product/discussions?ref=blog.meilisearch.com).

For anything else, join our developers community on [Discord](https://discord.meilisearch.com/?utm_campaign=release-v1-8&utm_source=blog).

Meilisearch is a partner of choice for OCTO Technology.

<Callout>
  Meilisearch 1.8 is available on [Meilisearch Cloud](https://cloud.meilisearch.com/?utm_campaign=release-v1-8&utm_source=blog) too—upgrade now!
</Callout>

New: negative search
--------------------

Meilisearch 1.8 allows you to exclude specific terms from your search results using the `-` operator. 

For example, to find documents that contain the word "room" but not "escape", place a `-` before "escape”:

```Bash
curl \
  -X POST 'https://edge.meilisearch.com/indexes/places/search' \
  -H 'Content-Type: application/json' \
  --data-binary '{ "q": "-escape room" }'

```

Experimental: Improved hybrid search
------------------------------------

<Callout>
  AI search is an experimental feature. Give feedback in this [GitHub discussion](https://github.com/orgs/meilisearch/discussions/677).
</Callout>

### New Ollama embedder

Meilisearch 1.8 introduces support for Ollama models. [Configure the embedder](https://www.meilisearch.com/docs/learn/experimental/vector_search?utm_campaign=release-v1-8&utm_source=blog#generate-vector-embeddings) in your index settings:

```JSON
"default": {
  "source": "ollama", // 👈 Use the ollama source
  "url": "http://localhost:11434/api/embeddings",
  // ☝️ optional, fetched from MEILI_OLLAMA_URL environment variable if missing
  "apiKey": "<your-api-key>",
  // ☝️ optional
  "model": "nomic-embed-text",
  "documentTemplate": "{{doc.field_name}}" // like for openAI and huggingFace sources
}

```

### New REST Embedder

Starting with Meilisearch 1.8, you can use any embedder accessible via a REST API. Here’s how to configure an OpenAI embedder with the `rest` source:

```JSON
{
  "source": "rest",
  "apiKey": "<your-openai-api-key>",
  "dimensions": 1536,
  "url": "https://api.openai.com/v1/embeddings",
  "query": {
    "model": "text-embedding-ada-002"
  },
  "inputField": ["input"],
  "inputType": "textArray",
  "pathToEmbeddings": ["data"],
  "embeddingObject": ["embedding"]
}

```

For more details on the API and available parameters, please consult the documentation on [REST embedders](https://www.meilisearch.com/docs/learn/experimental/vector_search#generate-auto-embeddings?utm_campaign=release-v1-8&utm_source=blog).

### New embedder setting: distribution shift

<Callout>
  This is an advanced API for custom models. OpenAI and Hugging Face models don't require this setting.
</Callout>

Depending on the embedding model, [vector embeddings](__GHOST_URL__/what-are-vector-embeddings/?utm_campaign=release-v1-8&utm_source=blog) are not always uniformly distributed within the [vector space](__GHOST_URL__/what-are-vector-embeddings/?utm_campaign=release-v1-8&utm_source=blog). When vectors are overly clustered, semantic search scores can be very closely ranked. As a result of semantic scores being packed, hybrid results might have sub-optimal relevancy.

To address this issue, you can now shift the distribution of results with an affine transformation. This transformation normalizes semantic scores to prevent them from clustering too high or too tightly, making them more comparable to the [keyword scores](https://www.meilisearch.com/docs/reference/api/search#ranking-score).

```JSON
"default": {
  "source": "huggingFace",
  "model": "MODEL_NAME",
  "distribution": {  // describes the natural distribution of results
    "mean": 0.7, // mean value
    "sigma": 0.3 // standard deviation
  }
}

```

*   The **mean** value represents the central point around which you observe semantic search scores to be densely packed. In the provided code sample, the setting indicates that semantic scores from this model tend to cluster around a mean of 0.7.
*   The **sigma** value determines the spread of ranking scores from the mean (standard deviation). The sigma provided in the code sample above means that before shifting the distribution, the results tend to have an average distance of 0.3 to the mean.

This allows for a more effective comparison of `_rankingScore` between semantic and keyword search results, leading to improved results ranking.

### Breaking: trimmed API response

To speed up the search response and reduce bandwidth usage, Meilisearch v1.8 no longer returns the following fields in the search responses:

*   the `vector` field, which returned the query vector 
*   the `value` field included in the [`vectorSort`](https://www.meilisearch.com/docs/reference/api/search?utm_campaign=release-v1-8&utm_source=blog#vectorsorttarget_vector) object, which is returned when you add \`”showRankingScoreDetails”: true”\` to a search query
*   the `_semanticScore` field; its value is the same as `_rankingScore`, which you should use instead. Set [`”showRankingScore”: true”`](https://www.meilisearch.com/docs/reference/api/search?utm_source=blog&utm_campaign=release-v1-8#ranking-score) in the search parameters

Search robustness improvements
------------------------------

### New search cut-off setting

To enhance stability and performance, Meilisearch automatically terminates any search request that exceeds 1500 milliseconds. 

To adjust this timeout threshold, you can customize the `searchCutoffMs` setting:

```bash
curl \
  -X PATCH 'https://edge.meilisearch.com/indexes/movies/settings' \
  -H 'Content-Type: application/json' \
  --data-binary '{
    "searchCutoffMs": 150
  }'

```

### Experimental: concurrent search requests limit

This release introduces a limit on concurrent search requests to prevent Meilisearch from using excessive amounts of RAM and crashing. The default queue limit is set to 1000 requests. 

To modify this limit, restart your self-hosted instance with the `--experimental-search-queue-size` option:

```bash
./meilisearch --experimental-search-queue-size 100
```

👉 This limit does not impact the search performance. It only controls the number of search requests that can be queued simultaneously to enhance system security.

<Callout>
  Give feedback regarding concurrent search requests limit in this [GitHub discussion](https://github.com/orgs/meilisearch/discussions/729).
</Callout>

Contributors shout-out
----------------------

We are really grateful for all the community members who participated in this release. We would like to thank: [@availhang](https://github.com/availhang), [@hack3ric](https://github.com/hack3ric), [@jakobklemm](https://github.com/jakobklemm), [@mohsen-alizadeh](https://github.com/mohsen-alizadeh), [@redistay](https://github.com/redistay), [@rohankmr414](https://github.com/rohankmr414), [@shuangcui](https://github.com/shuangcui), [@Toromyx](https://github.com/Toromyx), [@6543](https://github.com/6543) for their help with Meilisearch, and [@Gusted](https://github.com/Gusted), [@mosuka](https://github.com/mosuka), [@6543](https://github.com/6543), for their work on [Charabia](https://github.com/meilisearch/charabia).

Special thanks to our SDKs maintainers, thanks to whom Meilisearch is available across so many languages. 🫶

* * *

And that’s a wrap for v1.8! This release post highlights the most significant updates. For an exhaustive listing, read the [changelog on Github](https://github.com/meilisearch/meilisearch/releases/tag/v1.8.0).

Stay in the loop of everything Meilisearch by subscribing to our [newsletter](https://dub.sh/meili-newsletter). To learn more about Meilisearch's future and help shape it, take a look at our [roadmap](https://roadmap.meilisearch.com/?utm_campaign=release-v1-8&utm_source=blog) and come participate in our [Product Discussions](https://github.com/meilisearch/product/discussions).

For anything else, join our developers community on [Discord](https://discord.meilisearch.com/?utm_campaign=release-v1-8&utm_source=blog).

Meilisearch 1.8

🚀 Announcing Meilisearch 1.7
-----------------------------

[![](https://unable-actionable-car.media.strapiapp.com/138_84d37ea815.jpg)](__GHOST_URL__/meilisearch-1-7/?utm_campaign=newsletter&utm_source=blog&utm_medium=march-updates-24)

Meilisearch 1.7 **stabilizes ranking score details**, adds **GPU support for Hugging Face embeddings**, and integrates the **latest OpenAI embedding models**. It’s time to upgrade. 👇

[Read release notes](__GHOST_URL__/meilisearch-1-7/?utm_campaign=newsletter&utm_source=blog&utm_medium=march-updates-24)

⚖️ Introducing hybrid search
----------------------------

[![](https://unable-actionable-car.media.strapiapp.com/Group_143725858_1cf2d7de65.png)](__GHOST_URL__/introducing-hybrid-search/?utm_campaign=newsletter&utm_source=blog&utm_medium=march-updates-24)

Meilisearch 1.7 introduced new OpenAI embedders, adding to Meilisearch's **AI-enhanced hybrid search capabilities**. Read the blog post to learn more about hybrid search and [**sign up to our waiting list**](https://share-eu1.hsforms.com/1RN-rGYhHQPu5BNivfknlwAfg3aq?utm_campaign=newsletter&utm_source=blog&utm_medium=march-updates-24).

[Read announcement](__GHOST_URL__/introducing-hybrid-search/?utm_campaign=newsletter&utm_source=blog&utm_medium=march-updates-24)

🍿 Watch our launch event
-------------------------

On March 7th, we discussed our roadmap, our recent releases and **showcased** Meilisearch Cloud’s new features, including **monitoring metrics** and **hybrid search**.

[Watch it!](https://youtu.be/Lgv8iTdWDQc?si=GOZ7HiV0jM0KJQ7y)

🤔 Full-text search vs vector search
------------------------------------

[![](https://unable-actionable-car.media.strapiapp.com/fts_vs_vs_71fd339c9c.jpg)](__GHOST_URL__/full-text-search-vs-vector-search/?utm_campaign=newsletter&utm_source=blog&utm_medium=march-updates-24)

Join [@StriftCodes](https://twitter.com/StriftCodes) in this **comparative analysis** as he delves into the pros and cons of each solution and how **hybrid search** might offer the best of both worlds.

[Read blog post](__GHOST_URL__/full-text-search-vs-vector-search/?utm_campaign=newsletter&utm_source=blog&utm_medium=march-updates-24)

* * *

🖇️ Links, links, links
-----------------------

### 🟧 [Laravel multitenancy guide](https://www.meilisearch.com/docs/learn/cookbooks/laravel_multitenancy?utm_campaign=newsletter&utm_source=blog&utm_medium=march-updates-24)

Implement search in a multitenant Laravel application.

### 💻 [Computing Hugging Face embeddings with the GPU](https://www.meilisearch.com/docs/learn/cookbooks/computing_hugging_face_embeddings_gpu?utm_campaign=newsletter&utm_source=blog&utm_medium=march-updates-24)

Learn to compile a Meilisearch binary that leverages an Nvidia GPU to generate Hugging Face embeddings.

### 🔢 [From ranking to scoring](__GHOST_URL__/from-ranking-to-scoring/?utm_campaign=newsletter&utm_source=blog&utm_medium=march-updates-24)

Our journey to add relevancy scores to search results in Meilisearch.

### 🪼 [Meilisearch v1.8 release candidate](https://github.com/meilisearch/meilisearch/releases/tag/v1.8.0-rc.0)

Get a preview of what’s coming next (and give your feedback!)

### 🚀 [Strapi v4 guide](https://www.meilisearch.com/docs/learn/cookbooks/strapi_v4?utm_campaign=newsletter&utm_source=blog&utm_medium=march-updates-24)

Learn how to integrate Meilisearch with Strapi.

* * *

🥰 Made with Meilisearch
------------------------

The folks at **Blazity** have created this impressive **Next.js storefront** designed for high-performance eCommerce, featuring AI capabilities and one-click deployment.

[![](https://unable-actionable-car.media.strapiapp.com/blazity_e97c54672a.png)](https://github.com/Blazity/enterprise-commerce)

_We’re always looking to promote new projects from the community. Post yours in the_ [_#built-with-meilisearch_](https://discord.meilisearch.com/?ref=blog.meilisearch.com) _Discord channel!_

🏡 Life at Meili
----------------

Join our Head of Engineering, [@clurquizar](https://twitter.com/clurquizar), as she walks us through key points of Meilisearch’s history and our evolution as an open-source company.

[![](https://unable-actionable-car.media.strapiapp.com/clem_itw_af4c14f089.png)](__GHOST_URL__/how-meilisearch-became-open-source-a-retrospective/?utm_campaign=newsletter&utm_source=blog&utm_medium=march-updates-24)

[Read blog post](__GHOST_URL__/how-meilisearch-became-open-source-a-retrospective/?utm_campaign=newsletter&utm_source=blog&utm_medium=march-updates-24)

* * *

🦸 We're hiring!
----------------

See our open positions:

*   [Growth Marketing Manager](https://www.meilisearch.com/careers?utm_campaign=newsletter&utm_source=blog&utm_medium=march-updates-24)

* * *

Get these monthly updates straight to your inbox by subscribing to our [newsletter](https://dub.sh/meili-newsletter).

For more things Meilisearch, join our developers community on [Discord](https://discord.meilisearch.com/?ref=blog.meilisearch.com). You can learn more about the product by checking out the [roadmap](https://roadmap.meilisearch.com/?ref=blog.meilisearch.com) and participating in [product discussions](https://github.com/meilisearch/product/discussions?ref=blog.meilisearch.com).

Until next time!

Meilisearch March Updates

_This is part 4 of a series of blog posts_ [_originally published on Clément Renault's blog_](https://blog.kerollmops.com/how-meilisearch-updates-a-millions-vector-embeddings-database-in-under-a-minute)_. Begin the journey with_ [_part 1_](__GHOST_URL__/spotify-inspired-hybrid-search-and-rust/)_,_ [_part 2_](__GHOST_URL__/refining-ann-performance-with-rust/)_, and_ [_part 3_](__GHOST_URL__/arroy-filtered-disk-ann/)_. This blog post considers that you’ve already read parts 1 and 2._

In this blog post, we’ll talk about how we implemented incremental indexing in [Arroy](https://github.com/meilisearch/arroy). By incremental, we mean that when inserting or deleting items in a tree, we can update only the required parts instead of rebuilding everything.

At Meilisearch, that’s essential because our users usually send updates regularly, and their content constantly changes. Rebuilding everything from scratch is very costly and won’t give a good experience; that’s why we already spent a lot of time on our latest release to make all the [data structures support incremental indexing](https://github.com/meilisearch/meiliSearch/releases/tag/v1.6.0) and arroy being one of them, we _HAD TO_ implement it as well!

The theory
----------

The following schema shows a high-level view of what should happen when adding and deleting documents into an existing tree.

As a reminder, arroy is a vector store based on [LMDB](http://www.lmdb.tech/doc/), an embedded transactional memory-mapped key-value store. It stores your vectors in trees composed of three kinds of nodes:

*   The split nodes - Store information on which item IDs are spread out between its left and right.
*   The descendants' nodes - Store multiple item IDs.
*   The items - A single vector.

For the following examples, we’ll consider the descendants node cannot contain **more** than two items and that the closer the IDs are, the closer the items are in space. For example, the item `1` is close to the item `2` but is farther from the item `10`.

![First step before inserting elements in the tree](https://unable-actionable-car.media.strapiapp.com/tamo_1_9ea29125b9.png)

Here, we’re trying to insert items `2` and `7` and delete items `3` and `11`.

Following what we do on the search, we split the items to insert between the left and right split nodes. But the deleted items are no longer present in the database! We don't have their associated vector, and therefore we'll need to iterate over all the leaves to find and delete them.

Fortunately, since we use roaring bitmaps, this list doesn't consume much memory, and since we never update it, we can share it with every thread without ever copying it 🎉

![Second step inserting element in the tree](https://unable-actionable-car.media.strapiapp.com/tamo_2_e4cd099c77.png)

Notice how the items to insert were balanced between the left and right split nodes. In the next step, we’ll follow the same process and split both lists of items to insert on the split node.

![Third step inserting into elements in the tree](https://unable-actionable-car.media.strapiapp.com/tamo_3_ab539d2465.png)

It’s on this step that all the good stuff happens. From the left to the right:

*   The first descendant node gets its item `3` removed, and item `2` added.
*   The second descendant node gets too big and must be replaced by a new split node.
*   The item `8` becomes a descendant containing both the items `7` and `8`.
*   After deleting item `10` in the last descendant node, we replace it with a direct link to an item to reduce the number of nodes in the tree (and improve search time by reducing the number of lookups).

![Last step inserting elements in the tree](https://unable-actionable-car.media.strapiapp.com/tamo_4_b07ebb4b8f.png)

Now that you’ve seen all the steps of our incremental indexing process, we’re still going to take a few notes on what happened:

*   In the first three steps, we had to read all of the tree nodes, which, by default, is equal to the total number of items in the database.
*   In the last step, we had to write four new tree nodes. Previously we would have rewritten the whole tree. The number of writes, which is the slowest operation in databases, is reduced to the bare minimum.
*   The process works on a single tree which means we can still multi-thread the computation of every tree.
*   The IDs aren’t sequential anymore, which doesn't play well with the ID generation strategy described in [Writing the tree in parallel](https://blog.kerollmops.com/multithreading-and-memory-mapping-refining-ann-performance-with-arroy).

How did we fix the ID generation?
---------------------------------

At the start of the indexing process, we need to gather all the existing tree IDs.  
The information we need to generate new nodes are:

*   The total number of tree nodes used to know when to stop building trees.
*   The « holes » in the used IDs that we want to re-use. This fragmentation is produced when we edit the trees.
*   The highest ID existing in the trees to generate the next new IDs.

![Selecting the next ID in the Roaring Bitmap](https://unable-actionable-car.media.strapiapp.com/tamo_5_1f7bb6a273.png)

The « simple » idea is to have a counter for the total number of tree nodes. A counter that we can increment to generate new fresh IDs. The most challenging part is to pick an available ID shared between threads and without big mutexes.

It took me quite some time to find a solution for this last point; I implemented a whole complex solution (150loc) that I ended up [rewriting entirely from scratch](https://github.com/meilisearch/arroy/pull/56/commits/0bf68e21cfd97ff00c7ef5a0ec116ba34c43c6a5) in one hour for an easy and more efficient solution. We're going to see all the ideas we went through before finding our final solution.

### Sharing an iterator over the available IDs

The first and most straightforward idea that comes to mind is to create and share an iterator over all the available IDs.

```rust
let mut available_ids = available_ids.into_iter();

roots.into_par_iter().for_each(|root| {
    // stuff
    let next_id = available_ids.next();
    // stuff
});
```

If you’re familiar with Rust, you’ll notice immediately that the `for_each` _cannot_ capture a mutable reference over the `available_ids` because we use the `into_par_iter` method from Rayon that execute the next methods on the iterator in a multi-threaded way. Which means we can't call `.next()`.

By using a mutex (short for [mutual exclusion](https://en.wikipedia.org/wiki/Mutual_exclusion)), we can make it compile:

```rust
let mut available_ids = Mutex::new(available_ids.into_iter());

roots.into_par_iter().for_each(|root| {
    // stuff
    let next_id = available_ids.lock().unwrap().next();
    // stuff
});
```

That is safe and will yield the right results, but won’t scale well as all the threads have to wait for each other on the lock.

### Let each thread own its list of IDs

When you need to remove a lock, a common solution is to split the work beforehand so each thread can work to its full capacity without ever having to know what’s happening on the other threads. That’s the solution [I implemented initially](https://github.com/meilisearch/arroy/pull/56/files/dc782ea61ef84f0ad8ad20e092476494d6eacc0c). The idea is to explode the roaring bitmap into as many smaller roaring bitmaps as there are trees to update.

The following function can split a roaring bitmap into `n` sub-bitmaps of equal size:

```rust
pub fn split_ids_in(available: RoaringBitmap, n: usize) -> Vec<RoaringBitmap> {
    // Define the number of elements per sub-bitmap
    let chunk_size = available.len() / n as u64;
    let mut ret = Vec::new();

    let mut iter = available.into_iter();
    for _ in 0..(n - 1) {
        // Create a roaring bitmap of `chunk_size` elements from the iterator without consuming it
        let available: RoaringBitmap = iter.by_ref().take(chunk_size as usize).collect();
        ret.push(available);
    }
    // the last element is going to contain everything remaining
    ret.extend(iter);

    ret
}
```

With this function available, it's then trivial to use one bitmap per tree to update with the [`zip`](https://docs.rs/rayon/latest/rayon/iter/trait.IndexedParallelIterator.html#method.zip) method on the parallel iterators of Rayon:

```rust
let available_ids = split_ids_in(available_ids, roots.len());

roots.into_par_iter().zip(available_ids).for_each(|(root, available_ids)| {
    let mut available_ids = available_ids.into_iter();
    // stuff
    let next_id = available_ids.next();
    // Here, we can use the available_ids without any locks or inter-thread synchronization
    // stuff
});
```

This works well while we're updating the already existing root trees. But later on, we may want to create new trees from scratch, and we can't know in advance how many new trees we’ll need to create. That's an issue because, without this information, we don't know how many sub-bitmaps we need to create.

At this point, I couldn’t see an easy fix to this problem, but I made the assumption that we rarely create a lot of new trees and that all new trees would use a lot of IDs. This means, giving all the IDs to the **first** new tree would _probably work out_ (it’s hard to be 100% sure without monitoring it in prod).

### The final solution

The previous solution was complex and did not even work perfectly. While I was scrolling through the documentation of the Roaring Bitmap, I saw [the `select` method](https://docs.rs/roaring/latest/roaring/bitmap/struct.RoaringBitmap.html#method.select) and immediately understood how I could make the initial approach works.

This method lets you get the values in your bitmap by index in an efficient way.  
For example, with the following values in a bitmap: `13, 14, 15, 98, 99, 100`:

*   `select(0)` returns `13`.
*   `select(3)` returns `98`.
*   `select(5)` returns `100`.
*   `select(6)` returns `None`.

With that in mind, if we share the bitmap with all threads in **read-only** and share the index in the bitmap as an atomic number, we can have multiple threads get available IDs at the same time with lock-free synchronization.

Once the `select` method returns `None`, it means we can stop picking IDs from the bitmap and instead simply generate new IDs from scratch with the old method.  
Even if it's super fast, calling the `select` method still takes some time, so once a thread gets a `None` returned from the method, we're going to update an atomic boolean telling us if there are still values to look at in the bitmap. If not, we can skip calling `select` and directly generate a new ID.

With that in mind, the cool `ConcurrentNodeIds` structure we showed you before in part 2 got a little bit more complex but it still let us generate IDs fairly without any locks!

```rust
/// A concurrent ID generator that will never return the same ID twice.
pub struct ConcurrentNodeIds {
    /// The current tree node ID we should use if no other IDs are available.
    current: AtomicU32,
    /// The total number of tree node IDs used.
    used: AtomicU64,

    /// A list of IDs to exhaust before picking IDs from `current`.
    available: RoaringBitmap,
    /// The current Nth ID to select in the bitmap.
    select_in_bitmap: AtomicU32,
    /// Tells if you should look in the roaring bitmap or if all the IDs are already exhausted.
    look_into_bitmap: AtomicBool,
}

impl ConcurrentNodeIds {
    /// Creates an ID generator returning unique IDs, avoiding the specified used IDs.
    pub fn new(used: RoaringBitmap) -> ConcurrentNodeIds {
        let last_id = used.max().map_or(0, |id| id + 1);
        let used_ids = used.len();
        let available = RoaringBitmap::from_sorted_iter(0..last_id).unwrap() - used;

        ConcurrentNodeIds {
            current: AtomicU32::new(last_id),
            used: AtomicU64::new(used_ids),
            select_in_bitmap: AtomicU32::new(0),
            look_into_bitmap: AtomicBool::new(!available.is_empty()),
            available,
        }
    }

    /// Returns a new unique ID and increases the count of IDs used.
    pub fn next(&self) -> Result<u32> {
        if self.look_into_bitmap.load(Ordering::Relaxed) {
            let current = self.select_in_bitmap.fetch_add(1, Ordering::Relaxed);
            match self.available.select(current) {
                Some(id) => Ok(id),
                None => {
                    self.look_into_bitmap.store(false, Ordering::Relaxed);
                    Ok(self.current.fetch_add(1, Ordering::Relaxed))
                }
            }
        } else {
            Ok(self.current.fetch_add(1, Ordering::Relaxed))
        }
    }
}
```

Performances in practice
------------------------

I haven’t run a lot of benchmarks yet (I would like to, though, and may update this post later), but on my benchmarks with a few hundred thousand items, here are the results I get:

![Performance improvement showed on a graph](https://unable-actionable-car.media.strapiapp.com/tamo_6_8c737d7e95.png)

*   On average, we get more than a 10x speedup after three batches of items indexed.
*   The little bumps we observe afterward that make us go from ~700ms to ~3s are due to the creation of new trees as the number of items increases.
*   I expect this benchmark to represent a worst-case scenario:
    *   All my item IDs are randomly generated, which means there are very few duplicates/updates but always new insertions and more writing.
    *   My vectors are also randomly generated with a uniform distribution, which means there shouldn’t be clusters as we observe in a real-life dataset.

This feature is now being used on production data with millions of documents, and we've seen numbers in the range of 9000 items added on a 2M items database in under a minute.

* * *

[Meilisearch](https://meilisearch.com/?ref=blog.meilisearch.com) is an open-source search engine enabling developers to build state-of-the-art experiences while enjoying simple, intuitive DX. 

For more things Meilisearch, you can join the community on [Discord](https://discord.gg/meilisearch/?ref=blog.meilisearch.com) or subscribe to the [newsletter](https://dub.sh/meili-newsletter). You can learn more about the product by checking out the [roadmap](https://roadmap.meilisearch.com/?ref=blog.meilisearch.com) and participating in [product discussions](https://github.com/orgs/meilisearch/discussions?ref=blog.meilisearch.com).

How Meilisearch updates a database with millions of vector embeddings in under a minute

Navigating today’s ocean of information can be both a marvel and a maze. Full-text and vector search enable us to build search experiences empowering users to find relevant products, content, and more. As our quest for precision and context in search evolves, a question arises: can we balance the lexical flexibility of full-text search with the semantic depth of vector search?

Let’s explore together the pros and cons of each solution and discover the synergies that are redefining modern search & discovery experiences.

Full-text search
----------------

[Full-text search](https://www.meilisearch.com/docs/learn/what_is_meilisearch/overview?utm_campaign=vector-search&utm_source=blog&utm_medium=full-text-vs-vector#features) refers to matching some or all of a text query with documents stored in a database. Compared to traditional database queries, full-text search provides results even in case of partial matches. It allows building more flexible search interfaces for users, thus enabling them to find accurate results more quickly.

In practice, an efficient full-text search solution comes with tolerance to typos, synonyms, prefix search, and fuzzy matching. These features enable end users to find what they’re looking for even when the information at their disposal is limited. Allowing searches with incomplete information makes discovery faster.

Although modern SQL and NoSQL databases come with some full-text search capabilities, they often fall short compared to search-focused databases. For a deeper dive on this topic, consider reading on areas where [Postgres falls short for full-text search](__GHOST_URL__/postgres-full-text-search-limitations/?utm_campaign=vector-search&utm_source=blog&utm_medium=full-text-vs-vector).

<Callout>
  Learn about the [inner workings of a search engine](__GHOST_URL__/how-full-text-search-engines-work/?utm_campaign=vector-search&utm_source=blog&utm_medium=full-text-vs-vector) in our dedicated article.
</Callout>

Vector search
-------------

[Vector search](https://www.meilisearch.com/solutions/vector-search?utm_campaign=vector-search&utm_source=blog&utm_medium=full-text-vs-vector) is the AI-powered approach to search. Instead of looking for documents matching a text query, it allows finding documents that have a similar _semantic_ meaning. This is achieved by building upon the semantic understanding of text that [Large Language Models](https://en.wikipedia.org/wiki/Large_language_model) (LLMs) provide.

LLMs can process database records and generate [vector embeddings](__GHOST_URL__/what-are-vector-embeddings/?utm_campaign=vector-search&utm_source=blog&utm_medium=full-text-vs-vector)—the numeric representation of the semantic meaning of documents. These embeddings are then stored in [vector databases](__GHOST_URL__/what-is-a-vector-database/?utm_campaign=vector-search&utm_source=blog&utm_medium=full-text-vs-vector). As the name suggests, vector search works by comparing similar vectors.

By performing vector comparisons, vector search effectively enables [semantic search](__GHOST_URL__/what-is-a-vector-database/?utm_campaign=vector-search&utm_source=blog&utm_medium=full-text-vs-vector#what-is-similarity-search). Despite relying on different underlying technologies, it also solves a similar purpose as full-text search: empowering users by improving the discovery aspect of search. It notably enables multi-modal search—searching with non-textual input, like in Google Images.

Comparative analysis: full-text search vs vector search
-------------------------------------------------------

Full-text search works by finding words in documents. That makes it good at understanding the **lexical intent** behind a user query. But it falls short when it has to work with vague queries that cannot be fuzzy-matched. The lack of semantic understanding of documents makes it unable to understand context and make generalizations.

To summarize, **full-text search**:

*   ✅ Corrects typos — e.g. `reutrn of the jedi`
*   ✅ Handles precise queries — e.g. `an exact product name`
*   ✅ Handles incomplete queries — e.g. `return of the j`
*   ❌ Cannot handle vague queries — e.g. `folks fighting with lightsabers`
*   ❌ Cannot understand context — e.g. `winter clothes`

On the other hand, vector search works by finding documents with similar meanings. That makes it good at understanding the **semantic intent** behind a user query. It’s great at understanding synonyms and descriptions of documents. It’s also adapted to the discovery of similar documents.

To summarize, **vector search**:

*   ✅ Handles vague queries — e.g. first released star wars movie
*   ✅ Understands context — e.g. winter clothes
*   ✅ Can suggest similar documents
*   ❌ Cannot handle precise queries
*   👎 Requires more resources

As we can see, neither solution is perfect. They each come with clear advantages for given use cases. They complement each other. Building state-of-the-art search experiences would require combining the benefits of both full-text and vector search.

![“Why don't we have both?“ meme](https://unable-actionable-car.media.strapiapp.com/why_dont_we_have_both_1e31f67a5c.gif)

Indeed, can we have both?

The case for hybrid search
--------------------------

Hybrid search combines the strengths of full-text search and vector search. It builds upon the accessible, search-as-you-type experience of full-text search and integrates the enhanced discovery capabilities that AI search enables.

Solutions like Meilisearch’s [hybrid search](https://www.meilisearch.com/solutions/hybrid-search?utm_campaign=vector-search&utm_source=blog&utm_medium=full-text-vs-vector) bring the best of both worlds without compromise. This allows developers to use a unified API for all types of searches. The code sample below demonstrates a usage of such API.

```js
// Hybrid search with Meilisearch

const results = await client.multiSearch({
	queries: [{ 
		indexUid: 'movies', 
		q: 'batman',
		hybrid: { embedder: 'default', semanticRatio: 0.5 }
	}]
})

```
<Callout>
  Not using JavaScript? Meilisearch supports [multiple SDKs](https://www.meilisearch.com/docs/learn/what_is_meilisearch/sdks?utm_campaign=vector-search&utm_source=blog&utm_medium=full-text-vs-vector).
</Callout>

This code uses a Meilisearch client to search for the `batman` query in the movies index. It uses hybrid search, which is configured via the `hybrid` parameter. Let’s review the fields in the `hybrid` object and see what they enable.

The `embedder` field allows choosing which embedder will process the query (Meilisearch allows [configuring multiple embedders](https://www.meilisearch.com/docs/learn/experimental/vector_search?utm_campaign=vector-search&utm_source=blog&utm_medium=full-text-vs-vector#generate-vector-embeddings) for a given index.)

The `semanticRatio` field allows us to balance the importance of results coming from semantic search. A higher value will allow documents that are further distanced in the [semantic space](__GHOST_URL__/what-are-vector-embeddings/?utm_campaign=vector-search&utm_source=blog&utm_medium=full-text-vs-vector#what-is-the-semantic-space) to make it to the final search results.

The `semanticRatio` value works as follows:

*   Semantic ratio = 0 — it’s a full-text search
*   Semantic ratio = 1 — it’s a vector search
*   0 < semantic ratio < 1 — it’s a hybrid search ✨

As semantic search will always return _some_ documents, it’s important to strike a good balance to keep the overall search results relevant. The demo below allows you to search for movies and play with the semantic ratio to see how it affects results.

[![Searching for movies using hybrid search](https://unable-actionable-car.media.strapiapp.com/hybrid_search_gif_cfa6283c31.gif)](https://where2watch.meilisearch.com/?utm_campaign=vector-search&utm_source=blog&utm_medium=full-text-vs-vector)

Searching for ”green ogre in fairy tale parody” ([Live demo](https://where2watch.meilisearch.com/?utm_campaign=vector-search&utm_source=blog&utm_medium=full-text-vs-vector))

* * *

Hybrid search is coming to Meilisearch Cloud, join the waitlist:

[Get early access](https://share-eu1.hsforms.com/1RN-rGYhHQPu5BNivfknlwAfg3aq?utm_campaign=vector-search&utm_source=blog&utm_medium=full-text-vs-vector)

Conclusion
----------

As engineers will tell you, the question of “What is the best tool for X?” is always: _it depends_. On their own, neither full-text search nor vector search can meet all the criteria to build a fast, relevant search experience. Hybrid search merges the accessibility of full-text search with improved discovery that AI enables.

**Hybrid search** is the modern approach to search, unifying behind a single API the most advanced search capabilities. This hybrid approach blends full-text search with semantic search, enhancing both the accuracy and comprehensiveness of search results. It is a modern, flexible solution adapted to today’s search use cases.

* * *

[Meilisearch](https://meilisearch.com/?utm_campaign=vector-search&utm_source=blog&utm_medium=full-text-vs-vector) is an open-source search engine enabling developers to build state-of-the-art experiences while enjoying simple, intuitive DX. 

For more things Meilisearch, you can join the community on [Discord](https://discord.meilisearch.com/?utm_campaign=vector-search&utm_source=blog&utm_medium=full-text-vs-vector) or subscribe to the [newsletter](https://dub.sh/meili-newsletter?utm_campaign=vector-search&utm_source=blog&utm_medium=full-text-vs-vector). You can learn more about the product by checking out its [roadmap](https://roadmap.meilisearch.com/?utm_campaign=vector-search&utm_source=blog&utm_medium=full-text-vs-vector) and participating in [product discussions](https://github.com/orgs/meilisearch/discussions).

Full-text search vs vector search

We're excited to unveil Meilisearch v1.7. In this article, we’ll review the most impactful changes. For an exhaustive listing, check out the [changelog](https://github.com/meilisearch/meilisearch/releases/tag/v1.7.0) on Github.

<Callout>
  Meilisearch 1.7 is available on [Meilisearch Cloud](https://cloud.meilisearch.com/?utm_campaign=release-v1-7&utm_source=blog) too—upgrade now!
</Callout>

Improved hybrid & vector search
-------------------------------

<Callout>
  AI search is an experimental feature, read the docs to [enable AI search](https://www.meilisearch.com/docs/learn/experimental/vector_search?utm_campaign=release-v1-7&utm_source=blog#activate-vector-search).
</Callout>

### New OpenAI embedding models

Starting with Meilisearch 1.7, you can use OpenAI’s third-generation embedding models:

*   `text-embedding-3-small`
*   `text-embedding-3-large`

Compared to the previous `text-embedding-ada-002` model, these new models offer improvements allowing to balance search results relevancy and cost efficiency.

The `small` model is more cost-effective, generating smaller vector embeddings that require less storage. The `large` focuses on delivering higher precision in search results, making it ideal to optimize relevancy.

### Configuring OpenAI model dimensions

When using the new `text-embedding-3-small` and `text-embedding-3-large` OpenAI models, you can now configure **dimensions** for your model. When not provided, the model's defaults dimensions are used.

You can do so when updating your settings:

```json
{
  // other settings...

  "embedders": {
    "small": {
      "source": "openAi",
      "model": "text-embedding-3-large",
      "dimensions": 512 // must be >0, must be <= 3072 for "text-embedding-3-large"
    },
  }
}

```

Higher dimensions typically offer more complex embeddings which can lead to more accurate search results. However, increased dimensions are more costly to compute and require more storage.

<Callout>
  You cannot customize dimensions when using OpenAI `text-embedding-ada-002`.
</Callout>

### Use AI Search on Meilisearch Cloud

Meilisearch hybrid search and vector search are coming to Meilisearch Cloud. We're rolling out invites every day. Join the beta. 👇

[Get early access](https://share-eu1.hsforms.com/1RN-rGYhHQPu5BNivfknlwAfg3aq?utm_campaign=release-v1-7&utm_source=blog)

GPU support for computing Hugging Face embeddings
-------------------------------------------------

Meilisearch 1.7 comes with the ability to enable GPU support for [self-hosted Meilisearch](https://www.meilisearch.com/docs/learn/getting_started/installation?utm_campaign=release-v1-7&utm_source=blog) for computing Hugging Face embeddings. This can significantly speed up the embedding generation process.

Find detailed instructions in our dedicated guide on [Using GPU to compute Hugging Face embeddings](https://www.meilisearch.com/docs/learn/cookbooks/computing_hugging_face_embeddings_gpu?utm_campaign=release-v1-7&utm_source=blog).

Display ranking scores details at search
----------------------------------------

Meilisearch 1.7 stabilizes the `showRankingScoreDetails` search parameter that was introduced in [Meilisearch 1.3](__GHOST_URL__/v1-3-release/?utm_campaign=release-v1-7&utm_source=blog). This parameter adds a `_rankingScoreDetails` field to each document in search results, enabling further inspection of the search results.

You can include the parameter when making searches:

```sh
curl \
  -X POST 'https://edge.meilisearch.com/indexes/movies/search' \
  -H 'Content-Type: application/json' \
  --data-binary '{
    "q": "batman",
    "showRankingScoreDetails": true
  }'

```

Learn more about the [`_rankingScoreDetails` object](https://www.meilisearch.com/docs/reference/api/search#ranking-score-details?utm_campaign=release-v1-7&utm_source=blog) and how to use it with your preferred SDK in the API reference.

Contributors shout-out
----------------------

We want to give a massive thank you to all contributors that made this release possible: [@capJavert](https://github.com/capJavert), [@codesmith-emmy](https://github.com/codesmith-emmy), [@Elliot67](https://github.com/Elliot67) and [@Gosti](https://github.com/Gosti) for their PRs in Meilisearch, and [@agourlay](https://github.com/agourlay), [@choznerol](https://github.com/choznerol), [@ngdbao](https://github.com/ngdbao), [@timvisee](https://github.com/timvisee), and [@xshadowlegendx](https://github.com/xshadowlegendx) for their work on [charabia](https://github.com/meilisearch/charabia).

Special thanks to our SDKs maintainers, thanks to whom Meilisearch is available across so many languages. 🫶

* * *

And that’s a wrap for v1.7! This release post highlights the most significant updates. For an exhaustive listing, read the [changelog on Github](https://github.com/meilisearch/meilisearch/releases/tag/v1.7.0).

Stay in the loop of everything Meilisearch by subscribing to our [newsletter](https://dub.sh/meili-newsletter). To learn more about Meilisearch's future and help shape it, take a look at our [roadmap](https://roadmap.meilisearch.com/?utm_campaign=release-v1-7&utm_source=blog) and come participate in our [Product Discussions](https://github.com/meilisearch/product/discussions).

For anything else, join our developers community on [Discord](https://discord.meilisearch.com/?utm_campaign=release-v1-7&utm_source=blog).

Meilisearch latest news and company updates