🦎 Releasing Meilisearch 1.9
----------------------------

[![](https://unable-actionable-car.media.strapiapp.com/meilisearch_1_9_d1239d3d00.jpg)](/blog/meilisearch-1-9/?utm_campaign=newsletter&utm_source=blog&utm_medium=july-24-updates)

Meilisearch 1.9 brings **similar documents**, **ranking score threshold**, **grouping by attribute**, and **improved AI-powered search**.

[Discover updates](/blog/meilisearch-1-9/?utm_campaign=newsletter&utm_source=blog&utm_medium=july-24-updates)

⚖️ Algolia comparison
---------------------

![](https://unable-actionable-car.media.strapiapp.com/meilisearch_vs_algolia_cover_5c7691f971.jpg)

We published a **comparison of the key features** between Meilisearch and Algolia to help you choose search engine that fits your needs.

[Read comparison](/blog/meilisearch-vs-algolia/?utm_campaign=newsletter&utm_source=blog&utm_medium=july-24-updates)

📌 Community picks
------------------

### 📢 [RFC: Database synchronization tooling](https://github.com/orgs/meilisearch/discussions/765)

We're collecting feedback on tooling to synchronize Meilisearch with your database.

### 📢 [RFC: Improving the search preview](https://github.com/orgs/meilisearch/discussions/766)

We want to make testing your search easier, and we need your feedback!

### 🎮 [A better way to build Meilisearch filters in TypeScript](https://levelup.gitconnected.com/a-better-way-to-build-meilisearch-filters-d88846c280d1)

Check out the meilisearch-helper package to build filters with a fluent API.

### ⚡️ [Lightning-fast Laravel search with Meilisearch](https://www.youtube.com/watch?v=FY_x9_eX1AQ)

Watch this video to learn how to build full-text search for Laravel with Meilisearch.

### 📽️ [Product live event](/blog/heres-what-you-missed-q2-2024/?utm_campaign=newsletter&utm_source=blog&utm_medium=july-updates-24#check-out-the-recording-of-the-event-for-more-details)

Watch the replay to learn more about our recent releases and what's coming next.

🤗 Made with Meilisearch
------------------------

A community member built [RemoteJobs.org](https://remotejobs.org) and shared it with the Discord community. It's a remote jobs board focussing on quality offers. It is, of course,  powered by Meilisearch. 🎉

[![](https://unable-actionable-car.media.strapiapp.com/Screenshot_2024_08_01_at_18_55_39_59f37af3bf.png)](https://remotejobs.org)

_We’re always looking to promote new projects from the community. Post yours in the_ [_#built-with-meilisearch_](https://discord.meilisearch.com) _channel!_

🦸‍♀️ We're hiring!
-------------------

See our open positions:

*   [Growth Marketing Manager](https://www.meilisearch.com/careers?utm_campaign=newsletter&utm_source=blog&utm_medium=july-updates-24)
*   [Senior Full-Stack Software Engineer](https://www.meilisearch.com/careers?utm_campaign=newsletter&utm_source=blog&utm_medium=july-updates-24)
*   [Founding Sales & Business Developer](https://www.meilisearch.com/careers?utm_campaign=newsletter&utm_source=blog&utm_medium=july-updates-24)

* * *

Get these monthly updates straight to your inbox by subscribing to our [newsletter](https://dub.sh/meili-newsletter?utm_campaign=newsletter&utm_source=blog&utm_medium=july-24-updates). 💌

For more things Meilisearch, join our developers community on [Discord](https://discord.meilisearch.com/?ref=blog.meilisearch.com). You can learn more about the product by checking out the [roadmap](https://roadmap.meilisearch.com/?ref=blog.meilisearch.com) and participating in [product discussions](https://github.com/meilisearch/product/discussions?ref=blog.meilisearch.com).

Until next time!

Meilisearch July updates

Deciding between Algolia and Meilisearch for your search solution? We're here to help! In this article, we'll compare their features, capabilities, pricing models, and overall strengths and weaknesses.

What is Algolia?
----------------

Founded in 2012, Algolia is a hosted search engine that provides full-text and AI-powered search capabilities. It offers an API that integrates fast, customizable search experiences into websites and mobile applications.

Algolia is known for its remarkable speed and scalability, making it a go-to choice for enterprises looking for sophisticated search functionalities. However, these premium features come with a higher cost, Algolia tends to be more suitable for larger businesses with bigger budgets.

What is Meilisearch?
--------------------

Launched in 2018, [Meilisearch](https://meilisearch.com/?utm_campaign=comparisons&utm_source=blog&utm_medium=vs-algolia) is an open-source search engine that features full-text and AI-powered search capabilities Designed to be developer-friendly and plug-and-play, it addresses most common use cases straight out of the box. Built with Rust, it ensures exceptional performance, security, and flexibility.

Known for its lightning-fast search speeds and modern development environment, Meilisearch is an excellent choice for small-to-medium growing businesses seeking high-performance search functionalities.

Unlike Algolia, Meilisearch is [open-source](https://github.com/meilisearch/meilisearch). You can self-host it or use the [managed Meilisearch Cloud service](https://meilisearch.com/cloud?utm_campaign=comparisons&utm_source=blog&utm_medium=vs-algolia), allowing for flexibility and cost-efficiency.

Algolia vs Meilisearch - Key features compared
----------------------------------------------

### Developer experience

Algolia and Meilisearch both prioritize ease of use, offering straightforward setup processes and comprehensive documentation. These search engines work out-of-the-box, providing great relevancy in most cases without the need for additional configurations. They feature intuitive REST APIs and place a strong emphasis on the developer experience. Meilisearch's open-source nature is a significant asset, as it enables better feedback collection and faster implementations.

When comparing Meilisearch Cloud and Algolia dashboards, Algolia’s interface stands out for its advanced and comprehensive UI. This is understandable, given that Algolia is a fully-hosted search engine that has been in the market for over a decade. In contrast, Meilisearch, which launched its cloud offering much more recently, is still developing its dashboard.

Overall, the Algolia dashboard feels more beginner-friendly and integrated, guiding users smoothly from data addition to frontend integration. Meilisearch, on the other hand, partially relies on its [documentation](https://www.meilisearch.com/docs/learn/getting_started/cloud_quick_start?utm_campaign=comparisons&utm_source=blog&utm_medium=vs-algolia) to guide users through the process. However, it’s important to note that Meilisearch’s dashboard is continually evolving based on user feedback, showing significant potential for improvement.

### SDKs and integrations

Both Algolia and Meilisearch provide [API clients](https://www.meilisearch.com/docs/learn/what_is_meilisearch/sdks?utm_campaign=comparisons&utm_source=blog&utm_medium=vs-algolia) for a variety of popular programming languages, including JavaScript, .NET, Python, PHP, and more, ensuring developers can easily integrate search functionalities into their applications.

Algolia offers a significantly broader selection of SDKs and integrations, particularly in the realm of specialized e-commerce platforms such as Shopify and Adobe Commerce (formerly Magento). This extensive range of integrations makes Algolia a versatile choice for businesses with specific platform requirements.

### Relevance and typo tolerance

Meilisearch and Algolia both deliver excellent relevancy and typo tolerance right out of the box, with options for customization to fine-tune search results. They offer similar features, including synonyms, deduplication, sorting, filtering, faceting, and geosearch.

However, configuring some of these features in Meilisearch can feel more straightforward. Certain functionalities, such as [sorting](https://www.meilisearch.com/docs/learn/fine_tuning_results/sorting#sorting?utm_campaign=comparisons&utm_source=blog&utm_medium=vs-algolia), require fewer configuration steps compared to Algolia. This simplicity makes Meilisearch an attractive option for developers looking for ease of setup and use.

Moreover, unlike Algolia, Meilisearch does not require replicating or copying indexes for each sort criterion, making cost management much more manageable and fair for customers.

Algolia, on the other hand, provides a more extensive feature set with advanced configuration options. Features like filter weighting offer more customization, but this can also add complexity. While Algolia's features can sometimes offer more advanced configuration, Meilisearch is rapidly closing the gap, [regularly launching new features](https://github.com/meilisearch/meilisearch/releases?utm_campaign=comparisons&utm_source=blog&utm_medium=vs-algolia) that enhance its capabilities.

### AI-powered search

Both Algolia and Meilisearch offer advanced semantic search capabilities, but with notable differences in accessibility and customization.

Algolia provides NeuralSearch, its approach to hybrid search, which combines neural search and full-text search results. Algolia leverages AI at multiple levels, including search, ranking, and query suggestions. This feature allows you to choose between keyword search and neural search modes. When using Algolia, text embedding is managed by their proprietary model, which is automatically applied to the documents. However, NeuralSearch is not widely available; it is reserved for the highest plan (Elevate) and requires event data from the front end. In practice, many users have yet to see a running instance of NeuralSearch. 

Meilisearch, on the other hand, offers a more customizable [AI-powered search](https://www.meilisearch.com/solutions/vector-search?utm_campaign=comparisons&utm_source=blog&utm_medium=vs-algolia) experience. It uses state-of-the-art large language model (LLM) providers such as OpenAI and HuggingFace to generate vector embeddings. Additionally, Meilisearch allows you to configure multiple embedders for an index and supports custom embeddings, ensuring flexibility and adaptability. Its [hybrid search](https://www.meilisearch.com/solutions/hybrid-search?utm_campaign=comparisons&utm_source=blog&utm_medium=vs-algolia) approach is highly customizable, enabling you to adjust the semantic ratio for each hybrid search query, ranging [from full semantic to full-text search](/blog/full-text-search-vs-vector-search/?utm_campaign=comparisons&utm_source=blog&utm_medium=vs-algolia Meilisearch's AI-powered search is available both when self-hosting and on Meilisearch Cloud, making it accessible to a wide range of users.

### Analytics & data visualization

Both Meilisearch and Algolia provide robust analytics and data visualization tools to help you monitor and optimize search performance, but they offer these insights in slightly different ways.

Meilisearch offers a [comprehensive set of analytics](https://www.meilisearch.com/docs/learn/cloud/analytics?utm_campaign=comparisons&utm_source=blog&utm_medium=vs-algolia), including metrics such as total searches, user activity, and key search effectiveness indicators like no result rate and click-through rate. You can visualize search trends over time and gain insights into user behavior, including the most searched queries and geographic distribution of searches.

Algolia, on the other hand, provides a more extensive suite of analytics and data visualization tools. Its overview dashboard includes similar key metrics like total searches, user activity, and conversion rates. However, Algolia goes further by offering advanced analytics providing deeper insights into popular filter values and top-performing results.

A notable feature that sets Algolia apart is its A/B testing capability. This allows you to compare an index and its replica with different settings, evaluating metrics like click-through rates and conversion rates. 

In summary, while both Meilisearch and Algolia offer essential analytics tools, Algolia includes more detailed and advanced options, including A/B testing, which can be particularly beneficial for optimizing user interactions in ecommerce use cases. Meilisearch focuses on core metrics and delivers a straightforward visualization of search trends and user behavior.

### Community and support

Algolia offers a dedicated Discord community with 507 members, accessible to all users, providing a platform for discussions, questions, and support. For more personalized assistance, email and website support are available starting from the Grow plan. Additionally, Algolia provides Standard and Extended Support options, though pricing is not disclosed and is only available through sales inquiries.

Meilisearch, being an open-source search engine, has fostered a robust community. Its [Discord community](https://discord.meilisearch.com/?utm_campaign=comparisons&utm_source=blog&utm_medium=vs-algolia) boasts 2,441 members, offering a vibrant space for open-source enthusiasts and users to engage, share knowledge, and seek help. For Cloud users, Meilisearch ensures comprehensive support tailored to each plan: the Build plan includes Discord support, the Pro plan offers chat and email support, and the Custom plan provides priority support with dedicated channels. This tiered support structure reflects Meilisearch's commitment to accessible and responsive customer service.

### Pricing and commercial offerings

Algolia offers a mix of free, pay-as-you-go, and custom pricing plans. The Free Plan includes up to 10,000 search requests per month and 1 million records, while the Grow Plan features a clear pay-as-you-go structure with specific costs for additional search requests and records. However, their Premium and Elevate Plans do not have disclosed pricing and require users to request details from sales.  

In contrast, Meilisearch Cloud has fixed monthly pricing for its Build and Pro plans, priced at $30 and $300 per month, respectively, with specified features and resources included. For larger projects, Meilisearch has a Custom Plan that requires a custom quote.

While Meilisearch does not offer a free tier, it starts at a relatively low monthly cost with substantial resources and support. Additionally, it includes a 14-day free trial without requiring a credit card, making it easy for potential users to explore its features. Moreover, Meilisearch being open-source, it can also be [self-hosted for free](https://www.meilisearch.com/docs/learn/getting_started/installation#local-installation?utm_campaign=comparisons&utm_source=blog&utm_medium=vs-algolia).

Although Algolia might save some costs in specific scenarios, its complexity and potential hidden costs can lead to higher expenses in practice. Meilisearch, with its clear and predictable [pricing model](https://www.meilisearch.com/pricing?utm_campaign=comparisons&utm_source=blog&utm_medium=vs-algolia), is often more attractive for businesses that prefer straightforward budgeting and cost transparency.

### Global coverage

Meilisearch Cloud has server regions in major cities such as San Francisco, New York, London, Frankfurt, and Singapore, strategically positioning itself in key commercial hubs across North America, Europe, and Asia. This coverage allows Meilisearch to provide reliable performance and data residency options in these important regions.

Algolia has a much broader network with clusters deployed in 16 regions globally. This extensive coverage spans the US, Europe, Asia, and additional regions like Brazil, Japan, and Australia. Algolia’s expansive network allows for greater flexibility in data localization, enhancing performance and compliance with local regulations.

Looking ahead, Meilisearch plans to expand its reach by adding more server regions to its network. This expansion will broaden its global coverage, further enhancing its service offerings.

Algolia challenges
------------------

Algolia faces several notable challenges, one of which is its high cost. While Algolia does offer a free tier, it is limited, and scaling up can quickly become expensive. Algolia's powerful features are well-suited for large enterprises, but smaller businesses might find these advanced features and the associated costs overwhelming. The lack of transparency for its Premium and Elevate plans further complicates budgeting, as potential customers must engage with the sales team to get detailed pricing information, which can deter smaller businesses that need to understand costs upfront. 

Additionally, fully leveraging Algolia’s extensive features and capabilities often requires significant customization and technical expertise. This can be a barrier for businesses without dedicated technical teams. 

Furthermore, Algolia's reliance on its proprietary models for AI features means users are dependent on Algolia's technology stack. This dependency can limit businesses looking for more flexibility or integration with other AI solutions.

Meilisearch challenges
----------------------

Meilisearch, despite its strengths, also faces its own set of challenges. One notable challenge is the absence of a completely free tier in the managed cloud version, which might deter very small businesses or startups looking to minimize costs. While Meilisearch's pricing is transparent and reasonable, the lack of a free option in the managed cloud version can be a disadvantage compared to competitors like Algolia. However, being open-source, Meilisearch provides a free, self-hosted option that ensures cost savings for those willing to manage their own infrastructure.

While Meilisearch offers essential and robust features, it might lack some of the advanced features and integrations that Algolia provides, particularly for specialized use cases such as advanced e-commerce applications.

When should you consider Algolia over Meilisearch?
--------------------------------------------------

If your business requires sophisticated e-commerce search functionalities, such as merchandising tools, product recommendations, and advanced filtering options, Algolia is currently better equipped to handle these needs. The platform's features are designed to improve product discovery and conversion rates.

For businesses that want to optimize their search configurations through A/B testing, Algolia provides robust tools to compare different search setups. This is especially beneficial in e-commerce, where fine-tuning search performance can significantly impact sales and customer engagement. 

Algolia’s infrastructure is also optimized for global reach, offering high availability and low latency across different regions. If your business operates internationally and requires consistent performance worldwide, Algolia is a strong contender.

In summary, Algolia excels in delivering advanced e-commerce functionalities, comprehensive analytics, and a globally optimized infrastructure. While it may come with higher costs and a steeper learning curve, these features make it particularly suitable for large enterprises and businesses with complex, high-demand search requirements.

When should you consider Meilisearch over Algolia?
--------------------------------------------------

Meilisearch is an excellent choice for businesses that prioritize simplicity, transparency, and flexibility. Its straightforward setup process and developer-friendly design make it ideal for teams seeking a hassle-free implementation. With robust core features that work out of the box, Meilisearch ensures high performance, fast response times, and excellent relevancy without requiring extensive customization or technical expertise.

For organizations that value open-source solutions, Meilisearch is particularly appealing. It benefits from active community contributions and maintenance, fostering a collaborative environment that accelerates innovation. This allows businesses to adapt and extend the search engine according to their specific requirements, providing greater flexibility and control over their search infrastructure.

The open-source model also ensures the search engine remains aligned with user needs, adding a layer of transparency and flexibility. As a result, customers gain better confidence in their ability to shape the product and steer it in the direction they need, enhancing their overall control and adaptability.

Regular updates every two months ensure that Meilisearch users can benefit from the latest advancements and enhancements, keeping the search engine up-to-date with evolving needs and technologies. This continuous improvement cycle makes Meilisearch a strong contender for businesses looking for a dynamic and easy-to-use search solution.

In summary, choose Meilisearch over Algolia if you need a flexible, developer-friendly search engine with transparent pricing and the benefits of an open-source community. Meilisearch’s focus on simplicity, high performance, and continuous innovation makes it a compelling option for businesses of all sizes.

* * *

Learn more about what Meilisearch can bring to your business.

[Talk to a search expert](https://dub.sh/meili-sales?utm_campaign=comparisons&utm_source=blog&utm_medium=vs-algolia)

* * *

Conclusion
----------

Both Algolia and Meilisearch offer robust search solutions with a developer-friendly experience and high performance.

Algolia is particularly strong in the e-commerce space, offering advanced analytics and features tailored to that use case. Its extensive feature set and flexible scalability options make it a great choice for larger enterprises looking for comprehensive search capabilities.

Meilisearch excels with its advanced AI capabilities, fast development cycle, and transparent pricing, making it an excellent choice for tech teams and app builders who want to be efficient and achieve fast ROI while benefiting from a secure, scalable search solution.

If you're considering making the switch, check out our [Migrating from Algolia to Meilisearch](https://www.meilisearch.com/docs/learn/update_and_migration/algolia_migration?utm_campaign=comparisons&utm_source=blog&utm_medium=vs-algolia) guide for a smooth transition.

You can see Meilisearch in action using this online [demo](https://where2watch.meilisearch.com/?utm_campaign=comparisons&utm_source=blog&utm_medium=vs-algolia). Also, feel free to check out its [public roadmap](https://roadmap.meilisearch.com/?ref=blog.meilisearch.com) and [GitHub community](https://github.com/orgs/meilisearch/discussions?ref=blog.meilisearch.com) to see what new and upcoming features you can look forward to.

<Callout intent="tip">

Check out our detailed comparison of the best [Algolia alternatives](https://www.meilisearch.com/blog/algolia-alternatives) in 2025.

</Callout>

Meilisearch vs Algolia

We're excited to unveil Meilisearch v1.9. In this article, we’ll review the most impactful changes. For an exhaustive listing, check out the [changelog](https://github.com/meilisearch/meilisearch/releases/tag/v1.9.0) on GitHub.

<Callout>
  Meilisearch 1.9 is available on [Meilisearch](https://cloud.meilisearch.com/?utm_campaign=release-v1-9&utm_source=blog) Cloud, too—upgrade now!
</Callout>

New: ranking score threshold
----------------------------

Meilisearch 1.9 allows excluding search results with low ranking scores. When using the new `rankingScoreThreshold` option, Meilisearch will not return any documents below the defined threshold.

```bash
curl \
 -X POST 'http://localhost:7700/indexes/movies/search' \
 -H 'Content-Type: application/json' \
 --data-binary '{
    "q": "green ogre living in a swamp",
    "hybrid": { 
	    "semanticRatio": 0.9, 
	    "embedder": "default" 
	},
    "showRankingScore": true,
    "limit": 5,
    "rankingScoreThreshold": 0.2
 }'
```

Using the ranking score threshold when implementing [hybrid search](https://www.meilisearch.com/docs/reference/api/search?utm_campaign=release-v1-9&utm_source=blog#hybrid-search-experimental) removes irrelevant results and allows your [search analytics](https://www.meilisearch.com/docs/learn/analytics/analytics?utm_campaign=release-v1-9&utm_source=blog) to collect the _No search results_ metric properly.

<Callout>
  Excluded results do not count towards `estimatedTotalHits`, `totalHits`, or facet distribution.
</Callout>

New: distinct attribute at search time (group by)
-------------------------------------------------

Meilisearch 1.9 adds the ability to define a [distinct attribute](https://www.meilisearch.com/docs/learn/configuration/distinct?utm_campaign=release-v1-9&utm_source=blog) at search time. Meilisearch will only return one document with the specified attribute value when using the new `distinct` search parameter.

This feature is commonly used in ecommerce applications. Consider a `products` index containing multiple variants of the same product, e.g. `Blue iPhone 15` and `Red iPhone 15` documents that share the same `product_id`. The API call below will return _a single_ iPhone 15:

```bash
curl \
 -X POST 'http://localhost:7700/indexes/products/search' \
 -H 'Content-Type: application/json' \
 --data-binary '{
    "q": "iphone",
    "distinct": "product_id"
 }'
```
<Callout>
  When `distinct` is provided, Meilisearch ignores the index’s [distinct attribute](https://www.meilisearch.com/docs/learn/configuration/distinct?utm_campaign=release-v1-9&utm_source=blog).
</Callout>

New: `frequency` matching strategy
----------------------------------

Meilisearch 1.9 introduces a new [matching strategy](https://www.meilisearch.com/docs/reference/api/search?utm_campaign=release-v1-9&utm_source=blog#matching-strategy) to prioritize results that contain occurrences of the least frequent query terms. When using the `frequency` matching strategy, Meilisearch will deprioritize very common words.

Let’s take the example of the `"the little prince"` query. In our indexed documents, the words `"the"` and `"little"` likely have a lot of occurrences. Consequently, the matching strategy will prioritize documents containing `"prince"` instead.

Experimental: new similar documents API
---------------------------------------

Meilisearch 1.9 introduces a new [AI-powered search](https://www.meilisearch.com/docs/learn/ai-powered-search/getting_started_with_ai_search?utm_campaign=release-v1-9&utm_source=blog) feature that allows searching for documents similar to an existing document.

The following API call searches for documents similar to the document whose [primary key](https://www.meilisearch.com/docs/learn/core_concepts/primary_key?utm_campaign=release-v1-9&utm_source=blog) is `23` in the movies index:

```bash
curl \
  -X POST /indexes/movies/similar
  -H 'Content-Type: application/json' \
  --data-binary '{
    "id": "23",
    "embedder": "default",
  }'
```

Check out the [similar documents API](https://www.meilisearch.com/docs/reference/api/similar?utm_campaign=release-v1-9&utm_source=blog) for more information about additional parameters.

Experimental: avoid re-generating embeddings
--------------------------------------------

When importing a [dump](https://www.meilisearch.com/docs/learn/advanced/dumps?utm_campaign=release-v1-9&utm_source=blog) created with Meilisearch 1.9 or higher, Meilisearch will not re-generate the embeddings. This will avoid unnecessary computing when upgrading your Meilisearch database.

### New: `regenerate` parameter

Additionally, Meilisearch 1.9 introduces a new API to give more fine-grained control over [document embedding generation](https://www.meilisearch.com/docs/learn/ai-powered-search/getting_started_with_ai_search?utm_campaign=release-v1-9&utm_source=blog#generate-vector-embeddings-with-openai). Specifically, it enables embedding generation whenever a document is updated.

The document `_vectors` object now accepts objects in addition to arrays. The provided object accepts a `regenerate` boolean and an optional `embeddings` array.

Consider the example document below with user-provided embeddings:

```json
{
	"id": 42,
	"_vectors": {
		// Embeddings for the `default` embedder
		// Equivalent to `regenerate: true`
		"default": [0.1, 0.2 ],
		// Embeddings for the `text` embedder
		"text": {
			"embeddings": [[0.1, 0.2, 0.3], [0.4, 0.5, 0.6]],
			// Never regenerate embeddings
			"regenerate": false
		},
		"translation": {
			"embeddings": [0.1, 0.2, 0.3, 0.4],
			// Regenerate embeddings when document is updated
			"regenerate": true
		}
	}
}

```

While, in general, you might want to re-generate your embeddings whenever the document is updated, this facilitates migrating from user-provided embeddings to letting Meilisearch handle embedding without incurring unnecessary costs.

Experimental: hybrid search breaking changes
--------------------------------------------

As we move toward [stabilizing AI-powered search features](https://github.com/meilisearch/meilisearch/issues/4733), we introduced minor breaking changes to make the APIs less error-prone.

### Breaking: empty `embeddings` array

Following user feedback that the previous behavior was unexpected and unhelpful, providing an empty `embeddings` array will now tell Meilisearch that the document has no embeddings.

<Callout>
  Before Meilisearch 1.9, an empty `embeddings` array was interpreted as a single embedding of dimension 0.
</Callout>

### Breaking: removed `_vectors` in search results

Starting with Meilisearch 1.9, API responses to [vector search and hybrid search](https://www.meilisearch.com/docs/learn/ai-powered-search/getting_started_with_ai_search?utm_campaign=release-v1-9&utm_source=blog) requests will not include `_vectors` in the response.

However, you can now use the new `retrieveVectors` search parameter if you want API responses to include them:

```bash
curl \
  -X POST 'http://localhost:7700/indexes/movies/search' \
  -H 'Content-Type: application/json' \
  --data-binary '{
    "q": "star wars",
    "retrieveVectors": true
  }'
```

### Breaking: optimizing user-provided `embeddings`

Starting with Meilisearch 1.9, vector embeddings will no longer be stored as-is. Numbers will be cast to floats in canonicalized representation to save storage and optimize performance. Simply put, the vector `[3]` might be stored as `[3.0]`.

Contributors shout-out
----------------------

Thanks to all community members who participated in this release. Shout out to [@gh2k](https://github.com/gh2k), [@writegr](https://github.com/writegr), and [@yudrywet](https://github.com/yudrywet) for their contributions to [Meilisearch](https://github.com/meilisearch/meilisearch) and [@mosuka](https://github.com/mosuka), [@Soham1803](https://github.com/Soham1803), and [@tkhshtsh0917](https://github.com/tkhshtsh0917) for their contributions to [Charabia](https://github.com/meilisearch/charabia).

And, of course, many thanks to our SDK maintainers, thanks to whom Meilisearch is available across many languages. Special thanks to [@the-sinner](https://github.com/the-sinner) and [@norkunas](https://github.com/norkunas). 🫶

* * *

And that’s a wrap for v1.9! This release post highlights the most significant updates. For an exhaustive listing, read the [changelog on Github](https://github.com/meilisearch/meilisearch/releases/tag/v1.9.0).

Stay in the loop of everything Meilisearch by subscribing to our [monthly newsletter](https://dub.sh/meili-newsletter?utm_campaign=release-v1-9&utm_source=blog). To learn more about Meilisearch's future and help shape it, take a look at our [roadmap](https://roadmap.meilisearch.com/?utm_campaign=release-v1-0&utm_source=blog) and participate in our [Product Discussions](https://github.com/orgs/meilisearch/discussions).

For anything else, join our developers community on [Discord](https://discord.meilisearch.com/?utm_campaign=release-v1-9&utm_source=blog).

Meilisearch 1.9

On June 11, 2024, Meilisearch hosted a live event packed with product updates, sneak peeks, and roadmap discussions. If you couldn't make it, here's a quick recap of what went down.

**AI-powered search refresher and new embedders**
-------------------------------------------------

Our product team took the stage to share insights into developing our AI-powered search solution and outlined our vision for providing ready-to-use components for seamless AI-powered search integration. By adding semantic and AI capabilities to Meilisearch's core keyword and full-text product, we introduced a [hybrid search solution](https://www.meilisearch.com/solutions/hybrid-search) in early 2024.

To streamline the developer experience, we have since added [Ollama and REST embedders](/blog/meilisearch-1-8/ for easier integration within the v1.8 scope.

![](https://lh7-us.googleusercontent.com/docsz/AD_4nXd_QlnM1707ZYLjpOrhzxjZrWDxgdVo3TL5Dewz6dM1ZIbhZp4CcPUJdtDhV4xAA_ZZACDSv8h9JF_d2AoFSNGYNlShW_JOI_s1TrqbEtntN-7pvxW8UnTTdVMNml9oBroLhZfrVprM5dQmM0HiTP55MZPw?key=pr3OBspkNUkqXQYoKJ84OA)

You are welcome to try out the hybrid search feature using our [movie demo database](https://where2watch.meilisearch.com/). 

**New relevancy improvements and smarter search**
-------------------------------------------------

We also introduced several new features designed to _improve search relevancy_ in v1.9:

*   **Negative keyword search**: exclude unwanted results easily.
*   **New frequency matching strategy**: enhance search accuracy by removing the most common term found in the dataset.
*   **Filter by score**: set a baseline _rankingScore_ to return only results considered "good enough" in terms of relevancy.
*   **Get similar documents:** utilize AI to find similar documents within your dataset.

This new feature set is slated for launch on July 1st.

**Query suggestions sneak peek**
--------------------------------

Attendees also got a sneak peek into an upcoming release. The new _query suggestions_ feature aims to enhance search efficiency by returning real-time, relevant suggestions as users type, ordered by popularity. This feature leverages [cloud search analytics](/blog/search-analytics-release/ to generate and rank suggestions based on popularity and relevance. Designed for seamless integration, it requires no new endpoints or tools and is expected to be released in Q3. Ongoing improvements will be based on user feedback.

**Check out the recording of the event for more details:**
----------------------------------------------------------

* * *

Get these monthly updates straight to your inbox by subscribing to our [newsletter](https://dub.sh/meili-newsletter).

For more things Meilisearch, join our developers' community on [Discord](https://discord.meilisearch.com/?ref=blog.meilisearch.com). You can learn more about the product by checking out the [roadmap](https://roadmap.meilisearch.com/?ref=blog.meilisearch.com) and participating in [product discussions](https://github.com/meilisearch/product/discussions?ref=blog.meilisearch.com).

Until next time!

Here's what you missed in the latest Meilisearch live event: V1.8, V1.9, new embedders, query suggestions, and more.

🪼 Announcing Meilisearch 1.8
-----------------------------

[![Meilisearch 1.8 release notes](https://unable-actionable-car.media.strapiapp.com/1_8_1fa2c43277.jpeg)](/blog/meilisearch-1-8/?utm_campaign=newsletter&utm_source=mailchimp&utm_medium=april-24-updates)

Meilisearch 1.8 brings **negative keyword search**, improvements to **search robustness**, and **new embedders** for AI search. It’s time to upgrade. 👇

[Read release notes](/blog/meilisearch-1-8/?utm_campaign=newsletter&utm_source=mailchimp&utm_medium=april-24-updates)� Join our June 11th webinar
-----------------------------

[![What's new in Meilisearch: v1.8, AI search updates, open source embedders, and more!](https://unable-actionable-car.media.strapiapp.com/1f114ac0_8673_4b91_a4ea_d3f4467b2e59_43e7fe7365.png)](https://us06web.zoom.us/meeting/register/tZMudO-qpz8qGdDQ_MNn3NmGV0m8WLf1uECM?utm_campaign=newsletter&utm_source=blog&utm_medium=april-24-updates#/registration)

Join our product team as we dive into what our latest released features. You will learn more about the AI-powered search and get a sneak peek of what's coming soon. 👀

[Register now](https://us06web.zoom.us/meeting/register/tZMudO-qpz8qGdDQ_MNn3NmGV0m8WLf1uECM?utm_campaign=newsletter&utm_source=blog&utm_medium=april-24-updates#/registration)

🤝 OCTO customer story
----------------------

[![Meilisearch is a partner of choice for OCTO Technology](https://unable-actionable-car.media.strapiapp.com/octo_58ab726648.png)](/blog/meilisearch-is-a-partner-of-choice-for-octo/?utm_campaign=newsletter&utm_source=blog&utm_medium=april-24-updates)

This customer story discusses why **OCTO Technology** chose Meilisearch to answer its client's complex needs.

[Read customer story](/blog/meilisearch-is-a-partner-of-choice-for-octo/?utm_campaign=newsletter&utm_source=blog&utm_medium=april-24-updates)
-----------------------

### 💬 [RFC: Query suggestions](https://github.com/orgs/meilisearch/discussions/740)

We’re looking for feedback on our query suggestions proposal.

### 🤔 [Full-text search vs vector search](/blog/full-text-search-vs-vector-search/?utm_campaign=newsletter&utm_source=blog&utm_medium=april-24-updates) comparative analysis of full-text, vector, and hybrid search.

### 💡 [Indexing best practices](/blog/best-practices-for-faster-indexing/?utm_campaign=newsletter&utm_source=blog&utm_medium=april-24-updates)ll you need to know to speed up Meilisearch indexing.

### 🛍️ [Adding site search to your Nuxt 3 ecommerce](/blog/nuxt-ecommerce-search-guide/?utm_campaign=newsletter&utm_source=blog&utm_medium=april-24-updates)earn to build sorting, filtering, and faceted search in Vue & Nuxt apps.

### 🍿 [Next.js hybrid search demo](https://where2watch.meilisearch.com/?utm_campaign=newsletter&utm_source=blog&utm_medium=april-24-updates)

Find where to stream movies in our hybrid search demo for Next.js.

🥰 Made with Meilisearch
------------------------

[Arvid Kahl](https://twitter.com/arvidkahl/status/1780350234410746084) is building [Podscan.fm](https://podscan.fm/?ref=meilisearch), a tool allowing users to subscribe to notifications when keywords are mentioned in podcasts. Meilisearch powers the app’s [full-text search over 2.8 million episode transcripts](https://twitter.com/arvidkahl/status/1780350234410746084).

[![Podscan.fm screenshot](https://unable-actionable-car.media.strapiapp.com/Screenshot_2024_05_14_at_17_27_59_514db7168c.png)](https://podscan.fm/?ref=meilisearch)

_We’re always looking to promote new projects from the community. Post yours in the_ [_#built-with-meilisearch_](https://discord.com/invite/hTD5hU2yWV) _Discord channel!_

🏡 Life at Meili
----------------

Some Rust libraries are like oil and water, they just don't mix. Our very own [@dureuill](https://github.com/dureuill) learnt that the hard way. Read on to learn about the "funny" bug he created while working on Meilisearch.

[![Amateur scientist creating a mess by mixing products](https://unable-actionable-car.media.strapiapp.com/dontmix_a951985054.gif)](https://blog.dureuill.net/articles/dont-mix-rayon-tokio/?ref=meilisearch)

[Read blog post](https://blog.dureuill.net/articles/dont-mix-rayon-tokio/?ref=meilisearch)

🦸 We're hiring!
----------------

See our open positions:

*   [Growth Marketing Manager](https://www.meilisearch.com/careers?utm_campaign=newsletter&utm_source=mailchimp&utm_medium=april-updates-24&utm_content=growth-marketing-manager)

* * *

Get these monthly updates straight to your inbox by subscribing to our [newsletter](https://dub.sh/meili-newsletter).

For more things Meilisearch, join our developers community on [Discord](https://discord.meilisearch.com/?ref=blog.meilisearch.com). You can learn more about the product by checking out the [roadmap](https://roadmap.meilisearch.com/?ref=blog.meilisearch.com) and participating in [product discussions](https://github.com/meilisearch/product/discussions?ref=blog.meilisearch.com).

Until next time!

Meilisearch April Updates

[OCTO Technology](https://www.octo.com/) is a consulting firm specializing in new technologies and the challenges of digital transformation. Headquartered in Paris, it’s recognized for its technological expertise and innovative approach.

> "The searchable page with Meilisearch is really fast and efficient. Meilisearch allows us to meet the customers’ needs with a turnkey solution and reactive support." - Simon Belbeoch, Tech Lead/Manager at Octo Technology

**OCTO Technology uses Meilisearch for its client, an undisclosed customer. The customer runs a platform that aims to connect young people with relevant job opportunities, allowing them to search all kinds of information about integrating into working life. The project involved implementing a search feature to look through data consolidated from various sources.**

Given the specific needs of their sector, the client strongly preferred an open-source solution from the beginning. Other decision-making factors included good performance and greater control over data, allowing users to search through a comprehensive list of jobs to find their ideal offer quickly.

Challenge
---------

The client faced search challenges that required a specific solution, not just standard configurations or a generic search engine. Their website had custom fields that required custom programming. They aimed to replace their previous search engine, [Algolia](https://www.meilisearch.com/docs/learn/what_is_meilisearch/comparison_to_alternatives#algolia), with a new Meilisearch solution, without compromising on functionality or performance

The task became more complex due to the need to aggregate data from various partners in differing formats and manage incomplete data to ensure a consistent search experience across the platform. OCTO’s customer also required more control over search results and sought integration with their CMS, Strapi, for better data control.

Why OCTO chose Meilisearch
--------------------------

Before settling for a specific search provider, OCTO Technology ran a few different scenarios on behalf of its customer and explored the following solutions:

*   Building a proprietary, in-house solution was briefly considered but ultimately ruled out because of the cost-effectiveness considerations and high-performance expectations.
*   Leveraging their prior experience with the platform, the OCTO team initially considered Algolia integration. However, they ultimately favored the open-source transparency offered by Meilisearch, which provided them with greater control over technology and customization settings.
*   Although the team at OCTO considered Elasticsearch too complex for the given client's use case, they decided to use it for logging and configuring data. Ultimately, despite having previous Elasticsearch expertise, the product's complexity prevented the team from benefitting from the Elasticsearch’s offering fully.

Ultimately, the following factors played the most in the decision-making:

### Preference for an open-source

For transparency and to align with the specific needs of the OCTO's client vertical, there was a requirement for the tech stack to be primarily open-source and publicly accessible.

The stress tests were conducted with Meilisearch, and the performance provided better search results compared to the custom search solution that was initially in place.

### Seamless CMS integration

OCTO’s client utilized [Strapi](https://www.meilisearch.com/docs/learn/cookbooks/strapi_v4), a headless CMS solution, for data management and discovered that Meilisearch supports seamless integration via the CMS. Adding data and updating the front-end were straightforward, with the Strapi connection functioning smoothly.

### Consolidation of multiple data sources

This challenge highlighted the need for precise control and robust performance.  To demonstrate Meilisearch's capabilities, the OCTO team conducted a comprehensive proof of concept, focusing on aggregating various job offers from diverse sources and formats.

OCTO conducted a series of stress tests to evaluate the search page functionality under extreme conditions, including handling high data volumes and maintaining search functionality during indexing. While measuring response speed, they also tested the system’s stability under heavy data ingestion to assess whether the search capabilities would withstand extreme condition.

* * *

Ready to elevate your search experience?

[Contact sales](https://dub.sh/meili-sales)

* * *

Implementation
--------------

The platform, developed using Next.js and hosted on Github (though not publicly linked at OCTO’s team request for client anonymity), showcases OCTO’s client's self-managed instance. 

The search engine was straightforward to implement, even without specific Meilisearch expertise, and was integrated with Strapi through a [plugin](https://www.meilisearch.com/docs/learn/cookbooks/strapi_v4). In the front-end, OCTO developed custom Strapi components to allow their client to easily manage UI edits via the CMS.

> _“At the beginning, we had to create custom components, extending from open-source, to meet our client’s specific requirements. Since then, we have consistently reused these front-end components. The early stages of the Strapi to Meilisearch plugin encountered some bugs, but thanks to the responsive collaboration with the Meilisearch team, these issues were resolved swiftly.”_ - Simon Belbeoch, Tech Lead/Manager at Octo Technology.

Further comprehensive stress tests were included in the implementation process to ensure robust system performance. These tests involved data insertion, simultaneous searches, and data insertion while searching.

Results
-------

The outcome of implementing Meilisearch was positive. The comprehensive proof of concept and stress tests conducted before going live showed that the search performed well and the page did not break. OCTO’s customer now fully and independently manages the instance, enhancing the user experience with more relevant search results that reduce user frustration. Although specific metrics were not set beforehand, the implementation successfully met the project's goals, with OCTO’s client observing that Meilisearch's search performance exceeded that of their previous custom search solution.

Vision
------

The experience gained through the completed project with Meilisearch allows OCTO to integrate Meilisearch into its discussions with clients at an earlier stage. It positioned the OCTO team to more effectively meet customer requirements, leading them to consider adopting Meilisearch for internal usage as well as for its new clients. Additionally, the client has expressed strong interest in Meilisearch, prompting considerations for its application in other products.

* * *

Want to stay in the loop with everything Meilisearch? Subscribe to our [newsletter](https://dub.sh/meili-newsletter). Help us shape Meilisearch's future by checking out our [roadmap](https://roadmap.meilisearch.com/?utm_campaign=release-v1-8&utm_source=blog) and participating in our [product discussions](https://github.com/meilisearch/product/discussions?ref=blog.meilisearch.com).

For anything else, join our developers community on [Discord](https://discord.meilisearch.com/?utm_campaign=release-v1-8&utm_source=blog).

Meilisearch is a partner of choice for OCTO Technology

<Callout>
  Meilisearch 1.8 is available on [Meilisearch Cloud](https://cloud.meilisearch.com/?utm_campaign=release-v1-8&utm_source=blog) too—upgrade now!
</Callout>

New: negative search
--------------------

Meilisearch 1.8 allows you to exclude specific terms from your search results using the `-` operator. 

For example, to find documents that contain the word "room" but not "escape", place a `-` before "escape”:

```Bash
curl \
  -X POST 'https://edge.meilisearch.com/indexes/places/search' \
  -H 'Content-Type: application/json' \
  --data-binary '{ "q": "-escape room" }'

```

Experimental: Improved hybrid search
------------------------------------

<Callout>
  AI search is an experimental feature. Give feedback in this [GitHub discussion](https://github.com/orgs/meilisearch/discussions/677).
</Callout>

### New Ollama embedder

Meilisearch 1.8 introduces support for Ollama models. [Configure the embedder](https://www.meilisearch.com/docs/learn/experimental/vector_search?utm_campaign=release-v1-8&utm_source=blog#generate-vector-embeddings) in your index settings:

```JSON
"default": {
  "source": "ollama", // 👈 Use the ollama source
  "url": "http://localhost:11434/api/embeddings",
  // ☝️ optional, fetched from MEILI_OLLAMA_URL environment variable if missing
  "apiKey": "<your-api-key>",
  // ☝️ optional
  "model": "nomic-embed-text",
  "documentTemplate": "{{doc.field_name}}" // like for openAI and huggingFace sources
}

```

### New REST Embedder

Starting with Meilisearch 1.8, you can use any embedder accessible via a REST API. Here’s how to configure an OpenAI embedder with the `rest` source:

```JSON
{
  "source": "rest",
  "apiKey": "<your-openai-api-key>",
  "dimensions": 1536,
  "url": "https://api.openai.com/v1/embeddings",
  "query": {
    "model": "text-embedding-ada-002"
  },
  "inputField": ["input"],
  "inputType": "textArray",
  "pathToEmbeddings": ["data"],
  "embeddingObject": ["embedding"]
}

```

For more details on the API and available parameters, please consult the documentation on [REST embedders](https://www.meilisearch.com/docs/learn/experimental/vector_search#generate-auto-embeddings?utm_campaign=release-v1-8&utm_source=blog).

### New embedder setting: distribution shift

<Callout>
  This is an advanced API for custom models. OpenAI and Hugging Face models don't require this setting.
</Callout>

Depending on the embedding model, [vector embeddings](/blog/what-are-vector-embeddings/?utm_campaign=release-v1-8&utm_source=blog) are not always uniformly distributed within the [vector space](/blog/what-are-vector-embeddings/?utm_campaign=release-v1-8&utm_source=blog) When vectors are overly clustered, semantic search scores can be very closely ranked. As a result of semantic scores being packed, hybrid results might have sub-optimal relevancy.

To address this issue, you can now shift the distribution of results with an affine transformation. This transformation normalizes semantic scores to prevent them from clustering too high or too tightly, making them more comparable to the [keyword scores](https://www.meilisearch.com/docs/reference/api/search#ranking-score).

```JSON
"default": {
  "source": "huggingFace",
  "model": "MODEL_NAME",
  "distribution": {  // describes the natural distribution of results
    "mean": 0.7, // mean value
    "sigma": 0.3 // standard deviation
  }
}

```

*   The **mean** value represents the central point around which you observe semantic search scores to be densely packed. In the provided code sample, the setting indicates that semantic scores from this model tend to cluster around a mean of 0.7.
*   The **sigma** value determines the spread of ranking scores from the mean (standard deviation). The sigma provided in the code sample above means that before shifting the distribution, the results tend to have an average distance of 0.3 to the mean.

This allows for a more effective comparison of `_rankingScore` between semantic and keyword search results, leading to improved results ranking.

### Breaking: trimmed API response

To speed up the search response and reduce bandwidth usage, Meilisearch v1.8 no longer returns the following fields in the search responses:

*   the `vector` field, which returned the query vector 
*   the `value` field included in the [`vectorSort`](https://www.meilisearch.com/docs/reference/api/search?utm_campaign=release-v1-8&utm_source=blog#vectorsorttarget_vector) object, which is returned when you add \`”showRankingScoreDetails”: true”\` to a search query
*   the `_semanticScore` field; its value is the same as `_rankingScore`, which you should use instead. Set [`”showRankingScore”: true”`](https://www.meilisearch.com/docs/reference/api/search?utm_source=blog&utm_campaign=release-v1-8#ranking-score) in the search parameters

Search robustness improvements
------------------------------

### New search cut-off setting

To enhance stability and performance, Meilisearch automatically terminates any search request that exceeds 1500 milliseconds. 

To adjust this timeout threshold, you can customize the `searchCutoffMs` setting:

```bash
curl \
  -X PATCH 'https://edge.meilisearch.com/indexes/movies/settings' \
  -H 'Content-Type: application/json' \
  --data-binary '{
    "searchCutoffMs": 150
  }'

```

### Experimental: concurrent search requests limit

This release introduces a limit on concurrent search requests to prevent Meilisearch from using excessive amounts of RAM and crashing. The default queue limit is set to 1000 requests. 

To modify this limit, restart your self-hosted instance with the `--experimental-search-queue-size` option:

```bash
./meilisearch --experimental-search-queue-size 100
```

👉 This limit does not impact the search performance. It only controls the number of search requests that can be queued simultaneously to enhance system security.

<Callout>
  Give feedback regarding concurrent search requests limit in this [GitHub discussion](https://github.com/orgs/meilisearch/discussions/729).
</Callout>

Contributors shout-out
----------------------

We are really grateful for all the community members who participated in this release. We would like to thank: [@availhang](https://github.com/availhang), [@hack3ric](https://github.com/hack3ric), [@jakobklemm](https://github.com/jakobklemm), [@mohsen-alizadeh](https://github.com/mohsen-alizadeh), [@redistay](https://github.com/redistay), [@rohankmr414](https://github.com/rohankmr414), [@shuangcui](https://github.com/shuangcui), [@Toromyx](https://github.com/Toromyx), [@6543](https://github.com/6543) for their help with Meilisearch, and [@Gusted](https://github.com/Gusted), [@mosuka](https://github.com/mosuka), [@6543](https://github.com/6543), for their work on [Charabia](https://github.com/meilisearch/charabia).

Special thanks to our SDKs maintainers, thanks to whom Meilisearch is available across so many languages. 🫶

* * *

And that’s a wrap for v1.8! This release post highlights the most significant updates. For an exhaustive listing, read the [changelog on Github](https://github.com/meilisearch/meilisearch/releases/tag/v1.8.0).

Stay in the loop of everything Meilisearch by subscribing to our [newsletter](https://dub.sh/meili-newsletter). To learn more about Meilisearch's future and help shape it, take a look at our [roadmap](https://roadmap.meilisearch.com/?utm_campaign=release-v1-8&utm_source=blog) and come participate in our [Product Discussions](https://github.com/meilisearch/product/discussions).

For anything else, join our developers community on [Discord](https://discord.meilisearch.com/?utm_campaign=release-v1-8&utm_source=blog).

Meilisearch 1.8

🚀 Announcing Meilisearch 1.7
-----------------------------

[![](https://unable-actionable-car.media.strapiapp.com/138_84d37ea815.jpg)](/blog/meilisearch-1-7/?utm_campaign=newsletter&utm_source=blog&utm_medium=march-updates-24)

Meilisearch 1.7 **stabilizes ranking score details**, adds **GPU support for Hugging Face embeddings**, and integrates the **latest OpenAI embedding models**. It’s time to upgrade. 👇

[Read release notes](/blog/meilisearch-1-7/?utm_campaign=newsletter&utm_source=blog&utm_medium=march-updates-24)

⚖️ Introducing hybrid search
----------------------------

[![](https://unable-actionable-car.media.strapiapp.com/Group_143725858_1cf2d7de65.png)](/blog/introducing-hybrid-search/?utm_campaign=newsletter&utm_source=blog&utm_medium=march-updates-24)

Meilisearch 1.7 introduced new OpenAI embedders, adding to Meilisearch's **AI-enhanced hybrid search capabilities**. Read the blog post to learn more about hybrid search and [**sign up to our waiting list**](https://share-eu1.hsforms.com/1RN-rGYhHQPu5BNivfknlwAfg3aq?utm_campaign=newsletter&utm_source=blog&utm_medium=march-updates-24).

[Read announcement](/blog/introducing-hybrid-search/?utm_campaign=newsletter&utm_source=blog&utm_medium=march-updates-24)

🍿 Watch our launch event
-------------------------

On March 7th, we discussed our roadmap, our recent releases and **showcased** Meilisearch Cloud’s new features, including **monitoring metrics** and **hybrid search**.

[Watch it!](https://youtu.be/Lgv8iTdWDQc?si=GOZ7HiV0jM0KJQ7y)

🤔 Full-text search vs vector search
------------------------------------

[![](https://unable-actionable-car.media.strapiapp.com/fts_vs_vs_71fd339c9c.jpg)](/blog/full-text-search-vs-vector-search/?utm_campaign=newsletter&utm_source=blog&utm_medium=march-updates-24)

Join [@StriftCodes](https://twitter.com/StriftCodes) in this **comparative analysis** as he delves into the pros and cons of each solution and how **hybrid search** might offer the best of both worlds.

[Read blog post](/blog/full-text-search-vs-vector-search/?utm_campaign=newsletter&utm_source=blog&utm_medium=march-updates-24)

* * *

🖇️ Links, links, links
-----------------------

### 🟧 [Laravel multitenancy guide](https://www.meilisearch.com/docs/learn/cookbooks/laravel_multitenancy?utm_campaign=newsletter&utm_source=blog&utm_medium=march-updates-24)

Implement search in a multitenant Laravel application.

### 💻 [Computing Hugging Face embeddings with the GPU](https://www.meilisearch.com/docs/learn/cookbooks/computing_hugging_face_embeddings_gpu?utm_campaign=newsletter&utm_source=blog&utm_medium=march-updates-24)

Learn to compile a Meilisearch binary that leverages an Nvidia GPU to generate Hugging Face embeddings.

### 🔢 [From ranking to scoring](/blog/from-ranking-to-scoring/?utm_campaign=newsletter&utm_source=blog&utm_medium=march-updates-24)

Our journey to add relevancy scores to search results in Meilisearch.

### 🪼 [Meilisearch v1.8 release candidate](https://github.com/meilisearch/meilisearch/releases/tag/v1.8.0-rc.0)

Get a preview of what’s coming next (and give your feedback!)

### 🚀 [Strapi v4 guide](https://www.meilisearch.com/docs/learn/cookbooks/strapi_v4?utm_campaign=newsletter&utm_source=blog&utm_medium=march-updates-24)

Learn how to integrate Meilisearch with Strapi.

* * *

🥰 Made with Meilisearch
------------------------

The folks at **Blazity** have created this impressive **Next.js storefront** designed for high-performance eCommerce, featuring AI capabilities and one-click deployment.

[![](https://unable-actionable-car.media.strapiapp.com/blazity_e97c54672a.png)](https://github.com/Blazity/enterprise-commerce)

_We’re always looking to promote new projects from the community. Post yours in the_ [_#built-with-meilisearch_](https://discord.meilisearch.com/?ref=blog.meilisearch.com) _Discord channel!_

🏡 Life at Meili
----------------

Join our Head of Engineering, [@clurquizar](https://twitter.com/clurquizar), as she walks us through key points of Meilisearch’s history and our evolution as an open-source company.

[![](https://unable-actionable-car.media.strapiapp.com/clem_itw_af4c14f089.png)](/blog/how-meilisearch-became-open-source-a-retrospective/?utm_campaign=newsletter&utm_source=blog&utm_medium=march-updates-24)

[Read blog post](/blog/how-meilisearch-became-open-source-a-retrospective/?utm_campaign=newsletter&utm_source=blog&utm_medium=march-updates-24)

* * *

🦸 We're hiring!
----------------

See our open positions:

*   [Growth Marketing Manager](https://www.meilisearch.com/careers?utm_campaign=newsletter&utm_source=blog&utm_medium=march-updates-24)

* * *

Get these monthly updates straight to your inbox by subscribing to our [newsletter](https://dub.sh/meili-newsletter).

For more things Meilisearch, join our developers community on [Discord](https://discord.meilisearch.com/?ref=blog.meilisearch.com). You can learn more about the product by checking out the [roadmap](https://roadmap.meilisearch.com/?ref=blog.meilisearch.com) and participating in [product discussions](https://github.com/meilisearch/product/discussions?ref=blog.meilisearch.com).

Until next time!

Meilisearch March Updates

_This is part 4 of a series of blog posts_ [_originally published on Clément Renault's blog_](https://blog.kerollmops.com/how-meilisearch-updates-a-millions-vector-embeddings-database-in-under-a-minute)_. Begin the journey with_ [_part 1_](/blog/spotify-inspired-hybrid-search-and-rust/)_,_ [_part 2_](/blog/refining-ann-performance-with-rust/)_ and_ [_part 3_](/blog/arroy-filtered-disk-ann/)_ This blog post considers that you’ve already read parts 1 and 2._

In this blog post, we’ll talk about how we implemented incremental indexing in [Arroy](https://github.com/meilisearch/arroy). By incremental, we mean that when inserting or deleting items in a tree, we can update only the required parts instead of rebuilding everything.

At Meilisearch, that’s essential because our users usually send updates regularly, and their content constantly changes. Rebuilding everything from scratch is very costly and won’t give a good experience; that’s why we already spent a lot of time on our latest release to make all the [data structures support incremental indexing](https://github.com/meilisearch/meiliSearch/releases/tag/v1.6.0) and arroy being one of them, we _HAD TO_ implement it as well!

The theory
----------

The following schema shows a high-level view of what should happen when adding and deleting documents into an existing tree.

As a reminder, arroy is a vector store based on [LMDB](http://www.lmdb.tech/doc/), an embedded transactional memory-mapped key-value store. It stores your vectors in trees composed of three kinds of nodes:

*   The split nodes - Store information on which item IDs are spread out between its left and right.
*   The descendants' nodes - Store multiple item IDs.
*   The items - A single vector.

For the following examples, we’ll consider the descendants node cannot contain **more** than two items and that the closer the IDs are, the closer the items are in space. For example, the item `1` is close to the item `2` but is farther from the item `10`.

![First step before inserting elements in the tree](https://unable-actionable-car.media.strapiapp.com/tamo_1_9ea29125b9.png)

Here, we’re trying to insert items `2` and `7` and delete items `3` and `11`.

Following what we do on the search, we split the items to insert between the left and right split nodes. But the deleted items are no longer present in the database! We don't have their associated vector, and therefore we'll need to iterate over all the leaves to find and delete them.

Fortunately, since we use roaring bitmaps, this list doesn't consume much memory, and since we never update it, we can share it with every thread without ever copying it 🎉

![Second step inserting element in the tree](https://unable-actionable-car.media.strapiapp.com/tamo_2_e4cd099c77.png)

Notice how the items to insert were balanced between the left and right split nodes. In the next step, we’ll follow the same process and split both lists of items to insert on the split node.

![Third step inserting into elements in the tree](https://unable-actionable-car.media.strapiapp.com/tamo_3_ab539d2465.png)

It’s on this step that all the good stuff happens. From the left to the right:

*   The first descendant node gets its item `3` removed, and item `2` added.
*   The second descendant node gets too big and must be replaced by a new split node.
*   The item `8` becomes a descendant containing both the items `7` and `8`.
*   After deleting item `10` in the last descendant node, we replace it with a direct link to an item to reduce the number of nodes in the tree (and improve search time by reducing the number of lookups).

![Last step inserting elements in the tree](https://unable-actionable-car.media.strapiapp.com/tamo_4_b07ebb4b8f.png)

Now that you’ve seen all the steps of our incremental indexing process, we’re still going to take a few notes on what happened:

*   In the first three steps, we had to read all of the tree nodes, which, by default, is equal to the total number of items in the database.
*   In the last step, we had to write four new tree nodes. Previously we would have rewritten the whole tree. The number of writes, which is the slowest operation in databases, is reduced to the bare minimum.
*   The process works on a single tree which means we can still multi-thread the computation of every tree.
*   The IDs aren’t sequential anymore, which doesn't play well with the ID generation strategy described in [Writing the tree in parallel](https://blog.kerollmops.com/multithreading-and-memory-mapping-refining-ann-performance-with-arroy).

How did we fix the ID generation?
---------------------------------

At the start of the indexing process, we need to gather all the existing tree IDs.  
The information we need to generate new nodes are:

*   The total number of tree nodes used to know when to stop building trees.
*   The « holes » in the used IDs that we want to re-use. This fragmentation is produced when we edit the trees.
*   The highest ID existing in the trees to generate the next new IDs.

![Selecting the next ID in the Roaring Bitmap](https://unable-actionable-car.media.strapiapp.com/tamo_5_1f7bb6a273.png)

The « simple » idea is to have a counter for the total number of tree nodes. A counter that we can increment to generate new fresh IDs. The most challenging part is to pick an available ID shared between threads and without big mutexes.

It took me quite some time to find a solution for this last point; I implemented a whole complex solution (150loc) that I ended up [rewriting entirely from scratch](https://github.com/meilisearch/arroy/pull/56/commits/0bf68e21cfd97ff00c7ef5a0ec116ba34c43c6a5) in one hour for an easy and more efficient solution. We're going to see all the ideas we went through before finding our final solution.

### Sharing an iterator over the available IDs

The first and most straightforward idea that comes to mind is to create and share an iterator over all the available IDs.

```rust
let mut available_ids = available_ids.into_iter();

roots.into_par_iter().for_each(|root| {
    // stuff
    let next_id = available_ids.next();
    // stuff
});
```

If you’re familiar with Rust, you’ll notice immediately that the `for_each` _cannot_ capture a mutable reference over the `available_ids` because we use the `into_par_iter` method from Rayon that execute the next methods on the iterator in a multi-threaded way. Which means we can't call `.next()`.

By using a mutex (short for [mutual exclusion](https://en.wikipedia.org/wiki/Mutual_exclusion)), we can make it compile:

```rust
let mut available_ids = Mutex::new(available_ids.into_iter());

roots.into_par_iter().for_each(|root| {
    // stuff
    let next_id = available_ids.lock().unwrap().next();
    // stuff
});
```

That is safe and will yield the right results, but won’t scale well as all the threads have to wait for each other on the lock.

### Let each thread own its list of IDs

When you need to remove a lock, a common solution is to split the work beforehand so each thread can work to its full capacity without ever having to know what’s happening on the other threads. That’s the solution [I implemented initially](https://github.com/meilisearch/arroy/pull/56/files/dc782ea61ef84f0ad8ad20e092476494d6eacc0c). The idea is to explode the roaring bitmap into as many smaller roaring bitmaps as there are trees to update.

The following function can split a roaring bitmap into `n` sub-bitmaps of equal size:

```rust
pub fn split_ids_in(available: RoaringBitmap, n: usize) -> Vec<RoaringBitmap> {
    // Define the number of elements per sub-bitmap
    let chunk_size = available.len() / n as u64;
    let mut ret = Vec::new();

    let mut iter = available.into_iter();
    for _ in 0..(n - 1) {
        // Create a roaring bitmap of `chunk_size` elements from the iterator without consuming it
        let available: RoaringBitmap = iter.by_ref().take(chunk_size as usize).collect();
        ret.push(available);
    }
    // the last element is going to contain everything remaining
    ret.extend(iter);

    ret
}
```

With this function available, it's then trivial to use one bitmap per tree to update with the [`zip`](https://docs.rs/rayon/latest/rayon/iter/trait.IndexedParallelIterator.html#method.zip) method on the parallel iterators of Rayon:

```rust
let available_ids = split_ids_in(available_ids, roots.len());

roots.into_par_iter().zip(available_ids).for_each(|(root, available_ids)| {
    let mut available_ids = available_ids.into_iter();
    // stuff
    let next_id = available_ids.next();
    // Here, we can use the available_ids without any locks or inter-thread synchronization
    // stuff
});
```

This works well while we're updating the already existing root trees. But later on, we may want to create new trees from scratch, and we can't know in advance how many new trees we’ll need to create. That's an issue because, without this information, we don't know how many sub-bitmaps we need to create.

At this point, I couldn’t see an easy fix to this problem, but I made the assumption that we rarely create a lot of new trees and that all new trees would use a lot of IDs. This means, giving all the IDs to the **first** new tree would _probably work out_ (it’s hard to be 100% sure without monitoring it in prod).

### The final solution

The previous solution was complex and did not even work perfectly. While I was scrolling through the documentation of the Roaring Bitmap, I saw [the `select` method](https://docs.rs/roaring/latest/roaring/bitmap/struct.RoaringBitmap.html#method.select) and immediately understood how I could make the initial approach works.

This method lets you get the values in your bitmap by index in an efficient way.  
For example, with the following values in a bitmap: `13, 14, 15, 98, 99, 100`:

*   `select(0)` returns `13`.
*   `select(3)` returns `98`.
*   `select(5)` returns `100`.
*   `select(6)` returns `None`.

With that in mind, if we share the bitmap with all threads in **read-only** and share the index in the bitmap as an atomic number, we can have multiple threads get available IDs at the same time with lock-free synchronization.

Once the `select` method returns `None`, it means we can stop picking IDs from the bitmap and instead simply generate new IDs from scratch with the old method.  
Even if it's super fast, calling the `select` method still takes some time, so once a thread gets a `None` returned from the method, we're going to update an atomic boolean telling us if there are still values to look at in the bitmap. If not, we can skip calling `select` and directly generate a new ID.

With that in mind, the cool `ConcurrentNodeIds` structure we showed you before in part 2 got a little bit more complex but it still let us generate IDs fairly without any locks!

```rust
/// A concurrent ID generator that will never return the same ID twice.
pub struct ConcurrentNodeIds {
    /// The current tree node ID we should use if no other IDs are available.
    current: AtomicU32,
    /// The total number of tree node IDs used.
    used: AtomicU64,

    /// A list of IDs to exhaust before picking IDs from `current`.
    available: RoaringBitmap,
    /// The current Nth ID to select in the bitmap.
    select_in_bitmap: AtomicU32,
    /// Tells if you should look in the roaring bitmap or if all the IDs are already exhausted.
    look_into_bitmap: AtomicBool,
}

impl ConcurrentNodeIds {
    /// Creates an ID generator returning unique IDs, avoiding the specified used IDs.
    pub fn new(used: RoaringBitmap) -> ConcurrentNodeIds {
        let last_id = used.max().map_or(0, |id| id + 1);
        let used_ids = used.len();
        let available = RoaringBitmap::from_sorted_iter(0..last_id).unwrap() - used;

        ConcurrentNodeIds {
            current: AtomicU32::new(last_id),
            used: AtomicU64::new(used_ids),
            select_in_bitmap: AtomicU32::new(0),
            look_into_bitmap: AtomicBool::new(!available.is_empty()),
            available,
        }
    }

    /// Returns a new unique ID and increases the count of IDs used.
    pub fn next(&self) -> Result<u32> {
        if self.look_into_bitmap.load(Ordering::Relaxed) {
            let current = self.select_in_bitmap.fetch_add(1, Ordering::Relaxed);
            match self.available.select(current) {
                Some(id) => Ok(id),
                None => {
                    self.look_into_bitmap.store(false, Ordering::Relaxed);
                    Ok(self.current.fetch_add(1, Ordering::Relaxed))
                }
            }
        } else {
            Ok(self.current.fetch_add(1, Ordering::Relaxed))
        }
    }
}
```

Performances in practice
------------------------

I haven’t run a lot of benchmarks yet (I would like to, though, and may update this post later), but on my benchmarks with a few hundred thousand items, here are the results I get:

![Performance improvement showed on a graph](https://unable-actionable-car.media.strapiapp.com/tamo_6_8c737d7e95.png)

*   On average, we get more than a 10x speedup after three batches of items indexed.
*   The little bumps we observe afterward that make us go from ~700ms to ~3s are due to the creation of new trees as the number of items increases.
*   I expect this benchmark to represent a worst-case scenario:
    *   All my item IDs are randomly generated, which means there are very few duplicates/updates but always new insertions and more writing.
    *   My vectors are also randomly generated with a uniform distribution, which means there shouldn’t be clusters as we observe in a real-life dataset.

This feature is now being used on production data with millions of documents, and we've seen numbers in the range of 9000 items added on a 2M items database in under a minute.

* * *

[Meilisearch](https://meilisearch.com/?ref=blog.meilisearch.com) is an open-source search engine enabling developers to build state-of-the-art experiences while enjoying simple, intuitive DX. 

For more things Meilisearch, you can join the community on [Discord](https://discord.gg/meilisearch/?ref=blog.meilisearch.com) or subscribe to the [newsletter](https://dub.sh/meili-newsletter). You can learn more about the product by checking out the [roadmap](https://roadmap.meilisearch.com/?ref=blog.meilisearch.com) and participating in [product discussions](https://github.com/orgs/meilisearch/discussions?ref=blog.meilisearch.com).

Meilisearch latest news and company updates