When your AI application starts hemorrhaging $500 daily in OpenAI API calls and users are drumming their fingers waiting 2 seconds for responses, semantic caching isn't just a nice-to-have—it's a lifeline.

While traditional caches match queries word-for-word, semantic caching understands the deeper meaning.

In this guide, you'll discover how to implement semantic caching that slashes response times and API costs. Whether you're building a chatbot, search engine, or AI-powered application, these strategies will help you create lightning-fast, cost-effective semantic search that keeps users happy and your infrastructure lean. You'll learn how to:

- Reduce response times dramatically
- Achieve massive API cost savings
- Handle similar questions without repeatedly querying language models

## Introduction to semantic caching

Semantic caching revolutionizes how we store and retrieve information in modern applications. Traditional caching relies on exact matches, while semantic caching understands the meaning behind queries. This allows systems to recognize and respond to similar questions, even if they're worded differently.

This technique is valuable for AI-powered applications. Processing similar queries repeatedly can be both time-consuming and expensive.

### What is semantic caching

Semantic caching is a sophisticated data retrieval technique. It stores both the data and the contextual meaning of queries and their results. When a user asks a question, the system analyzes its semantic meaning rather than looking for an exact match.

For example, "What's the weather in New York?" and "Tell me New York's forecast" can pull from the same cached response, saving processing time and resources.

### Comparing semantic caching vs traditional caching

![Semantic Cache](https://unable-actionable-car.media.strapiapp.com/how_to_cache_semantic_search_1996414fc7.avif)

Traditional caching and semantic caching serve different purposes in modern applications. Here's a comparison:

- **Traditional caching**: Relies on exact matches and specific keys to retrieve data. Think of it as using a filing cabinet where you need the exact file name.
- **Semantic caching**: Understands context and meaning. It's like having a smart assistant who can find related information even when you ask in different ways.

### Advantages of semantic caching

Semantic caching offers several advantages for real-world applications:

- **Reduced API costs** by avoiding redundant queries.
- **Faster response times** for similar questions.
- **More consistent answers** across related queries.
- **Better scalability** for AI-powered applications.

### When to choose semantic caching

Consider semantic caching for your application if:

- **Users might ask the same question in different ways.**
- Response consistency is crucial.
- API costs need to be optimized.
- Real-time response is essential.

## Core concepts of semantic caching

Understanding the fundamental components and applications of semantic caching is crucial for successful implementation. Let's explore how these elements work together to create efficient caching systems that enhance your application's performance.

### Key components of semantic caching systems

Several critical components form the foundation of semantic caching.

- [**Embedding model**](https://www.meilisearch.com/blog/what-are-vector-embeddings): Converts queries into vector representations capturing their meaning.
- [**Vector database**](https://www.meilisearch.com/blog/what-is-a-vector-database): Stores these vectors, allowing for quick similarity searches for new queries.
- **Cache layer**: Sits between your application and underlying systems, managing query-response pairs and their semantic meanings. It decides whether to serve cached content or process new requests.
- **Vector search mechanism**: Evaluates incoming queries against stored embeddings using similarity thresholds to determine cache hits.

### Semantic cache in RAG

**Retrieval-Augmented Generation (RAG)** systems use semantic caching to improve performance and reduce costs. When a user submits a query, the system first checks the semantic cache for similar questions before going through the full retrieval and generation process. This is especially effective for handling common queries in customer support or knowledge base applications.

The integration process involves three main steps:

1. Embed the incoming query into a vector representation.
2. Search the cache for semantically similar queries.
3. If a match is found, return the cached response immediately. If not, proceed with the traditional RAG pipeline and store the new result for future use.

By understanding these core concepts and components, you can effectively implement and optimize semantic caching in your AI applications.

## Implementing semantic caching for an e-commerce site

Semantic caching can significantly enhance how e-commerce platforms handle product searches and customer queries. By adopting a smart caching mechanism, businesses can boost search performance and improve user experience.

![how to cache semantic search](https://unable-actionable-car.media.strapiapp.com/how_to_cache_semantic_search_ba174395a9.avif)

We will go through these key steps:

1. Configure embedding model by initializing OpenAI embeddings and Meilisearch client with appropriate connection parameters.
2. Create a vector store with custom configuration, specifying index name, embedding dimensions, and text key for efficient semantic search storage.
3. [Implement semantic search](https://www.meilisearch.com/docs/guides/ai/openai) function to convert queries into vector embeddings and perform similarity-based retrieval from the vector store.
4. Develop cache management logic to handle cache hits and misses, including fallback mechanisms for database searches and optional cache updates.

### Setting up the semantic caching infrastructure

Building a semantic caching system requires careful planning and the right tools. Here's a step-by-step guide to implementation.

#### Choosing core technologies

Start by [selecting the technologies for your system](https://www.meilisearch.com/blog/choosing-the-best-model-for-semantic-search). **Meilisearch is an excellent vector search engine** for semantic caching. For this example, we'll use Python with OpenAI embeddings to create a context-aware caching solution.

#### Installing required dependencies

Set up your project environment by installing the necessary libraries:

```bash
pip install meilisearch langchain openai python-dotenv
```

This command installs essential tools like Meilisearch for vector storage and LangChain for embedding management.

#### Configuring the embedding model

The core of semantic caching is converting queries into meaningful vector representations. Below is a sample configuration using OpenAI embeddings:

```python
from langchain.embeddings.openai import OpenAIEmbeddings
import meilisearch

# Set up OpenAI embeddings
embeddings = OpenAIEmbeddings()

# Initialize Meilisearch client
client = meilisearch.Client(
    url="your_meilisearch_url",
    api_key="your_api_key"
)
```

#### Creating a vector store for semantic caching

Setting up a vector store is crucial for implementing an effective semantic cache. Here's a comprehensive approach to creating a vector store using Meilisearch and OpenAI embeddings:

```python
# Create vector store with custom configuration
vector_store = Meilisearch(
    client=client,
    embedding=embeddings,
    index_name="semantic_cache",
    text_key="content"
)

# Optional: Define embedder configuration
embedder_config = {
    "custom": {
        "source": "userProvided",
        "dimensions": 1536  # OpenAI embedding dimensions
    }
}
```

This setup allows you to create a robust semantic cache that can efficiently store and retrieve contextually similar queries and their responses.

### Implementing the semantic cache mechanism

The key to semantic caching is creating a **retrieval system that understands query intent**.

#### Creating a semantic search function

Here’s an example of a semantic search function:

```python
def semantic_cache(query, k=3):
    # Convert query to vector embedding
    query_embedding = embeddings.embed_query(query)
    
    # Perform similarity search in Meilisearch
    results = vector_store.similarity_search(
        query=query,
        k=k
    )
    
    return results
```

This function:

- Converts the query into a vector representation.
- Searches for semantically similar results in the vector store.
- Returns the most relevant matches.

#### Handling cache hits and misses

A robust caching strategy must handle both cache hits and misses effectively:

```python
def process_product_search(query):
    # Check the semantic cache
    cached_results = semantic_cache(query)
    
    if cached_results:
        # Cache hit: return cached results
        return {
            "source": "cache",
            "results": cached_results
        }
    else:
        # Cache miss: perform full database search
        full_results = perform_database_search(query)
        
        # Optionally, store new results in cache
        update_semantic_cache(query, full_results)
        
        return {
            "source": "database",
            "results": full_results
        }
```

### Performance optimization techniques

To ensure efficient semantic caching, consider these strategies:

- **Set a similarity threshold** (e.g., 0.85–0.95) to filter results.
- Limit cache size to prevent memory issues.
- Use time-to-live (TTL) to refresh outdated cache entries.
- Regularly update your embedding model for better accuracy.

### Testing your semantic cache

Testing is critical to ensure reliability. Test scenarios should include:

- Exact match queries.
- Semantically similar queries.
- Completely new queries.
- Edge cases with minimal context.

**Thorough testing ensures your system performs well across diverse search patterns.**

Meilisearch’s fast vector search capabilities make semantic caching not only feasible but also highly efficient for e-commerce platforms.

## Adding semantic caching to RAG system

Retrieval-Augmented Generation (RAG) systems can greatly benefit from semantic caching. This integration **improves efficiency, reduces costs, and enhances responsiveness** in AI applications.

### Understanding RAG and semantic caching integration

Semantic caching in RAG systems focuses on storing and retrieving query responses based on their meaning, not just exact text matches. This allows AI applications to quickly access previously generated responses while maintaining contextual relevance.

Key steps in integrating semantic caching into a RAG workflow include:

1. **Embedding Transformation**: Convert user queries into vector representations that capture semantic meaning. Use embedding models like BERT or OpenAI's embedding APIs for this transformation.

2. **Cache Lookup Mechanism**: Use a vector database for fast, similarity-based searches. **Meilisearch is a strong choice** for its efficient semantic search capabilities, enabling quick matches between incoming queries and cached responses.

### Implementing the semantic caching layer

Implementing a semantic caching layer requires creating a specialized class that manages query embeddings, similarity checks, and cached response retrieval. This code provides a practical approach to efficiently storing and accessing semantically similar query responses.

```python
class RAGSemanticCache:
    def __init__(self, embedding_model, vector_store):
        self.embedding_model = embedding_model
        self.vector_store = vector_store
        self.similarity_threshold = 0.85

    def retrieve_cached_response(self, query):
        query_embedding = self.embedding_model.embed_query(query)
        similar_results = self.vector_store.similarity_search(
            query_embedding, 
            threshold=self.similarity_threshold
        )
        return similar_results[0] if similar_results else None
```

This class:

- Embeds queries into vectors.
- Searches for semantically similar results in the vector store.
- Returns a cached response if it meets the similarity threshold.

<Callout intent="note">
	
	To ensure your semantic caching layer performs efficiently, apply these strategies:
	
	- **Dynamic Similarity Thresholds**: Adjust thresholds based on your domain and use case.
	- **Cache Maintenance**: Use expiration and refresh mechanisms to keep responses relevant.
	- **Multi-Turn Context Tracking**: For conversational systems, include conversation history in cache retrieval to maintain context.
	
	
</Callout>

### Practical implementation example

Here’s how semantic caching can be integrated into a RAG workflow:

```python
def rag_with_semantic_cache(query, rag_system, semantic_cache):
    # Check semantic cache first
    cached_response = semantic_cache.retrieve_cached_response(query)
    
    if cached_response:
        return cached_response
    
    # If no cached response, use traditional RAG
    retrieved_docs = rag_system.retrieve_documents(query)
    generated_response = rag_system.generate_response(query, retrieved_docs)
    
    # Store the new response in the semantic cache
    semantic_cache.store_response(query, generated_response)
    
    return generated_response
```

This function:

- Checks the semantic cache for a response.
- Falls back to the RAG system if no cached response exists.
- Stores new responses in the cache for future use.

### Benefits for different applications

Semantic caching is especially useful in:

- Customer support chatbots.
- Enterprise knowledge management systems.
- Educational platforms.
- Research and analytics tools.

## Optimization strategies

Once your semantic caching system is running, focus on optimization to ensure it continues to deliver maximum value. Let's explore key strategies to maintain and improve your system's performance over time.

### Ensuring accuracy & consistency

Maintaining high accuracy in semantic caching requires regular monitoring and updates.

- **Implement version control** for cached responses to track changes and ensure users receive the most current information.
- **Invalidate or update related cache entries** when source data updates to prevent serving outdated information.
- **Cache consistency** becomes particularly important in distributed systems. Use a publish-subscribe system to propagate updates across your infrastructure, ensuring all cache nodes remain synchronized.
- **Regular validation of cached responses** helps maintain quality. Set up automated checks to verify cached responses still match their original context and meaning. Periodically compare cached responses against fresh generations to identify any drift in accuracy.

### Improving performance and efficiency

**Performance optimization** starts with careful monitoring of key metrics. Track cache hit rates, response times, and system resource usage to identify bottlenecks and optimization opportunities. Use dashboards to **visualize these metrics** and establish baseline performance expectations.

Consider these advanced optimization techniques:

- **Predictive caching** for frequently accessed queries.
- **Tiered caching** with different storage solutions for hot and cold data.
- **Optimize vector similarity calculations** for faster retrieval.
- **Compress embeddings** while maintaining accuracy.

Regular performance audits help identify improvement areas. Compare your system's performance against benchmarks and adjust your caching strategy accordingly. Document performance improvements and their impact on user experience and operational costs.

### Common pitfalls and how to avoid them

**Over-aggressive caching** can lead to memory issues and degraded performance. Start with conservative cache sizes and grow based on actual usage patterns rather than theoretical maximums.

**Poor similarity threshold selection** represents another challenge. Setting thresholds too low leads to false positives and incorrect responses. Setting them too high results in unnecessary cache misses. Monitor your threshold settings and adjust based on user feedback and performance metrics.

Watch out for these additional pitfalls:

- **Neglecting cache maintenance and cleanup**.
- **Failing to handle edge cases** in query processing.
- **Ignoring the impact of cache invalidation** on system performance.
- **Not implementing proper error handling** and fallback mechanisms.

**Focusing on these optimization strategies** ensures your semantic caching system remains efficient and effective, providing consistent performance improvements over time.

## Semantic caching: the path to high performance search

**Implementing semantic caching for your search system is a powerful way to boost performance, reduce costs, and enhance user experience.**

By following this guide's systematic approach to setting up semantic caching—from infrastructure assessment to optimization strategies—you'll be well-equipped to handle growing query volumes while maintaining fast response times.

<Callout intent="tip">
	
	No search engine expertise required! [Get started with Meilisearch Cloud](https://meilisearch.com/get-started) and experience an intuitive platform that makes implementing powerful search effortless, even for developers new to search technologies.
	
	
</Callout>

How to cache semantic search: a complete guide

Building a RAG system with Meilisearch: a comprehensive guide

How personalization and recommendations enhance search and discovery

## 🚀 Meilisearch 1.12 release

![Meilisearch 1.12 release notes](https://unable-actionable-car.media.strapiapp.com/meilisearch_1_12_7139b280d2.jpg)

Meilisearch 1.12 release notes
Meilisearch 1.12 significantly **improves indexing speed** and introduces new APIs to **disable prefix search and facet search** and retrieve **task batches**.

[Learn more](https://www.meilisearch.com/blog/meilisearch-1-12)

## 💫 Recent releases wrap up 

![Group 143726039 (2).png](https://unable-actionable-car.media.strapiapp.com/Group_143726039_2_a316baf38d.png)

Catch up on everything we shipped recently—faster search, revamped search preview, new index settings UI, and more.

[Learn more](https://www.meilisearch.com/blog/q4-2024-release-wrap-up)

## 📰 In the news

### ⚡ Indexing vector embeddings 7x faster

How integrating binary quantization allowed Meilisearch to be even faster.

[Read the blog post](https://blog.kerollmops.com/meilisearch-indexes-embeddings-7x-faster-with-binary-quantization)

### 🔍 Semantic search with OpenAI embeddings

Learn to use OpenAI embeddings to build semantic search.

[Read the guide](https://www.meilisearch.com/docs/guides/ai/openai)

### 🛍️ Building ecommerce search with Nuxt

Learn how to integrate real time sorting, filtering, and faceted search with Nuxt 3.

[Read the guide](https://www.meilisearch.com/blog/nuxt-ecommerce-search-guide)

### 🎓 A complete guide to predictive search

Key concepts, optimization techniques, and real-world examples to enhance UX.

[Read the blog post](https://www.meilisearch.com/blog/predictive-search)

## 🤗 Made with Meilisearch

[Fintable](https://fintable.io/) enables syncing bank transactions to spreadsheets. On their [API Coverage page](https://fintable.io/coverage), they use Meilisearch to allow searching through more than 15,000+ bank API connections.

![Searching on Fintable](https://unable-actionable-car.media.strapiapp.com/fintable_d4d4483e90.gif)

We’re always looking to promote new projects from the community. Post yours in the [#built-with-meilisearch](https://discord.meilisearch.com/) channel!


## 🤓 Did you know?

The fastest way to get started is with Meilisearch Cloud. [Start a free trial](https://meilisearch.com/cloud).

---

Stay in the loop of everything Meilisearch by subscribing to our [monthly newsletter](https://dub.sh/meili-newsletter?utm_campaign=newsletter&utm_source=blog&utm_medium=december-24-updates). To learn more about Meilisearch's future and help shape it, take a look at our [roadmap](https://roadmap.meilisearch.com/?utm_campaign=newsletter&utm_source=blog&utm_medium=december-24-updates) and participate in our [Product Discussions](https://github.com/orgs/meilisearch/discussions).

For anything else, join our developers community on [Discord](https://dub.sh/meili-discord?utm_campaign=newsletter&utm_source=blog&utm_medium=december-24-updates).

Meilisearch December updates

We're excited to introduce Meilisearch v1.12. In this article, we'll cover the most significant changes. For a full list of updates, visit the [changelog](https://github.com/meilisearch/meilisearch/releases/tag/v1.12.0) on GitHub.

<Callout>
Meilisearch 1.12 is available on Meilisearch Cloud, too—upgrade now!
</Callout>

## New: `facetSearch` and `prefixSearch` index settings

Meilisearch 1.12 introduces new APIs to disable [facet search](https://www.meilisearch.com/docs/learn/filtering_and_sorting/search_with_facet_filters) and [prefix search](https://www.meilisearch.com/docs/learn/engine/prefix). When disabled, both these settings allow Meilisearch to skip building the required data structures to enable these features. As a result, the indexing process gets much faster.

<Callout>
Disabling these features can impact search result relevancy. Be sure to assess whether your product relies on them before proceeding.
</Callout>

### Updating facet search index settings

Use the new API to [update facet search](https://www.meilisearch.com/docs/reference/api/settings#update-facet-search-settings) index settings:

```ts
// disable facet search
client.index('my_index').setFacetSearch(false)
// enable facet search (default)
client.index('my_index').setFacetSearch(true)
```

### Updating prefix search index settings

Use the new API to [update prefix search](https://www.meilisearch.com/docs/reference/api/settings#update-prefix-search-settings) index settings:

```ts
// disable prefix search
client.index('my_index').setPrefixSearch('disabled')
// enable prefix search (default)
client.index('my_index').setPrefixSearch('indexingTime')
```

## New: batches API

Meilisearch 1.12 introduces a new [batches API](https://www.meilisearch.com/docs/reference/api/batches) to retrieve data about batches. A batch is a set of [tasks](https://www.meilisearch.com/docs/learn/async/asynchronous_operations) processed together by Meilisearch.

Here is an example batch object returned (as of Meilisearch 1.12.0):

```json
{
  "uid": 160,
  "progress": {
    "steps": [
      {
        "currentStep": "processing tasks",
        "finished": 0,
        "total": 2
      },
      {
        "currentStep": "indexing",
        "finished": 2,
        "total": 3
      },
      {
        "currentStep": "extracting words",
        "finished": 3,
        "total": 13
      },
      {
        "currentStep": "document",
        "finished": 12300,
        "total": 19546
      }
    ],
    "percentage": 37.986263
  },
  "details": {
    "receivedDocuments": 19547,
    "indexedDocuments": null
  },
  "stats": {
    "totalNbTasks": 1,
    "status": {
      "processing": 1
    },
    "types": {
      "documentAdditionOrUpdate": 1
    },
    "indexUids": {
      "movies": 1
    }
  },
  "duration": null,
  "startedAt": "2024-12-12T09:44:34.124726733Z",
  "finishedAt": null
}
```

Please note that the [get batches API](https://www.meilisearch.com/docs/reference/api/batches#get-batches) accepts the same parameters as the [get tasks API](https://www.meilisearch.com/docs/reference/api/tasks#get-tasks), allowing you to retrieve batches by filtering by task data. 

For example, you can use `GET /batches?uid=123` to retrieve batches containing tasks with a `taskUid` of `123`.

### Tasks API update

For better integration, the [tasks API](https://www.meilisearch.com/docs/reference/api/tasks) now includes a `batchUid` field in task objects. You can use this value to retrieve details about a specific batch using the API to [get one batch](https://www.meilisearch.com/docs/reference/api/batches#get-one-batch).
## Indexing speed improvements 

Meilisearch 1.12 brings indexing speed improvements across the board for servers making use of multiple cores and having good I/O throughput.

This new release brings:
- 2x faster raw document insertion
- 4x faster incremental document updates in large databases
- 1.5x faster embedding generation

Additionally, this release makes task cancellation faster. On less powerful machines, indexing time remains similar but may show slight improvements.

These advancements build upon the indexing customization and speed enhancements introduced in [Meilisearch 1.6](https://www.meilisearch.com/blog/meilisearch-1-6).

## Contributors shout-out

We want to give a massive thank you to the external contributors who made this release possible. Thanks to [@airycanon](https://github.com/airycanon), [@DerTimonius](https://github.com/DerTimonius), [@flevi29](https://github.com/flevi29), [@LukasKalbertodt](https://github.com/LukasKalbertodt), [@PedroTurik](https://github.com/PedroTurik), and [@wuast94](https://github.com/wuast94) for their contribution to [Meilisearch](https://github.com/meilisearch/meilisearch), and [@dqkqd](https://github.com/dqkqd) and [@tats-u](https://github.com/tats-u) for their work on [charabia](https://github.com/meilisearch/charabia).

We also want to extend special thanks to our SDK maintainers working hard to make Meilisearch available across many [languages and frameworks](https://www.meilisearch.com/docs/learn/what_is_meilisearch/sdks).

---

And that’s a wrap for v1.12! These release notes only highlight the most significant updates. For an exhaustive listing, read the [changelog](https://github.com/meilisearch/meilisearch/releases/tag/v1.12.0) on GitHub.

Upgrade with ease using Meilisearch Cloud.

[Open Meilisearch Cloud](https://cloud.meilisearch.com/login)

---

For more information, subscribe to our [monthly newsletter](https://dub.sh/meili-newsletter?utm_campaign=release-v1-12&utm_source=blog), check out the [roadmap](https://roadmap.meilisearch.com/?utm_campaign=release-v1-12&utm_source=blog), or join our [Product Discussions](https://github.com/orgs/meilisearch/discussions).

For anything else, join our developers community on [Discord](https://discord.meilisearch.com/?utm_campaign=release-v1-12&utm_source=blog).

Meilisearch 1.12

Fuzzy search: a comprehensive guide to implementation

## 🔍 Improved Search Preview

[![Meilisearch Search Preview](https://unable-actionable-car.media.strapiapp.com/meilisearch_search_preview_v2_3b8196bc8b.gif)](https://www.meilisearch.com/docs/learn/getting_started/search_preview)

We’re improving the **Search Preview**. This new version allows further testing of your full-text search **filtering, sorting, and faceting configuration**. Plus, it now comes with a built-in **hybrid search** and **semantic ratio customization**.

[Learn more](https://www.meilisearch.com/docs/learn/getting_started/search_preview)

## ✨ Revamped index settings

[![Meilisearch index settings](https://unable-actionable-car.media.strapiapp.com/meilisearch_index_settings_v2_c5c37c4b21.gif)](https://www.meilisearch.com/docs/learn/configuration/configuring_index_settings)

Our new index settings UI allows easier, **less error-prone configuration** of your  Meilisearch settings. We want to **make Meilisearch welcoming for everyone on your team**—even for those who don't speak JSON fluently.

[Learn more](https://www.meilisearch.com/docs/learn/configuration/configuring_index_settings)

## 💫 Release week recap

[![Meilisearch Q4 2024 release wrap-up](https://unable-actionable-car.media.strapiapp.com/q4_2_24_blog_1_ee4e231bba.png)](https://www.meilisearch.com/blog/q4-2024-release-wrap-up)

Missed our release week? We get it—it happens! Sure, we’re a little sad (we worked hard to ship all this), but no worries. You can catch up on everything you missed in our latest blog post. 

[Read the blog post](https://www.meilisearch.com/blog/q4-2024-release-wrap-up)

## 📰 In the news

### 🔧 [Opt-out settings in v1.12](https://github.com/meilisearch/meilisearch/issues/4979)

New settings to opt out of prefix search and facet search to get 15% faster indexing.

### ⚡ [6.5x faster document ingestion](https://x.com/Kerollmops/status/1859635534626201993)

[Meilisearch was too slow](https://blog.kerollmops.com/meilisearch-is-too-slow) for [@kerollmops](https://x.com/Kerollmops). So we fixed it. 

### 🔌 [Support for Strapi v5](https://github.com/meilisearch/strapi-plugin-meilisearch/releases)

Our Strapi plugin now supports Strapi v5. Shout-out to our contributors [@zoran995](https://github.com/zoran995) and [@candidosales](https://github.com/candidosales) for the awesome work. 🙌

### 🛍️ [Laravel SaaS search demo + repository](https://github.com/meilisearch/saas-demo)

Get started building with Laravel using this demo of multi-index, federated search.

### 🍿 [The greatest Meilisearch ad ever](https://x.com/meilisearch/status/1852030676813378042) (0% clickbait)

It appears our Head of Product [@ledoublegui](https://x.com/ledoublegui) has multiple... talents.

## 🤗 Made with Meilisearch

The [UTMB Index](https://utmb.world/fr/utmb-index/runner-search) allows you to search for any runner who participated in a trail running event. The site uses Meilisearch to power its instant search (with a short debounce!) Thanks to [@CartGauthier](https://x.com/CartGauthier/status/1852025207839449191) for sharing this with us.

[![UTMB index runner search](https://unable-actionable-car.media.strapiapp.com/utmb_runner_search_9b060121f2.gif)](https://x.com/CartGauthier/status/1852025207839449191)

_We’re always looking to promote new projects from the community. Post yours in the [#built-with-meilisearch](https://dub.sh/meili-discord?utm_campaign=newsletter&utm_source=blog&utm_medium=november-24-updates) channel!_

## 🦸 We're hiring

See our open positions:

- [Chief of Staff](https://www.meilisearch.com/careers)

## 🤓 Did you know?

The fastest way to get started is with Meilisearch Cloud. [Start a free trial.](https://meilisearch.com/cloud)

---

Stay in the loop of everything Meilisearch by subscribing to our [monthly newsletter](https://dub.sh/meili-newsletter?utm_campaign=newsletter&utm_source=blog&utm_medium=november-24-updates). To learn more about Meilisearch's future and help shape it, take a look at our [roadmap](https://roadmap.meilisearch.com/?utm_campaign=newsletter&utm_source=blog&utm_medium=november-24-updates) and participate in our [Product Discussions](https://github.com/orgs/meilisearch/discussions).

For anything else, join our developers community on [Discord](https://dub.sh/meili-discord?utm_campaign=newsletter&utm_source=blog&utm_medium=november-24-updates).

Meilisearch November updates


Have you ever noticed how Google suggests exactly what you're looking for before you finish typing? That's predictive search in action. It revolutionizes user interaction with software applications. In this guide, we'll explore implementing predictive search in your applications.

## Introduction to predictive search

### Definition and importance of predictive search

**Predictive search is an intelligent system that anticipates user intent and suggests relevant results in real-time as users type**. Unlike traditional search methods that wait for a complete query, predictive search processes each keystroke to provide immediate, relevant suggestions.

The importance of predictive search in software engineering is significant. It bridges user intent and content discovery, reducing the time and effort users spend searching. Modern applications leverage predictive search to enhance productivity and streamline workflows.

Software engineering teams implementing predictive search can expect improved user engagement metrics. For instance, e-commerce platforms often see a 25-30% increase in conversion rates with predictive search features. Developer tools report significant reductions in time spent searching for code snippets or documentation.

### Predictive search enhancing autocomplete

Traditional autocomplete completes words based on character matching. Predictive search, however, analyzes patterns in user behavior, search history, and contextual data to offer intelligent suggestions.

The personalization aspect of predictive search sets it apart from basic autocomplete. **Predictive search creates a more tailored experience by considering factors like previous searches, user location, and trending queries**.

For example, when a developer searches for "deploy" in a cloud platform's console, the system prioritizes suggestions based on their commonly used services or recent deployments.

<Callout intent="note">
	
	For example, in an e-commerce platform, **autocomplete might suggest "laptop bag"** when a user types "lap", predictive search might also suggest "computer accessories," "mobile workstation," or specific laptop models based on browsing history and market trends.
	
	
</Callout>

## Core concepts and components

### Predictive models in software engineering

Predictive models form the backbone of modern search capabilities. **They utilize complex algorithms to analyze user input patterns and generate relevant suggestions**.

Businesses can leverage large datasets, with the specific type of data varying depending on the industry and business objectives. It is possible to use various data sources like browsing history, user interactions, and market trends to improve predictive search strategies.

![Examples Of Data Predictive Search](https://unable-actionable-car.media.strapiapp.com/predictive_search_b4778b9493.avif)

These models process queries through multiple layers of analysis, including string matching, semantic understanding, and historical pattern recognition.

Predictive models often rely on prefix trees (or radix trees) for efficient string matching and retrieval. This structure enables rapid processing of partial queries, allowing the system to generate suggestions with minimal latency. The models continuously learn from user interactions, refining their prediction accuracy over time.

### Search engines and search queries

Search engines in predictive systems employ sophisticated indexing mechanisms to aid rapid query processing. Key components include query processors, suggestion generators, and ranking algorithms.

These elements work together to produce and prioritize results based on various relevance factors.

![Layers Of Logic Search Engine](https://unable-actionable-car.media.strapiapp.com/predictive_search_2c856b34f6.avif)

Modern search queries go through steps including tokenization, spell checking, and context analysis. **This ensures that even imperfect queries can yield useful suggestions**. Balancing immediate results and accuracy often involves using caching mechanisms to optimize frequently requested queries.

### Machine learning and AI power

Artificial intelligence significantly enhances predictive search through advanced natural language processing and machine learning algorithms.

These technologies help systems understand user intent beyond literal string matching, incorporating semantic understanding and contextual relevance.

AI improves search accuracy by:

- Capturing user intent with natural language understanding
- Adapting ranking algorithms to changing user behavior patterns
- Identifying relationships between different terms through semantic analysis
- Tailoring suggestions based on user profiles and historical interactions

Combining machine learning with traditional search algorithms creates a hybrid approach that delivers both speed and accuracy.

## Capabilities required for predictive search

A robust predictive search implementation requires several key capabilities to deliver a seamless user experience. Fast indexing and efficient query processing are foundational, but there's more to consider.

### Meilisearch's approach to enabling predictive search

Meilisearch combines traditional full-text search with AI-powered capabilities. Its architecture enables fast, relevant results and maintains simplicity in implementation.

**The integration process focuses on three key areas**: data indexing, search configuration, and result optimization. Here's an example of configuring search settings:

```typescript
// config/search.ts
const searchConfig = {
  settings: {
    rankingRules: [
      'words',
      'typo',
      'proximity',
      'attribute',
      'sort',
      'exactness'
    ],
    searchableAttributes: [
      'title',
      'description',
      'category'
    ],
    displayedAttributes: [
      'title',
      'description',
      'category',
      'thumbnail'
    ],
    // Enable AI-powered search with OpenAI
    embedder: {
      type: 'openai',
      documentTemplate: "Product: {{title}}nDescription: {{description}}"
    }
  }
};

await searchClient.index('products').updateSettings(searchConfig);
```

**This configuration enables both traditional search capabilities and AI-powered features**, creating a hybrid search experience that delivers both speed and relevance. The system handles typos, synonyms, and contextual understanding while maintaining sub-50ms response times.

<Callout intent="tip">
	
	Ready to supercharge your search experience? [Explore Meilisearch Cloud](https://meilisearch.com/cloud) and deliver lightning-fast search results that will keep your users engaged and boost your conversion rates.
	
	
</Callout>

## Applications and use cases

Modern software applications leverage predictive search in numerous ways, from e-commerce platforms to content management systems. Understanding these practical applications helps developers implement the right search strategies for their specific use cases.

### App search

**App search is a common implementation of predictive search in software applications**. Mobile apps benefit particularly due to limited screen space and typing constraints.

For instance, food delivery apps use predictive search to help users find restaurants or dishes quickly, considering factors like previous orders and current location.

### Site and media search

**Site and media search capabilities have evolved to handle complex content structures**. Content management systems and documentation platforms use predictive search to help users navigate large information hierarchies.

For example, technical documentation sites can suggest relevant API endpoints or code examples as developers type their queries.

<Callout intent="note">
	
	Check out how Meilisearch powers the discovery of 300,000+ AI models on [Hugging Face](https://www.meilisearch.com/blog/hugging-face-democratizes-ai), demonstrating the power of predictive search in organizing and accessing vast AI resources.
	
	
</Callout>

### Ecommerce search

E-commerce platforms showcase the potential of predictive search through sophisticated product discovery. Online stores suggest not only product names but also categories, brands, and specific features. **This capability extends to handling complex queries like "red summer dress under $50" accurately**.

<Callout intent="note">
	
	For another compelling example of predictive search in action, explore [Meilisearch's case study with Bookshop.org](https://www.meilisearch.com/blog/bookshop-increases-search-based-purchases), where they achieved a remarkable 43% increase in search-based purchases by enhancing their book discovery platform.
	
	
</Callout>

### Enterprise search

[Enterprise search](meilisearch.com/blog/enterprise-search) presents unique challenges because it handles diverse data sources while maintaining security constraints. Predictive search in enterprise environments must consider user permissions, department-specific terminology, and internal document hierarchies.

For instance, a company's internal tools might suggest different results for the same query based on the user's role and access level.

## Techniques and methodologies

### Genetic programming and test models

Genetic programming enhances predictive search capabilities by **evolving and refining search algorithms**. These techniques mimic natural selection to optimize search patterns and improve relevancy.

The system dynamically generates search algorithm variations. It then evaluates these algorithms by analyzing user interactions. Based on performance metrics, the system selects the most effective algorithm combinations to enhance future search experiences.

Test models developed through genetic programming adapt to changing search patterns. For instance, e-commerce search systems can automatically adjust prediction weights based on seasonal trends, user behavior shifts, and new product introductions. **The evolutionary nature of these algorithms ensures relevant search suggestions** as user preferences and patterns evolve.

### Base software and search base

Establishing a robust search base requires careful consideration of data structure and indexing strategies.

<Callout intent="note">
	
	For deeper insights into optimizing search indexing, check our comprehensive guide on [best practices for faster indexing](https://meilisearch.com/blog/best-practices-for-faster-indexing).
	
	
</Callout>

Typically, it consists of an inverted index system enabling quick lookup of terms while maintaining relationships between search elements. **This base structure must support both exact matches and fuzzy searching capabilities**.

Integration of predictive search within software frameworks demands:

- Efficient data storage and retrieval mechanisms to handle large volumes of search queries
- Flexible indexing strategies accommodating both structured and unstructured data
- Scalable architecture maintaining performance under increasing load
- Real-time update capabilities to reflect changes in underlying data

The search base must also handle edge cases and special scenarios. For example, with multilingual search capabilities, the base system must account for different character sets, writing systems, and language-specific patterns. **This ensures reliable and effective predictive search functionality across diverse use cases** and user requirements.

## Capabilities

Low latency is crucial for predictive search systems because users expect instant feedback while typing. Modern search implementations must maintain response times under 100ms to feel instantaneous. This speed requirement becomes even more critical when handling complex queries across large datasets.

Several optimization techniques help achieve these performance goals. **Intelligent caching strategies store frequently accessed results and partial queries**, significantly reducing database load and response times for common searches. Here's how response times typically break down:

| Operation | Target Latency | Optimization Method |
| --- | --- | --- |
| Query Processing | &lt; 20ms | In-memory processing |
| Index Lookup | &lt; 30ms | Cached indices |
| Result Ranking | &lt; 30ms | Pre-computed scores |
| Network Transfer | &lt; 20ms | Edge caching |

Ensuring reliability involves implementing robust error handling and failover mechanisms. **Search systems must gracefully handle scenarios like network issues, invalid queries, or backend service disruptions**. Implementing circuit breakers and fallback options helps maintain service availability.

Performance monitoring is vital for maintaining optimal search functionality. Modern search implementations use real-time metrics tracking to identify bottlenecks and optimization opportunities.

Key performance benchmarks for predictive search systems include:

- **Query response time:** 95th percentile under 200ms
- **Suggestion relevancy:** Minimum 80% accuracy rate for top 5 suggestions
- **System throughput:** Ability to handle 1000+ queries per second per node
- **Cache hit ratio:** At least 85% cache effectiveness

<Callout intent="tip">
	
	Meilisearch Cloud includes [built-in monitoring tools](https://www.meilisearch.com/docs/learn/analytics/monitoring) that enable real-time tracking of search performance metrics.
	
	
</Callout>

The architecture should scale horizontally to handle increasing load while maintaining performance. This typically involves:

1. Distributed index sharding for better resource utilization
2. Load balancing across multiple search nodes
3. Asynchronous index updates to prevent blocking operations
4. Smart query routing based on user location and server load

Reliability also extends to data consistency and accuracy. Search systems must handle index updates and real-time data changes without compromising search quality or availability. **This often requires sophisticated versioning and synchronization mechanisms** to ensure users always see the most relevant and up-to-date results.

## Future directions and innovations

As predictive search technology continues to evolve, several emerging trends and innovations are reshaping the future of search systems.

### Integration with advanced technologies

The future of predictive search is being shaped by **advancements in artificial intelligence and machine learning technologies**. Large language models (LLMs) are transforming how search systems understand and process user queries. These models can interpret natural language with high accuracy, enabling more conversational and context-aware search experiences.

Integration with advanced technologies is already showing promising results. Some systems are experimenting with multimodal search capabilities, processing text, images, and voice inputs simultaneously. This enables more intuitive search interfaces that match how humans naturally communicate.

### Trends in predictive search

LLMs are revolutionizing predictive search capabilities by fundamentally transforming how we interact with search engines. These sophisticated systems can now comprehend complex queries and generate natural, contextually appropriate suggestions that feel remarkably human-like.

Modern search systems now interpret both technical and informal language, moving beyond pattern matching to understand the true meaning behind user queries.

This advancement has enabled several crucial improvements in search functionality:

- More precise interpretation of user queries
- Sophisticated handling of ambiguous searches
- Enhanced recognition of specialized terminology
- Intelligent, context-aware result ranking

**These innovations are already being implemented in various forms**. Some search engines use hybrid approaches that combine traditional search algorithms with [AI-powered semantic understanding](meilisearch.com/blog/ai-powered-search). This allows them to maintain the speed and reliability of conventional search while leveraging advanced AI capabilities.

## Beyond horizons: the transformative potential of predictive search

Predictive search has evolved from a convenient feature to an essential component of modern software applications.

Successful implementation requires careful consideration of architecture, performance optimization, and user experience design.

The integration of advanced AI technologies and mobile-first approaches will continue to push the boundaries of what's possible in predictive search. This makes it an exciting area for continued innovation and development.

<Callout intent="tip">
	
	Unlock advanced search potential with customizable relevancy, typo tolerance, and more. [Enhance Your Search Strategy](https://meilisearch.com/features) with Meilisearch Cloud's powerful search capabilities.
	
	
</Callout>

Software engineering predictive search: a complete guide

_This blog post was originally published on kero's [personal blog](https://blog.kerollmops.com/meilisearch-indexes-embeddings-7x-faster-with-binary-quantization)._

<Callout intent="note">
This is one blog post in a series:
- [Spotify-Inspired: Elevating Meilisearch with Hybrid Search and Rust](https://blog.kerollmops.com/spotify-inspired-elevating-meilisearch-with-hybrid-search-and-rust)
- [Multithreading and Memory-Mapping: Refining ANN Performance with Arroy](https://blog.kerollmops.com/multithreading-and-memory-mapping-refining-ann-performance-with-arroy-in-rust)
- [Meilisearch Expands Search Power with Arroy's Filtered Disk ANN](https://blog.kerollmops.com/meilisearch-expands-search-power-with-arroy-s-filtered-disk-ann)
- [How Meilisearch Updates a Millions Vector Embeddings Database in Under a Minute](https://blog.kerollmops.com/how-meilisearch-updates-a-millions-vector-embeddings-database-in-under-a-minute)
- Meilisearch Indexes Embeddings 7x Faster with Binary Quantization.
</Callout>

For the past year at Meilisearch, [we’ve been working hard on hybrid search](https://github.com/meilisearch/meilisearch/issues/3838), which mixes keyword search with semantic search using [Arroy: our vector store, based on a Spotify technology](https://blog.kerollmops.com/spotify-inspired-elevating-meilisearch-with-hybrid-search-and-rust). A vector store is a data structure that efficiently stores embeddings (vectors) for fast and relevant retrieval, called Approximate Nearest Neighbors (ANN) search.

When our customers started to use arroy heavily, some of them started reaching the machine limits. For example, for a customer with a 768 dimensions model, we found out that we couldn’t index more than 15M of embeddings on a machine with 64GiB of RAM.

### Binary Quantization to the Rescue

The concept behind this elaborate term is that we are going to *quantize* the embeddings. Quantizing a number involves spreading it into a defined number of values; for *binary* quantization, two values can be represented on a single bit, e.g., 0.2561 becomes 1, and -0.568 becomes -1.

This implies that we could convert a 32-bit float number into a 1-bit number, which would divide the disk and RAM usage by 32. That would mean we could index up to 480M embeddings with 64GiB of RAM, instead of being limited to 15M embeddings.

An issue we faced is that bits cannot represent -1 and 1 but only 0 and 1. We had to use some tricks to make it work and fake 0s into -1s in our computations. For example, one of the functions that we had to optimize a lot was the function in charge of converting a binary quantized vector to 32-bit float vectors.

We do not publicly expose raw quantized embeddings and utilize this function to provide real 32-bit float embeddings to the users. However, the primary reason for converting 1-bit quantized embeddings to 32-bit floats is to ensure high precision when [generating hyperplanes to group embeddings together](https://blog.kerollmops.com/spotify-inspired-elevating-meilisearch-with-hybrid-search-and-rust#spotify-s-hyperplane-trees-for-efficient-anns) and maintaining relevant ANNs.

![Profiling backtrace showing slow 1-bit to 32-bit float conversion](https://unable-actionable-car.media.strapiapp.com/68747470733a2f2f692e696d6775722e636f6d2f497165713649562e706e67_596adc5fd5.png)

_Converting a quantized embedding back to 32-bit floating points takes up to 24.21s on a 1m53s run._

Below is the SIMD neon-specific version of this function. It processes raw bytes corresponding to a 1-bit quantized embedding, converting them into a properly aligned 32-bit float embedding. Using masks, the function separates the low and high bits, processing them in batches of four in 128-bit registers, which are then transformed into four 32-bit floats. These values are collectively written to the output vector.

```rust
unsafe fn to_vec_neon(bytes: &[u8]) -> Vec<f32> {
    let mut output: Vec<f32> = vec![0.0; bytes.len() / u32::BITS];
    let output_ptr = output.as_mut_ptr();
    let low_mask = [0b_0000_0001, 0b_0000_0010, 0b_0000_0100, 0b_0000_1000];
    let high_mask = [0b_0001_0000, 0b_0010_0000, 0b_0100_0000, 0b_1000_0000];
    let ones = unsafe { vld1q_dup_f32(&1.0) };
    let minus = unsafe { vld1q_dup_f32(&-1.0) };

    for (current_byte, base) in bytes.iter().enumerate() {
        unsafe {
            let base = *base as u32;
            let base = vld1q_dup_u32(&base);
            for (i, mask) in [low_mask, high_mask].iter().enumerate() {
                let mask = vld1q_u32(mask.as_ptr());
                let mask = vandq_u32(base, mask);
                // 0xffffffff if equal to zero and 0x00000000 otherwise
                let mask = vceqzq_u32(mask);
                let lane = vbslq_f32(mask, minus, ones);
                let offset = output_ptr.add(current_byte * 8 + i * 4);
                vst1q_f32(offset, lane);
            }
        }
    }

    output
}
```

_<a href="https://x.com/lemire/status/1835700368648323261">Daniel Lemire doesn't seem to have found a better algorithm</a>_ 🤭

### The Results

We chatted about the theoretical benefits of this approach earlier, but how does it work out in real life? Here are different tables summarizing the results for you.

<details>
  <summary>Open to see the original benchmarks</summary>
  <img src="https://unable-actionable-car.media.strapiapp.com/68747470733a2f2f692e696d6775722e636f6d2f67667765626b592e706e67_a96a995c7c.png"  alt="Original arroy benchmark results"/>
</details>

#### Embeddings with 1024 Dimensions

When examining small embeddings, we can observe a significant impact on relevancy due to binary quantization. The primary advantages of this approach lie in faster insertion times and reduced storage requirements. It is not advisable to enable this feature for small embeddings, as they already occupy minimal disk space. Storing over four million 1024-dimensional embeddings requires just 8 GiB of disk space.

| Version                 | Recall @1 | Recall @20 | Recall @100 | Indexing time | Search time | On disk size |
|-------------------------|-----------|------------|-------------|---------------|-------------|--------------|
| Cosine                  | **1.00**  | **0.72**   | **0.84**    | 491s          | **23s**     | 8 GiB        |
| Binary Quantized Cosine | 0.87      | 0.50       | 0.52        | **147s**      | 27s         | **850 MiB**  |

#### Embeddings with 1536 Dimensions

Binary quantization proves advantageous for managing larger embeddings. It significantly reduces search times, utilizing just 15% of the original disk space. Indexing about 500,000 1536-dimensional embeddings is achieved in a quarter of the time. Despite this, there remains a noticeable impact on relevancy...

| Version                 | Recall @1 | Recall @20 | Recall @100 | Indexing time | Search time | On disk size |
|-------------------------|-----------|------------|-------------|---------------|-------------|--------------|
| Cosine                  | **1.00**  | **0.82**   | **0.90**    | 1219s         | 50s         | 13 GiB       |
| Binary Quantized Cosine | **1.00**  | 0.70       | 0.70        | **235s**      | **47s**     | **2 GiB**    |

#### Embeddings with 3072 Dimensions

The impact of binary quantization on large embeddings is evident. While relevance may be slightly affected, the reduction in disk usage, search time, and indexing time is significant. Being able to store the same number of embeddings with just 15% of the space and index 6 times faster is a major achievement! It takes half as much disk space to store a million 3072-dimensional binary quantized embeddings than four million 1024-dimensional embeddings.

| Version                 | Recall @1 | Recall @20 | Recall @100 | Indexing time | Search time | On disk size |
|-------------------------|-----------|------------|-------------|---------------|-------------|--------------|
| Cosine                  | **1.00**  | **0.86**   | **0.92**    | 4000s         | 150s        | 26 GiB       |
| Binary Quantized Cosine | **1.00**  | 0.77       | 0.77        | **560s**      | **100s**    | **4 GiB**    |

### Conclusion

The impact of binary quantization on recall was expected to decrease, given the reduced information in the algorithm. However, for datasets with over 1500 dimensions, the impact on recall is relatively minor compared to the significant benefits in disk space and indexing speed.

In practice, we observed that space utilization decreased not by 32 times but by just 15% of the original space, a notable achievement. This is primarily due to arroy storing not only user vectors but also internal nodes for efficient searching. Split nodes (hyperplanes) are retained as full 32-bit floats to maintain precision.

- Indexing speed improves nearly 5 times with 1536 dimensions and about 7 times with 3072 dimensions.
- Disk size shrinks to around 6.5 times smaller.
- Search time shows slight improvements for embeddings starting at 1536 dimensions.

We recommend using Meilisearch/arroy's binary quantization feature for OpenAI users with large embeddings. In an upcoming article, we will compare Meilisearch/arroy with competitors, focusing on our Filtered DiskANN feature. Stay tuned!

Meilisearch latest news and company updates