Introducing Meilisearch's next-generation indexer: 4x faster updates, 30% less storage

When you're handling millions of documents in a search engine, every millisecond counts. That's why we've completely reimagined how Meilisearch processes data with our new indexer in v1.12. We've rewritten our indexer—the core component that processes your documents and builds search structures—from the ground up. The result? A faster, more efficient, and more scalable system.

This complete redesign fundamentally changes how our search engine handles data. In this article, we'll show you how we've achieved up to 4x faster document updates and 30% smaller database sizes, and what this means for your applications.

A foundation for the future

We've named this release Indexer edition 2024, reflecting our vision to build a foundation that will power Meilisearch's performance for years to come. Inspired by Rust's edition model, this isn't just an update—it's a new chapter in Meilisearch's evolution, setting the stage for future innovations in search technology.

(Also like Rust editions, we took the year it was developed rather than the year it was released 😁)

Why would you write a new indexer?

The Doctor would not rewrite an indexer

You need a very strong reason to re-write an indexer. And we had one.

As our CTO Kero put it bluntly in his blog post: Meilisearch is too slow. The new indexer is our answer to that challenge—a ground-up rewrite to dramatically improve speed.

The main goal of the new indexer is to improve the indexing speed of larger indexes with dozens of millions of documents, to relieve some of the stress on our infrastructure, to better exploit the resources of the machine, and to address some long-running pain points in a structured way.

The crux of the matter: performance improvements

Sometimes you really gotta go faster

In various scenarios, the new indexer is faster, sometimes multiple times faster, than the already optimized v1.11 indexer.

On machines with multiple cores, good IO and a lot of RAM, performance is vastly increased. Inserting new documents is twice as fast, while incrementally updating documents in a large database is x4 faster!

While the goal was to improve performance on bigger machines, we were successful in keeping similar or better performance for more modest machines with fewer cores and humble RAM (good IO remains vital to a database, though).

Additionally, for Cloud customers, we’re exploring optimizing our build by using native CPU instruction and link-time-optimization (LTO), yielding up to 12% improvements. We’re also considering advanced techniques like profile-guided optimization (PGO), which so far has yielded an additional 12% improvement over native instructions and LTO.

How it’s done: more parallelism, fewer I/O operations, pipelining writes

Meilisearch used to split a payload of documents into multiple chunks (roughly one per indexing thread), and would then work each chunk throughout the entire extraction process. Once a chunk was finished, it would then be persisted in the database.

This approach had a couple of drawbacks, performance-wise:

• As a lot of temporary data was extracted at the same time, it had to be written to disk and then read back to be persisted in the database, causing a lot of I/O operations.

• As the chunks were built independently of each other, they could result in us writing the same key to the database for each chunk, again causing unnecessary writes.

• As the chunks would finish computing at the same time on average, the indexing process would intensively use all the CPUs while computing the chunks, but then be solely single-threaded while writing everything to the database.

Simplified representation of the previous indexing process

In the new indexer, Meilisearch iterates in parallel over documents, performing a single extraction operation at a time. The result of that extraction is then merged in parallel, to compute the list of keys and values to send to the database. These keys and values are sent to the writer thread in a channel that acts as a pipeline, reducing the time where the merger threads wait for the writer thread.

The merger threads wait less thanks to pipelining

We successfully made the merger step parallel thanks to a key insight: while the extraction steps split documents between multiple threads, the merger step has to split database keys between multiple threads. For instance, to build the reverse index that matches a word with the list of documents that contain that word, each thread reads some of the documents to extract the words it contain, then all the documents containing a word are merged into a single list.

To make this possible, all threads during the extraction step deterministically hashes each word it encounters. Then during the merger step, words to handle are partitioned between threads depending on their hash value.

Simplified representation of indexing process edition 2024

The drawback to this new approach is that some CPU-intensive operations, such as document tokenization, have to be duplicated across all steps. Still, merging in parallel utilizes a machine's multiple cores much more, reducing the time Meilisearch spends single-threaded.

Furthermore, by not duplicating key writes, the database size has been reduced by more than 30%.

Better RAM control and usage

Meilisearch uses bumpalo to improve RAM control. Bumpalo is an arena allocator that allows to bulk deallocate memory and to query how much memory was allocated in an arena.

Arena allocators are often used in video games for objects that have to be recreated on each game frame, because allocating in an arena is very cheap (just a pointer bump), and deallocating can happen in bulk (yet another pointer bump) as long as no destructor has to be run.

In some regards, indexing documents with Meilisearch also has “frames”. In the new indexing approach, we can even detect two kinds of such frames: A very short-lived frame that corresponds to the work done on each document. And a longer-lived frame that starts at the beginning of each extraction step, and ends right after the merger is done sending everything to the writer. The “document” arena contains all allocations related to reading and tokenizing that document, while the “extraction” arena contains all allocations related to the data being extracted that will then be merged and written to the database.

During this process, Meilisearch queries the size of the “extraction” arena to detect when its RAM usage is above the threshold. When this happens, Meilisearch spills excess data to disk, so that the RAM usage stays in control.

If the extracted data fits in RAM, it never hits the disk, yielding fewer I/O operations.

Faster and more reliable task cancellation

Historically, in the previous indexer, chunks of documents used to be very small, so it made sense to allow cancellation only after a chunk was done processing. Unfortunately, small chunks were inefficient when writing temporary data to disk, so we decided to switch to a “one chunk per core” model a few versions ago. This had an adverse effect on cancellation, as it would be processed almost at the end of the indexing process after all threads finished processing their chunk.

As the new Meilisearch iterates on documents, and further split the process by each extraction step, task cancellation is faster than ever, routinely being processed under 1s!

For all of you who regret sending all these documents at once

Better error messages

The new architecture exposes the document id to all extractors. This means that error messages can be enriched with this information as was done in this PR: knowing which document caused the error allows for more effective troubleshooting.

Better observability with progress

Lastly, the new indexer brings better observability to the table: because we know both which step we’re at and how many documents have been handled for this step, we can expose this information in the newly added batches route as a progress object.

{
  "uid": 160,
  "progress": {
    "steps": [
      {
        "currentStep": "processing tasks",
        "finished": 0,
        "total": 2
      },
      {
        "currentStep": "indexing",
        "finished": 2,
        "total": 3
      },
      {
        "currentStep": "extracting words",
        "finished": 3,
        "total": 13
      },
      {
        "currentStep": "document",
        "finished": 12300,
        "total": 19546
      }
    ],
    "percentage": 37.986263
  },
  "details": {
    "receivedDocuments": 19547,
    "indexedDocuments": null
  },
  "stats": {
    "totalNbTasks": 1,
    "status": {
      "processing": 1
    },
    "types": {
      "documentAdditionOrUpdate": 1
    },
    "indexUids": {
      "mieli": 1
    }
  },
  "duration": null,
  "startedAt": "2024-12-12T09:44:34.124726733Z",
  "finishedAt": null
}

In the above, we know that we finished extracting the words of 12300 out of 19546 documents. There are a total of 13 indexing steps, and we finished 3 of them already, for a global completion of 37.986263%.

Note that the progress object should only be used for display purposes, we do not guarantee that the step names or number remain the same from one version to the next.

Adding the progress object is a boon for the engine's observability, which already scored us concrete wins. We identified the bottleneck in a customer’s workload just by repeatedly calling the batches route and looking at which step was frequently seen.

The best is yet to come!

The new indexer reaches new heights in maintainability with its structure that decouples the “business” extractors from its support functions (cancel, progress, reading a document), allowing us to quickly iterate on features, such as new settings to fine-tune the indexing behavior and the AI stabilization. In the future, we plan on adding progress to the settings tasks, enriching finished batches with the timing of each step. Feel free to share your ideas about what you’d like to see in Meilisearch on the product discussion. If you’re new to Meilisearch, you can also try it for free on our Cloud.