Impact of RAM and multi-threading on indexing performance

Adding new documents to an index is a multi-threaded and memory-intensive operation. Meilisearch's indexes are at the core of what makes our search engine fast, relevant, and reliable. This article explains some of the details regarding RAM consumption and multi-threading.

By default, our indexer uses the sysinfo Rust library to calculate a machine's total memory size. Meilisearch then adapts its behavior so indexing uses a maximum two thirds of available resources. Alternatively, you can use the --max-indexing-memory instance option to manually control the maximum amount of RAM Meilisearch can consume.

It is important to prevent Meilisearch from using all available memory during indexing. If that happens, there are two negative consequences:

1. Meilisearch may be killed by the OS for over-consuming RAM

2. Search performance may decrease while the indexer is processing an update

Memory overconsumption can still happen in two cases:

1. When letting Meilisearch automatically set the maximum amount of memory used during indexing, sysinfo may not be able to calculate the amount of available RAM for certain OSes. Meilisearch still makes an educated estimate and adapts its behavior based on that, but crashes may still happen in this case. Follow this link for an exhaustive list of OSes supported by sysinfo

2. Lower-end machines might struggle when processing huge datasets. Splitting your data payload into smaller batches can help in this case. For more information, consult the section below

In machines with multi-core processors, the indexer avoids using more than half of the available processing units. For example, if your machine has twelve cores, the indexer will try to use six of them at most. This ensures Meilisearch is always ready to perform searches, even while you are updating an index.

You can override Meilisearch's default threading limit by using the --max-indexing-threads instance option. Allowing Meilisearch to use all processor cores for indexing might negatively impact your users' search experience.

Multi-threading is unfortunately not possible in machines with only one processor core.

In some cases, the OS will interrupt Meilisearch and stop all its processes. Most of these crashes happen during indexing and are a result of a machine running out of RAM. This means your computer does not have enough memory to process your dataset.

Meilisearch is aware of this issue and actively trying to resolve it. If you are struggling with memory-related crashes, consider:

• Adding new documents in smaller batches
• Increasing your machine's RAM
• Following indexing best practices