Cracking the code: Symfony's search solution for 12 million words of documentation

Symfony is an open-source web development framework for PHP that enables users to create web applications by utilizing sets of reusable components. Originating from France, Symfony combines a web application framework, a philosophy, and a vibrant community.

With Meilisearch, Symfony gained better control over parsing and indexing of documentation content to speed up and improve the quality of search results.

Using Meilisearch has been a delight so far. It all starts with its quick and simple installation process, followed by its great and thorough documentation, and it ends with a powerful search engine with mind-blowing performance. Meilisearch feels fun, fresh and uncomplicated. - Javier Eguiluz, Symfony Evangelist.

Challenge

Fast and efficient information retrieval from Symfony’s documentation is essential for the success of its rapidly growing community. The Symfony Documentation, including code samples, comprises well over 12 million words, and throughout its history, Symfony has published 30 different version updates, making smooth information retrieval from the documentation crucial. In fact, to ensure the search function is as accessible as possible for symfony.com visitors, the team integrated multiple keyboard shortcuts to open the search input across the entire website.

In the past, Symfony’s team explored various products and services, including Apache Solr, Elasticsearch, Google Custom Search Engine, and Algolia Search. However, the team realized the need for better control over the parsing and indexing of search content in order to tweak the search rules and improve website navigation and the overall search experience as a result. While using a fully hosted service is convenient, the pre-existing configuration via JSON files proved cumbersome and did not provide Symfony with the flexibility it required.

Secondly, there was potential to improve the overall quality of the search results, with the Symfony team receiving some complaints about inaccurate or missing results from the documentation users. Unfortunately, due to the aforementioned lack of control over the parsing and indexing processes, significant improvements to these results were unattainable and the need for a better alternative became evident.

Why Symfony chose Meilisearch

Determined to find a better alternative, Symfony’s team started research on search engines. From the very beginning, the preference was for open-source projects developed in Go, because there was existing internal prior expertise in that language. Although there were many search solutions on the markets that matched those requirements, many of them lacked active development, didn’t have an active community, or missed essential features such as support for synonyms.

Finally, the team looked at Meilisearch, which provides an open-source search engine developed in Rust, along with an option to self-host, and quickly found a fit. Meilisearch’s established reputation as an open-source project with a vibrant community played a pivotal role in influencing the final decision.

Implementation

Meilisearch offers integrations with a wide range of programming languages and frameworks. In addition to hosting Symfony's index data, the Meilisearch team provided valuable assistance during the migration process so that a single developer who had no prior experience with Meilisearch was able to handle an entire migration and the implementation process, using the Meilisearch PHP integration with the Symfony HttpClient component.

Here’s a brief overview of the technical integration process:

1. Parsing Symfony Documentation: For Symfony’s documentation, the process included splitting the entire page contents into multiple small documents in order to produce better search results, closely related to the query terms.
2. Indexing Documents was the simplest part of the process and included configuring stopwords (list of words to be ignored in search queries) and synonyms to produce better search results.

For a full walkthrough of how Symfony implemented parsing of their documentation, documents indexing, and search engine interface, we recommend taking a look at their blog post on migrating symfony.com search engine to Meilisearch.

Results

The Symfony team focuses primarily on two different aspects of performance when evaluating the results:

• Performance of indexing contents: Meilisearch can index approximately 800 full documents per second, which includes the processing of hundreds of stopwords and tens of synonyms.
• Performance of queries: Symfony, including all versions of Symfony Docs and documentation for bundles, parses 12,754 files and generates 85,146 Meilisearch documents. The total time taken to parse, create, and index all these documents is 17 seconds. The search index consumes approximately 1.2 GB of disk space, with most queries returning results in less than 10 milliseconds.

Nicolas Grekas, Core team member at Symfony, summarizes it as follows: “These numbers are quite impressive and offer an excellent experience when updating search results in real-time as you type."

To complete the evaluation process, the Symfony team conducted a comprehensive comparison between the previous search engine results and the new Meilisearch engine. They tested typical queries, rare queries, long queries, and queries with typos. Below is a comparison of the search results obtained before (left) and after implementing Meilisearch (right):

Since the new search experience was deployed, the team hasn't received any negative feedback from the community, which normally served as an indicator of the quality of the search experience.

Vision

Currently, Symfony's team is satisfied with Meilisearch's feature offering, which offers all the essential features needed, including indexing arbitrary information, sorting, result weighting, handling stopwords, synonyms, facets, filtering, and more.

Javier Eguiluz, Symfony Evangelist, advises: "Before integrating it into your projects, be aware of some known limitations. If your project requirements are complex (e.g., cross-faceting between multiple indexes, etc.), make sure to check if these advanced features are already supported. Additionally, it's a good idea to review the Meilisearch public roadmap."

For a full walkthrough of how Symfony implemented parsing of their documentation, documents indexing, and search engine interface, we recommend taking a look at their blog post on migrating symfony.com search engine to Meilisearch.

Ready to optimize your documentation search? Start a free trial with Meilisearch today.