We are proud and thrilled to present our [open-source search engine](https://github.com/meilisearch/MeiliSearch). After one year of hard work, we are finally able to make our first official beta release.

We want to contribute to the open-source community by offering an ultra-relevant & instant search engine to improve the search experience in every application.

Meilisearch is the **first open-source instant search engine**. Instant means that our search engine is so fast to answer that you have the search-as-you-type experience - results are appearing and reordering on each keystroke.

![](https://unable-actionable-car.media.strapiapp.com/crates_io_demo_907b54a599.gif)

Meilisearch finds crates on [crates.meilisearch.com](https://crates.meilisearch.com)

We differ from other open-source search engines in many ways.Compared to Elastic Search (and other Lucene-based engines), we specialize in end-user search, and Meilisearch works out-of-the-box to that end. Compared to your database search engine, we are more relevant and way faster. We have typo-tolerance, and it works ;)

Motivations
-----------

We built Meilisearch with the end-users in mind, to reduce the friction between the user and the machine when using your app, browsing your website.

> You should not be Amazon or Google to have a good search engine.

It seems unfair to us that today, only the biggest companies could afford the luxury of a decent search engine. From the biggest corporations to your Sunday side-project, you should have a go-to search engine that does not cost you an arm and a leg. You should not need the financial means of Google or Amazon to afford that.

![](https://unable-actionable-car.media.strapiapp.com/photo_1571924035778_d53e10ef9858_dc0da161e2.jpeg)

Team
----

At the start, we were three developers. And because a developer knows how to recognize another good developer, we hired two other developers. Today, we are a team of five working on the engine, SDK's, front-end libraries, [documentation](https://docs.meilisearch.com/), articles, and social media.

How did we survive at 5 in Paris for one year, you may ask. Well! We participated in a Hackaton about search organized by Louis Vuitton. At that time, we were already working on our search engine, and it seemed to be a call of destiny. We finished first!

Our winning price was three Louis Vuitton bags 👜, and then, as a little bonus, we signed a contract with them to develop our product and run it in production on their servers.

![](https://unable-actionable-car.media.strapiapp.com/Hackaton_920e7ba818.jpeg)

What's coming next
------------------

We will continue to work on the search engine to make it faster and more relevant than ever and adding features such as geolocation or trendiness. We will work on different APIs wrapper and front-end libraries to make Meilisearch more and more accessible. In a few months, we may even offer our search engine as a service to free you from the hassle of maintaining your infrastructure.

If our story inspired you, please give a try to our search engine. You can follow the quick start and read more about how it works in the documentation. If you want to support us, you can star the project on Github.

![](https://unable-actionable-car.media.strapiapp.com/tenor_1_c11a2fa59b.gif)

Ressources
----------

*   [Documentation](https://docs.meilisearch.com)
*   [Github](https://github.com/meilisearch/MeiliSearch)
*   [Twitter](https://twitter.com/meilisearch)

Meilisearch: Beta release!

Today, I am about to guide you in the depths of crates.io and how [I made an alternative search bar](https://crates.meilisearch.com) using our instant search engine: [Meilisearch](https://github.com/meilisearch/MeiliDB).

![](https://unable-actionable-car.media.strapiapp.com/crates_io_demo_798a562ddd.gif)

The Meili instant, relevant, and typo-tolerant search engine

[Crates.io](https://crates.io) is the official website that stores Rust community crates (packages), and it's the place where the cargo package manager uploads, updates, and downloads those.

[Sean Griffin](https://twitter.com/sgrif) is part of the crates.io team and maintains the current search engine of it along with the whole website. [Kornel Lesinski](https://kornel.ski/) built [lib.rs](https://lib.rs/) an alternative to [crates.io](http://crates.io/) and uses [Tantivy](https://github.com/tantivy-search/tantivy/) to power its search bar. To be honest, I prefer its color design, that's why I took it for our search demo.  
  
I decided to run our instant search engine and test its relevancy overtime against these existing solutions. Our search engine uses completely different algorithms; it is based on prefix search and is typo tolerant.

![](https://unable-actionable-car.media.strapiapp.com/current_results_1_73457862c1.png)

Meili is typo-tolerant and supports a whole lot of other features

So, I asked myself: Why not use our new instant search engine and make it useful to my beloved community? It would give us lots of feedback and probably some pull requests during the process.

* * *

At Meili we manage an internal Kubernetes cluster, this is useful to host demos for clients. The Meilisearch server for this demo currently run on a pod in this cluster.

To make Meilisearch exhibit those crates, we needed to find all the currently available packages on crates.io. Fortunately, this index is available on GitHub in the form of several subfolders with the names and versions of the packages, containing something like _32 000_ files. A commit is done each time a crate is updated to a new version, or a version is yanked.

So, I used the [crates.io-index repository](https://github.com/rust-lang/crates.io-index) to **initialize** our newly created Meili search engine but needed more data first, like, the description, keywords, and categories of each of those crates. Again, the Rust crates.io team was here for us, I talked to [Pietro Albini](https://twitter.com/pietroalbini), and he pointed me to the not rate-limited servers that deliver packages content.

Now that we can retrieve useful data, I create an [async crawler](https://github.com/Kerollmops/meili-crates/blob/67770fe179436eee6478cdebff7d8613f9263049/src/lib.rs) that downloads, extracts, retrieves the Cargo.toml, and upload the essential data to Meilisearch.

![](https://unable-actionable-car.media.strapiapp.com/Screenshot_2019_11_08_Meili_Dashboard_c9d2ae42f1.png)

The Meili dashboard interface showing raw documents

Meilisearch now understands that data and gives us instant, relevant and typo tolerant responses. But, what about new crates? We want to be notified about new crates and be able to send those to Meilisearch.

[Docs.rs](https://docs.rs) is the official website to compute and store the documentation of all the Rust packages hosted by crates.io. It diffs the crates.io index each minute to know about new crates updates. Luckily, it provides an Atom feed of those updates.

It is where [Heroku](https://heroku.com) enters the game. Heroku offers something like _1000_ free hours per month of computing power on their servers and provides us schedulers. We can consume those credits and freely [ask docs.rs about newly updated crates](https://github.com/Kerollmops/meili-crates/blob/67770fe179436eee6478cdebff7d8613f9263049/src/bin/live.rs), every 10 minutes, by fetching the Atom feed, download the updated packages like previously, and finally update our search engine, **live**!

![](https://unable-actionable-car.media.strapiapp.com/heroku_live_timings_1_fbb7661c94.png)

Fetching new crates takes us 4 seconds

Search results were satisfying but not as expected, there was something wrong. When we wrote "serde", for example, the first result was relevant, but the next ones were not. It was related to the fact that Meilisearch did not have enough data to rate crates except the query matching words.

![](https://unable-actionable-car.media.strapiapp.com/Screenshot_2019_11_10_Meili_crates_browser_2000_1_5cfc7e603c.png)

Meilisearch doesn't know how to settle equal crates with only the matching words

**Downloads counts** amazingly improved the search results. [This data is available through crates.io](https://crates.io/data-access). A full database export is done each day, and it contains the number of downloads of each crate. I decided to use those as the last ranking criterion to help Meilisearch settle about crates considered equal.

![](https://unable-actionable-car.media.strapiapp.com/Screenshot_Meili_crates_browser_2000_good_1_21f3eb0098.png)

Meilisearch shows better results thanks to the number of downloads

I deployed a Heroku scheduler to run each day to update all the crates downloads; it takes something like 30 seconds to download the tarball, extract it, read the CSV and upload the _32 000_ crates downloads counts to Meilisearch. So we are far from _1000_ free hours per month.

* * *

I think this search demo is pretty good, but I also thought about adding synonyms and stop-words as Meilisearch supports those. For example, it would be pleasant to write "db" and see results associated with "database". Stop-words would help ignore useless words like "the" or "that" which could pollute search results sometimes, but we need to be careful as crates names could be composed of stop-words.

Meilisearch also supports basic filtering, and it would be great in the future to be able to search for crates in a category or with a specific keyword.

All these improvements can be made by you; [the demo is open source](https://github.com/Kerollmops/meili-crates). The [core engine source code](https://github.com/meilisearch/MeiliDB) is also available on GitHub, that's the whole point of this article! Take a look and talk about it, more people involved, more features!

Search for Rust crates with Meili

SpO₂ is oxygen saturation and is used in medical person monitoring.

I am Kerollmops, I am the CTO of [Meili](https://www.meilisearch.com/) and today I am releasing [SpO₂](https://github.com/Kerollmops/spo2/).

At Meili, we needed a tool that allows us to monitor our pods, we already have [vigil](https://github.com/valeriansaliou/vigil) which health checks our front page and backend, but the number of these services is limited. We do not pop new front or backend servers dynamically (for now). When we create new search engines for the user we instanciate a kubernetes pod, we need to monitor the health of this service. Adding each of those URLs by hand in the vigil config file is not a solution.

So we decided that we needed a simple tool, a tool that can accept HTTP requests to register/unregister URLs to health check. We use the [new async/await Rust syntax](https://blog.rust-lang.org/2019/09/30/Async-await-hits-beta.html) along with [tide](https://github.com/http-rs/tide) for the http server, no big deal here.

Our current cloud provider is [Digital Ocean](https://www.digitalocean.com/), therefore, we cannot host our SpO₂ service there. We chose [Scaleway](https://www.scaleway.com) as it is way cheaper, and it works out of the box. We need persitent storage of the health checked URLs. What would you do if those are only stored in RAM? What if the server restarts? I already worked on [a disk backed key-value store in Rust named Sled](https://sled.rs/). So we chose to rely on it.

![](https://unable-actionable-car.media.strapiapp.com/screenshot_d586f10f06.png)

[@qdequele](https://quentin.llmops.com) built the front-end with vanilla javascript. Using Websocket we are able to display in realtime the pods status. As most humans do not stand in front of a TV the whole day, spotting color changes to notify the devops, we decided to implement notifications. We are using Slack for all of our monitoring tools, and because it is as simple as a webhook to implement, we went with it.

In the last release, we made some improvements to the Slack notification system. We now batch status changes events by 40; this means that SpO₂ sends one message with at most 40 events and regulates channel spamming. It also displays the HTTP status related to an unhealthy measurement and the error message on an unreachable one.

SpO₂ does not support SSL/TLS by itself, neither for the HTTP nor the WebSocket endpoints. We needed this kind of security so we looked at NGINX, a tiny little obscure reverse proxy server which we configured with basic authentication. It is not an easy task and because we are cool we made the documentation to help you do the same.

Do not hesitate to share or star this project, pull requests are welcome 😊

And just as a side note, we do not measure humans but machines actually.

SpO₂ the little dynamic monitoring tool

We want to share our thoughts about open source and how we should license our code. This is entirely new to us, and we are still figuring out the best solution and opportunities so if you have any reading recommendation do not hesitate to tell us.

As a quick reminder: at Meili, we are building [an instant search engine](https://github.com/meilisearch/MeiliDB), in Rust 🦀, and we want to open source it. We are still wondering about the right business plan for us, and we may write about it too but the subject of the day is license!

Why did we choose to go open source?
------------------------------------

We believe that every application deserves a decent search engine. That means that search engine technology should become a commodity. We found a sweet spot between Algolia which is very developer-friendly but proprietary and Elastic Search which is difficult to set up for an instant search.
  
We think that if you want to become the standard technology in your field, you should be open source, that way you get to build your features with the people who use it and you gain trust around your code. Also, because we are cool developers, we are using a lot of open source code, and we want to give back by contributing to open source and also sharing our projects.

What is open source?
--------------------

Open source is complicated, and if you ask two persons a definition of open source, you may realize that there are many definitions of open source. [Steve Klabnik wrote a very interesting blog post](https://words.steveklabnik.com/the-culture-war-at-the-heart-of-open-source) about the culture war at the heart of open source in which we learned about open source history and why people are fighting over its definition.

In the beginning, there was the Free Software Foundation, which was created in 1985. The FSF was founded by Richard Stallman to support the GNU Project and the concept of Free Software: the idea that software should be free to use, free to copy it and free to redistribute it. However, the term Free is very restrictive. For the FSF, a Free Software must be under a copyleft license. Copyleft implies that all the work deriving from it must also be copyleft. We call this a “hereditary” or “contaminating” license, and we quickly understand that this kind of license could scare businesses.

Then there was the Open Source Initiative, a non-profit organization created in 1998. Eric S. Raymond and Bruce Perens wrote the Open Source Definition which frames less restrictive licenses (MIT, Apache, BSDs, etc.). These licenses all require to credit the author of the software you are using, but what you are building can have any other license. One of the goals of this Open Source Definition was to be more "business-friendly" because the term "Free" could scare companies, and by having a more friendly name and less restrictive licenses you can encourage companies to invest and participate in open source.

* * *

To illustrate the differences, we can take two OS based on Unix: On one side there is the Linux kernel which is licensed under the GPLv2 (copyleft), and this is why Linux-based Android is open source. On the other hand, there is Berkely Software Distribution (BSD) which is licensed under the BSD license, and its most famous fork are iOS and Mac OS which are proprietary and private.

These are today's definitions of open source, and the problems are:

*   Everyone does not agree on one definition because everyone has its ideology of open source
*   Everyone does not play by the same rules: while some companies are releasing their GPL's licensed projects on some obscure FTP, others are releasing their code publicly on GitHub without an OSI approved license. What is open source to you?

Open source nowadays
--------------------

As you may have read in the last months, some "open source" projects changed their license for the same reasons. We think that [MongoDB explains it very well](https://www.mongodb.com/licensing/server-side-public-license/faq#what-we-sell):

> This is a time of incredible opportunity for open source projects, with the potential to foster a new wave of great open source server-side software. The reality, however, is that once an open source project becomes interesting, it is too easy for large cloud vendors to capture all the value but contribute nothing back to the community.

And they are not the only project worrying about this situation.

Redis, for example, [is licensing some of its Redis Modules under its own license](https://redislabs.com/blog/redis-labs-modules-license-changes/), the Redis Source Available License. They were previously using the Commons Clause over an Apache license. The Commons Clause adds some restriction on commercial distribution over an already existing license like the Apache one. But [they changed it for confusion reason](https://www.zdnet.com/article/redis-labs-drops-commons-clause-for-a-new-license/); people were not sure whether it was open source or not.

[Confluent changed its license too](https://www.confluent.io/blog/license-changes-confluent-platform), from Apache they choose to create the Confluent Community License because they were afraid of some big cloud companies too.

Open source and Meili
---------------------

The point is not about whether big companies should contribute more or differently to open source but that open source is evolving, and we need a new definition, a new term to gather around and know what we are talking about.

Maybe we need labels to define and frame the ideology we are supporting as a company? Perhaps we need a patent-style system which states that every code that you release as a company will be under the MIT License or any other license 3 years after you wrote it. This way you could monetize your code for a period while enabling public disclosure of your code?

At Meili, we do not have a clear vision on these issues yet, but we are thinking about it a lot. We want our code to be available to anyone, free to use it and modify it for your projects. If you're going to use it for some personal project, go for it, if you want to use it for your company you should too. We do not wish to any cloud provider to git clone our project and be a competitor on a SaaS offering. Because we believe that our project will thrive more if we can build a sustainable business on it.

Today [our code is still under the MIT license](https://github.com/meilisearch/MeiliDB/blob/master/LICENSE), but tomorrow we might change it for the Commons Clause or write our public license as MongoDB and Confluent did. No, it won't be open source as the OSI state it, but we believe that making a publicly available source code is still better than closed source, too bad there is not a clear term of it today.

Open source or not?

Today we are releasing [MeiliES](https://github.com/meilisearch/MeiliES), our event store which is fully implemented in Rust.

Event sourcing is a way to _"Capture all changes to an application state as a sequence of events"_ as [Martin Fowler describes it](https://martinfowler.com/eaaDev/EventSourcing.html). The first purpose of the server is to publish events of a stream to subscribed clients. A client can also specify from which event number it wants to read, therefore it is possible to recover from a crash by rebuilding a state with new events only.

Since [a message queue is not made for event-sourcing](https://medium.com/serialized-io/apache-kafka-is-not-for-event-sourcing-81735c3cf5c), there are few available applications for event sourcing: the main option being [EventStore](https://eventstore.org/), an event sourcing database made in C#.  
So we began using EventStore for our event sourcing needs and ran against some troubles:

*   Portability: EventStore is built with C# and uses mono, making it quite difficult to install and run on UNIX
*   Support: We asked for information about EventStore commercial support and never got any response
*   Stability: the Rust client is not production-ready, and despite making some merge-requests there was still a lot to do while many of the available features were not useful to us.

So we decided to implement our own event store to resolve these issues.

MeiliES is an _Event Sourcing database_ that uses the _RESP_ (REdis Serialization Protocol) to communicate. We use the Redis protocol to simplify clients implementation. The portability problematics are resolved by using a full Rust implementation (we are using [Sled as internal storage](https://github.com/spacejam/sled)).

Installation is as simple as doing a `cargo build --release`. If you do not want to clone the repository you can [download the binaries from the Azure pipeline](https://dev.azure.com/thomas0884/thomas/_build) or run it using docker `docker run -p 6480:6480 --rm getmeili/meilies:latest`.

If you have any questions regarding MeiliES, you can tweet [@meilisearch](https://twitter.com/meilisearch) or make an [issue on GitHub](https://github.com/meilisearch/MeiliES) 🎉

MeiliES - Event sourcing in Rust

It is never easy to start a blog, we never know what to say on the first blog post 😱 so I will keep it simple here.

We are Meili, a team of three developers. In our previous job we worked on building a search engine aiming for the best end-user experience.

The most common alternatives were Algolia or ElasticSearch. Where the first one is only available as a service, you hand over your data and could be expensive. The second one is overkill for a simple end-user search,  complicated to setup and configure.

So here came our idea to develop [an open source alternative](https://github.com/meilisearch/MeiliDB) to Algolia, easy to setup, lightweight and fast.

So...

...Welcome on our blog! 🎉

I expect we will use it as often as possible to talk about what we build, company news and various thoughts.

  
(I just hope I won't be the only reader of this blog 🤭)

Meilisearch latest news and company updates

Meilisearch: Beta release!

Search for Rust crates with Meili

SpO₂ the little dynamic monitoring tool

Open source or not?

MeiliES - Event sourcing in Rust

Starting our blog