Go to homev1.11 Docs

    Implementing semantic search with LangChain

    In this guide, you’ll use OpenAI’s text embeddings to measure the similarity between document properties. Then, you’ll use the LangChain framework to seamlessly integrate Meilisearch and create an application with semantic search.

    Requirements

    This guide assumes a basic understanding of Python and LangChain. Beginners to LangChain will still find the tutorial accessible.

    On Meilisearch Cloud, enable the vector store feature via your project’s Settings page.

    A section of the project overview interface titled "Experimental features". There are two options: "Score details" and "Vector store". "Vector store" is turned on.

    If you are working with a self-hosted Meilisearch instance, activate the vector store with the API route.

    Creating the application

    Create a folder for your application with an empty setup.py file.

    Before writing any code, install the necessary dependencies:

    pip install langchain openai meilisearch python-dotenv

    First create a .env to store our credentials:

    # .env

MEILI_HTTP_ADDR="your Meilisearch host"
MEILI_API_KEY="your Meilisearch API key"
OPENAI_API_KEY="your OpenAI API key"

    Now that you have your environment variables available, create a setup.py file with some boilerplate code:

    # setup.py

import os
from dotenv import load_dotenv # remove if not using dotenv
from langchain.vectorstores import Meilisearch
from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.document_loaders import JSONLoader

load_dotenv() # remove if not using dotenv

# exit if missing env vars
if "MEILI_HTTP_ADDR" not in os.environ:
    raise Exception("Missing MEILI_HTTP_ADDR env var")
if "MEILI_API_KEY" not in os.environ:
    raise Exception("Missing MEILI_API_KEY env var")
if "OPENAI_API_KEY" not in os.environ:
    raise Exception("Missing OPENAI_API_KEY env var")

# Setup code will go here 👇

    Importing documents and embeddings

    Now that the project is ready, import some documents in Meilisearch. First, download this small movies dataset:

    movies-lite.json

    Then, update the setup.py file to load the JSON and store it in Meilisearch. You will also use the OpenAI text search models to generate vector embeddings.

    To use vector search, we need to set the embedders index setting. In this case, you are using an userProvided source which requires to specify the size of the vectors in a dimensions field. The default model used by OpenAIEmbeddings() is text-embedding-ada-002, which has 1,536 dimensions.

    # setup.py

# previous code

# Load documents
loader = JSONLoader(
    file_path="./movies-lite.json",
    jq_schema=".[] | {id: .id, overview: .overview, title: .title}",
    text_content=False,
)
documents = loader.load()
print("Loaded {} documents".format(len(documents)))

# Store documents in Meilisearch
embeddings = OpenAIEmbeddings()
embedders = { 
        "custom": {
            "source": "userProvided",
            "dimensions": 1536
        }
    }
embedder_name = "custom" 
vector_store = Meilisearch.from_documents(documents=documents, embedding=embeddings, embedders=embedders, embedder_name=embedder_name)

print("Started importing documents")

    Your Meilisearch instance will now contain your documents. Meilisearch runs tasks like document import asynchronously, so you might need to wait a bit for documents to be available. Consult the asynchronous operations explanation for more information on how tasks work.

    Your database is now populated with the data from the movies dataset. Create a new search.py file to make a semantic search query: searching for documents using similarity search.

    # search.py

import os
from dotenv import load_dotenv
from langchain.vectorstores import Meilisearch
from langchain.embeddings.openai import OpenAIEmbeddings
import meilisearch

load_dotenv()

# You can use the same code as `setup.py` to check for missing env vars

# Create the vector store
client = meilisearch.Client(
    url=os.environ.get("MEILI_HTTP_ADDR"),
    api_key=os.environ.get("MEILI_API_KEY"),
)
embeddings = OpenAIEmbeddings()
vector_store = Meilisearch(client=client, embedding=embeddings)

# Make similarity search
embedder_name = "custom"
query = "superhero fighting evil in a city at night"
results = vector_store.similarity_search(
    query=query,
    embedder_name=embedder_name,
    k=3,
)

# Display results
for result in results:
    print(result.page_content)

    Run search.py. If everything is working correctly, you should see an output like this:

    {"id": 155, "title": "The Dark Knight", "overview": "Batman raises the stakes in his war on crime. With the help of Lt. Jim Gordon and District Attorney Harvey Dent, Batman sets out to dismantle the remaining criminal organizations that plague the streets. The partnership proves to be effective, but they soon find themselves prey to a reign of chaos unleashed by a rising criminal mastermind known to the terrified citizens of Gotham as the Joker."}
{"id": 314, "title": "Catwoman", "overview": "Liquidated after discovering a corporate conspiracy, mild-mannered graphic artist Patience Phillips washes up on an island, where she's resurrected and endowed with the prowess of a cat -- and she's eager to use her new skills ... as a vigilante. Before you can say \"cat and mouse,\" handsome gumshoe Tom Lone is on her tail."}
{"id": 268, "title": "Batman", "overview": "Batman must face his most ruthless nemesis when a deformed madman calling himself \"The Joker\" seizes control of Gotham's criminal underworld."}

    Congrats 🎉 You managed to make a similarity search using Meilisearch as a LangChain vector store.

    Going further

    Using Meilisearch as a LangChain vector store allows you to load documents and search for them in different ways:

    For additional information, consult:

    Meilisearch Python SDK docs

    Finally, should you want to use Meilisearch's vector search capabilities without LangChain or its hybrid search feature, refer to the dedicated tutorial.