07 Sept 2023

How to boost your search results with filter scoring

Assign weights to your filters and prioritize documents based on how well they match your criteria.

Carolina FerreiraDeveloper Advocate @ Meilisearch@CarolainFG

How to boost your search results with filter scoring

In this guide, you’ll discover how to implement a filter scoring feature to enhance your search functionality within Meilisearch.

What is filter boosting?

Filter boosting, also referred to as filter scoring, is an advanced search optimization strategy designed to enhance the relevance and precision of returned documents. Instead of simply returning documents that match a single filter, this method uses a weighted system for multiple filters. The documents that align with the most filters—or those that match the most heavily-weighted filters—are given priority and returned at the top of the search results.

Generating filter boosted queries

Meilisearch allows users to refine their search queries by adding filters. Traditionally, only documents that precisely match these filters are returned in the search results.

With the implementation of filter boosting, you can optimize the document retrieval process by ranking documents based on the relevancy of multiple, weighted filters. This ensures a more tailored and effective search experience.

The idea behind this implementation is to associate a weight to each filter. The higher the value, the more important the filter should be. In this section, we’ll demonstrate how to implement a search algorithm that makes use of these weighted filters.

Step 1 — Setting up and prioritizing filters: weights assignment

To leverage the filter scoring feature, you’ll need to provide a list of filters along with their respective weights. This helps prioritize the search results according to the criteria that matter most to you.

Example input using JavaScript:

const filtersWeights = [
    { filter: "genres = Animation", weight: 3 },
    { filter: "genres = Family", weight: 1 },
    { filter: "release_date > 1609510226", weight: 10 }
]

In the example above:

The highest weight is assigned to the release date, indicating a preference for movies released after 2021
Movies in the “Animation” genre get the next level of preference
“Family” genre movies also receive a minor boost

Step 2. Combining filters

The goal is to create a list of all filter combinations, where each combination would be associated with its total weight.

Taking the previous example as a reference, the generated queries with their total weights would be as follows:

("genres = Animation AND genres = Family AND release_date > 1609510226", 14)
("genres = Animation AND NOT(genres = Family) AND release_date > 1609510226", 13)
("NOT(genres = Animation) AND genres = Family AND release_date > 1609510226", 11)
("NOT(genres = Animation) AND NOT(genres = Family) AND release_date > 1609510226", 10)
("genres = Animation AND genres = Family AND NOT(release_date > 1609510226)", 4)
("genres = Animation AND NOT(genres = Family) AND NOT(release_date > 1609510226)", 3)
("NOT(genres = Animation) AND genres = Family AND NOT(release_date > 1609510226)", 1)
("NOT(genres = Animation) AND NOT(genres = Family) AND NOT(release_date > 1609510226)", 0)

We can see, when filters match Criteria 1 + Criteria 2 + Criteria 3, the total weight is weight1 + weight2 + weight3 ( 3 + 1 +10 = 14).

Below, we'll explain how to build this list. For details on automating this process, refer to the Filter combination algorithm section.

Then, you can use Meilisearch’s multi-search API to perform queries based on these filters, arranging them in descending order according to their assigned weight.

Step 3. Using Meilisearch’s multi-search API

Don’t forget to install the Meilisearch JavaScript client, first:

npm install meilisearch
\\ or
yarn add meilisearch

const { MeiliSearch } = require('meilisearch')
// Or if you are in a ES environment
import { MeiliSearch } from 'meilisearch'

;(async () => {
    // Setup Meilisearch client
    const client = new MeiliSearch({
        host: 'http://localhost:7700',
        apiKey: 'apiKey',
    })
    
    const INDEX = "movies"
    const limit = 20
    
    const queries = [
        { indexUid: INDEX, limit: limit, filter: 'genres = Animation AND genres = Family AND release_date > 1609510226' },
        { indexUid: INDEX, limit: limit, filter: 'genres = Animation AND NOT(genres = Family) AND release_date > 1609510226' },
        { indexUid: INDEX, limit: limit, filter: 'NOT(genres = Animation) AND genres = Family AND release_date > 1609510226' },
        { indexUid: INDEX, limit: limit, filter: 'NOT(genres = Animation) AND NOT(genres = Family) AND release_date > 1609510226' },
        { indexUid: INDEX, limit: limit, filter: 'genres = Animation AND genres = Family AND NOT(release_date > 1609510226)' },
        { indexUid: INDEX, limit: limit, filter: 'genres = Animation AND NOT(genres = Family) AND NOT(release_date > 1609510226)' },
        { indexUid: INDEX, limit: limit, filter: 'NOT(genres = Animation) AND genres = Family AND NOT(release_date > 1609510226)' },
        { indexUid: INDEX, limit: limit, filter: 'NOT(genres = Animation) AND NOT(genres = Family) AND NOT(release_date > 1609510226)' }
    ]
    
    try {
        const results = await client.multiSearch({ queries });
        displayResults(results);
    } catch (error) {
        console.error("Error while fetching search results:", error);
    }
    
    function displayResults(data) {
        let i = 0;
        console.log("=== best filter ===");
        
        for (const resultsPerIndex of data.results) {
            for (const result of resultsPerIndex.hits) {
                if (i >= limit) {
                    break;
                }
                console.log(`${i.toString().padStart(3, '0')}: ${result.title}`);
                i++;
            }
            console.log("=== changing filter ===");
        }
    }
    
})();

We begin by importing the required libraries for our task. Then we initialize the Meilisearch client, which connects to our Meilisearch server, and defines the movie index we’ll be searching.

Next, we send our search criteria to the Meilisearch server and retrieve the results. The multiSearch function lets us send multiple search queries at once, which can be more efficient than sending them one by one.

Finally, we print out the results in a formatted manner. The outer loop iterates through the results of each filter. The inner loop iterates through the hits (actual search results) for a given filter. We print each movie title with a number prefix.

We get the following output:

=== best filter ===
000: Blazing Samurai
001: Minions: The Rise of Gru
002: Sing 2
003: The Boss Baby: Family Business
=== changing filter ===
004: Evangelion: 3.0+1.0 Thrice Upon a Time
005: Vivo
=== changing filter ===
006: Space Jam: A New Legacy
007: Jungle Cruise
=== changing filter ===
008: Avatar 2
009: The Flash
010: Uncharted
...
=== changing filter ===

Filter combination algorithm

While the manual filtering approach provides accurate results, it isn’t the most efficient method. Automating this process will significantly enhance both speed and efficiency. Let’s create a function that takes query parameters and a list of weighted filters as inputs and outputs a list of search hits.

Utility functions: the building blocks of filter manipulation

Before diving into the core function, it’s essential to create some utility functions to handle filter manipulation.

Negating filters

The negateFilter function, returns the opposite of a given filter. For example, if provided with genres = Animation, it would return NOT(genres = Animation).

function negateFilter(filter) {
  return `NOT(${filter})`;
}

Aggregating filters

The aggregateFilters function combines two filter strings with an “AND” operation. For instance, if given genres = Animation and release_date > 1609510226, it would return (genres = Animation) AND (release_date > 1609510226).

function aggregateFilters(left, right) {
  if (left === "") {
    return right;
  }
  if (right === "") {
    return left;
  }
  return `(${left}) AND (${right})`;
}

Generating combinations

The getCombinations function generates all possible combinations of a specified size from an input array. This is crucial for creating different sets of filter combinations based on their assigned weights.

function getCombinations(array, size) {
    const result = [];
    
    function generateCombination(prefix, remaining, size) {
        if (size === 0) {
            result.push(prefix);
            return;
        }
        
        for (let i = 0; i < remaining.length; i++) {
            const newPrefix = prefix.concat([remaining[i]]);
            const newRemaining = remaining.slice(i + 1);
            generateCombination(newPrefix, newRemaining, size - 1);
        }
    }
    
    generateCombination([], array, size);
    return result;
}

The core function: boostFilter

Now that we have our utility functions, we can now move on to generating filter combinations in a more dynamic fashion, according to their assigned weights. This is achieved with the boostFilter function, it combines and sorts filters based on their respective weights.

function boostFilter(filterWeights) {
    const totalWeight = filterWeights.reduce((sum, { weight }) => sum + weight, 0);
    const weightScores = {};
    
    const indexes = filterWeights.map((_, idx) => idx);
    
    for (let i = 1; i <= filterWeights.length; i++) {
        const combinations = getCombinations(indexes, i);
        
        for (const filterIndexes of combinations) {
            const combinationWeight = filterIndexes.reduce((sum, idx) => sum + filterWeights[idx].weight, 0);
            weightScores[filterIndexes] = combinationWeight / totalWeight;
        }
    }
    
    const filterScores = [];
    for (const [filterIndexes, score] of Object.entries(weightScores)) {
        let aggregatedFilter = "";
        const indexesArray = filterIndexes.split(",").map(idx => parseInt(idx));
        
        for (let i = 0; i < filterWeights.length; i++) {
            if (indexesArray.includes(i)) {
                aggregatedFilter = aggregateFilters(aggregatedFilter, filterWeights[i].filter);
            } else {
                aggregatedFilter = aggregateFilters(aggregatedFilter, negateFilter(filterWeights[i].filter));
            }
        }
        filterScores.push([aggregatedFilter, score]);
    }
    
    filterScores.sort((a, b) => b[1] - a[1]);
    return filterScores;
}

Breaking down the boostFilter function

Let’s dissect the function to better understand its components and operations.

1. Calculate total weight

The function begins by calculating the totalWeight, which is simply the sum of all the weights in the filterWeights array.

const totalWeight = filterWeights.reduce((sum, { weight }) => sum + weight, 0);

2. Create weight and indexes structures

Two essential structures are initialized here:

weightScores: holds combinations of filters and their associated relative scores
indexes: an array that maps each filter to its position in the original filterWeights array

const weightScores = {};
    
const indexes = filterWeights.map((_, idx) => idx);

3. Computation of weighted filter combinations

For each combination, we calculate its weight and store its relative score in the weightScores object.

for (let i = 1; i <= filterWeights.length; i++) {
    const combinations = getCombinations(indexes, i);
    
    for (const filterIndexes of combinations) {
        const combinationWeight = filterIndexes.reduce((sum, idx) => sum + filterWeights[idx].weight, 0);
        weightScores[filterIndexes] = combinationWeight / totalWeight;
    }
}

4. Aggregate and negate filters

Here, we form the aggregated filter string. Each combination from weightScores is processed and populated into the filterScores list, along with its relative score.

const filterScores = [];
for (const [filterIndexes, score] of Object.entries(weightScores)) {
    let aggregatedFilter = "";
    const indexesArray = filterIndexes.split(",").map(idx => parseInt(idx));
    
    for (let i = 0; i < filterWeights.length; i++) {
        if (indexesArray.includes(i)) {
            aggregatedFilter = aggregateFilters(aggregatedFilter, filterWeights[i].filter);
        } else {
            aggregatedFilter = aggregateFilters(aggregatedFilter, negateFilter(filterWeights[i].filter));
        }
    }
    filterScores.push([aggregatedFilter, score]);
}

5. Sort and return filter scores

Finally, the filterScores list is sorted in descending order based on scores. This ensures the most “important” filters (as determined by weight) are at the beginning.

filterScores.sort((a, b) => b[1] - a[1]);
return filterScores;

Using the filter boosting function

Now that we have the boostFilter function, we can demonstrate its efficacy on an example. This function returns an array of arrays, where each inner array contains:

A combined filter based on the input criteria
A score indicating the weighted importance of the filter

When we apply our function to an example:

boostFilter([["genres = Animation", 3], ["genres = Family", 1], ["release_date > 1609510226", 10]])

We receive the following output:

[
    [
      '((genres = Animation) AND (genres = Family)) AND (release_date > 1609510226)',
      1
    ],
    [
      '((genres = Animation) AND (NOT(genres = Family))) AND (release_date > 1609510226)',
      0.9285714285714286
    ],
    [
      '((NOT(genres = Animation)) AND (genres = Family)) AND (release_date > 1609510226)',
      0.7857142857142857
    ],
    [
      '((NOT(genres = Animation)) AND (NOT(genres = Family))) AND (release_date > 1609510226)',
      0.7142857142857143
    ],
    [
      '((genres = Animation) AND (genres = Family)) AND (NOT(release_date > 1609510226))',
      0.2857142857142857
    ],
    [
      '((genres = Animation) AND (NOT(genres = Family))) AND (NOT(release_date > 1609510226))',
      0.21428571428571427
    ],
    [
      '((NOT(genres = Animation)) AND (genres = Family)) AND (NOT(release_date > 1609510226))',
      0.07142857142857142
    ]
]

Generating search queries from boosted filters

Now that we have a prioritized list of filters from the boostFilter function, we can use it to generate search queries.Let’s create a searchBoostFilter function to automate the generation of search queries based on the boosted filters and execute the search queries using the provided Meilisearch client.

async function searchBoostFilter(client, filterScores, indexUid, q) {
    const searchQueries = filterScores.map(([filter, _]) => {
        const query = { ...q };
        query.indexUid = indexUid;
        query.filter = filter;
        return query;
    });
    
    const results = await client.multiSearch({ queries: searchQueries });
    return results;
}

The function takes the following parameters

client: the Meilisearch client instance.
filterScores: array of arrays of filters and their corresponding scores.
indexUid: the index you want to search within
q: base query parameters

For each filter in filterScores, we:

create a copy of the base query parameters q using the spread operator
update the indexUid and filter values for the current search query
add the modified query to our searchQueries array

The function then returns the raw results from the multi search route.

Example: extracting top movies using filter scores

Let’s create a function to display the top movie titles that fit within our defined search limits and based on our prioritized filter criteria: the bestMoviesFromFilters function.

async function bestMoviesFromFilters(client, filterWeights, indexUid, q) {
    
    const filterScores = boostFilter(filterWeights);
    const results = await searchBoostFilter(client, filterScores, indexUid, q);
    const limit = results.results[0].limit;
    let hitIndex = 0;
    let filterIndex = 0;
    
    for (const resultsPerIndex of results.results) {
        if (hitIndex >= limit) {
            break;
        }
        
        const [filter, score] = filterScores[filterIndex];
        console.log(`=== filter '${filter}' | score = ${score} ===`);
        
        for (const result of resultsPerIndex.hits) {
            if (hitIndex >= limit) {
                break;
            }
            
            console.log(`${String(hitIndex).padStart(3, '0')}: ${result.title}`);
            hitIndex++;
        }
        
        filterIndex++;
    }
}

The function uses the boostFilter function to get the list of filter combinations and their scores.

Then, the searchBoostFilter function obtains the results for the provided filters.
It also determines the maximum number of movie titles we wish to display based on the limit set in our base query.

Using a loop, the function iterates through the results:

If the current count of displayed movie titles (hitIndex) reaches the specified limit, the function stops processing further.
For each set of results from the multi-search query, the function displays the applied filter condition and its score.
It then goes through the search results (or hits) and displays the movie titles until the limit is reached or all results for the current filter are displayed.
The process continues for the next set of results with a different filter combination until the overall limit is reached or all results are displayed.

Let’s use our new function in an example:

bestMoviesFromFilters(client, 
    [
        { filter: "genres = Animation", weight: 3 }, 
        { filter: "genres = Family", weight: 1 }, 
        { filter: "release_date > 1609510226", weight: 10 }
    ],
    "movies", 
    { q: "Samurai", limit: 100 }
)

We get the following output:

=== filter '((genres = Animation) AND (genres = Family)) AND (release_date > 1609510226)' | score = 1.0 ===
000: Blazing Samurai
=== filter '((genres = Animation) AND (NOT(genres = Family))) AND (release_date > 1609510226)' | score = 0.9285714285714286 ===
=== filter '((NOT(genres = Animation)) AND (genres = Family)) AND (release_date > 1609510226)' | score = 0.7857142857142857 ===
=== filter '((NOT(genres = Animation)) AND (NOT(genres = Family))) AND (release_date > 1609510226)' | score = 0.7142857142857143 ===
=== filter '((genres = Animation) AND (genres = Family)) AND (NOT(release_date > 1609510226))' | score = 0.2857142857142857 ===
001: Scooby-Doo! and the Samurai Sword
002: Kubo and the Two Strings
=== filter '((genres = Animation) AND (NOT(genres = Family))) AND (NOT(release_date > 1609510226))' | score = 0.21428571428571427 ===
003: Samurai Jack: The Premiere Movie
004: Afro Samurai: Resurrection
005: Program
006: Lupin the Third: Goemon's Blood Spray
007: Hellboy Animated: Sword of Storms
008: Gintama: The Movie
009: Heaven's Lost Property the Movie: The Angeloid of Clockwork
010: Heaven's Lost Property Final – The Movie: Eternally My Master
=== filter '((NOT(genres = Animation)) AND (genres = Family)) AND (NOT(release_date > 1609510226))' | score = 0.07142857142857142 ===
011: Teenage Mutant Ninja Turtles III

Conclusion

In this guide, we walked through the process of implementing a scored filtering feature. We learned how to set up weighted filters and automatically generate filter combinations, which we then scored based on their weight. Following that, we explored how to create search queries using these boosted filters with the help of Meilisearch's multi-search API.

We plan to integrate scored filters in the Meilisearch engine. Give your feedback on the previous link to help us prioritize it.

For more things Meilisearch, you can subscribe to our newsletter. You can learn more about our product by checking out the roadmap and participating in our product discussions.

For anything else, join our developers community on Discord.

Using Meilisearch AI

How to add AI-powered search to a React app

Build a React movie search and recommendation app with Meilisearch's AI-powered search.

Carolina Ferreira24 Sept 2024

Using Meilisearch

Build your Next.js Shopify storefront with Blazity

Learn to build a Shopify storefront with Next.js and the Blazity commerce starter.

Laurent Cazanove19 Aug 2024

Release Using Meilisearch

Meilisearch 1.8

Meilisearch 1.8 brings negative keyword search, improvements in search robustness and AI search, including new embedders.

Carolina Ferreira07 May 2024