# Performance issue with Matches list. Caching + incremental updates

Canonical: https://airs0urce.com/blog/performance-issue-matches-list-caching

![Median execution time of the getMyMatches method: response time going down and peaks eliminated after the fix](/static/images/articles/matches-median-graph.png)

## Background information

On project I'm working now we have users matches list.

We have page with tinder style like/dislike (we call it "swipe") interface.

When users swipes (likes/dislikes) any profile on the page, we should stop showing swiped user on Quick Match page and some other pages.

The Matches list:

1. is endless, we show all matching users there
2. is unique for each user
3. each user sorted by distance from you
4. we have to show a fresh list of users
5. filtered by matching age, your gender preference and prefered genders of another user.
6. filtered by maximal radius from you (or country borders user selected that option).

#4 is necessary because our marketing team plans advertising campaigns, in one hour we can get 1000 new users and they won't be able to see each other for a while if we keep the results in cache.

## What the problem

We got user who reported issue to customer support team, she said website loads slow for her.

I checked her account and saw she swiped ~35000 users and I also found many other users with big number of swiped profiles.

For those users MongoDB had to exclude 35000 user ids (with MongoDB `$nin`) which generated a huge db query which was slow to execute because of amount of User IDs it had to exclude.

And this query is something that we run often because

1. Matches page is where user lands when open website
2. When user scrolling Matches page we run that query again and again to load more records, same complex sorting, filters + excluding applies.

Filtering of swiped ids is `$nin` part of big query with all other distance sorting and filters. Real query was looking much more complex, but here I show only part of query where we excluded swiped users:

```
{
  ...
  $nin: [ObjectId('...'), ObjectId('...'),ObjectId('...'),ObjectId('...')]
}
```

## What was done

For users online in last 2 days we store up to 400 ids of matched users in separate collection "quick_match_list" in simple document with format like below:

```
{
  user_id: <user-id>,
  matched_user_id, <user-id>,
  distance: <km>
}
```

Why we store only for users active last 2 days: as we have 7 millions of users and for each of them we can have thousands of matches this list can grow to billions records which is not acceptable.

Same time we have ~15000 DAU which is in worst case for initial pre-calculation is 15000 x 400 = 6,000,000 simple records in "quick_match_list" (not all users have >= 400 matches).

Exclusion of swiped users happens only once when we generate the full matches list.

On Matches page we query this simple collection to get user ids in one query like:

```
db.users.find({_id: {$in: [<user_ids>]}}, {sort: {distance: -1}})
```

If user tries to see over 400 results we are pre-calculating 400 more users and join them to existing matches in "quick_match_list".

The list is pre-generated, sorted, there is not so big number or records, query runs very fast. We dont need any geo-filtering or applying other filters like exclusion of swiped users.

We re-generate full list of matches for certain user when:

- the user requests Matches and pre-generated list is empty
- the user changes Match Preferencess (age, distance), location or user's status changes (deleted/deactivated/hidden by any reason).

So, that is perfect, but how do we keep the list updated? If new user creates an account or somebody deletes account, somebody changes location, that would be not so good if we re-generate full list for all users around changed one (as the changed one can be in their lists).

To prevent full re-generation we have incremental updates of "quick_match_list" collection:

- When UserA swipes UserB and UserB should be hidden for UserA further, we delete UserB from "quick_match_list" without full list recalculation
- When UserX deletes account we just delete him from lists of all users in "quick_match_list" collection, i.e `quick_match_list.deleteMany({"matched_user_id": <userX>})`
- When UserA creates account on website we check users who potentially can get UserA in list and compare if each of them matches UserA, if yes - we add them to list of another user.

So, this solution of caching + incremental updates improved 95th percentile execution time of one of most heavy/frequent query we had.

As it happened before I don't have screenshots of graph with percentile, but have median execution time graph where it's visible that response time went down and we got rid of peaks (which were those users with big number of swipes).

The median execution time of getMyMatches method is on screenshot I posted at the beginning of the article.