Dmitry Eroshenko

From Data Loss to Point-In-Time Recovery: Lessons from a MongoDB Driver Bug

The Problem

A few years ago, the MongoDB driver for Node.js (https://npmjs.org/mongodb) wasn't as polished as it is now.

One of the issues that caused a serious problem for me was the update query:

db.user_views.update(<filter>, <update_statement>);

We had "one-time" scripts that we ran on the server to migrate data to a new format. In one of those scripts, I wanted to edit a single field across all documents in the "user_views" collection, but I forgot to add the $set operator in <update_statement>, so the query ended up looking like this:

db.user_views.update(
 {}, // no filter — affects all documents
 {} // update part — left empty by mistake
);

The problem with the driver was that instead of throwing an error — like the official mongosh (MongoDB's command-line client) would — it replaced every document with the empty object I had passed. The result: all user_views were wiped from the database. It was terrible to see what I had done.

But it wasn't a total disaster, because we had daily backups created with mongodump. I stopped the API, restored the user_views collection, and started the API again. The only data we lost were user_view documents created in the last 10 hours before the incident.

What Was Done After That

  1. I checked the MongoDB documentation and confirmed that the driver was not behaving as expected.

    According to the docs, a query like this — without any update operators in the update part — should not be executed:

    db.user_views.update({...}, {});

    However, this was only enforced by the command-line utility, not by the Node.js driver.

    I created a ticket in MongoDB's Jira, described the problem, and it was later fixed by the MongoDB team.

  2. Since we had lost user_views from the last few hours, I needed to take steps to prevent this from happening again:

    • I forked the MongoDB driver locally and added a validation check for this case, so we could use it as a safeguard until the fix landed in the official driver.
    • I researched our options for incremental backups and found Percona Backup for MongoDB. I spent some time setting it up and wrote detailed documentation so that our team could follow clear instructions and act without delays if something like this ever happened again.

    Note: Before sharing this document, I removed everything related to our infrastructure: replaced real server addresses with <SERVER_ADDRESS_HERE> and changed usernames and paths.

    https://drive.google.com/file/d/1kMw5TEr8rQuxxQFNTY6Xsb8BCWiNuybY/view

What We Gained

  1. Incremental backups allowed us to significantly reduce backup storage, since we only run full backups every three days and use incremental backups in between.
  2. It became much easier to restore the most recent data if anything goes wrong with the database.
  3. These backups also turned out to be useful for our marketing team. They would occasionally need to check things like "what changed in the configuration of Feature X two weeks ago?" — they would compare subscription statistics and sometimes didn't have a log of the configuration changes they had made in our admin panel, but wanted to understand what had led to shifts in subscription numbers.

← Back to blog