First of all I can tell you that we will explain it for an architecture with two mongodb nodes in sharding, one of them with the role of router and the other with the role of config server.

In the tests that have been done with mongodump and mongorestore commands, the restoration  of mongod data files wasn’t  satisfactory and  didn’t ensure the integrity of the data because in some cases sharding configuration was loosed and in another cases some data was corrupted even using –oplog option which allow to write the most recent written data while backup is being done. In a few tests restoration crashed when  there weere 2.000.000 of entries in database.

Mongodb version used in these tests was 2.0. Filesystem used in data partitions was ext4  above LVM.

The best and most reliable way to  backing up mongod data files in sharding (but you have to remember it is a complete backup) is doing a LVM  snapshot, is blocking  write operations  and then making  a snapshot backup in a remote directory. Doing this  you only block database write operations during  LVM snapshot is being made.

This is the diagram of the architecture:

In each server we have to have the following partitions:

  • 1 lvm with  root partition (/) mounted
  • 1 lvm with mongodb data partition mounted (/var/lib/mongodb)
  • Free space for making  LVM snapshot (at least the same size as the LVM data partition)

If you want to know how resize a LVM you can check  this post  Resizing Logical Volumes in Linux (LVM).

Once you  meet this requirements we will be able to connect to mongodb console:

#mongo -port 30000

and block write operations on the database:

mongo1$ use admin
mongo1$ db.runCommand({fsync:1,lock:1})

then we have to clone the  LVM volume where we keep mongod data:

mongo1$  lvcreate -L40G -s -n mongosnapshot /dev/mapper/lvmcondatosmnongo

the next step is unblock write operations on the database:

mongo1$ db.$cmd.sys.unlock.findOne()

and finally we can make the backup on this LVM clone mounting it in a directory of our  choice:

#mkdir -p /mnt/backup/mongonapshot

#mount /dev/mapper/mongosnapshot /mnt/backup/mongosnapshot

In this step you have to choose between making a raw backup or a filesystem backup. I recommend you making  a filesystem backup because you will be able to restore the backup faster and optimize the disk space. If you make a raw backup you copy empty blocks too.

Filesystem backup: #tar -pczf /backups/mongo.tar.gz /mnt/backup/mongosnapshot

Raw backup: #dd if=/mnt/backup/mongosnapshot of=/backups/mongo.dd

We already have  the backup!

Final step is restore the mongod data files so we have to stop all services in each node and uncompress the backup:

# /etc/init.d/mongodb stop
#tar zxvf  backups/mongo.tar.gz /var/lib/

or restore the clone image:

#dd if=/backups/mongo.dd of=/var/lib/mongodb

References:

http://www.howtoforge.com/linux_lvm_snapshots_p1

http://www.mongodb.org/display/DOCS/Backups

http://equivocation.org/node/103

http://www.tcpdump.com/kb/os/linux/lvm-resizing-guide/shrink.html

Advertisements