Simple and efficient MongoDB Backup using script

by jagbir on May 9, 2012

MongoDB Backup types and strategies are neatly explained in its documentation, which you can check here. In case you are not familiar with MongoDB backup types and strategies, please have a look at its documentation.

What I am describing here is a simple script which we are using since months to take MongoDB backup and transfer it over to our Backup server. Here are few things its doing:

  • As we have multiple MongoDB Replica Sets, the script identify current replica set and check whether current server is Master or Slave, exit if its Master. We take backup only from Slave host.
  • Take Backup using mongodump command.
  • Upon successful completion of dump, transfer that to our Backup server. Ensure that ssh key based authentication is setup between both servers to implement seamless and secure transfer. It creates new directory based on current timestamp under replicaset directory in specified path at Backup server and transfer dump there.
  • Log each steps described above, send alert mail if any step fail with description or send confirmation mail upon successful execution with essential details.
  • Sample confirmation mail is below:
Subject: Backup for  done on  at 09/May/12 08:00:01 AM
Body: Mongo Backup Status for  on 09/May/12 08:00:01 AM. 
 
08:00:01 AM:  is slave and looks OK.
08:00:01 AM: Starting Dump, executing /usr/local/mongodb/mongo/bin/mongodump --out /databases/dump --host ...
08:34:06 AM: Mongodump command completed. Backup size is 25G. 
08:34:10 AM: Directory created on backup server, copying data using scp...
08:57:00 AM: Copied dump to backup server in directory /home/backup/Mongodump/replicaset/datestamp/.
08:57:00 AM: Mongo Backup process completed successfully.

Here is the full bash script, please note that you need to update variables properly and have to check and update it to run in your environment which can be entirely different from mine. Its just an idea to automate MongoDB backup:

##
# Script to take mongo backup using mongodump and store it in Backup Server
##
 
#!/bin/bash
 
## Set variables
TodayDate=`date +"%d/%b/%g %r"`
DateStamp=`date +%d%m%y%H%M%S`
CurrentTime=`date +"%r"`
 
MongoBinPath="/usr/local/mongodb/mongo/bin"
ReplicaSet=`echo 'rs.status()' | $MongoBinPath/mongo | egrep "set" | awk -F \" '{print $4}'`
MongoHost=`hostname`
LocalBackupPath="/databases/dump"
 
LogFile="/var/log/mongo-backup.log"
IsOK=0
CmdStatus=""
 
BackupHost=xx.xx.xx.xx
BackupHostPath="/home/backup/mongobackup"
BackupHostPort=22
 
MailNotification="admin@domain.com anotheradmin@domain.com"
 
echo -e "Mongo Backup Status for $ReplicaSet on $TodayDate. \n" > $LogFile
 
## Check whether host is slave and in good state for backup
for i in `echo "rs.status()" | $MongoBinPath/mongo | egrep "name" | awk -F \" '{print $4}'| cut -f 1 -d :`; do
 IsMaster=`echo "db.isMaster()"| $MongoBinPath/mongo --host $i | grep ismaster|awk -F ":" '{print $2}' | cut -f 1 -d ,`;
 TheState=`echo "rs.status()"| $MongoBinPath/mongo --host $i | grep -i mystate | awk -F ":" '{print $2}' | cut -f 1 -d ,`;
 if [ $IsMaster == "false" -a $TheState -eq 2  ]; then
  MongoHost=$i
  IsOK=1
  echo "$CurrentTime: $MongoHost is slave and looks OK." >> $LogFile
  break
 fi
done
 
CurrentTime=`date +"%r"`
 
## Exit if not good
if [ $IsOK -eq 0 ]; then
   echo "$CurrentTime: Error: Either $MongoHost is not slave or not in good state. Aborting Backup, Please check!" >> $LogFile
   mail -s "Backup error for $ReplicaSet from $MongoHost on $TodayDate" $MailNotification < $LogFile    exit 1; fi ## Remove earlier backup CmdStatus=$(rm -rf $LocalBackupPath/*) ## Start backup process echo "$CurrentTime: Starting Dump, executing $MongoBinPath/mongodump --out $LocalBackupPath --host $MongoHost..." >> $LogFile
 
CmdStatus=`$MongoBinPath/mongodump --out $LocalBackupPath --host $MongoHost`
if [ $? -ne 0 ]; then
  echo "$CurrentTime: There is an issue while trying to take dump in $MongoHost. Aborting dump process, please check! " >> $LogFile
  cat $CmdStatus >> $LogFile
  mail -s "Backup error for $ReplicaSet from $MongoHost on $TodayDate" $MailNotification < $LogFile   exit fi CurrentTime=`date +"%r"` BackupSize=$(du -sh $LocalBackupPath/. | awk '{ print $1 }') echo "$CurrentTime: Mongodump command completed. Backup size is $BackupSize. " >> $LogFile
 
## dump is fine then scp it to backup server
## create directory first
CmdStatus=$(ssh -p $BackupHostPort $BackupHost "mkdir -p  $BackupHostPath/$ReplicaSet/$DateStamp")
 
if [ $? -ne 0 ]; then
  echo "$CurrentTime: Either failing to connect Backup server using ssh or destination directory already exist!" >> $LogFile
  cat $CmdStatus >> $LogFile
  mail -s "Backup error for $ReplicaSet from $MongoHost on $TodayDate" $MailNotification < $LogFile   exit fi CurrentTime=`date +"%r"` echo "$CurrentTime: Directory created on backup server, copying data using scp..." >> $LogFile
 
CmdStatus=`scp -P $BackupHostPort -r $LocalBackupPath/* $BackupHost:/$BackupHostPath/$ReplicaSet/$DateStamp/`
if [ $? -ne 0 ]; then
  echo "$CurrentTime: Unable to scp dump to $BackupHost:/$BackupHostPath/$ReplicaSet/$DateStamp using port $BackupHostPort. " >> $LogFile
  cat $CmdStatus >> $LogFile
  mail -s "Backup error for $ReplicaSet from $MongoHost on $TodayDate" $MailNotification < $LogFile   exit fi CurrentTime=`date +"%r"` echo "$CurrentTime: Copied dump to backup server in directory $BackupHost$BackupHostPath/$ReplicaSet/$DateStamp/." >> $LogFile
echo "$CurrentTime: Mongo Backup process completed successfully." >> $LogFile
mail -s "Backup for $ReplicaSet done on $MongoHost at $TodayDate" $MailRecipients $MailNotification < $LogFile

Its just a basic script and may needs further enhancements. In case you have suggestion/queries, please put it below in comments.

More Related and helpful articles:

 

Tagged as: bash, mongodb, mongodump, nosql, script

Previous post:

Next post: