Linux Admin Zone » database

Simple and efficient MongoDB Backup using script

jagbir — Wed, 09 May 2012 10:17:54 +0000

MongoDB Backup types and strategies are neatly explained in its documentation, which you can check here. In case you are not familiar with MongoDB backup types and strategies, please have a look at its documentation.

What I am describing here is a simple script which we are using since months to take MongoDB backup and transfer it over to our Backup server. Here are few things its doing:

As we have multiple MongoDB Replica Sets, the script identify current replica set and check whether current server is Master or Slave, exit if its Master. We take backup only from Slave host.
Take Backup using mongodump command.
Upon successful completion of dump, transfer that to our Backup server. Ensure that ssh key based authentication is setup between both servers to implement seamless and secure transfer. It creates new directory based on current timestamp under replicaset directory in specified path at Backup server and transfer dump there.
Log each steps described above, send alert mail if any step fail with description or send confirmation mail upon successful execution with essential details.
Sample confirmation mail is below:

Subject: Backup for  done on  at 09/May/12 08:00:01 AM
Body: Mongo Backup Status for  on 09/May/12 08:00:01 AM. 
 
08:00:01 AM:  is slave and looks OK.
08:00:01 AM: Starting Dump, executing /usr/local/mongodb/mongo/bin/mongodump --out /databases/dump --host ...
08:34:06 AM: Mongodump command completed. Backup size is 25G. 
08:34:10 AM: Directory created on backup server, copying data using scp...
08:57:00 AM: Copied dump to backup server in directory /home/backup/Mongodump/replicaset/datestamp/.
08:57:00 AM: Mongo Backup process completed successfully.

Here is the full bash script, please note that you need to update variables properly and have to check and update it to run in your environment which can be entirely different from mine. Its just an idea to automate MongoDB backup:

##
# Script to take mongo backup using mongodump and store it in Backup Server
##
 
#!/bin/bash
 
## Set variables
TodayDate=`date +"%d/%b/%g %r"`
DateStamp=`date +%d%m%y%H%M%S`
CurrentTime=`date +"%r"`
 
MongoBinPath="/usr/local/mongodb/mongo/bin"
ReplicaSet=`echo 'rs.status()' | $MongoBinPath/mongo | egrep "set" | awk -F \" '{print $4}'`
MongoHost=`hostname`
LocalBackupPath="/databases/dump"
 
LogFile="/var/log/mongo-backup.log"
IsOK=0
CmdStatus=""
 
BackupHost=xx.xx.xx.xx
BackupHostPath="/home/backup/mongobackup"
BackupHostPort=22
 
MailNotification="admin@domain.com anotheradmin@domain.com"
 
echo -e "Mongo Backup Status for $ReplicaSet on $TodayDate. \n" &gt; $LogFile
 
## Check whether host is slave and in good state for backup
for i in `echo "rs.status()" | $MongoBinPath/mongo | egrep "name" | awk -F \" '{print $4}'| cut -f 1 -d :`; do
 IsMaster=`echo "db.isMaster()"| $MongoBinPath/mongo --host $i | grep ismaster|awk -F ":" '{print $2}' | cut -f 1 -d ,`;
 TheState=`echo "rs.status()"| $MongoBinPath/mongo --host $i | grep -i mystate | awk -F ":" '{print $2}' | cut -f 1 -d ,`;
 if [ $IsMaster == "false" -a $TheState -eq 2  ]; then
  MongoHost=$i
  IsOK=1
  echo "$CurrentTime: $MongoHost is slave and looks OK." &gt;&gt; $LogFile
  break
 fi
done
 
CurrentTime=`date +"%r"`
 
## Exit if not good
if [ $IsOK -eq 0 ]; then
   echo "$CurrentTime: Error: Either $MongoHost is not slave or not in good state. Aborting Backup, Please check!" &gt;&gt; $LogFile
   mail -s "Backup error for $ReplicaSet from $MongoHost on $TodayDate" $MailNotification &lt; $LogFile    exit 1; fi ## Remove earlier backup CmdStatus=$(rm -rf $LocalBackupPath/*) ## Start backup process echo "$CurrentTime: Starting Dump, executing $MongoBinPath/mongodump --out $LocalBackupPath --host $MongoHost..." >> $LogFile
 
CmdStatus=`$MongoBinPath/mongodump --out $LocalBackupPath --host $MongoHost`
if [ $? -ne 0 ]; then
  echo "$CurrentTime: There is an issue while trying to take dump in $MongoHost. Aborting dump process, please check! " &gt;&gt; $LogFile
  cat $CmdStatus &gt;&gt; $LogFile
  mail -s "Backup error for $ReplicaSet from $MongoHost on $TodayDate" $MailNotification &lt; $LogFile   exit fi CurrentTime=`date +"%r"` BackupSize=$(du -sh $LocalBackupPath/. | awk '{ print $1 }') echo "$CurrentTime: Mongodump command completed. Backup size is $BackupSize. " &gt;&gt; $LogFile
 
## dump is fine then scp it to backup server
## create directory first
CmdStatus=$(ssh -p $BackupHostPort $BackupHost "mkdir -p  $BackupHostPath/$ReplicaSet/$DateStamp")
 
if [ $? -ne 0 ]; then
  echo "$CurrentTime: Either failing to connect Backup server using ssh or destination directory already exist!" &gt;&gt; $LogFile
  cat $CmdStatus &gt;&gt; $LogFile
  mail -s "Backup error for $ReplicaSet from $MongoHost on $TodayDate" $MailNotification &lt; $LogFile   exit fi CurrentTime=`date +"%r"` echo "$CurrentTime: Directory created on backup server, copying data using scp..." &gt;&gt; $LogFile
 
CmdStatus=`scp -P $BackupHostPort -r $LocalBackupPath/* $BackupHost:/$BackupHostPath/$ReplicaSet/$DateStamp/`
if [ $? -ne 0 ]; then
  echo "$CurrentTime: Unable to scp dump to $BackupHost:/$BackupHostPath/$ReplicaSet/$DateStamp using port $BackupHostPort. " &gt;&gt; $LogFile
  cat $CmdStatus &gt;&gt; $LogFile
  mail -s "Backup error for $ReplicaSet from $MongoHost on $TodayDate" $MailNotification &lt; $LogFile   exit fi CurrentTime=`date +"%r"` echo "$CurrentTime: Copied dump to backup server in directory $BackupHost$BackupHostPath/$ReplicaSet/$DateStamp/." &gt;&gt; $LogFile
echo "$CurrentTime: Mongo Backup process completed successfully." &gt;&gt; $LogFile
mail -s "Backup for $ReplicaSet done on $MongoHost at $TodayDate" $MailRecipients $MailNotification &lt; $LogFile

Its just a basic script and may needs further enhancements. In case you have suggestion/queries, please put it below in comments.

Fix mysql memory table error: The table xtable is full

jagbir — Mon, 20 Feb 2012 08:34:55 +0000

Replication just stopped in one Slave server with error: The table xtable is full which means no more records are permitted to insert in this table by MySQL and hence this has broken the replication.

I checked that xtable is having storage engine as Memory. In such tables, the max. no. of records you can insert is controlled by variable max_heap_table_size. When checking the size of this variable, I found that this is having default value:

mysql> show variables like 'max_heap_table_size';
+---------------------+----------+
| Variable_name       | Value    |
+---------------------+----------+
| max_heap_table_size | 16777216 |
+---------------------+----------+
1 row in set (0.00 sec)

So we need to increase the value of this variable and then issue Alter Table command to make it effective. Also do not forget to add variable with new value in your my.cnf.

Change the variable value:

mysql> set  max_heap_table_size=268435456;
Query OK, 0 rows affected (0.00 sec)
 
mysql> show variables like 'max_heap_table_size';
+---------------------+-----------+
| Variable_name       | Value     |
+---------------------+-----------+
| max_heap_table_size | 268435456 |
+---------------------+-----------+
1 row in set (0.00 sec)

Now let’s truncate the table and issue Alter table on it to make the value effective for this table:

mysql> truncate table xtable;
Query OK, 0 rows affected (0.00 sec)
 
mysql> alter table xtable ENGINE=MEMORY;
Query OK, 88 rows affected (0.06 sec)
Records: 88  Duplicates: 0  Warnings: 0

After this fix, the issue has been resolved. Make sure that you do not run the truncate command on table which have important data. I have issued truncate because it contains temporary/transactional data so we are fine with removing all records here.

Speed up large MySQL backup, dump, restore processes

jagbir — Tue, 19 Apr 2011 07:34:55 +0000

In general, MySQL is quite fast at restoring data, but I observed that while restoring 20gb of backup its taking more than the usual time. This can happen when you don’t have enough memory or if key_buffer_size is not set high enough, then it can take very long time to re-index the data. In CentOS 5.2 server with 6gb of RAM, I noticed key_buffer_size is set to just 800M which is very low. You should set it at least 20-25% of total RAM. After increasing the value to 2gb, MySQL is able to reload data quite fast.

Though this does trick for me but there are other helpful suggestions also which you can try/check to speed up your backup/restore process, few are as follows:

* Increase value of key_buffer_size variable in my.cnf to 20-25% of available RAM as mentioned above. Be aware that, if you make the value too large like setting more than 50% of total RAM, your machine may start to page and become extremely slow. This is because MySQL relies on the operating system to perform file system caching for data reads, so you must leave some room for the file system cache. Anyhow, remember that on 32 bit system, the maximum value key_buffer_size can take is 4gb only and setting any greater might crash your MySQL server.

* For InnoDB engine, another very important setting you should consider is to set innodb_flush_log_at_trx_commit = 2. Reset it to 1 again after restore if you need ACID.

* Again for InnoDB and foreing keys, you could also disable foreign key checks and re-enable it at the end (using SET FOREIGN_KEY_CHECKS=0 and SET FOREIGN_KEY_CHECKS=1).

* Few maatkit tools like mk-parallel-dump / mk-parallel-restore will add capability to do dump/restore with multithreaded support utilizing available no. of CPUs but beware maatkit itself not recommending using these tools for backup.

* Try using Extended inserts in dumps but again as client has a fixed size buffer for each line it is reading and extended inserts may exceeds that limit, there’s possibility that you may not be able to restore your very large dumps.

* Backup/Dump data with –tab format to make use of mysqlimport, which is faster than mysql < dumpfile.

* You can try importing data with multiple threads, means start individual process to import each table.

* Change database engine if possible. Importing into a heavily transactional engine like innodb is painfully slow. Inserting into a non-transactional engine like MyISAM is much much faster.

* You can also try turning off foreign key checks and turn on auto-commit.

* Also check usage of --disable-keys with InnoDB.

* If you think network may be bottleneck, try --compress when using mysqldump.

* Last but not the least here, you can better take directory dump and restore it when required if your MySQL version is similar. You just need to stop your MySQL server, copy whole data or mysql directory, compress it and store at your backup location. Restore in same way when needed. Its one of the fastest method available and I would recommend using it if you are having separate slave hosts which is taking regular backups etc. You can also check lvmbackup/snapshot etc for more efficiency.

Do you know any other tip/tricks which can speed up taking/restore MySQL dump? Please share by putting a comment below.

quick web based php script to check replication status of MySQL

jagbir — Wed, 06 Apr 2011 16:42:14 +0000

This is a basic PHP script to check replication status of single or multiple MySQL servers. You have to update variables in the script to run in browser like user, password, ip of server etc.

Here is the script code:


##########
#
# dbrepstatus.php by Jagbir Singh (jags@jagbir.com)
#
#
#########

?>
 

 Online DB Server status 

 echo $_SERVER['PHP_SELF']; ?>" method="POST">



 

 Online DB Server Status