Rsync backup script for our web and MySQL servers

One of the most important matters for an IT company is taking care of backups. We make backups in twofold. We make full snapshots of our virtual machines in the data center and we make incremental backups to another location. This is absolutely necessary in case of a major disaster like theft or a fire. We keep 14 incremental backups so when a customer wants to recover a single file he deleted accidentally we can recover it easily as long as it didn't happen earlier than 14 days ago.

We used to have a script on each server that took care of the backup. With rsync and mysql dump. This worked well at first but as our number of servers increased it could happen that the process would fail due to multiple backups running at the same time. Multiple copy commands on the server made everything very slow. When you aborted a slow backup the incremental snapshot could only be half copied resulting in duplicate data being transferred the next backup. When this happpened I decided to turn the process around. I wrote a script that runs on the backup server that connects to each server one by one. The programming language I know best is PHP so I wrote it in that language.

Preparing the server that we need to backup

The server needs to support "rsync". I also want to backup MySQL so I also needed "mysqldump". "mysqldump" comes with the package "mysql-client-5.5" and is usually already installed.

I ran:

$ apt-get install rsync

That's all I needed to do!

Installing the backup server

The backup server can run any Linux distribution supporting PHP on the command line and rsync. I used SSH keys to connect to the servers.
In my case I installed a minimal Debian server with php and rsync on it. I ran:

$ apt-get install rsync php-cli

I used SSH to connect to the web servers with a private and public key pair. I generated the key pair with the following command:

$ ssh-keygen

The resulting key files are then located in /root/.ssh/id_rsa(.pub).

Adding a target server

Now it's time to connect to the target web server. For example. "web1.example.com". To install the public key I ran:

$ ssh-copy-id root@web1.example.com

I entered the root password so that my public key (/root/.ssh/id_rsa.pub) is added to the authorized keys on the web server.

I verified that this worked by logging in with the key. There's no need to enter a password:

$ ssh root@web1.example.com
$ exit

Now we have a backup server that can access our web server with public key authentication.

Installing the script

Put the script on the server. For example in /root/backup.php

Edit the following values on top of the script:

<?php

...


/**
 * The backup report will be mailed to this address.
 * 
 * @var String 
 */
private $emailTo = 'postmaster@example.com';

/**
 * The location of the private key to use for SSH
 * 
 * @var String 
 */
private $sshKey = '/root/.ssh/id_rsa'; 

/**
 * An array of server to backup
 * 
 * @var array 
 */
private $servers = array(
 array(
  'host' => 'web1.example.com', //hostname of target server to backup
  'user' => 'root', //the user on the target server.          
  'port' => 22, //SSH port
  'backupFolders' => array('/etc', '/home', '/var/www'), //folders to backup
  'targetFolder' => '/home/backup/web1', //folder on the backup server to store the backups
  'rotations' => 1, //number of rotations.
  'backupMysql' => true, //Backup MySQL database. Note that "mysqldump" must be installed on the target server,
  'mysqlUser' => 'root',
  'mysqlPassword' => 'secret',
  'mysqlBackupDir' => '/home/mysql_backup' //MySQL backup directory on the target server
 )
);
 
...

After configuration I ran a test:

$ php /root/backup.php

Then I scheduled it in a cron job:

$ crontab -e

I added this line to run it on midnight every day:

0 0 * * php /root/backup.php


I hope this little tutorial is useful! This is the full backup script:

<?php
/**
 * Backup script to backup multiple servers including MySQL support.
 * Author: Merijn Schering <info@intermesh.nl> 
 * 
 * 

The MIT License (MIT)

Copyright (c) 2014 Intermesh BV <info@intermesh.nl>

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
 *
 */

class backup{

 /**
  * The backup report will be mailed to this address.
  * 
  * @var String 
  */
 private $emailTo = 'postmaster@example.com';

 /**
  * The location of the private key to use for SSH
  * 
  * @var String 
  */
 private $sshKey = '/root/.ssh/id_rsa'; 

 /**
  * An array of server to backup
  * 
  * @var array 
  */
 private $servers = array(
   array(
     'host' => 'web1.example.com', //hostname of target server to backup
     'user' => 'root', //the user on the target server.          
     'port' => 22, //SSH port
     'backupFolders' => array('/etc', '/home', '/var/www'), //folders to backup
     'targetFolder' => '/home/backup/web1', //folder on the backup server to store the backups
     'rotations' => 1, //number of rotations.
     'backupMysql' => true, //Backup MySQL database. Note that "mysqldump" must be installed on the target server,
     'mysqlUser' => 'root',
     'mysqlPassword' => 'secret',
     'mysqlBackupDir' => '/home/mysql_backup' //MySQL backup directory on the target server
   )
 );
 
 
 private $logDir;
 private $sshCmd;
 private $dateFormat;
 private $logFile;
 private $errors=array();

 public function __construct() {

  //set the timezone to avoid date() notices from PHP
  date_default_timezone_set('europe/amsterdam');

  //Make sure files are created with the right permissions
  umask(000);

  $old_error_handler = set_error_handler(array($this, "errorHandler"));

  $this->dateFormat = 'Y-m-d_H-i-s';

  if (!isset($this->logDir)) {
   $this->logDir = dirname(__FILE__) . '/log/';
  }
  $this->logFile = $this->logDir . '/' . date($this->dateFormat) . '.log';

  $this->execCommand('mkdir -p ' . $this->logDir);
 }

 /**
  * Executes command line program
  * 
  * @param stirng $cmd
  * @param boolean $halt
  * @return mixed false on failure or command output
  */
 private function execCommand($cmd, $halt = true) {
  exec($cmd, $output, $ret);
  if ($ret != 0) {

   $this->error("Failed to run command: '" . $cmd . "'");
   $this->info(implode("\n", $output));


   if ($halt) {
    $this->exitBackup();
   } else {
    return false;
   }
  } else {
   return $output;
  }
 }

 /**
  * Executes command on the remote server.
  * 
  * @param string $cmd
  * @return mixed false on failure or command output
  */
 private function execRemoteCommand($cmd, $halt=true) {

  $cmd = $this->sshCmd . ' "' . str_replace('"', '\"', $cmd) . '"';

  return $this->execCommand($cmd, $halt);
 }

 /**
  * Exits the backup  script and sends backup report
  * @param string $error
  */
 private function exitBackup($error = '') {
  if ($error != ''){
   $this->error($error);
  }

  $subject = "Backup report (success)";
  $body="";
  
  if(count($this->errors)){
   $subject = "Backup report (ERRORS!)";
   
   $body = "Error summary: \n".implode("\n---\n", $this->errors);
  }else
  {
   $subject = "Backup report (success)";
  }
  
  
  $body .= file_get_contents($this->logFile);
  
  $header = "From: Backup server \r\n";

  mail($this->emailTo, $subject, $body, $header);

  exit();
 }

 /**
  * Writes log message
  * 
  * @param string $text
  */
 private function info($text) {
  $text = '[' . date('Y-m-d H:i') . '] ' . $text . "\n";

  echo $text;

  file_put_contents($this->logFile, $text, FILE_APPEND);
 }
 
 
 /**
  * Writes error message
  * 
  * @param string $text
  */
 private function error($text) {
  $text = '[' . date('Y-m-d H:i') . '] !!ERROR!! :' . $text . "\n";

  echo $text;
  
  $this->errors[]=$text;

  file_put_contents($this->logFile, $text, FILE_APPEND);
 }

 private function setSshCmd($server) {
  $this->sshCmd = 'ssh -i ' . $this->sshKey . ' ' . $server['user'] . '@' . $server['host'] . ' -p ' . $server['port'];

  $this->info("Testing SSH connection");
  if($this->execCommand($this->sshCmd . ' -o "PasswordAuthentication=no" "exit"' , false)!==false){
   $this->info("Test OK");
  }else
  {
   $this->error("SSH test for ".$server['host']." failed!");
   return false;
  }


  $this->sshCmd .= ' -o "ServerAliveInterval 60"';

  return true;
 }

 /**
  * Runs the backup
  */
 public function run() {

  //We never want this script to run twice at the same time
  $lockFile = dirname(__FILE__) . '/backup.lock';
  if (file_exists($lockFile)) {
   $this->exitBackup("Lock file '" . $lockFile . "' exists. Please check last backup! It failed or it is still running.");
  } else {
   touch($lockFile);
  }

  foreach ($this->servers as $server) {


   $this->info("\n\n\n----------------------------------------------------\n\n\n");
   $this->info("Backing up " . $server['host']);

   if($this->updateLatest){
    $server['backupMysql']=false;
    $server['rotations']=1;
   }

   if(!$this->setSshCmd($server)){
    continue;
   }

   //Create the backup folder
   $this->execCommand('mkdir -p ' . $server['targetFolder']);


   //Delete the oldest backup
   $ls = $this->execCommand('ls -dXr ' . $server['targetFolder'] . '/*/', false);

   $newestBackup = isset($ls[0]) ? $ls[0] : false;

   if ($server['rotations'] > 1) {
    $oldBackupIndex = $server['rotations'] - 1;
    if (isset($ls[$oldBackupIndex])) {
     $this->info("Deleting oldest backup: " . $ls[$oldBackupIndex]);
     $this->execCommand('rm -Rf ' . $ls[$oldBackupIndex]);
     $this->info("Delete done");
    }
   }

   //Make a copy using hard links so we don't duplicate in diskspace
   $backupDate = date($this->dateFormat);
   $currentBackupDir = $server['targetFolder'] . '/' . $backupDate;

   if ($newestBackup) {
    if ($server['rotations'] > 1) {
     $this->info("Copying all from " . $newestBackup . " to " . $currentBackupDir);
     $this->execCommand('cp -al ' . $newestBackup . ' ' . $currentBackupDir);
    } else {
     $this->info("Moving previous backup dir from " . $newestBackup . " to " . $currentBackupDir);
     $this->execCommand("mv " . $newestBackup . " " . $currentBackupDir);
    }
   } else {
    $this->info("Creating first backup directory " . $currentBackupDir);
    $this->execCommand('mkdir ' . $currentBackupDir);
   }

   if (!is_writable($currentBackupDir)) {
    $this->exitBackup("Error: backup directory $currentBackupDir is not writable!");
   }


   $this->backupMysql($server);


   //Now run rsync for each backup source
   foreach ($server['backupFolders'] as $remoteBackupFolder) {
    $this->info("Backing up " . $remoteBackupFolder);

    $localBackupFolder = $currentBackupDir . '/' . $remoteBackupFolder;
    $this->execCommand('mkdir -p ' . $localBackupFolder);

    $cmd = 'rsync -av --delete -e "ssh -i ' . $this->sshKey . ' -p' . $server['port'] . '" ' . $server['user'] . '@' . $server['host'] . ':' . $remoteBackupFolder . '/ ' . $localBackupFolder . '/';
    $this->info("Running " . $cmd);
    //system($cmd, $ret);
    //run the rsync command and read the response line by line
    $fp = popen($cmd, "r");
    while (!feof($fp)) {
     // send the current file part to the browser 
     $this->info(fread($fp, 1024));
    }
    $ret = pclose($fp);

    if ($ret != 0) {
    
     if($ret == 24){
      $this->info("Some source files vanished during transfer. This is probably normal because the system is live.");
     }else{    
      $this->error("Failed running rsync command!\ncmd: ".$cmd."\nReturn code: ".$ret);
     }
    }
   }

   $this->info("Done with " . $server['host']);

   $this->info("\n\n\n----------------------------------------------------\n\n\n");

   //$contents = ob_get_contents(); 
  }

  //remove lock file
  unlink($lockFile);

  $this->exitBackup();
 }

 private function backupMysql($server) {
  if ($server['backupMysql']) {

   $this->info("Starting MySQL backup");

   if (empty($server['mysqlBackupDir'])) {
    $this->error("Please set " . $server['mysqlBackupDir']);
    return false;
   }
   if($this->execRemoteCommand('mkdir -p ' . $server['mysqlBackupDir'], false)===false){
    return false;
   }

   $cmd = 'mysql --user=' . $server['mysqlUser'] . ' --password=' . $server['mysqlPassword'] . ' -e "SHOW DATABASES;" | tr -d "| " | grep -v Database';
   //$this->info($cmd);
   $databases = $this->execRemoteCommand($cmd, false);

   foreach ($databases as $database) {
    if ($database != "information_schema" && $database != 'performance_schema') {
     $this->info("Dumping $database");
     $cmd = 'mysqldump --force --opt --user=' . $server['mysqlUser'] . ' --password=' . $server['mysqlPassword'] . ' --databases ' . $database . ' > "' . $server['mysqlBackupDir'] . '/' . $database . '.sql"';

     if($this->execRemoteCommand($cmd, false)===false){
      continue;
     }

     $this->info("Compressing " . $database);
     $cmd = "tar -C " . $server['mysqlBackupDir'] . " -czf " . $server['mysqlBackupDir'] . "/$database.tar.gz $database.sql";
     if($this->execRemoteCommand($cmd, false)===false){
      continue;
     }

     $cmd = 'rm ' . $server['mysqlBackupDir'] . "/$database.sql";
     if($this->execRemoteCommand($cmd, false)===false){
      continue;
     }



     //break;
    }
   }
   $this->info("MySQL backup done");
  } else {
   $this->info("MySQL backup NOT enabled");
  }
 }

 public function errorHandler($errno, $errstr, $errfile, $errline) {
  switch ($errno) {
   case E_USER_ERROR:
    // Send an e-mail to the administrator
    $msg = "!!!!! Error: $errstr \n Fatal error on line $errline in file $errfile";

    // Write the error to our log file
    $msg = "!!!!! Error: $errstr \n Fatal error on line $errline in file $errfile";
    break;

   case E_USER_WARNING:
    // Write the error to our log file
    $msg = "!!!!! Warning: $errstr \n in $errfile on line $errline";
    break;

   case E_USER_NOTICE:
    // Write the error to our log file
    $msg = "!!!!! Notice: $errstr \n in $errfile on line $errline";
    break;

   default:
    // Write the error to our log file
    $msg = "!!!!! Unknown error [#$errno]: $errstr \n in $errfile on line $errline";
    break;
  }

  $this->error($msg);

  // Don't execute PHP's internal error handler
  return TRUE;
 }

}

$backup = new backup();
$backup->run();