Migrating email to your own mail server

To move your email to a new mail server you'll need setup the new server with all the existing accounts and copy the mailboxes.

Since this task is often at hand we had to find an efficient way to do this quickly so we wrote a script that would:

1. Create all the existing mailboxes on the new server.
2. Sync all the mail from the old server to the new one.
3. Create Users that have access to this mailboxes.

For this you will need:

• Group Office
• ImapSync
• The script at the bottom of this article

ImapSync packages are available for Debian here: http://archive.debian.net/etch/net/imapsync

Step 1: Setup your new mail server (together with Group Office)

The new email server needs to be installed first.
We have created a package for Debian/Ubuntu that would install and configure a postfix/dovecot mail server (almost) automatically

See our Wiki page for a link to these apt-get packages that would install and configure Group Office, Postfix (MTA for SMTP) and Dovecote (MDA for IMAP)
https://www.group-office.com/wiki/Installing_on_Debian_or_Ubuntu

When Group Office and the Mail server are installed open it in the browser and install the Postfixadmin module by going to Start menu (top right corner) -> Modules -> Install (top left) and choice Postfix Admin. This module lets you configure virtual mail domain and mailboxes. But don't create then yet. This will be done automatically in the next step.

Step 2: Create mailboxes on your new server

First you'll need a list of the existing mailboxes on the source server you want to fetch the mail from. You'll need to following details:

1. Email
2. Password
3. First name
4. Middle name
5. Last name

The above information will be used to create mailboxes in the new server (with the same username and password) and the fetch the mail from the old server. If you don't know the password you'll have to change it because you won't be able to read to mailbox without it.

Put the mailbox information into a mailboxes.csv file like this.

michael@group-office.com,S3cRetPassWD,Michael,de,Hart
merijn@group-office.com,S3cRetPassWD,Merijn,,Schering
wesley@group-office.com,S3cRetPassWD,Wesley,,Smits
wilmar@group-office.com,S3cRetPassWD,Wilmar,van,Beusekom

Step 3: Running the mailbox migration script

After creating the CSV file that contain the information for the mailboxes The script at the bottom of this article will do the rest of the work.

Run it on the command line (not in the browser)

$ php syncmail.php

The script will need to know the path the you mailboxes.csv file. It it is in the same directory press enter. Otherwise enter the path to this file first. It the file is found it will show you a list of mailboxes it is going to migrate and asks you the following question:
- Do you want to create to following mailboxes? [y/N]:

Type 'y' and press enter

This won't take long and the following question will be asked:
- Fill mailboxes from another imap host? [y/N]:

Before you continue confirm the mailboxes are created on your new mail server by logging in to Group Office and open the Postfix admin module you've installed in Step 1

Type 'y' and press enter

ImapSync will need to know the target and source server address to fetch from and copy to. You'll be prompted for this information:
Enter IMAP host to fetch from: mail.oldserver.com
Port number [143]:
Enter IMAP host to copy to: [localhost]:
Port number [143]:

When ImapSync is installed correctly the script will start syncing all the created mailboxes (this could take a while)

Step 4: Creating Group Office user accounts for each mailbox (optional)

The last part the script takes care of is creating Group Office users that may access these mailboxes. If this is already setup our you would rather do this manualy just hit 'n'

Do you want to add/create users with these mail accounts in this Group Office installation [y/N]:

When entering 'y' the script will create Group Office users accounts. The username will be what is before the @ of the mailbox and the password will be the same.

<?php
/*
 * The MIT License (MIT)
 * 
 * Copyright (c) 2014 Intermesh BV 
 * 
 * Permission is hereby granted, free of charge, to any person obtaining a copy
 * of this software and associated documentation files (the "Software"), to deal
 * in the Software without restriction, including without limitation the rights
 * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 * copies of the Software, and to permit persons to whom the Software is
 * furnished to do so, subject to the following conditions:
 * 
 * The above copyright notice and this permission notice shall be included in
 * all copies or substantial portions of the Software.
 * 
 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 * THE SOFTWARE.
 */

/**
 * Script askes the following questions:
 * 
 * - Do you want to create to following mailboxes? [y/N]: 
 * - Fill mailboxes from another imap host? [y/N]:  
 * - Do you want to add/create users with these mailaccounts in this GroupOffice installation [y/N]:
 * 
 * RUN ON CLI AS ROOT !!
 * YOU MOST LIKELY WANT TO CHANGE THE PATH TO GO AND CONFIG.PHP
 * 
 * CSV should look like this:
 * email,password,firstname,middlename,lastname\n
 * 
 * @author Michael de Hart 
 * @copyright Copyright Intermesh BV.
 */

//first argument is the path to config.php
$conf_path = '/etc/groupoffice/config.php';

define('GO_CONFIG_FILE',$conf_path);
define('NOLOG', true); // stop groupoffice from logging. 
require_once('/usr/share/groupoffice/GO.php');
GO::session()->runAsRoot();

$syncer = new ImapSync();
$syncer->start();

class ImapSync {

 protected $imap_sync = '/usr/bin/imapsync'; //binary
 protected $domain = 'superfun.nl';
 protected $quota = 524288; //quota in kb = (512MB)
 protected $source = array(
  'host'=>'',
  'port'=>'143',
 );
 protected $target = array(
  'host'=>'localhost',
  'port'=>'143',
  'smtp_host'=>'localhost',
  'smtp_port'=>'25',
 );
 // CSV Config
 protected $csv = array(
  'path'=>'mailboxes.csv',
  'delimiter'=>",",
  'enclosure'=>'"'
 );
 // Column configuration for Mailbox CSV file
 protected $col = array(
  'email'=>0,
  'password'=>1,
  'first_name'=>2,
  'middle_name'=>3,
  'last_name'=>4,
 );
 protected $log_file = 'imapsync';
 private $records = array();

 public function start() {

  try {
   $handle = fopen("php://stdin","r");
   
   echo "This script will create mailboxes and users based on a CSV file?\n"
   . "Enter the path to the CSV file to continue: [".$this->csv['path']."]: ";
   $line = trim(fgets($handle));
   if(!empty($line)){
    $this->csv['path'] = $line;
   }
   $this->records = $this->loadCsv();
   
   if(GO::modules()->isInstalled('postfixadmin')) {
    echo implode(array_keys($this->records),"\n")."\n".
    "Do you want to create to following mailboxes? [y/N]: ";
    $line = fgets($handle);
    if(trim($line) == 'y'){
     echo "\nCreating mailboxes...\n";
     $this->createMailBoxes();
    }

    echo "Fill mailboxes from another imap host? [y/N]: ";
    $line = fgets($handle);
    if(trim($line) == 'y'){
     $this->configureSync($handle);
     echo "\nSyncing IMAP mailboxes...\n";
     foreach($this->records as $record) {
      $this->syncImap($record);
     }
    }
   }

   echo "Do you want to add/create users with these mailaccounts in this GroupOffice installation [y/N]: ";
   $line = trim(fgets($handle));
   if($line == 'y'){
    echo "\nEnter SMTP host for the accounts [".$this->target['smtp_host']."]: ";
    $line = trim(fgets($handle));
    if(!empty($line))
     $this->target['smtp_host'] = $line;
    echo "\nPort number [".$this->target['smtp_port']."]: ";
    $line = trim(fgets($handle));
    if(!empty($line))
     $this->target['smtp_port'] = $line;
    echo "\nAdding/creating users and accounts...\n";
    $this->createUsersWithMailAccounts();
   }
   echo "All done!\n";

  } catch(Exception $e) {
   echo $e->getMessage()."\n";
  }
 }
 
 private function configureSync($handle) {
  echo "\nEnter IMAP host to fetch from: ";
  $line = trim(fgets($handle));
  if(!empty($line))
   $this->source['host'] = $line;
  echo "\nPort number [".$this->source['port']."]: ";
  $line = trim(fgets($handle));
  if(!empty($line))
   $this->source['port'] = $line;
  echo "\nEnter IMAP host to copy to [".$this->target['host']."]: ";
  $line = trim(fgets($handle));
  if(!empty($line))
   $this->target['host'] = $line;
  echo "\nPort number [".$this->target['port']."]: ";
  $line = trim(fgets($handle));
  if(!empty($line))
   $this->target['port'] = $line;
 }
 
 //Read CSV file to an array
 private function loadCsv() {
  $records=array();
  $fp = @fopen($this->csv['path'], "r");
  if(!$fp)
   die("Can't find: " . $this->csv['path']."\n");
  while ($record = fgetcsv($fp, 4096, $this->csv['delimiter'], $this->csv['enclosure'])) {
   $assocRecord = array();
   foreach($this->col as $key => $number)
    $assocRecord[$key] = $record[$number];
   $records[$assocRecord['email']] = $assocRecord;
  }
  //Get domain from last email account
  if(isset($records[0])) {
   $this->domain = end(explode('@',$records[0]['email'],2));
  }
  return $records;
 }

 //Fetch all Mail and copy to new IMAP server using ImapSync
 private function syncImap($record){
  echo "Syncing " . $record['email'] . "...\n\n";
  $cmd = $this->imap_sync.' --syncinternaldates --authmech1 LOGIN --authmech2 LOGIN ' .
   '--host1="'.$this->source['host'].'" --user1="'.$record['email'].'" --password1="'.$record['password'].'" ' .
   '--host2="'.$this->target['host'].'" --user2="'.$record['email'].'" --password2="'.$record['password'].'" '.
   '--subscribe --allowsizemismatch --nofoldersizes '.
   '--sep1 / --sep2 . --regextrans2 "s,/,_,g"'; // --folder "INBOX"
  if(!empty($this->log_file))
   $cmd .= ' > '.__DIR__.'/'.$this->log_file.'_'.date('Y-m-d\ H:i:s').'.log';
  system($cmd);
 }

 // Create GroupOffice Mailboxes
 private function createMailBoxes() {
  $domain = GO_Postfixadmin_Model_Domain::model()->findSingleByAttribute('domain',$this->domain);
  if(empty($domain)) {
   $domain = new GO_Postfixadmin_Model_Domain();
   $domain->transport='virtual';
   $domain->active='1';
   $domain->domain=$this->domain;
   $domain->default_quota=$this->quota;
   $domain->user_id=1;
   if(!$domain->save())
    throw new Exception('Error while saving mailbox model: '.$record['email']."\n".implode("\n",$domain->getValidationErrors()));
  }
  foreach($this->records as $record) {
   $username = current(explode("@", $record['email']));
   $mailbox = GO_Postfixadmin_Model_Mailbox::model()->findSingleByAttribute('username',$username);
   if(!$mailbox){
    $mailbox = new GO_Postfixadmin_Model_Mailbox();
    $mailbox->domain_id = $domain->id;
    $mailbox->username = $record['email']; //$username;
    $mailbox->quota = $domain->default_quota;
   }
   $mailbox->name = $record['email'];
   $mailbox->password = $record['password'];
   echo "Saving mailbox ".$mailbox->username." with quota ".($mailbox->quota/1024)." MB...\n";
   if(!$mailbox->save())
    throw new Exception('Error while saving mailbox model: '.$record['email']."\n".implode("\n",$mailbox->getValidationErrors())); 
  }
  echo "All mailboxes are created, time to fetch from source mail host...\n";
  
 }

 private function createUsersWithMailAccounts() {
  foreach($this->records as $record) {
   $username = current(explode("@", $record['email']));

   // Check if the user exists in Group-Office and if it doesn't create it.
   $user = GO_Base_Model_User::model()->findSingleByAttribute('username',$username);
   if(!$user){
    $user = new GO_Base_Model_User();
    $user->first_name = $record['first_name'];
    $user->middle_name = $record['middle_name'];
    $user->last_name = $record['last_name'];
    $user->email = $record['email'];
    $user->enabled = 1;
   }
   $user->username = $username;
   $user->password = $record['password'];
   if(!$user->save())
    throw new Exception('Error while saving user: '.$username."\n".implode("\n",$user->getValidationErrors()));

   // Create an e-mail account for the user
   $account = GO_Email_Model_Account::model()->findSingleByAttributes(array(
    'username'=>$record['email'],
    'user_id'=>$user->id
   ));
   if(!$account) {
    $account = new GO_Email_Model_Account();
    $account->user_id = $user->id;
    $account->mbroot = '';
    $account->use_ssl = 0;
    $account->type = 'imap';
    $account->smtp_encryption = '';
    $account->smtp_username = '';
    $account->smtp_password = '';
    $account->name = $user->email;
    $account->email = $user->email;
   }
   $account->host = $this->target['host'];
   $account->port = $this->target['post'];
   $account->smtp_host = $this->target['smtp_host'];
   $account->smtp_port = $this->target['smtp_port'];
   $account->username = $record['email'];
   $account->password = $record['password'];
   if(!$account->save())
    throw new Exception('Error while saving account: '.$username."\n".implode("\n",$account->getValidationErrors()));

   echo $username."\n";
  }
 }

}