Thursday, July 24, 2014

New Intermesh frameworks


New technologies for the web are rising rapidly. As an application development company it's important to keep up with new technologies. I've explored and finally started to develop two frameworks that we will use for new applications.


On the server I want to create an API that's just communicating with clients through JSON requests. The server and client should be completely decoupled in my opinion. The server runs with the Intermesh PHP framework that features:
  • Api documentation with ApiGen
  • Model View Controller design pattern
  • ActiveRecord models for database ORM
  • Simple routing of HTTP requests to controller action methods
  • User authentication and role based permissions
  • Unit testing with phpunit
  • Composer support
On the client side I've developed an AngularJS framework for building an Angular client. The framework features:
  • API documentation with ngdocs
  • AngularJS uses the Model View Controller design pattern
  • bower support
  • services, directives and filters to communicate with the Intermesh PHP server API.
  • grunt to build the application, documentation and maintain index.html so it's not needed to add new scripts all the time.

Notes about the tools used

Composer

Composer is an essential tool. It takes care of:
  1. Installing and maintaining 3rd party libraries
  2. Autoloading PHP classes using the PSR-04 standard
There's no need to write your own autoloading script any more. Just install packages and you're good to go just by using the class names.

ApiGen

Documentation was always something that was pushed to a later date. So it's important to get this right at the start of a new project. I've tried various documentation generators PHPDoc, SAMI and ApiGen. The first two we're disappointing in the result. PHPDoc is more active and has more feature but the out of the box templates are just not good enough. ApiGen was the only one that produced a very usable result out of the box!

PHPUnit

I think the only and best tool for unit testing. Works flawlessly out of the box.

Bower

An essential tool that is similar to Composer but for javascript packages.

Grunt

Grunt is a javascript task runner. I used it for a couple of things. When I started with the AngularJS project I wanted to keep each component in a separate Javascript file to keep the code organized. Each time a new file was created I had to add it to the index.html file. As I'm a lazy programmer I don't like to do such annoying tasks :) Grunt can do that for me using the "fileblocks" module. Each time I add a file it's automatically inserted. Apart from that you also don't want to deploy your application like that. To improve performance grunt can concatenate and minify the Javascript, CSS and HTML for you.

ngdocs

Getting AngularJS documentation was tricky to setup. First of all there is some package confusion. I've tried ngdoc, ngdocs and dgeni. I couldn't get ngdoc to work at all so I moved on to ngdocs which worked well out of the box. I noticed the dgeni project is more active and used by AngularJS but it doesn't seem to be ready for community use yet. Perhaps it will grow out to be better than ngdocs.
The hardest thing to get going were the live inline examples. I used grunt to build a special script that includes all the Intermesh angular modules. With the "scripts" option for the ngdocs grunt task I was able to add those scripts to the docs so they started to work. See my Gruntfile.js for an example.
Here's an example of the ngdoc syntax for live examples:
https://github.com/Intermesh/intermesh-angular-framework/blob/master/src/intermesh/core/translate-provider.js

Getting started

To get started with development you will need to install:
I use Ubuntu to develop on. If you want to install on Ubuntu you can install git and the Node Package Manager npm with this command:

$ sudo apt-get install git npm 

I had to work around some bug with (see https://github.com/joyent/node/issues/3911):

$ sudo ln -s /usr/bin/nodejs /usr/bin/node 

Install grunt, bower with the node package manager (use -g for global install):

$ sudo npm install -g grunt-cli bower

Install composer:

$ curl -sS https://getcomposer.org/installer | php 
$ sudo mv composer.phar /usr/local/bin/composer
Then install the server and client examples. Installation instructions are on the project pages on Github:

 

Wednesday, May 28, 2014

Building a Group-Office client with AngularJS and Bootstrap

To demonstrate how easy it is to build a client for Group-Office, I've created an AngularJS app. It demonstrates that the server side PHP code of Group-Office is in fact one big JSON API that can be talked to with a completely decoupled client.

It's a very simple interface that was just for a "Proof of concept" but here's a screenshot:

E-mail App in Angularjs Group-Office client

When we'll replace the current ExtJS 3 based user interface we'll use the same approach.

I've created a GitHub repository for the Group-Office client here:

https://github.com/mschering/angular-groupoffice

Note that this client will only work with Group-Office 6.0.3 which you can get here:

http://sourceforge.net/projects/group-office/files/6.0/


I hope you like this example! You can also integrate these kind of API calls into your website or another application. Let me know what you think about it!

 

Note about CORS

If you run the Group-Office server on a different domain than this client you'll need to enable CORS. Otherwise you'll get errors like:

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote
resource at http://localhost/groupoffice. This can be fixed by moving the resource
to the same domain or enabling CORS.

You can read more about enabling CORS on Group-Office here:

https://www.group-office.com/wiki/CORS

Friday, May 16, 2014

Installing fast indexed search for IMAP e-mail

With Dovecot and apache SOLR it's possible to install very fast mail search. You can search through the entire message contents within fractions of a second.

I've used Ubuntu and Debian to set this up.

Tomcat SOLR


Install the tomcat solr server with the following command:

$ apt-get install solr-tomcat

Find the SOLR schema on page http://hg.dovecot.org/dovecot-2.2/file/tip/doc/solr-schema.xml

Make sure your dovecot version matches.

Download it like this:

$ wget http://hg.dovecot.org/dovecot-2.2/raw-file/e99cd21e1f92/doc/solr-schema.xml

Move the file to the right location:

$ cp solr-schema.xml /etc/solr/conf/schema.xml

$ service tomcat6 restart

Check if tomcat works by browsing to:

http://localhost:8080

Dovecot

Now setup Dovecot. Modify:

/etc/dovecot/conf.d/10-mail.conf:

...
# Space separated list of plugins to load for all services. Plugins specific to
# IMAP, LDA, etc. are added to this list in their own .conf files.
#mail_plugins =
mail_plugins = $mail_plugins fts fts_solr
...


/etc/dovecot/conf.d/90-plugin.conf:

...
plugin {
  fts = solr
  fts_solr = break-imap-search url=http://localhost:8080/solr/
}
...

The "break-imap-search" option will use Solr also for indexing TEXT and BODY searches. This makes your server non-IMAP-compliant, but it's what people want ;). This is always enabled in v2.1+.

Now when an IMAP client does a "SEARCH TEXT keyword command" you should see these log entries in /var/log/mail.log:

May 15 14:19:29 mail.example.com dovecot: indexer-worker(admin@intermesh.dev): Indexed 294 messages in INBOX

Enjoy the fast searches!

Friday, May 9, 2014

Setting up a Chat server with Prosody and Group-Office


A lot of customers requested a chat module for Group-Office. In my search for a
solution I stumbled upon a great XMPP/Jabber chat server called Prosody.

It was very easy to connect this server with Group-Office. This is a short guide
on how to do it.

Become root and add the Prosody repository (replace squeeze with the debian or ubuntu version you are using):

$ echo deb http://packages.prosody.im/debian squeeze main | tee -a /etc/apt/sources.list

Add their key file:

$ wget https://prosody.im/files/prosody-debian-packages.key -O- | apt-key add -

Update APT sources:

$ apt-get update

No install Prosody XMPP server:

$ apt-get install prosody luarocks

Install lpy becuase it's required for the external external auth module:

$ luarocks install lpty

Download mod_auth_external for Prosody here:

https://code.google.com/p/prosody-modules/source/browse/mod_auth_external/mod_auth_external.lua

Put it in /usr/lib/prosody/modules/

Now create /usr/local/bin/GroupOfficeProsodyAuth.php. This script handles the
authentication requests from Prosody and it uses the Group-Office framework.

#!/usr/bin/php

 * @package GO.base
 */

// the logfile to which to write, should be writeable by the user 
// that is running the server
$sLogFile  = "/var/log/prosody/prosody.log";

// set true to debug if needed
$bDebug  = false;

$oAuth = new exAuth($sLogFile, $bDebug);

class exAuth
{

 private $sLogFile;

 private $bDebug;

 private $rLogFile;

 private function _includeGO($server){
  //Try to detect the config file.
        if(file_exists('/etc/groupoffice/'.$server.'/config.php')){
                define('GO_CONFIG_FILE', '/etc/groupoffice/'.$server.'/config.php');
                require(GO_CONFIG_FILE);
                require_once($config['root_path'].'GO.php');
        }else
        {

                require_once('/usr/share/groupoffice/GO.php');
        }

  
  $this->writeDebugLog("Config file: ".GO::config()->get_config_file());
 }
 
 public function __construct($sLogFile, $bDebug)
 {

  $this->sLogFile  = $sLogFile;
  $this->bDebug  = $bDebug;
  
  $this->rLogFile = fopen($this->sLogFile, "a") or die("Error opening log file: ". $this->sLogFile);

  $this->writeLog("start");

  do {  
   $sData = substr(fgets(STDIN),0,-1);
   
   if($sData){
    
    $this->writeDebugLog("received data: ". $sData);
    $aCommand = explode(":", $sData);
    if (is_array($aCommand)){
     switch ($aCommand[0]){
      case "isuser":
       if (!isset($aCommand[1])){
        $this->writeLog("invalid isuser command, no username given");
        $this->failure();
       } else {
   
        $sUser = str_replace(array("%20", "(a)"), array(" ", "@"), $aCommand[1]);
        $this->writeDebugLog("checking isuser for ". $sUser);

        $this->_includeGO($aCommand[2]);
        $user = \GO\Base\Model\User::model()->findSingleByAttribute('username', $sUser);

        if ($user){        
         $this->writeLog("valid user: ". $sUser);
         
         $this->success();

        } else {
         $this->writeLog("invalid user: ". $sUser);
         $this->failure();
        }
        
        
       }
      break;
      case "auth":
       
       if (sizeof($aCommand) != 4){
        $this->writeLog("invalid auth command, data missing");
        $this->failure();
       } else {
        

        $sUser = str_replace(array("%20", "(a)"), array(" ", "@"), $aCommand[1]);
        $sPassword = $aCommand[3];
        $this->writeDebugLog("doing auth for ". $sUser." with pass ".$sPassword);



        $this->_includeGO($aCommand[2]);

        $user = \GO::session()->login($sUser, $sPassword);

     
        if ($user) {
      
         $this->writeLog("authentificated user ". $sUser ." domain ". $aCommand[2]);
         $this->success();
        } else {
      
         $this->writeLog("authentification failed for user ". $sUser .
          " domain ". $aCommand[2]);
         $this->failure();
        }
       
       }
      break;
      case "setpass":
       $this->writeLog("setpass command disabled");
       $this->failure();
      break;
      default:
       $this->writeLog("unknown command ". $aCommand[0]);
       $this->failure();
      break;
     }
    } else {
     $this->writeDebugLog("invalid command string");
     $this->failure();
    }
   }

   unset($aCommand);
  } while (true);
 }

 public function __destruct()
 {
  $this->writeLog("stop");  
 }

 private function writeLog($sMessage)
 {
  if (is_resource($this->rLogFile)) {
   fwrite($this->rLogFile, date("r") ." [external_auth] ". $sMessage ."\n");
  }
 }

 private function writeDebugLog($sMessage)
 {
  if ($this->bDebug){
   $this->writeLog( date("r")." [external_auth_debug] ".$sMessage);
  }
 }
 
 
 private function success(){
  echo "1\n";
 }
 
 private function failure(){
  echo "0\n";
 }


 
}


Make this script executable:

$ chmod +x /usr/local/bin/GroupOfficeProsodyAuth.php

Edit /etc/prosody/prosody.cfg.lua and set:

authentication = "external"
external_auth_command = "/usr/local/bin/GroupOfficeProsodyAuth.php"
external_auth_protocol = "generic"

Also setup your VirtualHost with the domain you run Group-Office on.
(See https://prosody.im/doc/configure for more info on configuring Prosody).

Restart the Prosody XMPP server:

$ service prosody restart

Now try to connect with your XMPP/Jabber client.

Use your Group-Office user, password and domain.

You can check /var/log/prosody/ for log files.

Now all we need is an XMPP web interface for Group-Office!

Update: I found the excellent program Converse.js that could be integrated in Group-Office 6.0 easily:

Additionally, I used the Prosody "groups" module to automatically deliver all Group-Office Chat users as contacts to the chat clients.

Wednesday, April 9, 2014

Migrating email to your own mail server

To move your email to a new mail server you'll need setup the new server with all the existing accounts and copy the mailboxes.

Since this task is often at hand we had to find an efficient way to do this quickly so we wrote a script that would:
  1. Create all the existing mailboxes on the new server.
  2. Sync all the mail from the old server to the new one.
  3. Create Users that have access to this mailboxes.


For this you will need:
  • Group Office
  • ImapSync
  • The script at the bottom of this article


ImapSync packages are available for Debian here: http://archive.debian.net/etch/net/imapsync

Step 1: Setup your new mail server (together with Group Office)

The new email server needs to be installed first.
We have created a package for Debian/Ubuntu that would install and configure a postfix/dovecot mail server (almost) automatically

See our Wiki page for a link to these apt-get packages that would install and configure Group Office, Postfix (MTA for SMTP) and Dovecote (MDA for IMAP)
https://www.group-office.com/wiki/Installing_on_Debian_or_Ubuntu

When Group Office and the Mail server are installed open it in the browser and install the Postfixadmin module by going to Start menu (top right corner) -> Modules -> Install (top left) and choice Postfix Admin. This module lets you configure virtual mail domain and mailboxes. But don't create then yet. This will be done automatically in the next step.

Step 2: Create mailboxes on your new server

First you'll need a list of the existing mailboxes on the source server you want to fetch the mail from. You'll need to following details:
  1. Email
  2. Password
  3. First name
  4. Middle name
  5. Last name
The above information will be used to create mailboxes in the new server (with the same username and password) and the fetch the mail from the old server. If you don't know the password you'll have to change it because you won't be able to read to mailbox without it.

Put the mailbox information into a mailboxes.csv file like this.
michael@group-office.com,S3cRetPassWD,Michael,de,Hart
merijn@group-office.com,S3cRetPassWD,Merijn,,Schering
wesley@group-office.com,S3cRetPassWD,Wesley,,Smits
wilmar@group-office.com,S3cRetPassWD,Wilmar,van,Beusekom


Step 3: Running the mailbox migration script

After creating the CSV file that contain the information for the mailboxes The script at the bottom of this article will do the rest of the work.

Run it on the command line (not in the browser)

$ php syncmail.php

The script will need to know the path the you mailboxes.csv file. It it is in the same directory press enter. Otherwise enter the path to this file first. It the file is found it will show you a list of mailboxes it is going to migrate and asks you the following question:
- Do you want to create to following mailboxes? [y/N]:

Type 'y' and press enter

This won't take long and the following question will be asked:
- Fill mailboxes from another imap host? [y/N]:

Before you continue confirm the mailboxes are created on your new mail server by logging in to Group Office and open the Postfix admin module you've installed in Step 1

Type 'y' and press enter

ImapSync will need to know the target and source server address to fetch from and copy to. You'll be prompted for this information:
Enter IMAP host to fetch from: mail.oldserver.com
Port number [143]:
Enter IMAP host to copy to: [localhost]:
Port number [143]:

When ImapSync is installed correctly the script will start syncing all the created mailboxes (this could take a while)

Step 4: Creating Group Office user accounts for each mailbox (optional)

The last part the script takes care of is creating Group Office users that may access these mailboxes. If this is already setup our you would rather do this manualy just hit 'n'

Do you want to add/create users with these mail accounts in this Group Office installation [y/N]:

When entering 'y' the script will create Group Office users accounts. The username will be what is before the @ of the mailbox and the password will be the same.

<?php
/*
 * The MIT License (MIT)
 * 
 * Copyright (c) 2014 Intermesh BV 
 * 
 * Permission is hereby granted, free of charge, to any person obtaining a copy
 * of this software and associated documentation files (the "Software"), to deal
 * in the Software without restriction, including without limitation the rights
 * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 * copies of the Software, and to permit persons to whom the Software is
 * furnished to do so, subject to the following conditions:
 * 
 * The above copyright notice and this permission notice shall be included in
 * all copies or substantial portions of the Software.
 * 
 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 * THE SOFTWARE.
 */

/**
 * Script askes the following questions:
 * 
 * - Do you want to create to following mailboxes? [y/N]: 
 * - Fill mailboxes from another imap host? [y/N]:  
 * - Do you want to add/create users with these mailaccounts in this GroupOffice installation [y/N]:
 * 
 * RUN ON CLI AS ROOT !!
 * YOU MOST LIKELY WANT TO CHANGE THE PATH TO GO AND CONFIG.PHP
 * 
 * CSV should look like this:
 * email,password,firstname,middlename,lastname\n
 * 
 * @author Michael de Hart 
 * @copyright Copyright Intermesh BV.
 */

//first argument is the path to config.php
$conf_path = '/etc/groupoffice/config.php';

define('GO_CONFIG_FILE',$conf_path);
define('NOLOG', true); // stop groupoffice from logging. 
require_once('/usr/share/groupoffice/GO.php');
GO::session()->runAsRoot();

$syncer = new ImapSync();
$syncer->start();

class ImapSync {

 protected $imap_sync = '/usr/bin/imapsync'; //binary
 protected $domain = 'superfun.nl';
 protected $quota = 524288; //quota in kb = (512MB)
 protected $source = array(
  'host'=>'',
  'port'=>'143',
 );
 protected $target = array(
  'host'=>'localhost',
  'port'=>'143',
  'smtp_host'=>'localhost',
  'smtp_port'=>'25',
 );
 // CSV Config
 protected $csv = array(
  'path'=>'mailboxes.csv',
  'delimiter'=>",",
  'enclosure'=>'"'
 );
 // Column configuration for Mailbox CSV file
 protected $col = array(
  'email'=>0,
  'password'=>1,
  'first_name'=>2,
  'middle_name'=>3,
  'last_name'=>4,
 );
 protected $log_file = 'imapsync';
 private $records = array();

 public function start() {

  try {
   $handle = fopen("php://stdin","r");
   
   echo "This script will create mailboxes and users based on a CSV file?\n"
   . "Enter the path to the CSV file to continue: [".$this->csv['path']."]: ";
   $line = trim(fgets($handle));
   if(!empty($line)){
    $this->csv['path'] = $line;
   }
   $this->records = $this->loadCsv();
   
   if(GO::modules()->isInstalled('postfixadmin')) {
    echo implode(array_keys($this->records),"\n")."\n".
    "Do you want to create to following mailboxes? [y/N]: ";
    $line = fgets($handle);
    if(trim($line) == 'y'){
     echo "\nCreating mailboxes...\n";
     $this->createMailBoxes();
    }

    echo "Fill mailboxes from another imap host? [y/N]: ";
    $line = fgets($handle);
    if(trim($line) == 'y'){
     $this->configureSync($handle);
     echo "\nSyncing IMAP mailboxes...\n";
     foreach($this->records as $record) {
      $this->syncImap($record);
     }
    }
   }

   echo "Do you want to add/create users with these mailaccounts in this GroupOffice installation [y/N]: ";
   $line = trim(fgets($handle));
   if($line == 'y'){
    echo "\nEnter SMTP host for the accounts [".$this->target['smtp_host']."]: ";
    $line = trim(fgets($handle));
    if(!empty($line))
     $this->target['smtp_host'] = $line;
    echo "\nPort number [".$this->target['smtp_port']."]: ";
    $line = trim(fgets($handle));
    if(!empty($line))
     $this->target['smtp_port'] = $line;
    echo "\nAdding/creating users and accounts...\n";
    $this->createUsersWithMailAccounts();
   }
   echo "All done!\n";

  } catch(Exception $e) {
   echo $e->getMessage()."\n";
  }
 }
 
 private function configureSync($handle) {
  echo "\nEnter IMAP host to fetch from: ";
  $line = trim(fgets($handle));
  if(!empty($line))
   $this->source['host'] = $line;
  echo "\nPort number [".$this->source['port']."]: ";
  $line = trim(fgets($handle));
  if(!empty($line))
   $this->source['port'] = $line;
  echo "\nEnter IMAP host to copy to [".$this->target['host']."]: ";
  $line = trim(fgets($handle));
  if(!empty($line))
   $this->target['host'] = $line;
  echo "\nPort number [".$this->target['port']."]: ";
  $line = trim(fgets($handle));
  if(!empty($line))
   $this->target['port'] = $line;
 }
 
 //Read CSV file to an array
 private function loadCsv() {
  $records=array();
  $fp = @fopen($this->csv['path'], "r");
  if(!$fp)
   die("Can't find: " . $this->csv['path']."\n");
  while ($record = fgetcsv($fp, 4096, $this->csv['delimiter'], $this->csv['enclosure'])) {
   $assocRecord = array();
   foreach($this->col as $key => $number)
    $assocRecord[$key] = $record[$number];
   $records[$assocRecord['email']] = $assocRecord;
  }
  //Get domain from last email account
  if(isset($records[0])) {
   $this->domain = end(explode('@',$records[0]['email'],2));
  }
  return $records;
 }

 //Fetch all Mail and copy to new IMAP server using ImapSync
 private function syncImap($record){
  echo "Syncing " . $record['email'] . "...\n\n";
  $cmd = $this->imap_sync.' --syncinternaldates --authmech1 LOGIN --authmech2 LOGIN ' .
   '--host1="'.$this->source['host'].'" --user1="'.$record['email'].'" --password1="'.$record['password'].'" ' .
   '--host2="'.$this->target['host'].'" --user2="'.$record['email'].'" --password2="'.$record['password'].'" '.
   '--subscribe --allowsizemismatch --nofoldersizes '.
   '--sep1 / --sep2 . --regextrans2 "s,/,_,g"'; // --folder "INBOX"
  if(!empty($this->log_file))
   $cmd .= ' > '.__DIR__.'/'.$this->log_file.'_'.date('Y-m-d\ H:i:s').'.log';
  system($cmd);
 }

 // Create GroupOffice Mailboxes
 private function createMailBoxes() {
  $domain = GO_Postfixadmin_Model_Domain::model()->findSingleByAttribute('domain',$this->domain);
  if(empty($domain)) {
   $domain = new GO_Postfixadmin_Model_Domain();
   $domain->transport='virtual';
   $domain->active='1';
   $domain->domain=$this->domain;
   $domain->default_quota=$this->quota;
   $domain->user_id=1;
   if(!$domain->save())
    throw new Exception('Error while saving mailbox model: '.$record['email']."\n".implode("\n",$domain->getValidationErrors()));
  }
  foreach($this->records as $record) {
   $username = current(explode("@", $record['email']));
   $mailbox = GO_Postfixadmin_Model_Mailbox::model()->findSingleByAttribute('username',$username);
   if(!$mailbox){
    $mailbox = new GO_Postfixadmin_Model_Mailbox();
    $mailbox->domain_id = $domain->id;
    $mailbox->username = $record['email']; //$username;
    $mailbox->quota = $domain->default_quota;
   }
   $mailbox->name = $record['email'];
   $mailbox->password = $record['password'];
   echo "Saving mailbox ".$mailbox->username." with quota ".($mailbox->quota/1024)." MB...\n";
   if(!$mailbox->save())
    throw new Exception('Error while saving mailbox model: '.$record['email']."\n".implode("\n",$mailbox->getValidationErrors())); 
  }
  echo "All mailboxes are created, time to fetch from source mail host...\n";
  
 }

 private function createUsersWithMailAccounts() {
  foreach($this->records as $record) {
   $username = current(explode("@", $record['email']));

   // Check if the user exists in Group-Office and if it doesn't create it.
   $user = GO_Base_Model_User::model()->findSingleByAttribute('username',$username);
   if(!$user){
    $user = new GO_Base_Model_User();
    $user->first_name = $record['first_name'];
    $user->middle_name = $record['middle_name'];
    $user->last_name = $record['last_name'];
    $user->email = $record['email'];
    $user->enabled = 1;
   }
   $user->username = $username;
   $user->password = $record['password'];
   if(!$user->save())
    throw new Exception('Error while saving user: '.$username."\n".implode("\n",$user->getValidationErrors()));

   // Create an e-mail account for the user
   $account = GO_Email_Model_Account::model()->findSingleByAttributes(array(
    'username'=>$record['email'],
    'user_id'=>$user->id
   ));
   if(!$account) {
    $account = new GO_Email_Model_Account();
    $account->user_id = $user->id;
    $account->mbroot = '';
    $account->use_ssl = 0;
    $account->type = 'imap';
    $account->smtp_encryption = '';
    $account->smtp_username = '';
    $account->smtp_password = '';
    $account->name = $user->email;
    $account->email = $user->email;
   }
   $account->host = $this->target['host'];
   $account->port = $this->target['post'];
   $account->smtp_host = $this->target['smtp_host'];
   $account->smtp_port = $this->target['smtp_port'];
   $account->username = $record['email'];
   $account->password = $record['password'];
   if(!$account->save())
    throw new Exception('Error while saving account: '.$username."\n".implode("\n",$account->getValidationErrors()));

   echo $username."\n";
  }
 }

}

Wednesday, March 26, 2014

Automatically upload a screenshot from the clipboard on paste using Javascript in Google Chrome, Internet Explorer 11 and Firefox

I was working on a new documentation site and had to make a lot of screenshots. I thought it would be useful to be able to hit print screen and then paste the image directly into Group-Office using the CTRL+V shortcut. Luckily with HTML5 you can do that!

With Google Chrome and Internet Explorer 11 you can use the Clipboard API. I created a "Paster" class that you can pass a DOM object and a callback function. So when you paste an image into that object the file is uploaded to a PHP script and then the callback function is called with the response.

In this working example it will call the PHP page itself and then put the response in the textarea. It should work on any PHP enabled web server.

You can add an onpaste event to any DOM element in Google Chrome. Then the "Paster" loops through all clipboard items and checks if there's an image:


Paster.prototype.handlePaste = function(paster, e) {

 //don't do this twice
 if (paster.processing) {
  return;
 }

 //loop through all clipBoardData items and upload it if it's a file.
 for (var i = 0; i < e.clipboardData.items.length; i++) {
  var item = e.clipboardData.items[i];
  if (item.kind === "file") {

   paster.processing = true;
   e.preventDefault();
   paster.uploadFile(paster, item.getAsFile());
  }
 }
};

Then it uploads the file using an "XMLHttpRequest" call. When the upload is finished the callback function is called.

Paster.prototype.uploadFile = function(paster, file) {

 var xhr = new XMLHttpRequest();

 //progress logging
 xhr.upload.onprogress = function(e) {
  var percentComplete = (e.loaded / e.total) * 100;

  console.log(percentComplete);
 };

 //called when finished
 xhr.onload = function() {
  if (xhr.status === 200) {
   alert("Sucess! Upload completed. PHP response will be put in the textarea.");
  } else {
   alert("Error! Upload failed");
  }

  paster.processing = false;
 };

 //error handling
 xhr.onerror = function() {
  alert("Error! Upload failed. Can not connect to server.");
 };

 //trigger a callback when it's successful
 xhr.onreadystatechange = function()
 {
  if (xhr.readyState === 4 && xhr.status === 200)
  {
   if (paster.callback) {
    paster.callback.call(paster.scope || paster, paster, xhr);
   }
  }
 };


 //prompt for the filename
 var filename = prompt("Please enter the file name", "Pasted image");

 if (filename) {

  //upload the file
  xhr.open("POST", "", {
   filename: filename,
   filetype: file.type
  });
  var formData = new FormData();
  formData.append("pastedFile", file);
  xhr.send(formData);
 }
}

Firefox

So far it was pretty simple! But then I started testing this in Firefox. Unfortunately it didn't work because Firefox doesn't support files in the clipboard. I came up with a work around though.

You can paste images in a content editable div in Firefox. The blob data is then available in the image source tag: <img src="data:image/png;base64,DATA" />. So in my workaround I create a hidden content editable div. When ever the textarea blurs, it will focus this hidden element. So when you click out of the textarea and then hit CTRL+V, the image is pasted in the hidden div. The paster then grabs that image source and creates a file blob object. This file can be uploaded just like in Google Chrome. Not as user friendly because you have to paste outside the textarea but it works:

Paster.prototype.init=function() {

 var paster = this;

 if (window.Clipboard) {
  //IE11, Chrome, Safari
  this.pasteEl.onpaste=function(e){
   paster.handlePaste(paster, e);
  };
 } else
 {
  //On Firefox use the contenteditable div hack
  this.canvas = document.createElement('canvas');
  this.pasteCatcher = document.createElement("div");
  this.pasteCatcher.setAttribute("id", "paste_ff");
  this.pasteCatcher.setAttribute("contenteditable", "");
  this.pasteCatcher.style.cssText = 'opacity:0;position:fixed;top:0px;left:0px;';
  this.pasteCatcher.style.marginLeft = "-20px";
  document.body.appendChild(this.pasteCatcher);


  this.pasteEl.onblur=function(e) {
   paster.pasteCatcher.focus();
  };

  this.pasteCatcher.onpaste=function(e) {

   paster.findImageEl(paster);
  };
 }
};

 
Paster.prototype.dataURItoBlob=function(dataURI, callback) {
// convert base64 to raw binary data held in a string
// doesn't handle URLEncoded DataURIs - see SO answer #6850276 for code that does this

 console.log(dataURI);

 var byteString = atob(dataURI.split(',')[1]);
// separate out the mime component
 var mimeString = dataURI.split(',')[0].split(':')[1].split(';')[0]

// write the bytes of the string to an ArrayBuffer
 var ab = new ArrayBuffer(byteString.length);
 var ia = new Uint8Array(ab);
 for (var i = 0; i < byteString.length; i++) {
  ia[i] = byteString.charCodeAt(i);
 }

// write the ArrayBuffer to a blob, and you're done

 return new Blob([ia], {type: mimeString});
};

Paster.prototype.findImageEl = function(paster) {

 if (paster.pasteCatcher.children.length > 0) {

  var dataURI = paster.pasteCatcher.firstElementChild.src;
  if (dataURI) {
   if (dataURI.indexOf('base64') === -1) {
    alert("Sorry, with Firefox you can only paste local screenshots and files. Use Chrome or IE11 if you need paster feature.");
    return;
   }

   var file = paster.dataURItoBlob(dataURI);
   paster.uploadFile(paster, file);
  }

  paster.pasteCatcher.innerHTML = '';

 } else
 {
  setTimeout(function() {
   paster.findImageEl(paster);
  }, 100);
 }
};


Complete example

I hope this is useful for you! It sure is useful for the Group-Office Content Management System and E-mail module! Here's the fully functional example that should work on any php server. Save it to pasteupload.php:


<?php
/**
 *
 * The MIT License (MIT)
 *
 * Copyright (c) 2014 Intermesh BV 
 *
 * Permission is hereby granted, free of charge, to any person obtaining a copy
 * of this software and associated documentation files (the "Software"), to deal
 * in the Software without restriction, including without limitation the rights
 * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 * copies of the Software, and to permit persons to whom the Software is
 * furnished to do so, subject to the following conditions:
 *
 * The above copyright notice and this permission notice shall be included in
 * all copies or substantial portions of the Software.
 *
 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 * THE SOFTWARE.
 *
 */

if($_SERVER['REQUEST_METHOD']=='POST'){
 
 var_dump($_FILES);
 
 exit();
}
?>
<html>
<head>
<script type="text/javascript">

//Define paster object with contructor
var Paster = function(config) {

 for(var key in config){
  this[key]=config[key];
 }
 
 this.init();
};

Paster.prototype.pasteEl=null;

Paster.prototype.init=function() {

 var paster = this;

 if (window.Clipboard) {
  //IE11, Chrome, Safari
  this.pasteEl.onpaste=function(e){
   paster.handlePaste(paster, e);
  };
 } else
 {
  //On Firefox use the contenteditable div hack
  this.canvas = document.createElement('canvas');
  this.pasteCatcher = document.createElement("div");
  this.pasteCatcher.setAttribute("id", "paste_ff");
  this.pasteCatcher.setAttribute("contenteditable", "");
  this.pasteCatcher.style.cssText = 'opacity:0;position:fixed;top:0px;left:0px;';
  this.pasteCatcher.style.marginLeft = "-20px";
  document.body.appendChild(this.pasteCatcher);


  this.pasteEl.onblur=function(e) {
   paster.pasteCatcher.focus();
  };

  this.pasteCatcher.onpaste=function(e) {

   paster.findImageEl(paster);
  };
 }
};

 
Paster.prototype.dataURItoBlob=function(dataURI, callback) {
// convert base64 to raw binary data held in a string
// doesn't handle URLEncoded DataURIs - see SO answer #6850276 for code that does this

 console.log(dataURI);

 var byteString = atob(dataURI.split(',')[1]);
// separate out the mime component
 var mimeString = dataURI.split(',')[0].split(':')[1].split(';')[0]

// write the bytes of the string to an ArrayBuffer
 var ab = new ArrayBuffer(byteString.length);
 var ia = new Uint8Array(ab);
 for (var i = 0; i < byteString.length; i++) {
  ia[i] = byteString.charCodeAt(i);
 }

// write the ArrayBuffer to a blob, and you're done

 return new Blob([ia], {type: mimeString});
};

Paster.prototype.findImageEl = function(paster) {

 if (paster.pasteCatcher.children.length > 0) {

  var dataURI = paster.pasteCatcher.firstElementChild.src;
  if (dataURI) {
   if (dataURI.indexOf('base64') === -1) {
    alert("Sorry, with Firefox you can only paste local screenshots and files. Use Chrome or IE11 if you need paster feature.");
    return;
   }

   var file = paster.dataURItoBlob(dataURI);
   paster.uploadFile(paster, file);
  }

  paster.pasteCatcher.innerHTML = '';

 } else
 {
  setTimeout(function() {
   paster.findImageEl(paster);
  }, 100);
 }
};

Paster.prototype.processing = false; //some wierd chrome bug makes the paste event fire twice when using javascript prompt for the filename

Paster.prototype.handlePaste = function(paster, e) {

 //don't do this twice
 if (paster.processing) {
  return;
 }

 //loop through all clipBoardData items and upload it if it's a file.
 for (var i = 0; i < e.clipboardData.items.length; i++) {
  var item = e.clipboardData.items[i];
  if (item.kind === "file") {

   paster.processing = true;
   e.preventDefault();
   paster.uploadFile(paster, item.getAsFile());
  }
 }
};


Paster.prototype.uploadFile = function(paster, file) {

 var xhr = new XMLHttpRequest();

 //progress logging
 xhr.upload.onprogress = function(e) {
  var percentComplete = (e.loaded / e.total) * 100;

  console.log(percentComplete);
 };

 //called when finished
 xhr.onload = function() {
  if (xhr.status === 200) {
   alert("Sucess! Upload completed. PHP response will be put in the textarea.");
  } else {
   alert("Error! Upload failed");
  }

  paster.processing = false;
 };

 //error handling
 xhr.onerror = function() {
  alert("Error! Upload failed. Can not connect to server.");
 };

 //trigger a callback when it's successful
 xhr.onreadystatechange = function()
 {
  if (xhr.readyState === 4 && xhr.status === 200)
  {
   if (paster.callback) {
    paster.callback.call(paster.scope || paster, paster, xhr);
   }
  }
 };


 //prompt for the filename
 var filename = prompt("Please enter the file name", "Pasted image");

 if (filename) {

  //upload the file
  xhr.open("POST", "<?php echo $_SERVER['PHP_SELF']; ?>", {
   filename: filename,
   filetype: file.type
  });
  
  //send it as multipart/form-data
  var formData = new FormData();
  formData.append("pastedFile", file);
  xhr.send(formData);
 }
};
</script>
</head>
<body>
 
 <textarea id="text" style="width:500px;height:500px;">
 
On chrome paste a screenshot in here and it will be uploaded.

On firefox first click outside the text area and paste the screenshot with CTRL+V

 </textarea>
 
 
 <script>
 
 //Create the paster object and connect it to the textarea.
 var paster = new Paster({
  pasteEl:document.getElementById("text"),
  callback:function(paster, xhr){
   
   //put the response in the textarea
   paster.pasteEl.value=xhr.responseText;
  }
 });
 
 //focus the textarea
 paster.pasteEl.focus();
 
 </script>
 
</body>
</html>

Thursday, March 6, 2014

Synchronize your Android phone with Group-Office without using Microsoft ActiveSync

Synchronizing Group-Office Professional using Microsoft ActiveSync is the easiest to setup and works great. But some might need an alternative. For those who don't know Group-Office yet here's a screenshot of the calendar:





I myself used to own a Samsung Galaxy S2 and I always happily synchronized this with Group-Office using Microsoft ActiveSync. I recently got myself a new LG Nexus 5 and I immediately setup ActiveSync again. But I immediately missed two important features:
  1. No tasks synchronization anymore! The stock Android OS doesn't have a tasks app. 
  2. I couldn't search for older mail anymore. The Samsung Galaxy had a very nice feature to search for mail on the server. So even when you only sync two weeks of mail, you could still find an older message by keyword. 
So I started looking for alternatives:

E-mail with IMAP

The e-mail solution was easy. I just setup an IMAP account that works out of the box with Android and Group-Office. Here's a screenshot of the Android e-mail app:



Contacts and Calendar sync

I found “Caldav Sync Free Beta” and “CardDav Sync Free beta” in the Google play store that were both easy to setup and work great. They are open source and free of charge as well!

The apps are implemented as a sync adapter so they seamlessly integrate with the standard android contacts and calendar application. The screenshot below just shows the standard Android calendar because that's all you will see and that's just the way it should be!


Tasks

Synchronizing tasks with Group-Office proved to be more difficult. I tried “CalDAV Tasksync beta free” and it worked, but I think it's in an early stage of development because the interface is not very good yet. But it looks promising for the future.

I also tried some other tasks apps that didn't work at all. I then decided to remove “Caldav Sync Free Beta” and replace it with “CalDAV-Sync” because it also has a nice tasks app with it. This app costs € 2,59 but it's worth the money! It also integrates nicely with the android calendar app and the tasks application looks and works great too!

Here's a screenshot of the Tasks app that synchronizes with Group-Office:



Ending

In lots of cases synchronizing Group-Office with Microsoft ActiveSync will be easier as you only have to setup one account for all. But the CalDAV alternatives work great too. It's just that you have to enter your new password four times when you setup a new one ;)

But as I did my research on these apps I came across great open source projects that might be the base of a combined tasks, calendar and contacts sync app for Group-Office!