Tuesday, August 12, 2014

Configuring a fresh Ubuntu for Group-Office development

Unfortunately my Asus Zenbook Prime's SSD drive failed so I had to install a fresh system for Group-Office development (twice...). I thought it would be useful to post my steps right after a fresh Ubuntu 14.04 installation:
  1. Install java for netbeans
     $ sudo add-apt-repository ppa:webupd8team/java
     $ sudo apt-get update
     $ sudo apt-get install oracle-java7-installer
    
  2. Install Netbeans from site.
     $ sh ~/Downloads/netbeans-8.0-linux.sh
    
  3. Install Group-Office
     $ sudo echo -e "\n## Group-Office repository\ndeb \
    http://repos.groupoffice.eu/ fivezero main" | sudo tee -a /etc/apt/sources.list
    $ gpg --keyserver hkp://keyserver.ubuntu.com:11371 --recv-keys 01F1AE44 $ gpg --export --armor 01F1AE44 | sudo apt-key add - $ sudo apt-get update $ sudo apt-get install groupoffice-com
  4. Install subversion
     $ sudo apt-get install subversion
    
  5. Own the /var/www directory
     $ sudo chown -R mschering:mschering /var/www
    
  6. Checkout groupoffice repository
     $ svn co svn+ssh://mscheering@svn.code.sf.net/p/group-office/code/branches/groupoffice-6.0
    
  7. Launch Netbeans and open the Group-Office project and start coding.
  8. Install Firebug add-on in Firefox
  9. Install Dark look and feel plugin for Netbeans. I prefer a dark screen when staring at it for 8 hours a day ;)
  10. Install phpmyadmin
    $ sudo apt-get install phpmyadmin
    
    DANGER Set auto root login in /etc/phpmyadmin/config.php
        $cfg['Servers'][$i]['auth_type']    = 'config';
        $cfg['Servers'][$i]['user']         = 'root';  
        $cfg['Servers'][$i]['password']         = '';
    
    Enable:
        $cfg['Servers'][$i]['AllowNoPassword'] = TRUE;
    
  11. Disable local sending in SSH to get rid of warnings that locale is not found on remote server. Cange /etc/ssh/ssh_config:
    #    SendEnv LANG LC_*
    
  12. Install shutter for taking screenshots
    $ sudo apt-get install shutter
    

Some extra stuff for fun

  1. Install Variety for automatic wallpapers
     $ sudo add-apt-repository ppa:peterlevi/ppa
     $ sudo apt-get update
     $ sudo apt-get install variety
    
  2. Install Weather indicator
     $ sudo add-apt-repository ppa:atareao/atareao
     $ sudo apt-get update
     $ sudo apt-get install my-weather-indicator
    
  3. Mount WebDAV
     $ sudo apt-get install davfs2
    
    To enable webdav for non root users:
     $ sudo dpkg-reconfigure davfs2
    
     $ sudo vi /etc/fstab
    
    Add the line:
     https://intermesh.group-office.com/webdav/ /home/mschering/Group-Office davfs rw,user,noauto 0 0
    
    Add your user to the davfs2 group
     $ sudo adduser mschering davfs2
    
    Refresh your group membership without logging out:
     $ exec su -l mschering
    
    Create secrets file because it fails without:
     $ touch .davfs2/secrets
     $ chmod 600 .davfs2/secrets
    
    Create local dir:
     $ mkdir ~/Group-Office
     $ mount ~/Group-Office
    
  4. Install Skype I downloaded and installed Skype from their website which works fine except for the indicator icon. I had to install this package:
     $ sudo apt-get install sni-qt:i386
    

Thursday, July 24, 2014

New Intermesh frameworks


New technologies for the web are rising rapidly. As an application development company it's important to keep up with new technologies. I've explored and finally started to develop two frameworks that we will use for new applications.


On the server I want to create an API that's just communicating with clients through JSON requests. The server and client should be completely decoupled in my opinion. The server runs with the Intermesh PHP framework that features:
  • Api documentation with ApiGen
  • Model View Controller design pattern
  • ActiveRecord models for database ORM
  • Simple routing of HTTP requests to controller action methods
  • User authentication and role based permissions
  • Unit testing with phpunit
  • Composer support
On the client side I've developed an AngularJS framework for building an Angular client. The framework features:
  • API documentation with ngdocs
  • AngularJS uses the Model View Controller design pattern
  • bower support
  • services, directives and filters to communicate with the Intermesh PHP server API.
  • grunt to build the application, documentation and maintain index.html so it's not needed to add new scripts all the time.

Notes about the tools used

Composer

Composer is an essential tool. It takes care of:
  1. Installing and maintaining 3rd party libraries
  2. Autoloading PHP classes using the PSR-04 standard
There's no need to write your own autoloading script any more. Just install packages and you're good to go just by using the class names.

ApiGen

Documentation was always something that was pushed to a later date. So it's important to get this right at the start of a new project. I've tried various documentation generators PHPDoc, SAMI and ApiGen. The first two we're disappointing in the result. PHPDoc is more active and has more feature but the out of the box templates are just not good enough. ApiGen was the only one that produced a very usable result out of the box!

PHPUnit

I think the only and best tool for unit testing. Works flawlessly out of the box.

Bower

An essential tool that is similar to Composer but for javascript packages.

Grunt

Grunt is a javascript task runner. I used it for a couple of things. When I started with the AngularJS project I wanted to keep each component in a separate Javascript file to keep the code organized. Each time a new file was created I had to add it to the index.html file. As I'm a lazy programmer I don't like to do such annoying tasks :) Grunt can do that for me using the "fileblocks" module. Each time I add a file it's automatically inserted. Apart from that you also don't want to deploy your application like that. To improve performance grunt can concatenate and minify the Javascript, CSS and HTML for you.

ngdocs

Getting AngularJS documentation was tricky to setup. First of all there is some package confusion. I've tried ngdoc, ngdocs and dgeni. I couldn't get ngdoc to work at all so I moved on to ngdocs which worked well out of the box. I noticed the dgeni project is more active and used by AngularJS but it doesn't seem to be ready for community use yet. Perhaps it will grow out to be better than ngdocs.
The hardest thing to get going were the live inline examples. I used grunt to build a special script that includes all the Intermesh angular modules. With the "scripts" option for the ngdocs grunt task I was able to add those scripts to the docs so they started to work. See my Gruntfile.js for an example.
Here's an example of the ngdoc syntax for live examples:
https://github.com/Intermesh/intermesh-angular-framework/blob/master/src/intermesh/core/translate-provider.js

Getting started

To get started with development you will need to install:
I use Ubuntu to develop on. If you want to install on Ubuntu you can install git and the Node Package Manager npm with this command:

$ sudo apt-get install git npm 

I had to work around some bug with (see https://github.com/joyent/node/issues/3911):

$ sudo ln -s /usr/bin/nodejs /usr/bin/node 

Install grunt, bower with the node package manager (use -g for global install):

$ sudo npm install -g grunt-cli bower

Install composer:

$ curl -sS https://getcomposer.org/installer | php 
$ sudo mv composer.phar /usr/local/bin/composer
Then install the server and client examples. Installation instructions are on the project pages on Github:

 

Wednesday, May 28, 2014

Building a Group-Office client with AngularJS and Bootstrap

To demonstrate how easy it is to build a client for Group-Office, I've created an AngularJS app. It demonstrates that the server side PHP code of Group-Office is in fact one big JSON API that can be talked to with a completely decoupled client.

It's a very simple interface that was just for a "Proof of concept" but here's a screenshot:

E-mail App in Angularjs Group-Office client

When we'll replace the current ExtJS 3 based user interface we'll use the same approach.

I've created a GitHub repository for the Group-Office client here:

https://github.com/mschering/angular-groupoffice

Note that this client will only work with Group-Office 6.0.3 which you can get here:

http://sourceforge.net/projects/group-office/files/6.0/


I hope you like this example! You can also integrate these kind of API calls into your website or another application. Let me know what you think about it!

 

Note about CORS

If you run the Group-Office server on a different domain than this client you'll need to enable CORS. Otherwise you'll get errors like:

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote
resource at http://localhost/groupoffice. This can be fixed by moving the resource
to the same domain or enabling CORS.

You can read more about enabling CORS on Group-Office here:

https://www.group-office.com/wiki/CORS

Friday, May 16, 2014

Installing fast indexed search for IMAP e-mail

With Dovecot and apache SOLR it's possible to install very fast mail search. You can search through the entire message contents within fractions of a second.

I've used Ubuntu and Debian to set this up.

Tomcat SOLR


Install the tomcat solr server with the following command:

$ apt-get install solr-tomcat

Find the SOLR schema on page http://hg.dovecot.org/dovecot-2.2/file/tip/doc/solr-schema.xml

Make sure your dovecot version matches.

Download it like this:

$ wget http://hg.dovecot.org/dovecot-2.2/raw-file/e99cd21e1f92/doc/solr-schema.xml

Move the file to the right location:

$ cp solr-schema.xml /etc/solr/conf/schema.xml

$ service tomcat6 restart

Check if tomcat works by browsing to:

http://localhost:8080

Dovecot

Now setup Dovecot. Modify:

/etc/dovecot/conf.d/10-mail.conf:

...
# Space separated list of plugins to load for all services. Plugins specific to
# IMAP, LDA, etc. are added to this list in their own .conf files.
#mail_plugins =
mail_plugins = $mail_plugins fts fts_solr
...


/etc/dovecot/conf.d/90-plugin.conf:

...
plugin {
  fts = solr
  fts_solr = break-imap-search url=http://localhost:8080/solr/
}
...

The "break-imap-search" option will use Solr also for indexing TEXT and BODY searches. This makes your server non-IMAP-compliant, but it's what people want ;). This is always enabled in v2.1+.

Now when an IMAP client does a "SEARCH TEXT keyword command" you should see these log entries in /var/log/mail.log:

May 15 14:19:29 mail.example.com dovecot: indexer-worker(admin@intermesh.dev): Indexed 294 messages in INBOX

Enjoy the fast searches!

Friday, May 9, 2014

Setting up a Chat server with Prosody and Group-Office


A lot of customers requested a chat module for Group-Office. In my search for a
solution I stumbled upon a great XMPP/Jabber chat server called Prosody.

It was very easy to connect this server with Group-Office. This is a short guide
on how to do it.

Become root and add the Prosody repository (replace squeeze with the debian or ubuntu version you are using):

$ echo deb http://packages.prosody.im/debian squeeze main | tee -a /etc/apt/sources.list

Add their key file:

$ wget https://prosody.im/files/prosody-debian-packages.key -O- | apt-key add -

Update APT sources:

$ apt-get update

No install Prosody XMPP server:

$ apt-get install prosody luarocks

Install lpy becuase it's required for the external external auth module:

$ luarocks install lpty

Download mod_auth_external for Prosody here:

https://code.google.com/p/prosody-modules/source/browse/mod_auth_external/mod_auth_external.lua

Put it in /usr/lib/prosody/modules/

Now create /usr/local/bin/GroupOfficeProsodyAuth.php. This script handles the
authentication requests from Prosody and it uses the Group-Office framework.

#!/usr/bin/php
<?php
/**
 * Group-Office
 * 
 * Copyright Intermesh BV. 
 * This file is part of Group-Office. You should have received a copy of the
 * Group-Office license along with Group-Office. See the file /LICENSE.TXT
 *
 * If you have questions write an e-mail to info@intermesh.nl
 * 
 * @license AGPL/Proprietary http://www.group-office.com/LICENSE.TXT
 * @link http://www.group-office.com
 * @copyright Copyright Intermesh BV
 * @version $Id: GroupOfficeProsodyAuth.php 17690 2014-06-17 07:02:34Z mschering $
 * @author Merijn Schering 
 * @package GO.base
 * 
 * test:
 * 
 * auth:username:groupoffice.domain.com:password
 */

// the logfile to which to write, should be writeable by the user which is running the server
$sLogFile  = "/var/log/prosody/prosody.log";

// set true to debug if needed
$bDebug  = false;

$oAuth = new exAuth($sLogFile, $bDebug);

class exAuth
{

 private $sLogFile;

 private $bDebug;

 private $rLogFile;

 private function _includeGO($server){
  //Try to detect a host based
  if(class_exists('GO')){
   return true;
  }
  if(file_exists('/etc/groupoffice/'.$server.'/config.php')){
   define('GO_CONFIG_FILE', '/etc/groupoffice/'.$server.'/config.php');
   require(GO_CONFIG_FILE);
   require_once($config['root_path'].'GO.php');
  }else
  {

   require_once('/usr/share/groupoffice/GO.php');
  }
  
  $this->writeDebugLog("Config file: ".GO::config()->get_config_file()." db: ".GO::config()->db_name." ".GO::config()->debug);
 }
 
 public function __construct($sLogFile, $bDebug)
 {

  $this->sLogFile  = $sLogFile;
  $this->bDebug  = $bDebug;
  
  $this->rLogFile = fopen($this->sLogFile, "a") or die("Error opening log file: ". $this->sLogFile);

  $this->writeLog("start");

  do {  
   $sData = substr(fgets(STDIN),0,-1);
   
   if($sData){
    
    $this->writeDebugLog("received data: ". $sData);
    $aCommand = explode(":", $sData);
    if (is_array($aCommand)){
     switch ($aCommand[0]){
      case "isuser":
       if (!isset($aCommand[1])){
        $this->writeLog("invalid isuser command, no username given");
        $this->failure();
       } else {
   
        $sUser = str_replace(array("%20", "(a)"), array(" ", "@"), $aCommand[1]);
        $this->writeDebugLog("checking isuser for ". $sUser);

        $this->_includeGO($aCommand[2]);
        $user = \GO\Base\Model\User::model()->findSingleByAttribute('username', $sUser);

        if ($user){        
         $this->writeLog("valid user: ". $sUser);
         
         $this->success();

        } else {
         $this->writeLog("invalid user: ". $sUser);
         $this->failure();
        }
        
        
       }
      break;
      case "auth":
       
       if (sizeof($aCommand) != 4){
        $this->writeLog("invalid auth command, data missing");
        $this->failure();
       } else {
        

        $sUser = str_replace(array("%20", "(a)"), array(" ", "@"), $aCommand[1]);
        $sPassword = $aCommand[3];
        $this->writeDebugLog("doing auth for ". $sUser);



        $this->_includeGO($aCommand[2]);

        $user = \GO::session()->login($sUser, $sPassword, false);

     
        if ($user) {
      
         $this->writeLog("authentificated user ". $sUser ." domain ". $aCommand[2]);
         $this->success();
        } else {
      
         $this->writeLog("authentification failed for user ". $sUser ." domain ". $aCommand[2]);
         $this->failure();
        }
       
       }
      break;
      case "setpass":
       // postavljanje zaporke, onemoguceno
       $this->writeLog("setpass command disabled");
       $this->failure();
      break;
      default:
       // ako je uhvaceno ista drugo
       $this->writeLog("unknown command ". $aCommand[0]);
       $this->failure();
      break;
     }
    } else {
     $this->writeDebugLog("invalid command string");
     $this->failure();
    }
   }

   unset($aCommand);
  } while (true);
 }

 public function __destruct()
 {
  $this->writeLog("stop");  
 }

 private function writeLog($sMessage)
 {
  if (is_resource($this->rLogFile)) {
   fwrite($this->rLogFile, date("r") ." [external_auth] ". $sMessage ."\n");
  }
 }

 private function writeDebugLog($sMessage)
 {
  if ($this->bDebug){
   $this->writeLog( date("r")." [external_auth_debug] ".$sMessage);
  }
 }
 
 
 private function success(){
  echo "1\n";
  if(class_exists('\GO')){
   \GO::unsetDbConnection();
  }

 }
 
 private function failure(){
  echo "0\n";
  if(class_exists('\GO')){
   \GO::unsetDbConnection();
  }
 }


 
}

Make this script executable:

$ chmod +x /usr/local/bin/GroupOfficeProsodyAuth.php

Edit /etc/prosody/prosody.cfg.lua and set:

authentication = "external"
external_auth_command = "/usr/local/bin/GroupOfficeProsodyAuth.php"
external_auth_protocol = "generic"

Also setup your VirtualHost with the domain you run Group-Office on.
(See https://prosody.im/doc/configure for more info on configuring Prosody).

Example config file:
-- Prosody XMPP Server Configuration
--
-- Information on configuring Prosody can be found on our
-- website at http://prosody.im/doc/configure
--
-- Tip: You can check that the syntax of this file is correct
-- when you have finished by running: luac -p prosody.cfg.lua
-- If there are any errors, it will let you know what and where
-- they are, otherwise it will keep quiet.
--
-- Good luck, and happy Jabbering!


---------- Server-wide settings ----------
-- Settings in this section apply to the whole server and are the default settings
-- for any virtual hosts

-- This is a (by default, empty) list of accounts that are admins
-- for the server. Note that you must create the accounts separately
-- (see http://prosody.im/doc/creating_accounts for info)
-- Example: admins = { "user1@example.com", "user2@example.net" }
admins = { }

-- Enable use of libevent for better performance under high load
-- For more information see: http://prosody.im/doc/libevent
--use_libevent = true;

-- This is the list of modules Prosody will load on startup.
-- It looks for mod_modulename.lua in the plugins folder, so make sure that exists too.
-- Documentation on modules can be found at: http://prosody.im/doc/modules
modules_enabled = {

        -- Generally required
                "roster"; -- Allow users to have a roster. Recommended ;)
                "saslauth"; -- Authentication for clients and servers. Recommended if you want to log in.
                "tls"; -- Add support for secure TLS on c2s/s2s connections
                "dialback"; -- s2s dialback support
                "disco"; -- Service discovery
                "posix"; -- POSIX functionality, sends server to background, enables syslog, etc.

        -- Not essential, but recommended
                "private"; -- Private XML storage (for room bookmarks, etc.)
                "vcard"; -- Allow users to set vCards

        -- These are commented by default as they have a performance impact
                --"privacy"; -- Support privacy lists
                --"compression"; -- Stream compression (requires the lua-zlib package installed)

        -- Nice to have
                "version"; -- Replies to server version requests
                "uptime"; -- Report how long server has been running
                "time"; -- Let others know the time here on this server
                "ping"; -- Replies to XMPP pings with pongs
                "pep"; -- Enables users to publish their mood, activity, playing music and more
"register"; -- Allow users to register on this server using a client and change passwords

        -- Admin interfaces
                "admin_adhoc"; -- Allows administration via an XMPP client that supports ad-hoc commands
                --"admin_telnet"; -- Opens telnet console interface on localhost port 5582

        -- HTTP modules
                "bosh"; -- Enable BOSH clients, aka "Jabber over HTTP"
                --"http_files"; -- Serve static files from a directory over HTTP

        -- Other specific functionality
                "groups"; -- Shared roster support
                --"announce"; -- Send announcement to all online users
                --"welcome"; -- Welcome users who register accounts
                --"watchregistrations"; -- Alert admins of registrations
                --"motd"; -- Send a message to users when they log in
                --"legacyauth"; -- Legacy authentication. Only used by some old clients and bots.
};

-- These modules are auto-loaded, but should you want
-- to disable them then uncomment them here:
modules_disabled = {
        -- "offline"; -- Store offline messages
        -- "c2s"; -- Handle client connections
        -- "s2s"; -- Handle server-to-server connections
};

-- Disable account creation by default, for security
-- For more information see http://prosody.im/doc/creating_accounts
allow_registration = false;

-- These are the SSL/TLS-related settings. If you don't want
-- to use SSL/TLS, you may comment or remove this
-- ssl = {
--      key = "/etc/prosody/certs/localhost.key";
--      certificate = "/etc/prosody/certs/localhost.crt";
--}

ssl = {
                key = "/etc/prosody/certs/group-office.com.key";
                certificate = "/etc/prosody/certs/group-office.com.crt";
        }

-- Force clients to use encrypted connections? This option will
-- prevent clients from authenticating unless they are using encryption.

c2s_require_encryption = false

-- Force certificate authentication for server-to-server connections?
-- This provides ideal security, but requires servers you communicate
-- with to support encryption AND present valid, trusted certificates.
-- NOTE: Your version of LuaSec must support certificate verification!
-- For more information see http://prosody.im/doc/s2s#security

s2s_secure_auth = false

-- Many servers don't support encryption or have invalid or self-signed
-- certificates. You can list domains here that will not be required to
-- authenticate using certificates. They will be authenticated using DNS.

--s2s_insecure_domains = { "gmail.com" }

-- Even if you leave s2s_secure_auth disabled, you can still require valid
-- certificates for some domains by specifying a list here.

--s2s_secure_domains = { "jabber.org" }

-- Required for init scripts and prosodyctl
pidfile = "/var/run/prosody/prosody.pid"

-- Select the authentication backend to use. The 'internal' providers
-- use Prosody's configured data storage to store the authentication data.
-- To allow Prosody to offer secure authentication mechanisms to clients, the
-- default provider stores passwords in plaintext. If you do not trust your
-- server please see http://prosody.im/doc/modules/mod_auth_internal_hashed
-- for information about using the hashed backend.

-- authentication = "internal_plain"

authentication = "external"
external_auth_command = "/usr/local/bin/GroupOfficeProsodyAuth.php"
external_auth_protocol = "generic"

--authentication = "anonymous"
cross_domain_bosh = true
-- Select the storage backend to use. By default Prosody uses flat files
-- in its configured data directory, but it also supports more backends
-- through modules. An "sql" backend is included by default, but requires
-- additional dependencies. See http://prosody.im/doc/storage for more info.

--storage = "sql" -- Default is "internal"

-- For the "sql" backend, you can uncomment *one* of the below to configure:
--sql = { driver = "SQLite3", database = "prosody.sqlite" } -- Default. 'database' is the filename.
--sql = { driver = "MySQL", database = "prosody", username = "prosody", password = "secret", host = "localhost" }
--sql = { driver = "PostgreSQL", database = "prosody", username = "prosody", password = "secret", host = "localhost" }

-- Logging configuration
-- For advanced logging see http://prosody.im/doc/logging
log = {
        info = "/var/log/prosody/prosody.log"; -- Change 'info' to 'debug' for verbose logging
        error = "/var/log/prosody/prosody.err";
        "*syslog";
}

----------- Virtual hosts -----------
-- You need to add a VirtualHost entry for each domain you wish Prosody to serve.
-- Settings under each VirtualHost entry apply *only* to that host.

--VirtualHost "localhost"

VirtualHost "intermesh.group-office.com"
        groups_file = "/home/govhosts/intermesh.group-office.com/data/chat/groups.txt";
        -- Assign this host a certificate for TLS, otherwise it would use the one
        -- set in the global section (if any).
        -- Note that old-style SSL on port 5223 only supports one certificate, and will always
        -- use the global one.
        ssl = {
                key = "/etc/prosody/certs/group-office.com.key";
                certificate = "/etc/prosody/certs/group-office.com.crt";
        }

------ Components ------
-- You can specify components to add hosts that provide special services,
-- like multi-user conferences, and transports.
-- For more information on components, see http://prosody.im/doc/components

---Set up a MUC (multi-user chat) room server on conference.example.com:
Component "conference.intermesh.group-office.com" "muc"

-- Set up a SOCKS5 bytestream proxy for server-proxied file transfers:
--Component "proxy.example.com" "proxy65"

---Set up an external component (default component port is 5347)
--
-- External components allow adding various services, such as gateways/
-- transports to other networks like ICQ, MSN and Yahoo. For more info
-- see: http://prosody.im/doc/components#adding_an_external_component
--
--Component "gateway.example.com"
--      component_secret = "password"

                                           
Restart the Prosody XMPP server:

$ service prosody restart

Now try to connect with your XMPP/Jabber client.

Use your Group-Office user, password and domain.

You can check /var/log/prosody/ for log files.

Also make sure config.php of Group-Office is readable by the prosody user!
Now enable the chat module in Group-Office.

Update: I found the excellent program Converse.js that could be integrated in Group-Office 6.0 easily:

Additionally, I used the Prosody "groups" module to automatically deliver all Group-Office Chat users as contacts to the chat clients.

Wednesday, April 9, 2014

Migrating email to your own mail server

To move your email to a new mail server you'll need setup the new server with all the existing accounts and copy the mailboxes.

Since this task is often at hand we had to find an efficient way to do this quickly so we wrote a script that would:
  1. Create all the existing mailboxes on the new server.
  2. Sync all the mail from the old server to the new one.
  3. Create Users that have access to this mailboxes.


For this you will need:
  • Group Office
  • ImapSync
  • The script at the bottom of this article


ImapSync packages are available for Debian here: http://archive.debian.net/etch/net/imapsync

Step 1: Setup your new mail server (together with Group Office)

The new email server needs to be installed first.
We have created a package for Debian/Ubuntu that would install and configure a postfix/dovecot mail server (almost) automatically

See our Wiki page for a link to these apt-get packages that would install and configure Group Office, Postfix (MTA for SMTP) and Dovecote (MDA for IMAP)
https://www.group-office.com/wiki/Installing_on_Debian_or_Ubuntu

When Group Office and the Mail server are installed open it in the browser and install the Postfixadmin module by going to Start menu (top right corner) -> Modules -> Install (top left) and choice Postfix Admin. This module lets you configure virtual mail domain and mailboxes. But don't create then yet. This will be done automatically in the next step.

Step 2: Create mailboxes on your new server

First you'll need a list of the existing mailboxes on the source server you want to fetch the mail from. You'll need to following details:
  1. Email
  2. Password
  3. First name
  4. Middle name
  5. Last name
The above information will be used to create mailboxes in the new server (with the same username and password) and the fetch the mail from the old server. If you don't know the password you'll have to change it because you won't be able to read to mailbox without it.

Put the mailbox information into a mailboxes.csv file like this.
michael@group-office.com,S3cRetPassWD,Michael,de,Hart
merijn@group-office.com,S3cRetPassWD,Merijn,,Schering
wesley@group-office.com,S3cRetPassWD,Wesley,,Smits
wilmar@group-office.com,S3cRetPassWD,Wilmar,van,Beusekom


Step 3: Running the mailbox migration script

After creating the CSV file that contain the information for the mailboxes The script at the bottom of this article will do the rest of the work.

Run it on the command line (not in the browser)

$ php syncmail.php

The script will need to know the path the you mailboxes.csv file. It it is in the same directory press enter. Otherwise enter the path to this file first. It the file is found it will show you a list of mailboxes it is going to migrate and asks you the following question:
- Do you want to create to following mailboxes? [y/N]:

Type 'y' and press enter

This won't take long and the following question will be asked:
- Fill mailboxes from another imap host? [y/N]:

Before you continue confirm the mailboxes are created on your new mail server by logging in to Group Office and open the Postfix admin module you've installed in Step 1

Type 'y' and press enter

ImapSync will need to know the target and source server address to fetch from and copy to. You'll be prompted for this information:
Enter IMAP host to fetch from: mail.oldserver.com
Port number [143]:
Enter IMAP host to copy to: [localhost]:
Port number [143]:

When ImapSync is installed correctly the script will start syncing all the created mailboxes (this could take a while)

Step 4: Creating Group Office user accounts for each mailbox (optional)

The last part the script takes care of is creating Group Office users that may access these mailboxes. If this is already setup our you would rather do this manualy just hit 'n'

Do you want to add/create users with these mail accounts in this Group Office installation [y/N]:

When entering 'y' the script will create Group Office users accounts. The username will be what is before the @ of the mailbox and the password will be the same.

<?php
/*
 * The MIT License (MIT)
 * 
 * Copyright (c) 2014 Intermesh BV 
 * 
 * Permission is hereby granted, free of charge, to any person obtaining a copy
 * of this software and associated documentation files (the "Software"), to deal
 * in the Software without restriction, including without limitation the rights
 * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 * copies of the Software, and to permit persons to whom the Software is
 * furnished to do so, subject to the following conditions:
 * 
 * The above copyright notice and this permission notice shall be included in
 * all copies or substantial portions of the Software.
 * 
 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 * THE SOFTWARE.
 */

/**
 * Script askes the following questions:
 * 
 * - Do you want to create to following mailboxes? [y/N]: 
 * - Fill mailboxes from another imap host? [y/N]:  
 * - Do you want to add/create users with these mailaccounts in this GroupOffice installation [y/N]:
 * 
 * RUN ON CLI AS ROOT !!
 * YOU MOST LIKELY WANT TO CHANGE THE PATH TO GO AND CONFIG.PHP
 * 
 * CSV should look like this:
 * email,password,firstname,middlename,lastname\n
 * 
 * @author Michael de Hart 
 * @copyright Copyright Intermesh BV.
 */

//first argument is the path to config.php
$conf_path = '/etc/groupoffice/config.php';

define('GO_CONFIG_FILE',$conf_path);
define('NOLOG', true); // stop groupoffice from logging. 
require_once('/usr/share/groupoffice/GO.php');
GO::session()->runAsRoot();

$syncer = new ImapSync();
$syncer->start();

class ImapSync {

 protected $imap_sync = '/usr/bin/imapsync'; //binary
 protected $domain = 'superfun.nl';
 protected $quota = 524288; //quota in kb = (512MB)
 protected $source = array(
  'host'=>'',
  'port'=>'143',
 );
 protected $target = array(
  'host'=>'localhost',
  'port'=>'143',
  'smtp_host'=>'localhost',
  'smtp_port'=>'25',
 );
 // CSV Config
 protected $csv = array(
  'path'=>'mailboxes.csv',
  'delimiter'=>",",
  'enclosure'=>'"'
 );
 // Column configuration for Mailbox CSV file
 protected $col = array(
  'email'=>0,
  'password'=>1,
  'first_name'=>2,
  'middle_name'=>3,
  'last_name'=>4,
 );
 protected $log_file = 'imapsync';
 private $records = array();

 public function start() {

  try {
   $handle = fopen("php://stdin","r");
   
   echo "This script will create mailboxes and users based on a CSV file?\n"
   . "Enter the path to the CSV file to continue: [".$this->csv['path']."]: ";
   $line = trim(fgets($handle));
   if(!empty($line)){
    $this->csv['path'] = $line;
   }
   $this->records = $this->loadCsv();
   
   if(GO::modules()->isInstalled('postfixadmin')) {
    echo implode(array_keys($this->records),"\n")."\n".
    "Do you want to create to following mailboxes? [y/N]: ";
    $line = fgets($handle);
    if(trim($line) == 'y'){
     echo "\nCreating mailboxes...\n";
     $this->createMailBoxes();
    }

    echo "Fill mailboxes from another imap host? [y/N]: ";
    $line = fgets($handle);
    if(trim($line) == 'y'){
     $this->configureSync($handle);
     echo "\nSyncing IMAP mailboxes...\n";
     foreach($this->records as $record) {
      $this->syncImap($record);
     }
    }
   }

   echo "Do you want to add/create users with these mailaccounts in this GroupOffice installation [y/N]: ";
   $line = trim(fgets($handle));
   if($line == 'y'){
    echo "\nEnter SMTP host for the accounts [".$this->target['smtp_host']."]: ";
    $line = trim(fgets($handle));
    if(!empty($line))
     $this->target['smtp_host'] = $line;
    echo "\nPort number [".$this->target['smtp_port']."]: ";
    $line = trim(fgets($handle));
    if(!empty($line))
     $this->target['smtp_port'] = $line;
    echo "\nAdding/creating users and accounts...\n";
    $this->createUsersWithMailAccounts();
   }
   echo "All done!\n";

  } catch(Exception $e) {
   echo $e->getMessage()."\n";
  }
 }
 
 private function configureSync($handle) {
  echo "\nEnter IMAP host to fetch from: ";
  $line = trim(fgets($handle));
  if(!empty($line))
   $this->source['host'] = $line;
  echo "\nPort number [".$this->source['port']."]: ";
  $line = trim(fgets($handle));
  if(!empty($line))
   $this->source['port'] = $line;
  echo "\nEnter IMAP host to copy to [".$this->target['host']."]: ";
  $line = trim(fgets($handle));
  if(!empty($line))
   $this->target['host'] = $line;
  echo "\nPort number [".$this->target['port']."]: ";
  $line = trim(fgets($handle));
  if(!empty($line))
   $this->target['port'] = $line;
 }
 
 //Read CSV file to an array
 private function loadCsv() {
  $records=array();
  $fp = @fopen($this->csv['path'], "r");
  if(!$fp)
   die("Can't find: " . $this->csv['path']."\n");
  while ($record = fgetcsv($fp, 4096, $this->csv['delimiter'], $this->csv['enclosure'])) {
   $assocRecord = array();
   foreach($this->col as $key => $number)
    $assocRecord[$key] = $record[$number];
   $records[$assocRecord['email']] = $assocRecord;
  }
  //Get domain from last email account
  if(isset($records[0])) {
   $this->domain = end(explode('@',$records[0]['email'],2));
  }
  return $records;
 }

 //Fetch all Mail and copy to new IMAP server using ImapSync
 private function syncImap($record){
  echo "Syncing " . $record['email'] . "...\n\n";
  $cmd = $this->imap_sync.' --syncinternaldates --authmech1 LOGIN --authmech2 LOGIN ' .
   '--host1="'.$this->source['host'].'" --user1="'.$record['email'].'" --password1="'.$record['password'].'" ' .
   '--host2="'.$this->target['host'].'" --user2="'.$record['email'].'" --password2="'.$record['password'].'" '.
   '--subscribe --allowsizemismatch --nofoldersizes '.
   '--sep1 / --sep2 . --regextrans2 "s,/,_,g"'; // --folder "INBOX"
  if(!empty($this->log_file))
   $cmd .= ' > '.__DIR__.'/'.$this->log_file.'_'.date('Y-m-d\ H:i:s').'.log';
  system($cmd);
 }

 // Create GroupOffice Mailboxes
 private function createMailBoxes() {
  $domain = GO_Postfixadmin_Model_Domain::model()->findSingleByAttribute('domain',$this->domain);
  if(empty($domain)) {
   $domain = new GO_Postfixadmin_Model_Domain();
   $domain->transport='virtual';
   $domain->active='1';
   $domain->domain=$this->domain;
   $domain->default_quota=$this->quota;
   $domain->user_id=1;
   if(!$domain->save())
    throw new Exception('Error while saving mailbox model: '.$record['email']."\n".implode("\n",$domain->getValidationErrors()));
  }
  foreach($this->records as $record) {
   $username = current(explode("@", $record['email']));
   $mailbox = GO_Postfixadmin_Model_Mailbox::model()->findSingleByAttribute('username',$username);
   if(!$mailbox){
    $mailbox = new GO_Postfixadmin_Model_Mailbox();
    $mailbox->domain_id = $domain->id;
    $mailbox->username = $record['email']; //$username;
    $mailbox->quota = $domain->default_quota;
   }
   $mailbox->name = $record['email'];
   $mailbox->password = $record['password'];
   echo "Saving mailbox ".$mailbox->username." with quota ".($mailbox->quota/1024)." MB...\n";
   if(!$mailbox->save())
    throw new Exception('Error while saving mailbox model: '.$record['email']."\n".implode("\n",$mailbox->getValidationErrors())); 
  }
  echo "All mailboxes are created, time to fetch from source mail host...\n";
  
 }

 private function createUsersWithMailAccounts() {
  foreach($this->records as $record) {
   $username = current(explode("@", $record['email']));

   // Check if the user exists in Group-Office and if it doesn't create it.
   $user = GO_Base_Model_User::model()->findSingleByAttribute('username',$username);
   if(!$user){
    $user = new GO_Base_Model_User();
    $user->first_name = $record['first_name'];
    $user->middle_name = $record['middle_name'];
    $user->last_name = $record['last_name'];
    $user->email = $record['email'];
    $user->enabled = 1;
   }
   $user->username = $username;
   $user->password = $record['password'];
   if(!$user->save())
    throw new Exception('Error while saving user: '.$username."\n".implode("\n",$user->getValidationErrors()));

   // Create an e-mail account for the user
   $account = GO_Email_Model_Account::model()->findSingleByAttributes(array(
    'username'=>$record['email'],
    'user_id'=>$user->id
   ));
   if(!$account) {
    $account = new GO_Email_Model_Account();
    $account->user_id = $user->id;
    $account->mbroot = '';
    $account->use_ssl = 0;
    $account->type = 'imap';
    $account->smtp_encryption = '';
    $account->smtp_username = '';
    $account->smtp_password = '';
    $account->name = $user->email;
    $account->email = $user->email;
   }
   $account->host = $this->target['host'];
   $account->port = $this->target['post'];
   $account->smtp_host = $this->target['smtp_host'];
   $account->smtp_port = $this->target['smtp_port'];
   $account->username = $record['email'];
   $account->password = $record['password'];
   if(!$account->save())
    throw new Exception('Error while saving account: '.$username."\n".implode("\n",$account->getValidationErrors()));

   echo $username."\n";
  }
 }

}

Wednesday, March 26, 2014

Automatically upload a screenshot from the clipboard on paste using Javascript in Google Chrome, Internet Explorer 11 and Firefox

I was working on a new documentation site and had to make a lot of screenshots. I thought it would be useful to be able to hit print screen and then paste the image directly into Group-Office using the CTRL+V shortcut. Luckily with HTML5 you can do that!

With Google Chrome and Internet Explorer 11 you can use the Clipboard API. I created a "Paster" class that you can pass a DOM object and a callback function. So when you paste an image into that object the file is uploaded to a PHP script and then the callback function is called with the response.

In this working example it will call the PHP page itself and then put the response in the textarea. It should work on any PHP enabled web server.

You can add an onpaste event to any DOM element in Google Chrome. Then the "Paster" loops through all clipboard items and checks if there's an image:


Paster.prototype.handlePaste = function(paster, e) {

 //don't do this twice
 if (paster.processing) {
  return;
 }

 //loop through all clipBoardData items and upload it if it's a file.
 for (var i = 0; i < e.clipboardData.items.length; i++) {
  var item = e.clipboardData.items[i];
  if (item.kind === "file") {

   paster.processing = true;
   e.preventDefault();
   paster.uploadFile(paster, item.getAsFile());
  }
 }
};

Then it uploads the file using an "XMLHttpRequest" call. When the upload is finished the callback function is called.

Paster.prototype.uploadFile = function(paster, file) {

 var xhr = new XMLHttpRequest();

 //progress logging
 xhr.upload.onprogress = function(e) {
  var percentComplete = (e.loaded / e.total) * 100;

  console.log(percentComplete);
 };

 //called when finished
 xhr.onload = function() {
  if (xhr.status === 200) {
   alert("Sucess! Upload completed. PHP response will be put in the textarea.");
  } else {
   alert("Error! Upload failed");
  }

  paster.processing = false;
 };

 //error handling
 xhr.onerror = function() {
  alert("Error! Upload failed. Can not connect to server.");
 };

 //trigger a callback when it's successful
 xhr.onreadystatechange = function()
 {
  if (xhr.readyState === 4 && xhr.status === 200)
  {
   if (paster.callback) {
    paster.callback.call(paster.scope || paster, paster, xhr);
   }
  }
 };


 //prompt for the filename
 var filename = prompt("Please enter the file name", "Pasted image");

 if (filename) {

  //upload the file
  xhr.open("POST", "", {
   filename: filename,
   filetype: file.type
  });
  var formData = new FormData();
  formData.append("pastedFile", file);
  xhr.send(formData);
 }
}

Firefox

So far it was pretty simple! But then I started testing this in Firefox. Unfortunately it didn't work because Firefox doesn't support files in the clipboard. I came up with a work around though.

You can paste images in a content editable div in Firefox. The blob data is then available in the image source tag: <img src="data:image/png;base64,DATA" />. So in my workaround I create a hidden content editable div. When ever the textarea blurs, it will focus this hidden element. So when you click out of the textarea and then hit CTRL+V, the image is pasted in the hidden div. The paster then grabs that image source and creates a file blob object. This file can be uploaded just like in Google Chrome. Not as user friendly because you have to paste outside the textarea but it works:

Paster.prototype.init=function() {

 var paster = this;

 if (window.Clipboard) {
  //IE11, Chrome, Safari
  this.pasteEl.onpaste=function(e){
   paster.handlePaste(paster, e);
  };
 } else
 {
  //On Firefox use the contenteditable div hack
  this.canvas = document.createElement('canvas');
  this.pasteCatcher = document.createElement("div");
  this.pasteCatcher.setAttribute("id", "paste_ff");
  this.pasteCatcher.setAttribute("contenteditable", "");
  this.pasteCatcher.style.cssText = 'opacity:0;position:fixed;top:0px;left:0px;';
  this.pasteCatcher.style.marginLeft = "-20px";
  document.body.appendChild(this.pasteCatcher);


  this.pasteEl.onblur=function(e) {
   paster.pasteCatcher.focus();
  };

  this.pasteCatcher.onpaste=function(e) {

   paster.findImageEl(paster);
  };
 }
};

 
Paster.prototype.dataURItoBlob=function(dataURI, callback) {
// convert base64 to raw binary data held in a string
// doesn't handle URLEncoded DataURIs - see SO answer #6850276 for code that does this

 console.log(dataURI);

 var byteString = atob(dataURI.split(',')[1]);
// separate out the mime component
 var mimeString = dataURI.split(',')[0].split(':')[1].split(';')[0]

// write the bytes of the string to an ArrayBuffer
 var ab = new ArrayBuffer(byteString.length);
 var ia = new Uint8Array(ab);
 for (var i = 0; i < byteString.length; i++) {
  ia[i] = byteString.charCodeAt(i);
 }

// write the ArrayBuffer to a blob, and you're done

 return new Blob([ia], {type: mimeString});
};

Paster.prototype.findImageEl = function(paster) {

 if (paster.pasteCatcher.children.length > 0) {

  var dataURI = paster.pasteCatcher.firstElementChild.src;
  if (dataURI) {
   if (dataURI.indexOf('base64') === -1) {
    alert("Sorry, with Firefox you can only paste local screenshots and files. Use Chrome or IE11 if you need paster feature.");
    return;
   }

   var file = paster.dataURItoBlob(dataURI);
   paster.uploadFile(paster, file);
  }

  paster.pasteCatcher.innerHTML = '';

 } else
 {
  setTimeout(function() {
   paster.findImageEl(paster);
  }, 100);
 }
};


Complete example

I hope this is useful for you! It sure is useful for the Group-Office Content Management System and E-mail module! Here's the fully functional example that should work on any php server. Save it to pasteupload.php:


<?php
/**
 *
 * The MIT License (MIT)
 *
 * Copyright (c) 2014 Intermesh BV 
 *
 * Permission is hereby granted, free of charge, to any person obtaining a copy
 * of this software and associated documentation files (the "Software"), to deal
 * in the Software without restriction, including without limitation the rights
 * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 * copies of the Software, and to permit persons to whom the Software is
 * furnished to do so, subject to the following conditions:
 *
 * The above copyright notice and this permission notice shall be included in
 * all copies or substantial portions of the Software.
 *
 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 * THE SOFTWARE.
 *
 */

if($_SERVER['REQUEST_METHOD']=='POST'){
 
 var_dump($_FILES);
 
 exit();
}
?>
<html>
<head>
<script type="text/javascript">

//Define paster object with contructor
var Paster = function(config) {

 for(var key in config){
  this[key]=config[key];
 }
 
 this.init();
};

Paster.prototype.pasteEl=null;

Paster.prototype.init=function() {

 var paster = this;

 if (window.Clipboard) {
  //IE11, Chrome, Safari
  this.pasteEl.onpaste=function(e){
   paster.handlePaste(paster, e);
  };
 } else
 {
  //On Firefox use the contenteditable div hack
  this.canvas = document.createElement('canvas');
  this.pasteCatcher = document.createElement("div");
  this.pasteCatcher.setAttribute("id", "paste_ff");
  this.pasteCatcher.setAttribute("contenteditable", "");
  this.pasteCatcher.style.cssText = 'opacity:0;position:fixed;top:0px;left:0px;';
  this.pasteCatcher.style.marginLeft = "-20px";
  document.body.appendChild(this.pasteCatcher);


  this.pasteEl.onblur=function(e) {
   paster.pasteCatcher.focus();
  };

  this.pasteCatcher.onpaste=function(e) {

   paster.findImageEl(paster);
  };
 }
};

 
Paster.prototype.dataURItoBlob=function(dataURI, callback) {
// convert base64 to raw binary data held in a string
// doesn't handle URLEncoded DataURIs - see SO answer #6850276 for code that does this

 console.log(dataURI);

 var byteString = atob(dataURI.split(',')[1]);
// separate out the mime component
 var mimeString = dataURI.split(',')[0].split(':')[1].split(';')[0]

// write the bytes of the string to an ArrayBuffer
 var ab = new ArrayBuffer(byteString.length);
 var ia = new Uint8Array(ab);
 for (var i = 0; i < byteString.length; i++) {
  ia[i] = byteString.charCodeAt(i);
 }

// write the ArrayBuffer to a blob, and you're done

 return new Blob([ia], {type: mimeString});
};

Paster.prototype.findImageEl = function(paster) {

 if (paster.pasteCatcher.children.length > 0) {

  var dataURI = paster.pasteCatcher.firstElementChild.src;
  if (dataURI) {
   if (dataURI.indexOf('base64') === -1) {
    alert("Sorry, with Firefox you can only paste local screenshots and files. Use Chrome or IE11 if you need paster feature.");
    return;
   }

   var file = paster.dataURItoBlob(dataURI);
   paster.uploadFile(paster, file);
  }

  paster.pasteCatcher.innerHTML = '';

 } else
 {
  setTimeout(function() {
   paster.findImageEl(paster);
  }, 100);
 }
};

Paster.prototype.processing = false; //some wierd chrome bug makes the paste event fire twice when using javascript prompt for the filename

Paster.prototype.handlePaste = function(paster, e) {

 //don't do this twice
 if (paster.processing) {
  return;
 }

 //loop through all clipBoardData items and upload it if it's a file.
 for (var i = 0; i < e.clipboardData.items.length; i++) {
  var item = e.clipboardData.items[i];
  if (item.kind === "file") {

   paster.processing = true;
   e.preventDefault();
   paster.uploadFile(paster, item.getAsFile());
  }
 }
};


Paster.prototype.uploadFile = function(paster, file) {

 var xhr = new XMLHttpRequest();

 //progress logging
 xhr.upload.onprogress = function(e) {
  var percentComplete = (e.loaded / e.total) * 100;

  console.log(percentComplete);
 };

 //called when finished
 xhr.onload = function() {
  if (xhr.status === 200) {
   alert("Sucess! Upload completed. PHP response will be put in the textarea.");
  } else {
   alert("Error! Upload failed");
  }

  paster.processing = false;
 };

 //error handling
 xhr.onerror = function() {
  alert("Error! Upload failed. Can not connect to server.");
 };

 //trigger a callback when it's successful
 xhr.onreadystatechange = function()
 {
  if (xhr.readyState === 4 && xhr.status === 200)
  {
   if (paster.callback) {
    paster.callback.call(paster.scope || paster, paster, xhr);
   }
  }
 };


 //prompt for the filename
 var filename = prompt("Please enter the file name", "Pasted image");

 if (filename) {

  //upload the file
  xhr.open("POST", "<?php echo $_SERVER['PHP_SELF']; ?>", {
   filename: filename,
   filetype: file.type
  });
  
  //send it as multipart/form-data
  var formData = new FormData();
  formData.append("pastedFile", file);
  xhr.send(formData);
 }
};
</script>
</head>
<body>
 
 <textarea id="text" style="width:500px;height:500px;">
 
On chrome paste a screenshot in here and it will be uploaded.

On firefox first click outside the text area and paste the screenshot with CTRL+V

 </textarea>
 
 
 <script>
 
 //Create the paster object and connect it to the textarea.
 var paster = new Paster({
  pasteEl:document.getElementById("text"),
  callback:function(paster, xhr){
   
   //put the response in the textarea
   paster.pasteEl.value=xhr.responseText;
  }
 });
 
 //focus the textarea
 paster.pasteEl.focus();
 
 </script>
 
</body>
</html>