While Heroku got its start from hosting Ruby on Rails applications, it nowadays supports many different environments in the Cedar stack. Node.js is what many use, but they also do support PHP.

Dependency management is easy for Node.js applications as Heroku recognizes your package.json files and automatically installs the libraries needed via NPM.

Until now PHP developers haven't had this convenience, but as Composer is emerging as the default PHP package manager, I've now added support for it. Before the pull request gets accepted, Composer dependency handling can already be used by specifying my custom PHP buildpack when creating Heroku apps.

I've written a simple example app to show how this works.

First you need to create the folder for your app and make it a Git repository:

$ mkdir myapp
$ cd myapp
$ git init

Then create the Heroku app using a custom buildpack (when the pull request is accepted you can skip the buildpack definition):

$ heroku create -s cedar --buildpack https://github.com/bergie/heroku-buildpack-php.git my-cool-app

Then it is time to write your composer.json file. In this case we'll only depend on the urlize library:

{
    "require": {
        "php": ">=5.2.0",
        "midgard/midgardmvc-helper-urlize": "*"
    }
}

For Heroku to recognize the app as a PHP one, you also need to have an index.php. In this case with the following code:

<?php
// URLizer service
require 'vendor/midgard/midgardmvc-helper-urlize/interface.php';
if (isset($_GET['urlize'])) {
    $data = array();
    $data['from'] = $_GET['urlize'];
    $data['to'] = midgardmvc_helper_urlize::string($_GET['urlize']);
    header('Content-type: application/json; charset=utf-8');
    die(json_encode($data));
}
header('Content-Type: text/html; charset=utf-8');
?>
<h1>Urlizer service</h1>
<form method="GET">
    <label>
        String to URLize
        <input name="urlize" type="text" />
    </label>
    <input type="submit" value="URLize" />
</form>

Now add and commit these files, and then deploy to Heroku:

$ git push heroku master

You should see that Heroku notices the Composer dependencies and installs them:

-----> Heroku receiving push
-----> Fetching custom buildpack... done
-----> PHP app detected
-----> Bundling Apache version 2.2.22
-----> Bundling PHP version 5.3.10
-----> Installing Composer dependencies
Installing dependencies
  - Package phptal/phptal (dev-master)
    Cloning e146361f25b8672d364695b757eddf1c169e05d2

  - Package midgard/midgardmvc-core (dev-master)
    Cloning 2b00d38cb2fea42c8f9791c5ecc7270dc81182e8

  - Package midgard/midgardmvc-helper-urlize (dev-master)
    Cloning 92d0c8c638c389b7be1887ca67cd334f51932912

midgard/midgardmvc-core suggests installing ext-midgard2 (>=10.05.5)
Writing lock file
Generating autoload files
-----> Discovering process types
       Procfile declares types -> (none)
       Default types for PHP   -> web
-----> Compiled slug size is 13.2MB
-----> Launching... done, v13

And that is it! You can see an example of this app at http://urlizer-service.herokuapp.com/.

While much of this has been made possible by the IKS project, it is also great to see the wider CMS community getting more active in the space. Part of this momentum is also apparent in how many communities want to learn more about these ideas.

Here are some upcoming events where decoupled CMS will be present:

• Typo3 Developer Days, April 12-15 in Munich, Germany. No official session, but I was invited to participate and chat with people
• World Wide Web Conference, April 16-20 in Lyon, France. We will present our paper on decoupled content management with Szaby Grünwald on Friday in the Developers track
• BerlinJS, May in Berlin, Germany. I will talk about Create
• Symfony Live, June 6-9 in Paris, France. I will talk about decoupling content management with Create and PHPCR, and Lukas Smith will talk about Symfony2 CMF
• IKS Technology in Action, June 12-13 in Salzburg, Austria. Lots of sessions about the different IKS tools
• DrupalCon, August 20-24 in Munich, Germany. My talk on decoupling content management is a featured session

If your community is interested in learning more, get in touch or follow my tweets from these events. And obviously I'm looking forward to meeting many of you in these events.

The book was edited by Lydia Pintscher and includes essays from 42 authors, many of whom you'll recognize if you tend to go to FOSS conferences. The LWN book review concludes:

Open Advice is a book that will be helpful to those who are new to FOSS, but, because of the individual voices, styles, and tones, it doesn't read like a "how to". It could even be recommended to those who aren't necessarily interested in contributing, but are curious about what this "free software thing" is all about. It is, in short, a great book for a variety of audiences and the (mostly) two or three page essays make it easy to read, while the anecdotes and recollections personalize it. The authors, editor, and everyone else who helped should be very pleased with the result. Readers will be too.

I probably shouldn't give the ending away, but my essay on cross-project collaboration, a subject I've also blogged about, ends with:

Good luck with breaking down the project boundaries! In most cases it works if your ideas are good and presented with an open mind. But even if you do not find a common ground, as long as your implementation solves the use case for you it has not been in vain. After all, delivering software, and delivering great user experience is what counts.

The book is licensed under CC-BY-SA, and is available as free download in ePub, mobi and PDF formats, and as paperback from Lulu. The book sources are available on GitHub, patches welcome!

• VIE is the base semantic interaction library that handles the site's content model through RDFa annotations and Backbone.js synchronization
• Create is a new kind of web editing interface built on top of that.

As the IKS project has entered its fourth year, both of these projects have gained maturity and contributions from many IKS partners and early adopters.

New UI for Create

While Create can be used for building any sort of custom user experiences (as seen in the CMS integration examples below), it also ships with a default user interface. Nemein's Riku Virta has designed a new UI concept that is currently being discussed on the CreateJS mailing list.

This interface builds on top of the original Create UI and Liip's UX work, and aims to provide more area for CMS-specific functionality and better touchscreen support:

See the full UI concept in the slideshow on Google Plus.

We hope that we will be able to land this new UI still within the March-April timeframe.

VIE: 2.0 and onwards

VIE is now nearing the 2.0 release, with the first RC expected for the end of this month. After that we'll have a hackathon in Saarbrucken, Germany where the plan is to focus on things that we've targeted for a 2.1.

The main feature of VIE 2.1 is a new way of handling RDF literals. This will make it easier to interface with services like DBpedia that give us data in multiple different languages. This will enable you to do things like:

var eiffel = vie.entities.get('dbp:Eiffel_Tower');
console.log(eiffel.get('label')); // Eiffel Tower

vie.setLanguage('fi');
console.log(eiffel.get('label')); // Eiffel-torni

The final API for this is still being discussed on the VIE mailing list.

Create and Hallo are now easier to integrate

Both Create and Hallo, our minimalist rich text editing tool now provide merged JavaScript files for easier integration. You can find the merged files, and also minified versions in:

• hallo.js
• create.js

Thanks to contributions from Alkacon, Create's widget selection mechanism is now much more configurable. This allows CMS developers to provide different editing tools for different types of information.

The currently bundled editing interfaces provide integration with Hallo, and also with the 0.20 version of Aloha Editor (though you will need to install Aloha separately to use it due to licensing restrictions).

CMS developers will also benefit from Blogsiple, the new integration testbed for Create and VIE. Blogsiple aims to be a very simple blog system built on top of Node.js that shows all the necessary integration points for supporting the whole range of VIE and Create features.

CMS adoption

As both VIE and Create improve, so does their adoption in different Content Management Systems. For example, here is the new OpenCms user interface built on these tools:

Polymedia's video annotation example is also interesting demonstration of VIE in a completely different kind of CMS environment, as is WordLift by InSideOut10.

The IKS Early Adopter program is still open if you're interested in getting support for using these tools in your CMS. There will also be an IKS event in Salzburg on June 12-13 where we will be able to show more.

As my office is where my laptop is, none of this work is particularly tied to a physical location. And as much of the work is communicating, I end up spending quite a lot of time traveling between meetings and conferences.

For this my previous home town of Helsinki, Finland wasn't quite ideal. From an European perspective, Finland is an island. And so I've accumulated airmiles instead of being able to travel more comfortably (and productively) on the European railway network.

I've previously dabbled with living in other locations like Istanbul and Saint Petersburg, but this time I hope the move will be more permanent: in the beginning of this month we packed our stuff, got rid of the Helsinki flat and hopped on a plane to Berlin.

The modern world makes this easy: my whole library fits on a Kindle and music collection on Spotify, version control and testing happens in the cloud, and banking and invoices are on the web. And thanks to EU there is minimal red tape in setting yourself up anywhere in Europe.

My current setup is such: I'm traveling with about two weeks' worth of clothes, my trusty work laptop, and tablet with an amazing 18 hour battery life. Accommodation happens in furnished, short-term rental apartments, and Internet comes from an old N900 serving as a WiFi access point via prepaid 3G.

While Berlin has excellent co-working spaces, I'm currently sharing office with Content Control, the local Midgard shop. Though at the time of writing my office is temporarily relocated to the nearby Hasenheide park where there is a nice sunny spring day.

I'm still planning on popping into Helsinki every now and then. With Air Berlin, the travel cost and distance is almost the same as if I lived in the Finnish city of Tampere (yeah, VR is expensive). Once the snows there melt, I'll also ride my motorcycle over.

If things work out, I may be eventually able to say: ich bin ein Berliner. This city is quite amazing - the amount of cultural activities and hacker meetups probably beats any other place in Europe right now. If you're around, ping me!

And if Berlin doesn't work, then there are lots of other interesting places for a working nomad...

Traditional content management has been very monolithic, meaning that by choosing a particular editing interface, CMS users also have to take the web framework, programming language and content storage mechanism mandated by the developers of their system. By splitting the CMS to the separate concepts of user interface, web framework, and content repository we can provide implementers a greater degree of freedom, and allow CMS developers to focus on the functionality where they can best make a difference.

What is Create?

With CreateJS, content management system developers can provide a simple, fast, and modern editing interface to their end-users. The UI is completely built in JavaScript, and can be integrated with three easy steps:

1. Mark up your pages with RDFa annotations
2. Include VIE and Create into the pages
3. Implement Backbone.sync with your CMS back-end

Create provides functionality like in-page content editing, managing of content collections (like article lists), running workflows for content, and handling images and content tagging. The jQuery UI plugin -based structure allows CMS developers also to implement their own additional functionality. This also makes it possible to either use the whole Create UI as-is, or just to take the parts of it that fit the UX concept of a system.

The Create UI was initially made for Midgard CMS, but has since been generalized so that it works anywhere. This approach has already gained some popularity, with CreateJS widgets being used in projects like Symfony CMF, Drupal, and OpenCMS.

The January hackathon

To push CreateJS forward we organized a hackathon in Zurich, Switzerland in the early January. Participants came from different IKS project partners and CreateJS early adopters.

Some of the results were:

• Workflow support, allowing CMSs to define per-content actions like publish/unpublish
• UI notifications, including the possibility to build scripted tutorials for site editors
• WYSIWYG editing of Markdown documents
• Much more flexible handling of editing widgets, for example to choose whether some content should be edited with a rich text editor, or a date picker
• Smart image insertion when using the Hallo editor
• Content tagging support
• Blogsiple, a CreateJS CMS integration testbed written for Node.js

Image insertion, link management, and content tagging have been designed to work together so that they can find about annotated entities thanks to the Apache Stanbol engine and provide intelligent suggestions on related content.

Moving forward

The important next step is to consolidate all these changes into the CreateJS codebase and to ensure that everything works smoothly together. Our continuous integration setup would also benefit from a larger number of tests.

After that we can consider new features, including things currently under discussion.

Helping CMSs to integrate this common user interface (or parts of it) is also a major task for this year. If you're interested in using CreateJS for your system, be sure to let us know! And also follow the progress on GitHub.

Our primary goal is to facilitate meetups, collaboration and awareness between different projects and communities within Open Mobile Linux and provide a place to present directions, ideas and your projects themselves.

By Open Mobile Linux we mean any open source projects revolving around typical non-desktop/server Linux, such as handsets, tablets, netbooks or other creative uses. Examples of such projects could be Qt5, Mer, MeeGo, Android, webOS, Plasma Active, Tizen, Boot to Gecko, SHR and other related efforts.

There are several exciting things happening in this space, including the recently announced Spark tablet, open sourcing of webOS's Enyo framework and continuing interest in the Maemo platform. Saturday's program includes:

• Mer ... and what you can do with it (David Greaves, 11:30-12:00)
• Openmoko Freerunner - Present and Future (Niels Heyvaert, 12:00-12:30)
• Towards Accelerated UI's on Mobile Linux With Power of Qt5 (Saija Eteläniemi, 12:30-13:00)
• Intro to Tizen and the Future of the Community (Dawn Foster, 13:00-13:30)
• buildroot: flexible building of a custom embedded system (Arnout Vandecappelle, 13:30-14:00)
• Deep Dive into KDE Mobile development on N9/N950 (Laszlo Papp, 14:00-14:30)
• Community Qt apps repository - way forward (Jukka Eklund, 14:30-15:00)
• Maliit - the open mobile text input project (Jon Nordby, 15:00-15:30)
• Nemo Mobile - How to contribute to the project (Marko Saukko, 15:30-16:00)
• Introduction to the QtOnPi project (Rajiv Ranganath, 16:00-16:30)
• An update on PIM storage and sync: SyncEvolution (Patrick Ohly, 16:30-17:00)
• OS in the Mobile App Stores (Thomas Bonte, 17:00-17:30)
• Rygel: Free and Open Mobile DLNA (Jens Georg, 17:30-18:00)

If there are any last-minute announcements or happenings that people want to discuss, we may be a ble to squeeze in a talk or two. Contact Carsten about this.

Also, if you want to chat other things (like PHPCR or CreateJS), I'll be around the whole weekend including the beer event. Drop me an SMS.

Looking forward to seeing as many of you there as possible!

• Your content model is constructed out of a tree of Nodes that can contain other Nodes or Properties
• Properties can be of many different types, including the typical strings and floats, but also binaries (similar to our blobs) and dates
• Properties can be multivalued (think of multiple images in same property, or a person having multiple emails)
• Queries can be either constructed with Query Object Model or SQL2
• Queries support tree constraints (ISDESCENDANTNODE, ISCHILDNODE) and joins
• Content can be exported and imported via a standard XML format
• There is a comprehensive Node Type system that allows you to define your content model. The full definitions can be read and introspected via PHP
• Nodes also support mixins, allowing you to add per-object metadata
• There are also the unstructured nodes (nt:unstructured) for cases where you don't want to have constraints on what to store and how

On top of PHPCR, there is also a Doctrine ODM that the Midgard2 provider also supports. That gives you more capabilities, like:

• More traditional object access with getters and setters to the repository
• Multilingual content storage

Some code examples: https://github.com/bergie/phpcr-midgard2/tree/master/examples

You can see Midgard's test status for both SQLite and MySQL storage back-ends in our Continuous Integration environment:

Once you have Midgard2 and our PHP extension available, installation is easy. Add a composer.json file into the root folder of your project:

{
  "require": {
    "midgard/phpcr": ">=1.0"
  }
}

Install with Composer:

$ wget http://getcomposer.org/composer.phar
$ php composer.phar install

Copy MgdSchemas:

$ sudo cp vendor/midgard/phpcr/data/share/schema/* /usr/share/midgard2/schema/
$ sudo cp vendor/midgard/phpcr/data/share/views/* /usr/share/midgard2/views/

...and then just start using the repository in your application. PHPCR can obviously be used side-by-side with traditional Midgard2 or other PHP code.

What will happen next:

• Support for PHP 5.4 and Postgres (needs work on Midgard2 level)
• Performance optimization
• Features work, including versioning
• More tools for PHPCR. Think of Asgard that would allow you to work with any repositories!
• PHPCR support built-in for MidCOM and MVC?

For those interested in learning more about PHPCR, there will also be a Paris meetup in mid-January.

# GObject Introspection APIs are available from imports.gi.Modulename
Gtk = imports.gi.Gtk

# For GNOME 3.2+ this should be Gtk.init null, 0
Gtk.init 0, null

win = new Gtk.Window
    type: Gtk.WindowType.TOPLEVEL
win.set_border_width 10
win.connect "destroy", (widget) ->
    Gtk.main_quit()

button = new Gtk.Button
    label: "Hello, world"
button.connect "clicked", ->
    button.set_label "Bar"
win.add button

button.show()
win.show()

Gtk.main()

Gjs doesn't run CoffeeScript directly, so you need to convert this before running:

$ coffee -c window.coffee 
$ gjs window.js

You should see something similar to:

Please send a short biography and an abstract for your talk to carsten.munk@gmail.com by Dec 31st 2011, and we'll get back to you at latest January 7th.

We're also grateful for volunteers helping to run the devroom. Contact Carsten if you're interested.

But now Composer, and its repository counterpart Packagist, promise to change all that. And obviously new conventions like PHP's namespacing support and the PSR-0 standard autoloader help.

Composer is heavily inspired by NPM which has built a strong culture of code-sharing and easy deployment in the Node.js community.

Easy for users

With Composer, managing dependencies in your project is very easy. Simply create a composer.json file where you state your dependencies, and let the package management system worry about the rest.

Packages that are registered with packagist.org are obviously easiest to depend on, but you can also state packages coming from custom repositories (like your company's internal version control system), or PHP extensions that you need.

Here is for example the composer.json from the Midgard PHPCR provider:

{
    "name": "midgard/phpcr",
    "type": "library",
    "require": {
        "php": ">=5.3.0",
        "ext-midgard2": ">=10.05.5",
        "phpcr/phpcr": ">=2.1.0-beta1"
    }
}

With this file, Composer knows that our PHPCR provider runs only on PHP 5.3 or newer (as it uses namespaces), and that it needs the Midgard PHP extension and the PHPCR interface classes to be available.

Now installing the project is easy:

$ wget http://getcomposer.org/composer.phar 
$ php composer.phar install

How about autoloading? Traditionally PHP required you to manually include or require all files you wanted to use in your code, with the possibility to write an autoloader to handle it automatically when you call an undefined class. But managing these autoloaders is also a chore.

Composer helps here too, by automatically generating an autoloader that will be able to load your own code, and the code from all your dependencies. So you can get rid of your own autoloaders and include statements, and just include the Composer-generated autoloader in your code:

require 'vendor/.composer/autoload.php';

After this all the classes you've stated your application needing will be available.

Easy for developers

While ease-of-installation is important, it isn't enough to build an ecosystem. The other thing that has to be easy is publishing code. Basically: if you've written a piece of functionality in PHP that you could see yourself using in another project, it should be effortless to publish it as a library.

This is where approaches like PEAR mostly failed, by making it too cumbersome to define your packages, to build them, and to upload them to the repository.

With Composer this is very easy. You again define a composer.json for your package, and push that to your project's Git repository. Then just register the Git repository URL with packagist.org.

After this Packagist will spider your repository and make it available as a package.

Publishing new versions is very easy: simply keep your composer.json up-to-date, and tag your releases in Git.

Where are we now?

It is still early days for Composer, and the project is being worked on at a hectic pace. However, it is already good enough for managing dependencies to modern, PSR-0 compatible libraries.

What I would like to see happen next is support for custom package roles and autoloaders. This would allow us to handle more specific cases, like for example installation of Midgard MVC components and their non-namespaced autoloading needs. After that we should be able to get rid of our custom installer code and just join the Composer crowd.

But if your code is already fully namespaced, this is a great time to get started with Composer.

Both environments have their strong points. Node.js is very fast and flexible, but PHP has a lot more mature tools and libraries available. So in a lot of projects it is hard to choose between the two. But now you might not have to.

Enter DNode

DNode is a remote method invocation protocol originally written for Node.js, as the name probably tells. But as the protocol itself is quite simple, just sending newline-terminated JSON packets over TCP connections, implementations have started popping up in other languages. You can talk DNode in Ruby, Perl, Python, Java, and now PHP.

I started working on the PHP DNode implementation in the Symfony CMF hackday in Cologne a week ago, and got it into a running stage on a train ride from there to Paris. The implementation isn't yet complete, but works already quite well.

With DNode you can expose Node.js functions to be available on PHP, and PHP class methods to be available on Node.

Like most Node.js functionality, DNode works asynchronously. So instead of waiting for return values you supply a callback function that will be called when the method completes.

PHP as client

Here is a simple DNode service for Node.js:

var dnode = require('dnode');
var server = dnode({
    zing: function (n, cb) { cb(n * 100) }
});
server.listen(7070);

This creates a DNode service running in TCP port 7070 that provides one method: zing that multiplies the value given to it by 100 and sends the result to the callback provided.

Calling this with PHP is easy:

// Connect to DNode server running in port 7070 and call 
// Zing with argument 33
$dnode = new DNode\DNode();
$dnode->connect(7070, function($remote, $connection) {
    // Remote is a proxy object that provides us all methods
    // from the server
    $remote->zing(33, function($n) use ($connection) {
        echo "n = {$n}\n";
        // Once we have the result we can close the connection
        $connection->end();
    });
});

Now just start the server:

$ node simple/server.js

And run the client. As you can see from the PHP code above, once we get the result the client will end the connection automatically:

$ php examples/simple/client.php 
n = 3300

Because only simple TCP connections and JSON packets are used, this is quite fast. Here are time results for the client on my MacBook Air:

real    0m0.064s
user    0m0.050s
sys     0m0.010s

PHP as a server

PHP can also act as a DNode server. You instantiate the DNode class and pass it the object you want to expose via DNode. All public methods of the object will be made available to the DNode clients:

// This is the class we're exposing to DNode
class Zinger
{
    // Public methods are made available to the network
    public function zing($n, $cb)
    {
        // Dnode is async, so we return via callback
        $cb($n * 100);
    }
}

// Create a DNode server
$server = new DNode\DNode(new Zinger());
$server->listen(7070);

This DNode service will obviously be visible for both Node.js and PHP clients.

Bidirectional communications

A DNode client can also expose methods to the server. In this example the server provides functionality for converting temperatures from Celsius to Fahrenheit, but actually gets the current Celsius temperature by asking it from a client.

Server:

// This is the class we're exposing to DNode
class Converter
{
    // Poll the client's own temperature() in celsius
    // and convert that value to fahrenheit in the supplied 
    // callback
    public function clientTempF($cb)
    {
        // The other side of DNode connection is exposed via
        // $this->remote proxy object
        $this->remote->temperature(function($degC) use ($cb) {
            $degF = round($degC * 9 / 5 + 32);
            $cb($degF);
        });
    }
}

// Create a DNode server that listens to port 6060
$server = new DNode\DNode(new Converter());
$server->listen(6060);

Client:

// This is the class we're exposing to DNode
class Temp
{
    // Compute the client's temperature and stuff that value
    // into the callback
    public function temperature($cb)
    {
        $degC = rand(-20, 50);
        echo "{$degC}° C\n";
        $cb($degC);
    }
}

$dnode = new DNode\DNode(new Temp());
$dnode->connect(6060, function($remote, $connection) {
    // Ask server for temperature in Fahrenheit
    $remote->clientTempF(function($degF) use ($connection) {
        echo "{$degF}° F\n";
        // Close the connection
        $connection->end();
    });
});

Then just start the server:

$ php examples/bidirectional/server.php

And run the client:

$ php examples/bidirectional/client.php 
28° C
82° F

The same will obviously work with a Node.js client:

$ node bidirectional/client.js 
23° C
73° F

Installing DNode

dnode-php can be installed using the Composer tool. You can either add dnode/dnode to your package dependencies, or if you want to install dnode-php as standalone, go to the main directory of its repository and run:

$ wget http://getcomposer.org/composer.phar 
$ php composer.phar install

You can then use the composer-generated autoloader to access the DNode classes:

require 'vendor/.composer/autoload.php';

Some DNode examples can be found from the examples folder. They are compatible with the similarly-named examples from Node.js DNode.

Contributing

php-dnode is developed under the MIT license in GitHub. If you're interested in it, please watch the repository and send issues or pull requests.

One option is to follow the lead of the big companies, hoping that Tizen works, or that Google again sees the benefit of working with others in the open.

The other is to take the matters in our own hands. There is precedent for this. Much of early Linux activity came from the efforts of the community, not on the initiative of corporate interests. And there have been OpenMoko and Mer, the latter an attempt to make a fully open version of Nokia's Maemo environment, suspended when MeeGo promised to bring the same benefits.

Well, now Mer is back.

The goals for Mer align pretty well with what the community would need:

• To be openly developed and openly governed as a meritocracy
• That primary customers of the platform are device vendors - not end-users.
• To provide a device manufacturer oriented structure, processes and tools: make life easy for them
• To have a device oriented architecture
• To be inclusive of technologies (such as MeeGo/Tizen/Qt/EFL/HTML5)
• To innovate in the mobile OS space

There have also been some other invitations to new potential homes for the community, ranging from openSUSE to Debian.

It will be interesting to see how this works out. But whatever we as a community do, we should ensure we look at more than just licensing.

For the last month we've been working in Salzburg to build such a system, the Intelligent Project Controlling Tool needed for running large collaborative research projects like IKS. Since the design we went with can be reused for other business analytics needs, I wanted to write a bit about it.

But first, here is how our system looks like:

Where does the data come from?

There are many ways to gather business data. Often the information systems already contain the data needed. But it may also be hidden in a jungle of spreadsheets. Or maybe some data is simply not available, and has to be filled in manually.

Handling all these cases in one system is a tricky question. To solve it, we went with a two-layered strategy:

• All data used for analytics is stored as Linked Data in a CouchDB system
• NoFlo workflows are used for gathering data from the diverse sources and convert it to the format needed

In IKS's case, much of the data was available in a series of spreadsheets. With these, we built the necessary workflows for first converting the spreadsheets into XML with Apache Tika, and then extracting the information from them in a sensible subset of JSON-LD.

Because IKS is a collaborative project, information needs to be gathered from a diverse group of partner organizations. Some of them have systems that provide the needed APIs (like Basecamp, which we use), and we can just periodically import the data. But with many we decided on a simple data interchange approach: spreadsheets handled over email.

In this approach, user files a data request into the system. This gets picked up by NoFlo, which sends an email with the appropriate spreadsheet template to the partner. Then it starts waiting for a reply. When a reply arrives, it extracts the data from the attached spreadsheet and imports it to the system.

Our NoFlo processes are mostly initiated by the CouchDB change notification API. We keep them running persistently using forever Node, so whenever some operation needs to be run it happens nearly immediately.

Ensuring data consistency

With any automation, and especially with the email-based data interchange, things can go wrong. Because of this we tag all data that we receive with its origin, whether it was some automated operation or an imported spreadsheet. These origins are called execution documents. Users can browse all completed workflow executions and see what data came in from them. These can then be either accepted or rejected.

This way if some partner accidentally sends faulty data, or something else breaks, the incorrect information received can be easily removed. CouchDB's versioning capabilities help here.

Analyzing the data

CouchDB is built on top of the concept of map/reduce. Here you can modify and combine the data in lots of different ways using simple JavaScript functions. In our case we elected to write all our CouchDB code in CoffeeScript for simplicity. For example, here is the reduce function in CoffeeScript that counts totals of time planned, time used, and time left per task or partner in a project:

(keys, values, rereduce) ->
    roundNumber = (rnum, rlength) ->
        Math.round(parseFloat(rnum) * Math.pow(10, rlength)) / Math.pow(10, rlength)
    data =
        planned: 0.0
        spent: 0.0
        left: 0.0

    if rereduce
        for reducedData in values
            data.planned += reducedData.planned
            data.spent += reducedData.spent
        data.left = data.planned - data.spent
        return data

    for doc in values
        if doc['@type'] is 'effortallocation'
            data.planned += roundNumber doc.value, 1
        if doc['@type'] is 'effort'
            data.spent += roundNumber doc.value, 1
    data.left = roundNumber data.planned - data.spent, 1
    return data

If you figure out a new way to look at the data you have, simply write the needed map and reduce functions and save them into the database. CouchDB will then run them against existing data and produce numbers.

Data visualizations

Numbers are good, but to really see the information buried in them you need some visualizations. For this we decided to follow the CouchApp idea where the user interface code is stored in the database together with the data itself. This way no application servers are needed, and you can take the whole system with you just by replicating the database. Think of the possibility of doing some analysis on your company while flying to a meeting!

The visuals are in our case provided by JavaScript InfoVis Toolkit, a nice, MIT-licensed interactive graph library.

CouchDB views handle the number crunching, then CouchDB list functions process the numbers into the format needed for visualization. This leaves only a minimal amount of work for the client side.

For consistency our application has been built with CoffeeApp, so all the database and user interface code is in CoffeeScript.

In a nutshell

Any business analytics system dealing with moderate amounts of data can be built following this approach.

• Apache CouchDB is the central data store
• All data is stored as JSON-LD entities
• NoFlo handles all data imports
• Analytics based on the data are done with CouchDB map/reduce
• Visualization happens with a CouchApp using JavaScript InfoVis Toolkit

This way you have a business analytics environment that is easy to extend with more data when it becomes available. New analysis can be done by writing reasonably simple map/reduce functions, and CouchDB's replication capabilities allow you to take the system and data with you.

Using JSON-LD for the data storage makes a lot of sense, as this way the relations between different pieces of information are easy to handle. And using URIs for data identifiers means you can easily mash up information coming from different sources together.

The two-layered approach of using NoFlo for data imports, and CouchDB for analysis also allows for clean separation of concerns. In our case, I did the workflow part of things, and Szaby built the visualizations.

Yesterday I merged the Zart codebase into the VIE repository. This blog post describes some of the improvements it brings.

VIE now has an instance

For VIE 1.x users the first visible change (and probably the only necessary API change) is that now VIE needs to be instantiated before being used. Singletons are evil, and so we are not a singleton any longer.

So, for existing VIE code, you need to:

var vie = new VIE();
// and then any traditional VIE calls, like:
var entities = vie.RDFaEntities.getInstances('div.article');
console.log("There are " + entities.length + " RDFa entities in your articles");

The VIE 1.0 API can be disabled by passing a setting when instantiating VIE:

var vie = new VIE({classic: false});

Services and VIE

The other big change in VIE is that now the API has been built in a service-oriented manner. This means that for example reading and writing RDFa is just a service you can enable and disable at will.

The benefit here is that we can easily add support for other formats and capabilities without having to touch VIE internals. Thanks to the schema.org situation, Microdata is getting more use, and so at some point we'll probably add a service for it.

Registering and accessing services is easy:

// Instantiate VIE
var vie = new VIE();

// Pass the service instance and a name you want to use for it
vie.use(new vie.RdfaService, 'rdfa');

// Call a method from the service using the name
// this one would give us the RDF subject of the
// element matched by the jQuery selector
vie.service('rdfa').getElementSubject('div.article');

An immediate benefit here is that we can have two RDFa parsing implementations. If you have problems with our own custom jQuery-based RDFa parser, then you can use the more strict rdfQuery powered implementation instead:

vie.use(new RdfaRdfQueryService, 'rdfa');

Using deferreds

For the new main VIE API we created a sort of a Domain-Specific Language for handling semantic entities. A core part of it is that now all operations utilize jQuery's Deferred objects. With them you can attach different callbacks to the results of your operation, and they will fire either when the operation completes, or immediately if the operation has already been run.

This gives a lot of flexibility in using the API, and allows us to provide same API for services that deal with the DOM, and services that talk to external APIs like Stanbol.

For example, parsing RDFa from a given DOM element (provided with a jQuery selector) happens like this:

vie.load({
        element: 'div.article'
    }).
    from('rdfa').
    execute().
    done(function(entities) {
        console.log(entities);
    });

The chain here is: operation (in this case, load), from service (rdfa), execute operation, then when done, do callback.

With the RDFa service we register Backbone Views for the elements our entities came from, so just like with VIE 1.x, they will update automatically when you change the contents of your entities. But manual writing is also available in case you need it. Here is how it works:

vie.save({
        element: 'div.article',
        entity: someBackboneModel
    }).
    to('rdfa').
    execute().
    done(function() {
        console.log("Saved!");
    });

In addition to done, which fires if the operation succeeds, you have fail for failed operations, and then which fires regardless of success or failure.

Accessing external services

The new VIE is not just about RDFa. In addition to working with the entities you have on a page, you can also access external repositories of semantic information, like DBpedia.

For example, to find out everything that Wikipedia knows about Salzburg, you could run:

vie.use(new vie.DBPediaService, 'dbpedia');
vie.load({
        entity: '<http://dbpedia.org/resource/Salzburg>'
    }).
    using('dbpedia').
    execute().
    done(function(entity) {
        console.log("This is what we know of Salzburg");
        console.log(entity);
    });

In browser usage these calls to external services are subject to cross-domain AJAX limitations. A way to work around those is to set up a proxy, and tell the DBpedia service to use that. To do this, pass the proxy URL to the service when instantiating:

vie.use(new vie.DBPediaService({proxyUrl: 'http://localhost:8080'});

With this, all the factual information from Wikipedia will be at your disposal. The size of every city, the height of every mountain. Birthdates and places of birth for famous people. Your web app can do quite a bit with this information.

Finding entities from text

Apache Stanbol is a semantic engine that can extract all kinds of entities from text documents. It can be used for auto-tagging and other things.

Here is how you can use it with VIE:

vie.use(new vie.StanbolService, 'stanbol');
vie.analyze({
        element: 'div.article'
    }).
    using('stanbol').
    execute().
    done(function(entities) {
        console.log("We got the following enhancements for article content");
        console.log(entities);
    });

Stanbol can tell you what a piece of content talks about. People mentioned, places, concepts. It will also give you the language of the text.

Moving forward

The new version of VIE is still under heavy development. Most of the thngs work, but some details may still change. It is a good idea to start taking a look at it now, but before a beta release at least, VIE 1.0 is the recommended tool to use.

If you already use VIE 1.0 for making your content editable, VIE 2.x will give you a lot of additional power. Enhancements, data queries, namespace handling, and much more.

Thanks to Szaby and Sebastian for helping to make this happen!

node-gir has been written by Tim Caswell, with help from Sebastian Wick and Piotr Pokora.

I've been following the progress for a while, and today, during a flight from Helsinki to Salzburg, I was finally able to open a Midgard repository connection with it. The API still is a bit weird, and lacks support for the asynchronous nature of Node, but those will hopefully change soon. Quick example:

var Midgard, gir, config, mgd;
gir = require("../gir");
gir.init();
Midgard = gir.load("Midgard");
Midgard.init();

// Use a local SQLite database file
config = new Midgard.Config();
config.__set_property__("dbdir", __dirname);
config.__set_property__("dbtype", "SQLite");
config.__set_property__("database", "midgard");

// Open connection to the database
mgd = new Midgard.Connection();
if (!mgd.__call__("open_config", config)) {
    console.error("Failed to open connection");
    process.exit();
}

node-gir is being developed on GitHub if you want to lend a hand or try it out. To build it, run npm install and you should be able to run the code examples.

Having GIR support for Node would make it a full-fledged GNOME environment, and mean that there would be proper GObject Introspection in all three major JavaScript runtimes - SpiderMonkey, JavaScriptCore and V8. And this way GNOME JavaScript developers could also utilize the wealth of existing Node.js modules.

Many of their plans to make web competitive against native development environments are good, there is indeed much to improve in the stack. But what I'm uneasy with is the unilateral way they go about it, preferring "big reveals" and post-facto standardization instead of the open conversation that built most of the Internet we have today. This is not the way to collaborate.

Consider some of their recent efforts:

• SPDY, a protocol to replace HTTP which Web is built on. Currently only supported by Chrome, which uses it to talk to several Google services
• Dart, their JavaScript-killer which recently surfaced through a leaked email
• Microdata and Schema.org that seek to replace last ten years of semantic web development with a spec cooked up by couple of big vendors in secret

These - together with WebSQL, NaCl, WebM and WebP - mean that Google has active efforts to replace practically every layer of the web (except HTML itself) with something of their own design.

The way all of these were introduced bears strong reminders of how Microsoft tried to embrace, extend, and extinguish the web in late 90s. That period brought horrors like ActiveX and the awful, unkillable IE6. Though, for the sake of fairness, it also brought us XmlHttpRequest which was the enabler of the AJAX revolution.

Google's new technologies may end up being beneficial for web developers, but they also threaten to fragment the platform. After all, as the competition in the "post-PC" space heats up, the competitors are unlikely to embrace Google's extensions of the web stack. That would be a loss to all.

Brendan Eich, the original author of JavaScript comments on Hacker News:

So "Works best in Chrome" and even "Works only in Chrome" are new norms promulgated intentionally by Google. We see more of this fragmentation every day. As a user of Chrome and Firefox (and Safari), I find it painful to experience, never mind the political bad taste.

Ok, counter-arguments. What's wrong with playing hardball to advance the web, you say? As my blog tries to explain, the standards process requires good social relations and philosophical balance among the participating competitors.

Google's approach with Dart is thus pretty much all wrong and doomed to leave Dart in excellent yet non-standardized and non-interoperable implementation status. Dart is GBScript to NaCl/Pepper's ActiveG.

Disclaimer: I've been a long-time fan of many of Google's services, and have visited some of their offices a few times. I like the company. Which is exactly why I'm so concerned about this unilateral approach at standards. I am also involved in some standards processes through the IKS Project.

MG Siegler from TechCrunch has a great post on the subject:

...I’ve been trained over time to think that the traditional PC is the way to do these things whether it’s for work or play. That’s simply not true. The tablet form factor is so. much. better. when you don’t have to do an excessive amount of typing. And during downtime, when I use a computer like a more regular human being, I’ve found that’s often...

Computing is changing. That’s just about the most obvious statement ever. We’ve been seeing this for years with the rise of the smartphones. But traditional computing is changing as well. As in, people are abandoning PCs for these newer devices. And this will keep happening.

My experience conforms with this. I rarely use my laptop outside of the work context of writing code, instead preferring to use the tablet with its great ergonomics, portability and long battery life. On some of my previous trips I noticed that already more than half of people sitting in airport lounges use a tablet instead of a laptop. Not bad for a product category that has existed in a mainstream manner for less than two years. Nokia's internet tablets blazed the trail years earlier, but were never marketed outside the geekdom.

Now, much of the attention in the tablet world has been focused on the couple of platforms that are winning in popularity, and therefore have most of the apps. But regardless of how well Apple and Google play their cards, the post-PC world will be a multiplatform one.

About a week after I got my webOS-powered TouchPad, HP went and killed the product. Yet this hasn't made the device useless. As Paul Rouget recently found out, as long as you have a good browser, your device will be relevant.

Some people can't or don't want to use Native Apps. Because their phones don't have Apps, or because there is just no good Apps for what they want to do, or because, well, because they don't need to...

While in the Western world we were looking at Apple bringing pretty Apps in an expensive device, in the Eastern world, Opera was bringing a working web browser to all the existing devices.

This is the big opportunity for free software to remain relevant in an environment of highly-locked devices. Much of the web already runs on free components, and by using the web as a universal runtime we can bypass almost any platform restrictions. As Paul Graham wrote back in 2001, no one can break web applications without breaking browsing.

The world of publishing is starting to understand this. Their revenue models can't take the heavy control that vendors like Apple imposes, and so Amazon's Kindle is a web app, and so is Financial Times. That "Next year HTML5 will replace native apps" is the new "Next year will be the year of Linux on the desktop" is already a Twitter joke, but there is certainly some movement in this direction. And interestingly, the Linux desktop is actually becoming more web-savvy and touch-friendly.

There are clearly sweet spots for something to be a web app, or for it to be a native application. Similarly, there are different situations where tablets will be the appropriate tool, and where PCs are. The tablet context will be more like this:

Than this:

The heavy lifting is a better fit for a system designed for that. As Steve Jobs said, the PC will be the truck.

Even in the ten years of history of our company this is quite a significant move - it allows us to combine Nemein's traditional expertise on content management with Infigo's mobile offerings. As smartphones and tablets are becoming popular, more and more services we build will have a mobile element, which is now easier with lots of in-house expertise.

This also means more focus on the interplay between the Midgard content repository, NoFlo workflows, Node.js and Symfony web services, and mobile applications built in Qt.

Petri Rajahalme (with me in the photo) will be the CEO of the merged company, and I will focus on leading the R&D efforts.

Yesterday Igor Wiedler published Evenement, a PHP port of the EventEmitter class from Node.js. As NoFlo builds quite heavily on EventEmitter, I decided to see how far the PHP port could be taken.

As result, there is now PhpFlo, a flow-based programming environment for PHP.

Example of how to define and run a flow (you can also use a JSON format for this):

// Add nodes to the graph
$graph = new PhpFlo\Graph("linecount");
$graph->addNode("Read File", "ReadFile");
$graph->addNode("Split by Lines", "SplitStr");
$graph->addNode("Count Lines", "Counter");
$graph->addNode("Display", "Output");

// Add connections between nodes
$graph->addEdge("Read File", "out", "Split by Lines", "in");
$graph->addEdge("Read File", "error", "Display", "in");
$graph->addEdge("Split by Lines", "out", "Count Lines", "in");
$graph->addEdge("Count Lines", "count", "Display", "in");

// Kick-start the process by sending filename to Read File
$graph->addInitial($fileName, "Read File", "source");

// Make the graph "live"
$network = PhpFlo\Network::create($graph);

The flow consists of processes, or instances simple "black box" components that have their own defined input and output ports. Program logic is defined by making connections between them. Here is a simple component that reads the contents of a file:

namespace PhpFlo\Component;
use PhpFlo\Component;
use PhpFlo\Port;
class ReadFile extends Component
{
    public function __construct()
    {
        $this->inPorts['source'] = new Port();
        $this->outPorts['out'] = new Port();
        $this->outPorts['error'] = new Port();

        $this->inPorts['source']->on('data', array($this, 'readFile'));
    }

    public function readFile($data)
    {
        if (!file_exists($data)) {
            $this->outPorts['error']->send("File {$data} doesn't exist");
            return;
        }

        $this->outPorts['out']->send(file_get_contents($data));
        $this->outPorts['out']->disconnect();
    }
}

I hope people find this system useful. If you're interested in FBP, then J. Paul Morrison's book is a good place to start.

And if you're in FrOSCon, feel free to come and chat with me :-)