For the last month we've been working in Salzburg to build such a system, the Intelligent Project Controlling Tool needed for running large collaborative research projects like IKS. Since the design we went with can be reused for other business analytics needs, I wanted to write a bit about it.

But first, here is how our system looks like:

Where does the data come from?

There are many ways to gather business data. Often the information systems already contain the data needed. But it may also be hidden in a jungle of spreadsheets. Or maybe some data is simply not available, and has to be filled in manually.

Handling all these cases in one system is a tricky question. To solve it, we went with a two-layered strategy:

• All data used for analytics is stored as Linked Data in a CouchDB system
• NoFlo workflows are used for gathering data from the diverse sources and convert it to the format needed

In IKS's case, much of the data was available in a series of spreadsheets. With these, we built the necessary workflows for first converting the spreadsheets into XML with Apache Tika, and then extracting the information from them in a sensible subset of JSON-LD.

Because IKS is a collaborative project, information needs to be gathered from a diverse group of partner organizations. Some of them have systems that provide the needed APIs (like Basecamp, which we use), and we can just periodically import the data. But with many we decided on a simple data interchange approach: spreadsheets handled over email.

In this approach, user files a data request into the system. This gets picked up by NoFlo, which sends an email with the appropriate spreadsheet template to the partner. Then it starts waiting for a reply. When a reply arrives, it extracts the data from the attached spreadsheet and imports it to the system.

Our NoFlo processes are mostly initiated by the CouchDB change notification API. We keep them running persistently using forever Node, so whenever some operation needs to be run it happens nearly immediately.

Ensuring data consistency

With any automation, and especially with the email-based data interchange, things can go wrong. Because of this we tag all data that we receive with its origin, whether it was some automated operation or an imported spreadsheet. These origins are called execution documents. Users can browse all completed workflow executions and see what data came in from them. These can then be either accepted or rejected.

This way if some partner accidentally sends faulty data, or something else breaks, the incorrect information received can be easily removed. CouchDB's versioning capabilities help here.

Analyzing the data

CouchDB is built on top of the concept of map/reduce. Here you can modify and combine the data in lots of different ways using simple JavaScript functions. In our case we elected to write all our CouchDB code in CoffeeScript for simplicity. For example, here is the reduce function in CoffeeScript that counts totals of time planned, time used, and time left per task or partner in a project:

(keys, values, rereduce) ->
    roundNumber = (rnum, rlength) ->
        Math.round(parseFloat(rnum) * Math.pow(10, rlength)) / Math.pow(10, rlength)
    data =
        planned: 0.0
        spent: 0.0
        left: 0.0

    if rereduce
        for reducedData in values
            data.planned += reducedData.planned
            data.spent += reducedData.spent
        data.left = data.planned - data.spent
        return data

    for doc in values
        if doc['@type'] is 'effortallocation'
            data.planned += roundNumber doc.value, 1
        if doc['@type'] is 'effort'
            data.spent += roundNumber doc.value, 1
    data.left = roundNumber data.planned - data.spent, 1
    return data

If you figure out a new way to look at the data you have, simply write the needed map and reduce functions and save them into the database. CouchDB will then run them against existing data and produce numbers.

Data visualizations

Numbers are good, but to really see the information buried in them you need some visualizations. For this we decided to follow the CouchApp idea where the user interface code is stored in the database together with the data itself. This way no application servers are needed, and you can take the whole system with you just by replicating the database. Think of the possibility of doing some analysis on your company while flying to a meeting!

The visuals are in our case provided by JavaScript InfoVis Toolkit, a nice, MIT-licensed interactive graph library.

CouchDB views handle the number crunching, then CouchDB list functions process the numbers into the format needed for visualization. This leaves only a minimal amount of work for the client side.

For consistency our application has been built with CoffeeApp, so all the database and user interface code is in CoffeeScript.

In a nutshell

Any business analytics system dealing with moderate amounts of data can be built following this approach.

• Apache CouchDB is the central data store
• All data is stored as JSON-LD entities
• NoFlo handles all data imports
• Analytics based on the data are done with CouchDB map/reduce
• Visualization happens with a CouchApp using JavaScript InfoVis Toolkit

This way you have a business analytics environment that is easy to extend with more data when it becomes available. New analysis can be done by writing reasonably simple map/reduce functions, and CouchDB's replication capabilities allow you to take the system and data with you.

Using JSON-LD for the data storage makes a lot of sense, as this way the relations between different pieces of information are easy to handle. And using URIs for data identifiers means you can easily mash up information coming from different sources together.

The two-layered approach of using NoFlo for data imports, and CouchDB for analysis also allows for clean separation of concerns. In our case, I did the workflow part of things, and Szaby built the visualizations.

Yesterday I merged the Zart codebase into the VIE repository. This blog post describes some of the improvements it brings.

VIE now has an instance

For VIE 1.x users the first visible change (and probably the only necessary API change) is that now VIE needs to be instantiated before being used. Singletons are evil, and so we are not a singleton any longer.

So, for existing VIE code, you need to:

var vie = new VIE();
// and then any traditional VIE calls, like:
var entities = vie.RDFaEntities.getInstances('div.article');
console.log("There are " + entities.length + " RDFa entities in your articles");

The VIE 1.0 API can be disabled by passing a setting when instantiating VIE:

var vie = new VIE({classic: false});

Services and VIE

The other big change in VIE is that now the API has been built in a service-oriented manner. This means that for example reading and writing RDFa is just a service you can enable and disable at will.

The benefit here is that we can easily add support for other formats and capabilities without having to touch VIE internals. Thanks to the schema.org situation, Microdata is getting more use, and so at some point we'll probably add a service for it.

Registering and accessing services is easy:

// Instantiate VIE
var vie = new VIE();

// Pass the service instance and a name you want to use for it
vie.use(new vie.RdfaService, 'rdfa');

// Call a method from the service using the name
// this one would give us the RDF subject of the
// element matched by the jQuery selector
vie.service('rdfa').getElementSubject('div.article');

An immediate benefit here is that we can have two RDFa parsing implementations. If you have problems with our own custom jQuery-based RDFa parser, then you can use the more strict rdfQuery powered implementation instead:

vie.use(new RdfaRdfQueryService, 'rdfa');

Using deferreds

For the new main VIE API we created a sort of a Domain-Specific Language for handling semantic entities. A core part of it is that now all operations utilize jQuery's Deferred objects. With them you can attach different callbacks to the results of your operation, and they will fire either when the operation completes, or immediately if the operation has already been run.

This gives a lot of flexibility in using the API, and allows us to provide same API for services that deal with the DOM, and services that talk to external APIs like Stanbol.

For example, parsing RDFa from a given DOM element (provided with a jQuery selector) happens like this:

vie.load({
        element: 'div.article'
    }).
    from('rdfa').
    execute().
    done(function(entities) {
        console.log(entities);
    });

The chain here is: operation (in this case, load), from service (rdfa), execute operation, then when done, do callback.

With the RDFa service we register Backbone Views for the elements our entities came from, so just like with VIE 1.x, they will update automatically when you change the contents of your entities. But manual writing is also available in case you need it. Here is how it works:

vie.save({
        element: 'div.article',
        entity: someBackboneModel
    }).
    to('rdfa').
    execute().
    done(function() {
        console.log("Saved!");
    });

In addition to done, which fires if the operation succeeds, you have fail for failed operations, and then which fires regardless of success or failure.

Accessing external services

The new VIE is not just about RDFa. In addition to working with the entities you have on a page, you can also access external repositories of semantic information, like DBpedia.

For example, to find out everything that Wikipedia knows about Salzburg, you could run:

vie.use(new vie.DBPediaService, 'dbpedia');
vie.load({
        entity: '<http://dbpedia.org/resource/Salzburg>'
    }).
    using('dbpedia').
    execute().
    done(function(entity) {
        console.log("This is what we know of Salzburg");
        console.log(entity);
    });

In browser usage these calls to external services are subject to cross-domain AJAX limitations. A way to work around those is to set up a proxy, and tell the DBpedia service to use that. To do this, pass the proxy URL to the service when instantiating:

vie.use(new vie.DBPediaService({proxyUrl: 'http://localhost:8080'});

With this, all the factual information from Wikipedia will be at your disposal. The size of every city, the height of every mountain. Birthdates and places of birth for famous people. Your web app can do quite a bit with this information.

Finding entities from text

Apache Stanbol is a semantic engine that can extract all kinds of entities from text documents. It can be used for auto-tagging and other things.

Here is how you can use it with VIE:

vie.use(new vie.StanbolService, 'stanbol');
vie.analyze({
        element: 'div.article'
    }).
    using('stanbol').
    execute().
    done(function(entities) {
        console.log("We got the following enhancements for article content");
        console.log(entities);
    });

Stanbol can tell you what a piece of content talks about. People mentioned, places, concepts. It will also give you the language of the text.

Moving forward

The new version of VIE is still under heavy development. Most of the thngs work, but some details may still change. It is a good idea to start taking a look at it now, but before a beta release at least, VIE 1.0 is the recommended tool to use.

If you already use VIE 1.0 for making your content editable, VIE 2.x will give you a lot of additional power. Enhancements, data queries, namespace handling, and much more.

Thanks to Szaby and Sebastian for helping to make this happen!

Many of their plans to make web competitive against native development environments are good, there is indeed much to improve in the stack. But what I'm uneasy with is the unilateral way they go about it, preferring "big reveals" and post-facto standardization instead of the open conversation that built most of the Internet we have today. This is not the way to collaborate.

Consider some of their recent efforts:

• SPDY, a protocol to replace HTTP which Web is built on. Currently only supported by Chrome, which uses it to talk to several Google services
• Dart, their JavaScript-killer which recently surfaced through a leaked email
• Microdata and Schema.org that seek to replace last ten years of semantic web development with a spec cooked up by couple of big vendors in secret

These - together with WebSQL, NaCl, WebM and WebP - mean that Google has active efforts to replace practically every layer of the web (except HTML itself) with something of their own design.

The way all of these were introduced bears strong reminders of how Microsoft tried to embrace, extend, and extinguish the web in late 90s. That period brought horrors like ActiveX and the awful, unkillable IE6. Though, for the sake of fairness, it also brought us XmlHttpRequest which was the enabler of the AJAX revolution.

Google's new technologies may end up being beneficial for web developers, but they also threaten to fragment the platform. After all, as the competition in the "post-PC" space heats up, the competitors are unlikely to embrace Google's extensions of the web stack. That would be a loss to all.

Brendan Eich, the original author of JavaScript comments on Hacker News:

So "Works best in Chrome" and even "Works only in Chrome" are new norms promulgated intentionally by Google. We see more of this fragmentation every day. As a user of Chrome and Firefox (and Safari), I find it painful to experience, never mind the political bad taste.

Ok, counter-arguments. What's wrong with playing hardball to advance the web, you say? As my blog tries to explain, the standards process requires good social relations and philosophical balance among the participating competitors.

Google's approach with Dart is thus pretty much all wrong and doomed to leave Dart in excellent yet non-standardized and non-interoperable implementation status. Dart is GBScript to NaCl/Pepper's ActiveG.

Disclaimer: I've been a long-time fan of many of Google's services, and have visited some of their offices a few times. I like the company. Which is exactly why I'm so concerned about this unilateral approach at standards. I am also involved in some standards processes through the IKS Project.

But there are also bright points. I've recently seen lots of exchange of ideas, and even potential code sharing between some communities including Symfony2, Midgard, TYPO3 and eZ Publish. Much of the vision in these systems is similar, as are many of the engineering principles. When everybody uses reasonable object-oriented design, namespaces, and test-driven development, it is much easier to share.

If I had to list three areas where there is most potential for collaboration, these would be:

Content model on the browser: VIE and RDFa

The age of communicating with your web audience via forms is almost over, and it is time to evolve. HTML5 includes support for the contentEditable attribute which allows rich editing interaction straight on the pages, and there are cool editors supporting that, including Aloha Editor and Mercury.

To do proper front-end editing, your CMS and the JavaScript environment have to agree on the content model. Fortunately there is a great solution for this: just annotate your content with some RDFa.

Having RDFa on a page allows the browser to understand the content. What is a collection of blog posts for instance, and what is the title of a blog post. With this, my VIE library will provide you with a nice in-browser content management API based on Backbone.js. Getting there is easy:

1. Annotate your pages with RDFa
2. Include vie.js to the pages
3. Implement Backbone.sync

This allows a great deal of decoupling in the CMS stack. Suddenly the server side just has to worry about content management and page generation, and newer in-browser technologies can be used for actual content authoring.

Using RDFa annotations in your content comes also with another benefit: suddenly your pages themselves are an API into your content model. And search engines can understand and present your content better.

If you want to learn more about this, watch my talk from the Aloha Editor Dev Con.

Content persistence and retrieval: PHPCR

Historically, all CMSs have implemented persistence in their own way. There have been systems using relational databases like MySQL, systems providing their own content repository APIs like Midgard, and also some systems just using XML and the file system. This has reduced integration and code re-use possibilities between systems. In the Java world, a solution exists for this: the Java Content Repository standard (JCR).

Now JCR has been ported to PHP. PHPCR provides a standard interface for all content management needs, and has multiple back-ends available. Depending on your deployment needs, you could store your content into a relational database, into Apache Jackrabbit, or into for example MongoDB.

PHPCR is great in that you can start small: just model your content with a simple, filesystem-like tree of nodes and properties. Then when you need it, a wealth of functionality is available. Versioning? Query builders? Access control? It is all there for you to use. And, depending on the PHPCR back-end, you'll have the ability to scale up to insane amounts of content.

While I've advocated using content repositories for years now, this is the first time PHP has a true standardized, vendor-neutral API for it. And PHPCR is even being integrated into the JCR specification, eventually making it an official standard.

Adoption is also picking up. Yesterday I was in a meeting where we had developers from TYPO3, Symfony2, Doctrine and Midgard discussing issues and solutions in the content repository space. I just hope the other projects also pick this specification up.

Improving performance: AppServer-in-PHP

Of the three, this is probably the most controversial idea. Traditionally PHP is run as a scripting environment on a regular web server, like Apache or Nginx. In such setup, when the server receives a request, it passes it on to the PHP environment. The PHP environment loads all the code needed to fulfill the request, runs it, sends the response back, and unloads everything loaded.

This is fine when PHP is being used in the way Rasmus originally intended, as a simple display layer. But nowadays most of PHP runs on a big framework, whether it is MVC or something custom like Drupal. And loading and then discarding a whole framework for each request is simply insane.

With AppServer-in-PHP (AiP), you have an environment where even a big framework can perform. AiP provides you with a full server environment for PHP, written in PHP. In this setup, your framework is loaded when the server boots up, and then each request just runs the request processing part of it.

During the San Francisco Aloha Dev Con we ported TYPO3 to run on AiP, and the performance results where staggering. A simpler request with not much I/O would run 3-4 times faster than the same code on regular PHP setup, and an I/O -intensive request would still be twice as fast. AiP can't do much about I/O performance, but at least the cost of having a framework is greatly reduced.

In short, AppServer-in-PHP is something any developer running web services with a PHP framework should consider. It is also a great way for framework developers to see if they have request isolation problems in their design.

This post has been written in the TYPO3 Developer Days 2011 event where I was invited to discuss these ideas, and also help run the RDFa part of the TYPO3 Goes Semantic workshop.

If you want something similar but without the Google tie-in, then a good option is my social impact calculator which is fully free software written in PHP. It was originally written in 2007, but the newer version has been cleaned of Midgard dependencies and updated to reflect the current popular social networking services. Usage example from my earlier post:

require('calculate.php');

$url = 'http://bergie.iki.fi/blog/introducing_the_midgard_create_user_interface/';

// Get the raw count for only one source
echo com_meego_planet_calculate::hackernews($url); // 145
echo com_meego_planet_calculate::facebook($url); // 1

// Get weighted total score for all sources
echo com_meego_planet_calculate::all($url); // 130.8

The old "open vs. proprietary" debate is over and open won. As IT infrastructure moves to the cloud, openness is not just a priority for source code but for standards and APIs as well. Almost every vendor in the IT market now wants to position its products as "open." Vendors that don't have an open source product instead emphasize having a product that uses "open standards" or has an "open API."

"Openwashing" is a term derived from "greenwashing" to refer to dubious vendor claims about openness. Openwashing brings the old "open vs. proprietary" debate back into play - not as "which one is better" but as "which one is which?"

Especially Google seems to be doing this quite a bit. If you want to be open, work in the open. This is the only way to ensure acceptance and sustainability for your code.

But there is one weakness in the JS ecosystem: as things are just now picking up, the amount of information on especially making larger applications is quite sparse. To help solving this issue, I decided to start a new blog dedicated to what I learn in that space.

Like my earlier piece on the importance of the language, the blog is called The Universal Runtime.

If you run into interesting tutorials on JavaScript, or are doing something cool in the space, let me know!

When I first presented the approach in the recent Aloha Editor Developer Conference, many CMSs wanted to do something similar. And so we decided to strip the Midgard-specific parts out and make the tool a generic JavaScript library. As part of this work, the library was adopted by the IKS project and named VIE, or "Vienna IKS Editables". There first CMS implementations of VIE included WordPress, TYPO3 and KaraCos, with more on their way.

To get started with VIE, check these pages out:

• VIE project page on IKS wiki
• VIE repository on GitHub
• VIE API documentation

In addition to Midgard Create, one of the first projects I'm implementing with VIE is Palsu, an interactive meeting tool powered by Node.js and Socket.io. It should explain the power of VIE outside of the traditional CMS space.

Update: VIE is now also available on npm:

npm install vie

With Cloud9IDE you get an excellent code editor thanks to the ACE project (formerly known as Mozilla Bespin), git integration and possibility to run your Node.js code right on the server. The editor supports multiple programming languages including PHP, Python and Ruby, but obviously JavaScript is the main target with very nice debugging tools. Cloud9IDE can either be used on the cloud, or you can run your own installation of the GPLd system.

What is good:

• Tight GitHub integration. All your GitHub projects are there, and you can register and log in using your GitHub credentials
• Fast and nice UI, at least on modern browsers like Chromium. It is probably also very fast on Firefox 4
• Integrated console and debugging tools
• No need to install anything, but still the possibility to run your own instance if you want
• Cloud9IDE is free software

What still needs love:

• In the current hosted version there is no way to commit stuff back to Git, or to work with branches
• No way to work with other Git repositories than GitHub
• Running code in Node.js doesn't seem to support any installable modules
• Some parts of the UI had glitches, like menus becoming transparent on hover

But none of this should be necessary. When web editing tools were more immature, it made sense for the same people to build the whole stack from database content models to web page generation and editing tools. But that was ten years ago, now we could do better.

Here is how a traditional CMS looks like:

As you can see, the whole system is a monolithic block. The CMS provides content storage, routing, templating, editing tools, the kitchen sink. Probably you're even tied to a particular relational database for content storage. Want to use a cool new editor like Aloha, or a different templating engine, or maybe a trendy NoSQL storage back-end? You'll have to convince the whole CMS project or vendor to switch over.

A much better picture would be something like the following:

In this scenario, the concept of Content Management is decoupled. There is a content repository that manages content models and how to store them. This could be something like JCR, PHPCR, CouchDB or Midgard2. Then there is a web framework, responsible of matching URL requests to particular content and generating corresponding web pages. This could be Drupal, Flow3, Django, CodeIgniter, Midgard MVC, or something similar. And finally there is the web editing tool. The web editing tool provides an interface for managing contents of the web pages. This includes functionalities like rich text editing, workflows and image handling.

The web editing tools have traditionally been part of the web framework, the framework serving forms and toolbars to the user as part of the generated web pages. But with modern browsers you could throw forms out of the window and just make pages editable as they are.

Common representation of content on HTML level

How would the communication between the web editing tool and the backend work, then?

First of all, the web editing tool has to understand the contents of the page. It has to understand what parts of the page should be editable, and how they connect together. If there is a list of news for instance, the tool needs to understand it enough to enable users to add new news items. The easy way of accomplishing this is to add some semantic annotations to the HTML pages. These annotations could be handled via Microformats, HTML5 microdata, but the most power lies with RDFa.

RDFa is a way to describe the meaning of particular HTML elements using simple attributes. For example:

<div typeof="http://rdfs.org/sioc/ns#Post" about="http://example.net/blog/news_item">
    <h1 property="dcterms:title">News item title</h1>
    <div property="sioc:content">News item contents</div>
</div>

Here we get all the necessary information for making a blog entry editable:

• typeof tells us the type of the editable object. On typical CMSs this would map to a content model or a database table
• about gives us the identifier of a particular object. On typical CMSs this would be the object identifier or database row primary key
• property ties a particular HTML element to a property of the content object. On a CMS this could be a database column

As a side effect, we also manage to make our page more understandable to search engines and other semantic tools. So the annotations are not just needed for UI, but also for SEO.

Common representation of content on JavaScript level

Having contents of a page described via RDFa makes it very easy to extract the content model into JavaScript. We can have a common utility library for doing this, but we also should have a common way of keeping track of these content objects. Enter Backbone.js:

Backbone supplies structure to JavaScript-heavy applications by providing models with key-value binding and custom events, collections with a rich API of enumerable functions, views with declarative event handling, and connects it all to your existing application over a RESTful JSON interface.

With Backbone, the content extracted from the RDFa-annotated HTML page is easily manageable via JavaScript. Consider for example:

objectInstance.set({title: 'Hello, world'});
objectInstance.save(null, {
  success: function(savedModel, response) {
    alert("Your article '" + savedModel.get('title') + "' was saved to server");
  }
});

This JS would work across all the different CMS implementations. Backbone.js provides a quite nice RESTful implementation of communicating with the server with JSON, but it can be easily overridden with CMS-specific implementation by just implementing a new Backbone.Sync method. Look for example at the localStorage Backbone.js Sync implementation.

New possibilities for collaboration

Once the different Content Management Systems describe their content with RDFa, and provide an unified JavaScript API to it, lots of things become possible. While most systems probably want to have their own look-and-feel, still many functionalities can be shared. Consider for example:

• Using browser's localStorage for storing drafts of content edited by user. Never lose content!
• Collaborative editing via XMPP or WebSockets
• Versioning and undo
• Semantic enrichment of content using tools like Apache Stanbol

All of these would be quite hard to implement by an individual CMS project. But if we have a common JS layer available, the effort can be shared by all CMS projects implementing these ideas.

There have been prior efforts at doing something similar. In the early 2000s, OSCOM made the Twingle tool that was able to edit and save content with multiple CMSs. Then there was the Atom Publishing Protocol and the Neutrol Protocol efforts, and also CMIS. But all of these mandated that the systems would have to implement some particular server-side protocol. The advantage of the approach promoted here is that the only server-side change needed is adding RDFa annotations to HTML templates, and then the rest happens on JavaScript level.

The new CMS interface we've built for Midgard2 already uses these concepts. Now here in the Aloha Editor Developer conference we're talking with Drupal and TYPO3 developers about rolling out the same ideas in their systems. Other systems and projects are also more than welcome to participate.

Update: The work is underway to generalize the RDFa-Backbone.js bridge I originally wrote for Midgard Create. You can find it on GitHub. We're currently experimenting with it on both Midgard2 and TYPO3.

The answer is sustainability. IKS is an EU-funded project which will eventually end. Proper project governance handled together with the Apache Software Foundation can help the software to survive and thrive for long after that.

Sustainability is something that is critical for all libraries and infrastructure software. If you want adoption, you need to ensure potential users and developers that the software will continue to be around. The way to accomplish that is to have a clear, open governance model. It is time to stop throwing code over the wall.

There are many examples of this happening, from Facebook's framing of web to netbooks systems like the JoliCloud not having an address bar. Certainly many companies are looking at Mozilla's search engine revenue and Apple's app store model and want to emulate that, moving the web into silos of their own control. But at the same time, we're thinking of Linked Data and open, interoperable web standards.

Web indeed is at new crossroads.

Chris Messina predicts the death of URLs:

a future without URLs and without the infinite organicity of the web frightens me. It’s not that I know what we’ll lose by removing this artifact of one of the most generative periods in history — and that’s exactly the point! The URL and the ability for anyone to mint a new one and then propagate it is what makes the web so resilient, so empowering, and so interesting! That I don’t need to ask anyone permission to create a new website or webpage is a kind of ideological freedom that few generations in history have known!

Tim O'Reilly presents a call to arms:

It could be that everyone will figure out how to play nicely with each other, and we'll see a continuation of the interoperable web model we've enjoyed for the past two decades. But I'm betting that things are going to get ugly. We're heading into a war for control of the web. And in the end, it's more than that, it's a war against the web as an interoperable platform. Instead, we're facing the prospect of Facebook as the platform, Apple as the platform, Google as the platform, Amazon as the platform, where big companies slug it out until one is king of the hill.

And it's time for developers to take a stand. If you don't want a repeat of the PC era, place your bets now on open systems. Don't wait till it's too late.

Core idea of the Social Desktop is to connect to your peers in the community, making sharing and exchanging knowledge easier to integrate into applications and the desktop itself. The concept behind the Social Desktop is to bring the power of online communities and group collaboration to desktop applications and the desktop shell itself.

This sounds exactly like the stuff I was talking about back in GUADEC 2003. I was there on behalf of OSCOM to see how the free desktop could be integrated with the various open source content management and collaboration systems developed by OSCOM members. It is great to see these ideas finally gain some traction.

There are many specifications to help us get there:

• OCS API caters for sharing things like activities, messages, and even file contents
• Tomboy API provides synchronization and sharing of notes, with some existing implementations like Snowy and its Midgard equivalent
• Atom Publishing Protocol was used by Nokia's Lifeblog tools
• Telepathy's peer-to-peer collaborative model is used in OLPC Sugar
• ...and of course WebDAV handles managing web content, potentially more usefully through Neutron introspection

With these the free desktop might become more than just an isolated island.

Such collaborative features will make the free applications much more compelling to users, especially if coupled with web interfaces that can be used when away from your own computer. Being built on open source server software and open standards they can be hosted by companies, schools, or even Linux distributions, instead of tying users to the big cloud vendors.

I'll be talking more about the relation between the desktop and the web and our approach to it in my Gran Canaria Desktop Summit talk on Tuesday Jul 7th.

Technorati Tags: freesoftware, gnome, opencollaborationservices, socialdesktop, web

My notes from the meeting days can be found on Qaiku:

• Day 1: general situation, tasks and requirements
• Day 2: tools, persistent storage, meeting the community

As things are happening, it is also possible to follow progress on the #iks-project Qaiku channel or the #iks-project Twitter hashtag.

Technorati Tags: iks-project, semanticweb

And they report it only in human-readable way, not in fashion a browser, a proxy or a search engine would understand it. While being down, Jaiku still responds with HTTP 200 OK:

HTTP 503 Service unavailable would be much nicer. For instance, that is what Midgard produces if the database goes down.

Technorati Tags: brightkite, jaiku, http, twitter

As always, the big issue is getting the semantic information out there. In a strongly typed CMS like Midgard, many semantics can be gathered from content structure directly, but to really get there we need users to add metadata. And as users are lazy, this will happen only if it provides some direct benefit: just look at how frequently people tag their photos on Facebook. Irritating or not, this happens because the tags are actually used to promote the pictures in the news feeds of tagged people.

For this to happen in the web in general, we need to start having the search engines leverage the semantic information. Yahoo! already does this to some extent, making use of microformats and RDFa in Yahoo! Query Language and in the Search Monkey engine. This means we can already do simple semantic queries like "pages mentioning Bergius in the Helsinki area":

Actually, the Yahoo! results are quite interesting:

• My web page (which shows my current location, so it may depend on time indexed)
• Some Chinese(?) spam site that syndicates my content, proving that also in the Semantic Web people lie
• Some Plazes entries about me located in Helsinki
• Midgard events held near Helsinki

Since search engines (well, Google really) are the way people access the web, search engines are the key for making semantic information more widely available. Just look at DeWitt Clinton's survey of rel values from yesterday: Google-defined rel="nofollow" is the most popular rel value out there, even surpassing style sheet declarations. This if nothing else shows the power of search engines in promoting new standards.

Technorati Tags: google, iks, microformats, yahoo, rdfa, search, semanticweb

Nemein is participating in the project as one of the six industrial partners. For the next four years we will be working together with cool CMS companies like Day and Nuxeo, as well as with some of the leading European researchers in the field of Semantic Web.

The plan is reasonably simple: we will try to figure out what kind of semantic features would make sense to CMS end-users, and then based on that develop the user interface conventions and tools that various participating CMSs can apply. Once the case has been proven with the initial participating systems, we will then help other CMS projects to implement the same stack in order to set off a wave of semantically-enhanced websites across the Europe.

For us in the Midgard community this is well-timed, as we're anyway concentrating much effort on the new "Vinland" series in order to build a set of content management tools that can carry us through the next ten years of web evolution, as the old Midgard series did since 1999.

The project will be a good chance to leverage our experiences with Microformats and apply them to the "uppercase semantic web". Use cases range from smarter queries into content inside a Midgard repository to automatic maintenance tools utilizing semantic content in external resources.

In addition to taking Midgard to a new level as a CMS, it will be great to collaborate again with several familiar faces from the old days of OSCOM - people like Bertrand from Day, Sandro from InitMarketing and Alexander from Alkacon.

Technorati Tags: iks-project, semanticweb

After giving this a little bit of thought, I decided, Flash must be the guilty party. Flash, as we know, is Adobe's proprietary framework for building multimedia-rich applications, and iPhone simply doesn't support it. Sometimes Flash is actually used for building interesting applications, but most usage online seems to be just fancy animated banners. So, I thought, there must be a smart way to get rid of Flash. And there is: With Firefox you can use the FlashBlock extension to excise all Flash content off the pages:

With other browsers you could off course uninstall the Flash plug-in, but the problem is that sometimes there is Flash content you actually want to see. And in these cases FlashBlock works perfectly: each Flash area on a site is replaced with a box that you can activate at will. So before Flash is made obsolete by the recent advancements in HTML and Javascript, this option is the best of both worlds.

According to Sir Tim Berners-Lee, the earliest web browser was also an editor. And the late 90s Netscape Communicator followed this ideal by including a HTML editor that could publish changes to pages using the HTTP PUT method. But then the idea of editing via the web browser transformed to clunky forms and Java applet -based WYSIWYG editors, brought about by the rise of content management systems.

The problems of the user interface being part of the web page were multiple: it cluttered the produced HTML, it was possible to break by layout changes, sometimes the login and editing options were hard to find. It also meant that each and every CMS would have a completely different user interface. A problem made especially difficult if you had to use multiple systems. For example:

If you are Quim Gil, working for Nokia's maemo.org, you're living in a world of many, many CMSs. The marketing and community parts of the website are run by Midgard, the documentation wiki by MediaWiki, your own blog by WordPress, and the list goes on. While in this kind of corporate setup it has been possible to mostly unify usernames and passwords, it still means each part of your work runs with different UIs, and different usage logic. Back in 2003, I named this syndrome Frankenstein CMS.

In addition to consistency and usability, offline editing has been a big issue with most CMSs. In typical situation, it simply doesn't exist, making copy-and-paste way of taking documents with you to edit on a laptop while traveling.

There have been several initiatives in solving these problems. A quite limited, but so far successful example is the Universal Edit Button specification, making rounds earlier this summer. The idea is that CMSs would include metadata on where the editing view of a particular page was in the page itself, and then browsers would display a button leading to it. The approach has been adopted by several big players, including Wikipedia, and we also made it a part of the new CMS that will be built on Midgard 2:

Another, also somewhat limited example is the MetaWeblog API provided by most blogging platforms. It has enabled vendors to produce great offline blogging software like Ecto and MarsEdit that provide offline editing and work with almost any blogging system out there. As I write this, I'm using Ecto on my MacBook Air, and will later use it to publish this entry to my Midgard server. The Atom Publishing Protocol will likely be the successor of MetaWeblog, but appears to still keep the blog-only focus.

Back in the earlier days of OSCOM we had another initiative called Twingle. It was a XUL-based desktop CMS client that utilized WebDAV and some XML introspection files to edit and publish data on different CMSs. After a March 2003 OSCOM Sprint in Zurich we were able to demo the same client browsing and editing resources on three CMSs, including Zope and Midgard. Unfortunately, then CMS market became so hot that vendors were simply too busy to pick this up and the project died.

Luckily Michael Wechner, of OSCOM and Apache Lenya fame didn't let the idea die. He worked it onwards, naming the protocol used for conversation between the client and the server Neutron Publishing Protocol, a play on Atom. He also built a new client, Yulup, as a Firefox extension that could manage content on compatible servers.

In summer 2007 I saw a demo of Yulup and discussed Neutron with Michi and was impressed. But was back then too depressed by the Czech episode and other personal issues to press on with the idea. Having watched another year of the directions the community is taking Midgard CMS to, and how clients view CMS deployments, I'm starting to think the time would be ripe for going full forward with Neutron.

In nutshell Neutron is a metadata layer that allows a CMS to specify the actions user can take, and the methods provided for those actions. The methods can either be full-blown WebDAV, or simpler GET-and-POST that more limited CMSs can support. Neutron also provides its own authentication mechanism, though I would gladly see that overtaken by a more secure and widely supported standard like OAuth.

Widely-supported Neutron would provide huge advantages: letting the developers of a CMS to focus on the actual management and presentation layers of the system, and allowing more innovation to happen in the client end due to combined efforts from multiple CMS projects. It would allow building of different content management interfaces for different audiences or usage scenarios. Simple editing and publishing clients that could run fully in browser's AJAX space for casual bloggers or wiki editors, workflow tools for gatekeepers of corporate publishing processes, and full-blown desktop content management tools for site editors. Other special cases like mobile content management could also be provided for.

So what is needed for this future to become reality? First of all, we need to make CMS developers aware of the possibility. Then we need a killer client that every CMS provider would want to support. And then we can expand the protocol, and build additional clients to cater for special needs.

The big question is how this will happen. Industry groups in CMS space are loose or non-existing, and so they lack the muscle to push any standards. Effort by single CMS vendor would probably stay partisan, as developers of different systems are quite suspicious or ignorant of each other. But maybe a combination of the two would work: industry group, such as OSCOM organizing some compatibility hacking events for the developer community, and a company dedicated at building a killer client application. It warrants serious consideration if Nemein should be that company.

This manifesto on transforming how CMSs are used was written in an Eurostar London-Brussels train, sipping Dourthe's Bordeaux white and listening to the Karelia cycle by Sibelius.