Weblog: category "midgard"

Jos Poortvliet did an interview with me for dot KDE in this summer's aKademy and it has been online for a while now. In it we discuss things like Midgard as a storage engine for desktop applications, and Maemo's open QA process for Downloads applications. Some excepts:

At maemo.org we have an appstore for FOSS applications on the Maemo platform. This appstore is enabled by default on all Nokia N900s so we wanted to have some quality control. We had to create our own appstore approval process, compatible with the FOSS philosophy. Now any developer can submit an app, and anyone can test and vote. The whole process is completely transparent, auditable and visible. And it also provides a feedback channel from testers and users to the developers!

...

Midgard is a data storage service. Whether you write desktop or web applications, instead of coming up with your own file format, you just use Midgard. You can work more easily and object-based. Users have many different devices these days, so Midgard has strong replication features to synchronize between different systems. Midgard is built on top of GObject; we provide bindings to a bunch of different languages so developers can choose the tools they like - PHP, Python, Javascript. Currently (as in now, while we're talking) Qt bindings are being developed here at Akademy.

Read the whole interview.

To bring CMS editing to the next level, the IKS project is working on a semantic HTML5 editor. This week we had a hackathon in Helsinki focusing on implementing our ideas with the Aloha Editor. In addition to enjoying the hot summer weather here, we accomplished quite a bit and in the end were able to present the whole pipeline of:

• Loading content from Midgard CMS to Aloha Editor
• Annotating our content with Google-compatible Person RDFa elements
• Saving the content back to Midgard
• ...and finally analysing the content with FISE to find more semantic information

The hackathon participants included developers from Nemein, Gentics, Infigo, Salzburg Research and the German Research Center for Artificial Intelligence. Some screenshots:

Editing content with Aloha in Midgard

Annotating persons with the Aloha RDFa plugin

RDFa annotation created with the semantic editor

Additional semantic information suggested by FISE

All the relevant code can be found from GitHub (see also the FISE Midgard integration).

Midgard 8.09 is an industrial CMS that is now in Long-Term Supported stage, with the community maintaining it until 2013. As we all know, performance is a feature, and with a CMS framework that has lived through many changes including transitions from PHP4 to 5.2 and from Classic Midgard era to the modern APIs, there is a lot to do.

For the next 8.09.10 release we decided to put quite a bit of efforts into performance tuning, with some excellent work done by Content Control to simplify ACL handling and cache navigation information. As you can see, the result is quite impressive:

What is left to be done is some work with the multilingual content database queries. After that we should be good to go with what is probably the fastest Midgard1 ever.

We tried to get the combined GUADEC and aKademy conferences to Tampere in 2009, but a warmer place unfortunately won. However, we will be hosting this year's aKademy so at least KDE and Qt fans will get to enjoy this beautiful northern industrial city.

The main conference will be held at the Tampere University over the weekend, and then the remaining hackweek will be in the nice Demola facility in the Finlayson district. Expect great connectivity and close proximity to all Tampere nightlife.

I'm involved with two aKademy activities:

• GeoClue presentation on the location service that can be used on any Linux desktop and is used in MeeGo. I'll also talk about what desktops can do when location is available
• Midgard Gathering during Mon-Wed of the hackweek. Midgard provides a content repository for applications that should be available also for Qt apps through GObject Introspection. With some proven performance benefits and replication support, Midgard should be interesting for any desktop or mobile app that deals with structured data

In addition there will be a Maemo / MeeGo meetup in the Plevna brewery on Friday evening. See you there!

Midgard2 10.05.1 was released yesterday, bringing a long-waited feature finally to the Midgard installation packages: the Midgard Runtime.

Midgard Runtime is an application that consists of a simple Qt WebKit viewer that, when run, starts a local Midgard web server on the background and connects to it. This means that you'll have the full Midgard MVC stack available on your own desktop, in a way that is easy to install and easy to run. Get it for your favorite Linux distribution from OBS!

As a whole, the Midgard Runtime stack is quite interesting:

• SQLite is used as the default storage through the libgda4 abstraction layer
• Midgard2 is used as the content repository, brought to PHP through an extension
• Midgard MVC provides the web development framework
• Alexey Zakhlestin's AppServer is used as the pure-PHP web server
• The Qt application starts server processes on the background and displays the site in a WebKit view
• Runtime's bundle setup tool uses libraries from the Pake build system

The AppServer is particularly something that other PHP projects might find useful. Since the server itself is written in PHP you don't have any additional dependencies besides php-cli. It also means your application can use a lot more efficient caching as files have to be loaded only once, and things can be kept in memory between requests. Of course the downside is that if your PHP script dies, then the whole server is down. But that is easy to deal with by a bit of defensive programming.

At this point the Runtime is targeted at web developers interested in trying out Midgard, but eventually we'll be using the bundle system also for distributing full Midgard-powered web applications to the desktop. The progress on this can be followed on GitHub.

Compared to Nokia's Web Runtime, the Midgard Runtime should be more familiar to developers as you'll be able to write also server-side PHP code, not just JavaScript. In addition to PHP, the whole system is also accessible via any language that can handle GObject Introspection.

So, what can I do with it?

• Install a Midgard environment with $ midgard2-runtime-bundle-simple-install
• Start the runtime with $ midgard2-runtime-bundle-simple-run
• Midgard MVC and components are located in ~/.midgard2/simple-bundle/. This is where you can also place your own code

Last year we became a partner in the European Commission -funded Interactive Knowledge Systems project aiming to increase semantic capabilities in Open Source CMSs and vendors.

The first year of IKS has focused mostly on research and requirements gathering. As a project participant we at the Midgard project have taken this time for making our own preparations as well. Here are some initial results:

• grokking the semantic web and linked data ideas and terminology obviously required some amounts of studying
• we did a comparison of features and concepts between three content repositories together with Day Software. The similarities between Midgard, JCR and CouchDB are remarkable
• while others in the IKS sphere are focused on Java, we've started collaborating with the developers of Tracker, a GObject-based RDF triple store. Tracker would fit the Midgard architecture much better and we would be able to avoid big external dependencies
• we chose Wymeditor as the new rich text editor in Midgard CMS. Wymeditor is fast, jQuery-based and relatively sparse of features, allowing us to build the functionality we need in an integrated manner. Wymeditor is also capable of dealing with RDFa
• MgdSchema, the XML syntax for describing Midgard content types was expanded to allow us to link objects and properties to appropriate RDF ontologies
• we participated in several IKS meetings around Europe: Salzburg, Rome, Salzburg, Rome and now Paderborn
• preparations were made to migrate Midgard's default HTML templates from Microformats to RDFa

All of this aims for making Midgard compatible with the designs of the IKS project, but building as much of it on top of the pure GNOME + PHP stack as possible.

Now that Midgard2 is at Long-Term Supported stage it was time to finally make the jump and migrate our development efforts to happen on top of Git, the fast version control system. To maximize project visibility and enable easy tool access we chose GitHub as the Git hosting provider.

While migrating to Git we also decided to implement the Distributed Version Control model where the MidgardProject account contains "blessed" repositories of various modules of the Midgard2 ecosystem, with development happening in personal clones. Maintainers of Midgard modules will be responsible for merging changes from developers back upstream, based on pull requests. More information can be found from the developer list thread.

You can get started by following the MidgardProject account on GitHub! See also Midgard2 and Midgard MVC on Ohloh.

Midgard1 Ragnaroek maintenance and development will still happen in our old SVN environment and the Trac tool.

Last week was the Ubuntu Developer Summit targeted at planning how the next iteration of the operating system, Maverick Meerkat, targeted at a October 10th 2010 release, would look like. The event was held in a spa resort off in the countryside near Brussels. A place where the developers were comfortably separated from the busy towns by forests and country roads.

I went to the UDS with two goals in my mind: to discuss making the Ubuntu desktop location-aware, and to figure out what sort of role Midgard, our content repository could play in the Ubuntu space. Normally such things would be better approached by talking to upstream projects and distributions, but as Ubuntu is taking an increasingly active role in developing itself to be a comprehensive and unique operating system, it would be a better place to accomplish such wide-ranging changes.

Location-awareness was a area that many were interested in. Ubuntu already had considered GeoClue, our location service, to be a component of the distribution in Lucid Lynx, but that plan had been abandoned due to the low-maintained status of the software. However, now that GeoClue has become the geolocation framework for MeeGo things are about to change, and so there is a significant possibility to get GeoClue into Ubuntu as well.

GeoClue by itself will obviously not do much, and so making applications use it is the important thing. Some applications already support location information and could be changed to include those features by default. This includes Empathy sharing location information with instant messaging buddies, and Getting Things GNOME TODO manager being able to tag action items with location. Zeitgeist is intending to start tagging and searching activity history with location, and Gwibber also wants to include location sharing to interface with social web services like Brightkite, Qaiku and Google Latitude. Another reasonably low-hanging fruit would be to make the Ubuntu panel clock automatically change time zones as you travel.

Midgard in Ubuntu is a bit trickier question. A lot of Ubuntu has now been developed to utilize the CouchDB document database, which provides quite similar storage services as Midgard does. CouchDB's cool map/reduce algorithms and replication capabilities have enabled Ubuntu developers to quickly integrate a content repository into many applications and to provide a cloud storage and synchronization service to users of the operating system.

But at the same time CouchDB introduces some problems:

CouchDB can't seem to handle the load of Gwibber's messages, leading to excessive CPU consumption and poor performance in certain cases. For example, the overhead of computing the views causes lag when the user switches streams after Gwibber refreshes. The cost of pulling the account configuration data out of the database can also sometimes cause a noticeable lag that lasts up to four or five seconds when opening Gwibber's account manager.

CouchDB has been designed with web servers in mind, and in many situations having a constantly-running Erlang process and having to do HTTP roundtrips to access data are undesirable on more constrained computers. We ported CouchDB to Maemo a few years ago and the performance was terrible. I'd imagine the same issues on netbooks and other small hardware that Ubuntu's Light initiative targets.

On such environments Midgard might be a better option. Midgard's recent LTS release uses regular relational databases for fast performance and low footprint, utilizes important pieces of free desktop architecture like D-Bus signalling and GObject Introspection, and has been proven to work well even on low-end devices like the Nokia N900. And yet it provides full content repository APIs comparable to CouchDB, and can even handle replication with Couch. One way to make Midgard possible, then, would be to provide Python DesktopCouch APIs to it.

Another interesting technology discussed much in UDS was Puppet, the  systems administration tool. It enables developers to describe their setups in a object-oriented declarative language, and then just let Puppet to make it so. I attended two Puppet sessions, and picked up a Kindle edition of the Puppet book which I read on the train ride to the Netherlands. As my company manages lots of Midgard web servers for our customers, automating system setup and maintenance on top of Puppet and Ubuntu Server might save us lots of time and effort.

All in all, quite an enlightening two days in the event. If things work out, I hope I'll be able to spend more time in the UDS targeting the N-series of Ubuntu next fall.

Midgard2 10.05 "Ratatoskr" was released yesterday, moving the Midgard Content Repository into long-term supported state as outlined in my recent post. Ratatoskr should provide a stable storage system for both desktop and mobile application developers.

Web developers will also benefit from Midgard MVC, the PHP framework that already runs services like Qaiku.com.

The release includes:

• Content Repository API bindings for the following programming languages: C, Python, PHP, C# and Vala. D-Bus signals are used to inform different Midgard2 applications about things happening in the repository, enabling for example a PHP website and a Python background process to communicate with each other.
• Midgard MVC, an elegant framework for PHP web applications. Midgard MVC includes interfaces for loadable components, hierarchical sub-requests, a forms system and much more.
• Midgard Runtime that combines the Midgard MVC, a PHP application server and a WebKit UI to provide a full Midgard web development environment on the desktop.

This release benefits greatly from new technologies happening in the GNOME sphere: some of the language bindings are created using GObject Introspection, Vala has made development of new features faster and libgda4 makes all database operations more efficient.

Packages for various Linux distributions are already trickling into Midgard's OpenSuse Build Service repositories and Maemo Packages.

A good place to start the Midgard2 journey is my Midgard2 Content Repository for Python tutorial.

Photo: Squirrel by John-Morgan on Flickr (the release codename Ratatoskr is a mythological squirrel)

Two weeks ago we had the Midgard Gathering in Poland, and some big decisions were made there. I've been meaning to blog about them, but the volcano eruption in Iceland kept me busy by providing an interesting trip through New York and Moscow.

Midgard2

The next Midgard2 release, 10.05 "Ratatoskr" will be a long-term support release, intended to provide a reliable base for building web and mobile applications utilizing the content repository. It will consist of:

Midgard Core, the actual content repository library that abstracts database connections, provides object-oriented storage and retrieval interfaces and handles things like D-Bus notifications of content changed by other applications.

Midgard PHP and Midgard Python, language bindings providing the Midgard content repository services to these popular scripting languages. This means the Midgard content repository can easily be used within any PHP web application, Python maemo application or with the Pylons MVC framework.

Midgard Mono, C# bindings to the Midgard content repository allowing better interfacing with Microsoft technologies.

GObject Introspection allowing us to provide other language bindings including Vala and Javascript.

Midgard MVC, an elegant framework for PHP web applications. Midgard MVC includes interfaces for loadable components, hierarchical sub-requests, a forms system and much more. You can see it in action for example on Qaiku.com

Midgard Runtime that combines the Midgard MVC, a PHP application server and a WebKit UI to provide a full Midgard web development environment on the desktop.

Once the release is in shape a major focus will be put in ease of installation and documentation. We already have Midgard2 running on Windows in addition to Linux and Mac, but it will remain to be seen if the port makes it in time for release.

Midgard3

After having a stable LTS release of Midgard2 out there the community will start working on Midgard3, which takes the concepts from Midgard2 even further with several important additions:

Workspaces, kind of "layered virtual databases" inside one content repository. These can be used on the CMS level of Midgard a bit like git branches are used.

Binary GUIDs for more efficient storage of the unique object identifiers in Midgard.

MgdSchemas stored in the database instead of XML files. This will make schema editing, sandboxing and replication between installations a lot easier.

MidgardQuerySelect, the new better database select interface to replace Midgard Query Builder.

Midgard CMS, the content management user interface built on top of Midgard3 and the Midgard MVC. We will provide interesting new concepts like UI constructed fully based on introspecting the CMS-generated pages and a git-like model where anybody can edit but pushing to production requires additional privileges.

Most applications will need to be adapted to work on Midgard3, which is why the LTS release will be there to ease the way, just like we did with Ragnaroek on the Midgard1 series.

The estimated first release of Midgard3 will be in December 2010.