At maemo.org we have an appstore for FOSS applications on the Maemo platform. This appstore is enabled by default on all Nokia N900s so we wanted to have some quality control. We had to create our own appstore approval process, compatible with the FOSS philosophy. Now any developer can submit an app, and anyone can test and vote. The whole process is completely transparent, auditable and visible. And it also provides a feedback channel from testers and users to the developers!

...

Midgard is a data storage service. Whether you write desktop or web applications, instead of coming up with your own file format, you just use Midgard. You can work more easily and object-based. Users have many different devices these days, so Midgard has strong replication features to synchronize between different systems. Midgard is built on top of GObject; we provide bindings to a bunch of different languages so developers can choose the tools they like - PHP, Python, Javascript. Currently (as in now, while we're talking) Qt bindings are being developed here at Akademy.

Read the whole interview.

• Loading content from Midgard CMS to Aloha Editor
• Annotating our content with Google-compatible Person RDFa elements
• Saving the content back to Midgard
• ...and finally analysing the content with FISE to find more semantic information

The hackathon participants included developers from Nemein, Gentics, Infigo, Salzburg Research and the German Research Center for Artificial Intelligence. Some screenshots:

Editing content with Aloha in Midgard

Annotating persons with the Aloha RDFa plugin

RDFa annotation created with the semantic editor

Additional semantic information suggested by FISE

All the relevant code can be found from GitHub (see also the FISE Midgard integration).

For the next 8.09.10 release we decided to put quite a bit of efforts into performance tuning, with some excellent work done by Content Control to simplify ACL handling and cache navigation information. As you can see, the result is quite impressive:

What is left to be done is some work with the multilingual content database queries. After that we should be good to go with what is probably the fastest Midgard1 ever.

The main conference will be held at the Tampere University over the weekend, and then the remaining hackweek will be in the nice Demola facility in the Finlayson district. Expect great connectivity and close proximity to all Tampere nightlife.

I'm involved with two aKademy activities:

• GeoClue presentation on the location service that can be used on any Linux desktop and is used in MeeGo. I'll also talk about what desktops can do when location is available
• Midgard Gathering during Mon-Wed of the hackweek. Midgard provides a content repository for applications that should be available also for Qt apps through GObject Introspection. With some proven performance benefits and replication support, Midgard should be interesting for any desktop or mobile app that deals with structured data

In addition there will be a Maemo / MeeGo meetup in the Plevna brewery on Friday evening. See you there!

Midgard Runtime is an application that consists of a simple Qt WebKit viewer that, when run, starts a local Midgard web server on the background and connects to it. This means that you'll have the full Midgard MVC stack available on your own desktop, in a way that is easy to install and easy to run. Get it for your favorite Linux distribution from OBS!

As a whole, the Midgard Runtime stack is quite interesting:

• SQLite is used as the default storage through the libgda4 abstraction layer
• Midgard2 is used as the content repository, brought to PHP through an extension
• Midgard MVC provides the web development framework
• Alexey Zakhlestin's AppServer is used as the pure-PHP web server
• The Qt application starts server processes on the background and displays the site in a WebKit view
• Runtime's bundle setup tool uses libraries from the Pake build system

The AppServer is particularly something that other PHP projects might find useful. Since the server itself is written in PHP you don't have any additional dependencies besides php-cli. It also means your application can use a lot more efficient caching as files have to be loaded only once, and things can be kept in memory between requests. Of course the downside is that if your PHP script dies, then the whole server is down. But that is easy to deal with by a bit of defensive programming.

At this point the Runtime is targeted at web developers interested in trying out Midgard, but eventually we'll be using the bundle system also for distributing full Midgard-powered web applications to the desktop. The progress on this can be followed on GitHub.

Compared to Nokia's Web Runtime, the Midgard Runtime should be more familiar to developers as you'll be able to write also server-side PHP code, not just JavaScript. In addition to PHP, the whole system is also accessible via any language that can handle GObject Introspection.

So, what can I do with it?

• Install a Midgard environment with $ midgard2-runtime-bundle-simple-install
• Start the runtime with $ midgard2-runtime-bundle-simple-run
• Midgard MVC and components are located in ~/.midgard2/simple-bundle/. This is where you can also place your own code

The first year of IKS has focused mostly on research and requirements gathering. As a project participant we at the Midgard project have taken this time for making our own preparations as well. Here are some initial results:

• grokking the semantic web and linked data ideas and terminology obviously required some amounts of studying
• we did a comparison of features and concepts between three content repositories together with Day Software. The similarities between Midgard, JCR and CouchDB are remarkable
• while others in the IKS sphere are focused on Java, we've started collaborating with the developers of Tracker, a GObject-based RDF triple store. Tracker would fit the Midgard architecture much better and we would be able to avoid big external dependencies
• we chose Wymeditor as the new rich text editor in Midgard CMS. Wymeditor is fast, jQuery-based and relatively sparse of features, allowing us to build the functionality we need in an integrated manner. Wymeditor is also capable of dealing with RDFa
• MgdSchema, the XML syntax for describing Midgard content types was expanded to allow us to link objects and properties to appropriate RDF ontologies
• we participated in several IKS meetings around Europe: Salzburg, Rome, Salzburg, Rome and now Paderborn
• preparations were made to migrate Midgard's default HTML templates from Microformats to RDFa

All of this aims for making Midgard compatible with the designs of the IKS project, but building as much of it on top of the pure GNOME + PHP stack as possible.

While migrating to Git we also decided to implement the Distributed Version Control model where the MidgardProject account contains "blessed" repositories of various modules of the Midgard2 ecosystem, with development happening in personal clones. Maintainers of Midgard modules will be responsible for merging changes from developers back upstream, based on pull requests. More information can be found from the developer list thread.

You can get started by following the MidgardProject account on GitHub! See also Midgard2 and Midgard MVC on Ohloh.

Midgard1 Ragnaroek maintenance and development will still happen in our old SVN environment and the Trac tool.

I went to the UDS with two goals in my mind: to discuss making the Ubuntu desktop location-aware, and to figure out what sort of role Midgard, our content repository could play in the Ubuntu space. Normally such things would be better approached by talking to upstream projects and distributions, but as Ubuntu is taking an increasingly active role in developing itself to be a comprehensive and unique operating system, it would be a better place to accomplish such wide-ranging changes.

Location-awareness was a area that many were interested in. Ubuntu already had considered GeoClue, our location service, to be a component of the distribution in Lucid Lynx, but that plan had been abandoned due to the low-maintained status of the software. However, now that GeoClue has become the geolocation framework for MeeGo things are about to change, and so there is a significant possibility to get GeoClue into Ubuntu as well.

GeoClue by itself will obviously not do much, and so making applications use it is the important thing. Some applications already support location information and could be changed to include those features by default. This includes Empathy sharing location information with instant messaging buddies, and Getting Things GNOME TODO manager being able to tag action items with location. Zeitgeist is intending to start tagging and searching activity history with location, and Gwibber also wants to include location sharing to interface with social web services like Brightkite, Qaiku and Google Latitude. Another reasonably low-hanging fruit would be to make the Ubuntu panel clock automatically change time zones as you travel.

Midgard in Ubuntu is a bit trickier question. A lot of Ubuntu has now been developed to utilize the CouchDB document database, which provides quite similar storage services as Midgard does. CouchDB's cool map/reduce algorithms and replication capabilities have enabled Ubuntu developers to quickly integrate a content repository into many applications and to provide a cloud storage and synchronization service to users of the operating system.

But at the same time CouchDB introduces some problems:

CouchDB can't seem to handle the load of Gwibber's messages, leading to excessive CPU consumption and poor performance in certain cases. For example, the overhead of computing the views causes lag when the user switches streams after Gwibber refreshes. The cost of pulling the account configuration data out of the database can also sometimes cause a noticeable lag that lasts up to four or five seconds when opening Gwibber's account manager.

CouchDB has been designed with web servers in mind, and in many situations having a constantly-running Erlang process and having to do HTTP roundtrips to access data are undesirable on more constrained computers. We ported CouchDB to Maemo a few years ago and the performance was terrible. I'd imagine the same issues on netbooks and other small hardware that Ubuntu's Light initiative targets.

On such environments Midgard might be a better option. Midgard's recent LTS release uses regular relational databases for fast performance and low footprint, utilizes important pieces of free desktop architecture like D-Bus signalling and GObject Introspection, and has been proven to work well even on low-end devices like the Nokia N900. And yet it provides full content repository APIs comparable to CouchDB, and can even handle replication with Couch. One way to make Midgard possible, then, would be to provide Python DesktopCouch APIs to it.

Another interesting technology discussed much in UDS was Puppet, the  systems administration tool. It enables developers to describe their setups in a object-oriented declarative language, and then just let Puppet to make it so. I attended two Puppet sessions, and picked up a Kindle edition of the Puppet book which I read on the train ride to the Netherlands. As my company manages lots of Midgard web servers for our customers, automating system setup and maintenance on top of Puppet and Ubuntu Server might save us lots of time and effort.

All in all, quite an enlightening two days in the event. If things work out, I hope I'll be able to spend more time in the UDS targeting the N-series of Ubuntu next fall.

Web developers will also benefit from Midgard MVC, the PHP framework that already runs services like Qaiku.com.

The release includes:

• Content Repository API bindings for the following programming languages: C, Python, PHP, C# and Vala. D-Bus signals are used to inform different Midgard2 applications about things happening in the repository, enabling for example a PHP website and a Python background process to communicate with each other.
• Midgard MVC, an elegant framework for PHP web applications. Midgard MVC includes interfaces for loadable components, hierarchical sub-requests, a forms system and much more.
• Midgard Runtime that combines the Midgard MVC, a PHP application server and a WebKit UI to provide a full Midgard web development environment on the desktop.

This release benefits greatly from new technologies happening in the GNOME sphere: some of the language bindings are created using GObject Introspection, Vala has made development of new features faster and libgda4 makes all database operations more efficient.

Packages for various Linux distributions are already trickling into Midgard's OpenSuse Build Service repositories and Maemo Packages.

A good place to start the Midgard2 journey is my Midgard2 Content Repository for Python tutorial.

Photo: Squirrel by John-Morgan on Flickr (the release codename Ratatoskr is a mythological squirrel)

Midgard2

The next Midgard2 release, 10.05 "Ratatoskr" will be a long-term support release, intended to provide a reliable base for building web and mobile applications utilizing the content repository. It will consist of:

Midgard Core, the actual content repository library that abstracts database connections, provides object-oriented storage and retrieval interfaces and handles things like D-Bus notifications of content changed by other applications.

Midgard PHP and Midgard Python, language bindings providing the Midgard content repository services to these popular scripting languages. This means the Midgard content repository can easily be used within any PHP web application, Python maemo application or with the Pylons MVC framework.

Midgard Mono, C# bindings to the Midgard content repository allowing better interfacing with Microsoft technologies.

GObject Introspection allowing us to provide other language bindings including Vala and Javascript.

Midgard MVC, an elegant framework for PHP web applications. Midgard MVC includes interfaces for loadable components, hierarchical sub-requests, a forms system and much more. You can see it in action for example on Qaiku.com

Midgard Runtime that combines the Midgard MVC, a PHP application server and a WebKit UI to provide a full Midgard web development environment on the desktop.

Once the release is in shape a major focus will be put in ease of installation and documentation. We already have Midgard2 running on Windows in addition to Linux and Mac, but it will remain to be seen if the port makes it in time for release.

Midgard3

After having a stable LTS release of Midgard2 out there the community will start working on Midgard3, which takes the concepts from Midgard2 even further with several important additions:

Workspaces, kind of "layered virtual databases" inside one content repository. These can be used on the CMS level of Midgard a bit like git branches are used.

Binary GUIDs for more efficient storage of the unique object identifiers in Midgard.

MgdSchemas stored in the database instead of XML files. This will make schema editing, sandboxing and replication between installations a lot easier.

MidgardQuerySelect, the new better database select interface to replace Midgard Query Builder.

Midgard CMS, the content management user interface built on top of Midgard3 and the Midgard MVC. We will provide interesting new concepts like UI constructed fully based on introspecting the CMS-generated pages and a git-like model where anybody can edit but pushing to production requires additional privileges.

Most applications will need to be adapted to work on Midgard3, which is why the LTS release will be there to ease the way, just like we did with Ragnaroek on the Midgard1 series.

The estimated first release of Midgard3 will be in December 2010.

...developers have become crippled by being able to only think of data in terms of Rows and Columns. There's a multitude of database paradigms: Graphs, Trees, Objects, and so on. Furthermore, databases limit developers to SQL, which is great for certain kinds of set mathematics and not much else. In order to overcome fundamental limitations of DBs, things like Conditionals, Iteration, String Manipulation, and more have been hacked into what was at first an elegant set mathematics language.

Algorithms (such as MapReduce) which make data analysis scalable are highly unintuitive in SQL. The limitations in the environment lead to a limitation in engineers' ability to solve "big data" problems.

This may be true if you're building the next Twitter or Facebook. But how many of us really do?
In the quite ranty article I Can't Wait for NoSQL to Die Ted Dziuba has a point:

by replacing MySQL or Postgres with a different, new data store, you have traded a well-enumerated list of limitations and warts for a newer, poorly understood list of limitations and warts, and that is a huge business risk.

This is exactly the reason we decided to build Midgard2 on top of a relational database. While it does constrain us in some cases, we can still provide a quite reasonably full-featured content repository this way.

And being built on SQL means we know the system scales down to mobile devices and all the way up to major business systems like Lufthansa's global marketing budget tracking. That is something very few NoSQL systems will be able to boast.

To access the template system just click Folder > Edit layout template in the Midgard toolbar:

This will take you into Asgard, the Midgard administration interface. There you'll be shown what elements are used by the components running on your site:

Clicking on an element name will let you create it. Then you'll be shown the default contents of that element to make overriding them easier. The language used in the templates is obviously PHP:

When you save your element it will replace the component's default output in the folders using that particular style. You can also do nice things like including contents from another page using Dynamic Loading.

What about file attachments?

Asgard also allows you to add files (images, CSS, javascript, whatever) to your template. To do this click the Attachments button when you're in the style. There you'll be able to manage existing file attachments and to add new ones:

If you're dealing with a text-based file (CSS, javascript, XML, ...) Asgard will also allow you to edit it directly on the web:

...and finally when you're done with your file and want to use it in some of your templates, the URL is available easily from the interface:

The new version looks quite nice and fresh. Notice the privacy information on the right-hand side, which is relevant as Qaiku allows channels and profiles that are private or invitation-only:

Technically the new version is also remarkable as it is the first major website to run fully on top of the legacy-free Midgard2 platform. So yes, every entry you see there is a GObject. And D-Bus signals fly when you post.

On to the challenge, then

To highlight Qaiku's threading, conversational nature I started a new "On my travels, I have" thread for sharing your most extraordinary travel experiences. This is not on Twitter or Buzz as with Qaiku it is so easy to keep the conversation together and accessible for the future as well.

To contribute, sign up on Qaiku, go to the thread and add your experiences as a comment. If you have a link or picture to include, you can also do so. My first entry was:

seen ice descend from the heavens and provide us with cold beer on a hot day in Lesotho

Will be interesting to see what comes out of this :-)

The idea of a content repository is that instead of coming up with new, isolated file formats or database setups for your application you can just work with objects and signals, and let Midgard handle the rest. This is something that lots of people are doing with CouchDB as well, but we feel Midgard, with its light footprint and native APIs for languages like Python, C, Vala and PHP fits better in the mobile applications context.

Installing Midgard

Midgard packages are available for many different Linux distributions through the OpenSuse Build Service. To find the right repository for your setup, go to the OBS project page. For example, on my Ubuntu Karmic netbook the URL to add to apt sources.list is deb http://download.opensuse.org/repositories/home:/midgardproject:/mjolnir/xUbuntu_9.10/ ./. Then I just:

sudo apt-get update
sudo apt-get install python-midgard2

Midgard is also available in Maemo extras and for OS X on MacPorts.

Defining a schema

The first thing when developing a Midgard application is to define your storage objects. This is done using the MgdSchema XML format. In this case we're doing a simple "attendee" object that amends Midgard's built-in person record with information related to the conference:

<?xml version="1.0" encoding="UTF-8"?>
<Schema xmlns="http://www.midgard-project.org/repligard/1.4">
    <type name="openbossa_attendee" table="openbossa_attendee">
        <property name="id" type="unsigned integer" primaryfield="id">
            <description>Local non-replication-safe database identifier</description>
        </property>
        <property name="person" type="unsigned integer" link="midgard_person:id">
            <description>Person attending the event</description>
        </property>
        <property name="registration" type="datetime">
            <description>Registration date of the attendee</description>
        </property>
        <property name="likesbeer" type="boolean">
            <description>Whether the attendee likes beer</description>
        </property>
    </type>
</Schema>

Then we just save this XML file into /usr/share/midgard2/schema/ so that Midgard will find it.

Initiating the repository connection

Once the MgdSchema is in place it is time to import antigravity and start hacking in Python. The code works pretty much in the same way in other languages Midgard is available for, but Python is used here for the sake of simplicity. First we load the Midgard extension:

import _midgard as midgard

Then we setup the repository connection. With these settings we will store our content into an SQLite database located in ~/.midgard2/data/midgardexample.db:

configuration = midgard.config()
configuration.dbtype = 'SQLite'
configuration.database = 'midgardexample'

# Open a Midgard repository connection with our config
connection = midgard.connection()
connection.open_config(configuration)

As this is the first time we're interacting with the repository we need to tell Midgard to prepare the storage for itself and also for our new openbossa_attendee class:

midgard.storage.create_base_storage()
midgard.storage.create_class_storage('midgard_person')
midgard.storage.create_class_storage('midgard_parameter')
midgard.storage.create_class_storage('openbossa_attendee')

Interacting with data

First we create a person object with our attendee:

person = midgard.mgdschema.midgard_person()
person.firstname = 'Leif'
person.lastname = 'Eriksson'
person.create()

Then we create our attendee object and link that with the person we just created:

attendee = midgard.mgdschema.openbossa_attendee()
attendee.person = person.id
attendee.likesbeer = True
attendee.create()

Querying data

Later we'll want to find out about all Leifs attending the event. We do this by using the Midgard query builder:

qb = midgard.query_builder('openbossa_attendee')
qb.add_constraint('person.firstname', '=', 'Leif')
attendees = qb.execute()

The query builder returns us a list of matching attendee objects. We can go through them and also fetch the associated persons:

for attendee in attendees:
    person = midgard.mgdschema.midgard_person()
    person.get_by_id(attendee.person)
    if attendee.likesbeer:
        print "%s, %s is attending the event" % (person.lastname, person.firstname)

Then we can update the persons with their email addresses:

    person.email = 'leif@vinland.no'
    person.update()

For basic data handling, that's it! When you need more, you can extend objects with file attachments or parameters. You can also create joined records using Midgard views. Midgard provides D-Bus signals, transactions, centralized metadata, synchronization and many other things.

The answer is, we use an open-source platform called "Midgard." We picked it nearly ten years ago, and it has held up fairly well.
...

One of the things we like about Midgard actually makes it rather unsuitable for many simpler publishing scenarios: it is highly object-oriented. This allows us to run multiple sites off largely a single codebase -- at the cost of quite user-unfriendly administrative and authoring facilities.

Also, Midgard is very much a development platform, and we have had to create a fair amount of custom code, especially to handle structured content. In that regard, our CMS experience probably resemble yours. As an industry we remain very far from plug-and-play content management technology for all but the simplest of websites.

While the post contains many negative points about older Midgard (the UIs are a bit better now than they used to be, quite a lot of development has since been happening especially in the LTS branch), it is remarkable that CMS Watch has been able to run their services through the same CMS setup for ten years. This really shows the durability and commitment to long-term stability we have in the Midgard community. We've been doing this for more than ten years, and will likely keep going for quite a bit longer.

As for usability and popularity of Midgard, there is quite little we can do about it in the Midgard1 area, as that is now in long-term support phase that won't allow major changes. But Midgard2 is a new world with new opportunities. Midgard's content repository is pretty much there already, as is the MVC layer, and this spring we should be able to unveil the new, quite revolutionary CMS concept as well. Watch this blog for updates!

First of all, go to meego.com and click login:

Select the "Log in using OpenID" option, and provide your maemo.org OpenID URL:

Then the request will be redirected to maemo.org where the site will check your credentials and ask whether to relay your information on to meego.com:

And that's it, suddenly you can use your maemo.org account with meego.com!

The same OpenID provider component can also be utilized on any other Midgard-powered website.

The idea behind the Midgard content repository is that instead of coming up with your own file formats you can just keep working with objects and signals, and let the repository deal with the rest.

It is always fun to go to Brazil and meet the vibrant free software community there. The plan is to fly over this weekend, spend a few days in Sao Paulo and then head for the Amazon. Feel free to ping me if you're around.

This is another case of location context being used to serve users better. To figure out where the user is coming from the site uses a combination of IP positioning and browser geolocation, while weather information comes from Foreca's feeds. Map visualization uses CloudMade's OpenStreetMap-based maps. And of course the whole thing runs on the stable Ragnaroek series of Midgard.

In other news, Jos et jaksa is another pretty interesting recent site launch, especially for the fact that it is the first-ever website running on the legacy-free Midgard2 platform and Midgard MVC.

If you’re thinking The Tablet is just a big iPhone, or just Apple’s take on the e-reader, or just a media player, or just anything, I say you’re thinking too small — the equivalent of thinking that the iPhone was going to be just a click wheel iPod that made phone calls. I think The Tablet is nothing short of Apple’s reconception of personal computing.

What I find most interesting are the view that the Tablet may bring new computer interaction paradigms. Again from Daring Fireball:

Our “desktop” computers’ human interfaces haven’t fundamentally changed since 1984 — keyboard and mouse/trackpad for input, overlapping draggable resizable windows on-screen, and a hierarchical file system where you create and manage “document files”. Have you ever sat back, scratched your chin, and wondered when the computer industry will break free of these current interfaces — which can be a hassle even for experts, and downright confusing (e.g. click vs. double-click) for the non-experts? Surely no one expects the computer interfaces of, say, 50 years hence to be based on these same metaphors and input methods. What’s the next step?

A touchscreen tablet isn't really suited for the WIMP paradigm as for example text entry is quite difficult, and you probably want larger, thumb-friendly user interface elements. This is where Microsoft's Tablet PC initiative failed, trying to bring the regular WIMP user interface to the tablet.

Instead what seems to be happening is that all the Wiis, iPhones, and N900s are now heading us towards a post-WIMP world. Instead of indirect manipulation by mouse and keyboard we can now interact with our applications using the more natural ways of touching things on screen or moving the device around.

This innovation will not be limited only to mobile APIs, web applications can already now know whether user is accessing them via a WIMP system or a touchscreen device thanks to CSS media queries and Javascript orientation events in latest Firefox.

The user interface innovation that is arriving thanks to these new interaction possibilities is quite promising, though it will probably take a while before we know what things actually work, and what are just fun demos.

If you're thinking about new kinds of user interfaces, it might be a good time to read papers like Noncommand User Interfaces (Jakob Nielsen, 1993) and Magic Ink (Bret Victor, 2006).

I certainly am as we are in the process of defining a new kind of CMS UI for Midgard 2.

Update: Gizmodo has a very nice article on Jef Raskin's information appliance concept and the evolution of GUIs.

• Midgard2 finally became a reality, bringing us a fully legacy-free modern Midgard implementation. There were two releases: 9.03 Vinland and 9.09 Mjolnir. Midgard2 works just fine also in desktop applications and mobile devices like the Nokia N900
• Ragnaroek, the long-term support version of Midgard1 kept chugging along with four new maintenance releases and total of 566 issues closed. Ragnaroek will be maintained until October 2013
• The Midgard Project turned ten years old in May. A gala evening and party was held in Helsinki, Finland
• Midgard's content repository gained bindings to two new programming languages alongside existing PHP, C and Python: Objective-C in March and Vala in December
• Midgard was one of the CMSs admitted to the EU-funded Interactive Knowlegde project that aims to increase semantic capabilities in content management
• Qaiku, a conversational microblogging platform powered by Midgard MVC was launched in March
• We had two Midgard Gatherings: in March in Linköping, Sweden and in October in Helsinki, Finland
• Ragnaland became a way to bridge the gap between Midgard1 and Midgard2 by enabling running of MidCOM applications on top of Midgard MVC
• Midgard was presented in several conferences including Gran Canaria Desktop Summit, FrOSCon, OpenMind and FSCONS
• Founding meeting for an association to support the Midgard project was held in December. The association will provide membership to all active Midgard contributors, and an alumni membership for people who have been part of the project in the past

Happy new year to everybody in the Midgard world!