Henri Bergius - Openpsa

Building a smarter workplace

henri.bergius@iki.fi (Henri Bergius) — Fri, 08 Mar 2013 00:00:00 +0000

As part of the SmarcoS project, we have been investigating how to make workplaces smarter through sensors and context awareness. Here is a video showing what we’ve built:

The idea here is to facilitate collaboration and smoother project communications through various different tools that I’ll describe below. While this already does a lot, it is obviously only the first step on the path to making offices smarter.

Office presence

An important part of collaboration is to know who is where. Maybe some people are having a lunch break, or are working remotely? The Office Presence Display system knows these things, thanks to various sensors:

Bluetooth sensors can see smartphones and other mobile devices in the space
WiFi sensor can see what computers are connected
Google Talk sensor knows who are connected and active on their work accounts

With these, we know pretty well when you arrive to the office, and when you leave. Thanks to the sensor watching the company instant messaging system, we also know when people working remotely are available.

In addition to letting people know where the other members of the team are, this system can help with the perpetual annoyance in many companies: having to fill timesheets. We have a logger process that listens to the sensors, and logs the data into a CouchDB database. From there you can easily visualize working hours and availability trends:

In many situations it is of course not enough to know whether people are present, but also to know what they’re working on. The next component of the system helps with that.

Electronic Kanban wall

Instead of the clumsy waterfall style, more and more companies are managing their projects in tight, recurring iterations. A Kanban wall is a great way to keep track of tasks as they move through the process, and to see who is doing what, and what could be possible to do next. The One day in Kanban Land post explains the concept well.

If everybody working on a project sits in the same room, and that room is also where all decisions about the project are made, then the traditional solution of whiteboards and Post-It notes is probably the best way to visualize Kanban. But for distributed teams and more flexible work, an electronic version is a lot better option. This way everybody can see the Kanban wall in its current state from their computer, a TV in an office room, or a tablet.

Our Kanban wall implementation provides exactly that. The same wall (or, in case of multiple projects or teams, a set of walls) is available through any web browser. You can also display it on a big screen in an office or a meeting room, and control that screen with the Kinect Air Cursor.

Now, for most companies this is not meant to be the master database of projects and tasks, but instead just to be an alternative view and controlling tool to wherever your project information resides. Maybe you’re already using Pivotal Tracker, Basecamp, or OpenPSA? The Kanban wall server has a full REST API available, and so it is easy to integrate with any existing system. If a task is moved on the Kanban wall, it can be updated to the project tracking system, and vice versa.

If you’re using both systems, then the Kanban wall is also aware of the presence information. We show a differently colored border around the “person tokens” depending on the availability state. You can also set different Work-in-Progress limits for different people and states.

Interested yet?

So far we’ve only trialed the system in our own office, but are now looking for some pilot customers to try the system out. If you’re interested in making your workplace smarter, get in touch.

For those who like to tinker with these things, all the sensor software is available as open source at http://github.com/nemein.

Open Advice

henri.bergius@iki.fi (Henri Bergius) — Mon, 19 Mar 2012 00:00:00 +0000

I seem to have not blogged about this, but Open Advice, our book on Free and Open Source Software: what we wish we had known when we started, was published last month.

The book was edited by Lydia Pintscher and includes essays from 42 authors, many of whom you'll recognize if you tend to go to FOSS conferences. The LWN book review concludes:

Open Advice is a book that will be helpful to those who are new to FOSS, but, because of the individual voices, styles, and tones, it doesn't read like a "how to". It could even be recommended to those who aren't necessarily interested in contributing, but are curious about what this "free software thing" is all about. It is, in short, a great book for a variety of audiences and the (mostly) two or three page essays make it easy to read, while the anecdotes and recollections personalize it. The authors, editor, and everyone else who helped should be very pleased with the result. Readers will be too.

I probably shouldn't give the ending away, but my essay on cross-project collaboration, a subject I've also blogged about, ends with:

Good luck with breaking down the project boundaries! In most cases it works if your ideas are good and presented with an open mind. But even if you do not find a common ground, as long as your implementation solves the use case for you it has not been in vain. After all, delivering software, and delivering great user experience is what counts.

The book is licensed under CC-BY-SA, and is available as free download in ePub, mobi and PDF formats, and as paperback from Lulu. The book sources are available on GitHub, patches welcome!

Business analytics with CouchDB and NoFlo

henri.bergius@iki.fi (Henri Bergius) — Wed, 21 Sep 2011 00:00:00 +0000

The purpose of business analytics is to find data from the company’s information systems that can be used to support decision making. What customers buy most? What do they do before a buying decision? What are the signs that a customer may be leaving?

For the last month we’ve been working in Salzburg to build such a system, the Intelligent Project Controlling Tool needed for running large collaborative research projects like IKS. Since the design we went with can be reused for other business analytics needs, I wanted to write a bit about it.

But first, here is how our system looks like:

Where does the data come from?

There are many ways to gather business data. Often the information systems already contain the data needed. But it may also be hidden in a jungle of spreadsheets. Or maybe some data is simply not available, and has to be filled in manually.

Handling all these cases in one system is a tricky question. To solve it, we went with a two-layered strategy:

All data used for analytics is stored as Linked Data in a CouchDB system
NoFlo workflows are used for gathering data from the diverse sources and convert it to the format needed

In IKS’s case, much of the data was available in a series of spreadsheets. With these, we built the necessary workflows for first converting the spreadsheets into XML with Apache Tika, and then extracting the information from them in a sensible subset of JSON-LD.

Because IKS is a collaborative project, information needs to be gathered from a diverse group of partner organizations. Some of them have systems that provide the needed APIs (like Basecamp, which we use), and we can just periodically import the data. But with many we decided on a simple data interchange approach: spreadsheets handled over email.

In this approach, user files a data request into the system. This gets picked up by NoFlo, which sends an email with the appropriate spreadsheet template to the partner. Then it starts waiting for a reply. When a reply arrives, it extracts the data from the attached spreadsheet and imports it to the system.

Our NoFlo processes are mostly initiated by the CouchDB change notification API. We keep them running persistently using forever Node, so whenever some operation needs to be run it happens nearly immediately.

Ensuring data consistency

With any automation, and especially with the email-based data interchange, things can go wrong. Because of this we tag all data that we receive with its origin, whether it was some automated operation or an imported spreadsheet. These origins are called execution documents. Users can browse all completed workflow executions and see what data came in from them. These can then be either accepted or rejected.

This way if some partner accidentally sends faulty data, or something else breaks, the incorrect information received can be easily removed. CouchDB’s versioning capabilities help here.

Analyzing the data

CouchDB is built on top of the concept of map/reduce. Here you can modify and combine the data in lots of different ways using simple JavaScript functions. In our case we elected to write all our CouchDB code in CoffeeScript for simplicity. For example, here is the reduce function in CoffeeScript that counts totals of time planned, time used, and time left per task or partner in a project:

(keys, values, rereduce) ->
    roundNumber = (rnum, rlength) ->
        Math.round(parseFloat(rnum) * Math.pow(10, rlength)) / Math.pow(10, rlength)
    data =
        planned: 0.0
        spent: 0.0
        left: 0.0

    if rereduce
        for reducedData in values
            data.planned += reducedData.planned
            data.spent += reducedData.spent
        data.left = data.planned - data.spent
        return data

    for doc in values
        if doc['@type'] is 'effortallocation'
            data.planned += roundNumber doc.value, 1
        if doc['@type'] is 'effort'
            data.spent += roundNumber doc.value, 1
    data.left = roundNumber data.planned - data.spent, 1
    return data

If you figure out a new way to look at the data you have, simply write the needed map and reduce functions and save them into the database. CouchDB will then run them against existing data and produce numbers.

Data visualizations

Numbers are good, but to really see the information buried in them you need some visualizations. For this we decided to follow the CouchApp idea where the user interface code is stored in the database together with the data itself. This way no application servers are needed, and you can take the whole system with you just by replicating the database. Think of the possibility of doing some analysis on your company while flying to a meeting!

The visuals are in our case provided by JavaScript InfoVis Toolkit, a nice, MIT-licensed interactive graph library.

CouchDB views handle the number crunching, then CouchDB list functions process the numbers into the format needed for visualization. This leaves only a minimal amount of work for the client side.

For consistency our application has been built with CoffeeApp, so all the database and user interface code is in CoffeeScript.

In a nutshell

Any business analytics system dealing with moderate amounts of data can be built following this approach.

Apache CouchDB is the central data store
All data is stored as JSON-LD entities
NoFlo handles all data imports
Analytics based on the data are done with CouchDB map/reduce
Visualization happens with a CouchApp using JavaScript InfoVis Toolkit

This way you have a business analytics environment that is easy to extend with more data when it becomes available. New analysis can be done by writing reasonably simple map/reduce functions, and CouchDB’s replication capabilities allow you to take the system and data with you.

Using JSON-LD for the data storage makes a lot of sense, as this way the relations between different pieces of information are easy to handle. And using URIs for data identifiers means you can easily mash up information coming from different sources together.

The two-layered approach of using NoFlo for data imports, and CouchDB for analysis also allows for clean separation of concerns. In our case, I did the workflow part of things, and Szaby built the visualizations.

Midgard in 2010

henri.bergius@iki.fi (Henri Bergius) — Tue, 11 Jan 2011 00:00:00 +0000

Wow, 2010 was quite a hectic year in the Midgard world. Here is a quick summary:

We held three Midgard Gatherings: one in Lodz, Poland in April, one in Tampere, Finland in July and one in Gothenburg, Sweden in November. In April we announced the new directions of the project.
The project completed a migration to Git (and GitHub) for Midgard2 and Midgard MVC, together with a change in the development process to facilitate a more decentralized way of working
The Long-Term Supported Midgard1 series 8.09 Ragnaroek had two releases: .8 and .9. After .9 there have been substantial performance improvements that still wait for release
Midgard2 had its first Long-Term Supported version 10.05 Ratatoskr, with three releases: .0, .1 and .2
Midgard2 started providing language bindings via GObject Introspection
Midgard2 landed into Debian unstable, Maemo Extras and MeeGo Community OBS
Third generation of Midgard Content Repository had its first release showcasing more flexible model definition and built-in RDF storage. The new MidgardCR is written in Vala
First version of the Midgard Create content management interface was shown
Midgard MVC gained a new application installer tool and the ability to be run on the PHP Application Server
The OpenPSA management suite was forked out from Ragnaroek tree, ported to Midgard2 and packaged for Ubuntu
On Maemo, Midgard-powered applications like The Tablet of Adventure and MaeCalories provided tens of thousands of Midgard installations on mobile phones

Unfortunately at the same time the Midgard developer community has stayed quite small and insular. This will hopefully improve through easier installation, availability of Midgard libraries in Linux distributions and closer collaboration with the rest of the PHP world as a participant of the Zeta Components ecosystem.

We still also need to solve the project governance question of either running our own association or joining a major organization like ASF. The relation between Midgard and the GNOME project on which we heavily rely on should also be clarified.

Help testing the OpenPSA business suite

henri.bergius@iki.fi (Henri Bergius) — Mon, 08 Nov 2010 00:00:00 +0000

OpenPSA is an web-based management suite for consulting companies. The GPLd suite includes functionalities like calendaring and contact management, product configurations, sales processes, project management and invoicing. The application was originally developed by Nemein but since 2008 has seen excellent maintenance work by the Content Control team.

As a result of last weekend's Midgard developer meeting in FSCONS, the OpenPSA suite has now been packaged for Debian and Ubuntu. Ubuntu users can easily test the package with:

$ sudo apt-add-repository ppa:midgard/ratatoskr
$ sudo apt-get update
$ sudo apt-get install openpsa2

After this, the OpenPSA suite should be running on your machine in http://localhost/openpsa2. Default user account is admin / password.

You can also check out the project on GitHub. Please add your issues and ideas there!

CouchDb and Midgard talking with each other

henri.bergius@iki.fi (Henri Bergius) — Mon, 06 Jul 2009 00:00:00 +0000

CouchDb is a really cool document-oriented map/reduce database that is nowadays an Apache project. Previously we created the distributed CRM application Ajatus on top of the system and ported CouchDb to Maemo.

Here in Gran Canaria Desktop Summit CouchDb has been somewhat a hot topic, as the Ubuntu project is planning to use it as the content repository for desktop applications.

We had a lunch with Jan Lehnardt today and discussed how to make Midgard2 and CouchDb interoperate better, and as it happens, it is actually very easy: CouchDb has a replication protocol that we can support also in Midgard, making the two repositories able to synchronize content with each other.

There is now a first test implementation of Midgard-to-CouchDb synchronization support, with better Midgard integration and CouchDb-to-Midgard coming soon. Check out the Midgard MVC component on Github. Anyway, already pretty cool!

Setting up replication on CouchDb admin UI:

Midgard record replicated successfully into CouchDb:

I'll talk more about this and repository-oriented application development in my Midgard2: Content repository for desktop and the web talk tomorrow at 16:45. Be there!

Technorati Tags: couchdb, midgard, replication

Come to celebrate the 10th Anniversary of the Midgard Project

henri.bergius@iki.fi (Henri Bergius) — Fri, 27 Mar 2009 00:00:00 +0000

May 8th 1999:

The Midgard Project has finally released the first public version of Midgard Application Server Suite. The new release contains Midgard core libraries, a PHP3-based web application server for the Apache platform and the needed web-based administration tools.

May 8th 2009:

It's time for celebration - Midgard CMS turns ten in May! The decade of Midgard will be celebrated with a guest gala that is to be held at restaurant Ostrobotnia in Helsinki, Finland.

Looking forward to seeing as many of you in the party as possible! Please register ASAP as we need to tell the restaurant how many tables are needed.

Free software at work: OpenPsa2 is making a return

henri.bergius@iki.fi (Henri Bergius) — Wed, 15 Oct 2008 00:00:00 +0000

In the recent OpenMind conference we were discussing how open source projects simply don't die as long as there is an interested user community: Even if the original company or individual who built the application stops working on it, somebody will always rise and start maintaining it.

Last few weeks have been a great example of this. Nemein is the company which originally built the OpenPsa2 suite of project management and CRM tools for consulting companies. In last two years there have been some shifts in strategy, and so we ceased working on the product.

Luckily for our users, OpenPsa has been out under the GNU General Public License since 2004. This meant that while commercial development had stopped, they were able to step in and continue where we left off. It took a while, but during this fall Andreas Flack from Content Control started maintaining the software.

Now the effort is starting to bear fruit. Midgard world has recently settled into a new synchronized release model, and OpenPsa is swiftly catching up with the rest of the architecture. We expect that it may be released with the Ragnaroek 8.09.2 revision.

Good work, Andreas!

ETech 2009 CFP: Building the peer-to-peer business network

henri.bergius@iki.fi (Henri Bergius) — Wed, 06 Aug 2008 00:00:00 +0000

ETech 2009, the O'Reilly conference on emerging technologies now has a call for papers. Here is my proposal for the "Nomadism & Shedworking" track:

The recent direction of business applications has been centralization to web-based systems, easing deployment, upgrades and management of application security. However, at same time centralization provides new risks like the introduction of a single point of failure for application usage and in most cases inability to work offline.

At the same time, working culture has started so shift more towards networked individuals, or "web workers" forming ad-hoc coalitions to work on various projects. For them, centralized applications might not be desirable due to data ownership and infrastructure requirements.

Moving the applications from centralized web servers to a peer-to-peer network allows web workers to be in control of their own data, stay productive even in unstable connectivity situations, and collaborate easily with their colleagues in an ad-hoc way.

This talk outlines some ways to move forward in building peer-to-peer networked business applications. There are many open source frameworks targeting the problem, including replicated databases like CouchDb and Prophet, and application-oriented P2P networks like Swallow/DBE. These will be discussed together with some real-world examples of business applications built with them:

Ajatus - a "Personal CRM" built on top of CouchDb, a RESTful replicated object database
OpenPsa - project management system doing resourcing over the DBE P2P network
Simple Defects - a P2P bug tracking system built on top of the Prophet replicated database

The first phase of P2P business applications will likely be services operating in closed networks of users' social networking or instant messaging contacts. But the P2P model can also provide opportunities for wider networking, making it possible to find new project partners or collaborators anywhere in the world. This wider-ranging business network will introduce new challenges like security and reputation management. Some ideas related to this will also be discussed.

Other interesting projects in this sphere I did not mention are Telepathy Tubes and F2F. We have also had some ideas for how Midgard could do this...

Scrum in management of a small software consultancy

henri.bergius@iki.fi (Henri Bergius) — Wed, 11 Jun 2008 00:00:00 +0000

Over the years we at Nemein have been experimenting with various ways of keeping our operations managed. Now with some personnel changes including Joe's departure it was a good time to change the way we work again.

I had some goals:

Keeping status of different projects up-to-date with more accuracy
Ensuring our sales and project management knows if some project is being blocked by missing materials
Enabling a more distributed, web working culture

We're a small company of less than 10 people, and as such most project management methods have not been very successful for us. In general they have been made for situations where same person or team keeps on working on a project for several months, whereas in our situation a person typically works on several projects every day.

I discussed this over some beers with Tero Heikkinen from Rohea, and he told me how they were implementing Scrum in their small company. While their number of different projects running at the same time is a bit smaller, their situation otherwise is quite similar: same technologies used, Ajatus for work tracking, etc.

We had a company sauna evening and I presented Tero's ideas there: we would partially implement the Scrum model, and keep tuning it to our needs. At the first phase this means:

Every morning we have an all-hands 15-20min meeting ("the daily Scrum") where everybody goes over what they have been doing the previous day, and what they were planning to do today. If they are being blocked by something missing: a software bug, missing information or other materials, this is also brought forward
Every project has a file in Google Docs where we keep the project status and task list (Backlog). This task list is updated based on what comes up in the morning meeting
Work hours are reported with Ajatus. Rohea also uses it for project burn-down charts, and once their add-on for that is finished we may do the same
Emilia, the project manager (or Scrum Master) is responsible for resolving possible impediments and maintaining the per-project status files

The approach we have taken should be quite pragmatic and low-tech. Instead of fancy project management software we use simple word processing for status data. And thanks to Google Docs the documents produced are accessible and editable from anywhere.

Similarly the actual meetings are quite easy to manage. The people who are at the office attend there, and others attend either via a Skype or mobile phone conference call, depending on network availability. We decided to have them at 10am so that everybody will be able to participate. Even if there is a Sprint or meeting scheduled for the same time, the short time needed for our all-hands meeting means it can be held over a "cigarette break".

Ajatus is the only more experimental piece of software in our puzzle. In our company, we use it for hour reporting, expense tracking and keeping meeting minutes. For these it works quite well, although more reporting tools are definitely needed. The alpha status of CouchDb, the database software powering Ajatus has bit us a few times by database corruption (caused by OSX-specific erlang bug) or simply difficult installation procedure, but these problems will hopefully improve over time.

We're now in the second week of this model, and at least the gut feel is that this has improved coordination inside the company. The next challenge then is to let the customer get involved in the process. This can mean just sharing the project status files, or even giving them access to actual meetings or the Ajatus data.