Henri Bergius - Geo

Two hackathons in a week: thoughts on NoFlo and MsgFlo

henri.bergius@iki.fi (Henri Bergius) — Mon, 19 Jun 2017 00:00:00 +0000

Last week I participated in two hackathons, events where a group of strangers would form a team for two or three days and build a product prototype. In the end all teams pitch their prototypes, and the best ones would be given some prizes.

Hackathons are typically organized to get feedback from developers on some new API or platform. Sometimes they’re also organized as a recruitment opportunity.

Apart from the free beer and camaraderie, I like going to hackathons since they’re a great way to battle test the developer tools I build. The time from idea to having to have a running prototype is short, people are used to different ways of working and different toolkits.

If our tools and flow-based programming work as intended, they should be ideal for these kind of situations.

Minds + Machines hackathon and Electrocute

Minds + Machines hackathon was held on a boat and focused on decarbonizing power and manufacturing industries. The main platform to work with was Predix, GE’s PaaS service.

Our project was Electrocute, a machine learning system for forecasting power consumption in a changing climate.

1.5°C is the global warming target set by the Paris Agreement. How will this affect energy consumption? What kind of generator assets should utilities deploy to meet these targets? When and how much renevable energy can be utilized?

The changing climate poses many questions to utilities. With Electrocute’s forecasting suite power companies can have accurate answers, on-demand.

The system was built with a NoFlo web API server talking over MsgFlo with a Python machine learning backend. We also built a frontend where users could see the energy usage forecasts on a heatmap.

Unfortunately we didn’t win this one.

Recoding Aviation and Skillport

Recoding Aviation was held at hub:raum and focused on improving the air travel experience through usage of open APIs offered by the various participating airports.

Skillport was our project to make long layovers more bearable by connecting people who’re stuck at the airport at the same time.

Long layovers suck. But there is ONE thing amazing about them: You are surrounded by highly skilled people with interesting stories from all over the world. It sometimes happens that you meet someone randomly - we all have a story like that. But usually we are too shy and lazy to communicate and see how we could create a valuable interaction. You never know if the other person feels the same.

We built a mobile app that turns airports into a networking, cultural exchange and knowledge sharing hub. Users tell each other through the app that they are available to meet and what value they can bring to an interaction.

The app connected with a J2EE API service that then communicated over MsgFlo with NoFlo microservices doing all the interactions with social and airport APIs. We also did some data enrichment in NoFlo to make smart recommendations on meeting venues.

This time our project went well with the judges and we were selected as the winner of the Life in between airports challenge. I’m looking forward to the helicopter ride over Berlin!

Skillport also won a space at hub:raum, so this might not be the last you’ll hear of the project…

Lessons learned

Benefits of a message queue architecture

I’ve written before on why to use message queues for microservices, but that post focused more on the benefits for real-life production usage.

The problems and tasks for a system architecture in a hackathon are different. Since the time is short, you want to enable people to work in parallel as much as possible without stepping on each other’s toes. Since people in the team come from different backgrounds, you want to enable a heterogeneous, polyglot architecture where each developer can use the tools they’re most productive with.

MsgFlo is by its nature very suitable for this. Components can be written in any language that supports the message queue used, and we have convenience libraries for many of them. The discovery mechanism makes new microservices appear on the Flowhub graph as soon as they start, enabling services to be wired together quickly.

Mock early, mock often

Mocks are a useful way to provide a microservice to the other team members even before the real implementation is ready.

For example in the GE Predix hackathon, we knew the machine learning team would need quite a bit of time to build their model. Until that point we ran their microservice with a simple msgflo-python component that just gave random() as the forecast.

This way everybody else was able to work with the real interface from the get-go. When the learning model was ready we just replaced that Python service, and everything was live.

Mocks can be useful also in situations where you have a misbehaving third-party API.

Don’t forget tests

While shooting for a full test coverage is probably not realistic within the time constraints of a hackathon, it still makes sense to have at least some “happy path” tests. When you’re working with multiple developers each building a different parts of the service, interface tests serve a dual purpose:

They show the other team members how to use your service
They verify that your service actually does what it is supposed to

And if you’re using a continuous integration tool like Travis, the tests will help you catch any breakages quickly, and also ensure the services work on a clean installation.

For a message queue architecture, fbp-spec is a great tool for writing and running these interface tests.

Talk with the API providers

The reason API and platform providers organize these events is to get feedback. As a developer that works with tons of different APIs, this is a great opportunity to make sure your ideas for improvement are heard.

On the flip side, this usually also means the APIs are in a pretty early stage, and you may be the first one using them in a real-world project. When the inevitable bugs arise, it is a good to have a channel of communications open with the API provider on site so you can get them resolved or worked around quickly.

Room for improvement

The downside of the NoFlo and MsgFlo stack is that there is still quite a bit of a learning curve. NoFlo documentation is now in a reasonable place, but with Flowhub and MsgFlo we have tons of work ahead on improving the onboarding experience.

Right now it is easy to work with if somebody sets it up properly first, but getting there is a bit tricky. Fixing this will be crucial for enabling others to benefit from these tools as well.

GeoClue rises again

henri.bergius@iki.fi (Henri Bergius) — Thu, 08 Aug 2013 00:00:00 +0000

Those that have been following my blog for a longer time know that I’ve been talking a lot about making the Linux desktop and mobile platforms location aware.

Thanks to the amazing advances in adoption of mobile platforms, this dream has more or less become true, especially in the more widespread Apple and Android ecosystems. All these devices know where they are, and developers are coming up with different smart applications to utilize this information.

The free software world has been at risk of getting left behind. GeoClue, the location framework designed for these environments was in a state of flux for a long time with very little happening to it. But now we have GeoClue2, a rewritten implementation of the original idea.

LWN has a good write-up:

Zeeshan Ali spoke about GNOME’s geo-awareness, which is undergoing a rewrite. Geo-awareness consists of four major pieces, he said. The first is geolocation, or the “where am I?” question. The second is the opposite; the user wants to find a different location: a particular address, a nearby restaurant or gas station, or other points of interest. The third issue is routing, finding the best way to get between locations. Finally, there is the user interface topic: locations, points of interest, and routes all need to be presented to the user on a map.

GeoClue2 can determine location from four different sources: coordinates from GPS devices (the most accurate), the location of nearby WiFi access points (which is accurate to just a few hundred meters), the location of 3G cellular towers (which are accurate only to a few kilometers), and IP addresses (which are accurate only down to the city level).

A major shortcoming that the new service addresses is privacy:

GeoClue2 also offers better privacy controls; the previous version of the library would provide the current location to any application; with GeoClue2, GNOME will require the user to confirm location requests from each application.

Kudos to Zeeshan and the others involved for keeping the location-aware dream alive!

I haven’t personally been involved much in the free desktop world lately. This is mainly because I’ve been busy trying to change the worlds of web publishing and software development, but also because I don’t really have a desktop at the moment. Instead, I do my work with an Android tablet and a web browser with an attached keyboard. But despite that, I hope this will be picked up not only by GNOME, but by the other Linux ecosystems as well.

Building a smarter workplace

henri.bergius@iki.fi (Henri Bergius) — Fri, 08 Mar 2013 00:00:00 +0000

As part of the SmarcoS project, we have been investigating how to make workplaces smarter through sensors and context awareness. Here is a video showing what we’ve built:

The idea here is to facilitate collaboration and smoother project communications through various different tools that I’ll describe below. While this already does a lot, it is obviously only the first step on the path to making offices smarter.

Office presence

An important part of collaboration is to know who is where. Maybe some people are having a lunch break, or are working remotely? The Office Presence Display system knows these things, thanks to various sensors:

Bluetooth sensors can see smartphones and other mobile devices in the space
WiFi sensor can see what computers are connected
Google Talk sensor knows who are connected and active on their work accounts

With these, we know pretty well when you arrive to the office, and when you leave. Thanks to the sensor watching the company instant messaging system, we also know when people working remotely are available.

In addition to letting people know where the other members of the team are, this system can help with the perpetual annoyance in many companies: having to fill timesheets. We have a logger process that listens to the sensors, and logs the data into a CouchDB database. From there you can easily visualize working hours and availability trends:

In many situations it is of course not enough to know whether people are present, but also to know what they’re working on. The next component of the system helps with that.

Electronic Kanban wall

Instead of the clumsy waterfall style, more and more companies are managing their projects in tight, recurring iterations. A Kanban wall is a great way to keep track of tasks as they move through the process, and to see who is doing what, and what could be possible to do next. The One day in Kanban Land post explains the concept well.

If everybody working on a project sits in the same room, and that room is also where all decisions about the project are made, then the traditional solution of whiteboards and Post-It notes is probably the best way to visualize Kanban. But for distributed teams and more flexible work, an electronic version is a lot better option. This way everybody can see the Kanban wall in its current state from their computer, a TV in an office room, or a tablet.

Our Kanban wall implementation provides exactly that. The same wall (or, in case of multiple projects or teams, a set of walls) is available through any web browser. You can also display it on a big screen in an office or a meeting room, and control that screen with the Kinect Air Cursor.

Now, for most companies this is not meant to be the master database of projects and tasks, but instead just to be an alternative view and controlling tool to wherever your project information resides. Maybe you’re already using Pivotal Tracker, Basecamp, or OpenPSA? The Kanban wall server has a full REST API available, and so it is easy to integrate with any existing system. If a task is moved on the Kanban wall, it can be updated to the project tracking system, and vice versa.

If you’re using both systems, then the Kanban wall is also aware of the presence information. We show a differently colored border around the “person tokens” depending on the availability state. You can also set different Work-in-Progress limits for different people and states.

Interested yet?

So far we’ve only trialed the system in our own office, but are now looking for some pilot customers to try the system out. If you’re interested in making your workplace smarter, get in touch.

For those who like to tinker with these things, all the sensor software is available as open source at http://github.com/nemein.

Open Advice

henri.bergius@iki.fi (Henri Bergius) — Mon, 19 Mar 2012 00:00:00 +0000

I seem to have not blogged about this, but Open Advice, our book on Free and Open Source Software: what we wish we had known when we started, was published last month.

The book was edited by Lydia Pintscher and includes essays from 42 authors, many of whom you'll recognize if you tend to go to FOSS conferences. The LWN book review concludes:

Open Advice is a book that will be helpful to those who are new to FOSS, but, because of the individual voices, styles, and tones, it doesn't read like a "how to". It could even be recommended to those who aren't necessarily interested in contributing, but are curious about what this "free software thing" is all about. It is, in short, a great book for a variety of audiences and the (mostly) two or three page essays make it easy to read, while the anecdotes and recollections personalize it. The authors, editor, and everyone else who helped should be very pleased with the result. Readers will be too.

I probably shouldn't give the ending away, but my essay on cross-project collaboration, a subject I've also blogged about, ends with:

Good luck with breaking down the project boundaries! In most cases it works if your ideas are good and presented with an open mind. But even if you do not find a common ground, as long as your implementation solves the use case for you it has not been in vain. After all, delivering software, and delivering great user experience is what counts.

The book is licensed under CC-BY-SA, and is available as free download in ePub, mobi and PDF formats, and as paperback from Lulu. The book sources are available on GitHub, patches welcome!

Finnish MeeGo Summit starts tomorrow

henri.bergius@iki.fi (Henri Bergius) — Thu, 14 Apr 2011 00:00:00 +0000

This weekend is the first-ever Finnish MeeGo Summit, held in Tampere in the same venue where we had aKademy last summer. Despite some announcements, the conference sold out in a very short time. The program looks very interesting, too.

I'll give two talks:

Location awareness in MeeGo, Hacks & Tricks track Friday 15:30
Midgard Create - Content Management System without forms, Finhack Saturday 12:00

Finhack is a Finnish free software meetup co-organized with Free Software Foundation Europe and COSS as a one-day track within MeeGo Summit.

If you're not able to attend the Summit, there are also regular MeeGo meetup groups in Helsinki and Tampere.

Towards the geotagged web

henri.bergius@iki.fi (Henri Bergius) — Wed, 15 Sep 2010 00:00:00 +0000

There was a pretty interesting snippet in Engadget's coverage of yesterday's Nokia World keynote:

"By 2013 800 million people will be using GPS-enabled devices. Soon, everything on the Internet will have a location coordinate. This is a space we intend to own."

Indeed, this is a quite possible future, and one for which we with Midgard are quite well prepared for. Many smartphone browsers (and even desktops) nowadays can transmit their location to web services, more and more geotagged information is being published via microformats, feeds and RDF. From the humble beginnings with Plazes, social web services like Twitter, Qaiku and Facebook can now store and display location where different status updates were made.

The missing piece for a wider-scale geotagged web however is actual usage of geodata. We need easy tools for tagging content with locations, but also web services that actually serve their users better thanks to location information. This can mean helping the user to catch a bus, or recommending them clothes purchases based on local weather. Anything that goes beyond just showing dots on a map.

For that, Midgard has had the necessary infrastructure in place. What we need is your ideas and use cases for how to improve web services through location.

buscatcher: Never miss another tram

henri.bergius@iki.fi (Henri Bergius) — Thu, 02 Sep 2010 00:00:00 +0000

Opening public data is a hot topic in Finland at the moment. As a small experiment with the data that is available I wrote buscatcher, a simple N900 app that displays Helsinki trams (and some buses) moving on a map in real time. This makes it easy to determine when your next tram is coming to the stop, or where it is stuck.

Updated 2010-09-28: Buscatcher now has a stable release that is available from Maemo Downloads with already more than 10,000 installations.

For other platforms, you can grab and run the application from the GitHub repo. It should run on regular Linux desktops, and there have been reports of working on also platforms like the OpenMoko Freerunner.

Zeitgeist does location: what did I do while in Brussels?

henri.bergius@iki.fi (Henri Bergius) — Wed, 21 Jul 2010 00:00:00 +0000

Zeitgeist, the desktop activity logging engine is now becoming geo-aware. From Seif Lotfy's blog:

It allows you to ask Zeitgeist stuff like

“Get me the recent files I edited at university”

“Who do I contact most when I am at School?”

“Which pictures did I take in Brazil?”

“Where was I when an Email came in?”

“What files did I open during the conference?”

As I've been advocating since 2006, location is important for making applications smarter. While you might not remember where you stored some file, you probably remember where you were when working on it. Then Zeitgeist's location features, powered by GeoClue, will be able to get it for you.

This is especially cool since Zeitgeist is coming for Maemo as well. My laptop is quite mobile, but the N900 is even more so.

My GeoClue talk from aKademy 2010

henri.bergius@iki.fi (Henri Bergius) — Fri, 09 Jul 2010 00:00:00 +0000

aKademy 2010 was hosted in the sunny city of Tampere by the Finnish Centre for Open Source Solutions, an organization that I'm a steering group member of. In addition to helping a bit with the arrangements and organizing the Midgard Gathering there, I also gave a talk about GeoClue, the positioning framework for Linux desktops.

We initially started the push for location-aware desktops around 2006, and now the efforts are finally starting to bear fruit. Both Zeitgeist and Nepomuk are looking at indexing documents based on where you accessed them, Telepathy can share your location with your friends, and hopefully soon also your desktop clock will switch timezones when you travel.

It is very cool that this development seems to be happening on both GNOME and KDE at a reasonably similar pace. GeoClue is also a service in MeeGo and I've been told another major mobile phone manufacturer uses it. Maybe soon Mac OS X will not be the only platform with location APIs built in?

Photo by Alexey Zakhlestin.

Notes from my first Ubuntu Developer Summit

henri.bergius@iki.fi (Henri Bergius) — Sat, 15 May 2010 00:00:00 +0000

Last week was the Ubuntu Developer Summit targeted at planning how the next iteration of the operating system, Maverick Meerkat, targeted at a October 10th 2010 release, would look like. The event was held in a spa resort off in the countryside near Brussels. A place where the developers were comfortably separated from the busy towns by forests and country roads.

I went to the UDS with two goals in my mind: to discuss making the Ubuntu desktop location-aware, and to figure out what sort of role Midgard, our content repository could play in the Ubuntu space. Normally such things would be better approached by talking to upstream projects and distributions, but as Ubuntu is taking an increasingly active role in developing itself to be a comprehensive and unique operating system, it would be a better place to accomplish such wide-ranging changes.

Location-awareness was a area that many were interested in. Ubuntu already had considered GeoClue, our location service, to be a component of the distribution in Lucid Lynx, but that plan had been abandoned due to the low-maintained status of the software. However, now that GeoClue has become the geolocation framework for MeeGo things are about to change, and so there is a significant possibility to get GeoClue into Ubuntu as well.

GeoClue by itself will obviously not do much, and so making applications use it is the important thing. Some applications already support location information and could be changed to include those features by default. This includes Empathy sharing location information with instant messaging buddies, and Getting Things GNOME TODO manager being able to tag action items with location. Zeitgeist is intending to start tagging and searching activity history with location, and Gwibber also wants to include location sharing to interface with social web services like Brightkite, Qaiku and Google Latitude. Another reasonably low-hanging fruit would be to make the Ubuntu panel clock automatically change time zones as you travel.

Midgard in Ubuntu is a bit trickier question. A lot of Ubuntu has now been developed to utilize the CouchDB document database, which provides quite similar storage services as Midgard does. CouchDB's cool map/reduce algorithms and replication capabilities have enabled Ubuntu developers to quickly integrate a content repository into many applications and to provide a cloud storage and synchronization service to users of the operating system.

But at the same time CouchDB introduces some problems:

CouchDB can't seem to handle the load of Gwibber's messages, leading to excessive CPU consumption and poor performance in certain cases. For example, the overhead of computing the views causes lag when the user switches streams after Gwibber refreshes. The cost of pulling the account configuration data out of the database can also sometimes cause a noticeable lag that lasts up to four or five seconds when opening Gwibber's account manager.

CouchDB has been designed with web servers in mind, and in many situations having a constantly-running Erlang process and having to do HTTP roundtrips to access data are undesirable on more constrained computers. We ported CouchDB to Maemo a few years ago and the performance was terrible. I'd imagine the same issues on netbooks and other small hardware that Ubuntu's Light initiative targets.

On such environments Midgard might be a better option. Midgard's recent LTS release uses regular relational databases for fast performance and low footprint, utilizes important pieces of free desktop architecture like D-Bus signalling and GObject Introspection, and has been proven to work well even on low-end devices like the Nokia N900. And yet it provides full content repository APIs comparable to CouchDB, and can even handle replication with Couch. One way to make Midgard possible, then, would be to provide Python DesktopCouch APIs to it.

Another interesting technology discussed much in UDS was Puppet, the systems administration tool. It enables developers to describe their setups in a object-oriented declarative language, and then just let Puppet to make it so. I attended two Puppet sessions, and picked up a Kindle edition of the Puppet book which I read on the train ride to the Netherlands. As my company manages lots of Midgard web servers for our customers, automating system setup and maintenance on top of Puppet and Ubuntu Server might save us lots of time and effort.

All in all, quite an enlightening two days in the event. If things work out, I hope I'll be able to spend more time in the UDS targeting the N-series of Ubuntu next fall.