Henri Bergius - Bestof

Building an IoT dashboard with NASA Open MCT

henri.bergius@iki.fi (Henri Bergius) — Thu, 05 Oct 2017 00:00:00 +0000

One important aspect of any Internet of Things setup is being able to collect and visualize data for analysis. Seeing trends in sensor readings over time can be useful for identifying problems, and for coming up with new ways to use the data.

We wanted an easy solution for this for the c-base IoT setup. Since the c-base backstory is that of a crashed space station, using space technology for this made sense.

NASA Open MCT is a framework for building web-based mission control tools and dashboards that they’ve released as open source. It is intended for bringing together tools and both historical and real-time data, as can be seen in their Mars Science Laboratory dashboard demo.

c-beam telemetry server

As a dashboard framework, Open MCT doesn’t really come with batteries included. You get a bunch of widgets and library functionality, but out of the box there is no integration with data sources.

However, they do provide a tutorial project for integrating data sources. We started with that, and built the cbeam-telemetry-server project which gives a very easy way to integrate Open MCT with an existing IoT setup.

With the c-beam telemetry server we combine Open MCT with the InfluxDB timeseries database and the MQTT messaging bus. This gives a “turnkey” setup for persisting and visualizing IoT information.

Getting started

The first step is to install the c-beam telemetry server. If you want to do a manual setup, first install a MQTT broker, InfluxDB and Node.js. Optionally you can also install CouchDB for sharing custom dashboard layouts between users.

Then just clone the c-beam telemetry server repo:

$ git clone https://github.com/c-base/cbeam-telemetry-server.git

Install the dependencies and build Open MCT with:

$ npm install

Now you should be able to start the service with:

$ npm start

Running with Docker

There is also an easier way to get going: we provide pre-built Docker images of the c-beam telemetry server for both x86 and ARM.

There are also docker-compose configuration files for both environments. To install and start the whole service with all its dependencies, grab the docker-compose.yml file (or the Raspberry Pi 3 version) and start with:

$ docker-compose up -d

We’re building these images as part of our continuous integration pipeline (ARM build with this recipe), so they should always be reasonably up-to-date.

Configuring your data

The next step is to create a JavaScript configuration file for your Open MCT. This is where you need to provide a “dictionary” listing all data you want your dashboard to track.

Data sets are configured like the following (configuring a temperature reading tracked for the 2nd floor):

var floor2 = new app.Dictionary('2nd floor', 'floor2');
floor2.addMeasurement('temperature', 'floor2_temperature', [
  {
    units: 'degrees',
    format: 'float'
  }
], {
  topic: 'bitraf/temperature/1'
});

You can have multiple dictionaries in the same Open MCT installation, allowing you to group related data sets. Each measurement needs to have a name and a unit.

Getting data in

In the example above we also supply a MQTT topic to read the measurement from. Now sending data to the dashboard is as easy as writing numbers to that MQTT topic. On command-line that would be done with:

$ mosquitto_pub -t bitraf/temperature/1 -m 27.3

If you were running the telemetry server when you sent that message, you should’ve seen it appear in the appropriate dashboard.

There are MQTT libraries available for most programming languages, making it easy to connect existing systems with this dashboard.

The telemetry server is also compatible with our MsgFlo framework, meaning that you can also configure the connections between your data sources and Open MCT visually in Flowhub.

This makes it possible to utilize the existing MsgFlo libraries for implementing data sources. For example, with msgflo-arduino you can transmit sensor data from Tiva-C or NodeMcu microcontrollers to the dashboard.

Status and how you can help

The c-beam telemetry server is currently in production use in a couple of hackerspaces, and seems to run quite happily.

We’d love to get feedback from other deployments!

If you’d like to help with the project, here are couple of areas that would be great:

Adding tests to the project
Implementing downsampling of historical data
Figuring out ways to control IoT devices via the dashboard (so, to write to MQTT instead of just reading)

Please file issues or make pull requests to the repository.

NoFlo: six years of JavaScript dataflow

henri.bergius@iki.fi (Henri Bergius) — Mon, 05 Jun 2017 00:00:00 +0000

Quite a bit of time has passed since my two years of NoFlo post, and it is time to take another look at the state of the NoFlo ecosystem. To start with the basics, NoFlo is a JavaScript implementation of Flow-Based Programming:

In computer programming, flow-based programming (FBP) is a programming paradigm that defines applications as networks of “black box” processes, which exchange data across predefined connections by message passing, where the connections are specified externally to the processes. These black box processes can be reconnected endlessly to form different applications without having to be changed internally. FBP is thus naturally component-oriented.

With NoFlo software is built by creating graphs that contain reusable components and define the program logic by determining how these components talk to each other.

I started the NoFlo open source project six years ago in Mountain View, California. My aim was to improve the JavaScript programming experience by bringing the FBP paradigm to the ecosystem. At the time the focus was largely on web API servers and extract, transform, load (ETL) programs, but the scope has since expanded quite a bit:

NoFlo is not a web framework or a UI toolkit. It is a way to coordinate and reorganize data flow in any JavaScript application. As such, it can be used for whatever purpose JavaScript can be used for. We know of NoFlo being used for anything from building web servers and build tools, to coordinating events inside GUI applications, driving robots, or building Internet-connected art installations.

Flowhub

Four years ago I wrote how UI was the missing part of NoFlo. Later the same year we launched a Kickstarter campaign to fix this.

Our promise was to design a new way to build software & manage complexity - a visual development environment for all.

This was wildly successful, being at the time the 5th highest funded software crowdfunding campaign. The result — Flowhub — was released to the public in 2014. Big thanks to all of our backers!

Here is how Fast Company wrote about NoFlo:

If NoFlo succeeds, it could herald a new paradigm of web programming. Imagine a world where anyone can understand and build web applications, and developers can focus on programming efficient components to be put to work by this new class of application architects. In a way, this is the same promise as the “learn to code” movement, which wants to teach everyone to be a programmer. Just without the programming.

With Flowhub you can manage full NoFlo projects in your browser. This includes writing components in JavaScript or CoffeeScript, editing graphs and subgraphs, running and introspecting the software, and creating unit tests. You can keep your project in sync with the GitHub integration.

Celebrating six years of NoFlo

Earlier this year we incorporated Flowhub in Germany. Now, to celebrate six years of NoFlo we’re offering a perpetual 30% discount on Flowhub plans. To lock in the discount, subscribe to a Flowhub plan before June 12th 2017 using the code noflo6.

Ecosystem

While NoFlo itself has by no means taken over the world yet, the overall ecosystem it is part of is looking very healthy. Sure, JavaScript fatigue is real, but at the same time it has gone through a pretty dramatic expansion.

JavaScript

As I wrote around the time I started NoFlo, JavaScript has indeed become a universal runtime. It is used on web browsers, server-side, as well as for building mobile and desktop applications. And with NoFlo you can target all those platforms with a single programming model and toolchain.

The de-facto standard for sharing JavaScript libraries — NPM has become the most popular software repository for open source modules. Apart from the hundreds of thousands of other packages, you can also get prebuild NoFlo components from NPM to cover almost any use case.

Dataflow

After a long period of semi-obscurity, our Kickstarter campaign greatly increased the awareness in FBP and dataflow programming. Several open source projects expanded the reach of FBP to other platforms, like MicroFlo to microcontroller programming, or PhpFlo to data conversion pipelines in PHP. To support more of these with common tooling, we standardized the FBP protocol that allows IDEs like Flowhub manage flow-based programs across different runtimes.

Dataflow also saw uptake in the bigger industry. Facebook’s Flux architecture brought flow-based programming to reactive web applications. Google’s TensorFlow made dataflow the way to build machine learning applications. And Google’s Cloud Dataflow uses these techniques for stream processing.

Tooling for flow-based programming

One big area of focus for us has been improving the tooling around NoFlo, as well as the other FBP systems. The FBP protocol has been a big enabler for both building better tools, and for collaboration between different FBP and dataflow systems.

Here are some of the tools currently available for NoFlo developers:

Flowhub — browser-based visual programming IDE for NoFlo and other flow-based systems
noflo-nodejs — command-line interface for running NoFlo programs on Node.js
noflo-browser-app — template for building browser applications in NoFLo
MsgFlo — for running NoFlo and other FBP runtimes as a distributed system
fbp-spec — data-driven tests for NoFlo and other FBP environments
flowtrace — tool for retroactive debugging of NoFlo programs. Supports visual replay with Flowhub

NoFlo 0.8

NoFlo 0.8, released in March this year is probably our most important release so far. It introduced a new component API and greatly clarified the component and network lifecycle.

With this release, it is easier than ever to build well-behaved NoFlo components and to deal with the mixture of asynchronous and synchronous data processing. It also brings NoFlo a lot closer to the classical FBP concepts.

As part of the release process, we also fully overhauled the NoFlo documentation and wrote a new data transformations tutorial project.

To find out more about NoFlo 0.8, watch my recent NoFlo talk from Berlin Node.js meetup:

Road to 1.0

In addition to providing lots of new APIs and functionality, NoFlo 0.8 also acts as the transitional release as we head towards the big 1.0 release. In this version we marked many old APIs as deprecated.

NoFlo 1.0 will essentially be 0.8, but with all the deprecated APIs removed. If you haven’t done so already, now is a good time to upgrade your NoFlo projects to 0.8 and make sure everything runs without deprecation warnings.

We intend to release NoFlo 1.0 later this summer once more of our open source component libraries have been updated to utilize the new features.

Working on an Android tablet, 2017 edition

henri.bergius@iki.fi (Henri Bergius) — Fri, 10 Feb 2017 00:00:00 +0000

Back in 2013 I was working exclusively on an Android tablet. Then with the NoFlo Kickstarter I needed a device with a desktop browser. What followed were brief periods working on a Chromebook, on a 12” MacBook, and even an iPad Pro.

But from April 2016 onwards I’ve been again working with an Android device. Some people have asked me about my setup, and so here is an update.

Why work on a tablet?

When I started on this path in 2013, using a tablet for “real work” was considered crazy. While every story on tablet productivity still brings out the people claiming it is not a real computer for real work, using tablets for real work is becoming more and more common.

A big contributor to this has been the plethora of work-oriented tablets and convertibles released since then. Microsoft’s popular Surface Pro line brought the PC to tablet form factor, and Apple’s iPad Pro devices gave the iPad a keyboard.

Here are couple of great posts talking about how it feels to work on an iPad:

With all the activity going on, one could claim using a tablet for work has been normalized. But why work on a tablet instead of a “real computer”? Here are some reasons, at least for me:

Free of legacy cruft

Desktop operating systems have become clunky. Window management. File management. Multiple ways to discover, install, and uninstall applications. Broken notification mechanisms.

With a tablet you can bypass pretty much all of that, and jump into a simpler, cleaner interface designed for the modern connected world.

I think this is also the reason driving some developers back to Linux and tiling window managers — cutting manual tweaking and staying focused.

Amazing endurance

Admittedly, laptop battery life has increased a lot since 2013. But with some manufacturers using this an excuse to ship thinner devices, tablets still win the endurance game.

With my current work tablet, I’m customarily getting 12 or more hours of usage. This means I can power through the typical long days of a startup founder without having to plug in. And when traveling, I really don’t have to care where power sockets are located on trains, airplanes, and conference centers.

Low power usage also means that I can really get a lot of more runtime by utilizing the mobile battery pack I originally bought to use with my phone. While I’ve never actually had to try this, back-of-the-envelope math claims I should be able to get a full workweek from the combo without plugging in.

Work and play

The other aspect of using a tablet is that it becomes a very nice content consumption device after I’m done working. Simply disconnect the keyboard and lean back, and the same device you used for writing software becomes a great e-reader, video player, or a gaming machine.

This combined with the battery life has meant that I’ve actually stopped carrying a Kindle with me. While an e-ink screen is still nicer to read, not needing an extra device has its benefits, especially for a frequent one-bag traveller.

The setup

I’m writing this on a Pixel C, a 10.2” Android tablet made by Google. I got the device last spring when there were developer discounts available at ramp-up to the Android 7 release, and have been using it full-time since.

Software

Surprisingly little has changed in my software use since 2013 — I still spend the most of the time writing software in either Flowhub or terminal. Here are the apps I use on daily basis:

JuiceSSH for mosh access to my remote development servers
Termux for local and offline development
Flowhub for visual programming
The Grid for updating my various websites
Slack, Hangouts, and Inbox by Gmail for communications
Google Drive and the associated applications for budgeting and planning
Chrome for web

Looking back to the situation in early 2013, the biggest change is that Slack has pretty much killed work email.

Termux is a new app that has done a lot to improve the local development situation. By starting the app you get a very nice Linux chroot environment where a lot of software is only a quick apt install away.

Since much of my non-Flowhub work is done in tmux and vim, I get the exactly same working environment on both local chroot and cloud machines by simply installing my dotfiles on each of them.

Keyboard

When I’m on the road I’m using the Pixel C keyboard. This doubles as a screen protector, and provides a reasonable laptop-like typing environment. It attaches to the tablet with very strong magnets and allows a good amount of flexibility on the screen angles.

However, when stationary, no laptop keyboard compares to a real mechanical keyboard. When I’m in the office I use a Filco MiniLa Air, a bluetooth keyboard with quiet-ish Cherry MX brown switches.

This tenkeyless (60%) keyboard is extremely comfortable to type on. However, the sturdy metal case means that it is a little too big and heavy to carry on a daily basis.

In practice I’ve only taken the mechanical keyboard with me when there has been a longer trip where I know that I’ll be doing a lot of typing. To solve this, I’m actually looking to build a more compact custom mechanical keyboard so I could always have it with me. (Update: here is the keyboard I built).

Comparison with iOS

So, why work on Android instead of getting an iPad Pro? I’ve actually worked on both, and here are my reasons:

Communications between apps: while iOS has extensions now, the ability to send data from an app to another is still a hit-or-miss. Android had intents from day one, meaning pretty much any app can talk to any other app
Standard charging: all of my other devices charge with the same USB-C chargers and cables. iPads still use the proprietary Lightnight plug, requiring custom dongles for everything
Standard accessories: this boils down to USB-C just like charging. With Android I can plug in a network adapter or even a mouse, and it’ll just work
Ecosystem lock-in: we’re moving to a world where everything — from household electronics to cars — is either locked to the Apple ecosystem or following standards. I don’t want to be locked to a single vendor for everything digital
Browser choice: with iOS you only get one web renderer, the rather dated Safari. On Android I can choose between Chrome, Firefox, or any other browser that has been ported to the platform

Of course, iOS has its own benefits. Apple has a stronger stance on privacy than Google. And there is more well-made tablet software available for iPads than Android. But when almost everything I use is available on the web, this doesn’t matter that much.

The future

As a software developer working on Android tablets, the weakest point of the platform is still that there are no browser developer tools available. This was a problem in 2013, and it is still a problem now.

From my conversations with some Chrome developers, it seems Google has very little interest in addressing this. However, there is a bright spot: the new breed of convertible Chromebooks being released now. And they run Android apps:

Chrome OS is another clean, legacy free, modern computing interface. With these new devices you get the combination of a full desktop browser and the ability to run all Android tablet software.

The Samsung Chromebook Pro/Plus mentioned above is definitely interesting. A high-res 12” screen and a digital pen which I see as something very promising for visual programming purposes.

However, given that I already have a great mechanical keyboard, I’d love a device that shipped without an attached keyboard. We’ll see what kind of devices get out later this year.

Sailing across the Atlantic

henri.bergius@iki.fi (Henri Bergius) — Sun, 22 Nov 2015 00:00:00 +0000

Exactly a year ago today we flew from Berlin to Tenerife to depart on a sailing trip across the Atlantic on the Finnish sail training schooner T/S Helena. The Grid had just reached 10.000 founding members, and it was time to charge batteries before the next big push towards the beta.

I had sailed on Helena once before, on a weekend trip in the Turku archipelago. But crossing the Atlantic would be a more interesting adventure. For a month this two-masted tall ship would be our home on the ocean.

Departure

Arriving to the harbour, Helena was easy to find moored next to a Czech “pirate ship”. We registered with the captain and got assigned to the first watch, and then went out for a dinner dinner and some wine with some friends from Berlin who happened to be holidaying on the island.

The next day was busy with provisioning the ship and getting to know the rest of the crew. Every possible nook and cranny would be filled with fruits, fresh produce, water, and other things needed on the long trip.

The ship’s electricity system also needed some repairs, causing delays with the departure. But at the last moment to catch the high tide, on the evening of November 24th, Helena sounded the three long blasts on the foghorn and we were on the way!

Almost immediately after leaving port, we encountered the larger waves of the ocean. This caused some discomfort for many of our students. Our first watch was on kitchen duty, and there were several moments where there was a need to get out of the fumes and out in the fresh air to get one’s bearings.

But human beings being as adaptable as they are, by the next day any signs of seasickness would be gone from pretty much everybody, and the whole group could enjoy the adventure ahead.

T/S Helena

Built in 1992, the steel hulled schooner T/S Helena is the main training ship of the Finnish Sail Training Association. Displacing 110 tons and with length of 38 meters, it is not a small sailboat.

It has a capacity of accommodating 28 crew members and students, as has undertaken many epic adventures on its journey. Cape Horn following the trail of frigate Suomen Joutsen, north to Svalbard, and many many Atlantic crossings.

Inside the ship we’d have cabins for each of the four watches, and a separate area for the professional crew. Also, several restrooms, a shower room, and a kitchen and lounging area.

While the primary means of moving the ship is by sail, it is also outfitted with a powerful engine, as well as an auxiliary power unit for producing electricity and making drinking water.

Life on board

While underway, the life on board of Helena is dominated by the traditional naval watch system, where the crew is divided into four watches, each with their own cabin.

The watch on deck duty handles all the sailing tasks of steering the ship, adding or removing sails, as well as navigation and monitoring the radio and AIS. The watch on standby makes and serves the meals, as well as keeping the ship clean.

When on free watch there is time to sleep, read, and relax on the deck. The cabins are quite a tight fit, and so on the nice warm tropical nights many preferred to sleep inside the sail stash on the deck.

Alongside the normal duties there were lessons on sailing theory, navigation, safety equipment, knots, and other matters. Various students had also been tasked with giving lectures on the islands of Cape Verde and Barbados where we were stopping, as well as on marine life.

The three warm meals of the day were obviously a highlight, with the crews ending up in friendly competition on what kind of epic meals could be made of the supplies. And this being a Finnish institution, on Thursday the lunch is always pea soup and pancakes.

As always, when food is good, everything goes smoother. The last cooking shift of my First Watch ended on a high note, when we prepared rum-marinated swordfish with garlic mashed potatoes.

Along the way provisions would of course dwindle, and once all fresh fruit and bread had been consumed, the kitchen shift would also start to include baking bread.

While the space is tight on the ship, and you’re rubbing elbows with other crew members pretty much everywhere onboard, there are still some places that are more secluded and fitting for relaxation. Especially the bowsprit has proven to be a place where one can focus on meditation and the open ocean.

One thing that I didn’t consider before the trip is how sticky everything would be through the constant presence of sea salt and sweat. The cabins were hot, and since there were issues with the water maker, we had to wash ourselves and do laundry with sea water.

In these conditions we’d often freshen up with a “prison shower” where the whole crew lined up and were washed with the firehose. But a highlight was definitely when while becalmed we were able to stop the ship and go for a swim in the middle of the Atlantic. Only five kilometers deep below…

Night sailing

One of the most memorable parts of the trip were the nights when our watch was on deck duty. Steering the big ship by stars, just like ancient Phoenicians.

Or when somebody else was manning the wheel just enjoying the constant shooting stars of the Geminid meteor shower, the play of moonlight on the sails and the sea, and even an occasional dolphin jumping by the ship, lit with phytoplankton.

The route

Starting from Tenerife, we followed the traditional Age of Sail route along the coast of Africa to the Cape Verde islands, where we stopped for some provisions and recreation in the sleepy town of Mindelo.

After Cape Verde we headed west across the ocean, setting out sights for the island of Barbados.

After some sightseeing and sail repairs, the route continued past the Sail Rock to the Tobago Cays marine park, where we stopped to enjoy a very nice lobster dinner on the beach.

Then the trip continued northwards to St. Lucia, stopping in Marigot Bay for a farewell dinner.

Our final port of call was Le Marin on the French colony of Martinique. So, in a way we did a very long sailing trip to get from Spain to France!

A traditional “crossing of the Atlantic” ceremony was held and then it was time to leave the ship and return to being a pedestrian traveler.

Christmas in the Caribbean

It didn’t feel sensible to fly home directly, and so we decided to spend the Christmas on Martinique. This ended up being a very relaxed few days with nice cheese and wine and a candlelit balcony, as the power went out from the part of the island where we were on.

Since our flights were from St. Lucia, we ended up finding an Australian catamaran that we were able to commission to take us across. After the strict routines of a sail training vessel, it was quite a culture shock to get on a recreational sailboat.

We still got our last glimpse of Helena on the trip by accident. As we were watching the sun set behind the Piton mountains from our hotel balcony in Soufriere, the ship glid majestically across the bay, getting ready to anchor somewhere outside of the marina.

But that was another crew, embarking on an adventure of their own. For us it was time to fly home and prepare for another hectic Grid meetup in Hawaii.

Flowhub public beta: a better interface for Flow-Based Programming

henri.bergius@iki.fi (Henri Bergius) — Wed, 07 May 2014 00:00:00 +0000

Today I’m happy to announce the public beta of the Flowhub interface for Flow-Based Programming. This is the latest step in the adventure that started with some UI sketching early last year, went through our successful Kickstarter — and now — thanks to our 1 205 backers, it is available to the public.

Getting Started

This post will go into more detail on how the new Flowhub interface works in a bit, but for those who want to dive straight in, here are the relevant links:

Make sure to read the Getting Started guides and check out the Flowhub FAQ. There is also a new video available:

Both the web version and the Chrome app are built following the offline first philosophy, and keep everything you need stored locally inside your browser. The Chrome app and the upcoming iOS and Android builds will enable us to later introduce capabilities that are not possible inside regular browsers, like talking directly to MicroFlo runtimes over USB or Bluetooth. But other than that they’re similar in features and user experience.

New User Interface

If you read the NoFlo Update from last October, you might notice that the new Flowhub user interface looks and feels quite different from it.

This new design was implemented to improve touch-screen friendliness, as well as to give Flowhub a more focused, unique look. It also allowed us to follow some interesting UX paths that I’ll explain next.

Zooming

One typical problem in visual programming tools is that they can become quite cluttered with information. To solve this, we utilized the concept of Zooming User Interfaces, which allow us to show a clear overview of a program when zoomed out, and reveal all kinds of detail about it when zoomed in.

Zooming works with two-finger scroll on typical desktop computers, or with the pinch gesture on touch-enabled devices.

Another interface concept that we used to make interactions faster and more contextual is Pie Menus.

For example, you can easily navigate to subgraphs and component source code with the menu:

When you have selected multiple nodes, you can use the menu to group them or move them to a new subgraph:

The menu can also be used for removing edges or nodes:

You can activate the pie menu in the graph editor with a right mouse click, or with a long press on touch-enabled devices.

Component Editor

Another new major feature is in-app component editing. If your runtime supports it, you can at any time create or modify custom components for your project and they’ll become immediately available for your graphs.

The programming languages available for component creation depend on the runtime. With NoFlo these are JavaScript and CoffeeScript. With another runtime they might be C, Java, or Python.

Offline First

While some claim that in reality you’re never offline, the reality is that there are many situations where Internet connectivity is either not available, unreliable, or simply expensive. Think of a typical conference or a hackathon for instance.

Because of this — and to give software developers the privacy they deserve — Flowhub has been designed to work “offline first”. All your graphs, projects, and custom components are stored locally in your browser’s Indexed Database and only transmitted over the network when you wish to push to a GitHub project, or interact with a remote runtime.

We’re following a very similar UI concept as Amazon Kindle in that you can download projects locally to your device, or browse the ones you have available in the cloud:

At any point you can push your changes to a graph or a component to GitHub:

Runtime discovery happens through a central service, but once you know the address of your FBP runtime, the communications between it and your browser will happen directly. This makes it easy to work with Node.js projects running on your own machine even when offline.

Cross-platform, Full-stack

When we launched the NoFlo UI Kickstarter, we were initially only thinking about how to support NoFlo in different environments. But in the course of development we ended up defining a network protocol for FBP that enabled us to move past just a single FBP environment and towards supporting all of them. This is what prompted the Flowhub rebranding.

Since then, the number of supported FBP environments has been growing. Here is a list of the ones I’m aware of:

NoFlo
- browsers
- Node.js
- GNOME desktop
MicroFlo
- microcontrollers
ImgFlo
- image processing with GEGL
ProtoFlo
- Python

I hope that the developers of other FBP environments like JavaFBP and GoFlow add support for the FBP protocol soon so that they can also utilize the Flowhub interface.

Open Source vs. Paid

As promised in our Kickstarter, the NoFlo Development Environment is an open source project available under the MIT license.

Flowhub is a branded and supported instance of that with some additional network services like the Runtime Registry.

The Flowhub plans allow us to continue development of this Flow-Based Programming toolset, as well as to provide the various network services needed for making the experience smooth.

Just like with GitHub, Flowhub provides a free environment for anybody working on public and open source projects. Private projects need a paid plan.

Kickstarter & Pre-Ordered Plans

It is likely that many readers of this post already supported our Kickstarter or pre-ordered a Flowhub plan. Since Flowhub is still in beta, we haven’t activated your plans yet. So for now, everybody is using Flowhub with the free plan.

We will be rolling out the paid plans and Kickstarter rewards towards the end of the beta testing period.

Feel free to already log in and start using Flowhub, however! The plan will be added to your account when we feel the software is ready for it.

Examples

Here are some examples of things you can build with Flowhub targeting web browsers:

For a more comprehensive cross-platform project, see my Building an Ingress Table with Flowhub post.

There is also an ongoing Google Summer of Code project to port various Meemoo apps to Flowhub. This will hopefully result in a lot more demos.

Next Steps

The main purpose of this public beta is to allow our backers and other FBP enthusiasts an early access to the Flowhub user interface. Now we will focus on stabilization and bug fixing, aided by the NoFlo UI issue tracker. We’re also gathering feedback from beta testers in form of user surveys and will utilize those to prioritize both bug fixing and feature work.

Right now the main areas of focus are:

Improving the GitHub integration
Landing the graph autolayout feature
Compatibility with upcoming Chrome 35 and its native HTML5 Custom Elements support
Implementing real-time collaboration over WebRTC

We hope to release the stable version of Flowhub in summer 2014.

Building an Ingress Table with Flowhub

henri.bergius@iki.fi (Henri Bergius) — Tue, 18 Mar 2014 00:00:00 +0000

The c-base space station — a culture carbonite and a hackerspace — is the focal point of Berlin’s thriving tech scene. It is also the place where many of the city’s Ingress agents converge after an evening of hectic raiding or farming.

In February we came with an idea on combining our dual passions of open source software and Ingress in a new way. Jon Nordby from Bitraf hackerspace in Oslo had recently shown off the new full-stack development capabilities of Flowhub made possible by integrating my NoFlo flow-based programming framework for JavaScript and his MicroFlo giving similar abilities to microcontroller programming. So why not use them to build something awesome?

Since Flowhub is nearing a public beta, this would also give us a way to showcase some of the possibilities, as well as stress-test Flow-Based Programming in a Internet-connected hardware project. Often hackerspace projects tend to stretch from months to infinity; our experiences with NoFlo and flying drones already showed that with FBP we can easily parallelize development, challenging some of the central dogmas of the Mythical Man Month. It was worth a try to see if this would allow us to compress the time needed for such a project from a couple of months to a long weekend.

Introducing the Ingress Table

Before the actual hackathon we had two meetings with the project team. There were many decisions to be made, starting from the size and shape of the table to the features it should have. Looking at the different tables in the c-base main hall we settled on a square table of slightly less than 1m², as that would fit nicely in the area, and still seat the magical number of eight Ingress agents or other c-base regulars.

The tabletop would be a map of c-base and the surrounding area, and it would show the status of the portals nearby, as well as alert people sitting at it of attacks and other Ingress events of interest. Essentially, it’d be a physical world equivalent of the Intel Map.

We considered integrating a regular screen to have maximum flexibility in the face of the changing world of Ingress, but eventually decided that most people at c-base already spend much of their waking hours looking at a screen, and so we’d do something more ambient and just use a set of physical lights.

The hardware and software also needed some thought, especially since some of the parts needed might have long shipping times. Eventually we settled on the combination of a BeagleBone Black ARM computer as the brains of the system, and a LaunchPad Tiva as the microcontroller running the hardware. The computer would run NoFlo on Linux, and we’d flash the microcontroller with MicroFlo.

By the time of arriving to c-base, many Ingress agents have their phones and battery packs depleted, and so we incorporated eight USB power ports into the table design. Simply plug in your own cable and you can charge your device while enjoying the beer and the chat.

Once the plans had been set, a flurry of preparations began. We would need lots of things, ranging from wood and glass parts for the table shell, to various different electronics and computer parts for the insides. And some of these would have to be ordered from China. Would they arrive in time?

I spent the two weeks before the hackathon doing a project in Florence, and it was quite interesting to coordinate the logistics remotely. Thankfully our Berlin team did a stellar job of tracking missing shipments and collecting the things we needed!

The hackathon

I landed in Berlin in the early evening of Friday, March 14th. After negotiating the rush hour public transport of the Tegel airport, I arrived to the space station to see most of our team already there, unpacking and getting the supplies ready for the hackathon.

At this point we essentially had only the raw materials available. Planks of wood, plates of glass and plastic. And a lot of electronics components. No assembly had yet been done, and no lines of code had been written or graphs drawn for the project.

We quickly organized the hackathon into three tracks: hardware, software, and electronics. The hardware team got themselves busy building the table shell, as that would need to be finished early so that the paint would have time to dry before we’d start assembling the electronics into it. Over the next day they’d often call the other teams over to help in holding or moving things, and also for the very important task of test-sitting the table to figure out the optimal trade-off between table height and legroom.

While the hardware guys were working, we started designing the software part of it. Some basic decisions had to be taken on how we’d get the data, and how we would filter and transform the raw portal statuses to commands to the actual lights in the table.

Eventually we settled on a NoFlo graph that would poll the portal data in, and run it through a set of transformations to find the detect the data points of interest, like portals that have changed owners or are under attack. In parallel we would run some animation loops to create a more organic, shifting feel to the whole map by having the light shining through the streets be constantly shifting and moving.

(and yes, the graph you see above is the actuall running code of the table)

Since the electronics wouldn’t be working for a while still, we decided to build also a Ingress Table Emulator in HTML and NoFlo. This would give us something to test the data and our graphs while the other teams where still working on their things. This proved to be a very useful thing, as this way we were able to watch a big Ingress battle through our simulated blinking lights already in the Saturday evening, and see our emulated table go through pretty much all the different states we were interested in.

Once the table shell had been built and the paint was drying, the hardware team started preparing the other things like the map layer, the glass top, and the USB chargers.

For electronics we noticed that we had still some parts missing from the inventory, and so I had to do a quick supply run on Saturday. But once we got those, the team got into calculations and soldering.

Every project has its setbacks, and in this case it came in the form of running pre-released software. It turned out that the LaunchPad port of MicroFlo still had some issues, and so most of Sunday was spent debugging the communications protocol and tuning the components. But the end result is a much better improved MicroFlo, and eventually we got the major moment of triumph of seeing the street lights start animating for the first time. LED strips controlled by a LaunchPad Tiva, in turn controlled by animation loops running in a NoFlo graph on Node.js.

On Monday evening we convened at c-base for the final push. Street lights were ready, but there were still some issues with getting the table connected wirelessly to the space station network. And we would still need to implement the MicroFlo component for the portal lights. The latter resulting in an epic parallel programming and debugging session between Jon in Norway and Uwe in Berlin. But by the end of the evening we were able to test the full system for the first time, and carry the table to its new home.

It was time to celebrate. For an Ingress table, this meant sitting around the table enjoying cold beers, while hacking a level 8 blue portal and watching the lights change across the board as agents ventured out.

(We’re still in the process of collecting media about the project. The table will look a lot more awesome in video, and I hope I’ll be able to add some of those to this post soon)

Moving ahead

Having the first running version of the table is of course a big milestone. Now we should monitor it for some time (over beer, of course) and make adjustments as necessary. There are some things that obviously need to be changed with the brightness of the lights based on the location of the table in the main hall. And of course we’ll only know about the full system’s robustness once it has a bit more mileage.

Since we already have a HTML emulator of the table, it might be fun to release that to the public at some point. That way agents who are not at the c-base main hall could also see what is going on with this simple interface.

An interesting area of development is also to see how the table could integrate better with the rest of the space station. There are various screens ranging from the awesome Mate Light to smaller screens and gauges everywhere. And all of that is pretty much networked and available. Maybe we could visualize some events of interest in other parts of the station. This shows of the “Internet of Things” is never finished.

So far Niantic Labs — the makers of Ingress — have limited the availability of a portal data API to few selected parties, and so for now we had to work with a third-party to get the information needed. We hope this table will be another step in convincing Niantic of the creative potential that an official, open Ingress API would unleash.

I’d like to give big thanks especially to everybody who participated in hackathon — whether on location or remotely from Oslo — as well as to those who were cheering us on. I’m also grateful to Flowhub for sponsoring the project. And of course to c-base for being an awesome place where such things can happen.

The full source code for the Ingress Table can be found from https://github.com/c-base/ingress-table

Ijon also has a blog post about the project.

Full-Stack Flow-Based Programming

henri.bergius@iki.fi (Henri Bergius) — Thu, 06 Feb 2014 00:00:00 +0000

The idea of Full-Stack Development is quite popular at the moment — building things that run both the browser and the server side of web development, usually utilizing similar languages and frameworks.

With Flow-Based Programming and the emerging Flowhub ecosystem, we can take this even further. Thanks to the FBP network protocol we can build and monitor graphs spanning multiple devices and flow-based environments.

Jon Nordby gave a Flow-Based Programming talk in FOSDEM Internet of Things track last weekend. His demo was running a FBP network comprising of three different environments that talk together. You can find the talk online.

Here are some screenshots of the different graphs.

MicroFlo running on an Arduino Microcontroller and monitoring a temperature sensor:

NoFlo running on Node.js and communicating with the Arduino over a serial port:

NoFlo running in browser and communicating with the Node.js process over WebSockets:

(click to see the full-size picture)

Taking this further

While this setup already works, as you can see the three graphs are still treated separately. The next obvious step will be to utilize the subgraph features of NoFlo UI and allow different nodes of a graph represent different runtime environments.

This way you could introspect the data passing through all the wires in a single UI window, and “zoom in” to see each individual part of the system.

The FBP ecosystem is growing all the time, with different runtimes popping up for different languages and use cases. While NoFlo’s JavaScript focus makes it part of the Universal Runtime, there are many valid scenarios where other runtimes would be useful, especially on mobile, embedded, and desktop.

Work to be done

Interoperability between them is an area we should focus on. The network protocol needs more scrutiny to ensure all scenarios are covered, and more of the FBP/dataflow systems need to integrate it.

Some steps are already being taken in this direction. After Jon’s session in FOSDEM we had a nice meetup discussing better integration between MicroFlo on microcontrollers, NoFlo on browser and server, and Lionel Landwerlin’s work on porting NoFlo to the GNOME desktop.

If you’re interested in collaborating, please get in touch!

Photo by Forrest Oliphant.

Flowhub

henri.bergius@iki.fi (Henri Bergius) — Thu, 12 Dec 2013 00:00:00 +0000

We just opened the Flowhub website for pre-orders. Flowhub is the collaborative development environment for NoFlo and other flow-based programming systems.

Once it launches for general use, Flowhub is the fruition of the project initiated by our successful NoFlo Development Environment Kickstarter from last August.

Flowhub will provide a development environment for your flow-based programs, allowing you to collaborate and build things inside the same interface whether you’re targeting the client-side or the server-side of the web world. And thanks to the efforts on creating a standard protocol for flow-based runtimes, Flowhub will be able to work with other environments as well. To get a glimpse of this potential, take a look at Jon Nordby’s work on Arduino programming with NoFlo UI and what Lionel Landwerlin is doing with NoFlo in the Linux desktop development world.

This is the main reason why we went with a more generic name for the new service. Flowhub aims to be the central hub for any flow-based development, not just the JavaScript-centric world where my NoFlo runtime is focused. Our Kickstarter sparked a lot of interest in flow-based programming in general, as evidenced by the various new FBP runtimes, and the upcoming book on the subject.

While NoFlo is and will remain our main focus, we should embrace the different ideas and different projects out there. One of the key points of FBP is everything can be modeled as a black box with input and output ports. With this level of abstraction, and a standard communications protocol, the different systems will be able to work together. Flowhub can be the central point for enabling that.

Collaboration as a service

NoFlo is open source, and so is the development environment we’re building for it. For developing things on your own, this is pretty much all you’ll need, especially if you’re willing to set up things like the NoFlo environment for Node.js yourself.

The role of Flowhub as a service is analogous to what GitHub provides for traditional software development. Anybody can serve git repositories and issue trackers on their own, but having a third party to take care of that gives a much simpler, smoother experience. And even more importantly, having a central point where multiple projects reside enables much better collaboration and discovery between teams and projects.

This is the essence of what Flowhub will provide:

Hosted version of the NoFlo UI, able to communicate with multiple different FBP runtimes whether running on client or server
Integration with GitHub’s version control and issue tracking capabilities
Peer-to-peer environment for team collaboration based on WebRTC
Hosted environment for running server-side NoFlo programs and their tests

As promised in our Kickstarter, the service will open to the public in the early summer of 2014. Our Kickstarter backers will gain an early access and have already service plans provided for them. With our Flowhub pre-order campaign, those who missed the Kickstarter opportunity have still a way to get in with early adopter pricing.

New technologies and a new UI

Like NoFlo itself, Flowhub is also an exploration of various new technologies. From the layout technologies and the use of Web Components, and the flow-based payment processing infrastructure handling both Bitcoin and credit card transactions that we built in NoFlo — there are many things being dogfooded on the service.

At this stage of the game there are sure to be some rough edges, but by the time Flowhub opens to the public there should’ve been enough time at both our end, and with web browsers to mature to the point where these things provide a smooth experience.

The user interface we showed in September, and user-tested in the NodeCopter NoFlo event has since seen quite a lot of improvement based on what we’ve learned. The new version has been rebuilt ground-up with Polymer and NoFlo, and is a lot more efficient and touch-friendly.

It is still not perfect, but having used it on various devices from small tablets through laptops to huge touch-screen PCs, I feel we’re definitely on the right track. Having a user interface where you can see the connections and data flows of your software in real time, and can rewire any part when needed is incredibly powerful. And once component editing is working fully with the runtimes, I can’t really see myself wanting to go back to text-only development.

Here is a sneak peek (click to see a bigger version):

To see more of what we’re building, visit flowhub.io, play with the demo we have there, and make sure to watch the intro video. Preorder today to help fund the hosted and open source version of Flowhub!

Bitcoin as a Medium of Exchange

henri.bergius@iki.fi (Henri Bergius) — Thu, 28 Nov 2013 00:00:00 +0000

Bitcoin — the decentralized digital currency — has been making a lot of headlines lately. Much of this is driven by the current investment boom around it that has raised the exchange rates over the 1000 USD mark. But really, looking at Bitcoin as a medium for currency or asset speculation is a bad idea. Instead, we should see it purely as a medium of exchange.

Not a great holder of value

Much of the press stories about Bitcoin see it as a speculative investment free of currency manipulation by central banks. By its nature, it is a deflationary currency, given that the global supply of Bitcoins is capped. As long as there is interest in Bitcoin, it means the prices are bound to go up. But at the same time, this is a threat to the currency. Deflation disincentivizes people from spending their money, as tomorrow the value is likely to be higher.

This is a big difference to fiat currencies where central banks generally act to ensure that there is a little bit of inflation, giving people incentives to spend, invest, and keep the economy running.

Traditional currencies are given a value by the fact that there is a government requiring taxes to be paid in them. This creates a demand in the currency, as people have to acquire units of that currency somehow, usually by selling goods and services. So, the value of a currency is built on the power of a state to tax its subjects in that currency.

The other traditional holder of value, gold, also differs from Bitcoins in the sense that at least you can make some useful things out of it.

In contrast, the value of Bitcoin without an economy backing it up is exactly zero. Without an active network of miners, and business transactions that create a demand for them, there is not much you could do with a Bitcoin.

Given this setting, Bitcoin as a investment mechanism resembles a ponzi scheme where people make speculative investments in them in hopes of being able to pass them on to the next fool when the prices go up.

This is obviously not sustainable in the long term. But luckily there are things you can do with Bitcoin that still make them interesting if you deal with them as a medium of exchange and not a long-term holder of value.

Great tool for conducting transactions

Now that we’ve established that Bitcoins are not the best thing to put your life savings into, or to stash under your mattress, what is it exactly that makes them interesting?

Mobile money

Bitcoin is a way of making transactions quickly, safely, and without middlemen. A lot of the articles written about it compare it to gold, but in my view a much closer comparison is M-Pesa, the operator-driven mobile payment network that powers over 30% of the Kenyan economy.

M-Pesa on a phone, by Pixel Ballads

When you see people paying with their mobile phones in a African street market, you know they’ve leapfrogged us in technology.

Bitcoin has the exactly same advantages as M-Pesa: you can send it to anywhere with minimal costs and delays, you can keep it on your phone, and anybody can ask for a payment through it. Unlike M-Pesa, Bitcoin is truly decentralized and doesn’t rely on a mobile operator to act as the “central bank”.

Unlike M-Pesa, Bitcoin currently requires an internet-connected smartphone with a wallet application. This limits its usefulness in the world outside of the rich countries (though cheap Android phones are changing the situation). It would be a huge boost for mobile commerce with Bitcoins if there was a service that enabled paying and requesting Bitcoin over SMS.

The one to make Bitcoin work over SMS will be rich

Last summer I was talking with Joerg Patzer, the owner of Room 77, a restaurant in Berlin that has been one of the early adopters of Bitcoin. At the time about 10% of the his sales happened in Bitcoin, and the number has been constantly growing since. He told that he even pays some of his suppliers in Bitcoin, and only converts some of it to euros, solely because you can’t pay taxes in Bitcoin.

Patzer buys the beer for Room 77 from the nearby Rollberg brewery, owned and run by qualified brew and malt meister Wilko Bereit. He pays for the barrels with Bitcoins

Joerg also told me an interesting case where there was a group of tourists having a dinner in his restaurant. A friend of theirs that wasn’t able to make the trip insisted on paying the bill remotely from Israel. This is something that would be hard to do with the traditional credit cards or currencies. Paying in Bitcoin is quite straightforward:

He taps the amount he owes Room 77 into the virtual Bitcoin wallet on his Android phone and, aligning it with a code on the bar’s device, presses a button to process the payment. A theatrical “kerching” sound follows and Gallas is grinning from ear to ear. “It could hardly be easier,” he insisted.

Requesting a payment using Bitcoin Wallet for Android

Global currency for the internet

Map of global Bitcoin nodes

Requesting Bitcoins is as easy as providing a URL. This could be presented as a link on the web, or shared to mobile devices via QR Codes. For example, clicking this link on a device with a Bitcoin wallet would enable you to send me Bitcoins worth around 0.1 EUR (on today’s exchange rates). No need for dealing with payment gateways or complicated protocols.

Ease of paying, no middle men, minimal transaction fees, and no risk of chargebacks make a lot of interesting business scenarios possible online. Consider for example micropayments for online content. This is something that would make online publishing viable without the current ad-driven models, but has been held back by the risks and costs associated with credit card payments. Wider adoption of Bitcoin would suddenly make this a very realistic way of asking for money for the things you do online.

Other clever uses

Since Bitcoins can be exchanged in miniscule amounts, and there is a public record kept on each transaction, they enable all kinds of new use cases.

One such service that I ran into yesterday is Proof of Existence, a tool for storing the fact that you were in possession of a document at a given time, and that it hasn’t been revised since based on microtransactions stored in the Bitcoin block chain. This is a mechanism known as trusted timestamping that could be useful for things we currently utilize notaries for. The Hacker News thread on this listed some interesting use cases, like verifying scientific priority on new inventions, i.e. that you had written a document describing a new invention at a given time even if you publish it only later.

We could also use Bitcoin for solving the spam problem. Emails could be “stamped” with a Bitcoin microtransaction, and email servers could be configured to reject messages that don’t have a valid payment attached to them. This would enable bringing decentralization back to the internet as we could again use distributed systems like email and XMPP without having to resort to the walled gardens like Facebook to keep them spammers away.

Another interesting approach was to use Bitcoin transactions for making bets.

How to get started

A good way to get started with Bitcoin is to read Wired’s Bitcoin survival guide or the materials available on Bitcoins.com. Install a Bitcoin wallet application on your mobile device. Then get some Bitcoin either by buying locally, from an online exchange or by selling goods or services for Bitcoin.

This is all you need to get going. Now you just need to find a way to spend them, for example in online shops, by buying domains, or in shops and restaurants that accept them, like many do in my neighborhood.

Disclaimer: I’m a programmer, not an economist. There are better people to take investment advice from. I do hold a little bit of Bitcoin, giving me some vested interest in the matter. Not the kind of amount that would make it an investment, but just the right amount for being able to buy a few nice dinners, especially at the Berlin prices. Generally, I try to keep the amount I have in BTC low enough to be equivalent of what I might be comfortable with carrying in cash.

Why did I reimplement Jekyll using NoFlo

henri.bergius@iki.fi (Henri Bergius) — Mon, 12 Aug 2013 00:00:00 +0000

Jekyll is a delightful piece of software. A Ruby application that turns your Markdown and HTML files to a nicely constructed static website. Since the generated site is static, you can deploy and serve it from anywhere with no security or performance concerns. As a matter of fact, this site is built with Jekyll.

For websites that don’t need to offer dynamic functionality this is in many ways the culmination of Decoupled Content Management — you can write the post using whatever editor, use GitHub as the content repository, and deploy the pages to any web server you want.

If you’ve been to our Kickstarter project you probably know that I’ve reimplemented Jekyll using the NoFlo flow-based programming environment. If I’m so happy with Jekyll as it is, why would I do such a thing?

Introducing noflo-jekyll

Before I go to the reasons, let me briefly introduce the project itself: noflo-jekyll

Just like any flow-based program, the NoFlo reimplementation of Jekyll is built out of a graph. Here is how the main graph of the application looks like when loaded via the NoFlo UI prototype:

Quite a few of the boxes you see in that graph are actually subgraphs. Since the UI is still under work, the subgraphs have been defined in the more text editor friendly .fbp syntax.

In total at the current version the graphs contain 107 nodes. Most of these running a component from the extensive NoFlo component libraries, but there are also 4 custom components built for this project. These components put together are under 500 lines of code, and everything else is using 100% reusable code from existing libraries. Not bad compared to the size of the original Jekyll code base at over 16k LoC!

You can grab the first release of noflo-jekyll right now from NPM. Please refer to the README on how to use it.

Where this is coming from

When I started working with the rest of The Grid team last year, Jekyll development had practically stopped. It looked like nothing was happening in that community. Since Jekyll was something we very much wanted to utilize for various projects, the situation looked a little bit risky.

Since we’re very much a NoFlo shop, it felt natural to take Jekyll as sort of a specification, and reimplement it using FBP techniques. Initially this was an internal effort, but then we decided, very much in the nature of GitHub’s “open source (nearly) everything” philosophy, that this should be opened as well.

I got noflo-jekyll into a running state in about a week, and then moved my attention to other things we needed to build. And so the finishing touches and a public release were postponed to a better day.

As it happens, Jekyll development resumed pretty soon after that. But there are still many reasons why noflo-jekyll can be quite cool.

Benefits of the NoFlo port

Here are some reasons why especially Node.js developers should care about this project:

Pure JavaScript, no need for Ruby or other runtimes in your environment. Especially handy if you’re using Grunt for site generation
Other data sources, in NoFlo everything is just a flow of data. You could easily plug in other data sources than the file system. For example, database query results
Different converters, don’t want to use Markdown? Just plug in your own mark-up processor component
Different template engines, don’t want to use Liquid? Just plug in your own template processor component
Use as library or executable, this Jekyll implementation is just a NoFlo graph. You can use it in other NoFlo applications, as a Node.js module, or as a command-line executable

However, as with any reimplementation of a application being actively developed, there are also some potential caveats to observe.

A NoFlo example

Most of the existing NoFlo applications out there are dealing with various business processes, and so very little of that is available out in the open. As such, noflo-jekyll can probably show how to build bigger systems out of flow-based graphs, and also how to connect them with the rest of the Node.js ecosystem.

noflo-jekyll is now available under the MIT license for your static site generation needs. But as it happens, it wasn’t the only interesting NoFlo example to be released this week. You may want to also check out how to handle HTTP requests in NoFlo with the Woute module. There is even an example project available.

Keeping up with Jekyll

To realize the benefits of flow-based static site generation, it is quite important to keep the NoFlo reimplementation up-to-speed with things changing in Jekyll-land.

Because of this, the most important part of our automated tests is comparing results between a site generated by the original Jekyll program, and the NoFlo version. This means that when new features are added to Jekyll, we can follow the red-green-refactor style of test-driven development, and add these features to the test site.

I’ve been happily building my blog and the NoFlo site with this for a while, and except for some minor templating and Markdown handling differences between the Node.js and Ruby versions of those libraries, things are looking solid. If you find something that doesn’t work with the site or templates you have, please let us know!