NoFlo — the flow-based programming system I started — is now two years old. I pushed the first commits to GitHub on June 5th 2011 from Hacker Dojo in Mountain View. To get us started with the story, I’ll let Wikipedia summarize:

Flow-based programming (FBP) is a programming paradigm that defines applications as networks of “black box” processes, which exchange data across predefined connections by message passing, where the connections are specified externally to the processes. These black box processes can be reconnected endlessly to form different applications without having to be changed internally.

While flow-based programming is still far from mainstream, it has been great to watch to the community grow around NoFlo.

There are several start-ups using it as their base infrastructure, with several of their engineers contributing to the open source effort. I’ve personally used the system for quite a wide range of tasks from web services and document transformations to handling payments and user on-boarding processes.

Why I started NoFlo

Two years ago I was undergoing a transition from PHP and Python to the JavaScript world, largely lured by the benefits of a universal runtime and the event-based multi-protocol paradigm Node.js offers.

While new programming languages and environments are easy to get into, this transition provided yet another point where I would have to sit down and port over all the little libraries and utilities I’ve grown to rely on over the years.

I wondered if there could be a better way.

To figure that out, I spent the time I had when traveling between conferences and hackathons of the IKS Project reading various computer science papers on different programming paradigms. Eventually I stumbled upon some mentions of flow-based programming. I dug deeper, and finally read the canonical book on the subject — Paul Morrison’s Flow-based programming, 2nd edition.

Having worked with component architectures and Unix pipes before, the idea resonated with me. To think things through, I took the excuse of having some meetings in Portland to drive up there following the beautiful California 1 coastal road. Couple of days alone in a car is a great way to let your mind flow!

Implementing your own is usually the best method for learning a new concept, and so when I got back to the Bay Area, I decided to write an FBP system of my own on Node.js.

I also kept a Qaiku thread on the things I discovered, parts of which I later republished on this site.

Beyond OOP

Object-oriented programming and Model-View-Controller have been the dominant programming paradigms since the desktop computing era of the 90s, even while the world has become more connected and multi-device. While these concepts did improve some things, the big promises of programmer productivity and component reuse have mostly been left unrealised.

Software has become one of the most important aspects of business and the society. As an effect, the demand for programmers vastly outstrips the amount of computer science graduates we’re able to produce.

The tools side of things isn’t looking much better, either. While IDEs are admittedly improving all the time, most of the talented programmers have ditched them and moved back to the console-based editors of the 80s like vim and Emacs, many following the Unix as your IDE idea. Clearly the productivity boost given by IDEs doesn’t yet match the overhead of using them.

These areas are where flow-based programming can help.

The logic is in the graph

When we design software, we usually start by drawing boxes and arrows on a whiteboard. Later on these are then translated to actual software through various text files containing classes, methods, and configuration information.

As the software evolves, people rarely go back to the original drawing and update it, effectively making the source code only place documenting how different pieces of a system fit together.

With FBP, the logic of the software is designed as a graph — much like a flowchart — and stays as a graph.

The boxes of the graph depict various instances of reusable components, and the arrows the wiring connecting their ports together.

The graph is what FBP systems like NoFlo execute, and it is also something that can be easily drawn on the screen, or even edited visually.

We humans are visual creatures. We are much better at recognizing shapes and visual connections than at finding them from a jumble of text files.

Each node (or process in FBP terminology) of a graph is an instance of a component. Just like objects in OOP can receive and react to messages (method calls), NoFlo components react to packets they receive through their input ports, process them, and eventually send something to their output ports.

The graph decides where the output is sent, meaning that the overall behavior of a program can be modified by just rewiring some of these connections. In traditional OOP the connections between various objects are usually hardcoded, or managed by a complicated dependency injection systems.

Component reuse

Because FBP components are just black boxes performing some well-defined task on incoming packets, they can be connected with each other in multitude of different ways to produce a different behaviour. This gives FBP systems a much better scale of code reuse than traditional OOP or procedural environments.

As an example, my typical NoFlo applications only contain some 5-15% of components written specifically for that project. The rest are all coming from the growing ecosystem of ready-made NoFlo components.

Writing and publishing components is already quite easy, and is becoming even faster and more reliable through tools like the Grunt component scaffolder and noflo-test.

The more components are out there the less time we need to spend writing code, and the more we can focus on designing the software logic itself.

NoFlo on the browser

Apart of better tooling and more components, the other big improvement in NoFlo has been support for running FBP programs on the client. Originally NoFlo was designed for server-side flows, but thanks to the improving client-side Component ecosystem, it became feasible to expose the environment also to web browsers.

This is keeping true with the promises of the universal runtime. With the next release of NoFlo, flow-based programs can be run on pretty much any computing device, whether a Node.js equipped web server, or a smartphone with a browser.

The ability to run client-side flow-based programs presents new opportunities. There is a tradition of using tools like FilterForge for visual effects, or Quartz Composer for user interaction design. As a matter of fact, Facebook Home was designed using flow-based tools:

…something like Facebook Home is completely beyond the abilities of Photoshop as a design tool. How can we talk about physics-based UIs and panels and bubbles that can be flung across the screen if we’re sitting around looking at static mocks? (Hint: we can’t.) It’s no secret that many of us on the Facebook Design team are avid users of Quartz Composer, a visual prototyping tool that lets you create hi-fidelity demos that look and feel like exactly what you want the end product to be.

Since NoFlo graphs run on any device including the tablets and smartphones that the application being designed is likely to target, it can provide an even better environment for such prototyping. There are lots of opportunities for a new tool here, especially given that Quartz Composer’s future is quite uncertain.

We are already doing some work on visual interaction components for NoFlo. This could be huge for NoFlo and FBP in general!

UI is the missing part

What we have with NoFlo is already a quite solid programming environment:

• Flow-based engine that works well in both browser and Node.js
• Growing ecosystem of reusable open source components
• Framework for quickly scaffolding and testing new components
• Domain-specific language for defining NoFlo graphs

However, the missing part is a tool that would allow viewing and editing NoFlo graphs visually. Sure, DrawFBP is there, and can be used with NoFlo. But something fitting modern touchscreen interactions and more connected to the live graphs would be better.

I did some prototypes on this with noflo-ui, and wrote down bunch of thoughts on how the graphs would be best shown. There has also been some collaboration with Forrest Oliphant of Meemoo fame.

Last week I sat down with Leigh Taylor, the original designer behind the Medium blogging platform, to go through the various ideas and concepts we had. Based on this we’re starting to have a quite solid design to continue working with.

There will be more on that in the near future, but here is a sneak preview:

Telling the story of NoFlo

Apart from the user interface, the other important task ahead of us is to grow the community around NoFlo and FBP in general.

This means producing better documentation, and explaining the concept in conferences and meetups. I gave a talk on NoFlo in JS.Everywhere 2011, but the project has moved quite far ahead since then.

To tell the story of NoFlo and FBP we’re currently filming a set of interviews with the people involved from the different sides. We’re talking with NoFlo contributors, designers, and with people who have years (or even decades!) of experience with flow-based programming. This will hopefully come out next month.

Expect also more posts and tutorials here on my blog.