It is nearly a decade since NoFlo 1.0, and it is time for a new major release. NoFlo was on hiatus for a while as I worked on the boat but now on the world cruise there are good chunks of time sitting at anchor waiting for a weather window. And so we’re close to a 2.x release.

The NoFlo ecosystem is quite big, and the current breed of LLMs and the Pi Coding Harness have made the sizable mechanical conversion and upgrading tasks a lot more feasible.

TL;DR: ES Modules with zero external dependencies, no build needed. Backpressure, better firing patterns and network shutdown system. New Reticulum-based remote control protocol designed to work over WebSockets, TCP, and LoRa. Assembly Line Programming support is now first-party. Ports can be typed with JSON Schemas.

LLMs and Flow-Based Programming

Before going to the actual changes in the 2.x series, it is probably good to tackle the elephant in the room: are dataflow tools like NoFlo at all relevant in a world where you can just throw tokens at an LLM and have it churn out a PhotoShop clone without further guidance?

I genuinely think that the answer is yes.

LLMs mean more software, and probably more complex software being built. And even though any software can be understood by throwing enough tokens at it, the dataflow/component separation of Flow-Based Programming can make software much easier and hence cheaper to understand and modify.

Having the automatically up-to-date dataflow diagram and the observability of NoFlo means that the human driving the LLM, and the LLM itself can have a much better shared understanding of how a program works. It can also help when multiple agents need to coordinate work. I’ve seen agents discover bugs by looking at flowtraces without having to litter the code with debug output.

As I argued in an earlier post, Flow-Based Programming is designed ground-up for this sort of separation of labour. J. Paul Morrison, the inventor of FBP wrote:

Just as in the preparation and consumption of food there are the two roles of cook and diner, in FBP application development there are two distinct roles: the component builder and the component user or application designer.

No build, no dependencies

The biggest immediate change is how NoFlo is now structured. We migrated a lot of the “main” libraries to a single monorepo, so the NoFlo framework and supporting tooling like the FBP Protocol runtime can be versioned together.

Everything is now plain JavaScript with ES Modules. No CommonJS, no build. There’s been a radical reduction of runtime dependencies to the point where NoFlo doesn’t require anything except itself and its own zero-dependency graph library. We now use EventTarget and Web Streams for our lifecycle events and dataflow instead of node:events.

Development dependencies have similarly been pruned substantially. We use node:test for test automation, Biome for formatting, and verify our JSDoc type annotations using the TypeScript compiler.

This dependency-light setup leans on the standard web platform, meaning that a NoFlo 2.x project is likely to stay runnable unmodified for decades to come, and if changes are needed, they won’t need days of build tool archeology before you can change a graph or switch a component.

Apart from making NoFlo much more durable, this also makes it a lot more multiplatform. We now support not only Node.js and evergreen browsers, but also any WinterTC compliant JS runtime, Deno and Bun included.

New packages

NoFlo 2.x series has moved its packaging to the @noflo namespace. We have a set of alpha packages already up on NPM.

npm install @noflo/noflo@next

The next tag currently points to alphas, and will point to a beta once we get there.

We will also publish these on JSR assuming they get their publishing pipeline unstuck.

Existing component libraries are being migrated and published under the @noflo namespace as we go, prioritizing components needed for verifying various aspects of the 2.x series or used for the dogfooding projects.

Correctness and classical FBP

Back in the NoFlo 0.x era, J. Paul Morrison published a lengthy article contrasting NoFlo and “classical FBP”.

While a lot of credit is due the NoFlo team for bringing FBP to the attention of the computer world, the NoFlo implementation of FBP in fact differs in a number of respects from “classical” FBP as it has evolved over the last 50+ years, and in fact operates on a completely different paradigm, although it shares with “classical” FBP a number of technical and philosophical ideas.

While 1.x and the Process API eliminated a lot of the concerns, we’ve taken the opportunity to make NoFlo very close to classical FBP paradigms in the 2.x series, while still keeping it idiomatic JavaScript. Changes include:

• We now have backpressure support
• Firing patterns in Process API are now much more complete, removing deadlocking risk in complex asynchronous scenarios
• Information Packets are automatically cloned in fanout outports. This means IPs always have a single owner
• Network shutdown is now a cascade of “End of Transmission” signals starting from IIPs and flowing down the chain

The biggest remaining divergence is multi-process support, and we’re planning to add it, supporting running components either inline (NoFlo 1.x style), as their own Web Workers, or as remote instances connected via FBP Protocol.

There’s a living document analyzing the original critique and the (evolving) state of NoFlo 2.x against it. It is a shame JPM is no longer among us. I think this new version would spark an interesting conversation.

Assembly line programming

Vladimir Sibirov’s Assembly Line Pattern is a great way to make complex FBP programs manageable. Instead of building visual coding spaghetti, you can build a linear assembly line where message objects travel through the line, accumulating data and errors as they go along:

Consider whole application as an industrial factory producing complex objects. Parts of the application that implement specific features are assembly lines. Multiple assembly lines are connected together in a shape similar to a skeleton of fish. Assembly lines consist of worker components and incoming objects are passed from one worker to another, producing a complete or intermediate result at the end of the line.

In 2.x we’ve ingested the Assembly support library and utilities into the monorepo. This means they’ll stay automatically up to date and maintained as NoFlo evolves.

A big part of the 2.x push is the asAssembly utility that enables regular NoFlo components or graphs to be wrapped into Assembly paradigm. This means you can use any community components (say from noflo-filesystem or noflo-yaml) unmodified in your Assembly program.

import { asAssembly } from "@noflo/assembly";

const graph = asAssembly(innerComponent, {
  target: "filesystem/CopyFile",
  mapping: {
    // optional overrides; defaults map port name ↔ msg field name
    in: { source: "source" },
    out: { destination: "destination" },
  },
});

The result of the above is a NoFlo subgraph wrapping the filesystem/CopyFile component into standard Assembly message passing flow, including error handling and scope isolation. The same procedure will work with any modern NoFlo component out there.

Schemas

NoFlo has had basic data type declarations for a long time, enabling components to declare that their “in” port accepts Strings for example.

NoFlo 2.x expands this with end-to-end support for JSON Schemas. Schemas can be shipped together with a component library, and can be applied to particular ports or even Information Packet objects.

Schema support enables editors like NoFlo UI to produce richer input forms for entering Initial Information Packets. It also makes it possible to validate component compatibility better when building graphs.

See the Port Schemas document for more information.

New FBP Protocol

The old FBP Protocol had several problems. The security story with “runtime secrets” was weak, and the runtime and client had to be able to connect directly, meaning every client and runtime had to support WebSockets, WebRTC, etc, making for a complex fragmented setup. I wanted to fix this from ground up, and found an interesting networking stack:

NoFlo 2.x uses a new FBP Protocol built ground-up on top of the Reticulum Network Stack. The new protocol uses much more compact MsgPack binary payloads instead of JSON, and utilizes Reticulum identities for encrypting and signing all messages.

Access control is now handled with my Dacar library, enabling decentralized and granular access control to various protocol features. Reticulum guarantees that you know who is talking to who, Dacar determines what they are allowed to do.

Packet and network state introspection uses the Flowtrace format, making it possible to use the same tooling to observe a running program as a trace from a previous execution error or failed CI run.

The mesh nature of Reticulum means that the runtime and the client don’t need to support the same transport, or even be in the same network. Messages can travel from WebSockets to TCP to LoRa on their way from IDE to runtime.

For anybody interested in adopting the new FBP Protocol, the AsyncAPI contract is available in the monorepo. NoFlo UI is already being ported over.

@noflo/fbp-client provides a new command-line tool for driving FBP Protocol runtimes. Here’s an example session:

$ fbp-client scan
my-runtime 3f9a1c2e...
$ fbp-client add-node core/Repeat --id repeat
repeat applied at epoch 7c31...
$ fbp-client add-node core/Output --id out
out applied at epoch 9d02...
$ fbp-client add-edge repeat out out in
edge repeat:out -> out:in applied at epoch b447...
$ fbp-client add-iip repeat in 'Hello, world!'
iip -> repeat:in applied at epoch c1f8...
$ fbp-client run

The last command would actually execute the graph and output a Hello, world!. The epochs mentioned above are graph versions. Each modification is versioned so that runtime and client don’t always need to send a full state to get in sync.

Timeline

The alphas already work quite well. This is a good time to start experimenting with the new NoFlo.

I’m doing various dogfooding projects with NoFlo 2.x, and will be going forward with betas and eventual stable as soon as I’ve proven that we can commit to the current API.

The component library migration guide is a good place to start. We’ll add a separate document for migrating full applications, but generally it should be quite easy as long as you’re utilizing the public API in NoFlo, and are using Promises instead of callbacks. Consider adopting Assembly for anything more complex.