Henri Bergius - Nodejs

Flying a Quadrocopter with NoFlo

henri.bergius@iki.fi (Henri Bergius) — Mon, 09 Sep 2013 00:00:00 +0000

The Parrot AR.Drone is quite a lot of fun, and also quite hackable. We recently got one, and the first thing to do was to connect the excellent Node.js ar-drone module with the NoFlo flow-based programming framework.

While quite a lot of work remains, it was already very satisfying to see how the drone was able to fly patterns based on the NoFlo graphs we created.

This will obviously be more interesting when we have the rest of the motion commands implemented, and can start reading the output of the various sensors on the drone to make it react to its environment. Also interesting would be to connect this to the various other NoFlo libraries so that the Drone could be controlled by other protocols, or could react to things happening in other web services.

You can see some example flows in Susanna’s droning project.

NoFlo and flying robots. What could go wrong? Just a little hint from back in April:

The NoFlo Kickstarter reached 100% of the funding goal this morning! Now that things are going smoothly there, I can focus more on this and other examples. Stay tuned!

Why did I reimplement Jekyll using NoFlo

henri.bergius@iki.fi (Henri Bergius) — Mon, 12 Aug 2013 00:00:00 +0000

Jekyll is a delightful piece of software. A Ruby application that turns your Markdown and HTML files to a nicely constructed static website. Since the generated site is static, you can deploy and serve it from anywhere with no security or performance concerns. As a matter of fact, this site is built with Jekyll.

For websites that don’t need to offer dynamic functionality this is in many ways the culmination of Decoupled Content Management — you can write the post using whatever editor, use GitHub as the content repository, and deploy the pages to any web server you want.

If you’ve been to our Kickstarter project you probably know that I’ve reimplemented Jekyll using the NoFlo flow-based programming environment. If I’m so happy with Jekyll as it is, why would I do such a thing?

Introducing noflo-jekyll

Before I go to the reasons, let me briefly introduce the project itself: noflo-jekyll

Just like any flow-based program, the NoFlo reimplementation of Jekyll is built out of a graph. Here is how the main graph of the application looks like when loaded via the NoFlo UI prototype:

Quite a few of the boxes you see in that graph are actually subgraphs. Since the UI is still under work, the subgraphs have been defined in the more text editor friendly .fbp syntax.

In total at the current version the graphs contain 107 nodes. Most of these running a component from the extensive NoFlo component libraries, but there are also 4 custom components built for this project. These components put together are under 500 lines of code, and everything else is using 100% reusable code from existing libraries. Not bad compared to the size of the original Jekyll code base at over 16k LoC!

You can grab the first release of noflo-jekyll right now from NPM. Please refer to the README on how to use it.

Where this is coming from

When I started working with the rest of The Grid team last year, Jekyll development had practically stopped. It looked like nothing was happening in that community. Since Jekyll was something we very much wanted to utilize for various projects, the situation looked a little bit risky.

Since we’re very much a NoFlo shop, it felt natural to take Jekyll as sort of a specification, and reimplement it using FBP techniques. Initially this was an internal effort, but then we decided, very much in the nature of GitHub’s “open source (nearly) everything” philosophy, that this should be opened as well.

I got noflo-jekyll into a running state in about a week, and then moved my attention to other things we needed to build. And so the finishing touches and a public release were postponed to a better day.

As it happens, Jekyll development resumed pretty soon after that. But there are still many reasons why noflo-jekyll can be quite cool.

Benefits of the NoFlo port

Here are some reasons why especially Node.js developers should care about this project:

Pure JavaScript, no need for Ruby or other runtimes in your environment. Especially handy if you’re using Grunt for site generation
Other data sources, in NoFlo everything is just a flow of data. You could easily plug in other data sources than the file system. For example, database query results
Different converters, don’t want to use Markdown? Just plug in your own mark-up processor component
Different template engines, don’t want to use Liquid? Just plug in your own template processor component
Use as library or executable, this Jekyll implementation is just a NoFlo graph. You can use it in other NoFlo applications, as a Node.js module, or as a command-line executable

However, as with any reimplementation of a application being actively developed, there are also some potential caveats to observe.

A NoFlo example

Most of the existing NoFlo applications out there are dealing with various business processes, and so very little of that is available out in the open. As such, noflo-jekyll can probably show how to build bigger systems out of flow-based graphs, and also how to connect them with the rest of the Node.js ecosystem.

noflo-jekyll is now available under the MIT license for your static site generation needs. But as it happens, it wasn’t the only interesting NoFlo example to be released this week. You may want to also check out how to handle HTTP requests in NoFlo with the Woute module. There is even an example project available.

Keeping up with Jekyll

To realize the benefits of flow-based static site generation, it is quite important to keep the NoFlo reimplementation up-to-speed with things changing in Jekyll-land.

Because of this, the most important part of our automated tests is comparing results between a site generated by the original Jekyll program, and the NoFlo version. This means that when new features are added to Jekyll, we can follow the red-green-refactor style of test-driven development, and add these features to the test site.

I’ve been happily building my blog and the NoFlo site with this for a while, and except for some minor templating and Markdown handling differences between the Node.js and Ruby versions of those libraries, things are looking solid. If you find something that doesn’t work with the site or templates you have, please let us know!

My interview on the origins of NoFlo

henri.bergius@iki.fi (Henri Bergius) — Tue, 06 Aug 2013 00:00:00 +0000

Here is a video interview of me talking about the origins of NoFlo, the flow-based programming environment for JavaScript:

You’ve seen short pieces from this in the NoFlo Kickstarter video, but most of the material is new. This was shot when NoFlo turned two, and so I’m obviously spending some time talking about where things came from.

The video should work well to support the hacker’s perspective of our funding campaign. Please support us and spread the word!

It is pretty amazing to think that since my last post on Friday, we’ve almost doubled the amount of contributions. Thanks to all the nearly 500 backers who have already helped us to change the world of programming!

NoFlo Kickstarter, the hacker's perspective

henri.bergius@iki.fi (Henri Bergius) — Thu, 01 Aug 2013 00:00:00 +0000

This has been a big week for NoFlo, the flow-based programming environment for JavaScript. Yesterday we released NoFlo 0.4, which added support for running flow-based programs in web browsers. And today we launched our NoFlo Development Environment effort on Kickstarter. Before continuing, make sure to watch the video!

This is our effort to bring visual and collaborative flow-based tools into the world of mainstream software development. Similar tools are already in use in many specialized industries from movie special effects to hardware simulation, but we programmers still have to construct these complex maps of our software’s control flow inside our brains based on their textual representation. With your support, we could change that!

We’ve already reached a third of our goal on the first day. Clearly people see the need for these tools. Exciting times!

The stories from TechCrunch and GigaOm tell the story well for non-programmers. But based on the questions I’ve received today, I thought it would be good to clarify various points from a more programmer-centric point of view.

Hasn’t visual programming been tried before?

The story of visual programming tools started with the GRaIL system of the 60s, and has progressed to tools like LabView and Pure Data. So far none of these tools has reached mainstream acceptance outside of their (sometimes fanatic) industry niches.

This is partly because these tools were built originally with a particular problem domain in mind, and partly because of the user experience. Execution matters, as we’ve seen so many times in the tech industry. After all, there were tablets a lot before the iPads.

With our team I have the confidence that we have the necessary skills and vision to build something that is actually a pleasure to use, and that makes it truly easier to work with the control flow of your software than it is with the text editors.

This is not a rehash of UML. UML is a diagram mapping out object-oriented constructs, often used for code generation. NoFlo graphs instead are only the coordination layer that manages the control flow of your software. The components are still handcrafted and unit-tested code that NoFlo merely wires together at runtime, following the edges specified in a JSON file. No code generation here.

Why map out the control flow?

All software is inherently a graph. Functions call other functions, sending data along. Signals are emitted and connections are made. But outside of some debugging tools we rarely see this in a visual format. Instead, when starting to work on a program you have to parse the code and build this map in your head.

This imposes a lot of cognitive load that tools like NoFlo could avoid. When you can see visually how things are connected, you can focus on the bigger picture and build the software you need in a more efficient way. This is what Bret Victor talked about recently.

What is the role for code, then?

However, there still remains a role for text-based code. The actual components, the boxes in the graph, are still written out in JavaScript. But since they’re isolated from their surroundings until a NoFlo graph wires them together, each component can focus on accomplishing a single task well.

My original NoFlo UI prototype already included the code editor for modifying and creating new components. By the principles of TDD, each component is always edited alongside its unit tests, and the tests can be run at any point with a single click. We’re now bringing that back into the new UI:

For those who don’t want this UI, NoFlo is still fully usable also without it. As a matter of fact, we don’t have a UI before the Kickstarter succeeds, and yet many companies are already building their applications with NoFlo. One way is by using the .fbp language.

Why now?

Another reason why the NoFlo development environment may succeed where many others failed is that programming has changed.

We no longer target a single protocol — whether the Win32 API or HTTP — in our applications. Instead, we need to talk multiple protocols and with multiple devices at the same time. Even just dealing with both REST and WebSockets is difficult to many traditional programming environments.

The other big change is the usage of different web APIs as part of your applications. Authentication, handling asynchronous requests and services that are sometimes down can be a pain.

NoFlo brings a controlling layer to your software that allows you to map out these scenarios and isolate the handling of each protocol or API into its own set of small, simple components.

Where could this be used?

Many of the nodal editing tools built in the past have been very domain-specific. However, flow-based programming is a general software development paradigm. NoFlo has already been used for a wide variety of tasks, including:

Business data extraction and reporting
Automating billing workflows
Receiving, routing, and sending text messages
Static site generation
Building server-side web applications

Now that we added browser support it will be possible to also build physics-based user interfaces with NoFlo. There should be some interesting examples of that coming up soon.

NoFlo being a general JavaScript library could in future enable us to coordinate the flows also in new kinds of places like desktop applications. There are several desktop environments where JavaScript is a first-class citizen. Combining flow-based interactions with declarative UI definitions could be something very powerful!

Many of the giants of the software industry, like Google, Facebook, Microsoft, Adobe, and Mozilla are all working on improving the JavaScript development tools. It would be awesome to have their support for what we’re doing.

How about open source?

If you’re following this blog, you probably know that I feel strongly for software freedom. Everything I’ve done during my professional career has been possible only thanks to the open source community.

NoFlo and the UI we’re building are and will remain open source available under the MIT license. As a matter of fact, the UI is designed to not only work with NoFlo, but also to be adaptable to work with other flow-based systems. I feel this is an area with a lot of potential for collaboration with the various functional and dataflow projects out there.

We will however be offering a hosted version of the software for a fee. The various Kickstarter rewards will give our backers an early and cheaper access to that.

But still, you’ll always be able to run the whole stack on your own infrastructure if you choose to do so.

What happens next?

Our Kickstarter campaign has been up for less than a day, and we’re already above 30% of our goal. This makes me quite optimistic for the effort. But of course we won’t succeed without the help of the wider open source and JavaScript community!

There are still quite a lot of days left in the campaign. Tonight I’ll be flying back to Europe, and then we’ll focus on the next steps.

One important area of attention is publishing more demos and examples of NoFlo in real-world use. I hope by early next week we’ll be able to show a few applications running on both browser and Node.js. I will also publish our flow-based port of the Jekyll static site generator. These should help people getting started with this style of programming.

We also have quite a lot of content lined up and waiting for editing. Something of interest to everybody working with flow-based or functional programming will be the long cut of the interview we made with J. Paul Morrison, the father of FBP. My hope is that video will be live by the end of the next week.

If you’re interested in the developments, make sure to follow NoFlo on Twitter, Facebook, or Google+. If you back our campaign, you’ll also receive the updates via Kickstarter.

Thanks everybody for helping to make this possible! Keep spreading the world and giving your support!

NoFlo: two years of flow-based programming

henri.bergius@iki.fi (Henri Bergius) — Wed, 05 Jun 2013 00:00:00 +0000

NoFlo — the flow-based programming system I started — is now two years old. I pushed the first commits to GitHub on June 5th 2011 from Hacker Dojo in Mountain View. To get us started with the story, I’ll let Wikipedia summarize:

Flow-based programming (FBP) is a programming paradigm that defines applications as networks of “black box” processes, which exchange data across predefined connections by message passing, where the connections are specified externally to the processes. These black box processes can be reconnected endlessly to form different applications without having to be changed internally.

While flow-based programming is still far from mainstream, it has been great to watch to the community grow around NoFlo.

There are several start-ups using it as their base infrastructure, with several of their engineers contributing to the open source effort. I’ve personally used the system for quite a wide range of tasks from web services and document transformations to handling payments and user on-boarding processes.

Why I started NoFlo

Two years ago I was undergoing a transition from PHP and Python to the JavaScript world, largely lured by the benefits of a universal runtime and the event-based multi-protocol paradigm Node.js offers.

While new programming languages and environments are easy to get into, this transition provided yet another point where I would have to sit down and port over all the little libraries and utilities I’ve grown to rely on over the years.

I wondered if there could be a better way.

To figure that out, I spent the time I had when traveling between conferences and hackathons of the IKS Project reading various computer science papers on different programming paradigms. Eventually I stumbled upon some mentions of flow-based programming. I dug deeper, and finally read the canonical book on the subject — Paul Morrison’s Flow-based programming, 2nd edition.

Having worked with component architectures and Unix pipes before, the idea resonated with me. To think things through, I took the excuse of having some meetings in Portland to drive up there following the beautiful California 1 coastal road. Couple of days alone in a car is a great way to let your mind flow!

Implementing your own is usually the best method for learning a new concept, and so when I got back to the Bay Area, I decided to write an FBP system of my own on Node.js.

I also kept a Qaiku thread on the things I discovered, parts of which I later republished on this site.

Beyond OOP

Object-oriented programming and Model-View-Controller have been the dominant programming paradigms since the desktop computing era of the 90s, even while the world has become more connected and multi-device. While these concepts did improve some things, the big promises of programmer productivity and component reuse have mostly been left unrealised.

Software has become one of the most important aspects of business and the society. As an effect, the demand for programmers vastly outstrips the amount of computer science graduates we’re able to produce.

The tools side of things isn’t looking much better, either. While IDEs are admittedly improving all the time, most of the talented programmers have ditched them and moved back to the console-based editors of the 80s like vim and Emacs, many following the Unix as your IDE idea. Clearly the productivity boost given by IDEs doesn’t yet match the overhead of using them.

These areas are where flow-based programming can help.

The logic is in the graph

When we design software, we usually start by drawing boxes and arrows on a whiteboard. Later on these are then translated to actual software through various text files containing classes, methods, and configuration information.

As the software evolves, people rarely go back to the original drawing and update it, effectively making the source code only place documenting how different pieces of a system fit together.

With FBP, the logic of the software is designed as a graph — much like a flowchart — and stays as a graph.

The boxes of the graph depict various instances of reusable components, and the arrows the wiring connecting their ports together.

The graph is what FBP systems like NoFlo execute, and it is also something that can be easily drawn on the screen, or even edited visually.

We humans are visual creatures. We are much better at recognizing shapes and visual connections than at finding them from a jumble of text files.

Each node (or process in FBP terminology) of a graph is an instance of a component. Just like objects in OOP can receive and react to messages (method calls), NoFlo components react to packets they receive through their input ports, process them, and eventually send something to their output ports.

The graph decides where the output is sent, meaning that the overall behavior of a program can be modified by just rewiring some of these connections. In traditional OOP the connections between various objects are usually hardcoded, or managed by a complicated dependency injection systems.

Component reuse

Because FBP components are just black boxes performing some well-defined task on incoming packets, they can be connected with each other in multitude of different ways to produce a different behaviour. This gives FBP systems a much better scale of code reuse than traditional OOP or procedural environments.

As an example, my typical NoFlo applications only contain some 5-15% of components written specifically for that project. The rest are all coming from the growing ecosystem of ready-made NoFlo components.

Writing and publishing components is already quite easy, and is becoming even faster and more reliable through tools like the Grunt component scaffolder and noflo-test.

The more components are out there the less time we need to spend writing code, and the more we can focus on designing the software logic itself.

NoFlo on the browser

Apart of better tooling and more components, the other big improvement in NoFlo has been support for running FBP programs on the client. Originally NoFlo was designed for server-side flows, but thanks to the improving client-side Component ecosystem, it became feasible to expose the environment also to web browsers.

This is keeping true with the promises of the universal runtime. With the next release of NoFlo, flow-based programs can be run on pretty much any computing device, whether a Node.js equipped web server, or a smartphone with a browser.

The ability to run client-side flow-based programs presents new opportunities. There is a tradition of using tools like FilterForge for visual effects, or Quartz Composer for user interaction design. As a matter of fact, Facebook Home was designed using flow-based tools:

…something like Facebook Home is completely beyond the abilities of Photoshop as a design tool. How can we talk about physics-based UIs and panels and bubbles that can be flung across the screen if we’re sitting around looking at static mocks? (Hint: we can’t.) It’s no secret that many of us on the Facebook Design team are avid users of Quartz Composer, a visual prototyping tool that lets you create hi-fidelity demos that look and feel like exactly what you want the end product to be.

Since NoFlo graphs run on any device including the tablets and smartphones that the application being designed is likely to target, it can provide an even better environment for such prototyping. There are lots of opportunities for a new tool here, especially given that Quartz Composer’s future is quite uncertain.

We are already doing some work on visual interaction components for NoFlo. This could be huge for NoFlo and FBP in general!

UI is the missing part

What we have with NoFlo is already a quite solid programming environment:

Flow-based engine that works well in both browser and Node.js
Growing ecosystem of reusable open source components
Framework for quickly scaffolding and testing new components
Domain-specific language for defining NoFlo graphs

However, the missing part is a tool that would allow viewing and editing NoFlo graphs visually. Sure, DrawFBP is there, and can be used with NoFlo. But something fitting modern touchscreen interactions and more connected to the live graphs would be better.

I did some prototypes on this with noflo-ui, and wrote down bunch of thoughts on how the graphs would be best shown. There has also been some collaboration with Forrest Oliphant of Meemoo fame.

Last week I sat down with Leigh Taylor, the original designer behind the Medium blogging platform, to go through the various ideas and concepts we had. Based on this we’re starting to have a quite solid design to continue working with.

There will be more on that in the near future, but here is a sneak preview:

Telling the story of NoFlo

Apart from the user interface, the other important task ahead of us is to grow the community around NoFlo and FBP in general.

This means producing better documentation, and explaining the concept in conferences and meetups. I gave a talk on NoFlo in JS.Everywhere 2011, but the project has moved quite far ahead since then.

To tell the story of NoFlo and FBP we’re currently filming a set of interviews with the people involved from the different sides. We’re talking with NoFlo contributors, designers, and with people who have years (or even decades!) of experience with flow-based programming. This will hopefully come out next month.

Expect also more posts and tutorials here on my blog.

Writing reusable, multi-platform JavaScript with Component

henri.bergius@iki.fi (Henri Bergius) — Thu, 11 Apr 2013 00:00:00 +0000

I’m currently in the process of porting the NoFlo Flow-Based Programming environment to run also in the browser. While there are some obvious differences in things like filesystem interaction and component loading, the goal here is to reuse as much of the same code as possible between these two platforms.

Many of the building blocks are already in place, and so the port should be complete still this week. You can track the work in issue 63.

Update: the post below is mostly of historical value. Nowadays I would recommend using NPM for dependency management and Webpack for browser builds. For NoFlo, grunt-noflo-browser provides automation for this.

A fragmented ecosystem

The current client-side JavaScript ecosystem is quite fragmented. While on general level any code can be used anywhere, there are many different approaches at packaging, code loading, and templating. In many ways this resembles the PHP landscape before Composer solved the problem there.

On Node.js we haven’t had this problem for a while, as NPM provides an excellent way to share and install code. With more than 27000 modules available, it is truly the default solution for JavaScript package management server-side. Some frameworks like Meteor tried to deviate from this by introducing their own package managers, but eventually came back to the fold.

Module definitions

There are many different ways for handling loading and encapsulation of JavaScript code. The module pattern is quite popular building block. Many also expose their code as jQuery plugins even though that really only makes sense or DOM handling.

Here is how you would define a jQuery plugin:

(function ($) {
  $.fn.somePlugin = function () {
    // some code here
  };
})(jQuery);

Some efforts towards standardization have been made, including the Asynchronous Module Definition spec, and the synchronous CommonJS module spec that Node.js also uses.

Here is how an AMD wrapper for code looks like:

define('somePlugin', ['jquery'], function ($) {
  return function () {
    // some code here
  };
});

The CommonJS definition for something similar would be the following. This will look familiar to Node.js developers:

var $ = require('jquery');
exports.someFunction = function () {
  // some code here
});

Harmony is the proposed next generation of the JavaScript language, and it includes a new module syntax:

module 'myModule' {
  import 'jquery' as $;
  export function someFunction () {
    // some code here
  }
}

It is no wonder many developers feel a bit lost on how they should expose their code or widgets!

The Harmony module spec may eventually harmonize this, but it will take a while before you can actually ship code like that for even a reasonably majority of browsers.

Module installation and loading

Once you’ve picked the module pattern to follow, the next question is how to actually get your dependencies installed.

The traditional way is to fetch “known good” versions of the libraries, add them to your project repository, and then just include each library with its own script tag. But this means having to maintain all the libraries in your own project, and cluttering your repository and change log with them.

A variant of this is loading common libraries from a Content Delivery Network like Google. This has the advantage that your users won’t have to download something like jQuery separately from each web server, and that you don’t need to duplicate the library files in your own code repository. But at the same time, this relies on the CDN provider not breaking things, and complicates offline usage.

And you still have to write script tag for each of them.

Bower

Twitter’s Bower package manager aims to help with dependency management. You declare the libraries your code needs in a component.json file, run Bower, and you’ll get the correct versions downloaded to your system.

Bower only handles dependency resolution and downloading, and so you’ll still be writing script tags for all modules you installed. But at least this allows you to keep the library files out of your repository.

Require.js and volo

The Require.js project seeks to solve this by handling module loading for you in an automated manner. They even provide the volo package manager for installing all the libraries you need.

Require.js is a quite popular way of solving this, but means that you will need to follow the Asynchronous Module Definition specification with your code.

Component

Component is a newer solution for this started by TJ Holowaychuk of Express and Mocha fame.

The system is explained a lot better in the introductory blog post, but in nutshell Component is a combination of a package manager and a module loading system based on CommonJS.

With Component you can easily write and distribute reusable JavaScript modules, including user interface components that may include HTML templates and CSS. There is a component writing tutorial available.

The Component installer will pull all the dependencies for you, and construct a single, easy-to-include JavaScript file out of them and your own code. If you want to think of it in that way, this is sort of similar to Composer generating an autoload file for PHP code.

Which to choose?

Given the multitude of options available, it can be hard to choose which one to go with. Eventually a winner may emerge, but in the meanwhile, my approach is the following:

Component for client-side libraries and widgets
NPM for Node.js libraries

If I was just writing client-side code, Require.js and volo may have been just as good option, at least if I want to deal with AMD.

However, the big advantage of Component is that it is based on CommonJS modules, which Node.js also uses. With it, I can share library code a lot more easily between browser and the server, the two main platforms of the Universal Runtime.

CommonJS modules run nicely in the browser, on Node.js, and other server-side JavaScript runtimes.

Writing a multi-platform library

Writing widgets with Component is covered nicely in the building a date picker component tutorial, and so I’m focusing on how to build more general-purpose libraries here.

Getting Component

The first step with Component is getting the tooling in place. Component — like most of the JavaScript dependency managers — is written on top of Node.js. It would be possible to implement the Component spec in other languages for easier integration in their native toolchain, but for now Node.js is what you must install.

Once you have Node.js running, getting the Component tools is easy:

$ sudo npm install -g component

This will give you the component command. You can run it to see the functionality it provides.

Finding dependencies

The next step is to find the libraries you need. Quite a lot of libraries and widgets are already available, and can be found from the Component Wiki.

You can also use the Component command to look up modules:

$ component search underscore

component/underscore
url: https://github.com/component/underscore
desc: JavaScript's functional programming helper library.

nathan7/memoize
url: https://github.com/nathan7/memoize
desc: underscore's memoize

As you can see, the components use a “GitHub-like” naming scheme of <vendor>/<module>. This is again similar to vendor names in Composer:

The package name consists of a vendor name and the project’s name. Often these will be identical - the vendor name just exists to prevent naming clashes. It allows two different people to create a library named json, which would then just be named igorw/json and seldaek/json.

Since NoFlo relies heavily on Node’s event API, we need to find an equivalent library for Component. After a quick look through component search events, it turns out component/emitter does the job.

The component.json file

Each Component module must provide a component.json file where you declare things like the name of the package, the version number, the software license, the files provided, and the possible dependencies. This is quite similar to the package.json file in NPM.

I’m using a very simplified version of NoFlo’s Graph class as the example here, so I can call the library bergie/graph. Like most JavaScript libraries, this will be under the MIT license.

{
  "name": "graph",
  "repo": "bergie/graph",
  "description": "Simple graph class",
  "license": "MIT",
  "version": "1.0.0",
  "scripts": [
    "graph.js"
  ],
  "dependencies": {
    "component/emitter": "*"
  }
}

For Node.js support you’ll also need a corresponding package.json file:

{
  "name": "graph",
  "description": "Simple graph class",
  "main": "./graph.js",
  "version": "1.0.0"
}

Once the dependencies are declared, run the installation:

$ component install

The example only uses Node.js’s built-in libraries, so NPM installation is not yet needed. If you add third-party libraries, you need to install them also:

$ npm install

Module code

Writing a Component module is very similar to writing Node.js modules. Create the file we just declared in the JSON file and open it in your favourite editor.

Since we’re aiming for multi-platform code, the main difference is dealing with platform-specific differences. For example, the event emitter library in Node.js is called events, and the Component equivalent is called emitter.Luckily their APIs are exactly the same, so we only have to do loading conditionally:

var EventEmitter;
if (typeof process === 'object' && process.title === 'node') {
  // Node.js
  EventEmitter = require('events').EventEmitter;
} else {
  // Browser
  EventEmitter = require('emitter');
}

This way, we have the correct event emitter implementation available for our code. Now we just create a constructor function to inherit from that:

// The constructor, just call "super"
function Graph () {
  this.nodes = [];
  EventEmitter.call(this);
}

// Set up inheritance
Graph.prototype = Object.create(EventEmitter.prototype);

// Define methods
Graph.prototype.addNode = function (name) {
  this.nodes.push(name);
  this.emit('node', name);
};

Once the code is there, we need to expose it as a CommonJS module:

module.exports = Graph;

Running the module in Node.js

For Node, this is all we need to do to be able to use our Graph as a module:

// Include the module
var Graph = require('./graph');

// Instantiate
var g = new Graph();

// Hook into events
g.on('node', function (name) {
  console.log("Node added " + name);
});

// Call a method
g.addNode('Foo');

Running this should end up with message Node added Foo shown on the console.

Running the module in browser

To be able to run the module in the browser, we need to run Component’s build process.

$ component build

This will generate a JavaScript file build/build.js which provides CommonJS module loading support, and all the JavaScript code we’ve declared in the JSON file.

Now you can include that file in your HTML, and start using the Graph module:

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8">
    <script src="./build/build.js"></script>
    <script>
      var Graph = require('graph/graph.js');
      var g = new Graph();
      g.on('node', function (name) {
        alert("Node added " + name);
      });
      g.addNode('foo');
    </script>
  </head>
  <body>
  </body>
</html>

Conclusion

Component can be used for solving the JavaScript code-sharing problem. They let you build full applications out of smaller, reusable modules. And thanks to the CommonJS module specification, it is quite easy to write these modules in away which enables them to be used also under Node.js and other JavaScript runtimes.

This post, and the date picker tutorial should give you enough information needed for starting to decouple your front-end applications, and to participate in the emerging open source Component ecosystem.

Even with more than 800 components available, it is too early to declare Component the winner in the front-end dependency management space, but it is a well-designed system that works quite well.

I will be utilizing Component for some of my JavaScript projects in the future.

GObject Introspection and Node.js

henri.bergius@iki.fi (Henri Bergius) — Wed, 25 Jul 2012 00:00:00 +0000

Unfortunately I will not make it to GUADEC this year. However, here is something new for GNOME developers:

I wrote last year how GObject Introspection was coming to Node.js. Back then the API was still quite bad, and it was limited in what you could do with it. Since then things have moved forward quite a bit, and today Piotras released version 0.1.0 of node-gir.

This allows you to interface with any GObject Introspection capable libraries, including the ones you write yourself.

There are still some things to be done, especially with stability, but with node-gir Node.js essentially becomes a fully qualified GNOME development environment.

For example, here is a CoffeeScript port of the GNOME Guitar Tuner example. Still a bit crashy, but at least it shows how the node-gir API works:

#!/usr/bin/env coffee
# Guitar Tuner
# ------------
#
# This is a CoffeeScript and node-gir port of GNOME's guitar tuner example from
# <http://developer.gnome.org/gnome-devel-demos/unstable/guitar-tuner.js.html.en>
gir = require 'gir'
gtk = gir.load 'Gtk', '3.0'
gst = gir.load 'Gst', '0.10'

gtk.init 0
gst.init 0

guitarwindow = new gtk.Window
  type: gtk.WindowType.toplevel
  title: "Node.js Guitar Tuner"

guitarwindow.on 'destroy', ->
  gtk.mainQuit()
  process.exit()

guitar_box = new gtk.ButtonBox

playSound = (frequency) ->
  console.log frequency
  pipeline = new gst.Pipeline
    name: 'note'
  source = new gst.ElementFactory.make "audiotestsrc", "source"
  sink = new gst.ElementFactory.make "autoaudiosink", "output"
  source.set_property "freq", frequency
  pipeline.add source
  pipeline.add sink
  source.link sink
  pipeline.set_state gst.State.PLAYING

addButton = (tune, freq) ->
  button = new gtk.Button
    label: tune
  guitar_box.add button

  button.on 'clicked', ->
    playSound freq

tunes =
  E: 369.23
  A: 440
  D: 587.33
  G: 783.99
  B: 987.77
  e: 1318.5

addButton tune, freq for tune, freq of tunes
guitarwindow.add guitar_box
guitar_box.show_all()

guitarwindow.show()

gtk.main()

People interested in this effort should watch Piotras’ node-gir repo. Now the project even has continuous integration on Travis using the Midgard GI bindings to test things.

Of course you can do other cool stuff, like creating a web browser with just couple of lines of JavaScript. Find more from the examples folder.

Running CoffeeScript on Microsoft Azure

henri.bergius@iki.fi (Henri Bergius) — Fri, 16 Dec 2011 00:00:00 +0000

It is quite easy to run CoffeeScript applications on Azure now that they support Node.js. You can mostly follow the steps of their Node.js and Express tutorial, with couple of modifications.

Add a package.json into your application root and depend on CoffeeScript and Express:

{

  "name": "myapp",

  "version": "0.0.1",

  "dependencies": {

    "coffee-script": "*",
    "express": "*"
  }

}

Then just have NPM install CoffeeScript by running in the Windows Azure PowerShell for Node.js:

npm install

Once this is done, you can decouple your actual application from the server.js that Azure uses to start your app. The app (in this case, app.coffee) itself could look like this:

http = require 'express'



exports.application = app = http.createServer()

app.get '/', (req, res) ->
  res.send "Hello, CoffeeScript on Azure!"

Note that the application doesn’t start a server, it just exposes it through exports. Starting can happen in server.js:

// Include the CoffeeScript interpreter so that .coffee files will work

var coffee = require('coffee-script');



// Include our application file

var app = require('./app.coffee');



// Start the server

app.application.listen(process.env.port);

And that is it! Now you can work with CoffeeScript just like you would work with any Node.js code on Azure. The only limitation is that if you want to include CoffeeScript files, you need to add the .coffee suffix to the module require call.

Here is how the deployed application looks (note that deployment can take a long time. My first deployment was about 30 minutes):

For data storage and queue handling, the Azure storage services are exposed through the azure module which is available via NPM or GitHub.

Using Azure workers

With cloud services one important part is to split your application into different functional modules. The web server front-ends should only handle HTTP requests and display data, and any heavier computing should be offloaded to separate workers.

Azure’s Queue feature provides a handy way of offloading tasks to be handled by worker nodes. For example, we could use a queue to send information on every request received on the web front-end so that workers can process it. For this, we need to install the azure module. Add this to the dependencies section of your application:

  "dependencies": {

    ...

    "azure": "*"

  }

Then let NPM install the module:

npm install

Now we can easily work with the Queue service. First initialize a ‘request’ queue in your app.coffee:

# Get the queue service
cloud = require 'azure'
queue = cloud.createQueueService()

# Ensure the queue exists
queue.createQueueIfNotExists 'request', (error) ->
  return unless error
  console.log "Failed to create queue", error

Then start sending information on new requests to the queue by creating a new Express middleware:

# Send URL of each request to queue
app.use (req, res, next) ->
  queue.createMessage 'request', req.url, (error) ->
    return unless error
    console.log "Error sending", error
  do next

Creating a worker

Then we just need to create a worker. It can be initialized using the a PowerShell command:

C:\node\testapp> Add-AzureNodeWorkerRole queue

Now go to the created queue directory, and add a package.json:

{
  "name": "test-queue",
  "version": "0.0.1",
  "dependencies": {
    "coffee-script": "*",
    "azure": "*"
  }
}

Again we’re going to make the actual heavy lifting in CoffeeScript, so change your worker’s server.js to say:

var coffee = require('coffee-script');
var worker = require('./worker.coffee');

Then edit worker.coffee and add the following:

cloud = require 'azure'
queue = cloud.createQueueService()

# Function that receives new requests from queue and outputs them
checkQueue = ->
  queue.getMessages 'request', (error, messages) ->
    return if error

    for message in messages
      console.log message

      # Destroy the message after we've caught it
      queue.deleteMessage 'request', message.messageid, message.popreceipt, (error) ->
        return if error
        console.log "Message #{message.messageid} deleted from queue"

# Poll queue every 0.2 secs
console.log "Setting up queue polling"
setInterval checkQueue, 200

If you deploy now, you should have both a worker and a web server available. In the Azure Emulator you can see all web requests getting logged by the worker.

About pricing

Azure has an interesting pricing model: every running instance costs, and then for storage you have to pay for both the amount of data used, and per transaction.

The polling worker alone, without any requests to the server, would cost 148.50$ per month. Add to that two web front-ends, and say 400 requests per day and you’d be at 208.63$ per month.

NoFlo - Managing workflows with JavaScript

henri.bergius@iki.fi (Henri Bergius) — Sat, 10 Dec 2011 00:00:00 +0000

My presentation from the Boston JS.Everywhere conference.

The slides are also available.

Node.js for Java developers

henri.bergius@iki.fi (Henri Bergius) — Tue, 06 Dec 2011 00:00:00 +0000

Node.js for Java developers