Literate programming with PHP

15 January 2011
7 minute read

noweb.php is a PHP implementation of the tool needed for literate programming. Wikipedia says the following about literate programming:

The literate programming paradigm, as conceived by Knuth, represents a move away from writing programs in the manner and order imposed by the computer, and instead enables programmers to develop programs in the order demanded by the logic and flow of their thoughts. Literate programs are written as an uninterrupted exposition of logic in an ordinary human language, much like the text of an essay, in which macros which hide abstractions and traditional source code are included. Literate programming tools are used to obtain two representations from a literate source file: one suitable for further compilation or execution by a computer, the “tangled” code, and another for viewing as formatted documentation, which is said to be “woven” from the literate source. While the first generation of literate programming tools were computer language-specific, the later ones are language-agnostic and exist above the programming languages.

noweb.php is able to facilitate this model of working by being able to extract program code from textual documents describing how the program ought to work. This document itself is such a description, and noweb.php PHP code can be generated from it.

The inspiration for creating noweb.php comes from Jonathan Aquino’s work on implementing the same with Python. See his world’s first executable blog post and the actual noweb.py project on GitHub.

Download

If you’re interested in doing literate programming with PHP, grab the software produced by this document from GitHub:

https://github.com/bergie/noweb.php

noweb.php requires PHP 5.3 or newer.

Usage

noweb.php is a PHP tool that reads a text file with noweb-style annotated software code macros in it, parses it and writes the files defined in the document into the file system.

$ noweb.php tangle README.txt

The resulting code files will be written into the same directory where the document resides.

If you just want to see what code files the document defines, you can also run:

$ noweb.php list README.txt

Getting a file

When noweb.php starts, we check for the command line arguments to get a file. If no argument is found, we abort and give users instructions:

if (basename($_SERVER['argv'][0]) == 'php')
{
    // The script was run via $ php noweb.php, tune arguments
    array_shift($_SERVER['argv']);
}
if (count($_SERVER['argv']) != 3)
{
    die("Usage: noweb.php tangle <textfile>\n");
}

We get the command and filename from the arguments:

$command = $_SERVER['argv'][1];
$readfile = $_SERVER['argv'][2];

The we check that the given file actually exists:

if (!file_exists($readfile))
{
    die("File {$readfile} not found\n");
}

And then we parse the file for any literate programming code, and extract the code. The workings of this is explained in detail later.

$noweb = new noweb();
$noweb->read_chunks($readfile);

We check the command given by the user and perform it:

switch ($command)
{
    case 'list':
        $noweb->list_files();
        break;
    case 'tangle':
        $noweb->tangle_files(dirname($readfile));
        break;
    case 'weave':
        $noweb->weave($readfile);
        break;
    default:
        die("Unknown command {$command}. Try 'tangle'\n");
}

Reading the file

In a literate program the actual software code is stored in chunks inside the document. A chunk is defined by a name inside << and >>=, and it ends on a line containing just @.

For parsing the chunks we will need two regular expressions:

private $chunk_start_regexp = '/^<<([^>]+)>>=/';
private $chunk_end_regexp = '/^@$/';

When reading a literate programming file, we handle it on line-by-line basis, keeping track of whether we’re inside a chunk or documentation:

$lines = file($filename);
$chunk = null;
foreach ($lines as $line)
{
    <<reading line of the file>>
}

With each line of the document we first check if starts a chunk:

$matches = array();
if (preg_match($this->chunk_start_regexp, $line, $matches))
{
    // Entering a chunk
    $chunk = $matches[1];
    if (!isset($this->chunks[$chunk]))
    {
        $this->chunks[$chunk] = '';
    }
    <<markdownizing chunk start>>
    continue;
}

If this line did not start a chunk, and we’re not inside a chunk we can ignore the line as it is documentation, not source code:

if (is_null($chunk))
{
    <<markdownizing documentation line>>
    continue;
}

If this check passes we’re inside a chunk of code. Therefore we need to check whether the line ends the chunk:

if (preg_match($this->chunk_end_regexp, $line))
{
    // Exiting a chunk
    $chunk = null;
    <<markdownizing chunk end>>
    continue;
}

If the line didn’t end the chunk then we append it to the chunk:

$this->chunks[$chunk] .= $line;
<<markdownizing chunk line>>

Expanding chunk macros

It is possible to include contents of another chunk inside a chunk by calling it with its name. This is done using the <<chunkname>> syntax.

We need a regular expression for matching this:

private $chunk_include_regexp = '/(\s*)<<([^>]+)>>/';

For expanding the macros we run the contents of each chunk through the regular expression. To ensure that line indentation works correctly we will also pass the whitespace that we found from a line before the chunk inclusion macro, and also expand recursively:

$noweb = $this;
$content = preg_replace_callback
(
    $this->chunk_include_regexp,
    function ($matches) use ($noweb)
    {
        // We want to indent the chunk with whitespace found
        $indent = substr($matches[1], 1);

        // Expand chunk inclusion request using given indent
        return "\n" . $noweb->expand_chunk($matches[2], $indent);
    },
    $this->chunks[$name]
);

The actual indentation happens after any possible chunk inclusions have been processed. Finally we trim newlines from the end and return the result:

$indented = '';
foreach (preg_split("/(\r?\n)/", $content) as $line)
{
    $indented .= "{$indent}{$line}\n";
}
return rtrim($indented, " \n");

Checking if a code chunk is a file

We look at the names of code chunks to check if they should be written to a file. Code chunks with slashes (/) or dots (.) are treated as files.

if (   strpos($name, '.') !== false
    || strpos($name, '/') !== false)
{
    return true;
}
return false;

Listing code files

If user is interested in what files have been defined in the document we can list them.

foreach ($this->chunks as $name => $chunk)
{
    if (!$this->is_chunk_file($name))
    {
        continue;
    }
    echo "{$name}\n";
}

Writing the code files

After we have found and expanded all chunks from the document we will look for chunks that have names mapping them to a file:

foreach ($this->chunks as $name => $chunk)
{
    if (!$this->is_chunk_file($name))
    {
        continue;
    }
    <<writing code file from chunk>>
}

Once we have found filenames from the chunks, we will create the necessary directories for them:

if (substr($target_dir, -1) != '/')
{
    $target_dir .= '/';
}
$folder = $target_dir . dirname($name);
if (!file_exists($folder))
{
    if (!mkdir($folder, 0777, true))
    {
        die("Failed to create folder {$folder}\n");
    }
}

And then we can just write the chunk to a file:

$file = $target_dir . $name;
if (!file_put_contents($file, $this->expand_chunk($name)))
{
    die("Failed to write to {$file}\n");
}

Writing documentation in HTML

As the format supported by noweb.php is almost-but-not-quite valid Markdown, the tool has a weave command for converting the document to valid Markdown.

For this we need to convert chunk starts to HTML so that they will be valid:

$this->markdown .= "<pre id=\"{$chunk}\" title=\"{$chunk}\">\n";

Since chunks are shown via HTML we need to escape them:

$this->markdown .= htmlentities($line);

When chunk ends we close the HTML element:

$this->markdown .= "</pre>\n";

Regular documentation lines don’t need any special treatment:

$this->markdown .= $line;

The actual weave command writes the Markdown to a `.markdown´ file. First we check that the original documentation file doesn’t have the same name:

$fileinfo = pathinfo($file);
$new_extension = 'markdown';
if ($fileinfo['extension'] == 'markdown')
{
    $new_extension = 'markdown.generated';
}

Then we write the file:

$new_filename = dirname($file) . "/{$fileinfo['filename']}.{$new_extension}";
file_put_contents($new_filename, $this->markdown);

Appendix 1: Generating the script

To generate noweb.php from this document, you first need a tool to extract the code from it. The easiest way to do that is by fetching the pre-generated noweb.php from GitHub.

Then you can generate noweb.php from README.txt as follows:

$ noweb.php README.txt

Appendix 2: Summary of the program

#!/usr/bin/php
<?php
<<license information>>
class noweb
{
    <<chunk regular expressions>>
    public $chunks = array();
    public $markdown = '';

    public function read_chunks($filename)
    {
        <<reading the file>>
    }

    public function expand_chunk($name, $indent = '')
    {
        <<expanding chunk macros>>
    }

    public function is_chunk_file($name)
    {
        <<checking for files>>
    }

    public function list_files()
    {
        <<listing code files>>
    }

    public function tangle_files($target_dir)
    {
        <<writing code files>>
    }

    public function weave($file)
    {
        <<writing documentation file>>
    }
}
<<getting the file>>
<<running the command>>

Appendix 3: License and contributing

noweb.php is free software produced by Henri Bergius. The project is managed on GitHub where you can send pull requests or file issues.

/**
 * @package noweb.php
 * @author Henri Bergius, http://bergie.iki.fi
 * @copyright Henri Bergius, http://bergie.iki.fi
 * @license http://www.gnu.org/licenses/lgpl.html GNU Lesser General Public License
 */

Continue reading

cover image for DNode: Make PHP and Node.js talk to each other

DNode: Make PHP and Node.js talk to each other

31 October 2011. 5 minute read.
If you’ve been following my blog, you might have noticed that lately I’ve started doing quite a lot of Node.js development alongside PHP. Based on conversations I’ve had in various conferences, I’m by far not alone in this situation - using Node.js for real-time functionality, and PHP (or Django, or Rails) for the more traditional CRUD stuff.
cover image for My secret agenda for PHP Content Management Systems

My secret agenda for PHP Content Management Systems

08 July 2011. 5 minute read.
As I’ve written before, I’m concerned about the state of the PHP ecosystem. There are lots of good applications written in the language, but there is very little code sharing between different projects, mainly because of framework incompatibilities, but also because of quite a strong NIH culture.

Read more Midgard posts.