IO, coroutines, generators

Radian’s functional semantics make synchronous I/O operations difficult, perhaps impossible; I certainly haven’t figured out how one would implement them. Instead, all I/O operations are performed asynchronously. A program’s job is not to perform I/O actions, but to describe them. You could think of a program as a function which returns a list of changes the external world ought to apply to itself. Each change has a continuation: another program to run once the change has been applied.

This lets you construct chains of I/O actions: read the contents of a directory, open all the files in it, transform the contents of those files, write them back. Each step is a little program which runs as the callback for the previous action, and then returns the next action.

The scheme works well in a mathematical sense but it’s a right nuisance in actual practice. Javascript programmers will be familiar with the issue: you end up with callback nested in callback nested in callback, and you can’t use the language’s built in flow control structures. You have to decompose your loops and conditionals into groups of functions by hand, and the result is hard to read.

The whole point of the Radian project is that the compiler should perform this kind of mechanical, predictable translation for you.You should be able to write straightforward straight-line code that expresses the algorithm you want to execute, and let the compiler figure out how to decompose that into a set of interrelated callback functions.

Let’s start by describing the IO process as an interface. It looks a lot like a sequence: a sequence contains a chain of iterators, where each iterator has a current value and a method letting you move on to the next iterator. An IO task contains a chain of sub-tasks, where each one returns an IO action to perform, and offers a method letting you move on to the next sub-task. The chief difference is that the “next” method needs some value as input, that being whatever came back from the IO action. It might be the contents of a file, the state of a socket, or an error condition flag, but whatever it is, it determines what the next IO subtask is going to do.

I’ll call this thing a “task”, and the subtasks I’ll call “steps”. Task will have a “run” method, which returns the first step; this is analogous to the sequence’s “iterate” method. Each step will have a “current” function, returning some IO action, and a “next” method which accepts the result of that action. Finally, a “valid?” method will return true when the IO task is complete; the terminal step has no “current” or “next”.

It is obviously be possible to use the existing object system to construct such a task, but that doesn’t solve the problem of having to decompose all your flow control by hand. The language needs a new bit of syntax: a task block, which defines a new task, and a yield operation, which marks the divisions between steps. The yield operation is like a function which accepts a parameter – that’s the IO action returned by “current” – and yields a value – which is of course the parameter the caller will pass in to “next”. It’s a doubly apt name: seen as a generator, the yield operator yields the generator’s output; seen as a thread, or coroutine, the yield operator yields control back to some other process.

Once this structure exists, I plan to redefine the IO system in terms of the “task” and “step” interfaces. The program itself will become one big task block, broken down into steps between each “yield” operation. The task syntax block will let you define these asynchronous state-machine tasks in the same way you can currently define ordinary functions and methods, and you’ll be able to create complex sequential IO operations by composing task instances in much the same way that ordinary functions can call each other.

This will completely change Radian IO syntax, of course. Right now one uses the ‘io’ object to create sequences of actions: it notionally represents the state of the world, and you wrap it in a series of actions to transform it. Once you return the resulting, modified state-of-the-world, that commits your changes to reality, and a new state-of-the-world (still called ‘io’) gets passed in to the next continuation function.

In the new system, there’ll be no need to represent the state of the world explicitly. IO actions will be simple objects; you won’t need to have a world-token in order to create them. It will be much more practical to have modules that manage specific types of IO, and return specific types of IO actions. Instead of making calls on the “io” object, you’ll yield things from function calls into various modules. For example, the hello world program will change from this:
io->print("Hello, world!")
to this:
yield console.print("Hello, world!")

This does mean that “yield” and the task system will have to be one of the first things people learn about Radian, but I suppose that’s worth the vast increase in ease of IO programming.

This scheme also opens the door for a lot of useful concurrency features. Tasks are basically coroutines; they’re like lightweight, cooperative threads which don’t need permanently-allocated stacks. You could have thousands of them running at once, Erlang-style.