In the meantime, Google’s Dart language came along with a clever little twist that offers exactly what I need. In a nutshell, identifiers which begin with an underscore are private to the module that declared them: the compiler mangles the strings so that two modules both referring to _foo will actually end up using ABC_foo and XYZ_foo.

This solution neatly sidesteps much of the complexity that typically comes along with scope control by shifting its attention from the class level up to the module level. Instead of managing fine-grained scope control with public/private/protected and friend declarations, you just put related objects into a file together and separate unrelated objects into different files. There’s no need for any extra keywords, and it grounds the scoping system in something very concrete. It also aligns well with the compiler’s optimization needs, since compilation units are equivalent to scope boundaries.

The semantics I’ve chosen are based on symbols, not just on identifiers: that is, the compiler must apply name-mangling whenever it generates any symbol value, whether it came from some declaration or from a symbol literal. This means it’s possible to insert private keys into a map shared across modules, for example, or to return unique private cookie values. Other modules can certainly pass these private symbol values around, but there is no way for any other module to generate one.

My latest step is that one can now assign a new value from inside an if-block to a variable which was defined outside it, even when that if-block happens to live inside a generator. That is, if-then blocks inside generators now behave exactly the same as if-then blocks inside any other kind of function. (This completes the project I mentioned on the 17th.)

The next step in the generator project is to reimplement the while loop. You can put a while loop inside a generator right now, but you can’t yield from it; allowing you to yield from the loop means I need to provide a way to pause a loop and then resume it later, from any of a number of possible yield statements.

So why is all this so difficult? There is a fairly well-understood strategy for implementing asynchronous generators in an imperative, object-oriented language. You turn the whole generator into a state machine: you create a class, hoist all the local variables up as members in that class, and give the object a “state” member which tells you which yield statement executed last. “Yield” sets the state member and then returns a value. On the next call to the function, a big switch statement picks out the appropriate case branch, and you pick up where you left off. It’s not by any means trivial, but a number of languages have done this and the technique is well understood.

Asynchronous generators, in a functional language, are a matter of continuations and lazy evaluation. Again, it’s not a trivial thing, but you don’t have to worry about object mutation or context assignment, so once you have your dataflow graph you can just pick the node that returns the value and wrap it with all of its dependencies into a function, and there’s your continuation. This is, again, well understood, and has been done in many languages.

The problem with doing this in Radian is that it combines the difficulties of both approaches. It looks like an object-oriented language with mutable state, but its internal semantics are based on pure functions and immutable data. I can’t use the typical object-oriented strategy because the semantics offer no way to represent a ‘goto’, and therefore I can’t make a single function with multiple entrypoints. Nor is there any sensible way to add such a ‘goto’ given the existing constraints on the design. The generator has to be decomposed into as many separate fragments as there are yield statements, functional-style. That’s fine, but Radian lets you do crazy things like “assign a value to a variable”, which means that this function decomposer has to keep careful track of data flow across multiple invocations of multiple functions so that it can make sure that the right values show up in the right places with the right names. It makes my head hurt.

The current problem is finding a way of passing values computed in a branch of an if-block, which has been nested inside an async block, on to the asynchronous segment following the end of the if-block.

This design does complicate the actual language semantics, since the if, for, and while loops have two completely different implementations depending on where you use them, but the result, as perceived by the human programmer, should be comparable. That is, by reusing the same syntax for complementary but separate semantics, the async block scheme should be much easier to learn.

Constants bug me because they aren’t really constant in the sense I am used to from C++. The definition can’t be changed, but its value certainly may, when it is defined in terms of variables. What’s more, the difference between a constant and a one-line function without parameters is so slight that I can’t think of any way you could tell the difference in code.

The change I’m pondering would be to ditch const entirely, then split the one-line function form apart from the block form. The new one-line simple definition statement would use the keyword def, borrowed of course from Python and Ruby. The zero-parameter form of this statement would be equivalent to the current const and to the current single-line parameterless function.

This change would regularize the syntax somewhat, in that each leading keyword would then unambiguously introduce either a single-line statement or a multi-line block. I haven’t been using const much because it often feels wrong, especially as an object member, where the “constant” is often defined in terms of variables; I think I might use def somewhat more.

The downside is that programmers familiar with Python or Ruby might find it jarring to find their familiar def keyword set to a similar but rather more restricted use.

An identical feature showed up in Python a couple of weeks ago, though that language has offered the simple yield statement for a decade now. I chose to head straight for the enhanced form of yield because I intend to use it as a foundation for the rest of the generator feature: any control-flow block nested in a generator will itself become a generator, and its container will implicitly yield from this nested block.

• Removed debug spew when inserting or doing lookups on a map object.
• String literal comparison no longer returns spurious “false” results when comparing two equal strings. This also caused map lookups to fail.

• A string is a sequence of characters, not a sequence of bytes. When iterating over the characters in a string literal, the contents are no longer returned as individual UTF-8 bytes, but as Unicode code points.
• Symbol lookup time is now O(log n) in the worst case, not O(n), since the index is now an AA-tree rather than an unbalanced binary tree.
• A race condition in the symbol allocator has been corrected; it is no longer possible to get duplicate symbols when the first lookup happens simultaneously on different processors.
• io.print function emits UTF-8 byte streams instead of mangling non-ASCII characters.
• The identifiers task, generator, and yield have been reserved for use as syntax keywords.
• String literals implement a compare_to method, which allows them to be compared and used as keys in a map container. This method uses a simple ordinal comparison, not one of the specific Unicode-defined lexical comparisons.
• Import statement allows an optional aliasing clause, where the local symbol has a different name than the symbol defined in the target module. The complete syntax is:
  'import' identifier ['=' identifier] ['from' expression]
  As with all declarations, the first identifier token is the declared name; the optional second identifier specifies the target to import.
• Expressions can now be split and continued on the next line after any binop token, including the comma and period tokens. Comments between a binop token and the end of line will be ignored.
• Tuples implement the sequence interface.
• Map constructor no longer tries to double up as a set constructor. Entries must have both a key and a value.
• Unary operators (negate, not, poison, and yield) no longer parse an entire expression instead of a single term. This caused baffling precedence errors.
• Numerics package supports rational as well as integer arithmetic. Division of two integers will now yield a rational number instead of truncating the remainder and returning an integer. Rational numbers are represented as a pair of integers, numerator and denominator. Rational numbers are normalized such that the numerator and denominator have no common factors and the denominator is always positive and greater than 1.
  Number type has two new predicates: rational?, which is currently true for all numbers, and integer?, which is true only for integral values. The shifting and bitwise arithmetic operations are only defined for integers, not for all rationals.
• Functions in the number module allow rounding of rational numbers in three modes: ceiling rounds toward positive infinity, floor rounds toward negative infinity, and truncate rounds toward zero.
• Sign function in the number module returns -1 for negative numbers, 1 for positive numbers, and 0 for all numbers which are equal to zero.
• New generator block type lets you define a function which returns a
  sequence. The generator supplies each value in the sequence by executing a yield statement. This is not very useful, as you cannot put yield statements inside nested loops or conditionals, but it is a sign of things to come.