Celerity Progress Report - April 2023

[alexrp]

This is the first in a new series of progress reports. These will be posted with a monthly-ish cadence when there's notable progress made on the project.

As we're still in the early days of Celerity development, a lot of work is happening. This report will only go over the user-facing highlights.

Language Design

As we get closer to finalizing the language syntax and semantics for v1.0, a number of design changes have been made.

New Module Syntax

Modules used to look like this:

@doc "foo"
mod Foo;

fn main() {
    // ...
}

They now look like this:

@doc "foo"
mod {
    fn main() {
        // ...
    }
}

This change was made because writing the module name in the mod declaration didn't actually add any value. Additionally, it was confusing how attributes attached to the mod declaration could affect later declarations in the file, despite them not being lexically nested within the mod declaration.

any & unk Type Changes

A new unk type has been introduced. any and unk are defined as follows:

• any effectively turns off type checking in any operation it appears in. It's basically the "I know what I'm doing" escape hatch. The result of an operation on any is any. Any type can be assigned to any. any can be assigned to any other type - which is of course type-unsafe.
• unk is the type-safe counterpart to any. Very few operations are allowed on unk. unk can only be assigned to unk or any.

Additionally, these types are no longer part of the union type grammar, but are instead parsed as top-level types. This means that e.g. int or any is no longer a valid type. This change was made because such types make no sense anyway; any already implies int as a possibility.

agent Type Change

Previously, the type syntax for an agent without any messages specified was agent {}. This has now been simplified to just agent, which is more in line with how int and fn types are written.

This change also means that when the braces are present, at least one message must be specified.

Simplified Map Type

Map syntax has been greatly simplified. It is no longer possible to specify optional and required keys; instead, the syntax is now of the more typical #[key : value] form seen in other languages.

This change was motivated by the introduction of variable types to the language. Variable types made little sense in optional key/value position, for example.

Variable Types

Variable types can now be introduced in type syntax. This enables functions to be generic, as in:

fn map(array : [type source], mapper : fn((source) -> type result)) -> [result] {
    // ...
}

Conceptually, this is similar to a C# method declared like so:

static TResult[] Map<TSource, TResult>(TSource[] array, Func<TSource, TResult> mapper)
{
    // ...
}

A variable type can also be constrained. For example, type source : int or real behaves exactly like int or real but also introduces the source variable type for use in the rest of the signature.

Variable types only make sense when introduced in the parameter list of a function. The future type checker will complain if they're introduced anywhere else, e.g. in a return type annotation or in a type declaration.

Revised Error Handling Design

Error handling was previously done with a catch clause attached to a fallible call:

let result = work()? catch {
    err MyError {} -> // ...
};

Additionally, functions could be marked with err to indicate that they could raise errors:

fn work() err {
    // ...
}

This old model had some issues:

• It was impossible to catch an error raised locally with a raise expression.
• Handling the same errors from multiple calls was unwieldy and repetitive.
• The err annotation was optional, potentially misleading users.

To solve the first two issues, the old catch clause on fallible calls was removed, and a new try expression was introduced:

let result = try work()? catch {
    err MyError {} -> // ...
};

Crucially, the body of a try expression can be any expression - even a block. This makes it practical to handle similar errors for more than a single call. A try expression statically requires the body to contain at least one fallible call or raise expression.

To solve the third issue, the err annotation is now required on functions that have fallible calls or raise expressions which are not enclosed in a try expression. Also, the err keyword must now come before the fn keyword:

err fn work() {
    // ...
}

Finally, return type syntax can now specify type information for errors that can be raised:

err fn work() -> int raise err MyError {} {
    // ...
}

(The type on the right-hand side of the raise keyword could also refer to a union of potential error types by name.)

defer Statement Restrictions

defer statements no longer allow ret, next, or break expressions to occur within the body. This is in line with restrictions on finally blocks in various other programming languages, and prevents surprising control flow behavior.

rec with and err ... with Expressions

It's now possible to construct a record value from an existing record value using a with clause:

let two_dim = rec { x : 4, y : 8 };
let three_dim = rec with two_dim { z : 16 };

Conceptually, this works by first performing a shallow copy of the fields (and their mut status) from the source record to the result record, and then adding the new fields as specified, potentially overwriting any copied fields.

The same with clause is allowed for error values as well.

Metarecords

Metarecords were introduced as a lightweight variation on Lua's metatables with a more limited scope.

Conceptually, every value has a metarecord that can be accessed with the meta expression. For example, meta 42 returns the metarecord for integers. Metarecords will be used by the runtime when some kind of special behavior is requested for a value. For example, if you write for _ in collection { ... }, the runtime will call (meta collection).iterator(collection) to retrieve an iterator. Or, if you write left + right, the runtime will call (meta left).binary_operator("+", left, right). (Of course, in practice, these operations will be aggressively specialized, especially for primitive types.)

A custom metarecord can be specified when constructing a record, like so:

fn create_vector(x, y) {
    rec {
        x : x,
        y : y,
    } meta rec {
        binary_operator : fn(op, left, right) -> match op {
            "+" -> create_vector(left.x + right.x, left.y + right.y),
            // ...
        },
    };
}

(The metarecord can also be stored in a const declaration and be referred to by name. That approach is what we expect to become the norm.)

Record type syntax has similarly been expanded to allow a meta clause.

this Expression

A new this expression has been added to the language. This can be used within a lambda expression to get the function value of the enclosing lambda, thus simplifying authoring of recursive lambdas.

By design, a this expression can only refer to the innermost lambda expression.

assert Changed to Expression

The assert statement has been changed to an expression. The result of an assert expression is the value of the condition operand.

This change allows inserting assertions within complex expressions without having to wrap them in a block expression.

Simplified Attribute Syntax

The value portion of an attribute was previously optional and was specified with =:

// These were equivalent.
@foo
@foo = nil
fn bar() {
    // ...
}

Attributes now always require a value, and so the = has been removed from the syntax:

@foo nil
fn bar() {
    // ...
}

Semicolon Requirements

A terminating semicolon is now required for use, type, const, and ext fn declarations:

use Foo::Bar;
type array(t) = [t];
const x = 42;
ext fn external_helper(a, b, c);

For use, type, and const, this makes it clearer where the declaration actually ends. For ext fn, it also more explicitly indicates that the function has no body.

Raw String Literals

The language now has support for two kinds of raw string literals: Verbatim strings (single-line) and block strings (multi-line). Both forms are enclosed by 3 or more double quotes, allowing the user to select the amount of quotes needed to avoid clashes with quotes in the contained text. Neither form can contain escape sequences.

// Verbatim string literal:
let path = """C:\Windows\System32""";

// Block string literal:
let code =
    """
    while true {
        work();
    };
    """;

A verbatim string literal results in the exact text within the quotes.

A block string literal works slightly differently: Every line of text within the quotes will have indentation stripped up to the point of the closing quotes, and an LF will be appended regardless of the line ending format used in the Celerity source file.

Block string literals allow an optional language indicator:

let code =
    celerity"""
    while true {
        work();
    };
    """;

This can be used by sophisticated text editors to perform syntax highlighting for embedded languages.

More String Escape Sequences

Regular string literals have gained support for the following escape sequences:

• \a (BEL)
• \b (BS)
• \e (ESC)
• \f (FF)
• \v (VT)

These are typically seen in other programming languages. Supporting them should make the experience smoother for new users.

Blank Characters

The language now accepts a broader range of blank characters:

• White space is defined as any character in Unicode category Zs, as well as HT, VT, FF, and SP.
• A line break is defined as any character in Unicode categories Zl and Zp, as well as LF, CR, and NEL.

Note that the future code formatter will normalize all of these to CR/LF and SP. The main motivation for this change is to make the experience smoother for new users.

Tooling and Editor Support

We're aiming to have good tooling support for Celerity from day one.

Project System and Workspaces

A basic project system has been implemented based on the design described here:

• Support a celerity.json project configuration file #30

For now, this is a very simplistic project model that doesn't deal with issues of package management at all. Celerity aims to follow the "batteries included" model of Python, with a rich standard library that covers the vast majority of needs users might have. This means that we don't anticipate seeing the crazy package sprawl that some other ecosystems struggle with (e.g. Node.js). So, we expect Git submodules to serve most users well enough.

Tooling such as code formatters and language servers need to know what files they should work on. The term typically used for this is 'workspace'. Celerity now has APIs for creating, watching, and manipulating workspaces. The two main types of workspaces are:

• ProjectWorkspace: Bases its analysis configuration on a celerity.json file that must be present in the workspace directory.
• SimpleWorkspace: Can open any directory as a workspace and picks reasonable defaults for analysis configuration.

The CLI driver has already switched to using workspaces whenever applicable. The upcoming language server will also use them.

Visual Studio Code Extension

Editing Celerity code should be a good experience out of the box. We're aiming to support a range of editors with official language extensions included with Celerity. The first editor to be supported is Visual Studio Code, for which we've written a basic language extension. It currently provides syntax highlighting, code snippets, and celerity-json-based tasks.

The upcoming language server will also add support for semantic highlighting, real-time diagnostics, linting, and limited code navigation and completion. In the long term, support will also be added for document formatting, debugging, and testing.

Future Plans

The following issues are immediate priorities:

• Make workspace APIs more flexible #78
• Introduce a concept of project solutions #82
• Basic LSP implementation #62
• Design and implement the shared linear IRs (HIR, MIR, LIR) #60

The first three will provide a great initial editor experience. I consider this to be a top priority going into standard library development.

The fourth issue is the beginning of the effort to write the language runtime system. In my view, this is where most of the truly interesting engineering will begin.