Abilities: a New Way to Inject Behavior and State

Unison’s Abilities may represent the future of designing programs, letting you inject behavior and state into your pure functional code.

Any of these sound familiar?

• I hate having to pass values for things like database connections, logger instances, or configuration through umpteen levels of code just so the lower-levels can use them.

• I wish I could separate the way my data is used from the way it is created.

• I want to write pure, functional code, but that means I have to manage state in monads. I’ve tried, I’ve really tried, but I still don’t find monads intuitive.

That’s my personal list, but I suspect I’m not alone.

Good news! There’s hope on the horizon, in the form of Unison’s Abilities.

Just to be clear

Unison didn’t invent this idea. Abilities are an implementration of Algebraic Effects. Algebraic effects were firat discussed in the early 2000’s. Unison’s implement is based on work done for the Frank language in 2017, described in the wonderfully titled paper Do Be Do Be Do.

What Are Abilities?

At its most basic, an Ability has three parts:

1. A type definition that defines a set of (slightly special) functions.

2. One or more implementations that handle calls to these functions.

3. A handler that orchestrates how code using the ability calls back and forth with the implementations.

For example, maybe we want to add logging capabilities to our application.

• The type definition would define an ability called Logger. This ability includes th type signature of a log function.

• We might write an implementation of logger called ConsoleLogger that logs messages to the terminal, and NetworkLogger that logs to some remote endpoint.

We then write application code that uses the log function. As it stands, that wouldn’t work, because we haven’t bound that code to a particular implementation. That’s where we’d use the third component of abilities: a handler:

handle my_top_level_function with console_logger

This handle expression does something quite magical: it makes the console implementation of the log function available not just to the code in my_top_level_function, but also to all the code invoked below that function. The code in your app can just call log without knowing what kind of logger it is, and without having to pass some kind of logger instance down the call stack. This is an example of dynamic scoping.¹

Handlers Give Us Dependency Injection

We have client function that use an ability, and we have the potential to write multiple implementations of that ability. We saw this in the logger example. This means we can easily change which implementation our code uses:

handle my_top_level_function with console_logger

-- or

handle my_top_level_function with network_logger

The logger is selected with no change to the code that uses the log function.

Or, we could have an ability representing our app’s configuration. Changing from development to test to production is as simple as

handle my_app with dev_config
handle my_app with test_config
handle my_app with prod_config

Or, with a helper function or two:

config dev my_app

Injecting Down the Call Stack

The functions defined for an ability are available to any code executed during the execution of the wrapped function.

It’s Also About State

Having got this far, you might be thinking that the handler just arranges for the functions in your ability implementations to be called from your regular code. But it has one more trick up its sleeve.

When you call handle to associate an ability with a function, you can pass it some initial state. The handler then passes this state into the implementation functions along with any parameters passed by the client code. When the implementation function returns, it does so by calling handle again, which means it can pass in updated state.

We’ll look into this in more detail when we disect how abilities work.

Show Me The Code

Enough with the words: let’s make all this a little more concrete. We’ll write a simple ability that acts as a counter. We’ll be imaginative and call this ability Counter.

Counter implements a single function nextValue that returns an incrementing value each time it is called. To to this, it also maintains a state: the next number to be returned.

Get Yourself Set Up

As we did in the previous episode, we’ll run ucm in one window and an editor in another window, both started in the same directory. In the code fragments that follow, I’ll indicate which window we’re it at the top.

Start by opening ucm, creating a namespace, and adding the base library to it.

UCM

.> cd pragdave.abilities.counter
  ☝️  The namespace .pragdave.abilities.counter is empty.

.pragdave.abilities.counter> fork .base lib.base
  Done.

Declare the Counter Ability

Now we’ll open the editor in our second window, and add the declaration for our Counter ability.

Unison

unique ability Counter
  where
    nextValue: Nat

All three lines are part of the same expression. The first line names our ability. The actual interface provided by the ability follows the where keyword. In our case there’s just the one, nextValue.

The declaration here tells us that nextValue is used just like any other value, and its type is Nat.

This syntax is actually shorthand. You’re more likely to see it written (both by people and by ucm) as:

nextValue: {Counter} Nat

Ability names always appear inside braces in typ declarations. Here it’s just saying what we already know: when we use nextValue, we’re doing so in the context of the Counter ability.

Save the file in your editor, and you should see ucm spring to life:

UCM

⍟ These new definitions are ok to `add`:
   
   unique ability Counter

Create an Implementation

The implementation of an abiliy is just a function. It takes two parameters. The first is the current value of the state. The second represents a request from the client code. We’ll use pattern matching to decide what to do with this request:

Unison

implementCounter value request = 
  match request with
    { nextValue -> resume }
        -> handle resume value with implementCounter (value + 1)
    { result } 
        -> result

If the client code references nextValue, the first of the match patterns will fire. Let’s look at it more closely:

Unison

{ nextValue -> resume }

Notice the braces here: this is a special pattern match used just for ability requests.

This pattern matches an access of nextValue (which appears before the arrow in the pattern) and a second value, which I call resume. This value is a function, but what it does is almost the opposite of a function call. Rather than creating a stack frame and invoking a function, resume actually pops a stack frame, returning to the place where the client used nextValue. The parameter passed to resume will end up being the value the client sees for nextValue.²

If that pattern matches, we use our old friend handle to invoke some code in the context of an ability.

Unison

    { nextValue -> resume }
        -> handle resume value with implementCounter (value + 1)

The code we invoke is resume value: we call the continuation so that the client code gets control back, and the nextValue call returns the value we pass in. The ability is the thing after the with keyword: it just invokes the Counter implementation recursively, passing in a new state of value+1. Let’s have a look at the whole implementation again:

Unison

implementCounter value request = 
  match request with
    { nextValue -> resume }
        -> handle resume value with implementCounter (value + 1)
    { result } 
        -> result

See how value is passed as the state to implementCounter, and then is incremented before passing it in the recursive call. It’s a classic example of using parameters in a recursive pure function to emulate changing state. It’s really just a fancy reducer.

So that just leaves the second pattern match: {result} -> result. What it does is straightforward: it simply returns what gets passed in.

It’s there because our implementation needs to be able to handle our client code finishing. Remember that the ability implementation actually runs the client function. When that function returns, the ability has to forward its return value back up to whatever called us. That’s what this pattern match does: it simply makes the wrapped client function’s result the ability’s result.

Before moving on, save the editor buffer:

UCM

⍟ These new definitions are ok to `add`:
  implementCounter : Nat -> Request {g, Counter} x -> x

(We’ll talk about that g that snuck into the ability in a minute.)

Use The Ability

Here’s a function that uses the Counter ability to return a list containing three consecutive values.

unison

nextThree = '[ nextValue, nextValue, nextValue ]

OK, so that’s a little strange. What’s with the '?

If we’d just written

nextThree = [ nextValue, nextValue, nextValue ]

then we’d just be binding the expression on the right to the name on the left. It would be done once, and immediately. But that would be too soon, because we haven’t yet associated our nextThree function with an ability: it has no idea what nextValue should do.

So, rather than binding [ nextValue, ...] to nextThree, we instead turn the right hand side into a zero arity function. A what? Fortunately, there’s a sidebar for that.

Thunks, Deferred, Zero-Arity Functions

One of the tenets of functional programming is that, given a particular parameter, a function will always return the same value: inc 2 will always return 3.

This means that a function with no parameters will only have one return value: it’s no different to that value. So there’s no need for syntax to let us create such a function. Instead, answer = 42 just binds 42 to fred.

As a result, most functional languages do not have the concept of functions with no parameters: it’s the presence of a parameter to the left of the equzls sign that makes something a function.

As you’ll see, though, when we write abilities we need to be able to pass around chunks of code that can be executed later: deferred functions with no parameters. So Unison introduces a special syntax.

A single quote followed by an expression creates a value that is a zero-arity function whose body is that expression. You evaluate such a value by putting an exclamation mark in front of it.

Unison

answer = '(40 + 2)    -- answer is a zero-valued function
> answer              -- reports '(40 Nat.+ 2)
> !answer             -- reports 42

You’ll see this zero-arity functions called both thunks and deferred functions in the Unison documentation (although I don’t like the latter: it’s a deferred expression, not function, but…)

On with the show.

So we defined nextThree as a zero-arity function using

unison

nextThree = '[ nextValue, nextValue, nextValue ]

What happens if we try to run it?

unison

> nextThree

ucm reports:

UCM

15 | > nextThree
           ⧩
           '[nextValue, nextValue, nextValue]

That’s pretty smart. The value associated with nextThree is a zero-arity function, so it just shows that value.

If we want to evaluate it, we need to turn this into a function call. That’s where the exclamation point comes in:

unison

> !nextThree

This time, ucm flags an error:

UCM

The expression needs the {Counter} ability, but this location does not 
have access to any abilities.
  
     15 | > !nextThree
             ^^^^^^^^^

It noticed we were using nextValue, but the only place that nextValue is defined is in the declaration of the Counter ability. In order to run nextThree, we need to wrap the call to it inside the ability. And this is the point where we say which particular implementation of Counter we want to apply, and what initial state to pass it.

unison

> handle !nextThree with implementCounter 0

Et voilà. ucm shows us our three element result:

    ⍟ These new definitions are ok to `add`:
    
      unique ability Counter
      implementCounter : Nat -> Request {g, Counter} x -> x
      nextThree        : '{Counter} [Nat]
  
  Now evaluating any watch expressions (lines starting with `>`)...

    15 | > handle !nextThree with implementCounter 0
           ⧩
           [0, 1, 2]

If we change the initial state, the list will reflect the new starting value.

UCM

    15 | > handle !nextThree with implementCounter 97
           ⧩
           [97, 98, 99]

Our Code So Far

Unison

unique ability Counter
  where
    nextValue: Nat

implementCounter value request = 
  match request with
    { result } 
        -> result
    { nextValue -> resume }
        -> handle resume value with implementCounter (value + 1)

nextThree _ = 
  '[ nextValue, nextValue, nextValue ]

What About the Scoping?

I previously said that if a function is wrapped in an ability handler, then that handler is automatically made available to any other functions called while the wrapped function executes. Let’s try it.

Unison

nestedThree = 
  '[ nextValue, nestedNextValue "dummyParam", doubleNestedNextValue "dummyParam" ]

nestedNextValue _ = 
  nextValue

doubleNestedNextValue param
  = twoLevelsDeep param

twoLevelsDeep _ = 
  nextValue

> handle !nestedThree with implementCounter 10

and ucm reports

UCM

  doubleNestedNextValue : param ->{Counter} Nat
  nestedNextValue       : ∀ _. _ ->{Counter} Nat
  nestedThree           : '{Counter} [Nat]
  nextThree             : '{Counter} [Nat]
  twoLevelsDeep         : ∀ _. _ ->{Counter} Nat  nestedThree           : '{Counter} [Nat]

29 | > handle !nestedThree() with implementCounter 10
           ⧩
           [10, 11, 12]

It worked. And notice how every function has inheritied the need to be wrapped by Counter. That includes doubleNestedNextValue, which doesn’t directly use the ability.

You might be wondering about all the `“dummyParam” stuff in this code. It’s there because I needed each of the child functions to be a real function, which means they each have to take a parameter, which I then ignore.

Abstract Away the Ability

So far we’ve explicitly wrapped the client code in an ability implementation.

Unison

> handle nextThree! with implementCounter 97

But it’s all just code, so there’s nothing stopping us from adding some helper functions to make this read a little better.

Our first attempt looks reasonable, but there’s a compilation error:

Unison

countingFrom initialValue code =
  handle !code with implementCounter initialValue

> countingFrom 0 nextThree
  ^^^^^^^^^^^^
--  The expression needs the {Counter} ability, but this location does 
--  not have access to any abilities.

This is the kind of error that you either suss immediately or grind through a couple of hours figuring out.

It turns out that, so far, we’ve been able to rely on Unison’s incredible type inference: we haven’t written a single type signature, and it has derived the types of everything perfectly.

In this case, though, it can’t. Let’s have a look at the actual type it inferred for countingFrom.

countingFrom : Nat -> '{g} t ->{g} t

The first parameter, the initial value, is a natural number. The second parameter has the type '{g} t. This is a function which has some arbitrary ability g that returns a value of type t. The third element of the type signature says that countingFrom returns a similar function.

But thats not true. The implementation of our ability takes an ability-wrapped function that returns some type, and then it returns just that type.³ This means that the correct type for our wrapper is the type of nextFree

Unison

countingFrom: Nat -> '{Counter} [Nat] -> [Nat] 
countingFrom initialValue code =
  handle !code with implementCounter initialValue

> countingFrom 0 nextThree

and ucm rewards us with

    33 | > countingFrom 0 nextThree
           ⧩
           [0, 1, 2]

We can now go helper-crazy:

Unison

countingFromZero = countingFrom 0

> countingFromZero nextThree    --> [ 0, 1, 2 ]

One More Wafer-Thin Tidy

Look back at the type signature for countingFrom:

Unison

countingFrom: Nat -> '{Counter} [Nat] -> [Nat] 

It’s saying that any function we wrap must return [Nat]. But we don’t actually care what type it returns: it returns a value to use, and we return it to our caller. So the type declaration can be made generic:

Unison

countingFrom: Nat -> '{Counter} resultType -> resultType

Abilities and Error Handling

The syntax

handle some_code with ability_implementation

might remind you of something similar in more conventional languages:

try { some_code } catch { exception_handler }

Not only is the syntax similar; the semantics are too. Just like abilities, the scope of exceptions is determined by the lifetime of the code in the try block; exceptions are dynamically scoped. And, just like abilities, when an exception is raised, the runtime passes control to the handler.

Unlike abilities, though, exceptions are typically one way. You raise an exception, control passes to the handler, and execution continues from there. There’s no going back to the line of code following the raise.

So you can’t implement abilities using exceptions, but you can implement exceptions using abilities. Unison comes with the abilities Abort, Exception, and Throw, all of which give you some way to pass errors back to the caller.

So just how do you terminate the function wrapped by an ability? Just have the handler return the error value instead of calling itself recursively.

Making The countingFrom Helper Even More Generic

The type signature for our countingFrom helper contains an implicit restriction. The return type is just a value of the same type that was returned by the client code. But what if our client code used more that one ability?

For example, if you want to do I/O in Unison, you’ll need to wrap your function with the IO ability. You’ll likely also have to include Exception, too, as I/O operations can have unexpected side effects.

So, our chatty version of nextThree will look like this:⁴

Unison

chattyNextThree: '{Counter, IO, Exception} [Nat]
chattyNextThree = do
  result = [ nextValue, nextValue, nextValue ]
  result |> toText |> printLine
  result

runChatty = do
  (handle !chattyNextThree with implementCounter 0)

In this case we can’t use a watch expression to run our handle expression directly, because watch expressions do not know about any abilities. Instead we have to define a function runChatty which does what we want, and then use the run command in ucm to call it:

UCM

   ⍟ These new definitions are ok to `add`:
      chattyNextThree       : '{IO, Exception, Counter} [Nat]
      runChatty             : '{IO, Exception} [Nat]

.pragdave.abilities.counter> run runChatty
[0, 1, 2]
  [0, 1, 2]

The first line of output is the result of the printLine. The indented following line is the value returned by runChatty.

So now let’s change runChatty to use our countingFrom helper:

Unison

runChatty = 
  '(countingFrom 0 chattyNextThree)
--                 ^^^^^^^^^^^^^^^
--  The expression needs the abilities: {IO}
--  but was assumed to only require: {#satjs04snt}
--
--  This is likely a result of using an un-annotated function as an
--  argument with concrete abilities. Try adding an annotation to the
--  function definition whose body is red.

The problem is the definition of countingFrom:

Unison

countingFrom: Nat ->'{Counter} returnType -> returnType 

The handler correctly takes a function wrapped in the Counter ability and returns something with no abilities. But we need to pass on the IO and Exception abilities that our chattyNextThree function uses. We could explicitly add them:

Unison

countingFrom: Nat ->'{Counter, Exception, IO} returnType ->{Exception, IO} returnType 

If we do this, though, we won’t be able to use countFrom with any function that doesn’t need IO or Exception (such as our original nextThree function).

The solution is to add a generic type parameter to the ability list.

Unison

countingFrom: Nat ->'{Counter, rest} returnType ->{rest} returnType 

This type variable will match zero or more additional abilities used by the wrapped function, and will add then to the result. (I call that parameter rest, but the Unison documentation tends to use g.) In our case, the rest type variable matches {Exception, IO}, which it uses to annotate its return value, and everything bursts into life.

Abilities are Exciting

Why am I writing (so much) about a facility which is unlikely ever to be available in your current language?

It’s because I think it’s important. The idea that you can have pluggable behavior that is scoped to runtime opens up a lot of possibilities for our code. Sure, it is likely to have problems in practice. It makes me nervous that calls into the ability are not distinguished from other calls, but that’s easily fixed if it does become a problem. I wonder what the performance implications are, but it’s far too early to be thinking about that, and even if it is a problem, a clever compiler could probably inline much of the use of abilities.

But, the upsides far outweigh these hypothetical issues. The isolation, flexibility, and reusability will all make code better. Even performance issues might have a silver lining: the execution of an ability with state is pretty much like a call to a process in Erlang. Maybe the mythical clever compiler could parallelize some of these calls.

So keep an eye on Unison, and give some thought to how you might bring the benefits of abilities to other environments.

Footnotes

1. Dynamic Scoping

   Modern programming languages implement global and/or module scope and/or lexical scope. A name defined globally is available everywhere in the code. A name given module scope is only directly accessible within that module (and may be available outside if qualified with the module name). A name defined with lexical scope is available inside the current lexical block and (typically) the blocks it encloses.

   All three of these are statically defined: the meaning of a variable name can be determined at compilation time.

   In the past, languages such as Perl also offered dynamic scope. This looks a little like lexical scope, except the names defined in a block are available not just in that block but also in all the functions invoked by that block, and functions invoked below them, and so on.

   The scope is only determined at runtime: the name exists for the duration of the block that defines it, and it exists in all functions executed during that time.

   As you can imagine, this was both powerful and widely abused: it’s hard to know just what a name means when its definition depends on the execution flow. This is one reason we don’t often see dynamic scoping in current languages.

   Unison’s abilities are a form of dynamic scoping. However, they overcome many of the issues with previous kinds of dynamic scoping because they are fully type safe. You cannot accidentally use a name injected freom a higher context, and you always know where every name comes from.

2. This ability to resume execution back in the client’s context is called a continuation, and it implements something very similar to coroutines: two functions that swap execution back and forth between themselves.

3. In our case, that return type is [Nat].

4. In this code I’ve introduced the do keyword. This generates a zero-arity function, just like ', but while ' makes the next expression a zero-arity function, do wraps the entire next block.