Erlang inspired error handling with superv.async.
March 10, 2016
Clojure error handling
Proper error handling is hard. I have struggled with it often
without recognizing it. You might think that you can add
to sections you find prone to error, typically sections doing IO with remote
systems and services. While you realize such sections when you write
them personally, this context is lost when you use libraries in an impure
language. Clojure goes a long way in providing a pure state management concept
which shields you from errors. As long as you properly separate your IO
functions, there is a limited interface on which exceptions can propagate. But
is this assurance good enough? I have experienced that once your state becomes
more and more linked to external systems, code covering these systems also tends
to spread, yielding possibly distributed error states which need to be addressed
on higher levels.
Clojure has nice concepts of how to deal with errors beyond
mechanisms with exceptions. There
libraries providing the Common Lisp condition system for Clojure and
as Chris Houser has pointed out,
even a simple dynamic scoping will doAnd will compose
properly. . This approach has many benefits, importantly it can allow
robustness (restarts), which are not possible when an exception has unwound the
Erlang error handling
So does this cut it? I have faced a more profound problem with Clojure error handling in core.async. But this only lead me to investigate the design space a bit, most prominently the Erlang Let it crash principle.
So before diving into error handling with
core.async, I will briefly describe
the Erlang error handling concept. It is a succinct feature of Erlang, which
profoundly solid as a language to build error-prone distributed systems with very high reliability despite lacking a type system.
Clojure with CSP
core.async is very similar in this regard, but by default it
lacks the same quality error handling.
Erlang uses links and monitors to track different processes (actors) automatically and using these mechanisms preemptively terminates processes linking to a dying process. This importantly allows to easily restart whole subsystems if something goes wrong. You do not handle the error locally if you are not sure what to do about it, but let your process crash, knowing that somebody might be able to catch the error on their monitor and deal with appropriately. Error handling is done in general at higher levels and incrementally added when the programmer experiences errors. But despite this lazy approach, the restarting concept allows to keep the system available as long as some higher system entity can properly restart. Technically there are many Erlang specific concepts like the linking of processes at runtime by refering to their name. But this is not essential for the concept, it only allows easier mutable adaptation of a running system and fits Erlang’s CSP design.
We can hence summarize the requirements for robust error handling by two qualities:These are only the core requirements to recover robustness.
- All errors must be catched and dealt with somewhere
- Resources must be properly freed to allow restarts
So how does this relate to Clojure? core.async uses the concept of go-routines, which are not the same as Erlang processes as they lack the addressing by names (like PIDs). Instead they are passed around as values, expanding a call tree at construction. They can be bound to vars or managed in STM primitives, but typically are rather used as in go-lang as dedicated routines which are called. core.async hence also does not have the send/receive mechanism of Erlang. This is not critical though, as we can recover the error handling requirements above without them.
Erlang inspired error handling in
Coming back to error handling in core.async, it does not provide any error
handling by default and throwing
While it is easy to fix it by installing a default handler for all errors
somewhere, it does not recover the Erlang qualities, but only installs one
global handler. One might wonder whether
go-lang itself has a good error
handling, but I have found out that it has no comparable conceptsIt is not seen as a problem really. and advertises to deal with
errors locally when possible. This does not recover the robustness requirements
mentioned above and can leave subsystems in a broken state by wrongly handling
errors in a too local scope. So what we need to do is ensure that we catch all
exceptions and propagate them and also that once we decide to deal with an
exception, we are able to restart the subsystem affected by it. This requires
bidirectional communication, as we need to signal our restart intent.
Importantly it is hence not enough to put exceptions on an error channel with a
go-try macro construct.
To recover Erlang-like subsystem modelling of errors, we need to assume to
have been passed an entity into our scope which tracks and gets all the errors.
We will call this entity our supervisor in similar terminology to Erlang. This
entity needs to track all errors (1.) and all go-routines (2.). The latter is
necessary so it can wait for their termination, if it decides to restart the
subsystem. There is only one problem left here, which is nicely solved in Erlang
by its preemptive scheduling and error-handling support in the runtime. How to
terminate the go-routines to enforce a restart? The supervisor could easily wait
for a go-routinee.g. a
go-loop forever, effectively
defeating the whole purpose of robustness and availability. Since Clojure cannot
preemptively reschedule code in go-routines, we have to interrupt them
We can exploit a simple observation here, almost all go-routines are written to allow asynchronous operations on channels which typically cover most of their wallclock time. We can control these blocking operations, since we already intercept them with our error-handling mechanism, unblocking them by throwing an abortion in the context, recursively unwinding all parallel go-routines. While this behaviour goes a long way, the programmer needs to be aware that go-routines with code that can take a long time without asynchronous operations (e.g. with blocking IO, long computations etc.) is problematic. It is cooperative concurrency after all.
I haven’t found this to be a problem yet, but there is little we can do here,
except of doing some deep code transformation with autoinjection of code in the
go-macro, which still would not help with outside JVM
code. Pulsar uses bytecode
instrumentation on the JVM to achieve something similar, but they have also
found that preemption is not too important in practice. We cannot use this
low-level approach with
could do differnt things on the two hosts though.)
So my first attempt at
robust error handling was to
adapt full.async. It would have
been desirable to use dynamic binding for the supervisor as it is a typical
example for a good dynamic binding and the already available
constructs keep working transparently. While this dynamic binding approach
worked well for the JVM, it
incompatible with ClojureScripts dynamical binding.
Somewhat reluctantly I decided to fork full.async as superv.async then and add a lexical argument for the supervisor to all corresponding full.async operations. While this adds some inconvenience, the benefits outweigh the costs for me. Not only am I now able to track all exceptions in parallel go-routines appropriately, I can also do things like robust connection handling and provide error handling by functional composition to the whole replication system.
An example invocation of a supervisor with the corresponding async primitives and operations.
While the implementation definitely needs more testing to be really fleshed out, I am comfortable to say that it considerably improves on error-handling in core.async today. It also communicates the normal exceptions to an embedding environment, the supervisor just returns it in case it cannot deal with it. I hope you find it useful and report back in the gitter chat or on the Clojure mailing list :).