After 3 years of development, 316 contributors, & many production deployments, ZIO 1.0 is ready.
# SCALA
# ZIO
On June 5th, 2017, I opened an issue in the Scalaz repository on Github. I argued Scalaz 8 needed a powerful and fast data type for async programming, and encouraged to contribute to Scalaz by my friend Vincent Marquez, I volunteered to build one.
This was not my first foray into async and concurrent programming. Previously, I had designed and built the first version of Aff, and assisted the talented Nathan Faubion with the second iteration. The library quickly became the number one solution for async and concurrent programming in the Purescript ecosystem.
Further back, I had written a Future data type for Scala, which I abandoned after the Scala standard library incorporated a much better version. Before that, I wrote a Promise data type for haXe, and a Raincheck data type for Java. In every case, I made mistakes, and subsequently learned to make new mistakes!
That sunny afternoon in June 2017, I imagined spending a few months developing this new data type, at which point I would wrap it up in a tidy bow, and hand it over to the Scalaz organization.
What happened next, I never could have imagined.
As I spent some free time working on the core of the new project, I increasingly felt like the goal of my work should not be to just create a pure functional effect wrapper for side-effects.
That had been done before, and while performance could be improved over Scalaz 7, libraries built on pure functional programming cannot generally compete with bare-metal, abstraction-free, hand-optimized procedural code.
Now, for developers like me who believe in the One True Way of functional programming, an effect-wrapper is enough, but it's not going to win any new converts if it just slaps a Certified Pure label on imperative code. Although solving the async problem is still useful in a pre-Loom world, lots of other other data types already solved this problem, such as Scala's own Future.
Instead, I thought I needed to focus the library on concrete pains that are well-solved by functional programming, and one pain stood out among all others: concurrency, including the unsolved problem in Scala's Future of how to safely cancel executing code whose result is no longer necessary due to timeout or other failures.
Concurrency is a big enough space to play in that whole libraries and ecosystems have been formed to tame its wily ways. Indeed, some frameworks become popular precisely because they shield developers from concurrency, because it's complex, confusing, and error-prone.
Given the challenges, I thought I should bake concurrency into this new data type, and give it features that would be impossible to replicate in a procedural program without special language features.
This vision of a powerful concurrent library drove my early development.
At the time I began working on the Scalaz 8 project, the prevailing dogma in the functional Scala ecosystem was that an effect type should have little or no support for concurrency baked in.
Indeed, some argued that effect concurrency was inherently unsafe and must be left to streaming libraries, like FS2.
Nonetheless, having seen some of the amazing work coming out of Haskell and F#, I believed it was not only possible but very important for a modern effect type to solve four closely related concurrency concerns:
In the course of time, I developed a small prototype of the so-called "Scalaz 8 IO monad", which solved these problems in a fast and purely functional package: effects could be "forked" to yield a "fiber" (cooperatively-yielding virtual thread), which could be awaited or instantly interrupted, with a Haskell-inspired version of try/finally called bracket that provided resource safety.
I was excited about this design and talked about it publicly before I released the code, resulting in some backlash from competitors who didn't believe the benchmarks could possibly be correct. But on November 16, 2017, I presented the first version at Scale by the Bay, opening a pull request with full source code, including rigorously designed benchmarks, which allayed all concerns.
Despite initial skepticism and criticism, in time all effect systems in Scala adopted the concept of launching an effect to yield a Fiber, which could be safely and automatically interrupted, with a primitive to ensure resource safety.
This early prototype was not yet ZIO as we know it today, but the seeds of ZIO had been planted, and they grew quickly.
My initial design for the Scalaz 8 IO data type was inspired by Haskell and Scalaz Task. In due time, however, I found myself reevaluating some decisions made by these data types.
In particular, I was unhappy with the fact that most effect types have dynamically typed errors. Pragmatically speaking, the compiler can't help you reason about error behavior if you pretend that every computation can always fail in infinite ways.
As a statically-typed functional programmer, I want to use the compiler to help me write better code. I'm able to do a better job if I know where I haven't handled errors, and where I have, and if I can use typed data models for business errors.
Of course, Haskell an answer to adding statically-typed errors: use a monad transformer, and maybe type classes. Unfortunately, this solution increases barriers to entry, reduces performance, and bolts on a second, often confusing error channel.
Since I was starting from a clean slate in a different programming language, I had a different idea: what if instead of rigidly fixing the error type to Throwable, like the Scala Try data type, I let the user pick the type, like Either?
Initial results of my experiment were remarkable: just looking at type signatures, I could understand exactly how code was dealing with errors (or not dealing with them). Combinators precisely reflected error handling behavior in type signatures, and some laws that traditionally had to be checked using libraries like ScalaCheck were now checked statically, at compile time.
So on January 2, 2018, I committed what ended up being a radical departure from the status quo: introducing statically-typed errors into the Scalaz 8 effect type.
Over the months that followed, I worked on polish, optimization, bug fixes, unit tests, and documentation, and found growing demand to use the data type in production. When it became apparent that Scalaz 8 was a longer-term project, a few ambitious developers pulled the IO data type into a standalone library so they could begin using it in their projects.
I was excited about this early traction, and I didn't want any obstacles to using the data type for users of Scalaz 7.x or Cats, so on June 11, 2018, I decided to pull the project out into a new, standalone project with zero dependencies, completely separate from Scalaz 8. I chose the name "ZIO", combining the "Z" from "Scalaz", and the "IO" from "IO monad".
Around this time, the first significant wave of contributors started joining the project, including Regis Kuckaertz, Wiem Zine Elabidine, and Pierre Ricadat (among others)—many new to both open source and functional Scala, although some with deep backgrounds in both.
Through mentorship by me and other contributors, including in some cases weekly meetings and pull request reviews, a whole new generation of open source Scala contributors were born—highly talented functional Scala developers whose warmth, positivity and can-do spirit started to shape the community.
ZIO started accreting more features to make it easier to build concurrent applications, such as an asynchronous, doubly-back-pressured queue, better error tracking and handling, rigorous finalization, and lower-level resource-safety than bracket.
Although the increased contributions led to an increasingly capable effect type, I personally found that using ZIO was not very pleasant, because of the library's poor type inference.
As many functional Scala developers at the time, I had absorbed the prevailing wisdom about how functional programming should be done in Scala, and this meant avoiding subtyping and declaration-site variance (indeed, the presence of subtyping in a language does negatively impact type inference, and using declaration-site variance has a couple drawbacks).
However, because of this decision, using ZIO required specifying type parameters to many methods, resulting in an unforgiving and joyless style of programming, particularly with typed errors. In private, I wrote a small prototype showing that using declaration-site variance could significantly improve type inference, which made me want to implement the feature in ZIO.
At the time, however, ZIO still resided in the Scalaz organization, in a separate repository. I was aware that such a departure from the status quo would be controversial, so in a private fork, Wiem Zine Elabidine and I worked together on a massive refactoring in our first major collaboration.
On Friday July 20, 2018, we opened the pull request that added covariance. The results spoke for themselves: nearly all explicit type annotations were deleted, and although there was some expected controversy, it was difficult to argue with the results. With this change, ZIO started becoming pleasant to use, and the extra error type parameter no longer negatively impacted usability.
This experience emboldened me to start breaking other taboos: I started aggressively renaming methods and classes and removing jargon known only to pure functional programmers. At each step, this created yet more controversy, but also further differentiated ZIO from some of the other design choices, including those expressed in Scalaz 7.x.
From all this turbulent evolution, a new take on functional Scala entered the ZIO community: a contrarian but principled take that emphasizes practical concerns, solving real problems in an accessible and joyful way, using all parts of Scala, including subtyping and declaration-site variance.
Finally, the project began to feel like the ZIO of today, shaped by a rapidly growing community of fresh faces eager to build a new future for functional programming in Scala.
Toward the latter half of 2018, ZIO got compositional scheduling, with a powerful new data type that represents a schedule, equipped with rich compositional operators. Using this single data type, ZIO could either retry effects or repeat them according to near arbitrary schedules.
Artem Pyanykh implemented a blazing fast low-level ring-buffer, which, with the help of Pierre Ricadat, became the foundation of ZIO's asynchronous queue, demonstrating the ability of the ZIO ecosystem to create de novo high-performance JVM structures.
Itamar Ravid, a highly talented Scala developer, joined the ZIO project and added a Managed data type encapsulating resources. Inspired by Haskell, Managed provided compositional resource safety in a package that supported parallelism and safe interruption. With the help of Maxim Schuwalow, Managed has grown to become an extremely powerful data type.
Thanks to the efforts of Raas Ahsan, ZIO unexpectedly got an early version of what would later become FiberRef, a fiber-based version of ThreadLocal. Then Kai, a wizard-level Scala developer and type astronaut, labored to add compatibility with Cats Effect libraries, so that ZIO users could benefit from all the hard work put into libraries like Doobie, http4s, and FS2.
Thanks to the work of numerous contributors spread over more than a year, ZIO became a powerful solution to building concurrent applications—albeit, one without concurrent streams.
Although Akka Streams provides a powerful streaming solution for Scala developers, it's coupled to the Akka ecosystem and Scala's Future. In the functional Scala space, FS2 provides a streaming solution that works with ZIO, but it's based on Cats Effect, whose type classes can't benefit from ZIO-specific features.
I knew that a ZIO-specific streaming solution would be more expressive and more type safe, with a lower barrier of entry for existing ZIO users. Given the importance of streaming to modern applications, I decided that ZIO needed its own streaming solution, one unconstrained by the feature set of Cats Effect.
Bringing a new competitive streaming library into existence would be a lot of work, and so when Itamar Ravid volunteered to help, I instantly said yes.
Together, in the third quarter of 2018, Itamar and I worked in secret on ZIO Streams, an asynchronous, back-pressured, resource-safe, and compositional stream. Inspired by work that the remarkable Scala developer Eric Torreborre did, as well as work in Haskell on iteratees, the initial release of ZIO Streams delivered a high-performance, composable concurrent streams and sinks, with strong guarantees of resource safety, even in the presence of arbitrary interruption.
We unveiled the design at Scale by the Bay 2018, and since then, thanks to Itamar and his army of capable contributors (including Regis Kuckaertz), ZIO Streams has become one of the highlights of the ZIO library—every bit as capable as other streaming libraries, but with much smoother integration with the ZIO effect type and capabilities.
Toward the end of 2018, I decided to focus on the complexity of testing code written using effect systems, which led to the last major revision of the ZIO effect type.
When exploring a contravariant reader data type to model dependencies, I discovered that using intersection types (emulated by the with keyword in Scala 2.x), one could achieve flawless type inference when composing effects with different dependencies, which provided a possible solution to simplifying testing of ZIO applications.
Excitedly, I wrote up a simple toy prototype and shared it with Wiem Zine Elabidine. "Do you want to help work on this?" I asked. She said yes, and together we quietly added the third and final type parameter to the ZIO effect type: the environment type parameter.
I unveiled the third type parameter at a now-infamous talk, The Death of Finally Tagless, humorously presented with a cartoonish Halloween theme. In this talk, I argued that testability was the primary benefit of tagless-final, and that it could be obtained much more simply and in a more teachable way by just "passing interfaces"—the solution that object-oriented programmers have used for decades.
As with tagless-final, and under the assumption of discipline, ZIO Environment provided a way to reason about dependencies statically, and a way to code to interfaces without any pain at the use-site. Unlike tagless-final, it fully inferred and didn't require teaching type classes, category theory, higher-kinded types, or implicits.
Some ZIO users immediately started using ZIO Environment, appreciating the ability to describe dependencies using types without actually passing them (thus allowing dependency inference). Constructing ZIO environments, however, proved to be problematic—impossible to do generically, and difficult to do even when the structure of the environment was fully known.
A workable solution to these pains would not be identified until almost a year later.
Meanwhile, ZIO continued to benefit from numerous contributions, which added combinators, improved documentation, improved interop, and improved semantics for core data types.
The next major addition to ZIO was software transactional memory.
The first prototype of the Scalaz IO data type included MVar, a doubly-back-pressured queue with a maximum capacity of 1, inspired by Haskell's data type of the same name.
I really liked the fact that MVar was already "proven", and could be used to build many other concurrent data structures (such as queues, semaphores, and more).
Soon after that early prototype, however, the talented and eloquent Fabio Labella convinced me that two simpler primitives provided a more orthogonal basis for building concurrency structures:
Promise, a variable data type that can be set exactly one time (but can be awaited on asynchronously and retrieved any number of times);
Ref, a model of a mutable cell that can store any immutable value, with atomic operations for updating the cell.
This early refactoring allowed us to delete MVar and provided a much simpler foundation. However, after a year of using these structures, while I appreciated their orthogonality and power, it became apparent to me that they were the "assembly language" of concurrent data structures.
These structures could be used to build lots of other asynchronous concurrent data structures, such as semaphores, queues, and locks, but doing so was extremely tricky, and required hundreds of lines of fairly advanced code.
Most of the complexity stems from the requirement that operations on the data structures must be safely interruptible, without leaking resources or "deadlocking".
Moreover, although you can build concurrent structures with Promise and Ref with enough work, you cannot make coordinated changes across two or more such concurrent structures.
The transactional guarantees of structures built with Promise and Ref are non-compositional: they apply only to isolated data structures, because they are built with Ref, which has non-compositional transactional semantics. Strictly speaking, their transactional power is equivalent to actors with mutable state: each actor can safely mutate its own state, but no transactional changes can be made across multiple actors.
Familiar with Haskell's software transactional memory, and how it provides an elegant, compositional solution to the problem of developing concurrent structures, I decided to implement a version for ZIO with the help of my partner-in-crime Wiem Zine Elabidine, which we presented at Scalar Conf in April 2019.
Dejan Mijic, another fantastic and highly motivated developer with a keen interest in high-performance, concurrency, and distributed systems, joined the ZIO STM team. With my mentorship, Dejan helped make STM stack-safe for transactions of any size, added several new STM data structures, dramatically improved the performance of existing structures, and implemented retry-storm protection for supporting large transactions on hotly contested transactional references.
ZIO STM is the only STM in Scala with these features, and although the much older Scala STM is surely production-worthy, it doesn't integrate well with asynchronous and purely functional effect systems built using fiber-based concurrency.
The next major feature in ZIO would address a severe deficiency that had never been solved in the Scala ecosystem: the extreme difficulty of debugging async code, a problem present in Scala's Future for more than a decade.
Previously in presenting ZIO to new non-pure functional programmers (the primary audience for ZIO), I had received the question: how do we debug ZIO code?
The difficulty stems from the worthless nature of stack traces in highly asynchronous programming. Stack traces only capture the call stack, but in Future and ZIO and other heavily async environments, the call stack mainly shows you the "guts" of the execution environment, which is not very useful for tracking down errors.
I had thought about the problem and had become convinced it would be possible to implement async execution traces using information reconstructed from the call stack, so I began telling people we would implement something like this in ZIO soon.
I did not anticipate just how soon this would be.
Kai came to me with an idea to do execution tracing in a radically different way than I imagined: by dynamically parsing and executing the bytecode of class files. Although my recollection is a bit hazy, it seemed mere days before Kai had whipped up a prototype that seemed extremely promising, so I offered my assistance on hammering out the details of the full implementation, and we ended up doing a wonderful joint talk in Ireland to launch the feature.
Sometimes I have a tendency to focus on laws and abstractions, but seeing the phenomenally positive response to execution tracing was a good reminder to stay focused on the real world pains that developers have.
Beginning in the summer of 2019, ZIO began seeing its first significant commercial adoption, which led to many feature requests and bug reports, and much feedback from users.
The summer saw many performance improvements, bug fixes, naming improvements, and other tweaks to the library, thanks to Regis Kuckaertz and countless other contributors.
Thanks to the work of the ever-patient Honza Strnad and others, FiberRef evolved into its present-day form, which is a much more powerful, fiber-aware version of ThreadLocal—but one which can undergo specified transformations on forks, and merges on joins.
I was very pleased with these additions. However, as ZIO grew, the automated tests for ZIO were growing too, and they became an increasing source of pain across Scala.js, JVM, and Dotty (our test runners at the time did not natively support Dotty).
So in the summer of 2019, I began work on a purely functional testing framework, with the goal of addressing these pains, the result of which was ZIO Test.
Testing functional effects inside a traditional testing library is painful: there's no easy way to run effects, provide them with dependencies, or integrate with the host facilities of the functional effect system (using retries, repeats, and so forth).
I wanted to change that with a small, compositional library called ZIO Test, whose design I had been thinking about since even before ZIO existed.
Like the ground-breaking Specs2 before it, ZIO Test embraced a philosophy of tests as values, although ZIO Test retained a more traditional tree-like structure for specs, which allows nesting tests inside test suites, and suites inside other suites.
Early in the development of ZIO Test, the incredible and extremely helpful Adam Fraser joined the project as a core contributor. Instrumental to fleshing out, realizing, and greatly extending the vision for ZIO Test, Adam has since become the lead architect and maintainer for the project.
Piggybacking atop ZIO's powerful effect type, ZIO Test was implemented in comparatively few lines of code: concerns like retrying, repeating, composition, parallel execution, and so forth, were already implemented in a principled, performant, and type-safe way.
Indeed, ZIO Test also got a featherweight alternative to ScalaCheck based on ZIO Streams, since a generator of a value can be viewed as a stream. Unlike ScalaCheck, the ZIO Test generator has auto-shrinking baked in, inspired by the Haskell Hedgehog library; and it correctly handles filters on shrunk values and other edge case scenarios that ScalaCheck did not handle.
Toward the end of 2018, after nearly a year of real world usage, the ZIO community had been hard at work on solutions to the problem of making dynamic construction of ZIO environments easier.
This work directly led to the creation of ZLayer, the last major data type added to ZIO.
Two very talented Scala developers, Maxim Schuwalow and Piotr Gołębiewski, jointly worked on a ZIO Macros project, which, among other utilities, provided an easier way to construct larger ZIO environments from smaller pieces. This excellent work was independently replicated in the highly-acclaimed Polynote by Jeremy Smith in response to the same pain.
At Functional Scala 2019, several speakers presented on the pain of constructing ZIO Environments, which convinced me to take a hard look at the problem. Taking inspiration from an earlier attempt by Piotr, I created two new data types, Has and ZLayer.
Has can be thought of as a type-indexed heterogeneous map, which is type safe, but requires access to compile-time type tag information. ZLayerZLayer can be thought of as a more powerful constructor, which can build multiple services in terms of their dependencies.
Unlike constructors, ZLayer dependency graphs are ordinary values, built from other values using composable operators, and ZLayer supports resources, asynchronous creation and finalization, retrying, and other features not possible with constructors.
ZLayer provided a very clean solution to the problems developers were having with ZIO Environment—not perfect, mind you, and I don't think any solution prior to Scala 3 can be perfect (every solution in the design space has different tradeoffs). The good solution became even better when the excellent consultancy Septimal Mind donated Izumi Reflect to the ZIO organization.
The introduction of ZLayer was the last major change to any core data type in ZIO. Since then, although streams has seen some churn, the rest of ZIO has been very stable.
Yet despite the stability, until recently, there was still one major unresolved issue at the very heart of the ZIO runtime system: a full solution to the problem of structured concurrency.
Structured concurrency is a paradigm that provides strong guarantees around the lifespans of operations performed concurrently. These guarantees make it easier to build applications that have stable, predictable resource utilization.
Since I have long been a fan of Haskell structured concurrency (via Async and related), ZIO was the first effect system to support structured concurrency in numerous operations:
Some of these design decisions were contentious and have not been implemented in other effect systems until recently (if at all).
However, there was one notable area where ZIO did not provide structured concurrency by default: whenever an effect was forked (launched concurrently to execute on a new fiber), the lifespan of that executing effect was unconstrained.
Solving this problem turned out to require multiple major surgeries to ZIO's internal runtime system (which is a part of ZIO that few developers understand completely, and which tends to bottleneck on me).
In the end, we solved the problem in a satisfactory way, but it required learning from real world feedback and prototyping no less than 5 completely different solutions to the problem.
Today, ZIO is the only effect system in Scala with full structured concurrency by default.
Yesterday, on August 3rd, ZIO 1.0 was released live in an online Zoom-hosted launch party that brought together and paid tribute to contributors and users across the ZIO ecosystem. We laughed, we chatted, I monologued a bit, and we toasted a few times to users, contributors, and the past and future of ZIO.
As of today, the ZIO 1.0 artifacts are now available on Sonatype, and other parts of the ZIO ecosystem are working to rapidly release new versions of downstream libraries.
We expect full binary backward compatibility for the 1.x line, with two exceptions:
These guarantees of backward compatibility will help the ZIO ecosystem flourish, and encourage more corporate adoption.
The release notes for 1.0 are online.
Effect systems aren't right for everyone, and ZIO may not be the right choice for some teams, but I think there are compelling reasons for Scala developers to take a serious look at ZIO 1.0:
Whether you end up with ZIO or something else, there's no question that concurrent programming in Scala was never this much fun, correct-by-construction, or productive!
ZIO 1.0 would not be possible without the work of more than 316 contributors. I personally reviewed and thanked many of these contributors, but I want to thank each and every one of them now for their work, their creativity, and their inspiration.
In addition, I'd like to personally thank some of the core contributors, who have not only rolled up their sleeves and contributed directly, but have mentored other developers, given talks, inspired me, and contributed an incredible variety of creative suggestions, innovative features, bug fixes, automated tests, and documentation improvements.
In particular, I want to thank:
I want to thank all the early adopters of ZIO, who fearlessly rolled out ZIO into production since the 0.22 release, and who were willing to pay the tax of tirelessly updating their code base, sometimes with each new release candidate for ZIO. These adopters encountered problems, reported bugs, and proposed new features that helped shape ZIO into the library it has become.
I want to thank Septimal Mind, who donated Izumi Reflect and contributed execution tracing; SoftwareMill, for their financial support for CircleCI builds and for being outstanding community supporters; Scalac, for their ongoing work teaching ZIO, sponsorship of ZIO Hackathon Warsaw, and their contributions to multiple ZIO open source projects (including the amazing panopticon-tui); and Signify Technology, who has helped bring many ZIO jobs to Scala developers, and whose participation in the Scala community has greatly enriched it.
Finally, I want to acknowledge the Haskell ecosystem, in particular Simon Marlow, who pushed the boundaries of what's possible in functional concurrency and asynchronicity; Michael Snoyman, who invented RIO, which shares a common design and purpose with ZIO; Michael Pilquist, who came up with a brilliant trick for making asynchronous waiting in concurrent structures safely interruptible (we used this in ZIO for the original version of Semaphore, though it's not necessary with STM, which provides this property for free); Alex Nedelcu, whose work on Monix blew past Scalaz Task long before there was ZIO, and whose work inspired a wonderful peek-ahead optimization in ZIO's runtime; and Fabio Labella, who persuaded me to abandon in favor of more orthogonal primitives.
This list could never be complete, and if I have omitted any significant credit or contribution, it's only because I'm human.
This has been an amazing journey for me, one I can scarcely believe has concluded. I'm incredibly grateful for the experience and the opportunity to finally release of ZIO 1.0, and start a new journey to places unknown.
— John A. De Goes
Stay ahead with the latest insights and breakthroughs from the world of technology. Our newsletter delivers curated news, expert analysis, and exclusive updates right to your inbox. Join our community today and never miss out on what's next in tech.
