Scala's market may not grow any larger unless we take action to address the challenges the language faces
# SCALA
# SCALA3
# TOOLING
# IDES
This post was originally published on degoes.net.
As a long-time user and one of the earliest evangelists of the Scala programming language, and as CEO of a tech company that has heavily invested in Scala open source, I've been acutely interested in growing Scala adoption within industry.
Scala 3 has the most powerful static type system and metaprogramming facilities in any major programming language. The language's strength in designing robust internal domain-specific languages like Spark has opened new doors into concurrent programming, distributed programming, database access, and many other domains.
Yet, not all is sunshine and roses in the world of Scala:
These factors, combined with the scarcity and high cost of senior Scala talent, have pushed many companies to other languages like Go, Kotlin, Java, and Rust, depending on their use cases.
While there are too many commercial users of Scala for the language to "disappear", it is not clear that the Scala programming language will grow adoption over time. The ZIO community continues to grow, but upon close inspection, the majority of this growth is occurring within the existing Scala market, and it is therefore not a proxy for the commercial success of Scala.
In light of this uncertain future, I believe that the time for action has come. As passionate commercial Scala developers, evangelists, and OSS contributors, we have seen how quickly the language helps us build reliable systems at scale. We know that Scala has the potential to take cloud-native programming to a new level of robustness, performance, and productivity.
In the ashes of Scala 2 and the Lightbend OSS ecosystem, we need a rebirth for Scala. A Scala resurrection, so to speak, which can help the language and ecosystem soar to new heights.
In this post, I will discuss the issues standing in the way of Scala's growth in industry, and share my thoughts about how we can all help.
With the introduction of Scala 3, the language has new life.
Scala 3 simplifies the Scala language in many important ways, such as splitting different use cases for implicits into special-purposed language features. The compiler itself has a more modern and maintainable architecture than its predecessor, which allows improvements and bug fixes to be made more quickly and more reliably.
Scala 3 is powerful, and while by no means perfect, it packs significant improvements in the areas of type-safety, metaprogramming, and abstraction, compared to all mainstream programming languages. Moreover, the language learns from mistakes made in Scala 2, and responds to some (but not all) feedback from commercial Scala developers.
Despite the promises, there remain challenges with Scala 3:
Continued pain and attrition due to the Scala 3 upheaval is inevitable, but not fatal. With language and ecosystem stability, combined with tangible library benefits from upgrading to Scala 3, I believe we will eventually convince most serious commercial users.
However, I know that millions of euros are currently being invested into Caprese, which is a capability-based effect system built on capture tracking that could be a new language unto itself. If Caprese is transformed into a hypothetical Scala 4, it would upend not just the language, but the entire OSS ecosystem and all tooling, making the syntactical changes between Scala 2 and 3 seem tame by comparison.
The looming threat of a future radically different Scala 4 is the single greatest existential threat to Scala.
In the old days, a text editor and were considered sufficient "tooling", but as programming languages have matured, so has their tooling, and expectations from developers are vastly higher today than at any point in history.
Every successful commercial programming language must include four key tools:
A programming language might be able to acquire mainstream status with one deficiency. But achieving mainstream status with multiple deficiencies is extremely unlikely, given how far developer tooling has come.
Let's look at how Scala stacks up in each area.
Despite the existence of Coursier, Scala essentially has no native package manager or online registry. Instead, owing to its roots on the JVM, Scala relies heavily on infrastructure common to the platform.
In the JVM, repositories hold build artifacts and dependencies. The structure of these repositories and the protocol by which they are accessed ultimately derives from Apache Maven, a project management tool built around objects like artifacts, versions, groups, and projects.
With Maven, there does not exist a single online repository of all public packages. Rather, there are many repositories, such as Maven Central, Sonatype Releases, JCenter, JBoss, and so forth. In addition, anyone can create and maintain their own repository, although build tools will not be able to fetch projects from custom repositories without additional configuration.
Relying on existing JVM infrastructure offers some advantages:
On the other hand, piggybacking on Maven comes with some serious drawbacks:
Modern package managers such as Cargo or NPM are tailor-made for their respective languages. Publishing and consuming dependencies is simple, in many cases a one-liner that's baked into standard tooling. JSON and REST APIs are supported natively. And there is a definitive online registry of open source projects that is pre-wired into all developer tooling.
Maven may have been an advantage in earlier, simpler times, but in this day and age, it's a significant liability—a dinosaur from another era, which pushes people away from Scala.
Overall, package management in Scala gets a D, barely passing and that, based solely on Maven's maturity and its familiarity to Java developers.
Grade: D
Except for an IDE or text editor, there is no tool a developer spends more time with than a build tool: the tool responsible for running compilers, downloading dependencies, and assembling artifacts that can actually be executed or deployed.
A good build tool can keep developers happy and productive, all the way up from simple "Hello World" applications to complex, large-scale applications that are split across dozens and dozens of projects, with hundreds of dependencies and very custom build steps.
The de facto build tool for Scala is Simple Build Tool, better known by the acronym SBT. For well more than a decade, SBT has gotten the job done, providing a way to build small and large Scala projects, both simple and complex.
To its credit, SBT did not shy away from complex builds, unlike many other attempts at creating build tools for Scala. Furthermore, its wide adoption is a testament not merely to its first-mover advantage, but to its features, documentation, build model, and extensive ecosystem of plugins. I and many other Scala developers owe a huge debt of gratitude and appreciation for the countless developer years poured into this tool, both by individual contributors who generously donated their free time, as well as by companies.
Unfortunately, SBT falls far short of the standards for modern build tools. Indeed, SBT falls so far short that numerous frustrated developers over the past decade have attempted to write better build tools for Scala (Mill, CBT, Seed, Fury, Bleep, etc.).
At a high-level, SBT suffers from the following major drawbacks (among others):
In my opinion, the main reason developers choose SBT to build Scala projects is because developers choose SBT to build Scala projects. For all these drawbacks, Scala's build tool gets an F.
Grade: F
An Integrated Development Environment (IDE), once a luxury, is now standard for all modern mainstream programming languages. With the most popular programming languages, developers have a choice of multiple IDEs—some commercial, some open source; some desktop-based, some cloud-based.
Thanks to type- or receiver-directed auto-complete, inline code documentation, inline errors and warnings, intelligent code refactoring, go-to-definition, and many more features, IDEs greatly increase the productivity and joy of developers, making many developers refuse to go back to editing code files with primitive text editors (whose only tool is "copy and paste").
In recent history, Scala developers have had a choice of three IDE-like experiences:
Only one of these would claim to be an IDE (IntelliJ), and only the first two have roughly the right feature set for an IDE.
Before evaluating these solutions, a few words of praise are in order.
JetBrains deserves kudos for building the finest Java IDE I have ever used (I happily paid for licenses in my Java days). I'm also quite grateful that JetBrains has invested in so many language plug-ins. Similarly, I'm grateful to the Scala Metals team for bringing me the first solution that I felt like could become the IDE that I wanted for Scala.
Many talented and smart developers poured much laudable effort into both of these solutions.
Yet both IntelliJ IDEA for Scala and Scala Metals suffer drawbacks so serious, they are more prototypes than products, and the distance between them and something that delivers the experience of IntelliJ IDEA for Java is growing, not shrinking.
For quite plausible-sounding reasons, IntelliJ IDEA made a gigantic mistake when they decided to implement their own syntax, type, and kind-checking for the Scala plug-in. This same decision for Java resulted in an amazing IDE that could function incrementally and in the presence of type errors (it's quite useful to refactor code even when it doesn't fully compile).
However, Java has a very simple type system that is static and well-defined. Scala's type system is extremely complex and defined only by the implementation, which itself has numerous quirks, edge cases, and of course bugs. Moreover, the semantics, quirks, edge cases, and bugs all vary somewhat between minor versions of the Scala compiler, and are completely different between major versions. Attempting to emulate Scala's type system means that IDEA and Scalac often do not agree on whether code compiles, and if it does compile, what the error message is.
Rather than correct this mistake by using the Scala compiler itself for errors and warnings, IntelliJ IDEA has doubled-down on the mistake, time and time again. Perhaps their next-generation editor Fleet is an acknowledgement both of the fact that JVM desktop applications are truly dead, and duplicating independent syntax, type, and kind-checkers for every language is not viable.
The architectural issues with the IntelliJ IDEA Scala manifest themselves in ways that frustrate Scala developers: code that compiles with scalac does not compile with IntelliJ IDEA. Code that does not compile with scalac compiles just fine with IntelliJ IDEA. One never knows if one's code is truly compiling without running the build tool, which defeats the primary purpose of an IDE.
Beyond these frustrating issues with simple compilation, the Scala plug-in for IntelliJ IDEA simply is not up to the same standards as their Java IDE. Refactoring support is barely present, and doesn't always work. With so many resources spent merely trying to keep the plug-in's model of Scala consistent with all the editions of the two Scala compilers, it is hardly surprising the team can't implement other IDE features.
As a result, while many commercial Scala developers still use IntelliJ IDEA (because, for all its drawbacks, it is significantly more stable and robust than Scala Metals), it is not an acceptable development experience, and it is no where in the same league as modern IDEs for languages such as Java, Kotlin, C#, Go, TypeScript, or even Python.
My own frustrations with IntelliJ IDEA led me to try Scala Metals, and I was blown away by the early experience of using the tool: with just a few clicks, it would compile my project and start reporting errors in VS Code, which were powered directly by the real Scala compiler.
Yet having used Scala Metals for a few years now, I know well that even though Metals demos well in controlled environments, the experience of using Metals in production is extremely frustrating. Quite simply, Metals is an extremely buggy piece of software, which frequently gets into a state where it completely stops working.
In fact, so common is the experience of Metals breaking that some developers have made scripts to blow away all of its directories, kill its JVM processes, and restart Metals in hopes the incantation will work again for some (probably short!) period of time.
Beyond the painful experience of using Metals due to its longstanding instability, Metals has chosen an architecture which is fundamentally flawed. Although Metals does not have its own syntax, type, and kind-checker (like IntelliJ IDEA for Scala), Metals invented its own "Build Server Protocol" and a separate build engine called Bloop for understandable but ultimately technically unjustifiable reasons.
So not only does Scala Metals deliver a very poor experience, but it's built on a suboptimal architecture—a foundation that is far too large, and which will continue to accrete technical debt at an unsustainable rate.
In short, there are no IDEs for Scala right now with a modern experience suitable for commercial software development, nor any obvious path to a commercial-grade IDE. Consequently, Scala's grade for IDE support is a definite F.
Grade: F
The REPL, which stands for Read-Eval-Print-Loop, allows developers to explore a language's syntax, semantics, and libraries, and to solve simple programming problems in an interactive way that maximizes feedback and encourages iteration and experimentation. While often targeted at new developers, REPLs are often used by seniors.
To its credit, Scala had a REPL before it was cool for statically-typed languages to include one. The REPL grew increasingly outdated over the years, failing to keep up with modern REPLs (especially those of dynamically-typed programming languages).
Then Ammonite came along, and added numerous features, such as syntax highlighting, multi-line editing, dependency loading, and much more, giving Scala a much more impressive REPL story, albeit one laden with edge cases in the wake of Scala 3.
Overall, because Scala does have a modern REPL in Ammonite, but because Ammonite does not entirely support Scala 3 and the project is not integrated into the Scala compiler, Scala's REPL support gets a grade of B.
Grade: B
Scala's average rating is a D, well below what I would consider the bare minimum for a mainstream programming language. Moreover, this grade represents a lowering from the peak of the Scala 2 lifecycle, where tooling and in particular IDE support reached an all-time high.
It is worth pointing out that while these are my own ratings, they are consistent with the views of the broader Scala community, at least as collected by the non-random (and therefore certainly biased) 2022 Scala survey.
The state of tooling represents the most immediate and greatest challenge to working with the Scala programming language today.
Scala has multiple open source ecosystems, which target specific pain points in building backend applications. In the sections that follow, I overview the strengths and weaknesses of each, with an eye toward how these ecosystems affect Scala adoption.
The Akka ecosystem originally intended to bring Erlang-style actors to the JVM. Through excellent marketing, and partially by riding the early Scala hype train, Akka managed to find itself over-hyped and over-deployed, with many companies later needing to rip out the technology (quite simply, actors are a terrible solution for local concurrency).
As Akka matured, however, and valid use cases became clear (distributed, persistent actors and an async foundation for local streaming), the ecosystem found its niche and its stride, and was successfully adopted by many large companies in mission-critical use cases.
These days, however, Lightbend, the company behind Akka, decided to monetize the framework directly, by changing from an open source license to the Business Source License, which had proven monetizable for databases and other developer infrastructure.
Although there is an open source fork of Akka called Pekko, it is not well-maintained and stands on questionable legal foundations. Thus, every company I have talked to about Akka has told me they are ripping it all out. Apache Flink, which depended on Akka, also plans to rip it out, with no plans to depend on Pekko.
Akka is still a powerful framework for building distributed and streaming applications. However, I do not expect Akka to help grow the Scala ecosystem, and in fact, I expect the change in licensing to push some companies away from Scala.
Thus, with a Scala-centric lens, Akka gets a grade F, solely for its negative effects on Scala adoption.
Grade: F
The ZIO ecosystem has brought a fresh and innovative fusion of practicality, ergonomics, and functional programming to the domain of building scalable, bulletproof cloud applications.
Ported to multiple programming languages, and even inspiring a few companies to "Go Scala", ZIO has become widely adopted on its own merits, while providing a partial but increasingly viable alternative to Akka, Spring, and other modern application development frameworks.
The ZIO community actively creates many new open source contributors and speakers each year, mentors existing open source contributors, and creates and maintains integrations to other Scala ecosystems, without any politics and with old school professionalism.
Despite all the good that the ZIO ecosystem is doing for Scala, however, the ZIO ecosystem has multiple weaknesses:
Given these drawbacks, together with the positive aspects, the ZIO ecosystem receives a grade of B.
Grade: B
The TypeLevel community, which started life as a political fork of the much older Scalaz (the first port of Haskell libraries to Scala), attempts to bring category-theoretic functional programming to commercial software development in Scala.
Being the oldest still viable example of "functional programming in Scala", the TypeLevel ecosystem has slowly developed a full-stack solution for the backend, including solutions for persistence, HTTP, logging, and caching. With much better documentation than Scalaz ever had, TypeLevel has also made Haskell-style functional programming accessible to more Scala developers than ever, showing commercial software developers the practical benefits of referential transparency and algebraic abstractions.
While having a positive impact on adoption of functional Scala within industry, the TypeLevel ecosystem has weaknesses of its own, including:
In light of both the positive impact but also TypeLevel's challenges around usability and community, the ecosystem receives a grade of B.
Grade: B
Though Scala is home to many open source libraries, most of these are not part of any ecosystem. The two partial exceptions are SoftwareMill's range of libraries, including Tapir and sttp, and Li Haoyi's Python-esque libraries that focus on getting things done with a minimum of ceremony.
There is no doubt many of these libraries are high quality, especially in the aforementioned two ecosystems. Where they are present, weaknesses of libraries across the greater Scala ecosystem include:
Overall, Scala gets a B in this category, an above average score. For its size, Scala has a wonderful third-party OSS ecosystem.
Grade: B
Scala's modern OSS ecosystem suffered a blow when Lightbend took the Akka ecosystem proprietary. Akka provided a comprehensive toolkit for building resilient, distributed, cloud-native applications, and showed the world that Scala offerings could compete with the very best of what other languages offered.
Despite these tumultuous changes, the Scala open source ecosystem remains vibrant and powerful—definitely playing in a league well beyond what the size of the Scala community would suggest.
However, despite being above average, there is ample room for improvement, in ZIO, TypeLevel, and other ecosystems. With improvement in a few core areas, Scala could unquestionably have one of the finest open source ecosystems of any programming language.
We live in a multi-platform age. Our applications must run locally on different desktop operating systems, remotely on different server operating systems, on different virtual machines, and sometimes even in browsers or on the edge.
In this section, I will overview Scala's support for targeting multiple platforms.
The JVM was the first platform that Scala supported, and to this day, it remains the only officially supported platform for Scala, and the only one that is baked into the compiler and broadly assumed by ecosystem tooling and libraries.
Despite the JVM being the first-class platform for Scala, there are still some major deficiencies in Scala's support for the JVM:
Because Scala does support the JVM natively and directly, but also because its support is not best-in-class, Scala's support for the JVM platform gets a grade of C (average).
Grade: C
Supporting the Javascript platform for Scala has always puzzled me somewhat. Currently, Javascript is used primarily for web front-ends, and most Scala developers have neither interest nor skills in frontend web development.
Most web frontend developers use Javascript or TypeScript. They do not want to learn a backend programming language to write frontend code.
That said, it's quite nice to use the same language for both frontend and backend (if you are familiar with both), and doing so can often reduce code duplication. In addition, providing Scala support for the Javascript platform allows backend engineers to create and maintain applications that they would ordinarily have no interest in creating or maintaining.
The quality of the Scala.js platform is high. Sporting a small subset of the JDK, one can compile real world code on Scala.js, and get high-quality Javascript output. Method overloading and other Scala features are mostly emulated in the way you would want.
That said, however, Scala.js has a number of weaknesses:
Considering both the strengths and weaknesses of Scala's Javascript platform support, Scala.js gets a grade of B.
Grade: B
Scala's LLVM support is known as Scala Native, because the end result of using LLVM is native platform-specific executables that run without bytecode-based interpretation or just-in-time compilation.
Scala Native is an attempt to wean Scala from its long-standing addiction to the JVM, providing a viable alternative platform for backend applications. JVM applications suffer from high-memory usage, GC pauses, and lengthy application startup and warmup times, all of which prove problematic with many classes of applications (command-line applications, cloud-native applications, etc.).
Scala Native is an enormous undertaking, because LLVM is much lower-level than the JVM or Javascript platforms. As with Scala.js, Scala Native must provide an emulation of some JDK constructs for virtually any program at all to run. But more than that, Scala Native must provide memory management, multithreading, and many other functions that we take for granted on the JVM platform.
A number of highly talented engineers have poured their own personal time and effort into Scala Native. In addition, the project has seen some modest investment by EPFL and the Scala Center.
Despite all of this investment, however, Scala Native is no where near to a suitable replacement for Scala JVM. The vast majority of Scala applications are not even close to compiling, let alone executing reliably and performantly, on Scala Native. And as with Scala.js, the tooling that exists or at least partially works for the JVM is missing or buggy for Native, leading to a poor development experience.
As much as I see the promise of viable alternatives to the JVM, Scala's LLVM support receives a grade of D, simply because it is not there yet, and requires significant further investment to become viable for most JVM-based Scala applications.
Grade: D
WASM has emerged as a possible successor to the JVM specification, one built for a cloud-native era, in which sandboxing is the norm, and low-latency, low-memory consumption cloud apps rule the skies.
In theory, there is a way to translate from LLVM IR to WASM through Emscripten. In practice, this is so far away from anything usable for production that Scala gets an F for WASM support.
Grade: F
Scala's platform strengths reside squarely with the JVM, which is unsurprising given the origin of the language. What is increasingly surprising, however, is that Scala remains an essentially JVM-only language.
Scala.js is an excellent attempt to bring Scala to the front-end via Javascript, albeit with a limited market and challenges around JDK support. Scala Native (LLVM), on the other hand, while being a remarkable achievement, is such a vast undertaking that it's too early, too limited, and too immature to benefit the typical Scala application. The LLVM pathway to WASM (if you want to call it that!) is merely a hello-world toy.
If you believe the JVM is, without question, the future platform for cloud- and edge-native applications, then Scala's existing platform support may be everything you want and need. For those hoping for native applications or a more modern take on a cross-platform virtual machine, Scala's current cross-platform support leaves a lot to be desired.
The difference between a hater and a lover is that a hater shouts (destructive) criticism to tear down, while a lover offers (constructive) criticism to build up.
To go as far as I believe the language can go, Scala needs building up. Not from just one individual, such as myself, or even Martin Odersky. Rather, Scala needs all our help, as we work together to make the language as commercially successful as it deserves to be.
We need a coordinated campaign to rectify known weaknesses, and shore up emerging strengths. Such a campaign will not be easy. It will require discipline, collaboration, and sacrifice, acting with a sense of common purpose for the greater good of all.
In the sections that follow, I'm going to present what I think needs to happen, as well as a few ideas on how it could happen. If you disagree or have your own suggestions, then please post below!
In my opinion, mainstream adoption of Scala requires a commitment to language stability. There can not be more "Scala 3" editions, which re-imagine the semantics and syntax of the core language, break virtually every library ever developed, reset tooling to ground zero, and require a combination of automated and manual code rewrites.
It is absolutely true that Scala 3 is not a perfect language. It is also true that Java is not a perfect language. Yet Java remains backward compatible with the very first Java program ever written, which allows the ecosystem to continuously improve, and which gives companies confidence in investing in the Java programming language.
Scala has been a useful vehicle for exploring academically interesting ideas, such as the dot calculus which forms the basis of Scala 3. But in order to enter mainstream, academic experiments must live outside Scala. They must live in new languages, or new dialects of Scala that are well insulated from the commercial edition of Scala.
Scala now needs a firm commitment to maintain backward compatibility indefinitely. We need to be able to run the same Scala 3 programs now that we can run 10 years from now, without having to completely rewrite them (with or without partial automation).
This commitment might mean that new academic explorations might not make it into Scala, but it will also mean Scala's ecosystem (particularly its tooling, but also libraries) will bloom, and companies will have the confidence they need to invest seriously into the language.
Scala also needs a commitment to fixing bugs and improving performance in the core compiler. Scala developers need a compiler that works reliably and performantly, and while that's true in many cases, there is much opportunity for ongoing improvement.
Finally, Scala needs work on an optimizer that can minimize the cost of abstraction. There is little value in having a language as expressive as Scala if actually taking advantage of its full powers slows applications down. And there is no reason why it should have to, as most forms of abstraction can be eliminated via a sufficiently advanced optimizer.
Scala desperately needs an IDE that works reliably across complex, real world projects. An IDE is table-stakes for any mainstream programming language, and while IntelliJ IDEA and Scala Metals have both tried to give Scala developers the IDE they need, neither has yet attained that goal, and architectural choices likely prevent these projects from ever attaining this goal.
We need an ultra lean, lightweight architecture for a squeaky clean IDE that can be maintained by a small team with an exceptionally low defect count. Far more important than breadth of features is core stability and developer experience. By leveraging existing open source, by learning from the architectural mistakes of IDEA and Metals, we can engineer a new and extensible solution that delivers the joyful and productive experience that modern developers expect from their IDEs.
Scala also needs a modern build tool, and can chart one of three paths to get there:
Though we can wait on this one for a while, Scala still needs its own package manager, distinct from Maven, which is built for Scala libraries and applications, which knows about Scala's versioning and cross-platform support, and which provides the joyful experience required by new developers, and the extensible experience required by seasoned professionals.
Scala also could benefit from a first-class REPL experience like Ammonite, polished, extended with full support for Scala 3, and incorporated directly into the official Scala compiler, as a replacement for its existing REPL.
In discussing tooling, I must mention the work that Virtus Lab has done in creating Scala CLI, a simple way to compile, run, test, and package single-module Scala projects that was recently accepted as the new scala command.
While not really a build tool right now (for most real world projects), Scala CLI could potentially become a foundation for a build tool, with package management and integrated REPL, as well as IDE support.
Solving the tooling challenges will require investment into promising open source projects, investigating ways to integrate the best parts from multiple OSS projects (for example, Scala CLI, Ammonite, Fury, Ensime), and the creation of new open source projects. Moreover, it will require a commitment to ongoing, broad-based investment from the community at large, as well as partners from academia and industry.
The Scala open source ecosystem is strong, but needs to become stronger. Obviously, the respective ecosystems can and should embrace their respective strengths, while addressing their respective weaknesses.
However, I believe Scala can go much further than just ecosystem-siloed improvement.
With a finite resources, we need to make every hour of open source labor count. One of the most rational and effective ways to do this is to reduce duplication across Scala's open source ecosystems.
Does Scala really need 20 JSON libraries, 5 schema libraries, 9 CLI libraries, 4 effect system libraries, and so on? Probably not. We need to consolidate, and rally around strong projects; merge our strengths, cancel our weaknesses.
There's probably a way to do that, in some cases, that makes everyone happy.
Let's say we wanted to consolidate JSON libraries. Each ecosystem having a JSON library could appoint a person to collaborate, who would be an admin in a joint project. The best aspects of each library could be combined into a single library that meets the needs of all users.
In support of this goal, I will personally commit to helping organize a Scala Open Source Summit, which encourages all ecosystems to come together to discuss and hack on Scala OSS. This should help foster dialogue between different ecosystems to see if common ground can be found.
If the most brilliant minds in Scala open source work together, then surely, the whole ecosystem can rise to new heights!
The JVM is attempting to innovate, and to its credit, is doing so faster now than at any time in its history. However, the world has moved far past the JVM, with cloud-driven requirements including auto-scaling, fast startup times, and low memory usages, running on heavily virtualized and sandboxed infrastructure, for applications demanding increased efficiency for greater cost savings.
The JVM doesn't have a solution for these problems, and while Oracle can solve this problem, it's not clear they will. There is obvious tension between Oracle's desire to improve the JVM, and Oracle's desire to make money on the JVM, and some of the pain points in the JVM are already being monetized (see GraalVM Enterprise).
If we were to redo the JVM specification from the ground-up, it might look a lot more like the WASM specification. WASM points to a future where a new operating system or perhaps a single application on an existing operating system runs hundreds or thousands of WASM applications in a completely secure, sandboxed fashion, at low cost, with auto-scaling and low memory usage.
In any case, it's clear to me that sticking with the JVM is an increasingly risky strategy. It ties the fate of Scala to the fate of the JVM. But Scala doesn't need the JVM to be a powerful, compelling solution to building modern applications.
Maybe in the early days, Scala needed the JVM. But no longer.
Scala needs to support alternative platforms, and it needs to do so directly, integrated into the compiler. The JVM platform should not be specialized. Rather, each backend should take an intermediate representation (perhaps TASTY, or perhaps something lower-level) and generate platform-specific code, sharing optimization passes where it makes sense to do so.
A huge liability in supporting multiple platforms is the Scala ecosystem's heavy reliance on the JDK. Both Scala.js and Scala Native have to re-implement the JDK, which is a monstrous task to do exhaustively and correctly. Moreover, it's not even really necessary, since an increasing percentage of Scala applications are not written to the JDK, per se, but to Scala's own open source ecosystem.
In order to enable low-cost, bulletproof cross-platform support, Scala needs a low-level, cross-platform library that provides the bare-bones essentials for platform support: including threads, files, sockets, processes, and rudimentary time functionality (one should look to various preliminary WASM specs for inspiration).
Scala's vast OSS ecosystem would then adopt this cross-platform library to power their features.
In this fashion, Scala applications depending on Scala's own open source ecosystems would instantly become compatible with all of Scala's supported platforms, to the maximum extent possible. In addition, there would cease being such a thing as "cross-platform libraries" (which are hell to maintain), because all libraries would always be fully cross-platform.
Such a cross-platform library could be built by volunteers, and architected and overseen by leaders from all the major OSS ecosystems in Scala (since ultimately, they would be its direct users, and critical to weaning Scala applications off the JDK).
As CEO of Ziverge and a core contributor to ZIO, I am happy to play a role in developing a low-level cross-platform library, but we need key stakeholders from other ecosystems and companies in the space to rally behind the idea.
As the Scala programming language and broader ecosystem is not a commercial product, there is no single entity who can fund development and maintenance of critical infrastructure. Although in some cases, I've volunteered my own time and resources, or those of my company, ultimately, in order to address all of these challenges, we need much broader participation.
Fortunately, there are many organizations who can and do contribute to Scala. The involvement of some of these organizations will be crucial.
LAMP is the lab at EPFL that is run by Martin Odersky, and which contributes directly to Scala 3 and the broader ecosystem, including open source and tooling.
LAMP receives its funding from grants (such as the Advanced Grant program from the Swiss National Fund), which are targeted at research and development in specific areas of programming methods.
Due to the way that grants to LAMP work, there is pressure on projects to produce research that is academically novel. Yet, academic novelty is frequently at odds with the needs of industry: commercial developers want languages, tooling, and open source to solve practical problems in straightforward ways, without regard to academic novelty, theory, or value to the research community.
Beyond this basic tension, many projects coming from LAMP are spearheaded by PhD students, who, when they obtain their doctorate, abandon their project and accept a job in industry, creating a steady stream of half-baked abandonware that never realizes its full promise in industry.
Despite these drawbacks, there is no doubt that LAMP has powered tremendous positive change for the Scala programming language, including the Dot calculus which brings a sounder foundation to Scala's advanced type system. The students and assistants working at LAMP are quite brilliant and hard working, but these traits do not solve the tension between academic research and industry needs, nor address the finite tenure of students.
To improve upon its legacy, I believe LAMP could benefit from a new fusion of industry and academia:
By directly involving industry in the development and maintenance of projects, the work that LAMP does will be more relevant than ever, have a much broader impact on commercial software development, and result in long-lasting benefits for the entire Scala community.
The Scala Center is a not-for-profit organization backed by EPFL and corporate sponsors, dedicated to open source and education around the Scala programming language. The organization is closely connected to and shares some of the same leadership and team as LAMP.
As with LAMP, the Scala Center has made many positive contributions to the Scala programming language and ecosystem. Originally formed to fill the void in the Scala ecosystem after Lightbend (Typesafe) pivoted away from Scala, the Scala Center has contributed to Scala.js, Scala Metals, Scalafix, Scalafmt, and many other projects.
While numerous smart and talented engineers have worked at Scala Center on important projects, the organization itself has had major challenges, including:
While these problems are not particularly hard to solve, they do require effective organization and management, collaboration with industry partners and open source contributors, and a commitment to long-lived and boring (but essential!) open source maintenance.
Instead, at a time when Scala tooling has fallen behind because of Scala 3, the Scala Center chose to spend resources building an inclusive language guide. Personally, I don't know any Scala developers who would prefer an inclusive language guide over a working IDE.
Worse still, at a time when the language, tooling, and cross-platform support require more financial investment than ever before, the Scala Center rejected a generous offer of corporate sponsorship from Ziverge.
As a CEO, I know how to fix these problems, but unfortunately, there is nothing any of us can do because there is no democratic or open mechanism for influencing the organization, management, or priorities of the Scala Center.
In the best case scenario, the Scala Center continues to operate as before, which means the Scala community gets some value from the organization, but cannot rely on the Scala Center to move the needle of adoption or completely solve any of the challenges Scala has.
The main commercial organizations involved in contributing to the greater Scala ecosystem include:
Given the limited resources of LAMP and the Scala Center, some of these organizations (in addition to many individual contributors) must be a part of the solution to challenges facing the Scala programming language.
As a Scala enthusiast and investor, I am keen on increasing Scala's adoption in industry. While Scala has powerful features, it also has some challenges, which prevent Scala from reaching its commercial potential.
I believe that it is time for all of us Scala enthusiasts to take action to improve the following areas:
Given that Scala is not a commercial project, addressing these deficiencies will require coordinated effort across individual open source contributors, LAMP, Scala Center, and organizations like Ziverge. It may require organizational and operational improvements, and will most certainly require new levels of collaboration, as a diverse set of commercial and academic users come together for a common purpose.
Through discipline, hard work, and creative thinking, I believe we can create a new era in Scala's commercial adoption—a true Scala resurrection.
Stay tuned for more!
Stay ahead with the latest insights and breakthroughs from the world of technology. Our newsletter delivers curated news, expert analysis, and exclusive updates right to your inbox. Join our community today and never miss out on what's next in tech.
