At work, I just finished getting our in-house libraries to cross-compile to Scala 2.13 and 3.3. Of course, at some point I encountered a macro, and since the macro system got a complete overhaul in Scala 3, I needed to rewrite that.

I’ve had almost no contact with Scala 2 or 3 macros before, so I wasn’t quite looking forward to the task. Co-Pilot produced some good looking code, which had only one compilation error. However, after trying to get it to work for a few hours, I realized, that the code was rubbish. (See, AI won’t come for our jobs just yet ;) )

The Task

Goal

This particular macro is intended to generate test cases for translation keys. You define an object with a bunch of translation keys of type TranslationKeyN[M, I1..IN], where N is the arity of the key (how many interpolation parameters it has), M is a marker trait and I1..IN the types of the interpolation values. This is to make sure, that each key has a translation with the appropriate number of parameters.

Steps

Alright, so we have a test class with an overloaded method, one for each arity. Our users call the macro with some container object C, which contains the keys for some marker trait (i.e., the keys that belong grouped together, e.g., for some email).

We need to:

• Enumerate the members of C
• Filter the public value definitions (vals)
• Filter the ones of the right type (TranslationKeyN)
• Generate an expression calling the right overload

The details of the implementation look a bit complex, but I found it an interesting example for a macro, because we deal with reflection (enumerating members), calling methods on instances, overloaded methods even, type arguments and even contextual abstractions (implicits/using/given).

A First Try

This article is not going to be an in-depth introduction to Scala 3 macros. But here are some helpful resources:

• The official Tutorial
• Scala 3 macros tips & tricks (Softwaremill)
• lampepfl: dotty-macro-examples
• Actual Scala 3 library source code (this is, probably, the most useful one!)

Alright, so here is an outline of our TranslationTest class:

class TranslationsTest[A]:
  def testTranslationKey(key: TranslationKey0[A]): Unit =
    testKeyInternal(key.value)(key()(_))

  def testTranslationKey[I1: Arbitrary](key: TranslationKey1[A, I1]): Unit =
    testKeyInternal(key.value)(key(random[I1])(_))

  // Provide more overloads...

  inline def testAllTranslationKeys[C](container: C): Unit =
    ${ macroImpl[A, C]('this, 'container) }

The method testKeyInternal will use the translation bundles to generate a table driven test. But in my example, it just does some println-ing. Using the random function, we generate some random input for the interpolation arguments, i.e., for a key “Welcome ” of type TranslationKey1[User, String], we need an Arbitrary[String] instance to generate some random value. This comes from scalacheck.

In the original code, TranslationsTest serves as a base-class for tests and a call to the inline method will generate the test cases there.

The method is the entry point to the macro and contains a splice with the call to the macro. We are passing quoted references to this and the container object with the keys. (extra credit: you don’t need to pass this, because you can get a reference to the owner of a splice. In my case, it wasn’t necessary, because this is library internal code and the user facing code would be the same.)

Let’s give the actual macro a try now. The implementation has to be in a different compilation unit - in my example I put it into Macro.scala:

import scala.quoted.*

def macroImpl[T, C](
    testsExpr: Expr[TranslationsTest[T]],
    container: Expr[C]
)(using quotes: Quotes, tTpe: Type[T], cTpe: Type[C]): Expr[Unit] =
  import quotes.reflect.*

  val tkTypes = List(
    TypeRepr.of[TranslationKey0[T]],
    TypeRepr.of[TranslationKey1[T, _]],
    TypeRepr.of[TranslationKey2[T, _, _]],
    TypeRepr.of[TranslationKey3[T, _, _, _]],
    TypeRepr.of[TranslationKey4[T, _, _, _, _]]
  )

  def isTranslationsKey(t: TypeRepr): Boolean =
    tkTypes.exists(t <:< _)

  val containerType: TypeRepr = TypeRepr.of[C]
  val containerTerm: Term = container.asTerm

Let’s have a look at the start of the function. We have our type arguments for the maker type T and the container type C. The first parameter is our expression containing this and the second one our container object. We need to use Quotes because this is a macro and we need instances of Type[A] for any type A we want to reference in our quote. The way I understand this, this is due to the JVM using type erasure for generics and if we want to actually do something with the type, it needs to be reified.

Alright, so the rest is mostly machinery to check that a member value has the type we are looking for. Then we get a type representation of C, which we can use for reflection and a term of the container, so we can use that to access the actual members of it.

Moving on:

val calls = containerType.typeSymbol.fieldMembers.map { symbol =>
  // Val definition is `val name: tpt = _rhs`
  case ValDef(name, tpt, _) if isTranslationsKey(tpt.tpe) => {
    val field: Term = Select(containerTerm, symbol) // container.`name`
    Some('{$testsExpr.testTranslationKey($i{field.asExpr})})

  case _ => None
}.flatten

We just get the value definitions that match and access the value on the actual container, passing it to the function we want to call in quoted code. Easy.

The Actual Solution

Unfortunately, this doesn’t work…

I was hoping it would just figure out the correct overload, but the expression we get from the term field is just an Expr[Any]. I tried a few things to get this to work, but then I gave up and just constructed the AST manually, applying the type arguments, arguments and resolving the givens (implicits):

  // Map over all member fields of container type,
  // searching for all translation keys.
  val calls = containerType.typeSymbol.fieldMembers.map { symbol =>
    symbol.tree match
      // Val definition is `val name: tpt = _`
      case ValDef(name, tpt, _) if isTranslationsKey(tpt.tpe) => {
        val field: Term = Select(containerTerm, symbol) // container.`name`
        val interpolationTArgs = tpt.tpe.typeArgs.tail // I1, I2, I3, ...

        // Resolve the needed `testTranslationKey` overload
        val callOverload = Select.overloaded(
          testsExpr.asTerm,
          "testTranslationKey",
          interpolationTArgs,
          List(field)
        )

        // Resolve givens/implicit for Arbitrary value generation
        val arbitraryTpe =
          TypeIdent(Symbol.requiredClass("org.scalacheck.Arbitrary")).tpe
        val implicits = interpolationTArgs.map { t =>
          Implicits.search(arbitraryTpe.appliedTo(t)) match
            case success: ImplicitSearchSuccess =>
              success.tree
            case failure =>
              report.errorAndAbort(s"Could not find implicit: $failure")
        }
        val res =
          if (implicits.nonEmpty) Apply(callOverload, implicits)
          else callOverload

        Some(res.asExpr)
      }
      case _ => None
  }.flatten

We need the final conditional, because at arity 0 (TranslationKey0) we don’t have any implicits to apply.

Finally, we return a code block with all the calls:

// Block expression with all calls
Expr.block(calls, '{ () })

You can find the whole code in this GitHub repository.

Conclusion

Scala 3 macros are very powerful, but it takes some time getting used to thinking in this “meta” way. Furthermore, there aren’t a ton of resources online about this topic. Maybe this article can help someone trying to solve a similar problem.

If you have any suggestions on how to improve this macro, or generals suggestions, as well as questions, just contact me via Twitter or the GitHub discussion for this post.

We have an internal library and sbt plugin, which pins many library versions and provides some base functionality. The latest release features several larger updates, most notably Play 2.9.

However, there is a lot of churn related to library updates. We use the scala-steward Github action to keep our services up-to-date, but when there are code changes, incompatible versions, etc. a human has to intervene.

This new update in particular had a few mechanic changes related to Play 2.9 and a newer Tapir version. So to make this update less painful for my colleagues, I decided to invest some time and figure out how to build scalafix migrations.

I was fantasizing about an automatic process, in which this update would get applied, merged and just magically work, without anyone taking much notice. Alas, my expectations were soon dampened, but I did not expect it to go as badly as it did… Anyway, I still believe that this is an awesome tool and the investment was worthwhile and will come in handy in the future.

This report is intended as a supplement to the tools’ documentation and might help others in their journey to automatic scala migrations - and maybe a reader will have some suggestions for me.

The Project Setup

The Tools

For our scala-steward Github action to run migrations, we first have to write them. The tool used to create and apply linters and migrations is scalafix. It is based on scalameta, a library to represent and query scala programs as syntax trees, as well as storing semantic information, like types and symbols, in a database format (SemanticDB).

The Requirements

I did not set out to write some generally applicable scalafix rules, which should be published for anyone to use. The migration should be very specific to our company’s needs. Our artifacts are published to our own, private Sonatype Nexus Repository and our code is in our private GitHub Enterprise organization.

So I have to work with these constraints. That means, that all the tools need to be able to resolve artifacts from the private repo and GitHub resources cannot be accessed without authentication.

Project structure

Scalafix allows you to use predefined rules (both internal and community release ones), but we don’t care about this for now.

The developer guide tells us about our options to setup the project. You might want to have the rules as a sub-module in your project’s sbt build, or as a separate project all together.

Rules can be consumed in different forms by scalafix. You can publish them as an artifact, like any other scala code. Or you can consume their source code and let scalafix compile them on the fly. Sources can be read via the file:, http: and github: protocols. The tutorial contains some more information on this.

I chose to add them to the existing libraries project, as a sub-module. The migration is split up into several rules, each one taking care of a specific change. Rules are published to our Nexus repository as a compiled artifact.

Using the github: or http: schemes, I would have had to publish the rules on the public internet, e.g., in a gist on my personal GitHub account, because those do not work with authentication. That’s a big no-no.

So the folder structure started out like in the tutorial. A scalafix folder in my project folder and inside the input, output, rules and tests folders. The default package name for rules you find in the tutorial (and basically all rules you find linked) is fix. As I thought this is just the first migration of many, and all of them will live in this project, I named my package v2_5 - because it is the migration to version 2.5. If you do that and you use the github: scheme to run rules from source, be aware that you have to specify the package path.

Also make sure to use the correct package name in the resources/META-INF/services/scalafix.v1.Rule file.

The input directory holds your test files, which should be rewritten. After applying a rule from rules, the result should match the corresponding file in output. All of this is orchestrated by the test runner in tests.

Cross Compilation

In our case, I wanted to rewrite the Dependencies.scala of our sbt build. One of the changes was renaming an artifact. Tapir upgraded from Play 2.8 straight to Play 3.0, which replaces Akka with Pekko, in a bugfix release¹. So I needed to change that to the newly created play29 modules tapir-play29-server, etc.

To test this rule, the input file needs some sbt dependencies (e.g., for the %% functions). Sadly, sbt is still stuck on Scala 2.12. So I needed to support Scala 2.12 and 2.13 inputs (I skipped Scala 3 for now).

I was struggling quite a bit with the correct setup, so here the short version for you. I’m building the scalafix rules themselves with Scala 2.13 and the inputs and outputs are cross-compiled. Dependencies are managed on a per-Scala-version basis. Here is a slightly edited version of the relevant sections from build.sbt:

val sbtScalaVersion       = "2.12.18"
val scala2_13             = "2.13.12"
val scalafixCrossVersions = sbtScalaVersion :: List(scala2_13)

lazy val `scalafix-input` = (project in file("scalafix/input"))
  .settings(
    crossScalaVersions := scalafixCrossVersions,
    evictionErrorLevel := Level.Warn,
    publish / skip := true,
    libraryDependencies ++= {
      CrossVersion.partialVersion(scalaVersion.value) match {
        case Some((2, 13)) =>
          Seq(
            "com.softwaremill.sttp.tapir" %% "tapir-play-server"     % "1.8.2",
            "com.typesafe.play"           %% "play-slick"            % "5.0.2",
            "com.lightbend.akka"          %% "akka-persistence-jdbc" % "4.0.0"
          )
        case Some((2, 12)) => Seq("org.scala-sbt" % "sbt" % "1.9.8")
        case _             => Seq.empty
      }
    }
  )
  .disablePlugins(ScalafixPlugin)

lazy val `scalafix-output` = (project in file("scalafix/output"))
  .settings(
    crossScalaVersions := scalafixCrossVersions,
    publish / skip := true,
    libraryDependencies ++= {
      CrossVersion.partialVersion(scalaVersion.value) match {
        case Some((2, 13)) =>
          Seq(
            "com.softwaremill.sttp.tapir" %% "tapir-play29-server"   % "1.9.8",
            "com.typesafe.play"           %% "play-slick"            % "5.2.0",
            "com.lightbend.akka"          %% "akka-persistence-jdbc" % "5.3.0"
          )
        case Some((2, 12)) => Seq("org.scala-sbt" % "sbt" % "1.9.8")
        case _             => Seq.empty
      }
    }
  )
  .disablePlugins(ScalafixPlugin)

lazy val `scalafix-rules` = (project in file("scalafix/rules"))
  .settings(commonSettings)
  .settings(
    name               := "underpin-scalafix-rules",
    scalaVersion       := scala2_13,
    crossScalaVersions := scalafixCrossVersions,
    scalacOptions += "-Ywarn-unused",
    libraryDependencies +=
      "ch.epfl.scala" %%
        "scalafix-core" %
        _root_.scalafix.sbt.BuildInfo.scalafixVersion
  )
  .disablePlugins(ScalafixPlugin)

lazy val `scalafix-tests` = (project in file("scalafix/tests"))
  .settings(
    crossScalaVersions := scalafixCrossVersions,
    publish / skip := true,
    evictionErrorLevel := Level.Warn,
    scalafixTestkitOutputSourceDirectories :=
      (`scalafix-output` / Compile / sourceDirectories).value,
    scalafixTestkitInputSourceDirectories :=
      (`scalafix-input` / Compile / sourceDirectories).value,
    scalafixTestkitInputClasspath :=
      (`scalafix-input` / Compile / fullClasspath).value,
    scalafixTestkitInputScalacOptions :=
      (`scalafix-input` / Compile / scalacOptions).value,
    scalafixTestkitInputScalaVersion :=
      (`scalafix-input` / Compile / scalaVersion).value
  )
  .dependsOn(`scalafix-input`, `scalafix-rules`)
  .enablePlugins(ScalafixTestkitPlugin)

This means that I have all the sources for the rules in the usual src/main/scala folder, but the input and output sources are separated in src/main/scala-2.12 and src/main/scala-2.13 respectively.

Writing Rules

This is the actual fun part! Two important concepts to understand, is the Syntax Tree and Tokens. The former one is a tree representation of your program, which you will write matchers against. Tokens are the building blocks of your program and represent keywords, identifiers and even indentation etc. We typically want to modify these.

The scalafix guide describes how to write rules and there are many existing rules you can learn from.

Syntactic vs. Semantic Rules

First, we need to decide whether we want to write a syntactic or semantic rule.

A syntactic only needs a parsed program, i.e., the program structure and won’t need a semanticDB with resolved types and symbols. I’d recommend to use syntactic rules whenever possible. They are faster and they work on programs with (semantic) compilation errors. For example, you update some dependencies and multiple changes are necessary to get the program to compile again. I might be biased, because I had huge problems with semantic rules in this regard. You might also get a stale semanticDB error with semantic rules, if the code was modified but not compiled/a new semanticDB generated in the meantime.

Syntactic rules are, however, less powerful. For example, if your code contains unqualified uses of two types with the same name, e.g., com.someframework.Url and org.other.Url, your rule might only see an identifier Url. It’s up to you to make sure you are rewriting the correct ones. This might be easy, if you know your codebase and you can ascertain that such situations won’t come up, simply by grepping through your code base.

Semantic rules, on the other hand, have access to the resolved types and symbols. So you can, e.g., use scalafix’ SymbolMatcher to look for occurrences of com.someframework.Url. Depending on how you run your scalafix rules, you’ll need to set your build to generate semanticDB files, e.g.,

semanticdbEnabled := true, // enable SemanticDB
semanticdbVersion := scalafixSemanticdb.revision, // only required for Scala 2.x

What to Match

I’m typically starting out by building a minimal example of what I want to rewrite (as input) and what I would like it to look like afterwards (the output). If you can think of any edge cases, or similar looking code you don’t want to rewrite, put that into the test as well.

Then paste the code into the Scalameta AST Explorer, to get an idea what the syntax tree looks like. Besides that, I found looking at the Scalameta source for Tree nodes helpful to find out, what nodes I can match against.

Other than that, I start by writing my matchers and placing plenty of printlns inside, to orient myself (and see how wrong I got things).

A Concrete Example

Let’s start with a syntactic rule. Tapir’s DecodeFailureHandler has changed.

What used to be

trait DecodeFailureHandler {
  def apply(ctx: DecodeFailureContext): Option[ValuedEndpointOutput[_]]
}

got a type argument for an effect type:

trait DecodeFailureHandler[F[_]] {
  def apply(ctx: DecodeFailureContext)(implicit monad: MonadError[F]): F[Option[ValuedEndpointOutput[_]]]
}

So I took a look at our actual DecodeFailureHandlers and made this test input:

/*
rule = FailureHandler
*/
import sttp.tapir.server.interceptor.decodefailure.{DecodeFailureHandler, DefaultDecodeFailureHandler}
import sttp.tapir.server.model.ValuedEndpointOutput
import sttp.tapir.server.interceptor.DecodeFailureContext
import scala.concurrent.ExecutionContext

class FailureHandler extends DecodeFailureHandler {

  private def renderResponse(): ValuedEndpointOutput[_] = ???

  override def apply(ctx: DecodeFailureContext): Option[ValuedEndpointOutput[_]] = {
      DefaultDecodeFailureHandler.respond(ctx) match {
        case Some((sc, hs)) =>
          Some(renderResponse())
        case None =>
          None
      }
    }
}

class DifferentClass {
  // Dummy to keep EC import
  def bar(ec: ExecutionContext) = ???

  // Don't rewrite this
  def apply(ctx: DecodeFailureContext): Option[ValuedEndpointOutput[_]] = {
    ???
  }
}

As this rule is intended for our services using Play, we’ll use Future as our effect type. The second class was added, to make sure my rule wouldn’t rewrite similar looking code. The ExecutionContext import was added, because my first draft messed up some code which already had the EC import.

And what is our desired output? We need some imports (potentially), substitute F with Future, add the implicit MonadError and wrap the result in a Future:

import sttp.tapir.server.interceptor.decodefailure.{DecodeFailureHandler, DefaultDecodeFailureHandler}
import sttp.tapir.server.model.ValuedEndpointOutput
import sttp.tapir.server.interceptor.DecodeFailureContext
import scala.concurrent.ExecutionContext
import scala.concurrent.Future
import sttp.monad.MonadError

class FailureHandler()(implicit ec: ExecutionContext) extends DecodeFailureHandler[Future] {

  private def renderResponse(): ValuedEndpointOutput[_] = ???

  override def apply(ctx: DecodeFailureContext)(implicit monad: MonadError[Future]): Future[Option[ValuedEndpointOutput[_]]] = Future {
      DefaultDecodeFailureHandler.respond(ctx) match {
        case Some((sc, hs)) =>
          Some(renderResponse())
        case None =>
          None
      }
    }
}

class DifferentClass {
  // Dummy to keep EC import
  def bar(ec: ExecutionContext) = ???

  // Don't rewrite this
  def apply(ctx: DecodeFailureContext): Option[ValuedEndpointOutput[_]] = {
    ???
  }
}

Great. Let’s start with the rule:

package v2_5

import scalafix.v1._
import scala.meta._

class FailureHandler extends SyntacticRule("FailureHandler") {
  override def isRewrite: Boolean = true
  override def description: String = "You can add a description, if you like"
  override def fix(implicit doc: SyntacticDocument): Patch = ???
}

The fix method is where the magic happens. As a starting point, we want to match on a class definition, which extends sttp.tapir.server.interceptor.decodefailure.DecodeFailureHandler. I’m cutting some corners here. Syntactic rules are simpler, as mentioned above. I am able to grep our entire code base, so I know, that we don’t have any other DecodeFailureHandler this one could be confused with. This is actually a very specific scenario and I know very well which files this should affect. So I can get away with a syntactic rule, just matching on “some class” extending something named “DecodeFailureHandler”. You could use scalafix’ Symbol to lookup a class and its parents. But that needs semantic information.

Let’s match on a class definition:

override def fix(implicit doc: SemanticDocument): Patch = {
    val clazz = doc.tree.collect {
      case c @ Defn.Class.After_4_6_0(
            _,
            name,
            params,
            Ctor.Primary.After_4_6_0(_, _, ctorParams),
            Template.After_4_4_0(_,
                                 List(Init.After_4_6_0(Type.Name("DecodeFailureHandler"), Name.Anonymous(), Nil)),
                                 _,
                                 stats,
                                 _)) if params.values.isEmpty && ctorParams.isEmpty => ???
        // More to come...
  }
  ???
}

Note that the fix method returns a Patch, which represents the changes we are performing. This type is very important, you’ll need those methods to generate the changes you want.

Inside the method, we apply our partial function to the syntax tree with collect. The Defn.Class node represents a class definition. What about the ugly After_4_6_0? Scalameta uses these versioned matchers for backwards compatibility. This is a great thing, otherwise every new version might break your rules.

The class definition contains a bunch of stuff: Defn.Class(modifiers, name, paramClause, ctor, template). This Template structure is very confusing, but contains lots of things we care about. Looking at this in the AST explorer brings some clarity.

Note that this matches syntactically, i.e., this matches only if the class only extends DecodeFailureHandler. If you test for multiple super-classes, you’ll have to search the list of Inits (or use a semantic rule). I’m also testing for “no type parameters” and “empty constructor”, because that’s what I see in our code base. Your mileage may vary.

Now I want to modify the class definition, by adding an implicit argument and by adding [Future] to DecodeFailureHandler and by wrapping Future.apply around the method body. I know that all source files I want to rewrite use curly braces around the body, so I can get away with simply placing Future before the opening brace {. This means I want a patch that manipulates the tokens of the tree node we just matched:

// continued in the match above
val tokens     = c.tokens  // Tokens of the class definition
val identifier = tokens.find(_.is[Token.Ident]) // First identifier should be the name of declared class
val traitName  = tokens.find { case Token.Ident("DecodeFailureHandler") => true; case _ => false } // Find the token in the `extends` clause

Patch lets us add text left or right of tokens, replace tree nodes, remove tokens, etc. So given FailureHandler (remember, no type parameters, no constructors), we want FailureHandler()(implicit ec: ExecutionContext). We can just replace the identifier token: Patch.replaceToken(identifier.get, s"${name.value}()(implicit ec: ExecutionContext)")². Yes, we’re playing fast and loose with Options here, but I’m sure that token exists, since we matched the tree node.

Adding the type argument to DecodeFailureHandler is even easier: Patch.addRight(traitName.get, "[Future]").

How do we get to the apply method? See this stats field in the class definitions template above? These are the statements that make up the body of the class. We need to match on those statements, to find the right method definition:

  stats.collect { case m @ Defn.Def.After_4_7_3(mods, Term.Name("apply"), _, Some(tpe), body) =>
    val modsStr = mods.mkString(" ")
    Patch.replaceTree(
      m,
      s"$modsStr def apply(ctx: DecodeFailureContext)(implicit monad: MonadError[Future]): Future[$tpe] = Future $body")
  }.asPatch

Again, I can simplify things a bit, because I know what the code looks like, on which this rule will be applied. I’m simply replacing the whole tree node, instead of fiddling around with tokens.

These three patches need to be assembled into one, e.g., like so: Patch.fromIterable(Seq(p1, p2, p3)).

The last piece missing is the imports. I know that some of the sources have some of the required imports, so I don’t want to add them unconditionally (although this could be cleaned up with another scalafix rule, RemoveUnused), but the MonadError import is never present:

  val imports = Patch.addGlobalImport(importer"sttp.monad.MonadError") +
    (if (hasImport(doc.tree, "Future")) Patch.empty
      else Patch.addGlobalImport(importer"scala.concurrent.Future")) +
    (if (hasImport(doc.tree, "ExecutionContext")) Patch.empty
      else Patch.addGlobalImport(importer"scala.concurrent.ExecutionContext"))

The hasImport method just matches on the tree, looking for Import statements.

Since we only want to add the imports, if this file actually contains a failure handler, we assemble the final patch conditionally:

if (clazz.isEmpty) Patch.empty else (imports + clazz)

The complete rule can be found in this gist.

Semantic Rules

Additionally to what you can do in a syntactic rule, you may also use symbol matchers and so on.

You can use a symbol matcher in a pattern matching a tree node. For example, if we want to match an element in a parameter list of type play.api.ApplicationLoader.Context, we can do this like so:

  private def patchContextParam(paramClause: Term.ParamClause)(implicit doc: SemanticDocument) = {
    val contextMatcher = SymbolMatcher.exact("play/api/ApplicationLoader.Context#")

    paramClause.values.collect { case p @ Term.Param(_, name, Some(contextMatcher(tpe)), _) =>
      Patch.replaceTree(p, s"$name: $tpe")
    }.asPatch
  }

We place the matcher contextMatcher(_) where want to match a tree node in our match expression. The name in parenthesis is bound to the matched tree node.

Applying Rules

This is where things got complicated for me.

Private Repos

As outlined in the beginning, we need to publish rules to a private repository. This simply uses our normal release process. It just pushes some jar files to Nexus.

When consuming rules from a private repository though, you need to add a resolver. We have an sbt resolver configured for our normal dependencies, but we need an additional scalafixResolver.

If you want to use the rules from your sbt project, add the resolver to the build:

ThisBuild / scalafixResolvers += coursierapi.MavenRepository.of("https://private.reposit.ory")

Since I created these migrations for use with scala-steward, this is not necessary (I will explain scala-stewards mechanism soon). But it is useful for testing and applying rules manually. You can also add it to the current sbt session only, by typing set ThisBuild / scalafixResolvers += coursierapi.MavenRepository.of("https://private.reposit.ory") in the sbt console.

Sbt vs. scalafix-cli

When run from the sbt build, the scalafix rules can’t rewrite the sbt build files themselves. Like that Dependency.scala file I wanted to rewrite, remember? At least, I haven’t found a way to achieve this.

But there is a scalafix CLI tool, which can be applied to any file.

This is especially useful for standard rules and those published via http/GitHub. To use artifacts from a private repository, you need to resolve them yourself. Luckily, that is quite simple with coursier:

scalafix \
  --tool-classpath $(cs fetch my.org::my-scalafix-rules:2.5.0 -p) \
  -r TapirPlay29Dependencies \
  ~/myproject/project/Dependencies.scala

You may also want to use the --scala-version flag. I wrote and published my rules in Scala 2.13, which is the current default.

One important thing to note is the double semicolon :: in the artifact coordinates. This works similar to sbt’s %% for cross-compiling. Many of the existing rules and examples use only one semicolon. Apparently, this is from an older version, which didn’t support cross-publishing. You might have to add the version suffix (“_2.13”) if you use that.

Scala-Steward

The star of the show! Let’s put everything together to realize my dream of automagic upgrades!

Try Before You Buy!

The good thing about reading blog posts about things is, that you don’t have to go through the mistakes the author has gone through in order to write this.

When scala-steward opens a PR on GitHub - that’s it! If you mess up the config, or your rules don’t work quite like you wanted, you don’t get a second chance. You can’t delete PRs, only close them. Scala-steward will see a closed PR for the given version and that’s that.

So to test your config, you can issue updates against a different branch from main. To make testing simpler, I used the scala-steward docker container.

Here is the magical incantation:

docker run -v /home/ben/Projects/scala-steward-test:/opt/scala-steward \
  -v /home/ben/.sbt:/root/.sbt \
  -v /home/ben/.config/coursier:/root/.config/coursier \
  --env COURSIER_REPOSITORIES="ivy2Local|central|https://my.private.repo" \
  -it fthomas/scala-steward:latest \
  --workspace  "/opt/scala-steward/workspace" \
  --git-author-email "benjamin.maurer@noreply.com" \
  --do-not-fork \
  --forge-login "scala-steward-test" \
  --repos-file "/opt/scala-steward/repos.md" \
  --repo-config "/opt/scala-steward/default.scala-steward.conf" \
  --git-ask-pass "/opt/scala-steward/.github/askpass/pass.sh" \
  --scalafix-migrations "/opt/scala-steward/scalafix-migrations.conf"

What does this mean? I’m mounting the directory /home/ben/Projects/scala-steward-test as /opt/scala-steward in the container. I have all the configuration files in this folder. The folders ~/.sbt and ~/.config/coursier are mounted into the container, so it has my credentials for the private repository (this is not a very secure way of doing things, but for a quick test, it’s OK). The environment variable COURSIER_REPOSITORIES tells coursier which repositories to use to resolve artifacts. My pass.sh is a very janky construct, that just echos my personal GitHub access token.

repos.md contains the repository configuration. It’s a markdown list in the format - org/repo[:branch]. If you don’t specify a branch, it will use main/master.

default.scala-steward.conf the, well, default scala-steward config (can be overwritten with project specific configs). You might want to allow pre-release updates for your artifact, if you are testing migrations before the release: updates.allowPreReleases = [ { groupId = "my.org", artifactId = "my-artifact" } ]

scalafix-migrations.conf contains the list of migrations. I found the documentation of this a bit lacking, but you can take a look at the sources to find out what all the fields mean (I’m planning on improving the docs).

Here an example:

migrations = [
  {
    groupId: "my.org",
    artifactIds: ["my-artifact"],
    newVersion: "2.5.0-RC1",
    rewriteRules: [
        "dependency:FailureHandler@my.org::my-scalafix-rules:2.5.0-RC7",
        "replace:akka.persistence.jdbc.config.JournalTableConfiguration/akka.persistence.jdbc.config.LegacyJournalTableConfiguration"
    ],
    executionOrder: "post-update",
    target: "sources",
    doc: "https://cptwunderlich.github.io/"
  }
]

First you define for which artifact and version the migration applies. rewriteRules contains a list of scalafix rules, with a protocol prefix (http, github, dependency…), unless it is a built-in rule.

Now it gets interesting. executionOrder can be pre-update, i.e., before the artifact version is incremented, or post-update. According to the docs, a pre-update migration could affect the update, so it is resolved again (i.e., you may change Dependencies.scala).

target can be sources or build. This one’s interesting and not well documented. Basically, “sources” is a transformation of the project’s sources, just like the sbt-based scalafix invocation seen earlier. “build” can be applied to the build files. And if we look at the implementation, this actually maps to the sbt-scalafix vs scalafix-cli scenario above. There is a difference though. “source” creates a new sbt project and sets the scalafixResolvers from the coursier resolvers. That’s why you don’t need to set the scalafixResolver in your project for scala-steward - and on the flipside only setting it there won’t make it work in scala-steward.

This brings me to my first big defeat: the “build” variant invokes scalafix-cli, which can’t resolve artifacts from private repositories on its own. The trick above with downloading dependencies with coursier and using --tools-classpath doesn’t work, because scala-steward can’t do this. For now, this just doesn’t work and there is no way to run a shell script or similar. (But maybe we can make this work #3276, #3133). I also tried using an artifact migration for this, but this seems to only be triggered if there is a new version for the artifact to be migrated. It didn’t work either way.

Github Action

I added the scalafix-migration.conf to the root directory of our GitHub repo with the config for the scala-steward GitHub action.

To the job’s steps, I added the checkout action and then I can simply refer to the config file:

jobs:
  scala-steward:
    runs-on: ubuntu-22.04
    name: Launch Scala Steward
    steps:
      - name: Add SBT credentials for Nexus
        run: | # redacted

      - name: Checkout repo so migrations config is available
        uses: actions/checkout@v4
      - name: Launch Scala Steward
        uses: scala-steward-org/scala-steward-action@v2
        with:
          github-app-id: $
          github-app-installation-id: $
          github-app-key: $
          scalafix-migrations: 'scalafix-migrations.conf'

Conclusion

So, did I achieve an automatic upgrade? Far from it. Scalafix and scala-steward are amazing tools and I’m happy I invested the time. I’m sure this will make future migrations easier.

Limitations

There are a few limitations though:

1. I didn’t mention it so far, but you can’t change configuration files this way.
2. Migrations of build files don’t work with private repos - you’ll need to publish to maven central, or put the source on a public GitHub repo or web site.
3. Migrations can fail if the code doesn’t compile, or the semanticDB is stale (and the PR will be opened nonetheless #3273).

I’m not quite sure I understand nr. 3 completely. After upgrading the dependency, the code doesn’t compile anymore. Syntactic rules still work, bc. those are not parser errors, but type errors. It seems like semantic rules don’t work on sources that don’t compile (yet). Maybe bc. scala-steward checks the project out and doesn’t have old semanticDBs lying around? Speaking of which, some users report the error “stale semanticDB found”. This might be the case, if one rule changes the code, but it is not re-compiled - not quite sure.

In general have I seen issue when mixing multiple semantic rules. Sometimes it seems like the first rule works, but then the code doesn’t compile properly so the second rule doesn’t. This is just anecdotal, I have not investigated this. But the built-in replace rule seems never to work for me in combination with other rules. I just went back to using sed where possible.

Documentation

I think the docs can be improved. I had quite the hard start with scalafix. But it might be me. Now in hindsight, the docs make a lot of sense to me. But in the beginning, I didn’t quite get it and more importantly, I couldn’t find a lot of things.

A few words on cross-compilation, or rather which Scala versions to use might be helpful. I wasn’t sure whether rules had to be compiled with the same version as the code they are applied to (seems like they don’t). I also at first thought, that they require Scala 2.12, like sbt does. But the rules I looked at seem to be old. Scala 2.13 is the default, AFAIK.

The scala-steward documentation lacks a description of the config values for migrations. Private repos could also do with some better explanations.

I hope I can contribute this sometime soon.

If you start out with scalafix (and scala-steward), don’t make the same mistake I did - take more notes on which things caused you problems, or where the docs were not clear enough. That will help to improve them!

PEBKAC

Remember those double semicolons? They are important! After all the testing and releasing libraries and preparatory work, I messed up the scala-steward config in a flurry. I had single semicolons and scala-steward tried to resolve the wrong version. So instead of half-applied migrations, I got none…

My colleagues had to make due with a migration guide, containing lots of sed commands and big scalafix incantation.

In the first two parts of this series, I described my motivations for bringing SSA transformation to NCG, what SSA form is and an evaluation of my implementation.

My original implementation used the classic Cytron et al. algorithm. The reason for that was, that a) there was plenty of literature on that (e.g. in “Engineering a Compiler”) and b) that GHC already had functions to calculate the dominators etc. To be honest, I did look at the Braun et al.¹ paper (which is probably the most popular one these days), but I found it a bit confusing. While the base algorithm is quite simple, there are plenty of improvements and additions they describe.

Alas, as my implementation was just way too slow to be practical (up to 30% increase in compile times), I set out to reimplement it with the Braun et al. algorithm.

SSA à la Braun et al.

The basic algorithm is quite simple. Looking at all blocks, all instructions in those blocks, we “remember” the new value for any definition and we look up the current value for any use. The look-up starts out local, checking if there is a definition in the current block and recurses into the block’s predecessors if there is none. If we only have one predecessor, we can simply walk up until we find a definition or multiple predecessors. Otherwise we have to place Phi-nodes, bc. we have merging control-flow.

Phi nodes are placed “defensively”, to mark already visited blocks and their arguments may be added later (incomplete Phis). After adding a Phi’s arguments, it’s checked for “triviality” and removed if it is trivial. A Phi node is trivial, if it doesn’t merge at least two distinct values. The argument list is cleaned of any duplicates and the declaring Phi itself (a possible self-reference). If only one distinct value remains, they Phi node can be replaced with that value. For example: p1 = φ(p1, v2) is trivial and each occurrence of p1 can be replaced with v2. Whereas p2 = φ(v1, p1) actually merges two distinct values and therefore is not trivial.

Incomplete Phis are added for blocks, which are not yet “sealed”. I’ll simply quote the paper¹ here, section 2.3, as this is a very clear explanation:

We call a basic block sealed if no further predecessors will be added to the block. As only filled blocks may have successors, predecessors are always filled. Note that a sealed block is not necessarily filled. Intuitively, a filled block contains all its instructions and can provide variable definitions for its successors. Conversely, a sealed block may look up variable definitions in its predecessors as all predecessors are known.

The algorithm is capable of constructing SSA form as the CFG is being constructed. In NCG, we don’t need this, as instruction selection happens before and we have a complete CFG and filled blocks. I’m still using these two states², because you need to handle loop tail’s edges to the loop head. In that case, you won’t have the definitions from the loop body when you are processing the loop head. To avoid recursing into the whole loop body (and having to figure out when to stop, so you won’t get an infinite recursion), I simply consider blocks “not sealed” if not all of their predecessors are filled. My definition of “filled” is, that all instructions have been processed to find definitions (i.e., the block has been “visited”).

Value Numbering

I did not mention anything about generating new SSA names for definitions in the previous section. In fact, neither does the paper. You see, the algorithm performs a (global) value numbering as a natural side-effect. That is because the algorithm looks for the current value of a variable. This may be a simple value or an expression. Think of it as the right hand side of an expression.

# source
x <- 2
y <- 2
z <- x + y
u <- x + 2

# result
v1 <- 2

v2 <- v1 + v1
v3 <- v2

GVN is an optimization and also the basis for further optimizations.

The paper seems to assumes some expressions like in an AST or some IR. Unfortunately, things aren’t as easy with assembly.

To make it short, there are several challenges: You basically have to model the semantics of any instruction you want to consider for all supported ISAs. After all, you have addq $2,%v1 and you need to know that this means v1 <- 2 + v1, or that xor %v1,%v1 means v1 <- 0, or multiplication/division by bit-shifting. Further more, some instructions have implicit (result) operands, like status registers etc. Most of the time, you won’t simply add constants (immediate values) to virtual registers though. You will have to deal with memory access, like memory access relative to the base pointer, so you will need to keep track of the current offset. Also complex addressing modes accessing memory, e.g., ModR/M-Addressing for x86: mov (%esi,%ebx,4), %edx.

All in all, it seemed too complex. For that reason, my implementation simply generates a new SSA name for definitions of virtual registers.

Removing Trivial Phis

As I wrote earlier, Phi nodes are placed “defensively” and are later checked for “triviality”. If a Phi node is deemed trivial, it can be replaced by its only argument value. In turn, any Phi node having that trivial Phi as an argument could now become trivial:

p1 = φ(p1, v1)
p2 = φ(p1, v1)

# p1 is trivial: p1 -> v1
p2 = φ(v1, v1)

# p2 has become trivial: p2 -> v1

This is conceptually quite simple, but some things have to be considered when implementing this. As this may remove previously inserted (and used!) Phis, there will be instructions using these Phis. In the paper, it looks as if they are using def-use chains to find and rewrite all those uses. We don’t have those and computing them incurs an overhead. Instead, I simply keep a mapping of all renamings (e.g., {p2 -> p1, p1 -> v1, p3 -> v2}) and then perform a second rewriting pass over the code with the “canonized” names (e.g., {p2 -> v1, p1 -> v1, p3 -> v2}).

This also necessitates keeping a Phi dependency graph, to keep track of all Phi users of a Phi node, in case that node becomes trivial and all users have to be updated.

The whole procedure is quite imperative and “mutating” in nature, so I’m making extensive use of the state monad. I’d actually like to see the state be shrunk somewhat, but I’ll look into optimizations in the future.

SSA Flavor and Optimizations

In my previous articles, I wrote that I was using “pruned” SSA - precomputing liveness to make sure to only insert live Phi nodes. The Braun et al. algorithm naturally produces pruned SSA form, as it will only insert Phi nodes “on-demand”, i.e., when it encounters the use of a variable coming from merging control-flow.

As for minimality, - minimal SSA requires that Phi nodes for a variable are only placed in the first block where multiple definitions of that variable meet for the first time - the algorithm produces minimal SSA only for reducible control flow. Section 3.2 contains the description of a post-processing pass to remove any Phi nodes inserted because of irreducible control flow. So far, I have not implemented this though. Looking at some simple benchmarks, I could not see any excess of Phi nodes. I might have to revisit this later though.

The paper also contains some extensions to remove the number of temporary Phi nodes inserted, in section 3.3. It may be worth looking at these to improve performance, but I have not implemented them for now.

Other Implementations

I looked at SPIRV-Tools and Cranelift’s implementations for inspiration. (There is also LLVM, but the whole situation there is much more complicated and I didn’t really dive into that)

I found both implementations very readable, but my situation in Haskell somewhat different, so many things did not apply.

There is also LibFirm, which contains the original implementation by the paper authors. (Alas, it’s written in C and I find navigating the code base quite challenging).

Results

So, how did it turn out? To test my implementation, I made sure that both versions are rebased on master. Then I used “cabaltest” - just a simple script to compile Cabal with provided flags - for comparison. There is a big caveat here: I only did one compile per configuration, but by repeating some of them, I saw quite the variability. I’d say that you should add ~5% error bars mentally.

To my chagrin, I also found out, that my first version seems to perform particularly badly with -O2, but quite OK with -O1. I did my last evaluation only with baseline (no optimizations) and -O2 - which was a bad choice anyway. But it wasn’t all for naught, because the new implementation is still faster and uses way less memory.

By looking at this table you might think “hey! the new version is actually slower at -O1”. I’m not convinced that it is, see my comment regarding the “error bars”. I’ve seen changes of 10 seconds or more across runs, so I’m inclined to say that they are about equal in terms of speed for -O1, where the new version performs much fewer allocations. Either way, I’ll keep the new one anyway and rerunning the benchmarks would take quite a long time. Mind you that the new implementation is absolutely not optimized and I’m sure that there are quite some gains to be had.

I also wanted to look at the number of Phi arguments, to see how I can optimize their representation (currently it’s just a list).

The “count” columns refers to the total number of Phi functions in the compiled modules. Interestingly, they are the same for the three benchmarks I looked at from the “nofib” suite. So there aren’t any superfluous Phi nodes inserted by the new implementation (see minimality above). It’s curious that the old implementation seems to insert more arguments in the last program, as indicated by the higher “max args” number. Those may be duplicate elements, as the argument lists are de-duplicated for the triviality check in the new implementations, but I have not verified this hunch.

Conclusion

I’m somewhat satisfied with the new implementation, but I think there is still a lot of potential for improvement. Especially performance can always be better. I might have to look into the post-processing step to ensure minimality for irreducible control flow later, but for now I have my eyes on liveness analysis. But stay tuned, I might write about that soon.

For those who want to keep track of this effort, the ticket is still the same and this is a new MR.

If you spot any mistakes, omissions or want to provide feedback, comments, questions - you can reach me on Twitter, or via mail with my username at gmail.

In the first part, I explained the motivation to bring SSA-transformation to GHC’s NCG. Here, I want to look at the implementation challenges, decisions and results.

SSA in the Backend

I’ve used pseudo-code in the first part, but generally, you’ll have some intermediate representation (IR) in your compiler, which is in SSA. For example, a three address code (TAC), where each instruction represents an operation, taking two input arguments and one output argument, e.g., IADD x, y, z -> z = x + y.

Modern compilers usually use multiple IRs, adapted to different compilation phases and tasks. They may or may not be in SSA. GHC has been around for a while and has several compilation phases and IRs. I won’t describe them here in detail¹, but they are Core, STG, C–. C– is a low-level, imperative language with an explicit stack. NCG performs instruction selection on C– and spits out machine instructions for the selected platform. So at this point, we’re not dealing with an IR anymore².

There are actually quite a few challenges when trying to implement SSA-form on machine instructions.

Pre-colored Registers

Sometimes, the ISA or ABI requires you to use certain registers in specific places. For example, the x86_64 calling convention requires the passing of function arguments in specific registers. Or think about special purpose registers, like the x86_64 stack registers RBP and RSP.

Instruction selection already emits these physical register references where necessary. Obviously, we can’t promote these to SSA variables and start renaming them willy-nilly. In this case, i.e., for live range discovery, we can simply get a way with ignoring them all together, since we are not moving any code³. (FYI, this will be a common theme in the following subsections.)

Modifying and 2-Address Instructions

Most real world ISAs have the annoying property of not being neat, regular and simple. x86 has many instructions that modify an operand (i.e., in and out operands). Let’s say %v1 denotes a virtual register number 1, then inc %v1 increments %v1, or v1 := v1 + 1. Another example is the two-address addition instruction: add %v1, %v2 is %v2 = %v1 + %v2.

This is clearly not in SSA, as we are producing a value but we can’t assign a new name! One may introduce an intermediate representation to handle this case, constraining the in-out operand pair together somehow - but again, in our use case, we can ignore this!

If we only care about live ranges having unique vreg names, this is OK because the in-out operand pair belongs together anyway. There is no way we could assign different registers here.

Conditional Instructions

Some architectures have conditional (or predicated) instructions, which will only execute, if a specific condition is met, i.e., normally some status bits must be set in a status register. For example, x86_64 offers the cmov, or conditional move instruction. As an example, CMOVE %v1, %v2 is equal to if (x == y) { v2 := v1 }.

32-Bit ARM can basically make every instruction predicated, whereas AARCH64 dialed that down a bit.

I’m not going to discuss their benefits or drawbacks, but since NCG supports architectures with these instructions, we need to take them into account. Again, I’m choosing the most simple route. I consider them non-kill definitions and don’t rename their target.

Further Reading

If you’d like to read more about these challenges, then check-out the SSA Book and “Using the SSA-Form in a Code Generator” by Benoît Dupont de Dinechin.

Implementation

Let’s get down to business. The ticket tracking the implementation can be found here.

Overview

The general workflow, including the new SSA transformation, is as follows and can be found in CmmToAsm.hs:

Instruction Selection -> Liveness Analysis -> SSA Construction -> SSA Destruction -> Register Allocation

For construction, I employed the classic Cytron et al. algorithm⁴ and destruction handled in the simplified fashion outlined in the first part. I’m constructing pruned-SSA, i.e., only insert φ-nodes for values that are actually live and used. This yields fewer φ-nodes and only “productive” ones, that I can use to find all the webs. But to get that information, I’ve plugged the SSA-steps after the existing Liveness Analysis. That is a classic data-flow analysis, i.e., a fixed-point algorithm.

Modelling Data

Since I didn’t want to rewrite large parts of the code generator, I had to stay close to the original data types in a way.

I’m greatly simplifying here, but basically, after liveness analysis, a procedure is list of strongly connected components (SCCs) of basic blocks:

[SCC (LiveBasicBlock instr)]

We have our Live_In-Sets, i.e., sets of vregs live on the entry of a basic block, in the main procedure structure (and we need those to place only live φ-functions). The LiveBasicBlocks contain the ID (or label) of the block and a list of instructions with liveness information (which vregs are read/written/modified, are born/die). SCCs are either acyclic, i.e., just one block, or they are CyclicSCCs, containing a list of blocks. The cyclic SCCs are the top-level loops in a procedure (and may contain more loops).

I’m not really interested in the SCC structure, as I need the Control Flow Graph (CFG) anyway, as it is more complete and lets me access all the connections directly, without having to navigate lists and find jumps “manually”. N.B., the CFG in the backend is currently only supported for x86_64, that’s why I can’t support SPARC or PowerPC. I’m not sure what the status for the new AARCH64 support is.

For the construction algorithm, I need random access, so I’m putting everything into a map. But I need to be able to rebuild the exact same SCC-based structure after destruction, otherwise I can’t reuse the rest of the backend. To achieve that, I’m wrapping each block with information telling me, which SCC it belongs to and I’m using a “deterministic map”, which preserves iteration order:

type BlockLookupTable instr
    = UniqDFM BlockId (SccBits (SSABasicBlock instr))

data SccBits a
    = AcyclicBit a | CyclicBit Int a

data PhiFun = PhiF RegId RegId [RegId]
          -- ^ oldId newId args
          
data SSABasicBlock instr
    = SSABB [PhiFun] !(LiveBasicBlock instr)

For example:

[A -> B -> [C -> D -> E] -> F -> [G -> H]]

turns into:

[A: A, B: B, C: 1 C, D: 1 D, E: 1 E, F: F, G: 2 G, H: 2 H]

Construction

I don’t want to go into great detail here, as better descriptions of the Cytron et al. algorithm can be found in many compiler textbooks (e.g., “Engineering a Compiler”) or online. There may also be better algorithms around (see “Conclusion”).

The gist is, you need to compute the dominance frontiers to know where to place φ-functions and then you need to start renaming definitions in a depth first pre-order walk over the CFG.

One node d in a directed graph is said to dominate a node n, if every path from the start node to n must go through d. Every node d dominates itself, but d is said to strictly dominate n if d dominates n and d ≠ n. The dominance frontier is the set of nodes, that d just doesn’t dominate, i.e., if d dominates an immediate predecessor of n, but does not strictly dominate n itself, it is in the dominance frontier.

Looking at the graph below, we can see that b dominates both c, d and e - to reach them, you must go through b. But c and d do not dominate e (only themselves). e is in the dominance frontier of c and of d and a block, where we may have to place a φ-node. Looking at the graph, we can intuitively see that, as two conflicting variable definitions from c and d would reach e and we’d need to reconcile them.

Whether we actually need a φ-function here, depends on whether there really was a live variable at that point in the original program. If we bother to check that, we get pruned-SSA. Otherwise it would still be called minimal-SSA, as that only requires that we place φs at join points and not everywhere (which would lead to very pointless and inefficient maximal SSA).

For the renaming part, you need to manage some state and stacks of current names for the current “level” of the CFG you are on. This has actually proven to be the most expensive part of the whole algorithm.

If you are interested in details, check out the relevant chapters in “Engineering a Compiler”, the code or my bachelor’s thesis.

Evaluation

To evaluate my changes, I let them loose on GHC’s nofib benchmark suite, looking both at spill statistics and perf numbers. I compared both the linear scan and the graph coloring allocators with and without SSA transformation.

Graph coloring register allocators use a heuristic to decide, which live range to spill once no more free registers are available. -fregs-graph uses a very simple one - it just spills the longest live range. This works surprisingly well (I tested it against some other ones, before my SSA implementation), but my hunch is, that this has something to do with vregs representing disjoint live ranges, making them longer on average. Therefore, I also implemented a simple Chaitin-style spill heuristic. Looking at more sophisticated spill heuristics in combination with SSA-transformation may be worthwhile.

Unique vregs

The point was to verify my theory, that most programs contain disjoint live ranges with the same vreg name. The median increase in unique vreg names, across all benchmarks, is 5.44%, with the single largest increase being 21.04%. I looked at the results for the real group in detail and all programs showed an increase, except for the tiny programs in real/eff.

Spills - The Good

Spills are both loads and stores. But register-register moves (reg-reg moves) are also noteworthy. Even though they are a metric used in papers, their cost seems to have gone down quite a bit over the last decades. Most modern desktop CPUs contain a much larger register file internally, than just the architectural registers (16 for x86_64). So they can apply register renaming in their execution pipeline. It’s like switching the label in a shelf, instead of moving the merchandise lying there.

I’ve also added SPILLS*, where I am excluding never spilling programs. Since I’m looking at just the relative changes in spills, programs that spilled zero times across all runs only distort the picture.

The changes for the linear scan allocator are less impressive:

Runtime - The Bad

OK, so reducing spills is great and all, but what really matters are actually executed instructions. After all, you could remove all spills from rarely executed straight line code and it wouldn’t add up to anything, compared to a hot loop containing spill code.

The results here are not overwhelming. I’ve added all three columns to the same table, but the graph-* columns are measured against -fregs-graph baseline, whereas ‘linear-ssa’ is measured against the linear scan allocator baseline. The numbers are relative changes in CPU cycles as reported by perf.

So what does this mean? Nofib-compare gives you the geometric mean. I wanted to explore some other averages, because there is, unfortunately, a huge difference in runtime of the benchmark programs. That is not intentional, they are supposed to have similar lengths, but they drift over time. I included the weighted average (weighted by fraction of total runtime), because there are many programs that run for a few milliseconds on one end of the spectrum and one benchmark takes 58 seconds to complete on my machine. You just can’t compare that. The short program is so much more sensitive to interferences and 5% change of 200ms vs 58000ms can’t be lumped together, IMHO.

Alright, so what does that leave us with? If the results are correct, we might expect around 1% improvement for -fregs-graph and basically nothing for linear scan. Depending on the cost, I’d take 1%, since for compiler engineers working on mature compilers, 1-2% is already a win. I tried to drill down a bit in my thesis to explain some of the results, but I’m not a 100% sure. A hand full of results weren’t really stable across runs. Some seem to be due to higher cache misses (spill code placement and even changes in code layout can have that effect). Generally, I think that there are still so many things that can be improved in -fregs-graph, but live range renumbering seems to be a prerequisite anyway.

So what about linear scan? I’m less familiar with that algorithm and code, but I have a hunch. Linear scan will make allocation decisions going through the code and insert “fix-up” code/blocks when assigned registers have to be changed. If I understand that correctly, you are not constrained to assign only one real register to a given vreg and most of the fix-up code will be reg-reg moves (but spills are needed too). That would mean, that renumbering just isn’t such an issue to begin with and since reg-reg moves are mostly quite cheap, even the cases where it would lead to more code are not so bad.

If you are interested in the raw numbers, they can be found in the ticket and a more lengthy discussion of the results can be found in my thesis.

Compile times - The Ugly

OK, I said I would take a speedup of 1%, but only at an adequate cost. You see, I knew this wouldn’t be fast. We are doing strictly more work. The only gains would be fewer iterations of register allocation (for -fregs-graph only) and possibly a speedup by a faster compiler compiled with that new optimization. I’ve also used pretty basic algorithms and I only tuned them ever so slightly. (Plus I’m not an experience Haskell developer and I have no intuition on how to write well performing Haskell.)

Alright, so, the slowdown I have observed compiling Cabal was almost… 30%. That is absolutely terrible, I’m not going to lie. I got some feedback on my code and tried a few things, but it barely made a dent.

But fret not, not all is lost.

Conclusion

-fregs-graph could need some love, there is no renumbering step and SSA transformation not only could take care of that, but it could also enable future optimizations.

My implementation (hopefully) shows, that there is some room for this, but that the improvement is not guaranteed to be large and cost must be low.

-fregs-graph itself could see better results with a different spill heuristic (possibly) and some other improvements (coloring for spill slots? memory addressing instead of stores/loads?).

There are a few possible optimizations for my SSA transformation pass. But several people have let me know, that a better, more modern algorithm exists⁵. I’m looking into implementing that algorithm and placing it before liveness analysis. Then I can do that on SSA-form directly, where I don’t need to run analyses in a loop until a fixed-point is reach, due to the SSA properties (so that’s supposed to be faster). But the current liveness analysis also removes some dead code as well - so I just need to take a look at the current code and the papers and make sure that I produce correct code.

What excites me quite a bit more though, is the possibility of writing a new graph-based register allocator that operates on SSA directly. Work on bringing register allocation to SSA form has been done and the discovery of some interesting properties made in the last two decades.⁶ So my next big project will be a new graph coloring register allocator on SSA-form for GHC!

If you spot any mistakes, omissions or want to provide feedback, comments, questions - you can reach me on Twitter, or via mail with my username at gmail.

For my first big project on GHC, I wanted to try and improve the Graph Coloring Register allocator in NCG (GHC’s Native Code Generator), aka -fregs-graph.

When code generators emit instructions, they usually use an unlimited number of virtual registers (vregs) and map those to physical registers (or real regs), possibly after changing the code around with optimizations. This register allocation step is generally one of the last ones and extremely important for performance. Registers are a scarce resource and memory access is orders of magnitude slower.

Two of the dominant paradigms for register allocation are Graph Coloring¹ based and Linear Scan² register allocators. These have been around since the eighties, respectively nineties, but are still highly relevant and in wide use. Generally speaking, graph coloring based approaches are supposed to yield better quality code, but take more time and memory, whereas linear scan is faster, at a cost of quality. GHC currently only uses a (very good) linear scan based allocator, since -fregs-graph has been suffering from bad performance since ca. 2013.

So looking at AndreasK’s improvement ideas, I chose to try live range splitting.

Live Ranges and Graph Coloring

I’m borrowing the definition of live range from Cooper & Torczon’s “Engineering a Compiler”, chapter 13:

A single live range consists of a set of definitions and uses that are related to each other because their values flow together. That is, a live range contains a set of definitions and a set of uses.

Since “range” sounds so “linear”, I somewhat prefer the term webs over live ranges, as these are webs of intersecting def-use chains.

Basically, we want to assign each live range one physical register, but we generally have fewer real registers than live ranges. We can assign two live ranges to the same physical register, if they never “overlap”. This “overlapping” is called interference and two LRs interfere if they are ever live at the same point in the program. We construct an undirected graph, the interference graph, where each vertex is a live range and each edge an interference. Calculating precise liveness information and constructing the interference graph isn’t cheap, by the way.

How does this help assigning registers? Using the interference graph, we can formulate a Graph Coloring problem, i.e., we want to color each vertex in such a way, that two connected vertices have different colors. Physical registers are our colors and since we have limited registers, we are looking for a k-coloring, where k is the number of registers.

Not all programs will be k-colorable, so we may have to spill, i.e., put an LR in memory, by performing a store after each definition and a load before each use. Then we can remove that LR from the interference graph and try again³.

I’ve ignored many things here: Coalescing is the process of merging non-interfering, copy related live ranges together. I will talk later about renumbering.

Live Range Splitting

In the ticket linked above, AndreasK has observed, that live ranges are spilled, even though they may be fit into liveness holes and a coloring could be found, by splitting the LR. What does this mean? Let’s look at the following pseudo-code:

x <- 1 # def x
y <- 1 # def y
loop {
    y <- y + 1  # use y
}

loop {
    # use x
}

Imagine for a moment this code makes sense (or would you rather stare at real assembly?). If we had only two registers and our heuristic decides to spill either x or y, we’d have an additional load and a store in each loop. At this point you may say “hold up, there are no loops in Haskell”, or “Why not put the load before the loop and the store after it?”. Since we’re working on machine instructions at this stage in the compiler, loops do exist - in the form of jumps between blocks. Choosing which LRs to spill and spill code placement are subjects in their own right. Let’s just say that this example is simplified and that -fregs-graph places the spill code right around the instructions, later cleaning up unnecessary ones.

I jumped on this and tried to implement Passive Live Range Splitting, by Cooper and Simpson. This algorithm is supposed to figure out, that we can split x like this:

x <- 1 # def x
y <- 1 # def y
STORE x
loop {
    y <- y + 1  # use y
}

LOAD x

loop {
    # use x
}

The paper looked simple and like a great solution, so I went ahead with it.

Jumping the Gun

I wasn’t familiar with the code base (or the algorithm) and did some learning by doing. I’ll give you the quick run down of my revelation:

In my description of live ranges, I made it sound like we are assigning registers to live ranges. That isn’t quite correct. There is no explicit modelling of live ranges (“webs”) in GHC (and probably in most compilers?). We are operating on vregs and those are mapped to physical registers.

Remember that “renumbering” step in the diagram earlier? This is actually about live range discovery. This whole coloring process is repeated. Like I noted before, spilling technically creates tiny live ranges and -fregs-graph renames them on-the-fly. To split live ranges, inserting stores and loads is not enough, you have to rename (“renumber”) the vregs in the instructions, to reflect the new live ranges! But -fregs-graph doesn’t have a renumbering phase. While analyzing this simple reproducing example program, it dawned on me. I saw something similar to this in assembly:

{
  def x
  use x
  def x
  use x
}
...
{
  def x
  use x
}

There were no branches with different definitions of x. In fact, x was redefined in the same basic block, redefined and used in a later block. Those are disjoint live ranges, but by having the same vreg name, they are constrained to be assigned to the same physical register! We don’t need live range splitting, we need live range discovery!⁴ I falsely assumed that vreg = live range.⁵

Things going Sideways

I briefly and desperately tried to create some makeshift renumbering for my split LRs, but it was no use. Figuring out where the new live ranges are and how to name them is hard, even though you technically only have to look “before” your inserted store and “after” the inserted load. But in the presence of control flow (branches and loops), it’s not that simple:

def x
STORE x
if (..) {
  use y
} else {
  LOAD x'
}
use x

Which x should we use after the branches join? This code is wrong and we either need to place a LOAD x' into the if-block, or possibly a copy x' -> x in the else branch.

To do that, I’d need to build a reaching definition analysis anyway and then some.

Discovering Live Ranges

So I decided to address the problem at hand by implementing a “renumbering” phase.

The “classic” way of doing this, is by performing data-flow analysis to build explicit def-use chains. Basically, we want to find all vreg definitions, create a list for each of them and add all the positions in the program where this definition was used. The data-flow analysis to get the reaching definitions is generally performed by a fixed-point algorithm. Straight line code is simple, but for loops, you have to iterate until your data-flow facts no longer change (i.e., the fixed-point is reached). This due to the fact, that information from the loop body can flow back to the loop head.

Once you have your def-use chains, i.e., n lists for each LR, you’ll want to perform pairwise intersection tests for all lists (with the same vreg). There are probably ways to optimize this, but this takes a lot of time and space!

In fact, it seems like this is not the way to do it anymore. E.g., looking at “Engineering a Compiler”, after giving an overview of the problem it says:

Conversion of the code into ssa form simplifies the construction of live ranges; thus, we will assume that the allocator operates on ssa form.

Even Briggs in 1994 wrote, that their compiler uses SSA instead of the “classical” way that Chaitin used.

SSA - What & Why

What is SSA and how does it help us? An intermediate representation is in Single Static Assignment form, if it conforms to a simple property:

Each name is only defined once, respectively, each definition introduces a new name

This has many nice implications, simplifies data-flow analysis and even enables new optimizations⁶. Any use of an SSA-variable x is dominated by its definition, i.e., any path from the start of the control flow graph of the program to the use has to go execute that definition - and there is only one definition. Def-use chains are no longer necessary, each definition has to come from a straight line of predecessors, be within the same basic block, or defined by a φ-function at the beginning of a block or come from

What is a φ-function (or φ-node, “phi-function”)? Let’s say we have a variable x that is redefined several times. Each time, we update an index to create a new name, i.e., x₀, x₁, … What if you have two different definitions of x reaching your basic block? Which name do you have to choose? This is where φ-functions come into play. Think of them as magically “selecting” the right name, depending on where the control flow came from and defining a new name for the selected value: x₂ = φ(x₀, x₁) Semantically, these are considered to be executed as parallel copies, all at once, at the beginning of the block, before the first instruction.

Here some code, which is not in SSA-form:

x := 0
if (...) {
    x := x + 1
} else {
    x := x + 2
}
y := x

…and transformed:

x_0 := 0
if (...) {
    x_1 := x_0 + 1
} else {
    x_2 := x_0 + 2
}
x_3 = φ(x_1, x_2)
y := x_3

SSA Destruction and Live Range Discovery

Naturally, we can’t leave the code like this. CPU’s simply don’t have a φ instruction, we need to get rid of them.

The process of transforming a program in SSA-form out-of-SSA (thus resolving all φ-functions) is sometimes called SSA Destruction. There is quite a bit of literature on this and it gets complicated if you want fast and correct SSA destruction.

But for our case, things are simpler. For now, all we want is to rename our live ranges, so that each vreg represents a single, disjoint live range. Sreedhar et al.⁷ noted the distinction between Conventional SSA (CSSA) and Transformed SSA (TSSA). A program is in CSSA right after SSA transformation (or after repair) and in TSSA once any program transformations have moved, removed or added code. The difference is, that the φ arguments in CSSA do not interfere - whereas in TSSA they might. Resolving φ-functions with interfering arguments requires the insertion of copies and to keep your program fast, you will want to figure out which ones you need and which ones you can get rid of. For CSSA, you simply gather all names connected by φ-functions, using a union-find (disjoint-set union), then assigning one name to each disjoint set. Voilà, you’re done!

Further Reading

Cooper & Torczon’s “Engineering a Compiler” is one of my favorite textbooks on compilers and contains lots of information on register allocation and SSA.

The (unfortunately unpublished) “SSA Book”, by France’s INRIA, is a real treasure trove and shouldn’t be missed.

For a more exhaustive bibliography, check out Jeremy Singer’s SSA Bibliography.

Conclusion

To summarize, in order to improve GHC’s graph coloring register allocator, I wanted to implement passive live range splitting, but discovered that several disjoint live ranges may bear the same vreg name. I’ve described what SSA is and how it may solve the problem.

In the next part, I’ll describe how I implemented SSA-transformation in GHC and what the results were.

If you spot any mistakes, omissions or want to provide feedback, comments, questions - you can reach me on Twitter, or via mail with my username at gmail.

Intro

For quite a long time I’ve been interested in compilers and to deepen my superficial knowledge of Haskell. Since I’m a hands-on learner, the latter requires an interesting project. Getting started with contributing to an existing compiler on the other hand, is a daunting task, as these are usually large and complex projects.

As I was searching for a suitable project, I found AndreasK’s blog post on GHC backends. Digging deeper, I read his article on GHC’s Graph Coloring Register Allocator and a ticket with improvement ideas for -fregs-graph.

Since I’m very interested in code generation and I’ve learned a bit about register allocation before, I thought that’d be a perfect place to start!

You can find some other (older) blog posts on how to get started in the GHC Wiki, but I encountered many issues along the way. Especially since some things are in a bit of a transition phase, e.g., the Make-base build vs. Hadrian-build. That’s why I’ve decided to write about my experience - maybe someone will find the solution to a problem here or get motivated to start hacking on GHC!

Note that this is written from a Ubuntu Linux on x86_64 perspective.

Set-Up

First things first. You need a very recent version of GHC and other dependencies to build the latest GHC. The packages in your distribution’s repository may be way to old (GHC 9.1 requires at least 8.10.x, AFAIK). Luckily, there is a PPA by Herbert V. Riedel - the page also links to sources for Debian Stretch, Mac OS and WSL repositories. Otherwise you might want to look at GHC Up.

The following commands will add the PPA and install everything needed from the repositories (note that I’m not choosing GHC 9, as there are still some things incompatible with it, e.g., Haskell Language Server):

sudo add-apt-repository ppa:hvr/ghc
sudo apt update
sudo apt install build-essential autoconf git ghc-8.10.4 cabal-install-3.4

And now to checkout out the multi-module Git project into the current folder:

git clone --recurse-submodules https://gitlab.haskell.org/ghc/ghc.git

Add GHC/Cabal to your path:

echo -e '# For GHC\nPATH="/opt/ghc/bin:$PATH"' >> ~/.profile
source ~/.profile

Now we need to install some Haskell packages, namely Happy & Alex (lexer and parser generators):

cabal update
cabal install happy
cabal install alex
# Add '/home/<username>/.cabal/bin' to your ~/.profile

Don’t forget to add the path to .cabal/bin to your PATH in .profile and source it! Check the versions to make sure everything is working:

cabal --version
ghc --version
alex --version
happy --version

For building the docs you’ll need Sphinx:

sudo apt install python3-sphinx

If you want to use/work on the LLVM-backend, you’ll also need to install the right LLVM development packages. Check the latest release notes to see which LLVM versions are supported.

IDE

Now, I know that the “true Scotsmen” use emacs, or maybe vi, but despite having programmed for many years, I never gotten over that initial learning bump. That time investment just doesn’t seem justifiable to me. I prefer an IDE or at least a rich code editor.

VS Code is a popular and robust choice for this, so that’s what I’m using for Haskell. Install VS Code, e.g., via the Ubuntu store, or download it from here.

There are plenty of infos out there on how to install and configure VS Code, so I’ll stick to the Haskell specifics. You’ll only need to install one extension: Go to File -> Preferences -> Extensions and search for “Haskell”. You’ll find “Haskell - Haskell language support powered by…”, or alternatively, here the direct link. This will also install the required extension “Haskell Syntax Highlighting”.

I can’t stress enough just how great the Haskell Language Server is. When I started, I tried manually compiling ghcide etc. and could barely get it to work. From the first release of Haskell LS, I could see and feel the improvements with every release. Give it a try, even with emacs or vim.

In case Haskell LS still ends up having difficulties, it can help to hit Ctrl + Shift + P and search for “Restart Haskell LSP Server” and run that.

Another very useful command is “Trim Trailing Whitespace” - GHC’s CI has a linter that will fail your MR if you have trailing whitespace! You can run the lint rules locally with ./hadrian/build lint:compiler or lint:base, but you will need hlint on your PATH!

Opening the project with VS Code is pretty straight forward - just go to File -> Open Folder and open your checkout directory of GHC.

HLint

Speaking of hlint - you can simply install it with cabal. You might want to add your own .hlint.yaml to a subdirectory in GHC and add or ignore some rules.

Git

GHC’s Git project contains many submodules. Sometimes, when you want to get a branch completely up-to-date with upstream, you’ll want to update all submodules recursively:

 git pull --recurse-submodules

Build & Test

Many of the things I’m writing here can be found on the nifty cheatsheet ghc.dev - make sure to check it out.

But I’m getting ahead of myself. Let’s start with the basics and change into the GHC directory with your favorite terminal emulator (e.g., in VS Code). Be aware that building GHC takes quite some computing resources and time, depending on the chosen optimizations.

Make sure to read the Hadrian README. Basically, there are different build “Flavours” with different optimizations or additional configuration (e.g., profiling builds). You can find an overview of the flavours here. GHC compilation is done in stages. Confusingly, sometimes you’ll see stage0 and stage1, other times it starts with stage1. Anyway, the first stage is the bootstrap compiler, compiled with the GHC on your path. The next stage is compiled with the bootstrap compiler, containing all the latest GHC changes.

Before your first compile (and whenever you upgraded your system GHC, change to a very different branch or the like), run ./boot && ./configure, which configures the build with all your paths and tool versions.

Running ./hadrian/build -j --flavour=Quick will build in parallel (-j) and produce a “quick” build (-O0). Note that you can use --freeze1 to avoid rebuilding the bootstrap compiler (if you haven’t changed anything relevant).

Hadrian has a clean command (hadrian/build clean), there is also hadrian/ghci to load GHC into a GHCi session and you can run the hlint rules with hadrian/build lint:base and hadrian/build lint:compiler.

After your build has finished, verify it by running ./_build/stage1/bin/ghc --version.

To run GHC’s test suite, just run ./hadrian/build test (but you might want to pass a flavour and other options) and check out testsuite/README.md for more options.

Although you should check out the build Flavours for yourself to find all the useful ones, like “quick” and “perf”, there is a good productivity tip from Matthew Pickering, a flavour combination that should pass almost the whole test suite, but reduces re-compile times: ./hadrian/build --flavour=default+no_profiled_libs+omit_pragmas --freeze1 -j (adding test to that incantation if you want to run the test suite)

Read

Reading the READMEs of GHC, Hadrian and ghc.dev gives you some quick and hands-on infos and commands.

To get the big picture view of GHC, there are also quite a few resources around!

Takenobu’s GHC reading guide is highly recommended, as well as their GHC illustrated.

Stephen Diehl also wrote about GHC, this is the first of a three-part blog post series. Be aware that some of the folder/module structures have changed, since that was written in 2016.

The Haskell Wiki is a double edges sword. While there is lots of extremely useful information, much of it is out-of-date and sometimes it’s hard to tell them apart. Either way, you’ll want to look at the GHC Commentary and the reading list, which contains papers and other theoretical sources.

Get connected!

Now that you can compile GHC and have some reading material, you want to get in touch with the community.

Get yourself a Gitlab account, so you can participate on tickets and merge requests. For the latter, you might want to create a fork of GHC for your account and add that as a git remote locally.

Subscribe to the GHC mailing list, to stay informed and get new ideas for projects!

AFAIK, most of the communication happens on IRC, so pop in at #GHC on libera.chat - I have to say, people on their have been very friendly and extremely helpful! There is also a #Haskell-docs channel, where you can find support for your fervor to make Haskell documentation excellent.

You might also want to check out this Wiki page for new contributors.

What to work on?

This is probably the hardest part. It very much depends on your knowledge and interests. If you have an area of interest, search for that on the bug tracker or ask around on IRC.

Making GHC compile times faster, even by a tiny bit, or reducing memory consumption, will always be welcomed by the community. If you have any experience with LLVM, I’m sure that there are also lots of ways to improve and speed-up the LLVM-backend.

Documentation is also vital and you can contribute by fixing mistakes, clarifying or adding descriptions. One really simple and fast way to dip your toes into GHC development is this ticket on documenting yet undocumented command line flags. All you need is grep and some detective work.

Developer’s Life

Usually, developing not only means writing code and compiling it, but debugging, profiling, benchmarking. I wanted to touch on these topics here too, briefly, to help you start out.

Debugging

Let’s say you found yourself a project and you’re working hard at it. Sooner or later, you’ll need to debug things. Unfortunately, there aren’t an awful lot of resources on this.

As mentioned above, there is a way to load GHC into GHCi. To be honest, I haven’t used this yet, as it didn’t seem to practical for working on the code generator.

This reference of debug flags got quite a lot of use though. You can dump intermediate steps along the way to figure out, where things went awry.

In lieu of a proper debugger, GHC.Driver.Ppr with it’s pprTrace and friends - where ppr stands for “pretty print”. GHC uses the SDoc type for string representations of data structures. I.e., this is the good old “printf-debugging”. The compiler needs to be built with -DDEBUG, e.g., the devel1 flavour and you can get additional output using the flag -dppr-debug.

When you have to debug a crashing output binary, you’ll also want to use GDB and possibly the reverse debugger rr.

In some cases I have used ad-hoc scripts to visualize or search for interesting data in my traces. Banging my head against the desk and crying profusely have not proven efficient techniques, despite my repeated tries.

Profiling

To enable cost center profiling, you’ll have to compile GHC and libs with profiling support. Hadrian offers Flavour Transformers for that. Note the remark for the profiled_ghc transformer - you’ll want to also use no_dynamic_ghc and the flavour “Quick” won’t cut it. E.g., use something like:

./hadrian/build -j --flavour="default+profiled_ghc+no_dynamic_ghc"

You can then use the RTS flags, like +RTX -xc.

Check out the above linked GHC documentation for more on profiling and make sure to install Matthew Pickering’s eventlog2html for some graphical representation of profiling eventlogs.

Here just quickly two recipes I found myself using quite a bit:

Since GHC is quite big and I was working on a single module, it made sense to restrict the output to that module, using -hm. The -hy option will give you a heap profile broken down by type and -hc breaks down time profiles by cost-center stack.

Here the type example:

ghc -O2 Main.hs +RTS -hm"GHC.CmmToAsm.SSA" -hy -l-au

And cost-centers:

ghc -O2 Main.hs +RTS -hm"GHC.CmmToAsm.SSA" -hc -l-au

Using eventlog2html to visualize it:

/home/ben/.cabal/bin/eventlog2html ghc.eventlog --bands 100 --include-trace-events && firefox ghc.eventlog.html

Benchmarking

GHC’s nofib benchmark suite come with a whole slew of of benchmarks.

While there is a submodule in the GHC tree, I’ve been advised to create a separate checkout of nofib. Like GHC, it has an old Make-based build and a new Shake-based one. I’ll only describe the latter here.

To start a full run of all benchmarks, simply use the nofib-run tool, like this:

# Optional `cabal update`
cabal new-run -- nofib-run --compiler=/absolute/path/to/your/ghc --compiler-arg="-O2" --output=my-test

Note that the path to your GHC, used for compiling the tests, has to be an absolute path! You can add any number of compiler args using the --compiler-arg flag. --output is the name of the result folder. To run tests “t”-times, use the -t switch. There is a -j option for parallel runs, but this only makes sense if you want to stress test your compiler to find e.g. crashes, as parallelization makes runtimes less deterministic and reproducible. Refer to cabal new-run -- nofib-run --help for other options.

The Make-build only runs certain benchmark groups by default, AFAIK omitting gc and some other. Using Shake, it will run all of them though. I haven’t figured out how to get the same behavior for both, but you can simply specify a list of tests to run.

All tests come with three different “speed” settings: SLOW, NORMAL, FAST This is supposed to enable benchmarking with different runtimes and also help normalize runtimes across benchmarks. That being said, runtimes have become quite disparate…

Results from different runs can be compared with nofib-compare, e.g., to compare baseline to your optimization:

cabal new-run -- nofib-compare ./_make/baseline ./_make/my_opt

nofib-compare also supports different output formats (CSV, ascii, markdown, latex).

For some serious measurements, you might want to install perf, which should be part of the linux-tools-generic package (or the specific one for your kernel). That way, you get data from your CPUs hardware counters, which is more “objective” than pure wall-time. To (temporarily) allow access to more CPU events, you have to allow that: echo "0" | sudo tee /proc/sys/kernel/perf_event_paranoid Cache misses are not part of the default event set used by nofib - for some reason - but you can include them explicitly with these arguments to nofib-run: --perf --perf-arg="-ecache-misses,cycles,instructions"

Benchmarking is generally an arcane art. You’ll want to limit any interferences, like programs running in background and you’ll definitely want to deactivate CPU frequency scaling and on-demand overclocking (“turbo boost”) - otherwise you’ll never have reproducible results.

To set the “performance” CPU governor:

sudo cpupower frequency-set -g performance

(Temporarily) deactivating “turbo boost”:

echo "0" | sudo tee /sys/devices/system/cpu/cpufreq/boost

(only tested this with my AMD chip, but should work for all CPUs. See docs)

Last but not least, nofib may sometimes break with the latest changes from GHC head. Currently I can’t use the latest GHC release (9.0.1) for nofib-run, so I have to use 8.10.x. Sometimes packages may become incompatible, but there is a hackage overlay with pre-releases, called head.hackage.

Nofib-run now has an option to use head.hackage directly. Update the packages first with cabal v2-update --project-file=nofib.head head.hackage.ghc.haskell.org, then simply pass the --head flag to nofib-run. Also see nofib/shake/README.mkd.

Conclusion

I hope this provided a gentle introduction to GHC development for you all and that some of you will contribute to this great project. Maybe this saves someone time that I spent scratching my head.

If you spot any mistakes, omissions or want to provide feedback, comments, questions - you can reach me on Twitter, or via mail with my username at gmail.

Updates

I’ll try to keep this somewhat up-to-date, but things sometimes move fast.

??.??.???? - Removed description of Hadrian’s -c option, bc. I was made aware of problems with it and that calling ./boot && ./configure explicitly is preferable.

16.07.2021 - Added mention of test suite and mpickerings productivity tip, also nofib-run’s --head option.

24.03.2022 - Mentioned that you need -DDEBUG to see pprTrace messages.

Worum es geht

Nachdem jemand auf Twitter einen "Der Spiegel" link gepostet hatte, mit wirklich schlechten Datenvisualisierungen zu antisemtischen Straftaten in Deutschland, dachte ich mir - das geht doch besser.

Also hier - hoffentlich - lesbarere Tortendiagramme, in denen auch alle Kategorien unterschieden werden, sowie eine Aufschlüsselung nach Bundesländern.

Sollte jemand so einen Datensatz für Österreich kennen, lasst es mich bitte wissen!

Achtung: Ein moderner Webbrowser wird benötigt! Z.B. Internet Explorer wird nicht unterstützt!

Für die Richtigkeit der Daten wird keine Haftung übernommen!

Quelle sind Daten veröffentlicht von MdB Petra Pau.

Ich hab übrigens keinen Google API Key, also wenn die Karte unten nicht lädt, haben sie mir vermutlich den Hahn abgedreht :))

Gesamt

Gesamtzahl der gemeldeten Antisemitischen Straftaten, inklusive nicht zugeordneter Nachmeldungen bis 31.12.2020.

Nur Zugeordnete

Nur zugeordnete Straftaten, d.h. ohne Nachmeldungen.

Nur Gewaltstraftaten

Hierzu wurden 21 Verletzte (29 mit Nachmeldungen) gemeldet.

Nach Bundesland

Aufschlüsselung aller Straftaten nach Bundesland.