Optimizing SBT resolution times

SBT offers a powerful system for customizing and controlling your build. However, it has one major niggle: its dependency manager is based on Apache Ivy. This is can be both a blessing or a curse, depending on which day of day of the week it happens to be. In many Scala project builds - regardless of if its on travis-ci.org, on-premise Jenkins or otherwise - a large portion of the overall build time is spent in doing dependency resolution. This is further aggravated when builds used Ivy's dynamic revision feature, as updates need to be calculated for each build, regardless of the Ivy local cache. In addition to these issues, many projects make use of lots of dependencies, and in todays world of several competing artifact delivery systems (sonatatype, maven central, bintray etc), resolving every artifact at every remote in an exhaustive fashion can consume a large quantity of time. With all this in mind, i've arrived at the following recipe for speeding your builds.

I'm going to assume that you're running your build system on-premise, and that you have control over the build environment (regardless of if its Jenkins, Travis Enterprise, Bamboo or whatever).

  • First and foremost, you need to run a local Nexus installation on the same network as your build system (Artifactory will also work, but Nexus is free, so we'll assume that for the purpose of this article). Running it on the same network as the build system is key, as we're trying to reduce latency for artifact and metadata XML transfer.

  • Once your Nexus install is setup, you're going to need to do a little digging through your builds - as many as it takes to get most of your repository locations - and then you're going to add each one of these remote locations as a remote proxy repository. In short, this means the Nexus install you created is going to act as a local cache for any artifacts that your builds need to download, meaning the build process (i.e. Ivy) will only have to look at your local Nexus, and it won't have to fetch anything over the internet.

  • What i've mentioned so far is pretty common practice - particularly for any larger system installation. There is one hitch though: any build that defines an external resolver will still cause Ivy to fetch to it for artifacts in an exhaustive fashion, so be sure to update every SBT build definition to only point to the domain of your local Nexus installation. After doing that, your builds would have increased in speed by quite a fair bit. Great! However, your builds are still free to add external resolvers anytime they like, which gets you right back to square one, and relying on the whole engineering team to not add new resolvers can be extremely brittle. In order to fix this, you'll need to add a custom repositories file to the SBT configuration - located at $HOME/.sbt/repositories by default. The contents of the file should be:

[repositories]
  local
  nexus-ivy-proxy-releases: http://YOURDOMAIN/nexus/content/groups/internal-ivy/, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]
  nexus-maven-proxy-releases: http://YOURDOMAIN/nexus/content/groups/internal/
  • The previous point is pretty well documented. What is not well documented is that one must also tell SBT to restrict resolver locations to that which is defined in this file (which it will not do by default, as it will still use repo.typesafe.com and friends). Assuming you're using the sbt-extras script there are three ways to ensure that the repositories file is authoritative (and restrictive) about where the build fetches artifacts from:
  1. Define a .sbtopts file in the root of your project, and add -Dsbt.override.build.repos=true as the contents.
  2. Define an environment variable $SBT_OPTS that contains -Dsbt.override.build.repos=true
  3. Finally you can propagate -Dsbt.override.build.repos=true via the -sbt-opts command line option if that works for you. However, I typically prefer the aforementioned options for a build environment, as then users don't have to care in their build scripts.
  • With these fixes in place, your build times should drop off a cliff. I've seen 5-10x decreases in build times since implementing this.

Hope this helps someone and reduces your build times in the same way it reduced mine.

blog comments powered by Disqus