How to Run Tagged Scala Tests with SBT and ScalaTest

We use SBT to build all of our Scala projects, and we’re still scratching the surface of what the tool can do. One thing we needed to do recently was to separate a module’s tests (built on scalatest) into groups. This is possible using scalatest’s tagging capabilities, which can be trigged with a custom SBT configuration. Although the documentation is out there, we couldn’t find a complete example. This is a step-by-step guide. So why is this useful? Here’s an illustrative example. We use vagrant to easily spin up local infrastructure in a known state. Using vagrant we can test an application’s use of systems such as RabbitMQ, Riak, Redis, MongoDB etc. We build tests that use this infrastructure so we know when we break things, and by using actual running instances, we avoid the need for mocks.

This is effective, but we don’t necessarily want to run these tests on every integration, especially on a loaded build server. We use the test tagging technique to mark tests that require vagrant in an ad-hoc manner, and configure our build system to exclude them based on run tasks. We can use a separate, less frequently used task, to run them. This technique could easily be extended for other categories of tests (e.g. slow, requiring DB access etc). The bottom line is, it shows us when a test failure is a genuine coding error, and keeps our confidence in CI results high.

There are various stages to follow. In this worked example we are creating the com.example.RequiresVagrant tag to designate tests that depend on external vagrant hosts.

Tag your Tests

Create a Tag

Tags are descendents of org.scalatest.Tag. in our example we need to add the following object to src/test/scala/com/example/project.

1
2
3
4
5
package com.example.project

import org.scalatest.Tag

object RequiresVagrant extends Tag("com.example.RequiresVagrant")

Tag the tests

The tests are tagged at the level of individual tests, not suites. That means we are looking for it blocks. We simply need to add our tag to them as follows:

1
2
3
4
5
6
7
8
9
10
11
12
package com.example.project
import com.example.RequiresVagrant
// ...

class StorageSpec extends FunSpec with GivenWhenThen with BeforeAndAfterAll {
  describe("A Broker") {
    it("Should correctly forward valid requests to the storage actor",
        RequiresVagrant) {
        // ...
    }
  }
}

This will ensure that, when RequiresVagrant is a filtered tag, this test won’t be run.

Make fixtures safe

Filtering tests will prevent attempts to use pieces of infrastructure that aren’t there. However if there are fixtures or other pieces of setup code, we may still attempt to contact that infrastructure. To prevent this we need to take all that code and put it in a block that will only run when a test runs. If we want our fixtures to be set up once per test, we can use a function definition (i.e. the block will be run 0 to many times). If we want a value that’s shared among tests, we can put the block in a lazy val (i.e. the block will run 0 or 1 times).

Scalatest documentation recommends an anonymous inner object.

So, for example, if a test required some sort of storage infrastructure, it would need to be amended as below. Here we are using two akka actors: one that uses some logic to filter the messages it receives, and the other that receives messages and stores them. Our tests verify that the messages we expect are subsequently found in storage.

Original:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
package com.example.project
import com.example.RequiresVagrant
// ...

class StorageSpec extends FunSpec with GivenWhenThen with BeforeAndAfterAll {
  implicit val sys = ActorSystem("storageTests")

  val storageActor = sys.actorOf(Props[StorageActor])
  val broker = sys.actorOf(Props(new StorageBroker(storageActor)))

  describe("A Broker") {
    it("Should correctly forward valid requests to the storage actor",
        RequiresVagrant) {
      // Use broker
      // Test storage has happened
    }
    // other tests
  }
}

Changed:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
package com.example.project
import com.example.RequiresVagrant

class StorageSpec extends FunSpec with GivenWhenThen with BeforeAndAfterAll {
  implicit val sys = ActorSystem("storageTests")

    lazy val fixture = new {
      val storageActor = sys.actorOf(Props[StorageActor])
      val broker = sys.actorOf(Props(new StorageBroker(storageActor)))
    }

  describe("A Broker") {
    import fixture._

    it("Should correctly forward valid items to the storage actor",
        RequiresVagrant) {
      // use broker as before
      // test storage as before
    }
    // other
  }
}

Whereas the first example will skip the test but still try to instantiate the storage subsystem, in the second example the fixture-using code is never run, so the lazy val is never initialized. Note the import line. Although this pulls in fixture‘s namespace, and ensures most test code is unaffected by our change, it won’t cause the object to be instantiated until use. Were the test not excluded, the test actors would be used, and the enclosing object would be instantiated the first time that happened. If you are using the beforeAll method to do some initialization, you may also safely use the fixture. beforeAll will only be called if some tests are due to run.

Clean up safely

There may be more code using these fixtures that would be run even if individual tests aren’t run. The afterAll method is an example. There are two things you can do about these:

  • If running the code but ignoring errors won’t slow you down, just wrap it in a try and ignore caught exceptions.
  • If there are timeouts etc that would delay you if the code is run, use a var cleanUp = false flag. You can and set it in every test. Then in your afterAll function, simply test the cleanUp flag.

Test your tests

You should now be in a position to toggle the test on or off from the command line. ScalaTest provides an -l command line switch to exclude certain tags. SBT, in turn, provides a facility to pass command line arguments to the underlying test system. Anything after -- will be passed through.

Let’s try it out:

1
sbt "test-only com.example.project.BrokerSpec -- -l com.example.RequiresVagrant

If your test concludes rapidly, without failures, you’re where you want to be.

Configure SBT

As shown previously, it’s possible to send arguments to the underlying test library with SBT’s test-only task. However, it’s not possible do this for the test task. And, in any case, we don’t want to constantly provide command-line arguments for a frequently invoked command. test and test-only are built-in tasks provided by SBT’s test scope. It would be good to copy this scope into a new configuration, say local, and add our command-line argument to those tasks, to take effect when our scope is used. SBT allows you to do just that. It requires the following steps:

Add a new configuration

First we ensure our SBT configuration (in project/*Build.scala) is importing sbt._ and Keys._, to give us access to the Test built-in. We find our Project declaration and add a new configuration with:

1
lazy val kernel = Project(/*existing configuration*/).configs(LocalTest)

We define LocalTest config as

1
2
3
lazy val kernel = Project(/*existing configuration*/).configs(LocalTest)

lazy val LocalTest = config("local") extend(Test)

which tells SBT that we want to give the configuration a scope called local and expect to use the same task keys as the built-in test scope.

We then pull the regular test defaults in with:

1
2
3
4
lazy val kernel = Project( /*existing configuration*/).configs(LocalTest)
    .settings(inConfig(LocalTest)(Defaults.testTasks): _*)

lazy val LocalTest = config("local") extend(Test)

Finally, we define our custom task configuration:

1
2
3
4
5
6
lazy val kernel = Project( /*existing configuration*/).configs(LocalTest)
  .settings(inConfig(LocalTest)(Defaults.testTasks): _*)
  .settings(testOptions in LocalTest := Seq(Tests.Argument("-l",
    "com.example.RequiresVagrant")))
   
lazy val LocalTest = config("local") extend(Test)

That says, when you run local:test or local:test-only, add the tag excluding arguments we used earlier.

Test SBT

Check that your new configuration will exclude the tests you want to exclude:

1
sbt local:test

It should be fairly clear if this has been successful.

Check you didn’t break those excluded tests by running everything:

1
sbt test

Configure your build server

Thanks to all our previous work, this is pretty easy. Simply replace whatever arcane incantation you were using in your build job with local:test, e.g.:

1
sbt local:test package

And you’re done!

Further reading