Self-documenting tests

Even though it's 2020, self-documenting tests are still niche. There is Clairvoyance, a Scala flavour of YatSpec (that I mention exensively in this article for IBM). Its creator, Rhys Keepence is an old colleague and told me recently that it "is mostly up to date (although I haven’t yet published for Scala 2.13). The docs on my github page are not super up to date, but the latest version is 1.0.129".

However, introducing yet another new library to the codebase was too much an ask so I started using GivenWhenThen in ScalaTest. It was somewhat painful to get it to print just the Given, When, Then outputs to a separate file that can be version controlled for the edification of other data scientists modulo all the gubbins that are also spat out in your typical build.

I eventually did it using these resources from the ScalaTest docs. The top-and-bottom of it is that the GWT outputs are captured in a Reporter that may be bespoke (IntelliJ uses its own to seperate the GWTs from the logging). This Reporter can then spew out the events at the end of the test. But if you want them in a file produced by your build, you'll need something like this (in Maven):

    <build>

        <plugins>

            <plugin>

                <groupId>org.scalatest</groupId>
                <artifactId>scalatest-maven-plugin</artifactId>
                <version>2.0.0</version>
                <configuration>
                    <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
                    <junitxml>.</junitxml>
                    <stderr/>
                    <filereports>W ../docs/src/main/acceptance_tests/scenarios.txt</filereports>
                </configuration>

...

and then copy scenarios.txt where it can be versioned controlled with something like:

            <plugin>
                <artifactId>maven-resources-plugin</artifactId>
                <version>3.1.0</version>
                <executions>
                    <execution>
                        <id>copy-resources</id>
                        <phase>package</phase>
                        <goals>
                            <goal>copy-resources</goal>
                        </goals>
                        <configuration>
                            <outputDirectory>${basedir}/../docs/src/main/acceptance_tests/</outputDirectory>
                            <resources>
                                <resource>
                                    <directory>target/docs/src/main/acceptance_tests/</directory>
                                    <filtering>true</filtering>

...

The W in filereports means without colour since although this looks good on a Unix CLI, it just adds odd escape characters to a text file, which is what the data scientists want.

I'm unaware of a similar BDD framework for ZIO which means I need to mix my ZIO tests with ScalaTest. Unfortunately, I noticed that with Maven, some ZIO tests were failing but this did not stop the build. I documented this on ZIO's github here.

Scala Equality

The code:

Set(1) == List(1)

will always return false but will also always compile. This is pathological.

"Equality in Scala is a mess ... because the language designers decided Java interoperability trumped doing the reasonable thing in this case." (from here).

There are ways of solving this problem, the simplest being to use === that a number of libraries offer. Here are a three different ways of doing it:

  def scalazTripleEquals(): Unit = {
    import scalaz._
    import Scalaz._
//    println(List(1) === List("1")) // doesn't compile :)
  }

  def scalaUtilsTripleEquals(): Unit = {
    import org.scalautils.TypeCheckedTripleEquals._
//    println(List(1) === List("1")) // doesn't compile :)
  }

  def scalacticTripleEquals(): Unit = {
    import org.scalactic._
    import TypeCheckedTripleEquals._
//    println(List(1) === (List("1"))) // doesn't compile :)
  }

But what about incorporating it into the Scala language itself?

Scala creator, Martin Odersky's views can be found here here. He is proposing "it is opt-in. To get safe checking, developers have to annotate with @equalityClass ... So this means we still keep universal equality as it is in Scala now - we don’t have a choice here anyway, because of backwards compatibility."

Warning: ScalaTest

There is a horrible gotcha using Scalactic and ScalaTest (which is odd since they are stable mates). The problem is that you want compilation to fail for something likes this:

import org.scalatest.{FlatSpec, Matchers}

class MyTripleEqualsFlatSpec extends FlatSpec with Matchers {

  "triple equals" should "not compile" in {
    List(1)  should === (List("1"))
  }

}

Only it doesn't. It happily compiles! This is not what was expected at all given the code in scalacticTripleEquals() above. The solution can be found here. You must change the class signature to:

class MyTripleEqualsFlatSpec extends FlatSpec with Matchers with TypeCheckedTripleEquals {

for the compiler to detect this error.

Arrays

As an addendum, I just discovered Scala gives a nice way to compare arrays by value rather than by reference. It's:

a.deep == b.deep

(see here).

Automatic Boundary Checking

Surprises in Java's Math class

ScalaTest/ScalaCheck has a rather nice little feature. It can boundary check your unit tests. Let's take this code:

import org.scalatest.WordSpec
import org.scalatest.prop.GeneratorDrivenPropertyChecks
import org.scalatest.Matchers

class MyScalaTest extends WordSpec with GeneratorDrivenPropertyChecks with Matchers {

  "java.lang.Math.abs" should {
    "always return a non-negative value" in {
      forAll { x: Int =>
        Math.abs(x) should be >= 0
      }
    }
  }
}

It nicely runs the test many times, probing it with different values. Unfortunately, it fails with:

[info] MyScalaTest:

[info] Math.abs

[info] - should always return a non-negative value *** FAILED ***

[info]   TestFailedException was thrown during property evaluation.

[info]     Message: -2147483648 was not greater than or equal to 0

[info]     Location: (MyScalaTest.scala:12)

[info]     Occurred when passed generated values (

[info]       arg0 = -2147483648

[info]     )

What? How could java.lang.Math.abs fail to return a non-negative value? Well, we hit a boundary condition. ScalaCheck is clever enough to try Integer.MIN_VALUE. However, because there are more negative Integers than positive (the range is -2³¹ to 2³¹-1), not all negative integers map to positive ones.

If we are happy with a limited range, we can write:

import org.scalacheck.Gen

.

.     

      val sensibleRange = Gen.choose(0, 100)

      forAll(sensibleRange) {

.

.

to choose numbers inclusive of 0 to 99.