MUnit

MUnit

  • Docs
  • Blog
  • GitHub

›Overview

Overview

  • Getting started
  • Declaring tests
  • Writing assertions
  • Using fixtures
  • Filtering tests
  • Generating test reports
  • Coming from ScalaTest
  • Troubleshooting

Integrations

  • ScalaCheck
Edit

Generating test reports

The MUnit sbt plugin supports collecting historical data about your test reports and displaying HTML report summaries about your test suites. The test report data is stored in Google Cloud Storage and the HTML report is rendered with an MDoc markdown modifier.

Example

Below is the generated report from the tests in the MUnit codebase.

D: Average duration in milliseconds, P: Passed, E: Failed, F: Flaky, R: Flaky/Passed ratio, S: Skipped, I: Ignored, O: Other

Name D P E F R S I O
munit.AssertionsSuite.basic 24 2600 0 0 0.0 744 36 0
munit.AssertionsSuite.expr 0 2600 0 0 0.0 744 36 0
munit.AssertionsSuite.subexpr 2 2600 0 0 0.0 744 36 0
munit.AssertionsSuite.subtype 1 2010 0 0 0.0 700 44 0
munit.AsyncFixtureOrderSuite.teardown runs only after test completes 52 1570 0 0 0.0 0 0 0
munit.AsyncJSSuite.async-error 0 842 0 0 0.0 0 0 0
munit.AsyncJSSuite.async-ok 0 842 0 0 0.0 0 0 0
munit.ClueSuite.clue 0 2568 0 0 0.0 768 44 0
munit.ClueSuite.clues 0 2568 0 0 0.0 768 44 0
munit.ClueSuite.comment 0 3380 0 0 0.0 0 0 0
munit.ClueSuite.identifier 4 3380 0 0 0.0 0 0 0
munit.ClueSuite.lambda 0 2550 18 0 0.0053254437869822485 768 44 0
munit.ClueSuite.list 0 2568 0 0 0.0 768 44 0
munit.ClueSuite.product 0 1018 0 0 0.0 1691 766 0
munit.ClueSuite.select 0 3380 0 0 0.0 0 0 0
munit.ClueSuite.string-message 0 2568 0 0 0.0 768 44 0
munit.ComparisonFailExceptionSuite.comparison-failure 14 885 0 0 0.0 0 0 0
munit.DiffsSuite.ansi 4 3380 0 0 0.0 0 0 0
munit.DiffsSuite.trailing-whitespace 0 3380 0 0 0.0 0 0 0
munit.DiffsSuite.windows-crlf 0 3380 0 0 0.0 0 0 0
munit.DoubleAssertionsFrameworkSuite.All Infinities 0 1061 0 0 0.0 0 0 0
munit.DoubleAssertionsFrameworkSuite.All Infinities-1 0 1061 0 0 0.0 0 0 0
munit.DoubleAssertionsFrameworkSuite.Assert Equals NaN Fails 66 1061 0 0 0.0 0 0 0
munit.DoubleAssertionsFrameworkSuite.Assert Equals NaN Fails-1 2 1061 0 0 0.0 0 0 0
munit.DoubleAssertionsFrameworkSuite.Assert NaN Equals Fails 0 1061 0 0 0.0 0 0 0
munit.DoubleAssertionsFrameworkSuite.Assert NaN Equals Fails-1 0 1061 0 0 0.0 0 0 0
munit.DoubleAssertionsFrameworkSuite.Assert NaN Equals NaN 4 1061 0 0 0.0 0 0 0
munit.DoubleAssertionsFrameworkSuite.Assert NaN Equals NaN-1 0 1061 0 0 0.0 0 0 0
munit.DoubleAssertionsFrameworkSuite.Assert Neg Infinity Equals Infinity 0 1061 0 0 0.0 0 0 0
munit.DoubleAssertionsFrameworkSuite.Assert Neg Infinity Equals Infinity-1 0 1061 0 0 0.0 0 0 0
munit.DoubleAssertionsFrameworkSuite.Assert Pos Infinity Equals Infinity 0 1061 0 0 0.0 0 0 0
munit.DoubleAssertionsFrameworkSuite.Assert Pos Infinity Equals Infinity-1 0 1061 0 0 0.0 0 0 0
munit.DoubleAssertionsFrameworkSuite.Assert Pos Infinity Not Equals 0 1061 0 0 0.0 0 0 0
munit.DoubleAssertionsFrameworkSuite.Assert Pos Infinity Not Equals Neg Infinity 0 1061 0 0 0.0 0 0 0
munit.DoubleAssertionsFrameworkSuite.Assert Pos Infinity Not Equals Neg Infinity... 0 1061 0 0 0.0 0 0 0
munit.DoubleAssertionsFrameworkSuite.Assert Pos Infinity Not Equals-1 0 1061 0 0 0.0 0 0 0
munit.FailExceptionSuite.assertion-error 1 907 0 0 0.0 0 0 0
munit.FrameworkSuite.AssertionsFrameworkSuite 6 416 0 0 0.0 0 0 0
munit.FrameworkSuite.AsyncFixtureFrameworkSuite 19 501 0 0 0.0 0 0 0
munit.FrameworkSuite.AsyncFixtureTeardownFrameworkSuite 3 501 0 0 0.0 0 0 0
munit.FrameworkSuite.BoxedFrameworkSuite 4 36 0 0 0.0 0 0 0
munit.FrameworkSuite.CiOnlyFrameworkSuite 10 786 0 0 0.0 0 0 0
munit.FrameworkSuite.DiffProductFrameworkSuite 12 786 0 0 0.0 0 0 0
munit.FrameworkSuite.DuplicateNameFrameworkSuite 4 416 0 0 0.0 0 0 0
munit.FrameworkSuite.FailFrameworkSuite 4 786 0 0 0.0 0 0 0
munit.FrameworkSuite.FailSuiteFrameworkSuite 3 652 0 0 0.0 0 0 0
munit.FrameworkSuite.FixtureFrameworkSuite 6 786 0 0 0.0 0 0 0
munit.FrameworkSuite.InterceptFrameworkSuite 9 706 0 0 0.0 0 0 0
munit.FrameworkSuite.Issue179FrameworkSuite 2 167 0 0 0.0 0 0 0
munit.FrameworkSuite.ScalaCheckExceptionFrameworkSuite 6 263 0 0 0.0 0 148 0
munit.FrameworkSuite.ScalaCheckFrameworkSuite 195 616 0 0 0.0 0 0 0
munit.FrameworkSuite.ScalaVersionFrameworkSuite 2 786 0 0 0.0 0 0 0
munit.FrameworkSuite.StackTraceFrameworkSuite 4 266 0 0 0.0 0 150 0
munit.FrameworkSuite.StackTraceFrameworkSuite-1 1 266 0 0 0.0 0 150 0
munit.FrameworkSuite.SuiteTransformCrashFrameworkSuite 3 642 0 0 0.0 0 0 0
munit.FrameworkSuite.SuiteTransformFrameworkSuite 3 642 0 0 0.0 0 0 0
munit.FrameworkSuite.SwallowedExceptionSuite 7 701 0 0 0.0 0 0 0
munit.FrameworkSuite.TagsFrameworkSuite 3 786 0 0 0.0 0 0 0
munit.FrameworkSuite.TagsFrameworkSuite-1 1 786 0 0 0.0 0 0 0
munit.FrameworkSuite.TagsFrameworkSuite-2 1 786 0 0 0.0 0 0 0
munit.FrameworkSuite.TestNameFrameworkSuite 3 786 0 0 0.0 0 0 0
munit.FrameworkSuite.TestTransformCrashFrameworkSuite 5 642 0 0 0.0 0 0 0
munit.FrameworkSuite.TestTransformFrameworkSuite 1 642 0 0 0.0 0 0 0
munit.FrameworkSuite.ValueTransformCrashFrameworkSuite 3 642 0 0 0.0 0 0 0
munit.FrameworkSuite.ValueTransformFrameworkSuite 6 642 0 0 0.0 0 0 0
munit.FunFixtureSuite.basic 1 3380 0 0 0.0 0 0 0
munit.FutureSuite.nested 26 2048 0 0 0.0 0 0 0
munit.LazyFutureSuite.nested 1 2696 0 0 0.0 0 0 0
munit.LazyFutureSuite.ok-task 0 3027 0 0 0.0 0 0 0
munit.LinesSuite.basic 3 3380 0 0 0.0 0 0 0
munit.LinesSuite.end-of-file 0 819 0 0 0.0 0 0 0
munit.LinesSuite.multiline 0 3380 0 0 0.0 0 0 0
munit.PrintersSuite.array 1 3380 0 0 0.0 0 0 0
munit.PrintersSuite.basic 1 3380 0 0 0.0 0 0 0
munit.PrintersSuite.char-single-quote 0 420 0 0 0.0 0 0 0
munit.PrintersSuite.list 0 3380 0 0 0.0 0 0 0
munit.PrintersSuite.map 2 3380 0 0 0.0 0 0 0
munit.PrintersSuite.multiline 0 3380 0 0 0.0 0 0 0
munit.PrintersSuite.newline 0 3380 0 0 0.0 0 0 0
munit.PrintersSuite.single-quote 0 3380 0 0 0.0 0 0 0
munit.PrintersSuite.string-single-quote 0 420 0 0 0.0 0 0 0
munit.PrintersSuite.user1 1 3380 0 0 0.0 0 0 0
munit.PrintersSuite.user2 0 3380 0 0 0.0 0 0 0
munit.ScalaCheckInitialSeedSuite.generated int are not all the same 0 1347 0 0 0.0 0 0 0
munit.ScalaCheckInitialSeedSuite.generating int 84 1347 0 0 0.0 0 0 0
munit.ScalaCheckSeedSuite.generated int are not all the same 0 1958 0 0 0.0 0 0 0
munit.ScalaCheckSeedSuite.generating int 28 1958 0 0 0.0 0 0 0
munit.SuiteLocalFixtureSuite.1 0 3380 0 0 0.0 0 0 0
munit.SuiteLocalFixtureSuite.2 0 3380 0 0 0.0 0 0 0
munit.SuiteLocalFixtureSuite.3 0 3380 0 0 0.0 0 0 0
munit.SuiteLocalFixtureSuite.4 0 3380 0 0 0.0 0 0 0
munit.SuiteLocalFixtureSuite.5 0 3380 0 0 0.0 0 0 0
munit.TestLocalFixtureSuite.basic 1 3380 0 0 0.0 0 0 0
munit.TestLocalFixtureSuite.two 0 3380 0 0 0.0 0 0 0
munit.TimeoutSuite.fast 2 2264 0 0 0.0 0 0 0
munit.TimeoutSuite.slow 102 2264 0 0 0.0 0 0 0
munit.TypeCheckSuite.not a member 3 2909 0 0 0.0 0 0 0
munit.TypeCheckSuite.parse error 0 2909 0 0 0.0 0 0 0
munit.TypeCheckSuite.type mismatch 0 2909 0 0 0.0 0 0 0

Installation

These installation instructions are experimental and subject to change.

First, install the sbt-munit plugin.

// project/plugins.sbt
addSbtPlugin("org.scalameta" % "sbt-munit" % "0.7.20")

Next, setup up Google Cloud authentication credentials. You should have a JSON file that looks like this.

{
  "type": "service_account",
  "project_id": "...",
  "private_key": "---BEGIN PRIVATE KEY---...."
  // ...
}

Next, add the following two secret environment variables to your CI settings.

  • GOOGLE_APPLICATION_CREDENTIALS_JSON: the base64 encoded value of the JSON file containing the credentials.
export GCP_CREDENTIALS="/path/to/json/file/on/your/computer"
# macOS
cat $GCP_CREDENTIALS | base64 | pbcopy
# Ubuntu (assuming GNU base64)
cat $GCP_CREDENTIALS | base64 -w0 | xclip
# Arch
cat $GCP_CREDENTIALS | base64 | sed -z 's;\n;;g' | xclip -selection clipboard -i
# FreeBSD (assuming BSD base64)
cat $GCP_CREDENTIALS | base64 | xclip
  • GOOGLE_APPLICATION_CREDENTIALS: an arbitrary relative filename like "gcp.json". This should not match the path on your personal computer. The exact value of this variable doesn't matter as long as the file path is relative and the file does not exists in your repository. The MUnit sbt plugin will generate this file from the base64 secret variable.

Verify that your CI provider only exposes secret environment variables to the jobs that run on master branch and not pull requests.

If you use GitHub actions, open the following URL for your repository https://github.com/scalameta/munit/settings/secrets to configure secret environment variables. Once the secrets are correctly configured, the page should looks something like this:

Example screenshot from GitHub Actions secrets page

Merge some changes into master and verify that the test reports are getting uploaded to Google Cloud.

❯ gsutil ls -r 'gs://munit-test-reports/**'
gs://munit-test-reports/scalameta/munit/2020-01-26/refs/heads/master/670f475b49a44a87d515ec48b9749ec196336f78/testsJVM/0.21.0-RC1/1.8.0_232.json
...

Next, setup MDoc to generate documentation. Once you have MDoc setup, enable MUnitReportPlugin in the same project where MdocPlugin is enabled.

  // build.sbt
  lazy val docs = project
    .in(file("myproject-docs"))
-   .enablePlugins(MdocPlugin)
+   .enablePlugins(MdocPlugin, MUnitReportPlugin)
    .settings(...)

Next, you should be able to use the mdoc:munit modifier to generate test reports from the stored data in Google Cloud.

```scala mdoc:munit

```
← Filtering testsComing from ScalaTest →
  • Example
  • Installation
MUnit
Overview
Getting started
Social
Copyright © 2021 Scalameta