Generating test reports
The MUnit sbt plugin supports collecting historical data about your test reports and displaying HTML report summaries about your test suites. The test report data is stored in Google Cloud Storage and the HTML report is rendered with an MDoc markdown modifier.
Example
Below is the generated report from the tests in the MUnit codebase.
D: Average duration in milliseconds, P: Passed, E: Failed, F: Flaky, R: Flaky/Passed ratio, S: Skipped, I: Ignored, O: Other
| Name | D | P | E | F | R | S | I | O |
|---|---|---|---|---|---|---|---|---|
| munit.AssertionsSuite.basic | 24 | 2600 | 0 | 0 | 0.0 | 744 | 36 | 0 |
| munit.AssertionsSuite.expr | 0 | 2600 | 0 | 0 | 0.0 | 744 | 36 | 0 |
| munit.AssertionsSuite.subexpr | 2 | 2600 | 0 | 0 | 0.0 | 744 | 36 | 0 |
| munit.AssertionsSuite.subtype | 1 | 2010 | 0 | 0 | 0.0 | 700 | 44 | 0 |
| munit.AsyncFixtureOrderSuite.teardown runs only after test completes | 52 | 1570 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.AsyncJSSuite.async-error | 0 | 842 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.AsyncJSSuite.async-ok | 0 | 842 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.ClueSuite.clue | 0 | 2568 | 0 | 0 | 0.0 | 768 | 44 | 0 |
| munit.ClueSuite.clues | 0 | 2568 | 0 | 0 | 0.0 | 768 | 44 | 0 |
| munit.ClueSuite.comment | 0 | 3380 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.ClueSuite.identifier | 4 | 3380 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.ClueSuite.lambda | 0 | 2550 | 18 | 0 | 0.0053254437869822485 | 768 | 44 | 0 |
| munit.ClueSuite.list | 0 | 2568 | 0 | 0 | 0.0 | 768 | 44 | 0 |
| munit.ClueSuite.product | 0 | 1018 | 0 | 0 | 0.0 | 1691 | 766 | 0 |
| munit.ClueSuite.select | 0 | 3380 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.ClueSuite.string-message | 0 | 2568 | 0 | 0 | 0.0 | 768 | 44 | 0 |
| munit.ComparisonFailExceptionSuite.comparison-failure | 14 | 885 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.DiffsSuite.ansi | 4 | 3380 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.DiffsSuite.trailing-whitespace | 0 | 3380 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.DiffsSuite.windows-crlf | 0 | 3380 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.DoubleAssertionsFrameworkSuite.All Infinities | 0 | 1061 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.DoubleAssertionsFrameworkSuite.All Infinities-1 | 0 | 1061 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.DoubleAssertionsFrameworkSuite.Assert Equals NaN Fails | 66 | 1061 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.DoubleAssertionsFrameworkSuite.Assert Equals NaN Fails-1 | 2 | 1061 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.DoubleAssertionsFrameworkSuite.Assert NaN Equals Fails | 0 | 1061 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.DoubleAssertionsFrameworkSuite.Assert NaN Equals Fails-1 | 0 | 1061 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.DoubleAssertionsFrameworkSuite.Assert NaN Equals NaN | 4 | 1061 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.DoubleAssertionsFrameworkSuite.Assert NaN Equals NaN-1 | 0 | 1061 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.DoubleAssertionsFrameworkSuite.Assert Neg Infinity Equals Infinity | 0 | 1061 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.DoubleAssertionsFrameworkSuite.Assert Neg Infinity Equals Infinity-1 | 0 | 1061 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.DoubleAssertionsFrameworkSuite.Assert Pos Infinity Equals Infinity | 0 | 1061 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.DoubleAssertionsFrameworkSuite.Assert Pos Infinity Equals Infinity-1 | 0 | 1061 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.DoubleAssertionsFrameworkSuite.Assert Pos Infinity Not Equals | 0 | 1061 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.DoubleAssertionsFrameworkSuite.Assert Pos Infinity Not Equals Neg Infinity | 0 | 1061 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.DoubleAssertionsFrameworkSuite.Assert Pos Infinity Not Equals Neg Infinity... | 0 | 1061 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.DoubleAssertionsFrameworkSuite.Assert Pos Infinity Not Equals-1 | 0 | 1061 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.FailExceptionSuite.assertion-error | 1 | 907 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.FrameworkSuite.AssertionsFrameworkSuite | 6 | 416 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.FrameworkSuite.AsyncFixtureFrameworkSuite | 19 | 501 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.FrameworkSuite.AsyncFixtureTeardownFrameworkSuite | 3 | 501 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.FrameworkSuite.BoxedFrameworkSuite | 4 | 36 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.FrameworkSuite.CiOnlyFrameworkSuite | 10 | 786 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.FrameworkSuite.DiffProductFrameworkSuite | 12 | 786 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.FrameworkSuite.DuplicateNameFrameworkSuite | 4 | 416 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.FrameworkSuite.FailFrameworkSuite | 4 | 786 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.FrameworkSuite.FailSuiteFrameworkSuite | 3 | 652 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.FrameworkSuite.FixtureFrameworkSuite | 6 | 786 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.FrameworkSuite.InterceptFrameworkSuite | 9 | 706 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.FrameworkSuite.Issue179FrameworkSuite | 2 | 167 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.FrameworkSuite.ScalaCheckExceptionFrameworkSuite | 6 | 263 | 0 | 0 | 0.0 | 0 | 148 | 0 |
| munit.FrameworkSuite.ScalaCheckFrameworkSuite | 195 | 616 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.FrameworkSuite.ScalaVersionFrameworkSuite | 2 | 786 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.FrameworkSuite.StackTraceFrameworkSuite | 4 | 266 | 0 | 0 | 0.0 | 0 | 150 | 0 |
| munit.FrameworkSuite.StackTraceFrameworkSuite-1 | 1 | 266 | 0 | 0 | 0.0 | 0 | 150 | 0 |
| munit.FrameworkSuite.SuiteTransformCrashFrameworkSuite | 3 | 642 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.FrameworkSuite.SuiteTransformFrameworkSuite | 3 | 642 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.FrameworkSuite.SwallowedExceptionSuite | 7 | 701 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.FrameworkSuite.TagsFrameworkSuite | 3 | 786 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.FrameworkSuite.TagsFrameworkSuite-1 | 1 | 786 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.FrameworkSuite.TagsFrameworkSuite-2 | 1 | 786 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.FrameworkSuite.TestNameFrameworkSuite | 3 | 786 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.FrameworkSuite.TestTransformCrashFrameworkSuite | 5 | 642 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.FrameworkSuite.TestTransformFrameworkSuite | 1 | 642 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.FrameworkSuite.ValueTransformCrashFrameworkSuite | 3 | 642 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.FrameworkSuite.ValueTransformFrameworkSuite | 6 | 642 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.FunFixtureSuite.basic | 1 | 3380 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.FutureSuite.nested | 26 | 2048 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.LazyFutureSuite.nested | 1 | 2696 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.LazyFutureSuite.ok-task | 0 | 3027 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.LinesSuite.basic | 3 | 3380 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.LinesSuite.end-of-file | 0 | 819 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.LinesSuite.multiline | 0 | 3380 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.PrintersSuite.array | 1 | 3380 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.PrintersSuite.basic | 1 | 3380 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.PrintersSuite.char-single-quote | 0 | 420 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.PrintersSuite.list | 0 | 3380 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.PrintersSuite.map | 2 | 3380 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.PrintersSuite.multiline | 0 | 3380 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.PrintersSuite.newline | 0 | 3380 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.PrintersSuite.single-quote | 0 | 3380 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.PrintersSuite.string-single-quote | 0 | 420 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.PrintersSuite.user1 | 1 | 3380 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.PrintersSuite.user2 | 0 | 3380 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.ScalaCheckInitialSeedSuite.generated int are not all the same | 0 | 1347 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.ScalaCheckInitialSeedSuite.generating int | 84 | 1347 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.ScalaCheckSeedSuite.generated int are not all the same | 0 | 1958 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.ScalaCheckSeedSuite.generating int | 28 | 1958 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.SuiteLocalFixtureSuite.1 | 0 | 3380 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.SuiteLocalFixtureSuite.2 | 0 | 3380 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.SuiteLocalFixtureSuite.3 | 0 | 3380 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.SuiteLocalFixtureSuite.4 | 0 | 3380 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.SuiteLocalFixtureSuite.5 | 0 | 3380 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.TestLocalFixtureSuite.basic | 1 | 3380 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.TestLocalFixtureSuite.two | 0 | 3380 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.TimeoutSuite.fast | 2 | 2264 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.TimeoutSuite.slow | 102 | 2264 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.TypeCheckSuite.not a member | 3 | 2909 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.TypeCheckSuite.parse error | 0 | 2909 | 0 | 0 | 0.0 | 0 | 0 | 0 |
| munit.TypeCheckSuite.type mismatch | 0 | 2909 | 0 | 0 | 0.0 | 0 | 0 | 0 |
Installation
These installation instructions are experimental and subject to change.
First, install the sbt-munit plugin.
// project/plugins.sbt
addSbtPlugin("org.scalameta" % "sbt-munit" % "0.7.20")
Next, setup up Google Cloud authentication credentials. You should have a JSON file that looks like this.
{
"type": "service_account",
"project_id": "...",
"private_key": "---BEGIN PRIVATE KEY---...."
// ...
}
Next, add the following two secret environment variables to your CI settings.
GOOGLE_APPLICATION_CREDENTIALS_JSON: the base64 encoded value of the JSON file containing the credentials.
export GCP_CREDENTIALS="/path/to/json/file/on/your/computer"
# macOS
cat $GCP_CREDENTIALS | base64 | pbcopy
# Ubuntu (assuming GNU base64)
cat $GCP_CREDENTIALS | base64 -w0 | xclip
# Arch
cat $GCP_CREDENTIALS | base64 | sed -z 's;\n;;g' | xclip -selection clipboard -i
# FreeBSD (assuming BSD base64)
cat $GCP_CREDENTIALS | base64 | xclip
GOOGLE_APPLICATION_CREDENTIALS: an arbitrary relative filename like"gcp.json". This should not match the path on your personal computer. The exact value of this variable doesn't matter as long as the file path is relative and the file does not exists in your repository. The MUnit sbt plugin will generate this file from the base64 secret variable.
Verify that your CI provider only exposes secret environment variables to the jobs that run on master branch and not pull requests.
If you use GitHub actions, open the following URL for your repository https://github.com/scalameta/munit/settings/secrets to configure secret environment variables. Once the secrets are correctly configured, the page should looks something like this:
Merge some changes into master and verify that the test reports are getting uploaded to Google Cloud.
❯ gsutil ls -r 'gs://munit-test-reports/**'
gs://munit-test-reports/scalameta/munit/2020-01-26/refs/heads/master/670f475b49a44a87d515ec48b9749ec196336f78/testsJVM/0.21.0-RC1/1.8.0_232.json
...
Next, setup MDoc to
generate documentation. Once you have MDoc setup, enable MUnitReportPlugin in
the same project where MdocPlugin is enabled.
// build.sbt
lazy val docs = project
.in(file("myproject-docs"))
- .enablePlugins(MdocPlugin)
+ .enablePlugins(MdocPlugin, MUnitReportPlugin)
.settings(...)
Next, you should be able to use the mdoc:munit modifier to generate test
reports from the stored data in Google Cloud.
```scala mdoc:munit
```