Introducing Exemplar for Automated Samples Testing
Learn about the Exemplar library and how it helps ensure users are getting the expected outputs.
Join the DZone community and get the full member experience.Join For Free
This post introduces a new library called Exemplar. The goal of Exemplar is to ensure that users get outputs that you expect them to see. It handles sample discovery, normalization (semantically equivalent results, perhaps from different environments), and flexible output verification. It invokes any command-line tool in the environment to be invoked. You can also invoke
curl, for example, to verify service API responses.
Gradle uses this library to verify examples in docs and guides, and remove boilerplate from integration tests.
Exemplar can be configured using a JUnit test runner (recommended) or using its APIs. See examples below and in the Exemplar GitHub repo.
Use Cases for Exemplar
It's important that documentation samples are accurate. Imagine the frustration, trying to learn something new and having the examples fail to work.
We think Exemplar works best as a documentation checking mechanism. It is not meant to substitute other forms of integration testing, however.
Exemplar is unique because it allows discovery of samples embedded in structured documents, as well as
OutputNormalizers (bundled and custom ones), conveniences for using sample projects in "more traditional" integration tests (via
@UsesSample("path/to/sample")), and allows samples to be programmatically modified (for extra environment setup, perhaps) before execution (
Exemplar is focused on logging and exit code outputs, and verifying other side effects is cumbersome (checking created files may require extra commands). Furthermore, although Exemplar can be used for any tool, it only has APIs for JVM projects.
Sample Project Testing Primer
sample-discovery library in Exemplar will consider any directory containing a
*.sample.conf file under the samples root as a sample project.
A sample config file is written in HOCON and tells
sample-check how to run the sample project. It can be simple:
... or more complex, with multiple steps:See the samples configuration reference in the docs to learn what options are available.
A JUnit Test then declares which
@SampleRunner and other aspects of samples tests.
Executing this test will discover and run external (that is, not embedded) sample projects and generate a JUnit test report as you'd expect.
Embedded Samples Testing Primer
Exemplar can discover samples within Asciidoctor source files using Asciidoctor's AST. Here's an example that uses
SampleModifiers as well to setup Samples' environments as well.
We run this test using the
EmbeddedSamplesRunner pointing to the
docs/ root directory where Exemplar will look for documentation files.
Executing this test will extract the code from Asciidoctor into a temporary project, run it, and verify it matches the declared output in the doc.
Adopt and Contribute
Exemplar is at version 0.6 at the time of writing this post, which means that the APIs may change before v1.0. That said, given that the API has gone through some revisions already and has been adopted by Gradle, breaking changes are unlikely before version 1.0.
Here are some of the things that might be good expansions of the library, and I'd love your help if you're eager to file ideas or submit pull requests to the gradle/exemplar GitHub repo.
- More standard normalizers such as date, time, or duration normalization
- Generate structured metadata for search indexing
- Allow expected output to be inlined in sample config files
- Samples discovery embedded in Markdown or other documentation formats (#2938)
Please reach out to me (@eriwen on Twitter) if you'd like guidance adopting or contributing to this project.
Published at DZone with permission of Eric Wendelin, DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.