More on Puppet Module Unit-Testing
Join the DZone community and get the full member experience.
Join For FreeI’ve previously made presentations and blog posts about Puppet and module testing – my position is that you should treat Puppet code as just that: code. Just like mainstream programming languages, it is possible (and good practice) to test your Puppet manifests so that you have higher confidence in them working when it comes time to actually run them.
There are some other factors which play a part:
- Make your modules generic. Any environment or host specifics you
have baked into classes or definitions makes them that much harder to
test in a dissimilar (read: clean) environment like your CI pipeline.
- As a corollary to the first point, classes should be parameterised
so that they can be used in a variety of different ways – in your
production environment, in staging, in QA, development (etc etc) and of
course your test pipeline.
- Loosely couple your modules. Tight dependencies enforced with strict
ordering constraints means that you can’t test each class by itself
without pulling in all the dependencies as well. Speaking directly from
experience, this can mean errors are that much harder to track down when
you have to look in a bunch of places for one failing resource.
It is issues like this last point that seem to cause the most grief when testing Puppet modules in our environment. We have a collection of common modules called dist, which provide both re-usable functionality when required by application modules (e.g. the ability to easily set up a MySQL server, a standard way to provision Apache/Nginx etc) and configuration we expect to be standardized across all machines – in other words, the platform. In fact the wrapper class that pulls in the standardized configuration is called just that – “platform”.
Platform pulls in a lot of helper functionality which application modules can take for granted. An example is the yum class. Here we set up a standard /etc/yum.conf with some tunable values, the /etc/yum.repos.d fragments directory and a bunch of standard repository fragments such as OS, Updates and so on. The module also includes a defined type, yum::repo which acts much like the built-in version but works with the yum class to have a fully managed fragments directory – if a yum fragment is on disk but not managed by Puppet, it is removed.
Naturally, in developing an application module you will want to set
up a repository fragment to point to wherever you have your app packages
stored, so all application modules utilise yum::repo at least once. Now, in testing your application class foo as a unit test, you might have the following code:
class foo { yum::repo { 'myapp': descr => 'Repository for My App', ... } ... }
To test it, you’d have the following in the tests directory:
class { 'foo': }
This is, of course, a trivial example with no parameters. Here, we
already have a problem when attempting to unit test the class – it will
immediately fail due to the yum class not having been instantiated in the catalog, thus not satisfying the dependency the yum::repo defined type has on it. Typically we have worked around this by just adding it to the test:
class { 'yum': } class { 'foo': }
This is fine if you know which class you need to pull in, but if there are dependencies between resources in different classes it may not be so clear. The dependent resource will know what it needs, but not where it should retrieve it from. This pattern actually breaks encapsulation, so while it is acceptable in Puppet standard practice it is not very good practice from a developer standpoint.
Another idea we toyed with was automatically including a “stubbed out” version of the platform
in every test, thus satisfying all dependencies that application
classes may have without needing the user to specify them. I don’t like
this idea for a couple of main reasons:
- It will blow out compile (and thus test) time a lot. We can usually
get away with each test running for a few seconds, and a complete app
module (for the entire job) in maybe 30-60 seconds. Pull in all of
platform for every test and one application module will take a few
minutes. Multiply that by hundreds of application modules and you are
looking at a big increase in test time.
- This is no longer really unit testing. We’re doing full-blown
integration testing at this point. Don’t get me wrong – this is also
valuable, but there is a time and place for it, and I don’t want to
destroy the unit testing that we already have, with the implicit limited
scope (and thus easier fault-finding) that it provides.
In traditional unit-testing with external dependencies that we don’t
want to test, we would mock those dependencies. Good mocking libraries
will also allow us to be explicit about call ordering, inputs and
outputs expected in order to verify behaviour of our own code as well as
the relationship it establishes with the external dependencies. Is this
possible with Puppet? What would it look like?
define yum::repo ( $descr, $baseurl, $enabled, ... ) { } class { 'foo': }
Now we have a somewhat mocked version of yum::repo that our
class can use without having to worry about other chained dependencies
outside of its view of the world. This starts introducing some other
problems though:
- It’s quite clear that the Puppet language just doesn’t have the
capabilities for advanced mocking (which is no surprise – that’s not its
primary goal). It would be interesting if a third-party library
provided Puppet mocks though…
- We now have inconsistency in our testing methods. The only time you
need to mock out a class/define is when it has unresolved dependencies
you don’t want to have to worry about. In all other cases, we can still
use the real version (which will be on the modulepath already since we
install the dist modules into the testing VM with the app module). Now there are two slightly confusing mechanisms for testing.
- Will the mocked version be found before the real version that is
elsewhere on the modulepath? I haven’t looked into the code to know
whether it will be found immediately by virtue of it having just been
parsed, but it’s not an unknown factor that I like.
- One of the tenets of reusable, encapsulated functionality that we provide in our dist
modules is that you don’t need to know the details. In fact, thanks to
parameterised classes, defined types, custom types and providers it is
often not possible to tell what is built-in to Puppet and what is one of
the previously mentioned ways of extending it – and this is just how it
should be. Would you really want to mock out any one of these resource
types when it provides so much more than just a container with
parameters? Input validation, consistency between input parameters,
platform checking, built-in dependency handling between resources of the
same type are all valid reasons to stick with what these types give you
for free. It feels wrong to remove them (which effectively is the
reason you do the compile testing in the first place).
I haven’t spent us much time on this as on other Puppet problems previously, because at the moment (fortunately) it is mostly no more than an annoyance. We can add in the missing test dependencies by hand, and most of our users are becoming savvy enough to do it themselves. I’m interesting in what the community thinks about this topic though, and if you have solved this problem yourself? Please leave comments; I would love to know what you think!
Published at DZone with permission of Oliver Hookins, DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.
Comments