More on Puppet Module Unit-Testing

I’ve previously made presentations and blog posts about Puppet and module testing – my position is that you should treat Puppet code as just that: code. Just like mainstream programming languages, it is possible (and good practice) to test your Puppet manifests so that you have higher confidence in them working when it comes time to actually run them.

There are some other factors which play a part:

• Make your modules generic. Any environment or host specifics you have baked into classes or definitions makes them that much harder to test in a dissimilar (read: clean) environment like your CI pipeline.
  
• As a corollary to the first point, classes should be parameterised so that they can be used in a variety of different ways – in your production environment, in staging, in QA, development (etc etc) and of course your test pipeline.
  
• Loosely couple your modules. Tight dependencies enforced with strict ordering constraints means that you can’t test each class by itself without pulling in all the dependencies as well. Speaking directly from experience, this can mean errors are that much harder to track down when you have to look in a bunch of places for one failing resource.
  

It is issues like this last point that seem to cause the most grief when testing Puppet modules in our environment. We have a collection of common modules called dist, which provide both re-usable functionality when required by application modules (e.g. the ability to easily set up a MySQL server, a standard way to provision Apache/Nginx etc) and configuration we expect to be standardized across all machines – in other words, the platform. In fact the wrapper class that pulls in the standardized configuration is called just that – “platform”.

Platform pulls in a lot of helper functionality which application modules can take for granted. An example is the yum class. Here we set up a standard /etc/yum.conf with some tunable values, the /etc/yum.repos.d fragments directory and a bunch of standard repository fragments such as OS, Updates and so on. The module also includes a defined type, yum::repo which acts much like the built-in version but works with the yum class to have a fully managed fragments directory – if a yum fragment is on disk but not managed by Puppet, it is removed.

Naturally, in developing an application module you will want to set up a repository fragment to point to wherever you have your app packages stored, so all application modules utilise yum::repo at least once. Now, in testing your application class foo as a unit test, you might have the following code:

class foo {
  yum::repo { 'myapp':
    descr => 'Repository for My App',
    ...
  }
...
}

To test it, you’d have the following in the tests directory:

class { 'foo': }

This is, of course, a trivial example with no parameters. Here, we already have a problem when attempting to unit test the class – it will immediately fail due to the yum class not having been instantiated in the catalog, thus not satisfying the dependency the yum::repo defined type has on it. Typically we have worked around this by just adding it to the test:

class { 'yum': }
class { 'foo': }

This is fine if you know which class you need to pull in, but if there are dependencies between resources in different classes it may not be so clear. The dependent resource will know what it needs, but not where it should retrieve it from. This pattern actually breaks encapsulation, so while it is acceptable in Puppet standard practice it is not very good practice from a developer standpoint.

Another idea we toyed with was automatically including a “stubbed out” version of the platform in every test, thus satisfying all dependencies that application classes may have without needing the user to specify them. I don’t like this idea for a couple of main reasons:

1. It will blow out compile (and thus test) time a lot. We can usually get away with each test running for a few seconds, and a complete app module (for the entire job) in maybe 30-60 seconds. Pull in all of platform for every test and one application module will take a few minutes. Multiply that by hundreds of application modules and you are looking at a big increase in test time.
   
2. This is no longer really unit testing. We’re doing full-blown integration testing at this point. Don’t get me wrong – this is also valuable, but there is a time and place for it, and I don’t want to destroy the unit testing that we already have, with the implicit limited scope (and thus easier fault-finding) that it provides.
   

In traditional unit-testing with external dependencies that we don’t want to test, we would mock those dependencies. Good mocking libraries will also allow us to be explicit about call ordering, inputs and outputs expected in order to verify behaviour of our own code as well as the relationship it establishes with the external dependencies. Is this possible with Puppet? What would it look like?

define yum::repo (
  $descr,
  $baseurl,
  $enabled,
  ...
) { }
class { 'foo': }

Now we have a somewhat mocked version of yum::repo that our class can use without having to worry about other chained dependencies outside of its view of the world. This starts introducing some other problems though:

• It’s quite clear that the Puppet language just doesn’t have the capabilities for advanced mocking (which is no surprise – that’s not its primary goal). It would be interesting if a third-party library provided Puppet mocks though…
  
• We now have inconsistency in our testing methods. The only time you need to mock out a class/define is when it has unresolved dependencies you don’t want to have to worry about. In all other cases, we can still use the real version (which will be on the modulepath already since we install the dist modules into the testing VM with the app module). Now there are two slightly confusing mechanisms for testing.
  
• Will the mocked version be found before the real version that is elsewhere on the modulepath? I haven’t looked into the code to know whether it will be found immediately by virtue of it having just been parsed, but it’s not an unknown factor that I like.
  
• One of the tenets of reusable, encapsulated functionality that we provide in our dist modules is that you don’t need to know the details. In fact, thanks to parameterised classes, defined types, custom types and providers it is often not possible to tell what is built-in to Puppet and what is one of the previously mentioned ways of extending it – and this is just how it should be. Would you really want to mock out any one of these resource types when it provides so much more than just a container with parameters? Input validation, consistency between input parameters, platform checking, built-in dependency handling between resources of the same type are all valid reasons to stick with what these types give you for free. It feels wrong to remove them (which effectively is the reason you do the compile testing in the first place).
  
  

I haven’t spent us much time on this as on other Puppet problems previously, because at the moment (fortunately) it is mostly no more than an annoyance. We can add in the missing test dependencies by hand, and most of our users are becoming savvy enough to do it themselves. I’m interesting in what the community thinks about this topic though, and if you have solved this problem yourself? Please leave comments; I would love to know what you think!