Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

AWS Adventures: Infrastructure as Code and Microservices (Part 3)

DZone's Guide to

AWS Adventures: Infrastructure as Code and Microservices (Part 3)

It's testing time! Now that the basics are set up in AWS, it's time to make sure the pieces work. We'll run through unit tests, integration tests, and plenty more.

· Cloud Zone
Free Resource

See how the beta release of Kubernetes on DC/OS 1.10 delivers the most robust platform for building & operating data-intensive, containerized apps. Register now for tech preview.

Even. More. Tests. Now that you've got yourself underway, we're going to test like there's no tomorrow. Manual tests, unit tests, integration tests, we're going to test this like crazy and make sure our bases are covered.

But first, a bit of housekeeping...

Step 6: Delete Your Lambda Function

Your function is created. If you want to update code in it, you could simple make a new zip file and call updateFunctionCode. However, to make things truly immutable and atomic, meaning each individual aspect of our Lambda we can test and update individually, we’ll just delete the whole thing.

Remember, we’re not treating our server like a nice camera. Instead, we purchase a disposable one at the local drugstore/apothecary, and if it breaks, we get a new one instead of waisting time debugging a $9.25 single use electronic. This is important for deploying specific versions of code. If you deploy a git tag called “2.1.6”, but you later update code, you’ve negated the whole point of using a specific git tag since it’s not really 2.1.6, but your own version. If something goes wrong, you know for sure (mostly) that’s that version of the code and not your modification.

In build.test.js, import the non-existent deleteFunction:

const {
    listFunctions,
    createFunction,
    deleteFunction
} = require('./build');


Add a mock method to our mockLambda:

deleteFunction: (params, cb) => cb(undefined, {})


And a mock method to our mockBadLambda:

deleteFunction: (params, cb) => cb(new Error('boom'))


And finally our two tests:

describe('#deleteFunction', ()=>
{
    it('should delete our lambda if there', (done)=>
    {
        deleteFunction(mockLamba, (err)=>
        {
            _.isUndefined(err).should.be.true;
            done();
        });
    });
    it('should not delete our lambda if it', (done)=>
    {
        deleteFunction(mockBadLambda, (err)=>
        {
            err.should.exist;
            done();
        });
    });
});


If our Lambda works, we get no error. The call gives you an empty Object back which is worthless, so we just bank on “no error is working code”. Let’s write the implementation. In build.js, put in the following code above your module.exports:

const deleteFunction = (lambda, callback)=>
{
    var params = {
        FunctionName: FUNCTION_NAME
    };
    lambda.deleteFunction(params, (err, data)=>
    {
        if(err)
        {
            // log("lambda::deleteFunction, error:", err);
            return callback(err);
        }
        // log("lambda::deleteFunction, data:", data);
        callback(undefined, data);
    });
};


And then add to your module.exports:

module.exports = {
    listFunctions,
    createFunction,
    deleteFunction
};


Cool, now re-run npm test:

Image title

Let’s give her a spin. Hardcode deleteFunction(lambda, ()=>{});at the very bottom, run node build.js, then log into the AWS Console for Lambda, and you should no longer see ‘datMicro’ (or whatever you called it) in the left list.

Step 7: Making Testable Code by Testing It

There are a few more steps to go in making our Lambda fully functional with the API Gateway. However, we can at this point test her out in the AWS Console. That means we can test her out in JavaScript, too. Let’s take a look at the original Lambda function code:

exports.handler = (event, context, callback) =>
{
    const response = {
        statusCode: '200',
        body: JSON.stringify({result: true, data: 'Hello from Lambda'}),
        headers: {
            'Content-Type': 'application/json',
        }
    }
    callback(null, response);
};


A few problems with this handler. First, it’s not testable because it doesn’t return anything. Second, it doesn’t really take any inputs of note. Let’s do a few things. We’ll add some unit tests, a function that always returns true, and a random number function.

Always True

Create an index.test.js file, and add this code as a starting template:

const expect = require("chai").expect;
const should = require('chai').should();
const _ = require('lodash');
const {
    alwaysTrue
} = require('./index');
describe('#index', ()=>
{
    describe('#alwaysTrue', ()=>
    {
        it('is always true', ()=>
        {
            alwaysTrue().should.be.true;
        });
    });
});


Modify your package.json to point to this test for now:

"scripts": {
    "test": "mocha index.test.js",
    ...
},


Now run npm test. Hopefully, you get something along the lines of:

Image title

To make it pass, create the predicate:

const alwaysTrue = ()=> true;


Then export at the very bottom:

module.exports = {
    alwaysTrue
};


Re-run your npm test, and it should be green:

module.exports = {
    alwaysTrue
};


Testing random numbers is hard. For now, we’ll just verify the number is within the range we specified. In index.test.js import the new, non-existent, function:

const {
    alwaysTrue,
    getRandomNumberFromRange
} = require('./index');


And a basic test, as we’re not handling bounds or typing checks for now:

describe('#getRandomNumberFromRange', ()=>
{
    it('should give a number within an expected range', ()=>
    {
        const START = 1;
        const END = 10;
        const result = getRandomNumberFromRange(START, END);
        _.inRange(result, START, END).should.be.true;
    });
});


Re-run your tests and it should fail (or perhaps not even compile).

Now implement the function in index.js:

const getRandomNumberFromRange = (start, end)=>
{
    const range = end - start;
    let result = Math.random() * range;
    result += start;
    return Math.round(result);
};


And export her at the bottom:

module.exports = {
    alwaysTrue,
    getRandomNumberFromRange
};


Re-run your tests and she should be green:

Image title

Lastly, let’s rework our main Lambda function to always return a value, respond to a test, and become its own function as we’ll manually add it to the module.exports in a bit. In index.test.js, import the handler:

const {
    alwaysTrue,
    getRandomNumberFromRange,
    handler
} = require('./index');


And write the first test that expects it to return a response. Since we aren’t a typed language, we’ll create a loose one via a couple predicates to determine if it’s “response like”.

const responseLike = (o)=> _.isObjectLike(o) && _.has(o, 'statusCode') && _.has(o, 'body');


And the test:

describe('#handler', ()=>
{
    it('returns a response with basic inputs', ()=>
    {
        const result = handler({}, {}, ()=>{});
        responseLike(result).should.be.true;
    });
});


For now, the response is always an HTTP 200. We can add different ones later. Re-run your tests and she should fail:

Image title

Now let’s modify the function signature of our handler from:

exports.handler = (event, context, callback) =>

to:

const handler = (event, context, callback) =>


Move her above the module.exports and then add her to the exports. Final Lambda should look like this:

const alwaysTrue = ()=> true;
const getRandomNumberFromRange = (start, end)=>
{
    const range = end - start;
    let result = Math.random() * range;
    result += start;
    return Math.round(result);
};
const handler = (event, context, callback) =>
{
    const response = {
        statusCode: '200',
        body: JSON.stringify({result: true, data: 'Hello from Lambda'}),
        headers: {
            'Content-Type': 'application/json',
        }
    }
    callback(null, response);
};
module.exports = {
    alwaysTrue,
    getRandomNumberFromRange,
    handler
};


Our Lambda will return random numbers in the response based on the range you give it. We’ll have to create some predicates to ensure we actually get numbers, they are within range, and then return error messages appropriately. Our response in our handler will start to be different based on if someone passes in good numbers, bad numbers, bad data, or if it’s just a test. So, we’ll need to make him dynamic. Finally, we’ll add a flag in the event to make integration testing easier.

First, the litany of predicates for input checking. You’ll need 2 helper functions to make this easier. Create a new JavaScript file called predicates.js, and put this code into it:

const _ = require('lodash');
const validator = (errorCode, method)=>
{
    const valid = function(args)
    {
        return method.apply(method, arguments);
    };
    valid.errorCode = errorCode;
    return valid;
}
const checker = ()=>
{
    const validators = _.toArray(arguments);
    return (something)=>
    {
        return _.reduce(validators, (errors, checkerFunction)=>
        {
            if(checkerFunction(something))
            {
                return errors;
            }
            else
            {
                return _.chain(errors).push(checkerFunction.errorCode).value();
            }
        }, [])
    };
};
module.exports = {
    validator,
    checker
};


Now, let’s test the new, (soon to be) parameter checked handler in a few situations. At the top of index.test.js, import the handler function:

const {
    alwaysTrue,
    getRandomNumberFromRange,
    handler
} = require('./index');


Let’s add a new, more brutal negative test where we pass nothing:

it('passing nothing is ok', ()=>
{
    const result = handler();
    responseLike(result).should.be.true;
});


Let’s look at the responses and ensure we’re failing because of bad parameters, specifically, a malformed event. One test for a good event, one for a missing end, and one for our echo statement. Since the response is encoded JSON, we create a predicate to parse it out and check the result:

const responseSucceeded = (o)=>
{
    try
    {
        const body = JSON.parse(o.body);
        return body.result === true;
    }
    catch(err)
    {
        return false;
    }
};
// ...
it('succeeds if event has a start and end', ()=>
{
    const response = handler({start: 1, end: 10}, {}, ()=>{});
    responseSucceeded(response).should.be.true;
});
it('fails if event only has start', ()=>
{
    const response = handler({start: 1}, {}, ()=>{});
    responseSucceeded(response).should.be.false;
});
it('succeeds if event only has echo to true', ()=>
{
    const response = handler({echo: true}, {}, ()=>{});
    responseSucceeded(response).should.be.true;
});


None of those will pass. Let’s make ’em pass. Open index.js, and put in the predicates first. Import her up at the top:

// Note: the below only works in newer Node, 
// not the 4.x version AWS uses
// const { validator, checker } = require('./predicates');
const predicates = require('./predicates');
const validator = predicates.validator;
const checker = predicates.checker;


Then below put your predicate helpers:

// predicate helpers
const eventHasStartAndEnd = (o) => _.has(o, 'start') && _.has(o, 'end');
const eventHasTestEcho    = (o) => _.get(o, 'echo', false);
const isLegitNumber       = (o) => _.isNumber(o) && _.isNaN(o) === false


These check the event for both a start and end property, or an echo. Lodash _.isNumber counts NaN as a number, even though NaN stands for “Not a Number” and is a Number per the ECMAStandard because “design by committee”. I wrangle the insanity by writing my own predicate that… you know… makes sense: isLegitNumber.

We’ll use them to build our argument predicates:

// argument predicates
const legitEvent = (o)=> 
_.some([
    eventHasStartAndEnd, 
    eventHasTestEcho
],
(predicate) => predicate(o)
);
const legitStart = (o) => isLegitNumber(_.get(o, 'start'));
const legitEnd   = (o) => isLegitNumber(_.get(o, 'end'));


Now we have a lot of functions to verify if our event is acceptable. However, if it’s not acceptable, we don’t know why. Worse, users of your Lambda, both you in 2 weeks when you forgot your code, and other API consumers, won’t have any clue what they did either without cracking open the CloudWatch logs + your code and attempting to debug it.

We’ll take this a step further by using those validator and checker functions we imported above. Second, the validators:

// validators
const validObject = validator('Not an Object.', _.isObjectLike);
const validEvent  = validator('Invalid event, missing key properties.', legitEvent);
const validStart  = validator('start is not a valid number.', legitStart);
const validEnd    = validator('end is not a valid number.', legitEnd);


These functions are normal, they just take advantage of JavaScript and just about everything being a dynamic Object. That first parameter, the string error message, you can store on the function, so if it returns false, you know WHY it returned false. The checkers will accumulate those errors using a reduce function. Third, the checkers:

// checkers
const checkEvent       = checker(validObject, validEvent);
const checkStartAndEnd = checker(validStart, validEnd);


Two more predicates and we’re done. All Lambdas are required to have at least 1 response to not blow up. However, you and I know code either works or it doesn’t. There is middle ground, sure, but for simple stuff, it’s black and white. We’ll break those out into two predicates for creating our HTTP responses of errors:

const getErrorResponse = (errors)=>
{
    return {
        statusCode: '500',
        body: JSON.stringify({result: false, error: errors.join('\n')}),
        headers: {
            'Content-Type': 'application/json',
        }
    };
};


And success:

const getResponse = (data)=>
{
    return {
        statusCode: '200',
        body: JSON.stringify({result: true, data}),
        headers: {
            'Content-Type': 'application/json',
        }
    }
};


Armed with our predicates, we can have a flexible handler, and if something blows up, we will know why. Let’s break her down into 5 steps:

const handler = (event, context, callback) =>
{
    if(_.isNil(callback) === true)
    {
        return getErrorResponse(['No callback was passed to the handler.']);
    }
    ...


We’re ok with no event and context, but no callback!? That’s crazy talk. Here’s your t3h boom.

const errors = checkEvent(event);
if(errors.length > 0)
{
    callback(new Error(errors.join('\n')));
    return getErrorResponse(errors);
}


If our event isn’t legit (either and echo, or having start and end numbers), we send the array of errors we get back to whoever triggered us in an error callback. Instead of “I didn’t work”, they’ll have a fighting chance of knowing why since we sent them the validation messaging.

Quick Security Note

I should point out AWS walks the line of being secure and not giving you verbose errors while sometimes giving you what they can without compromising security to help you debug as a developer. You’ll note that my checkers tend to be verbose in the hope they’ll help whoever made a mistake. However, as things scale, you must be careful not to expose public information, or reveal too much about what you DON’T validate. I’m not a security guy, I don’t have the answers beyond lots of peer review of code, automated quality checks, and automated security scanning. You’ll note the obvious of not throwing stack traces back to the client. You can see those in CloudWatch if you wish.

if(event.echo === true)
{
    const echoResponse = getResponse('pong');
    callback(undefined, echoResponse);
    return echoResponse;
}


That’s for our future integration tests. It’s easier if our remote code is aware she’ll be pinged to see if she’s alive and well. We test for it to ensure it doesn’t negatively affect others. You can see the work that went into validating it as well as ensuring it played nice with others, yet still supported the ability to be tested without actually doing real work that could lead to leaky state.

const startEndErrors = checkStartAndEnd(event);
if(startEndErrors.length > 0)
{
    callback(new Error(startEndErrors.join('\n')));
    return getErrorResponse(startEndErrors);
}


Finally, we check to ensure if we’re going to do the random number generation, we have what we need from the event to do so, else, blow up and explain why. The real work is the end of the function:

const start        = _.get(event, 'start');
const end          = _.get(event, 'end');
const randomNumber = getRandomNumberFromRange(start, end);
const response     = getResponse(randomNumber);
    callback(undefined, randomNumber);
return response;
};


Now re-running your tests should result in them all passing:

Image title

Manual Test

One last manual test you can do as well is simply run her in the node REPL. In the Terminal, type “node” and hit enter.

Image title

Then import your index module by typing lambda =require('./index') and hitting enter:

Image title

You’ll see our three functions we exposed. AWS only cares about your handler, so let’s manually test ours with some bogus data. Type lambda.handler() and hit enter, and you should see a 500 response:

Image title

Now let’s mirror our unit test by using, and give it some good inputs via handler({echo: true}, {}, ()=>{}); to get a basic 200 response:

Image title

You can Control + C twice to get out of Node.

Skills. Unit tests work, and a manual test works. Now you can be assured if you upload to AWS and she breaks, it’s them not you. Yes, your problem, but thankfully your self-esteem shall remain intact. Remember, part of programming is deflecting blame to others, backing it up with fancy terms like “I have lots of TDD code coverage”, then fixing “their” problem and looking like a hero.

Deploy Testable Code to AWS To Test There

Speaking of AWS, let’s redeploy and test our more testable code up on AWS. This’ll be a common task you do again and again by testing code locally, then re-deploying to test it on AWS. We’ll suffer through it for a bit so we appreciate the automating of it later.

For now, let’s adjust your makezip script in package.json to add our new files. We have to add predicates.js and our libraries which are in node_modules:

"makezip": "zip -r -X deploy.zip index.js predicates.js node_modules",


We’ll hardcode our build script, for now, to destroy our stack first, then recreate it with whatever deploy.zip it finds locally. Open up build.js, and at the bottom, let’s chain together our deleteFunction & createFunction:

deleteFunction(lambda, (err, data)=>{
    log("deleteFunction");
    log("err:", err);
    log("data:", data);
    createFunction(lambda, fs, (err, data)=>
    {
        log("createFunction");
        log("err:", err);
        log("data:", data);
    });
});


You may get an error the first time since no function may be up there to delete and that’s ok. We’re creating and fixing one thing at a time. For now, it’s good enough if she creates your zip file and uploads it to your newly created Lambda function. We’re not differentiating between dependencies and development dependencies in node_modules, so your deploy.zip will be quite large, and may take more time to upload now that she’s not just under 1kb of text.

Run npm run deletezip, then npm run makezip, then node build… or just:

npm run deletezip && npm run makezip && node build


Log into your AWS Console and under Services choose Lambda. You should see your function in the list (they’re often sorted by newest up top). Notice she’s 4+ megs, t3h lulz. #myFLAFilesWereBigRaR #backInTheDay

Click it, and let’s test it. You’ll see a big blue button at the top called “Test”. Click it. It should blow up with our custom blow up message:

Image title

Hot, let’s see if she correctly responds to our manual integration test. Click “Actions” and “Configure Test Event”. Here, you can basically make up your own event JSON to test your Lambda and it’ll run on AWS infrastructure. Ours is pretty simple, echo true. When done click Save and Test.

Image title

Now be careful; sometimes this window has a glitch where it’ll save “echo”: “true” instead of “echo”: true. The “true” String is not the same as the true Boolean that we want. All goes well, you’ll see:

Image title

DAT PONG! Last manual test, let’s generate a random number. Again click Actions and Configure Test Event, and replace the fixture with:

{
    "start": 1,
    "end": 10
}


Save and Test…

Image title

I got a 5, what’d you get?

Until Next Time...

That's all for now! We dived in and tested our functions, our network communication, and plenty more. Next time, we'll build a command line tool for some easy of use. After that, get ready for blue green deployments.

New Mesosphere DC/OS 1.10: Production-proven reliability, security & scalability for fast-data, modern apps. Register now for a live demo.

Topics:
cloud ,aws ,infrastructure as code ,microservices ,tdd ,tutorial

Published at DZone with permission of Jesse Warden, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}