Posts Testable Lambda
Post
Cancel

Testable Lambda

Introduction

In this article I’ll show you how to easily run integration tests every time you deploy new Lambda code.

If you already have a full suite of integration tests that run whenever you deploy infrastructure changes and new code then you can stop reading now. You obviously already know the benefits to both the business and your blood pressure of having the changes you make to your production environment just plain work. Every time.

If, however, writing integration tests and getting them to run automatically has always seemed like more hassle than it’s worth, keep reading. This post will hopefully demystify the process and motivate you to add them to your testing tool belt.

For a working example of the code described in this article, visit the Testable Lambda Github repository.

Integration test definition

Before getting started, I want to make sure we are on the same page regarding the term “integration test.” For this article, an integration test is responsible for verifying that the function being tested can talk to (is integrated with) the backend services it uses (e.g., another microservice or a database).

Integration tests are not responsible for ensuring the function being tested actually does what it’s supposed to do. That’s what unit and functional tests are for.

Integration tests also differ from end-to-end tests in that an integration test validates an individual component while an end-to-end test exercises a path though the entire system, usually from the user interface’s perspective.

Architecture

There are two components to automated integration tests:

  1. The service that runs the tests on every deployment
  2. The integration tests themselves that validate the new deployment

This article describes using CodeDeploy to run the integration tests on each deployment. Since CodeDeploy is part of the infrastructure, I define it in code using the AWS CDK.

The test itself is just another Lambda function. The example test is written in Node, but you can use any language supported by Lambda.

Sequence

At a high level, here’s what happens during a successful deployment:

High-level integration test sequence diagram

  1. A DevOps person runs cdk deploy to create and deploy a CloudFormation template.

  2. CloudFormation creates a new version of the main Lambda. (If the main Lambda hasn’t changed, CloudFormation skips the CodeDeploy process entirely.)

  3. CloudFormation starts a CodeDeploy deployment.

  4. CodeDeploy invokes the test Lambda.

  5. The test Lambda invokes the new (but not yet live) version of the main Lambda.

  6. The test Lambda verifies the main Lambda behaved as expected.

  7. The test Lambda tells CodeDeploy if the test passed or failed.

  8. Assuming test passes, CodeDeploy starts to shift some of the live traffic to the new version of the main Lambda.

  9. CodeDeploy monitors CloudWatch Alarms to see if any new problems arise processing production traffic.

  10. If no problems are detected, CodeDeploy shifts all the remaining traffic to the new Lambda version.

  11. CodeDeploy tells CloudFormation the test passed so CloudFormation can finish with the stack deployment. If the test failed, CloudFormation will roll back any changes made from the new deployment

TestableLambda CDK Construct

AWS CodeDeploy pre-deployment hooks can be used to run a test Lambda on every deploy. In order to make it easy to define the infrastructure for this, I created the TestableLambda CDK construct.

Constructor Properties

The TestableLambda class is simple to create with just a few options:

1
2
3
4
5
6
7
interface TestableLambdaProps extends StackProps {
  mainFunction: lambda.Function
  testFunction: lambda.Function
  deploymentConfig: codedeploy.ILambdaDeploymentConfig
  alarms?: cloudwatch.Alarm[]
  liveAliasName?: string
}

mainFunction

mainFunction is the Lambda you want to test. This Lambda can be written in any language and is a black box to the test Lambda. The only constraint is that the Lambda’s source code must be built with the CDK’s fromAsset() or fromInline() methods so the CDK can automatically detect when the main Lambda changes.

testFunction

testFunction is the Lambda that will do the testing. The TestableLambda construct adds a FUNCTION_TO_INVOKE environment variable to the to the test Lambda for it knows which version of the main Lambda to test.

deploymentConfig

deploymentConfig defines how CodeDeploy should roll out the new Lambda version after the integration tests pass:

  • All-at-once: All traffic is shifted to the new version immediately
  • Linear: Traffic is shifted to the new lambda in even increments over time
  • Canary: A small percentage of the traffic is shifted to the new version immediately and the rest gets shifted a short while later assuming no errors were encountered

You can read more about the Lambda deployment options here.

alarms

alarms allow you to use CloudWatch Alarms to monitor the health of the system after some live traffic has been routed to the new deployment. This is handy for detecting things your integration tests may miss like overloading a new backend service with production workloads.

When using linear or canary deployments, CodeDeploy will roll back the deployment if any of these alarms are triggered before all the traffic has shifted to the new version. This really powerful feature is, unfortunately, beyond the scope of this post.

liveAliasName

liveAliasName is the name of a pointer to the version of the main Lambda that serves live (production) traffic. The default live alias name is ‘live.’ You can read more about Lambda aliases here.

Code

Thanks to the high-level CDK constructs the TestableLambda code is pretty straightforward:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
export class TestableLambda extends Construct {
  readonly liveAlias: lambda.Alias

  constructor(scope: Construct, id: string, props: TestableLambdaProps) {
    super(scope, id)

    const aliasName = props.liveAliasName || 'live'

    const newVersion = props.mainFunction.currentVersion
    this.liveAlias = newVersion.addAlias(aliasName)

    props.testFunction.addEnvironment('FUNCTION_TO_INVOKE', newVersion.functionArn)
    newVersion.grantInvoke(props.testFunction)

    new codedeploy.LambdaDeploymentGroup(this, `DeploymentGroup`, {
     alias: this.liveAlias,
     deploymentConfig: props.deploymentConfig,
     preHook: props.testFunction,
     alarms: props.alarms
    })
  }
}

The code above does the following:

  1. Detects if there is a new version of the code. This is handled by props.mainFunction.currentVersion.

  2. Assigns the live alias to the new Lambda version. If the main Lambda hasn’t changed then the live alias is already pointing at the currentVersion so the integration test won’t run for this deployment.

  3. Adds the FUNCTION_TO_INVOKE environment variable to the test Lambda

  4. Grants the test Lambda permission to run the main Lambda.

  5. Configures CodeDeploy to run the integration test and monitor CloudWatch Alarms during traffic shifting.

Working with other services

The TestableLambda class exposes aliveAlias property for the Lambda alias pointing to the function version serving production traffic. Use this property elsewhere in the CDK when integrating the main Lambda with other services like API Gateway to ensure they only invoke the latest tested code.

1
2
3
4
5
6
const api = new apig.HttpApi(this, 'testable-lambda-api')
api.addRoutes({
  methods: [apig.HttpMethod.POST],
  path: '/',
  integration: new apig.LambdaProxyIntegration({ handler: testableLambda.liveAlias })
})

Integration Test

While the Lambda doing the testing can be written in any language, this post uses Node. The techniques, however, are generally applicable to any runtime.

The test Lambda needs to invoke the main Lambda once for every test you want to run. For example, if the main Lambda handles GET, PUT and POST requests for a service, the test Lambda would invoke the main Lambda three times, once for each HTTP verb.

Test Lambda handler

Use the runIntegrationTests() helper function in the test Lambda’s handler to easily add and remove tests from the suite:

1
2
3
4
5
6
7
exports.handler = async (event) => {
  await runIntegrationTests(event, [
    testGET(),
    testPUT(),
    testPOST()
])
}

runIntegrationTests()

runIntegrationTests() runs all the passed in tests and updates CodeDeploy with the final status.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
exports.runIntegrationTests = async (event, tests) => {
  let status = 'Succeeded'
  try {
    // All tests run concurrently
    await Promise.all(tests)
  } catch (e) {
    console.log('Error running test: ' + e)
    status = 'Failed'
  }

  await codeDeploy.putLifecycleEventHookExecutionStatus({
    status,
    deploymentId: event.DeploymentId,
    lifecycleEventHookExecutionId: event.LifecycleEventHookExecutionId
  }).promise()

  console.log('Final deployment status: ' + status)
}

The await Promise.all(tests) lines lets all the tests run concurrently and doesn’t return until they have all completed. If one test fails the entire suite fails immediately.

Individual tests

Each test takes essentially the same form:

  1. Setup the test event data
  2. Invoke the main Lambda with the test event
  3. Check the results and return Promise.reject() if something’s wrong

Here’s an example of a very basic test that checks if the main Lambda returned the same text that what was passed to it:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
const { invokeLambda } = require('./invoke-lambda')
const testEvent = require('./test-one-event.json')

exports.testOne = async () => {
  const testString = 'this is a test'
  testEvent.body = testString

  const resp = await invokeLambda({
    Payload: JSON.stringify(testEvent)
  })

  if (!resp.Payload.includes(testString)) {
    console.log(`Unexpected response \nExpecting: ${testString} \nReceived: ${resp}`)
    return Promise.reject(`Unexpected response ${resp}`)
  }
}

This example uses a synchronous invocation where the test Lambda can determine success or failure based on the main Lambda’s response. For more sophisticated use cases or for asynchronous functions, the test Lambda can instead verify the main Lambda’s success by inspecting backend resources (e.g., query the database to look for the expected change).

testEvent

testEvent is an canned Lambda invocation event that exercises the path through the main Lambda code you want to test. For example, here’s a snippet of an API Gateway event:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
{
  "version": "2.0",
  "headers": {
    "accept": "*/*",
    ...
  },
  "body": "hello world",
  "requestContext": {
    "http": {
      "method": "POST",
      "path": "/",
      "protocol": "HTTP/1.1",
      "userAgent": "PostmanRuntime/7.22.0"
    },
    ...
  }
}

For cleanliness, I typically store these events in a separate file (e.g., test-get-event.json). I’ve found the easiest way to generate test events is to deploy, in a development environment, a version of the main Lambda that logs the passed in event to the console. Then I trigger the main Lambda via its event source (e.g., call an API Gateway route or drop a file into an S3 Bucket) and get the event details from CloudWatch Logs.

invokeLambda()

The invokeLambda helper function is a thin wrapper around the AWS SDK lambda.invoke() method:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
exports.invokeLambda = async (overrides) => {
  const params = {
    FunctionName: process.env.FUNCTION_TO_INVOKE,
    InvocationType: 'RequestResponse' // Pass in 'Event' for asynchronous invocations
  }

  const resp = await lambda.invoke(Object.assign(params, overrides)).promise()

  if (resp.StatusCode !== 200) {
    const errorMessage = `Invalid status code invoking ${params.FunctionName}`
    console.log(`${errorMessage} \nResponse = ${JSON.stringify(resp, null, 2)}`)
    return Promise.reject(errorMessage)
  }

  return Promise.resolve(resp)
}

invokeLambda simplifies writing integration tests by:

  • Automatically determining the the function to test (from the FUNCTION_TO_INVOKE environment variable)
  • Converting non-200 invocation status codes to Promise rejections so each individual test doesn’t need to handle these cases

Conclusion

I hope this post has demystified the process of running integration tests against your Lambda functions. With just a few helper functions, you can easily leverage the CDK, CloudFormation, CodeDeploy and CloudWatch Alarms to ensure all your production deployments will either work as expected or be aborted at the first sign of trouble.

Areas for improvement

The runIntegrationTests() function is pretty bare-bones. You may want to replace it with a test framework like Jest or Mocha.

This scheme requires a DevOps person to run cdk deploy on their local machine. As your organization matures and you become more confident in your test coverage, you can use CodePipeline to automatically start a deployment when new code is pushed to a Git repository.

This post is licensed under CC BY 4.0 by the author.