Serverless with AWS Lambda and Scala

About a year ago, I started looking at AWS's serverless offering, AWS Lambda. The premise is relatively simple: rather than a full server running that you manage and deploy your Docker/web servers to, you just define a single function endpoint and map that to the API gateway and you have an infinitely* scaleable endpoint.

The appeal is fairly obvious — no maintenance or upgrading servers, fully scalable, and pay-per-second of usage (so no cost for AWS Lambda functions that you define whilst not being called). I haven't looked into the performance of using the JVM-based Lambda functions, but my assumption is that there will be potential performance costs if your function isn't frequently used, as AWS will have to start up the function, load its dependencies, etc., so depending on your use case, it would be advisable to look at performance benchmarking before putting into production use.

When I first looked into AWS Lambda a year ago, it was less mature than it is today, and dealing with input/output JSON objects required annotating POJOs, so I decided to start putting together a small library to make it easier to work with AWS Lambda in a more idiomatic Scala way — using Circe and it's automatic encoder/decoder generation with Shapeless. The code is all available on GitHub.

Getting Started

To deploy on AWS I used a framework called Serverless — this is a really easy framework to set up serverless functions on a range of cloud providers. Once you have followed the pre-requisite install steps, you can simply run:

serverless create --template aws-java-gradle

This will generate you a Java (JVM)-based Gradle project template, with a YML configuration file in the root that defines your endpoints and function call. If you look in the src folder as well, you will also see the classes for a very simple function that you can deploy and check your Lambda works as expected. You should also take the time at this point to log in to your AWS console and have a look at what has been created in the Lambda and API Gateway sections. You should now be able to curl your API endpoint (or use the serverless CLI with a command like  serverless invoke -f YOUR_FUNCTION__NAME -l).

ScaLambda — AWS Lambda with Idiomatic Scala

Ok, so we have a nice simple Java-based AWS Lambda function deployed and working, let's looking at moving it to Scala. As you try to build an API in this way you will need to be able to define endpoints that can receive inbound JSON being posted as well as return fixed JSON structures — AWS provides its inbuilt de/serialisation support, but inevitably you will have a type that might need further customization of how it is de/serialized (UUIDs maybe, custom date formats, etc.) and there are a few nice libraries that can handle this stuff and Scala has some nice ways that can simplify this.

We can simply upgrade our new Java project to a Scala one (either convert the build.gradle to an sbt file, or just add Scala dependency/plugins to the build file as is) and then add the dependency:

repositories {
    maven {
        url  "https://dl.bintray.com/robhinds/snapshots" 
    }
}

dependencies {
    compile 'io.github.robhinds:ScaLambda:0.0.1'
}
view raw

We can now update the input/output classes so they are just normal Scala case classes:

case class TestInput(value: String)
case class TestOutput(value: String)

Not a huge change from the POJOs we had, but is both more idiomatic and also means you can use case classes that you have in other existing Scala projects/libraries elsewhere in your tech stack.

Next we can update the request handler — this will also result in quite similar looking code to the original generated Java code, but will be in Scala and will be backed by Circe and it's automatic JSON encoder/decoder derivation.

class TestFunction extends Controller[TestInput, TestOutput] 
  with DefaultResponseSerializerComponent 
  with DefaultExceptionHandler {
    
  override def handleRequest(in: TestInput): ApiResponse[TestOutput] = 
    success(TestOutput(s"OUTPUT:${in.value}"))
        
}

You will see that similar to the AWS Java class we define generic parameter types for the class that represents the input case class and the output case class and then you simply implement the handleRequest method which expects the input class and returns the output response.

You might notice the return type is wrapped in the ApiResponse  class; this is simply an alias for a Scala  Either[ Exception , T ] , which means if you need to respond with an error from your function you can just return an exception rather than the TestOutput . To simplify this, there is an ApiResponse  companion object that provides a success  and failure  method:

object ApiResponse {
  def failure[T](e :Exception): ApiResponse[T] = Left(e)
  def success[T](t: T): ApiResponse[T] = Right(t)
}

All the JSON serialization/de-serialization will use Circe's auto-derived code which relies on Shapeless — if you use custom types that cannot be automatically derived, then you can just define implicit encoder/decoders for your type and they will be used.

Error Handling

The library also has support for error handling — as the ApiResponse class supports returning exceptions, we need to map those exceptions back to something that can be returned by our API. To support this, the Controller class that we have implemented for our Lambda function expects (via self type annotations) to be provided an implementation of the ExceptionHandlerComponent trait and of the ResponseSerializerComponent trait.

Out of the box, the library provides a default implementation of each of these that can be used, but they can easily be replaced with custom implementations to handle any custom exception handling required:

trait DefaultExceptionHandler extends ExceptionHandlerComponent {
  private val l = Logger(classOf[DefaultExceptionHandler])
  override def exceptionHandler: ExceptionHandler = new ExceptionHandler {
    override def handle[B](e: ApiResponse[B]): Either[ErrorResponse, B] = e match {
      case Left(x: JsonError) => errorResponse("400", s"Error de-serialising JSON: ${x.getMessage}")
      case Left(x: NotFound) => errorResponse("404", x.message)
      case Left(x: BadRequest) => errorResponse("400", x.message)
      case Left(x: InternalServerError) => errorResponse("500", x.message)
      case Left(x) => errorResponse("500", x.getMessage)
      case Right(x) => Right(x)
    }
  }
}

Custom Response Envelopes

We mentioned above that we also need to provide an implementation of the ResponseSerializerComponent  trait. A common pattern in building APIs is the need to wrap all response messages in a custom envelope or response wrapper — we might want to include status codes or additional metadata (paging, rate limiting, etc.) — this is the job of the ResponseSerializerComponent. The default implementation simply wraps the response inside a basic response message with a status code included, but this could easily be extended/changed as needed.

trait DefaultResponseSerializerComponent extends ResponseSerializerComponent {
  override def responseSerializer: ResponseSerializer = new ResponseSerializer {
    override def serialiseResponse[B: Encoder](e: Either[ErrorResponse, B]): Json = e match {
      case Left(x) => x.asJson
      case Right(x) =>
        Json.fromFields(List(
          ("status", Json.fromString("200")),
          ("data", x.asJson)
      ))
    }
  }
}

Conclusion

The project is still in early stages of exploring the AWS Lambda stuff, but hopefully is starting to provide a useful approach to idiomatic Scala with AWS Lambda functions, allowing re-use of error handling and serialization so you can just focus on the business logic required for the function.