Using a custom runtime for Go-based Lambda functions
Despite Go having native support in AWS Lambda, switching to a custom runtime has its advantages
Go is a supported language in AWS Lambda, so you may find the title of this tutorial a bit of a paradox. Afterall, why would you want to use a feature traditionally used to run unsupported languages for your Go-based functions? It turns out Go support in Lambda has a few gaps when compared to other natively supported languages that can be filled by using a custom runtime.
However, we are getting ahead of ourselves. Before we dive in, we need to discuss Lambda’s mechanism for supporting languages and how native support looks for Go.
Lambda runtimes: native vs custom
A runtime is an option presented to you when you are creating a Lambda function. It defines how the Lambda function will invoke your handler. For the purposes of this discussion, they fall into two categories - native and custom.
- A native runtime - Defines everything needed to invoke your handler function for a given language. For example, the native runtime for Python includes the Python Interpreter for running your handler code. For languages that support a native runtime, using Lambda is as simple as packaging your code and uploading it.
- A custom runtime - Requires the user to provide the logic needed to invoke the handler code. This logic must be included in your Lambda code package as an executable called bootstrap. During initialization of your Lambda function, bootstrap is invoked to perform all the initialization logic of your provided handler. After doing so it interacts with the Lambda Runtime API to fetch events to forward to your handler. Custom runtimes are a way to add support to languages that do not provide a native runtime. Rust is an example of a language that currently must run in a custom runtime.
The native runtime for Go
AWS provides a native runtime for Go called go1.x. This native runtime is unique compared to other native runtimes as it requires you to wrap your handler code with a library provided by AWS.
package main
import (
"fmt"
"context"
"github.com/aws/aws-lambda-go/lambda"
)
type Event struct {
Name string `json:"name"`
}
func main() {
h := func HandleRequest(ctx context.Context, e Event) (string, error) {
return fmt.Sprintf("Invoked by: %s", e.Name ), nil
}
lambda.Start(h)
}
The lambda.Start function takes your handler and wraps it in a net/rpc server. During cold start, this server binds itself to a local socket and listens for a program built into the runtime to send it events.
Three advantages to using a custom runtime for Go
Given that AWS has provided a native runtime for Go-based Lambdas, you may be wondering why you would want to use a custom runtime instead. In my opinion, there are three advantages to building a custom runtime:
1. Support for Lambda extensions
Lambda Extensions are a way to provide extended functionality outside of your handler code. One example is Hashicorp's Vault extension which allows you to fetch secrets stored in Vault and add them to your Lambda function's environment. One of the major benefits of Extensions is they can be written in a single language and can be shared across all runtimes.
The native runtime for Go is the only runtime that does not support Lambda Extensions. Given that the goal of Extensions is to write once and use everywhere, this lack of support means workaround solutions have to be developed for Lambda functions using the native runtime. If you want to go all-in on Lambda Extensions for your common Lambda tasks, using a custom runtime for Go would provide a unified experience across all language runtimes.
2. Provides an Amazon Linux 2 execution environment
AWS is in the process of shifting newer Lambda runtimes to use an Amazon Linux 2 (AL2) execution environment. All languages with a native runtime have an AL2-based runtime with the exception of Go. Upgrading to AL2 is key to ensuring you have the most up to date execution environment as Amazon Linux (AL1) has transitioned to end of life support.
So what is the plan for Go? Per the AL2 migration guide, AWS has not made plans to update the native runtime and are recommending a switch to the AL2 variant of the Custom Runtime called provided.al2.
3. Unifies the runtime and handler code
As previously described, the native runtime for Go is split into two programs that communicate via RPC calls. A custom runtime provides an advantage as your handler code is compiled into the same binary as the runtime code. The benefit is all the RPC logic is removed and the handler code can be directly invoked. This helps improve execution time by cutting out the multiple encodings/decodings needed to transfer the event from the runtime to the handler in the native runtime.
Additionally, the library used for RPC was placed under code freeze by the Go team and is not accepting new features or bug fixes. Removing reliancy on this package in production code is worth considering.
Migrating from the Go native runtime to a custom runtime
If any of the advantages mentioned above sound appealing, then this section will help you migrate from the native runtime to a custom runtime.
Change #1 - Build bootstrap binary
For your handler code to run in a custom runtime you need to package it with code that handles the Lambda Runtime API into a binary called bootstrap. The aws-lambda-go library you are already using provides a custom runtime mode out of the box. The library has logic to determine whether it is running in a native or custom runtime. This means all you need to do is change your build process to rename the produced binary to bootstrap and it will work in a custom runtime without any changes to the handler code.
Example:
GOOS=linux go build -o ./bootstrap ./...
Change #2 - Remove RPC for faster cold-start
The aws-lambda-go library provides a build tag called lambda.norpc to remove all RPC logic from the built binary, reducing its size. Binary size contributes to Lambda cold-start time, so any reduction we make will help improve this metric.
Example:
GOOS=linux go build -o ./bootstrap ./... -tags lambda.norpc
Change #3 - Update Lambda configuration
The final change is to update the runtime and handler properties in your Lambda configuration to use your new bootstrap binary in a custom runtime. For runtime, you will switch from go1.x to provided.al2. The provided.al2 setting indicates that we will “provide” custom runtime code to run on an Amazon Linux 2 execution environment. For the handler switch from your current value to bootstrap.
Example (CloudFormation):
Resources:
Function:
Type: AWS::Serverless::Function
Properties:
Handler: bootstrap
Runtime: provided.al2
... # Other required properties
Putting it all together
With these three changes, you should be able to utilize all the advantages custom runtimes can bring to your Go-based Lambdas. So the next time you are writing a Go-based Lambda, consider using a custom runtime!