swift-serverless/aws-lambda-swift-sprinter

The goal of this project is to provide an environment to build an AWS Lambda Custom Runtime for the Swift programming language and provide the required support to run swift-nio 2.0.

The project helps building a Swift Lambda based on the framework LambdaSwiftSprinter.

The project contains also some Examples:

• HelloWorld: A basic Lambda Swift example
• HTTPSRequest: A basic example showing how to perform an HTTPS request from the Swift Lambda using the LambdaSwiftSprinterNioPlugin
• S3Test: A basic example showing how to access an S3 bucket from the Swift Lambda using https://.com/swift-aws/aws-sdk-swift.
• RedisDemo: A basic example showing how to connect to Redis from the Swift Lambda using the RediStack client.
• PostgreSQLDemo: A basic example showing how to connect to Redis from the Swift Lambda using the PostgresNIO client.

The AWS Lambdas run on Amazon Linux.

To build swift on Amazon Linux is required to:

• building the code on the official Docker Swift
• extracting the build and all the runtime's shared libraries
• packaging the artefacts and use them as AWS Lambda Custom Runtime

The artefacts required to run the Swift Lambda are split in two parts:

• Lambda Layer: A layer is a ZIP archive that contains libraries, a custom runtime, all the shared libraries required to run the Swift lambda.
• Lambda code depends on the LambdaSwiftSprinter framework: A zip package containing the Swift Lambda main executable and third-party libraries.

• Install Docker from here
• Clone this repository. From the command line type:

git clone https://.com/swift-sprinter/aws-lambda-swift-sprinter
cd aws-lambda-swift-sprinter

• Ensure you can run make:

make --version

the Makefile was developed with this version:

GNU Make 3.81
Copyright (C) 2006  Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

This program built for i386-apple-darwin11.3.0

The docker image swift:latest contains the latest release of Swift. This image does not contain the libraries libssl-dev and libicu-dev required to build swift-nio.

The Dockerfile contains the recipe to build a custom docker image based on the swift:latest with the addition of the swift-nio requirements.

To build the image use the command:

make docker_build

This will build and tag a local docker image called nio-swift:latest.

It's possible to check it by using the following command:

docker image ls

REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
nio-swift           latest              b2b9d0b2a68d        8 days ago          1.45GB
swift               latest              1e9a2a744b48        13 days ago         1.35GB

To prepare the AWS Lambda layer containing all the swift runtime libraries and the bootstrap file:

make package_layer

The output of this command is a ZIP file called swift-lambda-runtime.zip under the folder .build

Here a basic example of the Lambda code:

import Foundation
import LambdaSwiftSprinter

struct Event: Codable {
    let name: String
}

struct Response: Codable {
    let message: String
}

let syncLambda: SyncCodableLambda<Event, Response> = { (event, context) throws -> Response in
    let message = "Hello World! Hello \(event.name)!"
    return Response(message: message)
}

//...

do {
    let sprinter = try SprinterCURL()
    sprinter.register(handler: "helloWorld", lambda: syncLambda)
    try sprinter.run()
} catch {
    log(String(describing: error))
}

More details on how to code a Swift Lambda are documented under the Examples folder:

• HelloWorld
• HTTPSRequest
• S3Test

Refer to the LambdaSwiftSprinter framework documentation to know more.

By default, the Makefile will build the HelloWorld example contained in this repository.

The following command will build and zip the Lambda:

make package_lambda

The output of this command is a ZIP file called lambda.zip under the folder .build.

Can pass values to your make command either before or after the call like below. This will fit better in a script per example to set up continuous integration in your project.

SWIFT_EXECUTABLE=HTTPSRequest \
SWIFT_PROJECT_PATH=Examples/HTTPSRequest \
LAMBDA_FUNCTION_NAME=HTTPSRequest \
LAMBDA_HANDLER=${SWIFT_EXECUTABLE}.getHttps \
    make invoke_lambda

make invoke_lambda \
    SWIFT_EXECUTABLE=HTTPSRequest \
    SWIFT_PROJECT_PATH=Examples/HTTPSRequest \
    LAMBDA_FUNCTION_NAME=HTTPSRequest \
    LAMBDA_HANDLER=HTTPSRequest.getHttps

You can also edit the Makefile to build a different Example by commenting the following lines and uncommenting the line related to the example you want to build.

...

SWIFT_EXECUTABLE?=HelloWorld
SWIFT_PROJECT_PATH?=Examples/HelloWorld
LAMBDA_FUNCTION_NAME?=HelloWorld
LAMBDA_HANDLER?=$(SWIFT_EXECUTABLE).helloWorld

...

The following tutorial describes how to deploy the lambda in your AWS account from the command line using AWS Command Line Interface through the Makefile.

The goal of the deployment is to create and configure a lambda with lambda.zip and the layer with swift-lambda-runtime.zip.

Steps:

• Create a Lambda layer with the swift-lambda-runtime.zip
• Create the lambda with the lambda.zip code and the correct configuration
  • Name
  • Runtime
  • Handler
  • Execution role
  • Function code
  • Test event

There are many ways to achieve a lambda deployment (AWS Console, SAM, CloudFormation ...), please refer to the latest AWS Lambda documentation to know more.

• an AWS account for test purpose.

• aws cli: Install the aws cli. Here the instructions.

• If you want to deploy your lambdas and layers using S3 you need to make sure the bucket in the Makefile already exists. If it doesn't you can create it using the command make create_s3_bucket which will use the value of the variable AWS_BUCKET as a name.

• if your AWS account it doesn't have admin priviledges:

  • Review(*) the policy contained in the file LambdaDeployerPolicyExample.json
  • Attach the policy to your user. Here is the official documentation.

(*) Note: It would be better to restrict the policy to the function and layer you want to create. It's suggested to test the following scripts before using in production.

Create a new lambda layer using the swift-lambda-runtime.zip file. This step is required once, however you are free to run it if you need to update your layer.

make upload_lambda_layer_with_s3

Datetime based versions are created and uploaded to S3 every time your version is created.

make upload_lambda_layer

You can create a new lambda which might take a few minutes using one of the options below:

make create_lambda_with_s3

Datetime based versions are created and uploaded to S3 every time your version is created.

make create_lambda

The lambda is created with the following parameters:

• function-name: $(LAMBDA_FUNCTION_NAME)
  • HelloWorld
• runtime: provided
• handler: $(LAMBDA_HANDLER)
  • HelloWorld.helloWorld
• role: "$(IAM_ROLE_ARN)"
  • a new role will be created with the name: lambda_sprinter_basic_execution
• zip-file:$(LAMBDA_BUILD_PATH)/$(LAMBDA_ZIP) --> ./build/lambda/lambda.zip
• layers: $(LAMBDA_LAYER_ARN)
  • it will use the arn contained in the file generated by the make upload_lambda_layer

This step is required once, if you need to update the lambda use the step 5.

Now the lambda function is ready for testing. The following command invokes the lambda with using the file event.json contained in the project folder.

make invoke_lambda

The output:

{
    "StatusCode": 200,
    "ExecutedVersion": "$LATEST"
}

Result:

{"message":"Hello World! Hello Swift-Sprinter!"}

Note:

The lambda invocation may require some policy to access other AWS Resources. Check the S3Test example to know more.

If needed, you will also be able to update your Lambda using one of the commands below:

make update_lambda_with_s3

make update_lambda

Sometimes you want to go back to a clean slate and we have a command for that which will rely on the parameters you use.

make nuke

That command will clean your local build folders then delete lambdas and layers created based off the configuration you use.

You can delete the S3 bucket where you hold your lambdas and layers with the command below:

make delete_s3_bucket

The docker image could require security updates or could require updates.

To the lambda with a new layer:

1. Rebuild the docker image

make docker_build

2. package the layer

make package_layer

3. upload the layer to AWS

make upload_lambda_layer

4. update the lambda

make update_lambda

Development and test can be simplified by using a sandboxed local environment replicating of the AWS Lambda by using the lambci/lambda:provided Docker image.

The local sandboxed Lambda can be launched just to test the Lambda once or more than once.

To run the sandboxed Lambda environment it's required to share the files contained in the layer, the boostrap and the files required by the build with the lambci/lambda:provided Docker image.

docker run --rm \
-v "$(PWD)/$(LOCAL_LAMBDA_PATH)":/var/task:ro,delegated \
-v "$(PWD)/bootstrap":/opt/bootstrap:ro,delegated \
-v "$(PWD)/$(SHARED_LIBS_FOLDER)":/opt/swift-shared-libs:ro,delegated \
lambci/lambda:provided $(LAMBDA_HANDLER) $(LOCAL_LAMBDA_EVENT)

• $LOCAL_LAMBDA_PATH: the path containing the build
• $SHARED_LIBS_FOLDER: the path containing the extracted swift runtime libraries
• $LAMBDA_HANDLER: the lambda handler
• $LOCAL_LAMBDA_EVENT: the event JSON string

To run the Lambda locally follow the Lambda development workflow to the step 4:

1. Requirements: Clone the repository and install Docker
2. Prepare a custom docker image: make docker_build
3. Build the lambda layer: make package_layer
4. Write the lambda code

To simplify the local execution swift-sprinter added some useful commands:

Build the lambda and copy the artefacts under $LOCAL_LAMBDA_PATH

make build_lambda_local

Invoke the Lambda locally once with Docker and LambCI on port 9001

make invoke_lambda_local_once

Start the local environment with Docker and LambCI on port 9001

make start_lambda_local_env

Keep it running and open a new terminal window to invoke the lambda.

Use ctrl^C to stop it.

Invoke the lambda locally using the endpoint http://localhost:9001 with the aws cli.

make invoke_lambda_local

To test lambda locally with a more complex environment it's possible to use Docker Compose.

All the examples in the repository have docker-compose.yml file to run the example locally listening on the port 9001.

Start the docker-compose test environment

make start_docker_compose_env

Invoke the lambda locally using the endpoint http://localhost:9001 with the aws cli.

make invoke_lambda_local

Stop the docker-compose test environment

make stop_docker_compose_env

• Use MacOS or Linux
• Install Docker
• Clone the repo git clone https://.com/swift-sprinter/aws-lambda-swift-sprinter.git

• From the command line run make docker_build
• From the command line run make package_layer
• From the command line run make package_lambda

• Go to AWS Lambda -> Layers in AWS Console and create a new layer from scratch
• Enter layer name "swift-lambda-runtime-5-1-5"
• Upload the zip file build/swift-lambda-runtime-5-1-5.zip
• Leave "Compatible runtimes" empty.
• Click "Create"
• Copy the arn from the created layer, it's required to set up the lambda.
• Go to AWS Lambda in AWS Console and create a new function from scratch
• Enter function name "benchmark-swift-hello" and select "Provide your own bootstrap" as runtime
• Choose the execution role created above
• Upload the zip file created with function code, it can be found under build/lambda.zip
• Input "HelloWorld.helloWorld" in "Handler" (Executable.Handler)
• Click "Layers"
• Click "Add Layer"
• Click "Provide a layer version"
• Add the arn of the layer you have uploaded previously.
• Click "Add"
• Click "Save"

• Test the function by clicking Test in the top right corner and add the following event:

{
    "name": "Swift-Sprinter"
}

• Give a name to the event, save and then test it.

Contributions are more than welcome! Follow this guide to contribute.

This project has been inspired by the amazing work of the following people:

• Matthew Burke, Capital One : https://medium.com/capital-one-tech/serverless-computing-with-swift-f515ff052919

• Justin Sanders : https://medium.com/@gigq/using-swift-in-aws-lambda-6e2a67a27e03

• Claus Höfele : https://medium.com/@claushoefele/serverless-swift-2e8dce589b68

• Kohki Miki, Cookpad : https://.com/giginet/aws-lambda-swift-runtime

• Toni Sutter : https://.com/tonisuter/aws-lambda-swift

• Sébastien Stormacq : https://.com/sebsto/swift-custom-runtime-lambda

A special thanks to BJSS to sustain me in delivering this project.

This project has been awarded AWS Open Source Promotional Credits for 2020. @AWSOpen