File tree

5 files changed

+291
-1
lines changed

5 files changed

+291
-1
lines changed
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
MIT License
22

3-
Copyright (c) 2020 Tobi
3+
Copyright (c) 2020 TobiLG <[email protected]>
44

55
Permission is hereby granted, free of charge, to any person obtaining a copy
66
of this software and associated documentation files (the "Software"), to deal
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
# python-lambda-layer-builder
2+
3+
Creates an AWS Lambda Layers structure that is **optimized** for: [Lambda Layer directory structure](https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html#configuration-layers-path), compiled library compatibility, and minimal file size.
4+
5+
This repo was created to address these issues:
6+
7+
- Many methods of creating Lambda zip files for Python functions don't work for Lambda Layers
8+
- This is due to the fact Lambda Layers require specific library paths within the zip, where regular Lambda zips don't
9+
- Compiled dependencies must be created in an environment that matches the Lambda runtime
10+
- Reduce size of the layer by removing unnecessary libraries and files
11+
12+
**Note: This script requires Docker and uses a container to mimic the Lambda environment.**
13+
14+
## Features
15+
16+
- Builds either a zip file or a raw directory strucutre (e.g. if you want to use frameworks like Serverless for packaging) containing Python dependencies and places the libraries into the proper directory structure for lambda layers
17+
- Ensures compiled libraries are compatible with Lambda environment by using the [lambci/lambda](https://hub.docker.com/r/lambci/lambda) Docker container that mimics the lambda runtime environment
18+
- Optimized the zip size by removing `.pyc` files and unnecessary libraries
19+
- allows specifying lambda supported python versions: 2.7, 3.6 and 3.7
20+
- Automatically searches for requirements.txt file in several locations:
21+
- same directory as script
22+
- parent directory or script (useful when used as submodule)
23+
- function sub-directory of the parent directory
24+
25+
## Installation
26+
27+
This function can be **cloned** for standalone use, into a parent repo or added as a **submodule**.
28+
29+
Clone for standalone use or within a repo:
30+
31+
``` bash
32+
# If installing into an exisiting repo, navigate to repo dir
33+
git clone --depth 1 https://.com/tobilg/python-lambda-layer-builder _build_layer
34+
```
35+
36+
Alternatively, add as a submodule:
37+
38+
``` bash
39+
cd {repo root}
40+
git submodule add https://.com/tobilg/python-lambda-layer-builder _build_layer
41+
# Update submodule
42+
git submodule update --init --recursive --remote
43+
```
44+
45+
## Usage
46+
47+
```text
48+
$ ./build.sh -h
49+
AWS Lambda Layer Builder for Python Libraries
50+
51+
Usage: build.sh [-p PYTHON_VER] [-n NAME] [-r] [-h] [-v]
52+
-p PYTHON_VER : Python version to use: 2.7, 3.6, 3.7 (default 3.7)
53+
-n NAME : Name of the layer
54+
-r : Raw mode, don't zip layer contents
55+
-h : Help
56+
-v : Display build.sh version
57+
```
58+
59+
- Run the builder with the command `./build.sh`
60+
- or `_build_layer/build.sh` if installed in sub-dir
61+
- It uses the first requirements.txt file found in these locations (in order):
62+
- Same directory as script
63+
- Parent directory of script (useful when used as submodule)
64+
- Function sub-directory of the parent directory (useful when used as submodule)
65+
- Optionally specify Python Version
66+
- `-p PYTHON_VER` - specifies the Python version: 2.7, 3.6, 3.7 (default 3.6)
67+
68+
### Custom cleaning logic
69+
70+
You can edit the `_clean.sh` file if you want to add custom cleaning logic for the build of the Lambda layer. The above part of the file must stay intact:
71+
72+
```bash
73+
#!/usr/bin/env bash
74+
# Change to working directory
75+
cd $1
76+
# ----- DON'T CHANGE THE ABOVE -----
77+
78+
# Cleaning statements
79+
# ----- CHANGE HERE -----
80+
rm test.xt
81+
```
82+
83+
The `_make.sh` script will then execute the commands after the Python packages have been installed.
84+
85+
## Deinstallation
86+
87+
If installed as submodule and need to remove
88+
89+
```bash
90+
# Remove the submodule entry from .git/config
91+
git submodule deinit -f $submodulepath
92+
# Remove the submodule directory from the superproject's .git/modules directory
93+
rm -rf .git/modules/$submodulepath
94+
# Remove the entry in .gitmodules and remove the submodule directory located at path/to/submodule
95+
git rm -f $submodulepath
96+
# remove entry in submodules file
97+
git config -f .git/config --remove-section submodule.$submodulepath
98+
```
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
#!/usr/bin/env bash
2+
# Change to working directory
3+
cd $1
4+
# ----- DON'T CHANGE THE ABOVE -----
5+
6+
# Cleaning statements
7+
# ----- CHANGE HERE -----
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
#!/usr/bin/env bash
2+
3+
set -e
4+
5+
# Credits for initial version: https://.com/robertpeteuil/build-lambda-layer-python
6+
7+
# AWS Lambda Layer Zip Builder for Python Libraries
8+
# This script is executed inside a docker container by the "build_layer.sh" script
9+
# It builds the zip file with files in lambda layers dir structure
10+
# /python/lib/pythonX.X/site-packages
11+
12+
scriptname=$(basename "$0")
13+
scriptbuildnum="1.0.0"
14+
scriptbuilddate="2020-03-29"
15+
16+
### Variables
17+
CURRENT_DIR=$(reldir=$(dirname -- "$0"; echo x); reldir=${reldir%?x}; cd -- "$reldir" && pwd && echo x); CURRENT_DIR=${CURRENT_DIR%?x}
18+
PYTHON="python${PYTHON_VER}"
19+
ZIP_FILE="${NAME}_${PYTHON}.zip"
20+
21+
echo "Building layer: ${NAME} for ${PYTHON}"
22+
23+
# Delete build dir
24+
rm -rf /tmp/build
25+
26+
# Create build dir
27+
mkdir -p /tmp/build
28+
29+
# Create virtual environment and activate it
30+
virtualenv -p $PYTHON /tmp/build
31+
source /tmp/build/bin/activate
32+
33+
# Install requirements
34+
pip install -r /temp/build/requirements.txt --no-cache-dir
35+
36+
# Create staging area in dir structure req for lambda layers
37+
mkdir -p "/tmp/base/python/lib/${PYTHON}"
38+
39+
# Move dependancies to staging area
40+
mv "/tmp/build/lib/${PYTHON}/site-packages" "/tmp/base/python/lib/${PYTHON}"
41+
42+
# Remove unused stuff
43+
cd "/tmp/base/python/lib/${PYTHON}/site-packages"
44+
echo "Original layer size: $(du -sh . | cut -f1)"
45+
rm -rf easy-install*
46+
rm -rf wheel*
47+
rm -rf setuptools*
48+
rm -rf virtualenv*
49+
rm -rf pip*
50+
find . -type d -name "tests" -exec rm -rf {} +
51+
find . -type d -name "test" -exec rm -rf {} +
52+
find . -type d -name "__pycache__" -exec rm -rf {} +
53+
find -name "*.so" -not -path "*/PIL/*" | xargs strip
54+
find -name "*.so.*" -not -path "*/PIL/*" | xargs strip
55+
find . -name '*.pyc' -delete
56+
if [[ -f "/temp/build/_clean.sh" ]]; then
57+
echo "Running custom cleaning script"
58+
source /temp/build/_clean.sh $PWD
59+
fi
60+
echo "Final layer size: $(du -sh . | cut -f1)"
61+
62+
# Delete .pyc files from staging area
63+
cd "/tmp/base/python/lib/${PYTHON}"
64+
find . -name '*.pyc' -delete
65+
66+
# Produce output
67+
if [[ "$RAW_MODE" = true ]]; then
68+
# Copy raw files to layer directory
69+
echo "${CURRENT_DIR}/layer"
70+
mkdir -p "${CURRENT_DIR}/layer"
71+
cp -R /tmp/base/. "${CURRENT_DIR}/layer"
72+
echo "Raw layer contents have been copied to the 'layer' subdirectory"
73+
else
74+
# Add files from staging area to zip
75+
cd /tmp/base
76+
zip -q -r "${CURRENT_DIR}/${ZIP_FILE}" .
77+
echo "Zipped layer size: $(ls -s --block-size=1048576 ${CURRENT_DIR}/${ZIP_FILE} | cut -d' ' -f1)M"
78+
fi
79+
80+
echo -e "\n${NAME} layer creation finished"
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
#!/usr/bin/env bash
2+
3+
set -e
4+
5+
# Credits for initial version: https://.com/robertpeteuil/build-lambda-layer-python
6+
7+
# AWS Lambda Layer Zip Builder for Python Libraries
8+
# requires: docker, _make.zip.sh, build_layer.sh (this script)
9+
# Launches docker container from lambci/lambda:build-pythonX.X image
10+
# where X.X is the python version (2.7, 3.6, 3.7) - defaults to 3.7
11+
# Executes build script "_make.zip.sh" within container to create zip
12+
# with libs specified in requirements.txt
13+
# Zip filename includes python version used in its creation
14+
15+
scriptname=$(basename "$0")
16+
scriptbuildnum="1.0.0"
17+
scriptbuilddate="2020-03-29"
18+
19+
# Used to set destination of zip
20+
SUBDIR_MODE=""
21+
22+
# Display version
23+
displayVer() {
24+
echo -e "${scriptname} v${scriptbuildnum} (${scriptbuilddate})"
25+
}
26+
27+
# Display usage
28+
usage() {
29+
echo -e "AWS Lambda Layer Builder for Python Libraries\n"
30+
echo -e "Usage: ${scriptname} [-p PYTHON_VER] [-n NAME] [-r] [-h] [-v]"
31+
echo -e " -p PYTHON_VER\t: Python version to use: 2.7, 3.6, 3.7 (default 3.7)"
32+
echo -e " -n NAME\t: Name of the layer"
33+
echo -e " -r\t\t: Raw mode, don't zip layer contents"
34+
echo -e " -h\t\t: Help"
35+
echo -e " -v\t\t: Display ${scriptname} version"
36+
}
37+
38+
# Handle configuration
39+
while getopts ":p:n:rhv" arg; do
40+
case "${arg}" in
41+
p) PYTHON_VER=${OPTARG};;
42+
n) NAME=${OPTARG};;
43+
r) RAW_MODE=true;;
44+
h) usage; exit;;
45+
v) displayVer; exit;;
46+
\?) echo -e "Error - Invalid option: $OPTARG"; usage; exit;;
47+
:) echo "Error - $OPTARG requires an argument"; usage; exit 1;;
48+
esac
49+
done
50+
shift $((OPTIND-1))
51+
52+
# Default Python to 3.7 if not set by CLI params
53+
PYTHON_VER="${PYTHON_VER:-3.7}"
54+
NAME="${NAME:-base}"
55+
CURRENT_DIR=$(reldir=$(dirname -- "$0"; echo x); reldir=${reldir%?x}; cd -- "$reldir" && pwd && echo x); CURRENT_DIR=${CURRENT_DIR%?x}
56+
BASE_DIR=$(basename $CURRENT_DIR)
57+
PARENT_DIR=${CURRENT_DIR%"${BASE_DIR}"}
58+
RAW_MODE="${RAW_MODE:-false}"
59+
60+
# Find location of requirements.txt
61+
if [[ -f "${CURRENT_DIR}/requirements.txt" ]]; then
62+
REQ_PATH="${CURRENT_DIR}/requirements.txt"
63+
echo "Using requirements.txt from script dir"
64+
elif [[ -f "${PARENT_DIR}/requirements.txt" ]]; then
65+
REQ_PATH="${PARENT_DIR}/requirements.txt"
66+
SUBDIR_MODE="True"
67+
echo "Using requirements.txt from ../"
68+
elif [[ -f "${PARENT_DIR}/function/requirements.txt" ]]; then
69+
REQ_PATH="${PARENT_DIR}/function/requirements.txt"
70+
SUBDIR_MODE="True"
71+
echo "Using requirements.txt from ../function"
72+
else
73+
echo "Unable to find requirements.txt"
74+
exit 1
75+
fi
76+
77+
# Find location of _clean.sh
78+
if [[ -f "${CURRENT_DIR}/_clean.sh" ]]; then
79+
CLEAN_PATH="${CURRENT_DIR}/_clean.sh"
80+
echo "Using clean.sh from script dir"
81+
elif [[ -f "${PARENT_DIR}/_clean.sh" ]]; then
82+
CLEAN_PATH="${PARENT_DIR}/_clean.sh"
83+
echo "Using clean.sh from ../"
84+
else
85+
echo "Using default cleaning step"
86+
fi
87+
88+
if [[ "$RAW_MODE" = true ]]; then
89+
echo "Using RAW mode"
90+
else
91+
echo "Using ZIP mode"
92+
fi
93+
94+
# Run build
95+
docker run --rm -e PYTHON_VER="$PYTHON_VER" -e NAME="$NAME" -e RAW_MODE="$RAW_MODE" -v "$CURRENT_DIR":/var/task -v "$REQ_PATH":/temp/build/requirements.txt -v "$CLEAN_PATH":/temp/build/_clean.sh "lambci/lambda:build-python${PYTHON_VER}" bash /var/task/_make.sh
96+
97+
# Move ZIP to parent dir if SUBDIR_MODE set
98+
if [[ "$SUBDIR_MODE" ]]; then
99+
ZIP_FILE="${NAME}_python${PYTHON_VER}.zip"
100+
# Make backup of zip if exists in parent dir
101+
if [[ -f "${PARENT_DIR}/${ZIP_FILE}" ]]; then
102+
mv "${PARENT_DIR}/${ZIP_FILE}" "${PARENT_DIR}/${ZIP_FILE}.bak"
103+
fi
104+
mv "${CURRENT_DIR}/${ZIP_FILE}" "${PARENT_DIR}"
105+
fi

0 commit comments

Comments
 (0)