A Lambda Layer is a way to create reusable code that can be used by your Lambda Functions. For example, if you have multiple Lambda Functions that need to make a connection to a database, you could create a Lambda Layer that creates the database connection and then configure your Lambda Functions to use the Lambda Layer to create the database connection.

A Simple Example

For example, let's say your Lambda Functions are Python and let's say you have the following in a file name database_connection.py.

#!/usr/bin/python
import psycopg2


def db_connect(database_name, database_dns_name, username, password, port):



  connection = psycopg2.connect(database=database_name,
                        host=database_dns_name,
                        user=username,
                        password=password,
                        port=port)

  return connection

You will want to create a zip file that contains datbase_connection.py. For example, on a Linux system the zip command can be used to create the zip file.

zip database_connection.zip database_connection.py

Now you can use the aws lambda publish-layer-version command to create a Lambda Layer using the .zip file.

aws lambda publish-layer-version --layer-name database_connection --zip-file fileb://database_connection.zip --compatible-runtimes python3.13

Or in the AWS console, create a Lambda Layer using the zip file, something like this. Make note of the Amazon Resource Number (ARN) of the layer.

Then you can configure your Lambda Functions to use the Lambda Layer and then updating your Lambda Functions code to make a connection to the database using the database_connection function in the Lambda Layer. In your Lambda Function, you would add the following from/import statement.

from <the name of the Python file in your Lambda Layer> import <the name of the Python function in the Python file in your Lambda Layer>

In this example, since the name of the Python file in the Lambda Layer is database_connection.py and the name of the function in database_connection.py is db_connect the Lambda Function would have from database_connection import db_connect so that the Lambda Function can use the db_connect function.

from database_connection import db_connect

def lambda_handler(event, context):

    conn = db_connect()

    cursor.execute("select * from mytable")

    results = cursor.fetchall()

    for row in results:
        print(f"row 0 = {row[0]}")
        print(f"row 1 = {row[1]}")
        print(f"row 2 = {row[2]}")

    cursor.close()
    conn.close()  

A more complex example

Let's say you get something like this when invoking one of your Python Lambda Function. In this example, the Lambda Function cannot import the paramiko module.

{"errorMessage": "Unable to import module 'app': No module named 'paramiko'", "errorType": "Runtime.ImportModuleError", "requestId": "574780f6-db2d-4a5d-a9c2-3bc56613dea8", "stackTrace": []}

There are a few ways to go about resolving this.

• Use pip to install the module and create a zip archive
• Download module tar and create a zip archive

And then

• Use the aws lambda publish-layer-version command to create a Lambda Layer using the .zip file.

According to https://docs.aws.amazon.com/lambda/latest/dg/packaging-layers.html:

Lambda loads the layer content into the /opt directory of that execution environment.

For each Lambda runtime, the PATH variable already includes specific folder paths within the /opt directory.

To ensure that your layer content gets picked up by the PATH variable, include the content in the following folder paths:

In other words, when you create the .zip file, the base directory in the .zip file must be "python/" or "python/lib/python3.x/site-packages/" so that when the .zip file is extracted the files in the .zip file are extracted to /opt/python or /opt/python/lib/python3.x/site-packages/.

You could add the following to your lambda_handler function.

def lambda_handler(event, context):

    import os
        
    for root, dirs, files in os.walk("/opt"):
        print(f"root = {root}")
        print(f"dirs = {dirs}")
        print(f"files = {files}")    

Which should then return something like this.

root = /opt
directories = [python]
files = []

root = /opt/python
directories = [lib]
files = []

root = /opt/python/lib
directories = [python3.12]
files = []

root = /opt/python/lib/python3.12
directories = [site-packages]
files = []

root = /opt/python/lib/python3.12/site-packages
directories = ['pip', 'pip-23.3.2.dist-info']
files = []

Use pip to install the module and create a zip archive

This is usually the method I go with. On a Linux system, use the same version of Python as is being used by your Lambda Function to create a Python virtual environment. If your Linux system does not have the same version of Python as is being used by your Lambda Function, check out my article Install Python on Linux.

Since the base directory must be "python" or "python/lib/python3.x/site-packages" let's create a create a Python virtual environment named "python" so that the base directory name is "python".

python3 -m venv python

Activate the virtual environment.

source python/bin/activate

I almost always first issue the pip list command, and there is almost always a prompt to upgrade pip. 

(python) [john.doe@server1 tmp]$ pip list
pip (9.0.3)
setuptools (39.2.0)
You are using pip version 9.0.3, however version 23.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Let's upgrade pip.

(python) [john.doe@server1 tmp]$ pip install --upgrade pip

The site-packages directory in the Python virtual environment should contain something like this.

(python) [john.doe@server1 tmp]$ ls -l python/lib/python*/site-packages/
-rw-rw-r--. 1 john.doe john.doe  126 Dec 11 05:03 easy_install.py
drwxrwxr-x. 5 john.doe john.doe 4096 Dec 11 05:04 pip
drwxrwxr-x. 2 john.doe john.doe 4096 Dec 11 05:04 pip-21.3.1.dist-info
drwxrwxr-x. 5 john.doe john.doe   89 Dec 11 05:03 pkg_resources
drwxrwxr-x. 2 john.doe john.doe   40 Dec 11 05:03 __pycache__
drwxrwxr-x. 6 john.doe john.doe 4096 Dec 11 05:03 setuptools
drwxrwxr-x. 2 john.doe john.doe 4096 Dec 11 05:03 setuptools-39.2.0.dist-info

Now install the module that your AWS Lambda Function is saying cannot be found.

(python) [john.doe@server1 tmp]$ pip install sendgrid

Then re-list the contents of the site-packages directory in the Python virtual environment and there should be new directories.

(python) [john.doe@server1 tmp]$ ls -l python/lib/python*/site-packages/
-rw-rw-r--. 1 john.doe john.doe  126 Dec 11 05:03 easy_install.py
drwxrwxr-x. 2 john.doe john.doe 4096 Dec 11 05:07 ellipticcurve
drwxrwxr-x. 5 john.doe john.doe 4096 Dec 11 05:04 pip
drwxrwxr-x. 2 john.doe john.doe 4096 Dec 11 05:04 pip-21.3.1.dist-info
drwxrwxr-x. 5 john.doe john.doe   89 Dec 11 05:03 pkg_resources
drwxrwxr-x. 2 john.doe john.doe   40 Dec 11 05:03 __pycache__
drwxrwxr-x. 2 john.doe john.doe 4096 Dec 11 05:07 python_http_client
drwxrwxr-x. 2 john.doe john.doe 4096 Dec 11 05:07 python_http_client-3.3.7.dist-info
drwxrwxr-x. 2 john.doe john.doe 4096 Dec 11 05:07 sendgrid
drwxrwxr-x. 2 john.doe john.doe 4096 Dec 11 05:07 sendgrid-6.11.0.dist-info
drwxrwxr-x. 6 john.doe john.doe 4096 Dec 11 05:03 setuptools
drwxrwxr-x. 2 john.doe john.doe 4096 Dec 11 05:03 setuptools-39.2.0.dist-info
drwxrwxr-x. 2 john.doe john.doe 4096 Dec 11 05:07 starkbank_ecdsa-2.2.0-py3.6.egg-info
drwxrwxr-x. 2 john.doe john.doe 4096 Dec 11 05:07 test

The Python virtual environment can be deactivated.

(python) [john.doe@server1 tmp]$ deactivate

Next you'll create a zip archive. The base directory in the .zip file must be "python/" or "python/lib/python3.x/site-packages/" so that when the .zip file is extracted the files in the .zip file are extracted to /opt/python or /opt/python/lib/python3.x/site-packages/. If you want to go with python/lib/python3.x/site-packages/ as the base directory, issue a command like this.

zip --recurse-path /tmp/sendgrid.zip python/lib/python*/site-packages

Or if you want to go with having python/ as the base directory in the .zip archive, let's create the /tmp/python directory.

mkdir /tmp/python

And then move into the site-packages directory.

cd python/lib/python*/site-packages

And then copy all of the files and directories in the site-packages directory to /tmp/python.

cp -R . /tmp/python

And change into the /tmp directory.

cd /tmp

And create the .zip archive.

zip --recurse-path /tmp/sendgrid.zip python/

And now you can remove the /tmp/python directory.

rm -rf /tmp/python

Download module tar and create a zip archive

On a Linux system, use wget to download the paramiko tar.

wget https://files.pythonhosted.org/packages/44/03/158ae1dcb950bd96f04038502238159e116fafb27addf5df1ba35068f2d6/paramiko-3.3.1.tar.gz

Use tar to extract the tar archive.

tar -zxpf paramiko-3.3.1.tar.gz --directory /tmp

You can now remove the tar file.

rm paramiko-3.3.1.tar.gz

Rename the extracted directory from /tmp/paramiko-3.3.1/ to /tmp/python/. This is needed so that when the zip file is created, the base directory in the zip archive will be python and all of the files and directories that were extracted will be below the python directory. Refer to Packaging your layer content - AWS Lambda (amazon.com) for more details on why this is required.

mv /tmp/paramiko-3.3.1 /tmp/python

zip the extracted files.

cd /tmp/
zip --recurse-path /tmp/paramiko-3.3.1.zip python/

You can now remove the /tmp/python/ directory.

rm -rf /tmp/python

Create the Lambda Layer

Now you can use the aws lambda publish-layer-version command to create a Lambda Layer using the .zip file.

aws lambda publish-layer-version --layer-name paramiko --zip-file fileb://paramiko.zip --compatible-runtimes python3.9

Or, in the AWS console, create a Lambda Layer using the zip file, something like this. Make note of the Amazon Resource Number (ARN) of the layer.

You could add the following to your lambda_handler function.

def lambda_handler(event, context):

    import os
        
    for root, dirs, files in os.walk("/opt"):
        print(f"root = {root}")
        print(f"dirs = {dirs}")
        print(f"files = {files}")    

If your .zip file has python/paramiko.py then paramiko.py should be in the /opt/python directory.

root = /opt
directories = [python]
files = []

root = /opt/python
directories = [lib]
files = [paramiko.py]