Let's say you get something like this when invoking one of your Python Lambda Function. In this example, the Lambda Function cannot import the paramiko module.

{"errorMessage": "Unable to import module 'app': No module named 'paramiko'", "errorType": "Runtime.ImportModuleError", "requestId": "574780f6-db2d-4a5d-a9c2-3bc56613dea8", "stackTrace": []}

There are a few ways to go about resolving this.

• Use pip to install the module and create a zip archive
• Download module tar and create a zip archive

And then

• Use the aws lambda publish-layer-version command to create a Lambda Layer using the .zip file.

According to https://docs.aws.amazon.com/lambda/latest/dg/packaging-layers.html:

Lambda loads the layer content into the /opt directory of that execution environment.

For each Lambda runtime, the PATH variable already includes specific folder paths within the /opt directory.

To ensure that your layer content gets picked up by the PATH variable, include the content in the following folder paths:

In other words, when you create the .zip file, the base directory in the .zip file must be "python/" or "python/lib/python3.x/site-packages/" so that when the .zip file is extracted the files in the .zip file are extracted to /opt/python or /opt/python/lib/python3.x/site-packages/.

You could add the following to your lambda_handler function.

def lambda_handler(event, context):

    import os
        
    for root, dirs, files in os.walk("/opt"):
        print(f"root = {root}")
        print(f"dirs = {dirs}")
        print(f"files = {files}")    

Which should then return something like this.

root = /opt
directories = [python]
files = []

root = /opt/python
directories = [lib]
files = []

root = /opt/python/lib
directories = [python3.12]
files = []

root = /opt/python/lib/python3.12
directories = [site-packages]
files = []

root = /opt/python/lib/python3.12/site-packages
directories = ['pip', 'pip-23.3.2.dist-info']
files = []

Use pip to install the module and create a zip archive

This is usually the method I go with. On a Linux system, use the same version of Python as is being used by your Lambda Function to create a Python virtual environment. If your Linux system does not have the same version of Python as is being used by your Lambda Function, check out my article Install Python on Linux.

Since the base directory must be "python" or "python/lib/python3.x/site-packages" let's create a create a Python virtual environment named "python" so that the base directory name is "python".

python3 -m venv python

Activate the virtual environment.

source python/bin/activate

I almost always first issue the pip list command, and there is almost always a prompt to upgrade pip. 

(python) [john.doe@server1 tmp]$ pip list
pip (9.0.3)
setuptools (39.2.0)
You are using pip version 9.0.3, however version 23.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Let's upgrade pip.

(python) [john.doe@server1 tmp]$ pip install --upgrade pip

The site-packages directory in the Python virtual environment should contain something like this.

(python) [john.doe@server1 tmp]$ ls -l python/lib/python*/site-packages/
-rw-rw-r--. 1 john.doe john.doe  126 Dec 11 05:03 easy_install.py
drwxrwxr-x. 5 john.doe john.doe 4096 Dec 11 05:04 pip
drwxrwxr-x. 2 john.doe john.doe 4096 Dec 11 05:04 pip-21.3.1.dist-info
drwxrwxr-x. 5 john.doe john.doe   89 Dec 11 05:03 pkg_resources
drwxrwxr-x. 2 john.doe john.doe   40 Dec 11 05:03 __pycache__
drwxrwxr-x. 6 john.doe john.doe 4096 Dec 11 05:03 setuptools
drwxrwxr-x. 2 john.doe john.doe 4096 Dec 11 05:03 setuptools-39.2.0.dist-info

Now install the module that your AWS Lambda Function is saying cannot be found.

(python) [john.doe@server1 tmp]$ pip install sendgrid

Then re-list the contents of the site-packages directory in the Python virtual environment and there should be new directories.

(python) [john.doe@server1 tmp]$ ls -l python/lib/python*/site-packages/
-rw-rw-r--. 1 john.doe john.doe  126 Dec 11 05:03 easy_install.py
drwxrwxr-x. 2 john.doe john.doe 4096 Dec 11 05:07 ellipticcurve
drwxrwxr-x. 5 john.doe john.doe 4096 Dec 11 05:04 pip
drwxrwxr-x. 2 john.doe john.doe 4096 Dec 11 05:04 pip-21.3.1.dist-info
drwxrwxr-x. 5 john.doe john.doe   89 Dec 11 05:03 pkg_resources
drwxrwxr-x. 2 john.doe john.doe   40 Dec 11 05:03 __pycache__
drwxrwxr-x. 2 john.doe john.doe 4096 Dec 11 05:07 python_http_client
drwxrwxr-x. 2 john.doe john.doe 4096 Dec 11 05:07 python_http_client-3.3.7.dist-info
drwxrwxr-x. 2 john.doe john.doe 4096 Dec 11 05:07 sendgrid
drwxrwxr-x. 2 john.doe john.doe 4096 Dec 11 05:07 sendgrid-6.11.0.dist-info
drwxrwxr-x. 6 john.doe john.doe 4096 Dec 11 05:03 setuptools
drwxrwxr-x. 2 john.doe john.doe 4096 Dec 11 05:03 setuptools-39.2.0.dist-info
drwxrwxr-x. 2 john.doe john.doe 4096 Dec 11 05:07 starkbank_ecdsa-2.2.0-py3.6.egg-info
drwxrwxr-x. 2 john.doe john.doe 4096 Dec 11 05:07 test

The Python virtual environment can be deactivated.

(python) [john.doe@server1 tmp]$ deactivate

Next you'll create a zip archive. The base directory in the .zip file must be "python/" or "python/lib/python3.x/site-packages/" so that when the .zip file is extracted the files in the .zip file are extracted to /opt/python or /opt/python/lib/python3.x/site-packages/. If you want to go with python/lib/python3.x/site-packages/ as the base directory, issue a command like this.

zip --recurse-path /tmp/sendgrid.zip python/lib/python*/site-packages

Or if you want to go with having python/ as the base directory in the .zip archive, let's create the /tmp/python directory.

mkdir /tmp/python

And then move into the site-packages directory.

cd python/lib/python*/site-packages

And then copy all of the files and directories in the site-packages directory to /tmp/python.

cp -R . /tmp/python

And change into the /tmp directory.

cd /tmp

And create the .zip archive.

zip --recurse-path /tmp/sendgrid.zip python/

And now you can remove the /tmp/python directory.

rm -rf /tmp/python

Download module tar and create a zip archive

On a Linux system, use wget to download the paramiko tar.

wget https://files.pythonhosted.org/packages/44/03/158ae1dcb950bd96f04038502238159e116fafb27addf5df1ba35068f2d6/paramiko-3.3.1.tar.gz

Use tar to extract the tar archive.

tar -zxpf paramiko-3.3.1.tar.gz --directory /tmp

You can now remove the tar file.

rm paramiko-3.3.1.tar.gz

Rename the extracted directory from /tmp/paramiko-3.3.1/ to /tmp/python/. This is needed so that when the zip file is created, the base directory in the zip archive will be python and all of the files and directories that were extracted will be below the python directory. Refer to Packaging your layer content - AWS Lambda (amazon.com) for more details on why this is required.

mv /tmp/paramiko-3.3.1 /tmp/python

zip the extracted files.

cd /tmp/
zip --recurse-path /tmp/paramiko-3.3.1.zip python/

You can now remove the /tmp/python/ directory.

rm -rf /tmp/python

Create the Lambda Layer

Now you can use the aws lambda publish-layer-version command to create a Lambda Layer using the .zip file.

aws lambda publish-layer-version --layer-name paramiko --zip-file fileb://paramiko.zip --compatible-runtimes python3.9

Or, in the AWS console, create a Lambda Layer using the zip file, something like this. Make note of the Amazon Resource Number (ARN) of the layer.

You could add the following to your lambda_handler function.

def lambda_handler(event, context):

    import os
        
    for root, dirs, files in os.walk("/opt"):
        print(f"root = {root}")
        print(f"dirs = {dirs}")
        print(f"files = {files}")    

If your .zip file has python/paramiko.py then paramiko.py should be in the /opt/python directory.

root = /opt
directories = [python]
files = []

root = /opt/python
directories = [lib]
files = [paramiko.py]