Amazon Web Services (AWS) - Parse Cloudwatch Logs using Python boto3

by
Jeremy Canfield |
Updated: November 25 2024
| Amazon Web Services (AWS) articles
This assumes you are familar with the basic configurations needed to connect to Amazon Web Services (AWS) using Python boto3. If not, check out my article Python (Scripting) - Getting Started with Amazon Web Services (AWS) boto3.
The query here is basically the same query you would perform in the Amazon Web Services Cloudwatch Log Insights console. This will return results where @message contains "foo".
#!/usr/bin/python3
import boto3
from datetime import datetime, timedelta
import time
client = boto3.client('logs')
query = "fields @timestamp, @message | sort @timestamp desc | limit 20"
log_group = "my-log-group"
start_query_response = client.start_query(
logGroupName=log_group,
startTime=int((datetime.today() - timedelta(hours=5)).timestamp()),
endTime=int(datetime.now().timestamp()),
queryString=query,
)
query_id = start_query_response['queryId']
response = None
while response == None or response['status'] == 'Running':
print('Waiting for query to complete ...')
time.sleep(1)
response = client.get_query_results(
queryId=query_id
)
for result in response['results']:
for dict in result:
message = ""
if dict['field'] == "@timestamp":
message += f"{dict['value']} "
if dict['field'] == "@message":
message += f"{dict['value']}"
print(message)
Which should return something like this.
Waiting for query to complete ...
Waiting for query to complete ...
Waiting for query to complete ...
Waiting for query to complete ...
2024-02-01 02:12:16.154 172.31.19.227 - - [01/Feb/2024:02:12:16 +0000] "GET /base_images/favicon.ico HTTP/1.1" 304 0 "https://www.example.come/foo" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36" "115.163.124.49"
2024-02-01 02:12:16.154 [pid: 13|app: 0|req: 10470/16308] 172.31.19.227 () {60 vars in 1198 bytes} [Thu Feb 1 02:12:16 2024] GET /base_images/favicon.ico => generated 0 bytes in 1 msecs (HTTP/1.1 304) 4 headers in 185 bytes (0 switches on core 0)
2024-02-01 02:12:16.154 172.31.19.227 - - [01/Feb/2024:02:12:15 +0000] "GET /base_images/favicon.ico HTTP/1.1" 200 1150 "https://www.example.come/bar" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36" "115.163.124.49"
2024-02-01 02:12:16.154 [pid: 13|app: 0|req: 10469/16307] 172.31.19.227 () {56 vars in 1086 bytes} [Thu Feb 1 02:12:15 2024] GET /base_images/favicon.ico => generated 1150 bytes in 1 msecs via sendfile() (HTTP/1.1 200) 7 headers in 283 bytes (0 switches on core 0)
If you want to parse the logs for a particular string.
query = = "fields @timestamp, @message, @logStream | filter @message ~= /foo/ | sort @timestamp desc | limit 20"
If you want to parse the logs for a particular log stream.
query = "fields @timestamp, @message, @logStream | filter @logStream == 'my-log-stream' | sort @timestamp desc | limit 20"
Did you find this article helpful?
If so, consider buying me a coffee over at