There are several built in modules that are part of the Python Standard Library included with your Python installation, such as os (Operating System) and re (Regular Expression) and sys (System).

This assumes you are already familar with Python Logger. If not, check out my article Python (Scripting) - Logging to a log file.

Here is an example of how to use RotatingFileHandler to rotate the Python log files created by the logging Logger.

#!/usr/bin/python
import logging
from logging.handlers import RotatingFileHandler

log_file = "/path/to/log.file"

dirname = os.path.dirname(log_file)

if not os.path.exists(dirname):
    os.makedirs(dirname)

logger = logging.getLogger()
logger.setLevel(logging.INFO)
format = logging.Formatter(fmt="[%(asctime)s %(levelname)s] %(message)s", datefmt="%Y-%m-%d %H:%M:%S")

fileHandler = logging.FileHandler(log_file)
fileHandler = RotatingFileHandler(log_file, maxBytes=100000, backupCount=10)
fileHandler.setFormatter(format)
logger.addHandler(fileHandler)

Often, this is put in a function.

#!/usr/bin/python
import logging
from logging.handlers import RotatingFileHandler

def return_logger(log_file, console):

  logger = logging.getLogger()

  # Set Default Log Level
  #   DEBUG    - print/log critical, error, warning, info, debug events
  #   INFO     - print/log critical, error, warning, info events
  #   WARNING  - print/log critical, error, warning events
  #   ERROR    - print/log critical, error events
  #   CRITICAL - print/log critical events
  logger.setLevel(logging.INFO)

  format = logging.Formatter(fmt="[%(asctime)s %(levelname)s] %(message)s", datefmt="%Y-%m-%d %H:%M:%S")

  dirname = os.path.dirname(log_file)

  if not os.path.exists(dirname):
    os.makedirs(dirname)

  fileHandler = logging.FileHandler(log_file)
  fileHandler = RotatingFileHandler(log_file, maxBytes=100000, backupCount=10)
  fileHandler.setFormatter(format)
  logger.addHandler(fileHandler)

  if console == True:
    consoleHandler = logging.StreamHandler(sys.stdout)
    consoleHandler.setFormatter(format)
    logger.addHandler(consoleHandler)

  return logger

And here is what you could have in your child Python scripts, where the logging module is imported and used.

#!/usr/bin/python
import os
import re
import sys

sys.path.append("/path/to/the/directory/that/has/your/logger/module")
from logger import return_logger

log_file = os.path.basename(__file__)
log_file = re.sub(".py", ".log", log_file)

# True is used so that the events are displayed on the console
return_logger(log_file, True)

logging.debug('debug message')
logging.info('info message')
logging.warning('warning message')
logging.error('error message')
logging.critical('critical message')

AVOID TROUBLE

Let's say you have a script located in the /usr/local/scripts directory that is using RotatingFileHandler and the script is owned by john.doe:users.

~]$ ll /usr/local/scripts/
-rw-r--r--. 1 john.doe users 12335 Oct  7 11:46 backup_databases.py

And the /usr/local/scripts directory has drwxrwxr-x (755) permissions for ec2-user:ec2-user. This means that the ec2-user user and ec2-user group has write permission to the /usr/local/scripts directory and that other users do not have write permission to the /usr/local/scripts directory.

~]$ ls -ld /usr/local/scripts/
drwxrwxr-x. 3 ec2-user ec2-user 16384 Oct  7 11:49 /usr/local/scripts/

Assuming john.doe is not a member of the ec2-users group . . .

~]$ groups john.doe
john.doe users

When /usr/local/scripts/backup_databases.py is invoked by john.doe and an attempt is made to rotate the /usr/local/scripts/backup_databases.log file, something this will probably be returned.

--- Logging error ---
Traceback (most recent call last):
  File "/usr/lib64/python3.9/logging/handlers.py", line 74, in emit
    self.doRollover()
  File "/usr/lib64/python3.9/logging/handlers.py", line 177, in doRollover
    self.rotate(self.baseFilename, dfn)
  File "/usr/lib64/python3.9/logging/handlers.py", line 115, in rotate
    os.rename(source, dest)
PermissionError: [Errno 13] Permission denied: '/usr/local/scripts/backup_databases.log' -> '/usr/local/scripts/backup_databases.log.1'

This can be resolved in a number of ways. One option would be to use the chmod command to update the permissions of the /usr/local/scripts directory so that other users can write to the /usr/local/scripts directory.

chmod 0777 /usr/local/scripts

Or the chgrp command could be used to update the groups of the /usr/local/scripts directory to one of the groups john.doe is a member of, such as the users group.

chgrp users /usr/local/scripts