
Let's say something like this is being returned.
WARN received SIGTERM indicating exit request
I once got this with one of my Flask apps running in a Docker uwsgi supervisord nginx container. Check out my article FreeKB - Flask - Install uWSGI Nginx Flask on Docker for more details on running Flask in a Docker uwsgi supervisord nginx container.
This happened because the supervisord.conf file in my Docker container had eventlistener:quit_on_failure to run /etc/supervisor/stop-supervisor.sh when one of the following events occur:
- PROCESS_STATE_STOPPED
- PROCESS_STATE_EXITED
- PROCESS_STATE_FATAL
http://supervisord.org/subprocess.html#process-states has more details on the process states.
~]$ sudo docker exec my-container cat /etc/supervisor/conf.d/supervisord.conf
[supervisord]
nodaemon=true
user=root
[program:uwsgi]
command=/usr/local/bin/uwsgi --ini /etc/uwsgi/uwsgi.ini --disable-logging --log-master
stdout_logfile=/var/log/supervisor/uwsgi_stdout.log
stdout_logfile_maxbytes=0
stderr_logfile=/var/log/supervisor/uwsgi_stderr.log
stderr_logfile_maxbytes=0
startsecs = 0
autorestart=false
[program:nginx]
command=/usr/sbin/nginx
stdout_logfile=/var/log/supervisor/nginx_stdout.log
stdout_logfile_maxbytes=0
stderr_logfile=/var/log/supervisor/nginx_stderr.log
stderr_logfile_maxbytes=0
# Graceful stop, see http://nginx.org/en/docs/control.html
stopsignal=QUIT
startsecs = 0
autorestart=false
[eventlistener:quit_on_failure]
events=PROCESS_STATE_STOPPED,PROCESS_STATE_EXITED,PROCESS_STATE_FATAL
command=/etc/supervisor/stop-supervisor.sh
So, why did PROCESS_STATE_STOPPED or PROCESS_STATE_EXITED or PROCESS_STATE_FATAL occur? To determine this, supervisord will probably need to be set to log at log level debug, because log level debug includes:
Messages useful for users trying to debug process configuration and communications behavior (process output, listener state changes, event notifications).
For example, the supervisord.conf file could have something like this.
[supervisord]
nodaemon=true
user=root
loglevel=debug
Once the log level has been set to debug, your supervisord activity log which is /var/log/supervisor/supervisord.log by default, should have something like this, much more verbose logging. Often this additionally logging will help you determine why PROCESS_STATE_STOPPED or PROCESS_STATE_EXITED or PROCESS_STATE_FATAL occurred.
http://supervisord.org/logging.html
https://supervisord.org/configuration.html
~]$ sudo docker exec my-container cat /var/log/supervisor/supervisord.log
2024-08-07 20:29:18,923 INFO supervisord started with pid 1
2024-08-07 20:29:19,926 INFO spawned: 'quit_on_failure' with pid 8
2024-08-07 20:29:19,930 INFO spawned: 'nginx' with pid 9
2024-08-07 20:29:19,933 INFO spawned: 'uwsgi' with pid 10
2024-08-07 20:29:19,936 DEBG 'quit_on_failure' stdout output:
b'READY\n'
2024-08-07 20:29:19,937 DEBG quit_on_failure: ACKNOWLEDGED -> READY
2024-08-07 20:29:19,937 INFO success: nginx entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2024-08-07 20:29:19,937 INFO success: uwsgi entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2024-08-07 20:29:19,952 DEBG fd 14 closed, stopped monitoring <POutputDispatcher at 140657251989200 for <Subprocess at 140657252374896 with name nginx in state RUNNING> (stderr)>
2024-08-07 20:29:19,954 DEBG 'uwsgi' stderr output:
[uWSGI] getting INI configuration from /app/uwsgi.ini
2024-08-07 20:29:19,956 DEBG 'uwsgi' stderr output:
[uWSGI] getting INI configuration from /etc/uwsgi/uwsgi.ini
2024-08-07 20:29:19,956 DEBG fd 19 closed, stopped monitoring <POutputDispatcher at 140657251989920 for <Subprocess at 140657252480384 with name uwsgi in state RUNNING> (stderr)>
2024-08-07 20:29:20,957 INFO success: quit_on_failure entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-08-07 20:29:21,985 DEBG 'uwsgi' stdout output:
;uWSGI instance configuration
[uwsgi]
2024-08-07 20:29:21,985 DEBG 'uwsgi' stdout output:
ini = /app/uwsgi.ini
2024-08-07 20:29:21,986 DEBG 'uwsgi' stdout output:
module = main
callable = app
processes = 16
cheaper = 2
ini = /etc/uwsgi/uwsgi.ini
socket = /tmp/uwsgi.sock
chown-socket = nginx:nginx
chmod-socket = 664
hook-master-start = unix_signal:15 gracefully_kill_them_all
need-app = true
die-on-term = true
show-config = true
disable-logging = true
log-master = true
;end of configuration
*** Starting uWSGI 2.0.22 (64bit) on [Wed Aug 7 20:29:19 2024] ***
compiled with version: 10.2.1 20210110 on 27 May 2024 00:21:42
os: Linux-6.1.49-69.116.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Aug 30 18:09:24 UTC 2023
nodename: 3188fa8190ea
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 2
current working directory: /app
detected binary path: /usr/local/bin/uwsgi
your memory page size is 4096 bytes
detected max file descriptor number: 32768
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to UNIX address /tmp/uwsgi.sock fd 6
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
Python version: 3.11.9 (main, May 14 2024, 08:23:55) [GCC 10.2.1 20210110]
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0x7f911c484638
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 1239640 bytes (1210 KB) for 16 cores
*** Operational MODE: preforking ***
WSGI app 0 (mountpoint='') ready in 2 seconds on interpreter 0x7f911c484638 pid: 10 (default app)
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
*** uWSGI is running in multiple interpreter mode ***
2024-08-07 20:29:21,986 DEBG 'uwsgi' stdout output:
spawned uWSGI master process (pid: 10)
spawned uWSGI worker 1 (pid: 13, cores: 1)
spawned uWSGI worker 2 (pid: 14, cores: 1)
running "unix_signal:15 gracefully_kill_them_all" (master-start)...
I used the ps command to get the list of processes running in my Docker container. Notice PID 1 uses python3 to run /usr/bin/supervisord.
]$ sudo docker exec my-container ps -ef
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 00:17 ? 00:00:00 /usr/bin/python3 /usr/bin/supervisord
root 7 1 0 00:17 ? 00:00:00 sh /etc/supervisor/stop-supervisor.sh
root 8 1 0 00:17 ? 00:00:00 nginx: master process /usr/sbin/nginx
root 9 1 0 00:17 ? 00:00:19 /usr/local/bin/uwsgi --ini /etc/uwsgi/uwsgi.ini --disable-logging --log-master
nginx 10 8 0 00:17 ? 00:00:00 nginx: worker process
root 14 9 0 00:17 ? 00:00:00 /usr/local/bin/uwsgi --ini /etc/uwsgi/uwsgi.ini --disable-logging --log-master
root 15 9 0 00:17 ? 00:00:00 /usr/local/bin/uwsgi --ini /etc/uwsgi/uwsgi.ini --disable-logging --log-master
root 309 0 0 01:33 ? 00:00:00 ps -ef
The /usr/bin/supervisord file is simply a Python script.
~]$ sudo docker exec my-container cat /usr/bin/supervisord
#!/usr/bin/python3
# EASY-INSTALL-ENTRY-SCRIPT: 'supervisor==4.2.2','console_scripts','supervisord'
import re
import sys
# for compatibility with easy_install; see #2198
__requires__ = 'supervisor==4.2.2'
try:
from importlib.metadata import distribution
except ImportError:
try:
from importlib_metadata import distribution
except ImportError:
from pkg_resources import load_entry_point
def importlib_load_entry_point(spec, group, name):
dist_name, _, _ = spec.partition('==')
matches = (
entry_point
for entry_point in distribution(dist_name).entry_points
if entry_point.group == group and entry_point.name == name
)
return next(matches).load()
globals().setdefault('load_entry_point', importlib_load_entry_point)
if __name__ == '__main__':
sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
sys.exit(load_entry_point('supervisor==4.2.2', 'console_scripts', 'supervisord')())
This warning comes from the supervisord daemon. In my setup, I could see the warning in the supervisord.log in the Docker container.
~]$ sudo docker exec my-container tail -20 /var/log/supervisor/supervisord.log
2024-08-05 12:10:56,040 INFO spawned: 'uwsgi' with pid 10
2024-08-05 12:10:56,041 INFO success: nginx entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2024-08-05 12:10:56,042 INFO success: uwsgi entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2024-08-05 12:10:57,060 INFO success: quit_on_failure entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-08-07 00:16:52,604 WARN received SIGTERM indicating exit request
2024-08-07 00:16:52,605 INFO waiting for quit_on_failure, nginx, uwsgi to die
2024-08-07 00:16:55,609 INFO waiting for quit_on_failure, nginx, uwsgi to die
2024-08-07 00:16:58,613 INFO waiting for quit_on_failure, nginx, uwsgi to die
2024-08-07 00:17:01,616 INFO waiting for quit_on_failure, nginx, uwsgi to die
You can check /var/log/apt/history.log to see if there were any updates to any of the apt packages, such as the supervisord package. When I ran into this, there were no recent events in the apt history.log.
~]$ sudo docker exec my-container tail /var/log/apt/history.log
Start-Date: 2024-06-09 11:26:05
Commandline: apt install nfs-common -y
Install: libwrap0:amd64 (7.6.q-31, automatic), nfs-common:amd64 (1:1.3.4-6), libcap2:amd64 (1:2.44-1, automatic), keyutils:amd64 (1.6.1-2, automatic), dmsetup:amd64 (2:1.02.175-2.1, automatic), rpcbind:amd64 (1.2.5-9, automatic), libnfsidmap2:amd64 (0.25-6, automatic), libdevmapper1.02.1:amd64 (2:1.02.175-2.1, automatic)
End-Date: 2024-06-09 11:26:06
And the supervisord --version command returned the version of supervisord that has been running when I created the container.
]$ sudo docker exec my-container supervisord --version
4.2.2
Did you find this article helpful?
If so, consider buying me a coffee over at