Let's say your IBM IHS Web Server error log contains something like this.

[Tue Aug 20 04:25:48.090971 2024] [mpm_worker:alert] [pid 515716:tid 139639077524736] (11)Resource temporarily unavailable: AH00282: apr_thread_create: unable to create worker thread

As a super quick fix, I restarted the Operating System and then each instance was able to start up. But this was just a band aid because then if I waited a while and tried to restart instances, the problem would return. And of course, restarting your Operating System is not a practical fix!

Notice this log event contains mpm_worker, the Multi Processing Module (MPM). This means your IBM IHS Web Server instances are loading the mpm worker module.

LoadModule mpm_worker_module modules/mod_mpm_worker.so

And the instances configuration file should have something like this listing the mpm worker configurations.

<IfModule worker.c>
StartServers         2
MaxClients         300
MinSpareThreads     25
MaxSpareThreads     75
ThreadsPerChild     25
MaxRequestsPerChild  0
</IfModule>

Sometimes this occurs when:

• MaxClients has been reached (300 in this example)
• ThreadsPerChild has been reached (25 in this example)

The mpmstats module can be used to append mpm stats to the error log. On IHS version 7 or higher, the mod_mpmstats.so file should be located in the modules/debug directory such as /opt/IBMIHS/modules/debug/mod_mpmstats.so. Append the following to your httpd.conf file to use the mpmstats module.

LoadModule mpmstats_module modules/debug/mod_mpmstats.so

<IfModule mod_mpmstats.c>
ReportInterval 2
TrackModules On
</IfModule>

And restart the instance for this change to take effect.

/opt/IBMIHS/bin/httpd -d /opt/IBMIHS -f /opt/IBMIHS/conf/httpd90.conf -k restart

And validate the instance has successfully loaded the mpmstats module.

~]$ /opt/IBMIHS/bin/httpd -d /opt/IBMIHS -f /opt/IBMIHS/conf/httpd.conf -M
Loaded Modules:
 mpmstats_module (shared)

If no requests are being made to the instance, then mpmstats will not log any events to the error log. So, hit one of the HTML web pages being produced by your instance and you should then start to see mpmstats events in the error log, something like this.

• rdy = threads that are ready
• bsy = threads that are busy
• rd = threads that are reading
• wr = threads that are writing
• ka = threads that were told to keep alive
• log = threads that are logging
• dns = threads looking up a DNS hostname
• cls = threads that are in the process of closing

[Tue Aug 27 04:50:18.901781 2024] [mpmstats:notice] [pid 1076616:tid 140416169809152] mpmstats: rdy 99 bsy 1 rd 0 wr 0 ka 1 log 0 dns 0 cls 0
[Tue Aug 27 04:50:20.903365 2024] [mpmstats:notice] [pid 1076616:tid 140416169809152] mpmstats: rdy 99 bsy 1 rd 0 wr 0 ka 1 log 0 dns 0 cls 0
[Tue Aug 27 04:50:22.905513 2024] [mpmstats:notice] [pid 1076616:tid 140416169809152] mpmstats: rdy 99 bsy 1 rd 0 wr 0 ka 1 log 0 dns 0 cls 0
[Tue Aug 27 04:50:24.907676 2024] [mpmstats:notice] [pid 1076616:tid 140416169809152] mpmstats: rdy 99 bsy 1 rd 0 wr 0 ka 1 log 0 dns 0 cls 0

One thing to look for is a consistent reduction in rdy (ready), something like this. This suggests prior threads are not being released and new threads are being used which eventually results in MaxClients or ThreadsPerChild being reached, then "unable to create worker thread" occurs. In this scenario, I would start by looking at recent changes made to IHS, to see if this issue was due to some change. Keep in mind that the change may be outside of IHS, such as some patch to the Operating System.

[Tue Aug 27 04:56:51.733791 2024] [mpmstats:notice] [pid 2392646:tid 140662879765760] mpmstats: rdy 74 bsy 1 rd 0 wr 1 ka 0 log 0 dns 0 cls 0
[Tue Aug 27 04:56:52.733791 2024] [mpmstats:notice] [pid 2392646:tid 140662879765760] mpmstats: rdy 73 bsy 2 rd 0 wr 1 ka 0 log 0 dns 0 cls 0
[Tue Aug 27 04:56:53.733791 2024] [mpmstats:notice] [pid 2392646:tid 140662879765760] mpmstats: rdy 72 bsy 3 rd 0 wr 1 ka 0 log 0 dns 0 cls 0
[Tue Aug 27 04:56:53.733791 2024] [mpmstats:notice] [pid 2392646:tid 140662879765760] mpmstats: rdy 71 bsy 4 rd 0 wr 1 ka 0 log 0 dns 0 cls 0
[Tue Aug 27 04:56:53.733791 2024] [mpmstats:notice] [pid 2392646:tid 140662879765760] mpmstats: rdy 70 bsy 5 rd 0 wr 1 ka 0 log 0 dns 0 cls 0
[Tue Aug 27 04:56:53.733791 2024] [mpmstats:notice] [pid 2392646:tid 140662879765760] mpmstats: rdy 69 bsy 6 rd 0 wr 1 ka 0 log 0 dns 0 cls 0
[Tue Aug 27 04:56:53.733791 2024] [mpmstats:notice] [pid 2392646:tid 140662879765760] mpmstats: rdy 68 bsy 7 rd 0 wr 1 ka 0 log 0 dns 0 cls 0

I also saw this, where after "unable to create worker thread" was event "failed on thread" and the nproc (number of processes) and stack size values were listed.

[Tue Aug 27 04:56:51.742306 2024] [mpm_worker:alert] [pid 2393464:tid 140662723663616] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread (see http://publib.boulder.ibm.com/httpserv/ihsdiag/apr_thread_create.html)
[Tue Aug 27 04:56:51.742351 2024] [mpm_worker:alert] [pid 2393464:tid 140662723663616] (11)Resource temporarily unavailable: ... failed on thread 24 of 25, RLIMIT_NPROC is 2047, stack size 524288

The ps command can be used to determine the user that is starting the IBM IHS Web Server processes. In this example root started the IBM IHS Web Server process (the parent process is PID 2498 in this example) and the child processes have the "nobody" user.

[root@dlhoweb011 tmp]# ps -ef | grep httpd
root        2498       1  0 02:41 ?        00:00:00 /opt/IBMIHS/bin/httpd -d /opt/IBMIHS -f /opt/IBMIHS/conf/httpd.conf -k start
nobody      2499    2498  0 02:41 ?        00:00:00 /opt/IBMIHS/bin/httpd -d /opt/IBMIHS -f /opt/IBMIHS/conf/httpd.conf -k start
nobody      2501    2498  0 02:41 ?        00:00:01 /opt/IBMIHS/bin/httpd -d /opt/IBMIHS -f /opt/IBMIHS/conf/httpd.conf -k start
nobody      2502    2498  0 02:41 ?        00:00:01 /opt/IBMIHS/bin/httpd -d /opt/IBMIHS -f /opt/IBMIHS/conf/httpd.conf -k start
nobody      2503    2498  0 02:41 ?        00:00:00 /opt/IBMIHS/bin/httpd -d /opt/IBMIHS -f /opt/IBMIHS/conf/httpd.conf -k start

The reason the child processes were launched by the "nobody" user is because the User directive in the httpd.conf file has "nobody" in this example.

[root@server ~]$ grep ^User /opt/IBMIHS/conf/httpd.conf
User nobody

The ulimit command can be used to list the limits for the user (root in this example) that is starting the IBM IHS Web Server processes. Notice in this example that the max user processes limit is 2047 meaning root is limited to running a max of 2047 processes. If you want to adjust the limit, check out my article FreeKB - Linux Commands - ulimit (list and update a users limits).

[root@server1 tmp]# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 22577
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 8192
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 2047
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

The ps and wc (word count) commands can be used to list the current number of processes the user is running. In this example, root is running 395 processes, much less than the max user processes limit of 2047. Just keep in mind that this is just a view of the count at the current moment. There may be spikes that cause the system to reach the user processes limit.

[root@server1 tmp]# ps -ef | wc -l
395

The underlying Operating System may not have enough memory to create worker threads for your IBM IHS Web Server. If you are running IBM IHS Web Server on a Linux system, the free command can be used to see how much free memory you have. When I had this issue, the free command showed me that I had 262 MB of free memory which is not that much free memory but I had 1.2 GB of buff/cache memory and 1.1 GB of available memory, so it seemed like I had plenty of free memory.

~]# free -h
              total        used        free      shared  buff/cache   available
Mem:          3.6Gi       2.2Gi       262Mi        60Mi       1.2Gi       1.1Gi
Swap:         2.0Gi       1.1Gi       877Mi