Digging further, using LogLevel slotmem_shm:debug [Fri Aug 27 11:10:25.724715 2021] [slotmem_shm:debug] [pid 176913:tid 139917097439872] mod_slotmem_shm.c(408): AH02611: create: apr_shm_create(/var/run/apache2/slotmem-shm-pd38fd8d0_wdprd_ssn_example_org_2.shm) succeeded [Fri Aug 27 11:10:25.724768 2021] [slotmem_shm:debug] [pid 176913:tid 139917097439872] mod_slotmem_shm.c(378): AH02602: create didn't find /var/run/apache2/slotmem-shm-pd38fd8d0_www_config_example_org_8085_2.shm in global list [Fri Aug 27 11:10:25.724781 2021] [slotmem_shm:debug] [pid 176913:tid 139917097439872] mod_slotmem_shm.c(387): AH02300: create /var/run/apache2/slotmem-shm-pd38fd8d0_www_config_example_org_8085_2.shm: 1064/2 [Fri Aug 27 11:10:25.724824 2021] [slotmem_shm:debug] [pid 176913:tid 139917097439872] mod_slotmem_shm.c(408): AH02611: create: apr_shm_create(/var/run/apache2/slotmem-shm-pd38fd8d0_www_config_example_org_8085_2.shm) succeeded [Fri Aug 27 11:10:25.724905 2021] [slotmem_shm:debug] [pid 176913:tid 139917097439872] mod_slotmem_shm.c(378): AH02602: create didn't find /var/run/apache2/slotmem-shm-pd38fd8d0_acc_ppm_signify_com_2.shm in global list [Fri Aug 27 11:10:25.724915 2021] [slotmem_shm:debug] [pid 176913:tid 139917097439872] mod_slotmem_shm.c(387): AH02300: create /var/run/apache2/slotmem-shm-pd38fd8d0_acc_ppm_signify_com_2.shm: 1064/2 [Fri Aug 27 11:10:25.724959 2021] [slotmem_shm:error] [pid 176913:tid 139917097439872] (28)No space left on device: AH02611: create: apr_shm_create(/var/run/apache2/slotmem-shm-pd38fd8d0_acc_ppm_signify_com_2.shm) failed [Fri Aug 27 11:10:25.725012 2021] [:emerg] [pid 176913:tid 139917097439872] AH00020: Configuration Failed, exiting On Fri, Aug 27, 2021 at 11:26 AM Spil Oss <spil.oss@xxxxxxxxx> wrote: > > Hi, > > I've been experiencing a failed apache2 service on Ubuntu 21.04 when > performing a reload using the `systemctl reload apache2` command. The > command does not always fail, but seems to be failing more often as > the number of vhosts increases (currently ca 120). My > > The `systemctl reload apache2` command exits without error, but the > service ends up in a failed state. Running `systemctl start apache2` > after this failure starts the service without issues. > I had taken to do running `systemctl reload apache2; systemctl status > apache2` to validate that I have a running service, but this would > report "success" even when the service is "failed". > > Expecting some timing issue, I increased the "RestartSec" systemd > parameter to 500ms using > `/etc/systemd/system/apache2.service.d/override.conf` > [Service] > RestartSec=500ms > > This has not fixed the issue either. > > Testing the reload using `apachectl -k graceful` can also trigger the > "failed" state of the process. > > The consistent error is with the persistence of shared memory > segments. The indicated error is incorrect, there's plenty of space on > the filesystem. The configuration has been kept as close as possible > to the default Ubuntu config. > > My gut feeling is some weird interaction between graceful and systemd > as seen in the logs. The RestartSec change not solving the problem > kind of goes against that. > > Any help appreciated! Thanks, Bernard Spil. > > $ df -h /var/run/apache2/ > Filesystem Size Used Avail Use% Mounted on > tmpfs 1.6G 6.6M 1.6G 1% /run > > $ ls mods-enabled/*.load | sed 's/mods-enabled\///;s/\.load//' > access_compat > alias > auth_mellon > authn_core > authn_file > authz_core > authz_host > authz_user > brotli > deflate > dir > env > filter > headers > http2 > lbmethod_byrequests > mime > mpm_event > negotiation > proxy > proxy_balancer > proxy_http > proxy_http2 > proxy_wstunnel > remoteip > reqtimeout > rewrite > setenvif > slotmem_shm > socache_shmcb > ssl > status > > /var/log/apache2/error.log: > [Fri Aug 27 00:00:18.881934 2021] [mpm_event:notice] [pid 138928:tid > 140168396681856] AH00493: SIGUSR1 received. Doing graceful restart > [Fri Aug 27 00:00:19.155640 2021] [slotmem_shm:error] [pid 138928:tid > 140168396681856] (28)No space left on device: AH02611: create: > apr_shm_create(/var/run/apache2/slotmem-shm-pd38fd8d0_acc_example_org_6.shm) > failed > [Fri Aug 27 00:00:19.155679 2021] [:emerg] [pid 138928:tid > 140168396681856] AH00020: Configuration Failed, exiting > [Fri Aug 27 00:05:01.645184 2021] [core:warn] [pid 166984:tid > 140602886914688] AH00098: pid file /var/run/apache2/apache2.pid > overwritten -- Unclean shutdown of previous Apache run? > [Fri Aug 27 00:05:01.656692 2021] [mpm_event:notice] [pid 166984:tid > 140602886914688] AH00489: Apache/2.4.46 (Ubuntu) OpenSSL/1.1.1j > configured -- resuming normal operations > [Fri Aug 27 00:05:01.656748 2021] [core:notice] [pid 166984:tid > 140602886914688] AH00094: Command line: '/usr/sbin/apache2' > > journalctl: > Aug 27 02:00:13 web01.example.org systemd[1]: Starting Rotate log files... > Aug 27 02:00:18 web01.example.org systemd[1]: Reloading The Apache HTTP Server. > Aug 27 02:00:18 web01.example.org systemd[1]: Reloaded The Apache HTTP Server. > Aug 27 02:00:18 web01.example.org systemd[1]: logrotate.service: Succeeded. > Aug 27 02:00:18 web01.example.org systemd[1]: Finished Rotate log files. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Main > process exited, code=exited, status=1/FAILURE > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164444 (apache2) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 166489 (apache2) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164446 (apache2) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164448 (apache2) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164450 (apache2) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164452 (apache2) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164454 (apache2) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164456 (apache2) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164458 (apache2) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164459 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164460 (apache2) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164461 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164462 (apache2) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164463 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164464 (apache2) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164465 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164466 (apache2) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164467 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164468 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164469 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164470 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164471 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164472 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164473 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164474 (apache2) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164475 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164476 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164477 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164478 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164479 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164480 (apache2) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164481 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164482 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164483 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164484 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164485 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164486 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164487 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164488 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164489 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164490 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164491 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164490 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164491 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164492 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164493 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164494 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164495 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164496 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164497 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164498 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164499 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164500 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164501 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164502 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164503 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164504 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164505 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164506 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164507 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164508 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164509 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164510 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164511 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164512 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164513 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164514 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164515 (apache2) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164516 (apache2) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164517 (apache2) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164518 (apache2) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164519 (apache2) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164520 (apache2) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164521 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164522 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164523 (apache2) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164524 (apache2) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164525 (apache2) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164526 (apache2) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164527 (apache2) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164528 (apache2) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164529 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164530 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164531 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164532 (apache2) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164533 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164534 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164535 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164536 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164537 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164538 (apache2) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164539 (apache2) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164540 (apache2) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164573 (apache2) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing > process 164606 (n/a) with signal SIGKILL. > Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Failed > with result 'exit-code'. > Aug 27 02:05:01 web01.example.org CRON[166975]: > pam_unix(cron:session): session opened for user root by (uid=0) > Aug 27 02:05:01 web01.example.org CRON[166976]: (root) CMD > (/root/bin/apache-workaround) > Aug 27 02:05:01 web01.example.org systemd[1]: Starting The Apache HTTP Server... > Aug 27 02:05:01 web01.example.org systemd[1]: Started The Apache HTTP Server. > > When running a vanilla `apachectl -k graceful`: > Aug 27 10:47:18 web01.sias.intra.lighting.com sudo[176897]: sysadmin: > TTY=pts/0 ; PWD=/etc/apache2 ; USER=root ; COMMAND=/usr/sbin/apachectl > -k graceful > Aug 27 10:47:18 web01.sias.intra.lighting.com sudo[176897]: > pam_unix(sudo:session): session opened for user root by > sysadmin(uid=1000) > Aug 27 10:47:18 web01.sias.intra.lighting.com sudo[176897]: > pam_unix(sudo:session): session closed for user root > Aug 27 10:47:19 web01.sias.intra.lighting.com systemd[1]: > apache2.service: Main process exited, code=exited, status=1/FAILURE > Aug 27 10:47:19 web01.sias.intra.lighting.com systemd[1]: > apache2.service: Killing process 173239 (apache2) with signal SIGKILL. > Aug 27 10:47:19 web01.sias.intra.lighting.com systemd[1]: > apache2.service: Killing process 174924 (apache2) with signal SIGKILL. > ...snip... > Aug 27 10:47:19 web01.sias.intra.lighting.com systemd[1]: > apache2.service: Killing process 176797 (apache2) with signal SIGKILL. > Aug 27 10:47:19 web01.sias.intra.lighting.com systemd[1]: > apache2.service: Killing process 176838 (n/a) with signal SIGKILL. > Aug 27 10:47:19 web01.sias.intra.lighting.com systemd[1]: > apache2.service: Failed with result 'exit-code'. > Aug 27 10:47:20 web01.sias.intra.lighting.com sudo[176903]: sysadmin: > TTY=pts/0 ; PWD=/etc/apache2 ; USER=root ; COMMAND=/usr/bin/systemctl > status apache2 > Aug 27 10:47:20 web01.sias.intra.lighting.com sudo[176903]: > pam_unix(sudo:session): session opened for user root by > sysadmin(uid=1000) > Aug 27 10:47:20 web01.sias.intra.lighting.com sudo[176903]: > pam_unix(sudo:session): session closed for user root > Aug 27 10:47:34 web01.sias.intra.lighting.com sudo[176906]: sysadmin: > TTY=pts/0 ; PWD=/etc/apache2 ; USER=root ; COMMAND=/usr/bin/systemctl > start apache2 > Aug 27 10:47:34 web01.sias.intra.lighting.com sudo[176906]: > pam_unix(sudo:session): session opened for user root by > sysadmin(uid=1000) > Aug 27 10:47:34 web01.sias.intra.lighting.com systemd[1]: Starting The > Apache HTTP Server... > Aug 27 10:47:34 web01.sias.intra.lighting.com systemd[1]: Started The > Apache HTTP Server. --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
|