Hi, I've been experiencing a failed apache2 service on Ubuntu 21.04 when performing a reload using the `systemctl reload apache2` command. The command does not always fail, but seems to be failing more often as the number of vhosts increases (currently ca 120). My The `systemctl reload apache2` command exits without error, but the service ends up in a failed state. Running `systemctl start apache2` after this failure starts the service without issues. I had taken to do running `systemctl reload apache2; systemctl status apache2` to validate that I have a running service, but this would report "success" even when the service is "failed". Expecting some timing issue, I increased the "RestartSec" systemd parameter to 500ms using `/etc/systemd/system/apache2.service.d/override.conf` [Service] RestartSec=500ms This has not fixed the issue either. Testing the reload using `apachectl -k graceful` can also trigger the "failed" state of the process. The consistent error is with the persistence of shared memory segments. The indicated error is incorrect, there's plenty of space on the filesystem. The configuration has been kept as close as possible to the default Ubuntu config. My gut feeling is some weird interaction between graceful and systemd as seen in the logs. The RestartSec change not solving the problem kind of goes against that. Any help appreciated! Thanks, Bernard Spil. $ df -h /var/run/apache2/ Filesystem Size Used Avail Use% Mounted on tmpfs 1.6G 6.6M 1.6G 1% /run $ ls mods-enabled/*.load | sed 's/mods-enabled\///;s/\.load//' access_compat alias auth_mellon authn_core authn_file authz_core authz_host authz_user brotli deflate dir env filter headers http2 lbmethod_byrequests mime mpm_event negotiation proxy proxy_balancer proxy_http proxy_http2 proxy_wstunnel remoteip reqtimeout rewrite setenvif slotmem_shm socache_shmcb ssl status /var/log/apache2/error.log: [Fri Aug 27 00:00:18.881934 2021] [mpm_event:notice] [pid 138928:tid 140168396681856] AH00493: SIGUSR1 received. Doing graceful restart [Fri Aug 27 00:00:19.155640 2021] [slotmem_shm:error] [pid 138928:tid 140168396681856] (28)No space left on device: AH02611: create: apr_shm_create(/var/run/apache2/slotmem-shm-pd38fd8d0_acc_example_org_6.shm) failed [Fri Aug 27 00:00:19.155679 2021] [:emerg] [pid 138928:tid 140168396681856] AH00020: Configuration Failed, exiting [Fri Aug 27 00:05:01.645184 2021] [core:warn] [pid 166984:tid 140602886914688] AH00098: pid file /var/run/apache2/apache2.pid overwritten -- Unclean shutdown of previous Apache run? [Fri Aug 27 00:05:01.656692 2021] [mpm_event:notice] [pid 166984:tid 140602886914688] AH00489: Apache/2.4.46 (Ubuntu) OpenSSL/1.1.1j configured -- resuming normal operations [Fri Aug 27 00:05:01.656748 2021] [core:notice] [pid 166984:tid 140602886914688] AH00094: Command line: '/usr/sbin/apache2' journalctl: Aug 27 02:00:13 web01.example.org systemd[1]: Starting Rotate log files... Aug 27 02:00:18 web01.example.org systemd[1]: Reloading The Apache HTTP Server. Aug 27 02:00:18 web01.example.org systemd[1]: Reloaded The Apache HTTP Server. Aug 27 02:00:18 web01.example.org systemd[1]: logrotate.service: Succeeded. Aug 27 02:00:18 web01.example.org systemd[1]: Finished Rotate log files. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Main process exited, code=exited, status=1/FAILURE Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164444 (apache2) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 166489 (apache2) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164446 (apache2) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164448 (apache2) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164450 (apache2) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164452 (apache2) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164454 (apache2) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164456 (apache2) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164458 (apache2) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164459 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164460 (apache2) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164461 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164462 (apache2) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164463 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164464 (apache2) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164465 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164466 (apache2) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164467 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164468 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164469 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164470 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164471 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164472 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164473 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164474 (apache2) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164475 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164476 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164477 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164478 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164479 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164480 (apache2) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164481 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164482 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164483 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164484 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164485 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164486 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164487 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164488 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164489 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164490 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164491 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164490 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164491 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164492 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164493 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164494 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164495 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164496 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164497 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164498 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164499 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164500 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164501 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164502 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164503 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164504 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164505 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164506 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164507 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164508 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164509 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164510 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164511 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164512 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164513 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164514 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164515 (apache2) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164516 (apache2) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164517 (apache2) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164518 (apache2) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164519 (apache2) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164520 (apache2) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164521 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164522 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164523 (apache2) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164524 (apache2) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164525 (apache2) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164526 (apache2) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164527 (apache2) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164528 (apache2) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164529 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164530 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164531 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164532 (apache2) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164533 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164534 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164535 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164536 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164537 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164538 (apache2) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164539 (apache2) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164540 (apache2) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164573 (apache2) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Killing process 164606 (n/a) with signal SIGKILL. Aug 27 02:00:19 web01.example.org systemd[1]: apache2.service: Failed with result 'exit-code'. Aug 27 02:05:01 web01.example.org CRON[166975]: pam_unix(cron:session): session opened for user root by (uid=0) Aug 27 02:05:01 web01.example.org CRON[166976]: (root) CMD (/root/bin/apache-workaround) Aug 27 02:05:01 web01.example.org systemd[1]: Starting The Apache HTTP Server... Aug 27 02:05:01 web01.example.org systemd[1]: Started The Apache HTTP Server. When running a vanilla `apachectl -k graceful`: Aug 27 10:47:18 web01.sias.intra.lighting.com sudo[176897]: sysadmin: TTY=pts/0 ; PWD=/etc/apache2 ; USER=root ; COMMAND=/usr/sbin/apachectl -k graceful Aug 27 10:47:18 web01.sias.intra.lighting.com sudo[176897]: pam_unix(sudo:session): session opened for user root by sysadmin(uid=1000) Aug 27 10:47:18 web01.sias.intra.lighting.com sudo[176897]: pam_unix(sudo:session): session closed for user root Aug 27 10:47:19 web01.sias.intra.lighting.com systemd[1]: apache2.service: Main process exited, code=exited, status=1/FAILURE Aug 27 10:47:19 web01.sias.intra.lighting.com systemd[1]: apache2.service: Killing process 173239 (apache2) with signal SIGKILL. Aug 27 10:47:19 web01.sias.intra.lighting.com systemd[1]: apache2.service: Killing process 174924 (apache2) with signal SIGKILL. ...snip... Aug 27 10:47:19 web01.sias.intra.lighting.com systemd[1]: apache2.service: Killing process 176797 (apache2) with signal SIGKILL. Aug 27 10:47:19 web01.sias.intra.lighting.com systemd[1]: apache2.service: Killing process 176838 (n/a) with signal SIGKILL. Aug 27 10:47:19 web01.sias.intra.lighting.com systemd[1]: apache2.service: Failed with result 'exit-code'. Aug 27 10:47:20 web01.sias.intra.lighting.com sudo[176903]: sysadmin: TTY=pts/0 ; PWD=/etc/apache2 ; USER=root ; COMMAND=/usr/bin/systemctl status apache2 Aug 27 10:47:20 web01.sias.intra.lighting.com sudo[176903]: pam_unix(sudo:session): session opened for user root by sysadmin(uid=1000) Aug 27 10:47:20 web01.sias.intra.lighting.com sudo[176903]: pam_unix(sudo:session): session closed for user root Aug 27 10:47:34 web01.sias.intra.lighting.com sudo[176906]: sysadmin: TTY=pts/0 ; PWD=/etc/apache2 ; USER=root ; COMMAND=/usr/bin/systemctl start apache2 Aug 27 10:47:34 web01.sias.intra.lighting.com sudo[176906]: pam_unix(sudo:session): session opened for user root by sysadmin(uid=1000) Aug 27 10:47:34 web01.sias.intra.lighting.com systemd[1]: Starting The Apache HTTP Server... Aug 27 10:47:34 web01.sias.intra.lighting.com systemd[1]: Started The Apache HTTP Server. --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
|