On Fri, Dec 29, 2017 at 11:00:58AM -0700, Jason Gunthorpe wrote:
> On Fri, Dec 29, 2017 at 06:10:06PM +0800, Honggang LI wrote:
> > From: Honggang Li <honli@xxxxxxxxxx>
> >
> > The srp_daemon service will be started at the very beginning state
> > of systemd when boot/reboot the machine, in case srp_daemon.service
> > is not after network.target. As result, the srp_daemon.service will
> > be terminated because of SERVICE_FAILURE_RESOURCES.
>
> How is this possible? srp_daemon.service just runs a script that
> doesn't touch the network.
To reproduce it, you just need enable srp_daemon.serice and then reboot
the machine. Watch the serial console when you are waiting for machine
boot up. Please see attached /var/log/boot.log for details. After system
boot up, check the status of srp_daemon.serice.
============
systemctl status srp_daemon.service -l
● srp_daemon.service - Daemon that discovers and logs in to SRP target systems
Loaded: loaded (/usr/lib/systemd/system/srp_daemon.service; enabled; vendor preset: disabled)
Active: failed (Result: resources)
Docs: man:srp_daemon
file:/etc/srp_daemon.conf
Dec 30 08:45:25 localhost.localdomain systemd[1]: [/usr/lib/systemd/system/srp_daemon.service:12] Unknown lvalue 'MemoryDenyWriteExecute' in section 'Service'
Dec 30 08:45:25 localhost.localdomain systemd[1]: [/usr/lib/systemd/system/srp_daemon.service:15] Unknown lvalue 'ProtectKernelModules' in section 'Service'
Dec 30 08:45:25 localhost.localdomain systemd[1]: [/usr/lib/systemd/system/srp_daemon.service:16] Unknown lvalue 'RestrictRealtime' in section 'Service'
=============
Note, it does not make any difference after remove all unknown lvaule
from srp_daemon.serivce and srp_daemon_port@serivce. We are using an
old version of systemd, which does not support such lvaule.
>
> I can't see any way that srp_daemon.service should have this added,
> please explain more what is going on.
>
> I could potentially understand needing it in srp_daemon_port@.service,
No, it does not work with srp_daemon_port@.service.
$ grep -w -n network.target /usr/lib/systemd/system/srp_daemon*
/usr/lib/systemd/system/srp_daemon_port@.service:21:After=srp_daemon.service network.target
> but even that needs much more explaination about what exactly is
> causing this requirement.
>
> You said SERVICE_FAILURE_RESOURCES which is an internal systemd error
> code.
Yes, it is systemd-219-51.el7.x86_64 error code.
> Is this because of PrivateNetwork=yes or something similar?
How to test or verify this?
[[32m OK [0m] Started Show Plymouth Boot Screen.
[[32m OK [0m] Reached target Paths.
[[32m OK [0m] Reached target Basic System.
%G[[32m OK [0m] Found device MB0500GCEHF 7.
Starting File System Check on /dev/...7-f857-41d3-b7ad-5b9618f6f39e...
[[32m OK [0m] Started dracut initqueue hook.
[[32m OK [0m] Reached target Remote File Systems (Pre).
[[32m OK [0m] Reached target Remote File Systems.
[[32m OK [0m] Started File System Check on /dev/d...7b7-f857-41d3-b7ad-5b9618f6f39e.
Mounting /sysroot...
[[32m OK [0m] Mounted /sysroot.
[[32m OK [0m] Reached target Initrd Root File System.
Starting Reload Configuration from the Real Root...
[[32m OK [0m] Started Reload Configuration from the Real Root.
[[32m OK [0m] Reached target Initrd File Systems.
[[32m OK [0m] Reached target Initrd Default Target.
Starting dracut pre-pivot and cleanup hook...
[[32m OK [0m] Started dracut pre-pivot and cleanup hook.
Starting Cleaning Up and Shutting Down Daemons...
[[32m OK [0m] Stopped target Timers.
Starting Plymouth switch root service...
[[32m OK [0m] Stopped Cleaning Up and Shutting Down Daemons.
[[32m OK [0m] Stopped dracut pre-pivot and cleanup hook.
Stopping dracut pre-pivot and cleanup hook...
[[32m OK [0m] Stopped target Remote File Systems.
[[32m OK [0m] Stopped target Remote File Systems (Pre).
[[32m OK [0m] Stopped dracut initqueue hook.
Stopping dracut initqueue hook...
[[32m OK [0m] Stopped target Initrd Default Target.
[[32m OK [0m] Stopped target Basic System.
[[32m OK [0m] Stopped target Paths.
[[32m OK [0m] Stopped target Slices.
[[32m OK [0m] Stopped target System Initialization.
Stopping udev Kernel Device Manager...
[[32m OK [0m] Stopped target Local File Systems.
[[32m OK [0m] Stopped udev Coldplug all Devices.
Stopping udev Coldplug all Devices...
[[32m OK [0m] Stopped Apply Kernel Variables.
Stopping Apply Kernel Variables...
[[32m OK [0m] Stopped target Swap.
[[32m OK [0m] Stopped target Sockets.
[[32m OK [0m] Stopped udev Kernel Device Manager.
[[32m OK [0m] Stopped Create Static Device Nodes in /dev.
Stopping Create Static Device Nodes in /dev...
[[32m OK [0m] Stopped Create list of required sta...ce nodes for the current kernel.
Stopping Create list of required st... nodes for the current kernel...
[[32m OK [0m] Stopped dracut pre-udev hook.
Stopping dracut pre-udev hook...
[[32m OK [0m] Stopped dracut cmdline hook.
Stopping dracut cmdline hook...
[[32m OK [0m] Closed udev Control Socket.
[[32m OK [0m] Closed udev Kernel Socket.
Starting Cleanup udevd DB...
[[32m OK [0m] Started Cleanup udevd DB.
[[32m OK [0m] Reached target Switch Root.
[[32m OK [0m] Started Plymouth switch root service.
Starting Switch Root...
Welcome to [0;31mRed Hat Enterprise Linux Server 7.5 Beta (Maipo)[0m!
[[32m OK [0m] Stopped Switch Root.
[[32m OK [0m] Stopped Journal Service.
Starting Journal Service...
[[32m OK [0m] Listening on udev Control Socket.
[[32m OK [0m] Listening on udev Kernel Socket.
Mounting POSIX Message Queue File System...
[[1;31mFAILED[0m] Failed to start Daemon that discove...d logs in to SRP target systems.
See 'systemctl status srp_daemon.service' for details.
Starting Daemon that discovers and logs in to SRP target systems...
[[32m OK [0m] Created slice system-selinux\x2dpol...grate\x2dlocal\x2dchanges.slice.
[[32m OK [0m] Listening on /dev/initctl Compatibility Named Pipe.
[[32m OK [0m] Stopped File System Check on Root Device.
Stopping File System Check on Root Device...
[[32m OK [0m] Set up automount Arbitrary Executab...ats File System Automount Point.
[[32m OK [0m] Created slice User and Session Slice.
Mounting Huge Pages File System...
[[32m OK [0m] Reached target Slices.
[[32m OK [0m] Listening on Delayed Shutdown Socket.
[[32m OK [0m] Created slice system-serial\x2dgetty.slice.
[[32m OK [0m] Reached target Local Encrypted Volumes.
Mounting Debug File System...
[[32m OK [0m] Stopped target Switch Root.
[[32m OK [0m] Stopped target Initrd File Systems.
Starting Create list of required st... nodes for the current kernel...
[[32m OK [0m] Stopped target Initrd Root File System.
[[32m OK [0m] Created slice system-getty.slice.
Starting Initialize the iWARP/InfiniBand/RDMA stack in the kernel...
Starting Apply Kernel Variables...
Starting Remount Root and Kernel File Systems...
[[32m OK [0m] Created slice system-systemd\x2dfsck.slice.
[[32m OK [0m] Mounted Debug File System.
[[32m OK [0m] Mounted POSIX Message Queue File System.
[[32m OK [0m] Mounted Huge Pages File System.
[[32m OK [0m] Started Journal Service.
[[32m OK [0m] Started Create list of required sta...ce nodes for the current kernel.
[[32m OK [0m] Started Remount Root and Kernel File Systems.
Starting Load/Save Random Seed...
Starting Configure read-only root support...
Starting udev Coldplug all Devices...
Starting Create Static Device Nodes in /dev...
Starting Flush Journal to Persistent Storage...
[[32m OK [0m] Started Apply Kernel Variables.
[[32m OK [0m] Started udev Coldplug all Devices.
[[32m OK [0m] Started Load/Save Random Seed.
[[32m OK [0m] Started Flush Journal to Persistent Storage.
[[32m OK [0m] Started Create Static Device Nodes in /dev.
[[32m OK [0m] Reached target Local File Systems (Pre).
Starting udev Kernel Device Manager...
[[32m OK [0m] Started Configure read-only root support.
[[32m OK [0m] Started udev Kernel Device Manager.
[[32m OK [0m] Found device /dev/ttyS1.
%G%G[[32m OK [0m] Found device MB0500GCEHF 3.
[[32m OK [0m] Found device MB0500GCEHF 2.
[[32m OK [0m] Found device MB0500GCEHF 6.
[[32m OK [0m] Found device MB0500GCEHF 5.
Mounting /mnt/rdma-xfs...
Activating swap /dev/disk/by-uuid/a...6-f0ce-4513-99fa-72bbdb9f5309...
Mounting /boot...
Starting File System Check on /dev/...6-129a-4cb2-9040-a0484cbc8765...
[[32m OK [0m] Activated swap /dev/disk/by-uuid/add068f6-f0ce-4513-99fa-72bbdb9f5309.
[[32m OK [0m] Reached target Swap.
[ 17.739449] systemd-fsck[561]: /dev/sda3: clean, 11/1638400 files, 146893/6553600 blocks
[[32m OK [0m] Mounted /boot.
[[32m OK [0m] Started File System Check on /dev/d...386-129a-4cb2-9040-a0484cbc8765.
Mounting /mnt/rdma-ext4...
[[32m OK [0m] Mounted /mnt/rdma-xfs.
[[32m OK [0m] Mounted /mnt/rdma-ext4.
[[32m OK [0m] Reached target Local File Systems.
Starting Preprocess NFS configuration...
Starting Tell Plymouth To Write Out Runtime Data...
Starting Import network configuration from initramfs...
Starting RDMA Node Description Daemon...
[[32m OK [0m] Created slice system-rdma\x2dload\x2dmodules.slice.
Starting Load RDMA modules from /etc/rdma/modules/roce.conf...
Starting Load RDMA modules from /etc/rdma/modules/infiniband.conf...
Starting Load RDMA modules from /etc/rdma/modules/rdma.conf...
[[32m OK [0m] Started Preprocess NFS configuration.
[[32m OK [0m] Started Load RDMA modules from /etc/rdma/modules/roce.conf.
[[32m OK [0m] Started Import network configuration from initramfs.
Starting Create Volatile Files and Directories...
[[32m OK [0m] Started Tell Plymouth To Write Out Runtime Data.
[[32m OK [0m] Started RDMA Node Description Daemon.
[[32m OK [0m] Started Create Volatile Files and Directories.
Starting Security Auditing Service...
Mounting RPC Pipe File System...
[[32m OK [0m] Started Load RDMA modules from /etc/rdma/modules/infiniband.conf.
[[32m OK [0m] Mounted RPC Pipe File System.
[[32m OK [0m] Reached target rpc_pipefs.target.
[[32m OK [0m] Started Load RDMA modules from /etc/rdma/modules/rdma.conf.
[[32m OK [0m] Reached target RDMA Hardware.
[[32m OK [0m] Reached target Network (Pre).
[[32m OK [0m] Started Initialize the iWARP/InfiniBand/RDMA stack in the kernel.
Starting Starts the OpenSM InfiniBand fabric Subnet Manager...
[[32m OK [0m] Started Starts the OpenSM InfiniBand fabric Subnet Manager.
[[32m OK [0m] Started Security Auditing Service.
Starting Update UTMP about System Boot/Shutdown...
[[32m OK [0m] Started Update UTMP about System Boot/Shutdown.
[[32m OK [0m] Reached target System Initialization.
[[32m OK [0m] Reached target Paths.
[[32m OK [0m] Listening on D-Bus System Message Bus Socket.
[[32m OK [0m] Reached target Timers.
[[32m OK [0m] Listening on Open-iSCSI iscsid Socket.
[[32m OK [0m] Listening on RPCbind Server Activation Socket.
Starting RPC bind service...
[[32m OK [0m] Listening on Open-iSCSI iscsiuio Socket.
[[32m OK [0m] Reached target Sockets.
[[32m OK [0m] Reached target Basic System.
Starting GSSAPI Proxy Daemon...
[[32m OK [0m] Started D-Bus System Message Bus.
Starting D-Bus System Message Bus...
Starting Network Manager...
Starting Dump dmesg to /var/log/dmesg...
[[32m OK [0m] Started irqbalance daemon.
Starting irqbalance daemon...
Starting Login Service...
Starting Authorization Manager...
Starting Load CPU microcode update...
Starting NTP client/server...
[[32m OK [0m] Started RPC bind service.
[[32m OK [0m] Started GSSAPI Proxy Daemon.
[[32m OK [0m] Started Load CPU microcode update.
[[32m OK [0m] Reached target NFS client services.
[[32m OK [0m] Started Login Service.
[[32m OK [0m] Started Dump dmesg to /var/log/dmesg.
[[32m OK [0m] Started NTP client/server.
Starting Wait for chrony to synchronize system clock...
[[32m OK [0m] Started Authorization Manager.
Starting Hostname Service...
[[32m OK [0m] Started Hostname Service.
[[32m OK [0m] Started Network Manager.
[[32m OK [0m] Reached target Network.
Starting Enable periodic update of entitlement certificates....
Starting Login and scanning of iSCSI devices...
Starting OpenSSH server daemon...
Starting Dynamic System Tuning Daemon...
Starting Logout off all iSCSI sessions on shutdown...
Starting Postfix Mail Transport Agent...
[[32m OK [0m] Reached target Network is Online.
Starting System Logging Service...
Starting Notify NFS peers of a restart...
[[32m OK [0m] Started Enable periodic update of entitlement certificates..
[[32m OK [0m] Started Logout off all iSCSI sessions on shutdown.
Starting Network Manager Script Dispatcher Service...
[[32m OK [0m] Started Notify NFS peers of a restart.
[[32m OK [0m] Started System Logging Service.
Starting Open-iSCSI...
[[32m OK [0m] Started Open-iSCSI.
[[32m OK [0m] Started OpenSSH server daemon.
[[32m OK [0m] Started Network Manager Script Dispatcher Service.
[[0m[31m* [0m] (1 of 4) A start job is running for...ze system clock (12s / no limit)
[K[[32m OK [0m] Started Dynamic System Tuning Daemon.
[[32m OK [0m] Started Postfix Mail Transport Agent.
[[32m OK [0m] Reached target Remote File Systems (Pre).
[[32m OK [0m] Reached target Remote File Systems.
Starting Permit User Sessions...
Starting Crash recovery kernel arming...
Starting Availability of block devices...
[[32m OK [0m] Started Availability of block devices.
[[32m OK [0m] Started Permit User Sessions.
Starting Terminate Plymouth Boot Screen...
Starting Wait for Plymouth Boot Screen to Quit...
[[32m OK [0m] Started Job spooling tools.
Starting Job spooling tools...
