Hi ,
Our Linux based system uses an external storage disk called capture.
The capture disk is encrypted with LUKS to protect stored data. During system boot, system uses the LUKS utilities to decrypt the disk before mounting the file systems containing the stored data. Due to an issue with inconsistent ordering of decryption and mounting services during system bootup, during reboots the system with capture disk can get into emergency mode.
Do we have any configuration problem ?
System details, logs and unit files details :
systemd version: 252
Used distribution : AlmaLinux 9.4
Linux kernel version used : 5.15.160-1.11.10.0.x86_64
Journalctl -xb output for error message:
Jan 10 09:53:04 ps systemd[1]: Listening on udev Control Socket. Jan 10 09:53:04 ps systemd[1]: Listening on udev Kernel Socket. Jan 10 09:53:04 ps systemd[1]: capture.mount: Directory /capture to mount over is not empty, mounting anyway. Jan 10 09:53:04 ps systemd[1]: Mounting /capture... Jan 10 09:53:04 ps systemd[1]: Mounting Huge Pages File System... Jan 10 09:53:04 ps systemd[1]: Mounting POSIX Message Queue File System... Jan 10 09:53:04 ps systemd[1]: Mounting /sys/kernel/debug... Jan 10 09:53:04 ps systemd[1]: Mounting Kernel Trace File System... Jan 10 09:53:04 ps systemd[1]: Kernel Module supporting RPCSEC_GSS was skipped because of an unmet condition check (ConditionPathExists=/etc/krb5.keytab). Jan 10 09:53:04 ps systemd[1]: Starting Availability of block devices... Jan 10 09:53:04 ps systemd[1]: Starting Create List of Static Device Nodes... Jan 10 09:53:04 ps systemd[1]: Starting Load legacy module configuration... Jan 10 09:53:04 ps systemd[1]: Starting Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling... Jan 10 09:53:04 ps systemd[1]: Starting Load Kernel Module configfs... Jan 10 09:53:04 ps systemd[1]: Starting Load Kernel Module drm... Jan 10 09:53:04 ps systemd[1]: Starting Load Kernel Module efi_pstore... Jan 10 09:53:04 ps systemd[1]: Starting Load Kernel Module fuse... Jan 10 09:53:04 ps systemd[1]: Starting Read and set NIS domainname from /etc/sysconfig/network... Jan 10 09:53:04 ps systemd[1]: First Boot Wizard was skipped because of an unmet condition check (ConditionFirstBoot=yes). Jan 10 09:53:04 ps systemd[1]: Rebuild Hardware Database was skipped because of an unmet condition check (ConditionNeedsUpdate=/etc). Jan 10 09:53:04 ps systemd[1]: systemd-journald.service: unit configures an IP firewall, but the local system does not support BPF/cgroup firewalling. Jan 10 09:53:04 ps systemd[1]: (This warning is only shown for the first unit using IP firewalling.) Jan 10 09:53:04 ps systemd[1]: Starting Journal Service... Jan 10 09:53:04 ps systemd[1]: Starting Load Kernel Modules... Jan 10 09:53:04 ps systemd[1]: Starting Generate network units from Kernel command line... Jan 10 09:53:04 ps systemd[1]: TPM2 PCR Machine ID Measurement was skipped because of an unmet condition check (ConditionPathExists=/sys/firmware/efi/efivars/StubPcrKernelImage-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f). Jan 10 09:53:04 ps systemd[1]: Create System Users was skipped because no trigger condition checks were met. Jan 10 09:53:04 ps systemd[1]: Starting Coldplug All udev Devices... Jan 10 09:53:04 ps systemd[1]: capture.mount: Mount process exited, code=exited, status=32/n/a Jan 10 09:53:04 ps systemd[1]: capture.mount: Failed with result 'exit-code'. Jan 10 09:53:04 ps systemd[1]: Failed to mount /capture. Jan 10 09:53:04 ps systemd[1]: Dependency failed for Local File Systems. Jan 10 09:53:04 ps systemd[1]: Dependency failed for Systemd service ordering. Jan 10 09:53:04 ps systemd[1]: service-ordering.service: Job service-ordering.service/start failed with result 'dependency'. Jan 10 09:53:04 ps systemd[1]: Dependency failed for Create Volatile Files and Directories. Jan 10 09:53:04 ps systemd[1]: systemd-tmpfiles-setup.service: Job systemd-tmpfiles-setup.service/start failed with result 'dependency'. Jan 10 09:53:04 ps systemd[1]: Dependency failed for Mark the need to relabel after reboot. Jan 10 09:53:04 ps systemd[1]: selinux-autorelabel-mark.service: Job selinux-autorelabel-mark.service/start failed with result 'dependency'. Jan 10 09:53:04 ps systemd[1]: local-fs.target: Job local-fs.target/start failed with result 'dependency'. Jan 10 09:53:04 ps systemd[1]: local-fs.target: Triggering _OnFailure_= dependencies. Jan 10 09:53:04 ps systemd[1]: Unnecessary job was removed for /dev/ttyS0. Jan 10 09:53:04 ps systemd[1]: Mounted Huge Pages File System. systemctl list-units --failed UNIT LOAD ACTIVE SUB DESCRIPTION * capture.mount loaded failed failed /capture LOAD = Reflects whether the unit definition was properly loaded. ACTIVE = The high-level unit activation state, i.e. generalization of SUB. SUB = The low-level unit activation state, values depend on unit type. 1 loaded units listed. systemctl status capture.mount x capture.mount - /capture Loaded: loaded (/etc/fstab; generated) Active: failed (Result: exit-code) since Fri 2025-01-10 09:53:03 UTC; 39min ago Where: /capture What: /dev/mapper/encr-sdd Docs: man:fstab(5) man:systemd-fstab-generator(8) CPU: 5ms Jan 10 09:53:04 ps mount[784]: mount: /capture: special device /dev/mapper/encr-sdd does not exist. Notice: journal has been rotated since unit was started, output may be incomplete. systemctl cat capture.mount # /run/systemd/generator/capture.mount # Automatically generated by systemd-fstab-generator [Unit] Documentation=man:fstab(5) man:systemd-fstab-generator(8) SourcePath=/etc/fstab After=cryptsetup.target Requires=cryptsetup.target Before=local-fs.target Requires=systemd-fsck@dev-mapper-encr\x2dsdd.service After=systemd-fsck@dev-mapper-encr\x2dsdd.service After=blockdev@dev-mapper-encr\x2dsdd.target [Mount] What=/dev/mapper/encr-sdd Where=/capture Type=ext4 Options=defaults,nosuid,x-systemd.requires=cryptsetup.target cat /etc/crypttab encr-sdd UUID=6af9171e-82c7-4be2-bd70-14979067727f /.encr/sdd cat /etc/fstab /dev/disk/by-label/boot /boot vfat ro,defaults 1 2 /dev/disk/by-label/opt /opt ext4 ro,defaults 1 2 /dev/disk/by-label/root / ext4 ro,defaults 1 1 /dev/disk/by-label/notused /notused ext4 defaults,ro,noauto 0 0 /dev/disk/by-label/rescue /rescue ext4 ro,defaults,noauto 0 0 /dev/disk/by-label/var /var ext4 defaults 1 2 /dev/disk/by-label/var_log /var/log ext4 defaults 1 2 /dev/disk/by-label/config /config ext4 defaults,nosuid 1 2 /dev/disk/by-label/swap swap swap defaults /dev/mapper/encr-sdd /capture ext4 defaults,nosuid,x-systemd.requires=cryptsetup.target 1 2 /dev/mapper/v0-deferred /deferred ext4 defaults,noatime,noexec,nosuid 1 3 /dev/mapper/v0-scandir /scandir ext4 defaults,noatime,noexec,nosuid,nobarrier 1 3 /dev/mapper/v0-logs /logs ext4 defaults,noatime,noexec,nosuid 1 3 /dev/mapper/v0-wk /wk ext4 defaults,nosuid 1 3 /var/etc /etc none bind,x-systemd.requires-mounts-for="" 0 0 0 /wk/tmp /tmp none bind,x-systemd.requires-mounts-for="" 0 0 0 /wk/root /root none bind,x-systemd.requires-mounts-for="" 0 0 0 /wk/home /home none bind,x-systemd.requires-mounts-for="" 0 0 0 /wk/AgentDB/db /var/McAfee/agent/db none bind,x-systemd.requires-mounts-for="" 1 0 0 /dev/cdrom /media/cdrom iso9660 noexec,nosuid,noauto,owner,ro 0 0 /dev/cdrom-0 /media/cdrom-0 iso9660 noexec,nosuid,noauto,owner,ro 0 0 /dev/cdrom-usb /media/cdrom-usb iso9660 noexec,nosuid,noauto,owner,ro 0 0 none /proc/fs/nfsd nfsd noauto 0 0 nodev /sys/kernel/debug debugfs defaults 0 0 systemctl list-unit-files | grep systemd-cryptsetup systemd-cryptsetup@encr\x2dsdd.service generated - systemctl status systemd-cryptsetup@encr\x2dsdd.service * systemd-cryptsetup@encr\x2dsdd.service - Cryptography Setup for encr-sdd Loaded: loaded (/etc/crypttab; generated) Active: inactive (dead) Docs: man:crypttab(5) man:systemd-cryptsetup-generator(8) man:systemd-cryptsetup@.service(8) cryptsetup status encr-sdd /dev/mapper/encr-sdd is inactive. ls -lrt /dev/mapper/* crw------- 1 root root 10, 236 Jan 10 09:53 /dev/mapper/control lrwxrwxrwx 1 root root 7 Jan 10 09:53 /dev/mapper/v0-deferred -> ../dm-0 lrwxrwxrwx 1 root root 7 Jan 10 09:53 /dev/mapper/v0-scandir -> ../dm-1 lrwxrwxrwx 1 root root 7 Jan 10 09:53 /dev/mapper/v0-wk -> ../dm-3 lrwxrwxrwx 1 root root 7 Jan 10 09:53 /dev/mapper/v0-logs -> ../dm-2 systemctl show systemd-cryptsetup@encr\x2dsdd.service Type=oneshot ExitType=main Restart=no NotifyAccess=none RestartUSec=100ms TimeoutStartUSec=infinity TimeoutStopUSec=infinity TimeoutAbortUSec=infinity TimeoutStartFailureMode=terminate TimeoutStopFailureMode=terminate RuntimeMaxUSec=infinity RuntimeRandomizedExtraUSec=0 WatchdogUSec=infinity WatchdogTimestampMonotonic=0 RootDirectoryStartOnly=no RemainAfterExit=yes GuessMainPID=yes MainPID=0 ControlPID=0 FileDescriptorStoreMax=0 NFileDescriptorStore=0 StatusErrno=0 Result=success ReloadResult=success CleanResult=success UID=[not set] GID=[not set] NRestarts=0 OOMPolicy=stop ReloadSignal=1 ExecMainStartTimestampMonotonic=0 ExecMainExitTimestampMonotonic=0 ExecMainPID=0 ExecMainCode=0 ExecMainStatus=0 ExecStart={ path=/usr/lib/systemd/systemd-cryptsetup ; argv[]=/usr/lib/systemd/systemd-cryptsetup attach encr-sdd /dev/disk/by-uuid/6af9171e-82c7-4be2-bd70-14979067727f /.encr/sdd ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 } ExecStartEx={ path=/usr/lib/systemd/systemd-cryptsetup ; argv[]=/usr/lib/systemd/systemd-cryptsetup attach encr-sdd /dev/disk/by-uuid/6af9171e-82c7-4be2-bd70-14979067727f /.encr/sdd ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 } ExecStop={ path=/usr/lib/systemd/systemd-cryptsetup ; argv[]=/usr/lib/systemd/systemd-cryptsetup detach encr-sdd ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 } ExecStopEx={ path=/usr/lib/systemd/systemd-cryptsetup ; argv[]=/usr/lib/systemd/systemd-cryptsetup detach encr-sdd ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 } Slice=system-systemd\x2dcryptsetup.slice ControlGroupId=0 MemoryCurrent=[not set] MemoryAvailable=infinity CPUUsageNSec=[not set] TasksCurrent=[not set] IPIngressBytes=[no data] IPIngressPackets=[no data] IPEgressBytes=[no data] IPEgressPackets=[no data] IOReadBytes=18446744073709551615 IOReadOperations=18446744073709551615 IOWriteBytes=18446744073709551615 IOWriteOperations=18446744073709551615 Delegate=no CPUAccounting=yes CPUWeight=[not set] StartupCPUWeight=[not set] CPUShares=[not set] StartupCPUShares=[not set] CPUQuotaPerSecUSec=infinity CPUQuotaPeriodUSec=infinity IOAccounting=no IOWeight=[not set] StartupIOWeight=[not set] BlockIOAccounting=no BlockIOWeight=[not set] StartupBlockIOWeight=[not set] MemoryAccounting=yes DefaultMemoryLow=0 DefaultMemoryMin=0 MemoryMin=0 MemoryLow=0 MemoryHigh=infinity MemoryMax=infinity MemorySwapMax=infinity MemoryLimit=infinity DevicePolicy=auto TasksAccounting=yes TasksMax=50833 IPAccounting=no ManagedOOMSwap=auto ManagedOOMMemoryPressure=auto ManagedOOMMemoryPressureLimit=0 ManagedOOMPreference=none UMask=0022 LimitCPU=infinity LimitCPUSoft=infinity LimitFSIZE=infinity LimitFSIZESoft=infinity LimitDATA=infinity LimitDATASoft=infinity LimitSTACK=infinity LimitSTACKSoft=8388608 LimitCORE=infinity LimitCORESoft=0 LimitRSS=infinity LimitRSSSoft=infinity LimitNOFILE=524288 LimitNOFILESoft=1024 LimitAS=infinity LimitASSoft=infinity LimitNPROC=31771 LimitNPROCSoft=31771 LimitMEMLOCK=8388608 LimitMEMLOCKSoft=8388608 LimitLOCKS=infinity LimitLOCKSSoft=infinity LimitSIGPENDING=31771 LimitSIGPENDINGSoft=31771 LimitMSGQUEUE=819200 LimitMSGQUEUESoft=819200 LimitNICE=0 LimitNICESoft=0 LimitRTPRIO=0 LimitRTPRIOSoft=0 LimitRTTIME=infinity LimitRTTIMESoft=infinity OOMScoreAdjust=500 CoredumpFilter=0x23 Nice=0 IOSchedulingClass=2 IOSchedulingPriority=4 CPUSchedulingPolicy=0 CPUSchedulingPriority=0 CPUAffinityFromNUMA=no NUMAPolicy=n/a TimerSlackNSec=50000 CPUSchedulingResetOnFork=no NonBlocking=no StandardInput=null StandardOutput=journal StandardError=inherit TTYReset=no TTYVHangup=no TTYVTDisallocate=no SyslogPriority=30 SyslogLevelPrefix=yes SyslogLevel=6 SyslogFacility=3 LogLevelMax=-1 LogRateLimitIntervalUSec=0 LogRateLimitBurst=0 SecureBits=0 CapabilityBoundingSet=cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon cap_bpf cap_checkpoint_restore DynamicUser=no RemoveIPC=no PrivateTmp=no PrivateDevices=no ProtectClock=no ProtectKernelTunables=no ProtectKernelModules=no ProtectKernelLogs=no ProtectControlGroups=no PrivateNetwork=no PrivateUsers=no PrivateMounts=no PrivateIPC=no ProtectHome=no ProtectSystem=no SameProcessGroup=no UtmpMode=init IgnoreSIGPIPE=yes NoNewPrivileges=no SystemCallErrorNumber=2147483646 LockPersonality=no RuntimeDirectoryPreserve=no RuntimeDirectoryMode=0755 StateDirectoryMode=0755 CacheDirectoryMode=0755 LogsDirectoryMode=0755 ConfigurationDirectoryMode=0755 TimeoutCleanUSec=infinity MemoryDenyWriteExecute=no RestrictRealtime=no RestrictSUIDSGID=no RestrictNamespaces=no MountAPIVFS=no KeyringMode=shared ProtectProc=default ProcSubset=all ProtectHostname=no KillMode=control-group KillSignal=15 RestartKillSignal=15 FinalKillSignal=9 SendSIGKILL=yes SendSIGHUP=no WatchdogSignal=6 systemctl show cryptsetup.target Id=cryptsetup.target Names=cryptsetup.target Requires="systemd-cryptsetup@encr\\x2dsdd.service" RequiredBy=capture.mount WantedBy=sysinit.target Before=capture.mount After=cryptsetup-pre.target systemd-pcrphase-initrd.service "systemd-cryptsetup@encr\\x2dsdd.service" systemd-ask-password-console.path systemd-ask-password-wall.path Description=cryptsetup.target LoadState=masked ActiveState=inactive FreezerState=running SubState=dead FragmentPath=/etc/systemd/system/cryptsetup.target UnitFileState=masked UnitFilePreset=enabled StateChangeTimestamp=Fri 2025-01-10 09:53:06 UTC StateChangeTimestampMonotonic=11420678 InactiveExitTimestampMonotonic=0 ActiveEnterTimestampMonotonic=0 ActiveExitTimestampMonotonic=0 InactiveEnterTimestampMonotonic=0 CanStart=no CanStop=yes CanReload=no CanIsolate=no CanFreeze=no StopWhenUnneeded=no RefuseManualStart=no RefuseManualStop=no AllowIsolate=no DefaultDependencies=yes _OnSuccessJobMode_=fail _OnFailureJobMode_=replace IgnoreOnIsolate=no NeedDaemonReload=no JobTimeoutUSec=infinity JobRunningTimeoutUSec=infinity JobTimeoutAction=none ConditionResult=no AssertResult=no ConditionTimestampMonotonic=0 AssertTimestampMonotonic=0 LoadError=org.freedesktop.systemd1.UnitMasked "Unit cryptsetup.target is masked." Transient=no Perpetual=no StartLimitIntervalUSec=10s StartLimitBurst=5 StartLimitAction=none FailureAction=none SuccessAction=none CollectMode=inactive Also, Pasted all the logs in https://gist.github.com/sumitkumar1-trellix/cefb8477e9e6bde05419f8a0e842994d
Please let me know if any other config/logs required.
