Hi, The initrd side startup and shutdown of tee-supplicant works now correctly with: [Unit] Description=TEE Supplicant on %i DefaultDependencies=no After=dev-%i.device Wants=dev-%i.device Conflicts=shutdown.target Before=systemd-pcrextend.socket systemd-pcrextend@.service systemd-pcrfs-root.service systemd-pcrfs@.service systemd-pcrmachine.service systemd-pcrphase-initrd.service systemd-pcrphase-sysinit.service systemd-pcrphase.service systemd-tpm2-setup-early.service systemd-tpm2-setup.service tpm2.target sysinit.target shutdown.target [Service] Type=simple ExitType=cgroup EnvironmentFile=-@sysconfdir@/default/tee-supplicant ExecStart=@sbindir@/tee-supplicant $OPTARGS [Install] To stop the tee-supplicant@.service in initrd after all TPM users, I used the long setup with Before=systemd-pcrphase-initrd etc. This seems to work. But I'm failing to setup the start in main rootfs correctly before any of the TPM2 device using services. Since udev is starting the service somehow after initrd already did it once, then how to redo the same steps earlier also on main rootfs? Log from a run where main rootfs side tee-supplicant is started too late and some services already tried to use the TPM2 device which resulted in failures: [initrd boot finishing up] Stopping [0;1;39mTEE Supplicant on teepriv0[0m... [[0;32m OK [0m] Stopped [0;1;39mTEE Supplicant on teepriv0[0m. ^ initrd side tee-supplicant stopped almost last so that all users were shutdown already [[0;32m OK [0m] Finished [0;1;39mCleanup udev Database[0m. [[0;32m OK [0m] Reached target [0;1;39mSwitch Root[0m. Starting [0;1;39mSwitch Root[0m... [[0;32m OK [0m] Stopped [0;1;39mSwitch Root[0m. [[0;32m OK [0m] Created slice [0;1;39mSlice /system/getty[0m. [[0;32m OK [0m] Created slice [0;1;39mSlice /system/serial-getty[0m. [[0;32m OK [0m] Created slice [0;1;39mUser and Session Slice[0m. [[0;32m OK [0m] Started [0;1;39mDispatch Password Requests to Console Directory Watch[0m. [[0;32m OK [0m] Started [0;1;39mForward Password Requests to Wall Directory Watch[0m. Expecting device [0;1;39m/dev/ttyAMA0[0m... Expecting device [0;1;39m/dev/ttyS2[0m... [[0;32m OK [0m] Reached target [0;1;39mBlock Device Preparation for /dev/mapper/usr[0m. [[0;32m OK [0m] Reached target [0;1;39mLocal Encrypted Volumes[0m. [[0;32m OK [0m] Stopped target [0;1;39mSwitch Root[0m. [[0;32m OK [0m] Stopped target [0;1;39mInitrd File Systems[0m. [[0;32m OK [0m] Stopped target [0;1;39mInitrd Root File System[0m. [[0;32m OK [0m] Reached target [0;1;39mLocal Integrity Protected Volumes[0m. [[0;32m OK [0m] Reached target [0;1;39mPath Units[0m. [[0;32m OK [0m] Reached target [0;1;39mRemote Encrypted Volumes[0m. [[0;32m OK [0m] Reached target [0;1;39mRemote File Systems[0m. [[0;32m OK [0m] Reached target [0;1;39mSlice Units[0m. [[0;32m OK [0m] Reached target [0;1;39mSwaps[0m. [[0;32m OK [0m] Reached target [0;1;39mLocal Verity Protected Volumes[0m. [[0;32m OK [0m] Listening on [0;1;39mRPCbind Server Activation Socket[0m. [[0;32m OK [0m] Listening on [0;1;39mRPCbind Server Activation Socket[0m. [[0;32m OK [0m] Reached target [0;1;39mRPC Port Mapper[0m. [[0;32m OK [0m] Listening on [0;1;39mSyslog Socket[0m. [[0;32m OK [0m] Listening on [0;1;39minitctl Compatibility Named Pipe[0m. [[0;32m OK [0m] Listening on [0;1;39mNetwork Service Netlink Socket[0m. [[0;32m OK [0m] Listening on [0;1;39mTPM2 PCR Extension (Varlink)[0m. ^ this should only be started once tee-supplicant is running again from main rootfs [[0;32m OK [0m] Listening on [0;1;39mudev Control Socket[0m. [[0;32m OK [0m] Listening on [0;1;39mudev Kernel Socket[0m. [[0;32m OK [0m] Listening on [0;1;39mUser Database Manager Socket[0m. Mounting [0;1;39mTemporary Directory /tmp[0m... Starting [0;1;39mCreate List of Static Device Nodes[0m... Starting [0;1;39mLoad Kernel Module configfs[0m... Starting [0;1;39mLoad Kernel Module dm_mod[0m... Starting [0;1;39mLoad Kernel Module drm[0m... Starting [0;1;39mLoad Kernel Module fuse[0m... Starting [0;1;39mLoad Kernel Module loop[0m... Starting [0;1;39mRPC Bind[0m... [[0;32m OK [0m] Stopped [0;1;39msystemd-cryptsetup@root.service[0m. [[0;32m OK [0m] Stopped [0;1;39msystemd-fsck-usr.service[0m. Starting [0;1;39mJournal Service[0m... Starting [0;1;39mLoad Kernel Modules[0m... Starting [0;1;39mGenerate network units from Kernel command line[0m... Starting [0;1;39mTPM2 PCR Machine ID Measurement[0m... Starting [0;1;39mRemount Root and Kernel File Systems[0m... Starting [0;1;39mColdplug All udev Devices[0m... [[0;32m OK [0m] Mounted [0;1;39mTemporary Directory /tmp[0m. [[0;32m OK [0m] Finished [0;1;39mCreate List of Static Device Nodes[0m. [[0;32m OK [0m] Finished [0;1;39mLoad Kernel Module configfs[0m. [[0;32m OK [0m] Finished [0;1;39mLoad Kernel Module dm_mod[0m. [[0;32m OK [0m] Finished [0;1;39mLoad Kernel Module drm[0m. [[0;32m OK [0m] Finished [0;1;39mLoad Kernel Module loop[0m. [ 13.021539] tpm tpm0: ftpm_tee_tpm_op_send: SUBMIT_COMMAND invoke error: 0xffff3024 [ 13.022259] tpm tpm0: tpm_try_transmit: send(): error -53212 ^ here TPM usage from kernel fails since tee-supplicant is not running [[0;32m OK [0m] Finished [0;1;39mLoad Kernel Module fuse[0m. [[0;32m OK [0m] Finished [0;1;39mGenerate network units from Kernel command line[0m. [[0;32m OK [0m] Started [0;1;39mJournal Service[0m. [[0;1;31mFAILED[0m] Failed to start [0;1;39mTPM2 PCR Machine ID Measurement[0m. See 'systemctl status systemd-pcrmachine.service' for details. ^ systemd-pcrmachine.service failed due to tee-supplicant not running yet [[0;32m OK [0m] Finished [0;1;39mRemount Root and Kernel File Systems[0m. Mounting [0;1;39mFUSE Control File System[0m... Starting [0;1;39mGrow Root File System[0m... Starting [0;1;39mFlush Journal to Persistent Storage[0m... Starting [0;1;39mTPM2 PCR Root File System Measurement[0m... Starting [0;1;39mCreate Static Device Nodes in /dev gracefully[0m... Starting [0;1;39mTPM2 SRK Setup[0m... [[0;32m OK [0m] Mounted [0;1;39mFUSE Control File System[0m. [[0;32m OK [0m] Finished [0;1;39mFlush Journal to Persistent Storage[0m. [[0;32m OK [0m] Finished [0;1;39mLoad Kernel Modules[0m. [[0;32m OK [0m] Finished [0;1;39mColdplug All udev Devices[0m. Starting [0;1;39mApply Kernel Variables[0m... [ 29.247979] tpm tpm0: ftpm_tee_tpm_op_send: SUBMIT_COMMAND invoke error: 0xffff3024 [ 29.248733] tpm tpm0: tpm_try_transmit: send(): error -53212 [[0;1;31mFAILED[0m] Failed to start [0;1;39mTPM2 SRK Setup[0m. [ 29.264721] tpm tpm0: ftpm_tee_tpm_op_send: SUBMIT_COMMAND invoke error: 0xffff3024 [ 29.265445] tpm tpm0: tpm_try_transmit: send(): error -53212 See 'systemctl status systemd-tpm2-setup.service' for details. ^ systemd-tpm2-setup.service failed due to tee-supplicant not running yet [[0;1;31mFAILED[0m] Failed to start [0;1;39mTPM2 PCR Root File System Measurement[0m. See 'systemctl status systemd-pcrfs-root.service' for details. [[0;32m OK [0m] Finished [0;1;39mCreate Static Device Nodes in /dev gracefully[0m. Starting [0;1;39mCreate System Users[0m... [[0;32m OK [0m] Finished [0;1;39mApply Kernel Variables[0m. [[0;32m OK [0m] Finished [0;1;39mGrow Root File System[0m. Starting [0;1;39mUser Database Manager[0m... [[0;32m OK [0m] Started [0;1;39mUser Database Manager[0m. [[0;32m OK [0m] Finished [0;1;39mCreate System Users[0m. Starting [0;1;39mCreate Static Device Nodes in /dev[0m... [[0;32m OK [0m] Finished [0;1;39mCreate Static Device Nodes in /dev[0m. [[0;32m OK [0m] Reached target [0;1;39mPreparation for Local File Systems[0m. [[0;32m OK [0m] Set up automount [0;1;39mEFI System Partition Automount[0m. Mounting [0;1;39m/var/volatile[0m... Starting [0;1;39mRule-based Manager for Device Events and Files[0m... [[0;32m OK [0m] Mounted [0;1;39m/var/volatile[0m. Starting [0;1;39mLoad/Save OS Random Seed[0m... [[0;32m OK [0m] Reached target [0;1;39mLocal File Systems[0m. Starting [0;1;39mRebuild Dynamic Linker Cache[0m... Starting [0;1;39mAutomatic Boot Loader Update[0m... Starting [0;1;39mCreate Volatile Files and Directories[0m... [[0;32m OK [0m] Created slice [0;1;39mSlice /system/systemd-fsck[0m. Expecting device [0;1;39m/dev/disk/by-diskseq/27-part6[0m... [[0;32m OK [0m] Finished [0;1;39mCreate Volatile Files and Directories[0m. [[0;32m OK [0m] Finished [0;1;39mLoad/Save OS Random Seed[0m. [[0;32m OK [0m] Reached target [0;1;39mFirst Boot Complete[0m. Starting [0;1;39mUpdate Boot Loader Random Seed[0m... Starting [0;1;39mRebuild Journal Catalog[0m... Starting [0;1;39mCommit a transient machine-id on disk[0m... Starting [0;1;39mNetwork Name Resolution[0m... Starting [0;1;39mNetwork Time Synchronization[0m... Starting [0;1;39mRecord System Boot/Shutdown in UTMP[0m... [[0;32m OK [0m] Started [0;1;39mNetwork Time Synchronization[0m. [[0;32m OK [0m] Finished [0;1;39mRecord System Boot/Shutdown in UTMP[0m. [[0;32m OK [0m] Reached target [0;1;39mSystem Time Set[0m. [[0;32m OK [0m] Started [0;1;39mNetwork Name Resolution[0m. [[0;32m OK [0m] Reached target [0;1;39mHost and Network Name Lookups[0m. [[0;32m OK [0m] Finished [0;1;39mCommit a transient machine-id on disk[0m. [[0;32m OK [0m] Finished [0;1;39mRebuild Journal Catalog[0m. [[0;32m OK [0m] Started [0;1;39mRule-based Manager for Device Events and Files[0m. Starting [0;1;39mLoad Kernel Module configfs[0m... [[0;32m OK [0m] Finished [0;1;39mLoad Kernel Module configfs[0m. [[0;32m OK [0m] Found device [0;1;39m/dev/tpmrm0[0m. [[0;32m OK [0m] Reached target [0;1;39mTrusted Platform Module[0m. Starting [0;1;39mFile System Check on /dev/disk/by-diskseq/27-part6[0m... [[0;32m OK [0m] Listening on [0;1;39mLoad/Save RF Kill Switch Status /dev/rfkill Watch[0m. [[0;32m OK [0m] Started [0;1;39mTEE Supplicant on teepriv0[0m. ^ tee-supplicant@teepriv0.service is started only here. Is the udev rule/re-initialization of initrd device done at a later stage? How to delay the TPM2 users until tee-supplicant runs or run the tee-supplicant startup earlier? Starting [0;1;39mVirtual Console Setup[0m... Starting [0;1;39mLoad/Save RF Kill Switch Status[0m... [ 32.282198] Bluetooth: hci0: BCM: firmware Patch file not found, tried: [ 32.282806] Bluetooth: hci0: BCM: 'brcm/BCM4345C5.radxa,rockpi4b.hcd' [ 32.283449] Bluetooth: hci0: BCM: 'brcm/BCM4345C5.hcd' [ 32.283916] Bluetooth: hci0: BCM: 'brcm/BCM.radxa,rockpi4b.hcd' [ 32.284450] Bluetooth: hci0: BCM: 'brcm/BCM.hcd' Starting [0;1;39mTPM2 PCR Machine ID Measurement[0m... Starting [0;1;39mTPM2 SRK Setup[0m... [[0;32m OK [0m] Started [0;1;39mLoad/Save RF Kill Switch Status[0m. [ 32.326093] Bluetooth: hci0: Opcode 0x2031 failed: -22 [ 32.329322] tpm tpm0: ftpm_tee_tpm_op_send: SUBMIT_COMMAND invoke error: 0xffff3024 [ 32.330020] tpm tpm0: tpm_try_transmit: send(): error -53212 [[0;1;31mFAILED[0m] Failed to start [0;1;39mTPM2 PCR Machine ID Measurement[0m. See 'systemctl status systemd-pcrmachine.service' for details. [ 32.358485] tpm tpm0: ftpm_tee_tpm_op_send: SUBMIT_COMMAND invoke error: 0xffff3024 [ 32.359178] tpm tpm0: tpm_try_transmit: send(): error -53212 [[0;1;31mFAILED[0m] Failed to start [0;1;39mTPM2 SRK Setup[0m. See 'systemctl status systemd-tpm2-setup.service' for details. Starting [0;1;39mTPM2 PCR Machine ID Measurement[0m... Starting [0;1;39mTPM2 SRK Setup[0m... [ 32.744315] tpm tpm0: ftpm_tee_tpm_op_send: SUBMIT_COMMAND invoke error: 0xffff3024 [ 32.745008] tpm tpm0: tpm_try_transmit: send(): error -53212 [[0;1;31mFAILED[0m] Failed to start [0;1;39mTPM2 PCR Machine ID Measurement[0m. See 'systemctl status systemd-pcrmachine.service' for details. [[0;32m OK [0m] Finished [0;1;39mVirtual Console Setup[0m. [ 32.790197] tpm tpm0: ftpm_tee_tpm_op_send: SUBMIT_COMMAND invoke error: 0xffff3024 [ 32.790893] tpm tpm0: tpm_try_transmit: send(): error -53212 [[0;1;31mFAILED[0m] Failed to start [0;1;39mTPM2 SRK Setup[0m. See 'systemctl status systemd-tpm2-setup.service' for details. [[0;32m OK [0m] Finished [0;1;39mFile System Check on /dev/disk/by-diskseq/27-part6[0m. Mounting [0;1;39mEFI System Partition Automount[0m... [[0;32m OK [0m] Mounted [0;1;39mEFI System Partition Automount[0m. [[0;32m OK [0m] Finished [0;1;39mAutomatic Boot Loader Update[0m. [[0;32m OK [0m] Finished [0;1;39mUpdate Boot Loader Random Seed[0m. [ 33.316910] brcmfmac: brcmf_sdio_htclk: HT Avail timeout (1000000): clkctl 0x50 [[0;32m OK [0m] Finished [0;1;39mRebuild Dynamic Linker Cache[0m. Starting [0;1;39mUpdate is Completed[0m... [[0;32m OK [0m] Finished [0;1;39mUpdate is Completed[0m. [[0;32m OK [0m] Reached target [0;1;39mSystem Initialization[0m. [[0;32m OK [0m] Started [0;1;39mDaily rotation of log files[0m. [[0;32m OK [0m] Started [0;1;39mDaily Cleanup of Temporary Directories[0m. [[0;32m OK [0m] Reached target [0;1;39mTimer Units[0m. [[0;32m OK [0m] Listening on [0;1;39mAvahi mDNS/DNS-SD Stack Activation Socket[0m. [[0;32m OK [0m] Listening on [0;1;39mD-Bus System Message Bus Socket[0m. Starting [0;1;39mDocker Socket for the API[0m... [[0;32m OK [0m] Listening on [0;1;39mPC/SC Smart Card Daemon Activation Socket[0m. [ 33.511784] tpm tpm0: ftpm_tee_tpm_op_send: SUBMIT_COMMAND invoke error: 0xffff3024 [ 33.512552] tpm tpm0: tpm_try_transmit: send(): error -53212 Starting [0;1;39msshd.socket[0m... Starting [0;1;39mTPM2 PCR Barrier (Initialization)[0m... [[0;32m OK [0m] Listening on [0;1;39mDocker Socket for the API[0m. [[0;32m OK [0m] Listening on [0;1;39msshd.socket[0m. [[0;32m OK [0m] Reached target [0;1;39mSocket Units[0m. [ 33.607192] tpm tpm0: ftpm_tee_tpm_op_send: SUBMIT_COMMAND invoke error: 0xffff3024 [ 33.607959] tpm tpm0: tpm_try_transmit: send(): error -53212 [[0;1;31mFAILED[0m] Failed to start [0;1;39mTPM2 PCR Barrier (Initialization)[0m. See 'systemctl status systemd-pcrphase-sysinit.service' for details. [[0;32m OK [0m] Reached target [0;1;39mBasic System[0m. Starting [0;1;39mSave/Restore Sound Card State[0m... Starting [0;1;39mAvahi mDNS/DNS-SD Stack[0m... Starting [0;1;39mBluetooth service[0m... [[0;32m OK [0m] Started [0;1;39mKernel Logging Service[0m. [[0;32m OK [0m] Started [0;1;39mSystem Logging Service[0m. [[0;32m OK [0m] Started [0;1;39mPeriodic Command Scheduler[0m. Starting [0;1;39mD-Bus System Message Bus[0m... Starting [0;1;39mIPv6 Packet Filtering Framework[0m... Starting [0;1;39mIPv4 Packet Filtering Framework[0m... Starting [0;1;39mUser Login Management[0m... Starting [0;1;39mTPM2 PCR Barrier (User)[0m... Starting [0;1;39mOpenSSH Key Generation[0m... [[0;32m OK [0m] Finished [0;1;39mSave/Restore Sound Card State[0m. Cheers, -Mikko