Re: sysext merge of /usr/local

[Itxaka Serrano Garcia <itxaka.garcia@xxxxxxxxxxxxxxxx>]

For anyone that may stumbled upon this, we worked around this issue by setting a more fine-grained dirs on the SYSTEMD_SYSEXT_HIERARCHIES env var. So it would just match those instead of overlaying the full /usr and getting our mounts under it lost :D

root@localhost:~# SYSTEMD_SYSEXT_HIERARCHIES=/usr/local/bin:/usr/local/sbin:/usr/local/include:/usr/local/lib:/usr/local/share:/usr/local/src:/usr/bin:/usr/share:/usr/lib:/usr/include:/usr/src:/usr/sbin systemd-sysext refresh
Using extensions 'work'.
Merged extensions into '/usr/local/bin'.
Merged extensions into '/usr/lib'.

That made our sysext work AND respected our mount under /usr/local with our own writable mounts and such :D

On Mon, Jun 10, 2024 at 6:43 PM Itxaka Serrano Garcia <itxaka.garcia@xxxxxxxxxxxxxxxx> wrote:

Ah, I guess this is because overlayfs does not respect any underlying mounts in the /usr so they get lost

So this has nothing to do with sysext.

In any case, is this a valid use case that we can see happening? Or now that we are here already, if anyone got a suggestion it would be welcome, otherwise feel free to ignore this email :)

El lun, 10 jun 2024, 17:19, Itxaka Serrano Garcia <itxaka.garcia@xxxxxxxxxxxxxxxx> escribió:

Hey again,

I'm back with more annoying questions about sysext :D

According to the docs, sysext only "extends" the existing usr/opt/etc with the sysext contents but we are seeing a different thing here:

root@cos-recovery:~# stat /usr/local/file2
  File: /usr/local/file2
  Size: 0         Blocks: 0          IO Block: 4096   regular empty file
Device: 0,47 Inode: 171         Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-06-10 14:56:50.725904625 +0000
Modify: 2024-06-10 14:56:50.725904625 +0000
Change: 2024-06-10 14:56:50.725904625 +0000
 Birth: 2024-06-10 14:56:50.725904625 +0000
root@cos-recovery:~# systemd-sysext refresh
Using extensions 'work'.
Merged extensions into '/usr'.
root@cos-recovery:~# stat /usr/local/file2
stat: cannot statx '/usr/local/file2': No such file or directory

root@cos-recovery:~# systemd-sysext unmerge
Unmerged '/usr'.
root@cos-recovery:~# stat /usr/local/file2
  File: /usr/local/file2
  Size: 0         Blocks: 0          IO Block: 4096   regular empty file
Device: 0,47 Inode: 171         Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-06-10 14:56:50.725904625 +0000
Modify: 2024-06-10 14:56:50.725904625 +0000
Change: 2024-06-10 14:56:50.725904625 +0000
 Birth: 2024-06-10 14:56:50.725904625 +0000

Weirdly enough, any files under /usr/ that we create are NOT overridden when we load any sysext, it only seems to happen in the dirs inside /usr

It's true that we are using USI here and on a "normal" non-USI/UKI system this behaviour doesn't seem to happen.

The main difference I can see is that the root fs in the failure case is a tmpfs while in the working case it's an ext4 fs, but for example our /usr/local is mounted to a ext4 disk:

/dev/mapper/vda3 on /usr/local type ext4 (rw,relatime)

If I do create an overlay on /usr then it seems to work as expected but its too late for us, we lose anything mounted there and hidden dirs.

Is this a bug or its expected behaviour? Is there anything we could be missing that triggers this behaviour? Any pointers on why this would react like this?

Cheers,

Itxaka