Re: portable profile broken file bind mount (/etc/resolv.conf)

[Peter Morrow <pemorrow@xxxxxxxxxxxxxxxxxxx>]

On Wed, Sep 23, 2020 at 03:14:45AM -0700, Peter Morrow wrote:
> Hi,
> 
> I ran into an issue (v239 custom yocto based distro, though the code is the same with latest releases) where a portable service ends up with a broken file
> bind mount since the file is deleted and recreated on the host. This behaviour
> is expected for a file based bind mount, the issue is that the default
> portable profile makes use of this pattern when it might be better to bind
> mount the parent directory. This would allow changes to be reflected from the
> host to the portable service.
> 
> Taking a look at the default portable profile
> src/portable/profile/default/service.conf we see:
> 
> BindReadOnlyPaths=/etc/resolv.conf
> 
> /etc/resolv.conf is symlink to /etc/resolv-conf.systemd which is a
> symlink to /run/systemd/resolve/resolv.conf.
> 
> The issue comes via src/resolve/resolved.c:
> 
>         /* Write finish default resolv.conf to avoid a dangling symlink */
>         (void) manager_write_resolv_conf(m);
> 
> The above writes out /run/systemd/resolve/resolv.conf, though obviously
> any time manager_write_resolv_conf() is called then
> /run/systemd/resolve/resolv.conf is deleted since the file update does
> not happen in place:
> 
> manager_write_resolv_conf():
> 
> if (rename(temp_path_uplink, PRIVATE_UPLINK_RESOLV_CONF) < 0)
>                 r = log_error_errno(errno, "Failed to move new %s into
> place: %m", PRIVATE_UPLINK_RESOLV_CONF);
> 
> 
> This means if the DNS servers are updated after the portable service is
> started and the bind mount has completed then we still see the old
> version of /run/systemd/resolve/resolv.conf.
> 
> I was thinking that it would be better if the default portable profile
> instead had this line in it:
> 
> BindReadOnlyPaths=/run/systemd/resolve/
> 
> So that if /run/systemd/resolve/resolv.conf is deleted and recreated
> then the portable service will see the new version of the file.  This
> only works since the same heirarchy of symlinks exists in the portable
> service image. Is this an OK solution or is it fragile or something
> else? I am happy to send a PR to change this if it seems like a workable
> solution.

I spent some time recreating this on v239 and v246:

root@qemux86-64:/run/systemd/resolve# systemctl --version
systemd 246 (246.2+)
-PAM -AUDIT -SELINUX +IMA -APPARMOR -SMACK +SYSVINIT +UTMP
-LIBCRYPTSETUP -GCRYPT -GNUTLS +ACL +d
root@qemux86-64:/run/systemd/resolve# 

My findings are as follows:

1) This issue is limited to portable services, I don't see this
happening with a normal file bind mount with a "normal" service.

2) I can create the issue on v239 with a test portable service we use,
the recreate for v246 is based on the nifty example here:

http://0pointer.net/blog/walkthrough-for-portable-services.html

root@qemux86-64:~# systemctl status walkthroughd.service --no-pager
* walkthroughd.service - A simple example service
     Loaded: loaded (/etc/systemd/system.attached/walkthroughd.service;
enabled; vendor preset: )
    Drop-In: /etc/systemd/system.attached/walkthroughd.service.d
             `-10-profile.conf, 20-portable.conf
     Active: active (running) since Thu 2020-09-24 13:32:55 UTC; 3min 9s
ago
   Main PID: 251 (walkthroughd)
      Tasks: 1 (limit: 271)
     Memory: 3.2M
     CGroup: /system.slice/walkthroughd.service
             `-251 /usr/local/lib/walkthroughd/walkthroughd

Sep 24 13:32:55 qemux86-64 systemd[1]: Started A simple example service.
Sep 24 13:32:56 qemux86-64 walkthroughd[251]: Initializing.
root@qemux86-64:~# nsenter -a -t 251
# findmnt | grep resolv
|-/etc/resolv.conf
tmpfs[/systemd/resolve/resolv.conf]                       5
#

So this all looks good. To simulate what happens in systemd is
/run/systemd/resolve/resolv.conf is to updated C.F.:

manager_write_resolv_conf():

if (rename(temp_path_uplink, PRIVATE_UPLINK_RESOLV_CONF) < 0)
                r = log_error_errno(errno, "Failed to move new %s into
place: %m", PRIVATE_UPLINK_RESOLV_CONF);

I remove /run/systemd/resolve/resolv.conf and create a new file with
the same name but with some other contents.

root@qemux86-64:~# cd /run/systemd/resolve/
root@qemux86-64:/run/systemd/resolve# cp resolv.conf /tmp/
root@qemux86-64:/run/systemd/resolve# rm resolv.conf
root@qemux86-64:/run/systemd/resolve# echo "Updated file contents" >
resolv.conf
root@qemux86-64:/run/systemd/resolve# nsenter -a -t 251
# findmnt | grep resolv
|-/etc/resolv.conf
tmpfs[/systemd/resolve/resolv.conf//deleted]              5
#

The mount is marked as deleted and we still see the only file contents:

# cat /etc/resolv.conf
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients
# directly to
# all known uplink DNS servers. This file lists all configured search
# domains.
#
# Third party programs should typically not access this file directly,
# but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5)
# in a
# different way, replace this symlink by a static file or a different
# symlink.
#
# See man:systemd-resolved.service(8) for details about the supported
# modes of
# operation for /etc/resolv.conf.

nameserver 1.1.1.1
nameserver 8.8.8.8
nameserver 1.0.0.1
# Too many DNS servers configured, the following entries may be ignored.
nameserver 8.8.4.4
nameserver 2606:4700:4700::1111
nameserver 2001:4860:4860::8888
nameserver 2606:4700:4700::1001
nameserver 2001:4860:4860::8844
#

My original idea of bind mounting /run/systemd/resolve to the container
does not work with the walkthrough service since /etc/resolv.conf is not
set up the same way it is on the host. Thus this seems like it isn't a
workable solution (however it does work for our use case since our
container resolv.conf linkage is the same as on the host).

Maybe I'll take this over to github with a new issue...

Thanks,
Peter.

> 
> Thanks!
> Peter.
> _______________________________________________
> systemd-devel mailing list
> systemd-devel@xxxxxxxxxxxxxxxxxxxxx
> https://lists.freedesktop.org/mailman/listinfo/systemd-devel
_______________________________________________
systemd-devel mailing list
systemd-devel@xxxxxxxxxxxxxxxxxxxxx
https://lists.freedesktop.org/mailman/listinfo/systemd-devel