Hi Vivek, comments inline below Regards Mauro On 9/2/20 10:20 PM, Vivek Goyal wrote: > On Wed, Sep 02, 2020 at 05:33:53PM +0200, Mauro Condarelli wrote: >> Hi Vivek, >> comments inline below. >> >> On 9/2/20 3:29 PM, Vivek Goyal wrote: >>> On Wed, Sep 02, 2020 at 06:34:12AM +0300, Amir Goldstein wrote: >>>> On Wed, Sep 2, 2020 at 2:37 AM Mauro Condarelli <mc5686@xxxxxxxxx> wrote: >>>>> Thanks Amir, >>>>> comments inline below >>>>> >>>>> Regards >>>>> Mauro >>>>> >>>>> On 9/1/20 8:43 PM, Amir Goldstein wrote: >>>>> >>>>> On Tue, Sep 1, 2020 at 9:01 PM Mauro Condarelli <mc5686@xxxxxxxxx> wrote: >>>>> >>>>> Hi, >>>>> most likely this is not the right place to ask, please redirect me as needed. >>>>> >>>>> I'm trying to use OverlayFS to add (limited) write capability to a ReadOnly >>>>> rootfs (SquashFS) >>>>> >>>>> Essentially (actual script is more complex, of course) boot-sequence includes: >>>>> >>>>> # /dev/mmcblk0p5: ext4 (upper+work+nwwroot+newroot/oldroot) >>>>> # /dev/mmcblk0p6: SquashFS mounted on / >>>>> mount /dev/mmcblk0p5 /overlay >>>>> mount -t overlay overlay -o lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work /overlay/newroot >>>>> cd /overlay/newroot >>>>> pivot_root . oldroot >>>>> mount --move oldroot/dev /dev >>>>> mount --move oldroot/proc /proc >>>>> >>>>> This works as expected, but, too often for comfort, some file >>>>> (and sometime also directories) become unavailable due to error: >>>>> >>>>> overlayfs: invalid origin (ssh/sshd_config, ftype=8000, origin ftype=4000). >>>>> >>>>> File name changes, of course, but rest is fairly constant. >>>>> >>>>> This always happens when some file is written. >>>>> Error persists reboots. >>>>> Only way I found to "cure" the system is to go on "upper" and delete the file >>>>> thus going back to "lower" version (in this case I should delete "/oldroot/overlay/upper/etc/ssh/sshd_config") >>>>> >>>>> This is a self-built kernel (Linux vocore 5.7.0 #2 PREEMPT Mon Aug 3 09:19:06 CEST 2020 mips GNU/Linux) >>>>> on a custom target based on a SoC (MT7628). >>>>> >>>>> I am available to do any required test, but I have no idea about where to start. >>>>> >>>>> Any hint (or redirect) would be greatly appreciated. >>>>> >>>>> This is probably your problem: >>>>> https://lore.kernel.org/linux-unionfs/32532923.JtPX5UtSzP@fgdesktop/ >>>>> >>>>> It surely looks like it is. >>>>> I tried to follow the thread, but I'm unsure i grokked it . >>>>> >>>>> I am using OverlayFS (ext4 over a SquashFS) on rootfs in order to >>>>> be able to update the whole system (changing SquashFS) while >>>>> retaining customization (essentially tweaks in /etc/, host certificates >>>>> and similar stuff) in an embedded target. >>>>> >>>>> To complicate matters I also have a dual system for fallback in case >>>>> of faulty upgrade; this means I can switch from: >>>>> lower=part6, upper=part5, update will go into part7 >>>>> to >>>>> lower=part7, upper=part5, update will go into part6 >>>>> >>>>> I don't need nfs at all, so exporting OverlayFS is not an issue, >>>>> but it's unclear to me if this "lower swapping" is >>>>> actually supported as I see: >>>>> >>>>>>> /me is again wondering what's the use case of modifying lower layer >>>>>>> with an existing upper. Is it fair to say, no don't recreate/modify >>>>>>> lower layers and use with existing upper. >>>>>>> >>>>>> It's fine by me to document that this is not supported. >>>>> ... which is scary. >>>>> Note I do *not* need to modify lower on-the-fly, when I >>>>> swap systems, that will happen at reboot. >>>> Don't be scared :-) >>>> Your reaction is a good answer to Vivek's question above - >>>> No, it is not fair to say don't re-create lower and use existing upper >>> This seems like a wrong use case to me. So in above example, say >>> following happens. >>> >>> - Mount overlay >>> - modify sshd_config (it gets copied up). >>> - Lower squahsfs gets updated and /etc/sshd/sshd_config gets updated. >>> - Mount overlay and user now sees old copied up sshd_config. (If it >>> works). >>> >>> Conceptually it is not making much sense to me. What's the point of >>> upgrading lower because after mounting overlay you might still see >>> old file. And to make it worse, behavior is underministic beacuse >>> if file has not been copied up, you will see new file. >> It might not make much sense to you, but it's EXACTLY what I need. >> This OverlayFS is used *only* for permanent configuration. >> Most of the times I'm adding new files (like host certificates and >> similar things), but sometimes applications provide "default >> configuration" I might need to override. >> If/when applications "upgrade their defaults" I definitely do not >> want them to override my choices. > Some packages solved that problem by keeping system default > separate from user default files and over upgrade user files are > left untouched. > > So basically squashfs image will be udpated. I am not sure how this > will work for all the cases. What if application is updated and > it decides to rename its config file from foo.txt to bar.txt. Now > when application is launched, user defaults are lost anyway. I'm in a somewhat middle position. Targets are embedded systems and it wouldn't make any sense to implement a fully flexible update system (deb/rpm/ipkg/...); I just have two "packages": system and application. I *am* the "rootfs (aka: system) image provider". OTOH I'm not responsible for the zillions packages composing a RootFS. A vast majority of packages provides "default configurations", often completely commented-out and heavily commented "for reference". I don't think it would be a good idea to inspect each and all the packages to remove such "useless" files. In some instances such configuration I need to override "globally" (i.e.: for all targets in the same manner) and that's easy because I can override in generated RootFS (lower SquashFS). In other instances changes need to be done on a per-target basis (note: I have ~20k targets to handle in places I cannot physically reach after installation... I mean: i *can* reach, but it would be expensive, you don't want to do it if net is still up). Here I have two cases: There are "once in the life" configurations that could conceivably be done at a central point (e.g.: client certificates), but that would mean to send different update images to each target, a logistic nightmare. There are other configs that need to be done at installation and possibly changed later (e.g.: WiFI setup). In both cases configurations *must* survive a full upgrade. > IOW, what is rootfs image provider expecting. Are they expecting > overlayfs to be there and are they writing applications in backward > compatible manner. I don't generally boot into such systems so > I have no idea. Bottom line: I *need* to be able to change "lower" keeping "upper", otherwise OverlayFS is simply not the right tool for me. Alternative would be to save all configs somewhere and restore them after an update, which is highly "inconvenient" for many reasons, especially in system that may (and will!!) need to be updated in unforeseeable ways. Please do *not* enforce cleanup of "upper" on "lower" changes. > Thanks > Vivek Regards Mauro
