On Tue, Dec 15, 2020 at 9:29 PM Michael Labriola <michael.d.labriola@xxxxxxxxx> wrote: > > On Tue, Dec 15, 2020 at 11:31 AM Amir Goldstein <amir73il@xxxxxxxxx> wrote: > > > > On Tue, Dec 15, 2020 at 5:33 PM Michael D Labriola > > <michael.d.labriola@xxxxxxxxx> wrote: > > > > > > On Mon, Dec 14, 2020 at 6:06 PM Michael D Labriola <michael.d.labriola@xxxxxxxxx> wrote: > > > > > > > > I'm sporatically getting "no data available" as a reason to fail to > > > > open files on an overlay mount. Most obvious is during ln of backup > > > > file during apt install. Only seems to happen on copy_up from lower > > > > How do you know that? Do you have some more tracing info? > > I haven't done any tracing, perhaps I overstated. The problem I'm > seeing only happens when overlayfs goes to create a copy of a lower > layer file in the upper layer. When the problem occurs, it's always > on a file that exists on lower but not upper, and is about to be > modified. > > > > > > > layer. Lower layer is squashfs (I've seen it happen with both the > > > > default zlib and also zstd compression), upper is EXT4. > > > > > > > > I've only bumped into this problem recently with 5.9+ kernels. I'm > > > > gonna go see if I can reproduce in some older kernels I still have > > > > installed. > > > > > > Rebooting into 5.4 made the problem go away and I can apt upgrade > > > w/out any problems. Rebooting an affected virtual machine into 5.8 > > > also fixed the problem, so it looks to be something introduced in 5.9. > > > > There are no overlayfs changes v5.8..v5.9 nor squashfs changes. > > Are you sure that your reproducer is reliable enough for the bisection? > > If it is, please try to bisect the offending commit because I have no idea > > where it may be. > > I'm having a hard time reproducing the problem. It's only happening > frequently enough for me to be pretty sure it's a bug. I've been > using an overlay of squashfs/EXT4 on my development laptop for over a > year, using the squashfs image to fork off disposable virtual machines > for testing. It's worked flawlessly up until I started testing w/ > 5.9... but I couldn't correlate my problems to anything specific until > just recently. > > More than once now, my host system or a virtual machine has randomly > failed to process an apt update. Either a backup hardlink creation > fails or some other processing command fails, always with an error > message of "No data available", which makes no sense to me. Booting > back into my 5.4 or 5.8 kernel and performing the upgrade, then back > into my 5.9 kernel alleviates the problem until it happens again on > some other package. > > I have also seen "No data available" pop up in seemingly random > places. For example, yesterday postfix refused to send mail, and when > I went to restart the service I got this: > > postfix/bounce[24836]: fatal: open lock file pid/unix.bounce: cannot > open file: No data available > > Today, in 5.9.14, I did an apt upgrade which didn't fail creating > backup files, but instead failed doing a postinstall task like this: > > Setting up sudo (1.8.21p2-3ubuntu1.3) ... > chown: changing ownership of '/etc/sudoers': No data available > dpkg: error processing package sudo (--configure): > installed sudo package post-installation script subprocess returned > error exit status 1 > > Rebooting the vm resulted in the same problem. Booting into 5.8.18, > apt upgrade succeeded. Then I rebooted back into 5.9. > > > > > > > > > I suppose I should try 5.10 and see if this problem has already been > > > fixed. > > > > > > > Wouldn't hurt. > > Trying that shortly. Also trying to figure out how to force the > problem to happen... I'll never get to the bottom of it at this rate. > I was really hoping somebody on the list would recognize the > problem... :-/ Just my luck. The problem rings a bell, but I would expect it had something to do with the change: a888db310195 ovl: fix regression with re-formatted lower squashfs >From v5.8.0 that was also applied to stable v5.4.y, so there is no match to your report. 'No data available' means ENODATA error from getxattr() which is not expected to be returned from operations like chmod() and link() as you reported that is why the message makes no sense. I did not find any internal vfs_getxattr() call in overlayfs code where ENODATA can be exposed to the caller. I did find that ENODATA could be exposed from lookup via a call to ovl_verify_origin(), but that implies that the index feature is enabled and is expected to print "overlayfs: failed to verify..." to kmsg. We should probably replace the use visible lookup error with EIO, but that in itself won't help users to understand the problem. Do you use an existing upper (ext4) dir with a lower squashfs image that is not the original image used when first mounting overlay with that upper dir AND enable the index=on feature? I still don't see how that would be a regression of kernel v5.8 and certainly not v5.9. Do you use any non default overlay config/mount options? Please share the output of '/proc/self/mountinfo' with mounted overlay and 'grep Y /sys/module/overlay/parameters/*' Do you see any "overlayfs:" logs in kmsg? If you do not need nfs export, you could try to create squashfs image with -no-exports as a possible workaround. Thanks, Amir.
