Hello, I hope this is the write mailing list. I was debugging a container problem after upgrading kubernetes nodes on my cloud provider, and turns out it's an overlay fs related issue and has nothing to do with the container technology. I made a repro script and it consistently reproduce the issue. Thing breaks in the newer kernel version (5.10.134-13.1.al8.x86_64), and works in the older version (5.10.84-10.2.al8.x86_64). Test cases and the situation is explained in the script. --- #### Synopsis ## Things work as expected in 5.10.84-10.2.al8.x86_64 ## Things break as expected in 5.10.134-13.1.al8.x86_64 ## Note: bin, rpc is just some non-privilege user/group #### Preparation #### # Cleanup Folder umount /root/test/mount && cd / && rm -fr /root/test # Create empty folders mkdir -p /root/test && cd /root/test; mkdir -p lower/target_bad1 lower/target_bad2 lower/target_good1 lower/target_good2 upper work mount; # Prepare testing conditions in lower chmod 1730 lower/target_bad1; chown root:bin lower/target_bad1; chmod 0737 lower/target_bad2; chown root:bin lower/target_bad2; chmod 0777 lower/target_good1; chown root:bin lower/target_good1; chmod 0707 lower/target_good2; chown root:rpc lower/target_good2; # Create Overlay FS mount mount -t overlay -o lowerdir=/root/test/lower,upperdir=/root/test/upper,workdir=/root/test/work null /root/test/mount; #### Test Case #### # Expectation: all 4 case success. files are touched and created. # In 5.10.134-13.1.al8.x86_64 however, the bad1, bad2 fails with `permisison denied`. good1, good2 behaves as expected. # Case 1: group permission bits are -wx, no r. Not working in 5.10.134-13.1.al8.x86_64 # This is the original problem I am working on, command crontab fail to write to /var/spool/cron/crontabs which has permission 1730. sudo -u bin -g bin touch mount/target_bad1/RANDOM # Case 2: I simplified the case a bit. # group permission bits is -wx, no r. Not working in 5.10.134-13.1.al8.x86_64 sudo -u bin -g bin touch mount/target_bad2/RANDOM # Case 3: group permission bits are rwx. It works in both sudo -u bin -g bin touch mount/target_good1/RANDOM # Case 4: directory ownership is rpc, I am sudo'ing to bin. It works in both sudo -u bin -g bin touch mount/target_good2/RANDOM --- I have not yet bisect or upgraded to the latest kernel version, that's something I might be doing next, but since I am on Alicloud, upgrading the kernel is not the easiest thing I could do. I did some searches here and there for similar problems but to no avail. I am also reaching out to Alicloud support. I hope if this is a known problem and if someone could point me to a related bug / issue, that would be a very great help and I could use that argument to push them to upgrade their kernel. Thank you very much Sam Wong
