Unintuitive (buggy?) CephFS behaviour when dealing with pool_namespace layout attribute

[Florian Haas <florian.haas@xxxxxxxxxx>]

Hello everyone,

I'm seeing some behaviour in CephFS that strikes me as unexpected, and I 
wonder if others have thoughts about it.

Consider this scenario:

* Ceph Reef (18.2.4) deployed with Cephadm running on Ubuntu Jammy, 
CephFS client is running kernel 5.15.0-133-generic.
* CephFS is mounted to /mnt, using a CephX identity that has rwp 
permissions.
* CephFS is using a single data pool named cephfs_data.
* In the CephFS root directory, there is a subdirectory named 
"restricted". That subdirectory currently has a single file, 
"test2.bin", of 8 MiB.

There is no other data in the data pool. I can verify this with the 
following rados command:

rados -n client.cephfs -p cephfs_data ls
10000000003.00000001
10000000003.00000000

Now I run this, from the root of my mounted CephFS:

cd /mnt
setfattr -n ceph.dir.layout.pool_namespace -v restricted /mnt/restricted

I realise this is naughty, because there is currently a file named 
/mnt/restricted/test2.bin, and I'm not supposed to set layout attributes 
on a non-empty directory. However, the command does succeed, and I am 
able to read back the ceph.dir.layout xattr:

getfattr -n ceph.dir.layout restricted
# file: restricted
ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 
pool=cephfs_data pool_namespace=restricted"

If at this stage I try to read RADOS objects in the "restricted" 
namespace, I get an empty list:

rados -p cephfs_data -N restricted ls
(empty output)

Now I move my test file out of the restricted subdirectory and back in:

mv /mnt/restricted/test2.bin /mnt/
sync
mv /mnt/test2.bin /mnt/restricted/
sync

No change:

rados -p cephfs_data -N restricted ls
(empty output)

Next, I try moving the file out, and *copying* it back in:

mv /mnt/restricted/test2.bin /mnt/
sync
cp  /mnt/test2.bin /mnt/restricted/
sync

Now, I do see objects in the "restricted" namespace:

rados -p cephfs_data -N restricted ls
100000001fd.00000001
100000001fd.00000000

Also, if I create a *new* file in the "restricted" subdirectory, then 
its RADOS objects do end up in the correct namespace:

dd if=/dev/urandom of=/mnt/restricted/test3.bin count=3 bs=4M

rados -p cephfs_data -N restricted ls
100000001fe.00000000
100000001fe.00000002
100000001fe.00000001
100000001fd.00000001
100000001fd.00000000

By contrast, here's what happens to a non-empty *file,* when I try to 
set ceph.file.layout:

dd if=/dev/urandom of=/mnt/test4.bin count=4 bs=4M

setfattr -n ceph.file.layout.pool_namespace -v restricted test4.bin
setfattr: test4.bin: Directory not empty

So that fails, somewhat expectedly (although the error message is odd).

In summary, there are two things that confuse me here:

1. Why does setting ceph.dir.layout.pool_namespace on a non-empty 
directory succeed, when setting ceph.file.layout.pool_namespace on a 
non-empty file fails (and even confusingly with a "Directory not empty" 
message)?

2. Considering that setting ceph.dir.layout.pool_namespace on a 
non-empty directory does succeed, why does mv'ing a file to a directory 
with a different pool_namespace, and then mv'ing it back, not result in 
its RADOS objects moving to the other namespace?

So I'm curious: is this a bug (or two), or have I been misunderstanding 
what's actually the expected behaviour?

Cheers,
Florian

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature

_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx