Hi, I'm experiencing some really fishy issues with cephfs, or not particularly the filesystem it self, rather an combination of cephfs and the application gitlab. The issue seems to on "quite low level" hence the devel mailing list, please point me in another direction if I'm wrong. Background, - ceph version 12.2.8-89.el7cp (2f66ab2fa63b2879913db6d6cf314572a83fd1f0) luminous (stable) - ceph-fuse-12.2.8-89.el7cp.x86_64 - Linux 3.10.0-957.10.1.el7.x86_64 So the issue I'm seeing when using a cephfs mount on my gitlab server to store repositories / images / etc. (data that will be shared by other gitlab-servers), is that some files (I've only seen this with with the images (avatar/images connected to groups) ends up on the cephfs-filesystem as 0 bytes (empty files). One could think that this would be an issue in gitlab, and that I should use their mailinglist / support instead. However, I do think that there is something more serious to this issue. Since the file I upload (the avatar) is *not* 0 bytes on the actual gitlab server (its an valid png as you would expect) - however, if I look at the particular file from another server where I have the same cephfs filesystem mounted, it comes up as empty (0 bytes). Folder structure and file is there, but there is no content in it. So, to summarize, - I have two servers, mounting a cephfs filesystem, XXX,XXX,XXX:/volumes/_nogroup/XXXX /data ceph name=user,secretfile=/etc/ceph/ceph.client.user,_netdev,defaults 0 0 - I upload the file via an application (gitlab in this case), and it stores a png under the following location, /data/gitlab/uploads/- /system/group/avatar/6/2wey8k.jpg - If you look at the file from the server where the upload happened (the gitlab server). I will see that its valid png file and a reasonable size. - If i look at the file from another server, where I have the same cephfs filesystem mounted, the file is empty. - Now if I unmount the filesystem on the gitlab server (stopping the application and making sure no one is using the filesystem), and mount the filesystem again - the file is 0 bytes. So, somehow, I end up in a situation where the file exists on cephfs, (as confirmed by looking at the file from another server), but empty for all clients (I've verified with other servers as well having the same mount) *except the one server running the application*. The data for the file is there, until the filesystem is unmounted. Now, I have not been able to reproduce this behavior with a "regular" file from the shell (eg. echo "test" > /data/gitlab/uploads/- /system/group/avatar/6/foo). The file I just created shows up on the cephfs as expected, as in all other situations where we use it as well. I'm not familiar with the internals of the particular application - but even so - it seems like the filesystem is doing something wrong here. How is it possible that the size of the file is actually *one thing* on one server, and 0 bytes on all the others ? Nor I'm a particularly familiar with how the internals of kernel filesystem drivers, but running, $ > sync; echo 3 > /proc/sys/vm/drop_caches doesn't change anything. I've also tried a few mount options to the cephfs-mount, but I always end up in this situation. I've also tried the same setup using ceph-fuse - which *do not* have this problem. Saved image/avatar or whatever, shows up with the actual content on all the cephfs-clients. So it seems only limited to the kernel driver. So, I've gathered some debug logs from the servers in questions by $ > echo module ceph +p >/sys/kernel/debug/dynamic_debug/control Here's the log from when gitlab stores the particular image on cephfs, https://dpaste.de/aeTd/raw Here's the log from another server, when statting the particular file, https://dpaste.de/gdHp/raw (here it is 0 bytes) Here's the log from when I do (on the gitlab server), $ > echo "" > /data/gitlab/uploads/-/system/group/avatar/6/2wey8k.jpg https://dpaste.de/Va7b/raw Here's the logs from another server again statting the particular file (which now is one byte), https://dpaste.de/ZXXS/raw If I instead do, $ > echo "" >> /data/gitlab/uploads/-/system/group/avatar/6/2wey8k.jpg the data actually gets appended, so I will have a png-file + "". I will gladly take any input on this. Maybe I'm just misunderstanding something trivial here - but it seems off that I see one thing on one server, and another thing from *all the other clients*. Best regards, Patrik Martinsson Sweden
