Thanks for answering Dan. I updated to the CentOS plus kernel on my application server and another client, but unfortunately the problem persists. I opened a bug on the tracker to try to get some more understanding / help, https://tracker.ceph.com/issues/39938 Best regards, Patrik Martinsson On Tue, 2019-05-14 at 20:50 +0200, Dan van der Ster wrote: > Sounds like this append write bug > http://tracker.ceph.com/issues/37713 > > .. Dan > > > On Tue, 14 May 2019, 20:14 Patrik Martinsson, < > patrik.martinsson@xxxxxxxxxxxxx> wrote: > > Hi, > > > > I'm experiencing some really fishy issues with cephfs, or not > > particularly the filesystem it self, rather an combination of > > cephfs > > and the application gitlab. > > > > The issue seems to on "quite low level" hence the devel mailing > > list, > > please point me in another direction if I'm wrong. > > > > Background, > > - ceph version 12.2.8-89.el7cp > > (2f66ab2fa63b2879913db6d6cf314572a83fd1f0) luminous (stable) > > - ceph-fuse-12.2.8-89.el7cp.x86_64 > > - Linux 3.10.0-957.10.1.el7.x86_64 > > > > So the issue I'm seeing when using a cephfs mount on my gitlab > > server > > to store repositories / images / etc. (data that will be shared by > > other gitlab-servers), is that some files (I've only seen this with > > with the images (avatar/images connected to groups) ends up on the > > cephfs-filesystem as 0 bytes (empty files). > > > > One could think that this would be an issue in gitlab, and that I > > should use their mailinglist / support instead. However, I do think > > that there is something more serious to this issue. > > > > Since the file I upload (the avatar) is *not* 0 bytes on the actual > > gitlab server (its an valid png as you would expect) - however, if > > I > > look at the particular file from another server where I have the > > same > > cephfs filesystem mounted, it comes up as empty (0 bytes). Folder > > structure and file is there, but there is no content in it. > > > > So, to summarize, > > > > - I have two servers, mounting a cephfs filesystem, > > XXX,XXX,XXX:/volumes/_nogroup/XXXX /data ceph > > name=user,secretfile=/etc/ceph/ceph.client.user,_netdev,defaults > > > > 0 0 > > > > - I upload the file via an application (gitlab in this case), and > > it > > stores a png under the following location, /data/gitlab/uploads/- > > /system/group/avatar/6/2wey8k.jpg > > > > - If you look at the file from the server where the upload happened > > (the gitlab server). I will see that its valid png file and a > > reasonable size. > > > > - If i look at the file from another server, where I have the same > > cephfs filesystem mounted, the file is empty. > > > > - Now if I unmount the filesystem on the gitlab server (stopping > > the > > application and making sure no one is using the filesystem), and > > mount > > the filesystem again - the file is 0 bytes. > > > > So, somehow, I end up in a situation where the file exists on > > cephfs, > > (as confirmed by looking at the file from another server), but > > empty > > for all clients (I've verified with other servers as well having > > the > > same mount) *except the one server running the application*. The > > data > > for the file is there, until the filesystem is unmounted. > > > > Now, I have not been able to reproduce this behavior with a > > "regular" > > file from the shell (eg. echo "test" > /data/gitlab/uploads/- > > /system/group/avatar/6/foo). The file I just created shows up on > > the > > cephfs as expected, as in all other situations where we use it as > > well. > > > > I'm not familiar with the internals of the particular application - > > but > > even so - it seems like the filesystem is doing something wrong > > here. > > How is it possible that the size of the file is actually *one > > thing* on > > one server, and 0 bytes on all the others ? > > > > Nor I'm a particularly familiar with how the internals of kernel > > filesystem drivers, but running, > > $ > sync; echo 3 > /proc/sys/vm/drop_caches > > doesn't change anything. I've also tried a few mount options to the > > cephfs-mount, but I always end up in this situation. > > > > > > I've also tried the same setup using ceph-fuse - which *do not* > > have > > this problem. Saved image/avatar or whatever, shows up with the > > actual > > content on all the cephfs-clients. So it seems only limited to the > > kernel driver. > > > > > > So, I've gathered some debug logs from the servers in questions by > > $ > echo module ceph +p >/sys/kernel/debug/dynamic_debug/control > > > > Here's the log from when gitlab stores the particular image on > > cephfs, > > https://dpaste.de/aeTd/raw > > > > Here's the log from another server, when statting the particular > > file, > > https://dpaste.de/gdHp/raw (here it is 0 bytes) > > > > Here's the log from when I do (on the gitlab server), > > $ > echo "" > /data/gitlab/uploads/- > > /system/group/avatar/6/2wey8k.jpg > > https://dpaste.de/Va7b/raw > > > > Here's the logs from another server again statting the particular > > file > > (which now is one byte), > > https://dpaste.de/ZXXS/raw > > > > If I instead do, > > $ > echo "" >> /data/gitlab/uploads/- > > /system/group/avatar/6/2wey8k.jpg > > the data actually gets appended, so I will have a png-file + "". > > > > > > I will gladly take any input on this. Maybe I'm just > > misunderstanding > > something trivial here - but it seems off that I see one thing on > > one > > server, and another thing from *all the other clients*. > > > > Best regards, > > Patrik Martinsson > > Sweden > > > > -- Patrik Martinsson - Linux System Administrator Mäster Samuelsgatan 17, 111 44 Stockholm, Sweden E patrik.martinsson@xxxxxxxxxxxxx W www.trioptima.com YT Genuine Happiness ! Please see the disclaimer: www.trioptima.com/email
