On Wed, 2020-03-25 at 12:14 +0000, Mikael Öhman wrote: > Hi all, > > Using sendfile function to write data to cephfs, the data doesn't end up being written. > From the client that writes the file, it looks correct at first, but from all other ceph clients, the size is 0 bytes. Re-mounting the filesystem, the data is lost. > I didn't see any errors, the data just doesn't get written, as if it's just cached in cephfs client. > Writing just an extra byte at the end of the file (without sendfile), it seems to trigger the actual write of all the data. > > Could someone else confirm if they are also seeing such issue? I'm on ceph 13.2.8, using kernel module for mounting on CentOS7. > > I've used this sendfile-example for the example below: > https://github.com/pijewski/sendfile-example/blob/master/sendfile.c > > Using a small 27 byte source file. > # ls -lh examples/ > -rw-r--r-- 1 root c3-staff 27 Mar 24 18:04 src > # ./sendfile examples/src examples/dst 27 > # ls -lh examples/ > ------x--- 1 root c3-staff 27 Mar 24 18:12 dst > -rw-r--r-- 1 root c3-staff 27 Mar 24 18:04 src > > But, directory is still on 27 bytes: > # ls -lhd examples > drwxr-sr-x 1 root c3-staff 27 Mar 24 18:15 examples > > and on all other cephfs clients, the file is empty: > # ls -lh examples/ > ------x--- 1 root c3-staff 0 Mar 24 18:12 dst > -rw-r--r-- 1 root c3-staff 27 Mar 24 18:04 src > > Is this a bug in cephfs, or should I not expect sendfile to work (as it is not posix compliant). There are no error reported from what i can see, and it is 100% reproducible (sorry for the resend, the original got caught up in moderation as I sent it from wrong address) This sounds like a kernel client bug in an old Centos7 kernel. The program seems to work as expected on current mainline kernels, and on 3.10.0-1062.1.2.el7.x86_64 (the latest one I had on my client). What kernel version are you using on the client? If you're not on the latest version Centos7 kernel version, then it'd be good to try that and see if it's still reproducible. For the record: [jlayton@centos7 ~]$ sudo umount -f /mnt/cephfs ; sudo mount /mnt/cephfs ; ./sendfile ./testfile /mnt/cephfs/testfile 27 ; sudo umount /mnt/cephfs ; sudo mount /mnt/cephfs ; ls -l /mnt/cephfs Sent 0 KiB over sendfile(3EXT) of 0 KiB requested total 1 drwxr-xr-x 1 root root 1 Mar 25 08:53 foo drwxrwxrwx 1 root root 0 Mar 25 08:37 scratch drwxr-xr-x 1 root root 57 Mar 25 08:44 test -rw-r--r-- 1 jlayton jlayton 27 Mar 25 12:35 testfile [jlayton@centos7 ~]$ uname -r 3.10.0-1062.1.2.el7.x86_64 Attached is the cleaned-up version of sendfile.c that I was using. -- Jeff Layton <jlayton@xxxxxxxxxx> _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |