On Fri, Jan 22, 2016 at 6:24 AM, Gregory Farnum <gfarnum@xxxxxxxxxx> wrote: > On Fri, Jan 15, 2016 at 9:00 AM, HMLTH <hmlth@xxxxxxxxxx> wrote: >> Hello, >> >> I'm evaluating cephfs on a virtual machines cluster. I'm using Infernalis >> (9.2.0) on debian Jessie as client and server. >> >> I'm trying to get some performance numbers on operations like tar/untar on >> things like the linux kernel. I have an issue where tar displays this >> warning : 'file changed as we read it'. I reproduced the problem with just >> the Documentation dir. >> >> root@admin:/mnt/cephfs# rm -rf Documentation ; tar xf >> linux-documentation.tar.bz2 ; sync ; tar c Documentation>/dev/null ; >> tar: Documentation/parisc: file changed as we read it >> tar: Documentation/pcmcia: file changed as we read it >> tar: Documentation/phy: file changed as we read it >> tar: Documentation/platform: file changed as we read it >> tar: Documentation/power: file changed as we read it >> tar: Documentation/powerpc: file changed as we read it >> tar: Documentation/pps: file changed as we read it >> tar: Documentation/prctl: file changed as we read it >> tar: Documentation/pti: file changed as we read it >> tar: Documentation/ptp: file changed as we read it >> tar: Documentation/rapidio: file changed as we read it >> tar: Documentation/s390: file changed as we read it >> tar: Documentation/scheduler: file changed as we read it >> tar: Documentation/scsi: file changed as we read it >> tar: Documentation/security: file changed as we read it >> tar: Documentation/w1/slaves: file changed as we read it >> tar: Documentation/watchdog: file changed as we read it >> tar: Documentation/x86: file changed as we read it >> tar: Documentation/zh_CN: file changed as we read it >> tar: Documentation: file changed as we read it >> >> If I wait between the two commands, the error is reduced but not >> eliminated: >> >> root@admin:/mnt/cephfs# rm -rf Documenation ; tar xf >> linux-documentation.tar.bz2 ; sleep 10 ; tar c Documentation>/dev/null ; >> tar: Documentation/virtual: file changed as we read it >> tar: Documentation/w1: file changed as we read it >> tar: Documentation/watchdog: file changed as we read it >> tar: Documentation/x86: file changed as we read it >> tar: Documentation/zh_CN: file changed as we read it >> tar: Documentation: file changed as we read it >> >> >> root@admin:/mnt/cephfs# rm -rf Documentation ; tar xf >> linux-documentation.tar.bz2 ; sleep 120 ; tar c Documentation>/dev/null ; >> tar: Documentation: file changed as we read it >> >> I'm sure no other client process is not modifying the files. I have this >> problem with the fuse client and with the kernel (version in Jessie) client. >> >> By doing a "stat", I see some meta-data are changed: >> root@admin:/mnt/cephfs# rm -rf Documentation ; tar xf >> linux-documentation.tar.bz2 ;stat Documentation; tar -c >> Documentation>/dev/null ; stat Documentation >> File: ‘Documentation’ >> Size: 14740322 Blocks: 1 IO Block: 4096 directory >> Device: 23h/35d Inode: 1099511913288 Links: 1 >> Access: (0770/drwxrwx---) Uid: ( 1000/ XXXX) Gid: ( 1000/ XXXX) >> Access: 2016-01-15 16:51:40.882143334 +0000 >> Modify: 2015-05-12 09:34:49.000000000 +0100 >> Change: 2016-01-15 16:52:31.745684502 +0000 >> Birth: - >> tar: Documentation/scheduler: file changed as we read it >> tar: Documentation/scsi: file changed as we read it >> tar: Documentation/zh_CN/arm64: file changed as we read it >> tar: Documentation/zh_CN/filesystems: file changed as we read it >> tar: Documentation/zh_CN/video4linux: file changed as we read it >> tar: Documentation: file changed as we read it >> File: ‘Documentation’ >> Size: 15088573 Blocks: 1 IO Block: 4096 directory >> Device: 23h/35d Inode: 1099511913288 Links: 1 >> Access: (0770/drwxrwx---) Uid: ( 1000/ XXXX) Gid: ( 1000/ XXXX) >> Access: 2016-01-15 16:51:40.882143334 +0000 >> Modify: 2015-05-12 09:34:49.000000000 +0100 >> Change: 2016-01-15 16:52:31.745684502 +0000 >> Birth: - >> >> I know it's possible to silence this warning with a tar option but, I don't >> want to worry about that in every commands of every script, it changes the >> tar output. And above all, I don't find that very clean. >> >> Do you know any settings that guarantees that all pending async writes are >> terminated when a client opens a file ? > > We do stuff with tar in several of our nightly and acceptance tests so > I'm surprised to see these issues. I take it you're using ceph-fuse to > mount your clients? > Can you test this again, but after you've done the extraction and > before you've started tarring it back up, can you check (and share) > the output of the following admin socket commands: > mds_requests > objecter_requests > > And if those don't reveal anything obvious, you can call the > dump_cache command which will dump out the state of every inode in the > cache. > -Greg For ceph-fuse, directory's size can change between successive stat(1). This is because size of directory is recursive size, recursive size is not updated instantly. For kernel client, both directory's size and ctime can change. the size change can be avoided by 'norbytes' mount option. the ctime change is duo to a cap flush bug. > > >> >> Regards >> >> Thomas HAMEL >> >> _______________________________________________ >> ceph-users mailing list >> ceph-users@xxxxxxxxxxxxxx >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |