On Thu, Jan 21, 2016 at 11:52 PM, Yan, Zheng <ukernel@xxxxxxxxx> wrote: > On Fri, Jan 22, 2016 at 6:24 AM, Gregory Farnum <gfarnum@xxxxxxxxxx> wrote: >> On Fri, Jan 15, 2016 at 9:00 AM, HMLTH <hmlth@xxxxxxxxxx> wrote: >>> Hello, >>> >>> I'm evaluating cephfs on a virtual machines cluster. I'm using Infernalis >>> (9.2.0) on debian Jessie as client and server. >>> >>> I'm trying to get some performance numbers on operations like tar/untar on >>> things like the linux kernel. I have an issue where tar displays this >>> warning : 'file changed as we read it'. I reproduced the problem with just >>> the Documentation dir. >>> >>> root@admin:/mnt/cephfs# rm -rf Documentation ; tar xf >>> linux-documentation.tar.bz2 ; sync ; tar c Documentation>/dev/null ; >>> tar: Documentation/parisc: file changed as we read it >>> tar: Documentation/pcmcia: file changed as we read it >>> tar: Documentation/phy: file changed as we read it >>> tar: Documentation/platform: file changed as we read it >>> tar: Documentation/power: file changed as we read it >>> tar: Documentation/powerpc: file changed as we read it >>> tar: Documentation/pps: file changed as we read it >>> tar: Documentation/prctl: file changed as we read it >>> tar: Documentation/pti: file changed as we read it >>> tar: Documentation/ptp: file changed as we read it >>> tar: Documentation/rapidio: file changed as we read it >>> tar: Documentation/s390: file changed as we read it >>> tar: Documentation/scheduler: file changed as we read it >>> tar: Documentation/scsi: file changed as we read it >>> tar: Documentation/security: file changed as we read it >>> tar: Documentation/w1/slaves: file changed as we read it >>> tar: Documentation/watchdog: file changed as we read it >>> tar: Documentation/x86: file changed as we read it >>> tar: Documentation/zh_CN: file changed as we read it >>> tar: Documentation: file changed as we read it >>> >>> If I wait between the two commands, the error is reduced but not >>> eliminated: >>> >>> root@admin:/mnt/cephfs# rm -rf Documenation ; tar xf >>> linux-documentation.tar.bz2 ; sleep 10 ; tar c Documentation>/dev/null ; >>> tar: Documentation/virtual: file changed as we read it >>> tar: Documentation/w1: file changed as we read it >>> tar: Documentation/watchdog: file changed as we read it >>> tar: Documentation/x86: file changed as we read it >>> tar: Documentation/zh_CN: file changed as we read it >>> tar: Documentation: file changed as we read it >>> >>> >>> root@admin:/mnt/cephfs# rm -rf Documentation ; tar xf >>> linux-documentation.tar.bz2 ; sleep 120 ; tar c Documentation>/dev/null ; >>> tar: Documentation: file changed as we read it >>> >>> I'm sure no other client process is not modifying the files. I have this >>> problem with the fuse client and with the kernel (version in Jessie) client. >>> >>> By doing a "stat", I see some meta-data are changed: >>> root@admin:/mnt/cephfs# rm -rf Documentation ; tar xf >>> linux-documentation.tar.bz2 ;stat Documentation; tar -c >>> Documentation>/dev/null ; stat Documentation >>> File: ‘Documentation’ >>> Size: 14740322 Blocks: 1 IO Block: 4096 directory >>> Device: 23h/35d Inode: 1099511913288 Links: 1 >>> Access: (0770/drwxrwx---) Uid: ( 1000/ XXXX) Gid: ( 1000/ XXXX) >>> Access: 2016-01-15 16:51:40.882143334 +0000 >>> Modify: 2015-05-12 09:34:49.000000000 +0100 >>> Change: 2016-01-15 16:52:31.745684502 +0000 >>> Birth: - >>> tar: Documentation/scheduler: file changed as we read it >>> tar: Documentation/scsi: file changed as we read it >>> tar: Documentation/zh_CN/arm64: file changed as we read it >>> tar: Documentation/zh_CN/filesystems: file changed as we read it >>> tar: Documentation/zh_CN/video4linux: file changed as we read it >>> tar: Documentation: file changed as we read it >>> File: ‘Documentation’ >>> Size: 15088573 Blocks: 1 IO Block: 4096 directory >>> Device: 23h/35d Inode: 1099511913288 Links: 1 >>> Access: (0770/drwxrwx---) Uid: ( 1000/ XXXX) Gid: ( 1000/ XXXX) >>> Access: 2016-01-15 16:51:40.882143334 +0000 >>> Modify: 2015-05-12 09:34:49.000000000 +0100 >>> Change: 2016-01-15 16:52:31.745684502 +0000 >>> Birth: - >>> >>> I know it's possible to silence this warning with a tar option but, I don't >>> want to worry about that in every commands of every script, it changes the >>> tar output. And above all, I don't find that very clean. >>> >>> Do you know any settings that guarantees that all pending async writes are >>> terminated when a client opens a file ? >> >> We do stuff with tar in several of our nightly and acceptance tests so >> I'm surprised to see these issues. I take it you're using ceph-fuse to >> mount your clients? >> Can you test this again, but after you've done the extraction and >> before you've started tarring it back up, can you check (and share) >> the output of the following admin socket commands: >> mds_requests >> objecter_requests >> >> And if those don't reveal anything obvious, you can call the >> dump_cache command which will dump out the state of every inode in the >> cache. >> -Greg > > > For ceph-fuse, directory's size can change between successive stat(1). > This is because size of directory is recursive size, recursive size is > not updated instantly. It turns out we don't actually generate tarballs in our tests, and the rsync tests we run have a long interval in between checks that allows the lazy statistic changes to propagate. We'll need to figure out if there's something we can do for this scenario other than letting the user disable rstats. :/ > For kernel client, both directory's size and ctime can change. the > size change can be avoided by 'norbytes' mount option. the ctime > change is duo to a cap flush bug. Do you have a ticket for that ctime change bug? -Greg _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |