|
>
> On Mon, Oct 22, 2018 at 9:46 AM Dylan McCulloch <dmc@xxxxxxxxxxxxxx> wrote: > > > > On Mon, Oct 8, 2018 at 2:57 PM Dylan McCulloch <dmc@xxxxxxxxxxxxxx>> wrote: > > >> > > >> Hi all, > > >> > > >> > > >> We have identified some unexpected blocking behaviour by the ceph-fs kernel client. > > >> > > >> > > >> When performing 'rm' on large files (100+GB), there appears to be a significant delay of 10 seconds or more, before a 'stat' operation can be performed on the same directory on the filesystem. > > >> > > >> > > >> Looking at the kernel client's mds inflight-ops, we observe that there are pending > > >> > > >> UNLINK operations corresponding to the deleted files. > > >> > > >> > > >> We have noted some correlation between files being in the client page cache and the blocking behaviour. For example, if the cache is dropped or the filesystem remounted the blocking will not occur. > > >> > > >> > > >> Test scenario below: > > >> > > >> > > >> /mnt/cephfs_mountpoint type ceph (rw,relatime,name=ceph_filesystem,secret=<hidden>>,noshare,acl,wsize=16777216,rasize=268439552,caps_wanted_delay_min=1,caps_wanted_delay_max=1) > > >> > > >> > > >> Test1: > > >> > > >> 1) unmount & remount: > > >> > > >> > > >> 2) Add 10 x 100GB files to a directory: > > >> > > >> > > >> for i in {1..10}; do dd if=/dev/zero of=/mnt/cephfs_mountpoint/file$i.txt count=102400 bs=1048576; done > > >> > > >> > > >> 3) Delete all files in directory: > > >> > > >> > > >> for i in {1..10};do rm -f /mnt/cephfs_mountpoint/file$i.txt; done > > >> > > >> > > >> 4) Immediately perform ls on directory: > > >> > > >> > > >> time ls /mnt/cephfs_mountpoint/test1 > > >> > > >> > > >> Result: delay ~16 seconds > > >> > > >> real 0m16.818s > > >> > > >> user 0m0.000s > > >> > > >> sys 0m0.002s > > >> > > >> > > > > > Are cephfs metadata pool and data pool on the same set of OSD. Is is > > > possible that heavy data IO slowed down metadata IO? > > > > Test results are from a new pre-production cluster that does not have any significant data IO. We've also confirmed the same behaviour on another cluster with similar configuration. Both clusters have separate device-class/crush rule for metadata pool using NVME OSDs, while the data pool uses HDD OSDs. > > Most metadata operations are unaffected. It appears that it is only metadata operations on files that exist in client page cache prior to rm that are delayed. > > > > Ok. Please enable kernel debug when running 'ls' and send kernel log to us. > > echo module ceph +p > /sys/kernel/debug/dynamic_debug/control; > time /mnt/cephfs_mountpoint/test1 > echo module ceph -p > /sys/kernel/debug/dynamic_debug/control; > > Yan, Zheng Thanks Yan, Zheng I've attached two logfiles as I ran the test twice.
The first time as previously described Test1 - cephfs_kern.log
The second time I dropped caches prior to rm as in previous Test2 - cephfs_drop_caches_kern.log
> > >>
> > >> Test2: > > >> > > >> > > >> 1) unmount & remount > > >> > > >> > > >> 2) Add 10 x 100GB files to a directory > > >> > > >> for i in {1..10}; do dd if=/dev/zero of=/mnt/cephfs_mountpoint/file$i.txt count=102400 bs=1048576; done > > >> > > >> > > >> 3) Either a) unmount & remount; or b) drop caches > > >> > > >> > > >> echo 3 >>/proc/sys/vm/drop_caches > > >> > > >> > > >> 4) Delete files in directory: > > >> > > >> > > >> for i in {1..10};do rm -f /mnt/cephfs_mountpoint/file$i.txt; done > > >> > > >> > > >> 5) Immediately perform ls on directory: > > >> > > >> > > >> time ls /mnt/cephfs_mountpoint/test1 > > >> > > >> > > >> Result: no delay > > >> > > >> real 0m0.010s > > >> > > >> user 0m0.000s > > >> > > >> sys 0m0.001s > > >> > > >> > > >> Our understanding of ceph-fs’ file deletion mechanism, is that there should be no blocking observed on the client. http://docs.ceph.com/docs/mimic/dev/delayed-delete/ . > > >> > > >> It appears that if files are cached on the client, either by being created or accessed recently it will cause the kernel client to block for reasons we have not identified. > > >> > > >> > > >> Is this a known issue, are there any ways to mitigate this behaviour? > > >> > > >> Our production system relies on our client’s processes having concurrent access to the file system, and access contention must be avoided. > > >> > > >> > > >> An old mailing list post that discusses changes to client’s page cache behaviour may be relevant. > > >> > > >> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2015-October/005692.html > > >> > > >> > > >> Client System: > > >> > > >> > > >> OS: RHEL7 > > >> > > >> Kernel: 4.15.15-1 > > >> > > >> > > >> Cluster: Ceph: Luminous 12.2.8 > > >> > > >> > > > > > > > > >> Thanks, > > >> > > >> Dylan > > >> > > >> > > >> _______________________________________________ > > >> ceph-users mailing list > > >> ceph-users@xxxxxxxxxxxxxx > > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > |
[ 1987.356149] perf-issue-part (6640): drop_caches: 3
[ 1987.394578] ceph: do_getattr inode 000000005a6fde97 mask As mode 040775
[ 1987.394580] ceph: __ceph_caps_issued_mask 000000005a6fde97 cap 00000000ca9565a0 issued pAsLsXsFs (mask As)
[ 1987.394581] ceph: __touch_cap 000000005a6fde97 cap 00000000ca9565a0 mds0
[ 1987.394583] ceph: d_revalidate 00000000522328b3 'dmc' inode 00000000febc30a8 offset 0
[ 1987.394584] ceph: dentry_lease_is_valid - dentry 00000000522328b3 = 0
[ 1987.394585] ceph: __ceph_caps_issued_mask 000000005a6fde97 cap 00000000ca9565a0 issued pAsLsXsFs (mask Fs)
[ 1987.394586] ceph: __touch_cap 000000005a6fde97 cap 00000000ca9565a0 mds0
[ 1987.394587] ceph: dir_lease_is_valid dir 000000005a6fde97 v1 dentry 00000000522328b3 v1 = 1
[ 1987.394588] ceph: d_revalidate 00000000522328b3 valid
[ 1987.394589] ceph: dentry_lru_touch 00000000ade6eeb4 00000000522328b3 'dmc' (offset 0)
[ 1987.394590] ceph: do_getattr inode 00000000febc30a8 mask As mode 040755
[ 1987.394591] ceph: __ceph_caps_issued_mask 00000000febc30a8 cap 00000000a0261128 issued pAsLsXsFsx (mask As)
[ 1987.394591] ceph: __touch_cap 00000000febc30a8 cap 00000000a0261128 mds0
[ 1987.394592] ceph: d_revalidate 0000000040087e94 'test1' inode 0000000070e4a810 offset 0
[ 1987.394593] ceph: dentry_lease_is_valid - dentry 0000000040087e94 = 0
[ 1987.394594] ceph: __ceph_caps_issued_mask 00000000febc30a8 cap 00000000a0261128 issued pAsLsXsFsx (mask Fs)
[ 1987.394594] ceph: __touch_cap 00000000febc30a8 cap 00000000a0261128 mds0
[ 1987.394595] ceph: dir_lease_is_valid dir 00000000febc30a8 v1 dentry 0000000040087e94 v1 = 1
[ 1987.394595] ceph: d_revalidate 0000000040087e94 valid
[ 1987.394596] ceph: dentry_lru_touch 00000000cca4f3b4 0000000040087e94 'test1' (offset 0)
[ 1987.394597] ceph: do_getattr inode 0000000070e4a810 mask pAsLsXsFs mode 040755
[ 1987.394598] ceph: __ceph_caps_issued_mask 0000000070e4a810 cap 00000000a57451a3 issued pAsLsXsFsx (mask pAsLsXsFs)
[ 1987.394598] ceph: __touch_cap 0000000070e4a810 cap 00000000a57451a3 mds0
[ 1987.394605] ceph: do_getattr inode 000000005a6fde97 mask As mode 040775
[ 1987.394606] ceph: __ceph_caps_issued_mask 000000005a6fde97 cap 00000000ca9565a0 issued pAsLsXsFs (mask As)
[ 1987.394606] ceph: __touch_cap 000000005a6fde97 cap 00000000ca9565a0 mds0
[ 1987.394607] ceph: d_revalidate 00000000522328b3 'dmc' inode 00000000febc30a8 offset 0
[ 1987.394608] ceph: dentry_lease_is_valid - dentry 00000000522328b3 = 0
[ 1987.394608] ceph: __ceph_caps_issued_mask 000000005a6fde97 cap 00000000ca9565a0 issued pAsLsXsFs (mask Fs)
[ 1987.394609] ceph: __touch_cap 000000005a6fde97 cap 00000000ca9565a0 mds0
[ 1987.394610] ceph: dir_lease_is_valid dir 000000005a6fde97 v1 dentry 00000000522328b3 v1 = 1
[ 1987.394610] ceph: d_revalidate 00000000522328b3 valid
[ 1987.394611] ceph: dentry_lru_touch 00000000ade6eeb4 00000000522328b3 'dmc' (offset 0)
[ 1987.394611] ceph: do_getattr inode 00000000febc30a8 mask As mode 040755
[ 1987.394612] ceph: __ceph_caps_issued_mask 00000000febc30a8 cap 00000000a0261128 issued pAsLsXsFsx (mask As)
[ 1987.394612] ceph: __touch_cap 00000000febc30a8 cap 00000000a0261128 mds0
[ 1987.394613] ceph: d_revalidate 0000000040087e94 'test1' inode 0000000070e4a810 offset 0
[ 1987.394614] ceph: dentry_lease_is_valid - dentry 0000000040087e94 = 0
[ 1987.394614] ceph: __ceph_caps_issued_mask 00000000febc30a8 cap 00000000a0261128 issued pAsLsXsFsx (mask Fs)
[ 1987.394615] ceph: __touch_cap 00000000febc30a8 cap 00000000a0261128 mds0
[ 1987.394616] ceph: dir_lease_is_valid dir 00000000febc30a8 v1 dentry 0000000040087e94 v1 = 1
[ 1987.394616] ceph: d_revalidate 0000000040087e94 valid
[ 1987.394617] ceph: dentry_lru_touch 00000000cca4f3b4 0000000040087e94 'test1' (offset 0)
[ 1987.394617] ceph: do_getattr inode 0000000070e4a810 mask As mode 040755
[ 1987.394618] ceph: __ceph_caps_issued_mask 0000000070e4a810 cap 00000000a57451a3 issued pAsLsXsFsx (mask As)
[ 1987.394619] ceph: __touch_cap 0000000070e4a810 cap 00000000a57451a3 mds0
[ 1987.394621] ceph: open inode 0000000070e4a810 ino 100001ed749.fffffffffffffffe file 000000008c22b0dd flags 65536 (100352)
[ 1987.394622] ceph: __ceph_caps_issued 0000000070e4a810 cap 00000000a57451a3 issued pAsLsXsFsx
[ 1987.394623] ceph: open 0000000070e4a810 fmode 0 want p issued pAsLsXsFsx using existing
[ 1987.394624] ceph: init_file 0000000070e4a810 000000008c22b0dd 040755 (regular)
[ 1987.394627] ceph: readdir 0000000070e4a810 file 000000008c22b0dd pos 0
[ 1987.394628] ceph: readdir off 0 -> '.'
[ 1987.394629] ceph: readdir off 1 -> '..'
[ 1987.394630] ceph: readdir fetching 100001ed749.fffffffffffffffe frag 0 offset '(null)'
[ 1987.394631] ceph: do_request on 000000009d3b260b
[ 1987.394633] ceph: reserve caps ctx=000000001d0cb3f0 need=35
[ 1987.394637] ceph: reserve caps ctx=000000001d0cb3f0 2103 = 2068 used + 35 resv + 0 avail
[ 1987.394638] ceph: __register_request 000000009d3b260b tid 124
[ 1987.394639] ceph: __choose_mds 0000000070e4a810 is_hash=1 (0) mode 2
[ 1987.394640] ceph: choose_frag(0) = 0
[ 1987.394641] ceph: choose_mds 0000000070e4a810 100001ed749.fffffffffffffffe mds0 (auth cap 00000000a57451a3)
[ 1987.394642] ceph: lookup_mds_session 00000000c8eb37bf 16
[ 1987.394643] ceph: mdsc get_session 00000000c8eb37bf 16 -> 17
[ 1987.394644] ceph: mdsc get_session 00000000c8eb37bf 17 -> 18
[ 1987.394645] ceph: do_request mds0 session 00000000c8eb37bf state open
[ 1987.394646] ceph: prepare_send_request 000000009d3b260b tid 124 readdir (attempt 1)
[ 1987.394647] ceph: inode 0000000070e4a810 100001ed749.fffffffffffffffe
[ 1987.394648] ceph: r_parent = (null)
[ 1987.394649] ceph: mdsc get_session 00000000c8eb37bf 18 -> 19
[ 1987.394649] ceph: mdsc con_get 00000000c8eb37bf ok (19)
[ 1987.394650] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1987.394650] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1987.394652] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1987.394653] ceph: do_request waiting
[ 1987.394665] ceph: mdsc con_put 00000000c8eb37bf (18)
[ 1987.394665] ceph: mdsc put_session 00000000c8eb37bf 19 -> 18
[ 1987.394725] ceph: mdsc get_session 00000000c8eb37bf 18 -> 19
[ 1987.394726] ceph: mdsc con_get 00000000c8eb37bf ok (19)
[ 1987.394731] ceph: mdsc con_put 00000000c8eb37bf (18)
[ 1987.394732] ceph: mdsc put_session 00000000c8eb37bf 19 -> 18
[ 1987.394752] ceph: mdsc get_session 00000000c8eb37bf 18 -> 19
[ 1987.394753] ceph: mdsc con_get 00000000c8eb37bf ok (19)
[ 1987.394756] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1987.394756] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1987.394765] ceph: handle_caps from mds0
[ 1987.394767] ceph: op revoke ino 100001ed749.fffffffffffffffe inode 0000000070e4a810
[ 1987.394768] ceph: mds0 seq 2016 cap seq 37
[ 1987.394769] ceph: __ceph_caps_issued 0000000070e4a810 cap 00000000a57451a3 issued pAsLsXsFsx
[ 1987.394771] ceph: handle_cap_grant inode 0000000070e4a810 cap 00000000a57451a3 mds0 seq 37 pAsLsXsFs
[ 1987.394772] ceph: size 0 max_size 0, i_size 40
[ 1987.394772] ceph: __ceph_caps_issued 0000000070e4a810 cap 00000000a57451a3 issued pAsLsXsFsx
[ 1987.394773] ceph: 0000000070e4a810 mode 040755 uid.gid 0.0
[ 1987.394775] ceph: my wanted = p, used = p, dirty -
[ 1987.394776] ceph: mds wanted pAsLsXsFsx -> p
[ 1987.394777] ceph: revocation: pAsLsXsFsx -> pAsLsXsFs (revoking Fx)
[ 1987.394778] ceph: __ceph_caps_issued 0000000070e4a810 cap 00000000a57451a3 issued pAsLsXsFs
[ 1987.394779] ceph: check_caps 0000000070e4a810 file_want p used p dirty - flushing - issued pAsLsXsFs revoking Fx retain pAsxLsxXsxFsxcrwbl AUTHONLY NODELAY
[ 1987.394781] ceph: mds0 cap 00000000a57451a3 used p issued pAsLsXsFs implemented pAsLsXsFsx revoking Fx
[ 1987.394781] ceph: completed revocation of Fx
[ 1987.394783] ceph: __send_cap 0000000070e4a810 cap 00000000a57451a3 session 00000000c8eb37bf pAsLsXsFsx -> pAsLsXsFs (revoking Fx)
[ 1987.394784] ceph: delaying issued pAsLsXsFs -> pAsLsXsFs, wanted pAsLsXsFsx -> p on send
[ 1987.394786] ceph: send_cap_msg update 37bd31 100001ed749 caps pAsLsXsFs wanted pAsLsXsFsx dirty - seq 37/36 tid 0/0 mseq 0 follows 0 size 40/0 xattr_ver 18446744072109198816 xattr_len 0
[ 1987.394787] ceph: mdsc get_session 00000000c8eb37bf 20 -> 21
[ 1987.394787] ceph: mdsc con_get 00000000c8eb37bf ok (21)
[ 1987.394788] ceph: mdsc get_session 00000000c8eb37bf 21 -> 22
[ 1987.394789] ceph: mdsc con_get 00000000c8eb37bf ok (22)
[ 1987.394790] ceph: __ceph_caps_issued 0000000070e4a810 cap 00000000a57451a3 issued pAsLsXsFs
[ 1987.394791] ceph: check_caps 0000000070e4a810 file_want p used p dirty - flushing - issued pAsLsXsFs revoking - retain pAsxLsxXsxFsxcrwbl AUTHONLY NODELAY
[ 1987.394792] ceph: __cap_set_timeouts 0000000070e4a810 min 1216 max 1216
[ 1987.394793] ceph: __cap_delay_requeue 0000000070e4a810 flags 0 at 4296655961
[ 1987.394794] ceph: mdsc con_put 00000000c8eb37bf (21)
[ 1987.394794] ceph: mdsc put_session 00000000c8eb37bf 22 -> 21
[ 1987.394801] ceph: mdsc con_put 00000000c8eb37bf (20)
[ 1987.394801] ceph: mdsc put_session 00000000c8eb37bf 21 -> 20
[ 1987.394802] ceph: mdsc con_put 00000000c8eb37bf (19)
[ 1987.394803] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1987.394852] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1987.394853] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1987.394855] ceph: mdsc con_put 00000000c8eb37bf (19)
[ 1987.394856] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1987.394857] ceph: mdsc con_put 00000000c8eb37bf (18)
[ 1987.394857] ceph: mdsc put_session 00000000c8eb37bf 19 -> 18
[ 1987.396296] ceph: mdsc get_session 00000000c8eb37bf 18 -> 19
[ 1987.396296] ceph: mdsc con_get 00000000c8eb37bf ok (19)
[ 1987.396335] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1987.396336] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1987.396337] ceph: mdsc con_put 00000000c8eb37bf (19)
[ 1987.396337] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1987.396347] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1987.396348] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1987.396348] ceph: mdsc con_put 00000000c8eb37bf (19)
[ 1987.396349] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1987.396387] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1987.396387] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1987.396388] ceph: mdsc con_put 00000000c8eb37bf (19)
[ 1987.396388] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1987.396398] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1987.396399] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1987.396399] ceph: mdsc con_put 00000000c8eb37bf (19)
[ 1987.396400] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1987.396437] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1987.396437] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1987.396438] ceph: mdsc con_put 00000000c8eb37bf (19)
[ 1987.396439] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1987.396444] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1987.396445] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1987.396446] ceph: mdsc con_put 00000000c8eb37bf (19)
[ 1987.396446] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1987.396484] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1987.396484] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1987.396485] ceph: mdsc con_put 00000000c8eb37bf (19)
[ 1987.396485] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1987.396519] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1987.396519] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1987.396520] ceph: mdsc con_put 00000000c8eb37bf (19)
[ 1987.396521] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1987.397517] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1987.397518] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1987.397520] ceph: handle_caps from mds0
[ 1987.397522] ceph: op grant ino 100001ed784.fffffffffffffffe inode 00000000b9dd47ac
[ 1987.397523] ceph: mds0 seq 2017 cap seq 92
[ 1987.397524] ceph: __ceph_caps_issued 00000000b9dd47ac cap 0000000024c864e1 issued p
[ 1987.397525] ceph: handle_cap_grant inode 00000000b9dd47ac cap 0000000024c864e1 mds0 seq 92 pAsLsXsFscr
[ 1987.397526] ceph: size 107374182400 max_size 0, i_size 107374182400
[ 1987.397526] ceph: __ceph_caps_issued 00000000b9dd47ac cap 0000000024c864e1 issued p
[ 1987.397527] ceph: 00000000b9dd47ac mode 0100644 uid.gid 0.0
[ 1987.397529] ceph: my wanted = -, used = -, dirty -
[ 1987.397530] ceph: mds wanted Ls -> -
[ 1987.397531] ceph: grant: p -> pAsLsXsFscr
[ 1987.397531] ceph: invalidate_aliases inode 00000000b9dd47ac
[ 1987.397532] ceph: mdsc con_put 00000000c8eb37bf (19)
[ 1987.397533] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1987.397534] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1987.397535] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1987.397536] ceph: handle_reply 00000000ce76d613
[ 1987.397537] ceph: __unregister_request 00000000ce76d613 tid 119
[ 1987.397538] ceph: got safe reply 119, mds0
[ 1987.397543] ceph: mdsc get_session 00000000c8eb37bf 20 -> 21
[ 1987.397544] ceph: mdsc con_get 00000000c8eb37bf ok (21)
[ 1987.397546] ceph: mdsc con_put 00000000c8eb37bf (20)
[ 1987.397547] ceph: mdsc put_session 00000000c8eb37bf 21 -> 20
[ 1987.397548] ceph: mdsc con_put 00000000c8eb37bf (19)
[ 1987.397548] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1987.397550] ceph: put_cap_refs 0000000070e4a810 had p
[ 1987.397551] ceph: ceph_d_prune 00000000062c3ffd
[ 1987.397554] ceph: destroy_inode 00000000b9dd47ac ino 100001ed784.fffffffffffffffe
[ 1987.397555] ceph: __ceph_remove_cap 0000000024c864e1 from 00000000b9dd47ac
[ 1987.397556] ceph: put_snap_realm 1 000000008b101794 35 -> 34
[ 1987.397557] ceph: __cap_delay_cancel 00000000b9dd47ac
[ 1987.397558] ceph: __ceph_destroy_xattrs p= (null)
[ 1987.397559] ceph: d_release 00000000062c3ffd
[ 1987.397560] ceph: dentry_lru_del 00000000ee10f333 00000000062c3ffd 'file1.txt'
[ 1987.397561] ceph: mdsc put_session 00000000c8eb37bf 19 -> 18
[ 1987.397562] ceph: unreserve caps ctx=00000000f34ca0f9 count=0
[ 1987.397563] ceph: mdsc con_put 00000000c8eb37bf (17)
[ 1987.397564] ceph: mdsc put_session 00000000c8eb37bf 18 -> 17
[ 1987.397565] ceph: mdsc get_session 00000000c8eb37bf 17 -> 18
[ 1987.397565] ceph: mdsc con_get 00000000c8eb37bf ok (18)
[ 1987.397567] ceph: handle_caps from mds0
[ 1987.397568] ceph: op grant ino 100001ed785.fffffffffffffffe inode 00000000150130f5
[ 1987.397568] ceph: mds0 seq 2018 cap seq 83
[ 1987.397569] ceph: __ceph_caps_issued 00000000150130f5 cap 0000000061a546e6 issued p
[ 1987.397570] ceph: handle_cap_grant inode 00000000150130f5 cap 0000000061a546e6 mds0 seq 83 pAsLsXsFscr
[ 1987.397571] ceph: size 107374182400 max_size 0, i_size 107374182400
[ 1987.397571] ceph: __ceph_caps_issued 00000000150130f5 cap 0000000061a546e6 issued p
[ 1987.397572] ceph: 00000000150130f5 mode 0100644 uid.gid 0.0
[ 1987.397573] ceph: my wanted = -, used = -, dirty -
[ 1987.397573] ceph: mds wanted Ls -> -
[ 1987.397574] ceph: grant: p -> pAsLsXsFscr
[ 1987.397574] ceph: invalidate_aliases inode 00000000150130f5
[ 1987.397575] ceph: mdsc con_put 00000000c8eb37bf (17)
[ 1987.397576] ceph: mdsc put_session 00000000c8eb37bf 18 -> 17
[ 1987.397577] ceph: mdsc get_session 00000000c8eb37bf 17 -> 18
[ 1987.397578] ceph: mdsc con_get 00000000c8eb37bf ok (18)
[ 1987.397579] ceph: handle_reply 00000000b5a6a4b6
[ 1987.397579] ceph: __unregister_request 00000000b5a6a4b6 tid 120
[ 1987.397580] ceph: got safe reply 120, mds0
[ 1987.397581] ceph: mdsc con_put 00000000c8eb37bf (17)
[ 1987.397581] ceph: mdsc put_session 00000000c8eb37bf 18 -> 17
[ 1987.397583] ceph: mdsc con_put 00000000c8eb37bf (16)
[ 1987.397583] ceph: mdsc put_session 00000000c8eb37bf 17 -> 16
[ 1987.397584] ceph: put_cap_refs 0000000070e4a810 had p
[ 1987.397585] ceph: ceph_d_prune 000000003a9fb321
[ 1987.397586] ceph: destroy_inode 00000000150130f5 ino 100001ed785.fffffffffffffffe
[ 1987.397586] ceph: __ceph_remove_cap 0000000061a546e6 from 00000000150130f5
[ 1987.397587] ceph: put_snap_realm 1 000000008b101794 34 -> 33
[ 1987.397588] ceph: __cap_delay_cancel 00000000150130f5
[ 1987.397588] ceph: __ceph_destroy_xattrs p= (null)
[ 1987.397589] ceph: d_release 000000003a9fb321
[ 1987.397589] ceph: dentry_lru_del 000000009f8c7d76 000000003a9fb321 'file2.txt'
[ 1987.397590] ceph: mdsc put_session 00000000c8eb37bf 16 -> 15
[ 1987.397591] ceph: unreserve caps ctx=0000000060eb8e9b count=0
[ 1987.397591] ceph: mdsc con_put 00000000c8eb37bf (14)
[ 1987.397592] ceph: mdsc put_session 00000000c8eb37bf 15 -> 14
[ 1987.397593] ceph: mdsc get_session 00000000c8eb37bf 14 -> 15
[ 1987.397594] ceph: mdsc con_get 00000000c8eb37bf ok (15)
[ 1987.397595] ceph: handle_caps from mds0
[ 1987.397596] ceph: op grant ino 100001ed786.fffffffffffffffe inode 0000000035bcecb2
[ 1987.397596] ceph: mds0 seq 2019 cap seq 80
[ 1987.397597] ceph: __ceph_caps_issued 0000000035bcecb2 cap 00000000d1fcbbd9 issued p
[ 1987.397598] ceph: handle_cap_grant inode 0000000035bcecb2 cap 00000000d1fcbbd9 mds0 seq 80 pAsLsXsFscr
[ 1987.397598] ceph: size 107374182400 max_size 0, i_size 107374182400
[ 1987.397599] ceph: __ceph_caps_issued 0000000035bcecb2 cap 00000000d1fcbbd9 issued p
[ 1987.397600] ceph: 0000000035bcecb2 mode 0100644 uid.gid 0.0
[ 1987.397601] ceph: my wanted = -, used = -, dirty -
[ 1987.397601] ceph: mds wanted Ls -> -
[ 1987.397602] ceph: grant: p -> pAsLsXsFscr
[ 1987.397602] ceph: invalidate_aliases inode 0000000035bcecb2
[ 1987.397603] ceph: mdsc con_put 00000000c8eb37bf (14)
[ 1987.397604] ceph: mdsc put_session 00000000c8eb37bf 15 -> 14
[ 1987.397605] ceph: mdsc get_session 00000000c8eb37bf 14 -> 15
[ 1987.397605] ceph: mdsc con_get 00000000c8eb37bf ok (15)
[ 1987.397606] ceph: handle_reply 000000007df1685a
[ 1987.397607] ceph: __unregister_request 000000007df1685a tid 121
[ 1987.397608] ceph: got safe reply 121, mds0
[ 1987.397609] ceph: mdsc con_put 00000000c8eb37bf (14)
[ 1987.397609] ceph: mdsc put_session 00000000c8eb37bf 15 -> 14
[ 1987.397610] ceph: mdsc con_put 00000000c8eb37bf (13)
[ 1987.397611] ceph: mdsc put_session 00000000c8eb37bf 14 -> 13
[ 1987.397612] ceph: put_cap_refs 0000000070e4a810 had p
[ 1987.397612] ceph: ceph_d_prune 000000004fa570bf
[ 1987.397613] ceph: destroy_inode 0000000035bcecb2 ino 100001ed786.fffffffffffffffe
[ 1987.397614] ceph: __ceph_remove_cap 00000000d1fcbbd9 from 0000000035bcecb2
[ 1987.397615] ceph: put_snap_realm 1 000000008b101794 33 -> 32
[ 1987.397615] ceph: __cap_delay_cancel 0000000035bcecb2
[ 1987.397616] ceph: __ceph_destroy_xattrs p= (null)
[ 1987.397616] ceph: d_release 000000004fa570bf
[ 1987.397617] ceph: dentry_lru_del 0000000055b76e3b 000000004fa570bf 'file3.txt'
[ 1987.397618] ceph: mdsc put_session 00000000c8eb37bf 13 -> 12
[ 1987.397618] ceph: unreserve caps ctx=0000000065473d82 count=0
[ 1987.397619] ceph: mdsc con_put 00000000c8eb37bf (11)
[ 1987.397619] ceph: mdsc put_session 00000000c8eb37bf 12 -> 11
[ 1987.397620] ceph: mdsc get_session 00000000c8eb37bf 11 -> 12
[ 1987.397621] ceph: mdsc con_get 00000000c8eb37bf ok (12)
[ 1987.397622] ceph: handle_caps from mds0
[ 1987.397623] ceph: op grant ino 100001ed787.fffffffffffffffe inode 000000005e234bc9
[ 1987.397624] ceph: mds0 seq 2020 cap seq 83
[ 1987.397624] ceph: __ceph_caps_issued 000000005e234bc9 cap 0000000053d84a0b issued p
[ 1987.397625] ceph: handle_cap_grant inode 000000005e234bc9 cap 0000000053d84a0b mds0 seq 83 pAsLsXsFscr
[ 1987.397626] ceph: size 107374182400 max_size 0, i_size 107374182400
[ 1987.397626] ceph: __ceph_caps_issued 000000005e234bc9 cap 0000000053d84a0b issued p
[ 1987.397627] ceph: 000000005e234bc9 mode 0100644 uid.gid 0.0
[ 1987.397627] ceph: my wanted = -, used = -, dirty -
[ 1987.397628] ceph: mds wanted Ls -> -
[ 1987.397628] ceph: grant: p -> pAsLsXsFscr
[ 1987.397629] ceph: invalidate_aliases inode 000000005e234bc9
[ 1987.397629] ceph: mdsc con_put 00000000c8eb37bf (11)
[ 1987.397630] ceph: mdsc put_session 00000000c8eb37bf 12 -> 11
[ 1987.397631] ceph: mdsc get_session 00000000c8eb37bf 11 -> 12
[ 1987.397632] ceph: mdsc con_get 00000000c8eb37bf ok (12)
[ 1987.397633] ceph: handle_reply 00000000d74a8455
[ 1987.397633] ceph: __unregister_request 00000000d74a8455 tid 122
[ 1987.397634] ceph: got safe reply 122, mds0
[ 1987.397635] ceph: mdsc con_put 00000000c8eb37bf (11)
[ 1987.397635] ceph: mdsc put_session 00000000c8eb37bf 12 -> 11
[ 1987.397637] ceph: mdsc con_put 00000000c8eb37bf (10)
[ 1987.397637] ceph: mdsc put_session 00000000c8eb37bf 11 -> 10
[ 1987.397638] ceph: put_cap_refs 0000000070e4a810 had p
[ 1987.397638] ceph: ceph_d_prune 00000000298dfd11
[ 1987.397639] ceph: destroy_inode 000000005e234bc9 ino 100001ed787.fffffffffffffffe
[ 1987.397640] ceph: __ceph_remove_cap 0000000053d84a0b from 000000005e234bc9
[ 1987.397641] ceph: put_snap_realm 1 000000008b101794 32 -> 31
[ 1987.397641] ceph: __cap_delay_cancel 000000005e234bc9
[ 1987.397642] ceph: __ceph_destroy_xattrs p= (null)
[ 1987.397642] ceph: d_release 00000000298dfd11
[ 1987.397643] ceph: dentry_lru_del 00000000fc7a978f 00000000298dfd11 'file4.txt'
[ 1987.397643] ceph: mdsc put_session 00000000c8eb37bf 10 -> 9
[ 1987.397644] ceph: unreserve caps ctx=00000000c9ad5916 count=0
[ 1987.397644] ceph: mdsc con_put 00000000c8eb37bf (8)
[ 1987.397645] ceph: mdsc put_session 00000000c8eb37bf 9 -> 8
[ 1987.397646] ceph: mdsc get_session 00000000c8eb37bf 8 -> 9
[ 1987.397647] ceph: mdsc con_get 00000000c8eb37bf ok (9)
[ 1987.397648] ceph: handle_reply 000000009d3b260b
[ 1987.397649] ceph: __unregister_request 000000009d3b260b tid 124
[ 1987.397649] ceph: handle_reply tid 124 result 0
[ 1987.397650] ceph: update_snap_trace deletion=0
[ 1987.397651] ceph: lookup_snap_realm 1 000000008b101794
[ 1987.397652] ceph: get_realm 000000008b101794 31 -> 32
[ 1987.397653] ceph: update_snap_trace 1 000000008b101794 seq 1 unchanged
[ 1987.397654] ceph: done with 1 000000008b101794, invalidated=0, 00000000edc90cda 00000000edc90cda
[ 1987.397655] ceph: fill_trace 000000009d3b260b is_dentry 0 is_target 1
[ 1987.397657] ceph: get_inode on 1099513648969=100001ed749.fffffffffffffffe got 0000000070e4a810
[ 1987.397658] ceph: fill_inode 0000000070e4a810 ino 100001ed749.fffffffffffffffe v 9650292 had 9650273
[ 1987.397659] ceph: get_cap ctx=000000001d0cb3f0 (35) 2103 = 2068 used + 35 resv + 0 avail
[ 1987.397660] ceph: __ceph_caps_issued 0000000070e4a810 cap 00000000a57451a3 issued pAsLsXsFs
[ 1987.397661] ceph: 0000000070e4a810 mode 040755 uid.gid 0.0
[ 1987.397663] ceph: add_cap 0000000070e4a810 mds0 cap 37bd31 pAsLsXsFs seq 38
[ 1987.397663] ceph: __ceph_caps_issued 0000000070e4a810 cap 00000000a57451a3 issued pAsLsXsFs
[ 1987.397664] ceph: issued pAsLsXsFs, mds wanted pAsLsXsFsx, actual p, queueing
[ 1987.397665] ceph: __cap_set_timeouts 0000000070e4a810 min 1213 max 1213
[ 1987.397666] ceph: __cap_delay_requeue 0000000070e4a810 flags 0 at 4296655961
[ 1987.397667] ceph: add_cap inode 0000000070e4a810 (100001ed749.fffffffffffffffe) cap 00000000a57451a3 pAsLsXsFs now pAsLsXsFs seq 38 mds0
[ 1987.397668] ceph: put_cap 000000000df0548c 2103 = 2069 used + 34 resv + 0 avail
[ 1987.397669] ceph: fill_trace done err=0
[ 1987.397670] ceph: readdir_prepopulate 0 items under dn 0000000040087e94
[ 1987.397671] ceph: readdir_prepopulate done
[ 1987.397671] ceph: unreserve caps ctx=000000001d0cb3f0 count=34
[ 1987.397672] ceph: unreserve caps 2103 = 2068 used + 0 resv + 35 avail
[ 1987.397673] ceph: put_snap_realm 1 000000008b101794 32 -> 31
[ 1987.397676] ceph: mdsc get_session 00000000c8eb37bf 9 -> 10
[ 1987.397676] ceph: mdsc con_get 00000000c8eb37bf ok (10)
[ 1987.397677] ceph: do_request waited, got 0
[ 1987.397678] ceph: do_request 000000009d3b260b done, result 0
[ 1987.397679] ceph: readdir got and parsed readdir result=0 on frag 0, end=1, complete=1, hash_order=1
[ 1987.397679] ceph: handle_caps from mds0
[ 1987.397680] ceph: op grant ino 100001ed749.fffffffffffffffe inode 0000000070e4a810
[ 1987.397681] ceph: mds0 seq 2021 cap seq 39
[ 1987.397682] ceph: readdir frag 0 num 0 pos 2 chunk first 0
[ 1987.397683] ceph: __ceph_caps_issued 0000000070e4a810 cap 00000000a57451a3 issued pAsLsXsFs
[ 1987.397683] ceph: mdsc con_put 00000000c8eb37bf (9)
[ 1987.397684] ceph: mdsc put_session 00000000c8eb37bf 10 -> 9
[ 1987.397685] ceph: handle_cap_grant inode 0000000070e4a810 cap 00000000a57451a3 mds0 seq 39 pAsLsXsFsx
[ 1987.397686] ceph: size 0 max_size 0, i_size 40
[ 1987.397687] ceph: mdsc con_put 00000000c8eb37bf (8)
[ 1987.397688] ceph: mdsc put_session 00000000c8eb37bf 9 -> 8
[ 1987.397688] ceph: __ceph_caps_issued 0000000070e4a810 cap 00000000a57451a3 issued pAsLsXsFs
[ 1987.397690] ceph: 0000000070e4a810 mode 040755 uid.gid 0.0
[ 1987.397690] ceph: my wanted = p, used = p, dirty -
[ 1987.397691] ceph: mds wanted pAsLsXsFsx -> p
[ 1987.397691] ceph: grant: pAsLsXsFs -> pAsLsXsFsx
[ 1987.397692] ceph: mdsc con_put 00000000c8eb37bf (7)
[ 1987.397693] ceph: put_cap_refs 0000000070e4a810 had p
[ 1987.397694] ceph: mdsc put_session 00000000c8eb37bf 8 -> 7
[ 1987.397695] ceph: mdsc put_session 00000000c8eb37bf 8 -> 7
[ 1987.397695] ceph: unreserve caps ctx=000000001d0cb3f0 count=0
[ 1987.397697] ceph: mdsc get_session 00000000c8eb37bf 6 -> 7
[ 1987.397698] ceph: marking 0000000070e4a810 complete and ordered
[ 1987.397698] ceph: readdir 0000000070e4a810 file 000000008c22b0dd done.
[ 1987.397699] ceph: mdsc con_get 00000000c8eb37bf ok (7)
[ 1987.397701] ceph: handle_caps from mds0
[ 1987.397702] ceph: op grant ino 100001ed788.fffffffffffffffe inode 000000000be62518
[ 1987.397702] ceph: readdir 0000000070e4a810 file 000000008c22b0dd pos 2
[ 1987.397703] ceph: mds0 seq 2022 cap seq 84
[ 1987.397704] ceph: __ceph_caps_issued 000000000be62518 cap 0000000070d8cfca issued p
[ 1987.397705] ceph: handle_cap_grant inode 000000000be62518 cap 0000000070d8cfca mds0 seq 84 pAsLsXsFscr
[ 1987.397706] ceph: release inode 0000000070e4a810 file 000000008c22b0dd
[ 1987.397707] ceph: put_fmode 0000000070e4a810 fmode 0 {0,0,0,0}
[ 1987.397708] ceph: size 107374182400 max_size 0, i_size 107374182400
[ 1987.397709] ceph: __ceph_caps_issued 000000000be62518 cap 0000000070d8cfca issued p
[ 1987.397709] ceph: __ceph_caps_issued 0000000070e4a810 cap 00000000a57451a3 issued pAsLsXsFsx
[ 1987.397711] ceph: check_caps 0000000070e4a810 file_want - used p dirty - flushing - issued pAsLsXsFsx revoking - retain pAsLsXsFsx
[ 1987.397712] ceph: 000000000be62518 mode 0100644 uid.gid 0.0
[ 1987.397713] ceph: my wanted = -, used = -, dirty -
[ 1987.397713] ceph: mds0 cap 00000000a57451a3 used p issued pAsLsXsFsx implemented pAsLsXsFsx revoking -
[ 1987.397714] ceph: delaying issued pAsLsXsFsx -> pAsLsXsFsx, wanted pAsLsXsFsx -> AsLsXsFsx
[ 1987.397715] ceph: mds wanted Ls -> -
[ 1987.397716] ceph: __cap_set_timeouts 0000000070e4a810 min 1207 max 1207
[ 1987.397717] ceph: __cap_delay_requeue 0000000070e4a810 flags 0 at 4296655955
[ 1987.397718] ceph: grant: p -> pAsLsXsFscr
[ 1987.397719] ceph: invalidate_aliases inode 000000000be62518
[ 1987.397719] ceph: mdsc con_put 00000000c8eb37bf (6)
[ 1987.397720] ceph: mdsc put_session 00000000c8eb37bf 7 -> 6
[ 1987.397721] ceph: mdsc get_session 00000000c8eb37bf 6 -> 7
[ 1987.397722] ceph: mdsc con_get 00000000c8eb37bf ok (7)
[ 1987.397723] ceph: handle_reply 000000006fb8dc63
[ 1987.397724] ceph: __unregister_request 000000006fb8dc63 tid 123
[ 1987.397724] ceph: got safe reply 123, mds0
[ 1987.397725] ceph: mdsc con_put 00000000c8eb37bf (6)
[ 1987.397726] ceph: mdsc put_session 00000000c8eb37bf 7 -> 6
[ 1987.397727] ceph: mdsc con_put 00000000c8eb37bf (5)
[ 1987.397727] ceph: mdsc put_session 00000000c8eb37bf 6 -> 5
[ 1987.397728] ceph: put_cap_refs 0000000070e4a810 had p
[ 1987.397729] ceph: ceph_d_prune 000000006d799af9
[ 1987.397735] ceph: destroy_inode 000000000be62518 ino 100001ed788.fffffffffffffffe
[ 1987.397735] ceph: __ceph_remove_cap 0000000070d8cfca from 000000000be62518
[ 1987.397736] ceph: put_snap_realm 1 000000008b101794 31 -> 30
[ 1987.397736] ceph: __cap_delay_cancel 000000000be62518
[ 1987.397737] ceph: __ceph_destroy_xattrs p= (null)
[ 1987.397737] ceph: d_release 000000006d799af9
[ 1987.397738] ceph: dentry_lru_del 00000000fa1e88d3 000000006d799af9 'file5.txt'
[ 1987.397739] ceph: mdsc put_session 00000000c8eb37bf 5 -> 4
[ 1987.397740] ceph: unreserve caps ctx=000000005cb10386 count=0
[ 1987.397740] ceph: mdsc con_put 00000000c8eb37bf (3)
[ 1987.397741] ceph: mdsc put_session 00000000c8eb37bf 4 -> 3
[ 1987.397744] ceph: mdsc con_put 00000000c8eb37bf (2)
[ 1987.397744] ceph: mdsc put_session 00000000c8eb37bf 3 -> 2
[ 1987.397745] ceph: mdsc con_put 00000000c8eb37bf (1)
[ 1987.397745] ceph: mdsc put_session 00000000c8eb37bf 2 -> 1
[ 1280.529935] ceph: do_getattr inode 000000005a6fde97 mask As mode 040775
[ 1280.529938] ceph: __ceph_caps_issued_mask 000000005a6fde97 cap 00000000ca9565a0 issued pAsLsXsFs (mask As)
[ 1280.529939] ceph: __touch_cap 000000005a6fde97 cap 00000000ca9565a0 mds0
[ 1280.529942] ceph: d_revalidate 0000000006dc6eec 'dmc' inode 000000006611481f offset 1148534478444953602
[ 1280.529943] ceph: dentry_lease_is_valid - dentry 0000000006dc6eec = 0
[ 1280.529944] ceph: __ceph_caps_issued_mask 000000005a6fde97 cap 00000000ca9565a0 issued pAsLsXsFs (mask Fs)
[ 1280.529944] ceph: __touch_cap 000000005a6fde97 cap 00000000ca9565a0 mds0
[ 1280.529945] ceph: dir_lease_is_valid dir 000000005a6fde97 v1 dentry 0000000006dc6eec v1 = 1
[ 1280.529946] ceph: d_revalidate 0000000006dc6eec valid
[ 1280.529948] ceph: dentry_lru_touch 00000000d589fb13 0000000006dc6eec 'dmc' (offset 1148534478444953602)
[ 1280.529949] ceph: do_getattr inode 000000006611481f mask As mode 040755
[ 1280.529950] ceph: __ceph_caps_issued_mask 000000006611481f cap 000000003a118540 issued pAsLsXsFsx (mask As)
[ 1280.529951] ceph: __touch_cap 000000006611481f cap 000000003a118540 mds0
[ 1280.529952] ceph: d_revalidate 0000000010f74030 'test1' inode 000000007dbf994a offset 1152287136058179586
[ 1280.529953] ceph: dentry_lease_is_valid - dentry 0000000010f74030 = 0
[ 1280.529953] ceph: __ceph_caps_issued_mask 000000006611481f cap 000000003a118540 issued pAsLsXsFsx (mask Fs)
[ 1280.529954] ceph: __touch_cap 000000006611481f cap 000000003a118540 mds0
[ 1280.529955] ceph: dir_lease_is_valid dir 000000006611481f v1 dentry 0000000010f74030 v1 = 1
[ 1280.529956] ceph: d_revalidate 0000000010f74030 valid
[ 1280.529957] ceph: dentry_lru_touch 00000000bcae6752 0000000010f74030 'test1' (offset 1152287136058179586)
[ 1280.529958] ceph: do_getattr inode 000000007dbf994a mask pAsLsXsFs mode 040755
[ 1280.529959] ceph: __ceph_caps_issued_mask 000000007dbf994a cap 000000001044692e issued pAsLsXsFsx (mask pAsLsXsFs)
[ 1280.529959] ceph: __touch_cap 000000007dbf994a cap 000000001044692e mds0
[ 1280.529967] ceph: do_getattr inode 000000005a6fde97 mask As mode 040775
[ 1280.529968] ceph: __ceph_caps_issued_mask 000000005a6fde97 cap 00000000ca9565a0 issued pAsLsXsFs (mask As)
[ 1280.529969] ceph: __touch_cap 000000005a6fde97 cap 00000000ca9565a0 mds0
[ 1280.529969] ceph: d_revalidate 0000000006dc6eec 'dmc' inode 000000006611481f offset 1148534478444953602
[ 1280.529970] ceph: dentry_lease_is_valid - dentry 0000000006dc6eec = 0
[ 1280.529971] ceph: __ceph_caps_issued_mask 000000005a6fde97 cap 00000000ca9565a0 issued pAsLsXsFs (mask Fs)
[ 1280.529972] ceph: __touch_cap 000000005a6fde97 cap 00000000ca9565a0 mds0
[ 1280.529972] ceph: dir_lease_is_valid dir 000000005a6fde97 v1 dentry 0000000006dc6eec v1 = 1
[ 1280.529973] ceph: d_revalidate 0000000006dc6eec valid
[ 1280.529974] ceph: dentry_lru_touch 00000000d589fb13 0000000006dc6eec 'dmc' (offset 1148534478444953602)
[ 1280.529974] ceph: do_getattr inode 000000006611481f mask As mode 040755
[ 1280.529975] ceph: __ceph_caps_issued_mask 000000006611481f cap 000000003a118540 issued pAsLsXsFsx (mask As)
[ 1280.529976] ceph: __touch_cap 000000006611481f cap 000000003a118540 mds0
[ 1280.529977] ceph: d_revalidate 0000000010f74030 'test1' inode 000000007dbf994a offset 1152287136058179586
[ 1280.529977] ceph: dentry_lease_is_valid - dentry 0000000010f74030 = 0
[ 1280.529978] ceph: __ceph_caps_issued_mask 000000006611481f cap 000000003a118540 issued pAsLsXsFsx (mask Fs)
[ 1280.529978] ceph: __touch_cap 000000006611481f cap 000000003a118540 mds0
[ 1280.529979] ceph: dir_lease_is_valid dir 000000006611481f v1 dentry 0000000010f74030 v1 = 1
[ 1280.529980] ceph: d_revalidate 0000000010f74030 valid
[ 1280.529980] ceph: dentry_lru_touch 00000000bcae6752 0000000010f74030 'test1' (offset 1152287136058179586)
[ 1280.529981] ceph: do_getattr inode 000000007dbf994a mask As mode 040755
[ 1280.529982] ceph: __ceph_caps_issued_mask 000000007dbf994a cap 000000001044692e issued pAsLsXsFsx (mask As)
[ 1280.529983] ceph: __touch_cap 000000007dbf994a cap 000000001044692e mds0
[ 1280.529984] ceph: open inode 000000007dbf994a ino 100001ed749.fffffffffffffffe file 00000000fa762655 flags 65536 (100352)
[ 1280.529986] ceph: __ceph_caps_issued 000000007dbf994a cap 000000001044692e issued pAsLsXsFsx
[ 1280.529987] ceph: open 000000007dbf994a fmode 0 want p issued pAsLsXsFsx using existing
[ 1280.529988] ceph: init_file 000000007dbf994a 00000000fa762655 040755 (regular)
[ 1280.529992] ceph: readdir 000000007dbf994a file 00000000fa762655 pos 0
[ 1280.529993] ceph: readdir off 0 -> '.'
[ 1280.529993] ceph: readdir off 1 -> '..'
[ 1280.529994] ceph: readdir fetching 100001ed749.fffffffffffffffe frag 0 offset '(null)'
[ 1280.529996] ceph: do_request on 000000004f565148
[ 1280.529998] ceph: reserve caps ctx=000000008187b867 need=35
[ 1280.529999] ceph: reserve caps ctx=000000008187b867 2096 = 2061 used + 35 resv + 0 avail
[ 1280.530000] ceph: __register_request 000000004f565148 tid 110
[ 1280.530002] ceph: __choose_mds 000000007dbf994a is_hash=1 (0) mode 2
[ 1280.530003] ceph: choose_frag(0) = 0
[ 1280.530004] ceph: choose_mds 000000007dbf994a 100001ed749.fffffffffffffffe mds0 (auth cap 000000001044692e)
[ 1280.530005] ceph: lookup_mds_session 00000000c8eb37bf 16
[ 1280.530006] ceph: mdsc get_session 00000000c8eb37bf 16 -> 17
[ 1280.530007] ceph: mdsc get_session 00000000c8eb37bf 17 -> 18
[ 1280.530008] ceph: do_request mds0 session 00000000c8eb37bf state open
[ 1280.530009] ceph: prepare_send_request 000000004f565148 tid 110 readdir (attempt 1)
[ 1280.530010] ceph: inode 000000007dbf994a 100001ed749.fffffffffffffffe
[ 1280.530012] ceph: r_parent = (null)
[ 1280.530012] ceph: mdsc get_session 00000000c8eb37bf 18 -> 19
[ 1280.530013] ceph: mdsc con_get 00000000c8eb37bf ok (19)
[ 1280.530014] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1280.530015] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1280.530018] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1280.530019] ceph: do_request waiting
[ 1280.530040] ceph: mdsc con_put 00000000c8eb37bf (18)
[ 1280.530040] ceph: mdsc put_session 00000000c8eb37bf 19 -> 18
[ 1280.530084] ceph: mdsc get_session 00000000c8eb37bf 18 -> 19
[ 1280.530091] ceph: mdsc con_get 00000000c8eb37bf ok (19)
[ 1280.530120] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1280.530121] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1280.530122] ceph: mdsc con_put 00000000c8eb37bf (19)
[ 1280.530123] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1280.531194] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1280.531195] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1280.531205] ceph: handle_caps from mds0
[ 1280.531206] ceph: op revoke ino 100001ed749.fffffffffffffffe inode 000000007dbf994a
[ 1280.531207] ceph: mds0 seq 1198 cap seq 20
[ 1280.531208] ceph: __ceph_caps_issued 000000007dbf994a cap 000000001044692e issued pAsLsXsFsx
[ 1280.531209] ceph: handle_cap_grant inode 000000007dbf994a cap 000000001044692e mds0 seq 20 pAsLsXsFs
[ 1280.531210] ceph: size 0 max_size 0, i_size 40
[ 1280.531211] ceph: __ceph_caps_issued 000000007dbf994a cap 000000001044692e issued pAsLsXsFsx
[ 1280.531211] ceph: 000000007dbf994a mode 040755 uid.gid 0.0
[ 1280.531213] ceph: my wanted = p, used = p, dirty -
[ 1280.531213] ceph: mds wanted pAsLsXsFsx -> p
[ 1280.531215] ceph: revocation: pAsLsXsFsx -> pAsLsXsFs (revoking Fx)
[ 1280.531215] ceph: __ceph_caps_issued 000000007dbf994a cap 000000001044692e issued pAsLsXsFs
[ 1280.531217] ceph: check_caps 000000007dbf994a file_want p used p dirty - flushing - issued pAsLsXsFs revoking Fx retain pAsxLsxXsxFsxcrwbl AUTHONLY NODELAY
[ 1280.531218] ceph: mds0 cap 000000001044692e used p issued pAsLsXsFs implemented pAsLsXsFsx revoking Fx
[ 1280.531219] ceph: completed revocation of Fx
[ 1280.531220] ceph: __send_cap 000000007dbf994a cap 000000001044692e session 00000000c8eb37bf pAsLsXsFsx -> pAsLsXsFs (revoking Fx)
[ 1280.531221] ceph: delaying issued pAsLsXsFs -> pAsLsXsFs, wanted pAsLsXsFsx -> p on send
[ 1280.531223] ceph: send_cap_msg update 37bd31 100001ed749 caps pAsLsXsFs wanted pAsLsXsFsx dirty - seq 20/19 tid 0/0 mseq 0 follows 0 size 40/0 xattr_ver 18446744072109198816 xattr_len 0
[ 1280.531224] ceph: mdsc get_session 00000000c8eb37bf 20 -> 21
[ 1280.531224] ceph: mdsc con_get 00000000c8eb37bf ok (21)
[ 1280.531225] ceph: mdsc get_session 00000000c8eb37bf 21 -> 22
[ 1280.531225] ceph: mdsc con_get 00000000c8eb37bf ok (22)
[ 1280.531226] ceph: __ceph_caps_issued 000000007dbf994a cap 000000001044692e issued pAsLsXsFs
[ 1280.531227] ceph: check_caps 000000007dbf994a file_want p used p dirty - flushing - issued pAsLsXsFs revoking - retain pAsxLsxXsxFsxcrwbl AUTHONLY NODELAY
[ 1280.531228] ceph: __cap_set_timeouts 000000007dbf994a min 1082 max 1082
[ 1280.531229] ceph: __cap_delay_requeue 000000007dbf994a flags 0 at 4295948943
[ 1280.531230] ceph: mdsc con_put 00000000c8eb37bf (21)
[ 1280.531230] ceph: mdsc put_session 00000000c8eb37bf 22 -> 21
[ 1280.531236] ceph: mdsc con_put 00000000c8eb37bf (20)
[ 1280.531237] ceph: mdsc put_session 00000000c8eb37bf 21 -> 20
[ 1280.531239] ceph: mdsc con_put 00000000c8eb37bf (19)
[ 1280.531239] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1280.531297] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1280.531298] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1280.531300] ceph: mdsc con_put 00000000c8eb37bf (19)
[ 1280.531301] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1280.531302] ceph: mdsc con_put 00000000c8eb37bf (18)
[ 1280.531302] ceph: mdsc put_session 00000000c8eb37bf 19 -> 18
[ 1280.531946] ceph: mdsc get_session 00000000c8eb37bf 18 -> 19
[ 1280.531947] ceph: mdsc con_get 00000000c8eb37bf ok (19)
[ 1280.531949] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1280.531950] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1280.531954] ceph: handle_caps from mds0
[ 1280.531957] ceph: op grant ino 100001ed77f.fffffffffffffffe inode 00000000fdca32d0
[ 1280.531958] ceph: mds0 seq 1199 cap seq 114
[ 1280.531958] ceph: __ceph_caps_issued 00000000fdca32d0 cap 0000000054a99657 issued pFc
[ 1280.531959] ceph: handle_cap_grant inode 00000000fdca32d0 cap 0000000054a99657 mds0 seq 114 pAsLsXsFscr
[ 1280.531960] ceph: size 107374182400 max_size 0, i_size 107374182400
[ 1280.531960] ceph: __ceph_caps_issued 00000000fdca32d0 cap 0000000054a99657 issued pFc
[ 1280.531961] ceph: 00000000fdca32d0 mode 0100644 uid.gid 0.0
[ 1280.531962] ceph: my wanted = Fc, used = Fc, dirty -
[ 1280.531963] ceph: mds wanted LsFc -> Fc
[ 1280.531963] ceph: grant: pFc -> pAsLsXsFscr
[ 1280.531964] ceph: invalidate_aliases inode 00000000fdca32d0
[ 1280.531965] ceph: mdsc con_put 00000000c8eb37bf (19)
[ 1280.531965] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1280.531967] ceph: mdsc con_put 00000000c8eb37bf (18)
[ 1280.531968] ceph: mdsc put_session 00000000c8eb37bf 19 -> 18
[ 1280.531995] ceph: mdsc get_session 00000000c8eb37bf 18 -> 19
[ 1280.531995] ceph: mdsc con_get 00000000c8eb37bf ok (19)
[ 1280.531999] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1280.531999] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1280.532000] ceph: mdsc con_put 00000000c8eb37bf (19)
[ 1280.532000] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1280.532006] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1280.532006] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1280.532007] ceph: mdsc con_put 00000000c8eb37bf (19)
[ 1280.532007] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1280.532012] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1280.532012] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1280.532013] ceph: handle_reply 00000000de20b748
[ 1280.532014] ceph: __unregister_request 00000000de20b748 tid 105
[ 1280.532015] ceph: got safe reply 105, mds0
[ 1280.532016] ceph: mdsc con_put 00000000c8eb37bf (19)
[ 1280.532017] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1280.532017] ceph: mdsc con_put 00000000c8eb37bf (18)
[ 1280.532018] ceph: mdsc put_session 00000000c8eb37bf 19 -> 18
[ 1280.532019] ceph: put_cap_refs 000000007dbf994a had p
[ 1280.532020] ceph: ceph_d_prune 00000000c2665322
[ 1280.532042] ceph: mdsc get_session 00000000c8eb37bf 18 -> 19
[ 1280.532043] ceph: mdsc con_get 00000000c8eb37bf ok (19)
[ 1280.532187] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1280.532187] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1280.532188] ceph: mdsc con_put 00000000c8eb37bf (19)
[ 1280.532188] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1280.532190] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1280.532191] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1280.532191] ceph: mdsc con_put 00000000c8eb37bf (19)
[ 1280.532192] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1280.532194] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1280.532195] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1280.532195] ceph: mdsc con_put 00000000c8eb37bf (19)
[ 1280.532196] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1280.532198] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1280.532198] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1280.532199] ceph: mdsc con_put 00000000c8eb37bf (19)
[ 1280.532199] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1280.532202] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1280.532202] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1280.532203] ceph: mdsc con_put 00000000c8eb37bf (19)
[ 1280.532203] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1280.532205] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1280.532206] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1280.532206] ceph: mdsc con_put 00000000c8eb37bf (19)
[ 1280.532207] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1280.957403] ceph: mdsc delayed_work
[ 1280.957404] ceph: check_delayed_caps
[ 1280.957405] ceph: lookup_mds_session 00000000c8eb37bf 19
[ 1280.957406] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1280.957408] ceph: send_renew_caps to mds0 (up:active)
[ 1280.957409] ceph: mdsc get_session 00000000c8eb37bf 20 -> 21
[ 1280.957410] ceph: mdsc con_get 00000000c8eb37bf ok (21)
[ 1280.957410] ceph: mdsc get_session 00000000c8eb37bf 21 -> 22
[ 1280.957411] ceph: mdsc con_get 00000000c8eb37bf ok (22)
[ 1280.957412] ceph: mdsc con_put 00000000c8eb37bf (21)
[ 1280.957412] ceph: mdsc put_session 00000000c8eb37bf 22 -> 21
[ 1280.957413] ceph: mdsc put_session 00000000c8eb37bf 21 -> 20
[ 1284.592212] ceph: destroy_inode 00000000fdca32d0 ino 100001ed77f.fffffffffffffffe
[ 1284.592214] ceph: __ceph_remove_cap 0000000054a99657 from 00000000fdca32d0
[ 1284.592216] ceph: put_snap_realm 1 000000008b101794 2061 -> 2060
[ 1284.592217] ceph: __cap_delay_cancel 00000000fdca32d0
[ 1284.592218] ceph: __ceph_destroy_xattrs p= (null)
[ 1284.592219] ceph: d_release 00000000c2665322
[ 1284.592221] ceph: dentry_lru_del 00000000c348e53a 00000000c2665322 'file1.txt'
[ 1284.592223] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1284.592224] ceph: unreserve caps ctx=00000000134ad6f9 count=0
[ 1284.592226] ceph: mdsc con_put 00000000c8eb37bf (18)
[ 1284.592227] ceph: mdsc put_session 00000000c8eb37bf 19 -> 18
[ 1284.592234] ceph: mdsc get_session 00000000c8eb37bf 18 -> 19
[ 1284.592235] ceph: mdsc con_get 00000000c8eb37bf ok (19)
[ 1284.592240] ceph: handle_caps from mds0
[ 1284.592242] ceph: op grant ino 100001ed780.fffffffffffffffe inode 00000000bef38140
[ 1284.592243] ceph: mds0 seq 1200 cap seq 86
[ 1284.592245] ceph: __ceph_caps_issued 00000000bef38140 cap 0000000019e816a1 issued pFc
[ 1284.592246] ceph: handle_cap_grant inode 00000000bef38140 cap 0000000019e816a1 mds0 seq 86 pAsLsXsFscr
[ 1284.592247] ceph: size 107374182400 max_size 0, i_size 107374182400
[ 1284.592248] ceph: __ceph_caps_issued 00000000bef38140 cap 0000000019e816a1 issued pFc
[ 1284.592249] ceph: 00000000bef38140 mode 0100644 uid.gid 0.0
[ 1284.592251] ceph: my wanted = Fc, used = Fc, dirty -
[ 1284.592252] ceph: mds wanted LsFc -> Fc
[ 1284.592253] ceph: grant: pFc -> pAsLsXsFscr
[ 1284.592253] ceph: invalidate_aliases inode 00000000bef38140
[ 1284.592254] ceph: mdsc con_put 00000000c8eb37bf (18)
[ 1284.592255] ceph: mdsc put_session 00000000c8eb37bf 19 -> 18
[ 1284.592256] ceph: mdsc get_session 00000000c8eb37bf 18 -> 19
[ 1284.592257] ceph: mdsc con_get 00000000c8eb37bf ok (19)
[ 1284.592259] ceph: handle_reply 00000000db0c7bcc
[ 1284.592259] ceph: __unregister_request 00000000db0c7bcc tid 106
[ 1284.592260] ceph: got safe reply 106, mds0
[ 1284.592261] ceph: mdsc con_put 00000000c8eb37bf (18)
[ 1284.592262] ceph: mdsc put_session 00000000c8eb37bf 19 -> 18
[ 1284.592263] ceph: mdsc con_put 00000000c8eb37bf (17)
[ 1284.592263] ceph: mdsc put_session 00000000c8eb37bf 18 -> 17
[ 1284.592264] ceph: put_cap_refs 000000007dbf994a had p
[ 1284.592265] ceph: ceph_d_prune 0000000063d6dbc8
[ 1287.101272] ceph: mdsc delayed_work
[ 1287.101274] ceph: check_delayed_caps
[ 1287.101275] ceph: check_delayed_caps on 000000007dbf994a
[ 1287.101279] ceph: __ceph_caps_issued 000000007dbf994a cap 000000001044692e issued pAsLsXsFs
[ 1287.101280] ceph: check_caps 000000007dbf994a file_want p used p dirty - flushing - issued pAsLsXsFs revoking - retain pAsxLsxXsxFsxcrwbl NODELAY
[ 1287.101282] ceph: mds0 cap 000000001044692e used p issued pAsLsXsFs implemented pAsLsXsFs revoking -
[ 1287.101284] ceph: __send_cap 000000007dbf994a cap 000000001044692e session 00000000c8eb37bf pAsLsXsFs -> pAsLsXsFs (revoking -)
[ 1287.101287] ceph: send_cap_msg update 37bd31 100001ed749 caps pAsLsXsFs wanted p dirty - seq 20/19 tid 0/0 mseq 0 follows 0 size 40/0 xattr_ver 18446744072109198672 xattr_len 0
[ 1287.101289] ceph: mdsc get_session 00000000c8eb37bf 17 -> 18
[ 1287.101290] ceph: mdsc con_get 00000000c8eb37bf ok (18)
[ 1287.101291] ceph: __ceph_caps_issued 000000007dbf994a cap 000000001044692e issued pAsLsXsFs
[ 1287.101292] ceph: check_caps 000000007dbf994a file_want p used p dirty - flushing - issued pAsLsXsFs revoking - retain pAsxLsxXsxFsxcrwbl NODELAY
[ 1287.101293] ceph: lookup_mds_session 00000000c8eb37bf 18
[ 1287.101293] ceph: mdsc get_session 00000000c8eb37bf 18 -> 19
[ 1287.101295] ceph: put_cap 0000000054a99657 2096 = 2061 used + 35 resv + 0 avail
[ 1287.101296] ceph: send_cap_releases mds0 00000000f1f253c8
[ 1287.101297] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1287.101297] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1287.101298] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1288.648118] ceph: destroy_inode 00000000bef38140 ino 100001ed780.fffffffffffffffe
[ 1288.648121] ceph: __ceph_remove_cap 0000000019e816a1 from 00000000bef38140
[ 1288.648123] ceph: put_snap_realm 1 000000008b101794 2060 -> 2059
[ 1288.648124] ceph: __cap_delay_cancel 00000000bef38140
[ 1288.648125] ceph: __ceph_destroy_xattrs p= (null)
[ 1288.648126] ceph: d_release 0000000063d6dbc8
[ 1288.648128] ceph: dentry_lru_del 0000000041d9b9ba 0000000063d6dbc8 'file2.txt'
[ 1288.648130] ceph: mdsc put_session 00000000c8eb37bf 19 -> 18
[ 1288.648131] ceph: unreserve caps ctx=0000000023d04527 count=0
[ 1288.648133] ceph: mdsc con_put 00000000c8eb37bf (17)
[ 1288.648133] ceph: mdsc put_session 00000000c8eb37bf 18 -> 17
[ 1288.648139] ceph: mdsc get_session 00000000c8eb37bf 17 -> 18
[ 1288.648140] ceph: mdsc con_get 00000000c8eb37bf ok (18)
[ 1288.648144] ceph: handle_caps from mds0
[ 1288.648147] ceph: op grant ino 100001ed781.fffffffffffffffe inode 00000000211593be
[ 1288.648148] ceph: mds0 seq 1201 cap seq 81
[ 1288.648149] ceph: __ceph_caps_issued 00000000211593be cap 00000000651e8193 issued pFc
[ 1288.648151] ceph: handle_cap_grant inode 00000000211593be cap 00000000651e8193 mds0 seq 81 pAsLsXsFscr
[ 1288.648152] ceph: size 107374182400 max_size 0, i_size 107374182400
[ 1288.648152] ceph: __ceph_caps_issued 00000000211593be cap 00000000651e8193 issued pFc
[ 1288.648154] ceph: 00000000211593be mode 0100644 uid.gid 0.0
[ 1288.648155] ceph: my wanted = Fc, used = Fc, dirty -
[ 1288.648156] ceph: mds wanted LsFc -> Fc
[ 1288.648157] ceph: grant: pFc -> pAsLsXsFscr
[ 1288.648158] ceph: invalidate_aliases inode 00000000211593be
[ 1288.648159] ceph: mdsc con_put 00000000c8eb37bf (17)
[ 1288.648160] ceph: mdsc put_session 00000000c8eb37bf 18 -> 17
[ 1288.648161] ceph: mdsc get_session 00000000c8eb37bf 17 -> 18
[ 1288.648162] ceph: mdsc con_get 00000000c8eb37bf ok (18)
[ 1288.648163] ceph: handle_reply 000000003731a03d
[ 1288.648165] ceph: __unregister_request 000000003731a03d tid 107
[ 1288.648166] ceph: got safe reply 107, mds0
[ 1288.648167] ceph: mdsc con_put 00000000c8eb37bf (17)
[ 1288.648167] ceph: mdsc put_session 00000000c8eb37bf 18 -> 17
[ 1288.648168] ceph: mdsc con_put 00000000c8eb37bf (16)
[ 1288.648169] ceph: mdsc put_session 00000000c8eb37bf 17 -> 16
[ 1288.648170] ceph: put_cap_refs 000000007dbf994a had p
[ 1288.648171] ceph: ceph_d_prune 0000000053587378
[ 1292.680151] ceph: destroy_inode 00000000211593be ino 100001ed781.fffffffffffffffe
[ 1292.680153] ceph: __ceph_remove_cap 00000000651e8193 from 00000000211593be
[ 1292.680155] ceph: put_snap_realm 1 000000008b101794 2059 -> 2058
[ 1292.680156] ceph: __cap_delay_cancel 00000000211593be
[ 1292.680157] ceph: __ceph_destroy_xattrs p= (null)
[ 1292.680158] ceph: d_release 0000000053587378
[ 1292.680159] ceph: dentry_lru_del 00000000dc920853 0000000053587378 'file3.txt'
[ 1292.680161] ceph: mdsc put_session 00000000c8eb37bf 16 -> 15
[ 1292.680162] ceph: unreserve caps ctx=000000006c55643b count=0
[ 1292.680164] ceph: mdsc con_put 00000000c8eb37bf (14)
[ 1292.680165] ceph: mdsc put_session 00000000c8eb37bf 15 -> 14
[ 1292.680170] ceph: mdsc get_session 00000000c8eb37bf 14 -> 15
[ 1292.680170] ceph: mdsc con_get 00000000c8eb37bf ok (15)
[ 1292.680175] ceph: handle_caps from mds0
[ 1292.680178] ceph: op grant ino 100001ed782.fffffffffffffffe inode 00000000dc4173bc
[ 1292.680178] ceph: mds0 seq 1202 cap seq 79
[ 1292.680180] ceph: __ceph_caps_issued 00000000dc4173bc cap 00000000655c934a issued pFc
[ 1292.680181] ceph: handle_cap_grant inode 00000000dc4173bc cap 00000000655c934a mds0 seq 79 pAsLsXsFscr
[ 1292.680182] ceph: size 107374182400 max_size 0, i_size 107374182400
[ 1292.680183] ceph: __ceph_caps_issued 00000000dc4173bc cap 00000000655c934a issued pFc
[ 1292.680184] ceph: 00000000dc4173bc mode 0100644 uid.gid 0.0
[ 1292.680186] ceph: my wanted = Fc, used = Fc, dirty -
[ 1292.680187] ceph: mds wanted LsFc -> Fc
[ 1292.680188] ceph: grant: pFc -> pAsLsXsFscr
[ 1292.680189] ceph: invalidate_aliases inode 00000000dc4173bc
[ 1292.680190] ceph: mdsc con_put 00000000c8eb37bf (14)
[ 1292.680191] ceph: mdsc put_session 00000000c8eb37bf 15 -> 14
[ 1292.680192] ceph: mdsc get_session 00000000c8eb37bf 14 -> 15
[ 1292.680192] ceph: mdsc con_get 00000000c8eb37bf ok (15)
[ 1292.680194] ceph: handle_reply 0000000075473824
[ 1292.680195] ceph: __unregister_request 0000000075473824 tid 108
[ 1292.680197] ceph: got safe reply 108, mds0
[ 1292.680198] ceph: mdsc con_put 00000000c8eb37bf (14)
[ 1292.680198] ceph: mdsc put_session 00000000c8eb37bf 15 -> 14
[ 1292.680199] ceph: mdsc con_put 00000000c8eb37bf (13)
[ 1292.680199] ceph: mdsc put_session 00000000c8eb37bf 14 -> 13
[ 1292.680201] ceph: put_cap_refs 000000007dbf994a had p
[ 1292.680202] ceph: ceph_d_prune 00000000a0a3347f
[ 1292.733086] ceph: mdsc delayed_work
[ 1292.733087] ceph: check_delayed_caps
[ 1292.733089] ceph: lookup_mds_session 00000000c8eb37bf 13
[ 1292.733089] ceph: mdsc get_session 00000000c8eb37bf 13 -> 14
[ 1292.733093] ceph: put_cap 0000000019e816a1 2096 = 2060 used + 35 resv + 1 avail
[ 1292.733094] ceph: put_cap 00000000651e8193 2096 = 2059 used + 35 resv + 2 avail
[ 1292.733095] ceph: send_cap_releases mds0 00000000f77df325
[ 1292.733096] ceph: mdsc get_session 00000000c8eb37bf 14 -> 15
[ 1292.733096] ceph: mdsc con_get 00000000c8eb37bf ok (15)
[ 1292.733097] ceph: mdsc put_session 00000000c8eb37bf 15 -> 14
[ 1296.739311] ceph: destroy_inode 00000000dc4173bc ino 100001ed782.fffffffffffffffe
[ 1296.739312] ceph: __ceph_remove_cap 00000000655c934a from 00000000dc4173bc
[ 1296.739314] ceph: put_snap_realm 1 000000008b101794 2058 -> 2057
[ 1296.739315] ceph: __cap_delay_cancel 00000000dc4173bc
[ 1296.739316] ceph: __ceph_destroy_xattrs p= (null)
[ 1296.739317] ceph: d_release 00000000a0a3347f
[ 1296.739318] ceph: dentry_lru_del 00000000b8d3da88 00000000a0a3347f 'file4.txt'
[ 1296.739319] ceph: mdsc put_session 00000000c8eb37bf 14 -> 13
[ 1296.739320] ceph: unreserve caps ctx=000000001f1693a7 count=0
[ 1296.739322] ceph: mdsc con_put 00000000c8eb37bf (12)
[ 1296.739322] ceph: mdsc put_session 00000000c8eb37bf 13 -> 12
[ 1296.739327] ceph: mdsc get_session 00000000c8eb37bf 12 -> 13
[ 1296.739328] ceph: mdsc con_get 00000000c8eb37bf ok (13)
[ 1296.739332] ceph: handle_reply 000000004f565148
[ 1296.739332] ceph: __unregister_request 000000004f565148 tid 110
[ 1296.739333] ceph: handle_reply tid 110 result 0
[ 1296.739335] ceph: update_snap_trace deletion=0
[ 1296.739336] ceph: lookup_snap_realm 1 000000008b101794
[ 1296.739337] ceph: get_realm 000000008b101794 2057 -> 2058
[ 1296.739338] ceph: update_snap_trace 1 000000008b101794 seq 1 unchanged
[ 1296.739339] ceph: done with 1 000000008b101794, invalidated=0, 00000000d291aa69 00000000d291aa69
[ 1296.739341] ceph: fill_trace 000000004f565148 is_dentry 0 is_target 1
[ 1296.739343] ceph: get_inode on 1099513648969=100001ed749.fffffffffffffffe got 000000007dbf994a
[ 1296.739344] ceph: fill_inode 000000007dbf994a ino 100001ed749.fffffffffffffffe v 9648574 had 9648555
[ 1296.739345] ceph: get_cap ctx=000000008187b867 (35) 2096 = 2058 used + 35 resv + 3 avail
[ 1296.739346] ceph: __ceph_caps_issued 000000007dbf994a cap 000000001044692e issued pAsLsXsFs
[ 1296.739347] ceph: 000000007dbf994a mode 040755 uid.gid 0.0
[ 1296.739349] ceph: add_cap 000000007dbf994a mds0 cap 37bd31 pAsLsXsFs seq 21
[ 1296.739349] ceph: __ceph_caps_issued 000000007dbf994a cap 000000001044692e issued pAsLsXsFs
[ 1296.739351] ceph: issued pAsLsXsFs, mds wanted pAsLsXsFsx, actual p, queueing
[ 1296.739352] ceph: __cap_set_timeouts 000000007dbf994a min 873 max 873
[ 1296.739352] ceph: __cap_delay_requeue 000000007dbf994a flags 0 at 4295964943
[ 1296.739354] ceph: add_cap inode 000000007dbf994a (100001ed749.fffffffffffffffe) cap 000000001044692e pAsLsXsFs now pAsLsXsFs seq 21 mds0
[ 1296.739355] ceph: put_cap 00000000651e8193 2096 = 2059 used + 34 resv + 3 avail
[ 1296.739356] ceph: fill_trace done err=0
[ 1296.739357] ceph: readdir_prepopulate 0 items under dn 0000000010f74030
[ 1296.739357] ceph: readdir_prepopulate done
[ 1296.739358] ceph: unreserve caps ctx=000000008187b867 count=34
[ 1296.739359] ceph: unreserve caps 2096 = 2058 used + 0 resv + 38 avail
[ 1296.739359] ceph: put_snap_realm 1 000000008b101794 2058 -> 2057
[ 1296.739363] ceph: mdsc get_session 00000000c8eb37bf 13 -> 14
[ 1296.739363] ceph: mdsc con_get 00000000c8eb37bf ok (14)
[ 1296.739365] ceph: handle_caps from mds0
[ 1296.739367] ceph: op grant ino 100001ed749.fffffffffffffffe inode 000000007dbf994a
[ 1296.739367] ceph: mds0 seq 1203 cap seq 22
[ 1296.739368] ceph: __ceph_caps_issued 000000007dbf994a cap 000000001044692e issued pAsLsXsFs
[ 1296.739369] ceph: handle_cap_grant inode 000000007dbf994a cap 000000001044692e mds0 seq 22 pAsLsXsFsx
[ 1296.739370] ceph: size 0 max_size 0, i_size 40
[ 1296.739371] ceph: __ceph_caps_issued 000000007dbf994a cap 000000001044692e issued pAsLsXsFs
[ 1296.739371] ceph: 000000007dbf994a mode 040755 uid.gid 0.0
[ 1296.739373] ceph: my wanted = p, used = p, dirty -
[ 1296.739373] ceph: mds wanted pAsLsXsFsx -> p
[ 1296.739374] ceph: grant: pAsLsXsFs -> pAsLsXsFsx
[ 1296.739375] ceph: mdsc con_put 00000000c8eb37bf (13)
[ 1296.739375] ceph: mdsc put_session 00000000c8eb37bf 14 -> 13
[ 1296.739377] ceph: mdsc get_session 00000000c8eb37bf 13 -> 14
[ 1296.739377] ceph: mdsc con_get 00000000c8eb37bf ok (14)
[ 1296.739378] ceph: handle_caps from mds0
[ 1296.739379] ceph: op grant ino 100001ed783.fffffffffffffffe inode 000000006e7794e2
[ 1296.739380] ceph: mds0 seq 1204 cap seq 81
[ 1296.739381] ceph: __ceph_caps_issued 000000006e7794e2 cap 00000000e4d0a581 issued pFc
[ 1296.739381] ceph: handle_cap_grant inode 000000006e7794e2 cap 00000000e4d0a581 mds0 seq 81 pAsLsXsFscr
[ 1296.739382] ceph: size 107374182400 max_size 0, i_size 107374182400
[ 1296.739383] ceph: __ceph_caps_issued 000000006e7794e2 cap 00000000e4d0a581 issued pFc
[ 1296.739383] ceph: 000000006e7794e2 mode 0100644 uid.gid 0.0
[ 1296.739384] ceph: my wanted = Fc, used = Fc, dirty -
[ 1296.739384] ceph: mds wanted LsFc -> Fc
[ 1296.739385] ceph: grant: pFc -> pAsLsXsFscr
[ 1296.739386] ceph: invalidate_aliases inode 000000006e7794e2
[ 1296.739387] ceph: mdsc con_put 00000000c8eb37bf (13)
[ 1296.739387] ceph: mdsc put_session 00000000c8eb37bf 14 -> 13
[ 1296.739389] ceph: mdsc get_session 00000000c8eb37bf 13 -> 14
[ 1296.739389] ceph: mdsc con_get 00000000c8eb37bf ok (14)
[ 1296.739391] ceph: handle_reply 000000003cc99a51
[ 1296.739391] ceph: __unregister_request 000000003cc99a51 tid 109
[ 1296.739392] ceph: got safe reply 109, mds0
[ 1296.739393] ceph: mdsc con_put 00000000c8eb37bf (13)
[ 1296.739394] ceph: mdsc put_session 00000000c8eb37bf 14 -> 13
[ 1296.739394] ceph: mdsc con_put 00000000c8eb37bf (12)
[ 1296.739395] ceph: mdsc put_session 00000000c8eb37bf 13 -> 12
[ 1296.739396] ceph: put_cap_refs 000000007dbf994a had p
[ 1296.739397] ceph: ceph_d_prune 00000000780bc6fa
[ 1296.739419] ceph: do_request waited, got 0
[ 1296.739421] ceph: do_request 000000004f565148 done, result 0
[ 1296.739421] ceph: readdir got and parsed readdir result=0 on frag 0, end=1, complete=1, hash_order=1
[ 1296.739422] ceph: readdir frag 0 num 0 pos 2 chunk first 0
[ 1296.739424] ceph: mdsc con_put 00000000c8eb37bf (11)
[ 1296.739425] ceph: mdsc put_session 00000000c8eb37bf 12 -> 11
[ 1296.739426] ceph: mdsc con_put 00000000c8eb37bf (10)
[ 1296.739427] ceph: mdsc put_session 00000000c8eb37bf 11 -> 10
[ 1296.739429] ceph: put_cap_refs 000000007dbf994a had p
[ 1296.739430] ceph: mdsc put_session 00000000c8eb37bf 10 -> 9
[ 1296.739431] ceph: unreserve caps ctx=000000008187b867 count=0
[ 1296.739432] ceph: marking 000000007dbf994a complete and ordered
[ 1296.739433] ceph: readdir 000000007dbf994a file 00000000fa762655 done.
[ 1296.739436] ceph: readdir 000000007dbf994a file 00000000fa762655 pos 2
[ 1296.739439] ceph: release inode 000000007dbf994a file 00000000fa762655
[ 1296.739441] ceph: put_fmode 000000007dbf994a fmode 0 {0,0,0,0}
[ 1296.739442] ceph: __ceph_caps_issued 000000007dbf994a cap 000000001044692e issued pAsLsXsFsx
[ 1296.739444] ceph: check_caps 000000007dbf994a file_want - used - dirty - flushing - issued pAsLsXsFsx revoking - retain pAsLsXsFsx
[ 1296.739445] ceph: mds0 cap 000000001044692e used - issued pAsLsXsFsx implemented pAsLsXsFsx revoking -
[ 1296.739447] ceph: delaying issued pAsLsXsFsx -> pAsLsXsFsx, wanted pAsLsXsFsx -> AsLsXsFsx
[ 1296.739448] ceph: __cap_set_timeouts 000000007dbf994a min 864 max 864
[ 1296.739449] ceph: __cap_delay_requeue 000000007dbf994a flags 0 at 4295964934
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
