Hello all,
I have a problem with my CephFS that I’m stumped on. I recently had to rebuild a node who’s system disk failed. Once I did that, I re-created the osd directory structure in /var/lib/ceph/osd and the osds came back into the cluster, then had to backfill. However, I now have the problem that writing more than a few MB into the filesystem causes the node (I’ve tried it on 6 different nodes now,) to reboot. Note that this did not happen until I brought the osds back into the cluster. Shutting them down (or that entire node,) doesn’t fix the problem. Updating to the latest (Centos7) kernel didn’t fix the problem. This happens with both the kernel and fuse mounts, across several data pools, including a newly created data pool. I’m running 14.2.6.
Sometimes I get the following message just before the system reboots (while running an rsync job):
Message from syslogd@admin2 at Jan 24 17:21:07 ...
kernel:NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [rsync:4399]
after failing during write to kernel mounted FS, dmesg has (in the abrt crash report):
[ 243.911318] ------------[ cut here ]------------
[ 243.911350] kernel BUG at mm/slub.c:3772!
[ 243.911368] invalid opcode: 0000 [#1] SMP
[ 243.911390] Modules linked in: fuse nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 ceph nfs libceph lockd dns_resolver grace fscache nf_nat_ftp nf_conntrack_ftp nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ib_isert iscsi_target_mod ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt rpcrdma sunrpc rdma_ucm ib_iser rdma_cm ib_umad iw_cm ib_ipoib libiscsi scsi_transport_iscsi
[ 243.911780] ib_cm iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass mlx4_ib crc32_pclmul ghash_clmulni_intel ib_uverbs aesni_intel lrw gf128mul ib_core glue_helper ablk_helper cryptd pcspkr joydev lpc_ich i2c_i801 sg mei_me mei ioatdma wmi ipmi_ssif ipmi_si ipmi_devintf ipmi_msghandler pcc_cpufreq acpi_pad acpi_power_meter ip_tables xfs libcrc32c mlx4_en sd_mod crc_t10dif crct10dif_generic ast drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mlx4_core drm ahci igb libahci crct10dif_pclmul crct10dif_common libata crc32c_intel ptp pps_core devlink drm_panel_orientation_quirks dca i2c_algo_bit dm_mirror dm_region_hash dm_log dm_mod
[ 243.912143] CPU: 1 PID: 0 Comm: swapper/1 Kdump: loaded Not tainted 3.10.0-1062.4.3.el7.x86_64 #1
[ 243.912177] Hardware name: Supermicro SYS-6018R-WTR/X10DRW-i, BIOS 2.0b 04/13/2017
[ 243.912207] task: ffff96b7e95a20e0 ti: ffff96b7e95b0000 task.ti: ffff96b7e95b0000
[ 243.912236] RIP: 0010:[<ffffffffb322355c>] [<ffffffffb322355c>] kfree+0x13c/0x140
[ 243.912271] RSP: 0018:ffff96c6bf843d60 EFLAGS: 00010246
[ 243.912293] RAX: 002fffff00000000 RBX: ffff96c6aaa60000 RCX: 0000000000000ec0
[ 243.912321] RDX: 002fffff00000000 RSI: 0000000000000001 RDI: ffff96c6aaa60000
[ 243.912349] RBP: ffff96c6bf843d78 R08: ffff96c6aaa60000 R09: ffff96b7e95b3e60
[ 243.912377] R10: 7fffffffffffffff R11: ffffe81100aa9800 R12: ffff96c6aaa60ec0
[ 243.912405] R13: ffffffffb36376c5 R14: 000000000000003d R15: ffff96c7691186a8
[ 243.912433] FS: 0000000000000000(0000) GS:ffff96c6bf840000(0000) knlGS:0000000000000000
[ 243.912465] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 243.912488] CR2: 000055ba0b02bfb8 CR3: 000000203972e000 CR4: 00000000003607e0
[ 243.912516] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 243.912544] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 243.912572] Call Trace:
[ 243.912584] <IRQ>
[ 243.912598] [<ffffffffb36376c5>] skb_release_data+0xf5/0x140
[ 243.912625] [<ffffffffb3637734>] skb_release_all+0x24/0x30
[ 243.912650] [<ffffffffb3637c1c>] consume_skb+0x2c/0x90
[ 243.912674] [<ffffffffb364cb9d>] __dev_kfree_skb_any+0x3d/0x50
[ 243.912705] [<ffffffffc0ad8bd5>] ipoib_ib_handle_tx_wc+0x75/0x1d0 [ib_ipoib]
[ 243.912737] [<ffffffffc0ad8f3a>] ipoib_tx_poll+0x9a/0x100 [ib_ipoib]
[ 243.912765] [<ffffffffb364f57f>] net_rx_action+0x26f/0x390
[ 243.912790] [<ffffffffb30a5305>] __do_softirq+0xf5/0x280
[ 243.912815] [<ffffffffb379042c>] call_softirq+0x1c/0x30
[ 243.912840] [<ffffffffb302f715>] do_softirq+0x65/0xa0
[ 243.912862] [<ffffffffb30a5685>] irq_exit+0x105/0x110
[ 243.912884] [<ffffffffb3791886>] do_IRQ+0x56/0xf0
[ 243.913860] [<ffffffffb378336a>] common_interrupt+0x16a/0x16a
[ 243.914841] <EOI>
[ 243.914853] [<ffffffffb35c10f7>] ? cpuidle_enter_state+0x57/0xd0
[ 243.916792] [<ffffffffb35c124e>] cpuidle_idle_call+0xde/0x230
[ 243.917768] [<ffffffffb3037c6e>] arch_cpu_idle+0xe/0xc0
[ 243.918731] [<ffffffffb3100d3a>] cpu_startup_entry+0x14a/0x1e0
[ 243.919701] [<ffffffffb305a0b7>] start_secondary+0x1f7/0x270
[ 243.920650] [<ffffffffb30000d5>] start_cpu+0x5/0x14
[ 243.921567] Code: 49 8b 03 31 f6 f6 c4 40 74 04 41 8b 73 68 4c 89 df e8 89 2f fa ff eb 84 4c 8b 58 30 48 8b 10 80 e6 80 4c 0f 44 d8 e9 28 ff ff ff <0f> 0b 66 90 0f 1f 44 00 00 55 48 89 e5 41 57 41 56 41 55 41 54
[ 243.923534] RIP [<ffffffffb322355c>] kfree+0x13c/0x140
[ 243.924435] RSP <ffff96c6bf843d60>
Just before the crash, I get the following in my mds log (level 10):
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.server straydn is [dentry #0x100/stray7/10003178634 [2,head] auth NULL (dversion lock) pv=0 v=129495990 ino=(nil) state=1342177345|new|bottomlru | request=1 0x55d0053208c0]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.server witnesses , have
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker acquire_locks request(client.77200959:238582 cr=0x55d019672580)
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker must authpin [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json [2,head] auth (dversion lock) v=13929147 ino=0x10003178634 state=1610612736 | request=0 lock=0 inodepin=1 dirty=1 waiter=0 authpin=0 0x55cfeeef3dc0]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker must authpin [inode 0x607 [...2,head] ~mds0/stray7/ auth v89339680 f(v40 m2020-01-24 16:08:43.778298 211=160+51) n(v130 rc2020-01-24 16:08:43.778298 b81704593 212=160+52) (inest lock) (iversion lock) | dirtyscattered=0 lock=0 dirfrag=1 stickydirs=1 stray=1 dirtyrstat=0 dirty=1 authpin=0 0x55cfeff6f800]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker must authpin [inode 0x1 [...2,head] / auth v34216266 snaprealm=0x55cfeff4a280 f(v0 m2020-01-17 12:41:39.137684 5=0+5) n(v7229910 rc2020-01-24 16:08:38.595298 b441687361105836 23052011=22719256+332755)/n(v0 1=0+1) (inest lock dirty) (iversion lock) caps={48654103=pAsLsXsFs/-@0,49033239=pAsLsXsFs/-@0,68004815=pAsLsXsFs/-@4,77200959=pAsLsXsFs/-@0,77246902=pAsLsXsFs/-@0} | dirtyscattered=1 lock=0 dirfrag=1 caps=1 openingsnapparents=0 dirty=1 authpin=0 0x55cfeff94700]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker must authpin [inode 0x10000000001 [...2,head] /home/ auth v98722180 f(v0 m2020-01-20 09:56:37.772033 65=0+65) n(v7735020 rc2020-01-24 16:08:47.195918 b164276032854114 16350398=16221677+128721) (inest lock dirty) (iversion lock) caps={49033239=pAsLsXs/-@0,75454214=pAsLsXsFs/-@4,76406416=pAsLsXsFs/-@0,76407061=pAsLsXsFs/-@0,76407436=pAsLsXsFs/-@0,77200959=pAsLsXsFs/-@0,77246902=pAsLsXsFs/AsLsXsFsx@4,77586859=pAsLsXsFs/-@0,77605502=pAsLsXsFs/-@2,77643571=pAsLsXsFs/-@1} | dirtyscattered=1 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=0 0x55cfeff94e00]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker must authpin [inode 0x10000000002 [...2,head] /home/tjr22/ auth v131045958 f(v31 m2020-01-24 16:08:21.091241 189=91+98) n(v765289 rc2020-01-24 16:08:47.195918 b8869319870466 1292992=1236041+56951) (inest lock dirty) (iversion lock) caps={49033239=pAsLsXs/-@0,75454214=pAsLsXsFs/-@1513,77200959=pAsLsXsFs/-@327,77246902=pAsLsXs/-@2,77605502=pAsLsXs/p@50} | dirtyscattered=1 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 waiter=0 authpin=0 0x55cfeff95500]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker must authpin [inode 0x10000000a52 [...2,head] /home/tjr22/.config/ auth v129496029 f(v0 m2019-12-13 13:57:58.591154 31=6+25) n(v440444 rc2020-01-24 16:08:47.195918 b336426 108=63+45) (inest lock dirty) (iversion lock) caps={49033239=pAsLsXs/-@0,75454214=pAsLsXsFs/-@69,77200959=pAsLsXsFs/-@64,77605502=pAsLsXsFs/-@2} | dirtyscattered=1 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=0 0x55cfeff95c00]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker must authpin [inode 0x1000268e7cb [...2,head] /home/tjr22/.config/configstore/ auth v15636199 f(v0 m2020-01-24 16:08:47.195918 2=2+0) n(v8 rc2020-01-24 16:08:47.246973 b700 3=2+1) (inest lock) (ifile excl) (iversion lock) caps={77200959=pAsLsXsFsx/-@30413},l=77200959 | dirtyscattered=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=0 0x55cfeff96300]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker must authpin [dentry #0x100/stray7/10003178634 [2,head] auth NULL (dversion lock) pv=0 v=129495990 ino=(nil) state=1342177345|new|bottomlru | request=1 0x55d0053208c0]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker must authpin [inode 0x10003178634 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json auth v13929147 dirtyparent s=350 n(v0 rc2020-01-24 16:08:43.778298 b350 1=1+0)/n(v0 rc2020-01-24 16:08:43.773910 b350 1=1+0) (iversion lock) caps={77200959=pAsXsFscr/-@6} | ptrwaiter=0 request=0 lock=0 caps=1 dirtyparent=1 dirty=1 authpin=0 0x55d006c3a000]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker must authpin [inode 0x10003178635 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 auth v13929158 dirtyparent s=350 n(v0 rc2020-01-24 16:08:47.246973 b350 1=1+0) (iversion lock) caps={77200959=pAsXsFscr/-@4} | ptrwaiter=0 request=0 lock=0 caps=1 dirtyparent=1 dirty=1 authpin=0 0x55d006c3b500]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker must authpin [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 [2,head] auth (dversion lock) v=13929158 ino=0x10003178635 state=1610612736 | request=0 lock=0 inodepin=1 dirty=1 authpin=0 0x55d018f10000]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker auth_pinning [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json [2,head] auth (dversion lock) v=13929147 ino=0x10003178634 state=1610612736 | request=0 lock=0 inodepin=1 dirty=1 waiter=0 authpin=0 0x55cfeeef3dc0]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.den(0x1000268e7cb tus-node-server-0.1.1.json) auth_pin by 0x55d01a129400 on [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json [2,head] auth (dversion lock) v=13929147 ap=1 ino=0x10003178634 state=1610612736 | request=0 lock=0 inodepin=1 dirty=1 waiter=0 authpin=1 0x55cfeeef3dc0] now 1
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker auth_pinning [inode 0x607 [...2,head] ~mds0/stray7/ auth v89339680 f(v40 m2020-01-24 16:08:43.778298 211=160+51) n(v130 rc2020-01-24 16:08:43.778298 b81704593 212=160+52) (inest lock) (iversion lock) | dirtyscattered=0 lock=0 dirfrag=1 stickydirs=1 stray=1 dirtyrstat=0 dirty=1 authpin=0 0x55cfeff6f800]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.ino(0x607) auth_pin by 0x55d01a129400 on [inode 0x607 [...2,head] ~mds0/stray7/ auth v89339680 ap=1 f(v40 m2020-01-24 16:08:43.778298 211=160+51) n(v130 rc2020-01-24 16:08:43.778298 b81704593 212=160+52) (inest lock) (iversion lock) | dirtyscattered=0 lock=0 dirfrag=1 stickydirs=1 stray=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff6f800] now 1
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker auth_pinning [inode 0x1 [...2,head] / auth v34216266 snaprealm=0x55cfeff4a280 f(v0 m2020-01-17 12:41:39.137684 5=0+5) n(v7229910 rc2020-01-24 16:08:38.595298 b441687361105836 23052011=22719256+332755)/n(v0 1=0+1) (inest lock dirty) (iversion lock) caps={48654103=pAsLsXsFs/-@0,49033239=pAsLsXsFs/-@0,68004815=pAsLsXsFs/-@4,77200959=pAsLsXsFs/-@0,77246902=pAsLsXsFs/-@0} | dirtyscattered=1 lock=0 dirfrag=1 caps=1 openingsnapparents=0 dirty=1 authpin=0 0x55cfeff94700]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.ino(0x1) auth_pin by 0x55d01a129400 on [inode 0x1 [...2,head] / auth v34216266 ap=1 snaprealm=0x55cfeff4a280 f(v0 m2020-01-17 12:41:39.137684 5=0+5) n(v7229910 rc2020-01-24 16:08:38.595298 b441687361105836 23052011=22719256+332755)/n(v0 1=0+1) (inest lock dirty) (iversion lock) caps={48654103=pAsLsXsFs/-@0,49033239=pAsLsXsFs/-@0,68004815=pAsLsXsFs/-@4,77200959=pAsLsXsFs/-@0,77246902=pAsLsXsFs/-@0} | dirtyscattered=1 lock=0 dirfrag=1 caps=1 openingsnapparents=0 dirty=1 authpin=1 0x55cfeff94700] now 1
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker auth_pinning [inode 0x10000000001 [...2,head] /home/ auth v98722180 f(v0 m2020-01-20 09:56:37.772033 65=0+65) n(v7735020 rc2020-01-24 16:08:47.195918 b164276032854114 16350398=16221677+128721) (inest lock dirty) (iversion lock) caps={49033239=pAsLsXs/-@0,75454214=pAsLsXsFs/-@4,76406416=pAsLsXsFs/-@0,76407061=pAsLsXsFs/-@0,76407436=pAsLsXsFs/-@0,77200959=pAsLsXsFs/-@0,77246902=pAsLsXsFs/AsLsXsFsx@4,77586859=pAsLsXsFs/-@0,77605502=pAsLsXsFs/-@2,77643571=pAsLsXsFs/-@1} | dirtyscattered=1 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=0 0x55cfeff94e00]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.ino(0x10000000001) auth_pin by 0x55d01a129400 on [inode 0x10000000001 [...2,head] /home/ auth v98722180 ap=1 f(v0 m2020-01-20 09:56:37.772033 65=0+65) n(v7735020 rc2020-01-24 16:08:47.195918 b164276032854114 16350398=16221677+128721) (inest lock dirty) (iversion lock) caps={49033239=pAsLsXs/-@0,75454214=pAsLsXsFs/-@4,76406416=pAsLsXsFs/-@0,76407061=pAsLsXsFs/-@0,76407436=pAsLsXsFs/-@0,77200959=pAsLsXsFs/-@0,77246902=pAsLsXsFs/AsLsXsFsx@4,77586859=pAsLsXsFs/-@0,77605502=pAsLsXsFs/-@2,77643571=pAsLsXsFs/-@1} | dirtyscattered=1 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff94e00] now 1
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker auth_pinning [inode 0x10000000002 [...2,head] /home/tjr22/ auth v131045958 f(v31 m2020-01-24 16:08:21.091241 189=91+98) n(v765289 rc2020-01-24 16:08:47.195918 b8869319870466 1292992=1236041+56951) (inest lock dirty) (iversion lock) caps={49033239=pAsLsXs/-@0,75454214=pAsLsXsFs/-@1513,77200959=pAsLsXsFs/-@327,77246902=pAsLsXs/-@2,77605502=pAsLsXs/p@50} | dirtyscattered=1 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 waiter=0 authpin=0 0x55cfeff95500]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.ino(0x10000000002) auth_pin by 0x55d01a129400 on [inode 0x10000000002 [...2,head] /home/tjr22/ auth v131045958 ap=1 f(v31 m2020-01-24 16:08:21.091241 189=91+98) n(v765289 rc2020-01-24 16:08:47.195918 b8869319870466 1292992=1236041+56951) (inest lock dirty) (iversion lock) caps={49033239=pAsLsXs/-@0,75454214=pAsLsXsFs/-@1513,77200959=pAsLsXsFs/-@327,77246902=pAsLsXs/-@2,77605502=pAsLsXs/p@50} | dirtyscattered=1 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 waiter=0 authpin=1 0x55cfeff95500] now 1
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker auth_pinning [inode 0x10000000a52 [...2,head] /home/tjr22/.config/ auth v129496029 f(v0 m2019-12-13 13:57:58.591154 31=6+25) n(v440444 rc2020-01-24 16:08:47.195918 b336426 108=63+45) (inest lock dirty) (iversion lock) caps={49033239=pAsLsXs/-@0,75454214=pAsLsXsFs/-@69,77200959=pAsLsXsFs/-@64,77605502=pAsLsXsFs/-@2} | dirtyscattered=1 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=0 0x55cfeff95c00]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.ino(0x10000000a52) auth_pin by 0x55d01a129400 on [inode 0x10000000a52 [...2,head] /home/tjr22/.config/ auth v129496029 ap=1 f(v0 m2019-12-13 13:57:58.591154 31=6+25) n(v440444 rc2020-01-24 16:08:47.195918 b336426 108=63+45) (inest lock dirty) (iversion lock) caps={49033239=pAsLsXs/-@0,75454214=pAsLsXsFs/-@69,77200959=pAsLsXsFs/-@64,77605502=pAsLsXsFs/-@2} | dirtyscattered=1 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff95c00] now 1
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker auth_pinning [inode 0x1000268e7cb [...2,head] /home/tjr22/.config/configstore/ auth v15636199 f(v0 m2020-01-24 16:08:47.195918 2=2+0) n(v8 rc2020-01-24 16:08:47.246973 b700 3=2+1) (inest lock) (ifile excl) (iversion lock) caps={77200959=pAsLsXsFsx/-@30413},l=77200959 | dirtyscattered=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=0 0x55cfeff96300]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.ino(0x1000268e7cb) auth_pin by 0x55d01a129400 on [inode 0x1000268e7cb [...2,head] /home/tjr22/.config/configstore/ auth v15636199 ap=1 f(v0 m2020-01-24 16:08:47.195918 2=2+0) n(v8 rc2020-01-24 16:08:47.246973 b700 3=2+1) (inest lock) (ifile excl) (iversion lock) caps={77200959=pAsLsXsFsx/-@30413},l=77200959 | dirtyscattered=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff96300] now 1
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker auth_pinning [dentry #0x100/stray7/10003178634 [2,head] auth NULL (dversion lock) pv=0 v=129495990 ino=(nil) state=1342177345|new|bottomlru | request=1 0x55d0053208c0]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.den(0x607 10003178634) auth_pin by 0x55d01a129400 on [dentry #0x100/stray7/10003178634 [2,head] auth NULL (dversion lock) pv=0 v=129495990 ap=1 ino=(nil) state=1342177345|new|bottomlru | request=1 authpin=1 0x55d0053208c0] now 1
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker auth_pinning [inode 0x10003178634 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json auth v13929147 dirtyparent s=350 n(v0 rc2020-01-24 16:08:43.778298 b350 1=1+0)/n(v0 rc2020-01-24 16:08:43.773910 b350 1=1+0) (iversion lock) caps={77200959=pAsXsFscr/-@6} | ptrwaiter=0 request=0 lock=0 caps=1 dirtyparent=1 dirty=1 authpin=0 0x55d006c3a000]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.ino(0x10003178634) auth_pin by 0x55d01a129400 on [inode 0x10003178634 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json auth v13929147 ap=1 dirtyparent s=350 n(v0 rc2020-01-24 16:08:43.778298 b350 1=1+0)/n(v0 rc2020-01-24 16:08:43.773910 b350 1=1+0) (iversion lock) caps={77200959=pAsXsFscr/-@6} | ptrwaiter=0 request=0 lock=0 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3a000] now 1
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker auth_pinning [inode 0x10003178635 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 auth v13929158 dirtyparent s=350 n(v0 rc2020-01-24 16:08:47.246973 b350 1=1+0) (iversion lock) caps={77200959=pAsXsFscr/-@4} | ptrwaiter=0 request=0 lock=0 caps=1 dirtyparent=1 dirty=1 authpin=0 0x55d006c3b500]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.ino(0x10003178635) auth_pin by 0x55d01a129400 on [inode 0x10003178635 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 auth v13929158 ap=1 dirtyparent s=350 n(v0 rc2020-01-24 16:08:47.246973 b350 1=1+0) (iversion lock) caps={77200959=pAsXsFscr/-@4} | ptrwaiter=0 request=0 lock=0 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3b500] now 1
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker auth_pinning [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 [2,head] auth (dversion lock) v=13929158 ino=0x10003178635 state=1610612736 | request=0 lock=0 inodepin=1 dirty=1 authpin=0 0x55d018f10000]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.den(0x1000268e7cb tus-node-server-0.1.1.json.2465308719) auth_pin by 0x55d01a129400 on [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 [2,head] auth (dversion lock) v=13929158 ap=1 ino=0x10003178635 state=1610612736 | request=0 lock=0 inodepin=1 dirty=1 authpin=1 0x55d018f10000] now 1
2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker local_wrlock_start on (dversion lock) on [dentry #0x100/stray7/10003178634 [2,head] auth NULL (dversion lock) pv=0 v=129495990 ap=1 ino=(nil) state=1342177345|new|bottomlru | request=1 authpin=1 0x55d0053208c0]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got wrlock on (dversion lock w=1 last_client=77200959) [dentry #0x100/stray7/10003178634 [2,head] auth NULL (dversion lock w=1 last_client=77200959) pv=0 v=129495990 ap=1 ino=(nil) state=1342177345|new|bottomlru | request=1 lock=1 authpin=1 0x55d0053208c0]
2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker xlock_start on (dn sync) on [dentry #0x100/stray7/10003178634 [2,head] auth NULL (dversion lock w=1 last_client=77200959) pv=0 v=129495990 ap=1 ino=(nil) state=1342177345|new|bottomlru | request=1 lock=1 authpin=1 0x55d0053208c0]
2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker simple_lock on (dn sync) on [dentry #0x100/stray7/10003178634 [2,head] auth NULL (dversion lock w=1 last_client=77200959) pv=0 v=129495990 ap=1 ino=(nil) state=1342177345|new|bottomlru | request=1 lock=1 authpin=1 0x55d0053208c0]
2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker simple_xlock on (dn lock) on [dentry #0x100/stray7/10003178634 [2,head] auth NULL (dn lock) (dversion lock w=1 last_client=77200959) pv=0 v=129495990 ap=1 ino=(nil) state=1342177345|new|bottomlru | request=1 lock=1 authpin=1 0x55d0053208c0]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.den(0x607 10003178634) auth_pin by 0x55d005320980 on [dentry #0x100/stray7/10003178634 [2,head] auth NULL (dn lock) (dversion lock w=1 last_client=77200959) pv=0 v=129495990 ap=2 ino=(nil) state=1342177345|new|bottomlru | request=1 lock=1 authpin=1 0x55d0053208c0] now 2
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got xlock on (dn xlock x=1 by 0x55d01a129400) [dentry #0x100/stray7/10003178634 [2,head] auth NULL (dn xlock x=1 by 0x55d01a129400) (dversion lock w=1 last_client=77200959) pv=0 v=129495990 ap=2 ino=(nil) state=1342177345|new|bottomlru | request=1 lock=2 authpin=1 0x55d0053208c0]
2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker local_wrlock_start on (dversion lock) on [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json [2,head] auth (dversion lock) v=13929147 ap=1 ino=0x10003178634 state=1610612736 | request=0 lock=0 inodepin=1 dirty=1 waiter=0 authpin=1 0x55cfeeef3dc0]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got wrlock on (dversion lock w=1 last_client=77200959) [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json [2,head] auth (dversion lock w=1 last_client=77200959) v=13929147 ap=1 ino=0x10003178634 state=1610612736 | request=0 lock=1 inodepin=1 dirty=1 waiter=0 authpin=1 0x55cfeeef3dc0]
2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker xlock_start on (dn sync) on [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json [2,head] auth (dversion lock w=1 last_client=77200959) v=13929147 ap=1 ino=0x10003178634 state=1610612736 | request=0 lock=1 inodepin=1 dirty=1 waiter=0 authpin=1 0x55cfeeef3dc0]
2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker simple_lock on (dn sync) on [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json [2,head] auth (dversion lock w=1 last_client=77200959) v=13929147 ap=1 ino=0x10003178634 state=1610612736 | request=0 lock=1 inodepin=1 dirty=1 waiter=0 authpin=1 0x55cfeeef3dc0]
2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker simple_xlock on (dn lock) on [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json [2,head] auth (dn lock) (dversion lock w=1 last_client=77200959) v=13929147 ap=1 ino=0x10003178634 state=1610612736 | request=1 lock=1 inodepin=1 dirty=1 waiter=0 authpin=1 0x55cfeeef3dc0]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.den(0x1000268e7cb tus-node-server-0.1.1.json) auth_pin by 0x55cfeeef3e80 on [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json [2,head] auth (dn lock) (dversion lock w=1 last_client=77200959) v=13929147 ap=2 ino=0x10003178634 state=1610612736 | request=1 lock=1 inodepin=1 dirty=1 waiter=0 authpin=1 0x55cfeeef3dc0] now 2
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got xlock on (dn xlock x=1 by 0x55d01a129400) [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json [2,head] auth (dn xlock x=1 by 0x55d01a129400) (dversion lock w=1 last_client=77200959) v=13929147 ap=2 ino=0x10003178634 state=1610612736 | request=1 lock=2 inodepin=1 dirty=1 waiter=0 authpin=1 0x55cfeeef3dc0]
2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker local_wrlock_start on (dversion lock) on [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 [2,head] auth (dversion lock) v=13929158 ap=1 ino=0x10003178635 state=1610612736 | request=0 lock=0 inodepin=1 dirty=1 authpin=1 0x55d018f10000]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got wrlock on (dversion lock w=1 last_client=77200959) [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 [2,head] auth (dversion lock w=1 last_client=77200959) v=13929158 ap=1 ino=0x10003178635 state=1610612736 | request=0 lock=1 inodepin=1 dirty=1 authpin=1 0x55d018f10000]
2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker xlock_start on (dn sync) on [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 [2,head] auth (dversion lock w=1 last_client=77200959) v=13929158 ap=1 ino=0x10003178635 state=1610612736 | request=0 lock=1 inodepin=1 dirty=1 authpin=1 0x55d018f10000]
2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker simple_lock on (dn sync) on [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 [2,head] auth (dversion lock w=1 last_client=77200959) v=13929158 ap=1 ino=0x10003178635 state=1610612736 | request=0 lock=1 inodepin=1 dirty=1 authpin=1 0x55d018f10000]
2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker simple_xlock on (dn lock) on [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 [2,head] auth (dn lock) (dversion lock w=1 last_client=77200959) v=13929158 ap=1 ino=0x10003178635 state=1610612736 | request=1 lock=1 inodepin=1 dirty=1 authpin=1 0x55d018f10000]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.den(0x1000268e7cb tus-node-server-0.1.1.json.2465308719) auth_pin by 0x55d018f100c0 on [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 [2,head] auth (dn lock) (dversion lock w=1 last_client=77200959) v=13929158 ap=2 ino=0x10003178635 state=1610612736 | request=1 lock=1 inodepin=1 dirty=1 authpin=1 0x55d018f10000] now 2
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got xlock on (dn xlock x=1 by 0x55d01a129400) [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 [2,head] auth (dn xlock x=1 by 0x55d01a129400) (dversion lock w=1 last_client=77200959) v=13929158 ap=2 ino=0x10003178635 state=1610612736 | request=1 lock=2 inodepin=1 dirty=1 authpin=1 0x55d018f10000]
2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker rdlock_start on (isnap sync) on [inode 0x1 [...2,head] / auth v34216266 ap=1 snaprealm=0x55cfeff4a280 f(v0 m2020-01-17 12:41:39.137684 5=0+5) n(v7229910 rc2020-01-24 16:08:38.595298 b441687361105836 23052011=22719256+332755)/n(v0 1=0+1) (inest lock dirty) (iversion lock) caps={48654103=pAsLsXsFs/-@0,49033239=pAsLsXsFs/-@0,68004815=pAsLsXsFs/-@4,77200959=pAsLsXsFs/-@0,77246902=pAsLsXsFs/-@0} | dirtyscattered=1 lock=0 dirfrag=1 caps=1 openingsnapparents=0 dirty=1 authpin=1 0x55cfeff94700]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got rdlock on (isnap sync r=1) [inode 0x1 [...2,head] / auth v34216266 ap=1 snaprealm=0x55cfeff4a280 f(v0 m2020-01-17 12:41:39.137684 5=0+5) n(v7229910 rc2020-01-24 16:08:38.595298 b441687361105836 23052011=22719256+332755)/n(v0 1=0+1) (isnap sync r=1) (inest lock dirty) (iversion lock) caps={48654103=pAsLsXsFs/-@0,49033239=pAsLsXsFs/-@0,68004815=pAsLsXsFs/-@4,77200959=pAsLsXsFs/-@0,77246902=pAsLsXsFs/-@0} | dirtyscattered=1 lock=1 dirfrag=1 caps=1 openingsnapparents=0 dirty=1 authpin=1 0x55cfeff94700]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker wrlock_start (ifile sync) on [inode 0x607 [...2,head] ~mds0/stray7/ auth v89339680 ap=1 f(v40 m2020-01-24 16:08:43.778298 211=160+51) n(v130 rc2020-01-24 16:08:43.778298 b81704593 212=160+52) (inest lock) (iversion lock) | dirtyscattered=0 lock=0 dirfrag=1 stickydirs=1 stray=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff6f800]
2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker simple_lock on (ifile sync) on [inode 0x607 [...2,head] ~mds0/stray7/ auth v89339680 ap=1 f(v40 m2020-01-24 16:08:43.778298 211=160+51) n(v130 rc2020-01-24 16:08:43.778298 b81704593 212=160+52) (inest lock) (iversion lock) | dirtyscattered=0 lock=0 dirfrag=1 stickydirs=1 stray=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff6f800]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got wrlock on (ifile lock w=1) [inode 0x607 [...2,head] ~mds0/stray7/ auth v89339680 ap=1 f(v40 m2020-01-24 16:08:43.778298 211=160+51) n(v130 rc2020-01-24 16:08:43.778298 b81704593 212=160+52) (inest lock) (ifile lock w=1) (iversion lock) | dirtyscattered=0 lock=1 dirfrag=1 stickydirs=1 stray=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff6f800]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker wrlock_start (inest lock) on [inode 0x607 [...2,head] ~mds0/stray7/ auth v89339680 ap=1 f(v40 m2020-01-24 16:08:43.778298 211=160+51) n(v130 rc2020-01-24 16:08:43.778298 b81704593 212=160+52) (inest lock) (ifile lock w=1) (iversion lock) | dirtyscattered=0 lock=1 dirfrag=1 stickydirs=1 stray=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff6f800]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got wrlock on (inest lock w=1) [inode 0x607 [...2,head] ~mds0/stray7/ auth v89339680 ap=1 f(v40 m2020-01-24 16:08:43.778298 211=160+51) n(v130 rc2020-01-24 16:08:43.778298 b81704593 212=160+52) (inest lock w=1) (ifile lock w=1) (iversion lock) | dirtyscattered=0 lock=2 dirfrag=1 stickydirs=1 stray=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff6f800]
2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker rdlock_start on (isnap sync) on [inode 0x10000000001 [...2,head] /home/ auth v98722180 ap=1 f(v0 m2020-01-20 09:56:37.772033 65=0+65) n(v7735020 rc2020-01-24 16:08:47.195918 b164276032854114 16350398=16221677+128721) (inest lock dirty) (iversion lock) caps={49033239=pAsLsXs/-@0,75454214=pAsLsXsFs/-@4,76406416=pAsLsXsFs/-@0,76407061=pAsLsXsFs/-@0,76407436=pAsLsXsFs/-@0,77200959=pAsLsXsFs/-@0,77246902=pAsLsXsFs/AsLsXsFsx@4,77586859=pAsLsXsFs/-@0,77605502=pAsLsXsFs/-@2,77643571=pAsLsXsFs/-@1} | dirtyscattered=1 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff94e00]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got rdlock on (isnap sync r=1) [inode 0x10000000001 [...2,head] /home/ auth v98722180 ap=1 f(v0 m2020-01-20 09:56:37.772033 65=0+65) n(v7735020 rc2020-01-24 16:08:47.195918 b164276032854114 16350398=16221677+128721) (isnap sync r=1) (inest lock dirty) (iversion lock) caps={49033239=pAsLsXs/-@0,75454214=pAsLsXsFs/-@4,76406416=pAsLsXsFs/-@0,76407061=pAsLsXsFs/-@0,76407436=pAsLsXsFs/-@0,77200959=pAsLsXsFs/-@0,77246902=pAsLsXsFs/AsLsXsFsx@4,77586859=pAsLsXsFs/-@0,77605502=pAsLsXsFs/-@2,77643571=pAsLsXsFs/-@1} | dirtyscattered=1 request=0 lock=1 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff94e00]
2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker rdlock_start on (isnap sync) on [inode 0x10000000002 [...2,head] /home/tjr22/ auth v131045958 ap=1 f(v31 m2020-01-24 16:08:21.091241 189=91+98) n(v765289 rc2020-01-24 16:08:47.195918 b8869319870466 1292992=1236041+56951) (inest lock dirty) (iversion lock) caps={49033239=pAsLsXs/-@0,75454214=pAsLsXsFs/-@1513,77200959=pAsLsXsFs/-@327,77246902=pAsLsXs/-@2,77605502=pAsLsXs/p@50} | dirtyscattered=1 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 waiter=0 authpin=1 0x55cfeff95500]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got rdlock on (isnap sync r=1) [inode 0x10000000002 [...2,head] /home/tjr22/ auth v131045958 ap=1 f(v31 m2020-01-24 16:08:21.091241 189=91+98) n(v765289 rc2020-01-24 16:08:47.195918 b8869319870466 1292992=1236041+56951) (isnap sync r=1) (inest lock dirty) (iversion lock) caps={49033239=pAsLsXs/-@0,75454214=pAsLsXsFs/-@1513,77200959=pAsLsXsFs/-@327,77246902=pAsLsXs/-@2,77605502=pAsLsXs/p@50} | dirtyscattered=1 request=0 lock=1 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 waiter=0 authpin=1 0x55cfeff95500]
2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker rdlock_start on (isnap sync) on [inode 0x10000000a52 [...2,head] /home/tjr22/.config/ auth v129496029 ap=1 f(v0 m2019-12-13 13:57:58.591154 31=6+25) n(v440444 rc2020-01-24 16:08:47.195918 b336426 108=63+45) (inest lock dirty) (iversion lock) caps={49033239=pAsLsXs/-@0,75454214=pAsLsXsFs/-@69,77200959=pAsLsXsFs/-@64,77605502=pAsLsXsFs/-@2} | dirtyscattered=1 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff95c00]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got rdlock on (isnap sync r=1) [inode 0x10000000a52 [...2,head] /home/tjr22/.config/ auth v129496029 ap=1 f(v0 m2019-12-13 13:57:58.591154 31=6+25) n(v440444 rc2020-01-24 16:08:47.195918 b336426 108=63+45) (isnap sync r=1) (inest lock dirty) (iversion lock) caps={49033239=pAsLsXs/-@0,75454214=pAsLsXsFs/-@69,77200959=pAsLsXsFs/-@64,77605502=pAsLsXsFs/-@2} | dirtyscattered=1 request=0 lock=1 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff95c00]
2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker rdlock_start on (isnap sync) on [inode 0x1000268e7cb [...2,head] /home/tjr22/.config/configstore/ auth v15636199 ap=1 f(v0 m2020-01-24 16:08:47.195918 2=2+0) n(v8 rc2020-01-24 16:08:47.246973 b700 3=2+1) (inest lock) (ifile excl) (iversion lock) caps={77200959=pAsLsXsFsx/-@30413},l=77200959 | dirtyscattered=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff96300]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got rdlock on (isnap sync r=1) [inode 0x1000268e7cb [...2,head] /home/tjr22/.config/configstore/ auth v15636199 ap=1 f(v0 m2020-01-24 16:08:47.195918 2=2+0) n(v8 rc2020-01-24 16:08:47.246973 b700 3=2+1) (isnap sync r=1) (inest lock) (ifile excl) (iversion lock) caps={77200959=pAsLsXsFsx/-@30413},l=77200959 | dirtyscattered=0 lock=1 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff96300]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker wrlock_start (ifile excl) on [inode 0x1000268e7cb [...2,head] /home/tjr22/.config/configstore/ auth v15636199 ap=1 f(v0 m2020-01-24 16:08:47.195918 2=2+0) n(v8 rc2020-01-24 16:08:47.246973 b700 3=2+1) (isnap sync r=1) (inest lock) (ifile excl) (iversion lock) caps={77200959=pAsLsXsFsx/-@30413},l=77200959 | dirtyscattered=0 lock=1 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff96300]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got wrlock on (ifile excl w=1) [inode 0x1000268e7cb [...2,head] /home/tjr22/.config/configstore/ auth v15636199 ap=1 f(v0 m2020-01-24 16:08:47.195918 2=2+0) n(v8 rc2020-01-24 16:08:47.246973 b700 3=2+1) (isnap sync r=1) (inest lock) (ifile excl w=1) (iversion lock) caps={77200959=pAsLsXsFsx/-@30413},l=77200959 | dirtyscattered=0 lock=2 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff96300]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker wrlock_start (inest lock) on [inode 0x1000268e7cb [...2,head] /home/tjr22/.config/configstore/ auth v15636199 ap=1 f(v0 m2020-01-24 16:08:47.195918 2=2+0) n(v8 rc2020-01-24 16:08:47.246973 b700 3=2+1) (isnap sync r=1) (inest lock) (ifile excl w=1) (iversion lock) caps={77200959=pAsLsXsFsx/-@30413},l=77200959 | dirtyscattered=0 lock=2 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff96300]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got wrlock on (inest lock w=1) [inode 0x1000268e7cb [...2,head] /home/tjr22/.config/configstore/ auth v15636199 ap=1 f(v0 m2020-01-24 16:08:47.195918 2=2+0) n(v8 rc2020-01-24 16:08:47.246973 b700 3=2+1) (isnap sync r=1) (inest lock w=1) (ifile excl w=1) (iversion lock) caps={77200959=pAsLsXsFsx/-@30413},l=77200959 | dirtyscattered=0 lock=3 dirfrag=1 caps=1 dirtyrstat=0 dirty=1 authpin=1 0x55cfeff96300]
2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker local_wrlock_start on (iversion lock) on [inode 0x10003178634 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json auth v13929147 ap=1 dirtyparent s=350 n(v0 rc2020-01-24 16:08:43.778298 b350 1=1+0)/n(v0 rc2020-01-24 16:08:43.773910 b350 1=1+0) (iversion lock) caps={77200959=pAsXsFscr/-@6} | ptrwaiter=0 request=0 lock=0 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3a000]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got wrlock on (iversion lock w=1 last_client=77200959) [inode 0x10003178634 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json auth v13929147 ap=1 dirtyparent s=350 n(v0 rc2020-01-24 16:08:43.778298 b350 1=1+0)/n(v0 rc2020-01-24 16:08:43.773910 b350 1=1+0) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@6} | ptrwaiter=0 request=0 lock=1 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3a000]
2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker xlock_start on (isnap sync) on [inode 0x10003178634 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json auth v13929147 ap=1 dirtyparent s=350 n(v0 rc2020-01-24 16:08:43.778298 b350 1=1+0)/n(v0 rc2020-01-24 16:08:43.773910 b350 1=1+0) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@6} | ptrwaiter=0 request=0 lock=1 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3a000]
2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker simple_lock on (isnap sync) on [inode 0x10003178634 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json auth v13929147 ap=1 dirtyparent s=350 n(v0 rc2020-01-24 16:08:43.778298 b350 1=1+0)/n(v0 rc2020-01-24 16:08:43.773910 b350 1=1+0) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@6} | ptrwaiter=0 request=0 lock=1 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3a000]
2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker simple_xlock on (isnap lock) on [inode 0x10003178634 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json auth v13929147 ap=1 dirtyparent s=350 n(v0 rc2020-01-24 16:08:43.778298 b350 1=1+0)/n(v0 rc2020-01-24 16:08:43.773910 b350 1=1+0) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@6} | ptrwaiter=0 request=1 lock=1 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3a000]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.ino(0x10003178634) auth_pin by 0x55d006c3a5e0 on [inode 0x10003178634 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json auth v13929147 ap=2 dirtyparent s=350 n(v0 rc2020-01-24 16:08:43.778298 b350 1=1+0)/n(v0 rc2020-01-24 16:08:43.773910 b350 1=1+0) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@6} | ptrwaiter=0 request=1 lock=1 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3a000] now 2
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got xlock on (isnap xlock x=1 by 0x55d01a129400) [inode 0x10003178634 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json auth v13929147 ap=2 dirtyparent s=350 n(v0 rc2020-01-24 16:08:43.778298 b350 1=1+0)/n(v0 rc2020-01-24 16:08:43.773910 b350 1=1+0) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@6} | ptrwaiter=0 request=1 lock=2 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3a000]
2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker xlock_start on (ilink sync) on [inode 0x10003178634 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json auth v13929147 ap=2 dirtyparent s=350 n(v0 rc2020-01-24 16:08:43.778298 b350 1=1+0)/n(v0 rc2020-01-24 16:08:43.773910 b350 1=1+0) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@6} | ptrwaiter=0 request=1 lock=2 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3a000]
2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker simple_lock on (ilink sync) on [inode 0x10003178634 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json auth v13929147 ap=2 dirtyparent s=350 n(v0 rc2020-01-24 16:08:43.778298 b350 1=1+0)/n(v0 rc2020-01-24 16:08:43.773910 b350 1=1+0) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@6} | ptrwaiter=0 request=1 lock=2 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3a000]
2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker simple_xlock on (ilink lock) on [inode 0x10003178634 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json auth v13929147 ap=2 dirtyparent s=350 n(v0 rc2020-01-24 16:08:43.778298 b350 1=1+0)/n(v0 rc2020-01-24 16:08:43.773910 b350 1=1+0) (ilink lock) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@6} | ptrwaiter=0 request=1 lock=2 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3a000]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.ino(0x10003178634) auth_pin by 0x55d006c3a530 on [inode 0x10003178634 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json auth v13929147 ap=3 dirtyparent s=350 n(v0 rc2020-01-24 16:08:43.778298 b350 1=1+0)/n(v0 rc2020-01-24 16:08:43.773910 b350 1=1+0) (ilink lock) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@6} | ptrwaiter=0 request=1 lock=2 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3a000] now 3
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got xlock on (ilink xlock x=1 by 0x55d01a129400) [inode 0x10003178634 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json auth v13929147 ap=3 dirtyparent s=350 n(v0 rc2020-01-24 16:08:43.778298 b350 1=1+0)/n(v0 rc2020-01-24 16:08:43.773910 b350 1=1+0) (ilink xlock x=1 by 0x55d01a129400) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@6} | ptrwaiter=0 request=1 lock=3 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3a000]
2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker local_wrlock_start on (iversion lock) on [inode 0x10003178635 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 auth v13929158 ap=1 dirtyparent s=350 n(v0 rc2020-01-24 16:08:47.246973 b350 1=1+0) (iversion lock) caps={77200959=pAsXsFscr/-@4} | ptrwaiter=0 request=0 lock=0 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3b500]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got wrlock on (iversion lock w=1 last_client=77200959) [inode 0x10003178635 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 auth v13929158 ap=1 dirtyparent s=350 n(v0 rc2020-01-24 16:08:47.246973 b350 1=1+0) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@4} | ptrwaiter=0 request=0 lock=1 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3b500]
2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker xlock_start on (isnap sync) on [inode 0x10003178635 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 auth v13929158 ap=1 dirtyparent s=350 n(v0 rc2020-01-24 16:08:47.246973 b350 1=1+0) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@4} | ptrwaiter=0 request=0 lock=1 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3b500]
2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker simple_lock on (isnap sync) on [inode 0x10003178635 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 auth v13929158 ap=1 dirtyparent s=350 n(v0 rc2020-01-24 16:08:47.246973 b350 1=1+0) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@4} | ptrwaiter=0 request=0 lock=1 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3b500]
2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker simple_xlock on (isnap lock) on [inode 0x10003178635 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 auth v13929158 ap=1 dirtyparent s=350 n(v0 rc2020-01-24 16:08:47.246973 b350 1=1+0) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@4} | ptrwaiter=0 request=1 lock=1 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3b500]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.ino(0x10003178635) auth_pin by 0x55d006c3bae0 on [inode 0x10003178635 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 auth v13929158 ap=2 dirtyparent s=350 n(v0 rc2020-01-24 16:08:47.246973 b350 1=1+0) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@4} | ptrwaiter=0 request=1 lock=1 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3b500] now 2
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got xlock on (isnap xlock x=1 by 0x55d01a129400) [inode 0x10003178635 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 auth v13929158 ap=2 dirtyparent s=350 n(v0 rc2020-01-24 16:08:47.246973 b350 1=1+0) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@4} | ptrwaiter=0 request=1 lock=2 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3b500]
2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker xlock_start on (ilink sync) on [inode 0x10003178635 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 auth v13929158 ap=2 dirtyparent s=350 n(v0 rc2020-01-24 16:08:47.246973 b350 1=1+0) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@4} | ptrwaiter=0 request=1 lock=2 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3b500]
2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker simple_lock on (ilink sync) on [inode 0x10003178635 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 auth v13929158 ap=2 dirtyparent s=350 n(v0 rc2020-01-24 16:08:47.246973 b350 1=1+0) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@4} | ptrwaiter=0 request=1 lock=2 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3b500]
2020-01-24 16:08:47.250 7fbe050c6700 7 mds.0.locker simple_xlock on (ilink lock) on [inode 0x10003178635 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 auth v13929158 ap=2 dirtyparent s=350 n(v0 rc2020-01-24 16:08:47.246973 b350 1=1+0) (ilink lock) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@4} | ptrwaiter=0 request=1 lock=2 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3b500]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.ino(0x10003178635) auth_pin by 0x55d006c3ba30 on [inode 0x10003178635 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 auth v13929158 ap=3 dirtyparent s=350 n(v0 rc2020-01-24 16:08:47.246973 b350 1=1+0) (ilink lock) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@4} | ptrwaiter=0 request=1 lock=2 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3b500] now 3
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.locker got xlock on (ilink xlock x=1 by 0x55d01a129400) [inode 0x10003178635 [2,head] /home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 auth v13929158 ap=3 dirtyparent s=350 n(v0 rc2020-01-24 16:08:47.246973 b350 1=1+0) (ilink xlock x=1 by 0x55d01a129400) (iversion lock w=1 last_client=77200959) caps={77200959=pAsXsFscr/-@4} | ptrwaiter=0 request=1 lock=3 caps=1 dirtyparent=1 dirty=1 authpin=1 0x55d006c3b500]
2020-01-24 16:08:47.250 7fbe050c6700 10 MDSAuthCap is_capable inode(path /home/tjr22/.config/configstore owner 1000:1003 mode 040700) by caller 1000:1000 mask 2 new 0:0 cap: MDSAuthCaps[allow *]
2020-01-24 16:08:47.250 7fbe050c6700 10 MDSAuthCap is_capable inode(path /home/tjr22/.config/configstore owner 1000:1003 mode 040700) by caller 1000:1000 mask 2 new 0:0 cap: MDSAuthCaps[allow *]
2020-01-24 16:08:47.250 7fbe050c6700 10 MDSAuthCap is_capable inode(path /home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 owner 1000:1000 mode 0100600) by caller 1000:1000 mask 2 new 0:0 cap: MDSAuthCaps[allow *]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.snaprealm(0x1 seq 1 0x55cfeff4a280) have_past_parents_open [1,head]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.cache.snaprealm(0x1 seq 1 0x55cfeff4a280) have_past_parents_open [1,head]
2020-01-24 16:08:47.250 7fbe050c6700 10 mds.0.server _rename_prepare request(client.77200959:238582 cr=0x55d019672580) [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json.2465308719 [2,head] auth (dn xlock x=1 by 0x55d01a129400) (dversion lock w=1 last_client=77200959) v=13929158 ap=2 ino=0x10003178635 state=1610612736 | request=1 lock=2 inodepin=1 dirty=1 authpin=1 0x55d018f10000] [dentry #0x1/home/tjr22/.config/configstore/tus-node-server-0.1.1.json [2,head] auth (dn xlock x=1 by 0x55d01a129400) (dversion lock w=1 last_client=77200959) v=13929147 ap=2 ino=0x10003178634 state=1610612736 | request=1 lock=2 inodepin=1 dirty=1 waiter=0 authpin=1 0x55cfeeef3dc0]
I have no idea if any of these are red herrings (sorry.) Please help, at this point I’m baffled! Thank you.
-TJ
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
