Hello,
I have a Ceph Nautilus Cluster 14.2.1 for cephfs only on 40x 1.8T SAS disk (no SSD) in 20 servers.
> cluster:
> id: 778234df-5784-4021-b983-0ee1814891be
> health: HEALTH_WARN
> 2 MDSs report slow requests
>
> services:
> mon: 3 daemons, quorum icadmin006,icadmin007,icadmin008 (age 5d)
> mgr: icadmin008(active, since 18h), standbys: icadmin007, icadmin006
> mds: cephfs:3 {0=icadmin006=up:active,1=icadmin007=up:active,2=icadmin008=up:active}
> osd: 40 osds: 40 up (since 2w), 40 in (since 3w)
>
> data:
> pools: 3 pools, 672 pgs
> objects: 36.08M objects, 19 TiB
> usage: 51 TiB used, 15 TiB / 65 TiB avail
> pgs: 670 active+clean
> 2 active+clean+scrubbing
I often get "MDSs report slow requests" and plenty of "[WRN] 3 slow requests, 0 included below; oldest blocked for > 60281.199503 secs"
> HEALTH_WARN 2 MDSs report slow requests
> MDS_SLOW_REQUEST 2 MDSs report slow requests
> mdsicadmin007(mds.1): 3 slow requests are blocked > 30 secs
> mdsicadmin006(mds.0): 10 slow requests are blocked > 30 secs
After a few investigations, I saw that ALL ceph-osd process eat a lot of memory, up to 130GB RSS each. It this value normal? May this related to
slow requests? Is disk only increasing the probability to get slow requests?
> USER PID %CPU %MEM VSZ RSS TTY STAT STAR TIME COMMAND
> ceph 34196 3.6 35.0 156247524 138521572 ? Ssl Jul01 4173:18 /usr/bin/ceph-osd -f --cluster apollo --id 1 --setuser ceph --setgroup ceph
> ceph 34394 3.6 35.0 160001436 138487776 ? Ssl Jul01 4178:37 /usr/bin/ceph-osd -f --cluster apollo --id 32 --setuser ceph --setgroup ceph
> ceph 34709 3.5 35.1 156369636 138752044 ? Ssl Jul01 4088:57 /usr/bin/ceph-osd -f --cluster apollo --id 29 --setuser ceph --setgroup ceph
> ceph 34915 3.4 35.1 158976936 138715900 ? Ssl Jul01 3950:45 /usr/bin/ceph-osd -f --cluster apollo --id 3 --setuser ceph --setgroup ceph
> ceph 34156 3.4 35.1 158280768 138714484 ? Ssl Jul01 3984:11 /usr/bin/ceph-osd -f --cluster apollo --id 30 --setuser ceph --setgroup ceph
> ceph 34378 3.7 35.1 155162420 138708096 ? Ssl Jul01 4312:12 /usr/bin/ceph-osd -f --cluster apollo --id 8 --setuser ceph --setgroup ceph
> ceph 34161 3.5 35.0 159606788 138523652 ? Ssl Jul01 4128:17 /usr/bin/ceph-osd -f --cluster apollo --id 16 --setuser ceph --setgroup ceph
> ceph 34380 3.6 35.1 161465372 138670168 ? Ssl Jul01 4238:20 /usr/bin/ceph-osd -f --cluster apollo --id 35 --setuser ceph --setgroup ceph
> ceph 33822 3.7 35.1 163456644 138734036 ? Ssl Jul01 4342:05 /usr/bin/ceph-osd -f --cluster apollo --id 15 --setuser ceph --setgroup ceph
> ceph 34003 3.8 35.0 161868584 138531208 ? Ssl Jul01 4427:32 /usr/bin/ceph-osd -f --cluster apollo --id 38 --setuser ceph --setgroup ceph
> ceph 9753 2.8 24.2 96923856 95580776 ? Ssl Sep02 700:25 /usr/bin/ceph-osd -f --cluster apollo --id 31 --setuser ceph --setgroup ceph
> ceph 10120 2.5 24.0 96130340 94856244 ? Ssl Sep02 644:50 /usr/bin/ceph-osd -f --cluster apollo --id 7 --setuser ceph --setgroup ceph
> ceph 36204 3.6 35.0 159394476 138592124 ? Ssl Jul01 4185:36 /usr/bin/ceph-osd -f --cluster apollo --id 18 --setuser ceph --setgroup ceph
> ceph 36427 3.7 34.4 155699060 136076432 ? Ssl Jul01 4298:26 /usr/bin/ceph-osd -f --cluster apollo --id 36 --setuser ceph --setgroup ceph
> ceph 36622 4.1 35.1 158219408 138724688 ? Ssl Jul01 4779:14 /usr/bin/ceph-osd -f --cluster apollo --id 19 --setuser ceph --setgroup ceph
> ceph 36881 4.0 35.1 157748752 138719064 ? Ssl Jul01 4669:54 /usr/bin/ceph-osd -f --cluster apollo --id 37 --setuser ceph --setgroup ceph
> ceph 34649 3.7 35.1 159601580 138652012 ? Ssl Jul01 4337:20 /usr/bin/ceph-osd -f --cluster apollo --id 14 --setuser ceph --setgroup ceph
> ceph 34881 3.8 35.1 158632412 138764376 ? Ssl Jul01 4433:50 /usr/bin/ceph-osd -f --cluster apollo --id 33 --setuser ceph --setgroup ceph
> ceph 34646 4.2 35.1 155029328 138732376 ? Ssl Jul01 4831:24 /usr/bin/ceph-osd -f --cluster apollo --id 17 --setuser ceph --setgroup ceph
> ceph 34881 4.1 35.1 156801676 138763588 ? Ssl Jul01 4710:19 /usr/bin/ceph-osd -f --cluster apollo --id 39 --setuser ceph --setgroup ceph
> ceph 36766 3.7 35.1 158070740 138703240 ? Ssl Jul01 4341:42 /usr/bin/ceph-osd -f --cluster apollo --id 13 --setuser ceph --setgroup ceph
> ceph 37013 3.5 35.0 157767668 138272248 ? Ssl Jul01 4094:12 /usr/bin/ceph-osd -f --cluster apollo --id 34 --setuser ceph --setgroup ceph
> ceph 35007 3.4 35.1 160318780 138756404 ? Ssl Jul01 3963:21 /usr/bin/ceph-osd -f --cluster apollo --id 2 --setuser ceph --setgroup ceph
> ceph 35217 3.5 35.1 159023744 138626680 ? Ssl Jul01 4041:50 /usr/bin/ceph-osd -f --cluster apollo --id 22 --setuser ceph --setgroup ceph
> ceph 36962 3.2 35.1 158692228 138730292 ? Ssl Jul01 3772:35 /usr/bin/ceph-osd -f --cluster apollo --id 5 --setuser ceph --setgroup ceph
> ceph 2991351 2.6 22.9 92011392 90761128 ? Ssl Sep02 666:32 /usr/bin/ceph-osd -f --cluster apollo --id 21 --setuser ceph --setgroup ceph
> ceph 35503 3.2 35.0 158784940 138502100 ? Ssl Jul01 3766:33 /usr/bin/ceph-osd -f --cluster apollo --id 25 --setuser ceph --setgroup ceph
> ceph 35683 3.6 35.1 160927812 138678080 ? Ssl Jul01 4233:17 /usr/bin/ceph-osd -f --cluster apollo --id 4 --setuser ceph --setgroup ceph
> ceph 36969 3.7 35.1 158701188 138745028 ? Ssl Jul01 4348:06 /usr/bin/ceph-osd -f --cluster apollo --id 20 --setuser ceph --setgroup ceph
> ceph 1902641 2.5 24.1 96688368 95438808 ? Ssl Sep02 633:45 /usr/bin/ceph-osd -f --cluster apollo --id 0 --setuser ceph --setgroup ceph
> ceph 35576 3.7 35.1 156262424 138750552 ? Ssl Jul01 4338:09 /usr/bin/ceph-osd -f --cluster apollo --id 27 --setuser ceph --setgroup ceph
> ceph 1901746 2.5 24.8 99300108 98051192 ? Ssl Sep02 641:52 /usr/bin/ceph-osd -f --cluster apollo --id 6 --setuser ceph --setgroup ceph
> ceph 35735 3.7 35.1 156027400 138738076 ? Ssl Jul01 4350:00 /usr/bin/ceph-osd -f --cluster apollo --id 24 --setuser ceph --setgroup ceph
> ceph 35929 3.7 35.0 160626040 138511872 ? Ssl Jul01 4361:54 /usr/bin/ceph-osd -f --cluster apollo --id 9 --setuser ceph --setgroup ceph
> ceph 35699 3.1 35.1 158773084 138728576 ? Ssl Jul01 3631:13 /usr/bin/ceph-osd -f --cluster apollo --id 10 --setuser ceph --setgroup ceph
> ceph 2941709 2.5 24.2 97125336 95906728 ? Ssl Sep02 638:11 /usr/bin/ceph-osd -f --cluster apollo --id 28 --setuser ceph --setgroup ceph
> ceph 38429 3.2 35.1 156638164 138712612 ? Ssl Jul01 3687:45 /usr/bin/ceph-osd -f --cluster apollo --id 12 --setuser ceph --setgroup ceph
> ceph 38651 3.3 35.1 159650296 138735924 ? Ssl Jul01 3835:51 /usr/bin/ceph-osd -f --cluster apollo --id 26 --setuser ceph --setgroup ceph
> ceph 35890 2.9 35.1 156923512 138734428 ? Ssl Jul01 3361:21 /usr/bin/ceph-osd -f --cluster apollo --id 11 --setuser ceph --setgroup ceph
> ceph 36129 3.3 35.1 158782748 138739248 ? Ssl Jul01 3845:41 /usr/bin/ceph-osd -f --cluster apollo --id 23 --setuser ceph --setgroup ceph
some logs :
> 2019-09-19 08:52:33.960242 mds.icadmin006 [WRN] 10 slow requests, 0 included below; oldest blocked for > 62427.674399 secs
> 2019-09-19 08:52:37.527465 mds.icadmin007 [WRN] 3 slow requests, 0 included below; oldest blocked for > 62431.241789 secs
> 2019-09-19 08:52:42.527581 mds.icadmin007 [WRN] 3 slow requests, 0 included below; oldest blocked for > 62436.241899 secs
> 2019-09-19 08:52:38.960358 mds.icadmin006 [WRN] 10 slow requests, 0 included below; oldest blocked for > 62432.674515 secs
> 2019-09-19 08:52:43.960476 mds.icadmin006 [WRN] 10 slow requests, 0 included below; oldest blocked for > 62437.674620 secs
> 2019-09-19 08:52:47.527663 mds.icadmin007 [WRN] 3 slow requests, 0 included below; oldest blocked for > 62441.241987 secs
> 2019-09-19 08:52:52.527770 mds.icadmin007 [WRN] 3 slow requests, 2 included below; oldest blocked for > 62446.242061 secs
> 2019-09-19 08:52:52.527777 mds.icadmin007 [WRN] slow request 61444.792236 seconds old, received at 2019-09-18 17:48:47.735459: internal op exportdir:mds.1:13 currently failed to wrlock, waiting
> 2019-09-19 08:52:52.527783 mds.icadmin007 [WRN] slow request 61444.792163 seconds old, received at 2019-09-18 17:48:47.735533: internal op exportdir:mds.1:14 currently failed to wrlock, waiting
> 2019-09-19 08:52:48.960590 mds.icadmin006 [WRN] 10 slow requests, 0 included below; oldest blocked for > 62442.674748 secs
> 2019-09-19 08:52:53.960684 mds.icadmin006 [WRN] 10 slow requests, 2 included below; oldest blocked for > 62447.674825 secs
> 2019-09-19 08:52:53.960692 mds.icadmin006 [WRN] slow request 61441.895507 seconds old, received at 2019-09-18 17:48:52.065114: rejoin:mds.1:13 currently dispatched
> 2019-09-19 08:52:53.960697 mds.icadmin006 [WRN] slow request 61441.895489 seconds old, received at 2019-09-18 17:48:52.065131: rejoin:mds.1:14 currently dispatched
> 2019-09-19 08:52:57.527852 mds.icadmin007 [WRN] 3 slow requests, 0 included below; oldest blocked for > 62451.242174 secs
> 2019-09-19 08:53:02.527972 mds.icadmin007 [WRN] 3 slow requests, 0 included below; oldest blocked for > 62456.242289 secs
> 2019-09-19 08:52:58.960777 mds.icadmin006 [WRN] 10 slow requests, 0 included below; oldest blocked for > 62452.674936 secs
> 2019-09-19 08:53:03.960853 mds.icadmin006 [WRN] 10 slow requests, 0 included below; oldest blocked for > 62457.675011 secs
> 2019-09-19 08:53:07.528033 mds.icadmin007 [WRN] 3 slow requests, 0 included below; oldest blocked for > 62461.242354 secs
> 2019-09-19 08:53:12.528177 mds.icadmin007 [WRN] 3 slow requests, 0 included below; oldest blocked for > 62466.242487 secs
> 2019-09-19 08:53:08.960965 mds.icadmin006 [WRN] 10 slow requests, 0 included below; oldest blocked for > 62462.675123 secs
> 2019-09-19 08:53:13.961034 mds.icadmin006 [WRN] 10 slow requests, 0 included below; oldest blocked for > 62467.675195 secs
> 2019-09-19 08:53:17.528276 mds.icadmin007 [WRN] 3 slow requests, 0 included below; oldest blocked for > 62471.242592 secs
> 2019-09-19 08:53:22.528407 mds.icadmin007 [WRN] 3 slow requests, 0 included below; oldest blocked for > 62476.242729 secs
> 2019-09-19 08:53:18.961149 mds.icadmin006 [WRN] 10 slow requests, 0 included below; oldest blocked for > 62472.675310 secs
> 2019-09-19 08:53:23.961234 mds.icadmin006 [WRN] 10 slow requests, 0 included below; oldest blocked for > 62477.675392 secs
> 2019-09-19 08:53:27.528509 mds.icadmin007 [WRN] 3 slow requests, 0 included below; oldest blocked for > 62481.242832 secs
> 2019-09-19 08:53:32.528651 mds.icadmin007 [WRN] 3 slow requests, 0 included below; oldest blocked for > 62486.242961 secs
> 2019-09-19 08:53:28.961314 mds.icadmin006 [WRN] 10 slow requests, 0 included below; oldest blocked for > 62482.675471 secs
> 2019-09-19 08:53:33.961393 mds.icadmin006 [WRN] 10 slow requests, 0 included below; oldest blocked for > 62487.675549 secs
> 2019-09-19 08:53:37.528706 mds.icadmin007 [WRN] 3 slow requests, 0 included below; oldest blocked for > 62491.243031 secs
> 2019-09-19 08:53:42.528790 mds.icadmin007 [WRN] 3 slow requests, 0 included below; oldest blocked for > 62496.243105 secs
> 2019-09-19 08:53:38.961476 mds.icadmin006 [WRN] 10 slow requests, 1 included below; oldest blocked for > 62492.675617 secs
> 2019-09-19 08:53:38.961485 mds.icadmin006 [WRN] slow request 61441.151061 seconds old, received at 2019-09-18 17:49:37.810351: client_request(client.21441:176429 getattr pAsLsXsFs #0x10000f2b1b3 2019-09-18 17:49:37.806002 caller_uid=204878, caller_gid=11233{}) currently failed to rdlock, waiting
> 2019-09-19 08:53:43.961569 mds.icadmin006 [WRN] 10 slow requests, 0 included below; oldest blocked for > 62497.675728 secs
> 2019-09-19 08:53:47.528891 mds.icadmin007 [WRN] 3 slow requests, 0 included below; oldest blocked for > 62501.243214 secs
> 2019-09-19 08:53:52.529021 mds.icadmin007 [WRN] 3 slow requests, 0 included below; oldest blocked for > 62506.243337 secs
> 2019-09-19 08:53:48.961685 mds.icadmin006 [WRN] 10 slow requests, 0 included below; oldest blocked for > 62502.675839 secs
> 2019-09-19 08:53:53.961792 mds.icadmin006 [WRN] 10 slow requests, 0 included below; oldest blocked for > 62507.675948 secs
> 2019-09-19 08:53:57.529113 mds.icadmin007 [WRN] 3 slow requests, 0 included below; oldest blocked for > 62511.243437 secs
> 2019-09-19 08:54:02.529224 mds.icadmin007 [WRN] 3 slow requests, 0 included below; oldest blocked for > 62516.243546 secs
> 2019-09-19 08:53:58.961866 mds.icadmin006 [WRN] 10 slow requests, 0 included below; oldest blocked for > 62512.676025 secs
> 2019-09-19 08:54:03.961939 mds.icadmin006 [WRN] 10 slow requests, 0 included below; oldest blocked for > 62517.676099 secs
Thanks for your help.
Best regards,
--
Yoann Moulin
EPFL IC-IT
[
{
"id": 651292,
"num_leases": 0,
"num_caps": 0,
"state": "open",
"request_load_avg": 0,
"uptime": 65094.458896163,
"replay_requests": 0,
"completed_requests": 0,
"reconnecting": false,
"inst": "client.651292 v1:10.90.47.29:0/2037483206",
"client_metadata": {
"features": "00000000000000ff",
"entity_id": "labo04",
"hostname": "iccluster177.",
"kernel_version": "4.15.0-43-generic",
"root": "/labo04-scratch"
}
},
{
"id": 89226,
"num_leases": 0,
"num_caps": 0,
"state": "open",
"request_load_avg": 0,
"uptime": 65096.702874342001,
"replay_requests": 0,
"completed_requests": 1,
"reconnecting": false,
"inst": "client.89226 v1:10.90.47.16:0/1643772475",
"client_metadata": {
"features": "00000000000000ff",
"entity_id": "labo01",
"hostname": "iccluster164.",
"kernel_version": "4.13.0-26-generic",
"root": "/labo01-scratch"
}
},
{
"id": 625347,
"num_leases": 0,
"num_caps": 0,
"state": "open",
"request_load_avg": 0,
"uptime": 65096.702874342001,
"replay_requests": 0,
"completed_requests": 1,
"reconnecting": false,
"inst": "client.625347 v1:10.90.47.34:0/2068926848",
"client_metadata": {
"features": "00000000000000ff",
"entity_id": "labo01",
"hostname": "iccluster182.",
"kernel_version": "4.15.0-43-generic",
"root": "/labo01-scratch"
}
},
{
"id": 624960,
"num_leases": 0,
"num_caps": 0,
"state": "open",
"request_load_avg": 0,
"uptime": 65096.702874342001,
"replay_requests": 0,
"completed_requests": 1,
"reconnecting": false,
"inst": "client.624960 v1:10.90.47.23:0/3391918921",
"client_metadata": {
"features": "00000000000000ff",
"entity_id": "labo02",
"hostname": "iccluster171.",
"kernel_version": "4.15.0-43-generic",
"root": "/labo02-scratch"
}
},
{
"id": 36482,
"num_leases": 0,
"num_caps": 0,
"state": "open",
"request_load_avg": 0,
"uptime": 65096.702874342001,
"replay_requests": 0,
"completed_requests": 0,
"reconnecting": false,
"inst": "client.36482 v1:10.90.37.13:0/461389346",
"client_metadata": {
"features": "00000000000000ff",
"entity_id": "cephfsadmin",
"hostname": "iccluster012",
"kernel_version": "4.15.0-52-generic",
"root": "/"
}
},
{
"id": 21441,
"num_leases": 0,
"num_caps": 6292,
"state": "open",
"request_load_avg": 0,
"uptime": 65096.702874342001,
"replay_requests": 0,
"completed_requests": 0,
"reconnecting": false,
"inst": "client.21441 v1:10.90.47.19:0/1234150467",
"client_metadata": {
"features": "00000000000000ff",
"entity_id": "labo01",
"hostname": "iccluster167.",
"kernel_version": "4.13.0-26-generic",
"root": "/labo01-scratch"
}
},
{
"id": 458928,
"num_leases": 0,
"num_caps": 0,
"state": "open",
"request_load_avg": 0,
"uptime": 65096.702874342001,
"replay_requests": 0,
"completed_requests": 0,
"reconnecting": false,
"inst": "client.458928 v1:10.90.43.5:0/4126499073",
"client_metadata": {
"features": "00000000000000ff",
"entity_id": "labo03",
"hostname": "iclabo03srv16",
"kernel_version": "4.15.0-22-generic",
"root": "/labo03-scratch"
}
},
{
"id": 126456,
"num_leases": 0,
"num_caps": 0,
"state": "open",
"request_load_avg": 0,
"uptime": 65096.702874342001,
"replay_requests": 0,
"completed_requests": 3,
"reconnecting": false,
"inst": "client.126456 v1:10.90.45.2:0/902274091",
"client_metadata": {
"features": "00000000000000ff",
"entity_id": "labo02",
"hostname": "iccluster130",
"kernel_version": "4.15.0-47-generic",
"root": "/labo02-scratch"
}
},
{
"id": 59533,
"num_leases": 0,
"num_caps": 0,
"state": "open",
"request_load_avg": 0,
"uptime": 65096.702874342001,
"replay_requests": 0,
"completed_requests": 0,
"reconnecting": false,
"inst": "client.59533 v1:10.90.47.5:0/1578661632",
"client_metadata": {
"features": "00000000000000ff",
"entity_id": "labo02",
"hostname": "iccluster153.",
"kernel_version": "4.13.0-26-generic",
"root": "/labo02-scratch"
}
},
{
"id": 675490,
"num_leases": 0,
"num_caps": 6832,
"state": "open",
"request_load_avg": 0,
"uptime": 65092.494915260002,
"replay_requests": 0,
"completed_requests": 0,
"reconnecting": false,
"inst": "client.675490 v1:128.178.243.37:0/2728635645",
"client_metadata": {
"features": "00000000000000ff",
"entity_id": "cephfsadmin",
"hostname": "icitsrv5",
"kernel_version": "4.15.0-55-generic",
"root": "/"
}
},
{
"id": 136975,
"num_leases": 0,
"num_caps": 0,
"state": "open",
"request_load_avg": 0,
"uptime": 65096.702874342001,
"replay_requests": 0,
"completed_requests": 1,
"reconnecting": false,
"inst": "client.136975 v1:10.90.38.21:0/918498737",
"client_metadata": {
"features": "00000000000000ff",
"entity_id": "labo02",
"hostname": "iccluster045",
"kernel_version": "4.15.0-47-generic",
"root": "/labo02-scratch"
}
},
{
"id": 603998,
"num_leases": 0,
"num_caps": 0,
"state": "stale",
"request_load_avg": 0,
"uptime": 65096.702874342001,
"replay_requests": 0,
"completed_requests": 0,
"reconnecting": false,
"inst": "client.603998 v1:10.90.47.25:0/1985924462",
"client_metadata": {
"features": "00000000000000ff",
"entity_id": "labo05",
"hostname": "iccluster173.",
"kernel_version": "4.15.0-43-generic",
"root": "/labo05-scratch"
}
},
{
"id": 550857,
"num_leases": 0,
"num_caps": 0,
"state": "open",
"request_load_avg": 0,
"uptime": 65096.702874342001,
"replay_requests": 0,
"completed_requests": 1,
"reconnecting": false,
"inst": "client.550857 v1:10.90.47.32:0/689496241",
"client_metadata": {
"features": "00000000000000ff",
"entity_id": "labo04",
"hostname": "iccluster180.",
"kernel_version": "4.15.0-43-generic",
"root": "/labo04-scratch"
}
}
]
[
{
"id": 675490,
"num_leases": 0,
"num_caps": 4212,
"state": "open",
"request_load_avg": 0,
"uptime": 65091.644983728998,
"replay_requests": 0,
"completed_requests": 0,
"reconnecting": false,
"inst": "client.675490 v1:128.178.243.37:0/2728635645",
"client_metadata": {
"features": "00000000000000ff",
"entity_id": "cephfsadmin",
"hostname": "icitsrv5",
"kernel_version": "4.15.0-55-generic",
"root": "/"
}
},
{
"id": 89226,
"num_leases": 0,
"num_caps": 0,
"state": "open",
"request_load_avg": 0,
"uptime": 65095.492947045001,
"replay_requests": 0,
"completed_requests": 0,
"reconnecting": false,
"inst": "client.89226 v1:10.90.47.16:0/1643772475",
"client_metadata": {
"features": "00000000000000ff",
"entity_id": "labo01",
"hostname": "iccluster164.",
"kernel_version": "4.13.0-26-generic",
"root": "/labo01-scratch"
}
},
{
"id": 685982,
"num_leases": 0,
"num_caps": 0,
"state": "open",
"request_load_avg": 0,
"uptime": 65091.716983042999,
"replay_requests": 0,
"completed_requests": 0,
"reconnecting": false,
"inst": "client.685982 v1:10.90.36.16:0/79318340",
"client_metadata": {
"features": "00000000000000ff",
"entity_id": "cephfsadmin",
"hostname": "icadmin006.",
"kernel_version": "4.13.0-26-generic",
"root": "/"
}
},
{
"id": 651292,
"num_leases": 0,
"num_caps": 0,
"state": "open",
"request_load_avg": 0,
"uptime": 65095.492947045001,
"replay_requests": 0,
"completed_requests": 0,
"reconnecting": false,
"inst": "client.651292 v1:10.90.47.29:0/2037483206",
"client_metadata": {
"features": "00000000000000ff",
"entity_id": "labo04",
"hostname": "iccluster177.",
"kernel_version": "4.15.0-43-generic",
"root": "/labo04-scratch"
}
},
{
"id": 625347,
"num_leases": 0,
"num_caps": 0,
"state": "open",
"request_load_avg": 0,
"uptime": 65095.492947045001,
"replay_requests": 0,
"completed_requests": 0,
"reconnecting": false,
"inst": "client.625347 v1:10.90.47.34:0/2068926848",
"client_metadata": {
"features": "00000000000000ff",
"entity_id": "labo01",
"hostname": "iccluster182.",
"kernel_version": "4.15.0-43-generic",
"root": "/labo01-scratch"
}
},
{
"id": 624960,
"num_leases": 0,
"num_caps": 295,
"state": "open",
"request_load_avg": 3,
"uptime": 65095.492947045001,
"replay_requests": 0,
"completed_requests": 0,
"reconnecting": false,
"inst": "client.624960 v1:10.90.47.23:0/3391918921",
"client_metadata": {
"features": "00000000000000ff",
"entity_id": "labo02",
"hostname": "iccluster171.",
"kernel_version": "4.15.0-43-generic",
"root": "/labo02-scratch"
}
},
{
"id": 59533,
"num_leases": 0,
"num_caps": 4,
"state": "open",
"request_load_avg": 0,
"uptime": 65095.492947045001,
"replay_requests": 0,
"completed_requests": 1,
"reconnecting": false,
"inst": "client.59533 v1:10.90.47.5:0/1578661632",
"client_metadata": {
"features": "00000000000000ff",
"entity_id": "labo02",
"hostname": "iccluster153.",
"kernel_version": "4.13.0-26-generic",
"root": "/labo02-scratch"
}
},
{
"id": 21441,
"num_leases": 0,
"num_caps": 4517,
"state": "open",
"request_load_avg": 0,
"uptime": 65095.492947045001,
"replay_requests": 0,
"completed_requests": 0,
"reconnecting": false,
"inst": "client.21441 v1:10.90.47.19:0/1234150467",
"client_metadata": {
"features": "00000000000000ff",
"entity_id": "labo01",
"hostname": "iccluster167.",
"kernel_version": "4.13.0-26-generic",
"root": "/labo01-scratch"
}
},
{
"id": 126456,
"num_leases": 0,
"num_caps": 250,
"state": "open",
"request_load_avg": 0,
"uptime": 65095.492947045001,
"replay_requests": 0,
"completed_requests": 1,
"reconnecting": false,
"inst": "client.126456 v1:10.90.45.2:0/902274091",
"client_metadata": {
"features": "00000000000000ff",
"entity_id": "labo02",
"hostname": "iccluster130",
"kernel_version": "4.15.0-47-generic",
"root": "/labo02-scratch"
}
},
{
"id": 603998,
"num_leases": 0,
"num_caps": 0,
"state": "stale",
"request_load_avg": 0,
"uptime": 65095.492947045001,
"replay_requests": 0,
"completed_requests": 0,
"reconnecting": false,
"inst": "client.603998 v1:10.90.47.25:0/1985924462",
"client_metadata": {
"features": "00000000000000ff",
"entity_id": "labo05",
"hostname": "iccluster173.",
"kernel_version": "4.15.0-43-generic",
"root": "/labo05-scratch"
}
},
{
"id": 136975,
"num_leases": 0,
"num_caps": 402,
"state": "open",
"request_load_avg": 0,
"uptime": 65095.492947045001,
"replay_requests": 0,
"completed_requests": 1,
"reconnecting": false,
"inst": "client.136975 v1:10.90.38.21:0/918498737",
"client_metadata": {
"features": "00000000000000ff",
"entity_id": "labo02",
"hostname": "iccluster045",
"kernel_version": "4.15.0-47-generic",
"root": "/labo02-scratch"
}
},
{
"id": 458928,
"num_leases": 0,
"num_caps": 0,
"state": "open",
"request_load_avg": 0,
"uptime": 65095.492947045001,
"replay_requests": 0,
"completed_requests": 0,
"reconnecting": false,
"inst": "client.458928 v1:10.90.43.5:0/4126499073",
"client_metadata": {
"features": "00000000000000ff",
"entity_id": "labo03",
"hostname": "iclabo03srv16",
"kernel_version": "4.15.0-22-generic",
"root": "/labo03-scratch"
}
},
{
"id": 356929,
"num_leases": 0,
"num_caps": 0,
"state": "open",
"request_load_avg": 0,
"uptime": 65095.492947045001,
"replay_requests": 0,
"completed_requests": 0,
"reconnecting": false,
"inst": "client.356929 v1:10.90.47.4:0/2986438734",
"client_metadata": {
"features": "00000000000000ff",
"entity_id": "labo02",
"hostname": "iccluster152.",
"kernel_version": "4.13.0-26-generic",
"root": "/labo02-scratch"
}
},
{
"id": 550857,
"num_leases": 0,
"num_caps": 0,
"state": "open",
"request_load_avg": 0,
"uptime": 65095.492947045001,
"replay_requests": 0,
"completed_requests": 1,
"reconnecting": false,
"inst": "client.550857 v1:10.90.47.32:0/689496241",
"client_metadata": {
"features": "00000000000000ff",
"entity_id": "labo04",
"hostname": "iccluster180.",
"kernel_version": "4.15.0-43-generic",
"root": "/labo04-scratch"
}
},
{
"id": 585369,
"num_leases": 0,
"num_caps": 0,
"state": "open",
"request_load_avg": 0,
"uptime": 65095.492947045001,
"replay_requests": 0,
"completed_requests": 0,
"reconnecting": false,
"inst": "client.585369 v1:10.90.47.29:0/294922734",
"client_metadata": {
"features": "00000000000000ff",
"entity_id": "labo05",
"hostname": "iccluster177.",
"kernel_version": "4.15.0-43-generic",
"root": "/labo05-scratch"
}
},
{
"id": 515210,
"num_leases": 0,
"num_caps": 0,
"state": "open",
"request_load_avg": 0,
"uptime": 65095.492947045001,
"replay_requests": 0,
"completed_requests": 0,
"reconnecting": false,
"inst": "client.515210 v1:10.90.37.26:0/3975601407",
"client_metadata": {
"features": "00000000000009ff",
"entity_id": "cephfsadmin",
"hostname": "iccluster025",
"kernel_version": "5.0.0-20-generic",
"root": "/"
}
},
{
"id": 146846,
"num_leases": 0,
"num_caps": 0,
"state": "open",
"request_load_avg": 0,
"uptime": 65095.492947045001,
"replay_requests": 0,
"completed_requests": 0,
"reconnecting": false,
"inst": "client.146846 v1:10.90.47.13:0/28266276",
"client_metadata": {
"features": "00000000000000ff",
"entity_id": "labo02",
"hostname": "iccluster161.",
"kernel_version": "4.13.0-26-generic",
"root": "/labo02-scratch"
}
},
{
"id": 36482,
"num_leases": 0,
"num_caps": 0,
"state": "open",
"request_load_avg": 0,
"uptime": 65095.492947045001,
"replay_requests": 0,
"completed_requests": 0,
"reconnecting": false,
"inst": "client.36482 v1:10.90.37.13:0/461389346",
"client_metadata": {
"features": "00000000000000ff",
"entity_id": "cephfsadmin",
"hostname": "iccluster012",
"kernel_version": "4.15.0-52-generic",
"root": "/"
}
}
]
[
{
"id": 680860,
"num_leases": 0,
"num_caps": 1,
"state": "open",
"request_load_avg": 0,
"uptime": 8254.4797387510007,
"replay_requests": 0,
"completed_requests": 0,
"reconnecting": false,
"inst": "client.680860 v1:10.90.47.35:0/3082181526",
"client_metadata": {
"features": "00000000000000ff",
"entity_id": "labo02",
"hostname": "iccluster183.",
"kernel_version": "4.15.0-43-generic",
"root": "/labo02-scratch"
}
},
{
"id": 675490,
"num_leases": 0,
"num_caps": 9514,
"state": "open",
"request_load_avg": 0,
"uptime": 65094.048210403002,
"replay_requests": 0,
"completed_requests": 0,
"reconnecting": false,
"inst": "client.675490 v1:128.178.243.37:0/2728635645",
"client_metadata": {
"features": "00000000000000ff",
"entity_id": "cephfsadmin",
"hostname": "icitsrv5",
"kernel_version": "4.15.0-55-generic",
"root": "/"
}
},
{
"id": 685982,
"num_leases": 0,
"num_caps": 1,
"state": "open",
"request_load_avg": 0,
"uptime": 65094.380209141003,
"replay_requests": 0,
"completed_requests": 0,
"reconnecting": false,
"inst": "client.685982 v1:10.90.36.16:0/79318340",
"client_metadata": {
"features": "00000000000000ff",
"entity_id": "cephfsadmin",
"hostname": "icadmin006.",
"kernel_version": "4.13.0-26-generic",
"root": "/"
}
},
{
"id": 89226,
"num_leases": 0,
"num_caps": 5,
"state": "open",
"request_load_avg": 0,
"uptime": 65096.952199366002,
"replay_requests": 0,
"completed_requests": 1,
"reconnecting": false,
"inst": "client.89226 v1:10.90.47.16:0/1643772475",
"client_metadata": {
"features": "00000000000000ff",
"entity_id": "labo01",
"hostname": "iccluster164.",
"kernel_version": "4.13.0-26-generic",
"root": "/labo01-scratch"
}
},
{
"id": 651292,
"num_leases": 0,
"num_caps": 3,
"state": "open",
"request_load_avg": 0,
"uptime": 65096.952199366002,
"replay_requests": 0,
"completed_requests": 0,
"reconnecting": false,
"inst": "client.651292 v1:10.90.47.29:0/2037483206",
"client_metadata": {
"features": "00000000000000ff",
"entity_id": "labo04",
"hostname": "iccluster177.",
"kernel_version": "4.15.0-43-generic",
"root": "/labo04-scratch"
}
},
{
"id": 585369,
"num_leases": 0,
"num_caps": 2,
"state": "open",
"request_load_avg": 0,
"uptime": 65096.952199366002,
"replay_requests": 0,
"completed_requests": 0,
"reconnecting": false,
"inst": "client.585369 v1:10.90.47.29:0/294922734",
"client_metadata": {
"features": "00000000000000ff",
"entity_id": "labo05",
"hostname": "iccluster177.",
"kernel_version": "4.15.0-43-generic",
"root": "/labo05-scratch"
}
},
{
"id": 550857,
"num_leases": 0,
"num_caps": 9,
"state": "open",
"request_load_avg": 0,
"uptime": 65096.952199366002,
"replay_requests": 0,
"completed_requests": 0,
"reconnecting": false,
"inst": "client.550857 v1:10.90.47.32:0/689496241",
"client_metadata": {
"features": "00000000000000ff",
"entity_id": "labo04",
"hostname": "iccluster180.",
"kernel_version": "4.15.0-43-generic",
"root": "/labo04-scratch"
}
},
{
"id": 624960,
"num_leases": 0,
"num_caps": 2,
"state": "open",
"request_load_avg": 0,
"uptime": 65096.952199366002,
"replay_requests": 0,
"completed_requests": 1,
"reconnecting": false,
"inst": "client.624960 v1:10.90.47.23:0/3391918921",
"client_metadata": {
"features": "00000000000000ff",
"entity_id": "labo02",
"hostname": "iccluster171.",
"kernel_version": "4.15.0-43-generic",
"root": "/labo02-scratch"
}
},
{
"id": 356929,
"num_leases": 0,
"num_caps": 1,
"state": "open",
"request_load_avg": 0,
"uptime": 65096.952199366002,
"replay_requests": 0,
"completed_requests": 0,
"reconnecting": false,
"inst": "client.356929 v1:10.90.47.4:0/2986438734",
"client_metadata": {
"features": "00000000000000ff",
"entity_id": "labo02",
"hostname": "iccluster152.",
"kernel_version": "4.13.0-26-generic",
"root": "/labo02-scratch"
}
},
{
"id": 458928,
"num_leases": 0,
"num_caps": 1,
"state": "open",
"request_load_avg": 0,
"uptime": 65096.952199366002,
"replay_requests": 0,
"completed_requests": 0,
"reconnecting": false,
"inst": "client.458928 v1:10.90.43.5:0/4126499073",
"client_metadata": {
"features": "00000000000000ff",
"entity_id": "labo03",
"hostname": "iclabo03srv16",
"kernel_version": "4.15.0-22-generic",
"root": "/labo03-scratch"
}
},
{
"id": 59533,
"num_leases": 0,
"num_caps": 2,
"state": "open",
"request_load_avg": 0,
"uptime": 65096.952199366002,
"replay_requests": 0,
"completed_requests": 0,
"reconnecting": false,
"inst": "client.59533 v1:10.90.47.5:0/1578661632",
"client_metadata": {
"features": "00000000000000ff",
"entity_id": "labo02",
"hostname": "iccluster153.",
"kernel_version": "4.13.0-26-generic",
"root": "/labo02-scratch"
}
},
{
"id": 126456,
"num_leases": 0,
"num_caps": 5,
"state": "open",
"request_load_avg": 0,
"uptime": 65096.952199366002,
"replay_requests": 0,
"completed_requests": 0,
"reconnecting": false,
"inst": "client.126456 v1:10.90.45.2:0/902274091",
"client_metadata": {
"features": "00000000000000ff",
"entity_id": "labo02",
"hostname": "iccluster130",
"kernel_version": "4.15.0-47-generic",
"root": "/labo02-scratch"
}
},
{
"id": 21441,
"num_leases": 0,
"num_caps": 7661,
"state": "open",
"request_load_avg": 0,
"uptime": 65096.952199366002,
"replay_requests": 0,
"completed_requests": 0,
"reconnecting": false,
"inst": "client.21441 v1:10.90.47.19:0/1234150467",
"client_metadata": {
"features": "00000000000000ff",
"entity_id": "labo01",
"hostname": "iccluster167.",
"kernel_version": "4.13.0-26-generic",
"root": "/labo01-scratch"
}
},
{
"id": 36482,
"num_leases": 0,
"num_caps": 1,
"state": "open",
"request_load_avg": 0,
"uptime": 65096.952199366002,
"replay_requests": 0,
"completed_requests": 0,
"reconnecting": false,
"inst": "client.36482 v1:10.90.37.13:0/461389346",
"client_metadata": {
"features": "00000000000000ff",
"entity_id": "cephfsadmin",
"hostname": "iccluster012",
"kernel_version": "4.15.0-52-generic",
"root": "/"
}
},
{
"id": 136975,
"num_leases": 0,
"num_caps": 6,
"state": "open",
"request_load_avg": 0,
"uptime": 65096.952199366002,
"replay_requests": 0,
"completed_requests": 0,
"reconnecting": false,
"inst": "client.136975 v1:10.90.38.21:0/918498737",
"client_metadata": {
"features": "00000000000000ff",
"entity_id": "labo02",
"hostname": "iccluster045",
"kernel_version": "4.15.0-47-generic",
"root": "/labo02-scratch"
}
},
{
"id": 625347,
"num_leases": 0,
"num_caps": 2,
"state": "open",
"request_load_avg": 0,
"uptime": 65096.952199366002,
"replay_requests": 0,
"completed_requests": 0,
"reconnecting": false,
"inst": "client.625347 v1:10.90.47.34:0/2068926848",
"client_metadata": {
"features": "00000000000000ff",
"entity_id": "labo01",
"hostname": "iccluster182.",
"kernel_version": "4.15.0-43-generic",
"root": "/labo01-scratch"
}
},
{
"id": 603998,
"num_leases": 0,
"num_caps": 2,
"state": "stale",
"request_load_avg": 0,
"uptime": 65096.952199366002,
"replay_requests": 0,
"completed_requests": 0,
"reconnecting": false,
"inst": "client.603998 v1:10.90.47.25:0/1985924462",
"client_metadata": {
"features": "00000000000000ff",
"entity_id": "labo05",
"hostname": "iccluster173.",
"kernel_version": "4.15.0-43-generic",
"root": "/labo05-scratch"
}
}
]
{
"ops": [
{
"description": "client_request(client.21441:177047 getattr pAsLsXsFs #0x10000f2b1b3 2019-09-18 18:34:07.876611 caller_uid=204878, caller_gid=11233{})",
"initiated_at": "2019-09-18 18:34:07.880457",
"age": 60155.081965347999,
"duration": 60155.082026488999,
"type_data": {
"flag_point": "failed to rdlock, waiting",
"reqid": "client.21441:177047",
"op_type": "client_request",
"client_info": {
"client": "client.21441",
"tid": 177047
},
"events": [
{
"time": "2019-09-18 18:34:07.880457",
"event": "initiated"
},
{
"time": "2019-09-18 18:34:07.880457",
"event": "header_read"
},
{
"time": "2019-09-18 18:34:07.880460",
"event": "throttled"
},
{
"time": "2019-09-18 18:34:07.880470",
"event": "all_read"
},
{
"time": "2019-09-18 18:34:07.880539",
"event": "dispatched"
},
{
"time": "2019-09-18 18:34:07.880602",
"event": "failed to rdlock, waiting"
}
]
}
},
{
"description": "client_request(client.675490:745 getattr pAsLsXsFs #0x10000f2b1b3 2019-09-19 09:22:40.361958 caller_uid=0, caller_gid=0{})",
"initiated_at": "2019-09-19 09:22:40.367750",
"age": 6842.5946725849999,
"duration": 6842.594808926,
"type_data": {
"flag_point": "failed to rdlock, waiting",
"reqid": "client.675490:745",
"op_type": "client_request",
"client_info": {
"client": "client.675490",
"tid": 745
},
"events": [
{
"time": "2019-09-19 09:22:40.367750",
"event": "initiated"
},
{
"time": "2019-09-19 09:22:40.367750",
"event": "header_read"
},
{
"time": "2019-09-19 09:22:40.367753",
"event": "throttled"
},
{
"time": "2019-09-19 09:22:40.367769",
"event": "all_read"
},
{
"time": "2019-09-19 09:22:40.367821",
"event": "dispatched"
},
{
"time": "2019-09-19 09:22:40.367899",
"event": "failed to rdlock, waiting"
}
]
}
},
{
"description": "client_request(client.675490:746 getattr pAsLsXsFs #0x10000f2b1b3 2019-09-19 09:29:49.743805 caller_uid=0, caller_gid=0{})",
"initiated_at": "2019-09-19 09:29:49.749383",
"age": 6413.2130391119999,
"duration": 6413.2132385490004,
"type_data": {
"flag_point": "failed to rdlock, waiting",
"reqid": "client.675490:746",
"op_type": "client_request",
"client_info": {
"client": "client.675490",
"tid": 746
},
"events": [
{
"time": "2019-09-19 09:29:49.749383",
"event": "initiated"
},
{
"time": "2019-09-19 09:29:49.749383",
"event": "header_read"
},
{
"time": "2019-09-19 09:29:49.749387",
"event": "throttled"
},
{
"time": "2019-09-19 09:29:49.749402",
"event": "all_read"
},
{
"time": "2019-09-19 09:29:49.749459",
"event": "dispatched"
},
{
"time": "2019-09-19 09:29:49.749554",
"event": "failed to rdlock, waiting"
}
]
}
},
{
"description": "client_request(client.675490:747 getattr pAsLsXsFs #0x10000f2b1b3 2019-09-19 09:36:01.859752 caller_uid=0, caller_gid=0{})",
"initiated_at": "2019-09-19 09:36:01.864560",
"age": 6041.0978630569998,
"duration": 6041.0981237579999,
"type_data": {
"flag_point": "failed to rdlock, waiting",
"reqid": "client.675490:747",
"op_type": "client_request",
"client_info": {
"client": "client.675490",
"tid": 747
},
"events": [
{
"time": "2019-09-19 09:36:01.864560",
"event": "initiated"
},
{
"time": "2019-09-19 09:36:01.864560",
"event": "header_read"
},
{
"time": "2019-09-19 09:36:01.864564",
"event": "throttled"
},
{
"time": "2019-09-19 09:36:01.864588",
"event": "all_read"
},
{
"time": "2019-09-19 09:36:01.864646",
"event": "dispatched"
},
{
"time": "2019-09-19 09:36:01.864755",
"event": "failed to rdlock, waiting"
}
]
}
},
{
"description": "client_request(client.21441:176427 rename #0x10000f2f817/16-72 #0x10000f2b1b3/16-72 2019-09-18 17:32:06.281271 caller_uid=204878, caller_gid=11233{})",
"initiated_at": "2019-09-18 17:32:06.285796",
"age": 63876.676626355002,
"duration": 63876.676952567002,
"type_data": {
"flag_point": "waiting for remote wrlocks",
"reqid": "client.21441:176427",
"op_type": "client_request",
"client_info": {
"client": "client.21441",
"tid": 176427
},
"events": [
{
"time": "2019-09-18 17:32:06.285796",
"event": "initiated"
},
{
"time": "2019-09-18 17:32:06.285796",
"event": "header_read"
},
{
"time": "2019-09-18 17:32:06.285798",
"event": "throttled"
},
{
"time": "2019-09-18 17:32:06.285804",
"event": "all_read"
},
{
"time": "2019-09-18 17:32:06.285820",
"event": "dispatched"
},
{
"time": "2019-09-18 17:32:06.286334",
"event": "requesting remote authpins"
},
{
"time": "2019-09-18 17:32:06.286763",
"event": "failed to xlock, waiting"
},
{
"time": "2019-09-18 17:32:06.287566",
"event": "waiting for remote wrlocks"
},
{
"time": "2019-09-18 17:32:06.287954",
"event": "waiting for remote wrlocks"
}
]
}
},
{
"description": "client_request(client.21441:176429 getattr pAsLsXsFs #0x10000f2b1b3 2019-09-18 17:49:37.806002 caller_uid=204878, caller_gid=11233{})",
"initiated_at": "2019-09-18 17:49:37.810351",
"age": 62825.152072056,
"duration": 62825.152482142003,
"type_data": {
"flag_point": "failed to rdlock, waiting",
"reqid": "client.21441:176429",
"op_type": "client_request",
"client_info": {
"client": "client.21441",
"tid": 176429
},
"events": [
{
"time": "2019-09-18 17:49:37.810351",
"event": "initiated"
},
{
"time": "2019-09-18 17:49:37.810351",
"event": "header_read"
},
{
"time": "2019-09-18 17:49:37.810355",
"event": "throttled"
},
{
"time": "2019-09-18 17:49:37.810374",
"event": "all_read"
},
{
"time": "2019-09-18 17:49:37.810454",
"event": "dispatched"
},
{
"time": "2019-09-18 17:49:37.810555",
"event": "failed to rdlock, waiting"
}
]
}
},
{
"description": "client_request(client.21441:176428 getattr pAsLsXsFs #0x10000f2b1b3 2019-09-18 17:32:09.881710 caller_uid=204878, caller_gid=11233{})",
"initiated_at": "2019-09-18 17:32:09.886578",
"age": 63873.075844398001,
"duration": 63873.076315585,
"type_data": {
"flag_point": "failed to rdlock, waiting",
"reqid": "client.21441:176428",
"op_type": "client_request",
"client_info": {
"client": "client.21441",
"tid": 176428
},
"events": [
{
"time": "2019-09-18 17:32:09.886578",
"event": "initiated"
},
{
"time": "2019-09-18 17:32:09.886578",
"event": "header_read"
},
{
"time": "2019-09-18 17:32:09.886582",
"event": "throttled"
},
{
"time": "2019-09-18 17:32:09.886606",
"event": "all_read"
},
{
"time": "2019-09-18 17:32:09.886662",
"event": "dispatched"
},
{
"time": "2019-09-18 17:32:09.886783",
"event": "failed to rdlock, waiting"
}
]
}
},
{
"description": "client_request(client.21441:176430 getattr pAsLsXsFs #0x10000f2b1b3 2019-09-18 18:32:19.165797 caller_uid=204878, caller_gid=11233{})",
"initiated_at": "2019-09-18 18:32:19.169824",
"age": 60263.792598510001,
"duration": 60263.793131118,
"type_data": {
"flag_point": "failed to rdlock, waiting",
"reqid": "client.21441:176430",
"op_type": "client_request",
"client_info": {
"client": "client.21441",
"tid": 176430
},
"events": [
{
"time": "2019-09-18 18:32:19.169824",
"event": "initiated"
},
{
"time": "2019-09-18 18:32:19.169824",
"event": "header_read"
},
{
"time": "2019-09-18 18:32:19.169828",
"event": "throttled"
},
{
"time": "2019-09-18 18:32:19.169849",
"event": "all_read"
},
{
"time": "2019-09-18 18:32:19.169908",
"event": "dispatched"
},
{
"time": "2019-09-18 18:32:19.170014",
"event": "failed to rdlock, waiting"
}
]
}
},
{
"description": "rejoin:mds.1:13",
"initiated_at": "2019-09-18 17:48:52.065114",
"age": 62870.897309002001,
"duration": 62870.897899360003,
"type_data": {
"flag_point": "dispatched",
"reqid": "mds.1:13",
"op_type": "no_available_op_found",
"events": [
{
"time": "2019-09-18 17:48:52.065114",
"event": "initiated"
},
{
"time": "2019-09-18 17:48:52.065114",
"event": "header_read"
},
{
"time": "2019-09-18 17:48:52.065110",
"event": "throttled"
},
{
"time": "2019-09-18 17:48:52.065125",
"event": "all_read"
},
{
"time": "2019-09-18 17:48:52.065191",
"event": "dispatched"
}
]
}
},
{
"description": "rejoin:mds.1:14",
"initiated_at": "2019-09-18 17:48:52.065131",
"age": 62870.897291588,
"duration": 62870.897927448001,
"type_data": {
"flag_point": "dispatched",
"reqid": "mds.1:14",
"op_type": "no_available_op_found",
"events": [
{
"time": "2019-09-18 17:48:52.065131",
"event": "initiated"
},
{
"time": "2019-09-18 17:48:52.065131",
"event": "header_read"
},
{
"time": "2019-09-18 17:48:52.065128",
"event": "throttled"
},
{
"time": "2019-09-18 17:48:52.065139",
"event": "all_read"
},
{
"time": "2019-09-18 17:48:52.065333",
"event": "dispatched"
}
]
}
}
],
"num_ops": 10
}
{
"ops": [
{
"description": "slave_request(client.21441:176427.0 wrlock)",
"initiated_at": "2019-09-18 17:32:06.285635",
"age": 63867.106892331998,
"duration": 63867.106928321999,
"type_data": {
"flag_point": "failed to wrlock, waiting",
"reqid": "client.21441:176427",
"op_type": "slave_request",
"master_info": {
"master": "mds.0"
},
"request_info": {
"attempt": 0,
"op_type": "wrlock",
"lock_type": 1024,
"object_info": "0x10000f2b1b3.head",
"srcdnpath": "",
"destdnpath": "",
"witnesses": "",
"has_inode_export": false,
"inode_export_v": 0,
"op_stamp": "0.000000"
},
"events": [
{
"time": "2019-09-18 17:32:06.285635",
"event": "initiated"
},
{
"time": "2019-09-18 17:32:06.285635",
"event": "header_read"
},
{
"time": "2019-09-18 17:32:06.285633",
"event": "throttled"
},
{
"time": "2019-09-18 17:32:06.285645",
"event": "all_read"
},
{
"time": "2019-09-18 17:32:06.285701",
"event": "dispatched"
},
{
"time": "2019-09-18 17:32:06.286195",
"event": "failed to xlock, waiting"
},
{
"time": "2019-09-18 17:32:06.286558",
"event": "acquired locks"
},
{
"time": "2019-09-18 17:32:06.287019",
"event": "acquired locks"
},
{
"time": "2019-09-18 17:32:06.287312",
"event": "failed to wrlock, waiting"
},
{
"time": "2019-09-18 17:32:06.289753",
"event": "failed to wrlock, waiting"
}
]
}
},
{
"description": "internal op exportdir:mds.1:13",
"initiated_at": "2019-09-18 17:48:47.735459",
"age": 62865.657068250999,
"duration": 62865.657216626001,
"type_data": {
"flag_point": "failed to wrlock, waiting",
"reqid": "mds.1:13",
"op_type": "internal_op",
"internal_op": 5377,
"op_name": "exportdir",
"events": [
{
"time": "2019-09-18 17:48:47.735459",
"event": "initiated"
},
{
"time": "2019-09-18 17:48:47.735459",
"event": "header_read"
},
{
"time": "2019-09-18 17:48:47.735459",
"event": "throttled"
},
{
"time": "2019-09-18 17:48:47.735459",
"event": "all_read"
},
{
"time": "2019-09-18 17:48:47.735459",
"event": "dispatched"
},
{
"time": "2019-09-18 17:48:52.064229",
"event": "requesting remote authpins"
},
{
"time": "2019-09-18 17:48:52.064953",
"event": "failed to wrlock, waiting"
}
]
}
},
{
"description": "internal op exportdir:mds.1:14",
"initiated_at": "2019-09-18 17:48:47.735533",
"age": 62865.656994645004,
"duration": 62865.657206085001,
"type_data": {
"flag_point": "failed to wrlock, waiting",
"reqid": "mds.1:14",
"op_type": "internal_op",
"internal_op": 5377,
"op_name": "exportdir",
"events": [
{
"time": "2019-09-18 17:48:47.735533",
"event": "initiated"
},
{
"time": "2019-09-18 17:48:47.735533",
"event": "header_read"
},
{
"time": "2019-09-18 17:48:47.735533",
"event": "throttled"
},
{
"time": "2019-09-18 17:48:47.735533",
"event": "all_read"
},
{
"time": "2019-09-18 17:48:47.735533",
"event": "dispatched"
},
{
"time": "2019-09-18 17:48:52.064255",
"event": "requesting remote authpins"
},
{
"time": "2019-09-18 17:48:52.064982",
"event": "failed to wrlock, waiting"
}
]
}
}
],
"num_ops": 3
}
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
