Re: High apply latency

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 

Hi Jakub,


Le 06/02/2018 à 16:03, Jakub Jaszewski a écrit :
​Hi Frederic,

I've not enable debug level logging on all OSDs, just on one for the test, need to double check that.
But looks that merging is ongoing on few OSDs or OSDs are faulty, I will dig into that tomorrow.
Write bandwidth is very random

I just reread the whole thread:

- Splitting is not happening anymore - if it ever did - that's for sure.
- Regarding the write bandwidth variations, it seems that these variations only concern EC 6+3 pools.
- As you get more than a 1.2 GB/s on replicated pools with 4MB iops, I would think that neither NVMe, nor PERC or HDDs is to blame.

Did you check CPU load during EC 6+3 writes on pool default.rgw.buckets.data ?

If you don't see any 100% CPU load, nor any 100% iostat issues on either the NVMe disk or HDDs, then I would benchmark the network for bandwidth or latency issues.

BTW, did you see that some of your OSDs were not tagged as 'hdd' (ceph osd df tree).

# rados bench -p default.rgw.buckets.data 120 write
hints = 1
Maintaining 16 concurrent writes of 4194432 bytes to objects of size 4194432 for up to 120 seconds or 0 objects
Object prefix: benchmark_data_sg08-09_59104
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16       155       139    555.93   556.017   0.0750027     0.10687
    2      16       264       248   495.936   436.013    0.154185    0.118693
    3      16       330       314   418.616   264.008    0.118476    0.142667
    4      16       415       399   398.953    340.01   0.0873379     0.15102
    5      16       483       467   373.557   272.008    0.750453    0.159819
    6      16       532       516   343.962   196.006   0.0298334    0.171218
    7      16       617       601   343.391    340.01    0.192698    0.177288
    8      16       700       684   341.963    332.01   0.0281355    0.171277
    9      16       762       746   331.521   248.008   0.0962037    0.163734
   10      16       804       788   315.167   168.005     1.40356    0.196298
   11      16       897       881    320.33   372.011   0.0369085     0.19496
   12      16       985       969   322.966   352.011   0.0290563    0.193986
   13      15      1106      1091   335.657   488.015   0.0617642    0.188703
   14      16      1166      1150   328.537   236.007   0.0401884    0.186206
   15      16      1251      1235   329.299    340.01    0.171256    0.190974
   16      16      1339      1323   330.716   352.011    0.024222    0.189901
   17      16      1417      1401   329.613    312.01   0.0289473    0.186562
   18      16      1465      1449   321.967   192.006    0.028123    0.189153
   19      16      1522      1506    317.02   228.007    0.265448    0.188288
2018-02-06 13:43:21.412512 min lat: 0.0204657 max lat: 3.61509 avg lat: 0.18918
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      16      1564      1548   309.568   168.005   0.0327581     0.18918
   21      16      1636      1620    308.54   288.009   0.0715159    0.187381
   22      16      1673      1657   301.242   148.005     1.57285    0.191596
   23      16      1762      1746   303.621   356.011     6.00352    0.206217
   24      16      1885      1869   311.468   492.015   0.0298435    0.203874
   25      16      2010      1994   319.008   500.015   0.0258761    0.199652
   26      16      2116      2100   323.044   424.013   0.0533319     0.19631
   27      16      2201      2185    323.67    340.01    0.134796    0.195953
   28      16      2257      2241    320.11   224.007    0.473629    0.196464
   29      16      2333      2317   319.554   304.009   0.0362741    0.198054
   30      16      2371      2355   313.968   152.005    0.438141    0.200265
   31      16      2459      2443   315.194   352.011   0.0610629    0.200858
   32      16      2525      2509   313.593   264.008   0.0234799    0.201008
   33      16      2612      2596   314.635   348.011    0.072019    0.199094
   34      16      2682      2666   313.615   280.009     0.10062    0.197586
   35      16      2757      2741   313.225   300.009   0.0552581    0.196981
   36      16      2849      2833   314.746   368.011    0.257323     0.19565
   37      16      2891      2875   310.779   168.005   0.0918386     0.19556
   38      16      2946      2930    308.39   220.007   0.0276621    0.195792
   39      16      2975      2959   303.456   116.004   0.0588971     0.19952
2018-02-06 13:43:41.415107 min lat: 0.0204657 max lat: 7.9873 avg lat: 0.198749
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   40      16      3060      3044   304.369    340.01   0.0217136    0.198749
   41      16      3098      3082   300.652   152.005   0.0717398    0.199052
   42      16      3141      3125   297.589   172.005   0.0257422    0.201899
   43      15      3241      3226   300.063   404.012   0.0733869    0.209446
   44      16      3332      3316   301.424   360.011   0.0327249    0.206686
   45      16      3430      3414   303.436   392.012   0.0413156    0.203727
   46      16      3534      3518   305.882   416.013    0.033638    0.202182
   47      16      3602      3586   305.161   272.008   0.0453557    0.200028
   48      16      3663      3647   303.886   244.007   0.0779019    0.199777
   49      16      3736      3720   303.643   292.009   0.0285231    0.206274
   50      16      3849      3833   306.609   452.014   0.0537071    0.208127
   51      16      3909      3893   305.303   240.007   0.0366709    0.207793
   52      16      3972      3956   304.277   252.008   0.0289131    0.207989
   53      16      4048      4032   304.272   304.009   0.0348617    0.207844
   54      16      4114      4098   303.525   264.008   0.0799526     0.20701
   55      16      4191      4175   303.606   308.009    0.034915    0.206882
   56      16      4313      4297   306.898   488.015    0.108777    0.205869
   57      16      4449      4433   311.057   544.017   0.0862092    0.205232
   58      16      4548      4532    312.52   396.012    0.135814    0.203753
   59      16      4678      4662   316.036   520.016   0.0307446    0.202156
2018-02-06 13:44:01.417687 min lat: 0.0204657 max lat: 13.0888 avg lat: 0.201848
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   60      16      4734      4718   314.502   224.007    0.223548    0.201848
   61      16      4783      4767   312.559   196.006    0.124237    0.203641
   62      16      4838      4822   311.066   220.007    0.270943    0.204074
   63      16      4929      4913   311.905   364.011    0.123486    0.204716
   64      16      5063      5047   315.406   536.016    0.101675    0.202283
   65      16      5124      5108   314.307   244.007   0.0504368    0.201181
   66      16      5243      5227   316.756   476.015    0.235348    0.201481
   67      16      5339      5323   317.759   384.012   0.0478478    0.200486
   68      16      5414      5398   317.497   300.009    0.258401    0.200194
   69      16      5461      5445    315.62   188.006    0.112516     0.20022
   70      16      5523      5507   314.654   248.008   0.0927405    0.201534
   71      16      5582      5566   313.546   236.007    0.333586    0.202304
   72      16      5690      5674    315.19   432.013   0.0706812    0.201498
   73      16      5780      5764   315.804   360.011   0.0306772    0.200354
   74      16      5850      5834    315.32   280.009   0.0261027    0.200627
   75      16      5905      5889   314.048   220.007    0.101282     0.20075
   76      16      6013      5997     315.6   432.013    0.161956    0.202154
   77      16      6130      6114   317.578   468.014    0.042322     0.20092
   78      16      6232      6216   318.737   408.012   0.0912166    0.199238
   79      16      6260      6244    316.12   112.003   0.0406156    0.198971
2018-02-06 13:44:21.420343 min lat: 0.0204657 max lat: 13.0888 avg lat: 0.200443
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   80      16      6304      6288   314.368   176.005   0.0409417    0.200443
   81      16      6375      6359   313.993   284.009    0.150617     0.19986
   82      16      6447      6431   313.676   288.009   0.0730616    0.200072
   83      16      6513      6497   313.077   264.008   0.0565517    0.202155
   84      16      6594      6578   313.206    324.01   0.0273074    0.202981
   85      16      6732      6716   316.015   552.017    0.134015    0.202301
   86      16      6878      6862   319.131   584.018    0.157684    0.200182
   87      16      6971      6955   319.738   372.011    0.327923    0.199431
   88      16      7018      7002   318.241   188.006   0.0317034    0.198737
   89      16      7056      7040   316.373   152.005   0.0874119    0.199579
   90      16      7095      7079    314.59   156.005    0.027061    0.198883
   91      16      7150      7134   313.551   220.007    0.247791    0.199385
   92      16      7201      7185    312.36   204.006    0.131592    0.202395
   93      16      7216      7200   309.646   60.0018   0.0678182    0.202927
   94      16      7276      7260   308.905   240.007    0.040735    0.204362
   95      16      7304      7288   306.832   112.003     0.04629    0.204849
   96      16      7402      7386   307.719   392.012    0.085536    0.206601
   97      16      7524      7508   309.577   488.015    0.904426    0.205316
   98      16      7682      7666   312.866   632.019   0.0580214    0.204241
   99      16      7840      7824   316.089   632.019     0.13412    0.202289
2018-02-06 13:44:41.423052 min lat: 0.0204657 max lat: 13.0888 avg lat: 0.200948
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  100      16      7949      7933   317.288   436.013    0.025568    0.200948
  101      16      7996      7980   316.007   188.006    0.328559    0.201496
  102      16      8020      8004    313.85   96.0029   0.0524293     0.20214
  103      16      8044      8028   311.735   96.0029   0.0628428     0.20236
  104      16      8062      8046    309.43   72.0022   0.0320078    0.202743
  105      16      8088      8072   307.473   104.003   0.0506497    0.204222
  106      16      8127      8111   306.044   156.005   0.0436112     0.20792
  107      16      8226      8210   306.885   396.012   0.0452614    0.207666
  108      16      8296      8280   306.635   280.009   0.0500199    0.207578
  109      16      8367      8351   306.428   284.009   0.0364288    0.207779
  110      16      8475      8459   307.569   432.013    0.140141    0.206752
  111      16      8559      8543   307.825    336.01   0.0250007    0.206032
  112      16      8696      8680   309.968   548.017   0.0249451    0.205808
  113      16      8804      8788   311.048   432.013     0.49485    0.205075
  114      16      8917      8901   312.284   452.014   0.0731665    0.204527
  115      16      9047      9031    314.09   520.016   0.0365535    0.203553
  116      16      9130      9114   314.244    332.01   0.0451302    0.203367
  117      16      9259      9243   315.968   516.016   0.0610521    0.202253
  118      16      9373      9357   317.154   456.014   0.0282051    0.201613
  119      16      9465      9449   317.581   368.011   0.0299575    0.200845
2018-02-06 13:45:01.425689 min lat: 0.0204657 max lat: 13.0888 avg lat: 0.200298
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  120      16      9545      9529   317.601    320.01   0.0264095    0.200298
  121       7      9546      9539   315.307   40.0012    0.659852    0.201486
  122       7      9546      9539   312.722         0           -    0.201486
Total time run:         122.442935
Total writes made:      9546
Write size:             4194432
Object size:            4194432
Bandwidth (MB/sec):     311.861
Stddev Bandwidth:       133.889
Max bandwidth (MB/sec): 632.019
Min bandwidth (MB/sec): 0
Average IOPS:           77
Stddev IOPS:            33
Max IOPS:               158
Min IOPS:               0
Average Latency(s):     0.203467
Stddev Latency(s):      0.531046
Max latency(s):         13.0888
Min latency(s):         0.0204657
Cleaning up (deleting benchmark objects)
Removed 9546 objects
Clean up completed and total clean up time :308.444893
#

Many slow requests occurred in log once Cleaning up started. Affected OSDs: osd.53 and osd.91 are located on different nodes.

Slow requests on cleanup is surely due to merging.

2018-02-06 13:42:49.881647 mon.sg01-06 mon.0 10.212.32.18:6789/0 3574 : cluster [WRN] overall HEALTH_WARN noscrub,nodeep-scrub flag(s) set
2018-02-06 13:44:49.777466 mon.sg01-06 mon.0 10.212.32.18:6789/0 3577 : cluster [INF] mon.1 10.212.32.19:6789/0 
2018-02-06 13:44:49.777627 mon.sg01-06 mon.0 10.212.32.18:6789/0 3578 : cluster [INF] mon.2 10.212.32.20:6789/0 
2018-02-06 13:45:22.221472 mon.sg01-06 mon.0 10.212.32.18:6789/0 3579 : cluster [WRN] Health check failed: 4 slow requests are blocked > 32 sec (REQUEST_SLOW)
2018-02-06 13:45:26.227333 mon.sg01-06 mon.0 10.212.32.18:6789/0 3580 : cluster [INF] Health check cleared: REQUEST_SLOW (was: 4 slow requests are blocked > 32 sec)
2018-02-06 13:45:40.325885 mon.sg01-06 mon.0 10.212.32.18:6789/0 3582 : cluster [WRN] Health check failed: 26 slow requests are blocked > 32 sec (REQUEST_SLOW)
2018-02-06 13:45:46.360383 mon.sg01-06 mon.0 10.212.32.18:6789/0 3583 : cluster [INF] Health check cleared: REQUEST_SLOW (was: 26 slow requests are blocked > 32 sec)
2018-02-06 13:46:08.518861 mon.sg01-06 mon.0 10.212.32.18:6789/0 3584 : cluster [WRN] Health check failed: 1 slow requests are blocked > 32 sec (REQUEST_SLOW)
2018-02-06 13:46:13.806928 mon.sg01-06 mon.0 10.212.32.18:6789/0 3585 : cluster [WRN] Health check update: 9 slow requests are blocked > 32 sec (REQUEST_SLOW)
2018-02-06 13:46:18.807371 mon.sg01-06 mon.0 10.212.32.18:6789/0 3586 : cluster [WRN] Health check update: 20 slow requests are blocked > 32 sec (REQUEST_SLOW)
2018-02-06 13:46:23.807835 mon.sg01-06 mon.0 10.212.32.18:6789/0 3587 : cluster [WRN] Health check update: 15 slow requests are blocked > 32 sec (REQUEST_SLOW)
2018-02-06 13:46:19.082992 osd.53 osd.53 10.212.32.22:6802/50082 167 : cluster [WRN] 1 slow requests, 1 included below; oldest blocked for > 30.071323 secs
2018-02-06 13:46:19.082999 osd.53 osd.53 10.212.32.22:6802/50082 168 : cluster [WRN] slow request 30.071323 seconds old, received at 2018-02-06 13:45:49.011626: osd_op(client.10075.0:25660466 3.50964d18 3:18b2690a:::rbd_data.bcc718ab7e41.000000000001018d:head [write 2703360~8192] snapc 0=[] ack+ondisk+write+known_if_redirected e5954) currently journaled_completion_queued
2018-02-06 13:46:21.083367 osd.53 osd.53 10.212.32.22:6802/50082 169 : cluster [WRN] 2 slow requests, 1 included below; oldest blocked for > 32.071703 secs
2018-02-06 13:46:21.083390 osd.53 osd.53 10.212.32.22:6802/50082 170 : cluster [WRN] slow request 30.141592 seconds old, received at 2018-02-06 13:45:50.941737: osd_op(client.841436.0:77497677 3.52e3e8ff 3:ff17c74a:::rbd_data.cd25332916c98.0000000000012b0b:head [write 245760~3612672] snapc 0=[] ack+ondisk+write+known_if_redirected e5954) currently commit_sent
2018-02-06 13:46:28.808214 mon.sg01-06 mon.0 10.212.32.18:6789/0 3589 : cluster [WRN] Health check update: 16 slow requests are blocked > 32 sec (REQUEST_SLOW)
2018-02-06 13:46:33.808664 mon.sg01-06 mon.0 10.212.32.18:6789/0 3590 : cluster [WRN] Health check update: 17 slow requests are blocked > 32 sec (REQUEST_SLOW)
2018-02-06 13:46:38.809093 mon.sg01-06 mon.0 10.212.32.18:6789/0 3591 : cluster [WRN] Health check update: 1 slow requests are blocked > 32 sec (REQUEST_SLOW)
2018-02-06 13:46:43.809537 mon.sg01-06 mon.0 10.212.32.18:6789/0 3592 : cluster [WRN] Health check update: 2 slow requests are blocked > 32 sec (REQUEST_SLOW)
2018-02-06 13:46:44.760767 mon.sg01-06 mon.0 10.212.32.18:6789/0 3593 : cluster [INF] Health check cleared: REQUEST_SLOW (was: 2 slow requests are blocked > 32 sec)
2018-02-06 13:46:52.819718 mon.sg01-06 mon.0 10.212.32.18:6789/0 3594 : cluster [WRN] Health check failed: 70 slow requests are blocked > 32 sec (REQUEST_SLOW)
2018-02-06 13:46:59.674390 mon.sg01-06 mon.0 10.212.32.18:6789/0 3595 : cluster [WRN] Health check update: 105 slow requests are blocked > 32 sec (REQUEST_SLOW)
2018-02-06 13:47:01.444356 osd.91 osd.91 10.212.32.25:6818/81813 5 : cluster [WRN] 1 slow requests, 1 included below; oldest blocked for > 30.028512 secs
2018-02-06 13:47:01.444365 osd.91 osd.91 10.212.32.25:6818/81813 6 : cluster [WRN] slow request 30.028512 seconds old, received at 2018-02-06 13:46:31.415754: osd_op(client.757449.0:155792306 3.92bf145a 3:5a28fd49:::rbd_data.8e125f170c6e.00000000000318b3:head [write 176128~4096] snapc 0=[] ack+ondisk+write+known_if_redirected e5954) currently sub_op_commit_rec from 72
2018-02-06 13:47:02.444612 osd.91 osd.91 10.212.32.25:6818/81813 7 : cluster [WRN] 12 slow requests, 5 included below; oldest blocked for > 31.028754 secs
2018-02-06 13:47:02.444637 osd.91 osd.91 10.212.32.25:6818/81813 12 : cluster [WRN] slow request 30.121723 seconds old, received at 2018-02-06 13:46:32.322785: osd_op(client.757449.0:155792349 3.92bf145a 3:5a28fd49:::rbd_data.8e125f170c6e.00000000000318b3:head [write 217088~4096] snapc 0=[] ack+ondisk+write+known_if_redirected e5954) currently sub_op_commit_rec from 72
2018-02-06 13:47:01.089688 osd.53 osd.53 10.212.32.22:6802/50082 171 : cluster [WRN] 1 slow requests, 1 included below; oldest blocked for > 30.546028 secs
2018-02-06 13:47:01.089695 osd.53 osd.53 10.212.32.22:6802/50082 172 : cluster [WRN] slow request 30.546028 seconds old, received at 2018-02-06 13:46:30.543617: osd_op(client.48366.0:29692284 3.859d4e2b 3.859d4e2b (undecoded) ack+ondisk+write+known_if_redirected e5954) currently queued_for_pg
2018-02-06 13:47:02.089919 osd.53 osd.53 10.212.32.22:6802/50082 173 : cluster [WRN] 9 slow requests, 5 included below; oldest blocked for > 31.546245 secs
2018-02-06 13:47:02.089926 osd.53 osd.53 10.212.32.22:6802/50082 174 : cluster [WRN] slow request 30.414557 seconds old, received at 2018-02-06 13:46:31.675305: osd_repop(client.757449.0:155792317 3.5a e5954/5842) currently queued_for_pg
2018-02-06 13:47:04.445182 osd.91 osd.91 10.212.32.25:6818/81813 24 : cluster [WRN] slow request 30.496769 seconds old, received at 2018-02-06 13:46:33.948299: osd_op(client.757449.0:155792384 3.92bf145a 3:5a28fd49:::rbd_data.8e125f170c6e.00000000000318b3:head [write 262144~4096] snapc 0=[] ack+ondisk+write+known_if_redirected e5954) currently sub_op_commit_rec from 72
2018-02-06 13:50:11.133054 osd.53 osd.53 10.212.32.22:6802/50082 864 : cluster [WRN] slow request 120.604868 seconds old, received at 2018-02-06 13:48:10.528083: osd_op(client.840950.0:169909382 3.cf2d18ff 3.cf2d18ff (undecoded) ack+ondisk+write+known_if_redirected e5954) currently queued_for_pg
2018-02-06 13:50:11.133058 osd.53 osd.53 10.212.32.22:6802/50082 865 : cluster [WRN] slow request 61.985051 seconds old, received at 2018-02-06 13:49:09.147900: osd_op(client.749595.0:78222152 3.4347d828 3.4347d828 (undecoded) ack+ondisk+write+known_if_redirected e5954) currently queued_for_pg
2018-02-06 13:50:11.133061 osd.53 osd.53 10.212.32.22:6802/50082 866 : cluster [WRN] slow request 31.256269 seconds old, received at 2018-02-06 13:49:39.876682: osd_op(client.749595.0:78223068 3.4347d828 3.4347d828 (undecoded) ack+ondisk+write+known_if_redirected e5954) currently queued_for_pg
2018-02-06 13:50:12.133218 osd.53 osd.53 10.212.32.22:6802/50082 867 : cluster [WRN] 48 slow requests, 1 included below; oldest blocked for > 168.672302 secs
2018-02-06 13:50:12.133223 osd.53 osd.53 10.212.32.22:6802/50082 868 : cluster [WRN] slow request 34.801266 seconds old, received at 2018-02-06 13:49:37.331905: osd_op(client.48192.0:97243662 3.7163d828 3.7163d828 (undecoded) ack+ondisk+write+known_if_redirected e5954) currently queued_for_pg
2018-02-06 13:50:18.139428 mon.sg01-06 mon.0 10.212.32.18:6789/0 3639 : cluster [INF] Health check cleared: REQUEST_SLOW (was: 247 slow requests are blocked > 32 sec)
2018-02-06 13:50:49.882654 mon.sg01-06 mon.0 10.212.32.18:6789/0 3641 : cluster [WRN] overall HEALTH_WARN noscrub,nodeep-scrub flag(s) set

Testing with dd

# dd if=/dev/zero of=/var/lib/ceph/osd/ceph-53/dd_test bs=4M count=1000 oflag=direct
1000+0 records in
1000+0 records out
4194304000 bytes (4,2 GB, 3,9 GiB) copied, 29,3667 s, 143 MB/s
#

# dd if=/dev/zero of=/var/lib/ceph/osd/ceph-91/dd_test bs=4M count=1000 oflag=direct
1000+0 records in
1000+0 records out
4194304000 bytes (4,2 GB, 3,9 GiB) copied, 33,4732 s, 125 MB/s
#

Device Model:     TOSHIBA MG04ACA400N
Firmware Version: FJ2D
User Capacity:    4,000,787,030,016 bytes [4,00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
ATA Version is:   ATA8-ACS (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)


Please see some lines with number of files per directory in affected filestore.
I think directories with 300+ files look like just merged or I get it wrong ?


That's probably right, but now that there's a random factor, you may have had more than 320 files per subdirectory with previous defaults split and merge values.

But yeah, the blocked requests surely come from merging operations.
...
337 in /var/lib/ceph/osd/ceph-91/current/20.83s4_head/DIR_3/DIR_8/DIR_4/DIR_E
21 in /var/lib/ceph/osd/ceph-91/current/20.83s4_head/DIR_3/DIR_8/DIR_4/DIR_F/DIR_0
18 in /var/lib/ceph/osd/ceph-91/current/20.83s4_head/DIR_3/DIR_8/DIR_4/DIR_F/DIR_1
21 in /var/lib/ceph/osd/ceph-91/current/20.83s4_head/DIR_3/DIR_8/DIR_4/DIR_F/DIR_6
25 in /var/lib/ceph/osd/ceph-91/current/20.83s4_head/DIR_3/DIR_8/DIR_4/DIR_F/DIR_2
18 in /var/lib/ceph/osd/ceph-91/current/20.83s4_head/DIR_3/DIR_8/DIR_4/DIR_F/DIR_3
22 in /var/lib/ceph/osd/ceph-91/current/20.83s4_head/DIR_3/DIR_8/DIR_4/DIR_F/DIR_4
20 in /var/lib/ceph/osd/ceph-91/current/20.83s4_head/DIR_3/DIR_8/DIR_4/DIR_F/DIR_5
27 in /var/lib/ceph/osd/ceph-91/current/20.83s4_head/DIR_3/DIR_8/DIR_4/DIR_F/DIR_7
22 in /var/lib/ceph/osd/ceph-91/current/20.83s4_head/DIR_3/DIR_8/DIR_4/DIR_F/DIR_8
25 in /var/lib/ceph/osd/ceph-91/current/20.83s4_head/DIR_3/DIR_8/DIR_4/DIR_F/DIR_9
19 in /var/lib/ceph/osd/ceph-91/current/20.83s4_head/DIR_3/DIR_8/DIR_4/DIR_F/DIR_A
22 in /var/lib/ceph/osd/ceph-91/current/20.83s4_head/DIR_3/DIR_8/DIR_4/DIR_F/DIR_B
28 in /var/lib/ceph/osd/ceph-91/current/20.83s4_head/DIR_3/DIR_8/DIR_4/DIR_F/DIR_C
20 in /var/lib/ceph/osd/ceph-91/current/20.83s4_head/DIR_3/DIR_8/DIR_4/DIR_F/DIR_D
19 in /var/lib/ceph/osd/ceph-91/current/20.83s4_head/DIR_3/DIR_8/DIR_4/DIR_F/DIR_E
28 in /var/lib/ceph/osd/ceph-91/current/20.83s4_head/DIR_3/DIR_8/DIR_4/DIR_F/DIR_F
338 in /var/lib/ceph/osd/ceph-91/current/20.83s4_head/DIR_3/DIR_8/DIR_8/DIR_0
344 in /var/lib/ceph/osd/ceph-91/current/20.83s4_head/DIR_3/DIR_8/DIR_8/DIR_1
320 in /var/lib/ceph/osd/ceph-91/current/20.83s4_head/DIR_3/DIR_8/DIR_8/DIR_2
50 in /var/lib/ceph/osd/ceph-91/current/20.83s4_head/DIR_3/DIR_8/DIR_8/DIR_3
21 in /var/lib/ceph/osd/ceph-91/current/20.83s4_head/DIR_3/DIR_8/DIR_8/DIR_3/DIR_0
26 in /var/lib/ceph/osd/ceph-91/current/20.83s4_head/DIR_3/DIR_8/DIR_8/DIR_3/DIR_1
20 in /var/lib/ceph/osd/ceph-91/current/20.83s4_head/DIR_3/DIR_8/DIR_8/DIR_3/DIR_2
13 in /var/lib/ceph/osd/ceph-91/current/20.83s4_head/DIR_3/DIR_8/DIR_8/DIR_3/DIR_3
28 in /var/lib/ceph/osd/ceph-91/current/20.83s4_head/DIR_3/DIR_8/DIR_8/DIR_3/DIR_4
19 in /var/lib/ceph/osd/ceph-91/current/20.83s4_head/DIR_3/DIR_8/DIR_8/DIR_3/DIR_5
20 in /var/lib/ceph/osd/ceph-91/current/20.83s4_head/DIR_3/DIR_8/DIR_8/DIR_3/DIR_6
17 in /var/lib/ceph/osd/ceph-91/current/20.83s4_head/DIR_3/DIR_8/DIR_8/DIR_3/DIR_7
21 in /var/lib/ceph/osd/ceph-91/current/20.83s4_head/DIR_3/DIR_8/DIR_8/DIR_3/DIR_8
22 in /var/lib/ceph/osd/ceph-91/current/20.83s4_head/DIR_3/DIR_8/DIR_8/DIR_3/DIR_9
23 in /var/lib/ceph/osd/ceph-91/current/20.83s4_head/DIR_3/DIR_8/DIR_8/DIR_3/DIR_B
23 in /var/lib/ceph/osd/ceph-91/current/20.83s4_head/DIR_3/DIR_8/DIR_8/DIR_3/DIR_E
26 in /var/lib/ceph/osd/ceph-91/current/20.83s4_head/DIR_3/DIR_8/DIR_8/DIR_3/DIR_D
17 in /var/lib/ceph/osd/ceph-91/current/20.83s4_head/DIR_3/DIR_8/DIR_8/DIR_3/DIR_F
325 in /var/lib/ceph/osd/ceph-91/current/20.83s4_head/DIR_3/DIR_8/DIR_8/DIR_4
...

Does ceph-objectstore-tool merge/split filestore structure ? I can try it tomorrow.

It only does the splitting part. You'd might want to set filestore_merge_threshold to -40 to not merge, OR provoke merging off peak hours with something like this :

while true ; do rados bench -p default.rgw.buckets.data 10 write -b 4K -t 16 ; sleep 60 ; done

Every of the 1024 PGs should rapidly see an object write and delete and then be merged.

Regards,

Frédéric.


Many thanks
Jakub
​

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]

  Powered by Linux