On 5/31/23 16:15, Igor Fedotov wrote:
On 31/05/2023 15:26, Stefan Kooman wrote:
On 5/29/23 15:52, Igor Fedotov wrote:
Hi Stefan,
given that allocation probes include every allocation (including
short 4K ones) your stats look pretty high indeed.
Although you omitted historic probes so it's hard to tell if there is
negative trend in it..
I did not omit them. We (currently) don't store logs for longer than 7
days. I will increase the interval in which the probes get created
(every hour).
Allocation probe contains historic data on its own, e.g.
allocation stats probe 33: cnt: 8148921 frags: 10958186 size: 1704348508>
probe -1: 35168547, 46401246, 1199516209152
probe -3: 27275094, 35681802, 200121712640
probe -5: 34847167, 52539758, 271272230912
probe -9: 44291522, 60025613, 523997483008
probe -17: 10646313, 10646313, 155178434560
in the snippet above probes -1 through -17 are historic data 1 through
17 days (or more correctly probe attempts) back.
Ah, I just blindly applied your suggested grep keywords, but then you
don't get the historic probes, here we go:
ceph-osd.55.log.1.gz:2023-05-30T18:42:02.340+0200 7ffb856f8700 0
bluestore(/var/lib/ceph/osd/ceph-55) allocation stats probe 15: cnt:
19940282 frags: 37474984 size: 331549941760
ceph-osd.55.log.1.gz:2023-05-30T18:42:02.340+0200 7ffb856f8700 0
bluestore(/var/lib/ceph/osd/ceph-55) probe -1: 19269005, 35807252,
265695309824
ceph-osd.55.log.1.gz:2023-05-30T18:42:02.340+0200 7ffb856f8700 0
bluestore(/var/lib/ceph/osd/ceph-55) probe -3: 18281841, 33452239,
321271795712
ceph-osd.55.log.1.gz:2023-05-30T18:42:02.340+0200 7ffb856f8700 0
bluestore(/var/lib/ceph/osd/ceph-55) probe -7: 21188454, 37298950,
278389411840
ceph-osd.55.log.1.gz:2023-05-30T18:42:02.340+0200 7ffb856f8700 0
bluestore(/var/lib/ceph/osd/ceph-55) probe -15: 20079763, 34770424,
306357551104
ceph-osd.55.log.1.gz:2023-05-30T18:42:02.340+0200 7ffb856f8700 0
bluestore(/var/lib/ceph/osd/ceph-55) probe -31: 0, 0, 0
ceph-osd.66.log.1.gz:2023-05-30T18:42:01.496+0200 7fce80741700 0
bluestore(/var/lib/ceph/osd/ceph-66) allocation stats probe 15: cnt:
18130163 frags: 32547753 size: 286495289344
ceph-osd.66.log.1.gz:2023-05-30T18:42:01.496+0200 7fce80741700 0
bluestore(/var/lib/ceph/osd/ceph-66) probe -1: 16671297, 28781657,
219492544512
ceph-osd.66.log.1.gz:2023-05-30T18:42:01.496+0200 7fce80741700 0
bluestore(/var/lib/ceph/osd/ceph-66) probe -3: 16781500, 31159184,
306530078720
ceph-osd.66.log.1.gz:2023-05-30T18:42:01.496+0200 7fce80741700 0
bluestore(/var/lib/ceph/osd/ceph-66) probe -7: 18475332, 33264944,
266053271552
ceph-osd.66.log.1.gz:2023-05-30T18:42:01.496+0200 7fce80741700 0
bluestore(/var/lib/ceph/osd/ceph-66) probe -15: 18799693, 32644833,
270106509312
ceph-osd.66.log.1.gz:2023-05-30T18:42:01.496+0200 7fce80741700 0
bluestore(/var/lib/ceph/osd/ceph-66) probe -31: 0, 0, 0
ceph-osd.67.log.1.gz:2023-05-30T18:42:03.784+0200 7fd3e6ee9700 0
bluestore(/var/lib/ceph/osd/ceph-67) allocation stats probe 15: cnt:
19073940 frags: 39349763 size: 350442409984
ceph-osd.67.log.1.gz:2023-05-30T18:42:03.784+0200 7fd3e6ee9700 0
bluestore(/var/lib/ceph/osd/ceph-67) probe -1: 19346357, 40814157,
291156762624
ceph-osd.67.log.1.gz:2023-05-30T18:42:03.784+0200 7fd3e6ee9700 0
bluestore(/var/lib/ceph/osd/ceph-67) probe -3: 18880370, 33120073,
329544183808
ceph-osd.67.log.1.gz:2023-05-30T18:42:03.784+0200 7fd3e6ee9700 0
bluestore(/var/lib/ceph/osd/ceph-67) probe -7: 18550989, 35236438,
273069948928
ceph-osd.67.log.1.gz:2023-05-30T18:42:03.784+0200 7fd3e6ee9700 0
bluestore(/var/lib/ceph/osd/ceph-67) probe -15: 18664247, 36208229,
327013150720
ceph-osd.67.log.1.gz:2023-05-30T18:42:03.784+0200 7fd3e6ee9700 0
bluestore(/var/lib/ceph/osd/ceph-67) probe -31: 0, 0, 0
ceph-osd.68.log.1.gz:2023-05-30T18:42:03.704+0200 7f6bb6c05700 0
bluestore(/var/lib/ceph/osd/ceph-68) allocation stats probe 15: cnt:
24318407 frags: 42340467 size: 324867026944
ceph-osd.68.log.1.gz:2023-05-30T18:42:03.704+0200 7f6bb6c05700 0
bluestore(/var/lib/ceph/osd/ceph-68) probe -1: 24202548, 41934313,
263141662720
ceph-osd.68.log.1.gz:2023-05-30T18:42:03.704+0200 7f6bb6c05700 0
bluestore(/var/lib/ceph/osd/ceph-68) probe -3: 24258584, 43985640,
348764803072
ceph-osd.68.log.1.gz:2023-05-30T18:42:03.704+0200 7f6bb6c05700 0
bluestore(/var/lib/ceph/osd/ceph-68) probe -7: 25312482, 43035393,
287792226304
ceph-osd.68.log.1.gz:2023-05-30T18:42:03.704+0200 7f6bb6c05700 0
bluestore(/var/lib/ceph/osd/ceph-68) probe -15: 25280732, 40994669,
298337083392
ceph-osd.68.log.1.gz:2023-05-30T18:42:03.704+0200 7f6bb6c05700 0
bluestore(/var/lib/ceph/osd/ceph-68) probe -31: 0, 0, 0
ceph-osd.75.log.1.gz:2023-05-30T18:42:03.664+0200 7ff67f434700 0
bluestore(/var/lib/ceph/osd/ceph-75) allocation stats probe 15: cnt:
21755123 frags: 42279513 size: 364872880128
ceph-osd.75.log.1.gz:2023-05-30T18:42:03.664+0200 7ff67f434700 0
bluestore(/var/lib/ceph/osd/ceph-75) probe -1: 20263259, 38412587,
291021299712
ceph-osd.75.log.1.gz:2023-05-30T18:42:03.664+0200 7ff67f434700 0
bluestore(/var/lib/ceph/osd/ceph-75) probe -3: 19733485, 38065426,
353624813568
ceph-osd.75.log.1.gz:2023-05-30T18:42:03.664+0200 7ff67f434700 0
bluestore(/var/lib/ceph/osd/ceph-75) probe -7: 23195936, 45095051,
326473826304
ceph-osd.75.log.1.gz:2023-05-30T18:42:03.664+0200 7ff67f434700 0
bluestore(/var/lib/ceph/osd/ceph-75) probe -15: 24094232, 45266621,
367246442496
ceph-osd.75.log.1.gz:2023-05-30T18:42:03.664+0200 7ff67f434700 0
bluestore(/var/lib/ceph/osd/ceph-75) probe -31: 0, 0, 0
ceph-osd.76.log.1.gz:2023-05-30T18:42:04.608+0200 7f7a57c77700 0
bluestore(/var/lib/ceph/osd/ceph-76) allocation stats probe 15: cnt:
21035351 frags: 34478758 size: 296729862144
ceph-osd.76.log.1.gz:2023-05-30T18:42:04.608+0200 7f7a57c77700 0
bluestore(/var/lib/ceph/osd/ceph-76) probe -1: 20215704, 32981561,
229284646912
ceph-osd.76.log.1.gz:2023-05-30T18:42:04.608+0200 7f7a57c77700 0
bluestore(/var/lib/ceph/osd/ceph-76) probe -3: 19533561, 31158296,
311621177344
ceph-osd.76.log.1.gz:2023-05-30T18:42:04.608+0200 7f7a57c77700 0
bluestore(/var/lib/ceph/osd/ceph-76) probe -7: 23539469, 39573256,
292528910336
ceph-osd.76.log.1.gz:2023-05-30T18:42:04.608+0200 7f7a57c77700 0
bluestore(/var/lib/ceph/osd/ceph-76) probe -15: 21308764, 34003659,
272235884544
ceph-osd.76.log.1.gz:2023-05-30T18:42:04.608+0200 7f7a57c77700 0
bluestore(/var/lib/ceph/osd/ceph-76) probe -31: 0, 0, 0
ceph-osd.77.log.1.gz:2023-05-30T18:42:05.092+0200 7f04997f6700 0
bluestore(/var/lib/ceph/osd/ceph-77) allocation stats probe 15: cnt:
21619552 frags: 38792793 size: 319570493440
ceph-osd.77.log.1.gz:2023-05-30T18:42:05.092+0200 7f04997f6700 0
bluestore(/var/lib/ceph/osd/ceph-77) probe -1: 19698381, 33288562,
245733462016
ceph-osd.77.log.1.gz:2023-05-30T18:42:05.092+0200 7f04997f6700 0
bluestore(/var/lib/ceph/osd/ceph-77) probe -3: 19345976, 35673735,
327406686208
ceph-osd.77.log.1.gz:2023-05-30T18:42:05.092+0200 7f04997f6700 0
bluestore(/var/lib/ceph/osd/ceph-77) probe -7: 21371937, 43158447,
312754188288
ceph-osd.77.log.1.gz:2023-05-30T18:42:05.092+0200 7f04997f6700 0
bluestore(/var/lib/ceph/osd/ceph-77) probe -15: 23144793, 41159338,
309643296768
ceph-osd.77.log.1.gz:2023-05-30T18:42:05.092+0200 7f04997f6700 0
bluestore(/var/lib/ceph/osd/ceph-77) probe -31: 0, 0, 0
ceph-osd.84.log.1.gz:2023-05-30T18:42:05.480+0200 7fba48407700 0
bluestore(/var/lib/ceph/osd/ceph-84) allocation stats probe 15: cnt:
17348416 frags: 32011874 size: 257342664704
ceph-osd.84.log.1.gz:2023-05-30T18:42:05.480+0200 7fba48407700 0
bluestore(/var/lib/ceph/osd/ceph-84) probe -1: 16688730, 29763231,
204166496256
ceph-osd.84.log.1.gz:2023-05-30T18:42:05.480+0200 7fba48407700 0
bluestore(/var/lib/ceph/osd/ceph-84) probe -3: 15778722, 28658535,
269868609536
ceph-osd.84.log.1.gz:2023-05-30T18:42:05.480+0200 7fba48407700 0
bluestore(/var/lib/ceph/osd/ceph-84) probe -7: 17700395, 31528725,
239652761600
ceph-osd.84.log.1.gz:2023-05-30T18:42:05.480+0200 7fba48407700 0
bluestore(/var/lib/ceph/osd/ceph-84) probe -15: 17331958, 29820789,
239322513408
ceph-osd.84.log.1.gz:2023-05-30T18:42:05.480+0200 7fba48407700 0
bluestore(/var/lib/ceph/osd/ceph-84) probe -31: 0, 0, 0
ceph-osd.85.log.1.gz:2023-05-30T18:42:05.192+0200 7f548ff43700 0
bluestore(/var/lib/ceph/osd/ceph-85) allocation stats probe 15: cnt:
21817405 frags: 35609259 size: 304181972992
ceph-osd.85.log.1.gz:2023-05-30T18:42:05.192+0200 7f548ff43700 0
bluestore(/var/lib/ceph/osd/ceph-85) probe -1: 21402843, 35001032,
247483416576
ceph-osd.85.log.1.gz:2023-05-30T18:42:05.192+0200 7f548ff43700 0
bluestore(/var/lib/ceph/osd/ceph-85) probe -3: 20555960, 33268576,
297290342400
ceph-osd.85.log.1.gz:2023-05-30T18:42:05.192+0200 7f548ff43700 0
bluestore(/var/lib/ceph/osd/ceph-85) probe -7: 22132156, 38534232,
281272401920
ceph-osd.85.log.1.gz:2023-05-30T18:42:05.192+0200 7f548ff43700 0
bluestore(/var/lib/ceph/osd/ceph-85) probe -15: 22188633, 37241809,
290561724416
ceph-osd.85.log.1.gz:2023-05-30T18:42:05.192+0200 7f548ff43700 0
bluestore(/var/lib/ceph/osd/ceph-85) probe -31: 0, 0, 0
ceph-osd.86.log.1.gz:2023-05-30T18:42:06.220+0200 7f294451a700 0
bluestore(/var/lib/ceph/osd/ceph-86) allocation stats probe 15: cnt:
20080223 frags: 33329658 size: 278508113920
ceph-osd.86.log.1.gz:2023-05-30T18:42:06.220+0200 7f294451a700 0
bluestore(/var/lib/ceph/osd/ceph-86) probe -1: 20682155, 32852611,
219409920000
ceph-osd.86.log.1.gz:2023-05-30T18:42:06.220+0200 7f294451a700 0
bluestore(/var/lib/ceph/osd/ceph-86) probe -3: 19216200, 34658404,
296857645056
ceph-osd.86.log.1.gz:2023-05-30T18:42:06.220+0200 7f294451a700 0
bluestore(/var/lib/ceph/osd/ceph-86) probe -7: 21393711, 35682241,
257189515264
ceph-osd.86.log.1.gz:2023-05-30T18:42:06.220+0200 7f294451a700 0
bluestore(/var/lib/ceph/osd/ceph-86) probe -15: 20233752, 31906022,
256953790464
ceph-osd.86.log.1.gz:2023-05-30T18:42:06.220+0200 7f294451a700 0
bluestore(/var/lib/ceph/osd/ceph-86) probe -31: 0, 0, 0
Thanks,
Gr. Stefan
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
