On Mon, Mar 25, 2024 at 7:37 PM Torkil Svensgaard <torkil@xxxxxxxx> wrote: > > > > On 24/03/2024 01:14, Torkil Svensgaard wrote: > > On 24-03-2024 00:31, Alexander E. Patrakov wrote: > >> Hi Torkil, > > > > Hi Alexander > > > >> Thanks for the update. Even though the improvement is small, it is > >> still an improvement, consistent with the osd_max_backfills value, and > >> it proves that there are still unsolved peering issues. > >> > >> I have looked at both the old and the new state of the PG, but could > >> not find anything else interesting. > >> > >> I also looked again at the state of PG 37.1. It is known what blocks > >> the backfill of this PG; please search for "blocked_by." However, this > >> is just one data point, which is insufficient for any conclusions. Try > >> looking at other PGs. Is there anything too common in the non-empty > >> "blocked_by" blocks? > > > > I'll take a look at that tomorrow, perhaps we can script something > > meaningful. > > Hi Alexander > > While working on a script querying all PGs and making a list of all OSDs > found in a blocked_by list, and how many times for each, I discovered > something odd about pool 38: > > " > [root@lazy blocked_by]# sh blocked_by.sh 38 |tee pool38 > OSDs blocking other OSDs: <snip> > All PGs in the pool are active+clean so why are there any blocked_by at > all? One example attached. I don't know. In any case, it doesn't match the "one OSD blocks them all" scenario that I was looking for. I think this is something bogus that can probably be cleared in your example by restarting osd.89 (i.e, the one being blocked). > > Mvh. > > Torkil > > >> I think we have to look for patterns in other ways, too. One tool that > >> produces good visualizations is TheJJ balancer. Although it is called > >> a "balancer," it can also visualize the ongoing backfills. > >> > >> The tool is available at > >> https://raw.githubusercontent.com/TheJJ/ceph-balancer/master/placementoptimizer.py > >> > >> Run it as follows: > >> > >> ./placementoptimizer.py showremapped --by-osd | tee remapped.txt > > > > Output attached. > > > > Thanks again. > > > > Mvh. > > > > Torkil > > > >> On Sun, Mar 24, 2024 at 5:50 AM Torkil Svensgaard <torkil@xxxxxxxx> > >> wrote: > >>> > >>> Hi Alex > >>> > >>> New query output attached after restarting both OSDs. OSD 237 is no > >>> longer mentioned but it unfortunately made no difference for the number > >>> of backfills which went 59->62->62. > >>> > >>> Mvh. > >>> > >>> Torkil > >>> > >>> On 23-03-2024 22:26, Alexander E. Patrakov wrote: > >>>> Hi Torkil, > >>>> > >>>> I have looked at the files that you attached. They were helpful: pool > >>>> 11 is problematic, it complains about degraded objects for no obvious > >>>> reason. I think that is the blocker. > >>>> > >>>> I also noted that you mentioned peering problems, and I suspect that > >>>> they are not completely resolved. As a somewhat-irrational move, to > >>>> confirm this theory, you can restart osd.237 (it is mentioned at the > >>>> end of query.11.fff.txt, although I don't understand why it is there) > >>>> and then osd.298 (it is the primary for that pg) and see if any > >>>> additional backfills are unblocked after that. Also, please re-query > >>>> that PG again after the OSD restart. > >>>> > >>>> On Sun, Mar 24, 2024 at 4:56 AM Torkil Svensgaard <torkil@xxxxxxxx> > >>>> wrote: > >>>>> > >>>>> > >>>>> > >>>>> On 23-03-2024 21:19, Alexander E. Patrakov wrote: > >>>>>> Hi Torkil, > >>>>> > >>>>> Hi Alexander > >>>>> > >>>>>> I have looked at the CRUSH rules, and the equivalent rules work on my > >>>>>> test cluster. So this cannot be the cause of the blockage. > >>>>> > >>>>> Thank you for taking the time =) > >>>>> > >>>>>> What happens if you increase the osd_max_backfills setting > >>>>>> temporarily? > >>>>> > >>>>> We already had the mclock override option in place and I re-enabled > >>>>> our > >>>>> babysitter script which sets osd_max_backfills pr OSD to 1-3 depending > >>>>> on how full they are. Active backfills went from 16 to 53 which is > >>>>> probably because default osd_max_backfills for mclock is 1. > >>>>> > >>>>> I think 53 is still a low number of active backfills given the large > >>>>> percentage misplaced. > >>>>> > >>>>>> It may be a good idea to investigate a few of the stalled PGs. Please > >>>>>> run commands similar to this one: > >>>>>> > >>>>>> ceph pg 37.0 query > query.37.0.txt > >>>>>> ceph pg 37.1 query > query.37.1.txt > >>>>>> ... > >>>>>> and the same for the other affected pools. > >>>>> > >>>>> A few samples attached. > >>>>> > >>>>>> Still, I must say that some of your rules are actually unsafe. > >>>>>> > >>>>>> The 4+2 rule as used by rbd_ec_data will not survive a > >>>>>> datacenter-offline incident. Namely, for each PG, it chooses OSDs > >>>>>> from > >>>>>> two hosts in each datacenter, so 6 OSDs total. When a datacenter is > >>>>>> offline, you will, therefore, have only 4 OSDs up, which is exactly > >>>>>> the number of data chunks. However, the pool requires min_size 5, so > >>>>>> all PGs will be inactive (to prevent data corruption) and will stay > >>>>>> inactive until the datacenter comes up again. However, please don't > >>>>>> set min_size to 4 - then, any additional incident (like a defective > >>>>>> disk) will lead to data loss, and the shards in the datacenter which > >>>>>> went offline would be useless because they do not correspond to the > >>>>>> updated shards written by the clients. > >>>>> > >>>>> Thanks for the explanation. This is an old pool predating the 3 DC > >>>>> setup > >>>>> and we'll migrate the data to a 4+5 pool when we can. > >>>>> > >>>>>> The 4+5 rule as used by cephfs.hdd.data has min_size equal to the > >>>>>> number of data chunks. See above why it is bad. Please set > >>>>>> min_size to > >>>>>> 5. > >>>>> > >>>>> Thanks, that was a leftover for getting the PGs to peer (stuck at > >>>>> creating+incomplete) when we created the pool. It's back to 5 now. > >>>>> > >>>>>> The rbd.ssd.data pool seems to be OK - and, by the way, its PGs are > >>>>>> 100% active+clean. > >>>>> > >>>>> There is very little data in this pool, that is probably the main > >>>>> reason. > >>>>> > >>>>>> Regarding the mon_max_pg_per_osd setting, you have a few OSDs that > >>>>>> have 300+ PGs, the observed maximum is 347. Please set it to 400. > >>>>> > >>>>> Copy that. Didn't seem to make a difference though, and we have > >>>>> osd_max_pg_per_osd_hard_ratio set to 5.000000. > >>>>> > >>>>> Mvh. > >>>>> > >>>>> Torkil > >>>>> > >>>>>> On Sun, Mar 24, 2024 at 3:16 AM Torkil Svensgaard > >>>>>> <torkil@xxxxxxxx> wrote: > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On 23-03-2024 19:05, Alexander E. Patrakov wrote: > >>>>>>>> Sorry for replying to myself, but "ceph osd pool ls detail" by > >>>>>>>> itself > >>>>>>>> is insufficient. For every erasure code profile mentioned in the > >>>>>>>> output, please also run something like this: > >>>>>>>> > >>>>>>>> ceph osd erasure-code-profile get prf-for-ec-data > >>>>>>>> > >>>>>>>> ...where "prf-for-ec-data" is the name that appears after the words > >>>>>>>> "erasure profile" in the "ceph osd pool ls detail" output. > >>>>>>> > >>>>>>> [root@lazy ~]# ceph osd pool ls detail | grep erasure > >>>>>>> pool 11 'rbd_ec_data' erasure profile DRCMR_k4m2 size 6 min_size 5 > >>>>>>> crush_rule 0 object_hash rjenkins pg_num 4096 pgp_num 4096 > >>>>>>> autoscale_mode off last_change 2257933 lfor 0/1291190/1755101 flags > >>>>>>> hashpspool,ec_overwrites,selfmanaged_snaps,bulk stripe_width 16384 > >>>>>>> fast_read 1 compression_algorithm snappy compression_mode aggressive > >>>>>>> application rbd > >>>>>>> pool 37 'cephfs.hdd.data' erasure profile > >>>>>>> DRCMR_k4m5_datacenter_hdd size > >>>>>>> 9 min_size 4 crush_rule 7 object_hash rjenkins pg_num 2048 > >>>>>>> pgp_num 2048 > >>>>>>> autoscale_mode off last_change 2257933 lfor 0/0/2139486 flags > >>>>>>> hashpspool,ec_overwrites,bulk stripe_width 16384 fast_read 1 > >>>>>>> compression_algorithm zstd compression_mode aggressive > >>>>>>> application cephfs > >>>>>>> pool 38 'rbd.ssd.data' erasure profile DRCMR_k4m5_datacenter_ssd > >>>>>>> size 9 > >>>>>>> min_size 5 crush_rule 8 object_hash rjenkins pg_num 32 pgp_num 32 > >>>>>>> autoscale_mode warn last_change 2198930 lfor 0/2198930/2198928 flags > >>>>>>> hashpspool,ec_overwrites,selfmanaged_snaps stripe_width 16384 > >>>>>>> compression_algorithm zstd compression_mode aggressive > >>>>>>> application rbd > >>>>>>> > >>>>>>> [root@lazy ~]# ceph osd erasure-code-profile get DRCMR_k4m2 > >>>>>>> crush-device-class=hdd > >>>>>>> crush-failure-domain=host > >>>>>>> crush-root=default > >>>>>>> jerasure-per-chunk-alignment=false > >>>>>>> k=4 > >>>>>>> m=2 > >>>>>>> plugin=jerasure > >>>>>>> technique=reed_sol_van > >>>>>>> w=8 > >>>>>>> [root@lazy ~]# ceph osd erasure-code-profile get > >>>>>>> DRCMR_k4m5_datacenter_hdd > >>>>>>> crush-device-class=hdd > >>>>>>> crush-failure-domain=datacenter > >>>>>>> crush-root=default > >>>>>>> jerasure-per-chunk-alignment=false > >>>>>>> k=4 > >>>>>>> m=5 > >>>>>>> plugin=jerasure > >>>>>>> technique=reed_sol_van > >>>>>>> w=8 > >>>>>>> [root@lazy ~]# ceph osd erasure-code-profile get > >>>>>>> DRCMR_k4m5_datacenter_ssd > >>>>>>> crush-device-class=ssd > >>>>>>> crush-failure-domain=datacenter > >>>>>>> crush-root=default > >>>>>>> jerasure-per-chunk-alignment=false > >>>>>>> k=4 > >>>>>>> m=5 > >>>>>>> plugin=jerasure > >>>>>>> technique=reed_sol_van > >>>>>>> w=8 > >>>>>>> > >>>>>>> But as I understand it those profiles are only used to create the > >>>>>>> initial crush rule for the pool, and we have manually edited > >>>>>>> those along > >>>>>>> the way. Here are the 3 rules in use for the 3 EC pools: > >>>>>>> > >>>>>>> rule rbd_ec_data { > >>>>>>> id 0 > >>>>>>> type erasure > >>>>>>> step set_chooseleaf_tries 5 > >>>>>>> step set_choose_tries 100 > >>>>>>> step take default class hdd > >>>>>>> step choose indep 0 type datacenter > >>>>>>> step chooseleaf indep 2 type host > >>>>>>> step emit > >>>>>>> } > >>>>>>> rule cephfs.hdd.data { > >>>>>>> id 7 > >>>>>>> type erasure > >>>>>>> step set_chooseleaf_tries 5 > >>>>>>> step set_choose_tries 100 > >>>>>>> step take default class hdd > >>>>>>> step choose indep 0 type datacenter > >>>>>>> step chooseleaf indep 3 type host > >>>>>>> step emit > >>>>>>> } > >>>>>>> rule rbd.ssd.data { > >>>>>>> id 8 > >>>>>>> type erasure > >>>>>>> step set_chooseleaf_tries 5 > >>>>>>> step set_choose_tries 100 > >>>>>>> step take default class ssd > >>>>>>> step choose indep 0 type datacenter > >>>>>>> step chooseleaf indep 3 type host > >>>>>>> step emit > >>>>>>> } > >>>>>>> > >>>>>>> Which should first pick all 3 datacenters in the choose step and > >>>>>>> then > >>>>>>> either 2 or 3 hosts in the chooseleaf step, matching EC 4+2 and 4+5 > >>>>>>> respectively. > >>>>>>> > >>>>>>> Mvh. > >>>>>>> > >>>>>>> Torkil > >>>>>>> > >>>>>>>> On Sun, Mar 24, 2024 at 1:56 AM Alexander E. Patrakov > >>>>>>>> <patrakov@xxxxxxxxx> wrote: > >>>>>>>>> > >>>>>>>>> Hi Torkil, > >>>>>>>>> > >>>>>>>>> I take my previous response back. > >>>>>>>>> > >>>>>>>>> You have an erasure-coded pool with nine shards but only three > >>>>>>>>> datacenters. This, in general, cannot work. You need either nine > >>>>>>>>> datacenters or a very custom CRUSH rule. The second option may > >>>>>>>>> not be > >>>>>>>>> available if the current EC setup is already incompatible, as > >>>>>>>>> there is > >>>>>>>>> no way to change the EC parameters. > >>>>>>>>> > >>>>>>>>> It would help if you provided the output of "ceph osd pool ls > >>>>>>>>> detail". > >>>>>>>>> > >>>>>>>>> On Sun, Mar 24, 2024 at 1:43 AM Alexander E. Patrakov > >>>>>>>>> <patrakov@xxxxxxxxx> wrote: > >>>>>>>>>> > >>>>>>>>>> Hi Torkil, > >>>>>>>>>> > >>>>>>>>>> Unfortunately, your files contain nothing obviously bad or > >>>>>>>>>> suspicious, > >>>>>>>>>> except for two things: more PGs than usual and bad balance. > >>>>>>>>>> > >>>>>>>>>> What's your "mon max pg per osd" setting? > >>>>>>>>>> > >>>>>>>>>> On Sun, Mar 24, 2024 at 1:08 AM Torkil Svensgaard > >>>>>>>>>> <torkil@xxxxxxxx> wrote: > >>>>>>>>>>> > >>>>>>>>>>> On 2024-03-23 17:54, Kai Stian Olstad wrote: > >>>>>>>>>>>> On Sat, Mar 23, 2024 at 12:09:29PM +0100, Torkil Svensgaard > >>>>>>>>>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>> The other output is too big for pastebin and I'm not > >>>>>>>>>>>>> familiar with > >>>>>>>>>>>>> paste services, any suggestion for a preferred way to share > >>>>>>>>>>>>> such > >>>>>>>>>>>>> output? > >>>>>>>>>>>> > >>>>>>>>>>>> You can attached files to the mail here on the list. > >>>>>>>>>>> > >>>>>>>>>>> Doh, for some reason I was sure attachments would be > >>>>>>>>>>> stripped. Thanks, > >>>>>>>>>>> attached. > >>>>>>>>>>> > >>>>>>>>>>> Mvh. > >>>>>>>>>>> > >>>>>>>>>>> Torkil > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> -- > >>>>>>>>>> Alexander E. Patrakov > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> -- > >>>>>>>>> Alexander E. Patrakov > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>>> -- > >>>>>>> Torkil Svensgaard > >>>>>>> Systems Administrator > >>>>>>> Danish Research Centre for Magnetic Resonance DRCMR, Section 714 > >>>>>>> Copenhagen University Hospital Amager and Hvidovre > >>>>>>> Kettegaard Allé 30, 2650 Hvidovre, Denmark > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> -- > >>>>> Torkil Svensgaard > >>>>> Systems Administrator > >>>>> Danish Research Centre for Magnetic Resonance DRCMR, Section 714 > >>>>> Copenhagen University Hospital Amager and Hvidovre > >>>>> Kettegaard Allé 30, 2650 Hvidovre, Denmark > >>>> > >>>> > >>>> > >>> > >>> -- > >>> Torkil Svensgaard > >>> Systems Administrator > >>> Danish Research Centre for Magnetic Resonance DRCMR, Section 714 > >>> Copenhagen University Hospital Amager and Hvidovre > >>> Kettegaard Allé 30, 2650 Hvidovre, Denmark > >> > >> > >> > > > > -- > Torkil Svensgaard > Sysadmin > MR-Forskningssektionen, afs. 714 > DRCMR, Danish Research Centre for Magnetic Resonance > Hvidovre Hospital > Kettegård Allé 30 > DK-2650 Hvidovre > Denmark > Tel: +45 386 22828 > E-mail: torkil@xxxxxxxx -- Alexander E. Patrakov _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |