Hi Igor, I had taken the OSDs out already, so bringing them up allowed a full rebalance to occur. I verified that they were not exhibiting ATA or SMART-reportable errors, wiped them and re-added. I will deep scrub. Thanks again! Jeremy On Mon, Dec 21, 2020 at 11:39 PM Igor Fedotov <ifedotov@xxxxxxx> wrote: > Hi Jeremy, > > good to know you managed to bring your OSDs up. > > Have you been able to reweight them to 0 and migrate data out of these > "broken" OSDs? > > If so I suggest to redeploy them - the corruption is still in the DB and > it might pop-up one day. > > If not please do that first - you might still hit into issues along the > process since DB is still corrupted. Disabling compaction just bypassed the > reads from broken files - but this might happen again during different > operations. > > > On 12/22/2020 7:17 AM, Jeremy Austin wrote: > > Igor, > > You're a bloomin' genius, as they say. > > Disabling auto compaction allowed OSDs 11 and 12 to spin up/out. The 7 > down PGs recovered; there were a few unfound items previously which I went > ahead and deleted, given that this is EC, revert not being an option. > > HEALTH OK :) > > I'm now intending to re-enable auto compaction. Should I also fsck the > rest of the OSDs, or is the typical scrub/deep scrub cycle sufficient? (No > PGs are behind on scrubbing, whereas they were during the degraded period.) > > Please do not re-enable the compaction before you're ready to kill > "broken" OSDs... > > There is no much sense in fsck-ing other OSDs - this is unlikely to detect > inconsistencies at PG level if any. > > Hence it's [deep]-scrub which you might want to apply manually once > reweight/rebalance is done. PGs located at "broken" OSDs are of the top > priority to scrub. > > > > Time will tell if I actually lost data, I guess. > > On Mon, Dec 21, 2020 at 8:37 AM Igor Fedotov <ifedotov@xxxxxxx> wrote: > >> Hi Jeremy, >> >> you might want to try RocksDB's disable_auto_compactions option for that. >> >> To adjust rocksdb's options one should edit/insert >> bluestore_rocksdb_options in ceph.conf. >> >> E.g. >> >> bluestore_rocksdb_options = >> "disable_auto_compactions=true,compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824" >> >> >> Please note bluestore specific defaults for rocksdb settings are >> re-provided to make sure they aren't reset to rocksdb's ones. >> >> Hope this helps. >> >> Thanks, >> >> Igor >> >> >> >> >> >> >> >> On 12/21/2020 2:56 AM, Jeremy Austin wrote: >> >> >> >> On Sun, Dec 20, 2020 at 2:25 PM Jeremy Austin <jhaustin@xxxxxxxxx> wrote: >> >>> Will attempt to disable compaction and report. >>> >> >> Not sure I'm doing this right. In [osd] section of ceph.conf, I added >> periodic_compaction_seconds=0 >> >> and attempted to start the OSDs in question. Same error as before. Am I >> setting compaction options correctly? >> >> >> -- >> Jeremy Austin >> jhaustin@xxxxxxxxx >> >> > > -- > Jeremy Austin > jhaustin@xxxxxxxxx > > -- Jeremy Austin jhaustin@xxxxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |