Re: blustore osd nearfull but no pgs on it

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



The large amount of osdmaps is what i was suspecting. "ceph tell osd.158
status" (or any osd other than 158) would show us how many osdmaps the osds
are currently holding on to.

Respectfully,

*Wes Dillingham*
wes@xxxxxxxxxxxxxxxxx
LinkedIn <http://www.linkedin.com/in/wesleydillingham>


On Mon, Nov 20, 2023 at 6:15 AM Debian <debian@xxxxxxxxxx> wrote:

> Hi,
>
> yes all of my small osds are affected
>
> i found the issue, my cluster is healthy and my rebalance finished - i
> have only to wait that my old osdmaps get cleaned up.
>
> like in the thread "Disks are filling up even if there is not a single
> placement group on them"
>
> thx!
>
> On 20.11.23 11:36, Eugen Block wrote:
> > You provide only a few details at a time, it would help to get a full
> > picture if you provided the output Wesley asked for (ceph df detail,
> > ceph tell osd.158 status, ceph osd df tree). Is osd.149 now the
> > problematic one or did you just add output from a different osd?
> > It's not really clear what you're doing without the necessary context.
> > You can just add the 'ceph daemon osd.{OSD} perf dump' output here or
> > in some pastebin.
> >
> > Zitat von Debian <debian@xxxxxxxxxx>:
> >
> >> Hi,
> >>
> >> the block.db size ist default and not custom configured:
> >>
> >> current:
> >>
> >> bluefs.db_used_bytes: 9602859008
> >> bluefs.db_used_bytes: 469434368
> >>
> >> ceph daemon osd.149 config show
> >>
> >>     "bluestore_bitmapallocator_span_size": "1024",
> >>     "bluestore_block_db_size": "0",
> >>     "bluestore_block_size": "107374182400",
> >>     "bluestore_block_wal_size": "100663296",
> >>     "bluestore_cache_size": "0",
> >>     "bluestore_cache_size_hdd": "1073741824",
> >>     "bluestore_cache_size_ssd": "3221225472",
> >>     "bluestore_compression_max_blob_size": "0",
> >>     "bluestore_compression_max_blob_size_hdd": "524288",
> >>     "bluestore_compression_max_blob_size_ssd": "65536",
> >>     "bluestore_compression_min_blob_size": "0",
> >>     "bluestore_compression_min_blob_size_hdd": "131072",
> >>     "bluestore_compression_min_blob_size_ssd": "8192",
> >>     "bluestore_extent_map_inline_shard_prealloc_size": "256",
> >>     "bluestore_extent_map_shard_max_size": "1200",
> >>     "bluestore_extent_map_shard_min_size": "150",
> >>     "bluestore_extent_map_shard_target_size": "500",
> >>     "bluestore_extent_map_shard_target_size_slop": "0.200000",
> >>     "bluestore_max_alloc_size": "0",
> >>     "bluestore_max_blob_size": "0",
> >>     "bluestore_max_blob_size_hdd": "524288",
> >>     "bluestore_max_blob_size_ssd": "65536",
> >>     "bluestore_min_alloc_size": "0",
> >>     "bluestore_min_alloc_size_hdd": "65536",
> >>     "bluestore_min_alloc_size_ssd": "4096",
> >>     "bluestore_prefer_deferred_size": "0",
> >>     "bluestore_prefer_deferred_size_hdd": "32768",
> >>     "bluestore_prefer_deferred_size_ssd": "0",
> >>     "bluestore_rocksdb_options":
> >>
> "compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2",
> >>
> >>     "bluefs_alloc_size": "1048576",
> >>     "bluefs_allocator": "hybrid",
> >>     "bluefs_buffered_io": "false",
> >>     "bluefs_check_for_zeros": "false",
> >>     "bluefs_compact_log_sync": "false",
> >>     "bluefs_log_compact_min_ratio": "5.000000",
> >>     "bluefs_log_compact_min_size": "16777216",
> >>     "bluefs_max_log_runway": "4194304",
> >>     "bluefs_max_prefetch": "1048576",
> >>     "bluefs_min_flush_size": "524288",
> >>     "bluefs_min_log_runway": "1048576",
> >>     "bluefs_preextend_wal_files": "false",
> >>     "bluefs_replay_recovery": "false",
> >>     "bluefs_replay_recovery_disable_compact": "false",
> >>     "bluefs_shared_alloc_size": "65536",
> >>     "bluefs_sync_write": "false",
> >>
> >> which the osd performance counter i cannot determine who is using the
> >> memory,...
> >>
> >> thx & best regards
> >>
> >>
> >> On 18.11.23 09:05, Eugen Block wrote:
> >>> Do you have a large block.db size defined in the ceph.conf (or
> >>> config store)?
> >>>
> >>> Zitat von Debian <debian@xxxxxxxxxx>:
> >>>
> >>>> thx for your reply, it shows nothing,... there are no pgs on the
> >>>> osd,...
> >>>>
> >>>> best regards
> >>>>
> >>>> On 17.11.23 23:09, Eugen Block wrote:
> >>>>> After you create the OSD, run ‚ceph pg ls-by-osd {OSD}‘, it should
> >>>>> show you which PGs are created there and then you’ll know which
> >>>>> pool they belong to, then check again the crush rule for that
> >>>>> pool. You can paste the outputs here.
> >>>>>
> >>>>> Zitat von Debian <debian@xxxxxxxxxx>:
> >>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>> after a massive rebalance(tunables) my small SSD-OSDs are getting
> >>>>>> full, i changed my crush rules so there are actual no pgs/pools
> >>>>>> on it, but the disks stay full:
> >>>>>>
> >>>>>> ceph version 14.2.21 (5ef401921d7a88aea18ec7558f7f9374ebd8f5a6)
> >>>>>> nautilus (stable)
> >>>>>>
> >>>>>> ID     CLASS WEIGHT     REWEIGHT SIZE    RAW USE DATA OMAP
> >>>>>> META     AVAIL    %USE  VAR  PGS STATUS TYPE NAME
> >>>>>> 158   ssd    0.21999  1.00000 224 GiB 194 GiB 193 GiB 22 MiB 1002
> >>>>>> MiB   30 GiB 86.68 1.49   0     up osd.158
> >>>>>>
> >>>>>> inferring bluefs devices from bluestore path
> >>>>>> 1 : device size 0x37e4400000 : own 0x[1ad3f00000~23c600000] =
> >>>>>> 0x23c600000 : using 0x39630000(918 MiB) : bluestore has
> >>>>>> 0x46e2d0000(18 GiB) available
> >>>>>>
> >>>>>> when i recreate the osd the osd gets full again
> >>>>>>
> >>>>>> any suggestion?
> >>>>>>
> >>>>>> thx & best regards
> >>>>>> _______________________________________________
> >>>>>> ceph-users mailing list -- ceph-users@xxxxxxx
> >>>>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
> >>>>>
> >>>>>
> >>>>> _______________________________________________
> >>>>> ceph-users mailing list -- ceph-users@xxxxxxx
> >>>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
> >>>> _______________________________________________
> >>>> ceph-users mailing list -- ceph-users@xxxxxxx
> >>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
> >>>
> >>>
> >>> _______________________________________________
> >>> ceph-users mailing list -- ceph-users@xxxxxxx
> >>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
> >> _______________________________________________
> >> ceph-users mailing list -- ceph-users@xxxxxxx
> >> To unsubscribe send an email to ceph-users-leave@xxxxxxx
> >
> >
> > _______________________________________________
> > ceph-users mailing list -- ceph-users@xxxxxxx
> > To unsubscribe send an email to ceph-users-leave@xxxxxxx
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux