Indeed it shows ssd in the OSD's metadata.
"bluestore_bdev_type": "ssd",
Then
I misunderstood the role of the device class in CRUSH, I expected the
OSD would actually set its settings according to the CRUSH device class."bluestore_bdev_type": "ssd",
Thanks,
HansOn Wed, Oct 18, 2017 at 11:56 AM, Wido den Hollander <wido@xxxxxxxx> wrote:
> Op 18 oktober 2017 om 11:41 schreef Hans van den Bogert <hansbogert@xxxxxxxxx>:
>
>
> Hi All,
>
> I've converted 2 nodes with 4 HDD/OSDs each from Filestore to Bluestore. I
> expected somewhat higher memory usage/RSS values, however I see, imo, a
> huge memory usage for all OSDs on both nodes.
>
> Small snippet from `top`
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
> COMMAND
> 4652 ceph 20 0 9840236 8.443g 21364 S 0.7 27.1 31:21.15
> /usr/bin/ceph-osd -f --cluster ceph --id 5 --setuser ceph --setgroup ceph
>
>
> The only deviation from a conventional install is that we use bcache for
> our HDDs. Bcache by default is recognized as an 'ssd' in CRUSH. I've
> manually set the class to 'hdd'.
>
> Small snippet from `ceph osd tree`
> -3 7.27399 host osd02
> 5 hdd 1.81850 osd.5 up 1.00000 1.00000
>
> So I would expect around 2GB of usage according to rules of thumb in the
> documentation and Sage's comments about the bluestore cache parameters for
> HDDs; yet we're now seeing a usage of more than 8GB after less than 1 day
> of runtime for this OSD. Is this a memory leak?
Although you've set the class to HDD it's the OSD which probably sees itself as an SSD backed OSD.
Test with:
$ ceph osd metadata 5
It will show:
"bluestore_bdev_rotational": "0",
"bluestore_bdev_type": "ssd",
The default for SSD OSDs is 3GB, see: http://docs.ceph.com/docs/master/rados/configuration/ bluestore-config-ref/
bluestore_cache_size_ssd is set to 3GB, so it will use at least 3GB.
I agree, 5GB above the 3GB is a lot of memory, but could you check the OSD metadata first?
> ______________________________
>
> Having read the other threads Sage recommends to also send the mempool dump
>
> {
> "bloom_filter": {
> "items": 0,
> "bytes": 0
> },
> "bluestore_alloc": {
> "items": 5732656,
> "bytes": 5732656
> },
> "bluestore_cache_data": {
> "items": 10659,
> "bytes": 481820672
> },
> "bluestore_cache_onode": {
> "items": 1106714,
> "bytes": 752565520
> },
> "bluestore_cache_other": {
> "items": 412675997,
> "bytes": 1388849420
> },
> "bluestore_fsck": {
> "items": 0,
> "bytes": 0
> },
> "bluestore_txc": {
> "items": 5,
> "bytes": 3600
> },
> "bluestore_writing_deferred": {
> "items": 21,
> "bytes": 225280
> },
> "bluestore_writing": {
> "items": 2,
> "bytes": 188146
> },
> "bluefs": {
> "items": 951,
> "bytes": 50432
> },
> "buffer_anon": {
> "items": 14440810,
> "bytes": 1804695070
> },
> "buffer_meta": {
> "items": 10754,
> "bytes": 946352
> },
> "osd": {
> "items": 155,
> "bytes": 1869920
> },
> "osd_mapbl": {
> "items": 16,
> "bytes": 288280
> },
> "osd_pglog": {
> "items": 284680,
> "bytes": 91233440
> },
> "osdmap": {
> "items": 14287,
> "bytes": 731680
> },
> "osdmap_mapping": {
> "items": 0,
> "bytes": 0
> },
> "pgmap": {
> "items": 0,
> "bytes": 0
> },
> "mds_co": {
> "items": 0,
> "bytes": 0
> },
> "unittest_1": {
> "items": 0,
> "bytes": 0
> },
> "unittest_2": {
> "items": 0,
> "bytes": 0
> },
> "total": {
> "items": 434277707,
> "bytes": 4529200468
> }
> }
>
> Regards,
>
> Hans
_________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph. com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
