> Op 18 oktober 2017 om 11:41 schreef Hans van den Bogert <hansbogert@xxxxxxxxx>: > > > Hi All, > > I've converted 2 nodes with 4 HDD/OSDs each from Filestore to Bluestore. I > expected somewhat higher memory usage/RSS values, however I see, imo, a > huge memory usage for all OSDs on both nodes. > > Small snippet from `top` > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ > COMMAND > 4652 ceph 20 0 9840236 8.443g 21364 S 0.7 27.1 31:21.15 > /usr/bin/ceph-osd -f --cluster ceph --id 5 --setuser ceph --setgroup ceph > > > The only deviation from a conventional install is that we use bcache for > our HDDs. Bcache by default is recognized as an 'ssd' in CRUSH. I've > manually set the class to 'hdd'. > > Small snippet from `ceph osd tree` > -3 7.27399 host osd02 > 5 hdd 1.81850 osd.5 up 1.00000 1.00000 > > So I would expect around 2GB of usage according to rules of thumb in the > documentation and Sage's comments about the bluestore cache parameters for > HDDs; yet we're now seeing a usage of more than 8GB after less than 1 day > of runtime for this OSD. Is this a memory leak? Although you've set the class to HDD it's the OSD which probably sees itself as an SSD backed OSD. Test with: $ ceph osd metadata 5 It will show: "bluestore_bdev_rotational": "0", "bluestore_bdev_type": "ssd", The default for SSD OSDs is 3GB, see: http://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/ bluestore_cache_size_ssd is set to 3GB, so it will use at least 3GB. I agree, 5GB above the 3GB is a lot of memory, but could you check the OSD metadata first? > > Having read the other threads Sage recommends to also send the mempool dump > > { > "bloom_filter": { > "items": 0, > "bytes": 0 > }, > "bluestore_alloc": { > "items": 5732656, > "bytes": 5732656 > }, > "bluestore_cache_data": { > "items": 10659, > "bytes": 481820672 > }, > "bluestore_cache_onode": { > "items": 1106714, > "bytes": 752565520 > }, > "bluestore_cache_other": { > "items": 412675997, > "bytes": 1388849420 > }, > "bluestore_fsck": { > "items": 0, > "bytes": 0 > }, > "bluestore_txc": { > "items": 5, > "bytes": 3600 > }, > "bluestore_writing_deferred": { > "items": 21, > "bytes": 225280 > }, > "bluestore_writing": { > "items": 2, > "bytes": 188146 > }, > "bluefs": { > "items": 951, > "bytes": 50432 > }, > "buffer_anon": { > "items": 14440810, > "bytes": 1804695070 > }, > "buffer_meta": { > "items": 10754, > "bytes": 946352 > }, > "osd": { > "items": 155, > "bytes": 1869920 > }, > "osd_mapbl": { > "items": 16, > "bytes": 288280 > }, > "osd_pglog": { > "items": 284680, > "bytes": 91233440 > }, > "osdmap": { > "items": 14287, > "bytes": 731680 > }, > "osdmap_mapping": { > "items": 0, > "bytes": 0 > }, > "pgmap": { > "items": 0, > "bytes": 0 > }, > "mds_co": { > "items": 0, > "bytes": 0 > }, > "unittest_1": { > "items": 0, > "bytes": 0 > }, > "unittest_2": { > "items": 0, > "bytes": 0 > }, > "total": { > "items": 434277707, > "bytes": 4529200468 > } > } > > Regards, > > Hans > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |