Hi Igor We are curious why blob garbage collection is not backported to mimic or luminous? https://github.com/ceph/ceph/pull/28229 Thanks Jerry Jerry Pu <yician1000ceph@xxxxxxxxx> 於 2020年7月8日 週三 下午6:04寫道： > OK. Thanks for your reminder. We will think about how to make the > adjustment to our cluster. > > Best > Jerry Pu > > Igor Fedotov <ifedotov@xxxxxxx> 於 2020年7月8日 週三 下午5:40寫道： > >> Please note that simple min_alloc_size downsizing might negatively impact >> OSD performance. That's why this modification has been postponed till >> Pacific - we've made a bunch of additional changes to eliminate the drop. >> >> >> Regards, >> >> Igor >> On 7/8/2020 12:32 PM, Jerry Pu wrote: >> >> Thanks for your reply. It's helpful! We may consider to adjust >> min_alloc_size to a lower value or take other actions based on >> your analysis for space overhead with EC pools. Thanks. >> >> Best >> Jerry Pu >> >> Igor Fedotov <ifedotov@xxxxxxx> 於 2020年7月7日 週二 下午4:10寫道： >> >>> I think you're facing the issue covered by the following ticket: >>> >>> https://tracker.ceph.com/issues/44213 >>> >>> >>> Unfortunately the only known solution is migrating to 4K min alloc size >>> which to be available since Pacific. >>> >>> >>> Thanks, >>> >>> Igor >>> >>> On 7/7/2020 6:38 AM, Jerry Pu wrote: >>> > Hi: >>> > >>> > We have a cluster (v13.2.4), and we do some tests on a EC >>> k=2, m=1 >>> > pool "VMPool0". We deploy some VMs (Windows, CentOS7) on the pool and >>> then >>> > use IOMeter to write data to these VMs. After a period of time, we >>> observe >>> > a strange thing that pool actual usage is much larger than stored data >>> * >>> > 1.5 (stored_raw). >>> > >>> > [root@Sim-str-R6-4 ~]# ceph df >>> > GLOBAL: >>> > CLASS SIZE AVAIL USED RAW USED %RAW >>> USED >>> > hdd 3.5 TiB 1.8 TiB 1.7 TiB 1.7 TiB >>> 48.74 >>> > TOTAL 3.5 TiB 1.8 TiB 1.7 TiB 1.7 TiB >>> 48.74 >>> > POOLS: >>> > NAME ID USED %USED MAX AVAIL >>> OBJECTS >>> > cephfs_data 1 29 GiB 100.00 0 B >>> 2597 >>> > cephfs_md 2 831 MiB 100.00 0 B >>> 133 >>> > erasure_meta_hdd 3 22 MiB 100.00 0 B >>> 170 >>> > VMPool0 4 1.2 TiB 56.77 644 GiB >>> 116011 >>> > stresspool 5 2.6 MiB 100.00 0 B >>> 32 >>> > >>> > [root@Sim-str-R6-4 ~]# ceph df detail -f json-pretty >>> > -----snippet----- >>> > { >>> > "name": "VMPool0", >>> > "id": 4, >>> > "stats": { >>> > "kb_used": 1328888832, >>> > "bytes_used": 1360782163968, <---------------- >>> > "percent_used": 0.567110, >>> > "max_avail": 692481687552, >>> > "objects": 116011, >>> > "quota_objects": 0, >>> > "quota_bytes": 0, >>> > "dirty": 116011, >>> > "rd": 27449034, >>> > "rd_bytes": 126572760064, >>> > "wr": 20675381, >>> > "wr_bytes": 1006460652544, >>> > "comp_ratio": 1.000000, >>> > "stored": 497657610240, >>> > "stored_raw": 746486431744, <---------------- >>> > } >>> > }, >>> > >>> > The perf counters of all osds (all hdd) used by VMPool0 also >>> show >>> > that bluestore_allocated is much larger than bluestore_stored. >>> > >>> > [root@Sim-str-R6-4 ~]# for i in {0..3}; do echo $i; ceph daemon >>> osd.$i perf >>> > dump | grep bluestore | head -6; done >>> > 0 >>> > "bluestore": { >>> > "bluestore_allocated": 175032369152, <---------------- >>> > "bluestore_stored": 83557936482, <---------------- >>> > "bluestore_compressed": 958795770, >>> > "bluestore_compressed_allocated": 6431965184, >>> > "bluestore_compressed_original": 18576584704, >>> > 1 >>> > "bluestore": { >>> > "bluestore_allocated": 119943593984, <---------------- >>> > "bluestore_stored": 53325238866, <---------------- >>> > "bluestore_compressed": 670158436, >>> > "bluestore_compressed_allocated": 4751818752, >>> > "bluestore_compressed_original": 13752328192, >>> > 2 >>> > "bluestore": { >>> > "bluestore_allocated": 155444707328, <---------------- >>> > "bluestore_stored": 69067116553, <---------------- >>> > "bluestore_compressed": 565170876, >>> > "bluestore_compressed_allocated": 4614324224, >>> > "bluestore_compressed_original": 13469696000, >>> > 3 >>> > "bluestore": { >>> > "bluestore_allocated": 128179240960, <---------------- >>> > "bluestore_stored": 60884752114, <---------------- >>> > "bluestore_compressed": 1653455847, >>> > "bluestore_compressed_allocated": 9741795328, >>> > "bluestore_compressed_original": 27878768640, >>> > >>> > [root@Sim-str-R6-5 osd]# for i in {4..7}; do echo $i; sh -c "ceph >>> daemon >>> > osd.$i perf dump | grep bluestore | head -6"; done >>> > 4 >>> > "bluestore": { >>> > "bluestore_allocated": 165950652416, <---------------- >>> > "bluestore_stored": 80255191687, <---------------- >>> > "bluestore_compressed": 1526871060, >>> > "bluestore_compressed_allocated": 8900378624, >>> > "bluestore_compressed_original": 25324142592, >>> > 5 >>> > admin_socket: exception getting command descriptions: [Errno 111] >>> > Connection refused >>> > 6 >>> > "bluestore": { >>> > "bluestore_allocated": 166022152192, <---------------- >>> > "bluestore_stored": 84645390708, <---------------- >>> > "bluestore_compressed": 1169055606, >>> > "bluestore_compressed_allocated": 8647278592, >>> > "bluestore_compressed_original": 25135091712, >>> > 7 >>> > "bluestore": { >>> > "bluestore_allocated": 204633604096, <---------------- >>> > "bluestore_stored": 100116382041, <---------------- >>> > "bluestore_compressed": 1081260422, >>> > "bluestore_compressed_allocated": 6510018560, >>> > "bluestore_compressed_original": 18654052352, >>> > >>> > [root@Sim-str-R6-6 osd]# for i in {8..12}; do echo $i; ceph daemon >>> osd.$i >>> > perf dump | grep bluestore | head -6; done >>> > 8 >>> > "bluestore": { >>> > "bluestore_allocated": 106330193920, <---------------- >>> > "bluestore_stored": 45282848089, <---------------- >>> > "bluestore_compressed": 1136610231, >>> > "bluestore_compressed_allocated": 7248609280, >>> > "bluestore_compressed_original": 20882960384, >>> > 9 >>> > "bluestore": { >>> > "bluestore_allocated": 120657412096, <---------------- >>> > "bluestore_stored": 52550745942, <---------------- >>> > "bluestore_compressed": 1321632665, >>> > "bluestore_compressed_allocated": 7401504768, >>> > "bluestore_compressed_original": 21073027072, >>> > 10 >>> > "bluestore": { >>> > "bluestore_allocated": 155985772544, <---------------- >>> > "bluestore_stored": 73236910054, <---------------- >>> > "bluestore_compressed": 98351920, >>> > "bluestore_compressed_allocated": 772210688, >>> > "bluestore_compressed_original": 2242043904, >>> > 11 >>> > "bluestore": { >>> > "bluestore_allocated": 106040524800, <---------------- >>> > "bluestore_stored": 45353612134, <---------------- >>> > "bluestore_compressed": 874216443, >>> > "bluestore_compressed_allocated": 4962844672, >>> > "bluestore_compressed_original": 14160310272, >>> > 12 >>> > "bluestore": { >>> > "bluestore_allocated": 118751363072, <---------------- >>> > "bluestore_stored": 52194408691, <---------------- >>> > "bluestore_compressed": 782919969, >>> > "bluestore_compressed_allocated": 5546311680, >>> > "bluestore_compressed_original": 16043233280, >>> > >>> > The config min_alloc_size_hdd of all osds are 64K and rados >>> objects >>> > of VMPool0 are all 4M rbd_data.x.xxxxxxx.xxxxxxx objects. It's kind of >>> > strange that allocation space is much larger than stored data. Can >>> anyone >>> > explains this? >>> > _______________________________________________ >>> > ceph-users mailing list -- ceph-users@xxxxxxx >>> > To unsubscribe send an email to ceph-users-leave@xxxxxxx >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@xxxxxxx >>> To unsubscribe send an email to ceph-users-leave@xxxxxxx >>> >> _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |