Thanks for your reply. It's helpful! We may consider to adjust
min_alloc_size to a lower value or take other actions based on
your analysis for space overhead with EC pools. Thanks.
Best
Jerry Pu
Igor Fedotov <ifedotov@xxxxxxx <mailto:ifedotov@xxxxxxx>> 於 2020年7月7日
週二 下午4:10寫道：
I think you're facing the issue covered by the following ticket:
https://tracker.ceph.com/issues/44213
Unfortunately the only known solution is migrating to 4K min alloc
size
which to be available since Pacific.
Thanks,
Igor
On 7/7/2020 6:38 AM, Jerry Pu wrote:
> Hi:
>
> We have a cluster (v13.2.4), and we do some tests on a
EC k=2, m=1
> pool "VMPool0". We deploy some VMs (Windows, CentOS7) on the
pool and then
> use IOMeter to write data to these VMs. After a period of time,
we observe
> a strange thing that pool actual usage is much larger than
stored data *
> 1.5 (stored_raw).
>
> [root@Sim-str-R6-4 ~]# ceph df
> GLOBAL:
> CLASS SIZE AVAIL USED RAW USED
%RAW USED
> hdd 3.5 TiB 1.8 TiB 1.7 TiB 1.7 TiB
48.74
> TOTAL 3.5 TiB 1.8 TiB 1.7 TiB 1.7 TiB
48.74
> POOLS:
> NAME ID USED %USED MAX AVAIL
OBJECTS
> cephfs_data 1 29 GiB 100.00 0 B
2597
> cephfs_md 2 831 MiB 100.00 0 B
133
> erasure_meta_hdd 3 22 MiB 100.00 0 B
170
> VMPool0 4 1.2 TiB 56.77 644 GiB
116011
> stresspool 5 2.6 MiB 100.00 0 B
32
>
> [root@Sim-str-R6-4 ~]# ceph df detail -f json-pretty
> -----snippet-----
> {
> "name": "VMPool0",
> "id": 4,
> "stats": {
> "kb_used": 1328888832,
> "bytes_used": 1360782163968, <----------------
> "percent_used": 0.567110,
> "max_avail": 692481687552,
> "objects": 116011,
> "quota_objects": 0,
> "quota_bytes": 0,
> "dirty": 116011,
> "rd": 27449034,
> "rd_bytes": 126572760064,
> "wr": 20675381,
> "wr_bytes": 1006460652544,
> "comp_ratio": 1.000000,
> "stored": 497657610240,
> "stored_raw": 746486431744, <----------------
> }
> },
>
> The perf counters of all osds (all hdd) used by VMPool0
also show
> that bluestore_allocated is much larger than bluestore_stored.
>
> [root@Sim-str-R6-4 ~]# for i in {0..3}; do echo $i; ceph daemon
osd.$i perf
> dump | grep bluestore | head -6; done
> 0
> "bluestore": {
> "bluestore_allocated": 175032369152, <----------------
> "bluestore_stored": 83557936482, <----------------
> "bluestore_compressed": 958795770,
> "bluestore_compressed_allocated": 6431965184,
> "bluestore_compressed_original": 18576584704,
> 1
> "bluestore": {
> "bluestore_allocated": 119943593984, <----------------
> "bluestore_stored": 53325238866, <----------------
> "bluestore_compressed": 670158436,
> "bluestore_compressed_allocated": 4751818752,
> "bluestore_compressed_original": 13752328192,
> 2
> "bluestore": {
> "bluestore_allocated": 155444707328, <----------------
> "bluestore_stored": 69067116553, <----------------
> "bluestore_compressed": 565170876,
> "bluestore_compressed_allocated": 4614324224,
> "bluestore_compressed_original": 13469696000,
> 3
> "bluestore": {
> "bluestore_allocated": 128179240960, <----------------
> "bluestore_stored": 60884752114, <----------------
> "bluestore_compressed": 1653455847,
> "bluestore_compressed_allocated": 9741795328,
> "bluestore_compressed_original": 27878768640,
>
> [root@Sim-str-R6-5 osd]# for i in {4..7}; do echo $i; sh -c
"ceph daemon
> osd.$i perf dump | grep bluestore | head -6"; done
> 4
> "bluestore": {
> "bluestore_allocated": 165950652416, <----------------
> "bluestore_stored": 80255191687, <----------------
> "bluestore_compressed": 1526871060,
> "bluestore_compressed_allocated": 8900378624,
> "bluestore_compressed_original": 25324142592,
> 5
> admin_socket: exception getting command descriptions: [Errno 111]
> Connection refused
> 6
> "bluestore": {
> "bluestore_allocated": 166022152192, <----------------
> "bluestore_stored": 84645390708, <----------------
> "bluestore_compressed": 1169055606,
> "bluestore_compressed_allocated": 8647278592,
> "bluestore_compressed_original": 25135091712,
> 7
> "bluestore": {
> "bluestore_allocated": 204633604096, <----------------
> "bluestore_stored": 100116382041, <----------------
> "bluestore_compressed": 1081260422,
> "bluestore_compressed_allocated": 6510018560,
> "bluestore_compressed_original": 18654052352,
>
> [root@Sim-str-R6-6 osd]# for i in {8..12}; do echo $i; ceph
daemon osd.$i
> perf dump | grep bluestore | head -6; done
> 8
> "bluestore": {
> "bluestore_allocated": 106330193920, <----------------
> "bluestore_stored": 45282848089, <----------------
> "bluestore_compressed": 1136610231,
> "bluestore_compressed_allocated": 7248609280,
> "bluestore_compressed_original": 20882960384,
> 9
> "bluestore": {
> "bluestore_allocated": 120657412096, <----------------
> "bluestore_stored": 52550745942, <----------------
> "bluestore_compressed": 1321632665,
> "bluestore_compressed_allocated": 7401504768,
> "bluestore_compressed_original": 21073027072,
> 10
> "bluestore": {
> "bluestore_allocated": 155985772544, <----------------
> "bluestore_stored": 73236910054, <----------------
> "bluestore_compressed": 98351920,
> "bluestore_compressed_allocated": 772210688,
> "bluestore_compressed_original": 2242043904,
> 11
> "bluestore": {
> "bluestore_allocated": 106040524800, <----------------
> "bluestore_stored": 45353612134, <----------------
> "bluestore_compressed": 874216443,
> "bluestore_compressed_allocated": 4962844672,
> "bluestore_compressed_original": 14160310272,
> 12
> "bluestore": {
> "bluestore_allocated": 118751363072, <----------------
> "bluestore_stored": 52194408691, <----------------
> "bluestore_compressed": 782919969,
> "bluestore_compressed_allocated": 5546311680,
> "bluestore_compressed_original": 16043233280,
>
> The config min_alloc_size_hdd of all osds are 64K and
rados objects
> of VMPool0 are all 4M rbd_data.x.xxxxxxx.xxxxxxx objects. It's
kind of
> strange that allocation space is much larger than stored data.
Can anyone
> explains this?
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
<mailto:ceph-users@xxxxxxx>
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
<mailto:ceph-users-leave@xxxxxxx>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
<mailto:ceph-users@xxxxxxx>
To unsubscribe send an email to ceph-users-leave@xxxxxxx
<mailto:ceph-users-leave@xxxxxxx>
