Re: radosgw sync using more capacity on secondary than on master zone

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hello ,

It could be related to the « erasure-code-profile » which is defined with different k+m on master and secondary .
For the size of buckets I guess its probalbly due to the parameter : compression enabled on secondary radosgw « rgw compression : yes » .

Regards

De : Scheurer François <francois.scheurer@xxxxxxxxxxxx>
Envoyé : jeudi 31 décembre 2020 15:45
À : ceph-users@xxxxxxx
Objet :  radosgw sync using more capacity on secondary than on master zone

Dear Ceph contributors


While our (new) rgw secondary zone is doing the initial data sync from our master zone,
we noticed that the reported capacity usage was getting higher than on primary zone:

    Master Zone:
        ceph version 14.2.5
        zone parameters:
            "log_meta": "false",
            "log_data": "true",
            "bucket_index_max_shards": 0,
            "read_only": "false",
            "tier_type": "",
            "sync_from_all": "false",
            "sync_from": [],
            "redirect_zone": ""
        bucket stats:
         => "size_actual GiB": 269868.9823989868,
            "size_utilized GiB": 17180102071.868008, <= (cf. other issue below)
            "num_objects": 100218788,
            "Compression Rate": 63660.899148714125
        pool stats:
         => "stored": 403834132234240,
            "objects": 191530416,
            "kb_used": 692891724288,
            "bytes_used": 709521125670912,
            "percent_used": 0.7570806741714478,
            "max_avail": 136595529269248
        erasure-code-profile:
            crush-device-class=nvme
            crush-failure-domain=host
            crush-root=default
            jerasure-per-chunk-alignment=false
            k=8
            m=2
            plugin=jerasure
            technique=reed_sol_van
            w=8
        osd parameters:
            bluefs_alloc_size                                1048576  default
            bluefs_shared_alloc_size                         65536    default
            bluestore_extent_map_inline_shard_prealloc_size  256      default
            bluestore_max_alloc_size                         0        default
            bluestore_min_alloc_size                         0        default
            bluestore_min_alloc_size_hdd                     65536    default
            bluestore_min_alloc_size_ssd                     16384    default
        rgw compression: no

    Secondary Zone:
        ceph version 14.2.12
        zone parameters:
            "log_meta": "false",
            "log_data": "true",
            "bucket_index_max_shards": 11,
            "read_only": "true",
            "tier_type": "",
            "sync_from_all": "true",
            "sync_from": [],
            "redirect_zone": ""
        bucket stats:
         => "size_actual GiB": 65282.37313461304,
            "size_utilized GiB": 60779.72828538809,
            "num_objects": 23074921,
            "Compression Rate": 0.9310281683550253
        pool stats:
         => "stored": 407816305115136,
            "objects": 118637638,
            "kb_used": 497822635396,
            "bytes_used": 509770378645504,
            "percent_used": 0.7275146245956421,
            "max_avail": 152744706965504
        erasure-code-profile:
            crush-device-class=
            crush-failure-domain=host
            crush-root=default
            jerasure-per-chunk-alignment=false
            k=3
            m=2
            plugin=jerasure
            technique=reed_sol_van
            w=8
        osd parameters:
            EC k=8 m=2
            bluefs_alloc_size                                1048576  default
            bluefs_shared_alloc_size                         65536    default
            bluestore_extent_map_inline_shard_prealloc_size  256      default
            bluestore_max_alloc_size                         0        default
            bluestore_min_alloc_size                         0        default
            bluestore_min_alloc_size_hdd                     65536    default
            bluestore_min_alloc_size_ssd                     4096     default
        rgw compression: yes

As you see, the secondary zone is using 408 TB vs 404 TB on master zone.
Summing size_actual for each buckets gives only 65 TB vs 270 TB on master zone.
Any idea about what could cause such a difference?
Is it a known issue?

The are some known issue with Space Overhead with EC Pools for alloc_size > 4 KiB, cf. :
    https://www.mail-archive.com/ceph-users@xxxxxxx/msg06191.html
    https://lists.ceph.io/hyperkitty/list/dev@xxxxxxx/thread/OHPO43J54TPBEUISYCK3SRV55SIZX2AT/
    https://www.spinics.net/lists/ceph-users/msg59587.html
But our secondary zone is on nvme osd's with bluestore_min_alloc_size_ssd=4096, so that should be fine.

I will also investigate further with "radosgw-admin bucket radoslist" and rgw-orphan-list.


Thank you in advance for any help.
And Happy New Year very soon ;-)



Best Regards
Francois


PS:
The value on master zone "size_utilized GiB": 17180102071.868008 is wrong.
This is due to a bucket with wrong stats:
    {
        "bucket": "XXX",
        "tenant": "",
        "zonegroup": "d29ea82c-4f77-40af-952b-2ab0705ad268",
        "placement_rule": "default-placement",
        "explicit_placement": {
            "data_pool": "",
            "data_extra_pool": "",
            "index_pool": ""
        },
        "id": "cb1594b3-a782-49d0-a19f-68cd48870a63.3119460.1",
        "marker": "cb1594b3-a782-49d0-a19f-68cd48870a63.63841.1481",
        "index_type": "Normal",
        "owner": "e13d054f6e9c4eea881f687923d7d380",
        "ver": "0#373075,1#286613,2#290913,3#341069,4#360862,5#341416,6#279526,7#352172,8#255944,9#314797,10#317650,11#305698,12#289557,13#345344,14#345273,15#294708,16#241001,17#298577,18#274866,19#293952,20#307635,21#334606,22#265355,23#302567,24#277505,25#307278,26#266963,27#297452,28#332274,29#319133,30#361027,31#314294,32#282887,33#324849,34#278560,35#307506,36#287269,37#344789,38#345389,39#323814,40#386483,41#280319,42#358072,43#336651,44#339176,45#248079,46#356784,47#381496,48#295152,49#251661,50#318661,51#330530,52#263564,53#332005,54#332937,55#320163,56#300485,57#296138,58#343271,59#359351,60#295711,61#275751,62#332264,63#351532",
        "master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0,27#0,28#0,29#0,30#0,31#0,32#0,33#0,34#0,35#0,36#0,37#0,38#0,39#0,40#0,41#0,42#0,43#0,44#0,45#0,46#0,47#0,48#0,49#0,50#0,51#0,52#0,53#0,54#0,55#0,56#0,57#0,58#0,59#0,60#0,61#0,62#0,63#0",
        "mtime": "2020-06-03 12:13:01.207610Z",
        "max_marker": "0#,1#,2#,3#00000341068.137793799.5,4#00000360861.156871075.5,5#00000341415.97799619.5,6#00000279525.155275260.5,7#,8#,9#00000314796.95564320.5,10#,11#,12#00000289556.137784091.5,13#,14#,15#,16#00000241000.126242121.5,17#,18#00000274865.124884405.5,19#,20#00000307634.137793798.5,21#00000334605.93836734.5,22#,23#00000302566.125226103.5,24#,25#,26#,27#00000297451.125229375.5,28#,29#00000319132.155275278.5,30#00000361026.98341455.5,31#,32#,33#00000324848.126242117.5,34#00000278559.124884417.5,35#,36#,37#00000344788.125945123.5,38#00000345388.97796715.5,39#,40#00000386482.98341457.5,41#,42#00000358071.124884415.5,43#,44#00000339175.135084366.5,45#00000248078.155263912.5,46#00000356783.98341461.5,47#00000381495.94538350.5,48#00000295151.138701826.5,49#00000251660.137793803.5,50#00000318660.93848186.5,51#,52#,53#00000332004.126242119.5,54#00000332936.138701824.5,55#,56#00000300484.156871073.5,57#,58#00000343270.98341459.5,59#00000359350.94257302.5,60#,61#,62#,63#00000351531.135084368.5",
        "usage": {
            "rgw.none": {
                "size": 0,
                "size_actual": 0,
                "size_utilized": 0,
                "size_kb": 0,
                "size_kb_actual": 0,
                "size_kb_utilized": 0,
                "num_objects": 18446744073709551604
            },
            "rgw.main": {
                "size": 22615692324233,
                "size_actual": 22617004142592,
                "size_utilized": 18446736471689615156,
                "size_kb": 22085637036,
                "size_kb_actual": 22086918108,
                "size_kb_utilized": 18014391085634390,
                "num_objects": 521927
            },
            "rgw.multimeta": {
                "size": 0,
                "size_actual": 0,
                "size_utilized": 0,
                "size_kb": 0,
                "size_kb_actual": 0,
                "size_kb_utilized": 0,
                "num_objects": 977
            }
        },
        "bucket_quota": {
            "enabled": false,
            "check_on_raw": false,
            "max_size": -1,
            "max_size_kb": 0,
            "max_objects": -1
        }
    }
This is another issue, probably not related to our sync issue.


--


EveryWare AG
François Scheurer
Senior Systems Engineer
Zurlindenstrasse 52a
CH-8003 Zürich

tel: +41 44 466 60 00
fax: +41 44 466 60 10
mail: francois.scheurer@xxxxxxxxxxxx<mailto:francois.scheurer@xxxxxxxxxxxx>
web: http://www.everyware.ch

_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux