Thanks I will start testing and adding some datas this afternoon, to check if I got the same errors regards ----- Mail original ----- > De: "Eugen Block" <eblock@xxxxxx> > À: "ceph-users" <ceph-users@xxxxxxx> > Envoyé: Mercredi 8 Juin 2022 10:49:44 > Objet: Re: Many errors about PG deviate more than 30% on a new cluster deployed by cephadm > It looks like your cluster doesn't have any "real" data in it yet, > only metadata. I see something similar in an Octopus cluster where one > device class is not used yet. I misread your output from the first > email. The autoscaler will increase pg_num as soon as you push data > into it, no need to tear the cluster down. > > Zitat von Christophe BAILLON <cb@xxxxxxx>: > >> Hello, >> >> thanks for your reply >> >> No did not stop autoscaler >> >> root@store-par2-node01:/home/user# ceph osd pool autoscale-status >> POOL SIZE TARGET SIZE RATE RAW >> CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW >> PG_NUM AUTOSCALE BULK >> .mgr 15488k 3.0 >> 2763T 0.0000 1.0 1 >> on False >> cephfs_data 0 1.3333333730697632 >> 2763T 0.0000 1.0 32 >> on False >> cephfs_metadata 5742 3.0 >> 2763T 0.0000 4.0 32 >> on False >> >> I not in production, I can destroy the pool and recreate it, or >> destroy the cluster and rebuild it >> >> >> ----- Mail original ----- >>> De: "Eugen Block" <eblock@xxxxxx> >>> À: "ceph-users" <ceph-users@xxxxxxx> >>> Envoyé: Mardi 7 Juin 2022 15:00:39 >>> Objet: Re: Many errors about PG deviate more than 30% >>> on a new cluster deployed by cephadm >> >>> Hi, >>> >>> please share the output of 'ceph osd pool autoscale-status'. You have >>> very low (too low) PG numbers per OSD (between 0 and 6), did you stop >>> the autoscaler at an early stage? If you don't want to use the >>> autoscaler you should increase the pg_num, but you could set >>> autoscaler to warn mode and see what it suggests. >>> >>> >>> Zitat von Christophe BAILLON <cb@xxxxxxx>: >>> >>>> Hi all >>>> >>>> I got many error about PG deviation more than 30% on a new >>>> installed cluster >>>> >>>> This cluster is managed by cephadm >>>> >>>> all box 15 have : >>>> 12 x 18Tb >>>> 2 x nvme >>>> 2 x ssd for boot >>>> >>>> Our main pool is on EC 6 + 2 for exclusive use with cephfs >>>> created with this method : >>>> ceph orch apply -i osd_spec.yaml >>>> >>>> with this conf >>>> >>>> osd_spec.yaml >>>> service_type: osd >>>> service_id: osd_spec_default >>>> placement: >>>> host_pattern: '*' >>>> data_devices: >>>> rotational: 1 >>>> db_devices: >>>> paths: >>>> - /dev/nvme0n1 >>>> - /dev/nvme1n1 >>>> >>>> root@store-par2-node01:/home/user# ceph -s >>>> cluster: >>>> id: cf37418e-e0b9-11ec-95a5-f1f73d5801cb >>>> health: HEALTH_OK >>>> >>>> services: >>>> mon: 5 daemons, quorum >>>> store-par2-node01,store-par2-node02,store-par2-node03,store-par2-node04,store-par2-node05 >>>> (age >>>> 5d) >>>> mgr: store-par2-node02.osbvrb(active, since 3d), standbys: >>>> store-par2-node01.oerpqs >>>> mds: 1/1 daemons up, 3 standby >>>> osd: 168 osds: 168 up (since 3d), 168 in (since 5d) >>>> >>>> data: >>>> volumes: 1/1 healthy >>>> pools: 3 pools, 65 pgs >>>> objects: 27 objects, 14 MiB >>>> usage: 13 TiB used, 2.7 PiB / 2.7 PiB avail >>>> pgs: 65 active+clean >>>> >>>> root@store-par2-node01:/home/user# ceph osd df tree >>>> ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP >>>> META AVAIL %USE VAR PGS STATUS TYPE NAME >>>> -1 2763.38159 - 2.7 PiB 13 TiB 9.6 GiB 9 KiB >>>> 9.7 GiB 2.7 PiB 0.47 1.00 - root default >>>> -3 197.38440 - 197 TiB 955 GiB 696 MiB 7 KiB >>>> 608 MiB 196 TiB 0.47 1.00 - host store-par2-node01 >>>> 0 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 60 MiB 16 TiB 0.47 1.00 2 up osd.0 >>>> 1 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 55 MiB 16 TiB 0.47 1.00 1 up osd.1 >>>> 2 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 55 MiB 16 TiB 0.47 1.00 1 up osd.2 >>>> 3 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 1 KiB >>>> 46 MiB 16 TiB 0.47 1.00 5 up osd.3 >>>> 4 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 1 KiB >>>> 45 MiB 16 TiB 0.47 1.00 2 up osd.4 >>>> 5 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 1 KiB >>>> 45 MiB 16 TiB 0.47 1.00 1 up osd.5 >>>> 6 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 1 KiB >>>> 45 MiB 16 TiB 0.47 1.00 0 up osd.6 >>>> 7 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 1 KiB >>>> 46 MiB 16 TiB 0.47 1.00 1 up osd.7 >>>> 8 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 1 KiB >>>> 47 MiB 16 TiB 0.47 1.00 4 up osd.8 >>>> 9 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 1 KiB >>>> 49 MiB 16 TiB 0.47 1.00 2 up osd.9 >>>> 10 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 55 MiB 16 TiB 0.47 1.00 4 up osd.10 >>>> 11 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 60 MiB 16 TiB 0.47 1.00 3 up osd.11 >>>> -7 197.38440 - 197 TiB 955 GiB 696 MiB 2 KiB >>>> 625 MiB 196 TiB 0.47 1.00 - host store-par2-node02 >>>> 13 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 54 MiB 16 TiB 0.47 1.00 5 up osd.13 >>>> 15 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 54 MiB 16 TiB 0.47 1.00 2 up osd.15 >>>> 17 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 54 MiB 16 TiB 0.47 1.00 3 up osd.17 >>>> 19 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 54 MiB 16 TiB 0.47 1.00 2 up osd.19 >>>> 21 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 54 MiB 16 TiB 0.47 1.00 3 up osd.21 >>>> 23 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 54 MiB 16 TiB 0.47 1.00 1 up osd.23 >>>> 25 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 54 MiB 16 TiB 0.47 1.00 3 up osd.25 >>>> 27 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 54 MiB 16 TiB 0.47 1.00 1 up osd.27 >>>> 29 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 54 MiB 16 TiB 0.47 1.00 1 up osd.29 >>>> 31 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 54 MiB 16 TiB 0.47 1.00 4 up osd.31 >>>> 33 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 1 KiB >>>> 43 MiB 16 TiB 0.47 1.00 2 up osd.33 >>>> 35 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 1 KiB >>>> 42 MiB 16 TiB 0.47 1.00 2 up osd.35 >>>> -5 197.38440 - 197 TiB 955 GiB 710 MiB 0 B >>>> 994 MiB 196 TiB 0.47 1.00 - host store-par2-node03 >>>> 12 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 54 MiB 16 TiB 0.47 1.00 0 up osd.12 >>>> 14 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 54 MiB 16 TiB 0.47 1.00 2 up osd.14 >>>> 16 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 54 MiB 16 TiB 0.47 1.00 0 up osd.16 >>>> 18 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 54 MiB 16 TiB 0.47 1.00 3 up osd.18 >>>> 20 hdd 16.44870 1.00000 16 TiB 80 GiB 71 MiB 0 B >>>> 400 MiB 16 TiB 0.47 1.00 4 up osd.20 >>>> 22 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 54 MiB 16 TiB 0.47 1.00 0 up osd.22 >>>> 24 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 54 MiB 16 TiB 0.47 1.00 3 up osd.24 >>>> 26 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 54 MiB 16 TiB 0.47 1.00 2 up osd.26 >>>> 28 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 54 MiB 16 TiB 0.47 1.00 4 up osd.28 >>>> 30 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 54 MiB 16 TiB 0.47 1.00 3 up osd.30 >>>> 32 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 54 MiB 16 TiB 0.47 1.00 1 up osd.32 >>>> 34 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 54 MiB 16 TiB 0.47 1.00 4 up osd.34 >>>> -29 197.38440 - 197 TiB 955 GiB 710 MiB 0 B >>>> 978 MiB 196 TiB 0.47 1.00 - host store-par2-node04 >>>> 38 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 2 up osd.38 >>>> 53 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 1 up osd.53 >>>> 65 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 2 up osd.65 >>>> 78 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 0 up osd.78 >>>> 89 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 0 up osd.89 >>>> 101 hdd 16.44870 1.00000 16 TiB 80 GiB 71 MiB 0 B >>>> 395 MiB 16 TiB 0.47 1.00 3 up osd.101 >>>> 112 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 4 up osd.112 >>>> 123 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 0 up osd.123 >>>> 134 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 3 up osd.134 >>>> 145 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 3 up osd.145 >>>> 156 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 2 up osd.156 >>>> 167 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 2 up osd.167 >>>> -11 197.38440 - 197 TiB 955 GiB 696 MiB 0 B >>>> 636 MiB 196 TiB 0.47 1.00 - host store-par2-node05 >>>> 46 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 2 up osd.46 >>>> 51 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 3 up osd.51 >>>> 60 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 3 up osd.60 >>>> 70 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 1 up osd.70 >>>> 81 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 3 up osd.81 >>>> 92 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 2 up osd.92 >>>> 103 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 1 up osd.103 >>>> 114 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 2 up osd.114 >>>> 125 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 0 up osd.125 >>>> 136 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 1 up osd.136 >>>> 147 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 5 up osd.147 >>>> 158 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 3 up osd.158 >>>> -13 197.38440 - 197 TiB 955 GiB 696 MiB 0 B >>>> 636 MiB 196 TiB 0.47 1.00 - host store-par2-node06 >>>> 36 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 2 up osd.36 >>>> 47 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 1 up osd.47 >>>> 59 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 1 up osd.59 >>>> 69 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 1 up osd.69 >>>> 82 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 2 up osd.82 >>>> 94 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 1 up osd.94 >>>> 106 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 2 up osd.106 >>>> 115 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 1 up osd.115 >>>> 128 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 2 up osd.128 >>>> 137 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 1 up osd.137 >>>> 148 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 2 up osd.148 >>>> 159 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 3 up osd.159 >>>> -23 197.38440 - 197 TiB 955 GiB 696 MiB 0 B >>>> 636 MiB 196 TiB 0.47 1.00 - host store-par2-node07 >>>> 41 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 3 up osd.41 >>>> 48 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 2 up osd.48 >>>> 58 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 1 up osd.58 >>>> 71 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 3 up osd.71 >>>> 83 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 4 up osd.83 >>>> 93 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 3 up osd.93 >>>> 105 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 2 up osd.105 >>>> 117 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 1 up osd.117 >>>> 129 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 3 up osd.129 >>>> 138 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 0 up osd.138 >>>> 150 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 5 up osd.150 >>>> 163 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 2 up osd.163 >>>> -17 197.38440 - 197 TiB 955 GiB 696 MiB 0 B >>>> 636 MiB 196 TiB 0.47 1.00 - host store-par2-node08 >>>> 43 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 2 up osd.43 >>>> 56 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 0 up osd.56 >>>> 68 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 1 up osd.68 >>>> 79 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 0 up osd.79 >>>> 90 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 1 up osd.90 >>>> 100 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 0 up osd.100 >>>> 111 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 0 up osd.111 >>>> 121 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 2 up osd.121 >>>> 132 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 7 up osd.132 >>>> 143 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 4 up osd.143 >>>> 154 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 2 up osd.154 >>>> 165 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 0 up osd.165 >>>> -27 197.38440 - 197 TiB 955 GiB 696 MiB 0 B >>>> 636 MiB 196 TiB 0.47 1.00 - host store-par2-node09 >>>> 37 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 4 up osd.37 >>>> 49 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 3 up osd.49 >>>> 63 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 3 up osd.63 >>>> 77 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 1 up osd.77 >>>> 88 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 2 up osd.88 >>>> 98 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 3 up osd.98 >>>> 110 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 4 up osd.110 >>>> 120 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 1 up osd.120 >>>> 131 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 4 up osd.131 >>>> 141 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 1 up osd.141 >>>> 152 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 2 up osd.152 >>>> 164 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 2 up osd.164 >>>> -9 197.38440 - 197 TiB 955 GiB 710 MiB 0 B >>>> 982 MiB 196 TiB 0.47 1.00 - host store-par2-node10 >>>> 45 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 4 up osd.45 >>>> 54 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 3 up osd.54 >>>> 64 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 1 up osd.64 >>>> 74 hdd 16.44870 1.00000 16 TiB 80 GiB 71 MiB 0 B >>>> 399 MiB 16 TiB 0.47 1.00 5 up osd.74 >>>> 87 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 0 up osd.87 >>>> 97 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 2 up osd.97 >>>> 107 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 4 up osd.107 >>>> 116 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 1 up osd.116 >>>> 127 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 2 up osd.127 >>>> 142 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 3 up osd.142 >>>> 153 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 1 up osd.153 >>>> 161 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 1 up osd.161 >>>> -15 197.38440 - 197 TiB 955 GiB 696 MiB 0 B >>>> 637 MiB 196 TiB 0.47 1.00 - host store-par2-node11 >>>> 40 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 54 MiB 16 TiB 0.47 1.00 6 up osd.40 >>>> 50 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 2 up osd.50 >>>> 61 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 2 up osd.61 >>>> 73 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 4 up osd.73 >>>> 84 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 1 up osd.84 >>>> 95 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 1 up osd.95 >>>> 104 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 5 up osd.104 >>>> 118 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 2 up osd.118 >>>> 126 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 1 up osd.126 >>>> 139 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 1 up osd.139 >>>> 149 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 3 up osd.149 >>>> 160 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 2 up osd.160 >>>> -25 197.38440 - 197 TiB 955 GiB 696 MiB 0 B >>>> 636 MiB 196 TiB 0.47 1.00 - host store-par2-node12 >>>> 39 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 1 up osd.39 >>>> 52 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 2 up osd.52 >>>> 66 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 2 up osd.66 >>>> 76 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 3 up osd.76 >>>> 86 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 3 up osd.86 >>>> 99 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 3 up osd.99 >>>> 109 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 2 up osd.109 >>>> 122 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 3 up osd.122 >>>> 133 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 2 up osd.133 >>>> 144 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 3 up osd.144 >>>> 155 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 0 up osd.155 >>>> 166 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 2 up osd.166 >>>> -19 197.38440 - 197 TiB 955 GiB 696 MiB 0 B >>>> 636 MiB 196 TiB 0.47 1.00 - host store-par2-node13 >>>> 42 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 1 up osd.42 >>>> 55 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 1 up osd.55 >>>> 62 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 0 up osd.62 >>>> 72 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 1 up osd.72 >>>> 80 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 3 up osd.80 >>>> 91 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 5 up osd.91 >>>> 102 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 1 up osd.102 >>>> 113 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 2 up osd.113 >>>> 124 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 1 up osd.124 >>>> 135 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 4 up osd.135 >>>> 146 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 3 up osd.146 >>>> 157 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 1 up osd.157 >>>> -21 197.38440 - 197 TiB 955 GiB 696 MiB 0 B >>>> 636 MiB 196 TiB 0.47 1.00 - host store-par2-node14 >>>> 44 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 2 up osd.44 >>>> 57 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 1 up osd.57 >>>> 67 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 4 up osd.67 >>>> 75 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 1 up osd.75 >>>> 85 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 3 up osd.85 >>>> 96 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 3 up osd.96 >>>> 108 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 1 up osd.108 >>>> 119 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 1 up osd.119 >>>> 130 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 3 up osd.130 >>>> 140 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 0 up osd.140 >>>> 151 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 2 up osd.151 >>>> 162 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >>>> 53 MiB 16 TiB 0.47 1.00 2 up osd.162 >>>> TOTAL 2.7 PiB 13 TiB 9.6 GiB 9.6 KiB >>>> 9.7 GiB 2.7 PiB 0.47 >>>> MIN/MAX VAR: 1.00/1.00 STDDEV: 0 >>>> >>>> >>>> root@store-par2-node01:/home/user# ceph pg dump >>>> version 140443 >>>> stamp 2022-06-06T12:27:35.382577+0000 >>>> last_osdmap_epoch 0 >>>> last_pg_scan 0 >>>> PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND >>>> BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE >>>> STATE_STAMP VERSION REPORTED UP >>>> UP_PRIMARY ACTING >>>> ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP >>>> LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN >>>> LAST_SCRUB_DURATION SCRUB_SCHEDULING >>>> OBJECTS_SCRUBBED OBJECTS_TRIMMED >>>> 3.f 1 0 0 0 0 >>>> 22 0 0 2 2 active+clean >>>> 2022-06-06T02:22:55.899870+0000 773'2 1191:813 >>>> [112,132,142] 112 [112,132,142] >>>> 112 773'2 2022-06-06T02:22:55.899728+0000 >>>> 773'2 2022-06-03T11:08:04.230774+0000 0 >>>> 1 periodic scrub scheduled @ 2022-06-07T02:26:26.925460+0000 >>>> 1 0 >>>> 2.e 0 0 0 0 0 >>>> 0 0 0 0 0 active+clean >>>> 2022-06-05T18:09:54.859041+0000 0'0 1191:818 >>>> [8,3,44,13,64,67,132,31] 8 >>>> [8,3,44,13,64,67,132,31] 8 0'0 >>>> 2022-06-05T18:09:54.858891+0000 0'0 >>>> 2022-05-31T19:51:41.372150+0000 0 1 >>>> periodic scrub scheduled @ 2022-06-07T02:48:10.931317+0000 >>>> 0 0 >>>> 3.c 0 0 0 0 0 >>>> 0 0 0 0 0 active+clean >>>> 2022-06-05T22:21:13.363161+0000 0'0 1191:813 >>>> [24,8,52] 24 [24,8,52] >>>> 24 0'0 2022-06-05T22:21:13.363041+0000 >>>> 0'0 2022-05-31T19:51:59.873880+0000 0 >>>> 1 periodic scrub scheduled @ 2022-06-07T05:12:34.695987+0000 >>>> 0 0 >>>> 2.d 0 0 0 0 0 >>>> 0 0 0 0 0 active+clean >>>> 2022-06-05T21:38:59.942761+0000 0'0 1191:819 >>>> [51,134,3,86,96,38,1,41] 51 >>>> [51,134,3,86,96,38,1,41] 51 0'0 >>>> 2022-06-05T21:38:59.942666+0000 0'0 >>>> 2022-05-31T19:51:41.372150+0000 0 1 >>>> periodic scrub scheduled @ 2022-06-06T22:34:01.326089+0000 >>>> 0 0 >>>> 3.d 1 0 0 0 0 >>>> 70 0 0 2 2 active+clean >>>> 2022-06-06T06:48:17.160609+0000 771'2 1191:816 >>>> [122,71,132] 122 [122,71,132] >>>> 122 771'2 2022-06-06T06:48:17.160469+0000 >>>> 771'2 2022-06-03T10:46:08.465824+0000 0 >>>> 1 periodic scrub scheduled @ 2022-06-07T10:22:35.365062+0000 >>>> 1 0 >>>> 2.c 0 0 0 0 0 >>>> 0 0 0 0 0 active+clean >>>> 2022-06-06T04:17:49.079390+0000 0'0 1191:819 >>>> [149,19,48,11,158,60,143,110] 149 >>>> [149,19,48,11,158,60,143,110] 149 0'0 >>>> 2022-06-06T04:17:49.079269+0000 0'0 >>>> 2022-05-31T19:51:41.372150+0000 0 1 >>>> periodic scrub scheduled @ 2022-06-07T06:16:29.279337+0000 >>>> 0 0 >>>> 3.a 0 0 0 0 0 >>>> 0 0 0 0 0 active+clean >>>> 2022-06-06T02:51:41.932102+0000 0'0 1191:813 >>>> [18,148,131] 18 [18,148,131] >>>> 18 0'0 2022-06-06T02:51:41.931974+0000 >>>> 0'0 2022-06-04T15:55:16.683239+0000 0 >>>> 1 periodic scrub scheduled @ 2022-06-07T10:39:23.180043+0000 >>>> 0 0 >>>> 2.b 0 0 0 0 0 >>>> 0 0 0 0 0 active+clean >>>> 2022-06-06T10:08:33.237436+0000 0'0 1191:818 >>>> [46,159,28,74,137,37,91,127] 46 >>>> [46,159,28,74,137,37,91,127] 46 0'0 >>>> 2022-06-06T10:08:33.237340+0000 0'0 >>>> 2022-06-06T10:08:33.237340+0000 0 1 >>>> periodic scrub scheduled @ 2022-06-07T10:13:44.177114+0000 >>>> 0 0 >>>> 3.b 1 0 0 0 0 >>>> 0 0 0 3 3 active+clean >>>> 2022-06-06T00:54:52.451573+0000 671'3 1191:814 >>>> [73,159,147] 73 [73,159,147] >>>> 73 671'3 2022-06-06T00:54:52.451459+0000 >>>> 671'3 2022-06-03T03:41:08.026476+0000 0 >>>> 1 periodic scrub scheduled @ 2022-06-07T12:45:10.753628+0000 >>>> 1 0 >>>> 2.a 0 0 0 0 0 >>>> 0 0 0 0 0 active+clean >>>> 2022-06-05T22:53:09.039828+0000 0'0 1191:815 >>>> [44,86,132,94,71,81,80,166] 44 >>>> [44,86,132,94,71,81,80,166] 44 0'0 >>>> 2022-06-05T22:53:09.039734+0000 0'0 >>>> 2022-06-03T12:58:37.531167+0000 0 1 >>>> periodic scrub scheduled @ 2022-06-07T03:20:37.899744+0000 >>>> 0 0 >>>> 3.8 0 0 0 0 0 >>>> 0 0 0 0 0 active+clean >>>> 2022-06-05T14:40:36.607866+0000 0'0 1191:811 >>>> [115,38,68] 115 [115,38,68] >>>> 115 0'0 2022-06-05T14:40:36.607760+0000 >>>> 0'0 2022-05-31T19:51:59.873880+0000 0 >>>> 1 periodic scrub scheduled @ 2022-06-06T14:45:02.993139+0000 >>>> 0 0 >>>> 2.9 0 0 0 0 0 >>>> 0 0 0 0 0 active+clean >>>> 2022-06-06T03:20:19.920382+0000 0'0 1191:817 >>>> [104,84,110,40,82,73,147,17] 104 >>>> [104,84,110,40,82,73,147,17] 104 0'0 >>>> 2022-06-06T03:20:19.920257+0000 0'0 >>>> 2022-05-31T19:51:41.372150+0000 0 1 >>>> periodic scrub scheduled @ 2022-06-07T06:24:06.295277+0000 >>>> 0 0 >>>> 3.9 0 0 0 0 0 >>>> 0 0 0 0 0 active+clean >>>> 2022-06-05T22:14:34.542404+0000 0'0 1191:811 >>>> [59,45,129] 59 [59,45,129] >>>> 59 0'0 2022-06-05T22:14:34.542285+0000 >>>> 0'0 2022-06-03T00:14:09.631316+0000 0 >>>> 1 periodic scrub scheduled @ 2022-06-07T06:57:37.776582+0000 >>>> 0 0 >>>> 2.8 0 0 0 0 0 >>>> 0 0 0 0 0 active+clean >>>> 2022-06-05T22:12:11.172949+0000 0'0 1191:815 >>>> [45,101,104,13,143,157,63,61] 45 >>>> [45,101,104,13,143,157,63,61] 45 0'0 >>>> 2022-06-05T22:12:11.172853+0000 0'0 >>>> 2022-05-31T19:51:41.372150+0000 0 1 >>>> periodic scrub scheduled @ 2022-06-07T06:52:53.792844+0000 >>>> 0 0 >>>> 3.6 1 0 0 0 0 >>>> 0 0 0 5 5 active+clean >>>> 2022-06-05T10:01:53.243050+0000 1043'5 1191:818 >>>> [98,66,132] 98 [98,66,132] >>>> 98 1043'5 2022-06-05T10:01:53.242948+0000 >>>> 1043'5 2022-06-05T10:01:53.242948+0000 0 >>>> 1 periodic scrub scheduled @ 2022-06-06T18:00:59.024882+0000 >>>> 1 0 >>>> 2.7 0 0 0 0 0 >>>> 0 0 0 0 0 active+clean >>>> 2022-06-06T08:28:41.853562+0000 0'0 1191:818 >>>> [163,57,151,110,116,118,30,147] 163 >>>> [163,57,151,110,116,118,30,147] 163 0'0 >>>> 2022-06-06T08:28:41.853404+0000 0'0 >>>> 2022-05-31T19:51:41.372150+0000 0 1 >>>> periodic scrub scheduled @ 2022-06-07T18:31:27.727201+0000 >>>> 0 0 >>>> 3.7 1 0 0 0 0 >>>> 0 0 0 2 2 active+clean >>>> 2022-06-06T11:35:38.186470+0000 437'2 1191:826 >>>> [95,31,55] 95 [95,31,55] >>>> 95 437'2 2022-06-06T11:35:38.186365+0000 >>>> 0'0 2022-05-31T19:51:59.873880+0000 0 >>>> 1 periodic scrub scheduled @ 2022-06-07T12:34:45.361534+0000 >>>> 1 0 >>>> 2.6 0 0 0 0 0 >>>> 0 0 0 0 0 active+clean >>>> 2022-06-06T06:13:11.860365+0000 0'0 1191:817 >>>> [135,112,19,98,114,88,119,8] 135 >>>> [135,112,19,98,114,88,119,8] 135 0'0 >>>> 2022-06-06T06:13:11.860264+0000 0'0 >>>> 2022-06-06T06:13:11.860264+0000 0 1 >>>> periodic scrub scheduled @ 2022-06-07T08:33:30.327642+0000 >>>> 0 0 >>>> 3.4 0 0 0 0 0 >>>> 0 0 0 0 0 active+clean >>>> 2022-06-05T11:53:13.051840+0000 0'0 1191:813 >>>> [10,13,131] 10 [10,13,131] >>>> 10 0'0 2022-06-05T11:53:13.051678+0000 >>>> 0'0 2022-06-02T02:18:09.375232+0000 0 >>>> 1 periodic scrub scheduled @ 2022-06-06T14:15:18.261225+0000 >>>> 0 0 >>>> 2.5 0 0 0 0 0 >>>> 0 0 0 0 0 active+clean >>>> 2022-06-05T15:15:43.099447+0000 0'0 1191:817 >>>> [15,4,8,144,81,96,73,10] 15 >>>> [15,4,8,144,81,96,73,10] 15 0'0 >>>> 2022-06-05T15:15:43.099314+0000 0'0 >>>> 2022-06-04T11:38:14.307983+0000 0 1 >>>> periodic scrub scheduled @ 2022-06-06T22:44:13.570864+0000 >>>> 0 0 >>>> 2.0 0 0 0 0 0 >>>> 0 0 0 0 0 active+clean >>>> 2022-06-05T23:46:06.362681+0000 0'0 1191:844 >>>> [30,109,13,40,133,136,37,65] 30 >>>> [30,109,13,40,133,136,37,65] 30 0'0 >>>> 2022-06-05T23:46:06.362567+0000 0'0 >>>> 2022-06-05T23:46:06.362567+0000 0 1 >>>> periodic scrub scheduled @ 2022-06-07T05:50:48.092572+0000 >>>> 0 0 >>>> 3.1 0 0 0 0 0 >>>> 0 0 0 0 0 active+clean >>>> 2022-06-05T16:32:59.830885+0000 0'0 1191:831 >>>> [73,150,54] 73 [73,150,54] >>>> 73 0'0 2022-06-05T16:32:59.830683+0000 >>>> 0'0 2022-06-05T16:32:59.830683+0000 0 >>>> 1 periodic scrub scheduled @ 2022-06-07T02:38:21.783899+0000 >>>> 0 0 >>>> 1.0 5 0 0 0 0 >>>> 14156320 0 0 10005 10005 active+clean >>>> 2022-06-05T16:52:10.117602+0000 1146'21705 1191:26731 >>>> [74,101,20] 74 [74,101,20] >>>> 74 1012'18422 2022-06-05T16:52:10.117484+0000 >>>> 0'0 2022-05-31T17:19:28.164530+0000 0 >>>> 1 periodic scrub scheduled @ 2022-06-06T23:19:35.924762+0000 >>>> 4 0 >>>> 2.3 0 0 0 0 0 >>>> 0 0 0 0 0 active+clean >>>> 2022-06-05T15:59:18.303912+0000 0'0 1191:819 >>>> [83,121,122,34,92,21,32,72] 83 >>>> [83,121,122,34,92,21,32,72] 83 0'0 >>>> 2022-06-05T15:59:18.303794+0000 0'0 >>>> 2022-05-31T19:51:41.372150+0000 0 1 >>>> periodic scrub scheduled @ 2022-06-07T02:34:48.910045+0000 >>>> 0 0 >>>> 3.2 0 0 0 0 0 >>>> 0 0 0 0 0 active+clean >>>> 2022-06-06T03:07:55.854126+0000 0'0 1191:811 >>>> [164,31,34] 164 [164,31,34] >>>> 164 0'0 2022-06-06T03:07:55.854014+0000 >>>> 0'0 2022-05-31T19:51:59.873880+0000 0 >>>> 1 periodic scrub scheduled @ 2022-06-07T08:11:53.316960+0000 >>>> 0 0 >>>> 2.1 0 0 0 0 0 >>>> 0 0 0 0 0 active+clean >>>> 2022-06-05T13:03:19.274658+0000 0'0 1191:821 >>>> [40,76,99,128,43,135,10,96] 40 >>>> [40,76,99,128,43,135,10,96] 40 0'0 >>>> 2022-06-05T13:03:19.274545+0000 0'0 >>>> 2022-05-31T19:51:41.372150+0000 0 1 >>>> periodic scrub scheduled @ 2022-06-06T18:43:52.710554+0000 >>>> 0 0 >>>> 3.0 0 0 0 0 0 >>>> 0 0 0 0 0 active+clean >>>> 2022-06-05T13:33:06.917207+0000 0'0 1191:826 >>>> [35,40,70] 35 [35,40,70] >>>> 35 0'0 2022-06-05T13:33:06.917074+0000 >>>> 0'0 2022-05-31T19:52:08.234732+0000 0 >>>> 1 periodic scrub scheduled @ 2022-06-06T18:08:03.107087+0000 >>>> 0 0 >>>> 3.3 0 0 0 0 0 >>>> 0 0 0 0 0 active+clean >>>> 2022-06-06T11:43:56.320476+0000 0'0 1191:840 >>>> [40,18,11] 40 [40,18,11] >>>> 40 0'0 2022-06-06T11:43:56.320361+0000 >>>> 0'0 2022-06-04T06:13:59.535590+0000 0 >>>> 1 periodic scrub scheduled @ 2022-06-07T20:55:33.787766+0000 >>>> 0 0 >>>> 2.2 0 0 0 0 0 >>>> 0 0 0 0 0 active+clean >>>> 2022-06-05T16:42:20.204103+0000 0'0 1191:819 >>>> [71,150,24,82,144,48,161,91] 71 >>>> [71,150,24,82,144,48,161,91] 71 0'0 >>>> 2022-06-05T16:42:20.203967+0000 0'0 >>>> 2022-06-03T04:54:17.344478+0000 0 1 >>>> periodic scrub scheduled @ 2022-06-06T21:40:54.542796+0000 >>>> 0 0 >>>> 3.5 0 0 0 0 0 >>>> 0 0 0 0 0 active+clean >>>> 2022-06-05T20:06:50.152236+0000 0'0 1191:812 >>>> [152,3,93] 152 [152,3,93] >>>> 152 0'0 2022-06-05T20:06:50.152134+0000 >>>> 0'0 2022-06-03T08:27:56.053931+0000 0 >>>> 1 periodic scrub scheduled @ 2022-06-07T05:07:45.942271+0000 >>>> 0 0 >>>> 2.4 0 0 0 0 0 >>>> 0 0 0 0 0 active+clean >>>> 2022-06-05T23:44:27.330764+0000 0'0 1191:816 >>>> [11,132,21,7,104,149,49,134] 11 >>>> [11,132,21,7,104,149,49,134] 11 0'0 >>>> 2022-06-05T23:44:27.330642+0000 0'0 >>>> 2022-06-05T23:44:27.330642+0000 0 1 >>>> periodic scrub scheduled @ 2022-06-07T03:48:00.509825+0000 >>>> 0 0 >>>> 3.e 1 0 0 0 0 >>>> 0 0 0 3 3 active+clean >>>> 2022-06-05T20:45:57.476116+0000 543'3 1191:814 >>>> [37,97,26] 37 [37,97,26] >>>> 37 543'3 2022-06-05T20:45:57.476006+0000 >>>> 543'3 2022-06-02T00:50:17.270315+0000 0 >>>> 1 periodic scrub scheduled @ 2022-06-07T00:57:37.043277+0000 >>>> 1 0 >>>> 2.f 0 0 0 0 0 >>>> 0 0 0 0 0 active+clean >>>> 2022-06-06T10:18:51.955868+0000 0'0 1191:827 >>>> [92,85,101,53,25,24,113,145] 92 >>>> [92,85,101,53,25,24,113,145] 92 0'0 >>>> 2022-06-06T10:18:51.955714+0000 0'0 >>>> 2022-06-05T06:18:02.624483+0000 0 1 >>>> periodic scrub scheduled @ 2022-06-07T12:10:14.007061+0000 >>>> 0 0 >>>> 3.11 0 0 0 0 0 >>>> 0 0 0 0 0 active+clean >>>> 2022-06-05T20:15:43.246605+0000 0'0 1191:811 >>>> [33,30,41] 33 [33,30,41] >>>> 33 0'0 2022-06-05T20:15:43.246445+0000 >>>> 0'0 2022-05-31T19:51:59.873880+0000 0 >>>> 1 periodic scrub scheduled @ 2022-06-07T01:59:04.536885+0000 >>>> 0 0 >>>> 2.10 0 0 0 0 0 >>>> 0 0 0 0 0 active+clean >>>> 2022-06-06T03:41:55.049114+0000 0'0 1191:819 >>>> [130,167,43,143,66,65,45,145] 130 >>>> [130,167,43,143,66,65,45,145] 130 0'0 >>>> 2022-06-06T03:41:55.048887+0000 0'0 >>>> 2022-06-06T03:41:55.048887+0000 0 1 >>>> periodic scrub scheduled @ 2022-06-07T12:59:20.841601+0000 >>>> 0 0 >>>> 3.10 4 0 0 0 0 >>>> 0 0 0 12 12 active+clean >>>> 2022-06-05T06:44:51.783819+0000 845'12 1191:816 >>>> [130,103,93] 130 [130,103,93] >>>> 130 845'12 2022-06-05T06:44:51.783710+0000 >>>> 845'12 2022-06-03T19:13:59.475041+0000 0 >>>> 1 periodic scrub scheduled @ 2022-06-06T16:31:37.232782+0000 >>>> 4 0 >>>> 2.11 0 0 0 0 0 >>>> 0 0 0 0 0 active+clean >>>> 2022-06-06T08:38:43.779419+0000 0'0 1191:820 >>>> [107,20,129,122,27,113,25,83] 107 >>>> [107,20,129,122,27,113,25,83] 107 0'0 >>>> 2022-06-06T08:38:43.779269+0000 0'0 >>>> 2022-06-03T11:14:02.841282+0000 0 1 >>>> periodic scrub scheduled @ 2022-06-07T16:53:01.369556+0000 >>>> 0 0 >>>> 3.13 2 0 0 0 0 >>>> 34 0 0 2 2 active+clean >>>> 2022-06-05T15:00:46.595842+0000 437'2 1191:816 >>>> [149,14,85] 149 [149,14,85] >>>> 149 437'2 2022-06-05T15:00:46.595730+0000 >>>> 0'0 2022-05-31T19:51:59.873880+0000 0 >>>> 1 periodic scrub scheduled @ 2022-06-06T16:27:45.919806+0000 >>>> 2 0 >>>> 2.12 0 0 0 0 0 >>>> 0 0 0 0 0 active+clean >>>> 2022-06-06T07:10:51.207987+0000 0'0 1191:815 >>>> [35,139,147,106,60,28,126,127] 35 >>>> [35,139,147,106,60,28,126,127] 35 0'0 >>>> 2022-06-06T07:10:51.207861+0000 0'0 >>>> 2022-05-31T19:51:41.372150+0000 0 1 >>>> periodic scrub scheduled @ 2022-06-07T17:59:37.937977+0000 >>>> 0 0 >>>> 3.12 0 0 0 0 0 >>>> 0 0 0 0 0 active+clean >>>> 2022-06-05T09:38:30.626384+0000 0'0 1191:812 >>>> [135,154,131] 135 [135,154,131] >>>> 135 0'0 2022-06-05T09:38:30.626285+0000 >>>> 0'0 2022-06-02T22:21:59.376573+0000 0 >>>> 1 periodic scrub scheduled @ 2022-06-06T18:43:10.579197+0000 >>>> 0 0 >>>> 2.13 0 0 0 0 0 >>>> 0 0 0 0 0 active+clean >>>> 2022-06-06T01:39:10.599882+0000 0'0 1191:818 >>>> [74,36,3,158,39,77,75,98] 74 >>>> [74,36,3,158,39,77,75,98] 74 0'0 >>>> 2022-06-06T01:39:10.599742+0000 0'0 >>>> 2022-05-31T19:51:41.372150+0000 0 1 >>>> periodic scrub scheduled @ 2022-06-07T07:50:19.048812+0000 >>>> 0 0 >>>> 3.15 2 0 0 0 0 >>>> 90 0 0 3 3 active+clean >>>> 2022-06-06T00:06:19.711529+0000 437'3 1191:817 >>>> [54,15,102] 54 [54,15,102] >>>> 54 437'3 2022-06-06T00:06:19.711396+0000 >>>> 0'0 2022-05-31T19:51:59.873880+0000 0 >>>> 1 periodic scrub scheduled @ 2022-06-07T09:05:26.475289+0000 >>>> 2 0 >>>> 2.14 0 0 0 0 0 >>>> 0 0 0 0 0 active+clean >>>> 2022-06-05T13:23:51.227730+0000 0'0 1191:815 >>>> [63,18,104,31,28,80,107,47] 63 >>>> [63,18,104,31,28,80,107,47] 63 0'0 >>>> 2022-06-05T13:23:51.227619+0000 0'0 >>>> 2022-05-31T19:51:41.372150+0000 0 1 >>>> periodic scrub scheduled @ 2022-06-07T00:57:54.831856+0000 >>>> 0 0 >>>> 3.14 4 0 0 0 0 >>>> 4972 0 0 37 37 active+clean >>>> 2022-06-05T18:43:38.650054+0000 718'37 1191:851 >>>> [83,114,153] 83 [83,114,153] >>>> 83 718'37 2022-06-05T18:43:38.649890+0000 >>>> 718'37 2022-06-05T18:43:38.649890+0000 0 >>>> 1 periodic scrub scheduled @ 2022-06-07T01:34:37.932141+0000 >>>> 4 0 >>>> 2.15 0 0 0 0 0 >>>> 0 0 0 0 0 active+clean >>>> 2022-06-05T16:15:50.881954+0000 0'0 1191:819 >>>> [54,88,142,33,76,20,146,150] 54 >>>> [54,88,142,33,76,20,146,150] 54 0'0 >>>> 2022-06-05T16:15:50.881848+0000 0'0 >>>> 2022-06-01T21:50:03.827583+0000 0 1 >>>> periodic scrub scheduled @ 2022-06-06T16:24:10.967689+0000 >>>> 0 0 >>>> 3.17 0 0 0 0 0 >>>> 0 0 0 0 0 active+clean >>>> 2022-06-05T16:14:32.120728+0000 0'0 1191:814 >>>> [147,34,167] 147 [147,34,167] >>>> 147 0'0 2022-06-05T16:14:32.120636+0000 >>>> 0'0 2022-05-31T19:51:59.873880+0000 0 >>>> 1 periodic scrub scheduled @ 2022-06-07T01:17:17.942645+0000 >>>> 0 0 >>>> 2.16 0 0 0 0 0 >>>> 0 0 0 0 0 active+clean >>>> 2022-06-05T13:32:29.676884+0000 0'0 1191:815 >>>> [162,152,23,74,160,9,121,10] 162 >>>> [162,152,23,74,160,9,121,10] 162 0'0 >>>> 2022-06-05T13:32:29.676755+0000 0'0 >>>> 2022-06-01T23:18:50.309093+0000 0 1 >>>> periodic scrub scheduled @ 2022-06-06T15:29:19.493434+0000 >>>> 0 0 >>>> 3.16 0 0 0 0 0 >>>> 0 0 0 0 0 active+clean >>>> 2022-06-06T03:05:19.005785+0000 0'0 1191:813 >>>> [26,162,105] 26 [26,162,105] >>>> 26 0'0 2022-06-06T03:05:19.005679+0000 >>>> 0'0 2022-05-31T19:52:13.903980+0000 0 >>>> 1 periodic scrub scheduled @ 2022-06-07T05:38:41.233380+0000 >>>> 0 0 >>>> 2.17 0 0 0 0 0 >>>> 0 0 0 0 0 active+clean >>>> 2022-06-05T16:25:32.999773+0000 0'0 1191:821 >>>> [146,104,158,163,108,106,141,159] 146 >>>> [146,104,158,163,108,106,141,159] 146 0'0 >>>> 2022-06-05T16:25:32.999664+0000 0'0 >>>> 2022-05-31T19:51:41.372150+0000 0 1 >>>> periodic scrub scheduled @ 2022-06-07T03:44:47.468570+0000 >>>> 0 0 >>>> 3.19 0 0 0 0 0 >>>> 0 0 0 0 0 active+clean >>>> 2022-06-05T18:18:12.164305+0000 0'0 1191:810 >>>> [156,40,109] 156 [156,40,109] >>>> 156 0'0 2022-06-05T18:18:12.164187+0000 >>>> 0'0 2022-05-31T19:51:59.873880+0000 0 >>>> 1 periodic scrub scheduled @ 2022-06-06T20:18:30.481546+0000 >>>> 0 0 >>>> 2.18 0 0 0 0 0 >>>> 0 0 0 0 0 active+clean >>>> 2022-06-05T18:42:00.227559+0000 0'0 1191:820 >>>> [107,105,97,17,99,124,154,132] 107 >>>> [107,105,97,17,99,124,154,132] 107 0'0 >>>> 2022-06-05T18:42:00.227444+0000 0'0 >>>> 2022-05-31T19:51:41.372150+0000 0 1 >>>> periodic scrub scheduled @ 2022-06-07T00:48:30.997533+0000 >>>> 0 0 >>>> 3.18 0 0 0 0 0 >>>> 0 0 0 0 0 active+clean >>>> 2022-06-05T22:29:35.968669+0000 0'0 1191:813 >>>> [150,21,36] 150 [150,21,36] >>>> 150 0'0 2022-06-05T22:29:35.968576+0000 >>>> 0'0 2022-06-01T20:07:31.626680+0000 0 >>>> 1 periodic scrub scheduled @ 2022-06-07T08:00:54.614246+0000 >>>> 0 0 >>>> 2.19 0 0 0 0 0 >>>> 0 0 0 0 0 active+clean >>>> 2022-06-06T12:25:06.541967+0000 0'0 1192:820 >>>> [131,151,63,13,146,90,135,133] 131 >>>> [131,151,63,13,146,90,135,133] 131 0'0 >>>> 2022-06-06T12:25:06.541855+0000 0'0 >>>> 2022-05-31T19:51:41.372150+0000 0 1 >>>> periodic scrub scheduled @ 2022-06-07T16:49:45.723687+0000 >>>> 0 0 >>>> 3.1b 0 0 0 0 0 >>>> 0 0 0 1 1 active+clean >>>> 2022-06-05T12:05:36.929834+0000 437'1 1191:813 >>>> [91,120,142] 91 [91,120,142] >>>> 91 437'1 2022-06-05T12:05:36.929706+0000 >>>> 437'1 2022-06-02T00:32:32.667254+0000 0 >>>> 1 periodic scrub scheduled @ 2022-06-06T20:33:52.594948+0000 >>>> 0 0 >>>> 2.1a 0 0 0 0 0 >>>> 0 0 0 0 0 active+clean >>>> 2022-06-05T21:33:42.799425+0000 0'0 1191:817 >>>> [58,112,85,50,129,148,49,45] 58 >>>> [58,112,85,50,129,148,49,45] 58 0'0 >>>> 2022-06-05T21:33:42.799305+0000 0'0 >>>> 2022-05-31T19:51:41.372150+0000 0 1 >>>> periodic scrub scheduled @ 2022-06-07T08:52:06.385325+0000 >>>> 0 0 >>>> 3.1a 0 0 0 0 0 >>>> 0 0 0 1 1 active+clean >>>> 2022-06-05T14:57:02.326186+0000 437'1 1191:815 >>>> [25,9,93] 25 [25,9,93] >>>> 25 437'1 2022-06-05T14:57:02.326068+0000 >>>> 0'0 2022-05-31T19:51:59.873880+0000 0 >>>> 1 periodic scrub scheduled @ 2022-06-06T22:00:49.835756+0000 >>>> 0 0 >>>> 2.1b 0 0 0 0 0 >>>> 0 0 0 0 0 active+clean >>>> 2022-06-06T00:21:22.013136+0000 0'0 1191:819 >>>> [51,42,60,83,164,145,52,134] 51 >>>> [51,42,60,83,164,145,52,134] 51 0'0 >>>> 2022-06-06T00:21:22.013028+0000 0'0 >>>> 2022-06-06T00:21:22.013028+0000 0 1 >>>> periodic scrub scheduled @ 2022-06-07T08:38:12.431836+0000 >>>> 0 0 >>>> 3.1d 1 0 0 0 0 >>>> 0 0 0 2 2 active+clean >>>> 2022-06-06T00:17:33.395234+0000 437'2 1191:815 >>>> [150,67,20] 150 [150,67,20] >>>> 150 437'2 2022-06-06T00:17:33.395098+0000 >>>> 0'0 2022-05-31T19:51:59.873880+0000 0 >>>> 1 periodic scrub scheduled @ 2022-06-07T05:38:06.038546+0000 >>>> 1 0 >>>> 2.1c 0 0 0 0 0 >>>> 0 0 0 0 0 active+clean >>>> 2022-06-05T21:38:58.966683+0000 0'0 1191:819 >>>> [51,34,80,144,143,99,50,0] 51 >>>> [51,34,80,144,143,99,50,0] 51 0'0 >>>> 2022-06-05T21:38:58.966531+0000 0'0 >>>> 2022-06-03T07:43:07.143846+0000 0 1 >>>> periodic scrub scheduled @ 2022-06-07T04:01:52.286592+0000 >>>> 0 0 >>>> 3.1c 1 0 0 0 0 >>>> 0 0 0 3 3 active+clean >>>> 2022-06-06T05:23:02.301909+0000 724'3 1191:815 >>>> [76,156,128] 76 [76,156,128] >>>> 76 724'3 2022-06-06T05:23:02.301761+0000 >>>> 724'3 2022-06-03T07:49:44.813182+0000 0 >>>> 1 periodic scrub scheduled @ 2022-06-07T11:37:54.326001+0000 >>>> 1 0 >>>> 2.1d 0 0 0 0 0 >>>> 0 0 0 0 0 active+clean >>>> 2022-06-06T03:14:29.829206+0000 0'0 1191:816 >>>> [67,86,61,81,130,69,74,46] 67 >>>> [67,86,61,81,130,69,74,46] 67 0'0 >>>> 2022-06-06T03:14:29.829088+0000 0'0 >>>> 2022-06-03T06:14:14.581690+0000 0 1 >>>> periodic scrub scheduled @ 2022-06-07T12:37:45.026826+0000 >>>> 0 0 >>>> 3.1f 2 0 0 0 0 >>>> 554 0 0 4 4 active+clean >>>> 2022-06-05T13:34:03.534647+0000 437'4 1191:815 >>>> [160,112,49] 160 [160,112,49] >>>> 160 437'4 2022-06-05T13:34:03.534531+0000 >>>> 0'0 2022-05-31T19:51:59.873880+0000 0 >>>> 1 periodic scrub scheduled @ 2022-06-06T15:30:53.351210+0000 >>>> 2 0 >>>> 2.1e 0 0 0 0 0 >>>> 0 0 0 0 0 active+clean >>>> 2022-06-05T16:05:18.546234+0000 0'0 1191:817 >>>> [5,0,91,118,2,107,41,37] 5 >>>> [5,0,91,118,2,107,41,37] 5 0'0 >>>> 2022-06-05T16:05:18.546142+0000 0'0 >>>> 2022-05-31T19:51:41.372150+0000 0 1 >>>> periodic scrub scheduled @ 2022-06-06T23:22:10.205148+0000 >>>> 0 0 >>>> 3.1e 0 0 0 0 0 >>>> 0 0 0 1 1 active+clean >>>> 2022-06-06T02:05:23.255525+0000 437'1 1191:814 >>>> [3,166,110] 3 [3,166,110] >>>> 3 437'1 2022-06-06T02:05:23.255430+0000 >>>> 0'0 2022-05-31T19:51:59.873880+0000 0 >>>> 1 periodic scrub scheduled @ 2022-06-07T12:43:16.169680+0000 >>>> 0 0 >>>> 2.1f 0 0 0 0 0 >>>> 0 0 0 0 0 active+clean >>>> 2022-06-05T14:19:38.386679+0000 0'0 1191:816 >>>> [67,14,117,29,28,4,91,17] 67 >>>> [67,14,117,29,28,4,91,17] 67 0'0 >>>> 2022-06-05T14:19:38.386564+0000 0'0 >>>> 2022-06-03T08:33:01.050897+0000 0 1 >>>> periodic scrub scheduled @ 2022-06-06T23:02:03.102683+0000 >>>> 0 0 >>>> >>>> 3 22 0 0 0 0 5742 0 0 83 83 >>>> 1 5 0 0 0 0 14156320 0 0 10005 10005 >>>> 2 0 0 0 0 0 0 0 0 0 0 >>>> >>>> sum 27 0 0 0 0 14162062 0 0 10088 10088 >>>> OSD_STAT USED AVAIL USED_RAW TOTAL HB_PEERS >>>> >>>> PG_SUM PRIMARY_PG_SUM >>>> 167 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [0,11,34,35,90,91,92,93,94,95,96,97,98,133,166] 2 >>>> 166 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [9,32,33,80,81,82,83,84,85,87,88,111,112,165,167] 2 >>>> 0 >>>> 77 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [3,22,23,76,78,101,109,154,157,158,159,160,161,162,163] 1 >>>> 0 >>>> 76 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [2,20,21,75,77,96,98,128,143,145,146,147,148,149,150,153,156] >>>> 3 1 >>>> 75 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [11,18,19,74,76,86,87,132,134,135,136,137,138,139,141] 1 >>>> 0 >>>> 74 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [3,10,16,17,20,36,39,61,67,73,75,77,98,101,121,123,124,125,128,129,131,133,158] >>>> 5 >>>> 2 >>>> 73 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [1,14,15,54,55,72,74,111,112,114,115,117,119,120,122,147,150,159] >>>> 4 2 >>>> 72 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [0,12,13,40,41,71,73,100,101,103,106,107,108,109,110] 1 >>>> 0 >>>> 71 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [8,24,34,35,48,70,72,82,89,90,91,94,95,96,97,98,99,144,150,157,158,161] >>>> 3 1 >>>> 70 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [9,32,33,69,71,78,79,80,84,85,86,87,88,148,150] 1 >>>> 69 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [7,30,31,65,68,70,71,72,73,74,75,76,77,132,136] 1 >>>> 68 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [6,28,29,53,58,60,61,62,63,64,66,67,69,128,130] 1 >>>> 67 80 GiB 16 TiB 80 GiB 16 TiB >>>> [4,5,14,17,26,27,28,29,38,46,47,48,49,50,51,54,55,61,66,68,69,74,81,86,91,111,117,122,130] >>>> 4 >>>> 2 >>>> 66 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [4,24,25,36,37,40,41,42,45,46,65,67,101,108,165] 2 >>>> 0 >>>> 65 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [3,22,23,64,66,97,99,154,157,158,159,160,162,163,164] 2 >>>> 0 >>>> 64 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [2,20,21,63,65,78,88,143,146,147,148,149,150,151,155] 1 >>>> 0 >>>> 63 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [11,18,19,28,31,47,62,64,72,74,80,104,107,132,134,136,137,138,139,140,144] >>>> 3 1 >>>> 62 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [10,16,17,49,50,61,63,121,123,125,127,128,129,130,133] 0 >>>> 0 >>>> 61 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [1,14,15,42,46,60,62,111,112,115,116,117,119,120,122] 2 >>>> 0 >>>> 60 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [0,12,13,59,61,100,101,102,105,107,108,109,110,159,160] 3 >>>> 0 >>>> 59 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [8,34,35,45,58,60,89,90,91,95,96,97,98,99,129,147,150] 1 >>>> 1 >>>> 58 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [9,32,33,45,49,50,57,59,78,79,80,81,84,85,86,87,88,112,129,137,140,148] >>>> 1 1 >>>> 57 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [7,30,31,56,58,65,69,70,72,73,74,76,77,121,129] 1 >>>> 56 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [6,28,29,53,55,57,58,59,60,61,63,64,66,113,119] 0 >>>> 55 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [5,26,27,38,47,48,49,50,51,52,54,56,57,100,107] 1 >>>> 54 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [4,15,20,24,25,33,36,37,39,40,41,44,46,53,55,76,88,89,91,102,142,146,150,165] >>>> 3 >>>> 2 >>>> 53 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [3,22,23,52,54,86,87,154,157,158,159,160,162,163,164] 1 >>>> 0 >>>> 52 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [2,20,21,51,53,65,70,143,146,148,149,150,151,152,153] 2 >>>> 0 >>>> 51 80 GiB 16 TiB 80 GiB 16 TiB >>>> [0,1,3,11,18,19,34,38,41,42,50,52,60,61,66,80,83,86,96,99,132,134,135,137,138,140,141,142,143,144,145,164] >>>> 3 >>>> 3 >>>> 50 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [10,16,17,37,46,49,51,121,123,124,127,128,129,130,133] 2 >>>> 0 >>>> 49 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [1,14,15,48,50,111,112,113,114,115,116,119,122,160,163] 3 >>>> 0 >>>> 48 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [0,12,13,47,49,100,101,102,103,104,107,108,109,148,152] 2 >>>> 0 >>>> 47 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [8,34,35,46,48,89,90,91,95,96,97,98,99,136,138] 1 >>>> 46 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [9,28,32,33,37,45,47,74,78,79,80,83,84,85,86,88,91,127,128,137,159] >>>> 2 1 >>>> 45 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [7,13,30,31,44,46,61,63,65,68,69,71,72,73,76,77,101,104,114,119,143,157] >>>> 4 1 >>>> 44 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [6,28,29,43,45,53,58,59,60,61,62,63,66,71,80,81,86,94,100,107,132,166] >>>> 2 1 >>>> 43 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [5,26,27,38,42,44,47,48,49,50,51,52,54,91,96] 2 0 >>>> 42 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [4,24,25,36,37,39,40,41,43,44,45,46,79,83,167] 1 0 >>>> 41 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [3,22,23,40,42,72,73,154,156,158,159,161,162,164,166] 3 >>>> 0 >>>> 40 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [2,10,11,18,20,21,39,41,43,58,66,76,96,99,128,135,143,145,146,147,148,151,152,153] >>>> 6 >>>> 2 >>>> 39 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [11,18,19,38,40,50,132,135,136,137,138,140,141,142,167] 1 >>>> 0 >>>> 38 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [10,16,17,37,39,121,124,125,126,127,128,129,130,164,166] 2 >>>> 0 >>>> 37 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [1,14,15,26,36,38,97,111,113,114,116,117,118,119,122,145,148] >>>> 4 1 >>>> 36 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [0,12,19,35,37,100,101,102,103,104,105,107,108,109,141] 2 >>>> 0 >>>> 15 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [4,8,10,11,14,16,26,73,81,96,132,134,135,136,137,138,139,140,141,142,144] >>>> 2 1 >>>> 14 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [10,13,15,25,121,123,124,125,126,127,128,129,130,131,133] 2 >>>> 0 >>>> 13 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [1,12,14,22,111,112,113,114,115,116,117,118,119,120,122] 5 >>>> 0 >>>> 12 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [1,11,13,17,100,101,102,103,104,105,106,107,108,109,110] 0 >>>> 0 >>>> 11 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [7,10,12,14,21,35,49,89,90,91,92,93,94,95,96,97,98,99,104,132,134,149] >>>> 3 1 >>>> 10 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [9,11,13,32,33,78,79,80,81,82,83,84,85,86,87,88,131] 4 >>>> 1 >>>> 9 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [8,10,30,31,65,68,69,70,71,72,73,74,75,76,77] 2 0 >>>> 8 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [3,7,9,13,28,29,31,44,53,56,58,59,60,61,62,63,64,66,67,132] 4 >>>> 1 >>>> 7 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [6,8,26,27,38,43,47,48,49,50,51,52,54,55,57] 1 0 >>>> 6 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [5,7,24,25,36,37,39,40,41,42,44,45,46,165,167] 0 0 >>>> 1 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [0,2,14,15,111,112,113,114,115,116,117,118,119,120,122] 1 >>>> 0 >>>> 0 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [1,12,13,100,101,102,103,104,105,106,107,108,109,110,167] 2 >>>> 0 >>>> 2 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [1,3,16,17,121,123,124,125,126,127,128,129,130,131,133] 1 >>>> 0 >>>> 3 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [2,4,18,19,110,132,134,135,136,137,138,139,140,141,142,144,166] >>>> 5 1 >>>> 4 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [3,5,20,21,143,145,146,147,148,149,150,151,152,153,155] 2 >>>> 0 >>>> 5 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [0,2,4,6,22,23,37,41,91,107,118,154,156,157,158,159,160,161,162,163,164,166] >>>> 1 1 >>>> 16 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [2,15,17,29,143,145,146,147,148,149,150,151,152,153,155] 0 >>>> 0 >>>> 17 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [3,16,18,30,154,156,157,158,159,160,161,162,163,164,166] 3 >>>> 0 >>>> 18 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [4,17,19,33,36,37,39,40,41,42,44,45,46,131,148,165,167] 3 >>>> 1 >>>> 19 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [5,18,20,34,38,43,47,48,49,50,51,52,54,55,57] 2 0 >>>> 20 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [6,13,19,21,53,56,58,59,60,61,62,63,64,66,67] 4 0 >>>> 21 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [7,14,20,22,65,68,69,70,71,72,73,74,75,76,77] 3 0 >>>> 22 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [9,17,21,23,78,79,80,81,82,83,84,85,86,87,88] 0 0 >>>> 23 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [8,18,22,24,89,90,91,92,93,94,95,96,97,98,99] 1 0 >>>> 24 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [0,8,21,23,25,52,100,101,102,103,104,105,106,107,108,109,110] >>>> 3 1 >>>> 25 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [1,9,22,24,26,93,111,112,113,114,115,116,117,118,119,120,122] >>>> 3 1 >>>> 26 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [10,25,27,29,105,121,123,124,125,126,127,128,129,130,131,133,162] >>>> 2 1 >>>> 27 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [11,26,28,30,132,134,135,136,137,138,139,140,141,142,144] 1 >>>> 0 >>>> 28 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [2,27,29,33,143,145,146,147,148,149,150,151,152,153,155] 4 >>>> 0 >>>> 29 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [3,28,30,34,154,156,157,158,159,160,161,162,163,164,166] 1 >>>> 0 >>>> 30 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [4,13,29,31,36,37,39,40,41,42,44,45,46,65,109,133,136,165,167] >>>> 3 1 >>>> 31 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [5,14,30,32,38,43,47,48,49,50,51,52,54,55,57] 4 0 >>>> 32 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [6,17,31,33,53,56,58,59,60,61,62,63,64,66,67] 1 0 >>>> 33 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [7,18,30,32,34,41,65,68,69,70,71,72,73,74,75,76,77] 2 >>>> 1 >>>> 34 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [9,21,33,35,78,79,80,81,82,83,84,85,86,87,88] 4 0 >>>> 35 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [8,16,28,34,36,40,60,70,89,90,91,92,93,95,96,97,98,99,106,126,127,128,139,147] >>>> 2 >>>> 2 >>>> 78 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [4,24,25,36,39,40,41,42,44,45,46,77,79,111,120] 0 >>>> 79 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [5,26,27,47,48,49,50,51,52,54,57,78,80,123,124] 0 >>>> 80 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [6,28,29,53,58,59,61,63,64,66,67,79,81,132,136] 3 >>>> 81 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [7,30,31,65,68,71,73,74,75,76,77,80,82,146,148] 3 >>>> 82 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [9,32,33,78,79,80,81,83,84,85,86,87,88,158,163] 2 >>>> 83 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [8,21,32,34,35,36,40,72,82,84,89,90,91,92,96,97,98,99,114,121,122,153] >>>> 4 2 >>>> 84 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [0,12,13,48,57,83,85,100,101,102,103,106,107,109,110] 1 >>>> 0 >>>> 85 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [1,14,15,61,66,84,86,111,112,113,114,115,116,117,120] 3 >>>> 0 >>>> 86 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [10,16,17,74,75,85,87,121,123,124,125,126,128,129,131] 3 >>>> 0 >>>> 87 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [11,18,19,86,88,98,99,132,134,135,136,137,138,139,140] 0 >>>> 0 >>>> 88 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [2,20,21,87,89,101,107,143,146,147,148,149,150,151,155] 2 >>>> 0 >>>> 89 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [3,22,23,88,90,111,120,157,158,159,160,161,162,163,166] 0 >>>> 0 >>>> 90 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [4,24,25,36,37,39,40,41,44,45,46,89,91,123,124] 1 >>>> 91 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [5,26,27,38,47,48,49,50,52,54,57,90,92,120,132,136,142] 5 >>>> 1 >>>> 92 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [6,24,25,28,29,53,56,59,61,63,64,66,67,85,91,93,101,113,145,146,150] >>>> 2 1 >>>> 93 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [7,30,31,65,68,72,73,74,75,76,77,92,94,158,159] 3 >>>> 94 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [9,32,33,40,41,78,79,80,81,85,86,87,88,93,95] 1 0 >>>> 95 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [8,31,34,35,47,55,57,89,90,91,92,93,94,96,97,98,99] 1 >>>> 1 >>>> 96 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [0,12,13,61,64,95,97,100,101,102,103,105,106,109,110] 3 >>>> 0 >>>> 97 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [1,14,15,75,77,96,98,111,112,113,114,115,117,118,122] 2 >>>> 0 >>>> 98 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [10,16,17,66,86,87,97,99,121,123,124,125,126,128,129,130,132] >>>> 3 1 >>>> 99 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [11,18,19,98,100,110,111,134,135,136,137,138,139,140,142] 3 >>>> 0 >>>> 100 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [2,20,21,99,101,122,123,146,147,148,149,150,151,152,153] 0 >>>> 0 >>>> 101 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [3,20,22,23,33,74,100,102,132,135,158,159,160,161,162,163,164,166] >>>> 3 0 >>>> 102 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [4,24,25,36,37,39,40,41,44,45,101,103,145,147,165] 1 >>>> 0 >>>> 103 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [5,26,27,38,43,47,48,49,52,54,57,102,104,157,160] 1 >>>> 0 >>>> 104 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [6,17,28,29,40,41,46,53,56,59,62,63,64,66,67,73,82,84,103,105,110,147] >>>> 5 1 >>>> 105 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [7,30,31,47,50,65,68,70,72,74,75,76,77,104,106] 2 >>>> 106 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [9,32,33,58,64,78,79,80,81,84,85,86,88,105,107] 2 >>>> 107 80 GiB 16 TiB 80 GiB 16 TiB >>>> [8,17,20,25,27,34,35,69,75,83,89,90,91,92,93,95,97,98,99,105,106,108,113,122,124,129,132,154] >>>> 4 >>>> 2 >>>> 108 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [0,12,13,86,87,100,101,102,103,104,105,106,107,109,110] 1 >>>> 0 >>>> 109 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [1,14,15,96,98,108,110,111,112,113,114,115,116,117,118] 2 >>>> 0 >>>> 110 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [10,16,17,100,109,111,122,123,124,125,126,127,128,129,130] 4 >>>> 0 >>>> 111 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [11,18,19,110,112,123,131,135,136,137,138,139,140,142,144] 0 >>>> 0 >>>> 112 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [2,20,21,111,113,132,135,142,147,148,149,150,151,152,153,155] >>>> 4 1 >>>> 113 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [3,22,23,112,114,145,147,154,159,160,161,162,163,164,166] 2 >>>> 0 >>>> 114 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [4,24,25,37,39,40,41,44,45,113,115,157,159,165,167] 2 >>>> 0 >>>> 115 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [5,26,27,38,43,45,46,48,49,50,52,55,57,68,114,116] 1 >>>> 1 >>>> 116 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [6,28,29,47,48,53,56,60,61,62,63,66,67,115,117] 1 >>>> 117 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [7,30,31,61,64,65,68,69,70,72,75,76,77,116,118] 1 >>>> 118 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [9,32,33,71,75,78,79,80,81,82,86,87,88,117,119] 2 >>>> 119 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [8,34,35,84,88,89,90,91,92,93,94,97,99,118,120] 1 >>>> 120 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [0,12,13,90,96,101,102,103,104,105,106,107,109,119,121] 1 >>>> 0 >>>> 121 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [1,14,15,109,110,112,113,114,115,116,117,118,119,120,122] 2 >>>> 0 >>>> 122 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [10,16,17,71,111,112,121,123,124,125,126,127,128,129,130,131,132] >>>> 3 1 >>>> 123 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [11,18,19,122,124,132,135,136,137,138,139,140,141,142,144] 0 >>>> 0 >>>> 124 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [2,20,21,123,125,143,145,147,148,149,150,151,152,153,155] 1 >>>> 0 >>>> 125 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [3,22,23,124,126,154,156,157,159,160,161,162,163,164,166] 0 >>>> 0 >>>> 126 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [4,24,25,36,37,39,41,42,44,45,46,125,127,165,167] 1 >>>> 0 >>>> 127 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [5,26,27,38,43,47,48,49,50,51,52,55,57,126,128] 2 >>>> 128 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [6,28,29,53,56,58,60,61,62,63,64,66,67,127,129] 2 >>>> 129 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [7,30,31,65,68,69,70,72,73,74,75,76,77,128,130] 3 >>>> 130 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [9,32,33,43,45,65,66,78,79,80,81,82,83,84,86,87,88,93,103,129,131,143,145,167] >>>> 3 >>>> 2 >>>> 131 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [8,13,34,35,63,89,90,91,92,93,94,95,96,97,99,130,132,133,135,146,151] >>>> 4 1 >>>> 132 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [0,12,13,101,102,103,104,105,106,107,108,109,110,131,133] 7 >>>> 0 >>>> 133 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [1,14,15,111,112,113,114,115,116,117,118,119,120,132,134] 2 >>>> 0 >>>> 134 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [10,16,17,121,124,125,126,127,128,129,130,131,133,135,144] 3 >>>> 0 >>>> 135 80 GiB 16 TiB 80 GiB 16 TiB >>>> [8,11,18,19,88,98,112,114,119,131,132,134,136,137,138,139,140,141,142,144,145,147,154] >>>> 4 >>>> 2 >>>> 136 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [2,20,21,135,137,143,145,149,150,151,152,153,155,157,159] 1 >>>> 0 >>>> 137 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [3,22,23,41,46,136,138,154,156,157,160,161,162,164,166] 1 >>>> 0 >>>> 138 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [4,24,25,37,39,42,44,45,46,47,50,137,139,165,167] 0 >>>> 0 >>>> 139 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [5,26,27,38,43,47,49,51,52,54,55,58,67,138,140] 1 >>>> 140 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [6,28,29,53,56,58,59,60,62,64,66,73,77,139,141] 0 >>>> 141 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [7,30,31,65,68,69,70,71,72,73,76,85,87,140,142] 1 >>>> 142 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [9,32,33,78,80,81,82,83,84,85,86,90,98,141,143] 3 >>>> 143 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [8,34,35,89,91,92,93,94,95,96,98,107,109,142,144] 4 >>>> 0 >>>> 144 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [0,12,13,102,103,104,105,106,107,108,110,111,112,143,145] 3 >>>> 0 >>>> 145 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [1,14,15,111,114,115,116,117,118,119,120,124,133,144,146] 3 >>>> 0 >>>> 146 80 GiB 16 TiB 80 GiB 16 TiB >>>> [10,16,17,104,106,108,121,126,127,128,129,130,131,133,134,136,141,145,147,158,159,163] >>>> 3 >>>> 1 >>>> 147 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [11,18,19,34,132,134,138,139,140,141,142,144,146,148,157,159,167] >>>> 5 1 >>>> 148 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [2,20,21,40,46,143,145,146,147,149,150,151,152,153,155] 2 >>>> 0 >>>> 149 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [3,11,14,19,22,23,47,48,60,85,110,143,148,150,154,156,157,158,161,162,164,166] >>>> 3 >>>> 2 >>>> 150 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [4,20,21,24,25,36,37,39,42,45,46,61,67,149,151,165,167] 5 >>>> 2 >>>> 151 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [5,26,27,38,43,47,50,51,52,54,55,71,77,150,152] 2 >>>> 152 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [3,6,28,29,53,56,58,59,60,61,62,66,85,87,93,151,153] 2 >>>> 1 >>>> 153 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [7,30,31,65,69,70,71,72,73,75,76,90,98,152,154] 1 >>>> 154 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [9,32,33,78,80,81,82,83,84,85,88,107,109,153,155] 2 >>>> 0 >>>> 155 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [8,34,35,91,92,93,94,95,96,97,98,111,112,154,156] 0 >>>> 0 >>>> 156 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [0,12,13,40,100,103,104,105,106,107,108,109,110,124,133,155,157] >>>> 2 1 >>>> 157 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [1,14,15,111,115,116,117,118,119,120,122,134,136,156,158] 1 >>>> 0 >>>> 158 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [10,16,17,121,123,126,127,129,130,131,133,146,148,157,159] 3 >>>> 0 >>>> 159 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [11,18,19,40,46,132,134,135,138,140,141,142,144,158,160] 3 >>>> 0 >>>> 160 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [2,20,21,47,49,54,112,143,145,146,147,150,151,152,155,159,161] >>>> 2 1 >>>> 161 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [3,22,23,61,67,154,156,157,158,159,160,162,163,164,166] 1 >>>> 0 >>>> 162 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [4,9,10,23,24,25,36,37,39,40,42,46,71,74,121,152,160,161,163,165,167] >>>> 2 1 >>>> 163 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [5,26,27,30,38,43,47,50,51,52,54,55,57,85,88,110,116,118,147,151,162,164] >>>> 2 1 >>>> 164 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [6,28,29,31,34,53,59,60,61,62,64,66,67,90,93,163,165] 2 >>>> 1 >>>> 165 80 GiB 16 TiB 80 GiB 16 TiB >>>> >>>> [7,30,31,65,69,70,71,72,73,74,75,109,110,164,166] 0 >>>> 0 >>>> sum 13 TiB 2.7 PiB 13 TiB 2.7 PiB >>>> >>>> * NOTE: Omap statistics are gathered during deep scrub and may be >>>> inaccurate soon afterwards depending on utilization. See >>>> http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics >>>> for further details. >>>> dumped all >>>> >>>> >>>> root@store-par2-node01:/home/user# ceph fs status >>>> cephfs - 2 clients >>>> ====== >>>> RANK STATE MDS ACTIVITY DNS >>>> INOS DIRS CAPS >>>> 0 active cephfs.store-par2-node01.honsru Reqs: 0 /s 10 >>>> 13 12 2 >>>> POOL TYPE USED AVAIL >>>> cephfs_metadata metadata 5742 870T >>>> cephfs_data data 0 1959T >>>> STANDBY MDS >>>> cephfs.store-par2-node02.ttsfvq >>>> cephfs.store-par2-node03.vnxscc >>>> cephfs.store-par2-node04.iudcse >>>> >>>> >>>> If somebody have an idea how can I correct this, or good pointers to >>>> learn about PG with cephadm, It will be nice ... >>>> >>>> Thanks >>>> >>>> >>>> -- >>>> Christophe BAILLON >>>> Mobile :: +336 16 400 522 >>>> Work :: https://eyona.com >>>> Twitter :: https://twitter.com/ctof >>>> >>>> >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users@xxxxxxx >>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx >>> >>> >>> >>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@xxxxxxx >>> To unsubscribe send an email to ceph-users-leave@xxxxxxx >> >> -- >> Christophe BAILLON >> Mobile :: +336 16 400 522 >> Work :: https://eyona.com >> Twitter :: https://twitter.com/ctof >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users@xxxxxxx >> To unsubscribe send an email to ceph-users-leave@xxxxxxx > > > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx -- Christophe BAILLON Mobile :: +336 16 400 522 Work :: https://eyona.com Twitter :: https://twitter.com/ctof _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |