Hello, thanks for your reply No did not stop autoscaler root@store-par2-node01:/home/user# ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE BULK .mgr 15488k 3.0 2763T 0.0000 1.0 1 on False cephfs_data 0 1.3333333730697632 2763T 0.0000 1.0 32 on False cephfs_metadata 5742 3.0 2763T 0.0000 4.0 32 on False I not in production, I can destroy the pool and recreate it, or destroy the cluster and rebuild it ----- Mail original ----- > De: "Eugen Block" <eblock@xxxxxx> > À: "ceph-users" <ceph-users@xxxxxxx> > Envoyé: Mardi 7 Juin 2022 15:00:39 > Objet: Re: Many errors about PG deviate more than 30% on a new cluster deployed by cephadm > Hi, > > please share the output of 'ceph osd pool autoscale-status'. You have > very low (too low) PG numbers per OSD (between 0 and 6), did you stop > the autoscaler at an early stage? If you don't want to use the > autoscaler you should increase the pg_num, but you could set > autoscaler to warn mode and see what it suggests. > > > Zitat von Christophe BAILLON <cb@xxxxxxx>: > >> Hi all >> >> I got many error about PG deviation more than 30% on a new installed cluster >> >> This cluster is managed by cephadm >> >> all box 15 have : >> 12 x 18Tb >> 2 x nvme >> 2 x ssd for boot >> >> Our main pool is on EC 6 + 2 for exclusive use with cephfs >> created with this method : >> ceph orch apply -i osd_spec.yaml >> >> with this conf >> >> osd_spec.yaml >> service_type: osd >> service_id: osd_spec_default >> placement: >> host_pattern: '*' >> data_devices: >> rotational: 1 >> db_devices: >> paths: >> - /dev/nvme0n1 >> - /dev/nvme1n1 >> >> root@store-par2-node01:/home/user# ceph -s >> cluster: >> id: cf37418e-e0b9-11ec-95a5-f1f73d5801cb >> health: HEALTH_OK >> >> services: >> mon: 5 daemons, quorum >> store-par2-node01,store-par2-node02,store-par2-node03,store-par2-node04,store-par2-node05 >> (age >> 5d) >> mgr: store-par2-node02.osbvrb(active, since 3d), standbys: >> store-par2-node01.oerpqs >> mds: 1/1 daemons up, 3 standby >> osd: 168 osds: 168 up (since 3d), 168 in (since 5d) >> >> data: >> volumes: 1/1 healthy >> pools: 3 pools, 65 pgs >> objects: 27 objects, 14 MiB >> usage: 13 TiB used, 2.7 PiB / 2.7 PiB avail >> pgs: 65 active+clean >> >> root@store-par2-node01:/home/user# ceph osd df tree >> ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP >> META AVAIL %USE VAR PGS STATUS TYPE NAME >> -1 2763.38159 - 2.7 PiB 13 TiB 9.6 GiB 9 KiB >> 9.7 GiB 2.7 PiB 0.47 1.00 - root default >> -3 197.38440 - 197 TiB 955 GiB 696 MiB 7 KiB >> 608 MiB 196 TiB 0.47 1.00 - host store-par2-node01 >> 0 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 60 MiB 16 TiB 0.47 1.00 2 up osd.0 >> 1 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 55 MiB 16 TiB 0.47 1.00 1 up osd.1 >> 2 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 55 MiB 16 TiB 0.47 1.00 1 up osd.2 >> 3 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 1 KiB >> 46 MiB 16 TiB 0.47 1.00 5 up osd.3 >> 4 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 1 KiB >> 45 MiB 16 TiB 0.47 1.00 2 up osd.4 >> 5 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 1 KiB >> 45 MiB 16 TiB 0.47 1.00 1 up osd.5 >> 6 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 1 KiB >> 45 MiB 16 TiB 0.47 1.00 0 up osd.6 >> 7 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 1 KiB >> 46 MiB 16 TiB 0.47 1.00 1 up osd.7 >> 8 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 1 KiB >> 47 MiB 16 TiB 0.47 1.00 4 up osd.8 >> 9 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 1 KiB >> 49 MiB 16 TiB 0.47 1.00 2 up osd.9 >> 10 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 55 MiB 16 TiB 0.47 1.00 4 up osd.10 >> 11 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 60 MiB 16 TiB 0.47 1.00 3 up osd.11 >> -7 197.38440 - 197 TiB 955 GiB 696 MiB 2 KiB >> 625 MiB 196 TiB 0.47 1.00 - host store-par2-node02 >> 13 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 54 MiB 16 TiB 0.47 1.00 5 up osd.13 >> 15 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 54 MiB 16 TiB 0.47 1.00 2 up osd.15 >> 17 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 54 MiB 16 TiB 0.47 1.00 3 up osd.17 >> 19 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 54 MiB 16 TiB 0.47 1.00 2 up osd.19 >> 21 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 54 MiB 16 TiB 0.47 1.00 3 up osd.21 >> 23 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 54 MiB 16 TiB 0.47 1.00 1 up osd.23 >> 25 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 54 MiB 16 TiB 0.47 1.00 3 up osd.25 >> 27 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 54 MiB 16 TiB 0.47 1.00 1 up osd.27 >> 29 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 54 MiB 16 TiB 0.47 1.00 1 up osd.29 >> 31 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 54 MiB 16 TiB 0.47 1.00 4 up osd.31 >> 33 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 1 KiB >> 43 MiB 16 TiB 0.47 1.00 2 up osd.33 >> 35 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 1 KiB >> 42 MiB 16 TiB 0.47 1.00 2 up osd.35 >> -5 197.38440 - 197 TiB 955 GiB 710 MiB 0 B >> 994 MiB 196 TiB 0.47 1.00 - host store-par2-node03 >> 12 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 54 MiB 16 TiB 0.47 1.00 0 up osd.12 >> 14 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 54 MiB 16 TiB 0.47 1.00 2 up osd.14 >> 16 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 54 MiB 16 TiB 0.47 1.00 0 up osd.16 >> 18 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 54 MiB 16 TiB 0.47 1.00 3 up osd.18 >> 20 hdd 16.44870 1.00000 16 TiB 80 GiB 71 MiB 0 B >> 400 MiB 16 TiB 0.47 1.00 4 up osd.20 >> 22 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 54 MiB 16 TiB 0.47 1.00 0 up osd.22 >> 24 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 54 MiB 16 TiB 0.47 1.00 3 up osd.24 >> 26 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 54 MiB 16 TiB 0.47 1.00 2 up osd.26 >> 28 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 54 MiB 16 TiB 0.47 1.00 4 up osd.28 >> 30 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 54 MiB 16 TiB 0.47 1.00 3 up osd.30 >> 32 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 54 MiB 16 TiB 0.47 1.00 1 up osd.32 >> 34 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 54 MiB 16 TiB 0.47 1.00 4 up osd.34 >> -29 197.38440 - 197 TiB 955 GiB 710 MiB 0 B >> 978 MiB 196 TiB 0.47 1.00 - host store-par2-node04 >> 38 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 2 up osd.38 >> 53 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 1 up osd.53 >> 65 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 2 up osd.65 >> 78 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 0 up osd.78 >> 89 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 0 up osd.89 >> 101 hdd 16.44870 1.00000 16 TiB 80 GiB 71 MiB 0 B >> 395 MiB 16 TiB 0.47 1.00 3 up osd.101 >> 112 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 4 up osd.112 >> 123 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 0 up osd.123 >> 134 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 3 up osd.134 >> 145 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 3 up osd.145 >> 156 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 2 up osd.156 >> 167 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 2 up osd.167 >> -11 197.38440 - 197 TiB 955 GiB 696 MiB 0 B >> 636 MiB 196 TiB 0.47 1.00 - host store-par2-node05 >> 46 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 2 up osd.46 >> 51 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 3 up osd.51 >> 60 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 3 up osd.60 >> 70 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 1 up osd.70 >> 81 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 3 up osd.81 >> 92 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 2 up osd.92 >> 103 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 1 up osd.103 >> 114 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 2 up osd.114 >> 125 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 0 up osd.125 >> 136 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 1 up osd.136 >> 147 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 5 up osd.147 >> 158 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 3 up osd.158 >> -13 197.38440 - 197 TiB 955 GiB 696 MiB 0 B >> 636 MiB 196 TiB 0.47 1.00 - host store-par2-node06 >> 36 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 2 up osd.36 >> 47 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 1 up osd.47 >> 59 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 1 up osd.59 >> 69 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 1 up osd.69 >> 82 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 2 up osd.82 >> 94 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 1 up osd.94 >> 106 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 2 up osd.106 >> 115 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 1 up osd.115 >> 128 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 2 up osd.128 >> 137 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 1 up osd.137 >> 148 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 2 up osd.148 >> 159 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 3 up osd.159 >> -23 197.38440 - 197 TiB 955 GiB 696 MiB 0 B >> 636 MiB 196 TiB 0.47 1.00 - host store-par2-node07 >> 41 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 3 up osd.41 >> 48 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 2 up osd.48 >> 58 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 1 up osd.58 >> 71 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 3 up osd.71 >> 83 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 4 up osd.83 >> 93 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 3 up osd.93 >> 105 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 2 up osd.105 >> 117 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 1 up osd.117 >> 129 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 3 up osd.129 >> 138 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 0 up osd.138 >> 150 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 5 up osd.150 >> 163 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 2 up osd.163 >> -17 197.38440 - 197 TiB 955 GiB 696 MiB 0 B >> 636 MiB 196 TiB 0.47 1.00 - host store-par2-node08 >> 43 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 2 up osd.43 >> 56 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 0 up osd.56 >> 68 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 1 up osd.68 >> 79 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 0 up osd.79 >> 90 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 1 up osd.90 >> 100 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 0 up osd.100 >> 111 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 0 up osd.111 >> 121 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 2 up osd.121 >> 132 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 7 up osd.132 >> 143 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 4 up osd.143 >> 154 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 2 up osd.154 >> 165 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 0 up osd.165 >> -27 197.38440 - 197 TiB 955 GiB 696 MiB 0 B >> 636 MiB 196 TiB 0.47 1.00 - host store-par2-node09 >> 37 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 4 up osd.37 >> 49 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 3 up osd.49 >> 63 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 3 up osd.63 >> 77 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 1 up osd.77 >> 88 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 2 up osd.88 >> 98 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 3 up osd.98 >> 110 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 4 up osd.110 >> 120 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 1 up osd.120 >> 131 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 4 up osd.131 >> 141 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 1 up osd.141 >> 152 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 2 up osd.152 >> 164 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 2 up osd.164 >> -9 197.38440 - 197 TiB 955 GiB 710 MiB 0 B >> 982 MiB 196 TiB 0.47 1.00 - host store-par2-node10 >> 45 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 4 up osd.45 >> 54 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 3 up osd.54 >> 64 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 1 up osd.64 >> 74 hdd 16.44870 1.00000 16 TiB 80 GiB 71 MiB 0 B >> 399 MiB 16 TiB 0.47 1.00 5 up osd.74 >> 87 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 0 up osd.87 >> 97 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 2 up osd.97 >> 107 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 4 up osd.107 >> 116 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 1 up osd.116 >> 127 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 2 up osd.127 >> 142 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 3 up osd.142 >> 153 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 1 up osd.153 >> 161 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 1 up osd.161 >> -15 197.38440 - 197 TiB 955 GiB 696 MiB 0 B >> 637 MiB 196 TiB 0.47 1.00 - host store-par2-node11 >> 40 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 54 MiB 16 TiB 0.47 1.00 6 up osd.40 >> 50 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 2 up osd.50 >> 61 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 2 up osd.61 >> 73 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 4 up osd.73 >> 84 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 1 up osd.84 >> 95 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 1 up osd.95 >> 104 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 5 up osd.104 >> 118 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 2 up osd.118 >> 126 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 1 up osd.126 >> 139 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 1 up osd.139 >> 149 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 3 up osd.149 >> 160 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 2 up osd.160 >> -25 197.38440 - 197 TiB 955 GiB 696 MiB 0 B >> 636 MiB 196 TiB 0.47 1.00 - host store-par2-node12 >> 39 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 1 up osd.39 >> 52 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 2 up osd.52 >> 66 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 2 up osd.66 >> 76 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 3 up osd.76 >> 86 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 3 up osd.86 >> 99 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 3 up osd.99 >> 109 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 2 up osd.109 >> 122 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 3 up osd.122 >> 133 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 2 up osd.133 >> 144 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 3 up osd.144 >> 155 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 0 up osd.155 >> 166 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 2 up osd.166 >> -19 197.38440 - 197 TiB 955 GiB 696 MiB 0 B >> 636 MiB 196 TiB 0.47 1.00 - host store-par2-node13 >> 42 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 1 up osd.42 >> 55 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 1 up osd.55 >> 62 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 0 up osd.62 >> 72 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 1 up osd.72 >> 80 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 3 up osd.80 >> 91 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 5 up osd.91 >> 102 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 1 up osd.102 >> 113 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 2 up osd.113 >> 124 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 1 up osd.124 >> 135 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 4 up osd.135 >> 146 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 3 up osd.146 >> 157 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 1 up osd.157 >> -21 197.38440 - 197 TiB 955 GiB 696 MiB 0 B >> 636 MiB 196 TiB 0.47 1.00 - host store-par2-node14 >> 44 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 2 up osd.44 >> 57 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 1 up osd.57 >> 67 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 4 up osd.67 >> 75 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 1 up osd.75 >> 85 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 3 up osd.85 >> 96 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 3 up osd.96 >> 108 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 1 up osd.108 >> 119 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 1 up osd.119 >> 130 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 3 up osd.130 >> 140 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 0 up osd.140 >> 151 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 2 up osd.151 >> 162 hdd 16.44870 1.00000 16 TiB 80 GiB 58 MiB 0 B >> 53 MiB 16 TiB 0.47 1.00 2 up osd.162 >> TOTAL 2.7 PiB 13 TiB 9.6 GiB 9.6 KiB >> 9.7 GiB 2.7 PiB 0.47 >> MIN/MAX VAR: 1.00/1.00 STDDEV: 0 >> >> >> root@store-par2-node01:/home/user# ceph pg dump >> version 140443 >> stamp 2022-06-06T12:27:35.382577+0000 >> last_osdmap_epoch 0 >> last_pg_scan 0 >> PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND >> BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE >> STATE_STAMP VERSION REPORTED UP >> UP_PRIMARY ACTING >> ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP >> LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN >> LAST_SCRUB_DURATION SCRUB_SCHEDULING >> OBJECTS_SCRUBBED OBJECTS_TRIMMED >> 3.f 1 0 0 0 0 >> 22 0 0 2 2 active+clean >> 2022-06-06T02:22:55.899870+0000 773'2 1191:813 >> [112,132,142] 112 [112,132,142] >> 112 773'2 2022-06-06T02:22:55.899728+0000 >> 773'2 2022-06-03T11:08:04.230774+0000 0 >> 1 periodic scrub scheduled @ 2022-06-07T02:26:26.925460+0000 >> 1 0 >> 2.e 0 0 0 0 0 >> 0 0 0 0 0 active+clean >> 2022-06-05T18:09:54.859041+0000 0'0 1191:818 >> [8,3,44,13,64,67,132,31] 8 >> [8,3,44,13,64,67,132,31] 8 0'0 >> 2022-06-05T18:09:54.858891+0000 0'0 >> 2022-05-31T19:51:41.372150+0000 0 1 >> periodic scrub scheduled @ 2022-06-07T02:48:10.931317+0000 >> 0 0 >> 3.c 0 0 0 0 0 >> 0 0 0 0 0 active+clean >> 2022-06-05T22:21:13.363161+0000 0'0 1191:813 >> [24,8,52] 24 [24,8,52] >> 24 0'0 2022-06-05T22:21:13.363041+0000 >> 0'0 2022-05-31T19:51:59.873880+0000 0 >> 1 periodic scrub scheduled @ 2022-06-07T05:12:34.695987+0000 >> 0 0 >> 2.d 0 0 0 0 0 >> 0 0 0 0 0 active+clean >> 2022-06-05T21:38:59.942761+0000 0'0 1191:819 >> [51,134,3,86,96,38,1,41] 51 >> [51,134,3,86,96,38,1,41] 51 0'0 >> 2022-06-05T21:38:59.942666+0000 0'0 >> 2022-05-31T19:51:41.372150+0000 0 1 >> periodic scrub scheduled @ 2022-06-06T22:34:01.326089+0000 >> 0 0 >> 3.d 1 0 0 0 0 >> 70 0 0 2 2 active+clean >> 2022-06-06T06:48:17.160609+0000 771'2 1191:816 >> [122,71,132] 122 [122,71,132] >> 122 771'2 2022-06-06T06:48:17.160469+0000 >> 771'2 2022-06-03T10:46:08.465824+0000 0 >> 1 periodic scrub scheduled @ 2022-06-07T10:22:35.365062+0000 >> 1 0 >> 2.c 0 0 0 0 0 >> 0 0 0 0 0 active+clean >> 2022-06-06T04:17:49.079390+0000 0'0 1191:819 >> [149,19,48,11,158,60,143,110] 149 >> [149,19,48,11,158,60,143,110] 149 0'0 >> 2022-06-06T04:17:49.079269+0000 0'0 >> 2022-05-31T19:51:41.372150+0000 0 1 >> periodic scrub scheduled @ 2022-06-07T06:16:29.279337+0000 >> 0 0 >> 3.a 0 0 0 0 0 >> 0 0 0 0 0 active+clean >> 2022-06-06T02:51:41.932102+0000 0'0 1191:813 >> [18,148,131] 18 [18,148,131] >> 18 0'0 2022-06-06T02:51:41.931974+0000 >> 0'0 2022-06-04T15:55:16.683239+0000 0 >> 1 periodic scrub scheduled @ 2022-06-07T10:39:23.180043+0000 >> 0 0 >> 2.b 0 0 0 0 0 >> 0 0 0 0 0 active+clean >> 2022-06-06T10:08:33.237436+0000 0'0 1191:818 >> [46,159,28,74,137,37,91,127] 46 >> [46,159,28,74,137,37,91,127] 46 0'0 >> 2022-06-06T10:08:33.237340+0000 0'0 >> 2022-06-06T10:08:33.237340+0000 0 1 >> periodic scrub scheduled @ 2022-06-07T10:13:44.177114+0000 >> 0 0 >> 3.b 1 0 0 0 0 >> 0 0 0 3 3 active+clean >> 2022-06-06T00:54:52.451573+0000 671'3 1191:814 >> [73,159,147] 73 [73,159,147] >> 73 671'3 2022-06-06T00:54:52.451459+0000 >> 671'3 2022-06-03T03:41:08.026476+0000 0 >> 1 periodic scrub scheduled @ 2022-06-07T12:45:10.753628+0000 >> 1 0 >> 2.a 0 0 0 0 0 >> 0 0 0 0 0 active+clean >> 2022-06-05T22:53:09.039828+0000 0'0 1191:815 >> [44,86,132,94,71,81,80,166] 44 >> [44,86,132,94,71,81,80,166] 44 0'0 >> 2022-06-05T22:53:09.039734+0000 0'0 >> 2022-06-03T12:58:37.531167+0000 0 1 >> periodic scrub scheduled @ 2022-06-07T03:20:37.899744+0000 >> 0 0 >> 3.8 0 0 0 0 0 >> 0 0 0 0 0 active+clean >> 2022-06-05T14:40:36.607866+0000 0'0 1191:811 >> [115,38,68] 115 [115,38,68] >> 115 0'0 2022-06-05T14:40:36.607760+0000 >> 0'0 2022-05-31T19:51:59.873880+0000 0 >> 1 periodic scrub scheduled @ 2022-06-06T14:45:02.993139+0000 >> 0 0 >> 2.9 0 0 0 0 0 >> 0 0 0 0 0 active+clean >> 2022-06-06T03:20:19.920382+0000 0'0 1191:817 >> [104,84,110,40,82,73,147,17] 104 >> [104,84,110,40,82,73,147,17] 104 0'0 >> 2022-06-06T03:20:19.920257+0000 0'0 >> 2022-05-31T19:51:41.372150+0000 0 1 >> periodic scrub scheduled @ 2022-06-07T06:24:06.295277+0000 >> 0 0 >> 3.9 0 0 0 0 0 >> 0 0 0 0 0 active+clean >> 2022-06-05T22:14:34.542404+0000 0'0 1191:811 >> [59,45,129] 59 [59,45,129] >> 59 0'0 2022-06-05T22:14:34.542285+0000 >> 0'0 2022-06-03T00:14:09.631316+0000 0 >> 1 periodic scrub scheduled @ 2022-06-07T06:57:37.776582+0000 >> 0 0 >> 2.8 0 0 0 0 0 >> 0 0 0 0 0 active+clean >> 2022-06-05T22:12:11.172949+0000 0'0 1191:815 >> [45,101,104,13,143,157,63,61] 45 >> [45,101,104,13,143,157,63,61] 45 0'0 >> 2022-06-05T22:12:11.172853+0000 0'0 >> 2022-05-31T19:51:41.372150+0000 0 1 >> periodic scrub scheduled @ 2022-06-07T06:52:53.792844+0000 >> 0 0 >> 3.6 1 0 0 0 0 >> 0 0 0 5 5 active+clean >> 2022-06-05T10:01:53.243050+0000 1043'5 1191:818 >> [98,66,132] 98 [98,66,132] >> 98 1043'5 2022-06-05T10:01:53.242948+0000 >> 1043'5 2022-06-05T10:01:53.242948+0000 0 >> 1 periodic scrub scheduled @ 2022-06-06T18:00:59.024882+0000 >> 1 0 >> 2.7 0 0 0 0 0 >> 0 0 0 0 0 active+clean >> 2022-06-06T08:28:41.853562+0000 0'0 1191:818 >> [163,57,151,110,116,118,30,147] 163 >> [163,57,151,110,116,118,30,147] 163 0'0 >> 2022-06-06T08:28:41.853404+0000 0'0 >> 2022-05-31T19:51:41.372150+0000 0 1 >> periodic scrub scheduled @ 2022-06-07T18:31:27.727201+0000 >> 0 0 >> 3.7 1 0 0 0 0 >> 0 0 0 2 2 active+clean >> 2022-06-06T11:35:38.186470+0000 437'2 1191:826 >> [95,31,55] 95 [95,31,55] >> 95 437'2 2022-06-06T11:35:38.186365+0000 >> 0'0 2022-05-31T19:51:59.873880+0000 0 >> 1 periodic scrub scheduled @ 2022-06-07T12:34:45.361534+0000 >> 1 0 >> 2.6 0 0 0 0 0 >> 0 0 0 0 0 active+clean >> 2022-06-06T06:13:11.860365+0000 0'0 1191:817 >> [135,112,19,98,114,88,119,8] 135 >> [135,112,19,98,114,88,119,8] 135 0'0 >> 2022-06-06T06:13:11.860264+0000 0'0 >> 2022-06-06T06:13:11.860264+0000 0 1 >> periodic scrub scheduled @ 2022-06-07T08:33:30.327642+0000 >> 0 0 >> 3.4 0 0 0 0 0 >> 0 0 0 0 0 active+clean >> 2022-06-05T11:53:13.051840+0000 0'0 1191:813 >> [10,13,131] 10 [10,13,131] >> 10 0'0 2022-06-05T11:53:13.051678+0000 >> 0'0 2022-06-02T02:18:09.375232+0000 0 >> 1 periodic scrub scheduled @ 2022-06-06T14:15:18.261225+0000 >> 0 0 >> 2.5 0 0 0 0 0 >> 0 0 0 0 0 active+clean >> 2022-06-05T15:15:43.099447+0000 0'0 1191:817 >> [15,4,8,144,81,96,73,10] 15 >> [15,4,8,144,81,96,73,10] 15 0'0 >> 2022-06-05T15:15:43.099314+0000 0'0 >> 2022-06-04T11:38:14.307983+0000 0 1 >> periodic scrub scheduled @ 2022-06-06T22:44:13.570864+0000 >> 0 0 >> 2.0 0 0 0 0 0 >> 0 0 0 0 0 active+clean >> 2022-06-05T23:46:06.362681+0000 0'0 1191:844 >> [30,109,13,40,133,136,37,65] 30 >> [30,109,13,40,133,136,37,65] 30 0'0 >> 2022-06-05T23:46:06.362567+0000 0'0 >> 2022-06-05T23:46:06.362567+0000 0 1 >> periodic scrub scheduled @ 2022-06-07T05:50:48.092572+0000 >> 0 0 >> 3.1 0 0 0 0 0 >> 0 0 0 0 0 active+clean >> 2022-06-05T16:32:59.830885+0000 0'0 1191:831 >> [73,150,54] 73 [73,150,54] >> 73 0'0 2022-06-05T16:32:59.830683+0000 >> 0'0 2022-06-05T16:32:59.830683+0000 0 >> 1 periodic scrub scheduled @ 2022-06-07T02:38:21.783899+0000 >> 0 0 >> 1.0 5 0 0 0 0 >> 14156320 0 0 10005 10005 active+clean >> 2022-06-05T16:52:10.117602+0000 1146'21705 1191:26731 >> [74,101,20] 74 [74,101,20] >> 74 1012'18422 2022-06-05T16:52:10.117484+0000 >> 0'0 2022-05-31T17:19:28.164530+0000 0 >> 1 periodic scrub scheduled @ 2022-06-06T23:19:35.924762+0000 >> 4 0 >> 2.3 0 0 0 0 0 >> 0 0 0 0 0 active+clean >> 2022-06-05T15:59:18.303912+0000 0'0 1191:819 >> [83,121,122,34,92,21,32,72] 83 >> [83,121,122,34,92,21,32,72] 83 0'0 >> 2022-06-05T15:59:18.303794+0000 0'0 >> 2022-05-31T19:51:41.372150+0000 0 1 >> periodic scrub scheduled @ 2022-06-07T02:34:48.910045+0000 >> 0 0 >> 3.2 0 0 0 0 0 >> 0 0 0 0 0 active+clean >> 2022-06-06T03:07:55.854126+0000 0'0 1191:811 >> [164,31,34] 164 [164,31,34] >> 164 0'0 2022-06-06T03:07:55.854014+0000 >> 0'0 2022-05-31T19:51:59.873880+0000 0 >> 1 periodic scrub scheduled @ 2022-06-07T08:11:53.316960+0000 >> 0 0 >> 2.1 0 0 0 0 0 >> 0 0 0 0 0 active+clean >> 2022-06-05T13:03:19.274658+0000 0'0 1191:821 >> [40,76,99,128,43,135,10,96] 40 >> [40,76,99,128,43,135,10,96] 40 0'0 >> 2022-06-05T13:03:19.274545+0000 0'0 >> 2022-05-31T19:51:41.372150+0000 0 1 >> periodic scrub scheduled @ 2022-06-06T18:43:52.710554+0000 >> 0 0 >> 3.0 0 0 0 0 0 >> 0 0 0 0 0 active+clean >> 2022-06-05T13:33:06.917207+0000 0'0 1191:826 >> [35,40,70] 35 [35,40,70] >> 35 0'0 2022-06-05T13:33:06.917074+0000 >> 0'0 2022-05-31T19:52:08.234732+0000 0 >> 1 periodic scrub scheduled @ 2022-06-06T18:08:03.107087+0000 >> 0 0 >> 3.3 0 0 0 0 0 >> 0 0 0 0 0 active+clean >> 2022-06-06T11:43:56.320476+0000 0'0 1191:840 >> [40,18,11] 40 [40,18,11] >> 40 0'0 2022-06-06T11:43:56.320361+0000 >> 0'0 2022-06-04T06:13:59.535590+0000 0 >> 1 periodic scrub scheduled @ 2022-06-07T20:55:33.787766+0000 >> 0 0 >> 2.2 0 0 0 0 0 >> 0 0 0 0 0 active+clean >> 2022-06-05T16:42:20.204103+0000 0'0 1191:819 >> [71,150,24,82,144,48,161,91] 71 >> [71,150,24,82,144,48,161,91] 71 0'0 >> 2022-06-05T16:42:20.203967+0000 0'0 >> 2022-06-03T04:54:17.344478+0000 0 1 >> periodic scrub scheduled @ 2022-06-06T21:40:54.542796+0000 >> 0 0 >> 3.5 0 0 0 0 0 >> 0 0 0 0 0 active+clean >> 2022-06-05T20:06:50.152236+0000 0'0 1191:812 >> [152,3,93] 152 [152,3,93] >> 152 0'0 2022-06-05T20:06:50.152134+0000 >> 0'0 2022-06-03T08:27:56.053931+0000 0 >> 1 periodic scrub scheduled @ 2022-06-07T05:07:45.942271+0000 >> 0 0 >> 2.4 0 0 0 0 0 >> 0 0 0 0 0 active+clean >> 2022-06-05T23:44:27.330764+0000 0'0 1191:816 >> [11,132,21,7,104,149,49,134] 11 >> [11,132,21,7,104,149,49,134] 11 0'0 >> 2022-06-05T23:44:27.330642+0000 0'0 >> 2022-06-05T23:44:27.330642+0000 0 1 >> periodic scrub scheduled @ 2022-06-07T03:48:00.509825+0000 >> 0 0 >> 3.e 1 0 0 0 0 >> 0 0 0 3 3 active+clean >> 2022-06-05T20:45:57.476116+0000 543'3 1191:814 >> [37,97,26] 37 [37,97,26] >> 37 543'3 2022-06-05T20:45:57.476006+0000 >> 543'3 2022-06-02T00:50:17.270315+0000 0 >> 1 periodic scrub scheduled @ 2022-06-07T00:57:37.043277+0000 >> 1 0 >> 2.f 0 0 0 0 0 >> 0 0 0 0 0 active+clean >> 2022-06-06T10:18:51.955868+0000 0'0 1191:827 >> [92,85,101,53,25,24,113,145] 92 >> [92,85,101,53,25,24,113,145] 92 0'0 >> 2022-06-06T10:18:51.955714+0000 0'0 >> 2022-06-05T06:18:02.624483+0000 0 1 >> periodic scrub scheduled @ 2022-06-07T12:10:14.007061+0000 >> 0 0 >> 3.11 0 0 0 0 0 >> 0 0 0 0 0 active+clean >> 2022-06-05T20:15:43.246605+0000 0'0 1191:811 >> [33,30,41] 33 [33,30,41] >> 33 0'0 2022-06-05T20:15:43.246445+0000 >> 0'0 2022-05-31T19:51:59.873880+0000 0 >> 1 periodic scrub scheduled @ 2022-06-07T01:59:04.536885+0000 >> 0 0 >> 2.10 0 0 0 0 0 >> 0 0 0 0 0 active+clean >> 2022-06-06T03:41:55.049114+0000 0'0 1191:819 >> [130,167,43,143,66,65,45,145] 130 >> [130,167,43,143,66,65,45,145] 130 0'0 >> 2022-06-06T03:41:55.048887+0000 0'0 >> 2022-06-06T03:41:55.048887+0000 0 1 >> periodic scrub scheduled @ 2022-06-07T12:59:20.841601+0000 >> 0 0 >> 3.10 4 0 0 0 0 >> 0 0 0 12 12 active+clean >> 2022-06-05T06:44:51.783819+0000 845'12 1191:816 >> [130,103,93] 130 [130,103,93] >> 130 845'12 2022-06-05T06:44:51.783710+0000 >> 845'12 2022-06-03T19:13:59.475041+0000 0 >> 1 periodic scrub scheduled @ 2022-06-06T16:31:37.232782+0000 >> 4 0 >> 2.11 0 0 0 0 0 >> 0 0 0 0 0 active+clean >> 2022-06-06T08:38:43.779419+0000 0'0 1191:820 >> [107,20,129,122,27,113,25,83] 107 >> [107,20,129,122,27,113,25,83] 107 0'0 >> 2022-06-06T08:38:43.779269+0000 0'0 >> 2022-06-03T11:14:02.841282+0000 0 1 >> periodic scrub scheduled @ 2022-06-07T16:53:01.369556+0000 >> 0 0 >> 3.13 2 0 0 0 0 >> 34 0 0 2 2 active+clean >> 2022-06-05T15:00:46.595842+0000 437'2 1191:816 >> [149,14,85] 149 [149,14,85] >> 149 437'2 2022-06-05T15:00:46.595730+0000 >> 0'0 2022-05-31T19:51:59.873880+0000 0 >> 1 periodic scrub scheduled @ 2022-06-06T16:27:45.919806+0000 >> 2 0 >> 2.12 0 0 0 0 0 >> 0 0 0 0 0 active+clean >> 2022-06-06T07:10:51.207987+0000 0'0 1191:815 >> [35,139,147,106,60,28,126,127] 35 >> [35,139,147,106,60,28,126,127] 35 0'0 >> 2022-06-06T07:10:51.207861+0000 0'0 >> 2022-05-31T19:51:41.372150+0000 0 1 >> periodic scrub scheduled @ 2022-06-07T17:59:37.937977+0000 >> 0 0 >> 3.12 0 0 0 0 0 >> 0 0 0 0 0 active+clean >> 2022-06-05T09:38:30.626384+0000 0'0 1191:812 >> [135,154,131] 135 [135,154,131] >> 135 0'0 2022-06-05T09:38:30.626285+0000 >> 0'0 2022-06-02T22:21:59.376573+0000 0 >> 1 periodic scrub scheduled @ 2022-06-06T18:43:10.579197+0000 >> 0 0 >> 2.13 0 0 0 0 0 >> 0 0 0 0 0 active+clean >> 2022-06-06T01:39:10.599882+0000 0'0 1191:818 >> [74,36,3,158,39,77,75,98] 74 >> [74,36,3,158,39,77,75,98] 74 0'0 >> 2022-06-06T01:39:10.599742+0000 0'0 >> 2022-05-31T19:51:41.372150+0000 0 1 >> periodic scrub scheduled @ 2022-06-07T07:50:19.048812+0000 >> 0 0 >> 3.15 2 0 0 0 0 >> 90 0 0 3 3 active+clean >> 2022-06-06T00:06:19.711529+0000 437'3 1191:817 >> [54,15,102] 54 [54,15,102] >> 54 437'3 2022-06-06T00:06:19.711396+0000 >> 0'0 2022-05-31T19:51:59.873880+0000 0 >> 1 periodic scrub scheduled @ 2022-06-07T09:05:26.475289+0000 >> 2 0 >> 2.14 0 0 0 0 0 >> 0 0 0 0 0 active+clean >> 2022-06-05T13:23:51.227730+0000 0'0 1191:815 >> [63,18,104,31,28,80,107,47] 63 >> [63,18,104,31,28,80,107,47] 63 0'0 >> 2022-06-05T13:23:51.227619+0000 0'0 >> 2022-05-31T19:51:41.372150+0000 0 1 >> periodic scrub scheduled @ 2022-06-07T00:57:54.831856+0000 >> 0 0 >> 3.14 4 0 0 0 0 >> 4972 0 0 37 37 active+clean >> 2022-06-05T18:43:38.650054+0000 718'37 1191:851 >> [83,114,153] 83 [83,114,153] >> 83 718'37 2022-06-05T18:43:38.649890+0000 >> 718'37 2022-06-05T18:43:38.649890+0000 0 >> 1 periodic scrub scheduled @ 2022-06-07T01:34:37.932141+0000 >> 4 0 >> 2.15 0 0 0 0 0 >> 0 0 0 0 0 active+clean >> 2022-06-05T16:15:50.881954+0000 0'0 1191:819 >> [54,88,142,33,76,20,146,150] 54 >> [54,88,142,33,76,20,146,150] 54 0'0 >> 2022-06-05T16:15:50.881848+0000 0'0 >> 2022-06-01T21:50:03.827583+0000 0 1 >> periodic scrub scheduled @ 2022-06-06T16:24:10.967689+0000 >> 0 0 >> 3.17 0 0 0 0 0 >> 0 0 0 0 0 active+clean >> 2022-06-05T16:14:32.120728+0000 0'0 1191:814 >> [147,34,167] 147 [147,34,167] >> 147 0'0 2022-06-05T16:14:32.120636+0000 >> 0'0 2022-05-31T19:51:59.873880+0000 0 >> 1 periodic scrub scheduled @ 2022-06-07T01:17:17.942645+0000 >> 0 0 >> 2.16 0 0 0 0 0 >> 0 0 0 0 0 active+clean >> 2022-06-05T13:32:29.676884+0000 0'0 1191:815 >> [162,152,23,74,160,9,121,10] 162 >> [162,152,23,74,160,9,121,10] 162 0'0 >> 2022-06-05T13:32:29.676755+0000 0'0 >> 2022-06-01T23:18:50.309093+0000 0 1 >> periodic scrub scheduled @ 2022-06-06T15:29:19.493434+0000 >> 0 0 >> 3.16 0 0 0 0 0 >> 0 0 0 0 0 active+clean >> 2022-06-06T03:05:19.005785+0000 0'0 1191:813 >> [26,162,105] 26 [26,162,105] >> 26 0'0 2022-06-06T03:05:19.005679+0000 >> 0'0 2022-05-31T19:52:13.903980+0000 0 >> 1 periodic scrub scheduled @ 2022-06-07T05:38:41.233380+0000 >> 0 0 >> 2.17 0 0 0 0 0 >> 0 0 0 0 0 active+clean >> 2022-06-05T16:25:32.999773+0000 0'0 1191:821 >> [146,104,158,163,108,106,141,159] 146 >> [146,104,158,163,108,106,141,159] 146 0'0 >> 2022-06-05T16:25:32.999664+0000 0'0 >> 2022-05-31T19:51:41.372150+0000 0 1 >> periodic scrub scheduled @ 2022-06-07T03:44:47.468570+0000 >> 0 0 >> 3.19 0 0 0 0 0 >> 0 0 0 0 0 active+clean >> 2022-06-05T18:18:12.164305+0000 0'0 1191:810 >> [156,40,109] 156 [156,40,109] >> 156 0'0 2022-06-05T18:18:12.164187+0000 >> 0'0 2022-05-31T19:51:59.873880+0000 0 >> 1 periodic scrub scheduled @ 2022-06-06T20:18:30.481546+0000 >> 0 0 >> 2.18 0 0 0 0 0 >> 0 0 0 0 0 active+clean >> 2022-06-05T18:42:00.227559+0000 0'0 1191:820 >> [107,105,97,17,99,124,154,132] 107 >> [107,105,97,17,99,124,154,132] 107 0'0 >> 2022-06-05T18:42:00.227444+0000 0'0 >> 2022-05-31T19:51:41.372150+0000 0 1 >> periodic scrub scheduled @ 2022-06-07T00:48:30.997533+0000 >> 0 0 >> 3.18 0 0 0 0 0 >> 0 0 0 0 0 active+clean >> 2022-06-05T22:29:35.968669+0000 0'0 1191:813 >> [150,21,36] 150 [150,21,36] >> 150 0'0 2022-06-05T22:29:35.968576+0000 >> 0'0 2022-06-01T20:07:31.626680+0000 0 >> 1 periodic scrub scheduled @ 2022-06-07T08:00:54.614246+0000 >> 0 0 >> 2.19 0 0 0 0 0 >> 0 0 0 0 0 active+clean >> 2022-06-06T12:25:06.541967+0000 0'0 1192:820 >> [131,151,63,13,146,90,135,133] 131 >> [131,151,63,13,146,90,135,133] 131 0'0 >> 2022-06-06T12:25:06.541855+0000 0'0 >> 2022-05-31T19:51:41.372150+0000 0 1 >> periodic scrub scheduled @ 2022-06-07T16:49:45.723687+0000 >> 0 0 >> 3.1b 0 0 0 0 0 >> 0 0 0 1 1 active+clean >> 2022-06-05T12:05:36.929834+0000 437'1 1191:813 >> [91,120,142] 91 [91,120,142] >> 91 437'1 2022-06-05T12:05:36.929706+0000 >> 437'1 2022-06-02T00:32:32.667254+0000 0 >> 1 periodic scrub scheduled @ 2022-06-06T20:33:52.594948+0000 >> 0 0 >> 2.1a 0 0 0 0 0 >> 0 0 0 0 0 active+clean >> 2022-06-05T21:33:42.799425+0000 0'0 1191:817 >> [58,112,85,50,129,148,49,45] 58 >> [58,112,85,50,129,148,49,45] 58 0'0 >> 2022-06-05T21:33:42.799305+0000 0'0 >> 2022-05-31T19:51:41.372150+0000 0 1 >> periodic scrub scheduled @ 2022-06-07T08:52:06.385325+0000 >> 0 0 >> 3.1a 0 0 0 0 0 >> 0 0 0 1 1 active+clean >> 2022-06-05T14:57:02.326186+0000 437'1 1191:815 >> [25,9,93] 25 [25,9,93] >> 25 437'1 2022-06-05T14:57:02.326068+0000 >> 0'0 2022-05-31T19:51:59.873880+0000 0 >> 1 periodic scrub scheduled @ 2022-06-06T22:00:49.835756+0000 >> 0 0 >> 2.1b 0 0 0 0 0 >> 0 0 0 0 0 active+clean >> 2022-06-06T00:21:22.013136+0000 0'0 1191:819 >> [51,42,60,83,164,145,52,134] 51 >> [51,42,60,83,164,145,52,134] 51 0'0 >> 2022-06-06T00:21:22.013028+0000 0'0 >> 2022-06-06T00:21:22.013028+0000 0 1 >> periodic scrub scheduled @ 2022-06-07T08:38:12.431836+0000 >> 0 0 >> 3.1d 1 0 0 0 0 >> 0 0 0 2 2 active+clean >> 2022-06-06T00:17:33.395234+0000 437'2 1191:815 >> [150,67,20] 150 [150,67,20] >> 150 437'2 2022-06-06T00:17:33.395098+0000 >> 0'0 2022-05-31T19:51:59.873880+0000 0 >> 1 periodic scrub scheduled @ 2022-06-07T05:38:06.038546+0000 >> 1 0 >> 2.1c 0 0 0 0 0 >> 0 0 0 0 0 active+clean >> 2022-06-05T21:38:58.966683+0000 0'0 1191:819 >> [51,34,80,144,143,99,50,0] 51 >> [51,34,80,144,143,99,50,0] 51 0'0 >> 2022-06-05T21:38:58.966531+0000 0'0 >> 2022-06-03T07:43:07.143846+0000 0 1 >> periodic scrub scheduled @ 2022-06-07T04:01:52.286592+0000 >> 0 0 >> 3.1c 1 0 0 0 0 >> 0 0 0 3 3 active+clean >> 2022-06-06T05:23:02.301909+0000 724'3 1191:815 >> [76,156,128] 76 [76,156,128] >> 76 724'3 2022-06-06T05:23:02.301761+0000 >> 724'3 2022-06-03T07:49:44.813182+0000 0 >> 1 periodic scrub scheduled @ 2022-06-07T11:37:54.326001+0000 >> 1 0 >> 2.1d 0 0 0 0 0 >> 0 0 0 0 0 active+clean >> 2022-06-06T03:14:29.829206+0000 0'0 1191:816 >> [67,86,61,81,130,69,74,46] 67 >> [67,86,61,81,130,69,74,46] 67 0'0 >> 2022-06-06T03:14:29.829088+0000 0'0 >> 2022-06-03T06:14:14.581690+0000 0 1 >> periodic scrub scheduled @ 2022-06-07T12:37:45.026826+0000 >> 0 0 >> 3.1f 2 0 0 0 0 >> 554 0 0 4 4 active+clean >> 2022-06-05T13:34:03.534647+0000 437'4 1191:815 >> [160,112,49] 160 [160,112,49] >> 160 437'4 2022-06-05T13:34:03.534531+0000 >> 0'0 2022-05-31T19:51:59.873880+0000 0 >> 1 periodic scrub scheduled @ 2022-06-06T15:30:53.351210+0000 >> 2 0 >> 2.1e 0 0 0 0 0 >> 0 0 0 0 0 active+clean >> 2022-06-05T16:05:18.546234+0000 0'0 1191:817 >> [5,0,91,118,2,107,41,37] 5 >> [5,0,91,118,2,107,41,37] 5 0'0 >> 2022-06-05T16:05:18.546142+0000 0'0 >> 2022-05-31T19:51:41.372150+0000 0 1 >> periodic scrub scheduled @ 2022-06-06T23:22:10.205148+0000 >> 0 0 >> 3.1e 0 0 0 0 0 >> 0 0 0 1 1 active+clean >> 2022-06-06T02:05:23.255525+0000 437'1 1191:814 >> [3,166,110] 3 [3,166,110] >> 3 437'1 2022-06-06T02:05:23.255430+0000 >> 0'0 2022-05-31T19:51:59.873880+0000 0 >> 1 periodic scrub scheduled @ 2022-06-07T12:43:16.169680+0000 >> 0 0 >> 2.1f 0 0 0 0 0 >> 0 0 0 0 0 active+clean >> 2022-06-05T14:19:38.386679+0000 0'0 1191:816 >> [67,14,117,29,28,4,91,17] 67 >> [67,14,117,29,28,4,91,17] 67 0'0 >> 2022-06-05T14:19:38.386564+0000 0'0 >> 2022-06-03T08:33:01.050897+0000 0 1 >> periodic scrub scheduled @ 2022-06-06T23:02:03.102683+0000 >> 0 0 >> >> 3 22 0 0 0 0 5742 0 0 83 83 >> 1 5 0 0 0 0 14156320 0 0 10005 10005 >> 2 0 0 0 0 0 0 0 0 0 0 >> >> sum 27 0 0 0 0 14162062 0 0 10088 10088 >> OSD_STAT USED AVAIL USED_RAW TOTAL HB_PEERS >> >> PG_SUM PRIMARY_PG_SUM >> 167 80 GiB 16 TiB 80 GiB 16 TiB >> >> [0,11,34,35,90,91,92,93,94,95,96,97,98,133,166] 2 >> 166 80 GiB 16 TiB 80 GiB 16 TiB >> >> [9,32,33,80,81,82,83,84,85,87,88,111,112,165,167] 2 >> 0 >> 77 80 GiB 16 TiB 80 GiB 16 TiB >> >> [3,22,23,76,78,101,109,154,157,158,159,160,161,162,163] 1 >> 0 >> 76 80 GiB 16 TiB 80 GiB 16 TiB >> >> [2,20,21,75,77,96,98,128,143,145,146,147,148,149,150,153,156] >> 3 1 >> 75 80 GiB 16 TiB 80 GiB 16 TiB >> >> [11,18,19,74,76,86,87,132,134,135,136,137,138,139,141] 1 >> 0 >> 74 80 GiB 16 TiB 80 GiB 16 TiB >> >> [3,10,16,17,20,36,39,61,67,73,75,77,98,101,121,123,124,125,128,129,131,133,158] >> 5 >> 2 >> 73 80 GiB 16 TiB 80 GiB 16 TiB >> >> [1,14,15,54,55,72,74,111,112,114,115,117,119,120,122,147,150,159] >> 4 2 >> 72 80 GiB 16 TiB 80 GiB 16 TiB >> >> [0,12,13,40,41,71,73,100,101,103,106,107,108,109,110] 1 >> 0 >> 71 80 GiB 16 TiB 80 GiB 16 TiB >> >> [8,24,34,35,48,70,72,82,89,90,91,94,95,96,97,98,99,144,150,157,158,161] >> 3 1 >> 70 80 GiB 16 TiB 80 GiB 16 TiB >> >> [9,32,33,69,71,78,79,80,84,85,86,87,88,148,150] 1 >> 69 80 GiB 16 TiB 80 GiB 16 TiB >> >> [7,30,31,65,68,70,71,72,73,74,75,76,77,132,136] 1 >> 68 80 GiB 16 TiB 80 GiB 16 TiB >> >> [6,28,29,53,58,60,61,62,63,64,66,67,69,128,130] 1 >> 67 80 GiB 16 TiB 80 GiB 16 TiB >> [4,5,14,17,26,27,28,29,38,46,47,48,49,50,51,54,55,61,66,68,69,74,81,86,91,111,117,122,130] >> 4 >> 2 >> 66 80 GiB 16 TiB 80 GiB 16 TiB >> >> [4,24,25,36,37,40,41,42,45,46,65,67,101,108,165] 2 >> 0 >> 65 80 GiB 16 TiB 80 GiB 16 TiB >> >> [3,22,23,64,66,97,99,154,157,158,159,160,162,163,164] 2 >> 0 >> 64 80 GiB 16 TiB 80 GiB 16 TiB >> >> [2,20,21,63,65,78,88,143,146,147,148,149,150,151,155] 1 >> 0 >> 63 80 GiB 16 TiB 80 GiB 16 TiB >> >> [11,18,19,28,31,47,62,64,72,74,80,104,107,132,134,136,137,138,139,140,144] >> 3 1 >> 62 80 GiB 16 TiB 80 GiB 16 TiB >> >> [10,16,17,49,50,61,63,121,123,125,127,128,129,130,133] 0 >> 0 >> 61 80 GiB 16 TiB 80 GiB 16 TiB >> >> [1,14,15,42,46,60,62,111,112,115,116,117,119,120,122] 2 >> 0 >> 60 80 GiB 16 TiB 80 GiB 16 TiB >> >> [0,12,13,59,61,100,101,102,105,107,108,109,110,159,160] 3 >> 0 >> 59 80 GiB 16 TiB 80 GiB 16 TiB >> >> [8,34,35,45,58,60,89,90,91,95,96,97,98,99,129,147,150] 1 >> 1 >> 58 80 GiB 16 TiB 80 GiB 16 TiB >> >> [9,32,33,45,49,50,57,59,78,79,80,81,84,85,86,87,88,112,129,137,140,148] >> 1 1 >> 57 80 GiB 16 TiB 80 GiB 16 TiB >> >> [7,30,31,56,58,65,69,70,72,73,74,76,77,121,129] 1 >> 56 80 GiB 16 TiB 80 GiB 16 TiB >> >> [6,28,29,53,55,57,58,59,60,61,63,64,66,113,119] 0 >> 55 80 GiB 16 TiB 80 GiB 16 TiB >> >> [5,26,27,38,47,48,49,50,51,52,54,56,57,100,107] 1 >> 54 80 GiB 16 TiB 80 GiB 16 TiB >> >> [4,15,20,24,25,33,36,37,39,40,41,44,46,53,55,76,88,89,91,102,142,146,150,165] >> 3 >> 2 >> 53 80 GiB 16 TiB 80 GiB 16 TiB >> >> [3,22,23,52,54,86,87,154,157,158,159,160,162,163,164] 1 >> 0 >> 52 80 GiB 16 TiB 80 GiB 16 TiB >> >> [2,20,21,51,53,65,70,143,146,148,149,150,151,152,153] 2 >> 0 >> 51 80 GiB 16 TiB 80 GiB 16 TiB >> [0,1,3,11,18,19,34,38,41,42,50,52,60,61,66,80,83,86,96,99,132,134,135,137,138,140,141,142,143,144,145,164] >> 3 >> 3 >> 50 80 GiB 16 TiB 80 GiB 16 TiB >> >> [10,16,17,37,46,49,51,121,123,124,127,128,129,130,133] 2 >> 0 >> 49 80 GiB 16 TiB 80 GiB 16 TiB >> >> [1,14,15,48,50,111,112,113,114,115,116,119,122,160,163] 3 >> 0 >> 48 80 GiB 16 TiB 80 GiB 16 TiB >> >> [0,12,13,47,49,100,101,102,103,104,107,108,109,148,152] 2 >> 0 >> 47 80 GiB 16 TiB 80 GiB 16 TiB >> >> [8,34,35,46,48,89,90,91,95,96,97,98,99,136,138] 1 >> 46 80 GiB 16 TiB 80 GiB 16 TiB >> >> [9,28,32,33,37,45,47,74,78,79,80,83,84,85,86,88,91,127,128,137,159] >> 2 1 >> 45 80 GiB 16 TiB 80 GiB 16 TiB >> >> [7,13,30,31,44,46,61,63,65,68,69,71,72,73,76,77,101,104,114,119,143,157] >> 4 1 >> 44 80 GiB 16 TiB 80 GiB 16 TiB >> >> [6,28,29,43,45,53,58,59,60,61,62,63,66,71,80,81,86,94,100,107,132,166] >> 2 1 >> 43 80 GiB 16 TiB 80 GiB 16 TiB >> >> [5,26,27,38,42,44,47,48,49,50,51,52,54,91,96] 2 0 >> 42 80 GiB 16 TiB 80 GiB 16 TiB >> >> [4,24,25,36,37,39,40,41,43,44,45,46,79,83,167] 1 0 >> 41 80 GiB 16 TiB 80 GiB 16 TiB >> >> [3,22,23,40,42,72,73,154,156,158,159,161,162,164,166] 3 >> 0 >> 40 80 GiB 16 TiB 80 GiB 16 TiB >> >> [2,10,11,18,20,21,39,41,43,58,66,76,96,99,128,135,143,145,146,147,148,151,152,153] >> 6 >> 2 >> 39 80 GiB 16 TiB 80 GiB 16 TiB >> >> [11,18,19,38,40,50,132,135,136,137,138,140,141,142,167] 1 >> 0 >> 38 80 GiB 16 TiB 80 GiB 16 TiB >> >> [10,16,17,37,39,121,124,125,126,127,128,129,130,164,166] 2 >> 0 >> 37 80 GiB 16 TiB 80 GiB 16 TiB >> >> [1,14,15,26,36,38,97,111,113,114,116,117,118,119,122,145,148] >> 4 1 >> 36 80 GiB 16 TiB 80 GiB 16 TiB >> >> [0,12,19,35,37,100,101,102,103,104,105,107,108,109,141] 2 >> 0 >> 15 80 GiB 16 TiB 80 GiB 16 TiB >> >> [4,8,10,11,14,16,26,73,81,96,132,134,135,136,137,138,139,140,141,142,144] >> 2 1 >> 14 80 GiB 16 TiB 80 GiB 16 TiB >> >> [10,13,15,25,121,123,124,125,126,127,128,129,130,131,133] 2 >> 0 >> 13 80 GiB 16 TiB 80 GiB 16 TiB >> >> [1,12,14,22,111,112,113,114,115,116,117,118,119,120,122] 5 >> 0 >> 12 80 GiB 16 TiB 80 GiB 16 TiB >> >> [1,11,13,17,100,101,102,103,104,105,106,107,108,109,110] 0 >> 0 >> 11 80 GiB 16 TiB 80 GiB 16 TiB >> >> [7,10,12,14,21,35,49,89,90,91,92,93,94,95,96,97,98,99,104,132,134,149] >> 3 1 >> 10 80 GiB 16 TiB 80 GiB 16 TiB >> >> [9,11,13,32,33,78,79,80,81,82,83,84,85,86,87,88,131] 4 >> 1 >> 9 80 GiB 16 TiB 80 GiB 16 TiB >> >> [8,10,30,31,65,68,69,70,71,72,73,74,75,76,77] 2 0 >> 8 80 GiB 16 TiB 80 GiB 16 TiB >> >> [3,7,9,13,28,29,31,44,53,56,58,59,60,61,62,63,64,66,67,132] 4 >> 1 >> 7 80 GiB 16 TiB 80 GiB 16 TiB >> >> [6,8,26,27,38,43,47,48,49,50,51,52,54,55,57] 1 0 >> 6 80 GiB 16 TiB 80 GiB 16 TiB >> >> [5,7,24,25,36,37,39,40,41,42,44,45,46,165,167] 0 0 >> 1 80 GiB 16 TiB 80 GiB 16 TiB >> >> [0,2,14,15,111,112,113,114,115,116,117,118,119,120,122] 1 >> 0 >> 0 80 GiB 16 TiB 80 GiB 16 TiB >> >> [1,12,13,100,101,102,103,104,105,106,107,108,109,110,167] 2 >> 0 >> 2 80 GiB 16 TiB 80 GiB 16 TiB >> >> [1,3,16,17,121,123,124,125,126,127,128,129,130,131,133] 1 >> 0 >> 3 80 GiB 16 TiB 80 GiB 16 TiB >> >> [2,4,18,19,110,132,134,135,136,137,138,139,140,141,142,144,166] >> 5 1 >> 4 80 GiB 16 TiB 80 GiB 16 TiB >> >> [3,5,20,21,143,145,146,147,148,149,150,151,152,153,155] 2 >> 0 >> 5 80 GiB 16 TiB 80 GiB 16 TiB >> >> [0,2,4,6,22,23,37,41,91,107,118,154,156,157,158,159,160,161,162,163,164,166] >> 1 1 >> 16 80 GiB 16 TiB 80 GiB 16 TiB >> >> [2,15,17,29,143,145,146,147,148,149,150,151,152,153,155] 0 >> 0 >> 17 80 GiB 16 TiB 80 GiB 16 TiB >> >> [3,16,18,30,154,156,157,158,159,160,161,162,163,164,166] 3 >> 0 >> 18 80 GiB 16 TiB 80 GiB 16 TiB >> >> [4,17,19,33,36,37,39,40,41,42,44,45,46,131,148,165,167] 3 >> 1 >> 19 80 GiB 16 TiB 80 GiB 16 TiB >> >> [5,18,20,34,38,43,47,48,49,50,51,52,54,55,57] 2 0 >> 20 80 GiB 16 TiB 80 GiB 16 TiB >> >> [6,13,19,21,53,56,58,59,60,61,62,63,64,66,67] 4 0 >> 21 80 GiB 16 TiB 80 GiB 16 TiB >> >> [7,14,20,22,65,68,69,70,71,72,73,74,75,76,77] 3 0 >> 22 80 GiB 16 TiB 80 GiB 16 TiB >> >> [9,17,21,23,78,79,80,81,82,83,84,85,86,87,88] 0 0 >> 23 80 GiB 16 TiB 80 GiB 16 TiB >> >> [8,18,22,24,89,90,91,92,93,94,95,96,97,98,99] 1 0 >> 24 80 GiB 16 TiB 80 GiB 16 TiB >> >> [0,8,21,23,25,52,100,101,102,103,104,105,106,107,108,109,110] >> 3 1 >> 25 80 GiB 16 TiB 80 GiB 16 TiB >> >> [1,9,22,24,26,93,111,112,113,114,115,116,117,118,119,120,122] >> 3 1 >> 26 80 GiB 16 TiB 80 GiB 16 TiB >> >> [10,25,27,29,105,121,123,124,125,126,127,128,129,130,131,133,162] >> 2 1 >> 27 80 GiB 16 TiB 80 GiB 16 TiB >> >> [11,26,28,30,132,134,135,136,137,138,139,140,141,142,144] 1 >> 0 >> 28 80 GiB 16 TiB 80 GiB 16 TiB >> >> [2,27,29,33,143,145,146,147,148,149,150,151,152,153,155] 4 >> 0 >> 29 80 GiB 16 TiB 80 GiB 16 TiB >> >> [3,28,30,34,154,156,157,158,159,160,161,162,163,164,166] 1 >> 0 >> 30 80 GiB 16 TiB 80 GiB 16 TiB >> >> [4,13,29,31,36,37,39,40,41,42,44,45,46,65,109,133,136,165,167] >> 3 1 >> 31 80 GiB 16 TiB 80 GiB 16 TiB >> >> [5,14,30,32,38,43,47,48,49,50,51,52,54,55,57] 4 0 >> 32 80 GiB 16 TiB 80 GiB 16 TiB >> >> [6,17,31,33,53,56,58,59,60,61,62,63,64,66,67] 1 0 >> 33 80 GiB 16 TiB 80 GiB 16 TiB >> >> [7,18,30,32,34,41,65,68,69,70,71,72,73,74,75,76,77] 2 >> 1 >> 34 80 GiB 16 TiB 80 GiB 16 TiB >> >> [9,21,33,35,78,79,80,81,82,83,84,85,86,87,88] 4 0 >> 35 80 GiB 16 TiB 80 GiB 16 TiB >> >> [8,16,28,34,36,40,60,70,89,90,91,92,93,95,96,97,98,99,106,126,127,128,139,147] >> 2 >> 2 >> 78 80 GiB 16 TiB 80 GiB 16 TiB >> >> [4,24,25,36,39,40,41,42,44,45,46,77,79,111,120] 0 >> 79 80 GiB 16 TiB 80 GiB 16 TiB >> >> [5,26,27,47,48,49,50,51,52,54,57,78,80,123,124] 0 >> 80 80 GiB 16 TiB 80 GiB 16 TiB >> >> [6,28,29,53,58,59,61,63,64,66,67,79,81,132,136] 3 >> 81 80 GiB 16 TiB 80 GiB 16 TiB >> >> [7,30,31,65,68,71,73,74,75,76,77,80,82,146,148] 3 >> 82 80 GiB 16 TiB 80 GiB 16 TiB >> >> [9,32,33,78,79,80,81,83,84,85,86,87,88,158,163] 2 >> 83 80 GiB 16 TiB 80 GiB 16 TiB >> >> [8,21,32,34,35,36,40,72,82,84,89,90,91,92,96,97,98,99,114,121,122,153] >> 4 2 >> 84 80 GiB 16 TiB 80 GiB 16 TiB >> >> [0,12,13,48,57,83,85,100,101,102,103,106,107,109,110] 1 >> 0 >> 85 80 GiB 16 TiB 80 GiB 16 TiB >> >> [1,14,15,61,66,84,86,111,112,113,114,115,116,117,120] 3 >> 0 >> 86 80 GiB 16 TiB 80 GiB 16 TiB >> >> [10,16,17,74,75,85,87,121,123,124,125,126,128,129,131] 3 >> 0 >> 87 80 GiB 16 TiB 80 GiB 16 TiB >> >> [11,18,19,86,88,98,99,132,134,135,136,137,138,139,140] 0 >> 0 >> 88 80 GiB 16 TiB 80 GiB 16 TiB >> >> [2,20,21,87,89,101,107,143,146,147,148,149,150,151,155] 2 >> 0 >> 89 80 GiB 16 TiB 80 GiB 16 TiB >> >> [3,22,23,88,90,111,120,157,158,159,160,161,162,163,166] 0 >> 0 >> 90 80 GiB 16 TiB 80 GiB 16 TiB >> >> [4,24,25,36,37,39,40,41,44,45,46,89,91,123,124] 1 >> 91 80 GiB 16 TiB 80 GiB 16 TiB >> >> [5,26,27,38,47,48,49,50,52,54,57,90,92,120,132,136,142] 5 >> 1 >> 92 80 GiB 16 TiB 80 GiB 16 TiB >> >> [6,24,25,28,29,53,56,59,61,63,64,66,67,85,91,93,101,113,145,146,150] >> 2 1 >> 93 80 GiB 16 TiB 80 GiB 16 TiB >> >> [7,30,31,65,68,72,73,74,75,76,77,92,94,158,159] 3 >> 94 80 GiB 16 TiB 80 GiB 16 TiB >> >> [9,32,33,40,41,78,79,80,81,85,86,87,88,93,95] 1 0 >> 95 80 GiB 16 TiB 80 GiB 16 TiB >> >> [8,31,34,35,47,55,57,89,90,91,92,93,94,96,97,98,99] 1 >> 1 >> 96 80 GiB 16 TiB 80 GiB 16 TiB >> >> [0,12,13,61,64,95,97,100,101,102,103,105,106,109,110] 3 >> 0 >> 97 80 GiB 16 TiB 80 GiB 16 TiB >> >> [1,14,15,75,77,96,98,111,112,113,114,115,117,118,122] 2 >> 0 >> 98 80 GiB 16 TiB 80 GiB 16 TiB >> >> [10,16,17,66,86,87,97,99,121,123,124,125,126,128,129,130,132] >> 3 1 >> 99 80 GiB 16 TiB 80 GiB 16 TiB >> >> [11,18,19,98,100,110,111,134,135,136,137,138,139,140,142] 3 >> 0 >> 100 80 GiB 16 TiB 80 GiB 16 TiB >> >> [2,20,21,99,101,122,123,146,147,148,149,150,151,152,153] 0 >> 0 >> 101 80 GiB 16 TiB 80 GiB 16 TiB >> >> [3,20,22,23,33,74,100,102,132,135,158,159,160,161,162,163,164,166] >> 3 0 >> 102 80 GiB 16 TiB 80 GiB 16 TiB >> >> [4,24,25,36,37,39,40,41,44,45,101,103,145,147,165] 1 >> 0 >> 103 80 GiB 16 TiB 80 GiB 16 TiB >> >> [5,26,27,38,43,47,48,49,52,54,57,102,104,157,160] 1 >> 0 >> 104 80 GiB 16 TiB 80 GiB 16 TiB >> >> [6,17,28,29,40,41,46,53,56,59,62,63,64,66,67,73,82,84,103,105,110,147] >> 5 1 >> 105 80 GiB 16 TiB 80 GiB 16 TiB >> >> [7,30,31,47,50,65,68,70,72,74,75,76,77,104,106] 2 >> 106 80 GiB 16 TiB 80 GiB 16 TiB >> >> [9,32,33,58,64,78,79,80,81,84,85,86,88,105,107] 2 >> 107 80 GiB 16 TiB 80 GiB 16 TiB >> [8,17,20,25,27,34,35,69,75,83,89,90,91,92,93,95,97,98,99,105,106,108,113,122,124,129,132,154] >> 4 >> 2 >> 108 80 GiB 16 TiB 80 GiB 16 TiB >> >> [0,12,13,86,87,100,101,102,103,104,105,106,107,109,110] 1 >> 0 >> 109 80 GiB 16 TiB 80 GiB 16 TiB >> >> [1,14,15,96,98,108,110,111,112,113,114,115,116,117,118] 2 >> 0 >> 110 80 GiB 16 TiB 80 GiB 16 TiB >> >> [10,16,17,100,109,111,122,123,124,125,126,127,128,129,130] 4 >> 0 >> 111 80 GiB 16 TiB 80 GiB 16 TiB >> >> [11,18,19,110,112,123,131,135,136,137,138,139,140,142,144] 0 >> 0 >> 112 80 GiB 16 TiB 80 GiB 16 TiB >> >> [2,20,21,111,113,132,135,142,147,148,149,150,151,152,153,155] >> 4 1 >> 113 80 GiB 16 TiB 80 GiB 16 TiB >> >> [3,22,23,112,114,145,147,154,159,160,161,162,163,164,166] 2 >> 0 >> 114 80 GiB 16 TiB 80 GiB 16 TiB >> >> [4,24,25,37,39,40,41,44,45,113,115,157,159,165,167] 2 >> 0 >> 115 80 GiB 16 TiB 80 GiB 16 TiB >> >> [5,26,27,38,43,45,46,48,49,50,52,55,57,68,114,116] 1 >> 1 >> 116 80 GiB 16 TiB 80 GiB 16 TiB >> >> [6,28,29,47,48,53,56,60,61,62,63,66,67,115,117] 1 >> 117 80 GiB 16 TiB 80 GiB 16 TiB >> >> [7,30,31,61,64,65,68,69,70,72,75,76,77,116,118] 1 >> 118 80 GiB 16 TiB 80 GiB 16 TiB >> >> [9,32,33,71,75,78,79,80,81,82,86,87,88,117,119] 2 >> 119 80 GiB 16 TiB 80 GiB 16 TiB >> >> [8,34,35,84,88,89,90,91,92,93,94,97,99,118,120] 1 >> 120 80 GiB 16 TiB 80 GiB 16 TiB >> >> [0,12,13,90,96,101,102,103,104,105,106,107,109,119,121] 1 >> 0 >> 121 80 GiB 16 TiB 80 GiB 16 TiB >> >> [1,14,15,109,110,112,113,114,115,116,117,118,119,120,122] 2 >> 0 >> 122 80 GiB 16 TiB 80 GiB 16 TiB >> >> [10,16,17,71,111,112,121,123,124,125,126,127,128,129,130,131,132] >> 3 1 >> 123 80 GiB 16 TiB 80 GiB 16 TiB >> >> [11,18,19,122,124,132,135,136,137,138,139,140,141,142,144] 0 >> 0 >> 124 80 GiB 16 TiB 80 GiB 16 TiB >> >> [2,20,21,123,125,143,145,147,148,149,150,151,152,153,155] 1 >> 0 >> 125 80 GiB 16 TiB 80 GiB 16 TiB >> >> [3,22,23,124,126,154,156,157,159,160,161,162,163,164,166] 0 >> 0 >> 126 80 GiB 16 TiB 80 GiB 16 TiB >> >> [4,24,25,36,37,39,41,42,44,45,46,125,127,165,167] 1 >> 0 >> 127 80 GiB 16 TiB 80 GiB 16 TiB >> >> [5,26,27,38,43,47,48,49,50,51,52,55,57,126,128] 2 >> 128 80 GiB 16 TiB 80 GiB 16 TiB >> >> [6,28,29,53,56,58,60,61,62,63,64,66,67,127,129] 2 >> 129 80 GiB 16 TiB 80 GiB 16 TiB >> >> [7,30,31,65,68,69,70,72,73,74,75,76,77,128,130] 3 >> 130 80 GiB 16 TiB 80 GiB 16 TiB >> >> [9,32,33,43,45,65,66,78,79,80,81,82,83,84,86,87,88,93,103,129,131,143,145,167] >> 3 >> 2 >> 131 80 GiB 16 TiB 80 GiB 16 TiB >> >> [8,13,34,35,63,89,90,91,92,93,94,95,96,97,99,130,132,133,135,146,151] >> 4 1 >> 132 80 GiB 16 TiB 80 GiB 16 TiB >> >> [0,12,13,101,102,103,104,105,106,107,108,109,110,131,133] 7 >> 0 >> 133 80 GiB 16 TiB 80 GiB 16 TiB >> >> [1,14,15,111,112,113,114,115,116,117,118,119,120,132,134] 2 >> 0 >> 134 80 GiB 16 TiB 80 GiB 16 TiB >> >> [10,16,17,121,124,125,126,127,128,129,130,131,133,135,144] 3 >> 0 >> 135 80 GiB 16 TiB 80 GiB 16 TiB >> [8,11,18,19,88,98,112,114,119,131,132,134,136,137,138,139,140,141,142,144,145,147,154] >> 4 >> 2 >> 136 80 GiB 16 TiB 80 GiB 16 TiB >> >> [2,20,21,135,137,143,145,149,150,151,152,153,155,157,159] 1 >> 0 >> 137 80 GiB 16 TiB 80 GiB 16 TiB >> >> [3,22,23,41,46,136,138,154,156,157,160,161,162,164,166] 1 >> 0 >> 138 80 GiB 16 TiB 80 GiB 16 TiB >> >> [4,24,25,37,39,42,44,45,46,47,50,137,139,165,167] 0 >> 0 >> 139 80 GiB 16 TiB 80 GiB 16 TiB >> >> [5,26,27,38,43,47,49,51,52,54,55,58,67,138,140] 1 >> 140 80 GiB 16 TiB 80 GiB 16 TiB >> >> [6,28,29,53,56,58,59,60,62,64,66,73,77,139,141] 0 >> 141 80 GiB 16 TiB 80 GiB 16 TiB >> >> [7,30,31,65,68,69,70,71,72,73,76,85,87,140,142] 1 >> 142 80 GiB 16 TiB 80 GiB 16 TiB >> >> [9,32,33,78,80,81,82,83,84,85,86,90,98,141,143] 3 >> 143 80 GiB 16 TiB 80 GiB 16 TiB >> >> [8,34,35,89,91,92,93,94,95,96,98,107,109,142,144] 4 >> 0 >> 144 80 GiB 16 TiB 80 GiB 16 TiB >> >> [0,12,13,102,103,104,105,106,107,108,110,111,112,143,145] 3 >> 0 >> 145 80 GiB 16 TiB 80 GiB 16 TiB >> >> [1,14,15,111,114,115,116,117,118,119,120,124,133,144,146] 3 >> 0 >> 146 80 GiB 16 TiB 80 GiB 16 TiB >> [10,16,17,104,106,108,121,126,127,128,129,130,131,133,134,136,141,145,147,158,159,163] >> 3 >> 1 >> 147 80 GiB 16 TiB 80 GiB 16 TiB >> >> [11,18,19,34,132,134,138,139,140,141,142,144,146,148,157,159,167] >> 5 1 >> 148 80 GiB 16 TiB 80 GiB 16 TiB >> >> [2,20,21,40,46,143,145,146,147,149,150,151,152,153,155] 2 >> 0 >> 149 80 GiB 16 TiB 80 GiB 16 TiB >> >> [3,11,14,19,22,23,47,48,60,85,110,143,148,150,154,156,157,158,161,162,164,166] >> 3 >> 2 >> 150 80 GiB 16 TiB 80 GiB 16 TiB >> >> [4,20,21,24,25,36,37,39,42,45,46,61,67,149,151,165,167] 5 >> 2 >> 151 80 GiB 16 TiB 80 GiB 16 TiB >> >> [5,26,27,38,43,47,50,51,52,54,55,71,77,150,152] 2 >> 152 80 GiB 16 TiB 80 GiB 16 TiB >> >> [3,6,28,29,53,56,58,59,60,61,62,66,85,87,93,151,153] 2 >> 1 >> 153 80 GiB 16 TiB 80 GiB 16 TiB >> >> [7,30,31,65,69,70,71,72,73,75,76,90,98,152,154] 1 >> 154 80 GiB 16 TiB 80 GiB 16 TiB >> >> [9,32,33,78,80,81,82,83,84,85,88,107,109,153,155] 2 >> 0 >> 155 80 GiB 16 TiB 80 GiB 16 TiB >> >> [8,34,35,91,92,93,94,95,96,97,98,111,112,154,156] 0 >> 0 >> 156 80 GiB 16 TiB 80 GiB 16 TiB >> >> [0,12,13,40,100,103,104,105,106,107,108,109,110,124,133,155,157] >> 2 1 >> 157 80 GiB 16 TiB 80 GiB 16 TiB >> >> [1,14,15,111,115,116,117,118,119,120,122,134,136,156,158] 1 >> 0 >> 158 80 GiB 16 TiB 80 GiB 16 TiB >> >> [10,16,17,121,123,126,127,129,130,131,133,146,148,157,159] 3 >> 0 >> 159 80 GiB 16 TiB 80 GiB 16 TiB >> >> [11,18,19,40,46,132,134,135,138,140,141,142,144,158,160] 3 >> 0 >> 160 80 GiB 16 TiB 80 GiB 16 TiB >> >> [2,20,21,47,49,54,112,143,145,146,147,150,151,152,155,159,161] >> 2 1 >> 161 80 GiB 16 TiB 80 GiB 16 TiB >> >> [3,22,23,61,67,154,156,157,158,159,160,162,163,164,166] 1 >> 0 >> 162 80 GiB 16 TiB 80 GiB 16 TiB >> >> [4,9,10,23,24,25,36,37,39,40,42,46,71,74,121,152,160,161,163,165,167] >> 2 1 >> 163 80 GiB 16 TiB 80 GiB 16 TiB >> >> [5,26,27,30,38,43,47,50,51,52,54,55,57,85,88,110,116,118,147,151,162,164] >> 2 1 >> 164 80 GiB 16 TiB 80 GiB 16 TiB >> >> [6,28,29,31,34,53,59,60,61,62,64,66,67,90,93,163,165] 2 >> 1 >> 165 80 GiB 16 TiB 80 GiB 16 TiB >> >> [7,30,31,65,69,70,71,72,73,74,75,109,110,164,166] 0 >> 0 >> sum 13 TiB 2.7 PiB 13 TiB 2.7 PiB >> >> * NOTE: Omap statistics are gathered during deep scrub and may be >> inaccurate soon afterwards depending on utilization. See >> http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics >> for further details. >> dumped all >> >> >> root@store-par2-node01:/home/user# ceph fs status >> cephfs - 2 clients >> ====== >> RANK STATE MDS ACTIVITY DNS >> INOS DIRS CAPS >> 0 active cephfs.store-par2-node01.honsru Reqs: 0 /s 10 >> 13 12 2 >> POOL TYPE USED AVAIL >> cephfs_metadata metadata 5742 870T >> cephfs_data data 0 1959T >> STANDBY MDS >> cephfs.store-par2-node02.ttsfvq >> cephfs.store-par2-node03.vnxscc >> cephfs.store-par2-node04.iudcse >> >> >> If somebody have an idea how can I correct this, or good pointers to >> learn about PG with cephadm, It will be nice ... >> >> Thanks >> >> >> -- >> Christophe BAILLON >> Mobile :: +336 16 400 522 >> Work :: https://eyona.com >> Twitter :: https://twitter.com/ctof >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users@xxxxxxx >> To unsubscribe send an email to ceph-users-leave@xxxxxxx > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx -- Christophe BAILLON Mobile :: +336 16 400 522 Work :: https://eyona.com Twitter :: https://twitter.com/ctof _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |