Hi, I found my error, it was a mismatch between the monitor network ip address and the --cluster_network which were in different subnets. I misunderstood the --cluster_network subnet, I thought that when creating a cluster, the monitor IP designed the public Network, and if I wanted to separate public and private (cluster) networks, I needed to use the --cluster_network option. Maybe I was in over my head, but sometimes it is not that clear. Regards. Le ven. 15 mars 2024 à 07:18, wodel youchi <wodel.youchi@xxxxxxxxx> a écrit : > Hi, > > Note : Firewall is disabled on all hosts. > > Regards. > > Le ven. 15 mars 2024 à 06:42, wodel youchi <wodel.youchi@xxxxxxxxx> a > écrit : > >> Hi, >> >> I did recreate the cluster again, and it is the result. >> >> This is my initial bootstrap >> >> cephadm --image 192.168.2.36:4000/ceph/ceph:v18 bootstrap >> --initial-dashboard-user admin \ >> --initial-dashboard-password adminpass --dashboard-password-noupdate >> --registry-url 192.168.2.36:4000 \ >> --registry-username admin --registry-password admin --mon-ip 20.1.0.23 >> --cluster-network 20.2.0.0/16 \ >> --ssh-private-key /root/.ssh/id_rsa --ssh-public-key >> /root/.ssh/id_rsa.pub \ >> -c initial-ceph.conf >> >> This is my initial-ceph.conf >> [mgr] >> mgr/cephadm/container_image_prometheus = >> 192.168.2.36:4000/prometheus/prometheus:v2.43.0 >> mgr/cephadm/container_image_node_exporter = >> 192.168.2.36:4000/prometheus/node-exporter:v1.5.0 >> mgr/cephadm/container_image_grafana = >> 192.168.2.36:4000/ceph/ceph-grafana:9.4.7 >> mgr/cephadm/container_image_alertmanager = >> 192.168.2.36:4000/prometheus/alertmanager:v0.25.0 >> >> >> Then I added two managers and monitors >> # ceph orch host add controllerb 20.1.0.27 _admin >> # ceph orch host add controllerc 20.1.0.31 _admin >> # ceph orch apply mon --placement="3 controllera controllerb >> controllerc" >> # ceph orch apply mgr --placement="3 controllera controllerb controllerc" >> >> Then I added node-exporter, prometheus, grafana and crash >> Then I added osd hosts >> # ceph orch host add computehci01 20.1.0.2 >> # ceph orch host add computehci02 20.1.0.3 >> # ceph orch host add computehci03 20.1.0.4 >> ... >> # ceph orch host add computehci09 20.1.0.10 >> ... >> >> And finally I added osd daemons >> # ceph orch daemon add osd >> computehci01:/dev/nvme0n1,/dev/nvme1n1,/dev/nvme2n1,/dev/nvme3n1 >> # ceph orch daemon add osd >> computehci02:/dev/nvme0n1,/dev/nvme1n1,/dev/nvme2n1,/dev/nvme3n1 >> ... >> >> I created a pool >> # ceph osd pool create volumes replicated >> # ceph osd pool application enable volumes rbd >> >> I even created cephfs pools and added mds service, but still 100% pgs are >> unknown???? !!!! >> >> >> [root@controllera ~]# ceph -s >> cluster: >> id: df914aa2-e21a-11ee-b8df-3cecef2872f0 >> health: HEALTH_WARN >> 1 MDSs report slow metadata IOs >> Reduced data availability: 4 pgs inactive >> >> services: >> mon: 3 daemons, quorum controllera,controllerc,controllerb (age 13h) >> mgr: controllera.ajttxz(active, since 13h), standbys: >> controllerb.qtixeq, controllerc.pqyqqo >> mds: 1/1 daemons up, 2 standby >> osd: 36 osds: 36 up (since 7h), 36 in (since 7h) >> >> data: >> volumes: 1/1 healthy >> pools: 4 pools, 4 pgs >> objects: 0 objects, 0 B >> usage: 1.1 GiB used, 110 TiB / 110 TiB avail >> pgs: 100.000% pgs unknown >> 4 unknown >> >> [root@controllera ~]# ceph osd pool ls detail >> pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash >> rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 46 flags >> hashpspool,creating stripe_width 0 pg_num_max 32 pg_nu >> m_min 1 application mgr >> pool 2 'volumes' replicated size 3 min_size 2 crush_rule 0 object_hash >> rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 154 flags >> hashpspool,creating stripe_width 0 application rbd >> pool 3 'cephfs' replicated size 3 min_size 2 crush_rule 0 object_hash >> rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 157 flags >> hashpspool,creating stripe_width 0 application ceph >> fs >> pool 4 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 >> object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 157 >> flags hashpspool,creating stripe_width 0 pg_auto >> scale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs >> >> What am I missing, why PGs won't pair? >> >> >> >> Regards. >> >> Le jeu. 14 mars 2024 à 15:36, wodel youchi <wodel.youchi@xxxxxxxxx> a >> écrit : >> >>> Hi, >>> >>> I am creating a new ceph cluster using REEF. >>> >>> This is my host_specs file >>> [root@controllera config]# cat hosts-specs2.yml >>> service_type: host >>> hostname: computehci01 >>> addr: 20.1.0.2 >>> location: >>> chassis: chassis1 >>> --- >>> service_type: host >>> hostname: computehci02 >>> addr: 20.1.0.3 >>> location: >>> chassis: chassis1 >>> --- >>> service_type: host >>> hostname: computehci03 >>> addr: 20.1.0.4 >>> location: >>> chassis: chassis1 >>> --- >>> service_type: host >>> hostname: computehci04 >>> addr: 20.1.0.5 >>> location: >>> chassis: chassis2 >>> --- >>> service_type: host >>> hostname: computehci05 >>> addr: 20.1.0.6 >>> location: >>> chassis: chassis2 >>> --- >>> service_type: host >>> hostname: computehci06 >>> addr: 20.1.0.7 >>> location: >>> chassis: chassis2 >>> --- >>> service_type: host >>> hostname: computehci07 >>> addr: 20.1.0.8 >>> location: >>> chassis: chassis3 >>> --- >>> service_type: host >>> hostname: computehci08 >>> addr: 20.1.0.9 >>> location: >>> chassis: chassis3 >>> --- >>> service_type: host >>> hostname: computehci09 >>> addr: 20.1.0.10 >>> location: >>> chassis: chassis3 >>> --- >>> service_type: host >>> hostname: computehci10 >>> addr: 20.1.0.11 >>> location: >>> chassis: chassis3 >>> --- >>> service_type: host >>> hostname: computehci11 >>> addr: 20.1.0.12 >>> location: >>> chassis: chassis4 >>> --- >>> service_type: host >>> hostname: computehci12 >>> addr: 20.1.0.13 >>> location: >>> chassis: chassis4 >>> --- >>> service_type: host >>> hostname: computehci13 >>> addr: 20.1.0.14 >>> location: >>> chassis: chassis4 >>> --- >>> service_type: host >>> hostname: computehci14 >>> addr: 20.1.0.15 >>> location: >>> chassis: chassis4 >>> --- >>> service_type: host >>> hostname: computehci15 >>> addr: 20.1.0.16 >>> location: >>> chassis: chassis5 >>> --- >>> service_type: host >>> hostname: computehci16 >>> addr: 20.1.0.17 >>> location: >>> chassis: chassis5 >>> --- >>> service_type: host >>> hostname: computehci17 >>> addr: 20.1.0.18 >>> location: >>> chassis: chassis5 >>> --- >>> service_type: host >>> hostname: computehci18 >>> addr: 20.1.0.19 >>> location: >>> chassis: chassis5 >>> --- >>> service_type: host >>> hostname: computehci19 >>> addr: 20.1.0.20 >>> location: >>> chassis: chassis6 >>> --- >>> service_type: host >>> hostname: computehci20 >>> addr: 20.1.0.21 >>> location: >>> chassis: chassis6 >>> --- >>> service_type: host >>> hostname: computehci21 >>> addr: 20.1.0.22 >>> location: >>> chassis: chassis6 >>> --- >>> service_type: host >>> hostname: computehci22 >>> addr: 20.1.0.24 >>> location: >>> chassis: chassis7 >>> --- >>> service_type: host >>> hostname: computehci23 >>> addr: 20.1.0.25 >>> location: >>> chassis: chassis7 >>> --- >>> service_type: host >>> hostname: computehci24 >>> addr: 20.1.0.26 >>> location: >>> chassis: chassis7 >>> --- >>> service_type: host >>> hostname: computehci25 >>> addr: 20.1.0.28 >>> location: >>> chassis: chassis8 >>> --- >>> service_type: host >>> hostname: computehci26 >>> addr: 20.1.0.29 >>> location: >>> chassis: chassis8 >>> --- >>> service_type: host >>> hostname: computehci27 >>> addr: 20.1.0.30 >>> location: >>> chassis: chassis8 >>> --- >>> service_type: host >>> hostname: controllera >>> addr: 20.1.0.23 >>> --- >>> service_type: host >>> hostname: controllerb >>> addr: 20.1.0.27 >>> --- >>> service_type: host >>> hostname: controllerc >>> addr: 20.1.0.31 >>> --- >>> service_type: mon >>> placement: >>> hosts: >>> - controllera >>> - controllerb >>> - controllerc >>> --- >>> service_type: mgr >>> placement: >>> hosts: >>> - controllera >>> - controllerb >>> - controllerc >>> --- >>> service_type: osd >>> service_id: default_drive_group >>> placement: >>> hosts: >>> - computehci01 >>> - computehci02 >>> - computehci03 >>> - computehci04 >>> - computehci05 >>> - computehci06 >>> - computehci07 >>> - computehci08 >>> - computehci09 >>> - computehci10 >>> - computehci11 >>> - computehci12 >>> - computehci13 >>> - computehci14 >>> - computehci15 >>> - computehci16 >>> - computehci17 >>> - computehci18 >>> - computehci19 >>> - computehci20 >>> - computehci21 >>> - computehci22 >>> - computehci23 >>> - computehci24 >>> - computehci25 >>> - computehci26 >>> - computehci27 >>> spec: >>> data_devices: >>> rotational: 0 >>> >>> >>> All osds were added but, pg still unknown state >>> >>> I've created a pool, but it didn't change anything. >>> >>> [root@controllerb ~]# ceph -s >>> cluster: >>> id: be250ade-e1f2-11ee-a6ff-3cecef2872f0 >>> health: HEALTH_WARN >>> Reduced data availability: 1 pg inactive >>> >>> services: >>> mon: 3 daemons, quorum controllera,controllerc,controllerb (age 3h) >>> mgr: controllerc.jevbkl(active, since 21s), standbys: >>> controllera.zwlolp, controllerb.vqkdga >>> osd: 108 osds: 108 up (since 2m), 108 in (since 24m) >>> >>> data: >>> pools: 2 pools, 33 pgs >>> objects: 0 objects, 0 B >>> usage: 5.1 GiB used, 330 TiB / 330 TiB avail >>> >>> >>> * pgs: 100.000% pgs unknown 33 unknown* >>> >>> Did I miss something? >>> >>> Regards. >>> >> _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |