osd_crush_initial_weight = 0
In ceph.conf an decrease the Crush weight via
Ceph osd Crush reweight osd.36 0.05000
Step by step
Am 25. April 2017 23:19:08 MESZ schrieb Reed Dier <reed.dier@xxxxxxxxxxx>:
Others will likely be able to provide some better responses, but I’ll take a shot to see if anything makes sense.
With 10.2.6 you should be able to set 'osd scrub during recovery’ to false to prevent any new scrubs from occurring during a recovery event. Current scrubs will complete, but future scrubs will not being until recovery has completed.
Also, adding just one OSD on the new server, assuming all 6 are ready(?) will cause a good deal of unnecessary data reshuffling as you add more OSD’s.
And on top of that, assuming the pool’s crush ruleset is ‘chooseleaf first 0 type host’ then that should create a bit of an unbalanced weighting. Any reason you aren’t bringing in all 6 OSD’s at once?
You should be able to set noscrub, noscrub-deep, norebalance, nobackfill, and norecover flags (also probably want noout to prevent rebalance if OSDs flap), wait for scrubs to complete (especially deep), add your 6 OSD’s, unset your flags for recovery/rebalance/backfill, and it will then move data only once, and hopefully not have the scrub load. After recovery, unset the scrub flags, and be back to normal.
Caveat, no VM’s running on my cluster, but those seem like low hanging fruit for possible load lightening during a rebalance.
ReedOn Apr 25, 2017, at 3:47 PM, Ramazan Terzi <ramazanterzi@xxxxxxxxx> wrote:
Hello,
I have a Ceph Cluster with specifications below:
3 x Monitor node
6 x Storage Node (6 disk per Storage Node, 6TB SATA Disks, all disks have SSD journals)
Distributed public and private networks. All NICs are 10Gbit/s
osd pool default size = 3
osd pool default min size = 2
Ceph version is Jewel 10.2.6.
Current health status:
cluster ****************
health HEALTH_OK
monmap e9: 3 mons at {ceph-mon01=xxx:6789/0,ceph-mon02=xxx:6789/0,ceph-mon03=xxx:6789/0}
election epoch 84, quorum 0,1,2 ceph-mon01,ceph-mon02,ceph-mon03
osdmap e1512: 36 osds: 36 up, 36 in
flags sortbitwise,require_jewel_osds
pgmap v7698673: 1408 pgs, 5 pools, 37365 GB data, 9436 kobjects
83871 GB used, 114 TB / 196 TB avail
1408 active+clean
My cluster is active and a lot of virtual machines running on it (Linux and Windows VM's, database clusters, web servers etc).
When I want to add a new storage node with 1 disk, I'm getting huge problems. With new osd, crushmap updated and Ceph Cluster turns into recovery mode. Everything is OK. But after a while, some runnings VM's became unmanageable. Servers become unresponsive one by one. Recovery process would take an average of 20 hours. For this reason, I removed the new osd. Recovery process completed and everythink become normal.
When new osd added, health status:
cluster ****************
health HEALTH_WARN
91 pgs backfill_wait
1 pgs bacfilling
28 pgs degraded
28 pgs recovery_wait
28 phs stuck degraded
recovery 2195/18486602 objects degraded (0.012%)
recovery 1279784/18486602 objects misplaced (6.923%)
monmap e9: 3 mons at {ceph-mon01=xxx:6789/0,ceph-mon02=xxx:6789/0,ceph-mon03=xxx:6789/0}
election epoch 84, quorum 0,1,2 ceph-mon01,ceph-mon02,ceph-mon03
osdmap e1512: 37 osds: 37 up, 37 in
flags sortbitwise,require_jewel_osds
pgmap v7698673: 1408 pgs, 5 pools, 37365 GB data, 9436 kobjects
83871 GB used, 114 TB / 201 TB avail
2195/18486602 objects degraded (0.012%)
1279784/18486602 objects misplaced (6.923%)
1286 active+clean
91 active+remapped+wait_backfill
28 active+recovery_wait+degraded
2 active+clean+scrubbing+deep
1 active+remapped+backfilling
recovery io 430 MB/s, 119 objects/s
client io 36174 B/s rrd, 5567 kB/s wr, 5 op/s rd, 700 op/s wr
Some Ceph config parameters:
osd_max_backfills = 1
osd_backfill_full_ratio = 0.85
osd_recovery_max_active = 3
osd_recovery_threads = 1
How I can add new OSD's safely?
Best regards,
Ramazan
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
