Hi,
Answering my own question, the high load was related to the cpufreq kernel module. Unloaded the cpufreq module and the CPU load instantly dropped and the mirroring started to work.
Obviously there is a bug somewhere but for the moment I’m just happy it works.
/Magnus
Den tors 15 nov. 2018 kl 15:24 skrev Magnus Grönlund <magnus@xxxxxxxxxxx>:
Hi,I’m trying to setup one-way rbd-mirroring for a ceph-cluster used by an openstack cloud, but the rbd-mirror is unable to “catch up” with the changes. However it appears to me as if it's not due to the ceph-clusters or the network but due to the server running the rbd-mirror process running out of cpu?Is a high cpu load to be expected or is it a symptom of something else?Or in other words, what can I check/do to get the mirroring working? 😊# rbd mirror pool status novahealth: WARNINGimages: 596 total572 starting_replay24 replayingtop - 13:31:36 up 79 days, 5:31, 1 user, load average: 32.27, 26.82, 25.33Tasks: 360 total, 17 running, 182 sleeping, 0 stopped, 0 zombie%Cpu(s): 8.9 us, 70.0 sy, 0.0 ni, 18.5 id, 0.0 wa, 0.0 hi, 2.7 si, 0.0 stKiB Mem : 13205185+total, 12862490+free, 579508 used, 2847444 buff/cacheKiB Swap: 0 total, 0 free, 0 used. 12948856+avail MemPID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND2336553 ceph 20 0 17.1g 178160 20344 S 417.2 0.1 21:50.61 rbd-mirror2312698 root 20 0 0 0 0 I 70.2 0.0 70:11.51 kworker/12:22312851 root 20 0 0 0 0 R 69.2 0.0 62:29.69 kworker/24:12324627 root 20 0 0 0 0 I 68.4 0.0 40:36.77 kworker/14:12235817 root 20 0 0 0 0 I 68.0 0.0 469:14.08 kworker/8:02241720 root 20 0 0 0 0 R 67.3 0.0 437:46.51 kworker/9:12306648 root 20 0 0 0 0 R 66.9 0.0 109:27.44 kworker/25:02324625 root 20 0 0 0 0 R 66.9 0.0 40:37.53 kworker/13:12336318 root 20 0 0 0 0 R 66.7 0.0 14:51.96 kworker/27:32324643 root 20 0 0 0 0 I 66.5 0.0 36:21.46 kworker/15:22294989 root 20 0 0 0 0 I 66.3 0.0 134:09.89 kworker/11:12324626 root 20 0 0 0 0 I 66.3 0.0 39:44.14 kworker/28:22324019 root 20 0 0 0 0 I 65.3 0.0 44:51.80 kworker/26:12235814 root 20 0 0 0 0 R 65.1 0.0 459:14.70 kworker/29:22294174 root 20 0 0 0 0 I 64.5 0.0 220:58.50 kworker/30:12324355 root 20 0 0 0 0 R 63.3 0.0 45:04.29 kworker/10:12263800 root 20 0 0 0 0 R 62.9 0.0 353:38.48 kworker/31:12270765 root 20 0 0 0 0 R 60.2 0.0 294:46.34 kworker/0:02294798 root 20 0 0 0 0 R 59.8 0.0 148:48.23 kworker/1:22307128 root 20 0 0 0 0 R 59.8 0.0 86:15.45 kworker/6:22307129 root 20 0 0 0 0 I 59.6 0.0 85:29.66 kworker/5:02294826 root 20 0 0 0 0 R 58.2 0.0 138:53.56 kworker/7:32294575 root 20 0 0 0 0 I 57.8 0.0 155:03.74 kworker/2:32294310 root 20 0 0 0 0 I 57.2 0.0 176:10.92 kworker/4:22295000 root 20 0 0 0 0 I 57.2 0.0 132:47.28 kworker/3:22307060 root 20 0 0 0 0 I 56.6 0.0 87:46.59 kworker/23:22294931 root 20 0 0 0 0 I 56.4 0.0 133:31.47 kworker/17:22318659 root 20 0 0 0 0 I 56.2 0.0 55:01.78 kworker/16:22336304 root 20 0 0 0 0 I 56.0 0.0 11:45.92 kworker/21:22306947 root 20 0 0 0 0 R 55.6 0.0 90:45.31 kworker/22:22270628 root 20 0 0 0 0 I 53.8 0.0 273:43.31 kworker/19:32294797 root 20 0 0 0 0 R 52.3 0.0 141:13.67 kworker/18:02330537 root 20 0 0 0 0 R 52.3 0.0 25:33.25 kworker/20:2The main cluster has 12 nodes with 120 OSDs and the backup cluster has 6 nodes with 60 OSDs (but roughly the same amount of storage), the rbd-mirror runs on a separate server with 2* E5-2650v2 cpus and 128GB memory.Best regards/Magnus
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
