Hi,
I have some problems with my ceph monitor nodes in my Cluster.
I had 5 mons in the cluster. On all 5 nodes the leveldb store grew up to
about 80 – 90 GB in size. To get rid of it I triggerd a compaction with
the following command on one node.
ceph tell mon.d compact
The Monitor compacted his data to about 5 GB. After this the mon tried
to synchronize his data with the other mons. And start my problems.
After a short time (20 – 30 sec) of streaming data from another mon node
The stream breaks and further the sender reads oviously all the other
data in his store with maximum speed. At this point the cluster lost his
leader and trying to elect a new one. The leader election only works if
the reading of the data is done.
I tried to remove the mon from the cluster completly and rejoin it as a
new one but while syncing I experience the same issue. So currently the
cluster has only 4 Mons.
While further investigation and testing I lost another mon which wants
to sync data after starting with the same behavior.
It seems the Node which streams data while syncing is in stress with
reading and sending data. I tried to limit the network bandwith of the
joining node to reduce the load. I also tried to set ionice -c3 on the
process which does all the disk IO while reading. But nothing helped.
Because the cluster is productive I don't want to expermient further
more without knowing what's going on.
Does anyone have any ideas what's going on and how I can try to fix this?
I am using ceph version 0.67.11 (bc8b67bef6309a32361be76cd11fb56b057ea9d2)
5 Monitor Nodes with SSD as leveldb store
24 OSD Hosts with 1416 OSDs
Thank you
Manuel
--
Manuel Lausch
Systemadministrator
Cloud Backend Services
1&1 Mail & Media Development & Technology GmbH | Brauerstraße 48 | 76135 Karlsruhe | Germany
Phone: +49 721 91374-1847
E-Mail: manuel.lausch@xxxxxxxx | Web: www.1und1.de
Amtsgericht Montabaur, HRB 5452
Geschäftsführer: Frank Einhellinger, Hans-Henning Kettler, Jan Oetjen
Member of United Internet
Diese E-Mail kann vertrauliche und/oder gesetzlich geschützte Informationen enthalten. Wenn Sie nicht der bestimmungsgemäße Adressat sind oder diese E-Mail irrtümlich erhalten haben, unterrichten Sie bitte den Absender und vernichten Sie diese E-Mail. Anderen als dem bestimmungsgemäßen Adressaten ist untersagt, diese E-Mail zu speichern, weiterzuleiten oder ihren Inhalt auf welche Weise auch immer zu verwenden.
This e-mail may contain confidential and/or privileged information. If you are not the intended recipient of this e-mail, you are hereby notified that saving, distribution or use of the content of this e-mail in any way is prohibited. If you have received this e-mail in error, please notify the sender and delete the e-mail.
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
