Hi *,
I'm investigating an interesting issue on two customer clusters (used
for mirroring) I've not solved yet, but today we finally made some
progress. Maybe someone has an idea where to look next, I'd appreciate
any hints or comments.
These are two (latest) Octopus clusters, main usage currently is RBD
mirroring with snapshot mode (around 500 RBD images are synced every
30 minutes). They noticed very long startup times of MON daemons after
reboot, times between 10 and 30 minutes (reboot time already
subtracted). These delays are present on both sites. Today we got a
maintenance window and started to check in more detail by just
restarting the MON service (joins quorum within seconds), then
stopping the MON service and wait a few minutes (still joins quorum
within seconds). And then we stopped the service and waited for more
than 5 minutes, simulating a reboot, and then we were able to
reproduce it. The sync then takes around 15 minutes, we verified with
other MONs as well. The MON store is around 2 GB of size (on HDD), I
understand that the sync itself can take some time, but what is the
threshold here? I tried to find a hint in the MON config, searching
for timeouts with 300 seconds, there were only a few matches
(mon_session_timeout is one of them), but I'm not sure if they can
explain this behavior.
Investigating the MON store (ceph-monstore-tool dump-keys) I noticed
that there were more than 42 Million osd_snap keys, which is quite a
lot and would explain the size of the MON store. But I'm also not sure
if it's related to the long syncing process.
Does that sound familiar to anyone?
Thanks,
Eugen
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
