OSD Recovery Delay Start

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

We are having a lot of trouble with the SSD OSDs for our cache tier
when they reboot. It causes massive blocked I/O when booting the OSD
and the entire cluster I/O nearly stalls even when the OSD is only
down for 60 seconds.

I have noticed that when the OSD starts it uses massive amounts of
RAM, for  the one minute test it used almost 8 GB, another one earlier
this morning used 14 GB, some last night were in the 10GB range.
During this time the process is not using much CPU, but the disks are
very busy writing a good 120-250 MB/s and hundreds to low thousand
IOPs. Once the memory usage gets down to about 1.5 GB blocked I/O
starts clearing slowly. At first I thought this was due to preloading
jemalloc, but it also happens without it.

Looking through [1] I thought osd recovery delay start set to 60
seconds or longer would allow the OSD to come up, join the cluster, do
any housekeeping before being in and trying to service I/O requests.
However, setting the value to 60 does nothing, we see recovery
operations start less than 30 seconds after the monitor shows the boot
message. The osd log does not show any kind of delay either.

Is there a bug here or am I understanding this option incorrectly?

What I'm looking for is something to delay any I/O until the peering
is completed, the PGs have been scanned, all of the house keeping is
done so that the only load on the OSD/disk is client/recovery I/O. I
don't want it to try to do both at the same time.

Once the OSD finally comes in and the blocked I/O clears, we can
manage backfilling and recovery without much impact to the cluster, it
is just the initial minutes of terror (dozens of blocked I/O > 500
seconds) that we can't figure out how to get rid of. I understand that
there will be some impact for recovery, but on our cluster that on
average does about 10K IOPs, we have less than 5K for 5 minutes (for a
single OSD that was down for 60 seconds). A host with two SSDs brought
our cluster to less than 2K IOPs for 15 minutes and took ten minutes
to get back to normal performance.

[1] http://docs.ceph.com/docs/v0.94/rados/configuration/osd-config-ref/#recovery

Thanks,
- ----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
-----BEGIN PGP SIGNATURE-----
Version: Mailvelope v1.2.3
Comment: https://www.mailvelope.com

wsFcBAEBCAAQBQJWS9EvCRDmVDuy+mK58QAAzE0P/3jQt3RkDUetTyuu/E3v
wVwBtcxONs7RQHIEtamNk/eIoGsSS+PevsBK2hSvnIJWNZkhQ3U13HQQ7Hz1
awkVD3+nw72You09kC772MtAXOIcHDEQgzJHQGoxevLlJSRwIarzyMlkJqrP
g+WdAx+O3BjtOoPG+6SG1HMDqUjTw46yHkCC2iybjT9y7PBp6PZ8EN1GD+00
k2+FferROKg/VxKxwQmgWVlXIvnrSF2/bHuZeTOUybw7TWNt1q6ZSXr4ZZuY
1e0yUnj8lNMus3SC6Itdj9wBp6Ke1J4tdUZkWiTgMkK5Xykw6iAJCADPIrni
zck3SfI2XB8XXrNwvuEvuKyAleXAodPf/AbWQ9sfO88MoWFYZ3ibNbbIfAp4
SEKeZpipzxlvNCm/W2NiDD08jbcaYDqn6dj6fHSHvIelysRItLlojTXuAioZ
ORQ4JAxPnEfNCUtn/eAq46oVIjrmSPiHs2p2hMYjhANLNYz5tyAt/HNSHXzR
hnYH9y4TFIOyrB7JcAypkIKwiuGjmoMbR8RvF1hDEJRXAzj7rpePQ9FoNbU/
/uGIJlwSPEJ8UxK1TuqDJ13XvXfLbR+S1aPjt+Y5LOMYO0pFo4fsdtt8NakM
ayRsUysGM9n7hBRTWHV5zPg7MB3wjcFv2kaE5NZfwfMNbM2xXkdClYHJ53Zr
kbpr
=BJTH
-----END PGP SIGNATURE-----
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux