On 18/12/2018 18:28, Oliver Freyermuth wrote:
We have yet to observe these hangs, we are running this with ~5 VMs with ~10 disks for about half a year now with daily snapshots. But all of these VMs have very "low" I/O, since we put anything I/O intensive on bare metal (but with automated provisioning of course). So I'll chime in on your question, especially since there might be VMs on our cluster in the future where the inner OS may not be running an agent. Since we did not observe this yet, I'll also add: What's your "scale", is it hundreds of VMs / disks? Hourly snapshots? I/O intensive VMs?
5 hosts, 15 VMs, daily snapshots. I/O is variable (customer workloads); usually not that high, but it can easily peak at 100% when certain things happen. We don't have great I/O performance (RBD over 1gbps links to HDD OSDs).
I'm poring through monitoring graphs now and I think the issue this time around was just too much dirty data in the page cache of a guest. The VM that failed spent 3 minutes flushing out writes to disk before its I/O was quiesced, at around 100 IOPS throughput (the actual data throughput was low, though, so small writes). That exceeded our timeout and then things went south from there.
I wasn't sure if fsfreeze did a full sync to disk, but given the I/O behavior I'm seeing that seems to be the case. Unfortunately coming up with an upper bound for the freeze time seems tricky now. I'm increasing our timeout to 15 minutes, we'll see if the problem recurs.
Given this, it makes even more sense to just avoid the freeze if at all reasonable. There's no real way to guarantee that a fsfreeze will complete in a "reasonable" amount of time as far as I can tell.
-- Hector Martin (hector@xxxxxxxxxxxxxx) Public Key: https://marcan.st/marcan.asc _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
