Hello, Slow requests, blocked requests and blocked ops occur quite often in our cluster; too often, I'd say: several times during one day. I must say we are running some tests, but we are far from pushing the cluster to the limit (or at least, that's what I believe). Every time a blocked request/operation happened, restarting the affected OSD solved the problem. Yesterday, we wanted to see if it was possible to minimize the impact that backfills and recovery have over normal cluster performace. In our case, performance dropped from 1000 cluster IOPS (approx) to 10 IOPS (approx) when doing some kind of recovery. Thus, we reduced the parameters "osd max backfills" and "osd recovery max active" to 1 (defaults are 10 and 15, respectively). Cluster performance during recovery improved to 500-600 IOPS (approx), and overall recovery time stayed approximately the same (surprisingly). Since then, we have had no more slow/blocked requests/ops (and our tests are still running). It is soon to say this, but my guess is that osds/disks in our cluster cannot cope with all I/O: network bandwidth is not an issue (10 GbE interconnection, graphs show network usage is under control all the time), but spindles are not high-performance (WD Green). Eventually, this might lead to slow/blocked requests/ops (which shouldn't occur that often). Reducing I/O pressure caused by recovery and backfill undoubtedly helped on improving cluster performance during recovery, that was expected. But we did not expect that recovery time stayed the same... The only explanation for this is that, during recovery, there are lots of operations that fail due a timeout, are retried several times, etc. So if disks are the bottleneck, reducing such values may help as well in normal cluster operation (when propagating the replicas, for instance). And slow/blocked requests/ops do not occur (or at least, occur less frequently). Does this make sense to you? Any other thoughts? Thank you very much again for your time. - Xavier Serrano - LCAC, Laboratori de C?lcul - Departament d'Arquitectura de Computadors, UPC
 |