On Wed, 13 May 2015, Vasiliy Angapov wrote: > Hi, > > Well, I've managed to find out that correct stop of osd causes no IO > downtime (/etc/init.d/ceph stop osd). But that cannot be called a fault > tolerance, which Ceph is supposed to be.However, "killall -9 ceph-osd" > causes IO to stop for about 20 seconds. > > I've tried lowering some timeouts but without luck. Here is a related part > of my ceph.conf after lowering the timeout values: > > [global] > heartbeat interval = 5 > mon osd down out interval = 90 > mon pg warn max per osd = 2000 > mon osd adjust heartbeat grace = false > > [client] > rbd cache = false > > [mon] > mon clock drift allowed = .200 > mon osd min down reports = 1 > > [osd] > osd heartbeat interval = 3 > osd heartbeat grace = 5 > > Can you help me to reduce IO downtime somehow? Because 20 seconds for > production is just horrible. You'll need to restart ceph-osd daemons for that change to take effect, or ceph tell osd.\* injectargs '--osd-heartbeat-grace 5 --osd-heartbeat-interval 1' Just remember that this timeout is a tradeoff against false positives--be careful tuning it too low. Note that ext4 going ro after 5 seconds sounds like insanity to me. I've only seen this with older guest kernels, and iirc the problem is a 120s timeout with ide or something? Ceph is a CP system that trades availability for consistency--it will block IO as needed to ensure that it is handling reads or writes in a completely consistent manner. Even if you get the failure detection latency down, other recovery scenarios are likely to cross the magic 5s threshold at some point and cause the same problem. You need to fix your guests one way or another! sage > > Regards, Vasily. > > > On Wed, May 13, 2015 at 9:57 AM, Vasiliy Angapov <angapov@xxxxxxxxx> wrote: > Thanks, Gregory! > My Ceph version is 0.94.1. What I'm trying to test is the worst > situation when the node is loosing network or becomes inresponsive. So > what i do is "killall -9 ceph-osd", then reboot. > > Well, I also tried to do a clean reboot several times (just a "reboot" > command), but i saw no difference - there is always an IO freeze for > about 30 seconds. Btw, i'm using Fedora 20 on all nodes. > > Ok, I will play with timeouts more. > > Thanks again! > > On Wed, May 13, 2015 at 10:46 AM, Gregory Farnum <greg@xxxxxxxxxxx> > wrote: > On Tue, May 12, 2015 at 11:39 PM, Vasiliy Angapov > <angapov@xxxxxxxxx> wrote: > > Hi, colleagues! > > > > I'm testing a simple Ceph cluster in order to use it in > production > > environment. I have 8 OSDs (1Tb SATA drives) which are > evenly distributed > > between 4 nodes. > > > > I'v mapped rbd image on the client node and started > writing a lot of data to > > it. Then I just reboot one node and see what's > happening. What happens is > > very sad. I have a write freeze for about 20-30 seconds > which is enough for > > ext4 filesystem to switch to RO. > > > > I wonder, if there is any way to minimize this lag? > AFAIK, ext filesystems > > have 5 seconds timeout before switching to RO. So is > there any way to get > > that lag beyond 5 secs? I've tried lowering different > osd timeouts, but it > > doesn't seem to help. > > > > How do you deal with such a situations? 20 seconds of > downtime is not > > tolerable in production. > > What version of Ceph are you running, and how are you rebooting > it? > Any newish version that gets a clean reboot will notify the > cluster > that it's shutting down, so you shouldn't witness blocked rights > really at all. > > If you're doing a reboot that involves just ending the daemon, > you > will have to wait through the timeout period before the OSD gets > marked down, which defaults to 30 seconds. This is adjustable > (look > for docs on the "osd heartbeat grace" config option), although > if you > set it too low you'll need to change a bunch of other timeouts > which I > don't know off-hand... > -Greg > > > > >
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
