Thanks, Gregory!
My Ceph version is 0.94.1. What I'm trying to test is the worst situation when the node is loosing network or becomes inresponsive. So what i do is "killall -9 ceph-osd", then reboot.
Well, I also tried to do a clean reboot several times (just a "reboot" command), but i saw no difference - there is always an IO freeze for about 30 seconds. Btw, i'm using Fedora 20 on all nodes.
Ok, I will play with timeouts more.
Thanks again!
On Wed, May 13, 2015 at 10:46 AM, Gregory Farnum <greg@xxxxxxxxxxx> wrote:
What version of Ceph are you running, and how are you rebooting it?On Tue, May 12, 2015 at 11:39 PM, Vasiliy Angapov <angapov@xxxxxxxxx> wrote:
> Hi, colleagues!
>
> I'm testing a simple Ceph cluster in order to use it in production
> environment. I have 8 OSDs (1Tb SATA drives) which are evenly distributed
> between 4 nodes.
>
> I'v mapped rbd image on the client node and started writing a lot of data to
> it. Then I just reboot one node and see what's happening. What happens is
> very sad. I have a write freeze for about 20-30 seconds which is enough for
> ext4 filesystem to switch to RO.
>
> I wonder, if there is any way to minimize this lag? AFAIK, ext filesystems
> have 5 seconds timeout before switching to RO. So is there any way to get
> that lag beyond 5 secs? I've tried lowering different osd timeouts, but it
> doesn't seem to help.
>
> How do you deal with such a situations? 20 seconds of downtime is not
> tolerable in production.
Any newish version that gets a clean reboot will notify the cluster
that it's shutting down, so you shouldn't witness blocked rights
really at all.
If you're doing a reboot that involves just ending the daemon, you
will have to wait through the timeout period before the OSD gets
marked down, which defaults to 30 seconds. This is adjustable (look
for docs on the "osd heartbeat grace" config option), although if you
set it too low you'll need to change a bunch of other timeouts which I
don't know off-hand...
-Greg
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
