On 2020/2/18 22:59, Ilya Dryomov wrote:
On Tue, Feb 18, 2020 at 1:01 PM Jeff Layton <jlayton@xxxxxxxxxx> wrote:
On Tue, 2020-02-18 at 15:19 +0800, Xiubo Li wrote:
On 2020/2/17 21:04, Jeff Layton wrote:
On Sun, 2020-02-16 at 01:49 -0500, xiubli@xxxxxxxxxx wrote:
From: Xiubo Li <xiubli@xxxxxxxxxx>
This will simulate pulling the power cable situation, which will
do:
- abort all the inflight osd/mds requests and fail them with -EIO.
- reject any new coming osd/mds requests with -EIO.
- close all the mds connections directly without doing any clean up
and disable mds sessions recovery routine.
- close all the osd connections directly without doing any clean up.
- set the msgr as stopped.
URL: https://tracker.ceph.com/issues/44044
Signed-off-by: Xiubo Li <xiubli@xxxxxxxxxx>
There is no explanation of how to actually _use_ this feature? I assume
you have to remount the fs with "-o remount,halt" ? Is it possible to
reenable the mount as well? If not, why keep the mount around? Maybe we
should consider wiring this in to a new umount2() flag instead?
This needs much better documentation.
In the past, I've generally done this using iptables. Granted that that
is difficult with a clustered fs like ceph (given that you potentially
have to set rules for a lot of addresses), but I wonder whether a scheme
like that might be more viable in the long run.
How about fulfilling the DROP iptable rules in libceph ? Could you
foresee any problem ? This seems the one approach could simulate pulling
the power cable.
Yeah, I've mostly done this using DROP rules when I needed to test things.
But, I think I was probably just guilty of speculating out loud here.
I'm not sure what exactly Xiubo meant by "fulfilling" iptables rules
in libceph, but I will say that any kind of iptables manipulation from
within libceph is probably out of the question.
Sorry for confusing here.
I meant in libceph add some helpers to enable/disable dropping the any
new coming packet on the floor without responding anything to the ceph
cluster for a specified session.
I think doing this by just closing down the sockets is probably fine. I
wouldn't pursue anything relating to to iptables here, unless we have
some larger reason to go that route.
IMO investing into a set of iptables and tc helpers for teuthology
makes a _lot_ of sense. It isn't exactly the same as a cable pull,
but it's probably the next best thing. First, it will be external to
the system under test. Second, it can be made selective -- you can
cut a single session or all of them, simulate packet loss and latency
issues, etc. Third, it can be used for recovery and failover/fencing
testing -- what happens when these packets get delivered two minutes
later? None of this is possible with something that just attempts to
wedge the mount and acts as a point of no return.
Yeah, cool and this is what the tracker#44044 intends to.
Thanks
BRs
Xiubo
Thanks,
Ilya
