On Tue, Apr 6, 2021 at 4:20 PM Chuck Lever III <chuck.lever@xxxxxxxxxx> wrote: > > > > > On Apr 6, 2021, at 7:50 AM, Gioh Kim <gi-oh.kim@xxxxxxxxx> wrote: > > > > My colleagues and I would like to apply the fault injection > > of the Linux to test error handling of RTRS module. RTRS module > > consists of client and server modules that are connected via > > Infiniband network. So it is important for the client to receive > > the error of the server and handle it smoothly. > > I am a fan of fault injection. In fact I added a disconnect fault > injector for RPC that's in the kernel now, and it uses debugfs > as its control interface. > > But that was years ago. If I were doing this today, I'd consider > kprobes, since fault injection is generally not something that > is consumed by users or administrators in a distributed kernel. > > Have you considered injection via kprobes or eBPF instead of > adding permanent code? I have not considered the eBPF yet. I will have a discussion with my colleagues about that. Thank you for the information. > > > > When debugfs is enabled, RTRS is able to export interfaces > > to fail RTRS client and server. > > Following fault injection points are enabled: > > - fail a request processing on RTRS client side > > - fail a heart-beat transferation on RTRS server side > > > > This patch set is just a starting point. We will enable various > > faults and test as many error cases as possible. > > > > Best regards > > > > Gioh Kim (4): > > RDMA/rtrs: Enable the fault-injection > > RDMA/rtrs-clt: Inject a fault at request processing > > RDMA/rtrs-srv: Inject a fault at heart-beat sending > > docs: fault-injection: Add fault-injection manual of RTRS > > > > .../fault-injection/rtrs-fault-injection.rst | 83 +++++++++++++++++++ > > drivers/infiniband/ulp/rtrs/Makefile | 2 + > > drivers/infiniband/ulp/rtrs/rtrs-clt-sysfs.c | 44 ++++++++++ > > drivers/infiniband/ulp/rtrs/rtrs-clt.c | 7 ++ > > drivers/infiniband/ulp/rtrs/rtrs-clt.h | 13 +++ > > drivers/infiniband/ulp/rtrs/rtrs-fault.c | 52 ++++++++++++ > > drivers/infiniband/ulp/rtrs/rtrs-fault.h | 28 +++++++ > > drivers/infiniband/ulp/rtrs/rtrs-srv-sysfs.c | 44 ++++++++++ > > drivers/infiniband/ulp/rtrs/rtrs-srv.c | 5 ++ > > drivers/infiniband/ulp/rtrs/rtrs-srv.h | 13 +++ > > 10 files changed, 291 insertions(+) > > create mode 100644 Documentation/fault-injection/rtrs-fault-injection.rst > > create mode 100644 drivers/infiniband/ulp/rtrs/rtrs-fault.c > > create mode 100644 drivers/infiniband/ulp/rtrs/rtrs-fault.h > > > > -- > > 2.25.1 > > > > -- > Chuck Lever > > >
