On Monday, March 03/19/18, 2018 at 20:52:11 +0530, Stephen Hemminger wrote: > On Mon, 19 Mar 2018 13:25:56 +0530 > Rahul Lakkireddy <rahul.lakkireddy@xxxxxxxxxxx> wrote: > > > On Friday, March 03/16/18, 2018 at 16:42:03 +0530, Rahul Lakkireddy wrote: > > > On production servers running variety of workloads over time, kernel > > > panic can happen sporadically after days or even months. It is > > > important to collect as much debug logs as possible to root cause > > > and fix the problem, that may not be easy to reproduce. Snapshot of > > > underlying hardware/firmware state (like register dump, firmware > > > logs, adapter memory, etc.), at the time of kernel panic will be very > > > helpful while debugging the culprit device driver. > > > > > > This series of patches add new generic framework that enable device > > > drivers to collect device specific snapshot of the hardware/firmware > > > state of the underlying device in the crash recovery kernel. In crash > > > recovery kernel, the collected logs are exposed via /proc/crashdd/ > > > directory, which is copied by user space scripts for post-analysis. > > > > > > A kernel module crashdd is newly added. In crash recovery kernel, > > > crashdd exposes /proc/crashdd/ directory containing device specific > > > hardware/firmware logs. > > > > > > The sequence of actions done by device drivers to append their device > > > specific hardware/firmware logs to /proc/crashdd/ directory are as > > > follows: > > > > > > 1. During probe (before hardware is initialized), device drivers > > > register to the crashdd module (via crashdd_add_dump()), with > > > callback function, along with buffer size and log name needed for > > > firmware/hardware log collection. > > > > > > 2. Crashdd creates a driver's directory under /proc/crashdd/<driver>. > > > Then, it allocates the buffer with requested size and invokes the > > > device driver's registered callback function. > > > > > > 3. Device driver collects all hardware/firmware logs into the buffer > > > and returns control back to crashdd. > > > > > > 4. Crashdd exposes the buffer as a file via > > > /proc/crashdd/<driver>/<dump_file>. > > > > > > 5. User space script (/usr/lib/kdump/kdump-lib-initramfs.sh) copies > > > the entire /proc/crashdd/ directory to /var/crash/ directory. > > > > > > Patch 1 adds crashdd module to allow drivers to register callback to > > > collect the device specific hardware/firmware logs. The module also > > > exports /proc/crashdd/ directory containing the hardware/firmware logs. > > > > > > Patch 2 shows a cxgb4 driver example using the API to collect > > > hardware/firmware logs in crash recovery kernel, before hardware is > > > initialized. The logs for the devices are made available under > > > /proc/crashdd/cxgb4/ directory. > > > > > > Suggestions and feedback will be much appreciated. > > > > > > Thanks, > > > Rahul > > > > > > RFC v1: https://www.spinics.net/lists/netdev/msg486562.html > > > > > > --- > > > v2: > > > - Added new crashdd module that exports /proc/crashdd/ containing > > > driver's registered hardware/firmware logs in patch 1. > > > - Replaced the API to allow drivers to register their hardware/firmware > > > log collect routine in crash recovery kernel in patch 1. > > > - Updated patch 2 to use the new API in patch 1. > > > > > > Rahul Lakkireddy (2): > > > proc/crashdd: add API to collect hardware dump in second kernel > > > cxgb4: collect hardware dump in second kernel > > > > > > drivers/net/ethernet/chelsio/cxgb4/cxgb4.h | 4 + > > > drivers/net/ethernet/chelsio/cxgb4/cxgb4_cudbg.c | 25 +++ > > > drivers/net/ethernet/chelsio/cxgb4/cxgb4_cudbg.h | 3 + > > > drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c | 12 ++ > > > fs/proc/Kconfig | 11 + > > > fs/proc/Makefile | 1 + > > > fs/proc/crashdd.c | 263 +++++++++++++++++++++++ > > > include/linux/crashdd.h | 43 ++++ > > > 8 files changed, 362 insertions(+) > > > create mode 100644 fs/proc/crashdd.c > > > create mode 100644 include/linux/crashdd.h > > > > > > -- > > > 2.14.1 > > > > > > > Does anyone have any comments with this approach? If there are no > > comments, then I'll re-spin this RFC to Patch series. > > > > Thanks, > > Rahul > > This does look like it gives useful data, but it is not clear that this can > not already be done with existing API's or small extensions. > > Introducing a new /proc interface and one that is mostly device specific is > unlikely to be greeted with a warm reception by the current Linux kernel community. > > For example, getting firmware logs seems like something more related to > ethtool or sysfs. The /proc/crashdd/ is only exposed in 2nd (crash recovery) kernel. This is similar to /proc/vmcore. Since vmcore is exported via /proc/ interface, I've exported the device specific logs also via /proc/crashdd/. I can change to sysfs if that is the preferred approach. Does /sys/crashdd/ sound good? Thanks, Rahul _______________________________________________ kexec mailing list kexec@xxxxxxxxxxxxxxxxxxx http://lists.infradead.org/mailman/listinfo/kexec
