Hi Corey Thanks very much for your reply! I use raw_spinlock in test case, it hangs the system forever, without triggering softlock detector. Is it just as expected? <<The good news for you is that it should be entirely possible for you to trigger the softlockup detector on RT entirely in usermode. All you'd need to do is disable RT throttling, and set a thread SCHED_FIFO w/ prio 99 spinning.>> I am a bit lost here, in test case I had already test SCHED_FIFO & 99 prio with echo -1 > /proc/sys/kernel/sched_rt_runtime_us But it hangs again without triggering softlock detector. In production environment, our servers with rt patch hangs occasionally. Since no kdump triggered, so it's hard to find the root cause. Please help me!! Thanks all very much. Thanks very much. On Thu, Aug 17, 2017 at 8:19 PM, Corey Minyard <cminyard@xxxxxxxxxx> wrote: > On 08/17/2017 05:05 AM, Journey Journey wrote: >> >> Anyone reproduce this problem? > > > You need to read Julia's previous email on this. I think it has the answers > you need. > > -corey > > >> Thanks >> >> On Wed, Aug 16, 2017 at 12:39 PM, Journey Journey >> <journeywang123@xxxxxxxxx> wrote: >>> >>> Thank you all for replying my question. The Details following: >>> >>> Kernel Version: 4.4.32 from >>> https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.4.32.tar.xz >>> Rt Patch Version: 4.4.32-rt43 from >>> >>> https://www.kernel.org/pub/linux/kernel/projects/rt/4.4/older/patch-4.4.32-rt43.patch.gz >>> >>> Configs as attached, >>> (1) config.4.4.32 for non rt kernel >>> (2) config.4.4.32.rt for rt kernel >>> >>> Testcase code as attached, >>> (1) softlockup-nonrt.c for non rt kernel; >>> (2) softlockup-rt.c for rt kernel, using raw_spinlock >>> (3) softlockup-rt-fifo.c for rt kernel, using sched_fifo & priority 99 >>> (4) sample Makefile for building kernel module >>> >>> >>> Test Case 1: non rt kernel + softlockup-nonrt.c >>> Picture attached nonrt-1.png >>> The kernel had no response for keyboard input.... >>> Picture attached nonrt-2.png >>> After several minutes, the kdump is triggered. >>> >>> Test Case 2: rt kernel + softlockup-rt.c >>> Picture attached rt-1.png >>> The kernel had no response for keyboard input.... >>> Hangs forever ........ >>> >>> Test Case 3: rt kernel + softlockup-rt-fifo.c >>> Picture attached rt-2.png >>> The kernel had no response for keyboard input.... >>> Hangs forever ........ >>> >>> Test Case 4: rt kernel + softlockup-rt-fifo.c + throttling >>> Picture attached rt-3.png >>> The kernel had no response for keyboard input.... >>> Hangs forever ........ >>> >>> By the way, my test environment is virtualbox with ubuntu. >>> Virtualbox Configration: >>> Picture attached Virtualbox-1.png Virtualbox-2.png Virtualbox-3.png >>> >>> crashkernel config >>> crashkernel=512M >>> >>> Please help me, thanks all!! >>> >>> >>> On Wed, Aug 9, 2017 at 11:35 PM, Julia Cartwright <julia@xxxxxx> wrote: >>>> >>>> On Wed, Aug 09, 2017 at 05:05:42PM +0800, Journey Journey wrote: >>>>> >>>>> Test case: >>>>> https://github.com/saiyamd/softlockup_test/blob/master/softlockup_test.c >>>>> >>>> Please in the future provide more information as to what you are trying >>>> to do, what it is you are trying, what you are seeing, and why you think >>>> it's wrong. >>>> >>>> Because otherwise, we're forced to reverse engineer your intent. :( >>>> >>>> I'm _guessing_ that what you meant to say was: >>>> >>>> I have a testcase module that I use that attempts to trigger the >>>> kernels softlockup detection. Here is a link: >>>> >>>> >>>> https://github.com/saiyamd/softlockup_test/blob/master/softlockup_test.c >>>> >>>> I'm expecting it to fire the softlockup detector and panic the >>>> system >>>> (I use CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC in my test rig), however, >>>> when I load this module on an RT system, it doesn't appear to >>>> trigger >>>> the softlockup detector![1] >>>> >>>> What could be wrong? >>>> >>>> If that's what you meant, then the answer is: >>>> >>>> The way that you are "hogging" the CPU, on PREEMPT_RT, doesn't prevent >>>> the kernel from scheduling other tasks in and out (including the >>>> watchdog thread which pets the watchdog!). >>>> >>>> Your test case, effectively is: >>>> >>>> spin_lock(&lock) >>>> while (1) >>>> printk("..."); >>>> spin_unlock(&lock) >>>> >>>> On RT, spin-lock protected regions are preemptible. When the watchdog >>>> hrtimer fires and wakes up the softlockup thread, that thread preempts >>>> the thread stuck in this region and touches the watchdog. This prevents >>>> the watchdog threshold from being reached. >>>> >>>> The good news for you is that it should be entirely possible for you to >>>> trigger the softlockup detector on RT entirely in usermode. All you'd >>>> need to do is disable RT throttling, and set a thread SCHED_FIFO w/ prio >>>> 99 spinning. >>>> >>>> Julia >>>> >>>> 1: Note: this has nothing to do with Kdump whatsoever. > > > -- To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
