Thanks a lot, Petr and Greg. I saw the patches and just started to prepare the test on devices using the 4.14.y tree. will update the test result later. @Greg Kroah-Hartman Sorry for not providing the details in the public thread. I can sync w/ you offline. Thanks, Yi Fan On Tue, Nov 9, 2021 at 7:28 AM Petr Mladek <pmladek@xxxxxxxx> wrote: > > On Tue 2021-11-09 07:27:35, Greg KH wrote: > > On Mon, Nov 08, 2021 at 11:17:07AM -0800, Yi Fan wrote: > > > On Mon, Nov 8, 2021 at 12:00 AM Greg KH <gregkh@xxxxxxxxxxxxxxxxxxx> wrote: > > > > > > > > On Thu, Nov 04, 2021 at 12:40:32PM -0700, Yi Fan wrote: > > > > > Reply inline. > > > > > > > > > > On Thu, Nov 4, 2021 at 11:56 AM Greg KH <gregkh@xxxxxxxxxxxxxxxxxxx> wrote: > > > > > > > > > > > > On Thu, Nov 04, 2021 at 11:14:55AM -0700, Yi Fan wrote: > > > > > > > Resend the email using plain text. > > > > > > > > > > > > > > I found some kernel performance regression issues that might be > > > > > > > related w/ 4.14.y LTS commit. > > > > > > > > > > > > > > 4.14.y commit: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=v4.14.253&id=27d185697322f9547bfd381c71252ce0bc1c0ee4 > > > > > > > > > > > > > > The issue is observed when "console=" is used as a kernel parameter to > > > > > > > disable the kernel console. > > I think that I see the problem. linux-4.14.y stable branch currently > ignores "console=" parameter. As a result, a console (ttyX) is enabled > by default. > > > > > > > What exact "performance issue" are you seeing? > > > > > > > > > > > [YF] one kernel thread was randomly blocked for more than ~40 > > > > > milliseconds, causing a certain task to fail to process in time. > > > > > [YF] the issue is highly random on a single device. But it might > > > > > happen a few times per 24 hours on a certain percentage of devices. > > > > > The overall percentage of devices that show the issue seems quite > > > > > stable over a long period of time (somehow the magic number is ~40%.). > > > > > [YF] local test on a pool of devices does not show any correlation w/ > > > > > any particular devices. > > This might happen when there is a flood of messages to be printed to > the console. It does not happen when there is no console. > > It has been fixed by the upstream commit 3cffa06aeef7ece30f6b5ac0 > ("printk/console: Allow to disable console output by using console="" > or console=null") > > The fix needs some tweaking for the stable branches because > __add_preferred_console() has gained more parameters over time. > > It seems that all longterm stable branches are affected. I am going > to prepare the backports. > > Best Regards, > Petr
