Re: ❌ FAIL: Test report for kernel 5.6.0-rc4-61a0925.cki (mainline.kernel.org)

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi,
hoping this is a convenient solution for you, I have attached a
compressed archives with a pair of patches.  The first patch just does
a lot of checks while BFQ runs (BUG_ONs must be turned on for this to
work), while the second patch is a tentative fix.

Looking forward to your feedback,
Paolo

Attachment: debug_patches_for_5.6.tgz
Description: Binary data


> Il giorno 9 mar 2020, alle ore 18:09, Rachel Sibley <rasibley@xxxxxxxxxx> ha scritto:
> 
> 
> 
> On 3/9/20 12:42 PM, Paolo Valente wrote:
>> Hi Rachel,
>> IIUC, you can reproduce this bug reliably. If so, I'd need you to test a debugging patch (on top of one of the offending kernels).
> 
> Hi Paolo,
> 
> Yes seems we have seen it pretty consistently in the last three reports, but I'm cloning the job to be sure we can
> reproduce reliably. In the mean time, feel free to send me a pointer to your debugging patch so I can retry with
> the patch applied.
> 
> Thank you,
> Rachel
> 
>> Looking forward to your feedback,
>> Paolo
>>> Il giorno 9 mar 2020, alle ore 15:27, Rachel Sibley <rasibley@xxxxxxxxxx> ha scritto:
>>> 
>>> (cc'ing linux-block@xxxxxxxxxxxxxxx)
>>> 
>>> Hello,
>>> 
>>> We are seeing a kernel panic triggered with LTP and xfstests against a recent commit for mainline,
>>> wanted to share in case it's not already known.
>>> 
>>> Kernel repo: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
>>> Commit: 61a09258f2e5 - Merge tag 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma
>>> 
>>> We have also seen it with 2c523b344dfa and 378fee2e6b12 commits as well.
>>> 
>>> LTP: https://cki-artifacts.s3.us-east-2.amazonaws.com/datawarehouse/2020/03/08/477469/x86_64_1_console.log
>>> xfstests: https://cki-artifacts.s3.us-east-2.amazonaws.com/datawarehouse/2020/03/08/477469/x86_64_4_console.log
>>> 
>>> [-- MARK -- Sun Mar  8 02:45:00 2020]
>>> [  762.315610] BUG: kernel NULL pointer dereference, address: 0000000000000158
>>> [  762.323385] #PF: supervisor read access in kernel mode
>>> [  762.329119] #PF: error_code(0x0000) - not-present page
>>> [  762.334853] PGD 0 P4D 0
>>> [  762.337680] Oops: 0000 [#1] SMP PTI
>>> [  762.341575] CPU: 9 PID: 87 Comm: kworker/9:1 Not tainted 5.6.0-rc4-61a0925.cki #1
>>> [  762.349927] Hardware name: Cisco Systems, Inc. UCS-E160DP-M1/K9/UCS-E160DP-M1/K9, BIOS UCSED.1.5.0.2.051520131757 05/15/2013
>>> [  762.362453] Workqueue: cgroup_destroy css_killed_work_fn
>>> [  762.368387] RIP: 0010:bfq_bfqq_expire+0x1c/0x940
>>> [  762.373540] Code: 01 00 00 c7 80 f8 00 00 00 01 00 00 00 c3 66 66 66 66 90 41 57 41 56 41 55 41 54 41 89 cc 55 48 89 fd 53 48 89 f3 48 83 ec 28 <8b> be 58 01 00 00 65 48 8b 04 25 28 00 00 00 48 89 44 24 20 31 c0
>>> [  762.394500] RSP: 0018:ffff9927c03bbd50 EFLAGS: 00010086
>>> [  762.400331] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000004
>>> [  762.408301] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8965a3913800
>>> [  762.416270] RBP: ffff8965a3913800 R08: ffff896592d41098 R09: ffff89657aa8df00
>>> [  762.424233] R10: 0000000000000000 R11: ffff89657aa8df00 R12: 0000000000000004
>>> [  762.432200] R13: ffff89659f0cd9b0 R14: ffff8965a3913bf0 R15: ffff89659f0cd898
>>> [  762.440175] FS:  0000000000000000(0000) GS:ffff8965a7c40000(0000) knlGS:0000000000000000
>>> [  762.449211] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>> [  762.455622] CR2: 0000000000000158 CR3: 000000065afc6003 CR4: 00000000000606e0
>>> [  762.463599] Call Trace:
>>> [  762.466341]  ? bfq_idle_extract+0x40/0xb0
>>> [  762.470821]  bfq_bfqq_move+0x14f/0x160
>>> [  762.475011]  bfq_pd_offline+0xd3/0xf0
>>> [  762.479112]  blkg_destroy+0x52/0xf0
>>> [  762.483005]  blkcg_destroy_blkgs+0x4f/0xa0
>>> [  762.487582]  css_killed_work_fn+0x4d/0xd0
>>> [  762.492066]  process_one_work+0x1b5/0x360
>>> [  762.496547]  worker_thread+0x50/0x3c0
>>> [  762.500641]  kthread+0xf9/0x130
>>> [  762.504153]  ? process_one_work+0x360/0x360
>>> [  762.508813]  ? kthread_park+0x90/0x90
>>> [  762.512909]  ret_from_fork+0x35/0x40
>>> 
>>> Thanks,
>>> Rachel
>>> 
>>> On 3/7/20 9:59 PM, CKI Project wrote:
>>>> Hello,
>>>> We ran automated tests on a recent commit from this kernel tree:
>>>>        Kernel repo: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
>>>>             Commit: 61a09258f2e5 - Merge tag 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma
>>>> The results of these automated tests are provided below.
>>>>     Overall result: FAILED (see details below)
>>>>              Merge: OK
>>>>            Compile: OK
>>>>              Tests: FAILED
>>>> All kernel binaries, config files, and logs are available for download here:
>>>>   https://cki-artifacts.s3.us-east-2.amazonaws.com/index.html?prefix=datawarehouse/2020/03/08/477469
>>>> One or more kernel tests failed:
>>>>     x86_64:
>>>>      ❌ LTP
>>>>      ❌ xfstests - ext4
>>>> We hope that these logs can help you find the problem quickly. For the full
>>>> detail on our testing procedures, please scroll to the bottom of this message.
>>>> Please reply to this email if you have any questions about the tests that we
>>>> ran or if you have any suggestions on how to make future tests more effective.
>>>>         ,-.   ,-.
>>>>        ( C ) ( K )  Continuous
>>>>         `-',-.`-'   Kernel
>>>>           ( I )     Integration
>>>>            `-'
>>>> ______________________________________________________________________________
>>>> Compile testing
>>>> ---------------
>>>> We compiled the kernel for 1 architecture:
>>>>     x86_64:
>>>>       make options: -j30 INSTALL_MOD_STRIP=1 targz-pkg
>>>> Hardware testing
>>>> ----------------
>>>> We booted each kernel and ran the following tests:
>>>>   x86_64:
>>>>     Host 1:
>>>>        ✅ Boot test
>>>>        ✅ Podman system integration test - as root
>>>>        ✅ Podman system integration test - as user
>>>>        ❌ LTP
>>>>        ⚡⚡⚡ Loopdev Sanity
>>>>        ⚡⚡⚡ Memory function: memfd_create
>>>>        ⚡⚡⚡ AMTU (Abstract Machine Test Utility)
>>>>        ⚡⚡⚡ Networking bridge: sanity
>>>>        ⚡⚡⚡ Ethernet drivers sanity
>>>>        ⚡⚡⚡ Networking MACsec: sanity
>>>>        ⚡⚡⚡ Networking socket: fuzz
>>>>        ⚡⚡⚡ Networking sctp-auth: sockopts test
>>>>        ⚡⚡⚡ Networking: igmp conformance test
>>>>        ⚡⚡⚡ Networking route: pmtu
>>>>        ⚡⚡⚡ Networking route_func - local
>>>>        ⚡⚡⚡ Networking route_func - forward
>>>>        ⚡⚡⚡ Networking TCP: keepalive test
>>>>        ⚡⚡⚡ Networking UDP: socket
>>>>        ⚡⚡⚡ Networking tunnel: geneve basic test
>>>>        ⚡⚡⚡ Networking tunnel: gre basic
>>>>        ⚡⚡⚡ L2TP basic test
>>>>        ⚡⚡⚡ Networking tunnel: vxlan basic
>>>>        ⚡⚡⚡ Networking ipsec: basic netns - transport
>>>>        ⚡⚡⚡ Networking ipsec: basic netns - tunnel
>>>>        ⚡⚡⚡ audit: audit testsuite test
>>>>        ⚡⚡⚡ httpd: mod_ssl smoke sanity
>>>>        ⚡⚡⚡ tuned: tune-processes-through-perf
>>>>        ⚡⚡⚡ pciutils: sanity smoke test
>>>>        ⚡⚡⚡ ALSA PCM loopback test
>>>>        ⚡⚡⚡ ALSA Control (mixer) Userspace Element test
>>>>        ⚡⚡⚡ storage: SCSI VPD
>>>>        ⚡⚡⚡ trace: ftrace/tracer
>>>>        🚧 ⚡⚡⚡ CIFS Connectathon
>>>>        🚧 ⚡⚡⚡ POSIX pjd-fstest suites
>>>>        🚧 ⚡⚡⚡ jvm - DaCapo Benchmark Suite
>>>>        🚧 ⚡⚡⚡ jvm - jcstress tests
>>>>        🚧 ⚡⚡⚡ Memory function: kaslr
>>>>        🚧 ⚡⚡⚡ LTP: openposix test suite
>>>>        🚧 ⚡⚡⚡ Networking vnic: ipvlan/basic
>>>>        🚧 ⚡⚡⚡ iotop: sanity
>>>>        🚧 ⚡⚡⚡ Usex - version 1.9-29
>>>>        🚧 ⚡⚡⚡ storage: dm/common
>>>>     Host 2:
>>>>        ✅ Boot test
>>>>        ✅ Storage SAN device stress - mpt3sas driver
>>>>     Host 3:
>>>>        ✅ Boot test
>>>>        ✅ Storage SAN device stress - megaraid_sas
>>>>     Host 4:
>>>>        ✅ Boot test
>>>>        ❌ xfstests - ext4
>>>>        ⚡⚡⚡ xfstests - xfs
>>>>        ⚡⚡⚡ selinux-policy: serge-testsuite
>>>>        ⚡⚡⚡ lvm thinp sanity
>>>>        ⚡⚡⚡ storage: software RAID testing
>>>>        ⚡⚡⚡ stress: stress-ng
>>>>        🚧 ⚡⚡⚡ IOMMU boot test
>>>>        🚧 ⚡⚡⚡ IPMI driver test
>>>>        🚧 ⚡⚡⚡ IPMItool loop stress test
>>>>        🚧 ⚡⚡⚡ power-management: cpupower/sanity test
>>>>        🚧 ⚡⚡⚡ Storage blktests
>>>>   Test sources: https://github.com/CKI-project/tests-beaker
>>>>     💚 Pull requests are welcome for new tests or improvements to existing tests!
>>>> Waived tests
>>>> ------------
>>>> If the test run included waived tests, they are marked with 🚧. Such tests are
>>>> executed but their results are not taken into account. Tests are waived when
>>>> their results are not reliable enough, e.g. when they're just introduced or are
>>>> being fixed.
>>>> Testing timeout
>>>> ---------------
>>>> We aim to provide a report within reasonable timeframe. Tests that haven't
>>>> finished running yet are marked with ⏱.
>>> 
> 


[Index of Archives]     [Linux RAID]     [Linux SCSI]     [Linux ATA RAID]     [IDE]     [Linux Wireless]     [Linux Kernel]     [ATH6KL]     [Linux Bluetooth]     [Linux Netdev]     [Kernel Newbies]     [Security]     [Git]     [Netfilter]     [Bugtraq]     [Yosemite News]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Device Mapper]

  Powered by Linux