> On May 4, 2023, at 3:02 PM, Chuck Lever III <chuck.lever@xxxxxxxxxx> wrote: > > > >> On May 4, 2023, at 3:29 AM, Leon Romanovsky <leon@xxxxxxxxxx> wrote: >> >> On Wed, May 03, 2023 at 02:02:33PM +0000, Chuck Lever III wrote: >>> >>> >>>> On May 3, 2023, at 2:34 AM, Eli Cohen <elic@xxxxxxxxxx> wrote: >>>> >>>> Hi Chuck, >>>> >>>> Just verifying, could you make sure your server and card firmware are up to date? >>> >>> Device firmware updated to 16.35.2000; no change. >>> >>> System firmware is dated September 2016. I'll see if I can get >>> something more recent installed. >> >> We are trying to reproduce this issue internally. > > More information. I captured the serial console during boot. > Here are the last messages: > > [ 9.837087] mlx5_core 0000:02:00.0: firmware version: 16.35.2000 > [ 9.843126] mlx5_core 0000:02:00.0: 126.016 Gb/s available PCIe bandwidth (8.0 GT/s PCIe x16 link) > [ 10.311515] mlx5_core 0000:02:00.0: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps > [ 10.321948] mlx5_core 0000:02:00.0: E-Switch: Total vports 2, per vport: max uc(128) max mc(2048) > [ 10.344324] mlx5_core 0000:02:00.0: mlx5_pcie_event:301:(pid 88): PCIe slot advertised sufficient power (27W). > [ 10.354339] BUG: unable to handle page fault for address: ffffffff8ff0ade0 > [ 10.361206] #PF: supervisor read access in kernel mode > [ 10.366335] #PF: error_code(0x0000) - not-present page > [ 10.371467] PGD 81ec39067 P4D 81ec39067 PUD 81ec3a063 PMD 114b07063 PTE 800ffff7e10f5062 > [ 10.379544] Oops: 0000 [#1] PREEMPT SMP PTI > [ 10.383721] CPU: 0 PID: 117 Comm: kworker/0:6 Not tainted 6.3.0-13028-g7222f123c983 #1 > [ 10.391625] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0b 06/12/2017 > [ 10.398750] Workqueue: events work_for_cpu_fn > [ 10.403108] RIP: 0010:__bitmap_or+0x10/0x26 > [ 10.407286] Code: 85 c0 0f 95 c0 c3 cc cc cc cc 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 89 c9 31 c0 48 83 c1 3f 48 c1 e9 06 39 c8 73 11 <4c> 8b 04 c6 4c 0b 04 c2 4c 89 04 c7 48 ff c0 eb eb c3 cc cc cc cc > [ 10.426024] RSP: 0000:ffffb45a0078f7b0 EFLAGS: 00010097 > [ 10.431240] RAX: 0000000000000000 RBX: ffffffff8ff0adc0 RCX: 0000000000000004 > [ 10.438365] RDX: ffff9156801967d0 RSI: ffffffff8ff0ade0 RDI: ffff9156801967b0 > [ 10.445489] RBP: ffffb45a0078f7e8 R08: 0000000000000030 R09: 0000000000000000 > [ 10.452613] R10: 0000000000000000 R11: 0000000000000000 R12: 00000000000000ec > [ 10.459737] R13: ffffffff8ff0ade0 R14: 0000000000000001 R15: 0000000000000020 > [ 10.466862] FS: 0000000000000000(0000) GS:ffff9165bfc00000(0000) knlGS:0000000000000000 > [ 10.474936] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 10.480674] CR2: ffffffff8ff0ade0 CR3: 00000001011ae003 CR4: 00000000003706f0 > [ 10.487800] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > [ 10.494922] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > [ 10.502046] Call Trace: > [ 10.504493] <TASK> > [ 10.506589] ? matrix_alloc_area.constprop.0+0x43/0x9a > [ 10.511729] ? prepare_namespace+0x84/0x174 > [ 10.515914] irq_matrix_reserve_managed+0x56/0x10c > [ 10.520699] x86_vector_alloc_irqs+0x1d2/0x31e > [ 10.525146] irq_domain_alloc_irqs_hierarchy+0x39/0x3f > [ 10.530284] irq_domain_alloc_irqs_parent+0x1a/0x2a > [ 10.535155] intel_irq_remapping_alloc+0x59/0x5e9 > [ 10.539859] ? kmem_cache_debug_flags+0x11/0x26 > [ 10.544383] ? __radix_tree_lookup+0x39/0xb9 > [ 10.548649] irq_domain_alloc_irqs_hierarchy+0x39/0x3f > [ 10.553779] irq_domain_alloc_irqs_parent+0x1a/0x2a > [ 10.558650] msi_domain_alloc+0x8c/0x120 > [ rqs_hierarchy+0x39/0x3f > [ 10.567697] irq_domain_alloc_irqs_locked+0x11d/0x286 > [ 10.572741] __irq_domain_alloc_irqs+0x72/0x93 > [ 10.577179] __msi_domain_alloc_irqs+0x193/0x3f1 > [ 10.581789] ? __xa_alloc+0xcf/0xe2 > [ 10.585273] msi_domain_alloc_irq_at+0xa8/0xfe > [ 10.589711] pci_msix_alloc_irq_at+0x47/0x5c > [ 10.593987] mlx5_irq_alloc+0x99/0x319 [mlx5_core] > [ 10.598881] ? xa_load+0x5e/0x68 > [ 10.602112] irq_pool_request_vector+0x60/0x7d [mlx5_core] > [ 10.607668] mlx5_irq_request+0x26/0x98 [mlx5_core] > [ 10.612617] mlx5_irqs_request_vectors+0x52/0x82 [mlx5_core] > [ 10.618345] mlx5_eq_table_create+0x613/0x8d3 [mlx5_core] > [ 10.623806] ? kmalloc_trace+0x46/0x57 > [ 10.627549] mlx5_load+0xb1/0x33e [mlx5_core] > [ 10.631971] mlx5_init_one+0x497/0x514 [mlx5_core] > [ 10.636824] probe_one+0x2fa/0x3f6 [mlx5_core] > [ 10.641330] local_pci_probe+0x47/0x8b > [ 10.645073] work_for_cpu_fn+0x1a/0x25 > [ 10.648817] process_one_work+0x1e0/0x2e0 > [ 10.652822] process_scheduled_works+0x2c/0x37 > [ 10.657258] worker_thread+0x1e2/0x25e > [ 10.661003] ? __pfx_worker_thread+0x10/0x10 > [ 10.665267] kthread+0x10d/0x115 > [ 10.668501] ? __pfx_kthread+0x10/0x10 > [ 10.672244] ret_from_fork+0x2c/0x50 > [ 10.675824] </TASK> > [ 10.678007] Modules linked in: mlx5_core(+) ast drm_kms_helper crct10dif_pclmul crc32_pclmul drm_shmem_helper crc32c_intel drm ghash_clmulni_intel sha512_ssse3 igb dca i2c_algo_bit mlxfw pci_hyperv_intf pkcs8_key_parser > [ 10.697447] CR2: ffffffff8ff0ade0 > [ 10.700758] ---[ end trace 0000000000000000 ]--- > [ 10.707706] pstore: backend (erst) writing error (-28) > [ 10.712838] RIP: 0010:__bitmap_or+0x10/0x26 > [ 10.717014] Code: 85 c0 0f 95 c0 c3 cc cc cc cc 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 89 c9 31 c0 48 83 c1 3f 48 c1 e9 06 39 c8 73 11 <4c> 8b 04 c6 4c 0b 04 c2 4c 89 04 c7 48 ff c0 eb eb c3 cc cc cc cc > [ 10.735752] RSP: 0000:ffffb45a0078f7b0 EFLAGS: 00010097 > [ 10.740969] RAX: 0000000000000000 RBX: ffffffff8ff0adc0 RCX: 0000000000000004 > [ 10.748093] RDX: ffff9156801967d0 RSI: ffffffff8ff0ade0 RDI: ffff9156801967b0 > [ 10.755218] RBP: ffffb45a0078f7e8 R08: 0000000000000030 R09: 0000000000000000 > [ 10.762341] R10: 0000000000000000 R11: 0000000000000000 R12: 00000000000000ec > [ 10.769467] R13: ffffffff8ff0ade0 R14: 0000000000000001 R15: 0000000000000020 > [ 10.776590] FS: 0000000000000000(0000) GS:ffff9165bfc00000(0000) knlGS:0000000000000000 > [ 10.784666] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 10.790405] CR2: ffffffff8ff0ade0 CR3: 00000001011ae003 CR4: 00000000003706f0 > [ 10.797529] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > [ 10.804651] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > [ 10.811775] note: kworker/0:6[117] exited with irqs disabled > [ 10.817444] note: kworker/0:6[117] exited with preempt_count 1 > > HTH > > -- > Chuck Lever Following up. Jason shamed me into replacing a working CX-3Pro in one of my lab systems with a CX-5 VPI, and the same problem occurs. Removing the CX-5 from the system alleviates the problem. Supermicro SYS-6028R-T/X10DRi, v6.4-rc2 -- Chuck Lever
