On Mon, 2023-05-01 at 06:19 -0700, Ben Greear wrote: > On 4/30/23 1:54 PM, Greenman, Gregory wrote: > > Hi Ben, > > > > On Fri, 2023-04-28 at 15:17 -0700, Ben Greear wrote: > > > On 4/28/23 14:42, Ben Greear wrote: > > > > I saw this in kernel based on recent Linus tree: > > > > > > > > [ 154.760738] iwlwifi 0000:0c:00.0: Microcode SW error detected. Restarting 0x2. > > > > [ 154.766812] iwlwifi 0000:0c:00.0: Start IWL Error Log Dump: > > > > [ 154.771123] iwlwifi 0000:0c:00.0: Transport status: 0x0000004B, valid: 6 > > > > [ 154.776564] iwlwifi 0000:0c:00.0: Loaded firmware version: 78.3bfdc55f.0 ty-a0-gf-a0-78.ucode > > > > [ 154.783811] iwlwifi 0000:0c:00.0: 0x00000071 | NMI_INTERRUPT_UMAC_FATAL > > > > [ 154.789485] iwlwifi 0000:0c:00.0: 0x00808203 | trm_hw_status0 > > > > [ 154.793954] iwlwifi 0000:0c:00.0: 0x00000000 | trm_hw_status1 > > > > [ 154.798427] iwlwifi 0000:0c:00.0: 0x004DC220 | branchlink2 > > > > [ 154.802623] iwlwifi 0000:0c:00.0: 0x000088B8 | interruptlink1 > > > > [ 154.807108] iwlwifi 0000:0c:00.0: 0x000088B8 | interruptlink2 > > > > [ 154.811568] iwlwifi 0000:0c:00.0: 0x00016AEC | data1 > > > > [ 154.815257] iwlwifi 0000:0c:00.0: 0x00000010 | data2 > > > > [ 154.818949] iwlwifi 0000:0c:00.0: 0x00000000 | data3 > > > > [ 154.822656] iwlwifi 0000:0c:00.0: 0x41817919 | beacon time > > > > [ 154.826873] iwlwifi 0000:0c:00.0: 0x7E3EA72C | tsf low > > > > [ 154.830722] iwlwifi 0000:0c:00.0: 0x00000003 | tsf hi > > > > [ 154.834492] iwlwifi 0000:0c:00.0: 0x00000000 | time gp1 > > > > [ 154.838446] iwlwifi 0000:0c:00.0: 0x01CE3A83 | time gp2 > > > > [ 154.842385] iwlwifi 0000:0c:00.0: 0x00000001 | uCode revision type > > > > [ 154.847273] iwlwifi 0000:0c:00.0: 0x0000004E | uCode version major > > > > [ 154.852165] iwlwifi 0000:0c:00.0: 0x3BFDC55F | uCode version minor > > > > [ 154.857090] iwlwifi 0000:0c:00.0: 0x00000420 | hw version > > > > [ 154.861201] iwlwifi 0000:0c:00.0: 0x00C80002 | board version > > > > [ 154.865594] iwlwifi 0000:0c:00.0: 0x806EF400 | hcmd > > > > [ 154.869182] iwlwifi 0000:0c:00.0: 0xE6F4B610 | isr0 > > > > [ 154.872785] iwlwifi 0000:0c:00.0: 0x01448000 | isr1 > > > > [ 154.876378] iwlwifi 0000:0c:00.0: 0x40F8011A | isr2 > > > > [ 154.879994] iwlwifi 0000:0c:00.0: 0x00C00008 | isr3 > > > > [ 154.883611] iwlwifi 0000:0c:00.0: 0x00200000 | isr4 > > > > [ 154.887259] iwlwifi 0000:0c:00.0: 0x0311001C | last cmd Id > > > > [ 154.891459] iwlwifi 0000:0c:00.0: 0x00016AEC | wait_event > > > > [ 154.895573] iwlwifi 0000:0c:00.0: 0x00004288 | l2p_control > > > > [ 154.899806] iwlwifi 0000:0c:00.0: 0x00019C14 | l2p_duration > > > > [ 154.904092] iwlwifi 0000:0c:00.0: 0x00000000 | l2p_mhvalid > > > > [ 154.908304] iwlwifi 0000:0c:00.0: 0x00EF00F8 | l2p_addr_match > > > > [ 154.912777] iwlwifi 0000:0c:00.0: 0x00000009 | lmpm_pmg_sel > > > > [ 154.917065] iwlwifi 0000:0c:00.0: 0x00000000 | timestamp > > > > [ 154.921103] iwlwifi 0000:0c:00.0: 0x05F83838 | flow_handler > > > > [ 154.925473] iwlwifi 0000:0c:00.0: Start IWL Error Log Dump: > > > > [ 154.929778] iwlwifi 0000:0c:00.0: Transport status: 0x0000004B, valid: 7 > > > > [ 154.935190] iwlwifi 0000:0c:00.0: 0x20100246 | ADVANCED_SYSASSERT > > > > [ 154.940001] iwlwifi 0000:0c:00.0: 0x00000000 | umac branchlink1 > > > > [ 154.944643] iwlwifi 0000:0c:00.0: 0x8046D728 | umac branchlink2 > > > > [ 154.949291] iwlwifi 0000:0c:00.0: 0x8048DB02 | umac interruptlink1 > > > > [ 154.954190] iwlwifi 0000:0c:00.0: 0x00000000 | umac interruptlink2 > > > > [ 154.959117] iwlwifi 0000:0c:00.0: 0xDEADBEEF | umac data1 > > > > [ 154.963238] iwlwifi 0000:0c:00.0: 0xDEADBEEF | umac data2 > > > > [ 154.967367] iwlwifi 0000:0c:00.0: 0xDEADBEEF | umac data3 > > > > [ 154.971493] iwlwifi 0000:0c:00.0: 0x0000004E | umac major > > > > [ 154.975657] iwlwifi 0000:0c:00.0: 0x3BFDC55F | umac minor > > > > [ 154.979798] iwlwifi 0000:0c:00.0: 0x01CE3A7D | frame pointer > > > > [ 154.984177] iwlwifi 0000:0c:00.0: 0xC0887F00 | stack pointer > > > > [ 154.988574] iwlwifi 0000:0c:00.0: 0x00030303 | last host cmd > > > > [ 154.992985] iwlwifi 0000:0c:00.0: 0x00010008 | isr status reg > > > > [ 154.997471] iwlwifi 0000:0c:00.0: IML/ROM dump: > > > > [ 155.000722] iwlwifi 0000:0c:00.0: 0x00000B03 | IML/ROM error/state > > > > [ 155.005655] iwlwifi 0000:0c:00.0: 0x00007F8A | IML/ROM data1 > > > > [ 155.010041] iwlwifi 0000:0c:00.0: 0x00000080 | IML/ROM WFPM_AUTH_KEY_0 > > > > [ 155.015285] iwlwifi 0000:0c:00.0: Fseq Registers: > > > > [ 155.018704] iwlwifi 0000:0c:00.0: 0x60000000 | FSEQ_ERROR_CODE > > > > [ 155.023255] iwlwifi 0000:0c:00.0: 0x80440006 | FSEQ_TOP_INIT_VERSION > > > > [ 155.028328] iwlwifi 0000:0c:00.0: 0x00080009 | FSEQ_CNVIO_INIT_VERSION > > > > [ 155.033572] iwlwifi 0000:0c:00.0: 0x0000A652 | FSEQ_OTP_VERSION > > > > [ 155.038221] iwlwifi 0000:0c:00.0: 0x00000002 | FSEQ_TOP_CONTENT_VERSION > > > > [ 155.043557] iwlwifi 0000:0c:00.0: 0x4552414E | FSEQ_ALIVE_TOKEN > > > > [ 155.048196] iwlwifi 0000:0c:00.0: 0x00400410 | FSEQ_CNVI_ID > > > > [ 155.052486] iwlwifi 0000:0c:00.0: 0x00400410 | FSEQ_CNVR_ID > > > > [ 155.056776] iwlwifi 0000:0c:00.0: 0x00400410 | CNVI_AUX_MISC_CHIP > > > > [ 155.061611] iwlwifi 0000:0c:00.0: 0x00400410 | CNVR_AUX_MISC_CHIP > > > > [ 155.066453] iwlwifi 0000:0c:00.0: 0x00009061 | CNVR_SCU_SD_REGS_SD_REG_DIG_DCDC_VTRIM > > > > [ 155.073000] iwlwifi 0000:0c:00.0: 0x00000061 | CNVR_SCU_SD_REGS_SD_REG_ACTIVE_VDIG_MIRROR > > > > [ 155.079912] iwlwifi 0000:0c:00.0: UMAC CURRENT PC: 0xd05c18 > > > > [ 155.084204] iwlwifi 0000:0c:00.0: LMAC1 CURRENT PC: 0xd05c1c > > > > [ 155.089017] iwlwifi 0000:0c:00.0: WRT: Collecting data: ini trigger 4 fired (delay=0ms). > > > > [ 155.089026] ieee80211 wiphy5: Hardware restart was requested > > > > [ 155.089064] iwlwifi 0000:0c:00.0: FW error in SYNC CMD UNKNOWN > > > > > > v77 firmware crashes as well: > > > > > > [ 107.339313] iwlwifi 0000:0c:00.0: Microcode SW error detected. Restarting 0x100. > > > [ 107.347603] iwlwifi 0000:0c:00.0: Start IWL Error Log Dump: > > > [ 107.351904] iwlwifi 0000:0c:00.0: Transport status: 0x0000004B, valid: 6 > > > [ 107.357345] iwlwifi 0000:0c:00.0: Loaded firmware version: 77.f92b5fed.0 ty-a0-gf-a0-77.ucode > > > [ 107.364602] iwlwifi 0000:0c:00.0: 0x00000071 | NMI_INTERRUPT_UMAC_FATAL > > > [ 107.370297] iwlwifi 0000:0c:00.0: 0x00000C07 | trm_hw_status0 > > > [ 107.374772] iwlwifi 0000:0c:00.0: 0x00000000 | trm_hw_status1 > > > [ 107.379244] iwlwifi 0000:0c:00.0: 0x004DC582 | branchlink2 > > > [ 107.383459] iwlwifi 0000:0c:00.0: 0x00008B14 | interruptlink1 > > > [ 107.387929] iwlwifi 0000:0c:00.0: 0x00008B14 | interruptlink2 > > > [ 107.392407] iwlwifi 0000:0c:00.0: 0x0000C586 | data1 > > > [ 107.396090] iwlwifi 0000:0c:00.0: 0x00000010 | data2 > > > [ 107.399791] iwlwifi 0000:0c:00.0: 0x00000000 | data3 > > > [ 107.403482] iwlwifi 0000:0c:00.0: 0x08003A16 | beacon time > > > [ 107.407690] iwlwifi 0000:0c:00.0: 0xC22E862C | tsf low > > > [ 107.411556] iwlwifi 0000:0c:00.0: 0x00000003 | tsf hi > > > [ 107.415332] iwlwifi 0000:0c:00.0: 0x00000000 | time gp1 > > > [ 107.419283] iwlwifi 0000:0c:00.0: 0x006FCDBE | time gp2 > > > [ 107.423227] iwlwifi 0000:0c:00.0: 0x00000001 | uCode revision type > > > [ 107.428129] iwlwifi 0000:0c:00.0: 0x0000004D | uCode version major > > > [ 107.433039] iwlwifi 0000:0c:00.0: 0xF92B5FED | uCode version minor > > > [ 107.437937] iwlwifi 0000:0c:00.0: 0x00000420 | hw version > > > [ 107.442049] iwlwifi 0000:0c:00.0: 0x18C80002 | board version > > > [ 107.446427] iwlwifi 0000:0c:00.0: 0x80EAFF00 | hcmd > > > [ 107.450023] iwlwifi 0000:0c:00.0: 0xE6F48400 | isr0 > > > [ 107.453624] iwlwifi 0000:0c:00.0: 0x01000000 | isr1 > > > [ 107.457218] iwlwifi 0000:0c:00.0: 0x40F8001A | isr2 > > > [ 107.460812] iwlwifi 0000:0c:00.0: 0x04C3601E | isr3 > > > [ 107.464429] iwlwifi 0000:0c:00.0: 0x00200000 | isr4 > > > [ 107.468035] iwlwifi 0000:0c:00.0: 0x0402001C | last cmd Id > > > [ 107.472251] iwlwifi 0000:0c:00.0: 0x0000C586 | wait_event > > > [ 107.476374] iwlwifi 0000:0c:00.0: 0x00004A88 | l2p_control > > > [ 107.480573] iwlwifi 0000:0c:00.0: 0x0001BC34 | l2p_duration > > > [ 107.484863] iwlwifi 0000:0c:00.0: 0x000003BF | l2p_mhvalid > > > [ 107.489089] iwlwifi 0000:0c:00.0: 0x00EF00F8 | l2p_addr_match > > > [ 107.493559] iwlwifi 0000:0c:00.0: 0x00000009 | lmpm_pmg_sel > > > [ 107.497849] iwlwifi 0000:0c:00.0: 0x00000000 | timestamp > > > [ 107.501899] iwlwifi 0000:0c:00.0: 0x05F8C8C8 | flow_handler > > > [ 107.506240] iwlwifi 0000:0c:00.0: Start IWL Error Log Dump: > > > [ 107.510534] iwlwifi 0000:0c:00.0: Transport status: 0x0000004B, valid: 7 > > > [ 107.515953] iwlwifi 0000:0c:00.0: 0x20100246 | ADVANCED_SYSASSERT > > > [ 107.520774] iwlwifi 0000:0c:00.0: 0x00000000 | umac branchlink1 > > > [ 107.525405] iwlwifi 0000:0c:00.0: 0x8046D64C | umac branchlink2 > > > [ 107.530044] iwlwifi 0000:0c:00.0: 0x8048D8D6 | umac interruptlink1 > > > [ 107.534940] iwlwifi 0000:0c:00.0: 0x00000000 | umac interruptlink2 > > > [ 107.539852] iwlwifi 0000:0c:00.0: 0xDEADBEEF | umac data1 > > > [ 107.543982] iwlwifi 0000:0c:00.0: 0xDEADBEEF | umac data2 > > > [ 107.548090] iwlwifi 0000:0c:00.0: 0xDEADBEEF | umac data3 > > > [ 107.552198] iwlwifi 0000:0c:00.0: 0x0000004D | umac major > > > [ 107.556326] iwlwifi 0000:0c:00.0: 0xF92B5FED | umac minor > > > [ 107.560448] iwlwifi 0000:0c:00.0: 0x006FCDB8 | frame pointer > > > [ 107.564840] iwlwifi 0000:0c:00.0: 0xC0887F00 | stack pointer > > > [ 107.569214] iwlwifi 0000:0c:00.0: 0x007B0303 | last host cmd > > > [ 107.573588] iwlwifi 0000:0c:00.0: 0x00010008 | isr status reg > > > [ 107.578077] iwlwifi 0000:0c:00.0: IML/ROM dump: > > > [ 107.581340] iwlwifi 0000:0c:00.0: 0x00000B03 | IML/ROM error/state > > > [ 107.586286] iwlwifi 0000:0c:00.0: 0x00007D74 | IML/ROM data1 > > > [ 107.590699] iwlwifi 0000:0c:00.0: 0x00000080 | IML/ROM WFPM_AUTH_KEY_0 > > > [ 107.595963] iwlwifi 0000:0c:00.0: Fseq Registers: > > > [ 107.599385] iwlwifi 0000:0c:00.0: 0x60000100 | FSEQ_ERROR_CODE > > > [ 107.603950] iwlwifi 0000:0c:00.0: 0x80440006 | FSEQ_TOP_INIT_VERSION > > > [ 107.609024] iwlwifi 0000:0c:00.0: 0x00080009 | FSEQ_CNVIO_INIT_VERSION > > > [ 107.614282] iwlwifi 0000:0c:00.0: 0x0000A652 | FSEQ_OTP_VERSION > > > [ 107.618921] iwlwifi 0000:0c:00.0: 0x00000002 | FSEQ_TOP_CONTENT_VERSION > > > [ 107.624250] iwlwifi 0000:0c:00.0: 0x4552414E | FSEQ_ALIVE_TOKEN > > > [ 107.628903] iwlwifi 0000:0c:00.0: 0x00400410 | FSEQ_CNVI_ID > > > [ 107.633203] iwlwifi 0000:0c:00.0: 0x00400410 | FSEQ_CNVR_ID > > > [ 107.637503] iwlwifi 0000:0c:00.0: 0x00400410 | CNVI_AUX_MISC_CHIP > > > [ 107.642328] iwlwifi 0000:0c:00.0: 0x00400410 | CNVR_AUX_MISC_CHIP > > > [ 107.647155] iwlwifi 0000:0c:00.0: 0x00009061 | CNVR_SCU_SD_REGS_SD_REG_DIG_DCDC_VTRIM > > > [ 107.653702] iwlwifi 0000:0c:00.0: 0x00000061 | CNVR_SCU_SD_REGS_SD_REG_ACTIVE_VDIG_MIRROR > > > [ 107.660601] iwlwifi 0000:0c:00.0: UMAC CURRENT PC: 0xd05c18 > > > [ 107.664895] iwlwifi 0000:0c:00.0: LMAC1 CURRENT PC: 0xd05c1c > > > [ 107.669512] iwlwifi 0000:0c:00.0: WRT: Collecting data: ini trigger 4 fired (delay=0ms). > > > [ 107.669517] ieee80211 wiphy5: Hardware restart was requested > > > [ 107.669528] iwlwifi 0000:0c:00.0: FW error in SYNC CMD UNKNOWN > > > [ 107.674073] CPU: 1 PID: 193 Comm: kworker/1:3 Not tainted 6.3.0+ #1 > > > [ 107.674078] Hardware name: Default string Default string/SKYBAY, BIOS 5.12 02/19/2019 > > > [ 107.674081] Workqueue: events iwl_mvm_tcm_work [iwlmvm] > > > [ 107.674132] Call Trace: > > > [ 107.674135] <TASK> > > > [ 107.674137] dump_stack_lvl+0x43/0x60 > > > [ 107.674145] iwl_trans_txq_send_hcmd+0x578/0x760 [iwlwifi] > > > [ 107.674207] ? iwl_trans_txq_freeze_timer+0x1f0/0x1f0 [iwlwifi] > > > [ 107.674258] ? destroy_sched_domains_rcu+0x30/0x30 > > > [ 107.674268] ? stack_trace_save+0x8c/0xc0 > > > [ 107.674274] ? filter_irq_stacks+0x18/0x60 > > > [ 107.674277] ? __stack_depot_save+0x35/0x470 > > > [ 107.674283] iwl_trans_send_cmd+0xc4/0x180 [iwlwifi] > > > [ 107.674328] iwl_mvm_send_cmd+0x32/0x80 [iwlmvm] > > > [ 107.674368] iwl_mvm_send_low_latency_cmd+0x107/0x170 [iwlmvm] > > > [ 107.674406] ? iwl_mvm_rx_diversity_allowed+0x150/0x150 [iwlmvm] > > > [ 107.674442] ? enqueue_timer+0xbf/0x1e0 > > > [ 107.674447] ? detach_if_pending+0x1e/0x180 > > > [ 107.674452] iwl_mvm_update_low_latency+0xc9/0x120 [iwlmvm] > > > [ 107.674492] __iterate_interfaces+0x102/0x190 [mac80211] > > > [ 107.674633] ? iwl_mvm_update_low_latency+0x120/0x120 [iwlmvm] > > > [ 107.674671] ? iwl_mvm_update_low_latency+0x120/0x120 [iwlmvm] > > > [ 107.674710] ieee80211_iterate_interfaces+0x32/0x50 [mac80211] > > > [ 107.674811] iwl_mvm_recalc_tcm+0x1e7/0x990 [iwlmvm] > > > [ 107.674862] ? iwl_mvm_tcm_load_percentage+0x20/0x20 [iwlmvm] > > > [ 107.674916] ? __schedule+0x7b3/0x1250 > > > [ 107.674920] ? io_schedule_timeout+0xb0/0xb0 > > > [ 107.674924] ? strscpy+0x146/0x190 > > > [ 107.674928] process_one_work+0x463/0x790 > > > [ 107.674935] worker_thread+0x88/0x6f0 > > > [ 107.674939] ? migrate_enable+0x170/0x170 > > > [ 107.674945] ? process_one_work+0x790/0x790 > > > [ 107.674949] kthread+0x16d/0x1b0 > > > [ 107.674954] ? kthread_complete_and_exit+0x20/0x20 > > > [ 107.674958] ret_from_fork+0x1f/0x30 > > > [ 107.674965] </TASK> > > > [ 107.674974] iwlwifi 0000:0c:00.0: Failed to send low latency command > > > [ 122.878885] iwlwifi 0000:0c:00.0: mcast filter cmd error. ret=-5 > > > [ 122.883632] iwlwifi 0000:0c:00.0: Failed to synchronize multicast groups update > > > [ 123.791841] iwlwifi 0000:0c:00.0: Master Disable Timed Out, 100 usec > > > [ 234.009047] iwlwifi 0000:0c:00.0: WRT: Invalid buffer destination > > > [ 234.240960] iwlwifi 0000:0c:00.0: WFPM_UMAC_PD_NOTIFICATION: 0x1f > > > [ 234.258687] iwlwifi 0000:0c:00.0: WFPM_LMAC2_PD_NOTIFICATION: 0x1f > > > [ 234.262627] iwlwifi 0000:0c:00.0: WFPM_AUTH_KEY_0: 0x80 > > > [ 234.266550] iwlwifi 0000:0c:00.0: CNVI_SCU_SEQ_DATA_DW9: 0x0 > > > > > > > > > > > > > > Thanks, > > > > Ben > > > > > > > > > Do you have a way to reproduce it or it just happens randomly? > > The system I looked at was crashing often and regularly. I can get you a binary > crash dump from the firmware if that would help. Yes, it'll be great. Looks like some ARC MMU issue, so need to look on the firmware logs. > > Thanks, > Ben > > > > > Thanks, > > Gregory > > > >
