Hi Lei Ming,
在 2018/11/27 21:08, Ming Lei 写道:
On Tue, Nov 27, 2018 at 05:55:45PM +0800, chenxiang (M) wrote:
Hi all,
There is a issue which may be related to CONFIG_SCSI_MQ_DEFAULT: before we
developed DIF/DIX feature on kernel 4.18 (disable CONFIG_SCSI_MQ_DEFAULT
default), and
it works well.
I guess you are testing hisi_sas_v3_hw, does 4.18 work with
'scsi_mod.use_blk_mq=Y'? If yes, you may run 'git bisect' to figure out
which commit is the 1st bad one.
Yes, i am testing hisi_sas_v3_hw, if 4.18 with 'scsi_mod.use_blk_mq=Y' ,
there is still a issue, and
the log is as attached ;
But when we switch to kernel 4.19-rc1 and 4.20-rc1, Call
trace as follow occurs when running fio and if disable config
CONFIG_SCSI_MQ_DEFAULT,
then it works well. Also if switch ioengine=libaio to ioengine=psync, it
seems also work well. Do you have any idea or encounter similar issue?
I tested scsi-debug via 'dix=1 dif=1', looks everything is fine, are you
using direct io or not?
Yes, i use direct io.
I also test scsi-debug on kernel 4.19, and there seems be some other
issue, and i am not sure
whethere there is something wrong my operation:
root@(none)$ insmod scsi_debug.ko dif=1 dix=1 virtual_gb=120
[ 319.255051] scsi_debug:scsi_debug_init: dif_storep 131072 bytes @
00000000ba5f1637
[ 319.255051] scsi_debug:scsi_debug_init: dif_storep 131072 bytes @
00000000ba5f1637
[ 319.270661] scsi_debug:sdebug_driver_probe: host protection DIF1 DIX1
[ 319.270661] scsi_debug:sdebug_driver_probe: host protection DIF1 DIX1
[ 319.283562] scsi host3: scsi_debug: version 0188 [20180128]
[ 319.283562] dev_size_mb=8, opts=0x0, submit_queues=1, statistics=0
[ 319.283562] scsi host3: scsi_debug: version 0188 [20180128]
[ 319.283562] dev_size_mb=8, opts=0x0, submit_queues=1, statistics=0
[ 319.307871] scsi 3:0:0:0: Direct-Access Linux scsi_debug
0188 PQ: 0 ANSI: 7
[ 319.307871] scsi 3:0:0:0: Direct-Access Linux scsi_debug
0188 PQ: 0 ANSI: 7
[ 319.324173] sd 3:0:0:0: Power-on or device reset occurred
[ 319.324173] sd 3:0:0:0: Power-on or device reset occurred
[ 319.324242] sd 3:0:0:0: Attached scsi generic sg0 type 0
[ 319.324242] sd 3:0:0:0: Attached scsi generic sg0 type 0
[ 319.343021] sd 3:0:0:0: [sda] Enabling DIF Type 1 protection
[ 319.343021] sd 3:0:0:0: [sda] Enabling DIF Type 1 protection
[ 319.356952] sd 3:0:0:0: [sda] 251658240 512-byte logical blocks: (129
GB/120 GiB)
[ 319.356952] sd 3:0:0:0: [sda] 251658240 512-byte logical blocks: (129
GB/120 GiB)
[ 319.375948] sd 3:0:0:0: [sda] Write Protect is off
[ 319.375948] sd 3:0:0:0: [sda] Write Protect is off
[ 319.393551] sd 3:0:0:0: [sda] Write cache: enabled, read cache:
enabled, supports DPO and FUA
[ 319.393551] sd 3:0:0:0: [sda] Write cache: enabled, read cache:
enabled, supports DPO and FUA
[ 319.467940] sd 3:0:0:0: [sda] Enabling DIX T10-DIF-TYPE1-CRC protection
[ 319.467940] sd 3:0:0:0: [sda] Enabling DIX T10-DIF-TYPE1-CRC protection
[ 319.481189] sd 3:0:0:0: [sda] DIF application tag size 2
[ 319.481189] sd 3:0:0:0: [sda] DIF application tag size 2
[ 319.527902] sd 3:0:0:0: [sda] Attached SCSI disk
[ 319.527902] sd 3:0:0:0: [sda] Attached SCSI disk
root@(none)$
root@(none)$
root@(none)$
root@(none)$
root@(none)$ fio --ioengine=libaio --bs=4k --rw=rw --filename=/dev/sd a
--size=10G --runtime=100 --direct=1 -numjobs=1 -iodepth=1 --time_based
--nam e=job
job: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=1
fio 2.0.5
Starting 1 process
Jobs: 1 (f=1)
Jobs: 1 (f=1)
Jobs: 1 (f=1): [M] [4.0% done] [495K/520K /s] [121 /127 iops] [eta 01m:37s]
Jobs: 1 (f=1): [M] [5.0% done] [507K/507K /s] [124 /124 iops] [eta 01m:36s]
Jobs: 1 (f=1): [M] [5.9% done] [516K/499K /s] [126 /122 iops] [eta 01m:35s]
Jobs: 1 (f=1): [M] [6.9% done] [479K/536K /s] [117 /131 iops] [eta 01m:34s]
Jobs: 1 (f=1): [M] [7.9% done] [495K/520K /s] [121 /127 iops] [eta 01m:33s]
Jobs: 1 (f=1): [M] [8.9% done] [499K/516K /s] [122 /126 iops] [eta 01m:32s]
[ 353.298158] scsi_debug:dif_verify: REF check failed on sector 16456
[ 353.298158] scsi_debug:dif_verify: REF check failed on sector 16456
[ 353.314718] scsi_io_completion_action: 107 callbacks suppressed
[ 353.314718] scsi_io_completion_action: 107 callbacks suppressed
[ 353.314723] sd 3:0:0:0: [sda] tag#21 UNKNOWN(0x2003) Result:
hostbyte=0x05 driverbyte=0x08
[ 353.314723] sd 3:0:0:0: [sda] tag#21 UNKNOWN(0x2003) Result:
hostbyte=0x05 driverbyte=0x08
[ 353.343116] sd 3:0:0:0: [sda] tag#21 Sense Key : 0xb [current]
[ 353.343116] sd 3:0:0:0: [sda] tag#21 Sense Key : 0xb [current]
[ 353.354961] sd 3:0:0:0: [sda] tag#21 ASC=0x10 ASCQ=0x3
[ 353.354961] sd 3:0:0:0: [sda] tag#21 ASC=0x10 ASCQ=0x3
[ 353.365415] sd 3:0:0:0: [sda] tag#21 CDB: opcode=0x28 28 20 00 00 40
48 00 00 08 00
[ 353.365415] sd 3:0:0:0: [sda] tag#21 CDB: opcode=0x28 28 20 00 00 40
48 00 00 08 00
[ 353.380742] print_req_error: 107 callbacks suppressed
[ 353.380742] print_req_error: 107 callbacks suppressed
[ 353.380743] print_req_error: protection error, dev sda, sector 16456
[ 353.380743] print_req_error: protection error, dev sda, sector 16456
fio: io_u error on file /dev/sda: Invalid or incomplete multibyte or
wide character
read offset=8425472, buflen=4096
fio: pid=3609, err=84/file:io_u.c:1282, func=io_u error, error=Invalid
or incomplete multibyte or wide character
job: (groupid=0, jobs=1): err=84 (file:io_u.c:1282, func=io_u error,
error=Invalid or incomplete multibyte or wide character): pid=3609
read : io=4108.0KB, bw=500427 B/s, iops=122 , runt= 8406msec
slat (usec): min=5 , max=12568 , avg=18.52, stdev=391.79
clat (usec): min=4002 , max=4018 , avg=4008.55, stdev= 0.84
lat (usec): min=4013 , max=4043 , avg=4015.00, stdev= 1.46
clat percentiles (usec):
| 1.00th=[ 4016], 5.00th=[ 4016], 10.00th=[ 4016], 20.00th=[ 4016],
| 30.00th=[ 4016], 40.00th=[ 4016], 50.00th=[ 4016], 60.00th=[ 4016],
| 70.00th=[ 4016], 80.00th=[ 4016], 90.00th=[ 4016], 95.00th=[ 4016],
| 99.00th=[ 4016], 99.50th=[ 4016], 99.90th=[ 4016]
bw (KB/s) : min= 424, max= 544, per=100.00%, avg=495.50, stdev=35.83
write: io=4120.0KB, bw=501889 B/s, iops=122 , runt= 8406msec
slat (usec): min=47 , max=78 , avg=47.94, stdev= 1.40
clat (usec): min=4004 , max=4012 , avg=4005.23, stdev= 0.65
lat (usec): min=4051 , max=4086 , avg=4053.33, stdev= 1.60
clat percentiles (usec):
| 1.00th=[ 4016], 5.00th=[ 4016], 10.00th=[ 4016], 20.00th=[ 4016],
| 30.00th=[ 4016], 40.00th=[ 4016], 50.00th=[ 4016], 60.00th=[ 4016],
| 70.00th=[ 4016], 80.00th=[ 4016], 90.00th=[ 4016], 95.00th=[ 4016],
| 99.00th=[ 4016], 99.50th=[ 4016], 99.90th=[ 4016]
bw (KB/s) : min= 448, max= 568, per=100.00%, avg=496.50, stdev=35.83
lat (msec) : 10=99.95%
cpu : usr=0.19%, sys=0.69%, ctx=2058, majf=0, minf=14
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
issued : total=r=1028/w=1030/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=4108KB, aggrb=488KB/s, minb=500KB/s, maxb=500KB/s,
mint=8406msec, maxt=8406msec
WRITE: io=4120KB, aggrb=490KB/s, minb=501KB/s, maxb=501KB/s,
mint=8406msec, maxt=8406msec
Disk stats (read/write):
sda: ios=1027/1030, merge=0/0, ticks=4108/4120, in_queue=8348,
util=98.47%
root@(none)$ fio --ioengine=libaio --bs=4k --rw=rw --filename=/dev/sd a
--size=10G --runtime=100 --direct=1 -numjobs=1 -iodepth=1 --time_based
--nam e=job
job: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=1
fio 2.0.5
Starting 1 process
[ 359.592980] scsi_debug:dif_verify: REF check failed on sector 8
[ 359.592980] scsi_debug:dif_verify: REF check failed on sector 8
[ 359.608842] sd 3:0:0:0: [sda] tag#66 UNKNOWN(0x2003) Result:
hostbyte=0x05 driverbyte=0x08
[ 359.608842] sd 3:0:0:0: [sda] tag#66 UNKNOWN(0x2003) Result:
hostbyte=0x05 driverbyte=0x08
[ 359.625391] sd 3:0:0:0: [sda] tag#66 Sense Key : 0xb [current]
[ 359.625391] sd 3:0:0:0: [sda] tag#66 Sense Key : 0xb [current]
[ 359.637236] sd 3:0:0:0: [sda] tag#66 ASC=0x10 ASCQ=0x3
[ 359.637236] sd 3:0:0:0: [sda] tag#66 ASC=0x10 ASCQ=0x3
[ 359.647689] sd 3:0:0:0: [sda] tag#66 CDB: opcode=0x28 28 20 00 00 00
08 00 00 08 00
[ 359.647689] sd 3:0:0:0: [sda] tag#66 CDB: opcode=0x28 28 20 00 00 00
08 00 00 08 00
[ 359.663016] print_req_error: protection error, dev sda, sector 8
[ 359.663016] print_req_error: protection error, dev sda, sector 8
fio: io_u error on file /dev/sda: Invalid or incomplete multibyte or
wide character
read offset=4096, buflen=4096
fio: pid=3612, err=84/file:io_u.c:1282, func=io_u error, error=Invalid
or incomplete multibyte or wide character
job: (groupid=0, jobs=1): err=84 (file:io_u.c:1282, func=io_u error,
error=Invalid or incomplete multibyte or wide character): pid=3612
write: io=4096 B, bw=47080 B/s, iops=11 , runt= 87msec
slat (usec): min=61 , max=61 , avg=61.00, stdev= 0.00
clat (usec): min=4008 , max=4008 , avg=4008.00, stdev= 0.00
lat (usec): min=4070 , max=4070 , avg=4070.00, stdev= 0.00
clat percentiles (usec):
| 1.00th=[ 4016], 5.00th=[ 4016], 10.00th=[ 4016], 20.00th=[ 4016],
| 30.00th=[ 4016], 40.00th=[ 4016], 50.00th=[ 4016], 60.00th=[ 4016],
| 70.00th=[ 4016], 80.00th=[ 4016], 90.00th=[ 4016], 95.00th=[ 4016],
| 99.00th=[ 4016], 99.50th=[ 4016], 99.90th=[ 4016]
lat (msec) : 10=50.00%
cpu : usr=0.00%, sys=12.79%, ctx=2, majf=0, minf=10
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=33.3%, 4=66.7%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
issued : total=r=1/w=1/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=4KB, aggrb=45KB/s, minb=47KB/s, maxb=47KB/s, mint=87msec,
maxt=87msec
Disk stats (read/write):
sda: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
fio 2.0.5
Starting 12 processes
[ 629.210506] Unable to handle kernel paging request at virtual address
0000ffff8027e048
[ 629.210506] Unable to handle kernel paging request at virtual address
0000ffff8027e048
[ 629.226373] Mem abort info:
[ 629.226373] Mem abort info:
[ 629.231952] ESR = 0x96000006
[ 629.231952] ESR = 0x96000006
[ 629.238052] Exception class = DABT (current EL), IL = 32 bits
[ 629.238052] Exception class = DABT (current EL), IL = 32 bits
[ 629.249898] SET = 0, FnV = 0
[ 629.249898] SET = 0, FnV = 0
[ 629.255998] EA = 0, S1PTW = 0
[ 629.255998] EA = 0, S1PTW = 0
[ 629.262272] Data abort info:
[ 629.262272] Data abort info:
[ 629.268023] ISV = 0, ISS = 0x00000006
[ 629.268023] ISV = 0, ISS = 0x00000006
[ 629.275690] CM = 0, WnR = 0
[ 629.275690] CM = 0, WnR = 0
[ 629.281617] user pgtable: 4k pages, 48-bit VAs, pgdp = 0000000085c91728
[ 629.281617] user pgtable: 4k pages, 48-bit VAs, pgdp = 0000000085c91728
[ 629.294857] [0000ffff8027e048] pgd=00000027a8644003,
pud=00000027a85ea003, pmd=0000000000000000
[ 629.294857] [0000ffff8027e048] pgd=00000027a8644003,
pud=00000027a85ea003, pmd=0000000000000000
[ 629.312278] Internal error: Oops: 96000006 [#1] PREEMPT SMP
[ 629.312278] Internal error: Oops: 96000006 [#1] PREEMPT SMP
[ 629.323427] Modules linked in: hisi_sas_v3_hw [last unloaded:
hisi_sas_v3_hw]
[ 629.323427] Modules linked in: hisi_sas_v3_hw [last unloaded:
hisi_sas_v3_hw]
[ 629.337713] CPU: 13 PID: 4465 Comm: fio Not tainted
4.20.0-rc1-15093-ge876dec #1067
[ 629.337713] CPU: 13 PID: 4465 Comm: fio Not tainted
4.20.0-rc1-15093-ge876dec #1067
[ 629.353040] Hardware name: Huawei D06/D06, BIOS Hisilicon D06 UEFI RC0 -
B601 (V6.01) 11/08/2018
[ 629.353040] Hardware name: Huawei D06/D06, BIOS Hisilicon D06 UEFI RC0 -
B601 (V6.01) 11/08/2018
[ 629.370633] pstate: 80400009 (Nzcv daif +PAN -UAO)
[ 629.370633] pstate: 80400009 (Nzcv daif +PAN -UAO)
[ 629.380218] pc : deadline_remove_request+0x2c/0xd0
[ 629.380218] pc : deadline_remove_request+0x2c/0xd0
Could you use gdb to find where 'deadline_remove_request+0x2c' points
to?
Thanks,
Ming
.
root@(none)$ dmesg | grep chenxiang
[ 0.000000] Linux version 4.18.0-rc1-12443-g7e2d9d9-dirty (chenxiang@Turing-Arch-b) (gcc version 4.9.1 20140505 (prerelease) (crosstool-NG linaro-1.13.1-4.9-2014.05 - Linaro GCC 4.9-2014.05)) #1077 SMP PREEMPT Wed Nov 28 10:09:34 CST 2018
root@(none)$ zcat /proc/cn onfig.gz | grep scsi SCSI_MQ_DEFAULT
CONFIG_SCSI_MQ_DEFAULT=y
root@(none)$ insmod hisi_sas_v3_hw.ko enable_dif_di x_dif=0x77
[ 155.849629] iommu: Adding device 0000:74:02.0 to group 8
[ 155.849629] iommu: Adding device 0000:74:02.0 to group 8
[ 155.872319] scsi host3: hisi_sas_v3_hw
[ 155.872319] scsi host3: hisi_sas_v3_hw
[ 157.145261] hisi_sas_v3_hw 0000:74:02.0: phyup: phy0 link_rate=11
[ 157.145261] hisi_sas_v3_hw 0000:74:02.0: phyup: phy0 link_rate=11
[ 157.157461] hisi_sas_v3_hw 0000:74:02.0: phyup: phy1 link_rate=11
[ 157.157461] hisi_sas_v3_hw 0000:74:02.0: phyup: phy1 link_rate=11
[ 157.169658] hisi_sas_v3_hw 0000:74:02.0: phyup: phy2 link_rate=11
[ 157.169658] hisi_sas_v3_hw 0000:74:02.0: phyup: phy2 link_rate=11
[ 157.181854] hisi_sas_v3_hw 0000:74:02.0: phyup: phy4 link_rate=11
[ 157.181854] hisi_sas_v3_hw 0000:74:02.0: phyup: phy4 link_rate=11
[ 157.194190] hisi_sas_v3_hw 0000:74:02.0: dev[1:2] found
[ 157.194190] hisi_sas_v3_hw 0000:74:02.0: dev[1:2] found
[ 157.207613] hisi_sas_v3_hw 0000:74:02.0: dev[2:1] found
[ 157.207613] hisi_sas_v3_hw 0000:74:02.0: dev[2:1] found
[ 157.218154] hisi_sas_v3_hw 0000:74:02.0: dev[3:1] found
[ 157.218154] hisi_sas_v3_hw 0000:74:02.0: dev[3:1] found
[ 157.228691] hisi_sas_v3_hw 0000:74:02.0: dev[4:1] found
[ 157.228691] hisi_sas_v3_hw 0000:74:02.0: dev[4:1] found
[ 157.239225] hisi_sas_v3_hw 0000:74:02.0: dev[5:1] found
[ 157.239225] hisi_sas_v3_hw 0000:74:02.0: dev[5:1] found
[ 157.249862] hisi_sas_v3_hw 0000:74:02.0: dev[6:1] found
[ 157.249862] hisi_sas_v3_hw 0000:74:02.0: dev[6:1] found
[ 157.260394] hisi_sas_v3_hw 0000:74:02.0: dev[7:1] found
[ 157.260394] hisi_sas_v3_hw 0000:74:02.0: dev[7:1] found
[ 157.270928] hisi_sas_v3_hw 0000:74:02.0: dev[8:1] found
[ 157.270928] hisi_sas_v3_hw 0000:74:02.0: dev[8:1] found
[ 157.281463] hisi_sas_v3_hw 0000:74:02.0: dev[9:1] found
[ 157.281463] hisi_sas_v3_hw 0000:74:02.0: dev[9:1] found
[ 157.291995] hisi_sas_v3_hw 0000:74:02.0: dev[10:1] found
[ 157.291995] hisi_sas_v3_hw 0000:74:02.0: dev[10:1] found
[ 157.302795] hisi_sas_v3_hw 0000:74:02.0: dev[11:1] found
[ 157.302795] hisi_sas_v3_hw 0000:74:02.0: dev[11:1] found
[ 157.313503] hisi_sas_v3_hw 0000:74:02.0: dev[12:1] found
[ 157.313503] hisi_sas_v3_hw 0000:74:02.0: dev[12:1] found
[ 157.324209] hisi_sas_v3_hw 0000:74:02.0: dev[13:1] found
[ 157.324209] hisi_sas_v3_hw 0000:74:02.0: dev[13:1] found
[ 157.334938] hisi_sas_v3_hw 0000:74:02.0: dev[14:1] found
[ 157.334938] hisi_sas_v3_hw 0000:74:02.0: dev[14:1] found
[ 157.346128] scsi 3:0:0:0: Direct-Access HGST HUSMM1640ASS204 C2D0 PQ: 0 ANSI: 6
[ 157.346128] scsi 3:0:0:0: Direct-Access HGST HUSMM1640ASS204 C2D0 PQ: 0 ANSI: 6
[ 157.362695] sd 3:0:0:0: Attached scsi generic sg0 type 0
[ 157.362695] sd 3:0:0:0: Attached scsi generic sg0 type 0
[ 157.363016] sd 3:0:0:0: [sda] 781422768 512-byte logical blocks: (400 GB/373 GiB)
[ 157.363016] sd 3:0:0:0: [sda] 781422768 512-byte logical blocks: (400 GB/373 GiB)
[ 157.373862] scsi 3:0:1:0: Direct-Access HGST HUSMM1640ASS204 C2D0 PQ: 0 ANSI: 6
[ 157.373862] scsi 3:0:1:0: Direct-Access HGST HUSMM1640ASS204 C2D0 PQ: 0 ANSI: 6
[ 157.388326] sd 3:0:0:0: [sda] 4096-byte physical blocks
[ 157.388326] sd 3:0:0:0: [sda] 4096-byte physical blocks
[ 157.404878] sd 3:0:1:0: Attached scsi generic sg1 type 0
[ 157.404878] sd 3:0:1:0: Attached scsi generic sg1 type 0
[ 157.405184] sd 3:0:1:0: [sdb] 781422768 512-byte logical blocks: (400 GB/373 GiB)
[ 157.405184] sd 3:0:1:0: [sdb] 781422768 512-byte logical blocks: (400 GB/373 GiB)
[ 157.405186] sd 3:0:1:0: [sdb] 4096-byte physical blocks
[ 157.405186] sd 3:0:1:0: [sdb] 4096-byte physical blocks
[ 157.405287] sd 3:0:1:0: [sdb] Write Protect is off
[ 157.405287] sd 3:0:1:0: [sdb] Write Protect is off
[ 157.405413] sd 3:0:1:0: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 157.405413] sd 3:0:1:0: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 157.412447] sd 3:0:1:0: [sdb] Attached SCSI disk
[ 157.412447] sd 3:0:1:0: [sdb] Attached SCSI disk
[ 157.415088] sd 3:0:0:0: [sda] Write Protect is off
[ 157.415088] sd 3:0:0:0: [sda] Write Protect is off
[ 157.426120] scsi 3:0:2:0: Direct-Access HGST HUSMM1640ASS204 C2D0 PQ: 0 ANSI: 6
[ 157.426120] scsi 3:0:2:0: Direct-Access HGST HUSMM1640ASS204 C2D0 PQ: 0 ANSI: 6
[ 157.451394] sd 3:0:2:0: Attached scsi generic sg2 type 0
[ 157.451394] sd 3:0:2:0: Attached scsi generic sg2 type 0
[ 157.451702] sd 3:0:2:0: [sdc] 781422768 512-byte logical blocks: (400 GB/373 GiB)
[ 157.451702] sd 3:0:2:0: [sdc] 781422768 512-byte logical blocks: (400 GB/373 GiB)
[ 157.451704] sd 3:0:2:0: [sdc] 4096-byte physical blocks
[ 157.451704] sd 3:0:2:0: [sdc] 4096-byte physical blocks
[ 157.451807] sd 3:0:2:0: [sdc] Write Protect is off
[ 157.451807] sd 3:0:2:0: [sdc] Write Protect is off
[ 157.451935] sd 3:0:2:0: [sdc] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 157.451935] sd 3:0:2:0: [sdc] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 157.458868] sd 3:0:2:0: [sdc] Attached SCSI disk
[ 157.458868] sd 3:0:2:0: [sdc] Attached SCSI disk
[ 157.460765] sd 3:0:0:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 157.460765] sd 3:0:0:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 157.478217] scsi 3:0:3:0: Direct-Access HGST HUSMM1640ASS204 C2D0 PQ: 0 ANSI: 6
[ 157.478217] scsi 3:0:3:0: Direct-Access HGST HUSMM1640ASS204 C2D0 PQ: 0 ANSI: 6
[ 157.493974] sd 3:0:0:0: [sda] Attached SCSI disk
[ 157.493974] sd 3:0:0:0: [sda] Attached SCSI disk
[ 157.496881] sd 3:0:3:0: Attached scsi generic sg3 type 0
[ 157.496881] sd 3:0:3:0: Attached scsi generic sg3 type 0
[ 157.497177] sd 3:0:3:0: [sdd] 781422768 512-byte logical blocks: (400 GB/373 GiB)
[ 157.497177] sd 3:0:3:0: [sdd] 781422768 512-byte logical blocks: (400 GB/373 GiB)
[ 157.497179] sd 3:0:3:0: [sdd] 4096-byte physical blocks
[ 157.497179] sd 3:0:3:0: [sdd] 4096-byte physical blocks
[ 157.497280] sd 3:0:3:0: [sdd] Write Protect is off
[ 157.497280] sd 3:0:3:0: [sdd] Write Protect is off
[ 157.497404] sd 3:0:3:0: [sdd] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 157.497404] sd 3:0:3:0: [sdd] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 157.504326] sd 3:0:3:0: [sdd] Attached SCSI disk
[ 157.504326] sd 3:0:3:0: [sdd] Attached SCSI disk
[ 157.699585] scsi 3:0:4:0: Direct-Access HGST HUSMM1640ASS204 C2D0 PQ: 0 ANSI: 6
[ 157.699585] scsi 3:0:4:0: Direct-Access HGST HUSMM1640ASS204 C2D0 PQ: 0 ANSI: 6
[ 157.716140] sd 3:0:4:0: Attached scsi generic sg4 type 0
[ 157.716140] sd 3:0:4:0: Attached scsi generic sg4 type 0
[ 157.716418] sd 3:0:4:0: [sde] 781422768 512-byte logical blocks: (400 GB/373 GiB)
[ 157.716418] sd 3:0:4:0: [sde] 781422768 512-byte logical blocks: (400 GB/373 GiB)
[ 157.727321] scsi 3:0:5:0: Direct-Access HGST HUSMM1640ASS204 C2D0 PQ: 0 ANSI: 6
[ 157.727321] scsi 3:0:5:0: Direct-Access HGST HUSMM1640ASS204 C2D0 PQ: 0 ANSI: 6
[ 157.741756] sd 3:0:4:0: [sde] 4096-byte physical blocks
[ 157.741756] sd 3:0:4:0: [sde] 4096-byte physical blocks
[ 157.741859] sd 3:0:4:0: [sde] Write Protect is off
[ 157.741859] sd 3:0:4:0: [sde] Write Protect is off
[ 157.758308] sd 3:0:5:0: Attached scsi generic sg5 type 0
[ 157.758308] sd 3:0:5:0: Attached scsi generic sg5 type 0
[ 157.758344] sd 3:0:5:0: [sdf] Enabling DIF Type 1 protection
[ 157.758344] sd 3:0:5:0: [sdf] Enabling DIF Type 1 protection
[ 157.758616] sd 3:0:5:0: [sdf] 781422768 512-byte logical blocks: (400 GB/373 GiB)
[ 157.758616] sd 3:0:5:0: [sdf] 781422768 512-byte logical blocks: (400 GB/373 GiB)
[ 157.758617] sd 3:0:5:0: [sdf] 4096-byte physical blocks
[ 157.758617] sd 3:0:5:0: [sdf] 4096-byte physical blocks
[ 157.758720] sd 3:0:5:0: [sdf] Write Protect is off
[ 157.758720] sd 3:0:5:0: [sdf] Write Protect is off
[ 157.758845] sd 3:0:5:0: [sdf] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 157.758845] sd 3:0:5:0: [sdf] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 157.763644] sd 3:0:5:0: [sdf] Enabling DIX T10-DIF-TYPE1-CRC protection
[ 157.763644] sd 3:0:5:0: [sdf] Enabling DIX T10-DIF-TYPE1-CRC protection
[ 157.765652] sd 3:0:5:0: [sdf] Attached SCSI disk
[ 157.765652] sd 3:0:5:0: [sdf] Attached SCSI disk
[ 157.778527] scsi 3:0:6:0: Direct-Access HGST HUSMM1640ASS204 C2D0 PQ: 0 ANSI: 6
[ 157.778527] scsi 3:0:6:0: Direct-Access HGST HUSMM1640ASS204 C2D0 PQ: 0 ANSI: 6
[ 157.788756] sd 3:0:4:0: [sde] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 157.788756] sd 3:0:4:0: [sde] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 157.800299] sd 3:0:6:0: Attached scsi generic sg6 type 0
[ 157.800299] sd 3:0:6:0: Attached scsi generic sg6 type 0
[ 157.800330] sd 3:0:6:0: [sdg] Enabling DIF Type 1 protection
[ 157.800330] sd 3:0:6:0: [sdg] Enabling DIF Type 1 protection
[ 157.800605] sd 3:0:6:0: [sdg] 781422768 512-byte logical blocks: (400 GB/373 GiB)
[ 157.800605] sd 3:0:6:0: [sdg] 781422768 512-byte logical blocks: (400 GB/373 GiB)
[ 157.800606] sd 3:0:6:0: [sdg] 4096-byte physical blocks
[ 157.800606] sd 3:0:6:0: [sdg] 4096-byte physical blocks
[ 157.800708] sd 3:0:6:0: [sdg] Write Protect is off
[ 157.800708] sd 3:0:6:0: [sdg] Write Protect is off
[ 157.800835] sd 3:0:6:0: [sdg] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 157.800835] sd 3:0:6:0: [sdg] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 157.805572] sd 3:0:6:0: [sdg] Enabling DIX T10-DIF-TYPE1-CRC protection
[ 157.805572] sd 3:0:6:0: [sdg] Enabling DIX T10-DIF-TYPE1-CRC protection
[ 157.807571] sd 3:0:6:0: [sdg] Attached SCSI disk
[ 157.807571] sd 3:0:6:0: [sdg] Attached SCSI disk
[ 157.821516] sd 3:0:4:0: [sde] Attached SCSI disk
[ 157.821516] sd 3:0:4:0: [sde] Attached SCSI disk
[ 157.825907] scsi 3:0:7:0: Direct-Access HGST HUSMM1640ASS204 C2D0 PQ: 0 ANSI: 6
[ 157.825907] scsi 3:0:7:0: Direct-Access HGST HUSMM1640ASS204 C2D0 PQ: 0 ANSI: 6
[ 158.030092] sd 3:0:7:0: Attached scsi generic sg7 type 0
[ 158.030092] sd 3:0:7:0: Attached scsi generic sg7 type 0
[ 158.030377] sd 3:0:7:0: [sdh] 781422768 512-byte logical blocks: (400 GB/373 GiB)
[ 158.030377] sd 3:0:7:0: [sdh] 781422768 512-byte logical blocks: (400 GB/373 GiB)
[ 158.041249] scsi 3:0:8:0: Direct-Access HGST HUSMM1640ASS204 C2D0 PQ: 0 ANSI: 6
[ 158.041249] scsi 3:0:8:0: Direct-Access HGST HUSMM1640ASS204 C2D0 PQ: 0 ANSI: 6
[ 158.055709] sd 3:0:7:0: [sdh] 4096-byte physical blocks
[ 158.055709] sd 3:0:7:0: [sdh] 4096-byte physical blocks
[ 158.072246] sd 3:0:8:0: Attached scsi generic sg8 type 0
[ 158.072246] sd 3:0:8:0: Attached scsi generic sg8 type 0
[ 158.072283] sd 3:0:8:0: [sdi] Enabling DIF Type 1 protection
[ 158.072283] sd 3:0:8:0: [sdi] Enabling DIF Type 1 protection
[ 158.072555] sd 3:0:8:0: [sdi] 781422768 512-byte logical blocks: (400 GB/373 GiB)
[ 158.072555] sd 3:0:8:0: [sdi] 781422768 512-byte logical blocks: (400 GB/373 GiB)
[ 158.072556] sd 3:0:8:0: [sdi] 4096-byte physical blocks
[ 158.072556] sd 3:0:8:0: [sdi] 4096-byte physical blocks
[ 158.072656] sd 3:0:8:0: [sdi] Write Protect is off
[ 158.072656] sd 3:0:8:0: [sdi] Write Protect is off
[ 158.072786] sd 3:0:8:0: [sdi] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 158.072786] sd 3:0:8:0: [sdi] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 158.077515] sd 3:0:8:0: [sdi] Enabling DIX T10-DIF-TYPE1-CRC protection
[ 158.077515] sd 3:0:8:0: [sdi] Enabling DIX T10-DIF-TYPE1-CRC protection
[ 158.079511] sd 3:0:8:0: [sdi] Attached SCSI disk
[ 158.079511] sd 3:0:8:0: [sdi] Attached SCSI disk
[ 158.082469] sd 3:0:7:0: [sdh] Write Protect is off
[ 158.082469] sd 3:0:7:0: [sdh] Write Protect is off
[ 158.093576] scsi 3:0:9:0: Direct-Access HGST HUSMM1640ASS204 C2D0 PQ: 0 ANSI: 6
[ 158.093576] scsi 3:0:9:0: Direct-Access HGST HUSMM1640ASS204 C2D0 PQ: 0 ANSI: 6
[ 158.119650] sd 3:0:9:0: Attached scsi generic sg9 type 0
[ 158.119650] sd 3:0:9:0: Attached scsi generic sg9 type 0
[ 158.119965] sd 3:0:9:0: [sdj] 781422768 512-byte logical blocks: (400 GB/373 GiB)
[ 158.119965] sd 3:0:9:0: [sdj] 781422768 512-byte logical blocks: (400 GB/373 GiB)
[ 158.119967] sd 3:0:9:0: [sdj] 4096-byte physical blocks
[ 158.119967] sd 3:0:9:0: [sdj] 4096-byte physical blocks
[ 158.120068] sd 3:0:9:0: [sdj] Write Protect is off
[ 158.120068] sd 3:0:9:0: [sdj] Write Protect is off
[ 158.120195] sd 3:0:9:0: [sdj] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 158.120195] sd 3:0:9:0: [sdj] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 158.126842] sd 3:0:9:0: [sdj] Attached SCSI disk
[ 158.126842] sd 3:0:9:0: [sdj] Attached SCSI disk
[ 158.129891] sd 3:0:7:0: [sdh] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 158.129891] sd 3:0:7:0: [sdh] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 158.139787] scsi 3:0:10:0: Direct-Access HGST HUSMM1640ASS204 C2D0 PQ: 0 ANSI: 6
[ 158.139787] scsi 3:0:10:0: Direct-Access HGST HUSMM1640ASS204 C2D0 PQ: 0 ANSI: 6
[ 158.163040] sd 3:0:7:0: [sdh] Attached SCSI disk
[ 158.163040] sd 3:0:7:0: [sdh] Attached SCSI disk
[ 158.169998] sd 3:0:10:0: Attached scsi generic sg10 type 0
[ 158.169998] sd 3:0:10:0: Attached scsi generic sg10 type 0
[ 158.170319] sd 3:0:10:0: [sdk] 781422768 512-byte logical blocks: (400 GB/373 GiB)
[ 158.170319] sd 3:0:10:0: [sdk] 781422768 512-byte logical blocks: (400 GB/373 GiB)
[ 158.170321] sd 3:0:10:0: [sdk] 4096-byte physical blocks
[ 158.170321] sd 3:0:10:0: [sdk] 4096-byte physical blocks
[ 158.170423] sd 3:0:10:0: [sdk] Write Protect is off
[ 158.170423] sd 3:0:10:0: [sdk] Write Protect is off
[ 158.170548] sd 3:0:10:0: [sdk] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 158.170548] sd 3:0:10:0: [sdk] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 158.177200] sd 3:0:10:0: [sdk] Attached SCSI disk
[ 158.177200] sd 3:0:10:0: [sdk] Attached SCSI disk
[ 158.393029] scsi 3:0:11:0: Direct-Access HGST HUSMM1640ASS204 C2D0 PQ: 0 ANSI: 6
[ 158.393029] scsi 3:0:11:0: Direct-Access HGST HUSMM1640ASS204 C2D0 PQ: 0 ANSI: 6
[ 158.409751] sd 3:0:11:0: Attached scsi generic sg11 type 0
[ 158.409751] sd 3:0:11:0: Attached scsi generic sg11 type 0
[ 158.410055] sd 3:0:11:0: [sdl] 781422768 512-byte logical blocks: (400 GB/373 GiB)
[ 158.410055] sd 3:0:11:0: [sdl] 781422768 512-byte logical blocks: (400 GB/373 GiB)
[ 158.421317] scsi 3:0:12:0: Enclosure 12G SAS Expander RevB PQ: 0 ANSI: 6
[ 158.421317] scsi 3:0:12:0: Enclosure 12G SAS Expander RevB PQ: 0 ANSI: 6
[ 158.435901] sd 3:0:11:0: [sdl] 4096-byte physical blocks
[ 158.435901] sd 3:0:11:0: [sdl] 4096-byte physical blocks
[ 158.452794] ses 3:0:12:0: Attached Enclosure device
[ 158.452794] ses 3:0:12:0: Attached Enclosure device
[ 158.463002] sd 3:0:11:0: [sdl] Write Protect is off
[ 158.463002] sd 3:0:11:0: [sdl] Write Protect is off
[ 158.472715] ses 3:0:12:0: Attached scsi generic sg12 type 13
[ 158.472715] ses 3:0:12:0: Attached scsi generic sg12 type 13
[ 158.482547] sd 3:0:11:0: [sdl] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 158.482547] sd 3:0:11:0: [sdl] Write cache: enabled, read cache: enabled, supports DPO and FUA
root@(none)$ [ 158.517620] sd 3:0:11:0: [sdl] Attached SCSI disk
[ 158.517620] sd 3:0:11:0: [sdl] Attached SCSI disk
root@(none)$
root@(none)$
root@(none)$
root@(none)$
root@(none)$ fio --ioengine=libaio --bs=4k --rw=rw --filename=/dev/sd f --size=10G --runtime=100 --direct=1 -numjobs=1 -iodepth=128 --time_based --n ame=job
[ 192.477842] random: fio: uninitialized urandom read (64 bytes read)
[ 192.477842] random: fio: uninitialized urandom read (64 bytes read)
job: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
fio 2.0.5
Starting 1 process
[ 192.678356] random: crng init done
[ 192.678356] random: crng init done
[ 192.685280] Unable to handle kernel paging request at virtual address 0000000040000030
[ 192.685280] Unable to handle kernel paging request at virtual address 0000000040000030
[ 192.685601] WARNING: CPU: 0 PID: 3757 at drivers/iommu/io-pgtable-arm.c:598 __arm_lpae_unmap+0x3b4/0x4b4
[ 192.685601] WARNING: CPU: 0 PID: 3757 at drivers/iommu/io-pgtable-arm.c:598 __arm_lpae_unmap+0x3b4/0x4b4
[ 192.701141] Mem abort info:
[ 192.701141] Mem abort info:
[ 192.720118] Modules linked in: hisi_sas_v3_hw
[ 192.720118] Modules linked in: hisi_sas_v3_hw
[ 192.720123] CPU: 0 PID: 3757 Comm: kworker/0:2H Not tainted 4.18.0-rc1-12443-g7e2d9d9-dirty #1077
[ 192.720123] CPU: 0 PID: 3757 Comm: kworker/0:2H Not tainted 4.18.0-rc1-12443-g7e2d9d9-dirty #1077
[ 192.720125] Hardware name: Huawei D06/D06, BIOS Hisilicon D06 UEFI RC0 - B601 (V6.01) 11/08/2018
[ 192.720125] Hardware name: Huawei D06/D06, BIOS Hisilicon D06 UEFI RC0 - B601 (V6.01) 11/08/2018
[ 192.725702] ESR = 0x96000005
[ 192.725702] ESR = 0x96000005
[ 192.734412] Workqueue: kblockd blk_mq_run_work_fn
[ 192.734412] Workqueue: kblockd blk_mq_run_work_fn
[ 192.752182] Exception class = DABT (current EL), IL = 32 bits
[ 192.752182] Exception class = DABT (current EL), IL = 32 bits
[ 192.769769] pstate: 80400009 (Nzcv daif +PAN -UAO)
[ 192.769769] pstate: 80400009 (Nzcv daif +PAN -UAO)
[ 192.769771] pc : __arm_lpae_unmap+0x3b4/0x4b4
[ 192.769771] pc : __arm_lpae_unmap+0x3b4/0x4b4
[ 192.769774] lr : __arm_lpae_unmap+0x100/0x4b4
[ 192.769774] lr : __arm_lpae_unmap+0x100/0x4b4
[ 192.775872] SET = 0, FnV = 0
[ 192.775872] SET = 0, FnV = 0
[ 192.785276] sp : ffff0000080039a0
[ 192.785276] sp : ffff0000080039a0
[ 192.785277] x29: ffff0000080039a0 x28: ffff8027a4300d6c
[ 192.785277] x29: ffff0000080039a0 x28: ffff8027a4300d6c
[ 192.797127] EA = 0, S1PTW = 0
[ 192.797127] EA = 0, S1PTW = 0
[ 192.806703] x27: 00000000000012f0 x26: 0000000000000004
[ 192.806703] x27: 00000000000012f0 x26: 0000000000000004
[ 192.806705] x25: 0000000000000002 x24: 0000000000001000
[ 192.806705] x25: 0000000000000002 x24: 0000000000001000
[ 192.815418] Data abort info:
[ 192.815418] Data abort info:
[ 192.824125] x23: ffff8027dc070c00 x22: ffff8027a42e8b60
[ 192.824125] x23: ffff8027dc070c00 x22: ffff8027a42e8b60
[ 192.824128] x21: 00000000fd56c000
[ 192.824128] x21: 00000000fd56c000
[ 192.830227] ISV = 0, ISS = 0x00000005
[ 192.830227] ISV = 0, ISS = 0x00000005
[ 192.836844] x20: 0000000000000003
[ 192.836844] x20: 0000000000000003
[ 192.836846] x19: 0000000000000000 x18: 000000000000000e
[ 192.836846] x19: 0000000000000000 x18: 000000000000000e
[ 192.836848] x17: 0000000000000001
[ 192.836848] x17: 0000000000000001
[ 192.847475] CM = 0, WnR = 0
[ 192.847475] CM = 0, WnR = 0
[ 192.853744] x16: 0000000000000019
[ 192.853744] x16: 0000000000000019
[ 192.853746] x15: ffffffffffffffff x14: 0000000000000001
[ 192.853746] x15: ffffffffffffffff x14: 0000000000000001
[ 192.853748] x13: 00000000c0a80101 x12: 0000000000000001
[ 192.853748] x13: 00000000c0a80101 x12: 0000000000000001
[ 192.864377] user pgtable: 4k pages, 48-bit VAs, pgdp = 000000005038c217
[ 192.864377] user pgtable: 4k pages, 48-bit VAs, pgdp = 000000005038c217
[ 192.874999] x11: 0000000000001000 x10: 0000000000000000
[ 192.874999] x11: 0000000000001000 x10: 0000000000000000
[ 192.875001] x9 : 000000000000000c
[ 192.875001] x9 : 000000000000000c
[ 192.880753] [0000000040000030] pgd=00000027a9ab9003
[ 192.880753] [0000000040000030] pgd=00000027a9ab9003
[ 192.891375] x8 : 0000000000000009
[ 192.891375] x8 : 0000000000000009
[ 192.891376] x7 : 0000000000000001 x6 : 00000000000fd56c
[ 192.891376] x7 : 0000000000000001 x6 : 00000000000fd56c
[ 192.898173] , pud=0000000000000000
[ 192.898173] , pud=0000000000000000
[ 192.905836] x5 : 0000000000000b60 x4 : ffff8027a42e8000
[ 192.905836] x5 : 0000000000000b60 x4 : ffff8027a42e8000
[ 192.905838] x3 : 0000000000000003
[ 192.905838] x3 : 0000000000000003
[ 192.923256] x2 : 0000000000001000
[ 192.923256] x2 : 0000000000001000
[ 192.923258] x1 : 00000000fd56c000 x0 : ffff8027dc070c00
[ 192.923258] x1 : 00000000fd56c000 x0 : ffff8027dc070c00
[ 192.930055] Internal error: Oops: 96000005 [#1] PREEMPT SMP
[ 192.930055] Internal error: Oops: 96000005 [#1] PREEMPT SMP
[ 192.935976] Call trace:
[ 192.935976] Call trace:
[ 192.942769] Modules linked in:
[ 192.942769] Modules linked in:
[ 192.953395] __arm_lpae_unmap+0x3b4/0x4b4
[ 192.953395] __arm_lpae_unmap+0x3b4/0x4b4
[ 192.964018] hisi_sas_v3_hw
[ 192.964018] hisi_sas_v3_hw
[ 192.977256] __arm_lpae_unmap+0x100/0x4b4
[ 192.977256] __arm_lpae_unmap+0x100/0x4b4
[ 192.977258] __arm_lpae_unmap+0x100/0x4b4
[ 192.977258] __arm_lpae_unmap+0x100/0x4b4
[ 192.987884] CPU: 8 PID: 3755 Comm: fio Not tainted 4.18.0-rc1-12443-g7e2d9d9-dirty #1077
[ 192.987884] CPU: 8 PID: 3755 Comm: fio Not tainted 4.18.0-rc1-12443-g7e2d9d9-dirty #1077
[ 192.994676] __arm_lpae_unmap+0x100/0x4b4
[ 192.994676] __arm_lpae_unmap+0x100/0x4b4
[ 192.994678] arm_lpae_unmap+0x38/0x50
[ 192.994678] arm_lpae_unmap+0x38/0x50
[ 193.004431] Hardware name: Huawei D06/D06, BIOS Hisilicon D06 UEFI RC0 - B601 (V6.01) 11/08/2018
[ 193.004431] Hardware name: Huawei D06/D06, BIOS Hisilicon D06 UEFI RC0 - B601 (V6.01) 11/08/2018
[ 193.011226] arm_smmu_unmap+0x18/0x20
[ 193.011226] arm_smmu_unmap+0x18/0x20
[ 193.011228] __iommu_unmap+0xc4/0x140
[ 193.011228] __iommu_unmap+0xc4/0x140
[ 193.021852] pstate: 80400009 (Nzcv daif +PAN -UAO)
[ 193.021852] pstate: 80400009 (Nzcv daif +PAN -UAO)
[ 193.021857] pc : deadline_remove_request+0x5c/0xd0
[ 193.021857] pc : deadline_remove_request+0x5c/0xd0
[ 193.028647] iommu_unmap+0x10/0x18
[ 193.028647] iommu_unmap+0x10/0x18
[ 193.028649] __iommu_dma_unmap+0x40/0x8c
[ 193.028649] __iommu_dma_unmap+0x40/0x8c
[ 193.039273] lr : dd_dispatch_request+0x184/0x1e0
[ 193.039273] lr : dd_dispatch_request+0x184/0x1e0
[ 193.039275] sp : ffff000021afbaf0
[ 193.039275] sp : ffff000021afbaf0
[ 193.046069] iommu_dma_unmap_sg+0x88/0xa4
[ 193.046069] iommu_dma_unmap_sg+0x88/0xa4
[ 193.046072] __iommu_unmap_sg_attrs+0x38/0x88
[ 193.046072] __iommu_unmap_sg_attrs+0x38/0x88
[ 193.052864] x29: ffff000021afbaf0 x28: ffff8027dacb9a80
[ 193.052864] x29: ffff000021afbaf0 x28: ffff8027dacb9a80
[ 193.063491] hisi_sas_slot_task_free+0xd4/0x180
[ 193.063491] hisi_sas_slot_task_free+0xd4/0x180
[ 193.063495] slot_complete_v3_hw+0x1d8/0x5bc [hisi_sas_v3_hw]
[ 193.063495] slot_complete_v3_hw+0x1d8/0x5bc [hisi_sas_v3_hw]
[ 193.074640] x27: 0000000000000001
[ 193.074640] x27: 0000000000000001
[ 193.079520] cq_tasklet_v3_hw+0x8c/0x10c [hisi_sas_v3_hw]
[ 193.079520] cq_tasklet_v3_hw+0x8c/0x10c [hisi_sas_v3_hw]
[ 193.079523] tasklet_action_common.isra.14+0x50/0xd8
[ 193.079523] tasklet_action_common.isra.14+0x50/0xd8
[ 193.085617] x26: ffff8027dacb9a80
[ 193.085617] x26: ffff8027dacb9a80
[ 193.093631] tasklet_action+0x24/0x2c
[ 193.093631] tasklet_action+0x24/0x2c
[ 193.093633] __do_softirq+0x100/0x1f8
[ 193.093633] __do_softirq+0x100/0x1f8
[ 193.099207] x25: 0000000000000000 x24: ffff000009451000
[ 193.099207] x25: 0000000000000000 x24: ffff000009451000
[ 193.107221] irq_exit+0xb8/0xcc
[ 193.107221] irq_exit+0xb8/0xcc
[ 193.107223] __handle_domain_irq+0x64/0xac
[ 193.107223] __handle_domain_irq+0x64/0xac
[ 193.115235] x23: ffff8027a5235f40
[ 193.115235] x23: ffff8027a5235f40
[ 193.131433] gic_handle_irq+0xd4/0x184
[ 193.131433] gic_handle_irq+0xd4/0x184
[ 193.131434] el1_irq+0xb0/0x128
[ 193.131434] el1_irq+0xb0/0x128
[ 193.139446] x22: 0000000040000000
[ 193.139446] x22: 0000000040000000
[ 193.146766] _raw_spin_lock+0x34/0x44
[ 193.146766] _raw_spin_lock+0x34/0x44
[ 193.164354] x21: ffff7e009ecb6500 x20: 0000000040000000
[ 193.164354] x21: ffff7e009ecb6500 x20: 0000000040000000
[ 193.171672] blk_mq_do_dispatch_sched+0x4c/0x104
[ 193.171672] blk_mq_do_dispatch_sched+0x4c/0x104
[ 193.171674] blk_mq_sched_dispatch_requests+0x130/0x1bc
[ 193.171674] blk_mq_sched_dispatch_requests+0x130/0x1bc
[ 193.178990] x19: ffff8027a5235f00
[ 193.178990] x19: ffff8027a5235f00
[ 193.188570] __blk_mq_run_hw_queue+0xa4/0x11c
[ 193.188570] __blk_mq_run_hw_queue+0xa4/0x11c
[ 193.198149] x18: 0000ffffd7630e00
[ 193.198149] x18: 0000ffffd7630e00
[ 193.204944] blk_mq_run_work_fn+0x18/0x20
[ 193.204944] blk_mq_run_work_fn+0x18/0x20
[ 193.204946] process_one_work+0x140/0x2b0
[ 193.204946] process_one_work+0x140/0x2b0
[ 193.212784] x17: 0000ffff92d2a5a0 x16: ffff7e009fffdd00
[ 193.212784] x17: 0000ffff92d2a5a0 x16: ffff7e009fffdd00
[ 193.222017] worker_thread+0x130/0x3e4
[ 193.222017] worker_thread+0x130/0x3e4
[ 193.222020] kthread+0xfc/0x128
[ 193.222020] kthread+0xfc/0x128
[ 193.228638] x15: 000032402e000000
[ 193.228638] x15: 000032402e000000
[ 193.236652] ret_from_fork+0x10/0x18
[ 193.236652] ret_from_fork+0x10/0x18
[Jobs: 1 (f=1)
193.245360] x14: 0000000000000000
[ 193.245360] x14: 0000000000000000
[ 193.255986] ---[ end trace ae3b8c1166c99889 ]---
[ 193.255986] ---[ end trace ae3b8c1166c99889 ]---
[ 193.265043] x13: ffff7e009ecb6400 x12: ffff8027a92a35a0
[ 193.265043] x13: ffff7e009ecb6400 x12: ffff8027a92a35a0
[ 193.276559] WARNING: CPU: 0 PID: 3757 at drivers/iommu/dma-iommu.c:408 __iommu_dma_unmap+0x68/0x8c
[ 193.276559] WARNING: CPU: 0 PID: 3757 at drivers/iommu/dma-iommu.c:408 __iommu_dma_unmap+0x68/0x8c
[ 193.283335] x11: 0000000000000000
[ 193.283335] x11: 0000000000000000
[ 193.294133] Modules linked in: hisi_sas_v3_hw
[ 193.294133] Modules linked in: hisi_sas_v3_hw
[ 193.304062] x10: ffff8027a92a11c0
[ 193.304062] x10: ffff8027a92a11c0
[ 193.310858] CPU: 0 PID: 3757 Comm: kworker/0:2H Tainted: G W 4.18.0-rc1-12443-g7e2d9d9-dirty #1077
[ 193.310858] CPU: 0 PID: 3757 Comm: kworker/0:2H Tainted: G W 4.18.0-rc1-12443-g7e2d9d9-dirty #1077
[ 193.318173] x9 : ffff8027a92a07b8 x8 : 0000000000000007
[ 193.318173] x9 : ffff8027a92a07b8 x8 : 0000000000000007
[ 193.325490] Hardware name: Huawei D06/D06, BIOS Hisilicon D06 UEFI RC0 - B601 (V6.01) 11/08/2018
[ 193.325490] Hardware name: Huawei D06/D06, BIOS Hisilicon D06 UEFI RC0 - B601 (V6.01) 11/08/2018
[ 193.325493] Workqueue: kblockd blk_mq_run_work_fn
[ 193.325493] Workqueue: kblockd blk_mq_run_work_fn
[ 193.336117] x7 : ffff8027a92a11c0
[ 193.336117] x7 : ffff8027a92a11c0
[ 193.342389] pstate: 80400009 (Nzcv daif +PAN -UAO)
[ 193.342389] pstate: 80400009 (Nzcv daif +PAN -UAO)
[ 193.350575] x6 : 0000000000000000
[ 193.350575] x6 : 0000000000000000
[ 193.357370] pc : __iommu_dma_unmap+0x68/0x8c
[ 193.357370] pc : __iommu_dma_unmap+0x68/0x8c
[ 193.357373] lr : __iommu_dma_unmap+0x40/0x8c
[ 193.357373] lr : __iommu_dma_unmap+0x40/0x8c
[ 193.364862] x5 : 0000000000000049 x4 : ffff8027e0407c00
[ 193.364862] x5 : 0000000000000049 x4 : ffff8027e0407c00
[ 193.371133] sp : ffff000008003c60
[ 193.371133] sp : ffff000008003c60
[ 193.371135] x29: ffff000008003c60
[ 193.371135] x29: ffff000008003c60
[ 193.377929] x3 : ffff8027a5238540
[ 193.377929] x3 : ffff8027a5238540
[ 193.385245] x28: ffff8027a4300d6c
[ 193.385245] x28: ffff8027a4300d6c
[ 193.395870] x2 : 0000000000000000
[ 193.395870] x2 : 0000000000000000
[ 193.405103] x27: 00000000000012f0 x26: 0000000000000000
[ 193.405103] x27: 00000000000012f0 x26: 0000000000000000
[ 193.415554] x1 : 0000000000000000 x0 : ffff8027a5235f60
[ 193.415554] x1 : 0000000000000000 x0 : ffff8027a5235f60
[ 193.422350] x25: ffff8027d9cc3098
[ 193.422350] x25: ffff8027d9cc3098
[ 193.431060] Process fio (pid: 3755, stack limit = 0x000000002dd5ba1f)
[ 193.431060] Process fio (pid: 3755, stack limit = 0x000000002dd5ba1f)
[ 193.437853] x24: ffff8027a5b05c00
[ 193.437853] x24: ffff8027a5b05c00
[ 193.445866] Call trace:
[ 193.445866] Call trace:
[ 193.453879] x23: ffff8027d9cc3098
[ 193.453879] x23: ffff8027d9cc3098
[ 193.464505] deadline_remove_request+0x5c/0xd0
[ 193.464505] deadline_remove_request+0x5c/0xd0
[ 193.471994] x22: 00000000fd568580
[ 193.471994] x22: 00000000fd568580
[ 193.478266] dd_dispatch_request+0x184/0x1e0
[ 193.478266] dd_dispatch_request+0x184/0x1e0
[ 193.478269] blk_mq_do_dispatch_sched+0x4c/0x104
[ 193.478269] blk_mq_do_dispatch_sched+0x4c/0x104
[ 193.485061] x21: ffff8027a9745800 x20: 00000000fd568000
[ 193.485061] x21: ffff8027a9745800 x20: 00000000fd568000
[ 193.492204] blk_mq_sched_dispatch_requests+0x130/0x1bc
[ 193.492204] blk_mq_sched_dispatch_requests+0x130/0x1bc
[ 193.492206] __blk_mq_run_hw_queue+0xa4/0x11c
[ 193.492206] __blk_mq_run_hw_queue+0xa4/0x11c
[ 193.500215] x19: 00000000da118000
[ 193.500215] x19: 00000000da118000
[ 193.509447] __blk_mq_delay_run_hw_queue+0x1a4/0x208
[ 193.509447] __blk_mq_delay_run_hw_queue+0x1a4/0x208
[ 193.509449] blk_mq_run_hw_queue+0x9c/0xf8
[ 193.509449] blk_mq_run_hw_queue+0x9c/0xf8
[ 193.520072] x18: 000000000000000e
[ 193.520072] x18: 000000000000000e
[ 193.538013] blk_mq_sched_insert_requests+0x58/0x84
[ 193.538013] blk_mq_sched_insert_requests+0x58/0x84
[ 193.538015] blk_mq_flush_plug_list+0x144/0x16c
[ 193.538015] blk_mq_flush_plug_list+0x144/0x16c
[ 193.544808] x17: 0000000000000001 x16: 0000000000000019
[ 193.544808] x17: 0000000000000001 x16: 0000000000000019
[ 193.553519] blk_flush_plug_list+0xe8/0x244
[ 193.553519] blk_flush_plug_list+0xe8/0x244
[ 193.553521] blk_finish_plug+0x30/0x40
[ 193.553521] blk_finish_plug+0x30/0x40
[ 193.560315] x15: ffffffffffffffff
[ 193.560315] x15: ffffffffffffffff
[ 193.580868] sys_io_submit+0x110/0x214
[ 193.580868] sys_io_submit+0x110/0x214
[ 193.591489] x14: 0000000000000001
[ 193.591489] x14: 0000000000000001
[ 193.609082] el0_svc_naked+0x30/0x34
[ 193.609082] el0_svc_naked+0x30/0x34
[ 193.609084] Code: b9401661 12000021 2a0103e2 8b010ed4 (f9401a81)
[ 193.609084] Code: b9401661 12000021 2a0103e2 8b010ed4 (f9401a81)
[ 193.618488] x13: 00000000c0a80101 x12: 0000000000000001
[ 193.618488] x13: 00000000c0a80101 x12: 0000000000000001
[ 193.625283] ---[ end trace ae3b8c1166c9988a ]---
[ 193.625283] ---[ end trace ae3b8c1166c9988a ]---
[ 193.634863] x11: 0000000000001000
[ 193.634863] x11: 0000000000001000
[ 193.641663] note: fio[3755] exited with preempt_count 2
[ 193.641663] note: fio[3755] exited with preempt_count 2
[ 193.650191] x10: 0000000000000000
[ 193.650191] x10: 0000000000000000
[ 193.650192] x9 : 000000000000000c x8 : 00000000000000ff
[ 193.650192] x9 : 000000000000000c x8 : 00000000000000ff
[ 193.658765] WARNING: CPU: 24 PID: 3755 at kernel/rcu/tree_plugin.h:330 rcu_note_context_switch+0x30/0x370
[ 193.658765] WARNING: CPU: 24 PID: 3755 at kernel/rcu/tree_plugin.h:330 rcu_note_context_switch+0x30/0x370
[ 193.669353] x7 : 0000000000000000 x6 : 003a699d00000000
[ 193.669353] x7 : 0000000000000000 x6 : 003a699d00000000
[ 193.675977] Modules linked in:
[ 193.675977] Modules linked in:
[ 193.682772] x5 : 0000000000000018
[ 193.682772] x5 : 0000000000000018
[ 193.689565] hisi_sas_v3_hw
[ 193.689565] hisi_sas_v3_hw
[ 193.696358] x4 : 0000000000007440
[ 193.696358] x4 : 0000000000007440
[ 193.703156] CPU: 24 PID: 3755 Comm: fio Tainted: G D W 4.18.0-rc1-12443-g7e2d9d9-dirty #1077
[ 193.703156] CPU: 24 PID: 3755 Comm: fio Tainted: G D W 4.18.0-rc1-12443-g7e2d9d9-dirty #1077
[ 193.713778] x3 : 000000000a000000 x2 : 0000000000040ee6
[ 193.713778] x3 : 000000000a000000 x2 : 0000000000040ee6
[ 193.724405] Hardware name: Huawei D06/D06, BIOS Hisilicon D06 UEFI RC0 - B601 (V6.01) 11/08/2018
[ 193.724405] Hardware name: Huawei D06/D06, BIOS Hisilicon D06 UEFI RC0 - B601 (V6.01) 11/08/2018
[ 193.731199] x1 : 674238a07d45b300
[ 193.731199] x1 : 674238a07d45b300
[ 193.744089] pstate: 20400089 (nzCv daIf +PAN -UAO)
[ 193.744089] pstate: 20400089 (nzCv daIf +PAN -UAO)
[ 193.750881] x0 : 0000000000004000
[ 193.750881] x0 : 0000000000004000
[ 193.755761] pc : rcu_note_context_switch+0x30/0x370
[ 193.755761] pc : rcu_note_context_switch+0x30/0x370
[ 193.755764] lr : rcu_note_context_switch+0x1c/0x370
[ 193.755764] lr : rcu_note_context_switch+0x1c/0x370
[ 193.762557] Call trace:
[ 193.762557] Call trace:
[ 193.771440] sp : ffff000021afb3e0
[ 193.771440] sp : ffff000021afb3e0
[ 193.771441] x29: ffff000021afb3e0
[ 193.771441] x29: ffff000021afb3e0
[ 193.778236] __iommu_dma_unmap+0x68/0x8c
[ 193.778236] __iommu_dma_unmap+0x68/0x8c
[ 193.786769] x28: ffff8027dacb9a80
[ 193.786769] x28: ffff8027dacb9a80
[ 193.796003] iommu_dma_unmap_sg+0x88/0xa4
[ 193.796003] iommu_dma_unmap_sg+0x88/0xa4
[ 193.796006] __iommu_unmap_sg_attrs+0x38/0x88
[ 193.796006] __iommu_unmap_sg_attrs+0x38/0x88
[ 193.806629] x27: 0000000000000000 x26: ffff000008d36a14
[ 193.806629] x27: 0000000000000000 x26: ffff000008d36a14
[ 193.817081] hisi_sas_slot_task_free+0xd4/0x180
[ 193.817081] hisi_sas_slot_task_free+0xd4/0x180
[ 193.817084] slot_complete_v3_hw+0x1d8/0x5bc [hisi_sas_v3_hw]
[ 193.817084] slot_complete_v3_hw+0x1d8/0x5bc [hisi_sas_v3_hw]
[ 193.825793] x25: ffff000009451000
[ 193.825793] x25: ffff000009451000
[ 193.832587] cq_tasklet_v3_hw+0x8c/0x10c [hisi_sas_v3_hw]
[ 193.832587] cq_tasklet_v3_hw+0x8c/0x10c [hisi_sas_v3_hw]
[ 193.832589] tasklet_action_common.isra.14+0x50/0xd8
[ 193.832589] tasklet_action_common.isra.14+0x50/0xd8
[ 193.842516] x24: ffff8027dacb9a80
[ 193.842516] x24: ffff8027dacb9a80
[ 193.850704] tasklet_action+0x24/0x2c
[ 193.850704] tasklet_action+0x24/0x2c
[ 193.850707] __do_softirq+0x100/0x1f8
[ 193.850707] __do_softirq+0x100/0x1f8
[ 193.857498] x23: ffff000009432000 x22: ffff00000943e000
[ 193.857498] x23: ffff000009432000 x22: ffff00000943e000
[ 193.867254] irq_exit+0xb8/0xcc
[ 193.867254] irq_exit+0xb8/0xcc
[ 193.867257] __handle_domain_irq+0x64/0xac
[ 193.867257] __handle_domain_irq+0x64/0xac
[ 193.876314] x21: ffff8027dacb9a80
[ 193.876314] x21: ffff8027dacb9a80
[ 193.886938] gic_handle_irq+0xd4/0x184
[ 193.886938] gic_handle_irq+0xd4/0x184
[ 193.886940] el1_irq+0xb0/0x128
[ 193.886940] el1_irq+0xb0/0x128
[ 193.895299] x20: 0000000000000000
[ 193.895299] x20: 0000000000000000
[ 193.902792] _raw_spin_lock+0x34/0x44
[ 193.902792] _raw_spin_lock+0x34/0x44
[ 193.902795] blk_mq_do_dispatch_sched+0x4c/0x104
[ 193.902795] blk_mq_do_dispatch_sched+0x4c/0x104
[ 193.909586] x19: ffff8027dacb9a80 x18: 0000000000000000
[ 193.909586] x19: ffff8027dacb9a80 x18: 0000000000000000
[ 193.917079] blk_mq_sched_dispatch_requests+0x130/0x1bc
[ 193.917079] blk_mq_sched_dispatch_requests+0x130/0x1bc
[ 193.917081] __blk_mq_run_hw_queue+0xa4/0x11c
[ 193.917081] __blk_mq_run_hw_queue+0xa4/0x11c
[ 193.923874] x17: 0000000000000000
[ 193.923874] x17: 0000000000000000
[ 193.931015] blk_mq_run_work_fn+0x18/0x20
[ 193.931015] blk_mq_run_work_fn+0x18/0x20
[ 193.931018] process_one_work+0x140/0x2b0
[ 193.931018] process_one_work+0x140/0x2b0
[ 193.943208] x16: 0000000000000000
[ 193.943208] x16: 0000000000000000
[ 193.953835] worker_thread+0x130/0x3e4
[ 193.953835] worker_thread+0x130/0x3e4
[ 193.953837] kthread+0xfc/0x128
[ 193.953837] kthread+0xfc/0x128
[ 193.963067] x15: 0000000000000000 x14: 0000000000000000
[ 193.963067] x15: 0000000000000000 x14: 0000000000000000
[ 193.969862] ret_from_fork+0x10/0x18
[ 193.969862] ret_from_fork+0x10/0x18
[ 193.969864] ---[ end trace ae3b8c1166c9988b ]---
[ 193.969864] ---[ end trace ae3b8c1166c9988b ]---
[ 193.980315] x13: 0000000000000000 x12: 0000000000000020
[ 193.980315] x13: 0000000000000000 x12: 0000000000000020
[ 194.452303] x11: 0000000000024000 x10: 0000000000000950
[ 194.452303] x11: 0000000000024000 x10: 0000000000000950
[ 194.462929] x9 : ffff000021afb580 x8 : ffff8027dacba430
[ 194.462929] x9 : ffff000021afb580 x8 : ffff8027dacba430
[ 194.473556] x7 : 0000000000000000 x6 : ffff8027a42d81a8
[ 194.473556] x7 : 0000000000000000 x6 : ffff8027a42d81a8
[ 194.484182] x5 : ffff000021afb580 x4 : 0000000000000000
[ 194.484182] x5 : ffff000021afb580 x4 : 0000000000000000
Jobs: 1 (f=1[): [M] [3.0% done] [0K/0K /s] [0 /0 iops] [eta 01m:38s]
194.494808] x3 : 0000802fd63a6000 x2 : ffff00000943f780
[ 194.494808] x3 : 0000802fd63a6000 x2 : ffff00000943f780
[ 194.511421] x1 : ffff00000943f798 x0 : 0000000000000001
[ 194.511421] x1 : ffff00000943f798 x0 : 0000000000000001
[ 194.522047] Call trace:
[ 194.522047] Call trace:
[ 194.526927] rcu_note_context_switch+0x30/0x370
[ 194.526927] rcu_note_context_switch+0x30/0x370
[ 194.535986] __schedule+0x88/0x5cc
[ 194.535986] __schedule+0x88/0x5cc
[ 194.542780] schedule+0x38/0x9c
[ 194.542780] schedule+0x38/0x9c
[ 194.549053] schedule_timeout+0x1b8/0x320
[ 194.549053] schedule_timeout+0x1b8/0x320
[ 194.557066] wait_for_common+0xb4/0x164
[ 194.557066] wait_for_common+0xb4/0x164
[ 194.564731] wait_for_completion+0x14/0x1c
[ 194.564731] wait_for_completion+0x14/0x1c
[ 194.572921] exit_aio+0x100/0x114
[ 194.572921] exit_aio+0x100/0x114
[ 194.579542] mmput+0x48/0x118
[ 194.579542] mmput+0x48/0x118
[ 194.585466] do_exit+0x278/0x978
[ 194.585466] do_exit+0x278/0x978
[ 194.591915] die+0x1c8/0x1f4
[ 194.591915] die+0x1c8/0x1f4
[ 194.597665] die_kernel_fault+0x64/0x74
[ 194.597665] die_kernel_fault+0x64/0x74
[ 194.605331] __do_kernel_fault+0x84/0xac
[ 194.605331] __do_kernel_fault+0x84/0xac
[ 194.613170] do_page_fault+0x1dc/0x3e8
[ 194.613170] do_page_fault+0x1dc/0x3e8
[ 194.620661] do_translation_fault+0x68/0x74
[ 194.620661] do_translation_fault+0x68/0x74
[ 194.629022] do_mem_abort+0x4c/0xf4
[ 194.629022] do_mem_abort+0x4c/0xf4
[ 194.635991] el1_da+0x20/0x80
[ 194.635991] el1_da+0x20/0x80
[ 194.641916] deadline_remove_request+0x5c/0xd0
[ 194.641916] deadline_remove_request+0x5c/0xd0
[ 194.650800] dd_dispatch_request+0x184/0x1e0
[ 194.650800] dd_dispatch_request+0x184/0x1e0
[ 194.659337] blk_mq_do_dispatch_sched+0x4c/0x104
[ 194.659337] blk_mq_do_dispatch_sched+0x4c/0x104
[ 194.668570] blk_mq_sched_dispatch_requests+0x130/0x1bc
[ 194.668570] blk_mq_sched_dispatch_requests+0x130/0x1bc
[ 194.679022] __blk_mq_run_hw_queue+0xa4/0x11c
[ 194.679022] __blk_mq_run_hw_queue+0xa4/0x11c
[ 194.687731] __blk_mq_delay_run_hw_queue+0x1a4/0x208
[ 194.687731] __blk_mq_delay_run_hw_queue+0x1a4/0x208
[ 194.697660] blk_mq_run_hw_queue+0x9c/0xf8
[ 194.697660] blk_mq_run_hw_queue+0x9c/0xf8
[ 194.705848] blk_mq_sched_insert_requests+0x58/0x84
[ 194.705848] blk_mq_sched_insert_requests+0x58/0x84
[ 194.715603] blk_mq_flush_plug_list+0x144/0x16c
[ 194.715603] blk_mq_flush_plug_list+0x144/0x16c
[ 194.724663] blk_flush_plug_list+0xe8/0x244
[ 194.724663] blk_flush_plug_list+0xe8/0x244
[ 194.733024] blk_finish_plug+0x30/0x40
[ 194.733024] blk_finish_plug+0x30/0x40
[ 194.740515] sys_io_submit+0x110/0x214
[ 194.740515] sys_io_submit+0x110/0x214
[ 194.748006] el0_svc_naked+0x30/0x34
[ 194.748006] el0_svc_naked+0x30/0x34
[ 194.755148] ---[ end trace ae3b8c1166c9988c ]---
[ 194.755148] ---[ end trace ae3b8c1166c9988c ]---
Jobs: 1 (f=1): [M] [4.0% done] [0K/0K /s] [0 /0 iops] [eta 01m:37s]
Jobs: 1 (f=1): [M] [5.0% done] [0K/0K /s] [0 /0 iops] [eta 01m:36s]
Jobs: 1 (f=1): [M] [5.9% done] [0K/0K /s] [0 /0 iops] [eta 01m:35s]
Jobs: 1 (f=1): [M] [6.9% done] [0K/0K /s] [0 /0 iops] [eta 01m:34s]
Jobs: 1 (f=1): [M] [7.9% done] [0K/0K /s] [0 /0 iops] [eta 01m:33s]
Jobs: 1 (f=1): [M] [8.9% done] [0K/0K /s] [0 /0 iops] [eta 01m:32s]
Jobs: 1 (f=1): [M] [9.9% done] [0K/0K /s] [0 /0 iops] [eta 01m:31s]
Jobs: 1 (f=1): [M] [10.9% done] [0K/0K /s] [0 /0 iops] [eta 01m:30s]
Jobs: 1 (f=1): [M] [11.9% done] [0K/0K /s] [0 /0 iops] [eta 01m:29s]
Jobs: 1 (f=1): [M] [12.9% done] [0K/0K /s] [0 /0 iops] [eta 01m:28s]
Jobs: 1 (f=1): [M] [13.9% done] [0K/0K /s] [0 /0 iops] [eta 01m:27s]
Jobs: 1 (f=1): [M] [14.9% done] [0K/0K /s] [0 /0 iops] [eta 01m:26s]
Jobs: 1 (f=1): [M] [15.8% done] [0K/0K /s] [0 /0 iops] [eta 01m:25s]
Jobs: 1 (f=1): [M] [16.8% done] [0K/0K /s] [0 /0 iops] [eta 01m:24s]
Jobs: 1 (f=1): [M] [17.8% done] [0K/0K /s] [0 /0 iops] [eta 01m:23s]
Jobs: 1 (f=1): [M] [18.8% done] [0K/0K /s] [0 /0 iops] [eta 01m:22s]
Jobs: 1 (f=1): [M] [19.8% done] [0K/0K /s] [0 /0 iops] [eta 01m:21s]
Jobs: 1 (f=1): [M] [20.8% done] [0K/0K /s] [0 /0 iops] [eta 01m:20s]
Jobs: 1 (f=1): [M] [21.8% done] [0K/0K /s] [0 /0 iops] [eta 01m:19s]
[ 213.743524] INFO: rcu_preempt self-detected stall on CPU
[ 213.743524] INFO: rcu_preempt self-detected stall on CPU
[ 213.751524] INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 213.751524] INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 213.754153] 0-....: (1 GPs behind) idle=ac6/1/4611686018427387906 softirq=4726/4728 fqs=2625
[ 213.754153] 0-....: (1 GPs behind) idle=ac6/1/4611686018427387906 softirq=4726/4728 fqs=2625
[ 213.782717]
[ 213.782717]
[ 213.782719] Tasks blocked on level-1 rcu_node (CPUs 0-15): P3755
[ 213.782719] Tasks blocked on level-1 rcu_node (CPUs 0-15): P3755
[ 213.785859] (t=5260 jiffies g=1596 c=1595 q=86)
[ 213.785859] (t=5260 jiffies g=1596 c=1595 q=86)
[ 213.798054] 0-....: (1 GPs behind) idle=ac6/1/4611686018427387906 softirq=4726/4728 fqs=2626
[ 213.798054] 0-....: (1 GPs behind) idle=ac6/1/4611686018427387906 softirq=4726/4728 fqs=2626
[ 213.824526] Task dump for CPU 0:
[ 213.824526] Task dump for CPU 0:
[ 213.824527] (detected by 5, t=5270 jiffies, g=1596, c=1595, q=86)
[ 213.824527] (detected by 5, t=5270 jiffies, g=1596, c=1595, q=86)
[ 213.830973] kworker/0:2H R running task 0 3757 2 0x0000002a
[ 213.830973] kworker/0:2H R running task 0 3757 2 0x0000002a
[ 213.857451] Workqueue: kblockd blk_mq_run_work_fn
[ 213.857451] Workqueue: kblockd blk_mq_run_work_fn
[ 213.866858] Call trace:
[ 213.866858] Call trace:
[ 213.871738] dump_backtrace+0x0/0x180
[ 213.871738] dump_backtrace+0x0/0x180
[ 213.879056] show_stack+0x14/0x1c
[ 213.879056] show_stack+0x14/0x1c
[ 213.885677] sched_show_task+0x16c/0x184
[ 213.885677] sched_show_task+0x16c/0x184
[ 213.893516] dump_cpu_task+0x40/0x4c
[ 213.893516] dump_cpu_task+0x40/0x4c
[ 213.900660] rcu_dump_cpu_stacks+0x98/0xd8
[ 213.900660] rcu_dump_cpu_stacks+0x98/0xd8
[ 213.908848] rcu_check_callbacks+0x750/0x9c0
[ 213.908848] rcu_check_callbacks+0x750/0x9c0
[ 213.917384] update_process_times+0x2c/0x54
[ 213.917384] update_process_times+0x2c/0x54
[ 213.925747] tick_sched_handle.isra.14+0x24/0x68
[ 213.925747] tick_sched_handle.isra.14+0x24/0x68
[ 213.934979] tick_sched_timer+0x48/0x98
[ 213.934979] tick_sched_timer+0x48/0x98
[ 213.942644] __hrtimer_run_queues+0x100/0x158
[ 213.942644] __hrtimer_run_queues+0x100/0x158
[ 213.951354] hrtimer_interrupt+0xe4/0x248
[ 213.951354] hrtimer_interrupt+0xe4/0x248
[ 213.959370] arch_timer_handler_phys+0x2c/0x38
[ 213.959370] arch_timer_handler_phys+0x2c/0x38
[ 213.968254] handle_percpu_devid_irq+0x78/0x12c
[ 213.968254] handle_percpu_devid_irq+0x78/0x12c
[ 213.977313] generic_handle_irq+0x24/0x38
[ 213.977313] generic_handle_irq+0x24/0x38
[ 213.985327] __handle_domain_irq+0x60/0xac
[ 213.985327] __handle_domain_irq+0x60/0xac
[ 213.993513] gic_handle_irq+0xd4/0x184
[ 213.993513] gic_handle_irq+0xd4/0x184
[ 214.001005] el1_irq+0xb0/0x128
[ 214.001005] el1_irq+0xb0/0x128
[ 214.007277] _raw_spin_lock+0x34/0x44
[ 214.007277] _raw_spin_lock+0x34/0x44
[ 214.014595] blk_mq_do_dispatch_sched+0x4c/0x104
[ 214.014595] blk_mq_do_dispatch_sched+0x4c/0x104
[ 214.023828] blk_mq_sched_dispatch_requests+0x130/0x1bc
[ 214.023828] blk_mq_sched_dispatch_requests+0x130/0x1bc
[ 214.034279] __blk_mq_run_hw_queue+0xa4/0x11c
[ 214.034279] __blk_mq_run_hw_queue+0xa4/0x11c
[ 214.042989] blk_mq_run_work_fn+0x18/0x20
[ 214.042989] blk_mq_run_work_fn+0x18/0x20
[ 214.051002] process_one_work+0x140/0x2b0
[ 214.051002] process_one_work+0x140/0x2b0
[ 214.059015] worker_thread+0x130/0x3e4
[ 214.059015] worker_thread+0x130/0x3e4
[ 214.066506] kthread+0xfc/0x128
[ 214.066506] kthread+0xfc/0x128
[ 214.072778] ret_from_fork+0x10/0x18
[ 214.072778] ret_from_fork+0x10/0x18
[ 214.079922] Task dump for CPU 0:
[ 214.079922] Task dump for CPU 0:
[ 214.086368] kworker/0:2H R running task 0 3757 2 0x0000002a
[ 214.086368] kworker/0:2H R running task 0 3757 2 0x0000002a
[ 214.100484] Workqueue: kblockd blk_mq_run_work_fn
[ 214.100484] Workqueue: kblockd blk_mq_run_work_fn
[ 214.109892] Call trace:
[ 214.109892] Call trace:
[ 214.114773] __switch_to+0x90/0xcc
[ 214.114773] __switch_to+0x90/0xcc
[ 214.121568] 0x674238a07d45b300
[ 214.121568] 0x674238a07d45b300
[ 214.127842] fio D 0 3755 3753 0x0000002c
[ 214.127842] fio D 0 3755 3753 0x0000002c
[ 214.138816] Call trace:
[ 214.138816] Call trace:
[ 214.143696] __switch_to+0x90/0xcc
[ 214.143696] __switch_to+0x90/0xcc
[ 214.150491] __schedule+0x1b8/0x5cc
[ 214.150491] __schedule+0x1b8/0x5cc
[ 214.157460] schedule+0x38/0x9c
[ 214.157460] schedule+0x38/0x9c
[ 214.163732] schedule_timeout+0x1b8/0x320
[ 214.163732] schedule_timeout+0x1b8/0x320
[ 214.171745] wait_for_common+0xb4/0x164
[ 214.171745] wait_for_common+0xb4/0x164
[ 214.179411] wait_for_completion+0x14/0x1c
[ 214.179411] wait_for_completion+0x14/0x1c
[ 214.187600] exit_aio+0x100/0x114
[ 214.187600] exit_aio+0x100/0x114
[ 214.194222] mmput+0x48/0x118
[ 214.194222] mmput+0x48/0x118
[ 214.200146] do_exit+0x278/0x978
[ 214.200146] do_exit+0x278/0x978
[ 214.206592] die+0x1c8/0x1f4
[ 214.206592] die+0x1c8/0x1f4
[ 214.212343] die_kernel_fault+0x64/0x74
[ 214.212343] die_kernel_fault+0x64/0x74
[ 214.220009] __do_kernel_fault+0x84/0xac
[ 214.220009] __do_kernel_fault+0x84/0xac
[ 214.227848] do_page_fault+0x1dc/0x3e8
[ 214.227848] do_page_fault+0x1dc/0x3e8
[ 214.235340] do_translation_fault+0x68/0x74
[ 214.235340] do_translation_fault+0x68/0x74
[ 214.243702] do_mem_abort+0x4c/0xf4
[ 214.243702] do_mem_abort+0x4c/0xf4
[ 214.250670] el1_da+0x20/0x80
[ 214.250670] el1_da+0x20/0x80
[ 214.256594] deadline_remove_request+0x5c/0xd0
[ 214.256594] deadline_remove_request+0x5c/0xd0
[ 214.265478] dd_dispatch_request+0x184/0x1e0
[ 214.265478] dd_dispatch_request+0x184/0x1e0
[ 214.274015] blk_mq_do_dispatch_sched+0x4c/0x104
[ 214.274015] blk_mq_do_dispatch_sched+0x4c/0x104
[ 214.283247] blk_mq_sched_dispatch_requests+0x130/0x1bc
[ 214.283247] blk_mq_sched_dispatch_requests+0x130/0x1bc
[ 214.293698] __blk_mq_run_hw_queue+0xa4/0x11c
[ 214.293698] __blk_mq_run_hw_queue+0xa4/0x11c
[ 214.302408] __blk_mq_delay_run_hw_queue+0x1a4/0x208
[ 214.302408] __blk_mq_delay_run_hw_queue+0x1a4/0x208
[ 214.312337] blk_mq_run_hw_queue+0x9c/0xf8
[ 214.312337] blk_mq_run_hw_queue+0x9c/0xf8
[ 214.320525] blk_mq_sched_insert_requests+0x58/0x84
[ 214.320525] blk_mq_sched_insert_requests+0x58/0x84
[ 214.330280] blk_mq_flush_plug_list+0x144/0x16c
[ 214.330280] blk_mq_flush_plug_list+0x144/0x16c
[ 214.339339] blk_flush_plug_list+0xe8/0x244
[ 214.339339] blk_flush_plug_list+0xe8/0x244
[ 214.347700] blk_finish_plug+0x30/0x40
[ 214.347700] blk_finish_plug+0x30/0x40
[ 214.355191] sys_io_submit+0x110/0x214
[ 214.355191] sys_io_submit+0x110/0x214
[ 214.362682] el0_svc_naked+0x30/0x34
[ 214.362682] el0_svc_naked+0x30/0x34
Jobs: 1 (f=1): [M] [22.8% done] [0K/0K /s] [0 /0 iops] [eta 01m:18s]
Jobs: 1 (f=1): [M] [23.8% done] [0K/0K /s] [0 /0 iops] [eta 01m:17s]
[ 215.771524] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 215.771524] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 215.782502] 0-....: (1 GPs behind) idle=ac6/1/4611686018427387904 softirq=4021/4728 fqs=2627
[ 215.782502] 0-....: (1 GPs behind) idle=ac6/1/4611686018427387904 softirq=4021/4728 fqs=2627
[ 215.799745] (detected by 5, t=5259 jiffies, g=-107, c=-108, q=1)
[ 215.799745] (detected by 5, t=5259 jiffies, g=-107, c=-108, q=1)
[ 215.811944] Task dump for CPU 0:
[ 215.811944] Task dump for CPU 0:
[ 215.818389] kworker/0:2H R running task 0 3757 2 0x0000002a
[ 215.818389] kworker/0:2H R running task 0 3757 2 0x0000002a
[ 215.832502] Workqueue: kblockd blk_mq_run_work_fn
[ 215.832502] Workqueue: kblockd blk_mq_run_work_fn
[ 215.841909] Call trace:
[ 215.841909] Call trace:
[ 215.846789] __switch_to+0x90/0xcc
[ 215.846789] __switch_to+0x90/0xcc
[ 215.853583] 0x674238a07d45b300
[ 215.853583] 0x674238a07d45b300
Jobs: 1 (f=1): [M] [24.8% done] [0K/0K /s] [0 /0 iops] [eta 01m:16s]
Jobs: 1 (f=1): [M] [25.7% done] [0K/0K /s] [0 /0 iops] [eta 01m:15s]
Jobs: 1 (f=1): [M] [26.7% done] [0K/0K /s] [0 /0 iops] [eta 01m:14s]
Jobs: 1 (f=1): [M] [27.7% done] [0K/0K /s] [0 /0 iops] [eta 01m:13s]
Jobs: 1 (f=1): [M] [28.7% done] [0K/0K /s] [0 /0 iops] [eta 01m:12s]
Jobs: 1 (f=1): [M] [29.7% done] [0K/0K /s] [0 /0 iops] [eta 01m:11s]
Jobs: 1 (f=1): [M] [30.7% done] [0K/0K /s] [0 /0 iops] [eta 01m:10s]
Jobs: 1 (f=1): [M] [31.7% done] [0K/0K /s] [0 /0 iops] [eta 01m:09s]
Jobs: 1 (f=1): [M] [32.7% done] [0K/0K /s] [0 /0 iops] [eta 01m:08s]
Jobs: 1 (f=1): [M] [33.7% done] [0K/0K /s] [0 /0 iops] [eta 01m:07s]
Jobs: 1 (f=1): [M] [34.7% done] [0K/0K /s] [0 /0 iops] [eta 01m:06s]
Jobs: 1 (f=1): [M] [35.6% done] [0K/0K /s] [0 /0 iops] [eta 01m:05s]
Jobs: 1 (f=1): [M] [36.6% done] [0K/0K /s] [0 /0 iops] [eta 01m:04s]
Jobs: 1 (f=1): [M] [37.6% done] [0K/0K /s] [0 /0 iops] [eta 01m:03s]
Jobs: 1 (f=1): [M] [38.6% done] [0K/0K /s] [0 /0 iops] [eta 01m:02s]
Jobs: 1 (f=1): [M] [39.6% done] [0K/0K /s] [0 /0 iops] [eta 01m:01s]
Jobs: 1 (f=1): [M] [40.6% done] [0K/0K /s] [0 /0 iops] [eta 01m:00s]
Jobs: 1 (f=1): [M] [41.6% done] [0K/0K /s] [0 /0 iops] [eta 00m:59s]
Jobs: 1 (f=1): [M] [42.6% done] [0K/0K /s] [0 /0 iops] [eta 00m:58s]
Jobs: 1 (f=1): [M] [43.6% done] [0K/0K /s] [0 /0 iops] [eta 00m:57s]
Jobs: 1 (f=1): [M] [44.6% done] [0K/0K /s] [0 /0 iops] [eta 00m:56s]
Jobs: 1 (f=1): [M] [45.5% done] [0K/0K /s] [0 /0 iops] [eta 00m:55s]
Jobs: 1 (f=1): [M] [46.5% done] [0K/0K /s] [0 /0 iops] [eta 00m:54s]
Jobs: 1 (f=1): [M] [47.5% done] [0K/0K /s] [0 /0 iops] [eta 00m:53s]
Jobs: 1 (f=1): [M] [48.5% done] [0K/0K /s] [0 /0 iops] [eta 00m:52s]
[ 240.763523] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/0:2H:3757]
[ 240.763523] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/0:2H:3757]
[ 240.778502] Modules linked in: hisi_sas_v3_hw
[ 240.778502] Modules linked in: hisi_sas_v3_hw
[ 240.787214] CPU: 0 PID: 3757 Comm: kworker/0:2H Tainted: G D W 4.18.0-rc1-12443-g7e2d9d9-dirty #1077
[ 240.787214] CPU: 0 PID: 3757 Comm: kworker/0:2H Tainted: G D W 4.18.0-rc1-12443-g7e2d9d9-dirty #1077
[ 240.807766] Hardware name: Huawei D06/D06, BIOS Hisilicon D06 UEFI RC0 - B601 (V6.01) 11/08/2018
[ 240.807766] Hardware name: Huawei D06/D06, BIOS Hisilicon D06 UEFI RC0 - B601 (V6.01) 11/08/2018
[ 240.825358] Workqueue: kblockd blk_mq_run_work_fn
[ 240.825358] Workqueue: kblockd blk_mq_run_work_fn
[ 240.834766] pstate: 80c00009 (Nzcv daif +PAN +UAO)
[ 240.834766] pstate: 80c00009 (Nzcv daif +PAN +UAO)
[ 240.844346] pc : _raw_spin_lock+0x34/0x44
[ 240.844346] pc : _raw_spin_lock+0x34/0x44
[ 240.852361] lr : dd_dispatch_request+0x2c/0x1e0
[ 240.852361] lr : dd_dispatch_request+0x2c/0x1e0
[ 240.861418] sp : ffff00002462bc60
[ 240.861418] sp : ffff00002462bc60
[ 240.868038] x29: ffff00002462bc60 x28: ffff000009973ce8
[ 240.868038] x29: ffff00002462bc60 x28: ffff000009973ce8
[ 240.878664] x27: ffff8027dceac938 x26: ffff000009467558
[ 240.878664] x27: ffff8027dceac938 x26: ffff000009467558
[ 240.889291] x25: 0000000000000000 x24: ffff000009451000
[ 240.889291] x25: 0000000000000000 x24: ffff000009451000
[ 240.899917] x23: ffff000009451000 x22: ffff8027a5a37200
[ 240.899917] x23: ffff000009451000 x22: ffff8027a5a37200
[ 240.910544] x21: ffff8027b24f4c5c x20: ffff00002462bce8
[ 240.910544] x21: ffff8027b24f4c5c x20: ffff00002462bce8
[ 240.921170] x19: ffff8027b24f4c00 x18: 000000000000000e
[ 240.921170] x19: ffff8027b24f4c00 x18: 000000000000000e
[ 240.931797] x17: 0000000000000001 x16: 0000000000000019
[ 240.931797] x17: 0000000000000001 x16: 0000000000000019
[ 240.942423] x15: ffffffffffffffff x14: 0000000000000001
[ 240.942423] x15: ffffffffffffffff x14: 0000000000000001
[ 240.953050] x13: 00000000c0a80101 x12: 0000000000000001
[ 240.953050] x13: 00000000c0a80101 x12: 0000000000000001
[ 240.963676] x11: 0000000000001000 x10: 0000000000000950
[ 240.963676] x11: 0000000000001000 x10: 0000000000000950
[ 240.974303] x9 : ffff00002462bd50 x8 : 0000000000000018
[ 240.974303] x9 : ffff00002462bd50 x8 : 0000000000000018
[ 240.984929] x7 : fefefefefefefeff x6 : 0000000000000000
[ 240.984929] x7 : fefefefefefefeff x6 : 0000000000000000
[ 240.995556] x5 : 000000000000001d x4 : 000000000000001e
[ 240.995556] x5 : 000000000000001d x4 : 000000000000001e
[ 241.006182] x3 : 0000000000003996 x2 : 0000000000000001
[ 241.006182] x3 : 0000000000003996 x2 : 0000000000000001
[ 241.016808] x1 : 0000000039973996 x0 : ffff8027b24f4c5c
[ 241.016808] x1 : 0000000039973996 x0 : ffff8027b24f4c5c
[ 241.027435] Call trace:
[ 241.027435] Call trace:
[ 241.032314] _raw_spin_lock+0x34/0x44
[ 241.032314] _raw_spin_lock+0x34/0x44
[ 241.039631] blk_mq_do_dispatch_sched+0x4c/0x104
[ 241.039631] blk_mq_do_dispatch_sched+0x4c/0x104
[ 241.048864] blk_mq_sched_dispatch_requests+0x130/0x1bc
[ 241.048864] blk_mq_sched_dispatch_requests+0x130/0x1bc
[ 241.059316] __blk_mq_run_hw_queue+0xa4/0x11c
[ 241.059316] __blk_mq_run_hw_queue+0xa4/0x11c
[ 241.068025] blk_mq_run_work_fn+0x18/0x20
[ 241.068025] blk_mq_run_work_fn+0x18/0x20
[ 241.076039] process_one_work+0x140/0x2b0
[ 241.076039] process_one_work+0x140/0x2b0
[ 241.084052] worker_thread+0x130/0x3e4
[ 241.084052] worker_thread+0x130/0x3e4
[ 241.091543] kthread+0xfc/0x128
[ 241.091543] kthread+0xfc/0x128
[ 241.097815] ret_from_fork+0x10/0x18
[ 241.097815] ret_from_fork+0x10/0x18
Jobs: 1 (f=1): [M] [49.5% done] [0K/0K /s] [0 /0 iops] [eta 00m:51s]
Jobs: 1 (f=1): [M] [50.5% done] [0K/0K /s] [0 /0 iops] [eta 00m:50s]
Jobs: 1 (f=1): [M] [51.5% done] [0K/0K /s] [0 /0 iops] [eta 00m:49s]
Jobs: 1 (f=1): [M] [52.5% done] [0K/0K /s] [0 /0 iops] [eta 00m:48s]
Jobs: 1 (f=1): [M] [53.5% done] [0K/0K /s] [0 /0 iops] [eta 00m:47s]
Jobs: 1 (f=1): [M] [54.5% done] [0K/0K /s] [0 /0 iops] [eta 00m:46s]
Jobs: 1 (f=1): [M] [55.4% done] [0K/0K /s] [0 /0 iops] [eta 00m:45s]
Jobs: 1 (f=1): [M] [56.4% done] [0K/0K /s] [0 /0 iops] [eta 00m:44s]
Jobs: 1 (f=1): [M] [57.4% done] [0K/0K /s] [0 /0 iops] [eta 00m:43s]
Jobs: 1 (f=1): [M] [58.4% done] [0K/0K /s] [0 /0 iops] [eta 00m:42s]
Jobs: 1 (f=1): [M] [59.4% done] [0K/0K /s] [0 /0 iops] [eta 00m:41s]
Jobs: 1 (f=1): [M] [60.4% done] [0K/0K /s] [0 /0 iops] [eta 00m:40s]
Jobs: 1 (f=1): [M] [61.4% done] [0K/0K /s] [0 /0 iops] [eta 00m:39s]
Jobs: 1 (f=1): [M] [62.4% done] [0K/0K /s] [0 /0 iops] [eta 00m:38s]
Jobs: 1 (f=1): [M] [63.4% done] [0K/0K /s] [0 /0 iops] [eta 00m:37s]
Jobs: 1 (f=1): [M] [64.4% done] [0K/0K /s] [0 /0 iops] [eta 00m:36s]
Jobs: 1 (f=1): [M] [65.3% done] [0K/0K /s] [0 /0 iops] [eta 00m:35s]
Jobs: 1 (f=1): [M] [66.3% done] [0K/0K /s] [0 /0 iops] [eta 00m:34s]
Jobs: 1 (f=1): [M] [67.3% done] [0K/0K /s] [0 /0 iops] [eta 00m:33s]
Jobs: 1 (f=1): [M] [68.3% done] [0K/0K /s] [0 /0 iops] [eta 00m:32s]
Jobs: 1 (f=1): [M] [69.3% done] [0K/0K /s] [0 /0 iops] [eta 00m:31s]
Jobs: 1 (f=1): [M] [70.3% done] [0K/0K /s] [0 /0 iops] [eta 00m:30s]
Jobs: 1 (f=1): [M] [71.3% done] [0K/0K /s] [0 /0 iops] [eta 00m:29s]
Jobs: 1 (f=1): [M] [72.3% done] [0K/0K /s] [0 /0 iops] [eta 00m:28s]
Jobs: 1 (f=1): [M] [73.3% done] [0K/0K /s] [0 /0 iops] [eta 00m:27s]
Jobs: 1 (f=1): [M] [74.3% done] [0K/0K /s] [0 /0 iops] [eta 00m:26s]
Jobs: 1 (f=1): [M] [75.2% done] [0K/0K /s] [0 /0 iops] [eta 00m:25s]
Jobs: 1 (f=1): [M] [76.2% done] [0K/0K /s] [0 /0 iops] [eta 00m:24s]
Jobs: 1 (f=1): [M] [77.2% done] [0K/0K /s] [0 /0 iops] [eta 00m:23s]
Jobs: 1 (f=1): [M] [78.2% done] [0K/0K /s] [0 /0 iops] [eta 00m:22s]
Jobs: 1 (f=1): [M] [79.2% done] [0K/0K /s] [0 /0 iops] [eta 00m:21s]
Jobs: 1 (f=1): [M] [80.2% done] [0K/0K /s] [0 /0 iops] [eta 00m:20s]
Jobs: 1 (f=1): [M] [81.2% done] [0K/0K /s] [0 /0 iops] [eta 00m:19s]
Jobs: 1 (f=1): [M] [82.2% done] [0K/0K /s] [0 /0 iops] [eta 00m:18s]
Jobs: 1 (f=1): [M] [83.2% done] [0K/0K /s] [0 /0 iops] [eta 00m:17s]
Jobs: 1 (f=1): [M] [84.2% done] [0K/0K /s] [0 /0 iops] [eta 00m:16s]
[ 276.763523] INFO: rcu_preempt self-detected stall on CPU
[ 276.763523] INFO: rcu_preempt self-detected stall on CPU
[ 276.771524] INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 276.771524] INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 276.774151] 0-....: (1 GPs behind) idle=ac6/1/4611686018427387906 softirq=4726/4728 fqs=10422
[ 276.774151] 0-....: (1 GPs behind) idle=ac6/1/4611686018427387906 softirq=4726/4728 fqs=10422
[ 276.774153]
[ 276.774153]
[ 276.785476] Tasks blocked on level-1 rcu_node (CPUs 0-15):
[ 276.785476] Tasks blocked on level-1 rcu_node (CPUs 0-15):
[ 276.802894] (t=21014 jiffies g=1596 c=1595 q=339)
[ 276.802894] (t=21014 jiffies g=1596 c=1595 q=339)
[ 276.806028] P3755
[ 276.806028] P3755
[ 276.830765] 0-....: (1 GPs behind) idle=ac6/1/4611686018427387906 softirq=4726/4728 fqs=10424
[ 276.830765] 0-....: (1 GPs behind) idle=ac6/1/4611686018427387906 softirq=4726/4728 fqs=10424
[ 276.848183]
[ 276.848183]
[ 276.848184] Task dump for CPU 0:
[ 276.848184] Task dump for CPU 0:
[ 276.848186] kworker/0:2H R
[ 276.848186] kworker/0:2H R
[ 276.851324] (detected by 24, t=21026 jiffies, g=1596, c=1595, q=339)
[ 276.851324] (detected by 24, t=21026 jiffies, g=1596, c=1595, q=339)
[ 276.857767] running task 0 3757 2 0x0000002a
[ 276.857767] running task 0 3757 2 0x0000002a
[ 276.887553] Workqueue: kblockd blk_mq_run_work_fn
[ 276.887553] Workqueue: kblockd blk_mq_run_work_fn
[ 276.896960] Call trace:
[ 276.896960] Call trace:
[ 276.901839] dump_backtrace+0x0/0x180
[ 276.901839] dump_backtrace+0x0/0x180
[ 276.909157] show_stack+0x14/0x1c
[ 276.909157] show_stack+0x14/0x1c
[ 276.915777] sched_show_task+0x16c/0x184
[ 276.915777] sched_show_task+0x16c/0x184
[ 276.923616] dump_cpu_task+0x40/0x4c
[ 276.923616] dump_cpu_task+0x40/0x4c
[ 276.930759] rcu_dump_cpu_stacks+0x98/0xd8
[ 276.930759] rcu_dump_cpu_stacks+0x98/0xd8
[ 276.938946] rcu_check_callbacks+0x750/0x9c0
[ 276.938946] rcu_check_callbacks+0x750/0x9c0
[ 276.947482] update_process_times+0x2c/0x54
[ 276.947482] update_process_times+0x2c/0x54
[ 276.955845] tick_sched_handle.isra.14+0x24/0x68
[ 276.955845] tick_sched_handle.isra.14+0x24/0x68
[ 276.965077] tick_sched_timer+0x48/0x98
[ 276.965077] tick_sched_timer+0x48/0x98
[ 276.972742] __hrtimer_run_queues+0x100/0x158
[ 276.972742] __hrtimer_run_queues+0x100/0x158
[ 276.981452] hrtimer_interrupt+0xe4/0x248
[ 276.981452] hrtimer_interrupt+0xe4/0x248
[ 276.989466] arch_timer_handler_phys+0x2c/0x38
[ 276.989466] arch_timer_handler_phys+0x2c/0x38
[ 276.998350] handle_percpu_devid_irq+0x78/0x12c
[ 276.998350] handle_percpu_devid_irq+0x78/0x12c
[ 277.007408] generic_handle_irq+0x24/0x38
[ 277.007408] generic_handle_irq+0x24/0x38
[ 277.015422] __handle_domain_irq+0x60/0xac
[ 277.015422] __handle_domain_irq+0x60/0xac
[ 277.023609] gic_handle_irq+0xd4/0x184
[ 277.023609] gic_handle_irq+0xd4/0x184
[ 277.031100] el1_irq+0xb0/0x128
[ 277.031100] el1_irq+0xb0/0x128
[ 277.037372] _raw_spin_lock+0x34/0x44
[ 277.037372] _raw_spin_lock+0x34/0x44
[ 277.044689] blk_mq_do_dispatch_sched+0x4c/0x104
[ 277.044689] blk_mq_do_dispatch_sched+0x4c/0x104
[ 277.053922] blk_mq_sched_dispatch_requests+0x130/0x1bc
[ 277.053922] blk_mq_sched_dispatch_requests+0x130/0x1bc
[ 277.064373] __blk_mq_run_hw_queue+0xa4/0x11c
[ 277.064373] __blk_mq_run_hw_queue+0xa4/0x11c
[ 277.073083] blk_mq_run_work_fn+0x18/0x20
[ 277.073083] blk_mq_run_work_fn+0x18/0x20
[ 277.081096] process_one_work+0x140/0x2b0
[ 277.081096] process_one_work+0x140/0x2b0
[ 277.089109] worker_thread+0x130/0x3e4
[ 277.089109] worker_thread+0x130/0x3e4
[ 277.096601] kthread+0xfc/0x128
[ 277.096601] kthread+0xfc/0x128
[ 277.102872] ret_from_fork+0x10/0x18
[ 277.102872] ret_from_fork+0x10/0x18
[ 277.110016] Task dump for CPU 0:
[ 277.110016] Task dump for CPU 0:
[ 277.116462] kworker/0:2H R running task 0 3757 2 0x0000002a
[ 277.116462] kworker/0:2H R running task 0 3757 2 0x0000002a
[ 277.130576] Workqueue: kblockd blk_mq_run_work_fn
[ 277.130576] Workqueue: kblockd blk_mq_run_work_fn
[ 277.139983] Call trace:
[ 277.139983] Call trace:
[ 277.144862] __switch_to+0x90/0xcc
[ 277.144862] __switch_to+0x90/0xcc
[ 277.151657] 0x674238a07d45b300
[ 277.151657] 0x674238a07d45b300
[ 277.157931] fio D 0 3755 3753 0x0000002c
[ 277.157931] fio D 0 3755 3753 0x0000002c
[ 277.168905] Call trace:
[ 277.168905] Call trace:
[ 277.173784] __switch_to+0x90/0xcc
[ 277.173784] __switch_to+0x90/0xcc
[ 277.180579] __schedule+0x1b8/0x5cc
[ 277.180579] __schedule+0x1b8/0x5cc
[ 277.187547] schedule+0x38/0x9c
[ 277.187547] schedule+0x38/0x9c
[ 277.193819] schedule_timeout+0x1b8/0x320
[ 277.193819] schedule_timeout+0x1b8/0x320
[ 277.201832] wait_for_common+0xb4/0x164
[ 277.201832] wait_for_common+0xb4/0x164
[ 277.209497] wait_for_completion+0x14/0x1c
[ 277.209497] wait_for_completion+0x14/0x1c
[ 277.217685] exit_aio+0x100/0x114
[ 277.217685] exit_aio+0x100/0x114
[ 277.224306] mmput+0x48/0x118
[ 277.224306] mmput+0x48/0x118
[ 277.230229] do_exit+0x278/0x978
[ 277.230229] do_exit+0x278/0x978
[ 277.236676] die+0x1c8/0x1f4
[ 277.236676] die+0x1c8/0x1f4
[ 277.242426] die_kernel_fault+0x64/0x74
[ 277.242426] die_kernel_fault+0x64/0x74
[ 277.250091] __do_kernel_fault+0x84/0xac
[ 277.250091] __do_kernel_fault+0x84/0xac
[ 277.257930] do_page_fault+0x1dc/0x3e8
[ 277.257930] do_page_fault+0x1dc/0x3e8
[ 277.265421] do_translation_fault+0x68/0x74
[ 277.265421] do_translation_fault+0x68/0x74
[ 277.273782] do_mem_abort+0x4c/0xf4
[ 277.273782] do_mem_abort+0x4c/0xf4
[ 277.280750] el1_da+0x20/0x80
[ 277.280750] el1_da+0x20/0x80
[ 277.286674] deadline_remove_request+0x5c/0xd0
[ 277.286674] deadline_remove_request+0x5c/0xd0
[ 277.295558] dd_dispatch_request+0x184/0x1e0
[ 277.295558] dd_dispatch_request+0x184/0x1e0
[ 277.304094] blk_mq_do_dispatch_sched+0x4c/0x104
[ 277.304094] blk_mq_do_dispatch_sched+0x4c/0x104
[ 277.313327] blk_mq_sched_dispatch_requests+0x130/0x1bc
[ 277.313327] blk_mq_sched_dispatch_requests+0x130/0x1bc
[ 277.323778] __blk_mq_run_hw_queue+0xa4/0x11c
[ 277.323778] __blk_mq_run_hw_queue+0xa4/0x11c
[ 277.332488] __blk_mq_delay_run_hw_queue+0x1a4/0x208
[ 277.332488] __blk_mq_delay_run_hw_queue+0x1a4/0x208
[ 277.342417] blk_mq_run_hw_queue+0x9c/0xf8
[ 277.342417] blk_mq_run_hw_queue+0x9c/0xf8
[ 277.350605] blk_mq_sched_insert_requests+0x58/0x84
[ 277.350605] blk_mq_sched_insert_requests+0x58/0x84
[ 277.360360] blk_mq_flush_plug_list+0x144/0x16c
[ 277.360360] blk_mq_flush_plug_list+0x144/0x16c
[ 277.369418] blk_flush_plug_list+0xe8/0x244
[ 277.369418] blk_flush_plug_list+0xe8/0x244
[ 277.377780] blk_finish_plug+0x30/0x40
[ 277.377780] blk_finish_plug+0x30/0x40
[ 277.385271] sys_io_submit+0x110/0x214
[ 277.385271] sys_io_submit+0x110/0x214
[ 277.392762] el0_svc_naked+0x30/0x34
[ 277.392762] el0_svc_naked+0x30/0x34
Jobs: 1 (f=1): [M] [85.1% done] [0K/0K /s] [0 /0 iops] [eta 00m:15s]
Jobs: 1 (f=1): [M] [86.1% done] [0K/0K /s] [0 /0 iops] [eta 00m:14s]
[ 278.791524] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 278.791524] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 278.802504] 0-....: (1 GPs behind) idle=ac6/1/4611686018427387904 softirq=4021/4728 fqs=10497
[ 278.802504] 0-....: (1 GPs behind) idle=ac6/1/4611686018427387904 softirq=4021/4728 fqs=10497
[ 278.819922] (detected by 7, t=21013 jiffies, g=-107, c=-108, q=1)
[ 278.819922] (detected by 7, t=21013 jiffies, g=-107, c=-108, q=1)
[ 278.832294] Task dump for CPU 0:
[ 278.832294] Task dump for CPU 0:
[ 278.838740] kworker/0:2H R running task 0 3757 2 0x0000002a
[ 278.838740] kworker/0:2H R running task 0 3757 2 0x0000002a
[ 278.852855] Workqueue: kblockd blk_mq_run_work_fn
[ 278.852855] Workqueue: kblockd blk_mq_run_work_fn
[ 278.862262] Call trace:
[ 278.862262] Call trace:
[ 278.867142] __switch_to+0x90/0xcc
[ 278.867142] __switch_to+0x90/0xcc
[ 278.873937] 0x674238a07d45b300
[ 278.873937] 0x674238a07d45b300
Jobs: 1 (f=1): [M] [87.1% done] [0K/0K /s] [0 /0 iops] [eta 00m:13s]
Jobs: 1 (f=1): [M] [88.1% done] [0K/0K /s] [0 /0 iops] [eta 00m:12s]
Jobs: 1 (f=1): [M] [89.1% done] [0K/0K /s] [0 /0 iops] [eta 00m:11s]
Jobs: 1 (f=1): [M] [90.1% done] [0K/0K /s] [0 /0 iops] [eta 00m:10s]
Jobs: 1 (f=1): [M] [91.1% done] [0K/0K /s] [0 /0 iops] [eta 00m:09s]
Jobs: 1 (f=1): [M] [92.1% done] [0K/0K /s] [0 /0 iops] [eta 00m:08s]
Jobs: 1 (f=1): [M] [93.1% done] [0K/0K /s] [0 /0 iops] [eta 00m:07s]
Jobs: 1 (f=1): [M] [94.1% done] [0K/0K /s] [0 /0 iops] [eta 00m:06s]
Jobs: 1 (f=1): [M] [95.0% done] [0K/0K /s] [0 /0 iops] [eta 00m:05s]
Jobs: 1 (f=1): [M] [96.0% done] [0K/0K /s] [0 /0 iops] [eta 00m:04s]
Jobs: 1 (f=1): [M] [97.0% done] [0K/0K /s] [0 /0 iops] [eta 00m:03s]
Jobs: 1 (f=1): [M] [98.0% done] [0K/0K /s] [0 /0 iops] [eta 00m:02s]
Jobs: 1 (f=1): [M] [99.0% done] [0K/0K /s] [0 /0 iops] [eta 00m:01s]
Jobs: 1 (f=1): [M] [100.0% done] [0K/0K /s] [0 /0 iops] [eta 00m:00s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 15h:22m:50s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 15h:31m:58s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 15h:41m:06s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 15h:50m:15s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 15h:59m:23s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 16h:08m:31s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 16h:17m:39s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 16h:26m:48s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 16h:35m:56s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 16h:45m:04s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 16h:54m:12s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 17h:03m:20s]
[ 304.763524] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2H:3757]
[ 304.763524] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2H:3757]
[ 304.778503] Modules linked in: hisi_sas_v3_hw
[ 304.778503] Modules linked in: hisi_sas_v3_hw
[ 304.787215] CPU: 0 PID: 3757 Comm: kworker/0:2H Tainted: G D W L 4.18.0-rc1-12443-g7e2d9d9-dirty #1077
[ 304.787215] CPU: 0 PID: 3757 Comm: kworker/0:2H Tainted: G D W L 4.18.0-rc1-12443-g7e2d9d9-dirty #1077
[ 304.807766] Hardware name: Huawei D06/D06, BIOS Hisilicon D06 UEFI RC0 - B601 (V6.01) 11/08/2018
[ 304.807766] Hardware name: Huawei D06/D06, BIOS Hisilicon D06 UEFI RC0 - B601 (V6.01) 11/08/2018
[ 304.825359] Workqueue: kblockd blk_mq_run_work_fn
[ 304.825359] Workqueue: kblockd blk_mq_run_work_fn
[ 304.834766] pstate: 80c00009 (Nzcv daif +PAN +UAO)
[ 304.834766] pstate: 80c00009 (Nzcv daif +PAN +UAO)
[ 304.844347] pc : _raw_spin_lock+0x34/0x44
[ 304.844347] pc : _raw_spin_lock+0x34/0x44
[ 304.852360] lr : dd_dispatch_request+0x2c/0x1e0
[ 304.852360] lr : dd_dispatch_request+0x2c/0x1e0
[ 304.861417] sp : ffff00002462bc60
[ 304.861417] sp : ffff00002462bc60
[ 304.868037] x29: ffff00002462bc60 x28: ffff000009973ce8
[ 304.868037] x29: ffff00002462bc60 x28: ffff000009973ce8
[ 304.878663] x27: ffff8027dceac938 x26: ffff000009467558
[ 304.878663] x27: ffff8027dceac938 x26: ffff000009467558
[ 304.889290] x25: 0000000000000000 x24: ffff000009451000
[ 304.889290] x25: 0000000000000000 x24: ffff000009451000
[ 304.899916] x23: ffff000009451000 x22: ffff8027a5a37200
[ 304.899916] x23: ffff000009451000 x22: ffff8027a5a37200
[ 304.910543] x21: ffff8027b24f4c5c x20: ffff00002462bce8
[ 304.910543] x21: ffff8027b24f4c5c x20: ffff00002462bce8
[ 304.921169] x19: ffff8027b24f4c00 x18: 000000000000000e
[ 304.921169] x19: ffff8027b24f4c00 x18: 000000000000000e
[ 304.931796] x17: 0000000000000001 x16: 0000000000000019
[ 304.931796] x17: 0000000000000001 x16: 0000000000000019
[ 304.942422] x15: ffffffffffffffff x14: 0000000000000001
[ 304.942422] x15: ffffffffffffffff x14: 0000000000000001
[ 304.953049] x13: 00000000c0a80101 x12: 0000000000000001
[ 304.953049] x13: 00000000c0a80101 x12: 0000000000000001
[ 304.963675] x11: 0000000000001000 x10: 0000000000000950
[ 304.963675] x11: 0000000000001000 x10: 0000000000000950
[ 304.974302] x9 : ffff00002462bd50 x8 : 0000000000000018
[ 304.974302] x9 : ffff00002462bd50 x8 : 0000000000000018
[ 304.984928] x7 : fefefefefefefeff x6 : 0000000000000000
[ 304.984928] x7 : fefefefefefefeff x6 : 0000000000000000
[ 304.995555] x5 : 000000000000001d x4 : 000000000000001e
[ 304.995555] x5 : 000000000000001d x4 : 000000000000001e
[ 305.006181] x3 : 0000000000003996 x2 : 0000000000000001
[ 305.006181] x3 : 0000000000003996 x2 : 0000000000000001
[ 305.016808] x1 : 0000000039973996 x0 : ffff8027b24f4c5c
[ 305.016808] x1 : 0000000039973996 x0 : ffff8027b24f4c5c
[ 305.027434] Call trace:
[ 305.027434] Call trace:
[ 305.032313] _raw_spin_lock+0x34/0x44
[ 305.032313] _raw_spin_lock+0x34/0x44
[ 305.039631] blk_mq_do_dispatch_sched+0x4c/0x104
[ 305.039631] blk_mq_do_dispatch_sched+0x4c/0x104
[ 305.048863] blk_mq_sched_dispatch_requests+0x130/0x1bc
[ 305.048863] blk_mq_sched_dispatch_requests+0x130/0x1bc
[ 305.059315] __blk_mq_run_hw_queue+0xa4/0x11c
[ 305.059315] __blk_mq_run_hw_queue+0xa4/0x11c
[ 305.068025] blk_mq_run_work_fn+0x18/0x20
[ 305.068025] blk_mq_run_work_fn+0x18/0x20
[ 305.076038] process_one_work+0x140/0x2b0
[ 305.076038] process_one_work+0x140/0x2b0
[ 305.084051] worker_thread+0x130/0x3e4
[ 305.084051] worker_thread+0x130/0x3e4
[ 305.091542] kthread+0xfc/0x128
[ 305.091542] kthread+0xfc/0x128
[ 305.097815] ret_from_fork+0x10/0x18
[ 305.097815] ret_from_fork+0x10/0x18
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 17h:12m:29s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 17h:21m:37s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 17h:30m:45s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 17h:39m:53s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 17h:49m:02s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 17h:58m:10s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 18h:07m:18s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 18h:16m:26s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 18h:25m:34s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 18h:34m:43s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 18h:43m:51s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 18h:52m:59s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 19h:02m:07s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 19h:11m:16s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 19h:20m:24s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 19h:29m:32s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 19h:38m:40s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 19h:47m:48s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 19h:56m:57s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 20h:06m:05s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 20h:15m:13s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 20h:24m:21s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 20h:33m:30s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 20h:42m:38s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 20h:51m:46s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 21h:00m:54s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 21h:10m:02s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 21h:19m:11s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 21h:28m:19s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 21h:37m:27s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 21h:46m:35s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 21h:55m:44s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 22h:04m:52s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 22h:14m:00s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 22h:23m:08s]
[ 339.783523] INFO: rcu_preempt self-detected stall on CPU
[ 339.783523] INFO: rcu_preempt self-detected stall on CPU
[ 339.791524] INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 339.791524] INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 339.794152] 0-....: (1 GPs behind) idle=ac6/1/4611686018427387906 softirq=4726/4728 fqs=18226
[ 339.794152] 0-....: (1 GPs behind) idle=ac6/1/4611686018427387906 softirq=4726/4728 fqs=18226
[ 339.805476]
[ 339.805476]
[ 339.822894] Tasks blocked on level-1 rcu_node (CPUs 0-15):
[ 339.822894] Tasks blocked on level-1 rcu_node (CPUs 0-15):
[ 339.822897] (t=36769 jiffies g=1596 c=1595 q=621)
[ 339.822897] (t=36769 jiffies g=1596 c=1595 q=621)
[ 339.826032] P3755
[ 339.826032] P3755
[ 339.850768] 0-....: (1 GPs behind) idle=ac6/1/4611686018427387906 softirq=4726/4728 fqs=18230
[ 339.850768] 0-....: (1 GPs behind) idle=ac6/1/4611686018427387906 softirq=4726/4728 fqs=18230
[ 339.868186] Task dump for CPU 0:
[ 339.868186] Task dump for CPU 0:
[ 339.868186] (detected by 5, t=36781 jiffies, g=1596, c=1595, q=621)
[ 339.868186] (detected by 5, t=36781 jiffies, g=1596, c=1595, q=621)
[ 339.874633] kworker/0:2H R running task 0 3757 2 0x0000002a
[ 339.874633] kworker/0:2H R running task 0 3757 2 0x0000002a
[ 339.901458] Workqueue: kblockd blk_mq_run_work_fn
[ 339.901458] Workqueue: kblockd blk_mq_run_work_fn
[ 339.910864] Call trace:
[ 339.910864] Call trace:
[ 339.915744] dump_backtrace+0x0/0x180
[ 339.915744] dump_backtrace+0x0/0x180
[ 339.923061] show_stack+0x14/0x1c
[ 339.923061] show_stack+0x14/0x1c
[ 339.929682] sched_show_task+0x16c/0x184
[ 339.929682] sched_show_task+0x16c/0x184
[ 339.937521] dump_cpu_task+0x40/0x4c
[ 339.937521] dump_cpu_task+0x40/0x4c
[ 339.944664] rcu_dump_cpu_stacks+0x98/0xd8
[ 339.944664] rcu_dump_cpu_stacks+0x98/0xd8
[ 339.952852] rcu_check_callbacks+0x750/0x9c0
[ 339.952852] rcu_check_callbacks+0x750/0x9c0
[ 339.961388] update_process_times+0x2c/0x54
[ 339.961388] update_process_times+0x2c/0x54
[ 339.969750] tick_sched_handle.isra.14+0x24/0x68
[ 339.969750] tick_sched_handle.isra.14+0x24/0x68
[ 339.978982] tick_sched_timer+0x48/0x98
[ 339.978982] tick_sched_timer+0x48/0x98
[ 339.986647] __hrtimer_run_queues+0x100/0x158
[ 339.986647] __hrtimer_run_queues+0x100/0x158
[ 339.995357] hrtimer_interrupt+0xe4/0x248
[ 339.995357] hrtimer_interrupt+0xe4/0x248
[ 340.003371] arch_timer_handler_phys+0x2c/0x38
[ 340.003371] arch_timer_handler_phys+0x2c/0x38
[ 340.012254] handle_percpu_devid_irq+0x78/0x12c
[ 340.012254] handle_percpu_devid_irq+0x78/0x12c
[ 340.021313] generic_handle_irq+0x24/0x38
[ 340.021313] generic_handle_irq+0x24/0x38
[ 340.029326] __handle_domain_irq+0x60/0xac
[ 340.029326] __handle_domain_irq+0x60/0xac
[ 340.037514] gic_handle_irq+0xd4/0x184
[ 340.037514] gic_handle_irq+0xd4/0x184
[ 340.045005] el1_irq+0xb0/0x128
[ 340.045005] el1_irq+0xb0/0x128
[ 340.051277] _raw_spin_lock+0x34/0x44
[ 340.051277] _raw_spin_lock+0x34/0x44
[ 340.058594] blk_mq_do_dispatch_sched+0x4c/0x104
[ 340.058594] blk_mq_do_dispatch_sched+0x4c/0x104
[ 340.067826] blk_mq_sched_dispatch_requests+0x130/0x1bc
[ 340.067826] blk_mq_sched_dispatch_requests+0x130/0x1bc
[ 340.078278] __blk_mq_run_hw_queue+0xa4/0x11c
[ 340.078278] __blk_mq_run_hw_queue+0xa4/0x11c
[ 340.086988] blk_mq_run_work_fn+0x18/0x20
[ 340.086988] blk_mq_run_work_fn+0x18/0x20
[ 340.095001] process_one_work+0x140/0x2b0
[ 340.095001] process_one_work+0x140/0x2b0
[ 340.103014] worker_thread+0x130/0x3e4
[ 340.103014] worker_thread+0x130/0x3e4
[ 340.110505] kthread+0xfc/0x128
[ 340.110505] kthread+0xfc/0x128
[ 340.116777] ret_from_fork+0x10/0x18
[ 340.116777] ret_from_fork+0x10/0x18
[ 340.123920] Task dump for CPU 0:
[ 340.123920] Task dump for CPU 0:
[ 340.130367] kworker/0:2H R running task 0 3757 2 0x0000002a
[ 340.130367] kworker/0:2H R running task 0 3757 2 0x0000002a
[ 340.144482] Workqueue: kblockd blk_mq_run_work_fn
[ 340.144482] Workqueue: kblockd blk_mq_run_work_fn
[ 340.153889] Call trace:
[ 340.153889] Call trace:
[ 340.158769] __switch_to+0x90/0xcc
[ 340.158769] __switch_to+0x90/0xcc
[ 340.165563] 0x674238a07d45b300
[ 340.165563] 0x674238a07d45b300
[ 340.171837] fio D 0 3755 3753 0x0000002c
[ 340.171837] fio D 0 3755 3753 0x0000002c
[ 340.182811] Call trace:
[ 340.182811] Call trace:
[ 340.187690] __switch_to+0x90/0xcc
[ 340.187690] __switch_to+0x90/0xcc
[ 340.194486] __schedule+0x1b8/0x5cc
[ 340.194486] __schedule+0x1b8/0x5cc
[ 340.201454] schedule+0x38/0x9c
[ 340.201454] schedule+0x38/0x9c
[ 340.207726] schedule_timeout+0x1b8/0x320
[ 340.207726] schedule_timeout+0x1b8/0x320
[ 340.215739] wait_for_common+0xb4/0x164
[ 340.215739] wait_for_common+0xb4/0x164
[ 340.223404] wait_for_completion+0x14/0x1c
[ 340.223404] wait_for_completion+0x14/0x1c
[ 340.231593] exit_aio+0x100/0x114
[ 340.231593] exit_aio+0x100/0x114
[ 340.238214] mmput+0x48/0x118
[ 340.238214] mmput+0x48/0x118
[ 340.244138] do_exit+0x278/0x978
[ 340.244138] do_exit+0x278/0x978
[ 340.250584] die+0x1c8/0x1f4
[ 340.250584] die+0x1c8/0x1f4
[ 340.256335] die_kernel_fault+0x64/0x74
[ 340.256335] die_kernel_fault+0x64/0x74
[ 340.264000] __do_kernel_fault+0x84/0xac
[ 340.264000] __do_kernel_fault+0x84/0xac
[ 340.271840] do_page_fault+0x1dc/0x3e8
[ 340.271840] do_page_fault+0x1dc/0x3e8
[ 340.279332] do_translation_fault+0x68/0x74
[ 340.279332] do_translation_fault+0x68/0x74
[ 340.287693] do_mem_abort+0x4c/0xf4
[ 340.287693] do_mem_abort+0x4c/0xf4
[ 340.294662] el1_da+0x20/0x80
[ 340.294662] el1_da+0x20/0x80
[ 340.300586] deadline_remove_request+0x5c/0xd0
[ 340.300586] deadline_remove_request+0x5c/0xd0
[ 340.309470] dd_dispatch_request+0x184/0x1e0
[ 340.309470] dd_dispatch_request+0x184/0x1e0
[ 340.318006] blk_mq_do_dispatch_sched+0x4c/0x104
[ 340.318006] blk_mq_do_dispatch_sched+0x4c/0x104
[ 340.327239] blk_mq_sched_dispatch_requests+0x130/0x1bc
[ 340.327239] blk_mq_sched_dispatch_requests+0x130/0x1bc
[ 340.337690] __blk_mq_run_hw_queue+0xa4/0x11c
[ 340.337690] __blk_mq_run_hw_queue+0xa4/0x11c
[ 340.346400] __blk_mq_delay_run_hw_queue+0x1a4/0x208
[ 340.346400] __blk_mq_delay_run_hw_queue+0x1a4/0x208
[ 340.356329] blk_mq_run_hw_queue+0x9c/0xf8
[ 340.356329] blk_mq_run_hw_queue+0x9c/0xf8
[ 340.364517] blk_mq_sched_insert_requests+0x58/0x84
[ 340.364517] blk_mq_sched_insert_requests+0x58/0x84
[ 340.374272] blk_mq_flush_plug_list+0x144/0x16c
[ 340.374272] blk_mq_flush_plug_list+0x144/0x16c
[ 340.383330] blk_flush_plug_list+0xe8/0x244
[ 340.383330] blk_flush_plug_list+0xe8/0x244
[ 340.391692] blk_finish_plug+0x30/0x40
[ 340.391692] blk_finish_plug+0x30/0x40
[ 340.399183] sys_io_submit+0x110/0x214
[ 340.399183] sys_io_submit+0x110/0x214
[ 340.406674] el0_svc_naked+0x30/0x34
[ 340.406674] el0_svc_naked+0x30/0x34
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 22h:32m:16s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 22h:41m:25s]
[ 341.811524] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 341.811524] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 341.822505] 0-....: (1 GPs behind) idle=ac6/1/4611686018427387904 softirq=4021/4728 fqs=18368
[ 341.822505] 0-....: (1 GPs behind) idle=ac6/1/4611686018427387904 softirq=4021/4728 fqs=18368
[ 341.839924] (detected by 15, t=36769 jiffies, g=-107, c=-108, q=1)
[ 341.839924] (detected by 15, t=36769 jiffies, g=-107, c=-108, q=1)
[ 341.852471] Task dump for CPU 0:
[ 341.852471] Task dump for CPU 0:
[ 341.858917] kworker/0:2H R running task 0 3757 2 0x0000002a
[ 341.858917] kworker/0:2H R running task 0 3757 2 0x0000002a
[ 341.873032] Workqueue: kblockd blk_mq_run_work_fn
[ 341.873032] Workqueue: kblockd blk_mq_run_work_fn
[ 341.882439] Call trace:
[ 341.882439] Call trace:
[ 341.887319] __switch_to+0x90/0xcc
[ 341.887319] __switch_to+0x90/0xcc
[ 341.894114] 0x674238a07d45b300
[ 341.894114] 0x674238a07d45b300
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 22h:50m:33s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 22h:59m:41s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 23h:08m:49s]
[ 344.587526] ------------[ cut here ]------------
[ 344.587526] ------------[ cut here ]------------
[ 344.596761] NETDEV WATCHDOG: eth2 (hns3): transmit queue 7 timed out
[ 344.596761] NETDEV WATCHDOG: eth2 (hns3): transmit queue 7 timed out
[ 344.609496] WARNING: CPU: 5 PID: 0 at net/sched/sch_generic.c:461 dev_watchdog+0x2a8/0x2b0
[ 344.609496] WARNING: CPU: 5 PID: 0 at net/sched/sch_generic.c:461 dev_watchdog+0x2a8/0x2b0
[ 344.626042] Modules linked in: hisi_sas_v3_hw
[ 344.626042] Modules linked in: hisi_sas_v3_hw
[ 344.634756] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G D W L 4.18.0-rc1-12443-g7e2d9d9-dirty #1077
[ 344.634756] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G D W L 4.18.0-rc1-12443-g7e2d9d9-dirty #1077
[ 344.654263] Hardware name: Huawei D06/D06, BIOS Hisilicon D06 UEFI RC0 - B601 (V6.01) 11/08/2018
[ 344.654263] Hardware name: Huawei D06/D06, BIOS Hisilicon D06 UEFI RC0 - B601 (V6.01) 11/08/2018
[ 344.671854] pstate: 60400009 (nZCv daif +PAN -UAO)
[ 344.671854] pstate: 60400009 (nZCv daif +PAN -UAO)
[ 344.681436] pc : dev_watchdog+0x2a8/0x2b0
[ 344.681436] pc : dev_watchdog+0x2a8/0x2b0
[ 344.689449] lr : dev_watchdog+0x2a8/0x2b0
[ 344.689449] lr : dev_watchdog+0x2a8/0x2b0
[ 344.697462] sp : ffff00000802bd60
[ 344.697462] sp : ffff00000802bd60
[ 344.704081] x29: ffff00000802bd60 x28: 0000000000000001
[ 344.704081] x29: ffff00000802bd60 x28: 0000000000000001
[ 344.714708] x27: ffff0000094460c0 x26: 00000000ffffffff
[ 344.714708] x27: ffff0000094460c0 x26: 00000000ffffffff
[ 344.725335] x25: 0000000000000005 x24: ffff8027c06923b0
[ 344.725335] x25: 0000000000000005 x24: ffff8027c06923b0
[ 344.735961] x23: ffff8027c0690940 x22: ffff8027c069239c
[ 344.735961] x23: ffff8027c0690940 x22: ffff8027c069239c
[ 344.746588] x21: ffff8027c0692000 x20: 0000000000000008
[ 344.746588] x21: ffff8027c0692000 x20: 0000000000000008
[ 344.757214] x19: ffff000009447000 x18: 0000ffffe70a3f40
[ 344.757214] x19: ffff000009447000 x18: 0000ffffe70a3f40
[ 344.767841] x17: 00000000004df678 x16: ffff00000822d61c
[ 344.767841] x17: 00000000004df678 x16: ffff00000822d61c
[ 344.778467] x15: ffffffffffffffff x14: 0000000000000000
[ 344.778467] x15: ffffffffffffffff x14: 0000000000000000
[ 344.789093] x13: ffffffffffffff95 x12: 674238a07d45b300
[ 344.789093] x13: ffffffffffffff95 x12: 674238a07d45b300
[ 344.799720] x11: ffffffffffffffff x10: 0000000000000006
[ 344.799720] x11: ffffffffffffffff x10: 0000000000000006
[ 344.810347] x9 : 0000000000000006 x8 : 0000000000000851
[ 344.810347] x9 : 0000000000000006 x8 : 0000000000000851
[ 344.820973] x7 : 712074696d736e61 x6 : ffff0000085c8d3c
[ 344.820973] x7 : 712074696d736e61 x6 : ffff0000085c8d3c
[ 344.831599] x5 : 0000000000000000 x4 : 0000000000000000
[ 344.831599] x5 : 0000000000000000 x4 : 0000000000000000
[ 344.842225] x3 : ffffffffffffffff x2 : ffff00000946a728
[ 344.842225] x3 : ffffffffffffffff x2 : ffff00000946a728
[ 344.852851] x1 : 674238a07d45b300 x0 : 674238a07d45b300
[ 344.852851] x1 : 674238a07d45b300 x0 : 674238a07d45b300
[ 344.863478] Call trace:
[ 344.863478] Call trace:
[ 344.868357] dev_watchdog+0x2a8/0x2b0
[ 344.868357] dev_watchdog+0x2a8/0x2b0
[ 344.875675] call_timer_fn+0x1c/0x70
[ 344.875675] call_timer_fn+0x1c/0x70
[ 344.882818] expire_timers+0xb0/0xbc
[ 344.882818] expire_timers+0xb0/0xbc
[ 344.889962] run_timer_softirq+0x98/0x17c
[ 344.889962] run_timer_softirq+0x98/0x17c
[ 344.897975] __do_softirq+0x100/0x1f8
[ 344.897975] __do_softirq+0x100/0x1f8
[ 344.905292] irq_exit+0xb8/0xcc
[ 344.905292] irq_exit+0xb8/0xcc
[ 344.911566] __handle_domain_irq+0x64/0xac
[ 344.911566] __handle_domain_irq+0x64/0xac
[ 344.919753] gic_handle_irq+0xd4/0x184
[ 344.919753] gic_handle_irq+0xd4/0x184
[ 344.927244] el1_irq+0xb0/0x128
[ 344.927244] el1_irq+0xb0/0x128
[ 344.933516] arch_cpu_idle+0x10/0x18
[ 344.933516] arch_cpu_idle+0x10/0x18
[ 344.940660] default_idle_call+0x18/0x2c
[ 344.940660] default_idle_call+0x18/0x2c
[ 344.948500] do_idle+0x1a8/0x248
[ 344.948500] do_idle+0x1a8/0x248
[ 344.954946] cpu_startup_entry+0x1c/0x24
[ 344.954946] cpu_startup_entry+0x1c/0x24
[ 344.962786] secondary_start_kernel+0x134/0x158
[ 344.962786] secondary_start_kernel+0x134/0x158
[ 344.971844] ---[ end trace ae3b8c1166c9988d ]---
[ 344.971844] ---[ end trace ae3b8c1166c9988d ]---
[ 344.981082] hns3 0000:7d:00.2 eth2: tx_timeout count: 0, queue id: 7, SW_NTU: 0x7, SW_NTC: 0x5, HW_HEAD: 0x7, HW_TAIL: 0x7, INT: 0x0
[ 344.981082] hns3 0000:7d:00.2 eth2: tx_timeout count: 0, queue id: 7, SW_NTU: 0x7, SW_NTC: 0x5, HW_HEAD: 0x7, HW_TAIL: 0x7, INT: 0x0
[ 345.004946] hns3 0000:7d:00.2: received reset event , reset type is 2
[ 345.004946] hns3 0000:7d:00.2: received reset event , reset type is 2
[ 345.017846] hns3 0000:7d:00.2: PF Reset requested
[ 345.017846] hns3 0000:7d:00.2: PF Reset requested
[ 345.515587] hns3 0000:7d:00.2 eth2: link down
[ 345.515587] hns3 0000:7d:00.2 eth2: link down
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 23h:17m:58s]
[ 345.611530] Marvell 88E1510 mii-0000:7d:00.2:00: no link in PHY_RUNNING
[ 345.611530] Marvell 88E1510 mii-0000:7d:00.2:00: no link in PHY_RUNNING
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 23h:27m:06s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 23h:36m:14s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 23h:45m:22s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 23h:54m:30s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:00h:03m:39s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:00h:12m:47s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:00h:21m:55s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:00h:31m:03s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:00h:40m:12s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:00h:49m:20s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:00h:58m:28s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:01h:07m:36s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:01h:16m:44s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:01h:25m:53s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:01h:35m:01s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:01h:44m:09s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:01h:53m:17s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:02h:02m:26s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:02h:11m:34s]
[ 364.763523] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2H:3757]
[ 364.763523] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2H:3757]
[ 364.778502] Modules linked in: hisi_sas_v3_hw
[ 364.778502] Modules linked in: hisi_sas_v3_hw
[ 364.787214] CPU: 0 PID: 3757 Comm: kworker/0:2H Tainted: G D W L 4.18.0-rc1-12443-g7e2d9d9-dirty #1077
[ 364.787214] CPU: 0 PID: 3757 Comm: kworker/0:2H Tainted: G D W L 4.18.0-rc1-12443-g7e2d9d9-dirty #1077
[ 364.807765] Hardware name: Huawei D06/D06, BIOS Hisilicon D06 UEFI RC0 - B601 (V6.01) 11/08/2018
[ 364.807765] Hardware name: Huawei D06/D06, BIOS Hisilicon D06 UEFI RC0 - B601 (V6.01) 11/08/2018
[ 364.825358] Workqueue: kblockd blk_mq_run_work_fn
[ 364.825358] Workqueue: kblockd blk_mq_run_work_fn
[ 364.834765] pstate: 80c00009 (Nzcv daif +PAN +UAO)
[ 364.834765] pstate: 80c00009 (Nzcv daif +PAN +UAO)
[ 364.844345] pc : _raw_spin_lock+0x34/0x44
[ 364.844345] pc : _raw_spin_lock+0x34/0x44
[ 364.852359] lr : dd_dispatch_request+0x2c/0x1e0
[ 364.852359] lr : dd_dispatch_request+0x2c/0x1e0
[ 364.861416] sp : ffff00002462bc60
[ 364.861416] sp : ffff00002462bc60
[ 364.868036] x29: ffff00002462bc60 x28: ffff000009973ce8
[ 364.868036] x29: ffff00002462bc60 x28: ffff000009973ce8
[ 364.878662] x27: ffff8027dceac938 x26: ffff000009467558
[ 364.878662] x27: ffff8027dceac938 x26: ffff000009467558
[ 364.889289] x25: 0000000000000000 x24: ffff000009451000
[ 364.889289] x25: 0000000000000000 x24: ffff000009451000
[ 364.899915] x23: ffff000009451000 x22: ffff8027a5a37200
[ 364.899915] x23: ffff000009451000 x22: ffff8027a5a37200
[ 364.910542] x21: ffff8027b24f4c5c x20: ffff00002462bce8
[ 364.910542] x21: ffff8027b24f4c5c x20: ffff00002462bce8
[ 364.921168] x19: ffff8027b24f4c00 x18: 000000000000000e
[ 364.921168] x19: ffff8027b24f4c00 x18: 000000000000000e
[ 364.931795] x17: 0000000000000001 x16: 0000000000000019
[ 364.931795] x17: 0000000000000001 x16: 0000000000000019
[ 364.942421] x15: ffffffffffffffff x14: 0000000000000001
[ 364.942421] x15: ffffffffffffffff x14: 0000000000000001
[ 364.953048] x13: 00000000c0a80101 x12: 0000000000000001
[ 364.953048] x13: 00000000c0a80101 x12: 0000000000000001
[ 364.963674] x11: 0000000000001000 x10: 0000000000000950
[ 364.963674] x11: 0000000000001000 x10: 0000000000000950
[ 364.974301] x9 : ffff00002462bd50 x8 : 0000000000000018
[ 364.974301] x9 : ffff00002462bd50 x8 : 0000000000000018
[ 364.984927] x7 : fefefefefefefeff x6 : 0000000000000000
[ 364.984927] x7 : fefefefefefefeff x6 : 0000000000000000
[ 364.995554] x5 : 000000000000001d x4 : 000000000000001e
[ 364.995554] x5 : 000000000000001d x4 : 000000000000001e
[ 365.006180] x3 : 0000000000003996 x2 : 0000000000000001
[ 365.006180] x3 : 0000000000003996 x2 : 0000000000000001
[ 365.016807] x1 : 0000000039973996 x0 : ffff8027b24f4c5c
[ 365.016807] x1 : 0000000039973996 x0 : ffff8027b24f4c5c
[ 365.027433] Call trace:
[ 365.027433] Call trace:
[ 365.032312] _raw_spin_lock+0x34/0x44
[ 365.032312] _raw_spin_lock+0x34/0x44
[ 365.039629] blk_mq_do_dispatch_sched+0x4c/0x104
[ 365.039629] blk_mq_do_dispatch_sched+0x4c/0x104
[ 365.048862] blk_mq_sched_dispatch_requests+0x130/0x1bc
[ 365.048862] blk_mq_sched_dispatch_requests+0x130/0x1bc
[ 365.059313] __blk_mq_run_hw_queue+0xa4/0x11c
[ 365.059313] __blk_mq_run_hw_queue+0xa4/0x11c
[ 365.068023] blk_mq_run_work_fn+0x18/0x20
[ 365.068023] blk_mq_run_work_fn+0x18/0x20
[ 365.076037] process_one_work+0x140/0x2b0
[ 365.076037] process_one_work+0x140/0x2b0
[ 365.084050] worker_thread+0x130/0x3e4
[ 365.084050] worker_thread+0x130/0x3e4
[ 365.091541] kthread+0xfc/0x128
[ 365.091541] kthread+0xfc/0x128
[ 365.097813] ret_from_fork+0x10/0x18
[ 365.097813] ret_from_fork+0x10/0x18
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:02h:20m:42s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:02h:29m:50s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:02h:38m:58s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:02h:48m:07s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:02h:57m:15s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:03h:06m:23s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:03h:15m:31s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:03h:24m:40s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:03h:33m:48s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:03h:42m:56s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:03h:52m:04s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:04h:01m:12s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:04h:10m:21s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:04h:19m:29s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:04h:28m:37s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:04h:37m:45s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:04h:46m:54s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:04h:56m:02s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:05h:05m:10s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:05h:14m:18s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:05h:23m:26s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:05h:32m:35s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:05h:41m:43s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:05h:50m:51s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:05h:59m:59s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:06h:09m:08s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:06h:18m:16s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:06h:27m:24s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:06h:36m:32s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:06h:45m:40s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:06h:54m:49s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:07h:03m:57s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:07h:13m:05s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:07h:22m:13s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:07h:31m:22s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:07h:40m:30s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:07h:49m:38s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:07h:58m:46s]
[ 402.803523] INFO: rcu_preempt self-detected stall on CPU
[ 402.803523] INFO: rcu_preempt self-detected stall on CPU
[ 402.811528] INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 402.811528] INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 402.814151] 0-....: (1 GPs behind) idle=ac6/1/4611686018427387906 softirq=4726/4728 fqs=26033
[ 402.814151] 0-....: (1 GPs behind) idle=ac6/1/4611686018427387906 softirq=4726/4728 fqs=26033
[ 402.825474]
[ 402.825474]
[ 402.842891] Tasks blocked on level-1 rcu_node (CPUs 0-15):
[ 402.842891] Tasks blocked on level-1 rcu_node (CPUs 0-15):
[ 402.842894] (t=52524 jiffies g=1596 c=1595 q=872)
[ 402.842894] (t=52524 jiffies g=1596 c=1595 q=872)
[ 402.846029] P3755
[ 402.846029] P3755
[ 402.870766] 0-....: (1 GPs behind) idle=ac6/1/4611686018427387906 softirq=4726/4728 fqs=26037
[ 402.870766] 0-....: (1 GPs behind) idle=ac6/1/4611686018427387906 softirq=4726/4728 fqs=26037
[ 402.888184] Task dump for CPU 0:
[ 402.888184] Task dump for CPU 0:
[ 402.888184] (detected by 7, t=52536 jiffies, g=1596, c=1595, q=872)
[ 402.888184] (detected by 7, t=52536 jiffies, g=1596, c=1595, q=872)
[ 402.894630] kworker/0:2H R running task 0 3757 2 0x0000002a
[ 402.894630] kworker/0:2H R running task 0 3757 2 0x0000002a
[ 402.921456] Workqueue: kblockd blk_mq_run_work_fn
[ 402.921456] Workqueue: kblockd blk_mq_run_work_fn
[ 402.930862] Call trace:
[ 402.930862] Call trace:
[ 402.935742] dump_backtrace+0x0/0x180
[ 402.935742] dump_backtrace+0x0/0x180
[ 402.943059] show_stack+0x14/0x1c
[ 402.943059] show_stack+0x14/0x1c
[ 402.949680] sched_show_task+0x16c/0x184
[ 402.949680] sched_show_task+0x16c/0x184
[ 402.957519] dump_cpu_task+0x40/0x4c
[ 402.957519] dump_cpu_task+0x40/0x4c
[ 402.964662] rcu_dump_cpu_stacks+0x98/0xd8
[ 402.964662] rcu_dump_cpu_stacks+0x98/0xd8
[ 402.972849] rcu_check_callbacks+0x750/0x9c0
[ 402.972849] rcu_check_callbacks+0x750/0x9c0
[ 402.981385] update_process_times+0x2c/0x54
[ 402.981385] update_process_times+0x2c/0x54
[ 402.989747] tick_sched_handle.isra.14+0x24/0x68
[ 402.989747] tick_sched_handle.isra.14+0x24/0x68
[ 402.998980] tick_sched_timer+0x48/0x98
[ 402.998980] tick_sched_timer+0x48/0x98
[ 403.006645] __hrtimer_run_queues+0x100/0x158
[ 403.006645] __hrtimer_run_queues+0x100/0x158
[ 403.015355] hrtimer_interrupt+0xe4/0x248
[ 403.015355] hrtimer_interrupt+0xe4/0x248
[ 403.023369] arch_timer_handler_phys+0x2c/0x38
[ 403.023369] arch_timer_handler_phys+0x2c/0x38
[ 403.032253] handle_percpu_devid_irq+0x78/0x12c
[ 403.032253] handle_percpu_devid_irq+0x78/0x12c
[ 403.041311] generic_handle_irq+0x24/0x38
[ 403.041311] generic_handle_irq+0x24/0x38
[ 403.049325] __handle_domain_irq+0x60/0xac
[ 403.049325] __handle_domain_irq+0x60/0xac
[ 403.057512] gic_handle_irq+0xd4/0x184
[ 403.057512] gic_handle_irq+0xd4/0x184
[ 403.065003] el1_irq+0xb0/0x128
[ 403.065003] el1_irq+0xb0/0x128
[ 403.071275] _raw_spin_lock+0x34/0x44
[ 403.071275] _raw_spin_lock+0x34/0x44
[ 403.078592] blk_mq_do_dispatch_sched+0x4c/0x104
[ 403.078592] blk_mq_do_dispatch_sched+0x4c/0x104
[ 403.087825] blk_mq_sched_dispatch_requests+0x130/0x1bc
[ 403.087825] blk_mq_sched_dispatch_requests+0x130/0x1bc
[ 403.098276] __blk_mq_run_hw_queue+0xa4/0x11c
[ 403.098276] __blk_mq_run_hw_queue+0xa4/0x11c
[ 403.106986] blk_mq_run_work_fn+0x18/0x20
[ 403.106986] blk_mq_run_work_fn+0x18/0x20
[ 403.114999] process_one_work+0x140/0x2b0
[ 403.114999] process_one_work+0x140/0x2b0
[ 403.123012] worker_thread+0x130/0x3e4
[ 403.123012] worker_thread+0x130/0x3e4
[ 403.130503] kthread+0xfc/0x128
[ 403.130503] kthread+0xfc/0x128
[ 403.136775] ret_from_fork+0x10/0x18
[ 403.136775] ret_from_fork+0x10/0x18
[ 403.143918] Task dump for CPU 0:
[ 403.143918] Task dump for CPU 0:
[ 403.150364] kworker/0:2H R running task 0 3757 2 0x0000002a
[ 403.150364] kworker/0:2H R running task 0 3757 2 0x0000002a
[ 403.164480] Workqueue: kblockd blk_mq_run_work_fn
[ 403.164480] Workqueue: kblockd blk_mq_run_work_fn
[ 403.173887] Call trace:
[ 403.173887] Call trace:
[ 403.178767] __switch_to+0x90/0xcc
[ 403.178767] __switch_to+0x90/0xcc
[ 403.185562] 0x674238a07d45b300
[ 403.185562] 0x674238a07d45b300
[ 403.191835] fio D 0 3755 3753 0x0000002c
[ 403.191835] fio D 0 3755 3753 0x0000002c
[ 403.202810] Call trace:
[ 403.202810] Call trace:
[ 403.207689] __switch_to+0x90/0xcc
[ 403.207689] __switch_to+0x90/0xcc
[ 403.214485] __schedule+0x1b8/0x5cc
[ 403.214485] __schedule+0x1b8/0x5cc
[ 403.221453] schedule+0x38/0x9c
[ 403.221453] schedule+0x38/0x9c
[ 403.227726] schedule_timeout+0x1b8/0x320
[ 403.227726] schedule_timeout+0x1b8/0x320
[ 403.235739] wait_for_common+0xb4/0x164
[ 403.235739] wait_for_common+0xb4/0x164
[ 403.243404] wait_for_completion+0x14/0x1c
[ 403.243404] wait_for_completion+0x14/0x1c
[ 403.251592] exit_aio+0x100/0x114
[ 403.251592] exit_aio+0x100/0x114
[ 403.258214] mmput+0x48/0x118
[ 403.258214] mmput+0x48/0x118
[ 403.264137] do_exit+0x278/0x978
[ 403.264137] do_exit+0x278/0x978
[ 403.270584] die+0x1c8/0x1f4
[ 403.270584] die+0x1c8/0x1f4
[ 403.276334] die_kernel_fault+0x64/0x74
[ 403.276334] die_kernel_fault+0x64/0x74
[ 403.284000] __do_kernel_fault+0x84/0xac
[ 403.284000] __do_kernel_fault+0x84/0xac
[ 403.291839] do_page_fault+0x1dc/0x3e8
[ 403.291839] do_page_fault+0x1dc/0x3e8
[ 403.299331] do_translation_fault+0x68/0x74
[ 403.299331] do_translation_fault+0x68/0x74
[ 403.307692] do_mem_abort+0x4c/0xf4
[ 403.307692] do_mem_abort+0x4c/0xf4
[ 403.314661] el1_da+0x20/0x80
[ 403.314661] el1_da+0x20/0x80
[ 403.320585] deadline_remove_request+0x5c/0xd0
[ 403.320585] deadline_remove_request+0x5c/0xd0
[ 403.329469] dd_dispatch_request+0x184/0x1e0
[ 403.329469] dd_dispatch_request+0x184/0x1e0
[ 403.338005] blk_mq_do_dispatch_sched+0x4c/0x104
[ 403.338005] blk_mq_do_dispatch_sched+0x4c/0x104
[ 403.347238] blk_mq_sched_dispatch_requests+0x130/0x1bc
[ 403.347238] blk_mq_sched_dispatch_requests+0x130/0x1bc
[ 403.357690] __blk_mq_run_hw_queue+0xa4/0x11c
[ 403.357690] __blk_mq_run_hw_queue+0xa4/0x11c
[ 403.366400] __blk_mq_delay_run_hw_queue+0x1a4/0x208
[ 403.366400] __blk_mq_delay_run_hw_queue+0x1a4/0x208
[ 403.376328] blk_mq_run_hw_queue+0x9c/0xf8
[ 403.376328] blk_mq_run_hw_queue+0x9c/0xf8
[ 403.384516] blk_mq_sched_insert_requests+0x58/0x84
[ 403.384516] blk_mq_sched_insert_requests+0x58/0x84
[ 403.394271] blk_mq_flush_plug_list+0x144/0x16c
[ 403.394271] blk_mq_flush_plug_list+0x144/0x16c
[ 403.403330] blk_flush_plug_list+0xe8/0x244
[ 403.403330] blk_flush_plug_list+0xe8/0x244
[ 403.411692] blk_finish_plug+0x30/0x40
[ 403.411692] blk_finish_plug+0x30/0x40
[ 403.419183] sys_io_submit+0x110/0x214
[ 403.419183] sys_io_submit+0x110/0x214
[ 403.426674] el0_svc_naked+0x30/0x34
[ 403.426674] el0_svc_naked+0x30/0x34
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:08h:17m:03s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:08h:26m:11s]
[ 404.831524] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 404.831524] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 404.842505] 0-....: (1 GPs behind) idle=ac6/1/4611686018427387904 softirq=4021/4728 fqs=26238
[ 404.842505] 0-....: (1 GPs behind) idle=ac6/1/4611686018427387904 softirq=4021/4728 fqs=26238
[ 404.859924] (detected by 18, t=52524 jiffies, g=-107, c=-108, q=1)
[ 404.859924] (detected by 18, t=52524 jiffies, g=-107, c=-108, q=1)
[ 404.872470] Task dump for CPU 0:
[ 404.872470] Task dump for CPU 0:
[ 404.878916] kworker/0:2H R running task 0 3757 2 0x0000002a
[ 404.878916] kworker/0:2H R running task 0 3757 2 0x0000002a
[ 404.893031] Workqueue: kblockd blk_mq_run_work_fn
[ 404.893031] Workqueue: kblockd blk_mq_run_work_fn
[ 404.902438] Call trace:
[ 404.902438] Call trace:
[ 404.907319] __switch_to+0x90/0xcc
[ 404.907319] __switch_to+0x90/0xcc
[ 404.914114] 0x674238a07d45b300
[ 404.914114] 0x674238a07d45b300
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:08h:35m:19s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:08h:44m:27s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:08h:53m:36s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:09h:02m:44s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:09h:11m:52s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:09h:21m:00s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:09h:30m:08s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:09h:39m:17s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:09h:48m:25s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:09h:57m:33s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:10h:06m:41s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:10h:15m:50s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:10h:24m:58s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:10h:34m:06s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:10h:43m:14s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:10h:52m:23s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:11h:01m:31s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:11h:10m:39s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:11h:19m:47s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:11h:28m:55s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:11h:38m:04s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:11h:47m:12s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:11h:56m:20s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:12h:05m:28s]
[ 428.763523] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/0:2H:3757]
[ 428.763523] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/0:2H:3757]
[ 428.778502] Modules linked in: hisi_sas_v3_hw
[ 428.778502] Modules linked in: hisi_sas_v3_hw
[ 428.787214] CPU: 0 PID: 3757 Comm: kworker/0:2H Tainted: G D W L 4.18.0-rc1-12443-g7e2d9d9-dirty #1077
[ 428.787214] CPU: 0 PID: 3757 Comm: kworker/0:2H Tainted: G D W L 4.18.0-rc1-12443-g7e2d9d9-dirty #1077
[ 428.807766] Hardware name: Huawei D06/D06, BIOS Hisilicon D06 UEFI RC0 - B601 (V6.01) 11/08/2018
[ 428.807766] Hardware name: Huawei D06/D06, BIOS Hisilicon D06 UEFI RC0 - B601 (V6.01) 11/08/2018
[ 428.825359] Workqueue: kblockd blk_mq_run_work_fn
[ 428.825359] Workqueue: kblockd blk_mq_run_work_fn
[ 428.834766] pstate: 80c00009 (Nzcv daif +PAN +UAO)
[ 428.834766] pstate: 80c00009 (Nzcv daif +PAN +UAO)
[ 428.844346] pc : _raw_spin_lock+0x34/0x44
[ 428.844346] pc : _raw_spin_lock+0x34/0x44
[ 428.852360] lr : dd_dispatch_request+0x2c/0x1e0
[ 428.852360] lr : dd_dispatch_request+0x2c/0x1e0
[ 428.861418] sp : ffff00002462bc60
[ 428.861418] sp : ffff00002462bc60
[ 428.868037] x29: ffff00002462bc60 x28: ffff000009973ce8
[ 428.868037] x29: ffff00002462bc60 x28: ffff000009973ce8
[ 428.878664] x27: ffff8027dceac938 x26: ffff000009467558
[ 428.878664] x27: ffff8027dceac938 x26: ffff000009467558
[ 428.889290] x25: 0000000000000000 x24: ffff000009451000
[ 428.889290] x25: 0000000000000000 x24: ffff000009451000
[ 428.899917] x23: ffff000009451000 x22: ffff8027a5a37200
[ 428.899917] x23: ffff000009451000 x22: ffff8027a5a37200
[ 428.910543] x21: ffff8027b24f4c5c x20: ffff00002462bce8
[ 428.910543] x21: ffff8027b24f4c5c x20: ffff00002462bce8
[ 428.921170] x19: ffff8027b24f4c00 x18: 000000000000000e
[ 428.921170] x19: ffff8027b24f4c00 x18: 000000000000000e
[ 428.931796] x17: 0000000000000001 x16: 0000000000000019
[ 428.931796] x17: 0000000000000001 x16: 0000000000000019
[ 428.942423] x15: ffffffffffffffff x14: 0000000000000001
[ 428.942423] x15: ffffffffffffffff x14: 0000000000000001
[ 428.953049] x13: 00000000c0a80101 x12: 0000000000000001
[ 428.953049] x13: 00000000c0a80101 x12: 0000000000000001
[ 428.963676] x11: 0000000000001000 x10: 0000000000000950
[ 428.963676] x11: 0000000000001000 x10: 0000000000000950
[ 428.974302] x9 : ffff00002462bd50 x8 : 0000000000000018
[ 428.974302] x9 : ffff00002462bd50 x8 : 0000000000000018
[ 428.984929] x7 : fefefefefefefeff x6 : 0000000000000000
[ 428.984929] x7 : fefefefefefefeff x6 : 0000000000000000
[ 428.995555] x5 : 000000000000001d x4 : 000000000000001e
[ 428.995555] x5 : 000000000000001d x4 : 000000000000001e
[ 429.006182] x3 : 0000000000003996 x2 : 0000000000000001
[ 429.006182] x3 : 0000000000003996 x2 : 0000000000000001
[ 429.016808] x1 : 0000000039973996 x0 : ffff8027b24f4c5c
[ 429.016808] x1 : 0000000039973996 x0 : ffff8027b24f4c5c
[ 429.027435] Call trace:
[ 429.027435] Call trace:
[ 429.032313] _raw_spin_lock+0x34/0x44
[ 429.032313] _raw_spin_lock+0x34/0x44
[ 429.039630] blk_mq_do_dispatch_sched+0x4c/0x104
[ 429.039630] blk_mq_do_dispatch_sched+0x4c/0x104
[ 429.048863] blk_mq_sched_dispatch_requests+0x130/0x1bc
[ 429.048863] blk_mq_sched_dispatch_requests+0x130/0x1bc
[ 429.059315] __blk_mq_run_hw_queue+0xa4/0x11c
[ 429.059315] __blk_mq_run_hw_queue+0xa4/0x11c
[ 429.068024] blk_mq_run_work_fn+0x18/0x20
[ 429.068024] blk_mq_run_work_fn+0x18/0x20
[ 429.076038] process_one_work+0x140/0x2b0
[ 429.076038] process_one_work+0x140/0x2b0
[ 429.084051] worker_thread+0x130/0x3e4
[ 429.084051] worker_thread+0x130/0x3e4
[ 429.091542] kthread+0xfc/0x128
[ 429.091542] kthread+0xfc/0x128
[ 429.097814] ret_from_fork+0x10/0x18
[ 429.097814] ret_from_fork+0x10/0x18
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:12h:14m:37s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:12h:23m:45s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:12h:32m:53s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:12h:42m:01s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:12h:51m:09s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:13h:00m:18s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:13h:09m:26s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:13h:18m:34s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:13h:27m:42s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:13h:36m:51s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:13h:45m:59s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:13h:55m:07s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:14h:04m:15s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:14h:13m:23s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:14h:22m:32s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:14h:31m:40s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:14h:40m:48s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:14h:49m:56s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:14h:59m:05s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:15h:08m:13s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:15h:17m:21s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:15h:26m:29s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:15h:35m:37s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:15h:44m:46s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:15h:53m:54s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:16h:03m:02s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:16h:12m:10s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:16h:21m:19s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:16h:30m:27s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:16h:39m:35s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:16h:48m:43s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:16h:57m:51s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:17h:07m:00s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:17h:16m:08s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:17h:25m:16s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:17h:34m:24s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:17h:43m:33s]
[ 465.823523] INFO: rcu_preempt self-detected stall on CPU
[ 465.823523] INFO: rcu_preempt self-detected stall on CPU
[ 465.831525] INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 465.831525] INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 465.834150] 0-....: (1 GPs behind) idle=ac6/1/4611686018427387906 softirq=4726/4728 fqs=33760
[ 465.834150] 0-....: (1 GPs behind) idle=ac6/1/4611686018427387906 softirq=4726/4728 fqs=33760
[ 465.845475]
[ 465.845475]
[ 465.862892] Tasks blocked on level-1 rcu_node (CPUs 0-15):
[ 465.862892] Tasks blocked on level-1 rcu_node (CPUs 0-15):
[ 465.862895] (t=68279 jiffies g=1596 c=1595 q=1142)
[ 465.862895] (t=68279 jiffies g=1596 c=1595 q=1142)
[ 465.866030] P3755
[ 465.866030] P3755
[ 465.890941] 0-....: (1 GPs behind) idle=ac6/1/4611686018427387906 softirq=4726/4728 fqs=33764
[ 465.890941] 0-....: (1 GPs behind) idle=ac6/1/4611686018427387906 softirq=4726/4728 fqs=33764
[ 465.908358] Task dump for CPU 0:
[ 465.908358] Task dump for CPU 0:
[ 465.908359] (detected by 9, t=68291 jiffies, g=1596, c=1595, q=1142)
[ 465.908359] (detected by 9, t=68291 jiffies, g=1596, c=1595, q=1142)
[ 465.914806] kworker/0:2H R running task 0 3757 2 0x0000002a
[ 465.914806] kworker/0:2H R running task 0 3757 2 0x0000002a
[ 465.941805] Workqueue: kblockd blk_mq_run_work_fn
[ 465.941805] Workqueue: kblockd blk_mq_run_work_fn
[ 465.951212] Call trace:
[ 465.951212] Call trace:
[ 465.956091] dump_backtrace+0x0/0x180
[ 465.956091] dump_backtrace+0x0/0x180
[ 465.963408] show_stack+0x14/0x1c
[ 465.963408] show_stack+0x14/0x1c
[ 465.970029] sched_show_task+0x16c/0x184
[ 465.970029] sched_show_task+0x16c/0x184
[ 465.977868] dump_cpu_task+0x40/0x4c
[ 465.977868] dump_cpu_task+0x40/0x4c
[ 465.985011] rcu_dump_cpu_stacks+0x98/0xd8
[ 465.985011] rcu_dump_cpu_stacks+0x98/0xd8
[ 465.993198] rcu_check_callbacks+0x750/0x9c0
[ 465.993198] rcu_check_callbacks+0x750/0x9c0
[ 466.001734] update_process_times+0x2c/0x54
[ 466.001734] update_process_times+0x2c/0x54
[ 466.010096] tick_sched_handle.isra.14+0x24/0x68
[ 466.010096] tick_sched_handle.isra.14+0x24/0x68
[ 466.019329] tick_sched_timer+0x48/0x98
[ 466.019329] tick_sched_timer+0x48/0x98
[ 466.026994] __hrtimer_run_queues+0x100/0x158
[ 466.026994] __hrtimer_run_queues+0x100/0x158
[ 466.035704] hrtimer_interrupt+0xe4/0x248
[ 466.035704] hrtimer_interrupt+0xe4/0x248
[ 466.043718] arch_timer_handler_phys+0x2c/0x38
[ 466.043718] arch_timer_handler_phys+0x2c/0x38
[ 466.052602] handle_percpu_devid_irq+0x78/0x12c
[ 466.052602] handle_percpu_devid_irq+0x78/0x12c
[ 466.061660] generic_handle_irq+0x24/0x38
[ 466.061660] generic_handle_irq+0x24/0x38
[ 466.069674] __handle_domain_irq+0x60/0xac
[ 466.069674] __handle_domain_irq+0x60/0xac
[ 466.077861] gic_handle_irq+0xd4/0x184
[ 466.077861] gic_handle_irq+0xd4/0x184
[ 466.085352] el1_irq+0xb0/0x128
[ 466.085352] el1_irq+0xb0/0x128
[ 466.091624] _raw_spin_lock+0x34/0x44
[ 466.091624] _raw_spin_lock+0x34/0x44
[ 466.098941] blk_mq_do_dispatch_sched+0x4c/0x104
[ 466.098941] blk_mq_do_dispatch_sched+0x4c/0x104
[ 466.108174] blk_mq_sched_dispatch_requests+0x130/0x1bc
[ 466.108174] blk_mq_sched_dispatch_requests+0x130/0x1bc
[ 466.118625] __blk_mq_run_hw_queue+0xa4/0x11c
[ 466.118625] __blk_mq_run_hw_queue+0xa4/0x11c
[ 466.127335] blk_mq_run_work_fn+0x18/0x20
[ 466.127335] blk_mq_run_work_fn+0x18/0x20
[ 466.135348] process_one_work+0x140/0x2b0
[ 466.135348] process_one_work+0x140/0x2b0
[ 466.143362] worker_thread+0x130/0x3e4
[ 466.143362] worker_thread+0x130/0x3e4
[ 466.150853] kthread+0xfc/0x128
[ 466.150853] kthread+0xfc/0x128
[ 466.157125] ret_from_fork+0x10/0x18
[ 466.157125] ret_from_fork+0x10/0x18
[ 466.164268] Task dump for CPU 0:
[ 466.164268] Task dump for CPU 0:
[ 466.170715] kworker/0:2H R running task 0 3757 2 0x0000002a
[ 466.170715] kworker/0:2H R running task 0 3757 2 0x0000002a
[ 466.184830] Workqueue: kblockd blk_mq_run_work_fn
[ 466.184830] Workqueue: kblockd blk_mq_run_work_fn
[ 466.194237] Call trace:
[ 466.194237] Call trace:
[ 466.199117] __switch_to+0x90/0xcc
[ 466.199117] __switch_to+0x90/0xcc
[ 466.205912] 0x674238a07d45b300
[ 466.205912] 0x674238a07d45b300
[ 466.212185] fio D 0 3755 3753 0x0000002c
[ 466.212185] fio D 0 3755 3753 0x0000002c
[ 466.223160] Call trace:
[ 466.223160] Call trace:
[ 466.228039] __switch_to+0x90/0xcc
[ 466.228039] __switch_to+0x90/0xcc
[ 466.234835] __schedule+0x1b8/0x5cc
[ 466.234835] __schedule+0x1b8/0x5cc
[ 466.241803] schedule+0x38/0x9c
[ 466.241803] schedule+0x38/0x9c
[ 466.248076] schedule_timeout+0x1b8/0x320
[ 466.248076] schedule_timeout+0x1b8/0x320
[ 466.256089] wait_for_common+0xb4/0x164
[ 466.256089] wait_for_common+0xb4/0x164
[ 466.263754] wait_for_completion+0x14/0x1c
[ 466.263754] wait_for_completion+0x14/0x1c
[ 466.271942] exit_aio+0x100/0x114
[ 466.271942] exit_aio+0x100/0x114
[ 466.278563] mmput+0x48/0x118
[ 466.278563] mmput+0x48/0x118
[ 466.284487] do_exit+0x278/0x978
[ 466.284487] do_exit+0x278/0x978
[ 466.290933] die+0x1c8/0x1f4
[ 466.290933] die+0x1c8/0x1f4
[ 466.296684] die_kernel_fault+0x64/0x74
[ 466.296684] die_kernel_fault+0x64/0x74
[ 466.304349] __do_kernel_fault+0x84/0xac
[ 466.304349] __do_kernel_fault+0x84/0xac
[ 466.312189] do_page_fault+0x1dc/0x3e8
[ 466.312189] do_page_fault+0x1dc/0x3e8
[ 466.319680] do_translation_fault+0x68/0x74
[ 466.319680] do_translation_fault+0x68/0x74
[ 466.328042] do_mem_abort+0x4c/0xf4
[ 466.328042] do_mem_abort+0x4c/0xf4
[ 466.335010] el1_da+0x20/0x80
[ 466.335010] el1_da+0x20/0x80
[ 466.340934] deadline_remove_request+0x5c/0xd0
[ 466.340934] deadline_remove_request+0x5c/0xd0
[ 466.349818] dd_dispatch_request+0x184/0x1e0
[ 466.349818] dd_dispatch_request+0x184/0x1e0
[ 466.358355] blk_mq_do_dispatch_sched+0x4c/0x104
[ 466.358355] blk_mq_do_dispatch_sched+0x4c/0x104
[ 466.367587] blk_mq_sched_dispatch_requests+0x130/0x1bc
[ 466.367587] blk_mq_sched_dispatch_requests+0x130/0x1bc
[ 466.378039] __blk_mq_run_hw_queue+0xa4/0x11c
[ 466.378039] __blk_mq_run_hw_queue+0xa4/0x11c
[ 466.386749] __blk_mq_delay_run_hw_queue+0x1a4/0x208
[ 466.386749] __blk_mq_delay_run_hw_queue+0x1a4/0x208
[ 466.396678] blk_mq_run_hw_queue+0x9c/0xf8
[ 466.396678] blk_mq_run_hw_queue+0x9c/0xf8
[ 466.404865] blk_mq_sched_insert_requests+0x58/0x84
[ 466.404865] blk_mq_sched_insert_requests+0x58/0x84
[ 466.414620] blk_mq_flush_plug_list+0x144/0x16c
[ 466.414620] blk_mq_flush_plug_list+0x144/0x16c
[ 466.423678] blk_flush_plug_list+0xe8/0x244
[ 466.423678] blk_flush_plug_list+0xe8/0x244
[ 466.432040] blk_finish_plug+0x30/0x40
[ 466.432040] blk_finish_plug+0x30/0x40
[ 466.439531] sys_io_submit+0x110/0x214
[ 466.439531] sys_io_submit+0x110/0x214
[ 466.447022] el0_svc_naked+0x30/0x34
[ 466.447022] el0_svc_naked+0x30/0x34
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:17h:52m:41s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:18h:01m:49s]
[ 467.851524] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 467.851524] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 467.862505] 0-....: (1 GPs behind) idle=ac6/1/4611686018427387904 softirq=4021/4728 fqs=34108
[ 467.862505] 0-....: (1 GPs behind) idle=ac6/1/4611686018427387904 softirq=4021/4728 fqs=34108
[ 467.879923] (detected by 14, t=68279 jiffies, g=-107, c=-108, q=1)
[ 467.879923] (detected by 14, t=68279 jiffies, g=-107, c=-108, q=1)
[ 467.892469] Task dump for CPU 0:
[ 467.892469] Task dump for CPU 0:
[ 467.898915] kworker/0:2H R running task 0 3757 2 0x0000002a
[ 467.898915] kworker/0:2H R running task 0 3757 2 0x0000002a
[ 467.913029] Workqueue: kblockd blk_mq_run_work_fn
[ 467.913029] Workqueue: kblockd blk_mq_run_work_fn
[ 467.922436] Call trace:
[ 467.922436] Call trace:
[ 467.927316] __switch_to+0x90/0xcc
[ 467.927316] __switch_to+0x90/0xcc
[ 467.934110] 0x674238a07d45b300
[ 467.934110] 0x674238a07d45b300
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:18h:10m:57s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:18h:20m:05s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:18h:29m:14s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:18h:38m:22s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:18h:47m:30s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:18h:56m:38s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:19h:05m:47s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:19h:14m:55s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:19h:24m:03s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:19h:33m:11s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:19h:42m:19s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:19h:51m:28s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:20h:00m:36s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:20h:09m:44s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:20h:18m:52s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:20h:28m:01s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:20h:37m:09s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:20h:46m:17s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:20h:55m:25s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:21h:04m:33s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:21h:13m:42s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:21h:22m:50s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:21h:31m:58s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:21h:41m:06s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:21h:50m:15s]
[ 492.763523] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2H:3757]
[ 492.763523] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2H:3757]
[ 492.778502] Modules linked in: hisi_sas_v3_hw
[ 492.778502] Modules linked in: hisi_sas_v3_hw
[ 492.787213] CPU: 0 PID: 3757 Comm: kworker/0:2H Tainted: G D W L 4.18.0-rc1-12443-g7e2d9d9-dirty #1077
[ 492.787213] CPU: 0 PID: 3757 Comm: kworker/0:2H Tainted: G D W L 4.18.0-rc1-12443-g7e2d9d9-dirty #1077
[ 492.807765] Hardware name: Huawei D06/D06, BIOS Hisilicon D06 UEFI RC0 - B601 (V6.01) 11/08/2018
[ 492.807765] Hardware name: Huawei D06/D06, BIOS Hisilicon D06 UEFI RC0 - B601 (V6.01) 11/08/2018
[ 492.825357] Workqueue: kblockd blk_mq_run_work_fn
[ 492.825357] Workqueue: kblockd blk_mq_run_work_fn
[ 492.834764] pstate: 80c00009 (Nzcv daif +PAN +UAO)
[ 492.834764] pstate: 80c00009 (Nzcv daif +PAN +UAO)
[ 492.844345] pc : _raw_spin_lock+0x34/0x44
[ 492.844345] pc : _raw_spin_lock+0x34/0x44
[ 492.852358] lr : dd_dispatch_request+0x2c/0x1e0
[ 492.852358] lr : dd_dispatch_request+0x2c/0x1e0
[ 492.861415] sp : ffff00002462bc60
[ 492.861415] sp : ffff00002462bc60
[ 492.868035] x29: ffff00002462bc60 x28: ffff000009973ce8
[ 492.868035] x29: ffff00002462bc60 x28: ffff000009973ce8
[ 492.878662] x27: ffff8027dceac938 x26: ffff000009467558
[ 492.878662] x27: ffff8027dceac938 x26: ffff000009467558
[ 492.889288] x25: 0000000000000000 x24: ffff000009451000
[ 492.889288] x25: 0000000000000000 x24: ffff000009451000
[ 492.899914] x23: ffff000009451000 x22: ffff8027a5a37200
[ 492.899914] x23: ffff000009451000 x22: ffff8027a5a37200
[ 492.910541] x21: ffff8027b24f4c5c x20: ffff00002462bce8
[ 492.910541] x21: ffff8027b24f4c5c x20: ffff00002462bce8
[ 492.921167] x19: ffff8027b24f4c00 x18: 000000000000000e
[ 492.921167] x19: ffff8027b24f4c00 x18: 000000000000000e
[ 492.931794] x17: 0000000000000001 x16: 0000000000000019
[ 492.931794] x17: 0000000000000001 x16: 0000000000000019
[ 492.942421] x15: ffffffffffffffff x14: 0000000000000001
[ 492.942421] x15: ffffffffffffffff x14: 0000000000000001
[ 492.953047] x13: 00000000c0a80101 x12: 0000000000000001
[ 492.953047] x13: 00000000c0a80101 x12: 0000000000000001
[ 492.963673] x11: 0000000000001000 x10: 0000000000000950
[ 492.963673] x11: 0000000000001000 x10: 0000000000000950
[ 492.974300] x9 : ffff00002462bd50 x8 : 0000000000000018
[ 492.974300] x9 : ffff00002462bd50 x8 : 0000000000000018
[ 492.984926] x7 : fefefefefefefeff x6 : 0000000000000000
[ 492.984926] x7 : fefefefefefefeff x6 : 0000000000000000
[ 492.995553] x5 : 000000000000001d x4 : 000000000000001e
[ 492.995553] x5 : 000000000000001d x4 : 000000000000001e
[ 493.006179] x3 : 0000000000003996 x2 : 0000000000000001
[ 493.006179] x3 : 0000000000003996 x2 : 0000000000000001
[ 493.016806] x1 : 0000000039973996 x0 : ffff8027b24f4c5c
[ 493.016806] x1 : 0000000039973996 x0 : ffff8027b24f4c5c
[ 493.027432] Call trace:
[ 493.027432] Call trace:
[ 493.032311] _raw_spin_lock+0x34/0x44
[ 493.032311] _raw_spin_lock+0x34/0x44
[ 493.039628] blk_mq_do_dispatch_sched+0x4c/0x104
[ 493.039628] blk_mq_do_dispatch_sched+0x4c/0x104
[ 493.048861] blk_mq_sched_dispatch_requests+0x130/0x1bc
[ 493.048861] blk_mq_sched_dispatch_requests+0x130/0x1bc
[ 493.059312] __blk_mq_run_hw_queue+0xa4/0x11c
[ 493.059312] __blk_mq_run_hw_queue+0xa4/0x11c
[ 493.068022] blk_mq_run_work_fn+0x18/0x20
[ 493.068022] blk_mq_run_work_fn+0x18/0x20
[ 493.076035] process_one_work+0x140/0x2b0
[ 493.076035] process_one_work+0x140/0x2b0
[ 493.084049] worker_thread+0x130/0x3e4
[ 493.084049] worker_thread+0x130/0x3e4
[ 493.091540] kthread+0xfc/0x128
[ 493.091540] kthread+0xfc/0x128
[ 493.097811] ret_from_fork+0x10/0x18
[ 493.097811] ret_from_fork+0x10/0x18
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:21h:59m:23s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:22h:08m:31s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:22h:17m:39s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:22h:26m:47s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:22h:35m:56s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:22h:45m:04s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:22h:54m:12s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:23h:03m:20s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:23h:12m:29s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:23h:21m:37s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:23h:30m:45s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:23h:39m:53s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:23h:49m:01s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 01d:23h:58m:10s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 02d:00h:07m:18s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 02d:00h:16m:26s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 02d:00h:25m:34s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 02d:00h:34m:43s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 02d:00h:43m:51s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 02d:00h:52m:59s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 02d:01h:02m:07s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 02d:01h:11m:15s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 02d:01h:20m:24s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 02d:01h:29m:32s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 02d:01h:38m:40s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 02d:01h:47m:48s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 02d:01h:56m:57s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 02d:02h:06m:05s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 02d:02h:15m:13s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 02d:02h:24m:21s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 02d:02h:33m:29s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 02d:02h:42m:38s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 02d:02h:51m:46s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 02d:03h:00m:54s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 02d:03h:10m:02s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 02d:03h:19m:11s]
[ 528.843523] INFO: rcu_preempt self-detected stall on CPU
[ 528.843523] INFO: rcu_preempt self-detected stall on CPU
[ 528.851527] INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 528.851527] INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 528.854150] 0-....: (1 GPs behind) idle=ac6/1/4611686018427387906 softirq=4726/4728 fqs=41565
[ 528.854150] 0-....: (1 GPs behind) idle=ac6/1/4611686018427387906 softirq=4726/4728 fqs=41565
[ 528.882888]
[ 528.882888]
[ 528.882890] Tasks blocked on level-1 rcu_node (CPUs 0-15): P3755
[ 528.882890] Tasks blocked on level-1 rcu_node (CPUs 0-15): P3755
[ 528.886030] (t=84035 jiffies g=1596 c=1595 q=1394)
[ 528.886030] (t=84035 jiffies g=1596 c=1595 q=1394)
[ 528.898225] 0-....: (1 GPs behind) idle=ac6/1/4611686018427387906 softirq=4726/4728 fqs=41566
[ 528.898225] 0-....: (1 GPs behind) idle=ac6/1/4611686018427387906 softirq=4726/4728 fqs=41566
[ 528.925394] Task dump for CPU 0:
[ 528.925394] Task dump for CPU 0:
[ 528.925395] (detected by 4, t=84045 jiffies, g=1596, c=1595, q=1394)
[ 528.925395] (detected by 4, t=84045 jiffies, g=1596, c=1595, q=1394)
[ 528.931842] kworker/0:2H R running task 0 3757 2 0x0000002a
[ 528.931842] kworker/0:2H R running task 0 3757 2 0x0000002a
[ 528.958842] Workqueue: kblockd blk_mq_run_work_fn
[ 528.958842] Workqueue: kblockd blk_mq_run_work_fn
[ 528.968248] Call trace:
[ 528.968248] Call trace:
[ 528.973128] dump_backtrace+0x0/0x180
[ 528.973128] dump_backtrace+0x0/0x180
[ 528.980445] show_stack+0x14/0x1c
[ 528.980445] show_stack+0x14/0x1c
[ 528.987065] sched_show_task+0x16c/0x184
[ 528.987065] sched_show_task+0x16c/0x184
[ 528.994904] dump_cpu_task+0x40/0x4c
[ 528.994904] dump_cpu_task+0x40/0x4c
[ 529.002046] rcu_dump_cpu_stacks+0x98/0xd8
[ 529.002046] rcu_dump_cpu_stacks+0x98/0xd8
[ 529.010233] rcu_check_callbacks+0x750/0x9c0
[ 529.010233] rcu_check_callbacks+0x750/0x9c0
[ 529.018770] update_process_times+0x2c/0x54
[ 529.018770] update_process_times+0x2c/0x54
[ 529.027131] tick_sched_handle.isra.14+0x24/0x68
[ 529.027131] tick_sched_handle.isra.14+0x24/0x68
[ 529.036364] tick_sched_timer+0x48/0x98
[ 529.036364] tick_sched_timer+0x48/0x98
[ 529.044028] __hrtimer_run_queues+0x100/0x158
[ 529.044028] __hrtimer_run_queues+0x100/0x158
[ 529.052738] hrtimer_interrupt+0xe4/0x248
[ 529.052738] hrtimer_interrupt+0xe4/0x248
[ 529.060752] arch_timer_handler_phys+0x2c/0x38
[ 529.060752] arch_timer_handler_phys+0x2c/0x38
[ 529.069636] handle_percpu_devid_irq+0x78/0x12c
[ 529.069636] handle_percpu_devid_irq+0x78/0x12c
[ 529.078694] generic_handle_irq+0x24/0x38
[ 529.078694] generic_handle_irq+0x24/0x38
[ 529.086708] __handle_domain_irq+0x60/0xac
[ 529.086708] __handle_domain_irq+0x60/0xac
[ 529.094896] gic_handle_irq+0xd4/0x184
[ 529.094896] gic_handle_irq+0xd4/0x184
[ 529.102387] el1_irq+0xb0/0x128
[ 529.102387] el1_irq+0xb0/0x128
[ 529.108659] _raw_spin_lock+0x34/0x44
[ 529.108659] _raw_spin_lock+0x34/0x44
[ 529.115976] blk_mq_do_dispatch_sched+0x4c/0x104
[ 529.115976] blk_mq_do_dispatch_sched+0x4c/0x104
[ 529.125208] blk_mq_sched_dispatch_requests+0x130/0x1bc
[ 529.125208] blk_mq_sched_dispatch_requests+0x130/0x1bc
[ 529.135660] __blk_mq_run_hw_queue+0xa4/0x11c
[ 529.135660] __blk_mq_run_hw_queue+0xa4/0x11c
[ 529.144370] blk_mq_run_work_fn+0x18/0x20
[ 529.144370] blk_mq_run_work_fn+0x18/0x20
[ 529.152383] process_one_work+0x140/0x2b0
[ 529.152383] process_one_work+0x140/0x2b0
[ 529.160396] worker_thread+0x130/0x3e4
[ 529.160396] worker_thread+0x130/0x3e4
[ 529.167887] kthread+0xfc/0x128
[ 529.167887] kthread+0xfc/0x128
[ 529.174159] ret_from_fork+0x10/0x18
[ 529.174159] ret_from_fork+0x10/0x18
[ 529.181302] Task dump for CPU 0:
[ 529.181302] Task dump for CPU 0:
[ 529.187749] kworker/0:2H R running task 0 3757 2 0x0000002a
[ 529.187749] kworker/0:2H R running task 0 3757 2 0x0000002a
[ 529.201864] Workqueue: kblockd blk_mq_run_work_fn
[ 529.201864] Workqueue: kblockd blk_mq_run_work_fn
[ 529.211271] Call trace:
[ 529.211271] Call trace:
[ 529.216151] __switch_to+0x90/0xcc
[ 529.216151] __switch_to+0x90/0xcc
[ 529.222947] 0x674238a07d45b300
[ 529.222947] 0x674238a07d45b300
[ 529.229220] fio D 0 3755 3753 0x0000002c
[ 529.229220] fio D 0 3755 3753 0x0000002c
[ 529.240195] Call trace:
[ 529.240195] Call trace:
[ 529.245074] __switch_to+0x90/0xcc
[ 529.245074] __switch_to+0x90/0xcc
[ 529.251870] __schedule+0x1b8/0x5cc
[ 529.251870] __schedule+0x1b8/0x5cc
[ 529.258838] schedule+0x38/0x9c
[ 529.258838] schedule+0x38/0x9c
[ 529.265110] schedule_timeout+0x1b8/0x320
[ 529.265110] schedule_timeout+0x1b8/0x320
[ 529.273123] wait_for_common+0xb4/0x164
[ 529.273123] wait_for_common+0xb4/0x164
[ 529.280788] wait_for_completion+0x14/0x1c
[ 529.280788] wait_for_completion+0x14/0x1c
[ 529.288977] exit_aio+0x100/0x114
[ 529.288977] exit_aio+0x100/0x114
[ 529.295599] mmput+0x48/0x118
[ 529.295599] mmput+0x48/0x118
[ 529.301522] do_exit+0x278/0x978
[ 529.301522] do_exit+0x278/0x978
[ 529.307969] die+0x1c8/0x1f4
[ 529.307969] die+0x1c8/0x1f4
[ 529.313720] die_kernel_fault+0x64/0x74
[ 529.313720] die_kernel_fault+0x64/0x74
[ 529.321385] __do_kernel_fault+0x84/0xac
[ 529.321385] __do_kernel_fault+0x84/0xac
[ 529.329225] do_page_fault+0x1dc/0x3e8
[ 529.329225] do_page_fault+0x1dc/0x3e8
[ 529.336716] do_translation_fault+0x68/0x74
[ 529.336716] do_translation_fault+0x68/0x74
[ 529.345078] do_mem_abort+0x4c/0xf4
[ 529.345078] do_mem_abort+0x4c/0xf4
[ 529.352046] el1_da+0x20/0x80
[ 529.352046] el1_da+0x20/0x80
[ 529.357970] deadline_remove_request+0x5c/0xd0
[ 529.357970] deadline_remove_request+0x5c/0xd0
[ 529.366854] dd_dispatch_request+0x184/0x1e0
[ 529.366854] dd_dispatch_request+0x184/0x1e0
[ 529.375391] blk_mq_do_dispatch_sched+0x4c/0x104
[ 529.375391] blk_mq_do_dispatch_sched+0x4c/0x104
[ 529.384623] blk_mq_sched_dispatch_requests+0x130/0x1bc
[ 529.384623] blk_mq_sched_dispatch_requests+0x130/0x1bc
[ 529.395075] __blk_mq_run_hw_queue+0xa4/0x11c
[ 529.395075] __blk_mq_run_hw_queue+0xa4/0x11c
[ 529.403785] __blk_mq_delay_run_hw_queue+0x1a4/0x208
[ 529.403785] __blk_mq_delay_run_hw_queue+0x1a4/0x208
[ 529.413713] blk_mq_run_hw_queue+0x9c/0xf8
[ 529.413713] blk_mq_run_hw_queue+0x9c/0xf8
[ 529.421901] blk_mq_sched_insert_requests+0x58/0x84
[ 529.421901] blk_mq_sched_insert_requests+0x58/0x84
[ 529.431656] blk_mq_flush_plug_list+0x144/0x16c
[ 529.431656] blk_mq_flush_plug_list+0x144/0x16c
[ 529.440715] blk_flush_plug_list+0xe8/0x244
[ 529.440715] blk_flush_plug_list+0xe8/0x244
[ 529.449077] blk_finish_plug+0x30/0x40
[ 529.449077] blk_finish_plug+0x30/0x40
[ 529.456568] sys_io_submit+0x110/0x214
[ 529.456568] sys_io_submit+0x110/0x214
[ 529.464059] el0_svc_naked+0x30/0x34
[ 529.464059] el0_svc_naked+0x30/0x34
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 02d:03h:28m:19s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 02d:03h:37m:27s]
[ 530.871525] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 530.871525] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 530.882503] 0-....: (1 GPs behind) idle=ac6/1/4611686018427387904 softirq=4021/4728 fqs=41978
[ 530.882503] 0-....: (1 GPs behind) idle=ac6/1/4611686018427387904 softirq=4021/4728 fqs=41978
[ 530.899921] (detected by 14, t=84034 jiffies, g=-107, c=-108, q=1)
[ 530.899921] (detected by 14, t=84034 jiffies, g=-107, c=-108, q=1)
[ 530.912465] Task dump for CPU 0:
[ 530.912465] Task dump for CPU 0:
[ 530.918911] kworker/0:2H R running task 0 3757 2 0x0000002a
[ 530.918911] kworker/0:2H R running task 0 3757 2 0x0000002a
[ 530.933024] Workqueue: kblockd blk_mq_run_work_fn
[ 530.933024] Workqueue: kblockd blk_mq_run_work_fn
[ 530.942431] Call trace:
[ 530.942431] Call trace:
[ 530.947310] __switch_to+0x90/0xcc
[ 530.947310] __switch_to+0x90/0xcc
[ 530.954105] 0x674238a07d45b300
[ 530.954105] 0x674238a07d45b300
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 02d:03h:46m:35s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 02d:03h:55m:43s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 02d:04h:04m:52s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 02d:04h:14m:00s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 02d:04h:23m:08s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 02d:04h:32m:16s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 02d:04h:41m:25s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 02d:04h:50m:33s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 02d:04h:59m:41s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 02d:05h:08m:49s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 02d:05h:17m:57s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 02d:05h:27m:06s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s] [0 /0 iops] [eta 02d:05h:36m:14s]
Jobs: 1 (f=1): [M] [0.2% done] [0K/0K /s]
