On 07/04/14 05:10, Junichi Nomura wrote: > could you collect output of 'dmsetup table' on the test machine? > My particular interest is whether pg_init is involved or not > since Hannes's patch made major surgery on pg_init processing. > > Also, output of the followings while the problem is happening might help: > dmsetup status > dmsetup info -c > cat /proc/diskstat > multipathd -k"show paths" > multipathd -k"show maps status" > multipathd -k"show maps stats" Hello Junichi, Thanks for looking into this issue. I have attached the requested information to this e-mail for a test run with kernel 3.16-rc4 at the initiator side. Bart.
# dmsetup table 26133373139386639: 0 5368709120 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 65:16 1000 1 65:192 1000 1 26230643330303865: 0 5368709120 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 65:48 1000 1 65:208 1000 1 23834333461383137: 0 5368709120 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 8:208 1000 1 65:128 1000 1 26361626563396362: 0 5368709120 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 65:80 1000 1 65:224 1000 1 26363633765623538: 0 256000 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 8:32 1000 1 8:128 1000 1 23832346438613834: 0 256000 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 8:96 1000 1 65:0 1000 1 26638323034336331: 0 256000 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 8:80 1000 1 8:224 1000 1 23937393633303630: 0 5368709120 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 8:240 1000 1 65:160 1000 1 23931656631326633: 0 256000 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 8:112 1000 1 65:32 1000 1 26665353936313532: 0 5368709120 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 65:176 1000 1 66:16 1000 1 26135303862613661: 0 256000 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 8:144 1000 1 65:64 1000 1 26439316335316263: 0 5368709120 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 65:112 1000 1 65:240 1000 1 26564666266393235: 0 5368709120 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 65:144 1000 1 66:0 1000 1 26562383264626236: 0 256000 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 8:64 1000 1 8:192 1000 1 26236616132323164: 0 256000 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 8:176 1000 1 65:96 1000 1 26466363537333266: 0 256000 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 8:48 1000 1 8:160 1000 1 # dmsetup status 26133373139386639: 0 5368709120 multipath 2 0 0 0 1 1 A 0 2 2 65:16 A 0 0 1 65:192 A 0 0 1 26230643330303865: 0 5368709120 multipath 2 0 0 0 1 1 A 0 2 2 65:48 A 0 0 1 65:208 A 0 0 1 23834333461383137: 0 5368709120 multipath 2 1 0 0 1 1 E 0 2 2 8:208 A 0 0 1 65:128 A 0 0 1 26361626563396362: 0 5368709120 multipath 2 1 0 0 1 1 E 0 2 2 65:80 A 0 0 1 65:224 A 0 0 1 26363633765623538: 0 256000 multipath 2 0 0 0 1 1 A 0 2 2 8:32 A 0 0 1 8:128 A 0 0 1 23832346438613834: 0 256000 multipath 2 1 0 0 1 1 E 0 2 2 8:96 A 0 0 1 65:0 A 0 0 1 26638323034336331: 0 256000 multipath 2 1 0 0 1 1 E 0 2 2 8:80 A 0 0 1 8:224 A 0 0 1 23937393633303630: 0 5368709120 multipath 2 0 0 0 1 1 A 0 2 2 8:240 A 0 0 1 65:160 A 0 0 1 23931656631326633: 0 256000 multipath 2 1 0 0 1 1 E 0 2 2 8:112 A 0 0 1 65:32 A 0 0 1 26665353936313532: 0 5368709120 multipath 2 0 0 0 1 1 A 0 2 2 65:176 A 0 0 1 66:16 A 0 0 1 26135303862613661: 0 256000 multipath 2 0 0 0 1 1 A 0 2 2 8:144 A 0 0 1 65:64 A 0 0 1 26439316335316263: 0 5368709120 multipath 2 0 0 0 1 1 A 0 2 2 65:112 A 0 0 1 65:240 A 0 0 1 26564666266393235: 0 5368709120 multipath 2 0 0 0 1 1 A 0 2 2 65:144 A 0 0 1 66:0 A 0 0 1 26562383264626236: 0 256000 multipath 2 1 0 0 1 1 E 0 2 2 8:64 A 0 0 1 8:192 A 0 0 1 26236616132323164: 0 256000 multipath 2 0 0 0 1 1 A 0 2 2 8:176 A 0 0 1 65:96 A 0 0 1 26466363537333266: 0 256000 multipath 2 1 0 0 1 1 E 0 2 2 8:48 A 0 0 1 8:160 A 0 0 1 # dmsetup info -c Name Maj Min Stat Open Targ Event UUID 26133373139386639 253 10 L--w 0 1 0 mpath-26133373139386639 26230643330303865 253 11 L--w 0 1 0 mpath-26230643330303865 23834333461383137 253 9 L--w 1 1 0 mpath-23834333461383137 26361626563396362 253 13 L--w 1 1 0 mpath-26361626563396362 26363633765623538 253 0 L--w 1 1 5 mpath-26363633765623538 23832346438613834 253 4 L--w 1 1 0 mpath-23832346438613834 26638323034336331 253 3 L--w 1 1 0 mpath-26638323034336331 23937393633303630 253 8 L--w 0 1 0 mpath-23937393633303630 23931656631326633 253 5 L--w 1 1 0 mpath-23931656631326633 26665353936313532 253 15 L--w 0 1 0 mpath-26665353936313532 26135303862613661 253 6 L--w 0 1 0 mpath-26135303862613661 26439316335316263 253 12 L--w 0 1 0 mpath-26439316335316263 26564666266393235 253 14 L--w 0 1 0 mpath-26564666266393235 26562383264626236 253 2 L--w 1 1 0 mpath-26562383264626236 26236616132323164 253 7 L--w 0 1 0 mpath-26236616132323164 26466363537333266 253 1 L--w 1 1 0 mpath-26466363537333266 # cat /proc/diskstats 8 0 sda 63494 687 2082482 45080 11530 6225 362768 690500 0 23530 734810 8 1 sda1 170 0 1360 190 0 0 0 0 0 190 190 8 2 sda2 63143 687 2079674 44620 11509 6225 362768 690490 0 23500 734330 8 16 sdb 374 6 3040 460 0 0 0 0 0 460 460 8 17 sdb1 180 0 1440 190 0 0 0 0 0 190 190 11 0 sr0 0 0 0 0 0 0 0 0 0 0 0 253 0 dm-0 283565 1406 2268002 22758620 295490 134 2560844 532030 0 671930 23290480 8 32 sdc 1489 0 11912 30130 1168 0 9320 1020 0 1640 31150 8 48 sdd 362 0 2896 1310 0 0 0 0 0 1310 1310 8 64 sde 347 0 2776 1180 0 0 0 0 0 1170 1180 8 80 sdf 346 0 2768 910 0 0 0 0 0 900 910 8 96 sdg 389 0 3112 1020 0 0 0 0 0 1020 1020 8 112 sdh 417 0 3336 1080 0 0 0 0 0 1070 1080 8 128 sdi 234 0 1872 450 1132 0 9050 1590 0 2030 2040 8 144 sdj 422 0 3400 1050 0 0 0 0 0 1040 1050 8 160 sdk 234 0 1872 520 0 0 0 0 0 520 520 8 176 sdl 421 0 3392 1030 0 0 0 0 0 1020 1030 8 192 sdm 234 0 1872 510 0 0 0 0 0 510 510 8 208 sdn 369 0 2952 880 0 0 0 0 0 880 880 8 224 sdo 234 0 1872 500 0 0 0 0 0 500 500 253 1 dm-1 127 0 1016 1070 0 0 0 0 1 278360 278820 8 240 sdp 425 0 3424 890 0 0 0 0 0 880 890 65 0 sdq 234 0 1872 500 0 0 0 0 0 500 500 65 16 sdr 424 0 3416 910 0 0 0 0 0 910 910 253 2 dm-2 112 0 896 1050 0 0 0 0 1 278240 278710 65 32 sds 233 0 1864 590 0 0 0 0 0 590 590 65 48 sdt 426 0 3432 860 0 0 0 0 0 850 860 65 64 sdu 421 0 3392 830 0 0 0 0 0 830 830 253 3 dm-3 111 0 888 1120 0 0 0 0 1 278170 278700 65 96 sdw 422 0 3400 910 0 0 0 0 0 910 910 65 80 sdv 404 0 3232 900 0 0 0 0 0 900 900 253 4 dm-4 154 0 1232 1310 0 0 0 0 1 278110 278720 65 112 sdx 425 0 3424 960 0 0 0 0 0 960 960 65 128 sdy 235 0 1880 570 0 0 0 0 0 570 570 253 5 dm-5 183 0 1464 1410 0 0 0 0 1 278050 278700 65 144 sdz 425 0 3424 820 0 0 0 0 0 820 820 65 160 sdaa 425 0 3424 830 0 0 0 0 0 830 830 253 6 dm-6 374 6 3040 1750 0 0 0 0 0 940 1750 65 176 sdab 426 0 3432 820 0 0 0 0 0 820 820 65 192 sdac 428 0 3448 830 0 0 0 0 0 830 830 65 208 sdad 425 0 3424 740 0 0 0 0 0 740 740 253 7 dm-7 374 6 3040 1620 0 0 0 0 0 880 1620 65 224 sdae 236 0 1888 500 0 0 0 0 0 500 500 65 240 sdaf 427 0 3440 640 0 0 0 0 0 640 640 66 0 sdag 427 0 3440 530 0 0 0 0 0 530 530 66 16 sdah 426 0 3432 540 0 0 0 0 0 540 540 253 8 dm-8 378 6 3072 1190 0 0 0 0 0 640 1190 253 9 dm-9 134 0 1072 520 0 0 0 0 1 277420 277640 253 10 dm-10 378 6 3072 1180 0 0 0 0 0 660 1180 253 11 dm-11 378 6 3072 1000 0 0 0 0 0 530 1000 253 12 dm-12 378 6 3072 990 0 0 0 0 0 560 990 253 13 dm-13 168 0 1344 500 0 0 0 0 1 276920 277150 253 14 dm-14 378 6 3072 910 0 0 0 0 0 480 910 253 15 dm-15 378 6 3072 950 0 0 0 0 0 500 950 # multipathd -k"show paths" hcil dev dev_t pri dm_st chk_st dev_st next_check 15:0:0:0 sdc 8:32 1 active ready running XX........ 1/4 15:0:0:1 sdd 8:48 1 active ready running XX........ 1/4 15:0:0:2 sde 8:64 1 active ready running XX........ 1/4 15:0:0:3 sdf 8:80 1 active ready running XX........ 1/4 15:0:0:4 sdg 8:96 1 active ready running XX........ 1/4 15:0:0:5 sdh 8:112 1 active ready running XX........ 1/4 15:0:0:6 sdj 8:144 1 active ready running XX........ 1/4 16:0:0:0 sdi 8:128 1 active ready running XX........ 1/4 15:0:0:7 sdl 8:176 1 active ready running XX........ 1/4 16:0:0:1 sdk 8:160 1 active ready running XX........ 1/4 16:0:0:2 sdm 8:192 1 active ready running XX........ 1/4 16:0:0:3 sdo 8:224 1 active ready running XX........ 1/4 15:0:0:11 sdp 8:240 1 active ready running XX........ 1/4 15:0:0:10 sdn 8:208 1 active ready running XX........ 1/4 16:0:0:4 sdq 65:0 1 active ready running XX........ 1/4 15:0:0:12 sdr 65:16 1 active ready running XX........ 1/4 16:0:0:5 sds 65:32 1 active ready running XX........ 1/4 16:0:0:6 sdu 65:64 1 active ready running XX........ 1/4 15:0:0:13 sdt 65:48 1 active ready running XX........ 1/4 16:0:0:10 sdy 65:128 1 active ready running XX........ 1/4 16:0:0:7 sdw 65:96 1 active ready running XX........ 1/4 15:0:0:15 sdx 65:112 1 active ready running XX........ 1/4 15:0:0:14 sdv 65:80 1 active ready running XX........ 1/4 15:0:0:16 sdz 65:144 1 active ready running XX........ 1/4 15:0:0:17 sdab 65:176 1 active ready running XX........ 1/4 16:0:0:12 sdac 65:192 1 active ready running XX........ 1/4 16:0:0:11 sdaa 65:160 1 active ready running XX........ 1/4 16:0:0:13 sdad 65:208 1 active ready running XX........ 1/4 16:0:0:14 sdae 65:224 1 active ready running XX........ 1/4 16:0:0:15 sdaf 65:240 1 active ready running XX........ 1/4 16:0:0:17 sdah 66:16 1 active ready running XX........ 1/4 16:0:0:16 sdag 66:0 1 active ready running XX........ 1/4 # multipathd -k"show maps status" name failback queueing paths dm-st write_prot 26363633765623538 - on 2 active rw 26466363537333266 - on 2 active rw 26562383264626236 - on 2 active rw 26638323034336331 - on 2 active rw 23832346438613834 - on 2 active rw 23931656631326633 - on 2 active rw 26135303862613661 - on 2 active rw 26236616132323164 - on 2 active rw 23937393633303630 - on 2 active rw 23834333461383137 - on 2 active rw 26133373139386639 - on 2 active rw 26230643330303865 - on 2 active rw 26439316335316263 - on 2 active rw 26361626563396362 - on 2 active rw 26564666266393235 - on 2 active rw 26665353936313532 - on 2 active rw # multipathd -k"show maps stats" name path_faults switch_grp map_loads total_q_time q_timeouts 26363633765623538 5 0 9 0 0 26466363537333266 0 0 2 0 0 26562383264626236 0 0 2 0 0 26638323034336331 0 0 2 0 0 23832346438613834 0 0 2 0 0 23931656631326633 0 0 2 0 0 26135303862613661 0 0 2 0 0 26236616132323164 0 0 2 0 0 23937393633303630 0 0 2 0 0 23834333461383137 0 0 2 0 0 26133373139386639 0 0 2 0 0 26230643330303865 0 0 2 0 0 26439316335316263 0 0 2 0 0 26361626563396362 0 0 2 0 0 26564666266393235 0 0 2 0 0 26665353936313532 0 0 2 0 0 SysRq : Show Blocked State task PC stack pid father systemd-udevd D ffff88083b6f8000 0 8283 358 0x00000000 ffff8807c69f3b30 0000000000000002 ffff88083b6f8000 ffff8807c69f3fd8 00000000000131c0 00000000000131c0 ffffffff81a154c0 ffff8807d7578698 ffff8807d75786a0 0000000000000246 ffff88083b6f8000 0000000000000000 Call Trace: [<ffffffff814bc3a3>] schedule_preempt_disabled+0x33/0x80 [<ffffffff814bd3dc>] mutex_lock_nested+0x15c/0x3d0 [<ffffffff811d4b56>] ? __blkdev_get+0x66/0x4c0 [<ffffffff811d4b56>] __blkdev_get+0x66/0x4c0 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50 [<ffffffff811d5195>] blkdev_get+0x1e5/0x380 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50 [<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50 [<ffffffff811d53db>] blkdev_open+0x5b/0x80 [<ffffffff81195fc2>] do_dentry_open.isra.15+0x1e2/0x320 [<ffffffff81196210>] finish_open+0x30/0x40 [<ffffffff811a82cd>] do_last.isra.61+0xa3d/0x1150 [<ffffffff811a4725>] ? path_init+0x5d5/0x750 [<ffffffff810a6900>] ? trace_hardirqs_on_caller+0xc0/0x1c0 [<ffffffff811a8a58>] ? path_openat+0x78/0x620 [<ffffffff811a8a97>] path_openat+0xb7/0x620 [<ffffffff811a967a>] do_filp_open+0x3a/0x90 [<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50 [<ffffffff811b8a1c>] ? __alloc_fd+0xac/0x1b0 [<ffffffff81197a5e>] do_sys_open+0x12e/0x210 [<ffffffff8101209d>] ? syscall_trace_enter+0xfd/0x200 [<ffffffff81197b5e>] SyS_open+0x1e/0x20 [<ffffffff814c1ea2>] tracesys+0xd0/0xd5 systemd-udevd D ffff88083b322230 0 8317 358 0x00000000 ffff8807cfa27b30 0000000000000002 ffff88083b322230 ffff8807cfa27fd8 00000000000131c0 00000000000131c0 ffff88083b490000 ffff88083ec1a098 ffff88083ec1a0a0 0000000000000246 ffff88083b322230 0000000000000000 Call Trace: [<ffffffff814bc3a3>] schedule_preempt_disabled+0x33/0x80 [<ffffffff814bd3dc>] mutex_lock_nested+0x15c/0x3d0 [<ffffffff811d4b56>] ? __blkdev_get+0x66/0x4c0 [<ffffffff811d4b56>] __blkdev_get+0x66/0x4c0 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50 [<ffffffff811d5195>] blkdev_get+0x1e5/0x380 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50 [<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50 [<ffffffff811d53db>] blkdev_open+0x5b/0x80 [<ffffffff81195fc2>] do_dentry_open.isra.15+0x1e2/0x320 [<ffffffff81196210>] finish_open+0x30/0x40 [<ffffffff811a82cd>] do_last.isra.61+0xa3d/0x1150 [<ffffffff811a4725>] ? path_init+0x5d5/0x750 [<ffffffff810a6900>] ? trace_hardirqs_on_caller+0xc0/0x1c0 [<ffffffff811a8a58>] ? path_openat+0x78/0x620 [<ffffffff811a8a97>] path_openat+0xb7/0x620 [<ffffffff811a967a>] do_filp_open+0x3a/0x90 [<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50 [<ffffffff811b8a1c>] ? __alloc_fd+0xac/0x1b0 [<ffffffff81197a5e>] do_sys_open+0x12e/0x210 [<ffffffff8101209d>] ? syscall_trace_enter+0xfd/0x200 [<ffffffff81197b5e>] SyS_open+0x1e/0x20 [<ffffffff814c1ea2>] tracesys+0xd0/0xd5 systemd-udevd D ffff88083b00a230 0 8329 358 0x00000000 ffff8807ee1b3b30 0000000000000002 ffff88083b00a230 ffff8807ee1b3fd8 00000000000131c0 00000000000131c0 ffff88083baea230 ffff88083ec1b418 ffff88083ec1b420 0000000000000246 ffff88083b00a230 0000000000000000 Call Trace: [<ffffffff814bc3a3>] schedule_preempt_disabled+0x33/0x80 [<ffffffff814bd3dc>] mutex_lock_nested+0x15c/0x3d0 [<ffffffff811d4b56>] ? __blkdev_get+0x66/0x4c0 [<ffffffff811d4b56>] __blkdev_get+0x66/0x4c0 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50 [<ffffffff811d5195>] blkdev_get+0x1e5/0x380 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50 [<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50 [<ffffffff811d53db>] blkdev_open+0x5b/0x80 [<ffffffff81195fc2>] do_dentry_open.isra.15+0x1e2/0x320 [<ffffffff81196210>] finish_open+0x30/0x40 [<ffffffff811a82cd>] do_last.isra.61+0xa3d/0x1150 [<ffffffff811a4725>] ? path_init+0x5d5/0x750 [<ffffffff810a6900>] ? trace_hardirqs_on_caller+0xc0/0x1c0 [<ffffffff811a8a58>] ? path_openat+0x78/0x620 [<ffffffff811a8a97>] path_openat+0xb7/0x620 [<ffffffff811a967a>] do_filp_open+0x3a/0x90 [<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50 [<ffffffff811b8a1c>] ? __alloc_fd+0xac/0x1b0 [<ffffffff81197a5e>] do_sys_open+0x12e/0x210 [<ffffffff8101209d>] ? syscall_trace_enter+0xfd/0x200 [<ffffffff81197b5e>] SyS_open+0x1e/0x20 [<ffffffff814c1ea2>] tracesys+0xd0/0xd5 systemd-udevd D ffff8807d1290000 0 8330 358 0x00000000 ffff8807cee23b30 0000000000000002 ffff8807d1290000 ffff8807cee23fd8 00000000000131c0 00000000000131c0 ffff88083b492230 ffff8807d77a6e98 ffff8807d77a6ea0 0000000000000246 ffff8807d1290000 0000000000000000 Call Trace: [<ffffffff814bc3a3>] schedule_preempt_disabled+0x33/0x80 [<ffffffff814bd3dc>] mutex_lock_nested+0x15c/0x3d0 [<ffffffff811d4b56>] ? __blkdev_get+0x66/0x4c0 [<ffffffff811d4b56>] __blkdev_get+0x66/0x4c0 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50 [<ffffffff811d5195>] blkdev_get+0x1e5/0x380 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50 [<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50 [<ffffffff811d53db>] blkdev_open+0x5b/0x80 [<ffffffff81195fc2>] do_dentry_open.isra.15+0x1e2/0x320 [<ffffffff81196210>] finish_open+0x30/0x40 [<ffffffff811a82cd>] do_last.isra.61+0xa3d/0x1150 [<ffffffff811a4725>] ? path_init+0x5d5/0x750 [<ffffffff810a6900>] ? trace_hardirqs_on_caller+0xc0/0x1c0 [<ffffffff811a8a58>] ? path_openat+0x78/0x620 [<ffffffff811a8a97>] path_openat+0xb7/0x620 [<ffffffff811a967a>] do_filp_open+0x3a/0x90 [<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50 [<ffffffff811b8a1c>] ? __alloc_fd+0xac/0x1b0 [<ffffffff81197a5e>] do_sys_open+0x12e/0x210 [<ffffffff8101209d>] ? syscall_trace_enter+0xfd/0x200 [<ffffffff81197b5e>] SyS_open+0x1e/0x20 [<ffffffff814c1ea2>] tracesys+0xd0/0xd5 systemd-udevd D ffff8807d168c460 0 8334 358 0x00000000 ffff8807d13bfb30 0000000000000002 ffff8807d168c460 ffff8807d13bffd8 00000000000131c0 00000000000131c0 ffff88083b422230 ffff8807d77a5498 ffff8807d77a54a0 0000000000000246 ffff8807d168c460 0000000000000000 Call Trace: [<ffffffff814bc3a3>] schedule_preempt_disabled+0x33/0x80 [<ffffffff814bd3dc>] mutex_lock_nested+0x15c/0x3d0 [<ffffffff811d4b56>] ? __blkdev_get+0x66/0x4c0 [<ffffffff811d4b56>] __blkdev_get+0x66/0x4c0 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50 [<ffffffff811d5195>] blkdev_get+0x1e5/0x380 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50 [<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50 [<ffffffff811d53db>] blkdev_open+0x5b/0x80 [<ffffffff81195fc2>] do_dentry_open.isra.15+0x1e2/0x320 [<ffffffff81196210>] finish_open+0x30/0x40 [<ffffffff811a82cd>] do_last.isra.61+0xa3d/0x1150 [<ffffffff811a4725>] ? path_init+0x5d5/0x750 [<ffffffff810a6900>] ? trace_hardirqs_on_caller+0xc0/0x1c0 [<ffffffff811a8a58>] ? path_openat+0x78/0x620 [<ffffffff811a8a97>] path_openat+0xb7/0x620 [<ffffffff811a967a>] do_filp_open+0x3a/0x90 [<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50 [<ffffffff811b8a1c>] ? __alloc_fd+0xac/0x1b0 [<ffffffff81197a5e>] do_sys_open+0x12e/0x210 [<ffffffff8101209d>] ? syscall_trace_enter+0xfd/0x200 [<ffffffff81197b5e>] SyS_open+0x1e/0x20 [<ffffffff814c1ea2>] tracesys+0xd0/0xd5 systemd-udevd D ffff8807d1688000 0 8335 358 0x00000000 ffff8807cf5f3b30 0000000000000002 ffff8807d1688000 ffff8807cf5f3fd8 00000000000131c0 00000000000131c0 ffff88083b494460 ffff88083ec1a718 ffff88083ec1a720 0000000000000246 ffff8807d1688000 0000000000000000 Call Trace: [<ffffffff814bc3a3>] schedule_preempt_disabled+0x33/0x80 [<ffffffff814bd3dc>] mutex_lock_nested+0x15c/0x3d0 [<ffffffff811d4b56>] ? __blkdev_get+0x66/0x4c0 [<ffffffff811d4b56>] __blkdev_get+0x66/0x4c0 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50 [<ffffffff811d5195>] blkdev_get+0x1e5/0x380 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50 [<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50 [<ffffffff811d53db>] blkdev_open+0x5b/0x80 [<ffffffff81195fc2>] do_dentry_open.isra.15+0x1e2/0x320 [<ffffffff81196210>] finish_open+0x30/0x40 [<ffffffff811a82cd>] do_last.isra.61+0xa3d/0x1150 [<ffffffff811a4725>] ? path_init+0x5d5/0x750 [<ffffffff810a6900>] ? trace_hardirqs_on_caller+0xc0/0x1c0 [<ffffffff811a8a58>] ? path_openat+0x78/0x620 [<ffffffff811a8a97>] path_openat+0xb7/0x620 [<ffffffff811a967a>] do_filp_open+0x3a/0x90 [<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50 [<ffffffff811b8a1c>] ? __alloc_fd+0xac/0x1b0 [<ffffffff81197a5e>] do_sys_open+0x12e/0x210 [<ffffffff8101209d>] ? syscall_trace_enter+0xfd/0x200 [<ffffffff81197b5e>] SyS_open+0x1e/0x20 [<ffffffff814c1ea2>] tracesys+0xd0/0xd5 systemd-udevd D ffff88082df38000 0 8341 358 0x00000000 ffff8807d1763b30 0000000000000002 ffff88082df38000 ffff8807d1763fd8 00000000000131c0 00000000000131c0 ffff88083b422230 ffff8807d757e818 ffff8807d757e820 0000000000000246 ffff88082df38000 0000000000000000 Call Trace: [<ffffffff814bc3a3>] schedule_preempt_disabled+0x33/0x80 [<ffffffff814bd3dc>] mutex_lock_nested+0x15c/0x3d0 [<ffffffff811d4b56>] ? __blkdev_get+0x66/0x4c0 [<ffffffff811d4b56>] __blkdev_get+0x66/0x4c0 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50 [<ffffffff811d5195>] blkdev_get+0x1e5/0x380 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50 [<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50 [<ffffffff811d53db>] blkdev_open+0x5b/0x80 [<ffffffff81195fc2>] do_dentry_open.isra.15+0x1e2/0x320 [<ffffffff81196210>] finish_open+0x30/0x40 [<ffffffff811a82cd>] do_last.isra.61+0xa3d/0x1150 [<ffffffff811a4725>] ? path_init+0x5d5/0x750 [<ffffffff810a6900>] ? trace_hardirqs_on_caller+0xc0/0x1c0 [<ffffffff811a8a58>] ? path_openat+0x78/0x620 [<ffffffff811a8a97>] path_openat+0xb7/0x620 [<ffffffff811a967a>] do_filp_open+0x3a/0x90 [<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50 [<ffffffff811b8a1c>] ? __alloc_fd+0xac/0x1b0 [<ffffffff81197a5e>] do_sys_open+0x12e/0x210 [<ffffffff8101209d>] ? syscall_trace_enter+0xfd/0x200 [<ffffffff81197b5e>] SyS_open+0x1e/0x20 [<ffffffff814c1ea2>] tracesys+0xd0/0xd5 blkid D ffff8807d3110000 0 8422 1 0x00000006 ffff8807d59879b8 0000000000000002 ffff8807d3110000 ffff8807d5987fd8 00000000000131c0 00000000000131c0 ffff88083b4c4460 ffff88085fd33ad0 ffff88085ff83038 ffff8807d5987a40 0000000000000002 ffffffff81135db0 Call Trace: [<ffffffff81135db0>] ? wait_on_page_read+0x60/0x60 [<ffffffff814bc0ed>] io_schedule+0x9d/0x130 [<ffffffff81135dbe>] sleep_on_page+0xe/0x20 [<ffffffff814bc738>] __wait_on_bit_lock+0x48/0xb0 [<ffffffff81135eca>] __lock_page+0x6a/0x70 [<ffffffff8109c8e0>] ? autoremove_wake_function+0x40/0x40 [<ffffffff811474cf>] truncate_inode_pages_range+0x3ff/0x690 [<ffffffff81147775>] truncate_inode_pages+0x15/0x20 [<ffffffff811d3425>] kill_bdev+0x35/0x40 [<ffffffff811d49a9>] __blkdev_put+0x69/0x1b0 [<ffffffff811d5450>] blkdev_put+0x50/0x160 [<ffffffff811d5615>] blkdev_close+0x25/0x30 [<ffffffff8119a37a>] __fput+0xea/0x1f0 [<ffffffff8119a4ce>] ____fput+0xe/0x10 [<ffffffff81074ddc>] task_work_run+0xac/0xe0 [<ffffffff8104ff67>] do_exit+0x2c7/0xc60 [<ffffffff81064091>] ? get_signal_to_deliver+0xd1/0x940 [<ffffffff814c118c>] ? _raw_spin_unlock_irq+0x2c/0x60 [<ffffffff81051cac>] do_group_exit+0x4c/0xc0 [<ffffffff810642a1>] get_signal_to_deliver+0x2e1/0x940 [<ffffffff81002528>] do_signal+0x48/0x630 [<ffffffff811d3f77>] ? blkdev_read_iter+0x37/0x40 [<ffffffff8119809e>] ? new_sync_read+0x7e/0xb0 [<ffffffff8127f0d7>] ? debug_smp_processor_id+0x17/0x20 [<ffffffff810ba5d6>] ? rcu_eqs_exit_common.isra.43+0x26/0xe0 [<ffffffff8113539a>] ? context_tracking_user_exit+0x5a/0x1a0 [<ffffffff810a693d>] ? trace_hardirqs_on_caller+0xfd/0x1c0 [<ffffffff81002b81>] do_notify_resume+0x71/0xc0 [<ffffffff814c1f98>] int_signal+0x12/0x17 blkid D ffff8807d3632230 0 8458 1 0x00000006 ffff8807d360f9b8 0000000000000002 ffff8807d3632230 ffff8807d360ffd8 00000000000131c0 00000000000131c0 ffff88083b492230 ffff88085fc93ad0 ffff88085ff7fa38 ffff8807d360fa40 0000000000000002 ffffffff81135db0 Call Trace: [<ffffffff81135db0>] ? wait_on_page_read+0x60/0x60 [<ffffffff814bc0ed>] io_schedule+0x9d/0x130 [<ffffffff81135dbe>] sleep_on_page+0xe/0x20 [<ffffffff814bc738>] __wait_on_bit_lock+0x48/0xb0 [<ffffffff81135eca>] __lock_page+0x6a/0x70 [<ffffffff8109c8e0>] ? autoremove_wake_function+0x40/0x40 [<ffffffff811474cf>] truncate_inode_pages_range+0x3ff/0x690 [<ffffffff81147775>] truncate_inode_pages+0x15/0x20 [<ffffffff811d3425>] kill_bdev+0x35/0x40 [<ffffffff811d49a9>] __blkdev_put+0x69/0x1b0 [<ffffffff811d5450>] blkdev_put+0x50/0x160 [<ffffffff811d5615>] blkdev_close+0x25/0x30 [<ffffffff8119a37a>] __fput+0xea/0x1f0 [<ffffffff8119a4ce>] ____fput+0xe/0x10 [<ffffffff81074ddc>] task_work_run+0xac/0xe0 [<ffffffff8104ff67>] do_exit+0x2c7/0xc60 [<ffffffff81064091>] ? get_signal_to_deliver+0xd1/0x940 [<ffffffff814c118c>] ? _raw_spin_unlock_irq+0x2c/0x60 [<ffffffff81051cac>] do_group_exit+0x4c/0xc0 [<ffffffff810642a1>] get_signal_to_deliver+0x2e1/0x940 [<ffffffff81002528>] do_signal+0x48/0x630 [<ffffffff811d3f77>] ? blkdev_read_iter+0x37/0x40 [<ffffffff8119809e>] ? new_sync_read+0x7e/0xb0 [<ffffffff8127f0d7>] ? debug_smp_processor_id+0x17/0x20 [<ffffffff810ba5d6>] ? rcu_eqs_exit_common.isra.43+0x26/0xe0 [<ffffffff8113539a>] ? context_tracking_user_exit+0x5a/0x1a0 [<ffffffff810a693d>] ? trace_hardirqs_on_caller+0xfd/0x1c0 [<ffffffff81002b81>] do_notify_resume+0x71/0xc0 [<ffffffff814c1f98>] int_signal+0x12/0x17 blkid D ffff8807d4432230 0 8467 1 0x00000006 ffff8807d4daf9b8 0000000000000002 ffff8807d4432230 ffff8807d4daffd8 00000000000131c0 00000000000131c0 ffff88083b4c2230 ffff88085fd53ad0 ffff88085ffa4338 ffff8807d4dafa40 0000000000000002 ffffffff81135db0 Call Trace: [<ffffffff81135db0>] ? wait_on_page_read+0x60/0x60 [<ffffffff814bc0ed>] io_schedule+0x9d/0x130 [<ffffffff81135dbe>] sleep_on_page+0xe/0x20 [<ffffffff814bc738>] __wait_on_bit_lock+0x48/0xb0 [<ffffffff81135eca>] __lock_page+0x6a/0x70 [<ffffffff8109c8e0>] ? autoremove_wake_function+0x40/0x40 [<ffffffff811474cf>] truncate_inode_pages_range+0x3ff/0x690 [<ffffffff81147775>] truncate_inode_pages+0x15/0x20 [<ffffffff811d3425>] kill_bdev+0x35/0x40 [<ffffffff811d49a9>] __blkdev_put+0x69/0x1b0 [<ffffffff811d5450>] blkdev_put+0x50/0x160 [<ffffffff811d5615>] blkdev_close+0x25/0x30 [<ffffffff8119a37a>] __fput+0xea/0x1f0 [<ffffffff8119a4ce>] ____fput+0xe/0x10 [<ffffffff81074ddc>] task_work_run+0xac/0xe0 [<ffffffff8104ff67>] do_exit+0x2c7/0xc60 [<ffffffff81064091>] ? get_signal_to_deliver+0xd1/0x940 [<ffffffff814c118c>] ? _raw_spin_unlock_irq+0x2c/0x60 [<ffffffff81051cac>] do_group_exit+0x4c/0xc0 [<ffffffff810642a1>] get_signal_to_deliver+0x2e1/0x940 [<ffffffff81002528>] do_signal+0x48/0x630 [<ffffffff811d3f77>] ? blkdev_read_iter+0x37/0x40 [<ffffffff8119809e>] ? new_sync_read+0x7e/0xb0 [<ffffffff8127f0d7>] ? debug_smp_processor_id+0x17/0x20 [<ffffffff810ba5d6>] ? rcu_eqs_exit_common.isra.43+0x26/0xe0 [<ffffffff8113539a>] ? context_tracking_user_exit+0x5a/0x1a0 [<ffffffff810a693d>] ? trace_hardirqs_on_caller+0xfd/0x1c0 [<ffffffff81002b81>] do_notify_resume+0x71/0xc0 [<ffffffff814c1f98>] int_signal+0x12/0x17 blkid D ffff8807d275a230 0 8472 1 0x00000006 ffff8807d27bb9b8 0000000000000002 ffff8807d275a230 ffff8807d27bbfd8 00000000000131c0 00000000000131c0 ffff8807d1290000 ffff88085fc93ad0 ffff88085ff87838 ffff8807d27bba40 0000000000000002 ffffffff81135db0 Call Trace: [<ffffffff81135db0>] ? wait_on_page_read+0x60/0x60 [<ffffffff814bc0ed>] io_schedule+0x9d/0x130 [<ffffffff81135dbe>] sleep_on_page+0xe/0x20 [<ffffffff814bc738>] __wait_on_bit_lock+0x48/0xb0 [<ffffffff81135eca>] __lock_page+0x6a/0x70 [<ffffffff8109c8e0>] ? autoremove_wake_function+0x40/0x40 [<ffffffff811474cf>] truncate_inode_pages_range+0x3ff/0x690 [<ffffffff81147775>] truncate_inode_pages+0x15/0x20 [<ffffffff811d3425>] kill_bdev+0x35/0x40 [<ffffffff811d49a9>] __blkdev_put+0x69/0x1b0 [<ffffffff811d5450>] blkdev_put+0x50/0x160 [<ffffffff811d5615>] blkdev_close+0x25/0x30 [<ffffffff8119a37a>] __fput+0xea/0x1f0 [<ffffffff8119a4ce>] ____fput+0xe/0x10 [<ffffffff81074ddc>] task_work_run+0xac/0xe0 [<ffffffff8104ff67>] do_exit+0x2c7/0xc60 [<ffffffff81064091>] ? get_signal_to_deliver+0xd1/0x940 [<ffffffff814c118c>] ? _raw_spin_unlock_irq+0x2c/0x60 [<ffffffff81051cac>] do_group_exit+0x4c/0xc0 [<ffffffff810642a1>] get_signal_to_deliver+0x2e1/0x940 [<ffffffff81002528>] do_signal+0x48/0x630 [<ffffffff811d3f77>] ? blkdev_read_iter+0x37/0x40 [<ffffffff8119809e>] ? new_sync_read+0x7e/0xb0 [<ffffffff8127f0d7>] ? debug_smp_processor_id+0x17/0x20 [<ffffffff810ba5d6>] ? rcu_eqs_exit_common.isra.43+0x26/0xe0 [<ffffffff8113539a>] ? context_tracking_user_exit+0x5a/0x1a0 [<ffffffff810a693d>] ? trace_hardirqs_on_caller+0xfd/0x1c0 [<ffffffff81002b81>] do_notify_resume+0x71/0xc0 [<ffffffff814c1f98>] int_signal+0x12/0x17 blkid D ffff8807c6b10000 0 8482 1 0x00000006 ffff8807d4dbf9b8 0000000000000002 ffff8807c6b10000 ffff8807d4dbffd8 00000000000131c0 00000000000131c0 ffff88082df38000 ffff88085fd73ad0 ffff88085ff74638 ffff8807d4dbfa40 0000000000000002 ffffffff81135db0 Call Trace: [<ffffffff81135db0>] ? wait_on_page_read+0x60/0x60 [<ffffffff814bc0ed>] io_schedule+0x9d/0x130 [<ffffffff81135dbe>] sleep_on_page+0xe/0x20 [<ffffffff814bc738>] __wait_on_bit_lock+0x48/0xb0 [<ffffffff81135eca>] __lock_page+0x6a/0x70 [<ffffffff8109c8e0>] ? autoremove_wake_function+0x40/0x40 [<ffffffff811474cf>] truncate_inode_pages_range+0x3ff/0x690 [<ffffffff81147775>] truncate_inode_pages+0x15/0x20 [<ffffffff811d3425>] kill_bdev+0x35/0x40 [<ffffffff811d49a9>] __blkdev_put+0x69/0x1b0 [<ffffffff811d5450>] blkdev_put+0x50/0x160 [<ffffffff811d5615>] blkdev_close+0x25/0x30 [<ffffffff8119a37a>] __fput+0xea/0x1f0 [<ffffffff8119a4ce>] ____fput+0xe/0x10 [<ffffffff81074ddc>] task_work_run+0xac/0xe0 [<ffffffff8104ff67>] do_exit+0x2c7/0xc60 [<ffffffff81064091>] ? get_signal_to_deliver+0xd1/0x940 [<ffffffff814c118c>] ? _raw_spin_unlock_irq+0x2c/0x60 [<ffffffff81051cac>] do_group_exit+0x4c/0xc0 [<ffffffff810642a1>] get_signal_to_deliver+0x2e1/0x940 [<ffffffff81002528>] do_signal+0x48/0x630 [<ffffffff811d3f77>] ? blkdev_read_iter+0x37/0x40 [<ffffffff8119809e>] ? new_sync_read+0x7e/0xb0 [<ffffffff8127f0d7>] ? debug_smp_processor_id+0x17/0x20 [<ffffffff810ba5d6>] ? rcu_eqs_exit_common.isra.43+0x26/0xe0 [<ffffffff8113539a>] ? context_tracking_user_exit+0x5a/0x1a0 [<ffffffff810a693d>] ? trace_hardirqs_on_caller+0xfd/0x1c0 [<ffffffff81002b81>] do_notify_resume+0x71/0xc0 [<ffffffff814c1f98>] int_signal+0x12/0x17 blkid D ffff8807c6b88000 0 8549 1 0x00000006 ffff8807c6bb39b8 0000000000000002 ffff8807c6b88000 ffff8807c6bb3fd8 00000000000131c0 00000000000131c0 ffff8807d168c460 ffff88085fc33ad0 ffff88085ff88a38 ffff8807c6bb3a40 0000000000000002 ffffffff81135db0 Call Trace: [<ffffffff81135db0>] ? wait_on_page_read+0x60/0x60 [<ffffffff814bc0ed>] io_schedule+0x9d/0x130 [<ffffffff81135dbe>] sleep_on_page+0xe/0x20 [<ffffffff814bc738>] __wait_on_bit_lock+0x48/0xb0 [<ffffffff81135eca>] __lock_page+0x6a/0x70 [<ffffffff8109c8e0>] ? autoremove_wake_function+0x40/0x40 [<ffffffff811474cf>] truncate_inode_pages_range+0x3ff/0x690 [<ffffffff81147775>] truncate_inode_pages+0x15/0x20 [<ffffffff811d3425>] kill_bdev+0x35/0x40 [<ffffffff811d49a9>] __blkdev_put+0x69/0x1b0 [<ffffffff811d5450>] blkdev_put+0x50/0x160 [<ffffffff811d5615>] blkdev_close+0x25/0x30 [<ffffffff8119a37a>] __fput+0xea/0x1f0 [<ffffffff8119a4ce>] ____fput+0xe/0x10 [<ffffffff81074ddc>] task_work_run+0xac/0xe0 [<ffffffff8104ff67>] do_exit+0x2c7/0xc60 [<ffffffff81064091>] ? get_signal_to_deliver+0xd1/0x940 [<ffffffff814c118c>] ? _raw_spin_unlock_irq+0x2c/0x60 [<ffffffff81051cac>] do_group_exit+0x4c/0xc0 [<ffffffff810642a1>] get_signal_to_deliver+0x2e1/0x940 [<ffffffff81002528>] do_signal+0x48/0x630 [<ffffffff811d3f77>] ? blkdev_read_iter+0x37/0x40 [<ffffffff8119809e>] ? new_sync_read+0x7e/0xb0 [<ffffffff8127f0d7>] ? debug_smp_processor_id+0x17/0x20 [<ffffffff810ba5d6>] ? rcu_eqs_exit_common.isra.43+0x26/0xe0 [<ffffffff8113539a>] ? context_tracking_user_exit+0x5a/0x1a0 [<ffffffff810a693d>] ? trace_hardirqs_on_caller+0xfd/0x1c0 [<ffffffff81002b81>] do_notify_resume+0x71/0xc0 [<ffffffff814c1f98>] int_signal+0x12/0x17 blkid D ffff8807c7268000 0 8615 1 0x00000006 ffff8807d43b39b8 0000000000000002 ffff8807c7268000 ffff8807d43b3fd8 00000000000131c0 00000000000131c0 ffff88083b4a8000 ffff88085fcb3ad0 ffff88085ffb0038 ffff8807d43b3a40 0000000000000002 ffffffff81135db0 Call Trace: [<ffffffff81135db0>] ? wait_on_page_read+0x60/0x60 [<ffffffff814bc0ed>] io_schedule+0x9d/0x130 [<ffffffff81135dbe>] sleep_on_page+0xe/0x20 [<ffffffff814bc738>] __wait_on_bit_lock+0x48/0xb0 [<ffffffff81135eca>] __lock_page+0x6a/0x70 [<ffffffff8109c8e0>] ? autoremove_wake_function+0x40/0x40 [<ffffffff811474cf>] truncate_inode_pages_range+0x3ff/0x690 [<ffffffff81147775>] truncate_inode_pages+0x15/0x20 [<ffffffff811d3425>] kill_bdev+0x35/0x40 [<ffffffff811d49a9>] __blkdev_put+0x69/0x1b0 [<ffffffff811d5450>] blkdev_put+0x50/0x160 [<ffffffff811d5615>] blkdev_close+0x25/0x30 [<ffffffff8119a37a>] __fput+0xea/0x1f0 [<ffffffff8119a4ce>] ____fput+0xe/0x10 [<ffffffff81074ddc>] task_work_run+0xac/0xe0 [<ffffffff8104ff67>] do_exit+0x2c7/0xc60 [<ffffffff81064091>] ? get_signal_to_deliver+0xd1/0x940 [<ffffffff814c118c>] ? _raw_spin_unlock_irq+0x2c/0x60 [<ffffffff81051cac>] do_group_exit+0x4c/0xc0 [<ffffffff810642a1>] get_signal_to_deliver+0x2e1/0x940 [<ffffffff81002528>] do_signal+0x48/0x630 [<ffffffff811d3f77>] ? blkdev_read_iter+0x37/0x40 [<ffffffff8119809e>] ? new_sync_read+0x7e/0xb0 [<ffffffff8127f0d7>] ? debug_smp_processor_id+0x17/0x20 [<ffffffff810ba5d6>] ? rcu_eqs_exit_common.isra.43+0x26/0xe0 [<ffffffff8113539a>] ? context_tracking_user_exit+0x5a/0x1a0 [<ffffffff810a693d>] ? trace_hardirqs_on_caller+0xfd/0x1c0 [<ffffffff81002b81>] do_notify_resume+0x71/0xc0 [<ffffffff814c1f98>] int_signal+0x12/0x17
-- dm-devel mailing list dm-devel@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/dm-devel
