On Mon, 2018-07-02 at 08:53 +0000, Jiaojianbing wrote: > > Please describe your setup bottom-up. You have two scripts running > > periodically, > > one calling "rescan-scsi-bus.sh" and one "multipath", and they are > > (can be) > > running at the same time? How frequently are they running? Be aware > > that > > "multipath" is not a monitoring command, it basically causes a > > reconfiguration. > > It's not recommended to run it periodically. Of course running > > "multipath" > > shouldn't cause a system hang, but in your case I still think the > > root problem is > > that devices that can't respond to IO are seen in "running" state > > by the kernel. > > If that happens, other processes are allowed, actually supposed, to > > do probing > > on these devices. But it's hard to say more without knowing what > > exactly is > > going on. > > > > Also, please consider updating to more recent version of the tools. > > dm- devel a > > mailing list for discussing upstream issues, and your versions of > > both multipath > > and sg3_utils are rather ancient. I guess you're using some older > > distribution, > > in which case you may want to engage with your distro's support > > team. > > > > I don't think well make much progress without detailed logs of both > > the kernel > > (please activate scsi logging with MLCOMPLETE=1|ERROR=4|SCAN_BUS=4, > > run rescan_scsi_bus.sh with -d switch, and set multipath verbosity > > of both > > multipathd and "multipath" command to 3 at least, and put results > > on a > > pastebin somewhere, and provide us with links. > > > > Dear martin and Doug Gilbert, > Thanks for your return. I descript all the scene and question > below, and 6th question is the import thing I want to know. > > 1、First, my scene is, there are 500 LUN in remote IPSAN, and several > hosts connect to the IPSAN. Each host has found 500 dm devices. > I want to test the time for one host to rescan when we remove the 500 > LUNs map In remote IPSAN. That is the time to measure > for the host from 500 dm devices to zero dm devices by “rescan-scsi- > bus.sh” script. > > 2、There are two scripts running at the same time，one is “rescan-scsi- > bus.sh”, who just run once ,but take large time > (about serval hours) to run to complete when 500 LUNs are removed in > remote IPSAN. The other script we called "test.sh", > who call "multipath" every five minutes as below: > > #! /bin/bash > > while : > do > multipath > sleep 300 > done > > 3、 if we run two scripts at the same time, “rescan-scsi-bus.sh” will > take more larger time than running itself without "test.sh". > And after complete of “rescan-scsi-bus.sh” , serval processes of > "systemd-udevd" will be in D state as below. And D state of these > processes > can't be recovered automatically. And also there will be leftover of > multipath. > > # ps aux |grep D > root 2635 0.0 0.0 0 0 ? D Jul01 0:00 > [systemd-udevd] > root 2682 0.0 0.0 0 0 ? D Jul01 0:00 > [systemd-udevd] > root 2693 0.0 0.0 0 0 ? D Jul01 0:00 > [systemd-udevd] > root 3069 0.0 0.0 0 0 ? D Jul01 0:00 > [systemd-udevd] > root 3080 0.0 0.0 0 0 ? D Jul01 0:00 > [systemd-udevd] > root 3517 0.0 0.0 0 0 ? D Jul01 0:00 > [systemd-udevd] > root 3659 0.0 0.0 0 0 ? D Jul01 0:00 > [systemd-udevd] > > and one of D process's stack: > # cat /proc/3659 /stack > [<ffffffff8118e784>] __lock_page+0x74/0x90 > [<ffffffff811a0564>] truncate_inode_pages_range+0x704/0x740 > [<ffffffff811a05b5>] truncate_inode_pages+0x15/0x20 > [<ffffffff812519af>] kill_bdev+0x2f/0x40 > [<ffffffff81253464>] __blkdev_put+0x64/0x1a0 > [<ffffffff81253eee>] blkdev_put+0x4e/0x140 > [<ffffffff81254095>] blkdev_close+0x25/0x30 > [<ffffffff8121796c>] __fput+0xec/0x260 > [<ffffffff81217c1e>] ____fput+0xe/0x10 > [<ffffffff810b6867>] task_work_run+0xc7/0xe0 > [<ffffffff81095010>] do_exit+0x2e0/0xa50 > [<ffffffff810957ff>] do_group_exit+0x3f/0xa0 > [<ffffffff810a6d60>] get_signal_to_deliver+0x1d0/0x6e0 > [<ffffffff8102a527>] do_signal+0x57/0x6b0 > [<ffffffff8102abdf>] do_notify_resume+0x5f/0xb0 > [<ffffffff816c264f>] int_signal+0x12/0x17 > [<ffffffffffffffff>] 0xffffffffffffffff Hmm, that stack reminds me of two situations Bart has reported earlier: https://patchwork.kernel.org/patch/9261609/ https://patchwork.kernel.org/patch/9202331/ It doesn't seem to me that there has been agreement on Bart's proposed fixes so far, but I may have missed it. Maybe Bart can comment on that. It would be interesting to see full alt-sysrq-w output of that system. What kernel have you been using? > > Leftover (2 for sample, more than 2): > 36e02861100592fcd0e1d8024000000ce dm-385 , > size=6.0G features='1 queue_if_no_path' hwhandler='0' wp=rw > `-+- policy='service-time 0' prio=0 status=enabled > |- #:#:#:# - #:# active faulty running > |- #:#:#:# - #:# active faulty running > |- #:#:#:# - #:# active faulty running > `- #:#:#:# - #:# active faulty running > 36e02861100592fcd0e1d586400000069 dm-273 , > size=1.0G features='1 queue_if_no_path' hwhandler='0' wp=rw > `-+- policy='service-time 0' prio=0 status=active > |- #:#:#:# - #:# failed faulty running > |- #:#:#:# - #:# failed faulty running > |- #:#:#:# - #:# failed faulty running > `- #:#:#:# - #:# failed faulty running > > 4、if we run only "rescan-scsi-bus.sh " without "test.sh" script, > there is no problem. And the time for " rescan-scsi-bus.sh " is more > less. > > 5、I'll check the last version according to you and Doug Gilbert's > advice. And collect log. > > 6、Before we do 5th step above, one conclusion maybe made that > "multipath" called periodically in other script can't be ran in > process of " rescan-scsi-bus.sh ", is that true? > I think , while " rescan-scsi-bus.sh " remove the paths and change > to failed state, "multipath" reinstate paths' state . These two > actions confused the state of paths ? I said already, I don't see the reason why multipathd reinstates paths which are not UP, and I'm waiting for some of the people who have been around longer than myself to comment. Yet in your case, the path state is UP (aka "running"). At least this seems to be the case temporarily, while rescan-scsi-bus.sh is running. There's nothing wrong with multipathd trying to reinstate a path in UP state, AFAICT, so we need to look elswhere for the root cause. The above kernel stack is a high suspect. Martin -- Dr. Martin Wilck <mwilck@xxxxxxxx>, Tel. +49 (0)911 74053 2107 SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG Nürnberg) -- dm-devel mailing list dm-devel@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/dm-devel
