Vivek Goyal wrote:
> On Wed, Mar 13, 2013 at 11:46:29AM +0400, Konstantin Khlebnikov wrote:
>
> [..]
>>> Ok, some more observation.
>>>
>>> - Problem seems to be in during shutdown path. Because older kernel 3.8
>>> can kexec into newer kernel 3.9.rc1 but not vice-a-versa.
>>>
>>> I did git bisecting and following commit seems to be problem.
>>>
>>> commit 7897e6022761ace7377f0f784fca059da55f5d71
>>> Author: Konstantin Khlebnikov<khlebnikov at openvz.org>
>>> Date: Mon Feb 4 15:55:58 2013 +0400
>>>
>>> PCI: Disable Bus Master unconditionally in pci_device_shutdown()
>>>
>>> Commit b566a22c23 ("PCI: disable Bus Master on PCI device shutdown")
>>> used pci_disable_device(), but that doesn't disable Bus Mastering
>>> unconditionally; we allow nested enable/disable calls, and only the
>>> last disable call actually does anything.
>>>
>>> This uses pci_clear_master() to unconditionally clear the Bus Master
>>> bit.
>>>
>>> Matthew Garrett and Alan Cox said (see LKML link below) that clearing
>>> Bus
>>> Master for all PCI devices may lead to unpredictable consequences:
>>> some
>>> devices ignores this bit and continue DMA, some of them hang after
>>> that or
>>> crash the whole system. But we're already trying to clear Bus Master
>>> in
>>> general because of b566a22c23; this merely deals with the cases where
>>> drivers haven't shut down the device correctly.
>>>
>>> [bhelgaas: changelog]
>>> Link: https://lkml.org/lkml/2012/6/6/278
>>> Signed-off-by: Konstantin Khlebnikov<khlebnikov at openvz.org>
>>> Signed-off-by: Bjorn Helgaas<bhelgaas at google.com>
>>> Acked-by: Rafael J. Wysocki<rafael.j.wysocki at intel.com>
>>>
>>> I reverted above commit and things work again. Just that I get following
>>> warning during shutdown.
>>>
>>> [ 54.252516] ------------[ cut here ]------------
>>> [ 54.257199] WARNING: at drivers/pci/pci.c:1397
>>> pci_disable_device+0x90/0xa0()
>>> [ 54.264387] Hardware name: HP xw6600 Workstation
>>> [ 54.269061] Device pci
>>> disabling already-disabled device
>>> [ 54.274341] Modules linked in: floppy
>>> [ 54.278403] Pid: 5272, comm: kexec Not tainted 3.9.0-rc2+ #207
>>> [ 54.284289] Call Trace:
>>> [ 54.286801] [<ffffffff8133c600>] ? pci_disable_device+0x60/0xa0
>>> [ 54.292864] [<ffffffff8103e49f>] warn_slowpath_common+0x7f/0xc0
>>> [ 54.298926] [<ffffffff8103e596>] warn_slowpath_fmt+0x46/0x50
>>> [ 54.304727] [<ffffffff8133c592>] ? do_pci_disable_device+0x52/0x60
>>> [ 54.311050] [<ffffffff8133c630>] pci_disable_device+0x90/0xa0
>>> [ 54.316938] [<ffffffff8133e1a4>] pci_device_shutdown+0x44/0x50
>>> [ 54.322915] [<ffffffff81462b2d>] device_shutdown+0x1d/0x180
>>> [ 54.328631] [<ffffffff81056ba6>] kernel_restart_prepare+0x36/0x50
>>> [ 54.334866] [<ffffffff810a16c0>] kernel_kexec+0x50/0x80
>>> [ 54.340235] [<ffffffff81056e35>] sys_reboot+0x1f5/0x260
>>> [ 54.345604] [<ffffffff811621b9>] ? mntput_no_expire+0x49/0x160
>>> [ 54.351578] [<ffffffff811622f6>] ? mntput+0x26/0x40
>>> [ 54.356601] [<ffffffff81144539>] ? __fput+0x1a9/0x280
>>> [ 54.361798] [<ffffffff8105fae4>] ? task_work_run+0xc4/0xe0
>>> [ 54.367428] [<ffffffff810029a5>] ? do_notify_resume+0x75/0x80
>>> [ 54.373319] [<ffffffff81882742>] system_call_fastpath+0x16/0x1b
>>> [ 54.379382] ---[ end trace ea6ecbf97debf2e2 ]---
>>> [ 54.385157] Starting new kernel
>>>
>>>
>>> I am leaving the logs from previous mail intact so that newly CCed
>>> people can have a look at it and don't go hunting for old mail in
>>> lkml archives.
>>>
>>> Thanks
>>> Vivek
>>>
>>
>> Look like I fixed one bug and added another.
>> After ->shutdown() device can be in D3-cold state and config space is unreachable.
>>
>> try this patch
>>
>> --- a/drivers/pci/pci-driver.c
>> +++ b/drivers/pci/pci-driver.c
>> @@ -385,6 +385,12 @@ static void pci_device_shutdown(struct device *dev)
>>
>> if (drv&& drv->shutdown)
>> drv->shutdown(pci_dev);
>> +
>> + if (pci_dev->current_state == PCI_D3cold) {
>> + WARN_ON(pci_dev->msi_enabled || pci_dev->msix_enabled);
>> + return;
>> + }
>> +
>> pci_msi_shutdown(pci_dev);
>> pci_msix_shutdown(pci_dev);
>>
>>
>
> Hi,
>
> So this patch is supposed to fix the warning? This warning showed up
> only after reverting your patch. So do you agree that your original
> patch should be reverted?
Please see more accurate patch in attachment.
My patch for pci_device_shutdown() just uncovers another problem.
Also that warning actually also mine. =)
Or probably you found hardware where bus-master bit cannot be cleared.
Matthew Garrett and Alan Cox already warned us about such hardware.
(see your citation of my comment above)
My patch (which you want to revert) just fixes bug in potentially broken hack.
After that this hack actually starts working.
>
> I applied this patch and warning is still there (After reverting your
> original patch).
>
> I thought we would first address the issue of why kexec is not working
> with your patch.
>
> Thanks
> Vivek
>
> [ 38.048452] tg3 0000:0e:00.0: System wakeup enabled by ACPI
> [ 38.266774] sd 5:0:0:0: [sdd] Synchronizing SCSI cache
> [ 38.272116] sd 3:0:0:0: [sdc] Synchronizing SCSI cache
> [ 38.277361] sd 2:0:0:0: [sdb] Synchronizing SCSI cache
> [ 38.282661] sd 0:0:0:0: [sda] Synchronizing SCSI cache
> [ 38.288467] ------------[ cut here ]------------
> [ 38.293151] WARNING: at drivers/pci/pci.c:1397
> pci_disable_device+0x90/0xa0()
> [ 38.300339] Hardware name: HP xw6600 Workstation
> [ 38.305014] Device pci
> disabling already-disabled device
> [ 38.310294] Modules linked in: floppy
> [ 38.314356] Pid: 5258, comm: kexec Not tainted 3.9.0-rc2+ #209
> [ 38.320243] Call Trace:
> [ 38.322755] [<ffffffff8133c600>] ? pci_disable_device+0x60/0xa0
> [ 38.328818] [<ffffffff8103e49f>] warn_slowpath_common+0x7f/0xc0
> [ 38.334880] [<ffffffff8103e596>] warn_slowpath_fmt+0x46/0x50
> [ 38.340681] [<ffffffff8133c592>] ? do_pci_disable_device+0x52/0x60
> [ 38.347003] [<ffffffff8133c630>] pci_disable_device+0x90/0xa0
> [ 38.352892] [<ffffffff8133f2d4>] pci_device_shutdown+0x54/0x80
> [ 38.358868] [<ffffffff81462b5d>] device_shutdown+0x1d/0x180
> [ 38.364584] [<ffffffff81056ba6>] kernel_restart_prepare+0x36/0x50
> [ 38.370820] [<ffffffff810a16c0>] kernel_kexec+0x50/0x80
> [ 38.376188] [<ffffffff81056e35>] sys_reboot+0x1f5/0x260
> [ 38.381558] [<ffffffff811621b9>] ? mntput_no_expire+0x49/0x160
> [ 38.387532] [<ffffffff811622f6>] ? mntput+0x26/0x40
> [ 38.392555] [<ffffffff81144539>] ? __fput+0x1a9/0x280
> [ 38.397753] [<ffffffff8187a0ee>] ? _raw_spin_unlock_irq+0xe/0x30
> [ 38.403901] [<ffffffff8105fae4>] ? task_work_run+0xc4/0xe0
> [ 38.409531] [<ffffffff810029a5>] ? do_notify_resume+0x75/0x80
> [ 38.415420] [<ffffffff81882742>] system_call_fastpath+0x16/0x1b
> [ 38.421479] ---[ end trace 61d35d2d55ce5d3d ]---
> [ 38.427241] Starting new kernel
> [ 0.000000] Initializing cgroup subsys cpuset
> [ 0.000000] Initializing cgroup subsys cpu
>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: pci-fix-access-to-disconnected-devices-in-pci_device_shutdown
URL: <http://lists.infradead.org/pipermail/kexec/attachments/20130313/9c2dcff5/attachment-0001.ksh>
