> -----Original Message----- > From: Thomas Gleixner [mailto:tglx@xxxxxxxxxxxxx] > Sent: Tuesday, October 17, 2017 1:57 AM > To: YASUAKI ISHIMATSU > Cc: Kashyap Desai; Hannes Reinecke; Marc Zyngier; Christoph Hellwig; > axboe@xxxxxxxxx; mpe@xxxxxxxxxxxxxx; keith.busch@xxxxxxxxx; > peterz@xxxxxxxxxxxxx; LKML; linux-scsi@xxxxxxxxxxxxxxx; Sumit Saxena; > Shivasharan Srikanteshwara > Subject: Re: system hung up when offlining CPUs > > Yasuaki, > > On Mon, 16 Oct 2017, YASUAKI ISHIMATSU wrote: > > > Hi Thomas, > > > > > Can you please apply the patch below on top of Linus tree and retest? > > > > > > Please send me the outputs I asked you to provide last time in any > > > case (success or fail). > > > > The issue still occurs even if I applied your patch to linux 4.14.0-rc4. > > Thanks for testing. > > > --- > > [ ...] INFO: task setroubleshootd:4972 blocked for more than 120 seconds. > > [ ...] Not tainted 4.14.0-rc4.thomas.with.irqdebug+ #6 > > [ ...] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this > message. > > [ ...] setroubleshootd D 0 4972 1 0x00000080 > > [ ...] Call Trace: > > [ ...] __schedule+0x28d/0x890 > > [ ...] ? release_pages+0x16f/0x3f0 > > [ ...] schedule+0x36/0x80 > > [ ...] io_schedule+0x16/0x40 > > [ ...] wait_on_page_bit+0x107/0x150 > > [ ...] ? page_cache_tree_insert+0xb0/0xb0 [ ...] > > truncate_inode_pages_range+0x3dd/0x7d0 > > [ ...] ? schedule_hrtimeout_range_clock+0xad/0x140 > > [ ...] ? remove_wait_queue+0x59/0x60 > > [ ...] ? down_write+0x12/0x40 > > [ ...] ? unmap_mapping_range+0x75/0x130 [ ...] > > truncate_pagecache+0x47/0x60 [ ...] truncate_setsize+0x32/0x40 [ ...] > > xfs_setattr_size+0x100/0x300 [xfs] [ ...] > > xfs_vn_setattr_size+0x40/0x90 [xfs] [ ...] xfs_vn_setattr+0x87/0xa0 > > [xfs] [ ...] notify_change+0x266/0x440 [ ...] do_truncate+0x75/0xc0 > > [ ...] path_openat+0xaba/0x13b0 [ ...] ? > > mem_cgroup_commit_charge+0x31/0x130 > > [ ...] do_filp_open+0x91/0x100 > > [ ...] ? __alloc_fd+0x46/0x170 > > [ ...] do_sys_open+0x124/0x210 > > [ ...] SyS_open+0x1e/0x20 > > [ ...] do_syscall_64+0x67/0x1b0 > > [ ...] entry_SYSCALL64_slow_path+0x25/0x25 > > This is definitely a driver issue. The driver requests an affinity managed > interrupt. Affinity managed interrupts are different from non managed > interrupts in several ways: > > Non-Managed interrupts: > > 1) At setup time the default interrupt affinity is assigned to each > interrupt. The effective affinity is usually a subset of the online > CPUs. > > 2) User space can modify the affinity of the interrupt > > 3) If a CPU in the affinity mask goes offline and there are still online > CPUs in the affinity mask then the effective affinity is moved to a > subset of the online CPUs in the affinity mask. > > If the last CPU in the affinity mask of an interrupt goes offline then > the hotplug code breaks the affinity and makes it affine to the online > CPUs. The effective affinity is a subset of the new affinity setting, > > Managed interrupts: > > 1) At setup time the interrupts of a multiqueue device are evenly spread > over the possible CPUs. If all CPUs in the affinity mask of a given > interrupt are offline at request_irq() time, the interrupt stays shut > down. If the first CPU in the affinity mask comes online later the > interrupt is started up. > > 2) User space cannot modify the affinity of the interrupt > > 3) If a CPU in the affinity mask goes offline and there are still online > CPUs in the affinity mask then the effective affinity is moved a subset > of the online CPUs in the affinity mask. I.e. the same as with > Non-Managed interrupts. > > If the last CPU in the affinity mask of a managed interrupt goes > offline then the interrupt is shutdown. If the first CPU in the > affinity mask becomes online again then the interrupt is started up > again. > Hi Thomas, Thanks for the detailed explanation about the behavior of managed interrupts. This helped me to understand the issue better. This is first time I am checking CPU hotplug system, so my input is very preliminary. Please bear with my understanding and correct me where required. This issue is reproducible on our local setup as well, with managed interrupts. I have few queries on the requirements for device driver that you have mentioned. In managed-interrupts case, interrupts which were affine to the offlined CPU is not getting migrated to another available CPU. But the documentation at below link says that "all interrupts" are migrated to a new CPU. So not all interrupts are getting migrated to a new CPU then. https://www.kernel.org/doc/html/v4.11/core-api/cpu_hotplug.html#the-offlin e-case "- All interrupts targeted to this CPU are migrated to a new CPU" > So this has consequences: > > 1) The device driver has to make sure that no requests are targeted at a > queue whose interrupt is affine to offline CPUs and therefor shut > down. If the driver ignores that then this queue will not deliver an > interrupt simply because that interrupt is shut down. > > 2) When the last CPU in the affinity mask of a queue interrupt goes > offline the device driver has to make sure, that all outstanding > requests in the queue which have not yet delivered their interrupt are > completed. This is required because when the CPU is finally offline the > interrupt is shut down and wont deliver any more interrupts. > > If that does not happen then the not yet completed request will try to > send the completion interrupt which obviously gets not delivered > because it is shut down. > Once the last CPU in the affinity mask is offlined and a particular IRQ is shutdown, is there a way currently for the device driver to get callback to complete all outstanding requests on that queue? >From the ftrace that I captured, below were the various functions being called once the irq shutdown was initiated. There were no callbacks being called from the irq core that I could see. <...>-16 [001] d..1 9915.744040: irq_shutdown <-irq_migrate_all_off_this_cpu ^^^^^^^^^^^ <...>-16 [001] d..1 9915.744040: __irq_disable <-irq_shutdown <...>-16 [001] d..1 9915.744041: mask_irq.part.30 <-__irq_disable <...>-16 [001] d..1 9915.744041: pci_msi_mask_irq <-mask_irq.part.30 <...>-16 [001] d..1 9915.744041: msi_set_mask_bit <-pci_msi_mask_irq <...>-16 [001] d..1 9915.744042: irq_domain_deactivate_irq <-irq_shutdown <...>-16 [001] d..1 9915.744043: __irq_domain_deactivate_irq <-irq_domain_deactivate_irq <...>-16 [001] d..1 9915.744043: msi_domain_deactivate <-__irq_domain_deactivate_irq <...>-16 [001] d..1 9915.744044: pci_msi_domain_write_msg <-msi_domain_deactivate <...>-16 [001] d..1 9915.744044: __pci_write_msi_msg <-pci_msi_domain_write_msg <...>-16 [001] d..1 9915.744044: __irq_domain_deactivate_irq <-__irq_domain_deactivate_irq <...>-16 [001] d..1 9915.744045: intel_irq_remapping_deactivate <-__irq_domain_deactivate_irq <...>-16 [001] d..1 9915.744045: modify_irte <-intel_irq_remapping_deactivate <...>-16 [001] d..1 9915.744045: _raw_spin_lock_irqsave <-modify_irte <...>-16 [001] d..1 9915.744045: qi_submit_sync <-modify_irte <...>-16 [001] d..1 9915.744046: _raw_spin_lock_irqsave <-qi_submit_sync <...>-16 [001] d..1 9915.744046: _raw_spin_lock <-qi_submit_sync <...>-16 [001] d..1 9915.744047: _raw_spin_unlock_irqrestore <-qi_submit_sync <...>-16 [001] d..1 9915.744047: _raw_spin_unlock_irqrestore <-modify_irte <...>-16 [001] d..1 9915.744047: __irq_domain_deactivate_irq <-__irq_domain_deactivate_irq In my knowledge many device drivers in the kernel tree pass PCI_IRQ_AFFINITY flag to pci_alloc_irq_vectors and it is widely used feature (not limited to megaraid_sas driver). But I could not see any of the drivers working as per the constraints mentioned. Can you please point me to any existing driver to understand how above constraints can be implemented? Below is simple grep I ran on drivers passing PCI_IRQ_AFFINITY flag to pci_alloc_irq_vectors. # grep -R "PCI_IRQ_AFFINITY" drivers/* drivers/nvme/host/pci.c: PCI_IRQ_ALL_TYPES | PCI_IRQ_AFFINITY); drivers/pci/host/vmd.c: PCI_IRQ_MSIX | PCI_IRQ_AFFINITY); drivers/pci/msi.c: if (flags & PCI_IRQ_AFFINITY) { drivers/scsi/aacraid/comminit.c: PCI_IRQ_MSIX | PCI_IRQ_AFFINITY); drivers/scsi/be2iscsi/be_main.c: PCI_IRQ_MSIX | PCI_IRQ_AFFINITY, &desc) < 0) { drivers/scsi/csiostor/csio_isr.c: PCI_IRQ_MSIX | PCI_IRQ_AFFINITY, &desc); drivers/scsi/hpsa.c: PCI_IRQ_MSIX | PCI_IRQ_AFFINITY); drivers/scsi/lpfc/lpfc_init.c: vectors, PCI_IRQ_MSIX | PCI_IRQ_AFFINITY); drivers/scsi/megaraid/megaraid_sas_base.c: irq_flags |= PCI_IRQ_AFFINITY; drivers/scsi/megaraid/megaraid_sas_base.c: irq_flags |= PCI_IRQ_AFFINITY; drivers/scsi/mpt3sas/mpt3sas_base.c: irq_flags |= PCI_IRQ_AFFINITY; drivers/scsi/qla2xxx/qla_isr.c: ha->msix_count, PCI_IRQ_MSIX | PCI_IRQ_AFFINITY, drivers/scsi/smartpqi/smartpqi_init.c: PCI_IRQ_MSIX | PCI_IRQ_AFFINITY); drivers/virtio/virtio_pci_common.c: (desc ? PCI_IRQ_AFFINITY : 0), Thanks, Shivasharan > It's hard to tell from the debug information which of the constraints (#1 or #2 > or both) has been violated by the driver (or the device hardware / > firmware) but the effect that the task which submitted the I/O operation is > hung after an offline operation points clearly into that direction. > > The irq core code is doing what is expected and I have no clue about that > megasas driver/hardware so I have to punt and redirect you to the SCSI and > megasas people. > > Thanks, > > tglx > >
