dmesg: http://bugzilla.kernel.org/attachment.cgi?id=20895 lspci -vt: http://bugzilla.kernel.org/attachment.cgi?id=20896 lspci -v: http://bugzilla.kernel.org/attachment.cgi?id=20897 2009/4/8 Grant Grundler <grundler@xxxxxxxxxx>: > 2009/4/7 Данила Жукоцкий <optimusgd@xxxxxxxxx>: >> Dammit BUG is still here! >> >> Sorry guys, bug is STILL here, but it hard to trigger. After eight >> night hours network and 3ware raid activity (sata drive umounted, usb >> hdd unplugged) my dmesg full of "dma_map_sg overflow" messages. >> Computer work all-night without any iommu options in boot string, but >> DMA still leaking, as i see. > > Possibly, but not necessarily a leak though. As you've noticed, the IOMMU > mappings is a limited resource and we can run out if too many IOs are > in flight. I *think* a 64MB apperture means we can have 64MB of IO > mapped at the same time - assuming we have 100% efficiency and > no fragmentation or other forms of "unusable" space. > >> Anybody, who understand how IOMMU driver >> work, can say me how i can set very small apperture size, about 4mb or >> something, > > You can 't reduce the IOMMU mapping space to smaller than the > amount of IO can possibly be in flight. A starting point to compute this: > 1) count the number of block devices, the max size of each IO, and the > queue depth. > 2) plus number of NICs * RX depth * TX depth * 4k (page size) > > I'm ignoring pci_consistent (ie control data) data allocations. Those can be > determined by looking at idle system after reboot. Don't know where but > I had added debug code to parisc IOMMU code to dump that sort of info in > the past. > > I'm also ignore 64-bit capable devices completely bypassing the IOMMU > and not consuming any resources at all. > >> because with default 64mb apperture i get leak only after >> couple of hours, so leak is small. > > For a system with a RAID device and more than 1GB of RAM, it's not that > hard to get more than 64MB of IO in flight. Just depends on the IO > controllers. > > If it's a leak, the only way to track it is to add debug code that tracks > the IP of the caller along with each mapping. Enabling such debugging > code will substantially change the timing of DMA map/unmap calls. > >> And i again not understand source >> of leaking, it may be network, sata, usb, etc (i don't reboot computer >> after yesterday's IO tests to from usb\sata). So how i can painless >> catch the leak source without disabling subsystems one by one and >> hours-long tests? It is workstation, and i must do my job all-day with >> it. > > First off, you can eliminate 64-bit devices/drivers. Yinghai Lu submitted > a patch to print the DMA mask: > http://lkml.org/lkml/2008/10/8/274 > > Add that patch, rebuild the kernel and post the dmesg output would > be a starting point. Posting "lspci -vt" and "lspci -v" would also help. > > FYI: PCI drivers get 32-bit dma_mask by default until the driver calls > pci_set_dma_mask(). So inspecting drivers will tell you if they are > actually using the IOMMU (or not if they set dma_mask to 64-bit). > > hth, > grant > >> 2009/4/8 Grant Grundler <grundler@xxxxxxxxxx>: >>> Anyone understand why allowdac exists or if it's buggy? >>> >>> thanks, >>> grant >>> >>> >>> ---------- Forwarded message ---------- >>> From: Grant Grundler <grundler@xxxxxxxxxx> >>> Date: 2009/4/7 >>> Subject: Re: [Bugme-new] [Bug 13001] New: PCI-DMA: Out of IOMMU space >>> To: Данила Жукоцкий <optimusgd@xxxxxxxxx> >>> Cc: akpm@xxxxxxxxxxxxxxxxxxxx, linux-ide@xxxxxxxxxxxxxxx, >>> bugme-daemon@xxxxxxxxxxxxxxxxxxx, x86@xxxxxxxxxx >>> >>> >>> 2009/4/7 Данила Жукоцкий <optimusgd@xxxxxxxxx> >>>> >>>> Thank You, Grant, for Your simple questions! >>> >>> Welcome! >>> I was just trying to point out some additional information to folks >>> who might understand the code. I don't in this case but was looking >>> for more clues. >>> >>>> >>>> Without "allowdac" after >>>> couple of hours testing i cannot reproduce bug! So it is my stupid >>>> mistake, i don't understand Why i add this absolutely unusual >>>> parameter to boot string. >>> >>> I thought I understood IOMMUs having written code for 4 different >>> implementations. >>> I don't understand why "allowdac" parameter exists. >>> dma_mask stuff should be handling this already. >>> Can someone explain *why* (Данила already posted the docs) this >>> parameter exists? >>> (ie use case that dma_mask APIs don't work.) >>> >>>> I'm apologize to All off You for this stupid >>>> mindfuck. You may close the bug, because it is bug in my damn head. >>> >>> My first impression: the bug is either allocdma exists instead of using >>> DMA API (See Documentation/DMA-API.txt) OR the documentation for >>> allocdma is missing warnings about "this could break your system" >>> and clearly specify when it should be used. >>> >>> hth, >>> grant >>> >>>> >>>> 2009/4/7 Данила Жукоцкий <optimusgd@xxxxxxxxx>: >>>> > Forgot reply to all, sorry >>>> > >>>> > ---------- Forwarded message ---------- >>>> > From: Данила Жукоцкий <optimusgd@xxxxxxxxx> >>>> > Date: 2009/4/7 >>>> > Subject: Re: [Bugme-new] [Bug 13001] New: PCI-DMA: Out of IOMMU space >>>> > To: Grant Grundler <grundler@xxxxxxxxxx> >>>> > >>>> > >>>> > Yes, in attachment clear dmesg, warnings in bugreport body >>>> > >>>> >>Apr 3 13:38:46 rngmhpamd sata_nv 0000:00:05.0: PCI-DMA: Out of IOMMU space for 4096 bytes >>>> >>Apr 3 13:38:46 rngmhpamd sata_nv 0000:00:05.0: PCI-DMA: Out of IOMMU space for 4096 bytes >>>> > >>>> > I got "PCI-DMA: Out of IOMMU space" while trying write data to usb or >>>> > sata hdd before all other error messages. After that usb and sata >>>> > drives lost. All other noise is attempts to communicate with died >>>> > devices. >>>> > >>>> >>Kernel command line: >>>> >>mce=bootlog root=/dev/ram0 real_root=/dev/evms/root init=/linuxrc >>>> > I'm boot from 3ware raid with evmc >>>> >>iommu=allowdac,merge,memaper=3 >>>> > This is from Documentation/x86/x86_64/boot-options.txt >>>> > iommu=allowdac Im try to avoid DMA bug. May be that not need. >>>> > allowdac Allow double-address cycle (DAC) mode, i.e. DMA >4GB. >>>> > DAC is used with 32-bit PCI to push a 64-bit address in >>>> > two cycles. When off all DMA over >4GB is forced through >>>> > an IOMMU or software bounce buffering. >>>> > merge Do scatter-gather (SG) merging. >>>> > memaper[=<order>] Allocate an own aperture over RAM with size 32MB<<order. >>>> > (default: order=1, i.e. 64MB) >>>> > With default apperture, 64mb, DMA leak very fast, now i have >>>> > memaper=5, 1 gb, becouse i must do my job and can't rollback to 2.6.28 >>>> > due strange mysterious problem with forcedeth nics that i can't >>>> > explain and solve. If solution for DMA leak will not be found, i'm try >>>> > to fill bugreport about problem with nics. >>>> > >>>> >>3w_9xxx.use_msi=1 snd-hda-intel.enable_msi=1 doevms quiet >>>> > I prefer use msi on that hardware. >>>> >>... >>>> >>Your BIOS doesn't leave a aperture memory hole >>>> >>Please enable the IOMMU option in the BIOS setup >>>> >>This costs you 256 MB of RAM >>>> > >>>> > xw9400 BIOS do not have IOMMU option in the BIOS setup. Now this costs >>>> > me 1gb of ram >>>> > >>>> > Anyway, i can stable reproduce bug without all this whistlers >>>> > >>>> > 2009/4/7 Grant Grundler <grundler@xxxxxxxxxx>: >>>> >> 2009/4/5 Данила Жукоцкий <optimusgd@xxxxxxxxx>: >>>> >>> 2009/4/4 Andrew Morton <akpm@xxxxxxxxxxxxxxxxxxxx>: >>>> >>>> >>>> >>>> (switched to email. Please respond via emailed reply-to-all, not via the >>>> >>>> bugzilla web interface). >>>> >>>> >>>> >>>> On Fri, 3 Apr 2009 09:30:19 GMT bugzilla-daemon@xxxxxxxxxxxxxxxxxxx wrote: >>>> >>>> >>>> >>>>> http://bugzilla.kernel.org/show_bug.cgi?id=13001 >>>> >>>>> >>>> >>>>> Summary: PCI-DMA: Out of IOMMU space >>>> >>>>> Product: Platform Specific/Hardware >>>> >>>>> Version: 2.5 >>>> >>>>> Kernel Version: 2.6.29-gentoo >>>> >>>>> Platform: All >>>> >>>>> OS/Version: Linux >>>> >>>>> Tree: Mainline >>>> >>>>> Status: NEW >>>> >>>>> Severity: normal >>>> >>>>> Priority: P1 >>>> >>>>> Component: x86-64 >>>> >>>>> AssignedTo: platform_x86_64@xxxxxxxxxxxxxxxxxxxx >>>> >>>>> ReportedBy: optimusgd@xxxxxxxxx >>>> >>>>> Regression: Yes >>>> >>>>> >>>> >>>>> >>>> >>>>> Created an attachment (id=20789) >>>> >>>>> --> (http://bugzilla.kernel.org/attachment.cgi?id=20789) >>>> >>>>> hwreport generated info >>>> >>>>> >>>> >>>>> After some IO activity the "PCI-DMA: Out of IOMMU space" message appear. >>>> >>>>> 2.6.28-gentoo-r4 work ok, so it is regression. >>>> >>>> >>>> >>>> It is indeed a regression. >>>> >>>> >>>> >>>>> Dmesg fragments: >>>> >>>>> >>>> >>>>> >>>> >>>>> Apr 3 13:38:46 rngmhpamd sata_nv 0000:00:05.0: PCI-DMA: Out of IOMMU space for >>>> >>>>> 4096 bytes >>>> >> >>>> >> The bug report has a "dmesg" attachment but I wasn't able to find the >>>> >> "Out of IOMMU space" message in the dmesg. Can that be corrected? >>>> >> I was looking for IDE/SATA errors *before* the IOMMU errors. >>>> >> >>>> >> But I was surprised to find these bits: >>>> >> ... >>>> >> Kernel command line: mce=bootlog root=/dev/ram0 >>>> >> real_root=/dev/evms/root init=/linuxrc iommu=allowdac,merge,memaper=3 >>>> >> 3w_9xxx.use_msi=1 snd-hda-intel.enable_msi=1 doevms quiet >>>> >> Initializing CPU#0 >>>> >> ... >>>> >> Your BIOS doesn't leave a aperture memory hole >>>> >> Please enable the IOMMU option in the BIOS setup >>>> >> This costs you 256 MB of RAM >>>> >> ... >>>> >> >>>> >> I'm not familiar with iommu= parameter nor the warning about the BIOS. >>>> >> Any comments on that? >>>> >> >>>> >> thanks, >>>> >> grant >>>> >> >>>> >>>>> Apr 3 13:38:46 rngmhpamd sata_nv 0000:00:05.0: PCI-DMA: Out of IOMMU space for >>>> >>>>> 4096 bytes >>>> >>>>> Apr 3 13:38:46 rngmhpamd ata1: EH in SWNCQ mode,QC:qc_active 0x1 sactive 0x1 >>>> >>>>> Apr 3 13:38:46 rngmhpamd ata1: SWNCQ:qc_active 0x0 defer_bits 0x0 >>>> >>>>> last_issue_tag 0xfafbfcfd >>>> >>>>> Apr 3 13:38:46 rngmhpamd dhfis 0x0 dmafis 0x0 sdbfis 0x0 >>>> >>>>> Apr 3 13:38:46 rngmhpamd ata1: ATA_REG 0x50 ERR_REG 0x0 >>>> >>>>> Apr 3 13:38:46 rngmhpamd ata1: tag : dhfis dmafis sdbfis sacitve >>>> >>>>> Apr 3 13:38:46 rngmhpamd ata1.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action >>>> >>>>> 0x6 >>>> >>>>> Apr 3 13:38:46 rngmhpamd ata1.00: cmd 60/08:00:00:00:00/00:00:00:00:00/40 tag >>>> >>>>> 0 ncq 4096 in >>>> >>>>> Apr 3 13:38:46 rngmhpamd res 50/00:00:00:00:00/00:45:00:00:00/a0 Emask 0x40 >>>> >>>>> (internal error) >>>> >>>>> Apr 3 13:38:46 rngmhpamd ata1.00: status: { DRDY } >>>> >>>>> Apr 3 13:38:46 rngmhpamd ata1: hard resetting link >>>> >>>> >>>> >>>> Are these scary-looking messages also present in 2.6.28? >>>> >>>> >>>> >>>> If so, perhaps the ata code is leaking DMA memory on the error-handling path? >>>> >>>> >>>> >>>>> Apr 3 13:38:47 rngmhpamd ata1: SATA link up 3.0 Gbps (SStatus 123 SControl >>>> >>>>> 300) >>>> >>>>> Apr 3 13:38:47 rngmhpamd ata1.00: configured for UDMA/100 >>>> >>>>> Apr 3 13:38:47 rngmhpamd ata1: EH complete >>>> >>>>> Apr 3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] 488397168 512-byte hardware >>>> >>>>> sectors: (250 GB/232 GiB) >>>> >>>>> Apr 3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] Write Protect is off >>>> >>>>> Apr 3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00 >>>> >>>>> Apr 3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] Write cache: enabled, read cache: >>>> >>>>> enabled, doesn't support DPO or FUA >>>> >>>>> Apr 3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] 488397168 512-byte hardware >>>> >>>>> sectors: (250 GB/232 GiB) >>>> >>>>> Apr 3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] Write Protect is off >>>> >>>>> Apr 3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00 >>>> >>>>> Apr 3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] Write cache: enabled, read cache: >>>> >>>>> enabled, doesn't support DPO or FUA >>>> >>>>> >>>> >>>>> And >>>> >>>>> >>>> >>>>> Mar 31 20:56:18 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space >>>> >>>>> for 4608 bytes >>>> >>>>> Mar 31 20:56:18 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space >>>> >>>>> for 69632 bytes >>>> >>>>> Mar 31 20:56:48 rngmhpamd usb 1-4: reset high speed USB device using ehci_hcd >>>> >>>>> and address 8 >>>> >>>>> Mar 31 20:56:48 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space >>>> >>>>> for 11776 bytes >>>> >>>>> Mar 31 20:56:48 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space >>>> >>>>> for 69632 bytes >>>> >>>>> Mar 31 20:57:19 rngmhpamd usb 1-4: reset high speed USB device using ehci_hcd >>>> >>>>> and address 8 >>>> >>>>> Mar 31 20:57:19 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space >>>> >>>>> for 11776 bytes >>>> >>>>> Mar 31 20:57:19 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space >>>> >>>>> for 69632 bytes >>>> >>>>> Mar 31 20:57:50 rngmhpamd usb 1-4: reset high speed USB device using ehci_hcd >>>> >>>>> and address 8 >>>> >>>>> Mar 31 20:57:50 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space >>>> >>>>> for 11776 bytes >>>> >>>>> Mar 31 20:57:50 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space >>>> >>>>> for 69632 bytes >>>> >>>>> Mar 31 20:58:21 rngmhpamd usb 1-4: reset high speed USB device using ehci_hcd >>>> >>>>> and address 8 >>>> >>>>> Mar 31 20:58:21 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space >>>> >>>>> for 11776 bytes >>>> >>>>> Mar 31 20:58:21 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space >>>> >>>>> for 69632 bytes >>>> >>>>> Mar 31 20:58:52 rngmhpamd usb 1-4: reset high speed USB device using ehci_hcd >>>> >>>>> and address 8 >>>> >>>>> Mar 31 20:58:52 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space >>>> >>>>> for 11776 bytes >>>> >>>>> Mar 31 20:58:52 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space >>>> >>>>> for 69632 bytes >>>> >>>>> Mar 31 20:59:01 rngmhpamd sd 8:0:0:0: [sdc] Unhandled error code >>>> >>>>> Mar 31 20:59:01 rngmhpamd sd 8:0:0:0: [sdc] Result: hostbyte=0x07 >>>> >>>>> driverbyte=0x00 >>>> >>>>> Mar 31 20:59:01 rngmhpamd end_request: I/O error, dev sdc, sector 1137 >>>> >>>>> Mar 31 20:59:01 rngmhpamd __ratelimit: 246 callbacks suppressed >>>> >>>> >>>> >>>> Do we have any debugging option for dumping the current PCI DMA >>>> >>>> allocations, find out where it has all gone? >>>> >>>> >>>> >>>> >>>> >>> >>>> >>> Upgrade to 2.6.29-gentoo-r1 (2.6.29.1), problem is still here, can >>>> >>> easyly trigger it. I boot with default apperture, 64mb, and while >>>> >>> write to usb-hdd get this: >>>> >>> >>>> >>> Apr 5 14:28:56 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU >>>> >>> space for 65536 bytes >>>> >>> Apr 5 14:28:56 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU >>>> >>> space for 65536 bytes >>>> >>> Apr 5 14:29:27 rngmhpamd usb 1-4: reset high speed USB device using >>>> >>> ehci_hcd and address 6 >>>> >>> -- >>>> >>> To unsubscribe from this list: send the line "unsubscribe linux-ide" in >>>> >>> the body of a message to majordomo@xxxxxxxxxxxxxxx >>>> >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>> >>> >>>> >> >>>> > >>>> > >>>> > >>>> > -- >>>> > С уважением Данила Жукоцкий, системный администратор ЗАО "Роснефтегазмаш" >>>> > >>>> > >>>> > >>>> > -- >>>> > С уважением Данила Жукоцкий, системный администратор ЗАО "Роснефтегазмаш" >>>> > >>>> >>>> >>>> >>>> -- >>>> С уважением Данила Жукоцкий, системный администратор ЗАО "Роснефтегазмаш" >>>> -- >>>> To unsubscribe from this list: send the line "unsubscribe linux-ide" in >>>> the body of a message to majordomo@xxxxxxxxxxxxxxx >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>> >> >> >> >> -- >> С уважением Данила Жукоцкий, системный администратор ЗАО "Роснефтегазмаш" >> > -- С уважением Данила Жукоцкий, системный администратор ЗАО "Роснефтегазмаш" -- To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
