Thank you for answer, Grant. I try tomorrow this patch. What done before you wrote me answer. I build all external non critical to boot stuff (sata,pata,usb,firewire,network drivers) to modules. All that stuff, without forcedeth, use 32bit dma AFAIC. Reboot in single user mode with extremely low gart apperture, iommu=4194304 (4mb). I presume that so small apperture help my quickly trigger bug. Unload all modules exept ide\sata ( ahci, libata, sata_nv) stuff and perform dd in and out sata hdd. I can read all hdd to /dev/null without problem, but while i perform dd from raw raid volume to sata hdd computer hung hard immediately with "kernel panic - not syncing: dma_map_area overflow 65536 bytes". No oops, no register dump, nothing, just this string in console. Capslock and scroll lock leds blinking, magic sysrq not work. I reboot, remove all modules and all sata stuff, but insert usb stuff and perform some test on usb drive without any problem. After that i perform test without any modules loaded to and from /dev/null and 3ware raid test partition. No problem. So it may be sata\ide bug? Anyway, i can reproduce hung with 1gb apperture size, but it take couple hours. And again. Very strange behavior forcedeth nics. When i correctly poweroff 2.6.29 kernel and boot 2.6.28 all my forcedeth nics down! In dmesg "no link during initialization" without "link up", leds off. Remove\insert modules, unplug and plug cables, warm reboots, cold reboots, poweroff with power cord off, reseting BIOS!!!!! wont help. Only one way i can boot 2.6.28 with working net is boot to 2.6.29 (nic up, leds light), then hard poweroff pressing power button 5 sec, and then powerup computer. Nic leds still light during this barbarian procedure, and 2.6.28 kernel boot with working network. After that reboot, poweron\poweroff and so on work as all years before 2.6.29 is released :) And i'm not alone with that, i know 3 people with similar forcedeth problem. One wrote this message http://forums.gentoo.org/viewtopic-p-5594777.html#5594777 , one this http://lkml.org/lkml/2009/1/11/403 (and return motherboard to dealer, as i understand), an one crying me to icq. I can wrote bugreport about this, but if you can explain me this mystic it whill be prefect, because debugging 2.6.29 with this glitches is pain in the ass. >2009/4/8 Grant Grundler <grundler@xxxxxxxxxx>: > 2009/4/7 Данила Жукоцкий <optimusgd@xxxxxxxxx>: >> Dammit BUG is still here! >> >> Sorry guys, bug is STILL here, but it hard to trigger. After eight >> night hours network and 3ware raid activity (sata drive umounted, usb >> hdd unplugged) my dmesg full of "dma_map_sg overflow" messages. >> Computer work all-night without any iommu options in boot string, but >> DMA still leaking, as i see. > > Possibly, but not necessarily a leak though. As you've noticed, the IOMMU > mappings is a limited resource and we can run out if too many IOs are > in flight. I *think* a 64MB apperture means we can have 64MB of IO > mapped at the same time - assuming we have 100% efficiency and > no fragmentation or other forms of "unusable" space. > >> Anybody, who understand how IOMMU driver >> work, can say me how i can set very small apperture size, about 4mb or >> something, > > You can 't reduce the IOMMU mapping space to smaller than the > amount of IO can possibly be in flight. A starting point to compute this: > 1) count the number of block devices, the max size of each IO, and the > queue depth. > 2) plus number of NICs * RX depth * TX depth * 4k (page size) > > I'm ignoring pci_consistent (ie control data) data allocations. Those can be > determined by looking at idle system after reboot. Don't know where but > I had added debug code to parisc IOMMU code to dump that sort of info in > the past. > > I'm also ignore 64-bit capable devices completely bypassing the IOMMU > and not consuming any resources at all. > >> because with default 64mb apperture i get leak only after >> couple of hours, so leak is small. > > For a system with a RAID device and more than 1GB of RAM, it's not that > hard to get more than 64MB of IO in flight. Just depends on the IO > controllers. > > If it's a leak, the only way to track it is to add debug code that tracks > the IP of the caller along with each mapping. Enabling such debugging > code will substantially change the timing of DMA map/unmap calls. > >> And i again not understand source >> of leaking, it may be network, sata, usb, etc (i don't reboot computer >> after yesterday's IO tests to from usb\sata). So how i can painless >> catch the leak source without disabling subsystems one by one and >> hours-long tests? It is workstation, and i must do my job all-day with >> it. > > First off, you can eliminate 64-bit devices/drivers. Yinghai Lu submitted > a patch to print the DMA mask: > http://lkml.org/lkml/2008/10/8/274 > > Add that patch, rebuild the kernel and post the dmesg output would > be a starting point. Posting "lspci -vt" and "lspci -v" would also help. > > FYI: PCI drivers get 32-bit dma_mask by default until the driver calls > pci_set_dma_mask(). So inspecting drivers will tell you if they are > actually using the IOMMU (or not if they set dma_mask to 64-bit). > > hth, > grant > >> 2009/4/8 Grant Grundler <grundler@xxxxxxxxxx>: >>> Anyone understand why allowdac exists or if it's buggy? >>> >>> thanks, >>> grant >>> >>> >>> ---------- Forwarded message ---------- >>> From: Grant Grundler <grundler@xxxxxxxxxx> >>> Date: 2009/4/7 >>> Subject: Re: [Bugme-new] [Bug 13001] New: PCI-DMA: Out of IOMMU space >>> To: Данила Жукоцкий <optimusgd@xxxxxxxxx> >>> Cc: akpm@xxxxxxxxxxxxxxxxxxxx, linux-ide@xxxxxxxxxxxxxxx, >>> bugme-daemon@xxxxxxxxxxxxxxxxxxx, x86@xxxxxxxxxx >>> >>> >>> 2009/4/7 Данила Жукоцкий <optimusgd@xxxxxxxxx> >>>> >>>> Thank You, Grant, for Your simple questions! >>> >>> Welcome! >>> I was just trying to point out some additional information to folks >>> who might understand the code. I don't in this case but was looking >>> for more clues. >>> >>>> >>>> Without "allowdac" after >>>> couple of hours testing i cannot reproduce bug! So it is my stupid >>>> mistake, i don't understand Why i add this absolutely unusual >>>> parameter to boot string. >>> >>> I thought I understood IOMMUs having written code for 4 different >>> implementations. >>> I don't understand why "allowdac" parameter exists. >>> dma_mask stuff should be handling this already. >>> Can someone explain *why* (Данила already posted the docs) this >>> parameter exists? >>> (ie use case that dma_mask APIs don't work.) >>> >>>> I'm apologize to All off You for this stupid >>>> mindfuck. You may close the bug, because it is bug in my damn head. >>> >>> My first impression: the bug is either allocdma exists instead of using >>> DMA API (See Documentation/DMA-API.txt) OR the documentation for >>> allocdma is missing warnings about "this could break your system" >>> and clearly specify when it should be used. >>> >>> hth, >>> grant >>> >>>> >>>> 2009/4/7 Данила Жукоцкий <optimusgd@xxxxxxxxx>: >>>> > Forgot reply to all, sorry >>>> > >>>> > ---------- Forwarded message ---------- >>>> > From: Данила Жукоцкий <optimusgd@xxxxxxxxx> >>>> > Date: 2009/4/7 >>>> > Subject: Re: [Bugme-new] [Bug 13001] New: PCI-DMA: Out of IOMMU space >>>> > To: Grant Grundler <grundler@xxxxxxxxxx> >>>> > >>>> > >>>> > Yes, in attachment clear dmesg, warnings in bugreport body >>>> > >>>> >>Apr 3 13:38:46 rngmhpamd sata_nv 0000:00:05.0: PCI-DMA: Out of IOMMU space for 4096 bytes >>>> >>Apr 3 13:38:46 rngmhpamd sata_nv 0000:00:05.0: PCI-DMA: Out of IOMMU space for 4096 bytes >>>> > >>>> > I got "PCI-DMA: Out of IOMMU space" while trying write data to usb or >>>> > sata hdd before all other error messages. After that usb and sata >>>> > drives lost. All other noise is attempts to communicate with died >>>> > devices. >>>> > >>>> >>Kernel command line: >>>> >>mce=bootlog root=/dev/ram0 real_root=/dev/evms/root init=/linuxrc >>>> > I'm boot from 3ware raid with evmc >>>> >>iommu=allowdac,merge,memaper=3 >>>> > This is from Documentation/x86/x86_64/boot-options.txt >>>> > iommu=allowdac Im try to avoid DMA bug. May be that not need. >>>> > allowdac Allow double-address cycle (DAC) mode, i.e. DMA >4GB. >>>> > DAC is used with 32-bit PCI to push a 64-bit address in >>>> > two cycles. When off all DMA over >4GB is forced through >>>> > an IOMMU or software bounce buffering. >>>> > merge Do scatter-gather (SG) merging. >>>> > memaper[=<order>] Allocate an own aperture over RAM with size 32MB<<order. >>>> > (default: order=1, i.e. 64MB) >>>> > With default apperture, 64mb, DMA leak very fast, now i have >>>> > memaper=5, 1 gb, becouse i must do my job and can't rollback to 2.6.28 >>>> > due strange mysterious problem with forcedeth nics that i can't >>>> > explain and solve. If solution for DMA leak will not be found, i'm try >>>> > to fill bugreport about problem with nics. >>>> > >>>> >>3w_9xxx.use_msi=1 snd-hda-intel.enable_msi=1 doevms quiet >>>> > I prefer use msi on that hardware. >>>> >>... >>>> >>Your BIOS doesn't leave a aperture memory hole >>>> >>Please enable the IOMMU option in the BIOS setup >>>> >>This costs you 256 MB of RAM >>>> > >>>> > xw9400 BIOS do not have IOMMU option in the BIOS setup. Now this costs >>>> > me 1gb of ram >>>> > >>>> > Anyway, i can stable reproduce bug without all this whistlers >>>> > >>>> > 2009/4/7 Grant Grundler <grundler@xxxxxxxxxx>: >>>> >> 2009/4/5 Данила Жукоцкий <optimusgd@xxxxxxxxx>: >>>> >>> 2009/4/4 Andrew Morton <akpm@xxxxxxxxxxxxxxxxxxxx>: >>>> >>>> >>>> >>>> (switched to email. Please respond via emailed reply-to-all, not via the >>>> >>>> bugzilla web interface). >>>> >>>> >>>> >>>> On Fri, 3 Apr 2009 09:30:19 GMT bugzilla-daemon@xxxxxxxxxxxxxxxxxxx wrote: >>>> >>>> >>>> >>>>> http://bugzilla.kernel.org/show_bug.cgi?id=13001 >>>> >>>>> >>>> >>>>> Summary: PCI-DMA: Out of IOMMU space >>>> >>>>> Product: Platform Specific/Hardware >>>> >>>>> Version: 2.5 >>>> >>>>> Kernel Version: 2.6.29-gentoo >>>> >>>>> Platform: All >>>> >>>>> OS/Version: Linux >>>> >>>>> Tree: Mainline >>>> >>>>> Status: NEW >>>> >>>>> Severity: normal >>>> >>>>> Priority: P1 >>>> >>>>> Component: x86-64 >>>> >>>>> AssignedTo: platform_x86_64@xxxxxxxxxxxxxxxxxxxx >>>> >>>>> ReportedBy: optimusgd@xxxxxxxxx >>>> >>>>> Regression: Yes >>>> >>>>> >>>> >>>>> >>>> >>>>> Created an attachment (id=20789) >>>> >>>>> --> (http://bugzilla.kernel.org/attachment.cgi?id=20789) >>>> >>>>> hwreport generated info >>>> >>>>> >>>> >>>>> After some IO activity the "PCI-DMA: Out of IOMMU space" message appear. >>>> >>>>> 2.6.28-gentoo-r4 work ok, so it is regression. >>>> >>>> >>>> >>>> It is indeed a regression. >>>> >>>> >>>> >>>>> Dmesg fragments: >>>> >>>>> >>>> >>>>> >>>> >>>>> Apr 3 13:38:46 rngmhpamd sata_nv 0000:00:05.0: PCI-DMA: Out of IOMMU space for >>>> >>>>> 4096 bytes >>>> >> >>>> >> The bug report has a "dmesg" attachment but I wasn't able to find the >>>> >> "Out of IOMMU space" message in the dmesg. Can that be corrected? >>>> >> I was looking for IDE/SATA errors *before* the IOMMU errors. >>>> >> >>>> >> But I was surprised to find these bits: >>>> >> ... >>>> >> Kernel command line: mce=bootlog root=/dev/ram0 >>>> >> real_root=/dev/evms/root init=/linuxrc iommu=allowdac,merge,memaper=3 >>>> >> 3w_9xxx.use_msi=1 snd-hda-intel.enable_msi=1 doevms quiet >>>> >> Initializing CPU#0 >>>> >> ... >>>> >> Your BIOS doesn't leave a aperture memory hole >>>> >> Please enable the IOMMU option in the BIOS setup >>>> >> This costs you 256 MB of RAM >>>> >> ... >>>> >> >>>> >> I'm not familiar with iommu= parameter nor the warning about the BIOS. >>>> >> Any comments on that? >>>> >> >>>> >> thanks, >>>> >> grant >>>> >> >>>> >>>>> Apr 3 13:38:46 rngmhpamd sata_nv 0000:00:05.0: PCI-DMA: Out of IOMMU space for >>>> >>>>> 4096 bytes >>>> >>>>> Apr 3 13:38:46 rngmhpamd ata1: EH in SWNCQ mode,QC:qc_active 0x1 sactive 0x1 >>>> >>>>> Apr 3 13:38:46 rngmhpamd ata1: SWNCQ:qc_active 0x0 defer_bits 0x0 >>>> >>>>> last_issue_tag 0xfafbfcfd >>>> >>>>> Apr 3 13:38:46 rngmhpamd dhfis 0x0 dmafis 0x0 sdbfis 0x0 >>>> >>>>> Apr 3 13:38:46 rngmhpamd ata1: ATA_REG 0x50 ERR_REG 0x0 >>>> >>>>> Apr 3 13:38:46 rngmhpamd ata1: tag : dhfis dmafis sdbfis sacitve >>>> >>>>> Apr 3 13:38:46 rngmhpamd ata1.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action >>>> >>>>> 0x6 >>>> >>>>> Apr 3 13:38:46 rngmhpamd ata1.00: cmd 60/08:00:00:00:00/00:00:00:00:00/40 tag >>>> >>>>> 0 ncq 4096 in >>>> >>>>> Apr 3 13:38:46 rngmhpamd res 50/00:00:00:00:00/00:45:00:00:00/a0 Emask 0x40 >>>> >>>>> (internal error) >>>> >>>>> Apr 3 13:38:46 rngmhpamd ata1.00: status: { DRDY } >>>> >>>>> Apr 3 13:38:46 rngmhpamd ata1: hard resetting link >>>> >>>> >>>> >>>> Are these scary-looking messages also present in 2.6.28? >>>> >>>> >>>> >>>> If so, perhaps the ata code is leaking DMA memory on the error-handling path? >>>> >>>> >>>> >>>>> Apr 3 13:38:47 rngmhpamd ata1: SATA link up 3.0 Gbps (SStatus 123 SControl >>>> >>>>> 300) >>>> >>>>> Apr 3 13:38:47 rngmhpamd ata1.00: configured for UDMA/100 >>>> >>>>> Apr 3 13:38:47 rngmhpamd ata1: EH complete >>>> >>>>> Apr 3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] 488397168 512-byte hardware >>>> >>>>> sectors: (250 GB/232 GiB) >>>> >>>>> Apr 3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] Write Protect is off >>>> >>>>> Apr 3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00 >>>> >>>>> Apr 3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] Write cache: enabled, read cache: >>>> >>>>> enabled, doesn't support DPO or FUA >>>> >>>>> Apr 3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] 488397168 512-byte hardware >>>> >>>>> sectors: (250 GB/232 GiB) >>>> >>>>> Apr 3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] Write Protect is off >>>> >>>>> Apr 3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00 >>>> >>>>> Apr 3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] Write cache: enabled, read cache: >>>> >>>>> enabled, doesn't support DPO or FUA >>>> >>>>> >>>> >>>>> And >>>> >>>>> >>>> >>>>> Mar 31 20:56:18 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space >>>> >>>>> for 4608 bytes >>>> >>>>> Mar 31 20:56:18 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space >>>> >>>>> for 69632 bytes >>>> >>>>> Mar 31 20:56:48 rngmhpamd usb 1-4: reset high speed USB device using ehci_hcd >>>> >>>>> and address 8 >>>> >>>>> Mar 31 20:56:48 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space >>>> >>>>> for 11776 bytes >>>> >>>>> Mar 31 20:56:48 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space >>>> >>>>> for 69632 bytes >>>> >>>>> Mar 31 20:57:19 rngmhpamd usb 1-4: reset high speed USB device using ehci_hcd >>>> >>>>> and address 8 >>>> >>>>> Mar 31 20:57:19 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space >>>> >>>>> for 11776 bytes >>>> >>>>> Mar 31 20:57:19 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space >>>> >>>>> for 69632 bytes >>>> >>>>> Mar 31 20:57:50 rngmhpamd usb 1-4: reset high speed USB device using ehci_hcd >>>> >>>>> and address 8 >>>> >>>>> Mar 31 20:57:50 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space >>>> >>>>> for 11776 bytes >>>> >>>>> Mar 31 20:57:50 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space >>>> >>>>> for 69632 bytes >>>> >>>>> Mar 31 20:58:21 rngmhpamd usb 1-4: reset high speed USB device using ehci_hcd >>>> >>>>> and address 8 >>>> >>>>> Mar 31 20:58:21 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space >>>> >>>>> for 11776 bytes >>>> >>>>> Mar 31 20:58:21 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space >>>> >>>>> for 69632 bytes >>>> >>>>> Mar 31 20:58:52 rngmhpamd usb 1-4: reset high speed USB device using ehci_hcd >>>> >>>>> and address 8 >>>> >>>>> Mar 31 20:58:52 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space >>>> >>>>> for 11776 bytes >>>> >>>>> Mar 31 20:58:52 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space >>>> >>>>> for 69632 bytes >>>> >>>>> Mar 31 20:59:01 rngmhpamd sd 8:0:0:0: [sdc] Unhandled error code >>>> >>>>> Mar 31 20:59:01 rngmhpamd sd 8:0:0:0: [sdc] Result: hostbyte=0x07 >>>> >>>>> driverbyte=0x00 >>>> >>>>> Mar 31 20:59:01 rngmhpamd end_request: I/O error, dev sdc, sector 1137 >>>> >>>>> Mar 31 20:59:01 rngmhpamd __ratelimit: 246 callbacks suppressed >>>> >>>> >>>> >>>> Do we have any debugging option for dumping the current PCI DMA >>>> >>>> allocations, find out where it has all gone? >>>> >>>> >>>> >>>> >>>> >>> >>>> >>> Upgrade to 2.6.29-gentoo-r1 (2.6.29.1), problem is still here, can >>>> >>> easyly trigger it. I boot with default apperture, 64mb, and while >>>> >>> write to usb-hdd get this: >>>> >>> >>>> >>> Apr 5 14:28:56 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU >>>> >>> space for 65536 bytes >>>> >>> Apr 5 14:28:56 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU >>>> >>> space for 65536 bytes >>>> >>> Apr 5 14:29:27 rngmhpamd usb 1-4: reset high speed USB device using >>>> >>> ehci_hcd and address 6 >>>> >>> -- >>>> >>> To unsubscribe from this list: send the line "unsubscribe linux-ide" in >>>> >>> the body of a message to majordomo@xxxxxxxxxxxxxxx >>>> >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>> >>> >>>> >> >>>> > >>>> > >>>> > >>>> > -- >>>> > С уважением Данила Жукоцкий, системный администратор ЗАО "Роснефтегазмаш" >>>> > >>>> > >>>> > >>>> > -- >>>> > С уважением Данила Жукоцкий, системный администратор ЗАО "Роснефтегазмаш" >>>> > >>>> >>>> >>>> >>>> -- >>>> С уважением Данила Жукоцкий, системный администратор ЗАО "Роснефтегазмаш" >>>> -- >>>> To unsubscribe from this list: send the line "unsubscribe linux-ide" in >>>> the body of a message to majordomo@xxxxxxxxxxxxxxx >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>> >> >> >> >> -- >> С уважением Данила Жукоцкий, системный администратор ЗАО "Роснефтегазмаш" >> > -- С уважением Данила Жукоцкий, системный администратор ЗАО "Роснефтегазмаш" -- To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
