On Wed, Jul 20, 2011 at 06:04:45PM +0200, Kruno Mrak wrote: > Am 19.07.2011 18:17, schrieb Sarah Sharp: > >On Fri, Jul 15, 2011 at 04:42:44PM +0800, Andiry Xu wrote: > >>On Fri, Jul 15, 2011 at 3:58 PM, Kruno Mrak<kruno.mrak@xxxxxxxxxxxxxxxx> wrote: > >>>Hi, > >>> > >>>when running our cameras on USB3.0 ports we get > >>>"ERROR no room on ep ring". > >>>This only happens if camera's image size is bigger than > >>>~1MB or when using multiple image buffers (queueing). > >>>Capturing is made through libusb and async bulk transfers. > >What standard is your camera using? > > This camera does not comply to a specific standard. Our products are > for machine > vision industry and we provide our customers a programming API > covering all of our products. I see. > > The draft USB-AV standard? > > Never heard of that. What's this? It's a draft standard being developed in the USB-IF video class work group. If your company is a member of the USB-IF, you can participate in the development. They're trying to make the video class more generic, define protocols for compressed and uncompressed data (especially useful with the larger USB 3.0 bus bandwidth) and allow the host and device to swap video source/sink roles. > > Other > >than that, I don't know of any camera standards that use bulk endpoints. > >Are you submitting multiple bulk transfers to the same endpoint, or are > >you submitting one bulk transfer and waiting on it? > > At "continuous capture start" one thread is firing multiple bulk > read requests to a single > endpoint. Another thread is waiting for finished transfer. Then > image processing > takes place on returned transfer. And finally, the transfers are > re-submitted. > Each request represents an uncompressed image from the camera, up to 5 MB. > The amount of multiple requests may vary and depends on image post > processing. > By default, we use 4. Yes, ok, that would certainly fill up an endpoint ring. Do you re-submit each transfer after it is completed, or re-submit them all at once? > >>>Now, as more and more of our customers are using PCIe-USB3.0 extension > >>>cards we get under pressure. > >>>After looking into xhci driver sources, i have seen that > >>>TRBS_PER_SEGMENT is limited to 64 and segments per ep ring > >>>is fixed to 1. (right?) > >>For bulk transfer, it's 1 segment. Isochronous transfer needs a bigger ring > >>because it inserts multiple TDs to the ring when an isoc URB is submitted. > >>Currently it uses 8 segments. > >Yes, Andiry's right. We never had a clear case for why people needed > >bigger bulk rings. VmWare did complain a bit that some of their test > >cases didn't run properly for very large transfers, but I think they > >didn't have any customers that actually used transfers that big. > > We receive raw-data (bayer-mosaic) images from camera. > So, is there a limit specified on the size of an bulk ring? No theoretical limit, it's just the code doesn't expand the rings as necessary. > Aren't bulk endpoints used for a large amount of data transfer? Yes, but most in-kernel applications submit one bulk transfer at a time, since the overhead to re-submit is smaller than if you're going through userspace. > I just wonder, if we are the only one who use bulk transfers and > such a large data amount. You're the first real application I've heard of. :) > >>>Playing with this params (TRBS_PER_SEGMENT=256 and ep ring segments=8) > >>>and first tests led to good results. > >I would like to keep the TRBS_PER_SEGMENT parameter the same, because > >otherwise I might to go change the streams code that uses a radix tree to > >map DMA addresses to virtual addresses. If you keep that to 64, how > >many ring segments do you end up needing? > > 20 segments is the minimum, then we are able to receive complete transfers. Yeah, ok, it seems like we really need dynamic endpoint ring expansion for that case. > >>>So, my question is, do you agree with me that limited > >>>ep ring allocation might cause above mentioned error. > >>>Is there a chance to push development on ep ring allocation? > >>> > >>>Thank you in advance. > >>> > >>Maybe dynamic ring allocation helps in this situation. > >Yes, it could help in this situation. However, I don't have the > >bandwidth right now to work on dynamic ring expansion. The quick fix > >would be to add a module parameter to the xHCI driver that sets the > >number of ring segments to use for the four different types of > >endpoints. Perhaps parameters named bulk_segs, intr_segs, isoc_segs, > >and ctrl_segs? > > This could be a quick (and preliminary) fix. Ok, I'll get you a patch to test tomorrow. > >It sounds like you already know how to modify the xHCI driver, so do > >you want to create a patch for this? > > I am not experienced in committing patches to open source world, > and unfortunately, i am fighting with bandwith problems on xhci and > our cameras, too. > I would prefer, to understand all the problems first, before > creating any patch. > > The bandwith problem seems really curious. > We can't reach maximum frame rate from camera on USB3.0 port. Is this with a USB 2.0 device under an xHCI (USB 3.0) port? Or a USB 3.0 device? > It does not exist all the time and there are no warnings and no error logs. > Any advice, how to investigate this problem? This is still using bulk transfers, right? They're asynchronously scheduled, so you don't have any guarantees about when they're sent out. The xHCI host controller actually does the scheduling, so you may just be running to host controller bugs. Oh, and you do know that your buffers are getting copied (possibly twice) when they go through the libusb/usbfs kernel-userspace interface? I've been meaning to fix it to use usb_alloc_buffer or maybe pin the user pages, but haven't gotten around to it. However, you would see the same latency while using the device under EHCI if that were the real problem. > >Also, is your camera already available commercially (USB-IF certified > >and all that), or will it be soon? If many Linux users are going to run > >into this issue, the xHCI driver really needs dynamic ring expansion. > >If we have some time before your product hits the market and your > >customers are just testing out your product, I think the module > >parameter would be an acceptable quick fix. > > Our camera is out on market since 2007 and we have sold several thousands > of them ( for both, Linux and Windows users). I suspect that more and more > of our customers will complain in near future. Ok, good to know. I'll work on that patch to help out your existing Linux customers. Sarah Sharp -- To unsubscribe from this list: send the line "unsubscribe linux-usb" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
