Hi,
I've been working on the issue. This is what I found about multi-controller setup:
The problem comes from the usbip tool trying to connect usb2 devices to usb3 ports, like this:
- usbip tool asks vhci driver for free port.
- If the first port (usb2) is already occupied, vhci answers with usb3 port (instead of next controller's usb2 port)
- usbip tool tries to connect usb2 device to usb3 port and fails
- usbip tool asks for the next free port, which is still the same usb3 port.
- Loop the above forever.
the code at tools/usb/usbip/libsrc/vhci_driver.c :
329 int usbip_vhci_get_free_port(uint32_t speed)
330 {
331 for (int i = 0; i < vhci_driver->nports; i++) {
332 if (speed == USB_SPEED_SUPER &&
333 vhci_driver->idev[i].hub != HUB_SPEED_SUPER)
334 continue;
335
336 if (vhci_driver->idev[i].status == VDEV_ST_NULL)
337 return vhci_driver->idev[i].port;
338 }
339
340 return -1;
341 }
prevents usb3 devices connecting to usb2 ports, but not the other way around.
I've patched this preventing the opposite, and multi-controller starts working the way I think it should:
- usb2 devices connect exclusively to usb2 ports
- The above is also true for usb3 devices and ports
- When there is no compatible port for the device in the present controller, go for the next
At least, for connecting devices this seems to work, and I can get several usb2 and usb3 connected to the same machine in different orders.
The question is... is this the way it is supposed to work?
Another problem is... some of my test devices don't work. They detonate a kernel BUG line in the vhci controller:
Nov 14 14:35:29 kernel-tester kernel: [ 229.636543] kernel BUG at drivers/usb/usbip/vhci_hcd.c:683!
This is happening even using the vhci module as it is downloaded from Ubuntu (no intervention on my behalf in this experiment), so it seems they just stopped working some kernels ago.
I'm still bisecting the kernel for this, but for now I have filed a bug at bugzilla with the specific device and kernel versions: https://bugzilla.kernel.org/show_bug.cgi?id=197867
It looks like something else got broken since kernel 4.10 (in which the device works). Any clue on this?
Regards,
Juan
Juan Antonio Zea Herranz
Proyectos y consultoría | Qindel Group
E: juan.zea@xxxxxxxxxx
T: +34 91 766 24 21
M: +34 637 74 63 09
W: qindel.com
----- Mensaje original -----
De: "Bjørn Mork" <bjorn@xxxxxxx>
Para: "Juan Zea" <juan.zea@xxxxxxxxxx>
CC: linux-usb@xxxxxxxxxxxxxxx, "Valentina Manea" <valentina.manea.m@xxxxxxxxx>, "Shuah Khan" <shuah@xxxxxxxxxx>
Enviados: Martes, 31 de Octubre 2017 12:28:30
Asunto: Re: usbip port number limits
Juan Zea <juan.zea@xxxxxxxxxx> writes:
> Having a look at the code of the VHCI driver at
> https://github.com/torvalds/linux/tree/master/drivers/usb/usbip , I
> found these two parameters: USBIP_VHCI_HC_PORTS and USBIP_VHCI_NR_HCS
>
> I've compiled VHCI driver from latest source above, altering both
> parameters to get higher number of controllers and ports per
> controller (not only the driver, but also the usbip cli tool at
> https://github.com/torvalds/linux/tree/master/tools/usb/usbip ), and
> run into several problems:
>
> 1. USBIP_VHCI_HC_PORTS x USBIP_VHCI_NR_HCS <= 128
>
> It seems, whatever the combination of both parameters, usbip command
> line tool won't accept that your driver is compiled with numbers
> violating the above rule. Is this a problem of the tool code or is
> there a reason behind this? As far as I 've researched, the 127
> devices limit in Linux is per controller, so increasing the number of
> controllers in usbip driver shouldn't be limited this way, am I wrong?
>
> 2. I've compiled with 4 ports per controller and 32 controllers. Given
> that the drivers creates an usb2 and an usb3 controller, it results in
> 64 controllers with four ports each: 128 ports. Here, usbip cli tool
> works. But... if you try to attach a device, the kernel dumps a trace
> that starts with this:
>
> Oct 30 10 :43:14 test-mononode-ubuntu-16 kernel: [ 44.138068] usbcore: failed to get bus number
> Oct 30 10 :43:14 test-mononode-ubuntu-16 kernel: [ 44.138813] vhci_hcd: usb_add_hcd hs failed -7
> Oct 30 10 :43:14 test-mononode-ubuntu-16 kernel: [ 44.139559] vhci_hcd: probe of vhci_hcd.31 failed with error -7
> Oct 30 10 :44:23 test-mononode-ubuntu-16 kernel: [ 113.345824] BUG: unable to handle kernel NULL pointer dereference at 0000000000000230
> Oct 30 10 :44:23 test-mononode-ubuntu-16 kernel: [ 113.347584] IP: status_show+0x13c/0x360 [vhci_hcd]
> Oct 30 10 :44:23 test-mononode-ubuntu-16 kernel: [ 113.348592] PGD 1398cc067
> Oct 30 10 :44:23 test-mononode-ubuntu-16 kernel: [ 113.348593] PUD 136a96067
> Oct 30 10 :44:23 test-mononode-ubuntu-16 kernel: [ 113.349119] PMD 0
>
> This leads me to thinking there is some problem with such a big number
> of controllers, although the controller number parameters is supposed
> to have a max value of 128. Could this be related to the kernel I'm
> using? Maybe I should compile the full kernel to get higher numbers
> working?
You are correct. There are multiple issues here:
1) the drivers/usb/core/hcd.c has a limit on the number of USB buses.
This is currently 64, and not configurable without editing the
source.
So setting USBIP_VHCI_NR_HCS to anything higher than ~30 is futile and
has probably never been tested. It cannot work
2) The sysfs code does not consider the possibility that one or more of
the VHCIs could have failed to register. It will happily register
attributes for all "vhci_num_controllers", and does little to sanity
check anything when accessing those attributes.
So it should not come as a surprise that this goes Ooops if one or more
controllers have failed to register.
> 3. For this last test, I compiled with smaller numbers, just to test
> how the several controllers thing works. Specifically, one port per
> controller, 16 controllers in total. This leads to the module loading,
> the controllers listable with lsusb, and no problems in kernel log. I
> attach one device, and it goes to the first controller. Working... no
> problem at all. But, if you attach a second device, the command hangs
> and I get this in syslog:
>
> Oct 30 11 :07:14 test-mononode-ubuntu-16 kernel: [ 1078.239613] vhci_hcd vhci_hcd.0: port 0 already used
>
> This message was repeated for more than 7000 times before I could stop
> the command. So... my conclusion is that usbip won't work correctly if
> compiled with more than one controller. Is this related to the usbip
> cli tool or is there any problem at the driver?
I think it would be great if you could work a little on the usbip
drivers. The multi-controller support was very immature when it was
merged, and has not gotten much love since. I still think using a fixed
number of controllers was a big mistake. It would have been much better
to add controllers dynamically when you needed more ports.
The bugs you have found are fallout of the shortcuts taken when the code
was merged.
Bjørn
--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
