(Resending with reply-all, sorry for that)
Hi,
> a first, very simplified, linux kernel driver for the Kinect sensor
> device is now available, so you now can use it as a ridiculously
> expensive webcam with any v4l2 application.
Good work! Here are my thoughts on the kernel driver:
> As you can see this driver is just some "advanced copy&paste" from
> libfreenect, plus reducing the packets scan routine to the bare minimum.
> Taking some code from libfreenect should be OK as it comes under GPLv2
> (dual-licensed with an Apache license) but let me know if you think
> there could be any issues.
This is actually precisely the reason why I made sure we had GPLv2 as an
option :)
> Now the hard part begins, here's a loose TODO-list:
> - Discuss the "fragmentation problem":
> * the webcam kernel driver and the libusb backend of libfreenect
> are not going to conflict each other in practice, but code
> duplication could be avoided to some degree; we could start
> listing the advantages and disadvantages of a v4l2 backend
> opposed to a libusb backend for video data in libfreenct (don't
> think in terms of userspace/kernelspace for now).
I don't think we should exclusively move over Kinect support to current
standard kernel interfaces (v4l, input, ALSA...), as they are too
generic and people like to experiment with and use the features of the
Kinect to their fullest. However, kernel drivers do have their
advantages, and of course people also want to use the Kinect as a
standard input device where it makes sense.
I think the best solution would be to have a kernel driver that
implements the basics of the kinect (v4l RGB video for webcam use,
mainly, and possibly a simple depth feed), as well as provides the
streaming layer for libfreenect to use (iso packet header parsing and
frame reassembly, but no data processing). Libfreenect would hook into
this as a backend (the current abstraction is at the USB level but this
could be changed) and therefore get the benefits of kernel-side iso
handling and streaming (probably 10x less context switching and more
reliable performance for loaded systems) while still letting the lib
handle configuration and letting us experiment with the different modes
and formats without messing with the kernel code. And without
libfreenect, users get to use the Kinect as a regular webcam.
I'm not sure what this "power" kernel interface would look like, but it
could be v4l2 as long as we can make sure we can expose all the Kinect
specifics at a lower level (basically weird/compressed/packed frame
formats, the timestamps, and raw control commands/responses). Also, I
haven't checked, but I assume v4l2 supports select()/poll() for frames
(this is critical in order to interoperate with the way libusb does
things, for our other subdevices). Another advantage of not exposing all
the Kinect functionality at high level via v4l2 methods is that we don't
have to duplicate that code in the kernel (otherwise libfreenect would
just become one huge switch at the high level to select between doing
most everything in the kernel and doing most everything in userspace,
and then what's the point). The goal would be a thin kernel driver in
libfreenect mode, plus a thick but basic mode for general compatibility
with other v4l apps.
> * Would exposing the accelerometer as an input device make sense
> too? The only reason for that is to use the data in already
> existing applications. And what about led and motor?
I'm not sure we need an accelerometer kernel driver, but if someone has
a use case for accelerometer data delivered via evdev with existing code
it could be done. There's not much point in making libfreenect talk to
it though, for that subdevice we should just ask libusb to unbind the
kernel driver if one gets written.
> - Decide if we want two separate video nodes, or a
> combined RGB-D data stream coming from a single video device node.
> (I haven't even looked at the synchronization logic yet).
A combined video stream is hard because the sources aren't framelocked,
not to mention the video streams aren't aligned (and as far as I can
tell, no, the Kinect's standard firmware cannot either of these, even
though its chipset can). I think figuring out how to align things both
temporally and spatially belongs squarely in userspace, so the Kinect
should show up as two v4l2 devices: one for video, and one for depth.
This also lets us play with RGB/IR and depth parameters fully
independently (they really are two separate streams) and makes perfect
logical sense given how the Kinect hardware is implemented.
Audio is still up in the air, but an ALSA driver would of course make
sense for general microphone use; once we figure out what we want to do
with audio this will become clearer. I suspect that getting echo
cancellation to work at all will be near impossible without a hard
realtime framework extending outside the kernel (i.e. JACK), and that
won't play nice with multiple sound cards, so I think advanced features
have the greatest chance of working with a setup like this, and thus
without using a kernel driver:
ALSA card <-- JACK <-> kinect-jackmic <-> libfreenect <-> Kinect
|
+--> echo-cancelled audio to other JACK clients
i.e. we *could* expose the Kinect as a full-duplex ALSA sound card but
at best you'd need a proper realtime framework and server program doing
the echo cancellation and such, and somehow make software talk to two
ALSA cards at once (Kinect and the main soundcard), and a way to set up
the echo cancellation parameters (not to mention firmware issues), so
what's the point, just make libfreenect talk straight to the hardware.
Simple mic use (no echo cancellation, no downloading the filter matrix,
just a 4-channel mic), nonetheless, makes perfect sense as a bog
standard ALSA driver.
libusb-1.0 support will always be in there as a lowest common
denominator and to support other OSes, of course.
Cheers,
--
Hector Martin (hector@xxxxxxxxxxxxxx)
Public Key: http://www.marcansoft.com/marcan.asc
--
To unsubscribe from this list: send the line "unsubscribe linux-media" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
