RFC: v4l-dvb Development Models Version 1, February 21st 2009 ------------------------------- Introduction ------------ The current v4l-dvb development model is based on a mercurial repository. Patches are fed into the repo by developers, the v4l-dvb maintainer will backport relevant patches from the upstream git repository into the v4l-dvb master and push essential fixes from the v4l-dvb repo into the upstream git repo. When a new kernel merge window opens all our accumulated changes are fed into the new kernel and the cycle starts again. What makes our v4l-dvb special is the ability to support older kernels. Through the use of smart compat headers, scripts and #if KERNEL_VERSION lines in the code the backwards compatibility support is done in a way that keeps the code relatively untouched. When our code is fed to the kernel, all the compat code is stripped so none of this is visible upstream. The obvious advantage is that people can use and test our drivers on older kernels without having to upgrade to the latest bleeding edge kernel. Currently we support kernels from 2.6.16 onwards. However, the v4l-dvb tree has seen considerable growth. Below is a simple statistic: $ du -bs linux-2.6.*/drivers/media/ 5569219 linux-2.6.16.61/drivers/media/ 7189484 linux-2.6.17.14/drivers/media/ 7910318 linux-2.6.18.8/drivers/media/ 8081530 linux-2.6.19.5/drivers/media/ 8419446 linux-2.6.20.21/drivers/media/ 8556067 linux-2.6.21.7/drivers/media/ 9121077 linux-2.6.22.19/drivers/media/ 9334386 linux-2.6.23.12/drivers/media/ 9667264 linux-2.6.24.7/drivers/media/ 10121497 linux-2.6.25.11/drivers/media/ 11244052 linux-2.6.26/drivers/media/ 12322637 linux-2.6.27/drivers/media/ 12947608 linux-2.6.28/drivers/media/ 13719917 linux-2.6.29-rc4/drivers/media/ So the total amount of code in the 2.6.29 tree is about 2.5 times that of the 2.6.16 tree and that tree is almost three years old. The high rate of changes and new drivers means that keeping up the backwards compatibility becomes an increasingly heavy burden. This leads to two questions: 1) Can we change our development model in such a way that this burden is reduced? 2) How far back do we want to support older kernels anyway? These questions are related, since changes in the development model has implications for the answer to the second question. 1: Alternatives to the development model ---------------------------------------- I see the following options: A) Keep our current model. This also keeps the way we do our backwards compatibility unchanged. B) Switch to a git tree that tracks Linus' tree closely and we drop backwards compatibility completely. C) Switch to the ALSA approach (http://git.alsa-project.org/). ALSA uses three trees: alsa-kernel tracks Linus' git tree, alsa-kmirror contains only the alsa part (similar to the layout of our repository), and alsa-driver is equal to alsa-kmirror, but with compatibility patches. If I understand their method correctly new changes are pushed to alsa-kernel and a script merges the changes into alsa-kmirror. Patches meant for alsa-specific files (the equivalent to our v4l/scripts or v4l2-spec directories) have to be pushed to alsa-kmirror directly because there is of course no equivalent in alsa-kernel. Every so often the alsa-driver repository is merged with the latest changes from alsa-kmirror and someone goes in, checks what broke this time and fixes any problems manually through the creation of patches. D) Anyone? I can't think of other serious alternatives. 1.1 Pros and Cons ----------------- A: Keep our current model Pros: - we know it works. - having backwards compatibility is very helpful in testing the drivers, it makes it very easy to point testers to the latest code base and have them install it. Cons: - no direct access to a full kernel git tree which makes it hard to make changes that go outside the drivers/media directory (e.g. platform related changes). - received patches against the latest git tree can be hard to apply if they conflict with our compat code in the source. - over time the code becomes ugly and harder to maintain with all the backwards compatibility changes. - the lack of a kernel git tree as the main repository makes it laborious for the v4l-dvb maintainer to create patches for the upstream git tree. - it's a continuous effort to try and keep the compatibility working. Because the repository is such a quickly changing codebase any fixed compat issue is promptly replaced by new compat issues. B: Switch to a git tree and drop compatibility completely Pros: - makes driver development and v4l-dvb maintenance very easy. - no compatibility issues anymore, this saves a lot of time. - ability to change kernel code outside the driver/media part. - received patches against the latest git tree are easy to apply. Cons: - no compatibility means that the initial testbase will be reduced substantially since it will be too difficult for many users to install the bleeding-edge kernel. So real feedback won't come in until the code is released as part of the main distros kernels. - the same is true for ourselves: we need to continuously upgrade our kernel, which is not always an option. C: The ALSA model Pros: - ability to change kernel code outside the driver/media part. - the core code contains no compatibility code, thus making it very clean. - received patches against the latest git tree are easy to apply. - sending patches upstream is much easier. - if patches go in that would break compatibility, then that doesn't immediately affect the compatibility git tree since these patches aren't merged automatically. - the compatibility patches do not have to be that concerned about the mess they make of the code since the patched code will never go upstream. - for the compatibility tree we can reuse our existing compatibility mechanism. No need to reinvent the wheel. Cons: - someone has to go in on a regular basis and merge the latest changes into the compatibility tree and fix up everything that broke. This is a commitment that may be hard to keep up. - it is a fair amount of work to convert our model into this new model. - developers will probably have to run the latest kernel as well in order to test whether their changes can be compiled and run correctly on the latest git tree. This might be a Pro as well, since it will also allow you to catch bugs quicker. - if a change breaks compat in the compatibility tree, then that will make it hard for end users to test. You probably need to make a private copy of the compat tree just to fix that up, allowing end users to test your code. 1.2 Overall conclusion ---------------------- Looking at the options I think that the ALSA model is a very good candidate. Since it uses the standard git tree the maintenance of the repository when it comes to 'interfacing' with the Linus' tree is much simplified. Since getting fixes and new drivers into the mainline kernel is our prime task it is right that this part should be easy. I think it is also good that the compat tree is more reliable. Now our code breaks too often for older kernels due to new patches that were committed. Replacing this by a periodic merge-and-fix cycle will make it less fragile. And the fact that it isn't code that goes to the mainline kernel allows us to be less concerned about how ugly the patch will look like. We do need some way of assigning the fixing of broken compatibility to several people. It is not a task that should fall to only one or two persons. Whoever broke it should probably also fix it, in particular if that person was the maintainer of the driver as well. It's a matter of sharing the pain. 2 How many kernels to support? ------------------------------ Currently we go back three years to kernel 2.6.16. This is tested daily by running a compile on all versions from 2.6.16 (and on multiple architectures as well). While a successful compile doesn't mean that it will also run, it actually is a pretty good indicator. A quick check (v4l/scripts/gentree.pl 2.6.29 linux/ gentree; du -sb linux/drivers/ gentree/drivers/) shows that we have about 200 kB of compatibility code in the driver sources: 14249297 linux/drivers/ 14055008 gentree/drivers/ Keeping support for older kernels should come with an expiry date as at some point in time the effort involved outweighs the benefits. It also keeps the amount of compat code down and you know that particularly ugly patches can eventually be removed. We are not paid to maintain this code, and we have better things to do with our time than to keep the maintenance of an old kernel up. What are the benefits of having compatibility code? 1) It increases the test coverage: more users can try out the code since upgrading to a bleeding edge kernel is usually not an option. 2) It eases our own development efforts for pretty much the same reason: no need to upgrade to the latest git kernel to test your code. The drawbacks are obvious: 1) it's a lot of work to keep the compatibility working. 2) sometimes really large changes are made in the kernel that makes it very hard to keep support older kernels at all. Usually the changes are renamed data structures, fields that are removed/added or moved to another struct altogether. While annoying, it is usually not too hard to fix. However, in the particular case of the i2c model that was changed in kernel 2.6.22 (and that sparked this discussion), the changes are so intrusive that they could no longer be fixed with a simple #if KERNEL_VERSION. It would probably still be possible if we had only a handfew of i2c modules, but i2c is a integral part of the design of many devices and we currently have 50 of these to maintain (and more coming in all the time). At the moment keeping the compatibility alive for kernels older than 2.6.22 actually slows down development, both for us and for the i2c core. And due to the code needed to support these older kernels the upstream code is actually affected and it looks really weird in these newer kernels. 2.6.22 is now over 1.5 years old. It is readily available in all the major distros with the exception of the enterprise versions. I would like to propose some rules that we can use to decide how far we go in supporting older kernels. 1) As long as the three major distros do not yet support a certain kernel, we obviously have to keep compatibility code in. 2) Support for kernels older than the oldest supported kernel by the three major distros should be dropped. This explicitly excludes enterprise kernels (more about that below). 3) In case of *major* technical problems we should have the option to drop support for kernels older than the oldest kernel supported by the latest distro releases. This is as a last resort only, but in all the time I've been developing for v4l-dvb the i2c core changes are the first that would qualify for this rule. In my view enterprise distros are *not* our responsibility. First of all people pay for them, so if they want to have webcams supported they should go to their vendor. We are not paid for doing this, nor are we supported by the vendors of the distros. I am convinced that the group involved is small: I can't remember seeing any bug reports for kernels from enterprise distros here. And if there are a few, then it is such a small number that it really doesn't pay to accomodate them. I think it is also completely fair to expect that users of old unsupported non-enterprise distros should upgrade to something newer first. If only because for many developers it will can be hard or impossible to even install an old kernel on there machine anymore. Note that development model C would still make it possible to support enterprise kernels without compromising the code quality of the main tree. However, that will be a substantial amount of work. If RedHat or Suse thinks there is enough demand for that, then they are free to have one of their own developers to do the work. But that's not what we should spend our unpaid time on. Currently the oldest supported kernels are: Oldest supported Fedora kernel is 2.6.25 (Fedora 9): http://fedoraproject.org/wiki/Releases Oldest supported openSUSE kernel is 2.6.22 (10.3). I don't know when the end-of-life due date is for this openSUSE release. It might not be too long from now. (I can't find a decent webpage with the openSUSE releases, does anyone have one?) Oldest supported Ubuntu kernel is 2.6.22 (7.10): https://wiki.ubuntu.com/Releases. End of life for this one will be in April 2009, after that the oldest kernel for Ubuntu will be 2.6.27 (8.10). In my view these criteria strike a good balance between supporting our users so we can good test coverage, and limiting the effort involved in supporting the compatibility code. And they are also based on simple facts: whenever the oldest regularly supported kernel changes, we can go in and remove some of the compat code. No need for discussions, the rules are clear and consistent. And luckily, since the oldest kernel currently in regular use is 2.6.22 that makes a very good argument for dropping the i2c compatibility mess. In addition, limiting the number of supported kernels will ease the effort needed to switch development models, should we actually decide to do so. Comments? Hans -- Hans Verkuil - video4linux developer - sponsored by TANDBERG -- To unsubscribe from this list: send the line "unsubscribe linux-media" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
