2016-08-18 19:55 GMT+02:00 Luis R. Rodriguez <mcgrof@xxxxxxxxxx>: > On Wed, Aug 17, 2016 at 09:26:58PM +0200, Cristina Moraru wrote: > >> This patchset implements dynamic pegging of kconfig symbol >> into driver modinfo section > > First a little bit of motivation here helps, so let me try to > help fill in some gaps. This may help explain what you have > been working on a bit more. > > First, for those that were not Cc'd but curious about this work > so far, you can read the patches here: > > Original cover letter: > > https://lkml.kernel.org/r/1471462023-119645-1-git-send-email-cristina.moraru09@xxxxxxxxx > > https://marc.info/?l=linux-kernel&m=147146213519750&w=2 - patch 1 > https://marc.info/?l=linux-kernel&m=147146209019744&w=2 - patch 2 > https://marc.info/?l=linux-kernel&m=147146211819747&w=2 - patch 3 > https://marc.info/?l=linux-kernel&m=147146209119745&w=2 - patch 4 > https://marc.info/?l=linux-kernel&m=147146209519746&w=2 - patch 5 > > There are a few situations in which you may want to extract a > subset of Kconfig CONFIG_* symbols for a system. At least when > only considering modules: > > a) When optimizing build requirements for a kernel for a system. > That is you boot into a distro kernel and then want to build > a slim kernel only with sensible kernel configuration options. > > b) When you are on a distribution kernel but the distribution > kernel provided lacks hardware support for your device, you > may either want to upgrade the full kernel in which case you > want to do a) or -- you may want to just a backports release > which provides just the modules you need, you'd use it on top > of the distribution kernel. > > Are there other *uses* outside of the two above for having a direct > deterministic mapping between module and a kconfig symbol ? What > if we could also do this for built-in code as well ? kconfig-sat [0] is > work that is still being ironed out, but if that was in place and we > handed it a set of required options we want, it should in theory help > resolve the dependencies to give us a functional .config for a kernel. > This of course is a long term possible goal just used as an example > -- but are there other uses? > I can't think of any examples. I will add this ones in the cover letter as motivation. > Another possible gain here is if addressing a deterministic mapping might > help with some of the limitations observed in practice over years with > kconfig and semantics for keconfig. I go into some of this further below. > > There are not many options currently available to optimize kernel builds this > way for modules or choose only what kernel kconfig options you need. One of > the best known efforts is the upstream 'make localmodconfig' added by Steven > years ago to help with a). This has its own set of limitations though. For > backports we also need to address possible kconfig symbol name changes, and the > fact that you may boot a box without a driver loaded and as such not be sure > what new kernel kconfig symbol you may need. > > There are shared issues with both goals, addressing shared issues > can ultimately help both. One of the shared known issues is the > lack of a deterministic resolution of a module to a respective > kconfig symbol. Even answering if this is possible or should be > possible requires some work. > > The work Cristina has done here for this year's Google Summer of > Code is try to help start looking into addressing the question if > this is possible and then start looking into how we could annotate > such a thing, and later how we'd use it. > > [0] https://kernelnewbies.org/KernelProjects/kconfig-sat > >> * updates streamline_config.pl to generate the auxiliary file >> scripts/mod/Module.ksymb containing associations of driver file >> names and corresponding kconfig symbols CONFIG_* > > Here you make use of existing heuristics -- note however that > heuristics are needed as we have no direct mapping available > nor do we have any rules over how to make such a direct mapping > possible if we wanted this mapping to be deterministic. > >> * updates Makefiles to trigger streamline_config.pl call for >> Module.ksymb generation and pass the determined CONFIG_* symbol >> as compilation parameter -D in KBUILD_KSYMB variable > > Passing -D for define a new KBUILD_KSYMB when available would be a way to pass > to the build system a config symbol associated with a module, its the right > approach, however re-using streamline_config.pl seems questionable here, and it > seems an optimization might be / should be possible. Likewise we'd need to also > still look into and address *why* some modules don't end up with a respective > kconfig symbol -- and if we wanted to do something about this. Given the stated > goals more importantly would be what the costs are to have a deterministic > mapping. > Yes. I've noticed. I will change it completely. >> * adds kconfig_symb as module attribute > > This seems lightweight and follows the logic of the other attributes. > If its too much, this can be a configurable feature. > >> * updates modpost to set KBUILD_KSYMB macro as value for >> kbuild_symb attribute > > Neat. > >> Note: the content of the file Module.ksymb is generated for >> all modules that have only one associate CONFIG option. All >> others are considered to be components linked at the final >> modules but not final modules itselves. > > To be clear, you only peg the CONFIG_ option if you are > 100% certain of having a direct mapping, is that right ? > Yes. And there are some drivers which have duplicate values in the hash. foo.o -> CONFIG_FOO CONFIG_FOO Although this one has one match it's filtered as having 2 CONFIGs. >> The result of this patchset is the following. All modules from >> /sys expose the correct CONFIG_* symbol in the module attribute >> kconfig_symb with some exceptions. For a total number of 58 >> modules, > > What kernel configuration did you use ? What architecture ? > If you use allmodconfig, what do the numbers look like ? > I have a virtual machine in which I created the configuration with make localmodconfig and installed the new kernel. $ uname -a Linux ubuntu 4.7.0+ #28 SMP Mon Aug 15 20:39:19 CEST 2016 x86_64 x86_64 x86_64 GNU/Linux Should I send you the .config file ? >> 4 of them do not: >> >> snd_seq_midi_event > > Takashi, any reason for this to not have a Kconfig symbol ? > If we don't want one, is there perhaps any annotation we > could use to help make a deterministic association if a > respective kconfig symbol is needed ? > >> mptscsih > > Same thing -- for both I suppose we can infer that these > are just helpers, and can / should be enabled if any of > the devices are found to be present / required. > > I can imagine that in certain situations a helper like > the above might really just be optional rather than required, > but I'd hope such an optional nature should be reflected via > Kconfig, the default setting being the recommended option to > assume as safe then, unless input is provided to help fill > in the gap and disable / force enable certain options. > >> libahci > > In this case libahci is spinkled over a lot of lines on the Makefile, > its a module though, and there is no direct association of dependency > relationship other than the Makefile association. Using a kconfig > symbol would replace the Makefile clutter but make the assocation > and requirement of the dependency explicit through Kconfig. Is > that desirable or should we just make the assumption that in these > cases we could assume one Kconfig symbol that adds a module implicitly > is equivalent as if the module had a kconfig symbol and we had done > "select foo" for each ? Note though that select is heavy handed [1] -- > and its over use can lead to issues. For instance its known that > select does not check for dependencies, its a heavy handed way of > enabling options needed by architectures. So using select here may > not be desirable. Using "depends on" here would work but it can hide > from the menu drivers until you have enabled its dependency. > > [1] https://kernelnewbies.org/KernelProjects/kconfig-sat#head-d1734174081ec7a1612fb29277bfc850d82ba31e > >> mptbase > > Same situation as mptscsih. > >> After a short research: >> >> For mptscsih - ./drivers/message/fusion/Makefile >> obj-$(CONFIG_FUSION_SPI) += mptbase.o mptscsih.o mptspi.o >> obj-$(CONFIG_FUSION_FC) += mptbase.o mptscsih.o mptfc.o >> obj-$(CONFIG_FUSION_SAS) += mptbase.o mptscsih.o mptsas.o >> As appears in the Makefile mptscsi is part of more config >> options so it's excluded because it's not considered a module >> by itself but, somehow gets compiled as module. Same for the >> mptbase. I still have to understand better the Kbuild logic so >> I cannot state that this is loose practice. > > Another reason for this loose use could also be that using "select" > in Kconfig is heavy handed and can lead to recursive issues, refer > to commit 1c199f2878f6c1b8c52125ad9805e94fe2dde472 ("kbuild: document recursive > dependency limitation / resolution") for more information and > a clear example of the issue. There is also a list of commits > with possible resolutions to issues, so that can be inspected further > to try to understand if perhaps some of the lack of kconfig symbols > and respective deterministic mapping from module to kconfig is due > to prior issues with select. I doubt it given that otherwise we'd > have at least symbols for these -- but worth taking into consideration > *if* we do then want to add symbols for them, we do not want to lead > to recursive issues if we add these new symbols. > > If we add a kconfig symbol for modules lacking them, what is a > strategical way to do so in such a way we avoid known past > issues but also paves the way for the future ? > >> For libahci there are multiple CONFIG_ symbols for different >> architectures. >> >> Although past discussion recommended not to use streamline_config, >> this patchset still relies on it, for the sake of the proof of >> concept. > > OK. > >> Although the result is close to the target it still does >> not provide complete correctness. However it can be replaced by >> creating another script > > Or C code. > >> which tries to determine better the >> module <-> CONFIG associations and output them in auxiliary file >> Module.ksymb. Maybe this way we could also determine all CONFIGs >> for a particular driver, not only the exact one that enables it. >> >> The auxiliary file is necessary because the Makefile itself does >> not have a mapping. > > Clarification: at least not an easy O(1) quick mapping. > >> The makefile includes the config file with >> directive include which creates a series of internal variables >> CONFIG_FOO=y/m/n. Afterwards, when recursively descending into >> other Makefiles, lines like >> >> obj-$(CONFIG_FOO) = foo.o >> >> are resolved in place to obj-y, obj-m etc according to 'make' >> logic and the association is lost. >> >> Finally, what does this patchset provide is an infrastructure >> to dinamically peg CONFIG_* options to associate drivers using >> the mapping from Module.ksymb file. Generation of Module.ksymb >> can be replaced but keeping the same format permit the usage of >> the other patches. > > OK this makes sense, so as it stands it seems we need a better > way to generate the Module.ksymb and then also address those > modules which lack a current deterministic mapping, and address > a resolution with the community on that. > >> This patchset is part of a research project within >> Google Summer of Code of porting 'make localmodconfig' >> for backported drivers. The goal is to enable each >> module to expose in /sys its corresponding CONFIG_* option. >> The value of this attribute will be dynamically pegged by >> modpost without requiring extra work from the driver developers. >> Further, this information will be used by a hardware interogation >> tool to extract build information about the existing devices. > > Good stuff! > > Luis > >> Cristina Moraru (5): >> Add generation of Module.symb in streamline_config >> Add CONFIG symbol to module as compilation parameter >> Trigger Module.ksymb generation in Makefile >> Set KCONFIG_KSYMB as value for kconfig_ksymb module attribute >> Add kconf_symb as kernel module attribute >> >> Makefile | 4 ++++ >> include/linux/module.h | 1 + >> kernel/module.c | 2 ++ >> scripts/Makefile.lib | 9 ++++++++- >> scripts/kconfig/streamline_config.pl | 30 +++++++++++++++++++++++++++++- >> scripts/mod/modpost.c | 7 +++++++ >> 6 files changed, 51 insertions(+), 2 deletions(-) >> >> -- >> 2.7.4 >> >> -- To unsubscribe from this list: send the line "unsubscribe backports" in
