can someone explain how the kernel source, header files and /lib/modules directories work together?
My impression is that:
Your kernel should be compiled as a non-root user, in a directory that they have write access to (perhaps their own home dir), and then installed by the root user.
Modules and a symlink to the build tree are stored in /lib/modules/<version>. Third party kernel modules should look in that symlink to find the kernel's .config file and the headers that they need.
p.s. the above is in aid of a current discussion on the FHS mailing list regarding the appropriate location for the kernel source/build directory, and why /usr/src is not a great idea.
/usr should be able to be mounted read-only, so it's not a good idea to build kernels there. However, it's OK for a distribution to store a read-only copy of the kernel source and headers in that location for the reference of third party modules. The "build" symlink in the distributed modules directory should point to whatever location the distribution used for its kernel source.
