On 13/08/12 18:17, Dustin Kirkland wrote: > Howdy Douglas, > > Responses are inline below... > > On Wed, Aug 8, 2012 at 7:57 AM, Douglas Leeder > <douglas.leeder@xxxxxxxxxx> wrote: >> Hi Dustin, Eric; >> >> We are in the final stages of producing a version of Sophos >> Anti-Virus for Linux that uses fanotify, and have hit an issue with >> fanotify against ecryptfs mounts. > > Ah, interesting. Okay, so I've worked with Sophos in the past, to > add eCryptfs to the do-not-scan-at-mount list for talpa. I'm not > familiar with fanotify -- is it a replacement for what Sophos was > previously using talpa to do (watch filesystem events)? Yes, fanotify is the built-in kernel code, that allows user space to monitor file events for entire filesystems, similar to inotify/dnotify except for entire filesystems. http://lwn.net/Articles/339253/ It is an alternative to Talpa, for newer kernels. > >> Eric is the person who did the vast majority of the work in getting >> fanotify into the kernel. Dustin is the upstream developer for >> ecryptfs, as well as an Ubuntu core developer. >> >> (At least that was the case at my last contact - if there are more >> appropriate people to contact then please forward this on). > > Right, I am the eCryptfs userspace maintainer. I have CC'd Tyler > Hicks, who is the eCryptfs kernel module maintainer. > >> ecryptfs is used for encrypted home directories on (at least) >> Ubuntu. > > That's one place. eCryptfs is also used in Debian, Arch, Fedora, > CentOS, RHEL, Google's ChromeOS, Seagate's NAS, Motorola Android > phones, and Gazzang's encryption products. Indeed, that is why I added the "(at least)" :-) > >> This issue I'm seeing is on Ubuntu 12.04 with encrypted home >> directories. (3.2 kernel) >> >> After using Firefox for a short while with fanotify scanning >> happening (for both the host fs, and the ecryptfs mount), it will >> go grey (or gray :-)), signalling it has hung. >> >> Two threads in Firefox with be in the D state (IO sleep), and two >> of the scanner threads will be stuck reading from the fanotify fd. >> These remain stuck until the machine is rebooted. >> >> I've got a test 'scanner' that I've managed to reproduce the issue >> with: >> >> Attached tarfile: fanotify.ecrypyfs.tgz >> >> github: https://github.com/paperclip/fanotify-ecryptfs-test >> >> >> However it doesn't happen every time, and I've not reproduced it on >> a VM. >> >> Points on note: 1) Every file on ecryptfs is backed by a file on >> the host fs 2) Thus every access to an ecryptfs also generates a >> file access to the host fs. 3) fanotify can deliver multiple file >> accesses in a single event. > > What do you want to scan? The lower, encrypted files, or the upper, > decrypted (cleartext) ones? So the problem is that fanotify works on a filesystem level, so we scan the ecryptfs to actually scan the clear version of the files, and get the encrypted version as a side-effect of scanning the root/parent filesystem. (Since ecryptfs encrypted files are normally stored on the normal parent fs) We can't usefully scan the encrypted files, but can't exclude them. > >> Given the unreliable nature of the reproduction steps; and the fact >> that when things hang it is always 2 threads, rather than 1 or 3 >> that hang: I believe that there is some kind of race condition when >> more than one file on an ecryptfs mount is accessed at the same >> time. > > Tyler, you've looked a few race conditions recently... Does this > one ring a bell? I think it must be a race condition, at least in part, since it doesn't happen every time, and at a different point each time, and simple file access is fine. > >> I wonder if each fanotify event contains the ecryptfs access from >> one client thread, and the host fs access from the other? >> >> Is there any advice you can offer? I wonder if fanotify needs to >> deliver permission events one at a time? Or can you point me at >> appropriate sources to look at for further information. >> >> In an ideal world there would be some fix to the scanner to avoid >> the problem, but I'm not sure what it could be, so the solution >> might only be applicable to future kernels. > > Interesting. Perhaps an I/O synchronicity problem. I don't know > much about Sophos or fanotify, but at a high level, I would think > that Sophos would need to watch the upper, clear text files right? > Those clear text files, of course, don't actually exist as clear text > files. They're decrypted and rendered in memory on demand. Since > eCryptfs is a layered filesystem, perhaps fanotify might need to > handle things a little differently, watching events only on the lower > filesystem? > So as, I sent later, making fanotify only return one file event per fanotify read(), So that suggests that the request is getting stuck in a cycle or adding a request. Florian has also been looking at this problem, and has pointed out that FMODE_NONOTIFY should prevent the loop seen in the backtrace, so I'm wondering if it's getting lost somewhere in ecryptfs. Or perhaps it needs to be explicitly propagated to the open request for the lower file? Thanks. -- Douglas Leeder, Senior Software Engineer ________________________________ Sophos Limited, The Pentagon, Abingdon Science Park, Abingdon, OX14 3YP, United Kingdom. Company Reg No 2096520. VAT Reg No GB 991 2418 08. ��.n��������+%������w��{.n����r����y� ʇڙ���f���h������_�(�階�ݢj"��������G����?���&��
