Hello,
I've continued to investigate this issue, and got back-traces from the stuck processes (checked into the git
repository as dmesg.{1,2,3}.
From these it looks like the problem is an fanotify_read is itself calling into fanotify_handle_event via ecryptfs_get_lower_file:
[ 1197.922036] [<ffffffff8161366f>] schedule+0x3f/0x60
[ 1197.922042] [<ffffffff811b28b9>] fanotify_handle_event+0x119/0x160
[ 1197.922046] [<ffffffff81089180>] ? add_wait_queue+0x60/0x60
[ 1197.922050] [<ffffffff811aedda>] send_to_group.isra.1+0x13a/0x190
[ 1197.922055] [<ffffffff81183e80>] ? poll_freewait+0xe0/0xe0
[ 1197.922059] [<ffffffff811aef93>] fsnotify+0x163/0x2a0
[ 1197.922065] [<ffffffff81298229>] security_dentry_open+0x79/0x80
[ 1197.922068] [<ffffffff8116ed8c>] __dentry_open+0x10c/0x360
[ 1197.922073] [<ffffffff812d186c>] ? apparmor_file_alloc_security+0x2c/0x60
[ 1197.922076] [<ffffffff8116f58d>] vfs_open+0x3d/0x40
[ 1197.922080] [<ffffffff8116f5de>] dentry_open+0x4e/0x70
[ 1197.922084] [<ffffffff811b2900>] ? fanotify_handle_event+0x160/0x160
[ 1197.922089] [<ffffffff81271805>] ecryptfs_privileged_open+0x85/0x290
[ 1197.922093] [<ffffffff8126830d>] ecryptfs_get_lower_file+0xad/0x120
[ 1197.922097] [<ffffffff81265590>] ecryptfs_open+0xa0/0x2a0
[ 1197.922100] [<ffffffff8116ef10>] __dentry_open+0x290/0x360
[ 1197.922104] [<ffffffff812654f0>] ? ecryptfs_release+0x40/0x40
[ 1197.922108] [<ffffffff812d186c>] ? apparmor_file_alloc_security+0x2c/0x60
[ 1197.922111] [<ffffffff8116f58d>] vfs_open+0x3d/0x40
[ 1197.922115] [<ffffffff8116f5de>] dentry_open+0x4e/0x70
[ 1197.922118] [<ffffffff811b3246>] fanotify_read+0x2c6/0x510
[ 1197.922122] [<ffffffff81089180>] ? add_wait_queue+0x60/0x60
[ 1197.922126] [<ffffffff811713b0>] vfs_read+0xb0/0x180
[ 1197.922129] [<ffffffff811714ca>] sys_read+0x4a/0x90
[ 1197.922133] [<ffffffff810b3ee5>] ? compat_sys_time+0x25/0x60
[ 1197.922137] [<ffffffff8161fb70>] sysenter_dispatch+0x7/0x2e
However I have not been able to reproduce the problem once I applied the attached patch, which ensures that
only one file access in included in each fanotify_read event. With this patch I was able to run firefox 5 times, through the 2
minute test cycle of the test code, without causing a hang.
What do you think?
-----Original Message-----
From: Douglas Leeder [mailto:douglas.leeder@xxxxxxxxxx]
Sent: Thursday, August 09, 2012 2:33 PM
To: malware-list@xxxxxxxxxxxxxxxx; ecryptfs@xxxxxxxxxxxxxxx
Cc: douglas.leeder@xxxxxxxxx; dustin.kirkland@xxxxxxxxxxx; Eric Paris
Subject: Process/Scanner hang with ecryptfs and fanotify
Hello,
We are in the final stages of producing a version of Sophos Anti-Virus for Linux that uses fanotify, and have hit an issue with fanotify against ecryptfs mounts.
ecryptfs is used for encrypted home directories on (at least) Ubuntu.
This issue I'm seeing is on Ubuntu 12.04 with encrypted home directories. (3.2 kernel)
After using Firefox for a short while with fanotify scanning happening (for both the host fs, and the ecryptfs mount), it will go grey (or gray :-)), signalling it has hung.
Two threads in Firefox with be in the D state (IO sleep), and two of the scanner threads will be stuck reading from the fanotify fd.
These remain stuck until the machine is rebooted.
I've got a test 'scanner' that I've managed to reproduce the issue with:
github:
https://github.com/paperclip/fanotify-ecryptfs-test
(If anyone wants this tarred up, I can send it directly - but I can't send it to the mailing list due to size limits)
However it doesn't happen every time, and I've not reproduced it on a VM.
Points on note:
1) Every file on ecryptfs is backed by a file on the host fs
2) Thus every access to an ecryptfs also generates a file access to the host fs.
3) fanotify can deliver multiple file accesses in a single event.
Given the unreliable nature of the reproduction steps; and the fact that
when things hang it is always 2 threads, rather than 1 or 3 that hang: I
believe that there is some kind of race condition when more than one
file on an ecryptfs mount is accessed at the same time.
I wonder if each fanotify event contains the ecryptfs access from one
client thread, and the host fs access from the other?
In the git repository there are two dmesg outputs, from the stuck
threads' backtraces. Definitely stuck between fanotify and ecryptfs.
Is there any advice you can offer? I wonder if fanotify needs to deliver
permission events one at a time? Or can you point me at appropriate
sources to look at for further information.
In an ideal world there would be some fix to the scanner to avoid the
problem, but I'm not sure what it could be, so the solution might only
be applicable to future kernels.
(Sophos defect DEF83746)
Thanks.
--
Douglas Leeder, Senior Software Engineer
________________________________
Sophos Limited, The Pentagon, Abingdon Science Park, Abingdon, OX14 3YP, United Kingdom.
Company Reg No 2096520. VAT Reg No GB 991 2418 08.
Attachment:
0001-Make-fanotify-events-only-contain-one-file-access-if.patch
Description: 0001-Make-fanotify-events-only-contain-one-file-access-if.patch
