Hey folks,
I'm trying to track down a BUG() that seems to strike a particular
system configuration (unfortunately, an increasingly common
configuration), but does so with 100% reliability.
The system in question is a Xen instance (6 vcpus, 32G memory) running
3.2 on essentially stock ubuntu (10.04) system.
if I run the attached program with the crash dir set to any ext3 or
ext4 file system with any audit rules installed, I get an oops on the
second time through the while loop:
kernel BUG at fs/buffer.c:1267!
invalid opcode: 0000 [#1] SMP
CPU 1
Pid: 4146, comm: a.out Not tainted 3.2.5-will-break-2-ganetixenu #4
RIP: e030:[<ffffffff81696a6c>] [<ffffffff81696a6c>] check_irqs_on.part
.10+0x17/0x19
RSP: e02b:ffff8807c7339bf8 EFLAGS: 00010096
RAX: 000000000000001e RBX: ffff8807970840b0 RCX: 00000000000000e7
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000004
RBP: ffff8807c7339bf8 R08: 0000000000000000 R09: 0000000000000018
R10: 0000000000006a5d R11: 0000000000000001 R12: 0000000000000400
R13: ffff8807dee05040 R14: ffff8807c7339dc0 R15: 0000000000000124
FS: 00007fe7cde057c0(0000) GS:ffff8807fff44000(0063) knlGS:00000000000
00000
CS: e033 DS: 002b ES: 002b CR0: 000000008005003b
CR2: 00000000f76dc4b0 CR3: 00000007a769a000 CR4: 0000000000002660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process a.out (pid: 4146, threadinfo ffff8807c7338000, task ffff8807ab3
496b0)
Stack:
ffff8807c7339c68 ffffffff81161dc9 ffff8807c7339c90 ffff8807b6b909f0
ffff8807ab23901a ffff8807c7339d60 ffff880700000000 ffff8807c7339d30
ffff8807c7339d60 ffff8807c7339e78 ffff8807970840b0 0000000000000400
Call Trace:
[<ffffffff81161dc9>] __find_get_block+0x1f9/0x200
[<ffffffff81164c8f>] __getblk+0x1f/0x280
[<ffffffff811f35bb>] __ext4_get_inode_loc+0x10b/0x410
[<ffffffff81124935>] ? kmem_cache_alloc+0xa5/0x150
[<ffffffff811f9857>] ? ext4_evict_inode+0x177/0x450
[<ffffffff811f4cc7>] ext4_get_inode_loc+0x17/0x20
[<ffffffff811f75a8>] ext4_reserve_inode_write+0x28/0xa0
[<ffffffff811f9815>] ? ext4_evict_inode+0x135/0x450
[<ffffffff811f7673>] ext4_mark_inode_dirty+0x53/0x200
[<ffffffff811f9857>] ext4_evict_inode+0x177/0x450
[<ffffffff8114bfb1>] evict+0xa1/0x1a0
[<ffffffff8114cc61>] iput+0x101/0x210
[<ffffffff81148040>] d_kill+0xf0/0x130
[<ffffffff81148bd2>] dput+0xd2/0x1b0
[<ffffffff8113eb85>] path_put+0x15/0x30
[<ffffffff81693e39>] audit_free_names+0x96/0xb5
[<ffffffff810ac629>] audit_syscall_exit+0x139/0x1e0
[<ffffffff816a076a>] sysexit_audit+0x21/0x5f
Code: 5c 48 89 df e8 b6 20 ab ff 5b 41 5c 5d c3 55 48 89 e5 0f 0b 55 be
08 00 00 00 48 c7 c7 c4 fe a0 81 31 c0 48 89 e5 e8 91 cb ff ff <0f>
0b 55 48 89 e5 0f
0b 55 48 89 e5 0f 0b 55 48 89 e5 0f 0b 55
RIP [<ffffffff81696a6c>] check_irqs_on.part.10+0x17/0x19
RSP <ffff8807c7339bf8>
line 1267 of fs/buffer.c is
static inline void check_irqs_on(void)
{
#ifdef irqs_disabled
BUG_ON(irqs_disabled());
#endif
}
If I run the same code on the same system with the same audit rule(s)
on an ext2 filesystem, I get no such oops.
So it seems like something either in the ext3/ext4 or Xen codepath is
disabling interrupts. I'm getting an updated test Xen instance to test
on, but I while I'm waiting on that, I wanted to see if anyone one
here might have an idea of the ext3/4 codepath. whether something
there is doing the interrupt disabling or if there might be some other
race condition going on. I haven't had a chance to test with the large
"ext4 updates for v3.7" tytso recently posted, but I'll be doing that
later today in case something there fixes this.
So, does any one have any thoughts and/or pointers which might help me
get to the bottom of this?
Cheers,
peter
--
Peter Moody Google 1.650.253.7306
Security Engineer pgp:0xC3410038
/*
* steps:
* 1) compile with gcc -m32
* 2) start auditd, install any rule.
* /etc/init.d/auditd start ; auditctl -D ; auditctl -a exit,always -F arch=b64 -S chmod
* 3) run'n wait (this only loops twice for me before dying)
* ./a.out
* 4) bask in instantaneous kernel bugs.
[ 571.282777] ------------[ cut here ]------------
[ 571.282786] kernel BUG at fs/buffer.c:1263!
[ 571.282790] invalid opcode: 0000 [#1] SMP
[ 571.282795] last sysfs file: /sys/devices/system/cpu/sched_mc_power_savings
[ 571.282798] CPU 0
[ 571.282802] Pid: 7457, comm: a.out Not tainted 2.6.38.8-gg868-ganetixenu #1
[ 571.282808] RIP: e030:[<ffffffff81153853>] [<ffffffff81153853>] __find_get_block+0x1f3/0x200
[ 571.282819] RSP: e02b:ffff88079b7ddc78 EFLAGS: 00010046
[ 571.282822] RAX: ffff8807bc290000 RBX: ffff8806d9bb9a98 RCX: 00000000023dc17c
[ 571.282826] RDX: 0000000000001000 RSI: 00000000023dc17c RDI: ffff8807fec29a00
[ 571.282830] RBP: ffff88079b7ddcd8 R08: 0000000000000001 R09: ffff8806d9bb99c0
[ 571.282834] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8806d9bb99c4
[ 571.282839] R13: ffff8806d9bb99f0 R14: ffff8807feff9060 R15: 00000000023dc17c
[ 571.282845] FS: 00007f8f6a76a7c0(0000) GS:ffff8807fff26000(0063) knlGS:0000000000000000
[ 571.282849] CS: e033 DS: 002b ES: 002b CR0: 000000008005003b
[ 571.282853] CR2: 00000000f76c6970 CR3: 00000007a250b000 CR4: 0000000000002660
[ 571.282857] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 571.282861] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 571.282866] Process a.out (pid: 7457, threadinfo ffff88079b7dc000, task ffff8807786843e0)
[ 571.282870] Stack:
[ 571.282872] ffff88079b7ddc98 ffffffff81654cd1 ffff88079b7ddca8 ffff8806d9bba440
[ 571.282879] ffff88079b7ddd08 ffffffff811c9294 ffff8807ffffffc3 0000000000000014
[ 571.282887] ffff8806d9bb9a98 ffff8806d9bb99c4 ffff8806d9bb99f0 ffff8807feff9060
[ 571.282895] Call Trace:
[ 571.282901] [<ffffffff81654cd1>] ? down_read+0x11/0x30
[ 571.282907] [<ffffffff811c9294>] ? ext3_xattr_get+0xf4/0x2b0
[ 571.282913] [<ffffffff811baf88>] ext3_clear_blocks+0x128/0x190
[ 571.282918] [<ffffffff811bb104>] ext3_free_data+0x114/0x160
[ 571.282923] [<ffffffff811bbc0a>] ext3_truncate+0x87a/0x950
[ 571.282928] [<ffffffff812133f5>] ? journal_start+0xb5/0x100
[ 571.282933] [<ffffffff811bc840>] ext3_evict_inode+0x180/0x1a0
[ 571.282938] [<ffffffff8114065f>] evict+0x1f/0xb0
[ 571.282945] [<ffffffff81006d52>] ? check_events+0x12/0x20
[ 571.282949] [<ffffffff81140c14>] iput+0x1a4/0x290
[ 571.282955] [<ffffffff8113ed05>] dput+0x265/0x310
[ 571.282959] [<ffffffff81132435>] path_put+0x15/0x30
[ 571.282965] [<ffffffff810a5d31>] audit_syscall_exit+0x171/0x260
[ 571.282971] [<ffffffff8103ed9a>] sysexit_audit+0x21/0x5f
[ 571.282974] Code: 82 00 05 01 00 85 c0 75 de 65 48 89 1c 25 00 05 01 00 e9 87 fe ff ff 48 89 df e8 e9 fc ff ff 4c 89 f7 e9 02 ff ff ff 0f 0b eb fe <0f> 0b eb fe 0f 0b eb fe 0f 1f 44 00 00 55 48 89 e5 41 57 49 89
[ 571.283027] RIP [<ffffffff81153853>] __find_get_block+0x1f3/0x200
[ 571.283033] RSP <ffff88079b7ddc78>
[ 571.283036] ---[ end trace 5975ffe20808ecd2 ]---
*
*/
#include <stdio.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <unistd.h>
#define KILLDIR "/tmp/kill_dir/a"
int main(int argc, char **argv) {
FILE *f;
char fullpath[512], *kdir = KILLDIR;
int i = 0;
if (argc > 1) {
kdir = argv[1];
}
fprintf(stderr, "crashing in %s\n", kdir);
while (1) {
fprintf(stderr, "%d ", i++);
mkdir(kdir, 0777);
chdir(kdir);
sprintf(fullpath, "%s/file", kdir);
f = fopen(fullpath, "w+");
fprintf(f, "nothing to see here");
fclose(f);
unlink(fullpath);
rmdir(kdir);
if (i > 100)
break;
}
return 0;
}
