Hi all,
I just had a weird expirience ... In a server system, running a free webmail
service, i started getting weird oopsen.
On a quad p2/xeon box, 2gb ram, running rehdat 6.2 with sendmail 8.10 (patched
with latest stuff), cyrus 2.0.16, openldap 1.2 and sasl 1.5, with a custom
2.4.19-pre10 aa something kernel, it looked like this:
ksymoops 2.4.4 on i686 2.4.19-pre10. Options used
-V (default)
-k /proc/ksyms (default)
-l /proc/modules (default)
-o /lib/modules/2.4.19-pre10/ (default)
-m /boot/System.map-2.4.19-pre10 (default)
Warning: You did not tell me where to find symbol information. I will
assume that the log matches the kernel and modules that are running
right now and I'll use the default options above for symbol resolution.
If the current kernel and/or modules do not match the log, you can get
more accurate output by telling me the kernel version and where to find
map, modules, ksyms etc. ksymoops -h explains the options.
Warning (compare_maps): ksyms_base symbol set_cpus_allowed_R__ver_set_cpus_allowed not found in System.map.
Ignoring ksyms_base entry
Apr 7 12:55:23 castor kernel: de08d7db
Apr 7 12:55:23 castor kernel: *pde = 00000000
Apr 7 12:55:23 castor kernel: Oops: 0000
Apr 7 12:55:23 castor kernel: CPU: 0
Apr 7 12:55:23 castor kernel: EIP: 0010:[<de08d7db>] Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
Apr 7 12:55:23 castor kernel: EFLAGS: 00010286
Apr 7 12:55:23 castor kernel: eax: bfffff14 ebx: ec3f6000 ecx: 00000000 edx: 00000000
Apr 7 12:55:23 castor kernel: esi: c0108efb edi: 0000000b ebp: ec3f7fb8 esp: ec3f7f80
Apr 7 12:55:23 castor kernel: ds: 0018 es: 0018 ss: 0018
Apr 7 12:55:23 castor kernel: Process mail2sms (pid: 13314, stackpage=ec3f7000)
Apr 7 12:55:23 castor kernel: Stack: ec3f6000 c0108efb 0000000b 00000296 00000000 d8d23000 0000000b 00000296
Apr 7 12:55:23 castor kernel: d8d23000 bfffff14 c0107a5f 00000000 00000a3a 00000020 bffffd28 de08d9c4
Apr 7 12:55:23 castor kernel: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Apr 7 12:55:23 castor kernel: Call Trace: [system_call+47/52] [sys_execve+79/92]
Apr 7 12:55:23 castor kernel: Code: 8b 42 04 83 f8 ff 0f 84 69 01 00 00 83 f8 fc 77 07 c7 42 04
>>EIP; de08d7db <END_OF_CODE+1abe12e0/????> <=====
Code; de08d7db <END_OF_CODE+1abe12e0/????>
00000000 <_EIP>:
Code; de08d7db <END_OF_CODE+1abe12e0/????> <=====
0: 8b 42 04 mov 0x4(%edx),%eax <=====
Code; de08d7de <END_OF_CODE+1abe12e3/????>
3: 83 f8 ff cmp $0xffffffff,%eax
Code; de08d7e1 <END_OF_CODE+1abe12e6/????>
6: 0f 84 69 01 00 00 je 175 <_EIP+0x175> de08d950 <END_OF_CODE+1abe1455/????>
Code; de08d7e7 <END_OF_CODE+1abe12ec/????>
c: 83 f8 fc cmp $0xfffffffc,%eax
Code; de08d7ea <END_OF_CODE+1abe12ef/????>
f: 77 07 ja 18 <_EIP+0x18> de08d7f3 <END_OF_CODE+1abe12f8/????>
Code; de08d7ec <END_OF_CODE+1abe12f1/????>
11: c7 42 04 00 00 00 00 movl $0x0,0x4(%edx)
Apr 7 12:55:23 castor kernel: <1>Unable to handle kernel NULL pointer dereference at virtual address 00000004
Apr 7 12:55:23 castor kernel: de08d7db
Apr 7 12:55:23 castor kernel: *pde = 00000000
Apr 7 12:55:23 castor kernel: Oops: 0000
Apr 7 12:55:23 castor kernel: CPU: 2
Apr 7 12:55:23 castor kernel: EIP: 0010:[<de08d7db>] Not tainted
Apr 7 12:55:23 castor kernel: EFLAGS: 00010286
Apr 7 12:55:23 castor kernel: eax: bffffee4 ebx: c0846000 ecx: 00000000 edx: 00000000
Apr 7 12:55:23 castor kernel: esi: c0108efb edi: 0000000b ebp: c0847fb8 esp: c0847f80
Apr 7 12:55:23 castor kernel: ds: 0018 es: 0018 ss: 0018
Apr 7 12:55:23 castor kernel: Process sendmail (pid: 13313, stackpage=c0847000)
Apr 7 12:55:23 castor kernel: Stack: c0846000 c0108efb 0000000b 00000296 00000000 e9a6f000 0000000b 00000296
Apr 7 12:55:23 castor kernel: e9a6f000 bffffee4 c0107a5f 00000000 00000a3a 00000020 bffffb90 de08d9c4
Apr 7 12:55:23 castor kernel: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Apr 7 12:55:23 castor kernel: Call Trace: [system_call+47/52] [sys_execve+79/92]
Apr 7 12:55:23 castor kernel: Code: 8b 42 04 83 f8 ff 0f 84 69 01 00 00 83 f8 fc 77 07 c7 42 04
>>EIP; de08d7db <END_OF_CODE+1abe12e0/????> <=====
Code; de08d7db <END_OF_CODE+1abe12e0/????>
00000000 <_EIP>:
Code; de08d7db <END_OF_CODE+1abe12e0/????> <=====
0: 8b 42 04 mov 0x4(%edx),%eax <=====
Code; de08d7de <END_OF_CODE+1abe12e3/????>
3: 83 f8 ff cmp $0xffffffff,%eax
Code; de08d7e1 <END_OF_CODE+1abe12e6/????>
6: 0f 84 69 01 00 00 je 175 <_EIP+0x175> de08d950 <END_OF_CODE+1abe1455/????>
Code; de08d7e7 <END_OF_CODE+1abe12ec/????>
c: 83 f8 fc cmp $0xfffffffc,%eax
Code; de08d7ea <END_OF_CODE+1abe12ef/????>
f: 77 07 ja 18 <_EIP+0x18> de08d7f3 <END_OF_CODE+1abe12f8/????>
Code; de08d7ec <END_OF_CODE+1abe12f1/????>
11: c7 42 04 00 00 00 00 movl $0x0,0x4(%edx)
2 warnings issued. Results may not be reliable.
There were two oopsen at 12:55 and 41 from 15:00:01 to 15:01:32 localtime .
The machine appears to be running fine.
The other box, dual xeon 2.0ghz (HT enabled), 4gb ram, running 7.3 with
kernel 2.4.18-26 bigmem, started returning segmentation fault for every
command i wanted to run at about 15:15. It is running apache 1.3.26, php
4.3.1 and postfix 1.11. I was lucky to have two ssh sessions opened to it
before this started. The oops looked like this:
ksymoops 2.4.4 on i686 2.4.18-26.7.xbigmem. Options used
-V (default)
-k /proc/ksyms (default)
-l /proc/modules (default)
-o /lib/modules/2.4.18-26.7.xbigmem/ (default)
-m /boot/System.map-2.4.18-26.7.xbigmem (default)
Warning: You did not tell me where to find symbol information. I will
assume that the log matches the kernel and modules that are running
right now and I'll use the default options above for symbol resolution.
If the current kernel and/or modules do not match the log, you can get
more accurate output by telling me the kernel version and where to find
map, modules, ksyms etc. ksymoops -h explains the options.
Error (expand_objects): cannot stat(/lib/ext3.o) for ext3
ksymoops: No such file or directory
Error (expand_objects): cannot stat(/lib/jbd.o) for jbd
ksymoops: No such file or directory
Error (expand_objects): cannot stat(/lib/raid1.o) for raid1
ksymoops: No such file or directory
Error (expand_objects): cannot stat(/lib/aic7xxx.o) for aic7xxx
ksymoops: No such file or directory
Error (expand_objects): cannot stat(/lib/sd_mod.o) for sd_mod
ksymoops: No such file or directory
Error (expand_objects): cannot stat(/lib/scsi_mod.o) for scsi_mod
ksymoops: No such file or directory
/usr/bin/find: /lib/modules/2.4.18-26.7.xbigmem/build: No such file or directory
Error (pclose_local): find_objects pclose failed 0x100
Warning (map_ksym_to_module): cannot match loaded module ext3 to a unique module object. Trace may not be reliable.
Apr 7 15:12:31 www kernel: Unable to handle kernel NULL pointer dereference at virtual address 00000004
Apr 7 15:12:31 www kernel: d01597ae
Apr 7 15:12:31 www kernel: *pde = 1a8b6001
Apr 7 15:12:31 www kernel: Oops: 0000
Apr 7 15:12:31 www kernel: CPU: 2
Apr 7 15:12:31 www kernel: EIP: 0010:[<d01597ae>] Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
Apr 7 15:12:31 www kernel: EFLAGS: 00010286
Apr 7 15:12:31 www kernel: eax: bffff9d4 ebx: ce346000 ecx: 00000000 edx: 00000000
Apr 7 15:12:31 www kernel: esi: c0108c93 edi: 0000000b ebp: ce347fb8 esp: ce347f80
Apr 7 15:12:31 www kernel: ds: 0018 es: 0018 ss: 0018
Apr 7 15:12:31 www kernel: Process sh (pid: 15725, stackpage=ce347000)
Apr 7 15:12:31 www kernel: Stack: 00000000 bffffff4 00000292 f441c000 c014e48e f441c000 bffff9d4 0000000b
Apr 7 15:12:31 www kernel: 00000000 f441c000 0000000b ce346000 c0108c93 0000000b bffe6118 d01599a2
Apr 7 15:12:31 www kernel: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Apr 7 15:12:31 www kernel: Call Trace: [<c014e48e>] getname [kernel] 0x5e (0xce347f90))
Apr 7 15:12:31 www kernel: [<c0108c93>] system_call [kernel] 0x33 (0xce347fb0))
Apr 7 15:12:31 www kernel: Code: 8b 42 04 83 f8 ff c7 45 f0 20 00 00 00 c7 45 ec 3a 0a 00 00
>>EIP; d01597ae <_end+fd16d92/383ca5e4> <=====
Trace; c014e48e <getname+5e/a0>
Trace; c0108c93 <system_call+33/38>
Code; d01597ae <_end+fd16d92/383ca5e4>
00000000 <_EIP>:
Code; d01597ae <_end+fd16d92/383ca5e4> <=====
0: 8b 42 04 mov 0x4(%edx),%eax <=====
Code; d01597b1 <_end+fd16d95/383ca5e4>
3: 83 f8 ff cmp $0xffffffff,%eax
Code; d01597b4 <_end+fd16d98/383ca5e4>
6: c7 45 f0 20 00 00 00 movl $0x20,0xfffffff0(%ebp)
Code; d01597bb <_end+fd16d9f/383ca5e4>
d: c7 45 ec 3a 0a 00 00 movl $0xa3a,0xffffffec(%ebp)
2 warnings and 7 errors issued. Results may not be reliable.
The box also segfaulted on a shutdown -r now, so i had no other choice but to
press the reset button. Then, it started appearing again seconds after the box
came up. After another reboot at around 15:46, it stopped. There are 517 oopsen
recorded in the log from 15:12:31 to 15:39:34.
Any ideas?
Both machines have good hardware and are running problemfree. This *looks* like
it is remotely triggered, so i suspect it is some kind of attack. system_call
is the common point in oopsen on both machines. Any ideas how to mess with this
remotely?
--
Jure Pecar
_______________________________________________
Redhat-devel-list mailing list
Redhat-devel-list@xxxxxxxxxx
https://listman.redhat.com/mailman/listinfo/redhat-devel-list
