[1.] One line summary of the problem:
After many write/delete on ocfs2 both servers in cluster got oops kernel
[2.] Full description of the problem/report:
After create new ocfs2 cluster with kernel 3.1.5 after few hours
write/delete on both servers via my own script got oops kernel on both
servers:
I try disable all features, change kernel to 2.6.39.4 and 3.2RC5 but all the
time this same effect.
When i kill on both server this script - still there is a high Load Average
+2.00 and ~10Mbit/s Bandwitch between servers.
After run "TOP" i see what take so much resources:
23557 root 20 0 0 0 0 D 24 0.0 15:35.63 kworker/u:3
1458 root 20 0 0 0 0 S 17 0.0 4:27.58 kworker/u:2
32246 root 20 0 0 0 0 S 15 0.0 5:46.31 kworker/u:1
2933 root 20 0 0 0 0 S 14 0.0 3:09.25 kworker/u:0
Of Course i create other filesystem and working good - so looks like problem
with OCFS2.
[3.] Keywords (i.e., modules, networking, kernel):
[4.] Kernel version (from /proc/version):
TEST-MAIL1 ~ # cat /proc/version
Linux version 3.1.5 (root@TEST-MAIL1) (gcc version 4.4.5 (Gentoo 4.4.5 p1.3,
pie-0.4.5) ) #1 SMP Mon Dec 19 17:16:12 CET 2011
TEST-MAIL2 ~ # cat /proc/version
Linux version 3.1.5 (root@TEST-MAIL2) (gcc version 4.4.5 (Gentoo 4.4.5 p1.3,
pie-0.4.5) ) #1 SMP Mon Dec 19 17:16:06 CET 2011
[5.] Output of Oops.. message (if applicable) with symbolic information
resolved (see Documentation/oops-tracing.txt)
TEST-MAIL1 ~ #dmesg
[cut]
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
rm D ffff88107f472c40 0 16705 22512 0x00000000
ffff881014693810 0000000000000086 ffff881000000000 ffff88102013b040
0000000000012c40 ffff880471855fd8 0000000000012c40 ffff880471854010
ffff880471855fd8 0000000000012c40 ffff881017ff8e40 0000000100000000
Call Trace:
[<ffffffff8148d45d>] ? schedule_timeout+0x1ed/0x2d0
[<ffffffffa0b7d1ea>] ? dlmlock+0x8a/0xda0 [ocfs2_dlm]
[<ffffffff8148ce5c>] ? wait_for_common+0x12c/0x1a0
[<ffffffff81052230>] ? try_to_wake_up+0x280/0x280
[<ffffffffa0a3b9c0>] ? __ocfs2_cluster_lock+0x1f0/0x780 [ocfs2]
[<ffffffff8148ce80>] ? wait_for_common+0x150/0x1a0
[<ffffffffa0a9c6bc>] ? ocfs2_buffer_cached+0x8c/0x180 [ocfs2]
[<ffffffffa0a40bc6>] ? ocfs2_inode_lock_full_nested+0x126/0x540 [ocfs2]
[<ffffffffa0a5922e>] ? ocfs2_lookup_lock_orphan_dir+0x6e/0x1b0 [ocfs2]
[<ffffffffa0a5922e>] ? ocfs2_lookup_lock_orphan_dir+0x6e/0x1b0 [ocfs2]
[<ffffffffa0a5ba1a>] ? ocfs2_prepare_orphan_dir+0x4a/0x290 [ocfs2]
[<ffffffffa0a5e621>] ? ocfs2_unlink+0x6e1/0xbb0 [ocfs2]
[<ffffffff811bcfea>] ? may_link+0xda/0x170
[<ffffffff81141c8e>] ? vfs_unlink+0x9e/0x100
[<ffffffff81145881>] ? do_unlinkat+0x1a1/0x1d0
[<ffffffff81147b00>] ? vfs_readdir+0xa0/0xe0
[<ffffffff8116fedb>] ? fsnotify_find_inode_mark+0x2b/0x40
[<ffffffff81170c24>] ? dnotify_flush+0x54/0x110
[<ffffffff81133eec>] ? filp_close+0x5c/0x90
[<ffffffff81496912>] ? system_call_fastpath+0x16/0x1b
TEST-MAIL2 ~ #dmesg
[CUT]
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
rm D ffff88107f3b2c40 0 10499 23219 0x00000000
ffff88101531f080 0000000000000082 0000000000000000 ffff88102004f080
0000000000012c40 ffff88088a217fd8 0000000000012c40 ffff88088a216010
ffff88088a217fd8 0000000000012c40 ffff881012761f30 0000000115197f40
Call Trace:
[<ffffffff8148d45d>] ? schedule_timeout+0x1ed/0x2d0
[<ffffffffa0b891ea>] ? dlmlock+0x8a/0xda0 [ocfs2_dlm]
[<ffffffff8148ce5c>] ? wait_for_common+0x12c/0x1a0
[<ffffffff81052230>] ? try_to_wake_up+0x280/0x280
[<ffffffffa0a479c0>] ? __ocfs2_cluster_lock+0x1f0/0x780 [ocfs2]
[<ffffffff8148ce80>] ? wait_for_common+0x150/0x1a0
[<ffffffffa0aa86bc>] ? ocfs2_buffer_cached+0x8c/0x180 [ocfs2]
[<ffffffffa0a4cbc6>] ? ocfs2_inode_lock_full_nested+0x126/0x540 [ocfs2]
[<ffffffffa0a6522e>] ? ocfs2_lookup_lock_orphan_dir+0x6e/0x1b0 [ocfs2]
[<ffffffffa0a6522e>] ? ocfs2_lookup_lock_orphan_dir+0x6e/0x1b0 [ocfs2]
[<ffffffffa0a67a1a>] ? ocfs2_prepare_orphan_dir+0x4a/0x290 [ocfs2]
[<ffffffffa0a6a621>] ? ocfs2_unlink+0x6e1/0xbb0 [ocfs2]
[<ffffffff811bcfea>] ? may_link+0xda/0x170
[<ffffffff81141c8e>] ? vfs_unlink+0x9e/0x100
[<ffffffff81145881>] ? do_unlinkat+0x1a1/0x1d0
[<ffffffff81147b00>] ? vfs_readdir+0xa0/0xe0
[<ffffffff8116fedb>] ? fsnotify_find_inode_mark+0x2b/0x40
[<ffffffff81496912>] ? system_call_fastpath+0x16/0x1b
[6.] A small shell script or example program which triggers the
problem (if possible)
TEST-MAIL1 ~ #cat ocfs2-test.sh
#!/bin/bash
while true
do
rm -rf /mnt/EMC/TEST-MAIL1
mkdir /mnt/EMC/TEST-MAIL1
cp -r /usr/src/linux-3.1.5 /mnt/EMC/MAIL1
rm -rf /mnt/EMC/TEST-MAIL1
done;
TEST-MAIL2 ~ #cat ocfs2-test.sh
#!/bin/bash
while true
do
rm -rf /mnt/EMC/TEST-MAIL2
mkdir /mnt/EMC/TEST-MAIL2
cp -r /usr/src/linux-3.1.5 /mnt/EMC/MAIL2
rm -rf /mnt/EMC/TEST-MAIL2
done;
[7.] Environment
TEST-MAIL1 ~ # export
declare -x CONFIG_PROTECT="/var/bind
/var/www/localhost/htdocs/squirrelmail/config"
declare -x CONFIG_PROTECT_MASK="/etc/gentoo-release /etc/sandbox.d
/etc/php/cli-php5.3/ext-active/ /etc/php/cgi-php5.3/ext-active/
/etc/php/apache2-php5.3/ext-active/ /etc/terminfo /etc/ca-certificates.conf
/etc/revdep-rebuild"
declare -x EDITOR="/bin/nano"
declare -x GCC_SPECS=""
declare -x HOME="/root"
declare -x
INFOPATH="/usr/share/info:/usr/share/binutils-data/x86_64-pc-linux-gnu/2.21.1/info:/usr/share/gcc-data/x86_64-pc-linux-gnu/4.4.5/info"
declare -x LANG="pl_PL"
declare -x LC_ALL="pl_PL"
declare -x LESS="-R -M --shift 5"
declare -x LESSOPEN="|lesspipe %s"
declare -x LOGNAME="root"
declare -x
LS_COLORS="rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.pdf=00;32:*.ps=00;32:*.txt=00;32:*.patch=00;32:*.diff=00;32:*.log=00;32:*.tex=00;32:*.doc=00;32:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:"
declare -x MAIL="/var/mail/root"
declare -x
MANPATH="/usr/local/share/man:/usr/share/man:/usr/share/binutils-data/x86_64-pc-linux-gnu/2.21.1/man:/usr/share/gcc-data/x86_64-pc-linux-gnu/4.4.5/man:/usr/lib64/php5.3/man/"
declare -x OLDPWD
declare -x PAGER="/usr/bin/less"
declare -x
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.4.5"
declare -x PWD="/root"
declare -x SHELL="/bin/bash"
declare -x SHLVL="1"
declare -x SSH_CLIENT="172.23.1.146 57479 22"
declare -x SSH_CONNECTION="172.23.1.146 57479 172.17.1.251 22"
declare -x SSH_TTY="/dev/pts/2"
declare -x TERM="xterm"
declare -x USER="root"
[7.1.] Software (add the output of the ver_linux script here)
TEST-MAIL1 scripts # ./ver_linux
If some fields are empty or look unusual you may have an old version.
Compare to the current minimal requirements in Documentation/Changes.
Linux TEST-MAIL1 3.1.5 #1 SMP Mon Dec 19 17:16:12 CET 2011 x86_64 Intel(R)
Xeon(R) CPU X5650 @ 2.67GHz GenuineIntel GNU/Linux
Gnu C 4.4.5
Gnu make 3.82
binutils 2.21.1
util-linux 2.19.1
mount support
module-init-tools 3.16
e2fsprogs 1.41.14
Linux C Library 2.13
Dynamic linker (ldd) 2.13
Procps 3.2.8
Net-tools 1.60_p20110409135728
Kbd 1.15.3wip
Sh-utils 8.7
Modules Loaded ocfs2_stack_o2cb ip_vs_wrr ip_vs nf_conntrack ipv6
8021q garp stp llc dm_round_robin scsi_dh_emc dlm ocfs2_dlm ocfs2
ocfs2_dlmfs ocfs2_stackglue ocfs2_nodemanager configfs dm_multipath bfa
snd_pcm bnx2 snd_timer cdc_ether i2c_i801 i2c_core usbnet mii snd soundcore
snd_page_alloc ioatdma iTCO_wdt iTCO_vendor_support serio_raw i7core_edac
edac_core dca pcspkr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi
tg3 e1000 fuse xfs exportfs nfs nfs_acl auth_rpcgss fscache lockd sunrpc
reiserfs btrfs libcrc32c zlib_deflate ext4 jbd2 ext3 jbd ext2 mbcache raid10
raid456 async_raid6_recov async_memcpy async_pq async_xor xor async_tx
raid6_pq raid1 raid0 dm_snapshot dm_crypt dm_mirror dm_region_hash dm_log
dm_mod scsi_wait_scan sl811_hcd usb_storage aic94xx libsas lpfc qla2xxx
megaraid_sas megaraid_mbox megaraid_mm aacraid sx8 hpsa cciss 3w_9xxx
3w_xxxx mptsas scsi_transport_sas mptfc scsi_transport_fc scsi_tgt mptspi
mptscsih mptbase imm parport sym53c8xx initio arcmsr aic7xxx aic79xx
scsi_transport_spi sr_mod cdrom sg sd_mod crc_t10dif pdc_adma sata_inic162x
sata_mv ata_piix ahci libahci sata_qstor sata_vsc sata_uli sata_sis sata_sx4
sata_nv sata_via sata_svw sata_sil24 sata_sil sata_promise pata_via
pata_jmicron pata_marvell pata_sis pata_netcell pata_pdc202xx_old
pata_atiixp pata_amd pata_ali pata_it8213 pata_pcmcia pata_serverworks
pata_oldpiix pata_artop pata_it821x pata_hpt3x2n pata_hpt3x3 pata_hpt37x
pata_hpt366 pata_cmd64x pata_sil680 pata_pdc2027x
[7.2.] Processor information (from /proc/cpuinfo):
TEST-MAIL1 ~ # cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 44
model name : Intel(R) Xeon(R) CPU X5650 @ 2.67GHz
stepping : 2
cpu MHz : 2666.752
cache size : 12288 KB
physical id : 0
siblings : 12
core id : 0
cpu cores : 6
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx
pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology
nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2
ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt aes lahf_lm arat epb dts
tpr_shadow vnmi flexpriority ept vpid
bogomips : 5333.50
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
[CUT]
processor : 23
vendor_id : GenuineIntel
cpu family : 6
model : 44
model name : Intel(R) Xeon(R) CPU X5650 @ 2.67GHz
stepping : 2
cpu MHz : 2666.752
cache size : 12288 KB
physical id : 1
siblings : 12
core id : 10
cpu cores : 6
apicid : 53
initial apicid : 53
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx
pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology
nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2
ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt aes lahf_lm arat epb dts
tpr_shadow vnmi flexpriority ept vpid
bogomips : 5333.19
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
[7.3.] Module information (from /proc/modules):
TEST-MAIL1 ~ # cat /proc/modules
ocfs2_stack_o2cb 3678 1 - Live 0xffffffffa0043000
ip_vs_wrr 2001 12 - Live 0xffffffffa002f000
ip_vs 126566 14 ip_vs_wrr, Live 0xffffffffa0b20000
nf_conntrack 77276 1 ip_vs, Live 0xffffffffa05a7000
ipv6 315785 79 ip_vs, Live 0xffffffffa0b93000
8021q 19111 0 - Live 0xffffffffa0183000
garp 6985 1 8021q, Live 0xffffffffa008f000
stp 2059 1 garp, Live 0xffffffffa000d000
llc 5480 2 garp,stp, Live 0xffffffffa0029000
dm_round_robin 2347 2 - Live 0xffffffffa0003000
scsi_dh_emc 7487 4 - Live 0xffffffffa0049000
dlm 140410 0 - Live 0xffffffffa0870000
ocfs2_dlm 215959 1 ocfs2_stack_o2cb, Live 0xffffffffa0b5d000
ocfs2 1087698 3 - Live 0xffffffffa0a15000
ocfs2_dlmfs 18832 1 - Live 0xffffffffa015e000
ocfs2_stackglue 11208 3 ocfs2_stack_o2cb,ocfs2,ocfs2_dlmfs, Live
0xffffffffa0117000
ocfs2_nodemanager 217156 12 ocfs2_stack_o2cb,ocfs2_dlm,ocfs2,ocfs2_dlmfs,
Live 0xffffffffa09de000
configfs 27116 3 dlm,ocfs2_nodemanager, Live 0xffffffffa02d7000
dm_multipath 17309 2 dm_round_robin, Live 0xffffffffa0065000
bfa 515624 4 - Live 0xffffffffa095f000
snd_pcm 78397 0 - Live 0xffffffffa03e2000
bnx2 75992 0 - Live 0xffffffffa079a000
snd_timer 21141 1 snd_pcm, Live 0xffffffffa04d8000
cdc_ether 8563 0 - Live 0xffffffffa010a000
i2c_i801 10128 0 - Live 0xffffffffa00d5000
i2c_core 29192 1 i2c_i801, Live 0xffffffffa0539000
usbnet 27108 1 cdc_ether, Live 0xffffffffa0466000
mii 5102 1 usbnet, Live 0xffffffffa0079000
snd 63536 2 snd_pcm,snd_timer, Live 0xffffffffa06e9000
soundcore 7444 1 snd, Live 0xffffffffa003e000
snd_page_alloc 8132 1 snd_pcm, Live 0xffffffffa0019000
ioatdma 49259 48 - Live 0xffffffffa01cd000
iTCO_wdt 12226 0 - Live 0xffffffffa095b000
iTCO_vendor_support 2878 1 iTCO_wdt, Live 0xffffffffa00fe000
serio_raw 4352 0 - Live 0xffffffffa0956000
i7core_edac 17774 0 - Live 0xffffffffa094d000
edac_core 44779 4 i7core_edac, Live 0xffffffffa093a000
dca 6700 1 ioatdma, Live 0xffffffffa020a000
pcspkr 1886 0 - Live 0xffffffffa00b3000
iscsi_tcp 9442 0 - Live 0xffffffffa0193000
libiscsi_tcp 15334 1 iscsi_tcp, Live 0xffffffffa0255000
libiscsi 44632 2 iscsi_tcp,libiscsi_tcp, Live 0xffffffffa091f000
scsi_transport_iscsi 36811 2 iscsi_tcp,libiscsi, Live 0xffffffffa090f000
tg3 143267 0 - Live 0xffffffffa08e4000
e1000 145820 0 - Live 0xffffffffa08b5000
fuse 68687 1 - Live 0xffffffffa089c000
xfs 764761 0 - Live 0xffffffffa07b4000
exportfs 3934 1 xfs, Live 0xffffffffa0073000
nfs 394497 0 - Live 0xffffffffa0738000
nfs_acl 2597 1 nfs, Live 0xffffffffa006b000
auth_rpcgss 43042 1 nfs, Live 0xffffffffa072c000
fscache 43963 1 nfs, Live 0xffffffffa0719000
lockd 75497 1 nfs, Live 0xffffffffa06fe000
sunrpc 229010 5 nfs,nfs_acl,auth_rpcgss,lockd, Live 0xffffffffa06b0000
reiserfs 219451 0 - Live 0xffffffffa066f000
btrfs 589407 0 - Live 0xffffffffa05c6000
libcrc32c 1196 1 btrfs, Live 0xffffffffa005b000
zlib_deflate 21895 1 btrfs, Live 0xffffffffa05bf000
ext4 409404 0 - Live 0xffffffffa0542000
jbd2 78658 2 ocfs2,ext4, Live 0xffffffffa0524000
ext3 210941 1 - Live 0xffffffffa04e0000
jbd 67340 1 ext3, Live 0xffffffffa04c6000
ext2 64947 0 - Live 0xffffffffa04af000
mbcache 6958 3 ext4,ext3,ext2, Live 0xffffffffa0126000
raid10 26340 0 - Live 0xffffffffa04a7000
raid456 62193 0 - Live 0xffffffffa0496000
async_raid6_recov 6473 1 raid456, Live 0xffffffffa00f4000
async_memcpy 1898 2 raid456,async_raid6_recov, Live 0xffffffffa0033000
async_pq 4533 2 raid456,async_raid6_recov, Live 0xffffffffa009b000
async_xor 3682 3 raid456,async_raid6_recov,async_pq, Live 0xffffffffa0023000
xor 4541 1 async_xor, Live 0xffffffffa0087000
async_tx 2878 5 raid456,async_raid6_recov,async_memcpy,async_pq,async_xor,
Live 0xffffffffa0012000
raid6_pq 79891 2 async_raid6_recov,async_pq, Live 0xffffffffa0481000
raid1 25867 0 - Live 0xffffffffa0479000
raid0 8154 0 - Live 0xffffffffa0050000
dm_snapshot 29422 0 - Live 0xffffffffa0470000
dm_crypt 16644 0 - Live 0xffffffffa01db000
dm_mirror 13796 0 - Live 0xffffffffa01b5000
dm_region_hash 10323 1 dm_mirror, Live 0xffffffffa00e0000
dm_log 9246 2 dm_mirror,dm_region_hash, Live 0xffffffffa00ca000
dm_mod 75518 7 dm_multipath,dm_snapshot,dm_crypt,dm_mirror,dm_log, Live
0xffffffffa0436000
scsi_wait_scan 749 0 - Live 0xffffffffa0008000
sl811_hcd 10688 0 - Live 0xffffffffa00a5000
usb_storage 50158 0 - Live 0xffffffffa0428000
aic94xx 79911 0 - Live 0xffffffffa0413000
libsas 59814 1 aic94xx, Live 0xffffffffa0403000
lpfc 564581 0 - Live 0xffffffffa0357000
qla2xxx 369523 0 - Live 0xffffffffa02fb000
megaraid_sas 74186 1 - Live 0xffffffffa02e2000
megaraid_mbox 30716 0 - Live 0xffffffffa02ce000
megaraid_mm 9579 1 megaraid_mbox, Live 0xffffffffa02c7000
aacraid 78932 0 - Live 0xffffffffa02ac000
sx8 14069 0 - Live 0xffffffffa02a4000
hpsa 45153 0 - Live 0xffffffffa0293000
cciss 107100 0 - Live 0xffffffffa0272000
3w_9xxx 35289 0 - Live 0xffffffffa0265000
3w_xxxx 28359 0 - Live 0xffffffffa025a000
mptsas 52240 0 - Live 0xffffffffa0247000
scsi_transport_sas 31289 3 aic94xx,libsas,mptsas, Live 0xffffffffa0238000
mptfc 14347 0 - Live 0xffffffffa0230000
scsi_transport_fc 48157 4 bfa,lpfc,qla2xxx,mptfc, Live 0xffffffffa021d000
scsi_tgt 11726 1 scsi_transport_fc, Live 0xffffffffa0216000
mptspi 16078 0 - Live 0xffffffffa020e000
mptscsih 36316 3 mptsas,mptfc,mptspi, Live 0xffffffffa0200000
mptbase 95142 4 mptsas,mptfc,mptspi,mptscsih, Live 0xffffffffa01e1000
imm 10725 0 - Live 0xffffffffa01c9000
parport 34760 1 imm, Live 0xffffffffa01ba000
sym53c8xx 76324 0 - Live 0xffffffffa01a1000
initio 20670 0 - Live 0xffffffffa0197000
arcmsr 30044 0 - Live 0xffffffffa018a000
aic7xxx 117148 0 - Live 0xffffffffa0165000
aic79xx 128507 0 - Live 0xffffffffa013d000
scsi_transport_spi 24324 4 mptspi,sym53c8xx,aic7xxx,aic79xx, Live
0xffffffffa0132000
sr_mod 15342 0 - Live 0xffffffffa012a000
cdrom 40595 1 sr_mod, Live 0xffffffffa011b000
sg 29553 0 - Live 0xffffffffa010e000
sd_mod 36634 6 - Live 0xffffffffa0100000
crc_t10dif 1491 1 sd_mod, Live 0xffffffffa00fc000
pdc_adma 6277 0 - Live 0xffffffffa00f7000
sata_inic162x 8215 0 - Live 0xffffffffa00f0000
sata_mv 28039 0 - Live 0xffffffffa00e4000
ata_piix 23192 0 - Live 0xffffffffa00d9000
ahci 20980 0 - Live 0xffffffffa00ce000
libahci 23551 1 ahci, Live 0xffffffffa00c3000
sata_qstor 6155 0 - Live 0xffffffffa00be000
sata_vsc 4404 0 - Live 0xffffffffa00b9000
sata_uli 3335 0 - Live 0xffffffffa00b5000
sata_sis 4109 0 - Live 0xffffffffa00b0000
sata_sx4 10018 0 - Live 0xffffffffa00a9000
sata_nv 21903 0 - Live 0xffffffffa009e000
sata_via 9479 0 - Live 0xffffffffa0097000
sata_svw 4622 0 - Live 0xffffffffa0092000
sata_sil24 12295 0 - Live 0xffffffffa008a000
sata_sil 8556 0 - Live 0xffffffffa0083000
sata_promise 11313 0 - Live 0xffffffffa007c000
pata_via 9828 0 - Live 0xffffffffa0075000
pata_jmicron 2699 0 - Live 0xffffffffa0071000
pata_marvell 3273 0 - Live 0xffffffffa006d000
pata_sis 11936 1 sata_sis, Live 0xffffffffa0061000
pata_netcell 2492 0 - Live 0xffffffffa005d000
pata_pdc202xx_old 5185 0 - Live 0xffffffffa0058000
pata_atiixp 4237 0 - Live 0xffffffffa0053000
pata_amd 11719 0 - Live 0xffffffffa004c000
pata_ali 10460 0 - Live 0xffffffffa0045000
pata_it8213 3935 0 - Live 0xffffffffa0041000
pata_pcmcia 10194 0 - Live 0xffffffffa003a000
pata_serverworks 6429 0 - Live 0xffffffffa0035000
pata_oldpiix 3586 0 - Live 0xffffffffa0031000
pata_artop 5498 0 - Live 0xffffffffa002c000
pata_it821x 10439 0 - Live 0xffffffffa0025000
pata_hpt3x2n 6180 0 - Live 0xffffffffa0020000
pata_hpt3x3 3518 0 - Live 0xffffffffa001c000
pata_hpt37x 12638 0 - Live 0xffffffffa0014000
pata_hpt366 5761 0 - Live 0xffffffffa000f000
pata_cmd64x 6762 0 - Live 0xffffffffa000a000
pata_sil680 5756 0 - Live 0xffffffffa0005000
pata_pdc2027x 7098 0 - Live 0xffffffffa0000000
[7.4.] Loaded driver and hardware information (/proc/ioports, /proc/iomem)
TEST-MAIL1 ~ # cat /proc/ioports
0000-0cf7 : PCI Bus 0000:00
0000-001f : dma1
0020-0021 : pic1
0040-0043 : timer0
0050-0053 : timer1
0060-0060 : keyboard
0064-0064 : keyboard
0070-0077 : rtc0
0080-008f : dma page reg
00a0-00a1 : pic2
00c0-00df : dma2
00f0-00ff : fpu
0170-0177 : 0000:00:1f.2
0170-0177 : ata_piix
01f0-01f7 : 0000:00:1f.2
01f0-01f7 : ata_piix
02f8-02ff : serial
0376-0376 : 0000:00:1f.2
0376-0376 : ata_piix
03c0-03df : vga+
03f6-03f6 : 0000:00:1f.2
03f6-03f6 : ata_piix
03f8-03ff : serial
0400-043f : pnp 00:06
0440-045f : pnp 00:06
0460-047f : pnp 00:06
0580-05ff : pnp 00:06
0580-0583 : ACPI PM1a_EVT_BLK
0584-0587 : ACPI PM1a_CNT_BLK
0588-058b : ACPI PM_TMR
05a0-05af : ACPI GPE0_BLK
05b0-05b3 : iTCO_wdt
05d0-05d0 : ACPI PM2_CNT_BLK
05e0-05ff : pnp 00:06
05e0-05ff : iTCO_wdt
0ca0-0ca1 : pnp 00:0b
0ca2-0ca2 : pnp 00:0c
0ca3-0ca3 : pnp 00:0c
0ca4-0ca9 : pnp 00:0b
0caa-0cab : pnp 00:0b
0cac-0ccb : pnp 00:0b
0cf8-0cff : PCI conf1
0d00-ffff : PCI Bus 0000:00
1000-1fff : PCI Bus 0000:01
1000-10ff : 0000:01:00.0
1000-10ff : megasas: LSI
2000-201f : 0000:00:1f.3
2000-201f : i801_smbus
2020-203f : 0000:00:1d.2
2020-203f : uhci_hcd
2040-205f : 0000:00:1d.1
2040-205f : uhci_hcd
2060-207f : 0000:00:1d.0
2060-207f : uhci_hcd
2080-209f : 0000:00:1a.1
2080-209f : uhci_hcd
20a0-20bf : 0000:00:1a.0
20a0-20bf : uhci_hcd
20c0-20cf : 0000:00:1f.5
20c0-20cf : ata_piix
20d0-20df : 0000:00:1f.5
20d0-20df : ata_piix
20e0-20ef : 0000:00:1f.2
20e0-20ef : ata_piix
20f0-20ff : 0000:00:1f.2
20f0-20ff : ata_piix
2100-2107 : 0000:00:1f.5
2100-2107 : ata_piix
2108-210f : 0000:00:1f.5
2108-210f : ata_piix
2120-2123 : 0000:00:1f.5
2120-2123 : ata_piix
2124-2127 : 0000:00:1f.5
2124-2127 : ata_piix
TEST-MAIL1 ~ # cat /proc/iomem
00000000-0000ffff : reserved
00010000-0009bfff : System RAM
0009c000-0009ffff : reserved
000a0000-000bffff : PCI Bus 0000:00
000c0000-000c7fff : Video ROM
000cc800-000d29ff : Adapter ROM
000e0000-000fffff : reserved
000f0000-000fffff : System ROM
00100000-7cb90fff : System RAM
01000000-01499782 : Kernel code
01499783-01a979bf : Kernel data
01b32000-01c6afff : Kernel bss
7cb91000-7cc54fff : reserved
7cc55000-7d5a2fff : System RAM
7d5a3000-7d652fff : reserved
7d653000-7f68efff : System RAM
7f68f000-7f6defff : reserved
7f6df000-7f7defff : ACPI Non-volatile Storage
7f7df000-7f7fefff : ACPI Tables
7f7ff000-7f7fffff : System RAM
7f800000-8fffffff : reserved
80000000-8fffffff : PCI MMCONFIG 0000 [bus 00-ff]
80000000-8fffffff : pnp 00:0a
90000000-fdffffff : PCI Bus 0000:00
90000000-900fffff : PCI Bus 0000:15
90000000-9007ffff : 0000:15:00.0
90100000-901fffff : PCI Bus 0000:1a
90100000-9017ffff : 0000:1a:00.0
90200000-904fffff : PCI Bus 0000:01
90200000-9021ffff : 0000:01:00.0
92000000-95ffffff : PCI Bus 0000:0b
92000000-93ffffff : 0000:0b:00.0
92000000-93ffffff : bnx2
94000000-95ffffff : 0000:0b:00.1
94000000-95ffffff : bnx2
96000000-96ffffff : PCI Bus 0000:06
96000000-96ffffff : PCI Bus 0000:07
96000000-96ffffff : 0000:07:00.0
97000000-978fffff : PCI Bus 0000:06
97000000-978fffff : PCI Bus 0000:07
97000000-977fffff : 0000:07:00.0
97800000-97803fff : 0000:07:00.0
97900000-979fffff : PCI Bus 0000:01
97900000-9793ffff : 0000:01:00.0
97940000-97943fff : 0000:01:00.0
97a00000-97afffff : PCI Bus 0000:1a
97a00000-97a1ffff : 0000:1a:00.0
97a00000-97a1ffff : bfa
97a20000-97a3ffff : 0000:1a:00.1
97a40000-97a43fff : 0000:1a:00.0
97a40000-97a43fff : bfa
97a44000-97a47fff : 0000:1a:00.1
97b00000-97bfffff : PCI Bus 0000:15
97b00000-97b1ffff : 0000:15:00.0
97b00000-97b1ffff : bfa
97b20000-97b3ffff : 0000:15:00.1
97b40000-97b43fff : 0000:15:00.0
97b40000-97b43fff : bfa
97b44000-97b47fff : 0000:15:00.1
97c00000-97c03fff : 0000:00:16.0
97c00000-97c03fff : ioatdma
97c04000-97c07fff : 0000:00:16.1
97c04000-97c07fff : ioatdma
97c08000-97c0bfff : 0000:00:16.2
97c08000-97c0bfff : ioatdma
97c0c000-97c0ffff : 0000:00:16.3
97c0c000-97c0ffff : ioatdma
97c10000-97c13fff : 0000:00:16.4
97c10000-97c13fff : ioatdma
97c14000-97c17fff : 0000:00:16.5
97c14000-97c17fff : ioatdma
97c18000-97c1bfff : 0000:00:16.6
97c18000-97c1bfff : ioatdma
97c1c000-97c1ffff : 0000:00:16.7
97c1c000-97c1ffff : ioatdma
97c20000-97c203ff : 0000:00:1d.7
97c20000-97c203ff : ehci_hcd
97c21000-97c213ff : 0000:00:1a.7
97c21000-97c213ff : ehci_hcd
97c22000-97c220ff : 0000:00:1f.3
fc000000-fcffffff : reserved
fc000000-fcffffff : pnp 00:0a
fe710000-fe711fff : pnp 00:0a
fe800000-fe9fffff : pnp 00:0a
fea00000-feafffff : pnp 00:0a
feb00000-febfffff : pnp 00:0a
fec00000-fec003ff : IOAPIC 0
fec80000-fec803ff : IOAPIC 1
fed00000-fed003ff : HPET 0
fed00000-fed003ff : pnp 00:05
fed1c000-fed1ffff : reserved
fed1c000-fed1ffff : pnp 00:0a
fee00000-feefffff : pnp 00:0a
fee00000-fee00fff : Local APIC
ff800000-ffffffff : reserved
ffc00000-ffffffff : pnp 00:0a
100000000-107fffffff : System RAM
7f00000000-7f00000000 : PCI Bus 0000:00
[7.5.] PCI information ('lspci -vvv' as root)
Can`t paste here because message size exceeding 100 000 characters.
If need it i can upload somewhere.
[7.6.] SCSI information (from /proc/scsi/scsi)
TEST-MAIL1 ~ # cat /proc/scsi/scsi
Attached devices:
Host: scsi0 Channel: 00 Id: 00 Lun: 00
Vendor: TSSTcorp Model: CDDVDW TS-L633F Rev: IT03
Type: CD-ROM ANSI SCSI revision: 05
Host: scsi4 Channel: 02 Id: 00 Lun: 00
Vendor: IBM Model: ServeRAID M5015 Rev: 2.12
Type: Direct-Access ANSI SCSI revision: 05
Host: scsi5 Channel: 00 Id: 00 Lun: 00
Vendor: DGC Model: RAID 10 Rev: 0428
Type: Direct-Access ANSI SCSI revision: 04
Host: scsi5 Channel: 00 Id: 01 Lun: 00
Vendor: DGC Model: RAID 10 Rev: 0428
Type: Direct-Access ANSI SCSI revision: 04
Host: scsi6 Channel: 00 Id: 00 Lun: 00
Vendor: DGC Model: RAID 10 Rev: 0428
Type: Direct-Access ANSI SCSI revision: 04
Host: scsi6 Channel: 00 Id: 01 Lun: 00
Vendor: DGC Model: RAID 10 Rev: 0428
Type: Direct-Access ANSI SCSI revision: 04
[7.7.] Other information that might be relevant to the problem
(please look in /proc and include all information that you
think to be relevant):
After 4 hours when i kill write/delete scripts on both servers load is still
high:
TEST-MAIL1
11:53:39 up 17:50, 1 user, load average: 2,30, 1,99, 1,90
TEST-MAIL2
11:54:16 up 17:58, 1 user, load average: 1,54, 1,59, 1,65
Bandwith between servers is ~10Mbit/s all the time but nothing is copying
from/to OCFS2.
On both servers this same process take resources:
TEST-MAIL1
23557 root 20 0 0 0 0 D 15 0.0 63:30.18 kworker/u:3
22844 root 20 0 0 0 0 S 10 0.0 0:14.65 kworker/u:1
20912 root 20 0 0 0 0 S 8 0.0 2:08.38 kworker/u:2
22408 root 20 0 0 0 0 S 6 0.0 67:55.86 dlm_thread
7489 root 20 0 0 0 0 S 5 0.0 11:41.52 kworker/u:0
TEST-MAIL2
6512 root 20 0 0 0 0 D 33 0.0 30:12.34 kworker/u:2
20497 root 20 0 0 0 0 D 17 0.0 2:57.55 kworker/u:1
16801 root 20 0 0 0 0 R 15 0.0 6:44.95 kworker/u:3
20966 root 20 0 0 0 0 S 14 0.0 2:16.34 kworker/u:0
23127 root 20 0 0 0 0 S 4 0.0 77:55.52 dlm_thread
Anyone saw this problem?
[X.] Other notes, patches, fixes, workarounds:
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
