Re: endless sync on bdi_sched_wait()? 2.6.33.1

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



  Hi,

On Wed 31-03-10 19:07:31, Denys Fedorysychenko wrote:
> I have a proxy server with "loaded" squid. On some moment i did sync, and 
> expecting it to finish in reasonable time. Waited more than 30 minutes, still 
> "sync". Can be reproduced easily.
> 
> Here is some stats and info:
> 
> Linux SUPERPROXY 2.6.33.1-build-0051 #16 SMP Wed Mar 31 17:23:28 EEST 2010 
> i686 GNU/Linux
> 
> SUPERPROXY ~ # iostat -k -x -d 30
> Linux 2.6.33.1-build-0051 (SUPERPROXY)  03/31/10        _i686_  (4 CPU)
> 
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz 
> avgqu-sz   await  svctm  %util
> sda               0.16     0.01    0.08    0.03     3.62     1.33    88.94     
> 0.15 1389.89  59.15   0.66
> sdb               4.14    61.25    6.22   25.55    44.52   347.21    24.66     
> 2.24   70.60   2.36   7.49
> sdc               4.37   421.28    9.95   98.31   318.27  2081.95    44.34    
> 20.93  193.21   2.31  24.96
> sdd               2.34   339.90    3.97  117.47    95.48  1829.52    31.70     
> 1.73   14.23   8.09  98.20
> sde               2.29    71.40    2.34   27.97    22.56   397.81    27.74     
> 2.34   77.34   1.66   5.04
> dm-0              0.00     0.00    0.19    0.02     3.48     0.02    32.96     
> 0.05  252.11  28.05   0.60
> 
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz 
> avgqu-sz   await  svctm  %util
> sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     
> 0.00    0.00   0.00   0.00
> sdb               0.00    54.67    2.93   26.87    12.27   326.13    22.71     
> 2.19   73.49   1.91   5.68
> sdc               0.00   420.50    3.43  110.53   126.40  2127.73    39.56    
> 23.82  209.00   2.06  23.44
> sdd               0.00   319.63    2.30  122.03   121.87  1765.87    30.37     
> 1.72   13.83   7.99  99.37
> sde               0.00    71.67    0.83   30.63     6.93   409.33    26.46     
> 2.66   84.68   1.51   4.76
> dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     
> 0.00    0.00   0.00   0.00
> 
> 
> CPU:  8.4% usr  7.7% sys  0.0% nic 50.7% idle 27.7% io  0.6% irq  4.7% sirq
> Load average: 5.57 4.82 4.46 2/243 2032
>   PID  PPID USER     STAT   VSZ %MEM CPU %CPU COMMAND
>  1769  1552 squid    R     668m  8.3   3 11.7 /usr/sbin/squid -N
>  1546  1545 root     R    10800  0.1   2  6.0 /config/globax 
> /config/globax.conf
>  1549  1548 root     S    43264  0.5   2  1.5 /config/globax /config/globax-
> dld.conf
>  1531     2 root     DW       0  0.0   0  0.3 [jbd2/sdd1-8]
>  1418     1 root     S     2500  0.0   3  0.0 /sbin/syslogd -R 80.83.17.2
>  1524     2 root     SW       0  0.0   0  0.0 [flush-8:32]
>  1525     2 root     SW       0  0.0   1  0.0 [jbd2/sdc1-8]
>  1604     2 root     DW       0  0.0   0  0.0 [flush-8:48]
>  1537     2 root     SW       0  0.0   1  0.0 [jbd2/sde1-8]
>    18     2 root     SW       0  0.0   3  0.0 [events/3]
>  1545     1 root     S     3576  0.0   1  0.0 /config/globax 
> /config/globax.conf
>  1548     1 root     S     3576  0.0   0  0.0 /config/globax /config/globax-
> dld.conf
>  1918     1 ntp      S     3316  0.0   3  0.0 /usr/sbin/ntpd -s
>  1919     1 root     S     3268  0.0   3  0.0 /usr/sbin/ntpd -s
>     1     0 root     S     2504  0.0   0  0.0 /bin/sh /init trynew trynew 
> trynew trynew
>  1923  1257 root     S     2504  0.0   1  0.0 /sbin/getty 38400 tty1
>  1924  1257 root     S     2504  0.0   0  0.0 /sbin/getty 38400 tty2
>  1927  1257 root     S     2504  0.0   0  0.0 /sbin/getty 38400 tty3
>  2015  2014 root     S     2504  0.0   1  0.0 -ash
>  2032  2015 root     R     2504  0.0   3  0.0 top
>  1584     1 root     S     2500  0.0   1  0.0 /usr/bin/ifplugd -i eth0 -a -r 
> /etc/startup/rc.ifup -t 1 -u 1 -d 1
>  1592     1 root     S     2500  0.0   1  0.0 /usr/bin/ifplugd -i eth2 -a -r 
> /etc/startup/rc.ifup -t 1 -u 1 -d 1
>  1587     1 root     S     2500  0.0   1  0.0 /usr/bin/ifplugd -i eth1 -a -r 
> /etc/startup/rc.ifup -t 1 -u 1 -d 1
>  1595     1 root     S     2500  0.0   1  0.0 /usr/bin/ifplugd -i eth3 -a -r 
> /etc/startup/rc.ifup -t 1 -u 1 -d 1
>  1257     1 root     S     2500  0.0   0  0.0 init
>  1420     1 root     S     2500  0.0   1  0.0 /sbin/klogd
>  1432     1 root     S     2500  0.0   3  0.0 /usr/sbin/telnetd -f 
> /etc/issue.telnet
>  1552     1 root     S     2500  0.0   1  0.0 /bin/sh /bin/squidloop
>  1743  1742 root     S     2500  0.0   3  0.0 ash -c gs newkernel
>  1744  1743 root     S     2500  0.0   0  0.0 /bin/sh /bin/gs newkernel
>  1753  1744 root     D     2368  0.0   0  0.0 sync
> 
> 
> SUPERPROXY ~ # cat /proc/1753/stack
> [<c019a93c>] bdi_sched_wait+0x8/0xc
> [<c019a807>] wait_on_bit+0x20/0x2c
> [<c019a9af>] sync_inodes_sb+0x6f/0x10a
> [<c019dd53>] __sync_filesystem+0x28/0x49
> [<c019ddf3>] sync_filesystems+0x7f/0xc0
> [<c019de7a>] sys_sync+0x1b/0x2d
> [<c02f7a25>] syscall_call+0x7/0xb
> [<ffffffff>] 0xffffffff
  Hmm, I guess you are observing the problem reported in
https://bugzilla.kernel.org/show_bug.cgi?id=14830
  There seem to be several issues in the per-bdi writeback code that
cause sync on a busy filesystem to last almost forever. To that bug are
attached two patches that fix two issues but apparently it's not all.
I'm still looking into it...

								Honza
-- 
Jan Kara <jack@xxxxxxx>
SUSE Labs, CR
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[Index of Archives]     [Linux Ext4 Filesystem]     [Union Filesystem]     [Filesystem Testing]     [Ceph Users]     [Ecryptfs]     [AutoFS]     [Kernel Newbies]     [Share Photos]     [Security]     [Netfilter]     [Bugtraq]     [Yosemite News]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux Cachefs]     [Reiser Filesystem]     [Linux RAID]     [Samba]     [Device Mapper]     [CEPH Development]
  Powered by Linux