Hi list,
here's an updated post of my one yesterday.
We've updated some systems from 2.6.32 to 3.0.4 vanilla kernel. Since
then we're expecting freezes every now and then. All in memory apps are
still working but nothing which reads or writes from or to disk (at
least it seems like that).
If you're already conncted via ssh and running top suddenly idle is
99-100% and loads goes up to 500. I then cannot write to disk anymore.
I've seen this for now on 2-5 of 20 servers i've updated. I can't
believe they are all damaged. Also every night there comes another one
where the same happens. Also running them on 2.6.32 again works fine.
Luckily i was able to trigger a sysrq on one machine fast enough:
echo t >/proc/sysrq-trigger
sysrq output is attached
I hope somebody can help.
Please CC me i'm not on list.
Stefan
2011-09-08 14:07:44 [ 2316.054478] ffff8801855d7fd8
2011-09-08 14:07:44 [ 2316.054698] Call Trace:
2011-09-08 14:07:44 ffff8801855fba28 0000000000000000[ 2316.056366] [] ? handle_mm_fault+0x24c/0x263
2011-09-08 14:07:44 ffff88018561c010 0000000000010640[ 2316.057975] [] ? do_notify_parent+0x1ca/0x1df
2011-09-08 14:07:44 0000000000010640 0000000000010640 ffff880185733fd8 ffff880185733fd8[ 2316.067053] stunnel x 0000000100000001
2011-09-08 14:07:44 0000000000000001[ 2316.069520] [] ? do_page_fault+0x342/0x37d
2011-09-08 14:07:44 [ 2316.069520] [] ? do_page_fault+0x342/0x37d
2011-09-08 14:07:44 [ 2316.074587] [] ? ktime_get_ts+0xad/0xba
2011-09-08 14:07:44 [ 2316.074587] [] ? ktime_get_ts+0xad/0xba
2011-09-08 14:07:44
2011-09-08 14:07:44 0 18797 7005 0x00000000
2011-09-08 14:07:44 [ 2316.077931] [] ? poll_select_copy_remaining+0xc5/0xe9
2011-09-08 14:07:44 [ 2316.078979] Call Trace:
2011-09-08 14:07:44 [ 2316.079796] httpd S ffff88017c8c2010[ 2316.080372] ffff88017c8c3fd8[ 2316.080592] Call Trace:
2011-09-08 14:07:44 ffff88043ffd9e60 ffff88018565e9e8
2011-09-08 14:07:44 [ 2316.089337] [] ? vfsmount_lock_global_unlock_online+0x40/0x4f
2011-09-08 14:07:44 [ 2316.089337] [] ? vfsmount_lock_global_unlock_online+0x40/0x4f
2011-09-08 14:07:44 [ 2316.094430] [] ? mntput_no_expire+0x96/0x155
2011-09-08 14:07:44 [ 2316.094430] [] ? mntput_no_expire+0x96/0x155
2011-09-08 14:07:44 [ 2316.094494] [] ? vfs_read+0xab/0x14e
2011-09-08 14:07:44 [ 2316.094494] [] ? vfs_read+0xab/0x14e
2011-09-08 14:07:44 ffff88017ca9dfd8[ 2316.097222] [] sys_semtimedop+0x6c1/0x845
2011-09-08 14:07:44 [ 2316.099211] [] ? fd_install+0x30/0x60
2011-09-08 14:07:44 [ 2316.100314] [] ? lru_cache_add_lru+0x40/0x42
2011-09-08 14:07:44 [ 2316.100691] [] compat_sys_wait4+0x26/0xb6
2011-09-08 14:07:44 [ 2316.101736] Call Trace:
2011-09-08 14:07:44 [ 2316.102232] [] ? sys32_stat64+0x29/0x34
2011-09-08 14:07:44 [ 2316.102417] pop3passwdcheck D[ 2316.103894] stunnel4 S[ 2316.104690] Call Trace:
2011-09-08 14:07:44 [ 2316.104874] [] schedule_timeout+0x37/0x269
2011-09-08 14:07:44 [ 2316.108462] httpd S[ 2316.109257] Call Trace:
2011-09-08 14:07:44 [ 2316.109257] Call Trace:
2011-09-08 14:07:44 [ 2316.114316] [] ? __alloc_pages_nodemask+0x137/0x740
2011-09-08 14:07:44 [ 2316.114316] [] ? __alloc_pages_nodemask+0x137/0x740
2011-09-08 14:07:44 [ 2316.114381] [] ? sys_semtimedop+0x2f3/0x845
2011-09-08 14:07:44 [ 2316.114381] [] ? sys_semtimedop+0x2f3/0x845
2011-09-08 14:07:44 [ 2316.114445] [] sys_semtimedop+0x6c1/0x845
2011-09-08 14:07:44 [ 2316.114764] [] ? pde_users_dec+0x15/0x3a
2011-09-08 14:07:44 [ 2316.115804] [] ? __alloc_pages_nodemask+0x137/0x740
2011-09-08 14:07:44 ffff88017cba07a8 0 18819 7005 0x00000000
2011-09-08 14:07:44 [ 2316.121498] ffff88017c519ce8[ 2316.122458] [] sys_semtimedop+0x6c1/0x845
2011-09-08 14:07:44 [ 2316.122713] [] ? cp_new_stat+0xe2/0xee
2011-09-08 14:07:44 0 18821 7005 0x00000000
2011-09-08 14:07:44 0000000000010640[ 2316.123696] Call Trace:
2011-09-08 14:07:44 0000000000010640[ 2316.125496] [] sys_semtimedop+0x6c1/0x845
2011-09-08 14:07:44 [ 2316.126127] httpd S[ 2316.126263] ffff88017c60bce8[ 2316.127236] [] ? handle_pte_fault+0x2c2/0x770
2011-09-08 14:07:44 0000000000010640[ 2316.128964] [] ? handle_pte_fault+0x2c2/0x770
2011-09-08 14:07:44 [ 2316.128964] [] ? handle_pte_fault+0x2c2/0x770
2011-09-08 14:07:44 [ 2316.134036] [] ? handle_mm_fault+0x24c/0x263
2011-09-08 14:07:44 [ 2316.134036] [] ? handle_mm_fault+0x24c/0x263
2011-09-08 14:07:44 [ 2316.135273] Call Trace:
2011-09-08 14:07:44 0000000000010640 000000007c787bb8[ 2316.140418] [] ? handle_pte_fault+0x2c2/0x770
2011-09-08 14:07:44 [ 2316.140799] [] sys_semop+0xb/0xd
2011-09-08 14:07:44 [ 2316.142893] 0000000000004000[ 2316.143829] [] ? sys32_stat64+0x29/0x34
2011-09-08 14:07:44 [ 2316.144868] [] ? handle_mm_fault+0x24c/0x263
2011-09-08 14:07:44 [ 2316.145247] [] unix_dgram_sendmsg+0x3dc/0x56c
2011-09-08 14:07:44 [ 2316.145500] [] ? sys_newstat+0x2c/0x3b
2011-09-08 14:07:44 [ 2316.146540] Call Trace:
2011-09-08 14:07:44 0 18834 4737 0x00000000
2011-09-08 14:07:44
2011-09-08 14:07:44 [ 2316.153959] 0000000000004000
2011-09-08 14:07:44 [ 2316.153959] 0000000000004000 ffff8803c7bd2300 ffff8803c7bd2300[ 2316.154775] [] ? wake_up_bit+0x23/0x23
2011-09-08 14:07:44 ffff88017c514cd0
2011-09-08 14:07:44 [ 2316.160039] ffff88017c067fd8[ 2316.160039] ffff88017c067fd8[ 2316.161010] [] compat_sys_open+0x15/0x17
2011-09-08 14:07:44 [ 2316.162239] [] ? link_path_walk+0xa8/0x6f7
2011-09-08 14:07:44 [ 2316.162865] stunnel4 S ffff88017cb46010[ 2316.166265] stunnel x[ 2316.166401] ffff88017c167eb8 ffff88017c16e010
2011-09-08 14:07:44 ffff88017c16e010
2011-09-08 14:07:44 [ 2316.174085] 0000000000004000[ 2316.174643] Call Trace:
2011-09-08 14:07:44 0000000000010640 ffff88043f810080[ 2316.176442] [] ? sys_semtimedop+0x2f3/0x845
2011-09-08 14:07:44 [ 2316.176696] [] ? handle_mm_fault+0x24c/0x263
2011-09-08 14:07:44 [ 2316.178056] [] ? sys_semtimedop+0x2f3/0x845
2011-09-08 14:07:44 ffff88017c27c010[ 2316.180241] httpd S[ 2316.181338] [] sys_semtimedop+0x6c1/0x845
2011-09-08 14:07:44
2011-09-08 14:07:44 0000000000000082[ 2316.184695] [] ? do_page_fault+0x342/0x37d
2011-09-08 14:07:44 0000000000000000[ 2316.186734] [] system_call_fastpath+0x16/0x1b
2011-09-08 14:07:44 ffff88017c3f6010 ffff88017c326080[ 2316.189006] [] ? __hrtimer_start_range_ns+0x31f/0x331
2011-09-08 14:07:44 [ 2316.189006] [] ? __hrtimer_start_range_ns+0x31f/0x331
2011-09-08 14:07:44 [ 2316.194086] [] schedule_hrtimeout_range_clock+0xda/0x123
2011-09-08 14:07:44 [ 2316.194343] [] sys_epoll_wait+0x22a/0x387
2011-09-08 14:07:44 [ 2316.195452] [] ? sys_semtimedop+0x2f3/0x845
2011-09-08 14:07:44 ffff88017bcb1fd8[ 2316.197637] httpd S 0 18862 7005 0x00000000
2011-09-08 14:07:44 [ 2316.199054] [] ? cp_new_stat+0xe2/0xee
2011-09-08 14:07:44 [ 2316.200226] Call Trace:
2011-09-08 14:07:44 [ 2316.200920] [] sys_semop+0xb/0xd
2011-09-08 14:07:44 [ 2316.204146] [] ? sys_select+0xa7/0xbd
2011-09-08 14:07:44 0000000000000001[ 2316.204909] ffff88017be49fd8[ 2316.205128] Call Trace:
2011-09-08 14:07:44 ffff88043fc13460 00000000000280da[ 2316.206863] [] ? try_to_wake_up+0x2de/0x2f0
2011-09-08 14:07:44
2011-09-08 14:07:44 ffff88017bc05dd8[ 2316.208717] [] mutex_lock+0x16/0x27
2011-09-08 14:07:44 [ 2316.208717] [] mutex_lock+0x16/0x27
2011-09-08 14:07:44 [ 2316.213785] [] generic_file_llseek+0x27/0x57
2011-09-08 14:07:44 [ 2316.213785] [] generic_file_llseek+0x27/0x57
2011-09-08 14:07:44 [ 2316.215067] [] ? sys_semtimedop+0x2f3/0x845
2011-09-08 14:07:44 [ 2316.215637] httpd S 0 18870 7005 0x00000000
2011-09-08 14:07:44 [ 2316.216998] [] ? ktime_get_ts+0xad/0xba
2011-09-08 14:07:44 ffff88043ffd9e60[ 2316.218292] [] ? sys_semtimedop+0x2f3/0x845
2011-09-08 14:07:44 [ 2316.218800] [] ? sys_select+0xa7/0xbd
2011-09-08 14:07:44 ffff88017c0511f8 [ 2316.219342] 0000000000004000[ 2316.220346] [] ? sockfd_lookup_light+0x1b/0x54
2011-09-08 14:07:44 0000000000000046[ 2316.221627] [] ? perf_event_exit_task+0xc0/0x1cb
2011-09-08 14:07:44 [ 2316.221877] [] system_call_fastpath+0x16/0x1b
2011-09-08 14:07:44 [ 2316.222859] [] ? ipc_lock+0x2e/0x45
2011-09-08 14:07:44 [ 2316.226604] [] system_call_fastpath+0x16/0x1b
2011-09-08 14:07:44 [ 2316.227700] [] ? sys_semtimedop+0x2f3/0x845
2011-09-08 14:07:44
2011-09-08 14:07:44 [ 2316.233763] 0000000000004000
2011-09-08 14:07:44 ffff88017b8001c0[ 2316.233763] 0000000000004000 ffff88017b8001c0 0000000000010640[ 2316.234889] [] ? __hrtimer_start_range_ns+0x31f/0x331
2011-09-08 14:07:44 0 18881 7005 0x00000000
2011-09-08 14:07:44 [ 2316.235496] 0000000000004000[ 2316.237673] [] ? sys_semtimedop+0x2f3/0x845
2011-09-08 14:07:44 ffff88017ba08010 ffff88017ba09fd8 ffff88043ffd9e60 ffff88017ba07138 [ 2316.241078] [] sys_semtimedop+0x6c1/0x845
2011-09-08 14:07:44 [ 2316.241458] [] ? poll_select_copy_remaining+0xc5/0xe9
2011-09-08 14:07:44 0 18885 7005 0x00000000
2011-09-08 14:07:44 ffff88017ba062c0 ffff88017bab1be8[ 2316.242690] [] sys_semtimedop+0x6c1/0x845
2011-09-08 14:07:44 ffff88017baf5520 0000000000000001e9
2011-09-08 14:07:44 0000000000000082 ffff88017baf4c10 0 18889 7005 0x00000000
2011-09-08 14:07:44 0 18889 7005 0x00000000
2011-09-08 14:07:44 [ 2316.253354] ffff88017b409ce8[ 2316.253354] ffff88017b409ce8 0000000000000082 0000000000000082 ffff88043f810080[ 2316.255060] 0000000000004000[ 2316.256245] [] ? sys_select+0xa7/0xbd
2011-09-08 14:07:44 [ 2316.257475] [] ? core_sys_select+0x1a3/0x253
2011-09-08 14:07:44 0000000000010640[ 2316.261470] ffff88017b597ce8 ffff88017b597fd8[ 2316.262812] [] ? ktime_get_ts+0xad/0xba
2011-09-08 14:07:44 [ 2316.263920] Call Trace:
2011-09-08 14:07:44 ffff88017b680010 ffff88017b680010[ 2316.266351] httpd S[ 2316.266288] [] system_call_fastpath+0x16/0x1b
2011-09-08 14:07:44 [ 2316.267524] [] ? handle_mm_fault+0x24c/0x263
2011-09-08 14:07:44 [ 2316.267903] [] sys_semop+0xb/0xd
2011-09-08 14:07:44 [ 2316.267965] [] system_call_fastpath+0x16/0x1b
2011-09-08 14:07:44 [ 2316.273032] httpd S[ 2316.267965] [] system_call_fastpath+0x16/0x1b
2011-09-08 14:07:44 [ 2316.273032] httpd S 0 18898 7005 0x00000000
2011-09-08 14:07:44 ffff88017b6c6d10 0000000000000001[ 2316.273998] [] ? __alloc_pages_nodemask+0x137/0x740
2011-09-08 14:07:44 [ 2316.274318] [] ? handle_mm_fault+0x24c/0x263
2011-09-08 14:07:44 ffff88017b771fd8[ 2316.275611] [] ? timerqueue_add+0x81/0x9f
2011-09-08 14:07:44 ffff88017b7d7fd8[ 2316.277046] Call Trace:
2011-09-08 14:07:44 0000000000000082 ffff88017b01ffd8[ 2316.278658] Call Trace:
2011-09-08 14:07:44 [ 2316.278843] [] sys_semtimedop+0x6c1/0x845
2011-09-08 14:07:44 [ 2316.280775] [] ? do_page_fault+0x342/0x37d
2011-09-08 14:07:44 000000007b0e7bb8 ffff88017b0e7fd8[ 2316.282964] ffff88017b12ddc8[ 2316.284002] [] ? update_rmtp+0x60/0x60
2011-09-08 14:07:44 ffff88017b801038 [ 2316.285362] [] ? do_lookup+0x55/0x2ba
2011-09-08 14:07:44 [ 2316.285913] [] ? __hrtimer_start_range_ns+0x31f/0x331
2011-09-08 14:07:44 ffff88017b9630f8 [ 2316.286522] 0000000000004000 ffff88017b199b68[ 2316.287019] [] ? handle_mm_fault+0x24c/0x263
2011-09-08 14:07:44 0 18907 18906 0x00000000
2011-09-08 14:07:44 0 18907 18906 0x00000000
2011-09-08 14:07:44 [ 2316.292926] ffff88017b19beb8[ 2316.292926] ffff88017b19beb8 0000000000000046 0000000000000046[ 2316.293644] [] ? do_notify_parent+0x1ca/0x1df
2011-09-08 14:07:44 [ 2316.294021] qpsmtpd-prefork D[ 2316.295258] [] mutex_lock+0x16/0x27
2011-09-08 14:07:44 [ 2316.295696] [] sys_open+0x1b/0x1d
2011-09-08 14:07:44 [ 2316.297044] [] __mutex_lock_slowpath+0xd4/0x119
2011-09-08 14:07:44 ffff88017b275fd8[ 2316.298528] Call Trace:
2011-09-08 14:07:44
2011-09-08 14:07:44 [ 2316.299815] [] ? generic_write_end+0x65/0x73
2011-09-08 14:07:44 [ 2316.300134] [] mutex_lock+0x16/0x27
2011-09-08 14:07:44 [ 2316.303782] [] ? do_lookup+0x55/0x2ba
2011-09-08 14:07:44 [ 2316.304094] [] path_openat+0xc5/0x336
2011-09-08 14:07:44 [ 2316.304406] qpsmtpd-prefork D[ 2316.304762] 0000000000004000[ 2316.305642] [] mutex_lock+0x16/0x27
2011-09-08 14:07:44 ffff88017b3bbfd8[ 2316.306999] Call Trace:
2011-09-08 14:07:44
2011-09-08 14:07:44
2011-09-08 14:07:44 [ 2316.312730] 0000000000004000[ 2316.312730] 0000000000004000 ffff88017b3b89d0 ffff88017b3b89d0[ 2316.314508] [] ? xfs_get_blocks_direct+0x11/0x11
2011-09-08 14:07:44 [ 2316.315810] 0000000000004000[ 2316.316941] [] path_openat+0xc5/0x336
2011-09-08 14:07:44 [ 2316.318048] Call Trace:
2011-09-08 14:07:44 [ 2316.318994] ffff88017ad03fd8 0000000000010640 000000000000007f[ 2316.320552] [] ? __alloc_pages_nodemask+0x137/0x740
2011-09-08 14:07:44 ffff88017adc7fd8[ 2316.322231] Call Trace:
2011-09-08 14:07:44 ffff88017adea010
2011-09-08 14:07:44 [ 2316.328019] Call Trace:
2011-09-08 14:07:44 [ 2316.328019] Call Trace:
2011-09-08 14:07:44 [ 2316.333082] [] ? xfs_get_blocks_direct+0x11/0x11
2011-09-08 14:07:44 [ 2316.333082] [] ? xfs_get_blocks_direct+0x11/0x11
2011-09-08 14:07:44 [ 2316.333146] [] ? generic_write_end+0x65/0x73
2011-09-08 14:07:44 [ 2316.333276] [] ? xfs_vm_write_end+0x24/0x46
2011-09-08 14:07:44 [ 2316.334028] postimap.pl D ffff88017af6c010
2011-09-08 14:07:44 [ 2316.336612] [] ? link_path_walk+0xa8/0x6f7
2011-09-08 14:07:44 [ 2316.337238] httpd S[ 2316.338207] [] ? __alloc_pages_nodemask+0x137/0x740
2011-09-08 14:07:44 0000000000010640[ 2316.339762] [] ? sys_semtimedop+0x2f3/0x845
2011-09-08 14:07:44
2011-09-08 14:07:44
2011-09-08 14:07:44 ffff88017a82a010 0000000000010640[ 2316.343811] ffff88017a857e78[ 2316.344767] [] __mutex_lock_slowpath+0xd4/0x119
2011-09-08 14:07:44 ffff88043ffd9e60[ 2316.347739] [] ? sys_semtimedop+0x2f3/0x845
2011-09-08 14:07:44 [ 2316.352815] [] sys_semtimedop+0x6c1/0x845
2011-09-08 14:07:44 [ 2316.347739] [] ? sys_semtimedop+0x2f3/0x845
2011-09-08 14:07:44 [ 2316.352815] [] sys_semtimedop+0x6c1/0x845
2011-09-08 14:07:44 [ 2316.352879] [] ? handle_pte_fault+0x2c2/0x770
2011-09-08 14:07:44
2011-09-08 14:07:44 ffff88017a99c010 ffff88017a99c010[ 2316.356725] qmail-inject D 0 18942 15453 0x00020000
2011-09-08 14:07:44 ffff88017b1bb560 ffff88017b1bb560[ 2316.359376] [] ? handle_mm_fault+0x24c/0x263
2011-09-08 14:07:44 [ 2316.359817] httpd S 0000000000000001[ 2316.361116] [] ? ktime_get_ts+0xad/0xba
2011-09-08 14:07:44 0000000021b5000f[ 2316.363618] ffff88017aadbfd8 0000000000000001[ 2316.365956] [] sys_semop+0xb/0xd
2011-09-08 14:07:44 ffff88017aba4010 ffff88017aba4010[ 2316.367619] [] ? sys_select+0xa7/0xbd
2011-09-08 14:07:44 [ 2316.367619] [] ? sys_select+0xa7/0xbd
2011-09-08 14:07:44 [ 2316.372690] [] sys_semop+0xb/0xd
2011-09-08 14:07:44 [ 2316.372690] [] sys_semop+0xb/0xd
2011-09-08 14:07:44 [ 2316.372751] [] system_call_fastpath+0x16/0x1b
2011-09-08 14:07:44 [ 2316.372814] httpd S[ 2316.372751] [] system_call_fastpath+0x16/0x1b
2011-09-08 14:07:44 [ 2316.372814] httpd S ffff88017aba3238 0000000000010640 ffff88043ffd9e60 ffff88017a42e010[ 2316.375727] [] ? ktime_get_ts+0xad/0xba
2011-09-08 14:07:44
2011-09-08 14:07:44 ffff88043ffd9e60[ 2316.381067] ffff88017a57bce8[ 2316.382039] [] ? handle_pte_fault+0x2c2/0x770
2011-09-08 14:07:44 ffff88017a5c1cc0[ 2316.385819] [] system_call_fastpath+0x16/0x1b
2011-09-08 14:07:44 [ 2316.386238] 0000000000004000[ 2316.386863] [] sys_semtimedop+0x6c1/0x845
2011-09-08 14:07:44 [ 2316.387371] [] sys_semop+0xb/0xd
2011-09-08 14:07:44 [ 2316.387371] [] sys_semop+0xb/0xd
2011-09-08 14:07:44 [ 2316.392446] [] system_call_fastpath+0x16/0x1b
2011-09-08 14:07:44 [ 2316.392446] [] system_call_fastpath+0x16/0x1b
2011-09-08 14:07:44 [ 2316.392509] httpd S[ 2316.392509] httpd S[ 2316.393934] [] ? sys_select+0xa7/0xbd
2011-09-08 14:07:44 [ 2316.395039] [] ? sys_semtimedop+0x2f3/0x845
2011-09-08 14:07:44 0000000000010640 ffff88017a7345a8 [ 2316.398512] [] sys_semtimedop+0x6c1/0x845
2011-09-08 14:07:44 [ 2316.398830] [] sys_semop+0xb/0xd
2011-09-08 14:07:44 [ 2316.399870] [] ? enqueue_hrtimer+0x48/0x74
2011-09-08 14:07:44 0000000000010640[ 2316.402118] [] system_call_fastpath+0x16/0x1b
2011-09-08 14:07:44 0000000000000086[ 2316.405773] [] ? vfsmount_lock_global_unlock_online+0x40/0x4f
2011-09-08 14:07:44 [ 2316.406899] Call Trace:
2011-09-08 14:07:44 [ 2316.407020] [] ? tcp_recvmsg+0x7c6/0x8e4
2011-09-08 14:07:44 [ 2316.407020] [] ? tcp_recvmsg+0x7c6/0x8e4
2011-09-08 14:07:44 [ 2316.412087] [] sys_semtimedop+0x6c1/0x845
2011-09-08 14:07:44 [ 2316.412087] [] sys_semtimedop+0x6c1/0x845
2011-09-08 14:07:44 [ 2316.412150] [] ? pipe_read+0x32e/0x395
2011-09-08 14:07:44 [ 2316.412606] [] system_call_fastpath+0x16/0x1b
2011-09-08 14:07:44 ffff88017a1bffd8[ 2316.416127] qmail-queue D[ 2316.416127] qmail-queue D[ 2316.416981] [] ? in_group_p+0x2b/0x2d
2011-09-08 14:07:44 ffff88017a1f7fd8[ 2316.419469] [] sys_newstat+0x1a/0x3b
2011-09-08 14:07:44
2011-09-08 14:07:44 0000000000000086[ 2316.422682] [] ? do_page_fault+0x342/0x37d
2011-09-08 14:07:44 [ 2316.423786] [] ? __alloc_pages_nodemask+0x137/0x740
2011-09-08 14:07:44 [ 2316.424168] [] ? do_page_fault+0x342/0x37d
2011-09-08 14:07:44 [ 2316.426206] [] vfs_read+0xab/0x14e
2011-09-08 14:07:44 ffff88017a3c6010 ffff88017a3c6010
2011-09-08 14:07:44
2011-09-08 14:07:44 [ 2316.431750] 0000000000004000[ 2316.431750] 0000000000004000 ffff88042e563660 0000000000010640 ffff88017a263bf8 ffff88017a263fd8[ 2316.434313] [] sys_semop+0xb/0xd
2011-09-08 14:07:44 [ 2316.438158] [] ? do_page_fault+0x342/0x37d
2011-09-08 14:07:44 [ 2316.439267] Call Trace:
2011-09-08 14:07:44 0 18985 4735 0x00000000
2011-09-08 14:07:44 [ 2316.441569] [] sys_fchmod+0x88/0xb5
2011-09-08 14:07:44 0000000000000086 ffff88017a1e0a10 ffff880179db19c8 [ 2316.444341] [] sys_semtimedop+0x6c1/0x845
2011-09-08 14:07:44 [ 2316.444721] [] ? poll_select_copy_remaining+0xc5/0xe9
2011-09-08 14:07:44 ffff88043ffd9e60[ 2316.446146] [] ? handle_mm_fault+0x24c/0x263
2011-09-08 14:07:44 [ 2316.446146] [] ? handle_mm_fault+0x24c/0x263
2011-09-08 14:07:44 [ 2316.451222] [] ? do_page_fault+0x342/0x37d
2011-09-08 14:07:44 ffff880179db0240[ 2316.453025] [] ? sys_select+0xa7/0xbd
2011-09-08 14:07:44 ffff88043ffd9e60[ 2316.454256] [] ? core_sys_select+0x1a3/0x253
2011-09-08 14:07:44 ffff880179ee7fd8[ 2316.455855] [] ? __alloc_pages_nodemask+0x137/0x740
2011-09-08 14:07:44 [ 2316.456046] [] ? core_sys_select+0x1a3/0x253
2011-09-08 14:07:44 [ 2316.456363] [] ? poll_select_copy_remaining+0xc5/0xe9
2011-09-08 14:07:44 [ 2316.458352] httpd S ffff88017a6ae9e8 [ 2316.459842] [] ? sys_select+0xa7/0xbd
2011-09-08 14:07:44 0 18995 5906 0x00020000
2011-09-08 14:07:44 [ 2316.460164] ffff880179ffbde8[ 2316.461246] [] lookup_create+0x1e/0x97
2011-09-08 14:07:44 ffff88043ffd9e60[ 2316.462854] [] ? do_page_fault+0x342/0x37d
2011-09-08 14:07:44 [ 2316.463106] [] ? sys_select+0xa7/0xbd
2011-09-08 14:07:44 0000000000000082 ffff880179483fd8 0 18998 7005 0x00000000
2011-09-08 14:07:44 0000000000010640[ 2316.465887] [] ? __hrtimer_start_range_ns+0x31f/0x331
2011-09-08 14:07:44 [ 2316.465887] [] ? __hrtimer_start_range_ns+0x31f/0x331
2011-09-08 14:07:44 [ 2316.470961] [] schedule_hrtimeout_range_clock+0xda/0x123
2011-09-08 14:07:44 [ 2316.470961] [] schedule_hrtimeout_range_clock+0xda/0x123
2011-09-08 14:07:44 [ 2316.471027] [] ? update_rmtp+0x60/0x60
2011-09-08 14:07:44 [ 2316.471027] [] ? update_rmtp+0x60/0x60
2011-09-08 14:07:44 [ 2316.472100] [] system_call_fastpath+0x16/0x1b
2011-09-08 14:07:44 [ 2316.473077] Call Trace:
2011-09-08 14:07:44 [ 2316.473709] [] ? sys_select+0xa7/0xbd
2011-09-08 14:07:44 [ 2316.473896] httpd S[ 2316.475509] httpd S ffff8801795b7fd8[ 2316.476304] Call Trace:
2011-09-08 14:07:44 ffff88017961dfd8
2011-09-08 14:07:44 ffff880179660010 ffff88043ffd9e60[ 2316.483001] [] ? handle_mm_fault+0x24c/0x263
2011-09-08 14:07:44 [ 2316.484299] Call Trace:
2011-09-08 14:07:44 [ 2316.485586] 0000000000004000 ffff880179795fd8 ffff880179795fd8
2011-09-08 14:07:44
2011-09-08 14:07:44 [ 2316.490813] ffff880179795fd8[ 2316.490813] ffff880179795fd8 0000000000010640[ 2316.491318] [] ? core_sys_select+0x1a3/0x253
2011-09-08 14:07:44 [ 2316.492462] ffff880178c17fd8[ 2316.494672] [] ? handle_mm_fault+0x24c/0x263
2011-09-08 14:07:44 ffff880178ca3fd8[ 2316.496158] [] ? __alloc_pages_nodemask+0x137/0x740
2011-09-08 14:07:44 [ 2316.496919] httpd S 0000000000000001[ 2316.498089] [] ? handle_pte_fault+0x2c2/0x770
2011-09-08 14:07:44 [ 2316.498594] qmail-queue D ffff880178d6ffd8[ 2316.500326] postimap.pl D 000000023f5db2d8[ 2316.501557] [] ? vfs_stat+0x16/0x18
2011-09-08 14:07:44 [ 2316.502965] [] __mutex_lock_slowpath+0xd4/0x119
2011-09-08 14:07:44 [ 2316.505137] [] do_sys_open+0x114/0x1a3
2011-09-08 14:07:44 [ 2316.505137] [] do_sys_open+0x114/0x1a3
2011-09-08 14:07:44 [ 2316.510209] [] compat_sys_open+0x15/0x17
2011-09-08 14:07:44 [ 2316.510209] [] compat_sys_open+0x15/0x17
2011-09-08 14:07:44 [ 2316.510332] suexec S 0000000000000082[ 2316.510907] ffff88017a379fd8[ 2316.511810] [] ? set_current_blocked+0x31/0x45
2011-09-08 14:07:44
2011-09-08 14:07:44 [ 2316.513102] [] mutex_lock+0x16/0x27
2011-09-08 14:07:44 [ 2316.514834] ffff880178ea5ce8[ 2316.516178] [] system_call_fastpath+0x16/0x1b
2011-09-08 14:07:44 [ 2316.516596] 0000000000004000 ffffffff811d4cf6[ 2316.518766] [] ? nameidata_to_filp+0x53/0x61
2011-09-08 14:07:44 [ 2316.519893] ffff880178f69fd8[ 2316.520866] [] do_filp_open+0x38/0x88
2011-09-08 14:07:44 [ 2316.521967] [] ? xfs_get_blocks_direct+0x11/0x11
2011-09-08 14:07:44 0 19030 4733 0x00000000
2011-09-08 14:07:44 ffff880178f66080
2011-09-08 14:07:44
2011-09-08 14:07:44 [ 2316.530066] 0000000000004000[ 2316.530066] 0000000000004000 ffff88017704d2e0 ffff88017704d2e0 0000000000010640 0000000000010640[ 2316.530675] [] ? xfs_get_blocks_direct+0x11/0x11 2011-09-08 14:07:44 [ 2316.531247] [] do_last+0x64a/0x651 2011-09-08 14:07:44 [ 2316.531496] [] sys_open+0x1b/0x1d 2011-09-08 14:07:44 0 19032 4733 0x00000000 2011-09-08 14:07:44 ffff88017708ffd8 2011-09-08 14:07:44 [ 2316.537699] ffff880177133fd8[ 2316.538734] [] do_sys_open+0x114/0x1a3 2011-09-08 14:07:44 ffffffff811d4cf6 ffff880177173fd8[ 2316.540344] [] do_last+0x64a/0x651 2011-09-08 14:07:44 [ 2316.540655] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:44 ffff8801771b6010[ 2316.542075] [] vfs_write+0xae/0x151 2011-09-08 14:07:44 ffffffff811d4cf6[ 2316.542838] ffff8801771f7fd8[ 2316.544173] qpsmtpd-prefork D 2011-09-08 14:07:44 2011-09-08 14:07:44 [ 2316.549537] 0000000000004000[ 2316.549537] 0000000000004000 ffff8801771b4140 ffff8801771b4140 0000000000010640 0000000000010640[ 2316.550161] [] ? iov_iter_copy_from_user_atomic+0x83/0x107 2011-09-08 14:07:44 [ 2316.550354] [] ? do_lookup+0x55/0x2ba 2011-09-08 14:07:44 [ 2316.550915] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:44 [ 2316.550978] qpsmtpd-prefork D[ 2316.552775] qpsmtpd-prefork D 0000000000010640 2011-09-08 14:07:44 [ 2316.554136] [] ? nameidata_to_filp+0x53/0x61 2011-09-08 14:07:44 [ 2316.554685] qpsmtpd-prefork S ffffea0005212838[ 2316.556208] 0000000000004000[ 2316.558690] [] ia32_do_call+0x13/0x13 2011-09-08 14:07:44 [ 2316.559797] [] ? handle_pte_fault+0x2c2/0x770 2011-09-08 14:07:44 [ 2316.560176] qmail-queue D[ 2316.561273] [] ? do_lookup+0x55/0x2ba 2011-09-08 14:07:44 [ 2316.561586] [] path_openat+0xc5/0x336 2011-09-08 14:07:44 000000000000002f[ 2316.563381] [] do_filp_open+0x38/0x88 2011-09-08 14:07:44 [ 2316.563695] httpd S ffff88043ffd9e60 ffff88043ffd9e60 2011-09-08 14:07:44 2011-09-08 14:07:44 [ 2316.569500] Call Trace: 2011-09-08 14:07:44 [ 2316.569500] Call Trace: 2011-09-08 14:07:44 [ 2316.569750] [] ? core_sys_select+0x1a3/0x253 2011-09-08 14:07:44 [ 2316.570067] [] ? poll_select_copy_remaining+0xc5/0xe9 2011-09-08 14:07:44 [ 2316.571115] Call Trace: 2011-09-08 14:07:44 ffff880171d36010 ffff880171d77a48 0000000000010640 0000000000000001[ 2316.574707] [] ? sys_semtimedop+0x2f3/0x845 2011-09-08 14:07:44 0000000000010640[ 2316.576195] [] sys_semtimedop+0x6c1/0x845 2011-09-08 14:07:44 0000000000000001[ 2316.579286] Call Trace: 2011-09-08 14:07:44 [ 2316.579597] [] ? link_path_walk+0xa8/0x6f7 2011-09-08 14:07:44 [ 2316.579846] [] ? rb_insert_color+0xbd/0xe6 2011-09-08 14:07:44 [ 2316.580223] httpd S[ 2316.581268] [] ? core_sys_select+0x1a3/0x253 2011-09-08 14:07:44 [ 2316.582413] ffff880171ecbfd8[ 2316.583337] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:44 ffff880171f2e010 2011-09-08 14:07:44 0000000000000001 ffff88043ffd9e60 ffff88043ffd9e60 2011-09-08 14:07:44 2011-09-08 14:07:44 [ 2316.589310] Call Trace: 2011-09-08 14:07:44 ffff88043ffd9e60 ffff88016e01c010[ 2316.594713] [] ? sys_select+0xa7/0xbd 2011-09-08 14:07:44 [ 2316.595814] Call Trace: 2011-09-08 14:07:44 0000000000000001[ 2316.597801] [] ? core_sys_select+0x1a3/0x253 2011-09-08 14:07:44 ffff88016e10ffd8 ffff88016e10ffd8[ 2316.599856] httpd S ffff880178d9e280 0000000000000000[ 2316.601412] [] ? filemap_fault+0x62/0x336 2011-09-08 14:07:44 [ 2316.602156] httpd S[ 2316.603582] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:44 [ 2316.603582] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:44 [ 2316.608653] httpd S[ 2316.608653] httpd S ffff88016e1b3278 ffff88016e1b3278 ffff88043ffd9e60[ 2316.609697] [] ? core_sys_select+0x1a3/0x253 2011-09-08 14:07:44 2011-09-08 14:07:44 [ 2316.611879] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:44 ffff88016e2a4010 ffff88016e2a5fd8 ffff88043ffd9e60 ffff88016e308010 ffff88043ffd9e60 ffff88043ffd9e60[ 2316.617567] [] ? __alloc_pages_nodemask+0x137/0x740 2011-09-08 14:07:44 ffff88016d436000[ 2316.619687] [] ? hrtimer_start_range_ns+0xf/0x11 2011-09-08 14:07:44 [ 2316.620636] [] sys_poll+0x50/0xba 2011-09-08 14:07:44 [ 2316.621557] Call Trace: 2011-09-08 14:07:44 0000000000000001[ 2316.623358] [] ? core_sys_select+0x1a3/0x253 2011-09-08 14:07:44 [ 2316.623358] [] ? core_sys_select+0x1a3/0x253 2011-09-08 14:07:44 [ 2316.628428] [] ? handle_pte_fault+0x2c2/0x770 2011-09-08 14:07:44 [ 2316.628428] [] ? handle_pte_fault+0x2c2/0x770 2011-09-08 14:07:44 [ 2316.628492] [] ? handle_mm_fault+0x24c/0x263 2011-09-08 14:07:44 0000000000000001 0000000000010640[ 2316.630535] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:44 ffff88016d56bbb8 ffff88016d5cf808 [ 2316.633193] [] ? core_sys_select+0x1a3/0x253 2011-09-08 14:07:44 ffff88016d5ce990 0000000000010640[ 2316.634616] [] ? __alloc_pages_nodemask+0x137/0x740 2011-09-08 14:07:44 [ 2316.634935] [] ? handle_mm_fault+0x24c/0x263 2011-09-08 14:07:44 000000000000002f[ 2316.636292] [] ? acl_permission_check+0x69/0x8a 2011-09-08 14:07:44 ffff88016d66c010[ 2316.638453] [] path_openat+0xc5/0x336 2011-09-08 14:07:44 ffff88016d7c0010 ffff88016d7bef78 [ 2316.642893] Call Trace: 2011-09-08 14:07:44 [ 2316.643207] [] ? handle_mm_fault+0x24c/0x263 2011-09-08 14:07:44 [ 2316.643207] [] ? handle_mm_fault+0x24c/0x263 2011-09-08 14:07:44 [ 2316.648277] [] ? do_page_fault+0x342/0x37d 2011-09-08 14:07:44 [ 2316.648277] [] ? do_page_fault+0x342/0x37d 2011-09-08 14:07:44 [ 2316.648340] [] ? seq_release_private+0x3e/0x3e 2011-09-08 14:07:44 [ 2316.648340] [] ? seq_release_private+0x3e/0x3e 2011-09-08 14:07:44 [ 2316.648404] [] ? pde_users_dec+0x15/0x3a 2011-09-08 14:07:44 [ 2316.648404] [] ? pde_users_dec+0x15/0x3a 2011-09-08 14:07:44 [ 2316.648468] [] sys_semop+0xb/0xd 2011-09-08 14:07:44 [ 2316.648948] 0000000000004000 ffff88043ffd9e60[ 2316.649511] [] ? sys_semtimedop+0x2f3/0x845 2011-09-08 14:07:44 [ 2316.649830] [] ? do_page_fault+0x342/0x37d 2011-09-08 14:07:44 [ 2316.651438] [] do_last+0x269/0x651 2011-09-08 14:07:44 [ 2316.651688] [] ? __hrtimer_start_range_ns+0x31f/0x331 2011-09-08 14:07:44 0 19094 18904 0x00000000 2011-09-08 14:07:44 [ 2316.652981] [] ? prepare_to_wait+0x70/0x7b 2011-09-08 14:07:44 ffff88016b88fe38[ 2316.654949] [] vfs_llseek+0x35/0x37 2011-09-08 14:07:44 [ 2316.656242] [] ? do_sync_read+0xc7/0x10d 2011-09-08 14:07:44 ffff88043ffd9e60[ 2316.657680] [] sys_semtimedop+0x6c1/0x845 2011-09-08 14:07:44 ffff88016b974ad0 ffff88016b9d9dd8[ 2316.659228] [] ? timerqueue_add+0x81/0x9f 2011-09-08 14:07:44 [ 2316.659548] [] ? hrtimer_start_range_ns+0xf/0x11 2011-09-08 14:07:44 [ 2316.659929] httpd S 0 19099 7005 0x00000000 2011-09-08 14:07:44 2011-09-08 14:07:44 [ 2316.662829] [] ? do_page_fault+0x342/0x37d 2011-09-08 14:07:44 [ 2316.662892] [] ? ktime_get_ts+0xad/0xba 2011-09-08 14:07:44 [ 2316.662892] [] ? ktime_get_ts+0xad/0xba 2011-09-08 14:07:44 [ 2316.667968] [] ? poll_select_copy_remaining+0xc5/0xe9 2011-09-08 14:07:44 [ 2316.667968] [] ? poll_select_copy_remaining+0xc5/0xe9 2011-09-08 14:07:44 [ 2316.668033] [] ? sys_select+0xa7/0xbd 2011-09-08 14:07:44 [ 2316.668033] [] ? sys_select+0xa7/0xbd 2011-09-08 14:07:44 0 19101 7005 0x00000000 2011-09-08 14:07:44 [ 2316.669200] [] ? enqueue_hrtimer+0x48/0x74 2011-09-08 14:07:44 [ 2316.670088] [] ? handle_pte_fault+0x2c2/0x770 2011-09-08 14:07:44 ffff88016bb73fd8[ 2316.673058] Call Trace: 2011-09-08 14:07:44 [ 2316.673307] [] ? core_sys_select+0x1a3/0x253 2011-09-08 14:07:44 ffff88016bbb6010 2011-09-08 14:07:44 [ 2316.675431] httpd S 0 19105 7005 0x00000000 2011-09-08 14:07:44 [ 2316.675787] 0000000000004000[ 2316.676669] [] ? cp_new_stat+0xe2/0xee 2011-09-08 14:07:44 [ 2316.678147] [] ? do_page_fault+0x342/0x37d 2011-09-08 14:07:44 0000000000010640[ 2316.679441] [] ? __hrtimer_start_range_ns+0x31f/0x331 2011-09-08 14:07:44 2011-09-08 14:07:44 0000000000010640[ 2316.682485] [] ? sys_semtimedop+0x2f3/0x845 2011-09-08 14:07:44 [ 2316.682676] [] ? handle_mm_fault+0x24c/0x263 2011-09-08 14:07:44 [ 2316.682676] [] ? handle_mm_fault+0x24c/0x263 2011-09-08 14:07:44 [ 2316.687746] [] ? do_page_fault+0x342/0x37d 2011-09-08 14:07:44 [ 2316.687746] [] ? do_page_fault+0x342/0x37d 2011-09-08 14:07:44 [ 2316.687809] [] ? cp_new_stat+0xe2/0xee 2011-09-08 14:07:44 0000000000000086[ 2316.689091] [] ? sys_semtimedop+0x2f3/0x845 2011-09-08 14:07:44 [ 2316.689409] [] ? seq_release_private+0x3e/0x3e 2011-09-08 14:07:44 ffff88016adb9178 2011-09-08 14:07:44 [ 2316.690832] [] ? handle_pte_fault+0x2c2/0x770 2011-09-08 14:07:44 [ 2316.691086] [] ? poll_select_copy_remaining+0xc5/0xe9 2011-09-08 14:07:44 ffff88016aea9fd8[ 2316.694628] httpd S 0000000000000082[ 2316.695915] [] ? handle_mm_fault+0x24c/0x263 2011-09-08 14:07:44 0000000000010640[ 2316.697969] httpd S[ 2316.698822] [] ? _raw_spin_unlock_bh+0x10/0x12 2011-09-08 14:07:44 [ 2316.699077] [] ? do_sync_read+0xc7/0x10d 2011-09-08 14:07:44 [ 2316.699408] [] sys_semop+0xb/0xd 2011-09-08 14:07:44 [ 2316.701258] httpd S[ 2316.701834] ffff88016a421fd8 ffffffff8147ab8a[ 2316.702491] [] ? vfsmount_lock_global_unlock_online+0x40/0x4f 2011-09-08 14:07:44 [ 2316.702491] [] ? vfsmount_lock_global_unlock_online+0x40/0x4f 2011-09-08 14:07:44 [ 2316.707581] [] ? mntput_no_expire+0x96/0x155 2011-09-08 14:07:44 [ 2316.707581] [] ? mntput_no_expire+0x96/0x155 2011-09-08 14:07:44 [ 2316.707645] [] ? vfs_read+0xab/0x14e 2011-09-08 14:07:44 [ 2316.707645] [] ? vfs_read+0xab/0x14e 2011-09-08 14:07:44 [ 2316.707707] [] sys_semop+0xb/0xd 2011-09-08 14:07:44 [ 2316.707707] [] sys_semop+0xb/0xd 2011-09-08 14:07:44 [ 2316.710241] Call Trace: 2011-09-08 14:07:44 [ 2316.710618] [] ? handle_mm_fault+0x24c/0x263 2011-09-08 14:07:44 [ 2316.711059] httpd S 0 19121 7005 0x00000000 2011-09-08 14:07:44 [ 2316.714341] httpd S[ 2316.715261] [] ? sys_semtimedop+0x2f3/0x845 2011-09-08 14:07:44 [ 2316.715579] [] ? do_page_fault+0x342/0x37d 2011-09-08 14:07:44 2011-09-08 14:07:44 [ 2316.717434] [] ? do_mmap_pgoff+0x28d/0x2f0 2011-09-08 14:07:44 ffff88016a651a38[ 2316.718715] [] ? cpumask_next_and+0x2d/0x3e 2011-09-08 14:07:44 0000000000000082 2011-09-08 14:07:44 [ 2316.720961] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:44 [ 2316.721380] 0000000000004000[ 2316.722193] [] mutex_lock+0x16/0x27 2011-09-08 14:07:44 [ 2316.722193] [] mutex_lock+0x16/0x27 2011-09-08 14:07:44 [ 2316.727261] [] do_last+0x269/0x651 2011-09-08 14:07:44 [ 2316.727261] [] do_last+0x269/0x651 2011-09-08 14:07:44 [ 2316.727323] [] path_openat+0xc5/0x336 2011-09-08 14:07:44 [ 2316.727577] [] compat_sys_open+0x15/0x17 2011-09-08 14:07:44 [ 2316.728931] [] do_last+0x269/0x651 2011-09-08 14:07:44 [ 2316.729369] httpd S 0000000000010640[ 2316.730530] [] sys_semtimedop+0x6c1/0x845 2011-09-08 14:07:44 ffff880169405fd8[ 2316.732625] [] ? sys_select+0xa7/0xbd 2011-09-08 14:07:44 0 19132 4733 0x00000000 2011-09-08 14:07:44 [ 2316.735779] qpsmtpd-prefork S ffff8801694e9fd8[ 2316.736690] Call Trace: 2011-09-08 14:07:44 ffff880169528010[ 2316.738919] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:44 [ 2316.740022] [] ? wait_noreap_copyout+0xed/0xed 2011-09-08 14:07:44 [ 2316.741129] [] ? iov_iter_copy_from_user_atomic+0x83/0x107 2011-09-08 14:07:44 [ 2316.741947] qpsmtpd-prefork D 0 19138 4733 0x00000000 2011-09-08 14:07:44 [ 2316.747203] ffff8801695edc48 0 19138 4733 0x00000000 2011-09-08 14:07:44 [ 2316.747203] ffff8801695edc48 ffff88016956a080 0000000000000068[ 2316.747985] [] ? generic_write_end+0x65/0x73 2011-09-08 14:07:44 [ 2316.749002] ffff880169631c48[ 2316.749662] Call Trace: 2011-09-08 14:07:44 ffffffff811d4cf6 2011-09-08 14:07:44 [ 2316.751461] Call Trace: 2011-09-08 14:07:44 [ 2316.757627] [] ? generic_write_end+0x65/0x73 2011-09-08 14:07:44 2011-09-08 14:07:44 [ 2316.759415] Call Trace: 2011-09-08 14:07:44 [ 2316.759537] [] ? pagevec_strip+0x5d/0x5d 2011-09-08 14:07:44 [ 2316.759913] [] ? __set_task_blocked+0x60/0x67 2011-09-08 14:07:44 [ 2316.761018] [] ? do_sync_write+0xc7/0x10d 2011-09-08 14:07:44 000000000000007a[ 2316.762127] Call Trace: 2011-09-08 14:07:44 [ 2316.762127] Call Trace: 2011-09-08 14:07:44 [ 2316.767193] [] ? xfs_get_blocks_direct+0x11/0x11 2011-09-08 14:07:44 [ 2316.767193] [] ? xfs_get_blocks_direct+0x11/0x11 2011-09-08 14:07:44 [ 2316.767257] [] ? generic_write_end+0x65/0x73 2011-09-08 14:07:44 [ 2316.767257] [] ? generic_write_end+0x65/0x73 2011-09-08 14:07:44 [ 2316.767321] [] ? iov_iter_copy_from_user_atomic+0x83/0x107 2011-09-08 14:07:44 ffff88016807ffd8[ 2316.768935] Call Trace: 2011-09-08 14:07:44 [ 2316.769306] qpsmtpd-prefork D ffffea0006cc8098[ 2316.770596] [] ? do_lookup+0x55/0x2ba 2011-09-08 14:07:44 [ 2316.770908] [] path_openat+0xc5/0x336 2011-09-08 14:07:44 [ 2316.771158] [] sys_open+0x1b/0x1d 2011-09-08 14:07:44 [ 2316.772200] [] ? generic_write_end+0x65/0x73 2011-09-08 14:07:44 000000023f5db2d8[ 2316.774249] [] sys_mkdirat+0x3d/0xe8 2011-09-08 14:07:44 [ 2316.775354] Call Trace: 2011-09-08 14:07:44 [ 2316.776109] httpd S 2011-09-08 14:07:44 [ 2316.777524] [] ? pde_users_dec+0x15/0x3a 2011-09-08 14:07:44 ffff880168259fd8[ 2316.780246] Call Trace: 2011-09-08 14:07:44 [ 2316.780559] [] ? handle_pte_fault+0x2c2/0x770 2011-09-08 14:07:44 ffffffff8147ab8a 2011-09-08 14:07:44 2011-09-08 14:07:44 [ 2316.786929] Call Trace: 2011-09-08 14:07:44 [ 2316.786929] Call Trace: 2011-09-08 14:07:44 [ 2316.786986] [] ? _raw_spin_unlock_bh+0x10/0x12 2011-09-08 14:07:44 [ 2316.786986] [] ? _raw_spin_unlock_bh+0x10/0x12 2011-09-08 14:07:44 [ 2316.787050] [] ? tcp_recvmsg+0x7c6/0x8e4 2011-09-08 14:07:44 [ 2316.787050] [] ? tcp_recvmsg+0x7c6/0x8e4 2011-09-08 14:07:44 ffff880168322010[ 2316.789115] [] ? cp_new_stat+0xe2/0xee 2011-09-08 14:07:44 [ 2316.790067] ffff880168369fd8[ 2316.792717] httpd S[ 2316.794515] httpd S[ 2316.795311] Call Trace: 2011-09-08 14:07:44 [ 2316.796002] httpd S[ 2316.796002] httpd S ffff8801654ba010 0000000000010640[ 2316.797047] [] ? handle_pte_fault+0x2c2/0x770 2011-09-08 14:07:44 [ 2316.797489] httpd S[ 2316.798342] [] ? __pollwait+0xcb/0xcb 2011-09-08 14:07:44 ffff88016556bfd8[ 2316.800773] [] ? __hrtimer_start_range_ns+0x31f/0x331 2011-09-08 14:07:44 2011-09-08 14:07:44 2011-09-08 14:07:44 [ 2316.806611] ffff8801655d7fd8[ 2316.806611] ffff8801655d7fd8 0000000000010640 0000000000010640 ffff88043fd0d2e0 0000000000000082[ 2316.808803] [] generic_file_llseek+0x27/0x57 2011-09-08 14:07:44 0 19172 7005 0x00000000 2011-09-08 14:07:44 [ 2316.810034] [] sys_semtimedop+0x6c1/0x845 2011-09-08 14:07:44 [ 2316.810414] [] ? poll_select_copy_remaining+0xc5/0xe9 2011-09-08 14:07:44 ffff880168141038 [ 2316.811021] 0000000000004000[ 2316.812147] [] sys_semop+0xb/0xd 2011-09-08 14:07:44 000280da3de2d058 ffff8801655e5fd8[ 2316.816219] Call Trace: 2011-09-08 14:07:44 [ 2316.816967] httpd S ffff8801657abfd8 ffff8801657abcd8[ 2316.817998] [] ? enqueue_hrtimer+0x48/0x74 2011-09-08 14:07:44 [ 2316.819212] ffff88015fc0dfd8 ffff88016556f1b8 [ 2316.821106] [] sys_semtimedop+0x6c1/0x845 2011-09-08 14:07:44 [ 2316.821360] [] ? do_page_fault+0x342/0x37d 2011-09-08 14:07:44 [ 2316.821360] [] ? do_page_fault+0x342/0x37d 2011-09-08 14:07:44 [ 2316.826440] [] ? ktime_get_ts+0xad/0xba 2011-09-08 14:07:44 [ 2316.826440] [] ? ktime_get_ts+0xad/0xba 2011-09-08 14:07:44 ffff88016a706480 2011-09-08 14:07:44 [ 2316.827735] [] sys_semtimedop+0x6c1/0x845 2011-09-08 14:07:44 2011-09-08 14:07:44 0000000000000082 2011-09-08 14:07:44 2011-09-08 14:07:44 [ 2316.831763] [] compat_sys_open+0x15/0x17 2011-09-08 14:07:44 [ 2316.833373] suexec S 0000000000000082[ 2316.834726] [] ? mmap_region+0x3a9/0x4b2 2011-09-08 14:07:44 ffff88015fda7c08 [ 2316.835510] 0000000000004000 ffff88015fe1ba38[ 2316.836890] suexec S[ 2316.837686] Call Trace: 2011-09-08 14:07:44 ffff88015fe54010[ 2316.839720] [] vfs_llseek+0x35/0x37 2011-09-08 14:07:44 [ 2316.840953] [] ? handle_pte_fault+0x2c2/0x770 2011-09-08 14:07:44 [ 2316.841081] [] ? do_page_fault+0x342/0x37d 2011-09-08 14:07:44 [ 2316.841081] [] ? do_page_fault+0x342/0x37d 2011-09-08 14:07:44 [ 2316.846260] [] ? seq_release_private+0x3e/0x3e 2011-09-08 14:07:44 [ 2316.846260] [] ? seq_release_private+0x3e/0x3e 2011-09-08 14:07:44 [ 2316.846324] [] ? pde_users_dec+0x15/0x3a 2011-09-08 14:07:44 [ 2316.847558] [] ? handle_pte_fault+0x2c2/0x770 2011-09-08 14:07:44 [ 2316.847876] [] sys_semop+0xb/0xd 2011-09-08 14:07:44 [ 2316.848796] Call Trace: 2011-09-08 14:07:44 [ 2316.849173] [] ? do_page_fault+0x342/0x37d 2011-09-08 14:07:44 2011-09-08 14:07:44 2011-09-08 14:07:44 [ 2316.852655] [] ? sys_select+0xa7/0xbd 2011-09-08 14:07:44 ffff88015ffee010[ 2316.854141] [] ? seq_release_private+0x3e/0x3e 2011-09-08 14:07:44 ffff88015fe505a8 [ 2316.857227] [] pipe_read+0x315/0x395 2011-09-08 14:07:44 [ 2316.857541] [] vfs_read+0xab/0x14e 2011-09-08 14:07:44 [ 2316.859007] [] vfs_llseek+0x35/0x37 2011-09-08 14:07:44 0 19203 7005 0x00000000 2011-09-08 14:07:44 ffff88015ffa8080 ffff88015d93b888 ffff88015d93dfd8 ffff88015d93dfd8 2011-09-08 14:07:44 2011-09-08 14:07:44 [ 2316.866274] ffff88015d93dfd8[ 2316.866274] ffff88015d93dfd8 0000000000010640[ 2316.867311] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:44 [ 2316.868935] httpd S 2011-09-08 14:07:44 2011-09-08 14:07:44 ffffffff8147ab8a[ 2316.870367] [] ? mntput_no_expire+0x96/0x155 2011-09-08 14:07:44 [ 2316.870617] httpd S ffff88015da2bbe8[ 2316.871597] [] sys_semtimedop+0x6c1/0x845 2011-09-08 14:07:44 ffff88015da28a50[ 2316.873543] [] sys_semop+0xb/0xd 2011-09-08 14:07:44 0000000000000001[ 2316.874244] ffff88015dad3fd8[ 2316.874464] Call Trace: 2011-09-08 14:07:44 [ 2316.875094] [] sys_semop+0xb/0xd 2011-09-08 14:07:44 [ 2316.875218] httpd S[ 2316.876822] httpd S 2011-09-08 14:07:44 ffff88015dbc1fd8[ 2316.879355] [] ? core_sys_select+0x1a3/0x253 2011-09-08 14:07:44 [ 2316.879923] httpd S 0000000000000082 ffff88015d421fd8 2011-09-08 14:07:44 [ 2316.880840] [] ? ipc_lock+0x2e/0x45 2011-09-08 14:07:44 [ 2316.880840] [] ? ipc_lock+0x2e/0x45 2011-09-08 14:07:44 [ 2316.885909] [] ? sys_semtimedop+0x2f3/0x845 2011-09-08 14:07:44 [ 2316.885909] [] ? sys_semtimedop+0x2f3/0x845 2011-09-08 14:07:44 [ 2316.885973] [] sys_semtimedop+0x6c1/0x845 2011-09-08 14:07:44 [ 2316.885973] [] sys_semtimedop+0x6c1/0x845 2011-09-08 14:07:44 [ 2316.889453] [] ? vfsmount_lock_global_unlock_online+0x40/0x4f 2011-09-08 14:07:44 [ 2316.890577] Call Trace: 2011-09-08 14:07:44 0000000000010640 ffff88015d575fd8[ 2316.892782] [] sys_semop+0xb/0xd 2011-09-08 14:07:44 [ 2316.894126] [] ? handle_pte_fault+0x2c2/0x770 2011-09-08 14:07:44 [ 2316.894381] [] ? pde_users_dec+0x15/0x3a 2011-09-08 14:07:44 [ 2316.895487] [] ? sys_semtimedop+0x2f3/0x845 2011-09-08 14:07:44 [ 2316.895805] [] ? cp_new_stat+0xe2/0xee 2011-09-08 14:07:44 0000000000000082[ 2316.897099] [] ? handle_pte_fault+0x2c2/0x770 2011-09-08 14:07:44 [ 2316.897479] httpd S ffff88015d6cbc98[ 2316.900063] [] ? tcp_recvmsg+0x7c6/0x8e4 2011-09-08 14:07:44 [ 2316.900459] [] ? mntput_no_expire+0x96/0x155 2011-09-08 14:07:44 [ 2316.900459] [] ? mntput_no_expire+0x96/0x155 2011-09-08 14:07:44 [ 2316.905531] [] ? vfs_read+0xab/0x14e 2011-09-08 14:07:44 [ 2316.905531] [] ? vfs_read+0xab/0x14e 2011-09-08 14:07:44 [ 2316.905593] [] sys_semop+0xb/0xd 2011-09-08 14:07:44 [ 2316.905593] [] sys_semop+0xb/0xd 2011-09-08 14:07:44 [ 2316.905717] httpd S ffff88015d771bf8[ 2316.907029] [] ? mntput_no_expire+0x96/0x155 2011-09-08 14:07:44 [ 2316.909946] [] ? do_sync_read+0xc7/0x10d 2011-09-08 14:07:44 ffff88043ffd9e60 0000000000010640[ 2316.913731] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:44 [ 2316.914898] [] generic_file_llseek+0x27/0x57 2011-09-08 14:07:44 ffff88015cd2b4e0 ffff88043f592858[ 2316.917847] [] ? link_path_walk+0xa8/0x6f7 2011-09-08 14:07:44 [ 2316.918222] [] ? __hrtimer_start_range_ns+0x31f/0x331 2011-09-08 14:07:44 ffff88015fe63ac8 [ 2316.919642] [] mutex_lock+0x16/0x27 2011-09-08 14:07:44 [ 2316.920019] [] do_sys_open+0x114/0x1a3 2011-09-08 14:07:44 [ 2316.920019] [] do_sys_open+0x114/0x1a3 2011-09-08 14:07:44 [ 2316.925096] [] compat_sys_open+0x15/0x17 2011-09-08 14:07:44 [ 2316.925096] [] compat_sys_open+0x15/0x17 2011-09-08 14:07:44 [ 2316.925576] 0000000000004000 ffff88015cd2a2c0 0000000000010640[ 2316.928285] qpsmtpd-prefork D ffff88015cd2abd0 0000000000010640[ 2316.931757] cron S 0000000000000000[ 2316.932674] [] ? zone_watermark_ok+0x1a/0x1c 2011-09-08 14:07:44 ffff88043f592858[ 2316.934656] [] ? link_path_walk+0xa8/0x6f7 2011-09-08 14:07:44 [ 2316.935283] nrpe S[ 2316.936578] nrpe S[ 2316.937373] Call Trace: 2011-09-08 14:07:44 ffff88015fe628a8 [ 2316.939351] [] ? xfs_iext_bno_to_ext+0xb8/0x13d 2011-09-08 14:07:44 [ 2316.939729] [] user_path_at+0x54/0x95 2011-09-08 14:07:44 [ 2316.939729] [] user_path_at+0x54/0x95 2011-09-08 14:07:44 [ 2316.944801] [] vfs_fstatat+0x49/0x76 2011-09-08 14:07:44 [ 2316.944801] [] vfs_fstatat+0x49/0x76 2011-09-08 14:07:44 0000000000010640[ 2316.946961] [] ia32_do_call+0x13/0x13 2011-09-08 14:07:44 [ 2316.947936] [] ? acl_permission_check+0x69/0x8a 2011-09-08 14:07:44 [ 2316.949328] ffff88015cec7fd8[ 2316.949732] [] ? do_lookup+0x55/0x2ba 2011-09-08 14:07:44 0 19255 5906 0x00020000 2011-09-08 14:07:44 [ 2316.953057] [] ? vfs_stat+0x16/0x18 2011-09-08 14:07:44 0 19256 4735 0x00000000 2011-09-08 14:07:44 [ 2316.954471] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:44 [ 2316.954890] 0000000000004000 ffff88015cf32d10[ 2316.956558] Call Trace: 2011-09-08 14:07:44 0 19260 4724 0x00020000 2011-09-08 14:07:44 [ 2316.958781] [] ? hrtimer_start_range_ns+0xf/0x11 2011-09-08 14:07:44 [ 2316.959034] [] ? poll_freewait+0x8d/0x8d 2011-09-08 14:07:44 [ 2316.959538] [] ? lock_timer_base+0x26/0x4a 2011-09-08 14:07:44 [ 2316.959538] [] ? lock_timer_base+0x26/0x4a 2011-09-08 14:07:44 [ 2316.964613] [] ? common_interrupt+0xe/0x13 2011-09-08 14:07:44 [ 2316.964738] [] ? compat_get_fd_set+0x9e/0xae 2011-09-08 14:07:44 [ 2316.965305] qmail-popup S ffff88015ccc8c90 0000000000000082[ 2316.967953] [] sys_mkdirat+0x3d/0xe8 2011-09-08 14:07:44 [ 2316.968866] .sysctl_sched_wakeup_granularity : 4.000000 2011-09-08 14:07:44 [ 2316.969171] .sysctl_sched_tunable_scaling : 1 (logaritmic) 2011-09-08 14:07:44 [ 2316.970415] .ttwu_count : 3081398 2011-09-08 14:07:44 [ 2316.971468] .rt_runtime : 950.000000 2011-09-08 14:07:44 [ 2316.973244] 2011-09-08 14:07:44 [ 2316.974273] 2011-09-08 14:07:44 [ 2316.975344] .cpu_load[2] : 0 2011-09-08 14:07:44 [ 2316.976288] .spread : 0.000000 2011-09-08 14:07:44 [ 2316.976288] .spread : 0.000000 2011-09-08 14:07:44 [ 2316.981480] .spread0 : -4591764.655376 2011-09-08 14:07:44 [ 2316.981480] .spread0 : -4591764.655376 2011-09-08 14:07:44 [ 2316.981542] .nr_spread_over : 579 2011-09-08 14:07:44 [ 2316.981542] .nr_spread_over : 579 2011-09-08 14:07:44 [ 2316.981601] .nr_running : 0 2011-09-08 14:07:44 [ 2316.981601] .nr_running : 0 2011-09-08 14:07:44 [ 2316.981659] .load : 0 2011-09-08 14:07:44 [ 2316.981824] .rt_nr_running : 0 2011-09-08 14:07:44 [ 2316.982064] 2011-09-08 14:07:44 [ 2316.982898] .curr->pid : 0 2011-09-08 14:07:44 [ 2316.983611] .ttwu_count : 914456 2011-09-08 14:07:44 [ 2316.984603] .rt_time : 0.000000 2011-09-08 14:07:44 [ 2316.985559] .curr->pid : 0 2011-09-08 14:07:44 [ 2316.986559] .MIN_vruntime : 0.000001 2011-09-08 14:07:44 [ 2316.987753] 2011-09-08 14:07:44 [ 2316.988693] .sched_switch : 0 2011-09-08 14:07:44 [ 2316.990164] ---------------------------------------------------------------------------------------------------------- 2011-09-08 14:07:44 [ 2316.991411] .yld_count : 413 2011-09-08 14:07:44 [ 2316.992177] .spread : 0.000000 2011-09-08 14:07:44 [ 2316.997244] .spread0 : -6731533.885230 2011-09-08 14:07:44 [ 2316.992177] .spread : 0.000000 2011-09-08 14:07:44 [ 2316.997244] .spread0 : -6731533.885230 2011-09-08 14:07:44 [ 2316.997483] rt_rq[6]: 2011-09-08 14:07:44 [ 2316.997588] .rt_nr_running : 0 2011-09-08 14:07:44 [ 2316.998481] .nr_load_updates : 541083 2011-09-08 14:07:44 [ 2316.999493] cfs_rq[7]: 2011-09-08 14:07:44 [ 2317.000854] 2011-09-08 14:07:44 [ 2317.001740] .cpu_load[3] : 0 2011-09-08 14:07:44 [ 2317.002036] .sched_goidle : 843803 2011-09-08 14:07:44 [ 2317.003639] 2011-09-08 14:07:44 [ 2317.005692] .rt_nr_running : 0 2011-09-08 14:07:44 [ 2317.007004] .cpu_load[2] : 2 2011-09-08 14:07:44 [ 2317.007948] .spread : 0.000000 2011-09-08 14:07:44 [ 2317.007948] .spread : 0.000000 2011-09-08 14:07:44 [ 2317.013014] .spread0 : -7538872.633166 2011-09-08 14:07:44 [ 2317.013014] .spread0 : -7538872.633166 2011-09-08 14:07:44 [ 2317.013076] .nr_spread_over : 2743 2011-09-08 14:07:44 [ 2317.013076] .nr_spread_over : 2743 2011-09-08 14:07:44 [ 2317.013595] .rt_time : 0.000000 2011-09-08 14:07:44 [ 2317.014672] .cpu_load[0] : 1024 2011-09-08 14:07:44 [ 2317.016748] cpu#12, 2526.849 MHz 2011-09-08 14:07:44 [ 2317.017214] .curr->pid : 6497 2011-09-08 14:07:44 [ 2317.017513] .cpu_load[3] : 128 2011-09-08 14:07:44 [ 2317.018582] .nr_running : 2 2011-09-08 14:07:44 [ 2317.018701] rt_rq[12]: 2011-09-08 14:07:44 [ 2317.020267] .curr->pid : 0 2011-09-08 14:07:44 [ 2317.021325] .exec_clock : 138863.533297 2011-09-08 14:07:44 [ 2317.021750] .nr_running : 0 2011-09-08 14:07:44 [ 2317.021974] .rt_nr_running : 0 2011-09-08 14:07:44 [ 2317.023466] .yld_count : 219 2011-09-08 14:07:44 [ 2317.023525] .sched_switch : 0 2011-09-08 14:07:44 [ 2317.023989] .exec_clock : 85896.777518 2011-09-08 14:07:44 [ 2317.023989] .exec_clock : 85896.777518 2011-09-08 14:07:44 [ 2317.029055] .MIN_vruntime : 0.000001 2011-09-08 14:07:44 [ 2317.029642] .rt_nr_running : 0 2011-09-08 14:07:44 [ 2317.029882] runnable tasks: 2011-09-08 14:07:44 [ 2317.031016] .cpu_load[3] : 0 2011-09-08 14:07:44 [ 2317.031372] .avg_idle : 1000000 2011-09-08 14:07:44 [ 2317.031658] .exec_clock : 49523.385387 2011-09-08 14:07:44 [ 2317.031901] .spread : 0.000000 2011-09-08 14:07:44 [ 2317.032914] 2011-09-08 14:07:43 [ 2315.068303] [] ? tcp_rcv_established+0xe3/0x8aa 2011-09-08 14:07:43 [ 2315.073371] [] ? tcp_v4_do_rcv+0x6c/0x277 2011-09-08 14:07:43 [ 2315.073371] [] ? tcp_v4_do_rcv+0x6c/0x277 2011-09-08 14:07:43 [ 2315.073434] [] ? tcp_current_mss+0x5e/0x7d 2011-09-08 14:07:43 [ 2315.073434] [] ? tcp_current_mss+0x5e/0x7d 2011-09-08 14:07:43 ffff88068f1d72b8 [ 2315.075224] [] ? timerqueue_add+0x81/0x9f 2011-09-08 14:07:43 [ 2315.075546] [] ? hrtimer_start_range_ns+0xf/0x11 2011-09-08 14:07:43 [ 2315.076720] Call Trace: 2011-09-08 14:07:43 [ 2315.077097] [] ? update_rmtp+0x60/0x60 2011-09-08 14:07:43 ffff8801930cdfd8 2011-09-08 14:07:43 [ 2315.080001] [] ? core_sys_select+0x1a3/0x253 2011-09-08 14:07:43 [ 2315.080504] imap S ffff88068f3ce7e8 [ 2315.081596] [] ? enqueue_hrtimer+0x48/0x74 2011-09-08 14:07:43 ffff8801932b5fd8 ffff8802145c1908 [ 2315.084819] [] ? wake_up_bit+0x23/0x23 2011-09-08 14:07:43 ffffffff8121058d[ 2315.087173] [] ? tcp_event_new_data_sent+0x9a/0x9c 2011-09-08 14:07:43 [ 2315.087808] [] ? mntput+0x21/0x23 2011-09-08 14:07:43 [ 2315.087808] [] ? mntput+0x21/0x23 2011-09-08 14:07:43 [ 2315.092882] [] ? ktime_get_ts+0xad/0xba 2011-09-08 14:07:43 [ 2315.092882] [] ? ktime_get_ts+0xad/0xba 2011-09-08 14:07:43 [ 2315.092944] [] compat_sys_select+0x93/0xbb 2011-09-08 14:07:43 [ 2315.092944] [] compat_sys_select+0x93/0xbb 2011-09-08 14:07:43 [ 2315.095536] Call Trace: 2011-09-08 14:07:43 [ 2315.096349] stunnel x ffff8801858a1dd8[ 2315.097577] stunnel x[ 2315.097713] ffff88018589beb8[ 2315.098801] [] sys_exit_group+0x12/0x16 2011-09-08 14:07:43 0000000000000000 ffff88042e4731f8 [ 2315.105123] 0000000000004000[ 2315.106004] [] mutex_lock+0x16/0x27 2011-09-08 14:07:43 ffff88039f832c90[ 2315.107864] [] mutex_lock+0x16/0x27 2011-09-08 14:07:43 [ 2315.107864] [] mutex_lock+0x16/0x27 2011-09-08 14:07:43 [ 2315.112937] [] do_truncate+0x59/0x80 2011-09-08 14:07:43 [ 2315.112937] [] do_truncate+0x59/0x80 2011-09-08 14:07:43 ffff8801fb557238 [ 2315.114671] [] mutex_lock+0x16/0x27 2011-09-08 14:07:43 [ 2315.115232] qpsmtpd-prefork D[ 2315.116141] Call Trace: 2011-09-08 14:07:43 [ 2315.116456] [] __mutex_lock_slowpath+0xd4/0x119 2011-09-08 14:07:43 [ 2315.116769] [] do_last+0x64a/0x651 2011-09-08 14:07:43 [ 2315.117940] Call Trace: 2011-09-08 14:07:43 [ 2315.118941] php5 S[ 2315.119922] [] ? del_timer+0xc2/0xc2 2011-09-08 14:07:43 ffff88018593bfd8[ 2315.122153] [] do_select+0x56b/0x5bb 2011-09-08 14:07:43 [ 2315.124831] [] sys_open+0x1b/0x1d 2011-09-08 14:07:43 [ 2315.126187] [] ? seq_release_private+0x3e/0x3e 2011-09-08 14:07:43 [ 2315.127233] Call Trace: 2011-09-08 14:07:43 [ 2315.127233] Call Trace: 2011-09-08 14:07:43 [ 2315.132291] [] ? loadavg_proc_open+0x17/0x17 2011-09-08 14:07:43 [ 2315.132291] [] ? loadavg_proc_open+0x17/0x17 2011-09-08 14:07:43 [ 2315.132354] [] ? single_open+0x64/0x93 2011-09-08 14:07:43 [ 2315.132354] [] ? single_open+0x64/0x93 2011-09-08 14:07:43 [ 2315.132417] [] ? sys_semtimedop+0x2f3/0x845 2011-09-08 14:07:43 0 12456 4737 0x00000000 2011-09-08 14:07:43 [ 2315.136199] Call Trace: 2011-09-08 14:07:43 [ 2315.136446] [] do_group_exit+0x71/0x9a 2011-09-08 14:07:43 0000000000000082[ 2315.137928] [] ? fd_install+0x30/0x60 2011-09-08 14:07:43 [ 2315.139093] [] ? __hrtimer_start_range_ns+0x31f/0x331 2011-09-08 14:07:43 [ 2315.139471] [] schedule_hrtimeout_range+0xe/0x10 2011-09-08 14:07:43 [ 2315.139723] [] ? __pollwait+0xcb/0xcb 2011-09-08 14:07:43 0 13032 5879 0x00000000 2011-09-08 14:07:43 [ 2315.141183] ffff8802591fdfd8 0000000000000086[ 2315.143265] [] sys_wait4+0xa6/0xc3 2011-09-08 14:07:43 [ 2315.144436] Call Trace: 2011-09-08 14:07:43 ffff8801ab011fd8 ffff8801ab011fd8 2011-09-08 14:07:43 [ 2315.151783] ffff8801ab011fd8 2011-09-08 14:07:43 [ 2315.151783] ffff8801ab011fd8 ffffffff81210539[ 2315.152249] [] ? __hrtimer_start_range_ns+0x31f/0x331 2011-09-08 14:07:43 [ 2315.153618] Call Trace: 2011-09-08 14:07:43 [ 2315.153932] [] __mutex_lock_slowpath+0xd4/0x119 2011-09-08 14:07:43 ffff8801e6c07fd8[ 2315.158743] 0000000000004000 ffff8801a60f8400[ 2315.160349] Call Trace: 2011-09-08 14:07:43 [ 2315.160659] [] sk_wait_data+0x81/0xc8 2011-09-08 14:07:43 [ 2315.161797] ffff88068e019fd8[ 2315.161797] ffff88068e019fd8[ 2315.162962] ffff8801ab1edc88[ 2315.164046] [] __skb_recv_datagram+0x1ff/0x27b 2011-09-08 14:07:43 ffff8802f6ecc140[ 2315.165466] Call Trace: 2011-09-08 14:07:43 [ 2315.165649] [] ? enqueue_hrtimer+0x48/0x74 2011-09-08 14:07:43 [ 2315.166098] [] ? try_to_wake_up+0x2f0/0x2f0 2011-09-08 14:07:43 [ 2315.166098] [] ? try_to_wake_up+0x2f0/0x2f0 2011-09-08 14:07:43 [ 2315.171166] [] ? mntput+0x21/0x23 2011-09-08 14:07:43 [ 2315.171166] [] ? mntput+0x21/0x23 2011-09-08 14:07:43 ffff8804072d2a10 ffffffff811d75cc[ 2315.173085] [] do_filp_open+0x38/0x88 2011-09-08 14:07:43 [ 2315.174376] [] ? __hrtimer_start_range_ns+0x31f/0x331 2011-09-08 14:07:43 [ 2315.175578] ffff880185b35fd8[ 2315.175855] [] ? rb_insert_color+0xbd/0xe6 2011-09-08 14:07:43 ffff8801972f8010 0 13775 7100 0x00000000 2011-09-08 14:07:43 [ 2315.180433] [] ? enqueue_hrtimer+0x48/0x74 2011-09-08 14:07:43 [ 2315.180741] [] ? update_rmtp+0x60/0x60 2011-09-08 14:07:43 ffff8801df00bfd8[ 2315.183237] [] ? ktime_get_ts+0xad/0xba 2011-09-08 14:07:43 0 13914 13913 0x00000000 2011-09-08 14:07:43 [ 2315.183624] ffff8801a592beb8[ 2315.184717] stunnel4 S 0000000000000086 ffff8801a58a7fd8[ 2315.185824] [] unix_wait_for_peer+0x97/0xb4 2011-09-08 14:07:43 [ 2315.185824] [] unix_wait_for_peer+0x97/0xb4 2011-09-08 14:07:43 [ 2315.190892] [] ? wake_up_bit+0x23/0x23 2011-09-08 14:07:43 [ 2315.190892] [] ? wake_up_bit+0x23/0x23 2011-09-08 14:07:43 [ 2315.191395] stunnel x 0000000100000001 0 13932 13922 0x00020000 2011-09-08 14:07:43 0 13943 13913 0x00020000 2011-09-08 14:07:43 0000000000010640[ 2315.194770] Call Trace: 2011-09-08 14:07:43 [ 2315.195809] 0000000000004000[ 2315.197005] [] ? __kfree_skb+0x7c/0x80 2011-09-08 14:07:43 0 14036 7100 0x00000000 2011-09-08 14:07:43 2011-09-08 14:07:43 [ 2315.198642] ffff8801bf491fd8[ 2315.199609] [] do_last+0x304/0x651 2011-09-08 14:07:43 0000000000010640 0000000000010640 ffff88042dcbb278 [ 2315.204727] [] ? __kfree_skb+0x7c/0x80 2011-09-08 14:07:43 [ 2315.205293] [] vfs_read+0xbe/0x14e 2011-09-08 14:07:43 [ 2315.205293] [] vfs_read+0xbe/0x14e 2011-09-08 14:07:43 [ 2315.210366] [] sys_read+0x47/0x6f 2011-09-08 14:07:43 [ 2315.210366] [] sys_read+0x47/0x6f 2011-09-08 14:07:43 [ 2315.210428] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:43 ffffffff811a44c7[ 2315.211472] [] ? do_lookup+0x114/0x2ba 2011-09-08 14:07:43 [ 2315.212097] php5 S[ 2315.213765] php5 S ffff8803d8ba09e8 [ 2315.214120] 0000000000004000[ 2315.215245] [] vfs_read+0xbe/0x14e 2011-09-08 14:07:43 [ 2315.216341] Call Trace: 2011-09-08 14:07:43 [ 2315.216712] [] ? skb_recv_datagram+0x21/0x21 2011-09-08 14:07:43 [ 2315.218503] [] sk_wait_data+0x81/0xc8 2011-09-08 14:07:43 [ 2315.218753] [] sock_aio_read+0x114/0x12d 2011-09-08 14:07:43 ffff8801e9df7808 [ 2315.220239] [] __mutex_lock_slowpath+0xd4/0x119 2011-09-08 14:07:43 [ 2315.221503] ffff8801981f9fd8 0000000000010640[ 2315.224004] stunnel4 S ffff8802162dfab8[ 2315.224990] [] ? dev_queue_xmit+0x4b5/0x4c5 2011-09-08 14:07:43 [ 2315.224990] [] ? dev_queue_xmit+0x4b5/0x4c5 2011-09-08 14:07:43 [ 2315.230058] [] schedule_hrtimeout_range_clock+0xda/0x123 2011-09-08 14:07:43 [ 2315.230058] [] schedule_hrtimeout_range_clock+0xda/0x123 2011-09-08 14:07:43 [ 2315.230124] [] ? update_rmtp+0x60/0x60 2011-09-08 14:07:43 [ 2315.230124] [] ? update_rmtp+0x60/0x60 2011-09-08 14:07:43 [ 2315.230187] [] ? hrtimer_start_range_ns+0xf/0x11 2011-09-08 14:07:43 [ 2315.230814] [] ? sock_aio_write+0x108/0x121 2011-09-08 14:07:43 [ 2315.231989] Call Trace: 2011-09-08 14:07:43 [ 2315.232298] [] sys_exit_group+0x12/0x16 2011-09-08 14:07:43 [ 2315.233777] [] mutex_lock+0x16/0x27 2011-09-08 14:07:43 0000000000000000[ 2315.235257] [] ? page_add_new_anon_rmap+0x8f/0xa1 2011-09-08 14:07:43 [ 2315.235820] php5 S ffff88037cf88628 [ 2315.237049] [] skb_recv_datagram+0x1f/0x21 2011-09-08 14:07:43 [ 2315.237299] [] ? mntput+0x21/0x23 2011-09-08 14:07:43 0000000000000003[ 2315.239036] [] ? __pollwait+0xcb/0xcb 2011-09-08 14:07:43 [ 2315.239970] [] compat_sys_select+0x93/0xbb 2011-09-08 14:07:43 [ 2315.241072] [] ? in_group_p+0x2b/0x2d 2011-09-08 14:07:43 [ 2315.241513] [] xfs_remove+0x136/0x34f 2011-09-08 14:07:43 0000000000010640 0 15001 7100 0x00000000 2011-09-08 14:07:43 0 15001 7100 0x00000000 2011-09-08 14:07:43 [ 2315.249148] ffff88021471fb08[ 2315.249148] ffff88021471fb08 0000000000000082 0000000000000082 ffff88021471f9e8 0000000000010640[ 2315.250679] php5 S ffff8801ab3dd9e8[ 2315.251965] [] ? wake_up_bit+0x23/0x23 2011-09-08 14:07:43 2011-09-08 14:07:43 [ 2315.253884] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:43 0000000000000082[ 2315.256414] [] ? hrtimer_start_range_ns+0xf/0x11 2011-09-08 14:07:43 [ 2315.256668] [] ? poll_freewait+0x8d/0x8d 2011-09-08 14:07:43 [ 2315.256856] [] ? xfs_iunlock+0x7e/0xba 2011-09-08 14:07:43 0000000000000000[ 2315.259031] [] ? __do_fault+0x3e2/0x419 2011-09-08 14:07:43 [ 2315.260964] [] ? poll_freewait+0x8d/0x8d 2011-09-08 14:07:43 0000000100000001 0 15256 15249 0x00020000 2011-09-08 14:07:43 [ 2315.263855] Call Trace: 2011-09-08 14:07:43 [ 2315.263855] Call Trace: 2011-09-08 14:07:43 [ 2315.268926] [] ? lru_cache_add_lru+0x40/0x42 2011-09-08 14:07:43 [ 2315.268926] [] ? lru_cache_add_lru+0x40/0x42 2011-09-08 14:07:43 [ 2315.268991] [] ? page_add_new_anon_rmap+0x8f/0xa1 2011-09-08 14:07:43 [ 2315.269910] 0000000000004000 0000000000010640[ 2315.270408] [] ? timerqueue_add+0x81/0x9f 2011-09-08 14:07:43 [ 2315.270728] [] ? hrtimer_start_range_ns+0xf/0x11 2011-09-08 14:07:43 [ 2315.271550] [] compat_core_sys_select+0x178/0x213 2011-09-08 14:07:43 [ 2315.271801] [] ? ktime_get_ts+0xad/0xba 2011-09-08 14:07:43 ffff880298751b88 ffff8801a580c010 ffff8803fffd6320[ 2315.274196] qmail-remote S 2011-09-08 14:07:43 2011-09-08 14:07:43 ffff8803e8d5d8c8 0000000000010640 ffff8803cfc54768 [ 2315.280731] [] ? wait_noreap_copyout+0xed/0xed 2011-09-08 14:07:43 [ 2315.281766] Call Trace: 2011-09-08 14:07:43 ffff88042f6c90f8 ffff8802b1e56010 ffff8802b1e56010 2011-09-08 14:07:43 2011-09-08 14:07:43 [ 2315.288188] 0000000000004000[ 2315.289064] [] skb_recv_datagram+0x1f/0x21 2011-09-08 14:07:43 2011-09-08 14:07:43 [ 2315.290666] [] ? try_to_del_timer_sync+0xb7/0xc4 2011-09-08 14:07:43 0 15685 7100 0x00000000 2011-09-08 14:07:43 ffff88042d266ad0 0000000000010640[ 2315.293205] [] do_last+0x304/0x651 2011-09-08 14:07:43 [ 2315.294439] Call Trace: 2011-09-08 14:07:43 [ 2315.294812] [] ? skb_recv_datagram+0x21/0x21 2011-09-08 14:07:43 0000000000010640[ 2315.296979] [] sys_read+0x47/0x6f 2011-09-08 14:07:43 [ 2315.298212] [] ? link_path_walk+0xa8/0x6f7 2011-09-08 14:07:43 ffff8802f9921238 2011-09-08 14:07:43 [ 2315.307565] ffff88019705bfd8 2011-09-08 14:07:43 [ 2315.307565] ffff88019705bfd8 0000000000010640 0000000000010640 ffff8803aa0fc340[ 2315.307785] Call Trace: 2011-09-08 14:07:43 ffff8803d0f1ffd8[ 2315.309507] [] ? bictcp_cong_avoid+0x1c/0x281 2011-09-08 14:07:43 0000000000000082 2011-09-08 14:07:43 [ 2315.311288] [] ? pollwake+0x57/0x59 2011-09-08 14:07:43 ffff8803aa068728 [ 2315.313642] [] ? mntput+0x21/0x23 2011-09-08 14:07:43 0000000000010640[ 2315.316648] [] ? page_add_new_anon_rmap+0x8f/0xa1 2011-09-08 14:07:43 [ 2315.316899] [] ? wait_noreap_copyout+0xed/0xed 2011-09-08 14:07:43 ffff8801fb56a868 [ 2315.317568] 0000000000004000 ffff88019715f908[ 2315.318703] [] ? poll_freewait+0x8d/0x8d 2011-09-08 14:07:43 ffff8801c3adfb68 ffff8801c3adffd8[ 2315.321001] [] unix_accept+0x58/0xfb 2011-09-08 14:07:43 ffff8803fcb64728 2011-09-08 14:07:43 2011-09-08 14:07:43 [ 2315.326803] 0000000000004000[ 2315.326803] 0000000000004000[ 2315.327981] [] ? mntput_no_expire+0x96/0x155 2011-09-08 14:07:43 ffff8803ffc14ff8 ffff8801932a0010 0000000000010640[ 2315.330408] [] sys_poll+0x50/0xba 2011-09-08 14:07:43 0000000000000046 0000000000000046 ffff88040ff18968 [ 2315.333003] [] compat_sys_wait4+0x26/0xb6 2011-09-08 14:07:43 0 16366 16363 0x00020000 2011-09-08 14:07:43 [ 2315.334349] [] ? __hrtimer_start_range_ns+0x31f/0x331 2011-09-08 14:07:43 [ 2315.335558] [] ? mntput+0x21/0x23 2011-09-08 14:07:43 [ 2315.338433] ffff88017fb73fd8[ 2315.339087] [] tcp_recvmsg+0x457/0x8e4 2011-09-08 14:07:43 ffff8804094f4010[ 2315.340936] [] skb_recv_datagram+0x1f/0x21 2011-09-08 14:07:43 [ 2315.341060] [] sys_accept4+0x120/0x1f9 2011-09-08 14:07:43 [ 2315.341060] [] sys_accept4+0x120/0x1f9 2011-09-08 14:07:43 [ 2315.346129] [] ? mntput_no_expire+0x96/0x155 2011-09-08 14:07:43 [ 2315.346129] [] ? mntput_no_expire+0x96/0x155 2011-09-08 14:07:43 [ 2315.346192] [] ? mntput+0x21/0x23 2011-09-08 14:07:43 [ 2315.346192] [] ? mntput+0x21/0x23 2011-09-08 14:07:43 [ 2315.346439] php5 S[ 2315.347236] Call Trace: 2011-09-08 14:07:43 [ 2315.351804] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:43 0000000000000000 2011-09-08 14:07:43 [ 2315.352663] Call Trace: 2011-09-08 14:07:43 [ 2315.352911] [] ? del_timer+0xc2/0xc2 2011-09-08 14:07:43 0000000000010640 ffff88019833fbc8[ 2315.354450] [] ? wake_up_bit+0x23/0x23 2011-09-08 14:07:43 2011-09-08 14:07:43 ffff8801bf452928 ffff8801bf452928 [ 2315.357958] [] ? __hrtimer_start_range_ns+0x31f/0x331 2011-09-08 14:07:43 [ 2315.359219] stunnel x[ 2315.360016] Call Trace: 2011-09-08 14:07:43 [ 2315.360324] [] sys_exit_group+0x12/0x16 2011-09-08 14:07:43 [ 2315.360387] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:43 [ 2315.360387] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:43 [ 2315.365462] php5 S[ 2315.365462] php5 S ffff8803cfc55078 [ 2315.366564] [] __skb_recv_datagram+0x1ff/0x27b 2011-09-08 14:07:43 0000000000010640[ 2315.368779] qmail-pop3d S[ 2315.369696] [] ? enqueue_hrtimer+0x48/0x74 2011-09-08 14:07:43 [ 2315.370145] [] do_select+0x56b/0x5bb 2011-09-08 14:07:43 [ 2315.371090] [] compat_sys_select+0x93/0xbb 2011-09-08 14:07:43 ffff880214767b68[ 2315.374421] [] ? mntput_no_expire+0x96/0x155 2011-09-08 14:07:43 0000000000010640[ 2315.375701] [] ? bictcp_cong_avoid+0x1c/0x281 2011-09-08 14:07:43 ffff88017f899fd8[ 2315.378057] [] ? fput+0x1fb/0x208 2011-09-08 14:07:43 ffff88042e52029c ffff8801a5903fd8[ 2315.379725] [] ? sys_newstat+0x2c/0x3b 2011-09-08 14:07:43 [ 2315.379725] [] ? sys_newstat+0x2c/0x3b 2011-09-08 14:07:43 [ 2315.384799] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:43 [ 2315.384862] stunnel x[ 2315.384799] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:43 0000000100000001 ffff880310d3ceb8 [ 2315.387060] [] ? pollwake+0x57/0x59 2011-09-08 14:07:43 0000000000000086[ 2315.389165] [] do_last+0x269/0x651 2011-09-08 14:07:43 [ 2315.390391] [] ? do_notify_parent+0x1ca/0x1df 2011-09-08 14:07:43 ffffffff810336f9[ 2315.392428] [] vfs_read+0xbe/0x14e 2011-09-08 14:07:43 0000000000200082 0000000000010640 0000000000000010[ 2315.393594] [] sk_stream_wait_memory+0x214/0x21c 2011-09-08 14:07:43 [ 2315.395451] [] skb_recv_datagram+0x1f/0x21 2011-09-08 14:07:43 [ 2315.395761] [] ? fput+0x1fb/0x208 2011-09-08 14:07:43 [ 2315.396083] ffff8801933dbc88 ffff8801933dbfd8[ 2315.397239] [] unix_accept+0x58/0xfb 2011-09-08 14:07:43 [ 2315.397610] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:43 [ 2315.398703] [] ? page_add_new_anon_rmap+0x8f/0xa1 2011-09-08 14:07:43 ffff88017fa39fd8 2011-09-08 14:07:43 [ 2315.404848] ffff88017fa39fd8 2011-09-08 14:07:43 [ 2315.404848] ffff88017fa39fd8 0000000000010640[ 2315.405567] [] ? fd_install+0x30/0x60 2011-09-08 14:07:43 [ 2315.406669] [] ? page_add_new_anon_rmap+0x8f/0xa1 2011-09-08 14:07:43 0000000000010640[ 2315.408782] [] ? sk_reset_timer+0x17/0x23 2011-09-08 14:07:43 [ 2315.409036] [] ? tcp_current_mss+0x5e/0x7d 2011-09-08 14:07:43 ffff8803c82c8ad0 2011-09-08 14:07:43 [ 2315.411510] qmail-pop3d S[ 2315.412489] [] ? enqueue_hrtimer+0x48/0x74 2011-09-08 14:07:43 ffff8801a5aa77b8[ 2315.415241] [] ? hrtimer_start_range_ns+0xf/0x11 2011-09-08 14:07:43 [ 2315.416425] [] compat_sys_select+0x93/0xbb 2011-09-08 14:07:43 ffffffff810336f9[ 2315.418289] [] vfs_read+0xbe/0x14e 2011-09-08 14:07:43 ffff88017fa82010 ffff88017fa82010 2011-09-08 14:07:43 2011-09-08 14:07:43 [ 2315.423846] 0000000000004000[ 2315.423846] 0000000000004000 ffff88017fa83fd8[ 2315.424286] Call Trace: 2011-09-08 14:07:43 0 16927 7100 0x00000000 2011-09-08 14:07:43 [ 2315.426506] [] sys_accept4+0x120/0x1f9 2011-09-08 14:07:43 0000000000010640[ 2315.428477] [] sys_open+0x1b/0x1d 2011-09-08 14:07:43 [ 2315.429454] [] ? __wake_up_sync_key+0x4e/0x5a 2011-09-08 14:07:43 [ 2315.429707] [] ? skb_recv_datagram+0x21/0x21 2011-09-08 14:07:43 [ 2315.430019] [] ? mntput+0x21/0x23 2011-09-08 14:07:43 [ 2315.431062] Call Trace: 2011-09-08 14:07:43 ffff88040c51d660 ffff88017fb8fa58[ 2315.432736] [] ? tcp_ack+0x1bcc/0x1dd2 2011-09-08 14:07:43 ffff8801a6229b68 ffff880310d732a0[ 2315.435326] php5 S[ 2315.435682] 0000000000004000 0000000000010640[ 2315.436933] [] sys_read+0x47/0x6f 2011-09-08 14:07:43 [ 2315.438037] [] sys_semtimedop+0x6c1/0x845 2011-09-08 14:07:43 [ 2315.438224] [] ? loadavg_proc_show+0xb3/0xc7 2011-09-08 14:07:43 [ 2315.438224] [] ? loadavg_proc_show+0xb3/0xc7 2011-09-08 14:07:43 [ 2315.443295] [] ? seq_release_private+0x3e/0x3e 2011-09-08 14:07:43 [ 2315.443295] [] ? seq_release_private+0x3e/0x3e 2011-09-08 14:07:43 [ 2315.443359] [] ? pde_users_dec+0x15/0x3a 2011-09-08 14:07:43 [ 2315.443359] [] ? pde_users_dec+0x15/0x3a 2011-09-08 14:07:43 [ 2315.443423] [] sys_semop+0xb/0xd 2011-09-08 14:07:43 [ 2315.443423] [] sys_semop+0xb/0xd 2011-09-08 14:07:43 ffff8801d4567fd8[ 2315.444342] Call Trace: 2011-09-08 14:07:43 ffff88029bd188e8 [ 2315.448300] [] ? skb_recv_datagram+0x21/0x21 2011-09-08 14:07:43 ffff88034376ffd8 ffff8803cf11fac8 [ 2315.451306] [] ? timerqueue_add+0x81/0x9f 2011-09-08 14:07:43 [ 2315.452322] [] ? release_sock+0xfe/0x107 2011-09-08 14:07:43 [ 2315.452701] [] ? ktime_get_ts+0xad/0xba 2011-09-08 14:07:43 ffff88042879e928 [ 2315.454054] [] __skb_recv_datagram+0x1ff/0x27b 2011-09-08 14:07:43 0000000000010640 2011-09-08 14:07:43 0000000000000001[ 2315.457316] [] xlog_grant_log_space+0x18e/0x4a6 2011-09-08 14:07:43 [ 2315.457682] [] xfs_create+0x200/0x53a 2011-09-08 14:07:43 [ 2315.457682] [] xfs_create+0x200/0x53a 2011-09-08 14:07:43 [ 2315.462759] [] ? xfs_iunlock_map_shared+0x9/0xb 2011-09-08 14:07:43 [ 2315.462759] [] ? xfs_iunlock_map_shared+0x9/0xb 2011-09-08 14:07:43 [ 2315.462823] [] xfs_vn_mknod+0x9a/0xf5 2011-09-08 14:07:43 [ 2315.463384] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:43 ffff88022e5fdb68 ffff88022e5fdfd8[ 2315.464244] Call Trace: 2011-09-08 14:07:43 [ 2315.464244] Call Trace: 2011-09-08 14:07:43 [ 2315.466785] [] vfs_read+0xbe/0x14e 2011-09-08 14:07:43 [ 2315.468012] [] schedule_timeout+0x37/0x269 2011-09-08 14:07:43 ffff8802986cfa08[ 2315.470167] [] sock_aio_read+0x114/0x12d 2011-09-08 14:07:43 [ 2315.470669] php5 S 0000000000000086[ 2315.472655] php5 S 2011-09-08 14:07:43 0000000000010640[ 2315.474379] qmail-popup S ffff88017fa56010[ 2315.475660] [] ? wait_noreap_copyout+0xed/0xed 2011-09-08 14:07:43 [ 2315.477018] [] schedule_hrtimeout_range_clock+0xda/0x123 2011-09-08 14:07:43 [ 2315.477018] [] schedule_hrtimeout_range_clock+0xda/0x123 2011-09-08 14:07:43 [ 2315.482094] [] ? update_rmtp+0x60/0x60 2011-09-08 14:07:43 [ 2315.482094] [] ? update_rmtp+0x60/0x60 2011-09-08 14:07:43 [ 2315.482157] [] ? hrtimer_start_range_ns+0xf/0x11 2011-09-08 14:07:43 [ 2315.482157] [] ? hrtimer_start_range_ns+0xf/0x11 2011-09-08 14:07:43 [ 2315.482221] [] schedule_hrtimeout_range+0xe/0x10 2011-09-08 14:07:43 [ 2315.482914] [] ? tcp_sendmsg+0x9bc/0xadf 2011-09-08 14:07:43 0000000000010640[ 2315.486857] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:43 [ 2315.487835] [] ? wake_up_bit+0x23/0x23 2011-09-08 14:07:43 0000000000000082 ffff8801a5b63fd8[ 2315.489940] [] ? fd_install+0x30/0x60 2011-09-08 14:07:43 ffff880279321848 [ 2315.490481] 0000000000004000 ffff8801a5a47908[ 2315.491041] [] ? enqueue_hrtimer+0x48/0x74 2011-09-08 14:07:43 [ 2315.492248] [] ? mntput+0x21/0x23 2011-09-08 14:07:43 ffff88018684bfd8[ 2315.494217] [] ? fput+0x1fb/0x208 2011-09-08 14:07:43 [ 2315.496004] [] sys_accept+0xb/0xd 2011-09-08 14:07:43 [ 2315.496004] [] sys_accept+0xb/0xd 2011-09-08 14:07:43 [ 2315.501070] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:43 [ 2315.501132] php5 S[ 2315.501070] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:43 [ 2315.501132] php5 S ffff8803e8c8d888 ffff8803e8c8d888 0 17507 7100 0x00000000 2011-09-08 14:07:43 ffff8803fdf99bc8 ffff88017fa1bb68[ 2315.504136] [] ? skb_recv_datagram+0x21/0x21 2011-09-08 14:07:43 ffff880186969c38 ffff880186969c38 2011-09-08 14:07:43 [ 2315.507785] [] sys_epoll_wait+0x22a/0x387 2011-09-08 14:07:43 [ 2315.508807] Call Trace: 2011-09-08 14:07:43 [ 2315.509743] php5 S[ 2315.510771] [] ? wake_up_bit+0x23/0x23 2011-09-08 14:07:43 ffff880186a42010[ 2315.512159] ffff880186a43fd8[ 2315.513002] [] sys_write+0x47/0x6e 2011-09-08 14:07:43 ffff8803fdfc2010[ 2315.513702] ffff8803fdfc3fd8[ 2315.514726] [] sys_accept+0xb/0xd 2011-09-08 14:07:43 [ 2315.515703] [] ? pollwake+0x57/0x59 2011-09-08 14:07:43 [ 2315.515703] [] ? pollwake+0x57/0x59 2011-09-08 14:07:43 [ 2315.520774] [] ? wake_up_bit+0x23/0x23 2011-09-08 14:07:43 [ 2315.520774] [] ? wake_up_bit+0x23/0x23 2011-09-08 14:07:43 [ 2315.520836] [] schedule_timeout+0x37/0x269 2011-09-08 14:07:43 [ 2315.520836] [] schedule_timeout+0x37/0x269 2011-09-08 14:07:43 [ 2315.520900] [] ? sock_recvmsg+0xc1/0xda 2011-09-08 14:07:43 [ 2315.522052] 0000000000004000[ 2315.523365] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:43 2011-09-08 14:07:43 0000000000010640[ 2315.524282] [] ? bictcp_cong_avoid+0x1c/0x281 2011-09-08 14:07:43 [ 2315.524533] [] ? wake_up_bit+0x23/0x23 2011-09-08 14:07:43 [ 2315.524847] [] vfs_read+0xbe/0x14e 2011-09-08 14:07:43 [ 2315.525167] ffff8803e1f05b38[ 2315.525387] 0000000000004000[ 2315.526516] [] do_sync_read+0xc7/0x10d 2011-09-08 14:07:43 2011-09-08 14:07:43 ffff8801a58fa010 ffffffff8121058d[ 2315.530103] [] ? __alloc_skb+0x72/0x132 2011-09-08 14:07:43 ffff8803cded5b68[ 2315.532086] [] ? skb_recv_datagram+0x21/0x21 2011-09-08 14:07:43 [ 2315.532460] [] ? fput+0x1fb/0x208 2011-09-08 14:07:43 [ 2315.533559] [] ? __wake_up_sync_key+0x4e/0x5a 2011-09-08 14:07:43 2011-09-08 14:07:43 [ 2315.535068] ffff8801ab3b9fd8 ffff8801ab3b9c38 ffff8801ab3b9c38 2011-09-08 14:07:43 2011-09-08 14:07:43 [ 2315.540302] Call Trace: 2011-09-08 14:07:43 ffff88020a0d6010[ 2315.544696] ffff88017faf3c88 2011-09-08 14:07:43 ffff88017faf3c38[ 2315.546399] php5 S[ 2315.547505] [] sk_wait_data+0x81/0xc8 2011-09-08 14:07:43 2011-09-08 14:07:43 [ 2315.549358] [] unix_accept+0x58/0xfb 2011-09-08 14:07:43 [ 2315.550588] Call Trace: 2011-09-08 14:07:43 [ 2315.550771] [] ? sock_recvmsg+0xc1/0xda 2011-09-08 14:07:43 [ 2315.551085] [] sys_accept4+0x120/0x1f9 2011-09-08 14:07:43 [ 2315.551394] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:43 [ 2315.553055] qmail-pop3d S 2011-09-08 14:07:43 [ 2315.554293] [] schedule_hrtimeout_range+0xe/0x10 2011-09-08 14:07:43 [ 2315.554482] [] ? poll_freewait+0x8d/0x8d 2011-09-08 14:07:43 [ 2315.559554] [] ? __pollwait+0xcb/0xcb 2011-09-08 14:07:43 [ 2315.554482] [] ? poll_freewait+0x8d/0x8d 2011-09-08 14:07:43 [ 2315.559616] [] ? sk_reset_timer+0x17/0x23 2011-09-08 14:07:43 [ 2315.559554] [] ? __pollwait+0xcb/0xcb 2011-09-08 14:07:43 [ 2315.559616] [] ? sk_reset_timer+0x17/0x23 2011-09-08 14:07:43 [ 2315.559680] [] ? tcp_event_new_data_sent+0x9a/0x9c 2011-09-08 14:07:43 ffff8802f99f9520 ffff8802f99f9520[ 2315.562552] php5 S[ 2315.563252] ffff8803c8317fd8 ffff8803aa3a1a48 [ 2315.565756] [] sys_accept4+0x120/0x1f9 2011-09-08 14:07:43 [ 2315.566128] php5 S ffff880186b4c010 0 17780 7100 0x00000000 2011-09-08 14:07:43 [ 2315.569394] [] sock_aio_read+0x114/0x12d 2011-09-08 14:07:43 [ 2315.572228] Call Trace: 2011-09-08 14:07:43 [ 2315.572412] [] ? del_timer+0xc2/0xc2 2011-09-08 14:07:43 ffff88019328dfd8 2011-09-08 14:07:43 2011-09-08 14:07:43 [ 2315.578903] Call Trace: 2011-09-08 14:07:43 [ 2315.578903] Call Trace: 2011-09-08 14:07:43 [ 2315.578959] [] ? pollwake+0x57/0x59 2011-09-08 14:07:43 [ 2315.578959] [] ? pollwake+0x57/0x59 2011-09-08 14:07:43 [ 2315.579021] [] ? __wake_up_sync_key+0x4e/0x5a 2011-09-08 14:07:43 [ 2315.579021] [] ? __wake_up_sync_key+0x4e/0x5a 2011-09-08 14:07:43 [ 2315.579586] [] ? mntput+0x21/0x23 2011-09-08 14:07:43 0 17900 7100 0x00000000 2011-09-08 14:07:43 [ 2315.580799] [] ? bictcp_cong_avoid+0x1c/0x281 2011-09-08 14:07:43 [ 2315.581175] [] tcp_recvmsg+0x457/0x8e4 2011-09-08 14:07:43 2011-09-08 14:07:43 [ 2315.586493] [] ? mntput+0x21/0x23 2011-09-08 14:07:43 0 18002 7100 0x00000000 2011-09-08 14:07:43 [ 2315.588145] [] unix_accept+0x58/0xfb 2011-09-08 14:07:43 [ 2315.588516] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:43 [ 2315.589559] [] schedule_timeout+0x22b/0x269 2011-09-08 14:07:43 0000000000010640 ffff8801c3b1fc38[ 2315.591286] [] ? sock_recvmsg+0xc1/0xda 2011-09-08 14:07:43 [ 2315.591600] [] sys_accept4+0x120/0x1f9 2011-09-08 14:07:43 0000000000010640[ 2315.592940] [] ? do_dbs_timer+0x35f/0x36e 2011-09-08 14:07:43 [ 2315.593194] [] kthread+0x81/0x89 2011-09-08 14:07:43 [ 2315.593256] [] kernel_thread_helper+0x4/0x10 2011-09-08 14:07:43 [ 2315.593256] [] kernel_thread_helper+0x4/0x10 2011-09-08 14:07:43 0000000000000082 2011-09-08 14:07:43 [ 2315.599247] Call Trace: 2011-09-08 14:07:43 ffff8801e6d9a828 [ 2315.600533] 0000000000004000 0000000000010640[ 2315.601095] [] ? cpumask_next_and+0x2d/0x3e 2011-09-08 14:07:43 ffff880396a0d138 ffff880396a0cbd0 ffffffff810336f9[ 2315.603081] [] ? cpumask_next_and+0x2d/0x3e 2011-09-08 14:07:43 [ 2315.604024] [] sys_read+0x47/0x6f 2011-09-08 14:07:43 [ 2315.606784] Call Trace: 2011-09-08 14:07:43 ffff88042dcaa8e8 0 18274 7100 0x00000000 2011-09-08 14:07:43 [ 2315.609265] [] sys_open+0x1b/0x1d 2011-09-08 14:07:43 [ 2315.609389] qmail-popup S[ 2315.610486] [] ? do_wp_page+0x6ba/0x6d2 2011-09-08 14:07:43 [ 2315.610985] qmail-pop3d S[ 2315.611561] ffff880370f9ffd8[ 2315.612601] [] ? xfs_bmbt_get_all+0x16/0x18 2011-09-08 14:07:43 [ 2315.612601] [] ? xfs_bmbt_get_all+0x16/0x18 2011-09-08 14:07:43 [ 2315.617671] [] ? xfs_bmap_search_multi_extents+0x83/0xda 2011-09-08 14:07:43 [ 2315.617927] [] ? xfs_bmapi+0x2c4/0xd5d 2011-09-08 14:07:43 [ 2315.619478] [] unix_wait_for_peer+0x97/0xb4 2011-09-08 14:07:43 [ 2315.620620] ffff8801e9fe3fd8[ 2315.621020] [] schedule_timeout+0x37/0x269 2011-09-08 14:07:43 2011-09-08 14:07:43 [ 2315.623183] [] xfs_vn_unlink+0x3c/0x75 2011-09-08 14:07:43 [ 2315.623679] qmail-queue D ffff88041ec63f38 ffff8801a5aee010[ 2315.630749] [] ? sockfd_lookup_light+0x1b/0x54 2011-09-08 14:07:43 ffff88040c58fc08 [ 2315.631357] 0000000000004000[ 2315.631982] [] ? __hrtimer_start_range_ns+0x31f/0x331 2011-09-08 14:07:43 [ 2315.631982] [] ? __hrtimer_start_range_ns+0x31f/0x331 2011-09-08 14:07:43 [ 2315.637061] [] ? dev_queue_xmit+0x4b5/0x4c5 2011-09-08 14:07:43 [ 2315.637061] [] ? dev_queue_xmit+0x4b5/0x4c5 2011-09-08 14:07:43 [ 2315.637125] [] schedule_hrtimeout_range_clock+0xda/0x123 2011-09-08 14:07:43 [ 2315.637191] [] ? update_rmtp+0x60/0x60 2011-09-08 14:07:43 [ 2315.639480] [] ? wake_up_bit+0x23/0x23 2011-09-08 14:07:43 ffff8802f6c1ad90 0000000000010640[ 2315.642005] [] ? lru_cache_add_lru+0x40/0x42 2011-09-08 14:07:43 [ 2315.642319] [] ? wait_noreap_copyout+0xed/0xed 2011-09-08 14:07:43 [ 2315.642570] [] ia32_do_call+0x13/0x13 2011-09-08 14:07:43 [ 2315.643799] [] xfs_log_reserve+0x11a/0x122 2011-09-08 14:07:43 [ 2315.643987] [] ? xfs_vn_lookup+0x60/0x7e 2011-09-08 14:07:43 [ 2315.644298] [] ? vfs_stat+0x16/0x18 2011-09-08 14:07:43 [ 2315.645398] [] ? lru_cache_add_lru+0x40/0x42 2011-09-08 14:07:43 0000000000010640[ 2315.647557] [] ia32_do_call+0x13/0x13 2011-09-08 14:07:43 0000000000010640 2011-09-08 14:07:43 0000000000010640 2011-09-08 14:07:43 [ 2315.657033] 0000000000004000 2011-09-08 14:07:43 [ 2315.657033] 0000000000004000 ffff8803de204d50 ffff8803de204d50 0000000000010640 0000000000010640 ffff8801e9c29fd8 0000000000010640 2011-09-08 14:07:43 0000000000010640[ 2315.658825] [] ? path_lookupat+0x514/0x584 2011-09-08 14:07:43 ffff880425c614a0[ 2315.660182] Call Trace: 2011-09-08 14:07:43 [ 2315.661190] ffff8803c82e1fd8 0000000000010640 2011-09-08 14:07:43 ffff880186a46010 0000000000010640[ 2315.664547] [] ? complete_walk+0x8e/0xdd 2011-09-08 14:07:43 [ 2315.664799] [] lookup_create+0x1e/0x97 2011-09-08 14:07:43 0000000000010640[ 2315.667003] 0000000000004000[ 2315.668003] [] sys32_waitpid+0xb/0xd 2011-09-08 14:07:43 000000023f5db2d8 ffff8804285d8d10[ 2315.671262] [] vfs_read+0xab/0x14e 2011-09-08 14:07:43 2011-09-08 14:07:43 2011-09-08 14:07:43 [ 2315.677252] Call Trace: 2011-09-08 14:07:43 [ 2315.677252] Call Trace: 2011-09-08 14:07:43 [ 2315.677309] [] ? lru_cache_add_lru+0x40/0x42 2011-09-08 14:07:43 [ 2315.677309] [] ? lru_cache_add_lru+0x40/0x42 2011-09-08 14:07:43 [ 2315.677373] [] ? handle_pte_fault+0x268/0x770 2011-09-08 14:07:43 [ 2315.677373] [] ? handle_pte_fault+0x268/0x770 2011-09-08 14:07:43 [ 2315.678379] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:43 [ 2315.679424] [] ? do_wp_page+0x6ba/0x6d2 2011-09-08 14:07:43 [ 2315.679736] [] ? fd_install+0x30/0x60 2011-09-08 14:07:43 [ 2315.680889] [] ? complete_walk+0x8e/0xdd 2011-09-08 14:07:43 [ 2315.682311] Call Trace: 2011-09-08 14:07:43 [ 2315.682559] [] schedule_timeout+0x37/0x269 2011-09-08 14:07:43 2011-09-08 14:07:43 [ 2315.684044] [] ? do_notify_parent+0x1ca/0x1df 2011-09-08 14:07:43 0000000000010640[ 2315.685274] [] ? handle_mm_fault+0x24c/0x263 2011-09-08 14:07:43 [ 2315.686269] [] do_sys_open+0x114/0x1a3 2011-09-08 14:07:43 [ 2315.687310] [] ? lru_cache_add_lru+0x40/0x42 2011-09-08 14:07:43 [ 2315.687625] [] ? wait_noreap_copyout+0xed/0xed 2011-09-08 14:07:43 [ 2315.688513] ffff880290d1ffd8[ 2315.689044] [] ? link_path_walk+0xa8/0x6f7 2011-09-08 14:07:43 ffff8803f20174a0[ 2315.691882] Call Trace: 2011-09-08 14:07:43 [ 2315.691882] Call Trace: 2011-09-08 14:07:43 [ 2315.696948] [] ? in_group_p+0x2b/0x2d 2011-09-08 14:07:43 [ 2315.696948] [] ? in_group_p+0x2b/0x2d 2011-09-08 14:07:43 [ 2315.697011] [] ? acl_permission_check+0x69/0x8a 2011-09-08 14:07:43 [ 2315.697011] [] ? acl_permission_check+0x69/0x8a 2011-09-08 14:07:43 [ 2315.697198] [] ? do_lookup+0x55/0x2ba 2011-09-08 14:07:43 [ 2315.697198] [] ? do_lookup+0x55/0x2ba 2011-09-08 14:07:43 [ 2315.697261] [] __mutex_lock_slowpath+0xd4/0x119 2011-09-08 14:07:43 ffff8802591e0010[ 2315.699122] [] ? wake_up_bit+0x23/0x23 2011-09-08 14:07:43 2011-09-08 14:07:43 ffff88042d3d78c8 ffff88042d3d7360 ffff88020dfd5d68 0 18581 4736 0x00020000 2011-09-08 14:07:43 [ 2315.704439] ffff8801d44cffd8[ 2315.704659] Call Trace: 2011-09-08 14:07:43 [ 2315.704968] [] sys_wait4+0xa6/0xc3 2011-09-08 14:07:43 [ 2315.705219] [] sys32_waitpid+0xb/0xd 2011-09-08 14:07:43 [ 2315.705479] ffff8801e9e25de8[ 2315.706448] [] lookup_create+0x1e/0x97 2011-09-08 14:07:43 ffffffff8103e73b 0 18584 18583 0x00000000 2011-09-08 14:07:43 [ 2315.709461] [] ? do_notify_parent+0x1ca/0x1df 2011-09-08 14:07:43 [ 2315.710415] ffff8802c5199fd8 2011-09-08 14:07:43 2011-09-08 14:07:43 [ 2315.717117] Call Trace: 2011-09-08 14:07:43 [ 2315.717117] Call Trace: 2011-09-08 14:07:43 [ 2315.717174] [] ? complete_walk+0x8e/0xdd 2011-09-08 14:07:43 [ 2315.717174] [] ? complete_walk+0x8e/0xdd 2011-09-08 14:07:43 [ 2315.717237] [] ? path_lookupat+0x514/0x584 2011-09-08 14:07:43 2011-09-08 14:07:43 2011-09-08 14:07:43 [ 2315.719766] 0000000000004000 2011-09-08 14:07:43 [ 2315.720327] [] ? perf_event_exit_task+0xc0/0x1cb 2011-09-08 14:07:43 [ 2315.721552] Call Trace: 2011-09-08 14:07:43 0000000000000082 ffff8803c82cbfd8[ 2315.724956] [] ? handle_mm_fault+0x24c/0x263 2011-09-08 14:07:43 ffff88020a191a28 2011-09-08 14:07:43 0000000000000082 0000000000010640[ 2315.729115] [] ? user_path_at+0x66/0x95 2011-09-08 14:07:43 [ 2315.730160] Call Trace: 2011-09-08 14:07:43 0 18603 18592 0x00000000 2011-09-08 14:07:43 [ 2315.731300] ffff8802fbd11b38 2011-09-08 14:07:43 2011-09-08 14:07:43 [ 2315.736750] ffff8802fbd11fd8[ 2315.736750] ffff8802fbd11fd8[ 2315.737027] [] ? cpumask_next_and+0x2d/0x3e 2011-09-08 14:07:43 [ 2315.737410] [] ? __wake_up_sync_key+0x4e/0x5a 2011-09-08 14:07:43 0000000000010640[ 2315.739644] [] vfs_read+0xbe/0x14e 2011-09-08 14:07:43 [ 2315.740809] [] ? do_lookup+0x55/0x2ba 2011-09-08 14:07:43 0000000000000000[ 2315.742603] [] sys_wait4+0xa6/0xc3 2011-09-08 14:07:43 0000000000005d18[ 2315.745145] ffff880197355fd8[ 2315.745551] [] ? do_wp_page+0x6ba/0x6d2 2011-09-08 14:07:43 [ 2315.746049] qmail-queue D ffff8801e542ddc8[ 2315.747343] [] path_openat+0xc5/0x336 2011-09-08 14:07:43 [ 2315.747596] [] compat_sys_open+0x15/0x17 2011-09-08 14:07:43 [ 2315.748702] [] ? do_wp_page+0x6ba/0x6d2 2011-09-08 14:07:43 [ 2315.750052] [] ? complete_walk+0x8e/0xdd 2011-09-08 14:07:43 [ 2315.751180] 0000000000004000 ffff8803aa37cb90[ 2315.751620] Call Trace: 2011-09-08 14:07:43 [ 2315.751620] Call Trace: 2011-09-08 14:07:43 [ 2315.756688] [] ? lru_cache_add_lru+0x40/0x42 2011-09-08 14:07:43 ffff88017fa94010 ffff8803fddfe010 0 18619 18618 0x00020000 2011-09-08 14:07:43 ffff8803fdde9fd8[ 2315.761798] [] ia32_do_call+0x13/0x13 2011-09-08 14:07:43 [ 2315.762712] [] ? handle_mm_fault+0x24c/0x263 2011-09-08 14:07:43 [ 2315.763092] [] unix_dgram_sendmsg+0x3dc/0x56c 2011-09-08 14:07:43 [ 2315.763408] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:43 [ 2315.766316] ffff8801a6349d68 ffff88023cee9620 0000000000000000[ 2315.767096] [] ? page_add_new_anon_rmap+0x8f/0xa1 2011-09-08 14:07:43 0000000000010640 2011-09-08 14:07:43 [ 2315.770103] [] ? complete_walk+0x8e/0xdd 2011-09-08 14:07:43 0000000000010640[ 2315.771642] [] ? acl_permission_check+0x69/0x8a 2011-09-08 14:07:43 [ 2315.771642] [] ? acl_permission_check+0x69/0x8a 2011-09-08 14:07:43 [ 2315.776716] [] ? do_lookup+0x55/0x2ba 2011-09-08 14:07:43 [ 2315.776716] [] ? do_lookup+0x55/0x2ba 2011-09-08 14:07:43 [ 2315.776778] [] __mutex_lock_slowpath+0xd4/0x119 2011-09-08 14:07:43 [ 2315.776967] [] do_last+0x269/0x651 2011-09-08 14:07:43 [ 2315.777282] [] compat_sys_open+0x15/0x17 2011-09-08 14:07:43 0000000000000086[ 2315.778749] [] ? vfs_stat+0x16/0x18 2011-09-08 14:07:43 ffff8801bf453b48 [ 2315.779352] 0000000000004000[ 2315.780413] [] ? __hrtimer_start_range_ns+0x31f/0x331 2011-09-08 14:07:43 0 18631 4736 0x00020000 2011-09-08 14:07:43 [ 2315.781021] 0000000000004000[ 2315.782021] [] sys32_waitpid+0xb/0xd 2011-09-08 14:07:43 [ 2315.783621] qmail-queue D ffff880186a5ffd8[ 2315.784472] [] ? in_group_p+0x2b/0x2d 2011-09-08 14:07:43 ffff8802793209d0 0000000000000000[ 2315.786260] [] ? handle_mm_fault+0x24c/0x263 2011-09-08 14:07:43 ffff8801a60f8d10[ 2315.788248] qmail-popup S 0 18647 4736 0x00020000 2011-09-08 14:07:43 [ 2315.789415] [] ? wait_noreap_copyout+0xed/0xed 2011-09-08 14:07:43 [ 2315.790581] [] ? complete_walk+0x8e/0xdd 2011-09-08 14:07:43 2011-09-08 14:07:43 [ 2315.796687] 0000000000004000 2011-09-08 14:07:43 [ 2315.796687] 0000000000004000 ffff88029bd18c90 ffff88029bd18c90[ 2315.799290] pop3passwdcheck D 0 18651 18649 0x00020000 2011-09-08 14:07:43 0000000000010640 0000000000000082 0000000000010640[ 2315.807112] [] vfs_read+0xab/0x14e 2011-09-08 14:07:43 [ 2315.809994] [] ? vfs_stat+0x16/0x18 2011-09-08 14:07:43 [ 2315.811158] [] ? page_add_new_anon_rmap+0x8f/0xa1 2011-09-08 14:07:43 [ 2315.811472] [] compat_sys_wait4+0x26/0xb6 2011-09-08 14:07:43 [ 2315.811472] [] compat_sys_wait4+0x26/0xb6 2011-09-08 14:07:43 [ 2315.816541] [] ? fd_install+0x30/0x60 2011-09-08 14:07:43 [ 2315.816541] [] ? fd_install+0x30/0x60 2011-09-08 14:07:43 [ 2315.816604] [] sys32_waitpid+0xb/0xd 2011-09-08 14:07:43 [ 2315.816604] [] sys32_waitpid+0xb/0xd 2011-09-08 14:07:43 [ 2315.816665] [] ia32_do_call+0x13/0x13 2011-09-08 14:07:43 0000000000010640 0 18663 18660 0x00020000 2011-09-08 14:07:43 [ 2315.821090] [] ? sys32_stat64+0x29/0x34 2011-09-08 14:07:43 [ 2315.822381] [] lookup_create+0x1e/0x97 2011-09-08 14:07:43 ffff88040b0337c8 ffffffff811d4cf6 2011-09-08 14:07:43 ffff8802f6f967a8 [ 2315.825710] [] ? xfs_vm_write_end+0x24/0x46 2011-09-08 14:07:43 ffff8801a62e0010[ 2315.827038] ffff8801a62e1fd8[ 2315.827314] [] ? xfs_get_blocks_direct+0x11/0x11 2011-09-08 14:07:43 0000000000000086 2011-09-08 14:07:43 [ 2315.829054] Call Trace: 2011-09-08 14:07:43 [ 2315.831179] [] __mutex_lock_slowpath+0xd4/0x119 2011-09-08 14:07:43 [ 2315.831179] [] __mutex_lock_slowpath+0xd4/0x119 2011-09-08 14:07:43 [ 2315.836257] [] ? do_lookup+0x55/0x2ba 2011-09-08 14:07:43 [ 2315.836257] [] ? do_lookup+0x55/0x2ba 2011-09-08 14:07:43 [ 2315.836382] [] do_truncate+0x59/0x80 2011-09-08 14:07:43 0000000000010640[ 2315.838739] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:43 ffff880186b76010 0000000000000086[ 2315.842735] [] path_openat+0xc5/0x336 2011-09-08 14:07:43 [ 2315.843844] Call Trace: 2011-09-08 14:07:43 ffff8802f6ecca50[ 2315.845942] [] ? __set_task_blocked+0x60/0x67 2011-09-08 14:07:43 [ 2315.847232] [] ? do_filp_open+0x38/0x88 2011-09-08 14:07:43 ffff880186a9bb68 0 18681 18680 0x00000000 2011-09-08 14:07:43 ffff8803cf31c928 [ 2315.850986] 0000000000004000[ 2315.851206] ffff8803cf2d7fd8 2011-09-08 14:07:43 2011-09-08 14:07:43 [ 2315.856436] Call Trace: 2011-09-08 14:07:43 [ 2315.856436] Call Trace: 2011-09-08 14:07:43 [ 2315.856493] [] ? _raw_spin_unlock_bh+0x10/0x12 2011-09-08 14:07:43 [ 2315.856493] [] ? _raw_spin_unlock_bh+0x10/0x12 2011-09-08 14:07:43 [ 2315.856557] [] ? tcp_recvmsg+0x7c6/0x8e4 2011-09-08 14:07:43 [ 2315.857017] [] ? vfs_read+0xab/0x14e 2011-09-08 14:07:43 [ 2315.858057] [] ? __alloc_pages_nodemask+0x137/0x740 2011-09-08 14:07:43 ffff8802592bc010[ 2315.859613] Call Trace: 2011-09-08 14:07:43 [ 2315.860544] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:43 [ 2315.861402] Call Trace: 2011-09-08 14:07:43 2011-09-08 14:07:43 [ 2315.864934] [] sys_semtimedop+0x6c1/0x845 2011-09-08 14:07:43 2011-09-08 14:07:43 [ 2315.867117] [] sys_semop+0xb/0xd 2011-09-08 14:07:43 [ 2315.868037] Call Trace: 2011-09-08 14:07:43 0000000000000082[ 2315.869647] [] ? do_page_fault+0x342/0x37d 2011-09-08 14:07:43 2011-09-08 14:07:43 ffff88042f609460 ffff88042f609460 2011-09-08 14:07:43 [ 2315.876575] stunnel4 S ffff8803f23be300 ffff8803fcb65948 [ 2315.881018] [] unix_wait_for_peer+0x97/0xb4 2011-09-08 14:07:43 0000000000010640 ffff88030d1de5a8 2011-09-08 14:07:43 0000000000000000 0 18701 18696 0x00000000 2011-09-08 14:07:43 [ 2315.888109] Call Trace: 2011-09-08 14:07:43 [ 2315.888294] [] do_exit+0x74d/0x75b 2011-09-08 14:07:43 2011-09-08 14:07:43 [ 2315.889449] Call Trace: 2011-09-08 14:07:43 ffff8801e1e77dd8 ffff8801e1e54010 2011-09-08 14:07:43 ffff8801e1e54010 2011-09-08 14:07:43 [ 2315.896479] 0000000000004000[ 2315.896479] 0000000000004000 ffff8803b412a380 ffff8803b412a380[ 2315.897042] [] ? sys_semtimedop+0x2f3/0x845 2011-09-08 14:07:43 ffff88043ffd9e60 0 18708 7005 0x00000000 2011-09-08 14:07:43 ffff8801e1edffd8[ 2315.900831] [] sys_semop+0xb/0xd 2011-09-08 14:07:43 [ 2315.901875] [] ? sys_semtimedop+0x2f3/0x845 2011-09-08 14:07:43 2011-09-08 14:07:43 [ 2315.903741] [] ? handle_mm_fault+0x24c/0x263 2011-09-08 14:07:43 ffff8801e1fa1fd8[ 2315.905605] [] ? poll_select_copy_remaining+0xc5/0xe9 2011-09-08 14:07:43 0 18712 7005 0x00000000 2011-09-08 14:07:43 [ 2315.907142] [] ? handle_mm_fault+0x24c/0x263 2011-09-08 14:07:43 [ 2315.908755] [] ? handle_mm_fault+0x24c/0x263 2011-09-08 14:07:43 [ 2315.909133] [] sys_semop+0xb/0xd 2011-09-08 14:07:43 [ 2315.910053] Call Trace: 2011-09-08 14:07:43 ffff8801ae0ba010 ffff8801ae0ba010 2011-09-08 14:07:43 2011-09-08 14:07:43 [ 2315.916228] 0000000000004000[ 2315.916228] 0000000000004000[ 2315.916791] [] ? perf_event_exit_task+0xc0/0x1cb 2011-09-08 14:07:43 2011-09-08 14:07:43 ffff8801ae0dfe38[ 2315.919520] ffff8801ae0dffd8 ffff8801ae0212f8 ffff8801ae0212f8 [ 2315.921912] httpd S 0 18719 7005 0x00000000 2011-09-08 14:07:43 ffff8801ae020480[ 2315.923273] [] ? poll_select_copy_remaining+0xc5/0xe9 2011-09-08 14:07:43 ffff88043ffd9e60 ffff8801ae184010[ 2315.926300] [] ? handle_mm_fault+0x24c/0x263 2011-09-08 14:07:43 0000000000010640[ 2315.928292] [] sys_semop+0xb/0xd 2011-09-08 14:07:43 [ 2315.929461] [] ? core_sys_select+0x1a3/0x253 2011-09-08 14:07:43 [ 2315.929716] [] ? ktime_get_ts+0xad/0xba 2011-09-08 14:07:43 0 18724 7005 0x00000000 2011-09-08 14:07:43 [ 2315.931129] [] sys_semtimedop+0x6c1/0x845 2011-09-08 14:07:43 [ 2315.931129] [] sys_semtimedop+0x6c1/0x845 2011-09-08 14:07:43 [ 2315.936202] [] ? core_sys_select+0x1a3/0x253 2011-09-08 14:07:43 [ 2315.936202] [] ? core_sys_select+0x1a3/0x253 2011-09-08 14:07:43 [ 2315.936266] [] ? handle_pte_fault+0x2c2/0x770 2011-09-08 14:07:43 [ 2315.936907] ffff8801ae2b7ce8[ 2315.937879] [] ? handle_pte_fault+0x2c2/0x770 2011-09-08 14:07:43 [ 2315.938197] [] ? sys_select+0xa7/0xbd 2011-09-08 14:07:43 [ 2315.939429] [] ? sch_direct_xmit+0x96/0x197 2011-09-08 14:07:43 [ 2315.939749] [] poll_schedule_timeout+0x46/0x69 2011-09-08 14:07:43 0000000000010640[ 2315.943212] [] ? tcp_recvmsg+0x7c6/0x8e4 2011-09-08 14:07:43 ffff88043ffd9e60[ 2315.945471] httpd S ffff8801ae3ea9d0 0000000000000001[ 2315.946645] [] ? handle_mm_fault+0x24c/0x263 2011-09-08 14:07:43 [ 2315.946961] [] ? sys_select+0xa7/0xbd 2011-09-08 14:07:43 0 18731 7005 0x00000000 2011-09-08 14:07:43 ffff8801ae3ea0c0 0000000000010640 ffff88017ecb9888 ffff88017ecb9320[ 2315.950176] [] ? ktime_get_ts+0xad/0xba 2011-09-08 14:07:43 ffff88017ecb8f78 0 18733 7005 0x00000000 2011-09-08 14:07:43 [ 2315.955638] ffff88017ecffce8 0 18733 7005 0x00000000 2011-09-08 14:07:43 [ 2315.955638] ffff88017ecffce8 0000000000000086 000000007ecffbb8 0000000000000086 000000007ecffbb8 ffff88017ecb8a10 2011-09-08 14:07:43 [ 2315.956546] [] sys_semtimedop+0x6c1/0x845 2011-09-08 14:07:43 ffff88017ed65fd8[ 2315.958283] [] ? link_path_walk+0xa8/0x6f7 2011-09-08 14:07:43 [ 2315.958847] qmail-queue D 000000000000002f[ 2315.960707] httpd S[ 2315.962270] qmail-queue D 000000000000002f ffff88025725ded8[ 2315.965107] [] ? wait_noreap_copyout+0xed/0xed 2011-09-08 14:07:43 [ 2315.966200] [] ? __alloc_pages_nodemask+0x137/0x740 2011-09-08 14:07:43 ffff88017eeca010 ffff8801e6d9b138 [ 2315.969613] [] ? wake_up_bit+0x23/0x23 2011-09-08 14:07:43 2011-09-08 14:07:43 [ 2315.975412] 0000000000004000 2011-09-08 14:07:43 0000000000010640 ffff88017eee3dd8 2011-09-08 14:07:43 [ 2315.977692] [] ? kill_pid_info+0x3a/0x47 2011-09-08 14:07:43 0000000000010640[ 2315.979173] [] mutex_lock+0x16/0x27 2011-09-08 14:07:43 ffff88043ffd9e60 ffff88017ef18010[ 2315.982336] [] ? do_page_fault+0x342/0x37d 2011-09-08 14:07:43 ffff88043ffd9e60 ffff88018ec0a010 ffff88018ec18010 2011-09-08 14:07:43 [ 2315.988634] [] do_last+0x64a/0x651 2011-09-08 14:07:43 0000000000000065[ 2315.990370] [] ? nameidata_to_filp+0x53/0x61 2011-09-08 14:07:43 [ 2315.990370] [] ? nameidata_to_filp+0x53/0x61 2011-09-08 14:07:43 [ 2315.995440] [] do_last+0x64a/0x651 2011-09-08 14:07:43 [ 2315.995440] [] do_last+0x64a/0x651 2011-09-08 14:07:43 [ 2315.995502] [] path_openat+0xc5/0x336 2011-09-08 14:07:43 [ 2315.997362] [] do_filp_open+0x38/0x88 2011-09-08 14:07:43 2011-09-08 14:07:43 [ 2315.998655] [] pipe_wait+0x5d/0x79 2011-09-08 14:07:43 2011-09-08 14:07:43 [ 2316.000510] [] mutex_lock+0x16/0x27 2011-09-08 14:07:43 [ 2316.000795] [] path_openat+0xc5/0x336 2011-09-08 14:07:43 ffff88018ecbf0f8 ffff88018ed00010[ 2316.002404] [] do_truncate+0x59/0x80 2011-09-08 14:07:43 [ 2316.002904] qpsmtpd-prefork D[ 2316.003755] [] ? xfs_get_blocks_direct+0x11/0x11 2011-09-08 14:07:43 0 18760 4733 0x00000000 2011-09-08 14:07:43 [ 2316.005748] [] ? xfs_vm_write_end+0x24/0x46 2011-09-08 14:07:43 [ 2316.005937] [] mutex_lock+0x16/0x27 2011-09-08 14:07:43 2011-09-08 14:07:43 [ 2316.009770] [] ? nameidata_to_filp+0x53/0x61 2011-09-08 14:07:43 [ 2316.014835] [] do_last+0x64a/0x651 2011-09-08 14:07:43 [ 2316.009770] [] ? nameidata_to_filp+0x53/0x61 2011-09-08 14:07:43 [ 2316.014897] [] path_openat+0xc5/0x336 2011-09-08 14:07:43 [ 2316.015565] 0000000000004000 ffffffff811d4cf6[ 2316.018862] ffff88018eec9fd8[ 2316.019777] [] sys_semop+0xb/0xd 2011-09-08 14:07:43 [ 2316.020881] [] ? sys_semtimedop+0x2f3/0x845 2011-09-08 14:07:43 [ 2316.021933] 0000000000004000[ 2316.022890] [] ? mntput_no_expire+0x96/0x155 2011-09-08 14:07:43 ffff88043ffd9e60[ 2316.024299] [] ? core_sys_select+0x1a3/0x253 2011-09-08 14:07:43 ffff88018effffd8[ 2316.026420] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:43 [ 2316.027399] [] ? tcp_recvmsg+0x7c6/0x8e4 2011-09-08 14:07:43 [ 2316.027777] [] ? mnt_get_count+0x46/0x5a 2011-09-08 14:07:43 [ 2316.029149] [] sys_semtimedop+0x6c1/0x845 2011-09-08 14:07:43 [ 2316.029401] [] ? vfsmount_lock_global_unlock_online+0x40/0x4f 2011-09-08 14:07:43 [ 2316.029401] [] ? vfsmount_lock_global_unlock_online+0x40/0x4f 2011-09-08 14:07:43 [ 2316.034488] [] ? mntput_no_expire+0x96/0x155 2011-09-08 14:07:43 [ 2316.034488] [] ? mntput_no_expire+0x96/0x155 2011-09-08 14:07:43 [ 2316.034552] [] ? vfs_read+0xab/0x14e 2011-09-08 14:07:43 [ 2316.034552] [] ? vfs_read+0xab/0x14e 2011-09-08 14:07:43 [ 2316.036084] [] ? vfsmount_lock_global_unlock_online+0x40/0x4f 2011-09-08 14:07:43 [ 2316.036414] qmail-popup S[ 2316.037894] pop3passwdcheck D 0000000000010640[ 2316.040541] [] mutex_lock+0x16/0x27 2011-09-08 14:07:43 [ 2316.041221] httpd S ffff8801854e8cd0[ 2316.042586] [] sys_semop+0xb/0xd 2011-09-08 14:07:43 2011-09-08 14:07:42 2011-09-08 14:07:42 [ 2314.090431] Call Trace: 2011-09-08 14:07:42 [ 2314.091310] mysqld S[ 2314.092668] [] tcp_recvmsg+0x457/0x8e4 2011-09-08 14:07:42 [ 2314.094021] [] ? sk_reset_timer+0x17/0x23 2011-09-08 14:07:42 ffff88017fba6010[ 2314.096063] [] ? __kfree_skb+0x7c/0x80 2011-09-08 14:07:42 ffff880198346010 0000000000010640[ 2314.097921] [] ? __kfree_skb+0x7c/0x80 2011-09-08 14:07:42 [ 2314.098110] [] ? wake_up_bit+0x23/0x23 2011-09-08 14:07:42 [ 2314.099249] ffff88017f963fd8[ 2314.100278] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:42 [ 2314.103864] mysqld S 2011-09-08 14:07:42 [ 2314.104719] [] ? unix_destruct_scm+0xbc/0xc9 2011-09-08 14:07:42 [ 2314.104719] [] ? unix_destruct_scm+0xbc/0xc9 2011-09-08 14:07:42 [ 2314.109795] [] ? get_futex_key+0x99/0x229 2011-09-08 14:07:42 [ 2314.109795] [] ? get_futex_key+0x99/0x229 2011-09-08 14:07:42 [ 2314.109858] [] futex_wait_queue_me+0xb8/0xd3 2011-09-08 14:07:42 [ 2314.109858] [] futex_wait_queue_me+0xb8/0xd3 2011-09-08 14:07:42 [ 2314.109922] [] futex_wait+0x129/0x242 2011-09-08 14:07:42 [ 2314.110109] [] ? do_sync_read+0xc7/0x10d 2011-09-08 14:07:42 ffff880417cbdac8 ffff880417cbd560 0000000000010640[ 2314.113822] [] ? sched_setscheduler+0xe/0x10 2011-09-08 14:07:42 [ 2314.114927] [] ? tcp_send_delayed_ack+0xbd/0xbf 2011-09-08 14:07:42 [ 2314.115181] [] futex_wait_queue_me+0xb8/0xd3 2011-09-08 14:07:42 [ 2314.115495] [] ? cp_new_stat+0xe2/0xee 2011-09-08 14:07:42 [ 2314.116480] Call Trace: 2011-09-08 14:07:42 [ 2314.117556] ffff88068dd23be8[ 2314.118400] [] ? get_futex_key+0x99/0x229 2011-09-08 14:07:42 [ 2314.118712] [] ? do_sync_read+0xc7/0x10d 2011-09-08 14:07:42 0000000000010640 0000000000000086[ 2314.122167] [] ? do_sync_read+0xc7/0x10d 2011-09-08 14:07:42 [ 2314.123342] Call Trace: 2011-09-08 14:07:42 [ 2314.124090] [] ? do_device_not_available+0x9/0xb 2011-09-08 14:07:42 [ 2314.124090] [] ? do_device_not_available+0x9/0xb 2011-09-08 14:07:42 [ 2314.129164] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:42 [ 2314.129164] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:42 [ 2314.129226] mysqld S[ 2314.130142] [] ? tcp_send_delayed_ack+0xbd/0xbf 2011-09-08 14:07:42 ffff8802592d3a08 ffff8802592d3fd8[ 2314.132626] [] do_sync_read+0xc7/0x10d 2011-09-08 14:07:42 ffffffff813ebb3f ffff8803c78d9fd8[ 2314.135466] Call Trace: 2011-09-08 14:07:42 ffff8803cdc85bc8 [ 2314.137626] [] futex_wait_queue_me+0xb8/0xd3 2011-09-08 14:07:42 [ 2314.138003] [] sys_futex+0x10e/0x12c 2011-09-08 14:07:42 ffffffff8144426a ffff8803fc610828 [ 2314.140282] 0000000000004000 0000000000010640[ 2314.140908] [] ? __tcp_ack_snd_check+0x6f/0x7d 2011-09-08 14:07:42 ffff8801855b3ad8[ 2314.142956] [] ? do_sync_read+0xc7/0x10d 2011-09-08 14:07:42 ffff880185623fd8 2011-09-08 14:07:42 ffff880185623fd8 2011-09-08 14:07:42 ffff880185623af8[ 2314.150002] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:42 [ 2314.150861] Call Trace: 2011-09-08 14:07:42 ffff8803c78c70b8 [ 2314.152717] [] ? unix_destruct_scm+0xbc/0xc9 2011-09-08 14:07:42 [ 2314.154568] [] ? get_futex_key+0x99/0x229 2011-09-08 14:07:42 2011-09-08 14:07:42 [ 2314.156806] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:42 0000000000000086[ 2314.158224] [] tcp_recvmsg+0x457/0x8e4 2011-09-08 14:07:42 [ 2314.159301] ffff88017b1cffd8 0 18969 4425 0x00000000 2011-09-08 14:07:42 [ 2314.161426] [] ? __tcp_ack_snd_check+0x6f/0x7d 2011-09-08 14:07:42 ffff88017a1f2a50 0000000000010640[ 2314.163037] [] ? tcp_send_delayed_ack+0xbd/0xbf 2011-09-08 14:07:42 [ 2314.163290] [] futex_wait+0x129/0x242 2011-09-08 14:07:42 [ 2314.163290] [] futex_wait+0x129/0x242 2011-09-08 14:07:42 [ 2314.168361] [] ? inet_recvmsg+0x6b/0x80 2011-09-08 14:07:42 [ 2314.168361] [] ? inet_recvmsg+0x6b/0x80 2011-09-08 14:07:42 [ 2314.168423] [] do_futex+0x9c/0x9b4 2011-09-08 14:07:42 [ 2314.168865] mysqld S ffff880179ff5ad8 2011-09-08 14:07:42 2011-09-08 14:07:42 0000000000000086[ 2314.171703] [] ? wake_up_bit+0x23/0x23 2011-09-08 14:07:42 0000000000010640 0000000000010640[ 2314.174974] [] ? tcp_send_delayed_ack+0xbd/0xbf 2011-09-08 14:07:42 [ 2314.175227] [] futex_wait+0x129/0x242 2011-09-08 14:07:42 0000000000010640 ffff88042f707c08 ffff88042e438010[ 2314.178623] [] schedule_hrtimeout_range_clock+0xda/0x123 2011-09-08 14:07:42 [ 2314.178880] [] poll_schedule_timeout+0x46/0x69 2011-09-08 14:07:42 [ 2314.179814] [] ? sys_accept4+0x1d1/0x1f9 2011-09-08 14:07:42 ffff88042f6719a8[ 2314.181491] [] do_sys_poll+0x36f/0x410 2011-09-08 14:07:42 ffff88082d607978 ffff88082d606010 2011-09-08 14:07:42 ffff88082d606010 2011-09-08 14:07:42 [ 2314.187607] 0000000000004000[ 2314.187607] 0000000000004000 ffff88082e39a1c0 ffff88082e39a1c0 2011-09-08 14:07:42 [ 2314.188805] [] ? poll_freewait+0x8d/0x8d 2011-09-08 14:07:42 [ 2314.189752] [] sys_poll+0x50/0xba 2011-09-08 14:07:42 0000000000010640 0000000000010640[ 2314.191977] splogger S ffff88042d3c1fd8[ 2314.192830] [] ? check_preempt_curr+0x2d/0x6c 2011-09-08 14:07:42 [ 2314.193711] [] ? path_put+0x1d/0x22 2011-09-08 14:07:42 0000000000010640[ 2314.195632] [] ? do_page_fault+0x342/0x37d 2011-09-08 14:07:42 0 4724 4715 0x00020000 2011-09-08 14:07:42 ffff88042f706480 2011-09-08 14:07:42 [ 2314.197549] [] schedule_hrtimeout_range_clock+0x50/0x123 2011-09-08 14:07:42 [ 2314.198620] [] compat_core_sys_select+0x178/0x213 2011-09-08 14:07:42 [ 2314.199978] [] ? current_fs_time+0x22/0x29 2011-09-08 14:07:42 ffff88042b41ffd8 ffff88042b41feb8 ffff88042b41feb8 2011-09-08 14:07:42 2011-09-08 14:07:42 [ 2314.206470] Call Trace: 2011-09-08 14:07:42 ffff88042f6c1fd8 0000000000000082[ 2314.209932] [] ? wake_up_bit+0x23/0x23 2011-09-08 14:07:42 [ 2314.210851] 0000000000004000 ffff88015ccc8c90 ffff88042d259fd8[ 2314.212704] [] ? check_preempt_curr+0x2d/0x6c 2011-09-08 14:07:42 [ 2314.213657] ffff88080c875fd8[ 2314.214109] [] ? timerqueue_add+0x81/0x9f 2011-09-08 14:07:42 [ 2314.216236] [] ? try_to_wake_up+0x2f0/0x2f0 2011-09-08 14:07:42 0 5909 1 0x00000000 2011-09-08 14:07:42 [ 2314.217784] [] sys_nanosleep+0x54/0x6c 2011-09-08 14:07:42 0000000000000000[ 2314.219145] [] ? hrtimer_start_range_ns+0xf/0x11 2011-09-08 14:07:42 [ 2314.220211] ffff88080db61fd8[ 2314.220551] [] ? timerqueue_add+0x81/0x9f 2011-09-08 14:07:42 [ 2314.221188] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:42 [ 2314.221188] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:42 [ 2314.226264] imap-login S[ 2314.226264] imap-login S ffff88082e63da48 [ 2314.227241] [] ? enqueue_hrtimer+0x48/0x74 2011-09-08 14:07:42 [ 2314.227562] [] ? hrtimer_start_range_ns+0xf/0x11 2011-09-08 14:07:42 [ 2314.227815] [] ? mntput+0x21/0x23 2011-09-08 14:07:42 [ 2314.228736] Call Trace: 2011-09-08 14:07:42 [ 2314.229430] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:42 0000000000010640[ 2314.230855] [] sys_epoll_wait+0x22a/0x387 2011-09-08 14:07:42 0 6490 4294 0x00000000 2011-09-08 14:07:42 [ 2314.232076] [] ? wait_for_common+0xd3/0x13b 2011-09-08 14:07:42 0 6497 6490 0x00000008 2011-09-08 14:07:42 [ 2314.234880] [] ? system_call_fastpath+0x16/0x1b 2011-09-08 14:07:42 [ 2314.236914] 0000000000004000[ 2314.238164] [] ? sys_write+0x47/0x6e 2011-09-08 14:07:42 [ 2314.239205] [] ? timerqueue_add+0x81/0x9f 2011-09-08 14:07:42 [ 2314.239589] [] schedule_hrtimeout_range+0xe/0x10 2011-09-08 14:07:42 [ 2314.240699] Call Trace: 2011-09-08 14:07:42 [ 2314.240945] [] ? __hrtimer_start_range_ns+0x31f/0x331 2011-09-08 14:07:42 [ 2314.240945] [] ? __hrtimer_start_range_ns+0x31f/0x331 2011-09-08 14:07:42 [ 2314.246017] [] schedule_hrtimeout_range_clock+0xda/0x123 2011-09-08 14:07:42 [ 2314.246017] [] schedule_hrtimeout_range_clock+0xda/0x123 2011-09-08 14:07:42 [ 2314.246083] [] ? update_rmtp+0x60/0x60 2011-09-08 14:07:42 [ 2314.247033] [] ? apic_timer_interrupt+0xe/0x20 2011-09-08 14:07:42 [ 2314.247288] [] ? remove_wait_queue+0x48/0x4d 2011-09-08 14:07:42 [ 2314.248582] [] ? start_flush_work+0x19/0xab 2011-09-08 14:07:42 ffff880405a80010[ 2314.249903] ffff880405a81fd8[ 2314.250123] Call Trace: 2011-09-08 14:07:42 0 7030 1 0x00000000 2011-09-08 14:07:42 [ 2314.252226] [] n_tty_read+0x4d0/0x760 2011-09-08 14:07:42 ffff880405a7fa88 [ 2314.253829] [] n_tty_read+0x4d0/0x760 2011-09-08 14:07:42 000000000000000d[ 2314.255229] [] ? try_to_wake_up+0x2de/0x2f0 2011-09-08 14:07:42 ffff8803fffadd68 0000000000000082[ 2314.258758] [] ? hrtimer_start_range_ns+0xf/0x11 2011-09-08 14:07:42 [ 2314.259013] [] ? sys_read+0x47/0x6f 2011-09-08 14:07:42 0000000000000000 ffff88040fe8bfd8[ 2314.260310] [] ? update_rmtp+0x60/0x60 2011-09-08 14:07:42 [ 2314.260310] [] ? update_rmtp+0x60/0x60 2011-09-08 14:07:42 [ 2314.265381] [] ? hrtimer_start_range_ns+0xf/0x11 2011-09-08 14:07:42 [ 2314.265381] [] ? hrtimer_start_range_ns+0xf/0x11 2011-09-08 14:07:42 [ 2314.265446] [] schedule_hrtimeout_range+0xe/0x10 2011-09-08 14:07:42 [ 2314.265446] [] schedule_hrtimeout_range+0xe/0x10 2011-09-08 14:07:42 [ 2314.265509] [] sys_epoll_wait+0x22a/0x387 2011-09-08 14:07:42 [ 2314.265509] [] sys_epoll_wait+0x22a/0x387 2011-09-08 14:07:42 [ 2314.265572] [] ? try_to_wake_up+0x2f0/0x2f0 2011-09-08 14:07:42 2011-09-08 14:07:42 2011-09-08 14:07:42 2011-09-08 14:07:42 ffff8804073fe140 ffff8804005c9fd8 ffff88042f5b72f8 [ 2314.271888] [] ? __hrtimer_start_range_ns+0x31f/0x331 2011-09-08 14:07:42 ffff88040b032040[ 2314.274063] [] ? try_to_wake_up+0x2f0/0x2f0 2011-09-08 14:07:42 [ 2314.274386] ffff8803fdc3fdc8[ 2314.275296] [] schedule_hrtimeout_range_clock+0xda/0x123 2011-09-08 14:07:42 2011-09-08 14:07:42 2011-09-08 14:07:42 [ 2314.280193] [] schedule_hrtimeout_range_clock+0xda/0x123 2011-09-08 14:07:42 [ 2314.280193] [] schedule_hrtimeout_range_clock+0xda/0x123 2011-09-08 14:07:42 [ 2314.285269] [] ? update_rmtp+0x60/0x60 2011-09-08 14:07:42 [ 2314.285269] [] ? update_rmtp+0x60/0x60 2011-09-08 14:07:42 [ 2314.285332] [] ? hrtimer_start_range_ns+0xf/0x11 2011-09-08 14:07:42 [ 2314.285649] errorlogs.pl S 0000000000000082[ 2314.286816] [] do_sync_read+0xc7/0x10d 2011-09-08 14:07:42 000000000000037f ffff8803ffc15908 [ 2314.291487] [] ? hrtimer_start_range_ns+0xf/0x11 2011-09-08 14:07:42 [ 2314.291865] imap S 0000000000000082 0000000000010640 ffff88040e689dd8[ 2314.293101] [] ? hrtimer_start_range_ns+0xf/0x11 2011-09-08 14:07:42 [ 2314.293417] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:42 0000000000000000 ffff880424455fd8[ 2314.294458] [] ? enqueue_hrtimer+0x48/0x74 2011-09-08 14:07:42 2011-09-08 14:07:42 ffff88042f62bac8 ffff88042f62b560 ffff88042f62b560[ 2314.298374] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:42 [ 2314.298437] httpd D[ 2314.299858] [] xfs_vn_mknod+0x9a/0xf5 2011-09-08 14:07:42 [ 2314.299921] [] vfs_mknod+0x8f/0xc5 2011-09-08 14:07:42 [ 2314.299921] [] vfs_mknod+0x8f/0xc5 2011-09-08 14:07:42 [ 2314.304991] [] unix_bind+0x1c0/0x383 2011-09-08 14:07:42 [ 2314.304991] [] unix_bind+0x1c0/0x383 2011-09-08 14:07:42 [ 2314.305053] [] ? get_empty_filp+0x95/0x123 2011-09-08 14:07:42 2011-09-08 14:07:42 ffff8803ffcfd0b8 [ 2314.307764] Call Trace: 2011-09-08 14:07:42 [ 2314.307764] Call Trace: 2011-09-08 14:07:42 [ 2314.308869] 0000000000004000[ 2314.309871] [] sys_semop+0xb/0xd 2011-09-08 14:07:42 [ 2314.311037] [] ? core_sys_select+0x1a3/0x253 2011-09-08 14:07:42 ffff8803fd50dfd8 ffff8803fd6eb2b8 ffff8803fd750010[ 2314.313776] ffff8803fd751fd8 2011-09-08 14:07:42 [ 2314.314340] [] ? path_openat+0x30c/0x336 2011-09-08 14:07:42 [ 2314.317682] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:42 [ 2314.318976] [] ? wake_up_bit+0x23/0x23 2011-09-08 14:07:42 [ 2314.319227] [] vfs_read+0xab/0x14e 2011-09-08 14:07:42 ffff8803fd259808 2011-09-08 14:07:42 2011-09-08 14:07:42 2011-09-08 14:07:42 [ 2314.325117] ffff8803fd25bfd8[ 2314.325117] ffff8803fd25bfd8[ 2314.325647] [] ? nr_running+0x45/0x5f 2011-09-08 14:07:42 [ 2314.326663] ffff8803fcbabfd8 ffff8803fc454200 ffff8801f3176000[ 2314.328551] [] sys_semtimedop+0x6c1/0x845 2011-09-08 14:07:42 [ 2314.328803] [] ? ktime_get_ts+0xad/0xba 2011-09-08 14:07:42 [ 2314.329117] imap S[ 2314.329969] [] ? rb_insert_color+0xbd/0xe6 2011-09-08 14:07:42 [ 2314.330417] [] schedule_hrtimeout_range+0xe/0x10 2011-09-08 14:07:42 ffff8803f8175fd8[ 2314.332402] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:42 [ 2314.333637] [] ? update_rmtp+0x60/0x60 2011-09-08 14:07:42 [ 2314.334079] httpd S ffff8801d461e000[ 2314.335185] [] ? nr_running+0x45/0x5f 2011-09-08 14:07:42 [ 2314.336732] [] schedule_hrtimeout_range_clock+0xda/0x123 2011-09-08 14:07:42 [ 2314.337932] ffff8803e1e1dfd8 0000000000010640[ 2314.339879] [] sys_semtimedop+0x6c1/0x845 2011-09-08 14:07:42 [ 2314.339879] [] sys_semtimedop+0x6c1/0x845 2011-09-08 14:07:42 [ 2314.344948] [] ? core_sys_select+0x1a3/0x253 2011-09-08 14:07:42 [ 2314.344948] [] ? core_sys_select+0x1a3/0x253 2011-09-08 14:07:42 [ 2314.345011] [] ? nr_running+0x45/0x5f 2011-09-08 14:07:42 [ 2314.345011] [] ? nr_running+0x45/0x5f 2011-09-08 14:07:42 [ 2314.345073] [] ? loadavg_proc_show+0xb3/0xc7 2011-09-08 14:07:42 [ 2314.345326] [] sys_semop+0xb/0xd 2011-09-08 14:07:42 [ 2314.346931] httpd S[ 2314.347846] [] ? sch_direct_xmit+0x96/0x197 2011-09-08 14:07:42 2011-09-08 14:07:42 ffff88041ef81000[ 2314.351445] [] ? loadavg_proc_show+0xb3/0xc7 2011-09-08 14:07:42 0000000000000000 ffff8802f3d2b000 0 7381 7005 0x00000000 2011-09-08 14:07:42 ffff8803cff402c0 ffff8803cffe9cc0[ 2314.354591] [] ? cp_new_stat+0xe2/0xee 2011-09-08 14:07:42 [ 2314.355695] [] ? tcp_recvmsg+0x7c6/0x8e4 2011-09-08 14:07:42 [ 2314.356010] [] ? vfsmount_lock_global_unlock_online+0x40/0x4f 2011-09-08 14:07:42 [ 2314.356341] httpd S[ 2314.357444] [] ? wait_noreap_copyout+0xed/0xed 2011-09-08 14:07:42 ffff8803cf31fbb8 2011-09-08 14:07:42 2011-09-08 14:07:42 [ 2314.364922] Call Trace: 2011-09-08 14:07:42 [ 2314.364922] Call Trace: 2011-09-08 14:07:42 [ 2314.365533] [] ? poll_select_copy_remaining+0xc5/0xe9 2011-09-08 14:07:42 [ 2314.366953] [] ? do_sync_read+0xc7/0x10d 2011-09-08 14:07:42 [ 2314.367221] [] ? vfs_read+0xab/0x14e 2011-09-08 14:07:42 [ 2314.368385] [] sys_semtimedop+0x6c1/0x845 2011-09-08 14:07:42 ffff8803c7bd5fd8 0000000000000000 0000000000010640 ffff8803ae4d5dd8[ 2314.373015] [] ? timerqueue_add+0x81/0x9f 2011-09-08 14:07:42 0000000000010640 0000000000000082 ffff8803f23872a0[ 2314.376692] [] ? try_to_wake_up+0x2f0/0x2f0 2011-09-08 14:07:42 [ 2314.377175] 0000000000004000[ 2314.378295] [] sys_epoll_wait+0x22a/0x387 2011-09-08 14:07:42 [ 2314.379336] [] ? rb_insert_color+0x69/0xe6 2011-09-08 14:07:42 [ 2314.379656] [] ? update_rmtp+0x60/0x60 2011-09-08 14:07:42 [ 2314.379656] [] ? update_rmtp+0x60/0x60 2011-09-08 14:07:42 [ 2314.384735] [] ? hrtimer_start_range_ns+0xf/0x11 2011-09-08 14:07:42 0 7550 5879 0x00000000 2011-09-08 14:07:42 [ 2314.386286] [] ? hrtimer_start_range_ns+0xf/0x11 2011-09-08 14:07:42 [ 2314.387398] Call Trace: 2011-09-08 14:07:42 ffff8803ae549c98 0000000000010640 ffff8803ae549dd8[ 2314.389256] [] ? timerqueue_add+0x81/0x9f 2011-09-08 14:07:42 ffff8803ae636a50[ 2314.390313] 0000000000004000[ 2314.391448] [] ? sys_write+0x47/0x6e 2011-09-08 14:07:42 [ 2314.392616] [] ? __hrtimer_start_range_ns+0x31f/0x331 2011-09-08 14:07:42 [ 2314.393703] ffff8803aa08ffd8[ 2314.395147] 0000000000004000 0000000000010640 ffff8803ae7f6768 [ 2314.397515] [] schedule_hrtimeout_range_clock+0xda/0x123 2011-09-08 14:07:42 [ 2314.397834] [] ? try_to_wake_up+0x2f0/0x2f0 2011-09-08 14:07:42 ffff8803a4e03c98[ 2314.399065] [] ? __hrtimer_start_range_ns+0x31f/0x331 2011-09-08 14:07:42 [ 2314.399450] [] ? try_to_wake_up+0x2f0/0x2f0 2011-09-08 14:07:42 [ 2314.399450] [] ? try_to_wake_up+0x2f0/0x2f0 2011-09-08 14:07:42 [ 2314.404524] [] ? sys_write+0x47/0x6e 2011-09-08 14:07:42 [ 2314.404524] [] ? sys_write+0x47/0x6e 2011-09-08 14:07:42 [ 2314.404649] imap S ffff8803ae681b48 [ 2314.406077] [] ? try_to_wake_up+0x2f0/0x2f0 2011-09-08 14:07:42 [ 2314.406377] imap S 0000000000000000[ 2314.407678] [] schedule_hrtimeout_range+0xe/0x10 2011-09-08 14:07:42 0 7699 5879 0x00000000 2011-09-08 14:07:42 ffff8803ae5cffd8[ 2314.411036] imap S ffff8803a4db6d50[ 2314.413684] [] ? enqueue_hrtimer+0x48/0x74 2011-09-08 14:07:42 0000000000010640[ 2314.415235] [] ? timerqueue_add+0x81/0x9f 2011-09-08 14:07:42 [ 2314.415746] [] ? try_to_wake_up+0x2f0/0x2f0 2011-09-08 14:07:42 ffff8803a4ff7c98 ffff8803a4ff7c98[ 2314.417361] [] ? mntput+0x21/0x23 2011-09-08 14:07:42 [ 2314.417486] imap S[ 2314.418838] [] schedule_hrtimeout_range+0xe/0x10 2011-09-08 14:07:42 ffff880382ff1fd8 2011-09-08 14:07:42 ffff880382ff1fd8 2011-09-08 14:07:42 [ 2314.424741] ffff880382ff1fd8[ 2314.424741] ffff880382ff1fd8[ 2314.425466] [] schedule_hrtimeout_range+0xe/0x10 2011-09-08 14:07:42 [ 2314.425719] imap S[ 2314.426761] [] ? __hrtimer_start_range_ns+0x31f/0x331 2011-09-08 14:07:42 [ 2314.428825] imap S ffff88039f899320[ 2314.430434] imap S[ 2314.430790] 0000000000004000[ 2314.431734] [] schedule_hrtimeout_range+0xe/0x10 2011-09-08 14:07:42 0000000000000082[ 2314.434840] [] schedule_hrtimeout_range+0xe/0x10 2011-09-08 14:07:42 [ 2314.435793] ffff88035fc5ffd8[ 2314.436126] Call Trace: 2011-09-08 14:07:42 [ 2314.436372] [] ? __hrtimer_start_range_ns+0x31f/0x331 2011-09-08 14:07:42 [ 2314.437582] ffff880373c93fd8 0000000000000000[ 2314.439475] [] ? rb_insert_color+0xbd/0xe6 2011-09-08 14:07:42 [ 2314.439475] [] ? rb_insert_color+0xbd/0xe6 2011-09-08 14:07:42 [ 2314.444541] [] ? timerqueue_add+0x81/0x9f 2011-09-08 14:07:42 [ 2314.444667] [] ? __hrtimer_start_range_ns+0x31f/0x331 2011-09-08 14:07:42 [ 2314.445052] [] ? try_to_wake_up+0x2f0/0x2f0 2011-09-08 14:07:42 0000000000000000[ 2314.446478] [] schedule_hrtimeout_range+0xe/0x10 2011-09-08 14:07:42 2011-09-08 14:07:42 [ 2314.447829] [] ? enqueue_hrtimer+0x48/0x74 2011-09-08 14:07:42 ffff880311229fd8[ 2314.449199] Call Trace: 2011-09-08 14:07:42 [ 2314.449445] [] ? __hrtimer_start_range_ns+0x31f/0x331 2011-09-08 14:07:42 0000000000010640[ 2314.451698] [] ? wake_up_state+0xb/0xd 2011-09-08 14:07:42 [ 2314.452136] clamd S 0000000000000086 ffff88031126dfd8[ 2314.453545] [] ? default_wake_function+0xd/0xf 2011-09-08 14:07:42 0000000000010640[ 2314.455602] [] ? __pollwait+0xcb/0xcb 2011-09-08 14:07:42 [ 2314.455916] [] ? move_addr_to_user+0x42/0x58 2011-09-08 14:07:42 0000000000010640[ 2314.457146] [] ? enqueue_hrtimer+0x48/0x74 2011-09-08 14:07:42 [ 2314.457782] httpd S 0000000000010640[ 2314.458579] Call Trace: 2011-09-08 14:07:42 [ 2314.458887] [] ? core_sys_select+0x1a3/0x253 2011-09-08 14:07:42 [ 2314.458887] [] ? core_sys_select+0x1a3/0x253 2011-09-08 14:07:42 [ 2314.463958] [] ? nr_running+0x45/0x5f 2011-09-08 14:07:42 [ 2314.463958] [] ? nr_running+0x45/0x5f 2011-09-08 14:07:42 [ 2314.464020] [] ? loadavg_proc_show+0xb3/0xc7 2011-09-08 14:07:42 [ 2314.464020] [] ? loadavg_proc_show+0xb3/0xc7 2011-09-08 14:07:42 [ 2314.464084] [] ? ktime_get_ts+0xad/0xba 2011-09-08 14:07:42 [ 2314.464084] [] ? ktime_get_ts+0xad/0xba 2011-09-08 14:07:42 [ 2314.464398] httpd S ffff88030881c010 0000000000010640 2011-09-08 14:07:42 ffff880308b3fbb8 0000000000000000 0000000000000000 0000000000000086[ 2314.471039] 0000000000004000 ffff8802d6571c98[ 2314.473400] [] ? update_rmtp+0x60/0x60 2011-09-08 14:07:42 ffff8802d66ef9b8[ 2314.475512] [] path_openat+0xc5/0x336 2011-09-08 14:07:42 [ 2314.475762] [] sys_open+0x1b/0x1d 2011-09-08 14:07:42 0000000000000000 ffff8802cc1ebfd8[ 2314.477552] imap S[ 2314.478724] [] ? update_rmtp+0x60/0x60 2011-09-08 14:07:42 [ 2314.478724] [] ? update_rmtp+0x60/0x60 2011-09-08 14:07:42 [ 2314.483798] [] ? hrtimer_start_range_ns+0xf/0x11 2011-09-08 14:07:42 [ 2314.483798] [] ? hrtimer_start_range_ns+0xf/0x11 2011-09-08 14:07:42 [ 2314.483989] [] ? try_to_wake_up+0x2f0/0x2f0 2011-09-08 14:07:42 0000000000000082 ffff88082ee29360 0000000000010640 ffff880795bee010 ffff880795befdd8 0000000000000000[ 2314.488632] [] ? hrtimer_start_range_ns+0xf/0x11 2011-09-08 14:07:42 [ 2314.490057] [] schedule_hrtimeout_range_clock+0xda/0x123 2011-09-08 14:07:42 [ 2314.493478] [] schedule_hrtimeout_range+0xe/0x10 2011-09-08 14:07:42 0 9860 5879 0x00000000 2011-09-08 14:07:42 [ 2314.494822] [] ? timerqueue_add+0x81/0x9f 2011-09-08 14:07:42 [ 2314.495143] [] ? hrtimer_start_range_ns+0xf/0x11 2011-09-08 14:07:42 [ 2314.496375] [] ? rb_insert_color+0xbd/0xe6 2011-09-08 14:07:42 [ 2314.497449] [] ? __wake_up_common+0x46/0x75 2011-09-08 14:07:42 0 10607 10529 0x00000000 2011-09-08 14:07:42 0 10607 10529 0x00000000 2011-09-08 14:07:42 [ 2314.503288] ffff88037e8c98e8[ 2314.503288] ffff88037e8c98e8 0000000000000082 0000000000000082 0000000000000000[ 2314.504005] [] ? rb_insert_color+0x69/0xe6 2011-09-08 14:07:42 [ 2314.505021] [] core_sys_select+0x1a3/0x253 2011-09-08 14:07:42 [ 2314.505272] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:42 [ 2314.506304] [] ? timerqueue_add+0x81/0x9f 2011-09-08 14:07:42 [ 2314.507258] [] core_sys_select+0x1a3/0x253 2011-09-08 14:07:42 [ 2314.508487] [] ? timerqueue_add+0x81/0x9f 2011-09-08 14:07:42 0000000000010640[ 2314.510972] 0000000000004000 2011-09-08 14:07:42 ffff8807a5bc0010[ 2314.515007] [] ? update_rmtp+0x60/0x60 2011-09-08 14:07:42 [ 2314.516426] [] ? enqueue_hrtimer+0x48/0x74 2011-09-08 14:07:42 ffffffff81210539[ 2314.517915] [] ? rb_insert_color+0x69/0xe6 2011-09-08 14:07:42 [ 2314.517915] [] ? rb_insert_color+0x69/0xe6 2011-09-08 14:07:42 [ 2314.522990] [] ? timerqueue_add+0x81/0x9f 2011-09-08 14:07:42 [ 2314.522990] [] ? timerqueue_add+0x81/0x9f 2011-09-08 14:07:42 [ 2314.525295] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:42 ffff88080e7fb808 [ 2314.528016] [] schedule_hrtimeout_range_clock+0xda/0x123 2011-09-08 14:07:42 [ 2314.529319] Call Trace: 2011-09-08 14:07:42 [ 2314.529501] [] ? enqueue_hrtimer+0x48/0x74 2011-09-08 14:07:42 [ 2314.529809] [] ? update_rmtp+0x60/0x60 2011-09-08 14:07:42 0000000000010640[ 2314.531859] [] do_last+0x304/0x651 2011-09-08 14:07:42 2011-09-08 14:07:42 [ 2314.533148] [] ? loadavg_proc_open+0x17/0x17 2011-09-08 14:07:42 0000000000010640 ffff88072ca35dd8[ 2314.537259] ffff88074de498e8[ 2314.537699] ffff88074de49fd8 ffffffff810b0d26 ffffffff810b0d26 2011-09-08 14:07:42 2011-09-08 14:07:42 [ 2314.542922] Call Trace: 2011-09-08 14:07:42 [ 2314.542922] Call Trace: 2011-09-08 14:07:42 [ 2314.542980] [] ? handle_pte_fault+0x725/0x770 2011-09-08 14:07:42 [ 2314.542980] [] ? handle_pte_fault+0x725/0x770 2011-09-08 14:07:42 [ 2314.543876] [] ? igb_xmit_frame_adv+0x3c/0x3e 2011-09-08 14:07:42 [ 2314.544899] ffff8806edd73dc8[ 2314.547415] [] ? timerqueue_add+0x81/0x9f 2011-09-08 14:07:42 [ 2314.548969] [] ? timerqueue_add+0x81/0x9f 2011-09-08 14:07:42 [ 2314.550184] ffff88072c091fd8 0 15590 5879 0x00000000 2011-09-08 14:07:42 t_range_ns+0x31f/0x331 2011-09-08 14:07:42 ffff8807e4bb5fd8[ 2314.554351] [] fput+0xfd/0x208 2011-09-08 14:07:42 [ 2314.555578] [] ? enqueue_hrtimer+0x48/0x74 2011-09-08 14:07:42 [ 2314.557129] [] ? timerqueue_add+0x81/0x9f 2011-09-08 14:07:42 [ 2314.557256] [] ? __hrtimer_start_range_ns+0x31f/0x331 2011-09-08 14:07:42 [ 2314.557256] [] ? __hrtimer_start_range_ns+0x31f/0x331 2011-09-08 14:07:42 [ 2314.562333] [] schedule_hrtimeout_range_clock+0xda/0x123 2011-09-08 14:07:42 [ 2314.562333] [] schedule_hrtimeout_range_clock+0xda/0x123 2011-09-08 14:07:42 [ 2314.562399] [] ? update_rmtp+0x60/0x60 2011-09-08 14:07:42 [ 2314.562399] [] ? update_rmtp+0x60/0x60 2011-09-08 14:07:42 0000000000000086[ 2314.564079] [] schedule_hrtimeout_range+0xe/0x10 2011-09-08 14:07:42 [ 2314.565556] [] ? __hrtimer_start_range_ns+0x31f/0x331 2011-09-08 14:07:42 ffff88035ff04cf0 0000000000000086[ 2314.570528] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:42 [ 2314.571559] [] ? acl_permission_check+0x69/0x8a 2011-09-08 14:07:42 [ 2314.571813] [] ? cp_new_stat+0xe2/0xee 2011-09-08 14:07:42 [ 2314.572065] [] sys_semop+0xb/0xd 2011-09-08 14:07:42 [ 2314.573042] [] ? loadavg_proc_open+0x17/0x17 2011-09-08 14:07:42 [ 2314.573421] [] ? ktime_get_ts+0xad/0xba 2011-09-08 14:07:42 [ 2314.573734] httpd S[ 2314.574588] [] ? acl_permission_check+0x69/0x8a 2011-09-08 14:07:42 [ 2314.574905] [] ? cp_new_stat+0xe2/0xee 2011-09-08 14:07:42 ffff88082e1ab1f8 ffff8806c42ba010 ffff8806c42bbfd8[ 2314.576709] [] ? try_to_wake_up+0x2f0/0x2f0 2011-09-08 14:07:42 2011-09-08 14:07:42 [ 2314.582384] 0000000000004000 2011-09-08 14:07:42 [ 2314.582384] 0000000000004000 ffff8806bc5da3c0 ffff8806bc5da3c0 0000000000010640 ffff8806c4b33fd8[ 2314.583456] [] ? try_to_wake_up+0x2f0/0x2f0 2011-09-08 14:07:42 ffff8806c2fd6010[ 2314.584689] [] schedule_hrtimeout_range_clock+0xda/0x123 2011-09-08 14:07:42 2011-09-08 14:07:42 [ 2314.586684] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:42 [ 2314.587601] [] ? dev_hard_start_xmit+0x3fa/0x598 2011-09-08 14:07:42 0000000000000000[ 2314.590668] [] ? update_rmtp+0x60/0x60 2011-09-08 14:07:42 0000000000010640[ 2314.591964] [] ? timerqueue_add+0x81/0x9f 2011-09-08 14:07:42 0000000000010640[ 2314.594154] imap S ffff88029864a010[ 2314.595446] [] ? update_rmtp+0x60/0x60 2011-09-08 14:07:42 [ 2314.596465] ffff880298763fd8[ 2314.596742] [] ? rb_insert_color+0xbd/0xe6 2011-09-08 14:07:42 [ 2314.596931] [] ? __hrtimer_start_range_ns+0x31f/0x331 2011-09-08 14:07:42 [ 2314.596931] [] ? __hrtimer_start_range_ns+0x31f/0x331 2011-09-08 14:07:42 [ 2314.602003] [] schedule_hrtimeout_range_clock+0xda/0x123 2011-09-08 14:07:42 [ 2314.602003] [] schedule_hrtimeout_range_clock+0xda/0x123 2011-09-08 14:07:42 ffff8802987c9dd8 0000000000010640[ 2314.605427] [] sys_epoll_wait+0x22a/0x387 2011-09-08 14:07:42 ffff8803d0c55dd8[ 2314.606770] [] ? enqueue_hrtimer+0x48/0x74 2011-09-08 14:07:42 ffff8802b1134010 ffff88042f6351f8 [ 2314.610066] [] ? update_rmtp+0x60/0x60 2011-09-08 14:07:42 [ 2314.613715] [] sys_semop+0xb/0xd 2011-09-08 14:07:42 [ 2314.613839] httpd S[ 2314.615319] httpd S ffff8803140b75a0[ 2314.616550] [] ? ktime_get_ts+0xad/0xba 2011-09-08 14:07:42 [ 2314.616678] [] ? sys_select+0xa7/0xbd 2011-09-08 14:07:42 [ 2314.616678] [] ? sys_select+0xa7/0xbd 2011-09-08 14:07:42 [ 2314.621754] [] sys_semop+0xb/0xd 2011-09-08 14:07:42 [ 2314.621754] [] sys_semop+0xb/0xd 2011-09-08 14:07:42 [ 2314.621816] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:42 [ 2314.621878] httpd S 0000000000000000 0000000000000000 2011-09-08 14:07:42 ffff88020df98000[ 2314.625935] [] ? single_open+0x64/0x93 2011-09-08 14:07:42 2011-09-08 14:07:42 [ 2314.627985] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:42 ffff88025922dbb8 ffff88025922dfd8[ 2314.629395] [] ? ktime_get_ts+0xad/0xba 2011-09-08 14:07:42 [ 2314.630750] [] ? core_sys_select+0x1a3/0x253 2011-09-08 14:07:42 ffff8806c2d89dd8 0 21626 5879 0x00000000 2011-09-08 14:07:42 2011-09-08 14:07:42 [ 2314.635964] [] sys_epoll_wait+0x22a/0x387 2011-09-08 14:07:42 ffff88025751fcd8 ffff88025751fdd8 ffff88025751fdd8 2011-09-08 14:07:42 2011-09-08 14:07:42 [ 2314.642020] Call Trace: 2011-09-08 14:07:42 [ 2314.642020] Call Trace: 2011-09-08 14:07:42 [ 2314.642077] [] ? timerqueue_add+0x81/0x9f 2011-09-08 14:07:42 [ 2314.642204] [] ? __hrtimer_start_range_ns+0x31f/0x331 2011-09-08 14:07:42 ffff880257581fd8[ 2314.643629] [] ? loadavg_proc_open+0x17/0x17 2011-09-08 14:07:42 [ 2314.644008] [] ? seq_release_private+0x3e/0x3e 2011-09-08 14:07:42 [ 2314.645056] Call Trace: 2011-09-08 14:07:42 0000000000000082 ffff8802576a1bf8[ 2314.647358] httpd S ffff8802575ddbf8[ 2314.648670] [] ? mntput_no_expire+0x96/0x155 2011-09-08 14:07:42 0000000000000086[ 2314.650214] httpd S[ 2314.651759] httpd S ffff8803b23e0000 ffff8802514ce010 0 21852 7005 0x00000000 2011-09-08 14:07:42 [ 2314.656133] [] ? ktime_get_ts+0xad/0xba 2011-09-08 14:07:42 ffff88024ec12010 ffff88024ec12010 2011-09-08 14:07:42 2011-09-08 14:07:42 [ 2314.661809] 0000000000004000[ 2314.661809] 0000000000004000 ffff880310de0140 ffff880310de0140 0000000000010640 0000000000010640 0000000000000000[ 2314.662495] [] ? core_sys_select+0x1a3/0x253 2011-09-08 14:07:42 ffff88024ec2dcc0 ffff88024ec2dcc0 ffff880427af1988 [ 2314.665705] [] sys_semop+0xb/0xd 2011-09-08 14:07:42 [ 2314.666874] [] ? handle_pte_fault+0x725/0x770 2011-09-08 14:07:42 ffff88024eee7fd8[ 2314.669059] ffff88024eff1ce8[ 2314.669957] [] ? single_open+0x64/0x93 2011-09-08 14:07:42 0000000000010640[ 2314.671382] Call Trace: 2011-09-08 14:07:42 [ 2314.671629] [] ? __hrtimer_start_range_ns+0x31f/0x331 2011-09-08 14:07:42 [ 2314.671951] [] sys_epoll_wait+0x22a/0x387 2011-09-08 14:07:42 ffff8806ba167dd8[ 2314.673502] [] schedule_hrtimeout_range+0xe/0x10 2011-09-08 14:07:42 [ 2314.673817] imap S ffffffff8121058d ffff88016b93201d 2011-09-08 14:07:42 [ 2314.676400] [] ? acl_permission_check+0x69/0x8a 2011-09-08 14:07:42 [ 2314.676717] [] ? ktime_get_ts+0xad/0xba 2011-09-08 14:07:42 [ 2314.676717] [] ? ktime_get_ts+0xad/0xba 2011-09-08 14:07:42 [ 2314.681790] [] ? poll_select_copy_remaining+0xc5/0xe9 2011-09-08 14:07:42 [ 2314.681790] [] ? poll_select_copy_remaining+0xc5/0xe9 2011-09-08 14:07:42 [ 2314.681854] [] ? sys_select+0xa7/0xbd 2011-09-08 14:07:42 [ 2314.681854] [] ? sys_select+0xa7/0xbd 2011-09-08 14:07:42 0 23398 5879 0x00000000 2011-09-08 14:07:42 ffff88072c9138c8 [ 2314.683083] [] ? __hrtimer_start_range_ns+0x31f/0x331 2011-09-08 14:07:42 [ 2314.685270] imap S 2011-09-08 14:07:42 [ 2314.685846] ffff88072cb83fd8[ 2314.686761] [] ? sys_write+0x47/0x6e 2011-09-08 14:07:42 ffff88074ad48010 0000000000010640[ 2314.688368] [] sys_epoll_wait+0x22a/0x387 2011-09-08 14:07:42 2011-09-08 14:07:42 [ 2314.690046] [] ? try_to_wake_up+0x2f0/0x2f0 2011-09-08 14:07:42 ffff88083d968fb8 ffff88083d968fb8 [ 2314.691598] [] sys_epoll_wait+0x22a/0x387 2011-09-08 14:07:42 [ 2314.691849] imap S[ 2314.692764] [] ? timerqueue_add+0x81/0x9f 2011-09-08 14:07:42 [ 2314.696489] [] ? update_rmtp+0x60/0x60 2011-09-08 14:07:42 [ 2314.696489] [] ? update_rmtp+0x60/0x60 2011-09-08 14:07:42 [ 2314.701558] [] ? hrtimer_start_range_ns+0xf/0x11 2011-09-08 14:07:42 [ 2314.701558] [] ? hrtimer_start_range_ns+0xf/0x11 2011-09-08 14:07:42 [ 2314.701622] [] schedule_hrtimeout_range+0xe/0x10 2011-09-08 14:07:42 [ 2314.701622] [] schedule_hrtimeout_range+0xe/0x10 2011-09-08 14:07:42 [ 2314.701686] [] sys_epoll_wait+0x22a/0x387 2011-09-08 14:07:42 [ 2314.701686] [] sys_epoll_wait+0x22a/0x387 2011-09-08 14:07:42 ffff8806abaa7fd8[ 2314.702734] Call Trace: 2011-09-08 14:07:42 [ 2314.703045] [] schedule_hrtimeout_range_clock+0xda/0x123 2011-09-08 14:07:42 0000000000010640[ 2314.704466] [] ? timerqueue_add+0x81/0x9f 2011-09-08 14:07:42 ffff880717ace010 0000000000010640[ 2314.706328] [] schedule_hrtimeout_range_clock+0xda/0x123 2011-09-08 14:07:42 0000000000010640 ffffffff81210539 2011-09-08 14:07:42 [ 2314.709184] Call Trace: 2011-09-08 14:07:42 ffff8806ba0c4010 ffff8806ba0c5dd8[ 2314.711671] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:42 [ 2314.712716] [] sys_semtimedop+0x6c1/0x845 2011-09-08 14:07:42 [ 2314.713174] [] sys_semop+0xb/0xd 2011-09-08 14:07:42 [ 2314.713298] httpd S[ 2314.714276] [] sys_semtimedop+0x6c1/0x845 2011-09-08 14:07:42 [ 2314.715417] ffff88020a3c3fd8[ 2314.716199] [] sys_semop+0xb/0xd 2011-09-08 14:07:42 [ 2314.716199] [] sys_semop+0xb/0xd 2011-09-08 14:07:42 [ 2314.721271] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:42 [ 2314.721333] qmail-popup S[ 2314.721910] ffff880251489fd8 0 29050 4734 0x00000000 2011-09-08 14:07:42 [ 2314.724036] [] ? __kfree_skb+0x7c/0x80 2011-09-08 14:07:42 [ 2314.724352] [] inet_recvmsg+0x6b/0x80 2011-09-08 14:07:42 0000000000010640 ffff8803fc946fb8 ffff88020a158010[ 2314.727896] [] sys_epoll_wait+0x22a/0x387 2011-09-08 14:07:42 [ 2314.728085] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:42 [ 2314.728283] ffff88025932bdc8 ffff88025932bfd8[ 2314.729624] [] sys_epoll_wait+0x22a/0x387 2011-09-08 14:07:42 0 29073 29034 0x00020000 2011-09-08 14:07:42 [ 2314.731049] [] ? hrtimer_start_range_ns+0xf/0x11 2011-09-08 14:07:42 [ 2314.732372] imap S[ 2314.732729] 0000000000004000[ 2314.733801] [] ? try_to_wake_up+0x2f0/0x2f0 2011-09-08 14:07:42 ffff880221859bb8 ffff880221859fd8[ 2314.735402] [] ? poll_select_copy_remaining+0xc5/0xe9 2011-09-08 14:07:42 ffff880201508010 ffff880201508010 2011-09-08 14:07:42 2011-09-08 14:07:42 [ 2314.741899] [] ? hrtimer_start_range_ns+0xf/0x11 2011-09-08 14:07:42 [ 2314.743130] [] ? rb_insert_color+0x69/0xe6 2011-09-08 14:07:42 [ 2314.744249] 0000000000004000[ 2314.744249] 0000000000004000[ 2314.745510] proftpd D 0 29872 4735 0x00000000 2011-09-08 14:07:42 2011-09-08 14:07:42 [ 2314.746199] ffff8801f336ffd8[ 2314.748958] stunnel4 S[ 2314.748958] stunnel4 S ffff88042e52049c 2011-09-08 14:07:42 [ 2314.749754] Call Trace: 2011-09-08 14:07:42 [ 2314.750381] [] ? sys_newstat+0x2c/0x3b 2011-09-08 14:07:42 [ 2314.750506] stunnel x[ 2314.751737] httpd S ffff8802f9b4c968 [ 2314.752773] [] ? single_open+0x64/0x93 2011-09-08 14:07:42 0000000000010640[ 2314.754200] Call Trace: 2011-09-08 14:07:42 [ 2314.754508] [] sys_exit_group+0x12/0x16 2011-09-08 14:07:42 [ 2314.754767] ffff8806c4bf1dc8[ 2314.755804] [] ? update_rmtp+0x60/0x60 2011-09-08 14:07:42 [ 2314.755804] [] ? update_rmtp+0x60/0x60 2011-09-08 14:07:42 [ 2314.760870] [] ? hrtimer_start_range_ns+0xf/0x11 2011-09-08 14:07:42 [ 2314.760870] [] ? hrtimer_start_range_ns+0xf/0x11 2011-09-08 14:07:42 [ 2314.760934] [] schedule_hrtimeout_range+0xe/0x10 2011-09-08 14:07:42 [ 2314.760934] [] schedule_hrtimeout_range+0xe/0x10 2011-09-08 14:07:42 [ 2314.760997] [] sys_epoll_wait+0x22a/0x387 2011-09-08 14:07:42 ffff8806972f0968 ffff880697328010[ 2314.761825] ffff880697329fd8 0000000000000046 0000000000000046[ 2314.764085] [] ? manage_workers+0x186/0x186 2011-09-08 14:07:42 [ 2314.764973] ffff8806940fffd8 ffff8806ba072868 [ 2314.766865] [] ? timerqueue_add+0x81/0x9f 2011-09-08 14:07:42 ffff8801e4c7ffd8 0000000000000082[ 2314.770332] [] ? wake_up_bit+0x23/0x23 2011-09-08 14:07:42 ffff8801fb703fd8 0000000100000001[ 2314.772579] ffff88024d80dfd8[ 2314.772799] Call Trace: 2011-09-08 14:07:42 [ 2314.773107] [] sys_exit_group+0x12/0x16 2011-09-08 14:07:42 0000000000000082 ffff880746d86340[ 2314.774842] [] do_sys_open+0x114/0x1a3 2011-09-08 14:07:42 [ 2314.775997] [] ? sch_direct_xmit+0x96/0x197 2011-09-08 14:07:42 [ 2314.775997] [] ? sch_direct_xmit+0x96/0x197 2011-09-08 14:07:42 [ 2314.781070] [] ? cpumask_any_but+0x2b/0x38 2011-09-08 14:07:42 [ 2314.781070] [] ? cpumask_any_but+0x2b/0x38 2011-09-08 14:07:42 [ 2314.781133] [] ? flush_tlb_page+0x4f/0x6e 2011-09-08 14:07:42 [ 2314.781133] [] ? flush_tlb_page+0x4f/0x6e 2011-09-08 14:07:42 [ 2314.781758] qpsmtpd-prefork S ffff880695f48010 ffff880695f49fd8 0000000000010640[ 2314.786023] [] mutex_lock+0x16/0x27 2011-09-08 14:07:42 [ 2314.786335] [] do_filp_open+0x38/0x88 2011-09-08 14:07:42 [ 2314.788064] [] do_truncate+0x59/0x80 2011-09-08 14:07:42 ffffea0017112640 ffff88082e627320[ 2314.791089] [] ? nameidata_to_filp+0x53/0x61 2011-09-08 14:07:42 ffffea0017160fa0 0000000000000000 0 1749 4737 0x00000000 2011-09-08 14:07:42 [ 2314.795372] [] ? __hrtimer_start_range_ns+0x31f/0x331 2011-09-08 14:07:42 [ 2314.795758] [] poll_schedule_timeout+0x46/0x69 2011-09-08 14:07:42 [ 2314.795758] [] poll_schedule_timeout+0x46/0x69 2011-09-08 14:07:42 [ 2314.800829] [] do_sys_poll+0x36f/0x410 2011-09-08 14:07:42 [ 2314.800829] [] do_sys_poll+0x36f/0x410 2011-09-08 14:07:42 [ 2314.800891] [] ? poll_freewait+0x8d/0x8d 2011-09-08 14:07:42 [ 2314.800891] [] ? poll_freewait+0x8d/0x8d 2011-09-08 14:07:42 [ 2314.801078] [] ? tcp_sendmsg+0x9bc/0xadf 2011-09-08 14:07:42 [ 2314.801581] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:42 0000000100000001 0000000000010640[ 2314.802498] [] ? do_notify_parent+0x1ca/0x1df 2011-09-08 14:07:42 2011-09-08 14:07:42 ffff8803fde0c010[ 2314.806086] [] ? tcp_rcv_established+0x6b5/0x8aa 2011-09-08 14:07:42 0000000000000086[ 2314.808201] [] ? vfsmount_lock_global_unlock_online+0x40/0x4f 2011-09-08 14:07:42 ffffffff81210539[ 2314.810147] imap S 0000000000000086 0000000000000086[ 2314.811439] [] ? update_rmtp+0x60/0x60 2011-09-08 14:07:42 [ 2314.815104] [] vfs_create+0x72/0xa4 2011-09-08 14:07:42 [ 2314.815104] [] vfs_create+0x72/0xa4 2011-09-08 14:07:42 [ 2314.820181] [] do_last+0x304/0x651 2011-09-08 14:07:42 [ 2314.820181] [] do_last+0x304/0x651 2011-09-08 14:07:42 [ 2314.820243] [] path_openat+0xc5/0x336 2011-09-08 14:07:42 [ 2314.820243] [] path_openat+0xc5/0x336 2011-09-08 14:07:42 ffff88042e52049c[ 2314.821776] [] unix_wait_for_peer+0x97/0xb4 2011-09-08 14:07:42 [ 2314.822854] ffff8801f3189fd8 ffff88021eccda08 0000000000000082[ 2314.826027] [] inet_recvmsg+0x6b/0x80 2011-09-08 14:07:42 ffffffff81050705 0 4773 4736 0x00020000 2011-09-08 14:07:42 0000000000010640 ffff8803ae773848 [ 2314.830908] [] ? hrtimer_start_range_ns+0xf/0x11 2011-09-08 14:07:42 [ 2314.831162] [] ? poll_freewait+0x8d/0x8d 2011-09-08 14:07:42 [ 2314.831478] [] ? __alloc_skb+0x72/0x132 2011-09-08 14:07:42 ffff8803ae79aa10[ 2314.833712] [] ? mntput_no_expire+0x96/0x155 2011-09-08 14:07:42 [ 2314.834931] Call Trace: 2011-09-08 14:07:42 [ 2314.834931] Call Trace: 2011-09-08 14:07:42 [ 2314.839996] [] ? lru_cache_add_lru+0x40/0x42 2011-09-08 14:07:42 [ 2314.839996] [] ? lru_cache_add_lru+0x40/0x42 2011-09-08 14:07:42 [ 2314.840059] [] ? page_add_new_anon_rmap+0x8f/0xa1 2011-09-08 14:07:42 [ 2314.840059] [] ? page_add_new_anon_rmap+0x8f/0xa1 2011-09-08 14:07:42 [ 2314.840499] [] sys32_waitpid+0xb/0xd 2011-09-08 14:07:42 0000000000000082 ffff88042f5b6480 0000000000000000 0000000000003d18[ 2314.843143] [] do_wait+0x1b1/0x21f 2011-09-08 14:07:42 [ 2314.843456] [] sys32_waitpid+0xb/0xd 2011-09-08 14:07:42 [ 2314.844620] [] ? __hrtimer_start_range_ns+0x31f/0x331 2011-09-08 14:07:42 [ 2314.845373] [] ? tcp_event_new_data_sent+0x9a/0x9c 2011-09-08 14:07:42 [ 2314.845626] [] ? _raw_spin_unlock_bh+0x10/0x12 2011-09-08 14:07:42 [ 2314.845816] [] compat_core_sys_select+0x178/0x213 2011-09-08 14:07:42 [ 2314.846131] [] compat_sys_select+0x93/0xbb 2011-09-08 14:07:42 [ 2314.847233] [] ? timerqueue_add+0x81/0x9f 2011-09-08 14:07:42 [ 2314.847491] [] ? update_rmtp+0x60/0x60 2011-09-08 14:07:42 [ 2314.848375] [] ? tcp_sendmsg+0x9bc/0xadf 2011-09-08 14:07:42 ffff8802145c0a90[ 2314.850303] [] do_select+0x56b/0x5bb 2011-09-08 14:07:42 [ 2314.851374] [] ? sys_write+0x47/0x6e 2011-09-08 14:07:42 ffff88034ae7b148[ 2314.852659] [] ? lock_timer_base+0x26/0x4a 2011-09-08 14:07:42 [ 2314.853038] [] ? wake_up_bit+0x23/0x23 2011-09-08 14:07:42 ffff8802b1e6dfd8 ffff8802b1e6dfd8 2011-09-08 14:07:42 2011-09-08 14:07:42 [ 2314.858994] ffff8802b1e6dfd8[ 2314.858994] ffff8802b1e6dfd8 0000000000010640 0000000000010640[ 2314.859521] [] __skb_recv_datagram+0x1ff/0x27b 2011-09-08 14:07:42 2011-09-08 14:07:42 2011-09-08 14:07:42 [ 2314.861688] [] sys_read+0x47/0x6f 2011-09-08 14:07:42 ffff88020df98000 ffff88020df98000 0000000000000000[ 2314.864586] [] sys_wait4+0xa6/0xc3 2011-09-08 14:07:42 2011-09-08 14:07:42 [ 2314.866319] [] sys32_waitpid+0xb/0xd 2011-09-08 14:07:42 [ 2314.866443] qmail-popup S[ 2314.867296] [] ? lru_cache_add_lru+0x40/0x42 2011-09-08 14:07:42 ffff88022e723fd8[ 2314.869027] [] sys_wait4+0xa6/0xc3 2011-09-08 14:07:42 [ 2314.869330] [] ? fd_install+0x30/0x60 2011-09-08 14:07:42 [ 2314.870432] [] ? page_add_new_anon_rmap+0x8f/0xa1 2011-09-08 14:07:42 [ 2314.871569] ffff8801f338dfd8 0000000000010640[ 2314.873301] [] compat_sys_select+0x93/0xbb 2011-09-08 14:07:42 [ 2314.873301] [] compat_sys_select+0x93/0xbb 2011-09-08 14:07:42 [ 2314.878378] [] ? sys_write+0x47/0x6e 2011-09-08 14:07:42 [ 2314.878440] [] ia32_do_call+0x13/0x13 2011-09-08 14:07:42 0000000000000001 ffff880221ab5908[ 2314.880246] [] ? xfs_bmap_search_extents+0x56/0xb9 2011-09-08 14:07:42 [ 2314.880502] [] ? xfs_iunlock+0x5d/0xba 2011-09-08 14:07:42 [ 2314.881784] [] ? rb_insert_color+0x69/0xe6 2011-09-08 14:07:42 [ 2314.882991] [] ? do_sync_write+0xc7/0x10d 2011-09-08 14:07:42 [ 2314.885926] [] ? sys_write+0x47/0x6e 2011-09-08 14:07:42 ffff88020146a010 2011-09-08 14:07:42 [ 2314.887592] [] do_select+0x56b/0x5bb 2011-09-08 14:07:42 [ 2314.888536] [] compat_sys_select+0x93/0xbb 2011-09-08 14:07:42 0000000000000082[ 2314.889895] [] ? update_rmtp+0x60/0x60 2011-09-08 14:07:42 [ 2314.891318] [] sys_semtimedop+0x6c1/0x845 2011-09-08 14:07:42 ffff8801e6f84010 ffff8801e6f84010 2011-09-08 14:07:42 2011-09-08 14:07:42 [ 2314.897269] 0000000000004000 0000000000010640[ 2314.898206] [] sys_accept4+0x120/0x1f9 2011-09-08 14:07:42 ffff8801e4de3ca8 0000000000000082[ 2314.901217] [] ? __hrtimer_start_range_ns+0x31f/0x331 2011-09-08 14:07:42 [ 2314.901410] [] ? hrtimer_start_range_ns+0xf/0x11 2011-09-08 14:07:42 [ 2314.902425] [] ? fsnotify+0x255/0x27b 2011-09-08 14:07:42 [ 2314.903650] [] ? lru_cache_add_lru+0x40/0x42 2011-09-08 14:07:42 ffff880417d7b5a0 2011-09-08 14:07:42 [ 2314.905692] [] schedule_hrtimeout_range+0xe/0x10 2011-09-08 14:07:42 [ 2314.905943] [] ? __pollwait+0xcb/0xcb 2011-09-08 14:07:42 0000000000000082 ffff880279371fd8[ 2314.908505] [] ? poll_freewait+0x8d/0x8d 2011-09-08 14:07:42 [ 2314.908754] [] ? __pollwait+0xcb/0xcb 2011-09-08 14:07:42 [ 2314.909067] [] ? poll_select_set_timeout+0x5c/0x77 2011-09-08 14:07:42 0000000000000046 ffff88042f65eeb8 [ 2314.911453] [] ? lru_cache_add_lru+0x40/0x42 2011-09-08 14:07:42 [ 2314.911453] [] ? lru_cache_add_lru+0x40/0x42 2011-09-08 14:07:42 [ 2314.916523] [] ? page_add_new_anon_rmap+0x8f/0xa1 2011-09-08 14:07:42 [ 2314.916523] [] ? page_add_new_anon_rmap+0x8f/0xa1 2011-09-08 14:07:42 [ 2314.916588] [] ? do_wp_page+0x6ba/0x6d2 2011-09-08 14:07:42 [ 2314.916588] [] ? do_wp_page+0x6ba/0x6d2 2011-09-08 14:07:42 [ 2314.916650] [] do_wait+0x1b1/0x21f 2011-09-08 14:07:42 [ 2314.916650] [] do_wait+0x1b1/0x21f 2011-09-08 14:07:42 [ 2314.916711] [] sys_wait4+0xa6/0xc3 2011-09-08 14:07:42 [ 2314.916711] [] sys_wait4+0xa6/0xc3 2011-09-08 14:07:42 [ 2314.917086] qmail-pop3d S[ 2314.919644] php5 D ffff88040c51d2b8 [ 2314.920000] 0000000000004000 ffff8801d452fbb8[ 2314.921190] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:42 [ 2314.922169] [] do_wait+0x1b1/0x21f 2011-09-08 14:07:42 ffff880691a811b8 2011-09-08 14:07:42 0000000000000082 0000000000010640 ffff88082e1be6e8 [ 2314.926407] [] ? wait_noreap_copyout+0xed/0xed 2011-09-08 14:07:42 [ 2314.927387] [] ? xfs_get_blocks_direct+0x11/0x11 2011-09-08 14:07:42 [ 2314.927709] [] ? do_lookup+0x55/0x2ba 2011-09-08 14:07:42 [ 2314.930471] Call Trace: 2011-09-08 14:07:42 [ 2314.930843] [] ? loadavg_proc_show+0xb3/0xc7 2011-09-08 14:07:42 [ 2314.931157] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:42 [ 2314.931157] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:42 [ 2314.936234] imap S[ 2314.936234] imap S ffff88081c406828 ffff88081c406828 ffff88081c4062c0[ 2314.937725] [] ? sys_write+0x47/0x6e 2011-09-08 14:07:42 [ 2314.938828] [] ? enqueue_hrtimer+0x48/0x74 2011-09-08 14:07:42 ffff88082e0d8ef8 ffff88016b93201d 2011-09-08 14:07:42 ffff880690825bf8 ffffffff8147ab8a ffff880690051bf8 2011-09-08 14:07:42 ffff880690091808 ffff880690093fd8 2011-09-08 14:07:42 ffffffff8147ab8a 0000000000000086 2011-09-08 14:07:42 [ 2314.955994] 0000000000004000 2011-09-08 14:07:42 [ 2314.955994] 0000000000004000 0000000000010640[ 2314.956491] [] ? lru_cache_add_lru+0x40/0x42 2011-09-08 14:07:42 ffff88042f5ef3a0[ 2314.958660] [] ? poll_freewait+0x8d/0x8d 2011-09-08 14:07:42 ffff8803c81d8ff8 [ 2314.960645] [] ? iov_iter_copy_from_user_atomic+0x83/0x107 2011-09-08 14:07:42 0000000000010640 2011-09-08 14:07:42 [ 2314.963206] ffff880690a89fd8[ 2314.964235] [] ? sys_write+0x47/0x6e 2011-09-08 14:07:42 [ 2314.965405] [] ? lock_timer_base+0x26/0x4a 2011-09-08 14:07:42 [ 2314.967522] [] schedule_hrtimeout_range+0xe/0x10 2011-09-08 14:07:42 ffff8801e6e9e010 2011-09-08 14:07:42 [ 2314.969807] qmail-popup S ffff8802219b6010[ 2314.970603] Call Trace: 2011-09-08 14:07:42 [ 2314.970603] Call Trace: 2011-09-08 14:07:42 [ 2314.975667] [] ? lru_cache_add_lru+0x40/0x42 2011-09-08 14:07:42 [ 2314.975667] [] ? lru_cache_add_lru+0x40/0x42 2011-09-08 14:07:42 [ 2314.975731] [] ? page_add_new_anon_rmap+0x8f/0xa1 2011-09-08 14:07:42 [ 2314.975731] [] ? page_add_new_anon_rmap+0x8f/0xa1 2011-09-08 14:07:42 [ 2314.975795] [] ? do_wp_page+0x6ba/0x6d2 2011-09-08 14:07:42 [ 2314.975795] [] ? do_wp_page+0x6ba/0x6d2 2011-09-08 14:07:42 [ 2314.975981] [] ? wait_noreap_copyout+0xed/0xed 2011-09-08 14:07:42 [ 2314.977090] Call Trace: 2011-09-08 14:07:42 [ 2314.978289] [] ? tcp_sendmsg+0x9bc/0xadf 2011-09-08 14:07:42 ffff8803ffc00280 0000000000000246[ 2314.980819] 0000000000004000[ 2314.981157] ffff8802160d7fd8 ffff8803d8a93fd8 ffff8803d8a93fd8 ffff8801e6e6dd38 0000000000000086 2011-09-08 14:07:42 [ 2314.985002] Call Trace: 2011-09-08 14:07:42 ffff880690188010[ 2314.987296] [] ? update_rmtp+0x60/0x60 2011-09-08 14:07:42 0000000000010640[ 2314.988530] [] ? xfs_vm_write_end+0x24/0x46 2011-09-08 14:07:42 [ 2314.988974] [] xfs_release+0x1c7/0x202 2011-09-08 14:07:42 [ 2314.989285] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:42 [ 2314.990329] [] ? do_lookup+0x114/0x2ba 2011-09-08 14:07:42 [ 2314.990329] [] ? do_lookup+0x114/0x2ba 2011-09-08 14:07:42 [ 2314.995408] [] __mutex_lock_slowpath+0xd4/0x119 2011-09-08 14:07:42 [ 2314.995408] [] __mutex_lock_slowpath+0xd4/0x119 2011-09-08 14:07:42 [ 2314.995472] [] ? link_path_walk+0xa8/0x6f7 2011-09-08 14:07:42 [ 2314.995472] [] ? link_path_walk+0xa8/0x6f7 2011-09-08 14:07:42 [ 2314.995721] [] do_filp_open+0x38/0x88 2011-09-08 14:07:42 ffff880695f2af78 [ 2314.997206] [] ? hrtimer_start_range_ns+0xf/0x11 2011-09-08 14:07:42 [ 2314.997523] imap S ffff8806901e8010[ 2314.998809] [] ? update_rmtp+0x60/0x60 2011-09-08 14:07:42 0 9254 5879 0x00000000 2011-09-08 14:07:42 [ 2314.999606] 0000000000004000[ 2315.000613] [] sys_epoll_wait+0x22a/0x387 2011-09-08 14:07:42 2011-09-08 14:07:42 [ 2315.003839] [] ? wake_up_bit+0x23/0x23 2011-09-08 14:07:42 [ 2315.004966] ffff8801c3a97fd8[ 2315.005243] [] ? pollwake+0x57/0x59 2011-09-08 14:07:42 [ 2315.005493] [] __skb_recv_datagram+0x1ff/0x27b 2011-09-08 14:07:42 [ 2315.006115] php5 S[ 2315.007838] php5 S 0000000000010640 ffff88023cd19bc8[ 2315.009315] [] ? mntput+0x21/0x23 2011-09-08 14:07:42 0 10262 7100 0x00000000 2011-09-08 14:07:42 2011-09-08 14:07:42 [ 2315.014923] 0000000000004000 2011-09-08 14:07:42 0000000000010640 0000000000000292[ 2315.016226] [] vfs_read+0xbe/0x14e 2011-09-08 14:07:42 [ 2315.016411] php5 S[ 2315.017704] [] inet_recvmsg+0x6b/0x80 2011-09-08 14:07:42 ffff88068e80fdd8 ffff88068e80fdd8[ 2315.020112] 0000000000004000 ffff8801a6224ff0 ffff8802701a5fd8[ 2315.022776] [] do_select+0x56b/0x5bb 2011-09-08 14:07:42 [ 2315.024824] [] ? rb_insert_color+0x69/0xe6 2011-09-08 14:07:42 0000000000010640[ 2315.027009] [] worker_thread+0x1dc/0x35b 2011-09-08 14:07:42 ffffea000522b380[ 2315.029176] [] sys_open+0x1b/0x1d 2011-09-08 14:07:42 [ 2315.029176] [] sys_open+0x1b/0x1d 2011-09-08 14:07:42 [ 2315.034248] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:42 [ 2315.034248] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:42 [ 2315.034311] php5 S[ 2315.034311] php5 S ffff88042d3d6fb8 0 10681 7100 0x00000000 2011-09-08 14:07:42 ffff88042d3d6fb8 [ 2315.034446] ffff8801980adb38[ 2315.035389] [] ? lock_timer_base+0x26/0x4a 2011-09-08 14:07:42 [ 2315.035705] [] ? __wake_up_sync_key+0x4e/0x5a 2011-09-08 14:07:42 0000000000010640[ 2315.037188] [] schedule_timeout+0x22b/0x269 2011-09-08 14:07:42 [ 2315.037689] [] ? xfs_free_eofblocks+0xac/0x1fb 2011-09-08 14:07:42 [ 2315.038736] Call Trace: 2011-09-08 14:07:42 [ 2315.039779] php5 S ffff8802fc695b68 2011-09-08 14:07:42 [ 2315.040576] Call Trace: 2011-09-08 14:07:42 ffff8802571c6010[ 2315.042864] [] ? user_path_at+0x66/0x95 2011-09-08 14:07:42 [ 2315.044034] Call Trace: 2011-09-08 14:07:42 0 11794 5879 0x00000000 2011-09-08 14:07:42 [ 2315.048112] [] ? hrtimer_start_range_ns+0xf/0x11 2011-09-08 14:07:42 ffff88068e3c0010 ffff88068e3c0010 2011-09-08 14:07:42 2011-09-08 14:07:42 [ 2315.053860] 0000000000004000[ 2315.053860] 0000000000004000 ffff8806c2f45520 ffff8806c2f45520 0000000000010640 0 11884 4733 0x00000000 2011-09-08 14:07:42 0000000000010640 ffff8802f9b048a8 [ 2315.058023] [] ? xfs_vm_write_end+0x24/0x46 2011-09-08 14:07:42 [ 2315.058274] [] do_truncate+0x59/0x80 2011-09-08 14:07:42 [ 2315.058587] [] do_sys_open+0x114/0x1a3 2011-09-08 14:07:42 [ 2315.059627] [] ? do_sync_write+0xc7/0x10d 2011-09-08 14:07:42 [ 2315.060736] Call Trace: 2011-09-08 14:07:42 ffff88042f6d5090[ 2315.062855] [] ? hrtimer_start_range_ns+0xf/0x11 2011-09-08 14:07:42 0000000100000001 0000000000000082[ 2315.068303] [] ? tcp_rcv_established+0xe3/0x8aa 2011-09-08 14:07:41 [ 2313.687909] SysRq : Show State 2011-09-08 14:07:41 [ 2313.687909] SysRq : Show State 2011-09-08 14:07:41 [ 2313.688015] task PC stack pid father 2011-09-08 14:07:41 [ 2313.688015] task PC stack pid father 2011-09-08 14:07:41 [ 2313.688078] init S[ 2313.688078] init S ffff88042fbeb7c8 ffff88042fbeb7c8 0 1 0 0x00000000 2011-09-08 14:07:41 0 1 0 0x00000000 2011-09-08 14:07:41 [ 2313.688219] ffff88042fbed8e8[ 2313.688219] ffff88042fbed8e8 0000000000000086 0000000000000086 0000000000000000 0000000000000000 ffff88042fbec010 ffff88042fbec010 2011-09-08 14:07:41 2011-09-08 14:07:41 [ 2313.688440] 0000000000004000 ffff88042fbeb260[ 2313.688440] 0000000000004000 ffff88042fbeb260 0000000000010640 0000000000010640 ffff88042fbedfd8 ffff88042fbedfd8 2011-09-08 14:07:41 2011-09-08 14:07:41 [ 2313.688663] ffff88042fbedfd8[ 2313.688663] ffff88042fbedfd8 0000000000010640 0000000000010640 ffffffff81210539 ffffffff81210539 ffff88042fbed8f8 ffff88042fbed8f8 2011-09-08 14:07:41 2011-09-08 14:07:41 [ 2313.688886] Call Trace: 2011-09-08 14:07:41 [ 2313.688886] Call Trace: 2011-09-08 14:07:41 [ 2313.688951] [] ? rb_insert_color+0x69/0xe6 2011-09-08 14:07:41 [ 2313.688951] [] ? rb_insert_color+0x69/0xe6 2011-09-08 14:07:41 [ 2313.689015] [] ? timerqueue_add+0x81/0x9f 2011-09-08 14:07:41 [ 2313.689015] [] ? timerqueue_add+0x81/0x9f 2011-09-08 14:07:41 [ 2313.689082] [] ? enqueue_hrtimer+0x48/0x74 2011-09-08 14:07:41 [ 2313.689082] [] ? enqueue_hrtimer+0x48/0x74 2011-09-08 14:07:41 [ 2313.689146] [] ? __hrtimer_start_range_ns+0x31f/0x331 2011-09-08 14:07:41 [ 2313.689146] [] ? __hrtimer_start_range_ns+0x31f/0x331 2011-09-08 14:07:41 [ 2313.689215] [] schedule_hrtimeout_range_clock+0xda/0x123 2011-09-08 14:07:41 [ 2313.689215] [] schedule_hrtimeout_range_clock+0xda/0x123 2011-09-08 14:07:41 [ 2313.689282] [] ? update_rmtp+0x60/0x60 2011-09-08 14:07:41 [ 2313.689282] [] ? update_rmtp+0x60/0x60 2011-09-08 14:07:41 [ 2313.689346] [] ? hrtimer_start_range_ns+0xf/0x11 2011-09-08 14:07:41 [ 2313.689346] [] ? hrtimer_start_range_ns+0xf/0x11 2011-09-08 14:07:41 [ 2313.689410] [] schedule_hrtimeout_range+0xe/0x10 2011-09-08 14:07:41 [ 2313.689410] [] schedule_hrtimeout_range+0xe/0x10 2011-09-08 14:07:41 [ 2313.689477] [] poll_schedule_timeout+0x46/0x69 2011-09-08 14:07:41 [ 2313.689477] [] poll_schedule_timeout+0x46/0x69 2011-09-08 14:07:41 [ 2313.689543] [] ? pipe_poll+0x2e/0x88 2011-09-08 14:07:41 [ 2313.689543] [] ? pipe_poll+0x2e/0x88 2011-09-08 14:07:41 [ 2313.689605] [] do_select+0x56b/0x5bb 2011-09-08 14:07:41 [ 2313.689605] [] do_select+0x56b/0x5bb 2011-09-08 14:07:41 [ 2313.689667] [] ? poll_freewait+0x8d/0x8d 2011-09-08 14:07:41 [ 2313.689667] [] ? poll_freewait+0x8d/0x8d 2011-09-08 14:07:41 [ 2313.689730] [] ? __pollwait+0xcb/0xcb 2011-09-08 14:07:41 [ 2313.689730] [] ? __pollwait+0xcb/0xcb 2011-09-08 14:07:41 [ 2313.689792] [] ? number+0x14e/0x26d 2011-09-08 14:07:41 [ 2313.689792] [] ? number+0x14e/0x26d 2011-09-08 14:07:41 [ 2313.689857] [] ? call_rcu_sched+0x10/0x12 2011-09-08 14:07:41 [ 2313.689857] [] ? call_rcu_sched+0x10/0x12 2011-09-08 14:07:41 [ 2313.689923] [] ? proc_destroy_inode+0x17/0x19 2011-09-08 14:07:41 [ 2313.689923] [] ? proc_destroy_inode+0x17/0x19 2011-09-08 14:07:41 [ 2313.689988] [] ? destroy_inode+0x35/0x4e 2011-09-08 14:07:41 [ 2313.689988] [] ? destroy_inode+0x35/0x4e 2011-09-08 14:07:41 [ 2313.690051] [] ? do_lookup+0x55/0x2ba 2011-09-08 14:07:41 [ 2313.690051] [] ? do_lookup+0x55/0x2ba 2011-09-08 14:07:41 [ 2313.690114] [] ? do_lookup+0x55/0x2ba 2011-09-08 14:07:41 [ 2313.690114] [] ? do_lookup+0x55/0x2ba 2011-09-08 14:07:41 [ 2313.690178] [] ? complete_walk+0x8e/0xdd 2011-09-08 14:07:41 [ 2313.690178] [] ? complete_walk+0x8e/0xdd 2011-09-08 14:07:41 [ 2313.690241] [] ? path_lookupat+0x514/0x584 2011-09-08 14:07:41 [ 2313.690241] [] ? path_lookupat+0x514/0x584 2011-09-08 14:07:41 [ 2313.690304] [] core_sys_select+0x1a3/0x253 2011-09-08 14:07:41 [ 2313.690304] [] core_sys_select+0x1a3/0x253 2011-09-08 14:07:41 [ 2313.690368] [] ? cp_new_stat+0xe2/0xee 2011-09-08 14:07:41 [ 2313.690368] [] ? cp_new_stat+0xe2/0xee 2011-09-08 14:07:41 [ 2313.690433] [] ? ktime_get_ts+0xad/0xba 2011-09-08 14:07:41 [ 2313.690433] [] ? ktime_get_ts+0xad/0xba 2011-09-08 14:07:41 [ 2313.690496] [] sys_select+0x94/0xbd 2011-09-08 14:07:41 [ 2313.690496] [] sys_select+0x94/0xbd 2011-09-08 14:07:41 [ 2313.690559] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:41 [ 2313.690622] kthreadd S[ 2313.690559] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:41 [ 2313.690622] kthreadd S ffff88042fbeaeb8 0 2 0 0x00000000 2011-09-08 14:07:41 ffff88042fbeaeb8 0 2 0 0x00000000 2011-09-08 14:07:41 [ 2313.690760] ffff88042fbefef0[ 2313.690760] ffff88042fbefef0 0000000000000046 0000000000000000 0000000000000046 0000000000000000 ffff88042fbee010 2011-09-08 14:07:41 ffff88042fbee010 2011-09-08 14:07:41 [ 2313.690981] 0000000000004000[ 2313.690981] 0000000000004000 ffff88042fbea950 ffff88042fbea950 0000000000010640 0000000000010640 ffff88042fbeffd8 ffff88042fbeffd8 2011-09-08 14:07:41 2011-09-08 14:07:41 [ 2313.691202] ffff88042fbeffd8[ 2313.691202] ffff88042fbeffd8 0000000000010640 0000000000010640 0000000000000000 0000000000000000 0000000000000000 0000000000000000 2011-09-08 14:07:41 2011-09-08 14:07:41 [ 2313.691423] Call Trace: 2011-09-08 14:07:41 [ 2313.691423] Call Trace: 2011-09-08 14:07:41 [ 2313.691483] [] ? kernel_thread+0x6b/0x6d 2011-09-08 14:07:41 [ 2313.691483] [] ? kernel_thread+0x6b/0x6d 2011-09-08 14:07:41 [ 2313.691547] [] ? kthread_worker_fn+0x117/0x117 2011-09-08 14:07:41 [ 2313.691613] [] ? gs_change+0xb/0xb 2011-09-08 14:07:41 [ 2313.691675] [] kthreadd+0x7a/0x103 2011-09-08 14:07:41 [ 2313.691547] [] ? kthread_worker_fn+0x117/0x117 2011-09-08 14:07:41 [ 2313.691613] [] ? gs_change+0xb/0xb 2011-09-08 14:07:41 [ 2313.691675] [] kthreadd+0x7a/0x103 2011-09-08 14:07:41 [ 2313.691737] [] kernel_thread_helper+0x4/0x10 2011-09-08 14:07:41 [ 2313.691737] [] kernel_thread_helper+0x4/0x10 2011-09-08 14:07:41 [ 2313.691800] [] ? kthread+0x89/0x89 2011-09-08 14:07:41 [ 2313.691800] [] ? kthread+0x89/0x89 2011-09-08 14:07:41 [ 2313.691862] [] ? gs_change+0xb/0xb 2011-09-08 14:07:41 [ 2313.696931] ksoftirqd/0 S[ 2313.691862] [] ? gs_change+0xb/0xb 2011-09-08 14:07:41 [ 2313.696931] ksoftirqd/0 S ffff88042fbea5a8 ffff88042fbea5a8 0 3 2 0x00000000 2011-09-08 14:07:41 0 3 2 0x00000000 2011-09-08 14:07:41 [ 2313.697068] ffff88042fbf3ea0[ 2313.697068] ffff88042fbf3ea0 0000000000000046 0000000000000046 0000000000000100 0000000000000100 ffff88042fbf2010 ffff88042fbf2010 2011-09-08 14:07:41 2011-09-08 14:07:41 [ 2313.697290] 0000000000004000[ 2313.697290] 0000000000004000 ffff88042fbea040 ffff88042fbea040 0000000000010640 0000000000010640 ffff88042fbf3fd8 ffff88042fbf3fd8 2011-09-08 14:07:41 2011-09-08 14:07:41 [ 2313.697512] ffff88042fbf3fd8[ 2313.697512] ffff88042fbf3fd8 0000000000010640 0000000000010640 ffff88042fbf3fd8 ffff88042fbf3fd8 ffff88015ceb1ce0 ffff88015ceb1ce0 2011-09-08 14:07:41 2011-09-08 14:07:41 [ 2313.697734] Call Trace: 2011-09-08 14:07:41 [ 2313.697734] Call Trace: 2011-09-08 14:07:41 [ 2313.697794] [] ? __do_softirq+0x184/0x193 2011-09-08 14:07:41 [ 2313.697794] [] ? __do_softirq+0x184/0x193 2011-09-08 14:07:41 [ 2313.697857] [] run_ksoftirqd+0x52/0x11d 2011-09-08 14:07:41 [ 2313.697857] [] run_ksoftirqd+0x52/0x11d 2011-09-08 14:07:41 [ 2313.697920] [] ? __do_softirq+0x193/0x193 2011-09-08 14:07:41 [ 2313.697920] [] ? __do_softirq+0x193/0x193 2011-09-08 14:07:41 [ 2313.697983] [] kthread+0x81/0x89 2011-09-08 14:07:41 [ 2313.697983] [] kthread+0x81/0x89 2011-09-08 14:07:41 [ 2313.698045] [] kernel_thread_helper+0x4/0x10 2011-09-08 14:07:41 [ 2313.698045] [] kernel_thread_helper+0x4/0x10 2011-09-08 14:07:41 [ 2313.698108] [] ? kthread_worker_fn+0x117/0x117 2011-09-08 14:07:41 [ 2313.698108] [] ? kthread_worker_fn+0x117/0x117 2011-09-08 14:07:41 [ 2313.698172] [] ? gs_change+0xb/0xb 2011-09-08 14:07:41 [ 2313.698233] kworker/u:0 S[ 2313.698172] [] ? gs_change+0xb/0xb 2011-09-08 14:07:41 [ 2313.698233] kworker/u:0 S ffff88042fbf4ef8 ffff88042fbf4ef8 0 5 2 0x00000000 2011-09-08 14:07:41 0 5 2 0x00000000 2011-09-08 14:07:41 [ 2313.698369] ffff88042fbf9e60[ 2313.698369] ffff88042fbf9e60 0000000000000046 0000000000000046 ffffffff81d87f00 ffffffff81d87f00 ffff88042fbf8010 2011-09-08 14:07:41 ffff88042fbf8010 2011-09-08 14:07:41 [ 2313.698590] 0000000000004000[ 2313.698590] 0000000000004000 ffff88042fbf4990 0000000000010640 ffff88042fbf4990 0000000000010640 ffff88042fbf9fd8 ffff88042fbf9fd8 2011-09-08 14:07:41 2011-09-08 14:07:41 [ 2313.698812] ffff88042fbf9fd8[ 2313.698812] ffff88042fbf9fd8 0000000000010640 0000000000010640 0000000000000000 0000000000000000 0000000000000000 0000000000000000 2011-09-08 14:07:41 2011-09-08 14:07:41 [ 2313.699032] Call Trace: 2011-09-08 14:07:41 [ 2313.699032] Call Trace: 2011-09-08 14:07:41 [ 2313.699091] [] ? ____call_usermodehelper+0xff/0xff 2011-09-08 14:07:41 [ 2313.699091] [] ? ____call_usermodehelper+0xff/0xff 2011-09-08 14:07:41 [ 2313.699155] [] ? gs_change+0xb/0xb 2011-09-08 14:07:41 [ 2313.699155] [] ? gs_change+0xb/0xb 2011-09-08 14:07:41 [ 2313.699218] [] ? process_one_work+0x380/0x38f 2011-09-08 14:07:41 [ 2313.699218] [] ? process_one_work+0x380/0x38f 2011-09-08 14:07:41 [ 2313.699282] [] ? call_usermodehelper_exec+0xc4/0xc4 2011-09-08 14:07:41 [ 2313.699282] [] ? call_usermodehelper_exec+0xc4/0xc4 2011-09-08 14:07:41 [ 2313.699348] [] worker_thread+0x356/0x35b 2011-09-08 14:07:41 [ 2313.699348] [] worker_thread+0x356/0x35b 2011-09-08 14:07:41 [ 2313.699411] [] ? manage_workers+0x186/0x186 2011-09-08 14:07:41 [ 2313.699411] [] ? manage_workers+0x186/0x186 2011-09-08 14:07:41 [ 2313.699474] [] kthread+0x81/0x89 2011-09-08 14:07:41 [ 2313.699474] [] kthread+0x81/0x89 2011-09-08 14:07:41 [ 2313.699535] [] kernel_thread_helper+0x4/0x10 2011-09-08 14:07:41 [ 2313.699535] [] kernel_thread_helper+0x4/0x10 2011-09-08 14:07:41 [ 2313.699599] [] ? kthread_worker_fn+0x117/0x117 2011-09-08 14:07:41 [ 2313.699599] [] ? kthread_worker_fn+0x117/0x117 2011-09-08 14:07:41 [ 2313.699663] [] ? gs_change+0xb/0xb 2011-09-08 14:07:41 [ 2313.699724] migration/0 S[ 2313.699663] [] ? gs_change+0xb/0xb 2011-09-08 14:07:41 [ 2313.699724] migration/0 S ffff88042fbf45e8 ffff88042fbf45e8 0 6 2 0x00000000 2011-09-08 14:07:41 0 6 2 0x00000000 2011-09-08 14:07:41 [ 2313.699860] ffff88042fbfdde0[ 2313.699860] ffff88042fbfdde0 0000000000000046 0000000000000046 ffff88043fd106b0 ffff88043fd106b0 ffff88042fbfc010 ffff88042fbfc010 2011-09-08 14:07:41 2011-09-08 14:07:41 [ 2313.700080] 0000000000004000[ 2313.700080] 0000000000004000 ffff88042fbf4080 ffff88042fbf4080 0000000000010640 0000000000010640 ffff88042fbfdfd8 ffff88042fbfdfd8 2011-09-08 14:07:41 2011-09-08 14:07:41 [ 2313.700303] ffff88042fbfdfd8[ 2313.700303] ffff88042fbfdfd8 0000000000010640 0000000000010640 ffff880171c76c98 ffff880171c76c98 ffff880171c76c50 ffff880171c76c50 2011-09-08 14:07:41 2011-09-08 14:07:41 [ 2313.700525] Call Trace: 2011-09-08 14:07:41 [ 2313.700525] Call Trace: 2011-09-08 14:07:41 [ 2313.700583] [] ? enqueue_task+0x5a/0x62 2011-09-08 14:07:41 [ 2313.700583] [] ? enqueue_task+0x5a/0x62 2011-09-08 14:07:41 [ 2313.700648] [] ? check_preempt_curr+0x3e/0x6c 2011-09-08 14:07:41 [ 2313.700648] [] ? check_preempt_curr+0x3e/0x6c 2011-09-08 14:07:41 [ 2313.700712] [] ? pull_task+0x42/0x4a 2011-09-08 14:07:41 [ 2313.700712] [] ? pull_task+0x42/0x4a 2011-09-08 14:07:41 [ 2313.700775] [] ? active_load_balance_cpu_stop+0x11b/0x16f 2011-09-08 14:07:41 [ 2313.700775] [] ? active_load_balance_cpu_stop+0x11b/0x16f 2011-09-08 14:07:41 [ 2313.700841] [] ? run_rebalance_domains+0x12d/0x12d 2011-09-08 14:07:41 [ 2313.700841] [] ? run_rebalance_domains+0x12d/0x12d 2011-09-08 14:07:41 [ 2313.700906] [] cpu_stopper_thread+0x187/0x1a3 2011-09-08 14:07:41 [ 2313.700906] [] cpu_stopper_thread+0x187/0x1a3 2011-09-08 14:07:41 [ 2313.700970] [] ? default_wake_function+0xd/0xf 2011-09-08 14:07:41 [ 2313.700970] [] ? default_wake_function+0xd/0xf 2011-09-08 14:07:41 [ 2313.701034] [] ? __wake_up_common+0x46/0x75 2011-09-08 14:07:41 [ 2313.701034] [] ? __wake_up_common+0x46/0x75 2011-09-08 14:07:41 [ 2313.701098] [] ? ikconfig_read_current+0x24/0x24 2011-09-08 14:07:41 [ 2313.701098] [] ? ikconfig_read_current+0x24/0x24 2011-09-08 14:07:41 [ 2313.701162] [] ? ikconfig_read_current+0x24/0x24 2011-09-08 14:07:41 [ 2313.701162] [] ? ikconfig_read_current+0x24/0x24 2011-09-08 14:07:41 [ 2313.701226] [] kthread+0x81/0x89 2011-09-08 14:07:41 [ 2313.701226] [] kthread+0x81/0x89 2011-09-08 14:07:41 [ 2313.701287] [] kernel_thread_helper+0x4/0x10 2011-09-08 14:07:41 [ 2313.701287] [] kernel_thread_helper+0x4/0x10 2011-09-08 14:07:41 [ 2313.701351] [] ? kthread_worker_fn+0x117/0x117 2011-09-08 14:07:41 [ 2313.701351] [] ? kthread_worker_fn+0x117/0x117 2011-09-08 14:07:41 [ 2313.701415] [] ? gs_change+0xb/0xb 2011-09-08 14:07:41 [ 2313.701415] [] ? gs_change+0xb/0xb 2011-09-08 14:07:41 [ 2313.701475] migration/1 S[ 2313.701475] migration/1 S ffff88042f41f848 ffff88042f41f848 0 7 2 0x00000000 2011-09-08 14:07:41 [ 2313.701613] ffff88042f421de0 0 7 2 0x00000000 2011-09-08 14:07:41 [ 2313.701613] ffff88042f421de0 0000000000000046 0000000000000046 ffff88043fcd06b0 ffff88043fcd06b0 ffff88042f420010 ffff88042f420010 2011-09-08 14:07:41 2011-09-08 14:07:41 [ 2313.701833] 0000000000004000 ffff88042f421fd8[ 2313.702274] Call Trace: 2011-09-08 14:07:41 [ 2313.702520] [] ? active_load_balance_cpu_stop+0x11b/0x16f 2011-09-08 14:07:41 [ 2313.702842] [] ? ikconfig_read_current+0x24/0x24 2011-09-08 14:07:41 [ 2313.703159] [] ? gs_change+0xb/0xb 2011-09-08 14:07:41 2011-09-08 14:07:41 2011-09-08 14:07:41 [ 2313.704021] Call Trace: 2011-09-08 14:07:41 [ 2313.704272] [] ? manage_workers+0x186/0x186 2011-09-08 14:07:41 [ 2313.705444] [] ? __rcu_process_callbacks+0x7c/0x2ed 2011-09-08 14:07:41 ffff880178d9f660[ 2313.707320] [] ? ikconfig_read_current+0x24/0x24 2011-09-08 14:07:41 [ 2313.707633] kworker/2:0 S ffff88042f43bd40 2011-09-08 14:07:41 [ 2313.708435] Call Trace: 2011-09-08 14:07:41 [ 2313.708688] [] ? process_one_work+0x380/0x38f 2011-09-08 14:07:41 [ 2313.709069] [] ? kthread_worker_fn+0x117/0x117 2011-09-08 14:07:41 [ 2313.710053] [] ? __rcu_process_callbacks+0x7c/0x2ed 2011-09-08 14:07:41 ffff88042f467fd8[ 2313.711799] [] ? default_wake_function+0xd/0xf 2011-09-08 14:07:41 [ 2313.711799] [] ? default_wake_function+0xd/0xf 2011-09-08 14:07:41 [ 2313.716869] [] ? __wake_up_common+0x46/0x75 2011-09-08 14:07:41 [ 2313.716869] [] ? __wake_up_common+0x46/0x75 2011-09-08 14:07:41 [ 2313.716932] [] ? ikconfig_read_current+0x24/0x24 2011-09-08 14:07:41 [ 2313.716932] [] ? ikconfig_read_current+0x24/0x24 2011-09-08 14:07:41 0000000000010640 ffff88082d40ccd0 0000000000010640 ffff88083e2d7fd8[ 2313.724708] [] ? ikconfig_read_current+0x24/0x24 2011-09-08 14:07:41 2011-09-08 14:07:41 2011-09-08 14:07:41 2011-09-08 14:07:41 0000000000010640[ 2313.730857] [] ? manage_workers+0x186/0x186 2011-09-08 14:07:41 ffff88083e367fd8 ffff88083e367fd8 2011-09-08 14:07:41 2011-09-08 14:07:41 [ 2313.736757] ffff88083e367fd8[ 2313.736757] ffff88083e367fd8 0000000000010640 2011-09-08 14:07:41 [ 2313.738844] [] ? __wake_up_common+0x46/0x75 2011-09-08 14:07:41 ffff88083e392010[ 2313.742136] migration/8 S[ 2313.743058] [] ? default_wake_function+0xd/0xf 2011-09-08 14:07:41 [ 2313.744180] 0000000000004000[ 2313.744180] 0000000000004000[ 2313.745061] [] kernel_thread_helper+0x4/0x10 2011-09-08 14:07:41 [ 2313.746108] [] ? __do_softirq+0x184/0x193 2011-09-08 14:07:41 0000000000010640[ 2313.747663] [] cpu_stopper_thread+0x187/0x1a3 2011-09-08 14:07:41 [ 2313.749155] [] ? process_one_work+0x380/0x38f 2011-09-08 14:07:41 2011-09-08 14:07:41 2011-09-08 14:07:41 2011-09-08 14:07:41 [ 2313.756709] Call Trace: 2011-09-08 14:07:41 [ 2313.756709] Call Trace: 2011-09-08 14:07:41 [ 2313.756766] [] ? enqueue_task+0x5a/0x62 2011-09-08 14:07:41 [ 2313.757531] [] ? kthread_worker_fn+0x117/0x117 2011-09-08 14:07:41 [ 2313.757654] kworker/10:0 S[ 2313.758639] [] ? queue_delayed_work_on+0x102/0x116 2011-09-08 14:07:41 [ 2313.761693] [] cpu_stopper_thread+0x187/0x1a3 2011-09-08 14:07:41 ffff88042f501560[ 2313.763631] [] kernel_thread_helper+0x4/0x10 2011-09-08 14:07:41 ffff88083e3946a8 [ 2313.766230] [] cpu_stopper_thread+0x187/0x1a3 2011-09-08 14:07:41 [ 2313.767378] ffff88083e399fd8 0 43 2 0x00000000 2011-09-08 14:07:41 ffff88083e3c0010[ 2313.771273] [] ? gs_change+0xb/0xb 2011-09-08 14:07:41 [ 2313.771273] [] ? gs_change+0xb/0xb 2011-09-08 14:07:41 [ 2313.776338] kworker/13:0 S[ 2313.776338] kworker/13:0 S ffff88083e3beff8 [ 2313.777197] [] ? queue_delayed_work_on+0x102/0x116 2011-09-08 14:07:41 [ 2313.777453] [] ? process_one_work+0x380/0x38f 2011-09-08 14:07:41 2011-09-08 14:07:41 [ 2313.778694] Call Trace: 2011-09-08 14:07:41 [ 2313.779318] migration/14 S ffff88083e3edfd8[ 2313.781055] kworker/14:0 S 0 48 2 0x00000000 2011-09-08 14:07:41 ffff88083e3effd8[ 2313.781850] Call Trace: 2011-09-08 14:07:41 [ 2313.782352] [] kthread+0x81/0x89 2011-09-08 14:07:41 0 49 2 0x00000000 2011-09-08 14:07:41 [ 2313.782955] 0000000000004000 0000000000000282 ffff88083e3f9fd8[ 2313.784748] Call Trace: 2011-09-08 14:07:41 [ 2313.785631] [] kernel_thread_helper+0x4/0x10 2011-09-08 14:07:41 ffff88083e3fbd40 ffff88083e3fbfd8 ffffffff810ab0b9[ 2313.786798] [] ? queue_delayed_work_on+0x102/0x116 2011-09-08 14:07:41 [ 2313.787241] [] ? kthread_worker_fn+0x117/0x117 2011-09-08 14:07:41 [ 2313.787500] ffff88083d81fea0 ffff88083d81ffd8[ 2313.788160] Call Trace: 2011-09-08 14:07:41 [ 2313.789947] [] ? kthread_worker_fn+0x117/0x117 2011-09-08 14:07:41 [ 2313.790987] [] ? gs_change+0xb/0xb 2011-09-08 14:07:41 [ 2313.791482] [] ? kthread_worker_fn+0x117/0x117 2011-09-08 14:07:41 [ 2313.791482] [] ? kthread_worker_fn+0x117/0x117 2011-09-08 14:07:41 [ 2313.796557] [] ? gs_change+0xb/0xb 2011-09-08 14:07:41 [ 2313.796557] [] ? gs_change+0xb/0xb 2011-09-08 14:07:41 [ 2313.796617] netns S[ 2313.796617] netns S ffff88042f59dbc8 ffff88042f59dbc8 0 57 2 0x00000000 2011-09-08 14:07:41 0 57 2 0x00000000 2011-09-08 14:07:41 2011-09-08 14:07:41 [ 2313.798327] 0000000000004000[ 2313.799142] [] bdi_sync_supers+0x3a/0x51 2011-09-08 14:07:41 ffff88042f597ca0[ 2313.801065] [] ? kthread_worker_fn+0x117/0x117 2011-09-08 14:07:41 2011-09-08 14:07:41 [ 2313.801984] Call Trace: 2011-09-08 14:07:41 0000000000000001[ 2313.804017] [] ? gs_change+0xb/0xb 2011-09-08 14:07:41 [ 2313.805122] [] ? try_to_wake_up+0x2de/0x2f0 2011-09-08 14:07:41 [ 2313.807392] 0000000000004000[ 2313.808268] [] kernel_thread_helper+0x4/0x10 2011-09-08 14:07:41 [ 2313.809422] [] ? try_to_del_timer_sync+0xb7/0xc4 2011-09-08 14:07:41 0000000000000000[ 2313.811542] [] kswapd+0x262/0x9ca 2011-09-08 14:07:41 [ 2313.811542] [] kswapd+0x262/0x9ca 2011-09-08 14:07:41 [ 2313.816610] [] ? try_to_wake_up+0x2de/0x2f0 2011-09-08 14:07:41 [ 2313.816610] [] ? try_to_wake_up+0x2de/0x2f0 2011-09-08 14:07:41 [ 2313.816673] [] ? wake_up_bit+0x23/0x23 2011-09-08 14:07:41 [ 2313.816673] [] ? wake_up_bit+0x23/0x23 2011-09-08 14:07:41 [ 2313.816735] [] ? zone_reclaim+0x3fb/0x3fb 2011-09-08 14:07:41 [ 2313.816735] [] ? zone_reclaim+0x3fb/0x3fb 2011-09-08 14:07:41 [ 2313.816798] [] kthread+0x81/0x89 2011-09-08 14:07:41 [ 2313.816798] [] kthread+0x81/0x89 2011-09-08 14:07:41 [ 2313.817047] khugepaged S[ 2313.817182] ffff88042e9c1cb0 0000000000000002[ 2313.818150] [] schedule_timeout_interruptible+0x19/0x1b 2011-09-08 14:07:41 2011-09-08 14:07:41 [ 2313.820130] [] kthread+0x81/0x89 2011-09-08 14:07:41 0 662 2 0x00000000 2011-09-08 14:07:41 ffff88042f59c440 2011-09-08 14:07:41 [ 2313.821295] [] ? enqueue_task+0x5a/0x62 2011-09-08 14:07:41 [ 2313.821672] [] ? kthread_worker_fn+0x117/0x117 2011-09-08 14:07:41 0000000000000046 0000000000010640 0000000000000000 0000000000000046[ 2313.824379] [] kthread+0x81/0x89 2011-09-08 14:07:41 [ 2313.824627] xfsdatad S ffff88042eb12010 0000000000010640[ 2313.825480] [] ? enqueue_task_fair+0x322/0x331 2011-09-08 14:07:41 [ 2313.826033] [] ? kthread_worker_fn+0x117/0x117 2011-09-08 14:07:41 ffff88083fd50640 ffff88042eb15fd8[ 2313.827513] [] ? gs_change+0xb/0xb 2011-09-08 14:07:41 [ 2313.828552] [] ? set_user_nice+0x132/0x159 2011-09-08 14:07:41 2011-09-08 14:07:41 ffff88042e49ffd8[ 2313.831581] [] worker_thread+0x356/0x35b 2011-09-08 14:07:41 [ 2313.831581] [] worker_thread+0x356/0x35b 2011-09-08 14:07:41 [ 2313.836768] [] ? manage_workers+0x186/0x186 2011-09-08 14:07:41 [ 2313.836768] [] ? manage_workers+0x186/0x186 2011-09-08 14:07:41 [ 2313.836831] [] kthread+0x81/0x89 2011-09-08 14:07:41 ffff88042e42bfd8[ 2313.838502] [] ? kthread_worker_fn+0x117/0x117 2011-09-08 14:07:41 [ 2313.839731] [] ? manage_workers+0x186/0x186 2011-09-08 14:07:41 [ 2313.840994] [] ? process_one_work+0x380/0x38f 2011-09-08 14:07:41 2011-09-08 14:07:41 2011-09-08 14:07:41 ffff88083e15a140[ 2313.844394] [] ? kthread_worker_fn+0x117/0x117 2011-09-08 14:07:41 [ 2313.845370] [] ? do_dbs_timer+0x35f/0x36e 2011-09-08 14:07:41 [ 2313.845749] [] ? kthread_worker_fn+0x117/0x117 2011-09-08 14:07:41 [ 2313.847355] kworker/6:1 S 0 1319 2 0x00000000 2011-09-08 14:07:41 2011-09-08 14:07:41 ffff88082e9afd90 0 1320 2 0x00000000 2011-09-08 14:07:41 [ 2313.850562] [] ? kthread_worker_fn+0x117/0x117 2011-09-08 14:07:41 0000000000000000 ffff88042e562440 0000000000000400[ 2313.851665] [] scsi_error_handler+0x9c/0x432 2011-09-08 14:07:41 [ 2313.851792] [] ? scsi_eh_get_sense+0xe0/0xe0 2011-09-08 14:07:41 [ 2313.851792] [] ? scsi_eh_get_sense+0xe0/0xe0 2011-09-08 14:07:41 [ 2313.856862] [] ? scsi_eh_get_sense+0xe0/0xe0 2011-09-08 14:07:41 [ 2313.856862] [] ? scsi_eh_get_sense+0xe0/0xe0 2011-09-08 14:07:41 [ 2313.856925] [] kthread+0x81/0x89 2011-09-08 14:07:41 [ 2313.856925] [] kthread+0x81/0x89 2011-09-08 14:07:41 [ 2313.856986] [] kernel_thread_helper+0x4/0x10 2011-09-08 14:07:41 [ 2313.856986] [] kernel_thread_helper+0x4/0x10 2011-09-08 14:07:41 [ 2313.857050] [] ? kthread_worker_fn+0x117/0x117 2011-09-08 14:07:41 [ 2313.857113] [] ? gs_change+0xb/0xb 2011-09-08 14:07:41 [ 2313.857050] [] ? kthread_worker_fn+0x117/0x117 2011-09-08 14:07:41 [ 2313.858026] [] ? do_dbs_timer+0x35f/0x36e 2011-09-08 14:07:41 0000000000010640 2011-09-08 14:07:41 [ 2313.859570] [] ? try_to_del_timer_sync+0xb7/0xc4 2011-09-08 14:07:41 [ 2313.859886] [] ? aac_fib_send+0x30f/0x30f 2011-09-08 14:07:41 [ 2313.860189] [] kernel_thread_helper+0x4/0x10 2011-09-08 14:07:41 0 1420 2 0x00000000 2011-09-08 14:07:41 ffff88042f600200 0000000000000000[ 2313.861293] [] ? process_one_work+0x380/0x38f 2011-09-08 14:07:41 [ 2313.863340] kworker/15:1 S[ 2313.864886] xfsbufd/sda6 S 00000000000024d0[ 2313.870856] [] ? gs_change+0xb/0xb 2011-09-08 14:07:41 2011-09-08 14:07:41 2011-09-08 14:07:41 [ 2313.876504] ffff88042d755fd8[ 2313.876504] ffff88042d755fd8[ 2313.877474] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:41 [ 2313.878698] [] kmsg_read+0x45/0x50 2011-09-08 14:07:41 [ 2313.879805] Call Trace: 2011-09-08 14:07:41 [ 2313.880987] [] core_sys_select+0x1a3/0x253 2011-09-08 14:07:41 ffff88042d0fbfd8 ffff88042d0fbfd8[ 2313.883038] [] path_openat+0xc5/0x336 2011-09-08 14:07:41 [ 2313.884323] Call Trace: 2011-09-08 14:07:41 [ 2313.885458] ffff88082d473fd8[ 2313.888648] Call Trace: 2011-09-08 14:07:41 [ 2313.889879] 0000000000004000[ 2313.890684] [] futex_wait_queue_me+0xb8/0xd3 2011-09-08 14:07:41 [ 2313.890999] [] ? mnt_get_count+0x46/0x5a 2011-09-08 14:07:41 [ 2313.890999] [] ? mnt_get_count+0x46/0x5a 2011-09-08 14:07:41 [ 2313.896071] [] ? vfsmount_lock_global_unlock_online+0x40/0x4f 2011-09-08 14:07:41 [ 2313.896151] [] ? mntput_no_expire+0x96/0x155 2011-09-08 14:07:41 [ 2313.896071] [] ? vfsmount_lock_global_unlock_online+0x40/0x4f 2011-09-08 14:07:41 0000000000000082 ffff88042d2abfd8 0000000000000000[ 2313.897380] [] futex_wait_queue_me+0xb8/0xd3 2011-09-08 14:07:41 [ 2313.898586] ffff88082d4a3fd8 ffff88082e3d9fd8[ 2313.900036] Call Trace: 2011-09-08 14:07:41 [ 2313.900219] [] ? common_interrupt+0xe/0x13 2011-09-08 14:07:41 [ 2313.900601] [] ? poll_freewait+0x8d/0x8d 2011-09-08 14:07:41 [ 2313.901106] [] ? check_preempt_curr+0x3e/0x6c 2011-09-08 14:07:41 [ 2313.901413] [] ? sys_clone+0x23/0x25 2011-09-08 14:07:41 0 4399 4360 0x00000000 2011-09-08 14:07:41 ffff88082e6ee480[ 2313.903019] [] ? finish_task_switch+0x64/0x87 2011-09-08 14:07:41 [ 2313.903999] [] ? get_futex_key+0x99/0x229 2011-09-08 14:07:41 [ 2313.904442] [] ? xfs_trans_add_item+0x1e/0x5b 2011-09-08 14:07:41 [ 2313.904694] [] ? vfs_fsync_range+0x5f/0x71 2011-09-08 14:07:41 0000000000000086 ffff88082e626100 0000000000002806[ 2313.906115] [] do_futex+0x9c/0x9b4 2011-09-08 14:07:41 [ 2313.906429] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:41 [ 2313.907522] [] ? get_futex_key+0x99/0x229 2011-09-08 14:07:41 [ 2313.910333] [] ? schedule+0x965/0x9ff 2011-09-08 14:07:41 [ 2313.910333] [] ? schedule+0x965/0x9ff 2011-09-08 14:07:41 [ 2313.915403] [] ? ktime_get_ts+0xad/0xba 2011-09-08 14:07:41 [ 2313.915403] [] ? ktime_get_ts+0xad/0xba 2011-09-08 14:07:41 [ 2313.915465] [] ? ktime_get_ts+0xad/0xba 2011-09-08 14:07:41 [ 2313.915465] [] ? ktime_get_ts+0xad/0xba 2011-09-08 14:07:41 [ 2313.915528] [] sys_select+0x94/0xbd 2011-09-08 14:07:41 [ 2313.915528] [] sys_select+0x94/0xbd 2011-09-08 14:07:41 ffff88083e208010[ 2313.916888] [] ? hrtimer_start_range_ns+0xf/0x11 2011-09-08 14:07:41 [ 2313.918017] mysqld S 2011-09-08 14:07:41 ffff88082e53f620[ 2313.921223] [] do_sigtimedwait+0xe6/0x15d 2011-09-08 14:07:41 ffff88042dcc3660[ 2313.923154] [] inet_recvmsg+0x6b/0x80 2011-09-08 14:07:41 ffff88040b38a080[ 2313.926864] [] ? sched_setscheduler+0xe/0x10 2011-09-08 14:07:41 [ 2313.927970] [] ? tcp_send_delayed_ack+0xbd/0xbf 2011-09-08 14:07:41 ffff8803e8c51a08[ 2313.929706] [] ? ip_local_out+0x24/0x29 2011-09-08 14:07:41 [ 2313.929706] [] ? ip_local_out+0x24/0x29 2011-09-08 14:07:41 [ 2313.934779] [] schedule_timeout+0x37/0x269 2011-09-08 14:07:41 [ 2313.934779] [] schedule_timeout+0x37/0x269 2011-09-08 14:07:41 [ 2313.935596] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:41 [ 2313.936627] [] ? sk_reset_timer+0x17/0x23 2011-09-08 14:07:41 [ 2313.937803] 0000000000004000[ 2313.937803] 0000000000004000[ 2313.938866] [] do_filp_open+0x38/0x88 2011-09-08 14:07:41 ffff880405b3e080[ 2313.940787] mysqld S[ 2313.941830] [] ? get_futex_key+0x99/0x229 2011-09-08 14:07:41 ffff88019837da08[ 2313.949126] [] ? tcp_rcv_established+0xe3/0x8aa 2011-09-08 14:07:41 [ 2313.949189] [] sk_wait_data+0x81/0xc8 2011-09-08 14:07:41 [ 2313.949189] [] sk_wait_data+0x81/0xc8 2011-09-08 14:07:41 [ 2313.954261] [] ? wake_up_bit+0x23/0x23 2011-09-08 14:07:41 [ 2313.954261] [] ? wake_up_bit+0x23/0x23 2011-09-08 14:07:41 [ 2313.954324] [] tcp_recvmsg+0x457/0x8e4 2011-09-08 14:07:41 ffff88068e81f3e0[ 2313.956435] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:41 0000000000000286[ 2313.959270] [] ? tcp_send_delayed_ack+0xbd/0xbf 2011-09-08 14:07:41 [ 2313.959588] [] futex_wait+0x129/0x242 2011-09-08 14:07:41 [ 2313.959839] [] ? cp_new_stat+0xe2/0xee 2011-09-08 14:07:41 2011-09-08 14:07:41 [ 2313.961873] mysqld S ffff88040b38b980[ 2313.963167] [] do_futex+0x9c/0x9b4 2011-09-08 14:07:41 [ 2313.963482] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:41 0000000000000086[ 2313.964121] ffff8801869b1fd8[ 2313.964342] Call Trace: 2011-09-08 14:07:41 [ 2313.964590] [] ? get_futex_key+0x99/0x229 2011-09-08 14:07:41 0000000000010640[ 2313.966818] [] ? do_sched_setscheduler+0x6a/0x75 2011-09-08 14:07:41 0 18684 4360 0x00000000 2011-09-08 14:07:41 0000000000010640[ 2313.968556] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:41 [ 2313.973626] mysqld S[ 2313.968556] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:41 [ 2313.973626] mysqld S ffff8801856f7848 0000000000010640[ 2313.976846] [] do_sync_read+0xc7/0x10d 2011-09-08 14:07:41 0000000000000286 ffff8803fae312f8 ffff8803fae30d90[ 2313.980364] [] inet_recvmsg+0x6b/0x80 2011-09-08 14:07:41 ffff8802f9921b48 ffff8801ab1de010[ 2313.982098] [] do_futex+0x9c/0x9b4 2011-09-08 14:07:41 ffff88017c064080 ffff880178df5c08 [ 2313.987169] [] ? inet_recvmsg+0x6b/0x80 2011-09-08 14:07:41 ffffffff8144426a ffffffff8144426a 2011-09-08 14:07:41 2011-09-08 14:07:41 [ 2313.993417] Call Trace: 2011-09-08 14:07:41 [ 2313.993417] Call Trace: 2011-09-08 14:07:41 [ 2313.993474] [] ? unix_write_space+0x54/0x7c 2011-09-08 14:07:41 [ 2313.993789] [] ? wake_up_bit+0x23/0x23 2011-09-08 14:07:41 ffff88016d764080[ 2313.995146] [] ? tcp_send_delayed_ack+0xbd/0xbf 2011-09-08 14:07:41 [ 2313.995578] [] ? inet_recvmsg+0x6b/0x80 2011-09-08 14:07:41 [ 2313.995766] [] sys_futex+0x10e/0x12c 2011-09-08 14:07:41 0 19182 4360 0x00000000 2011-09-08 14:07:41 [ 2314.000479] [] ? wait_noreap_copyout+0xed/0xed 2011-09-08 14:07:41 [ 2314.002839] mysqld S 0 4602 4425 0x00000000 2011-09-08 14:07:41 [ 2314.004258] mysqld S 0 4603 4425 0x00000000 2011-09-08 14:07:41 [ 2314.005299] [] do_futex+0x9c/0x9b4 2011-09-08 14:07:41 [ 2314.005867] [] ? do_fsync+0x39/0x44 2011-09-08 14:07:41 [ 2314.007035] [] futex_wait+0x129/0x242 2011-09-08 14:07:41 0 4605 4425 0x00000000 2011-09-08 14:07:41 0 4605 4425 0x00000000 2011-09-08 14:07:41 [ 2314.012868] ffff88081c493be8[ 2314.012868] ffff88081c493be8 0000000000000086 0000000000000086[ 2314.013088] 0000000000004000 0000000000000286 0000000000000286[ 2314.014477] ffff88081c495be8 0000000000010640 0000000000000286[ 2314.016074] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:41 [ 2314.017432] [] ? schedule+0x434/0x9ff 2011-09-08 14:07:41 0000000000000086[ 2314.018788] [] futex_wait+0x129/0x242 2011-09-08 14:07:41 ffff8801859cee18[ 2314.020826] [] ? reschedule_interrupt+0xe/0x20 2011-09-08 14:07:41 0000000000010640[ 2314.022874] mysqld S[ 2314.024606] mysqld S ffff8808120b0010[ 2314.026842] [] core_sys_select+0x1a3/0x253 2011-09-08 14:07:41 [ 2314.027092] [] ? ktime_get_ts+0xad/0xba 2011-09-08 14:07:41 [ 2314.027092] [] ? ktime_get_ts+0xad/0xba 2011-09-08 14:07:41 [ 2314.032163] [] sys_select+0x94/0xbd 2011-09-08 14:07:41 [ 2314.032163] [] sys_select+0x94/0xbd 2011-09-08 14:07:41 [ 2314.032226] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:41 [ 2314.032226] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:41 [ 2314.033084] Call Trace: 2011-09-08 14:07:41 [ 2314.033395] [] schedule_hrtimeout_range_clock+0xda/0x123 2011-09-08 14:07:41 [ 2314.034346] [] ? schedule+0x965/0x9ff 2011-09-08 14:07:41 ffff88082e0e1b48 ffffffff8128d7bc 0000000000000086[ 2314.039328] ffff8808120cffd8[ 2314.039547] Call Trace: 2011-09-08 14:07:41 [ 2314.039858] [] schedule_timeout_interruptible+0x19/0x1b 2011-09-08 14:07:41 ffff88022e50ba58 0000000000000292[ 2314.042287] mysqld S 0000000000000292[ 2314.044196] mysqld S ffff8802bdecc010 ffff8802bdecdfd8[ 2314.044991] Call Trace: 2011-09-08 14:07:41 0 1867 4425 0x00000000 2011-09-08 14:07:41 2011-09-08 14:07:41 2011-09-08 14:07:41 [ 2314.051352] 0000000000004000[ 2314.051352] 0000000000004000 ffff8803fc577460 0000000000010640[ 2314.051850] [] ? dev_queue_xmit+0x4b5/0x4c5 2011-09-08 14:07:41 ffff8802831fa010[ 2314.054335] [] sys_futex+0x10e/0x12c 2011-09-08 14:07:41 0 2456 4425 0x00000000 2011-09-08 14:07:41 2011-09-08 14:07:41 ffff8801bf429a38[ 2314.056190] [] do_sync_read+0xc7/0x10d 2011-09-08 14:07:41 ffffffff810336f9[ 2314.059217] [] schedule_timeout+0x37/0x269 2011-09-08 14:07:41 [ 2314.059908] mysqld S 0000000000010640[ 2314.061635] [] system_call_fastpath+0x16/0x1b 2011-09-08 14:07:41 [ 2314.062550] [] ? default_wake_function+0xd/0xf 2011-09-08 14:07:41 ffff88042d3d71c8[ 2314.064603] [] ? generic_write_end+0x54/0x73 2011-09-08 14:07:41 0000000000010640 0000000000000286 2011-09-08 14:07:41 0000000000000286 2011-09-08 14:07:41 [ 2314.070982] Call Trace: 2011-09-08 14:07:41 [ 2314.070982] Call Trace: 2011-09-08 14:07:41 [ 2314.071040] [] ? __blockdev_direct_IO+0x92f/0x95a 2011-09-08 14:07:41 ffff8803cf2e8010[ 2314.072486] ffff8803cf2e9fd8[ 2314.073705] mysqld S 2011-09-08 14:07:41 [ 2314.074809] [] ? __kfree_skb+0x7c/0x80 2011-09-08 14:07:41 ffff8801e6eddfd8 ffff88031128c628 ffff880198144010 ffff8803188a7340 0 12038 4425 0x00000000 2011-09-08 14:07:41 0000000000010640 ffff8803e8d5c140[ 2314.082148] [] sk_wait_data+0x81/0xc8 2011-09-08 14:07:41 [ 2314.083286] ffff880193151fd8[ 2314.083507] Call Trace: 2011-09-08 14:07:41 ffff88039faa2968 2011-09-08 14:07:41 [ 2314.090431] Call Trace: 2011-09-08 14:03:23 [ 2056.116317] php5[16371]: segfault at f ip 00007fa3239f22d0 sp 00007fffd77f1b90 error 6 in libpthread-2.7.so[7fa3239ea000+16000][ 2056.116317] php5[16371]: segfault at f ip 00007fa3239f22d0 sp 00007fffd77f1b90 error 6 in libpthread-2.7.so[7fa3239ea000+16000] 2011-09-08 14:03:23