On Mon, 24 Feb 2014, m.roth@xxxxxxxxx wrote:
> Every so often, one of our servers will go into what I can only describe
> as an undefined state: it pings, but there's zero access - you can't ssh
> in, and if I go plug a keyboard and monitor into the server itself, you
> can see the monitor's live, it's not the "monitor turned off" color, but
> there is zero response to the keyboard. The upshot is that I wind up
> having to power cycle it.
>
> Well, it just happened again on one of our servers Friday evening, as I
> found this morning. Looking at the logs this morning, I see that sar last
> shows
> 10:20:01 PM all 34.38 0.00 8.29 0.00 0.00
> 57.33
>
> On of my users dropped me an email at 22:45 that it was "off", and the
> last things I see in /var/log/messages are one of those annoying
> Feb 21 22:26:23 <server> kernel: INFO: task perl:20596 blocked for more
> than 120 seconds.
> Feb 21 22:26:23 <server> kernel: "echo 0 >
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>
> I also see
> Feb 21 22:26:23 <server> kernel: perl D ffffffff80158250 0
> 20596 20557
> which, as I just found by googling perl NOTLD, means that this is in a
> kernel uninterruptable state
> In addition, in the stack trace, some nfs messages
> Feb 21 22:26:23 <server> kernel: [<ffffffff886b58d1>]
> :nfs:nfs_wait_bit_uninterruptible+0x0/0xd
>
> So, it *appears* to be either an NFS issue, or a NIC issue. The user's
> home directory server is CentOS running 6.5, and the server that hung is
> 5.10. Mount on the formerly hung server, su-d to his account shows merely
> nfs, so I'm guessing it's NFS3. Looking at lsmod and /var/log/dmesg, I see
> it's running the tg3 NIC driver.
>
> Anyone else seeing this, and if so, any thoughts on the matter? Note that
> I've had this on Penguins, which are all Supermicro, and they're using the
> igb NIC driver, but the one this past weekend is a Dell, so it's not just
> one system.
>
>
> mark
>
> _______________________________________________
> CentOS mailing list
> CentOS@xxxxxxxxxx
> http://lists.centos.org/mailman/listinfo/centos
>
What CPU's do these systems have? AMD or Intel.
What kernel are the server and client running?
-Connie Sieh
_______________________________________________
CentOS mailing list
CentOS@xxxxxxxxxx
http://lists.centos.org/mailman/listinfo/centos
