Hi,
On 3/12/24 17:43, Dai Ngo wrote:
On 3/12/24 4:37 AM, Jeff Layton wrote:
On Mon, 2024-03-11 at 19:43 +0100, Rik Theys wrote:
Hi,
Since a few weeks our Rocky Linux 9 NFS server has periodically
logged hung nfsd tasks. The initial effect was that some clients
could no longer access the NFS server. This got worse and worse
(probably as more nfsd threads got blocked) and we had to restart
the server. Restarting the server also failed as the NFS server
service could no longer be stopped.
The initial kernel we noticed this behavior on was
kernel-5.14.0-362.18.1.el9_3.x86_64. Since then we've installed
kernel-5.14.0-419.el9.x86_64 from CentOS Stream 9. The same issue
happened again on this newer kernel version:
[Mon Mar 11 14:10:08 2024] Not tainted 5.14.0-419.el9.x86_64 #1
[Mon Mar 11 14:10:08 2024] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Mon Mar 11 14:10:08 2024] task:nfsd state:D stack:0
pid:8865 ppid:2 flags:0x00004000
[Mon Mar 11 14:10:08 2024] Call Trace:
[Mon Mar 11 14:10:08 2024] <TASK>
[Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
[Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:10:08 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:10:08 2024] nfsd4_shutdown_callback+0x49/0x120 [nfsd]
[Mon Mar 11 14:10:08 2024] ? nfsd4_cld_remove+0x54/0x1d0 [nfsd]
[Mon Mar 11 14:10:08 2024] ?
nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
[Mon Mar 11 14:10:08 2024] ? nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
[Mon Mar 11 14:10:08 2024] __destroy_client+0x1f3/0x290 [nfsd]
[Mon Mar 11 14:10:08 2024] nfsd4_exchange_id+0x75f/0x770 [nfsd]
[Mon Mar 11 14:10:08 2024] ? nfsd4_decode_opaque+0x3a/0x90 [nfsd]
[Mon Mar 11 14:10:08 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:10:08 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
[Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:10:08 2024] </TASK>
[Mon Mar 11 14:10:08 2024] INFO: task nfsd:8866 blocked for more
than 122 seconds.
[Mon Mar 11 14:10:08 2024] Not tainted 5.14.0-419.el9.x86_64 #1
[Mon Mar 11 14:10:08 2024] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Mon Mar 11 14:10:08 2024] task:nfsd state:D stack:0
pid:8866 ppid:2 flags:0x00004000
[Mon Mar 11 14:10:08 2024] Call Trace:
[Mon Mar 11 14:10:08 2024] <TASK>
[Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
[Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:10:08 2024] ? tcp_recvmsg+0x196/0x210
[Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:10:08 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:10:08 2024] nfsd4_destroy_session+0x1a4/0x240 [nfsd]
[Mon Mar 11 14:10:08 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:10:08 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
[Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:10:08 2024] </TASK>
The above is repeated a few times, and then this warning is also
logged:
[Mon Mar 11 14:12:04 2024] ------------[ cut here ]------------
[Mon Mar 11 14:12:04 2024] WARNING: CPU: 39 PID: 8844 at
fs/nfsd/nfs4state.c:4919 nfsd_break_deleg_cb+0x170/0x190 [nfsd]
[Mon Mar 11 14:12:05 2024] Modules linked in: nfsv4 dns_resolver
nfs fscache netfs rpcsec_gss_krb5 rpcrdma rdma_cm iw_cm ib_cm
ib_core binfmt_misc bonding tls rfkill nft_counter nft_ct
nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_reject_inet
nf_reject_ipv4 nf_reject_ipv6 nft_reject nf_tables nfnetlink vfat
fat dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio l
ibcrc32c dm_service_time dm_multipath intel_rapl_msr
intel_rapl_common intel_uncore_frequency
intel_uncore_frequency_common isst_if_common skx_edac nfit libnvdimm
ipmi_ssif x86_pkg_temp
_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass dcdbas
rapl intel_cstate mgag200 i2c_algo_bit drm_shmem_helper dell_smbios
drm_kms_helper dell_wmi_descriptor wmi_bmof intel_u
ncore syscopyarea pcspkr sysfillrect mei_me sysimgblt acpi_ipmi
mei fb_sys_fops i2c_i801 ipmi_si intel_pch_thermal lpc_ich
ipmi_devintf i2c_smbus ipmi_msghandler joydev acpi_power_meter
nfsd auth_rpcgss nfs_acl drm lockd grace fuse sunrpc ext4 mbcache
jbd2 sd_mod sg lpfc
[Mon Mar 11 14:12:05 2024] nvmet_fc nvmet nvme_fc nvme_fabrics
crct10dif_pclmul ahci libahci crc32_pclmul nvme_core crc32c_intel
ixgbe megaraid_sas libata nvme_common ghash_clmulni_int
el t10_pi wdat_wdt scsi_transport_fc mdio wmi dca dm_mirror
dm_region_hash dm_log dm_mod
[Mon Mar 11 14:12:05 2024] CPU: 39 PID: 8844 Comm: nfsd Not
tainted 5.14.0-419.el9.x86_64 #1
[Mon Mar 11 14:12:05 2024] Hardware name: Dell Inc. PowerEdge
R740/00WGD1, BIOS 2.20.1 09/13/2023
[Mon Mar 11 14:12:05 2024] RIP:
0010:nfsd_break_deleg_cb+0x170/0x190 [nfsd]
[Mon Mar 11 14:12:05 2024] Code: a6 95 c5 f3 e9 ff fe ff ff 48 89
df be 01 00 00 00 e8 34 b5 13 f4 48 8d bb 98 00 00 00 e8 c8 f9 00 00
84 c0 0f 85 2e ff ff ff <0f> 0b e9 27 ff ff ff be
02 00 00 00 48 89 df e8 0c b5 13 f4 e9 01
[Mon Mar 11 14:12:05 2024] RSP: 0018:ffff9929e0bb7b80 EFLAGS:
00010246
[Mon Mar 11 14:12:05 2024] RAX: 0000000000000000 RBX:
ffff8ada51930900 RCX: 0000000000000024
[Mon Mar 11 14:12:05 2024] RDX: ffff8ada519309c8 RSI:
ffff8ad582933c00 RDI: 0000000000002000
[Mon Mar 11 14:12:05 2024] RBP: ffff8ad46bf21574 R08:
ffff9929e0bb7b48 R09: 0000000000000000
[Mon Mar 11 14:12:05 2024] R10: ffff8aec859a2948 R11:
0000000000000000 R12: ffff8ad6f497c360
[Mon Mar 11 14:12:05 2024] R13: ffff8ad46bf21560 R14:
ffff8ae5942e0b10 R15: ffff8ad6f497c360
[Mon Mar 11 14:12:05 2024] FS: 0000000000000000(0000)
GS:ffff8b031fcc0000(0000) knlGS:0000000000000000
[Mon Mar 11 14:12:05 2024] CS: 0010 DS: 0000 ES: 0000 CR0:
0000000080050033
[Mon Mar 11 14:12:05 2024] CR2: 00007fafe2060744 CR3:
00000018e58de006 CR4: 00000000007706e0
[Mon Mar 11 14:12:05 2024] DR0: 0000000000000000 DR1:
0000000000000000 DR2: 0000000000000000
[Mon Mar 11 14:12:05 2024] DR3: 0000000000000000 DR6:
00000000fffe0ff0 DR7: 0000000000000400
[Mon Mar 11 14:12:05 2024] PKRU: 55555554
[Mon Mar 11 14:12:05 2024] Call Trace:
[Mon Mar 11 14:12:05 2024] <TASK>
[Mon Mar 11 14:12:05 2024] ? show_trace_log_lvl+0x1c4/0x2df
[Mon Mar 11 14:12:05 2024] ? show_trace_log_lvl+0x1c4/0x2df
[Mon Mar 11 14:12:05 2024] ? __break_lease+0x16f/0x5f0
[Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
[Mon Mar 11 14:12:05 2024] ? __warn+0x81/0x110
[Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
[Mon Mar 11 14:12:05 2024] ? report_bug+0x10a/0x140
[Mon Mar 11 14:12:05 2024] ? handle_bug+0x3c/0x70
[Mon Mar 11 14:12:05 2024] ? exc_invalid_op+0x14/0x70
[Mon Mar 11 14:12:05 2024] ? asm_exc_invalid_op+0x16/0x20
[Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
[Mon Mar 11 14:12:05 2024] __break_lease+0x16f/0x5f0
[Mon Mar 11 14:12:05 2024] ? nfsd_file_lookup_locked+0x117/0x160
[nfsd]
[Mon Mar 11 14:12:05 2024] ? list_lru_del+0x101/0x150
[Mon Mar 11 14:12:05 2024] nfsd_file_do_acquire+0x790/0x830 [nfsd]
[Mon Mar 11 14:12:05 2024] nfs4_get_vfs_file+0x315/0x3a0 [nfsd]
[Mon Mar 11 14:12:05 2024] nfsd4_process_open2+0x430/0xa30 [nfsd]
[Mon Mar 11 14:12:05 2024] ? fh_verify+0x297/0x2f0 [nfsd]
[Mon Mar 11 14:12:05 2024] nfsd4_open+0x3ce/0x4b0 [nfsd]
[Mon Mar 11 14:12:05 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:12:05 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:12:05 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:12:05 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:12:05 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:12:05 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:12:05 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:12:05 2024] kthread+0xdd/0x100
[Mon Mar 11 14:12:05 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:12:05 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:12:05 2024] </TASK>
[Mon Mar 11 14:12:05 2024] ---[ end trace 7a039e17443dc651 ]---
[Mon Mar 11 14:29:16 2024] task:kworker/u96:3 state:D stack:0
pid:2451130 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: nfsd4_callbacks
nfsd4_run_cb_work [nfsd]
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_timeout+0x88/0x160
[Mon Mar 11 14:29:16 2024] ? __pfx_process_timeout+0x10/0x10
[Mon Mar 11 14:29:16 2024] rpc_shutdown_client+0xb3/0x150 [sunrpc]
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] nfsd4_process_cb_update+0x3e/0x260 [nfsd]
[Mon Mar 11 14:29:16 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:16 2024] ? raw_spin_rq_lock_nested+0x19/0x80
[Mon Mar 11 14:29:16 2024] ? newidle_balance+0x26e/0x400
[Mon Mar 11 14:29:16 2024] ? pick_next_task_fair+0x41/0x500
[Mon Mar 11 14:29:16 2024] ? put_prev_task_fair+0x1e/0x40
[Mon Mar 11 14:29:16 2024] ? pick_next_task+0x861/0x950
[Mon Mar 11 14:29:16 2024] ? __update_idle_core+0x23/0xc0
[Mon Mar 11 14:29:16 2024] ? __switch_to_asm+0x3a/0x80
[Mon Mar 11 14:29:16 2024] ? finish_task_switch.isra.0+0x8c/0x2a0
[Mon Mar 11 14:29:16 2024] nfsd4_run_cb_work+0x9f/0x150 [nfsd]
[Mon Mar 11 14:29:16 2024] process_one_work+0x1e2/0x3b0
[Mon Mar 11 14:29:16 2024] worker_thread+0x50/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
The above is the main task that I see in the cb workqueue. It's
trying to call rpc_shutdown_client, which is waiting for this:
wait_event_timeout(destroy_wait,
list_empty(&clnt->cl_tasks), 1*HZ);
...so basically waiting for the cl_tasks list to go empty. It repeatedly
does a rpc_killall_tasks though, so possibly trying to kill this task?
18423 2281 0 0x18 0x0 1354 nfsd4_cb_ops [nfsd]
nfs4_cbv1 CB_RECALL_ANY a:call_start [sunrpc] q:delayq
I wonder why this task is on delayq. Could it be related to memory
shortage issue, or connection related problems?
Output of /proc/meminfo on the nfs server at time of the problem
would shed some light.
We don't have that anymore. I can check our monitoring host more closely
for more fine grained stats tomorrow, but when I look at the sar
statistics (see attachment) nothing special was going on memory or
network wise.
We start to get the cpu stall messages and the system load starts to
rise (starts around 2:10 PM). At 3:00 PM we restart the server as our
users can no longer work.
Looking at the stats, the cpu's were ~idle. The only thing that may be
related is that around 11:30 AM the write load (rx packets) starts to
get a lot higher than the read load (tx packets). This goes on for hours
(even after the server was restarted) and that workload was later
identified. It was a workload that was constantly rewriting a statistics
file.
Regards,
Rik
Currently there is only 1 active task allowed for the nfsd callback
workqueue at a time. If for some reasons a callback task is stuck in
the workqueue it will block all other callback tasks which can effect
multiple clients.
-Dai
Callbacks are soft RPC tasks though, so they should be easily killable.
--
Rik Theys
System Engineer
KU Leuven - Dept. Elektrotechniek (ESAT)
Kasteelpark Arenberg 10 bus 2440 - B-3001 Leuven-Heverlee
+32(0)16/32.11.07
----------------------------------------------------------------
<<Any errors in spelling, tact or fact are transmission errors>>
11:20:00 AM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15 blocked
11:30:00 AM 2 1198 3.63 3.33 2.77 1
11:40:01 AM 4 1197 3.29 3.87 3.36 3
11:50:00 AM 0 1203 2.34 3.32 3.50 0
12:00:00 PM 3 1184 2.31 2.64 3.06 3
12:10:00 PM 4 1192 5.66 4.29 3.58 1
12:20:00 PM 1 1197 7.46 4.69 3.86 1
12:30:00 PM 2 1200 3.65 3.87 3.83 0
12:40:00 PM 4 1209 3.15 3.57 3.77 1
12:50:00 PM 0 1204 5.15 4.33 3.95 1
01:00:00 PM 6 1192 3.26 3.57 3.78 1
01:10:00 PM 3 1215 3.50 3.12 3.41 0
01:20:00 PM 3 1196 3.15 3.11 3.27 5
01:30:00 PM 4 1194 4.13 3.71 3.52 0
01:40:00 PM 1 1200 2.97 2.99 3.19 0
01:50:00 PM 1 1210 3.55 3.54 3.44 0
02:00:00 PM 2 1220 4.61 4.56 3.93 0
02:10:00 PM 1 1229 16.94 9.96 6.44 0
02:20:00 PM 1 1208 57.03 45.47 26.17 0
02:30:00 PM 3 1218 66.46 58.40 41.85 0
02:40:00 PM 2 1218 71.43 67.15 54.03 1
02:50:00 PM 4 1222 87.71 81.15 67.21 1
03:00:00 PM 5 1228 98.74 93.68 80.13 2
Average: 2 1120 6.00 5.46 4.53 0
03:11:33 PM LINUX RESTART (48 CPU)
11:20:00 AM CPU %user %nice %system %iowait %steal %idle
11:20:00 AM all 0.59 0.00 2.61 0.48 0.00 96.31
11:30:00 AM all 0.55 0.00 3.78 3.05 0.00 92.62
11:40:01 AM all 0.52 0.00 4.48 2.71 0.00 92.29
11:50:00 AM all 0.51 0.00 4.54 2.91 0.00 92.04
12:00:00 PM all 0.52 0.00 4.17 1.13 0.00 94.18
12:10:00 PM all 0.59 0.00 4.51 2.10 0.00 92.79
12:20:00 PM all 0.54 0.00 4.67 0.87 0.00 93.92
12:30:00 PM all 0.50 0.00 5.40 1.61 0.00 92.50
12:40:00 PM all 0.50 0.00 6.04 1.27 0.00 92.20
12:50:00 PM all 0.53 0.00 5.43 1.39 0.00 92.65
01:00:00 PM all 0.61 0.00 5.56 1.90 0.00 91.93
01:10:00 PM all 0.60 0.00 5.38 0.91 0.00 93.11
01:20:00 PM all 0.59 0.00 5.69 1.12 0.00 92.59
01:30:00 PM all 0.60 0.00 6.06 1.30 0.00 92.04
01:40:00 PM all 0.57 0.00 5.15 1.32 0.00 92.95
01:50:00 PM all 0.62 0.00 5.58 1.86 0.00 91.94
02:00:00 PM all 0.62 0.00 6.01 2.45 0.00 90.93
02:10:00 PM all 0.65 0.00 9.16 2.95 0.00 87.23
02:20:00 PM all 0.62 0.00 9.68 4.55 0.00 85.15
02:30:00 PM all 0.68 0.00 7.05 2.24 0.00 90.03
02:40:00 PM all 1.00 0.00 5.92 2.19 0.00 90.89
02:50:00 PM all 2.00 0.00 6.49 3.23 0.00 88.27
03:00:00 PM all 1.37 0.00 5.90 3.01 0.00 89.72
Average: all 0.32 0.00 2.84 0.81 0.00 96.03
03:11:33 PM LINUX RESTART (48 CPU)
11:20:00 AM pgpgin/s pgpgout/s fault/s majflt/s pgfree/s pgscank/s pgscand/s pgsteal/s %vmeff
11:20:00 AM 4020.11 8605.06 2157.62 0.00 191933.03 1217.98 0.00 2442.82 200.56
11:30:00 AM 55268.90 50540.80 2096.02 0.04 637897.57 17648.39 0.00 35296.47 200.00
11:40:01 AM 79943.11 116231.90 1859.00 0.00 760848.68 23008.54 0.00 46020.39 200.01
11:50:00 AM 22594.29 173959.91 1676.84 0.00 1084371.55 14182.57 0.00 28364.99 200.00
12:00:00 PM 2769.56 164286.91 1450.51 0.00 896969.12 1416.75 0.00 2836.86 200.24
12:10:00 PM 8822.01 155296.22 1431.25 0.00 931734.21 2687.33 0.00 5378.07 200.13
12:20:00 PM 2577.91 178625.24 1379.77 0.00 1180545.30 3497.12 0.00 6994.22 200.00
12:30:00 PM 5954.85 226744.21 1253.30 0.00 1918863.89 7147.24 0.00 14297.88 200.05
12:40:00 PM 2377.24 289116.52 1194.98 0.00 2580662.81 6756.25 0.00 13512.50 200.00
12:50:00 PM 1939.94 270595.12 1579.08 0.00 2189766.74 2785.89 0.00 5571.77 200.00
01:00:00 PM 3975.65 280647.85 1960.25 0.00 2421730.66 2904.24 0.00 5808.49 200.00
01:10:00 PM 3061.46 285614.89 2077.81 0.00 2256171.33 2655.80 0.00 5311.57 200.00
01:20:00 PM 1522.98 289014.26 1878.51 0.00 2514450.25 1213.70 0.00 2427.40 200.00
01:30:00 PM 4979.56 272158.71 1622.52 0.00 2252781.20 2183.79 0.00 4367.58 200.00
01:40:00 PM 4364.82 266270.12 1411.46 0.00 2013371.12 2164.64 0.00 4329.28 200.00
01:50:00 PM 9045.98 274397.77 1958.44 0.01 1970749.12 3996.12 0.00 7995.65 200.09
02:00:00 PM 6472.46 280631.95 1453.42 0.00 2098466.10 3921.65 0.00 7843.27 200.00
02:10:00 PM 2577.67 491251.09 2247.01 0.00 2415029.61 75131.80 0.00 150276.28 200.02
02:20:00 PM 8279.88 523464.79 2216.22 0.00 2554580.80 85277.68 0.00 170554.78 200.00
02:30:00 PM 4016.00 378426.50 2704.81 0.32 2155706.27 36195.48 0.00 72400.96 200.03
02:40:00 PM 10939.12 268965.25 2279.79 0.01 1920982.26 4331.20 0.00 8676.01 200.31
02:50:00 PM 35906.38 286698.69 1951.03 0.00 1994827.72 13385.01 0.00 26776.99 200.05
03:00:00 PM 30307.83 284038.19 1873.97 0.02 2041174.51 12174.66 0.00 24352.69 200.03
Average: 11214.03 84754.74 1052.37 0.01 641224.50 7068.27 0.06 14139.06 200.03
03:11:33 PM LINUX RESTART (48 CPU)
11:20:00 AM kbmemfree kbavail kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty kbanonpg kbslab kbkstack kbpgtbl kbvmused
11:20:00 AM 1865420 171187348 1998368 1.02 14285620 138954960 4464588 1.70 1157788 152702956 163464 672516 39006492 19600 46112 305960
11:30:00 AM 3687156 171152312 2037216 1.04 13766560 138071112 4532556 1.72 1209572 151275684 483556 700220 38548816 19904 48504 306200
11:40:01 AM 1766552 170336640 2143452 1.09 12970608 141160884 4478464 1.70 1171124 153592108 542440 683992 38069364 19888 47428 306200
11:50:00 AM 7281384 171091800 2096236 1.07 12565028 136191740 4527272 1.72 1185148 148217324 57744 697772 37976472 20000 48268 306216
12:00:00 PM 8276692 171026364 2151216 1.10 12754360 134945564 4521328 1.72 1175520 147155948 1041800 689424 37983028 19664 48056 305800
12:10:00 PM 2571948 171107708 2052636 1.05 12886184 140612248 4546576 1.73 1182792 152958724 475408 700172 37987844 19808 48144 306072
12:20:00 PM 4397388 171101628 2036452 1.04 12975352 138459220 4539876 1.72 1171936 150894616 796324 687384 38242448 19904 47496 306104
12:30:00 PM 7087352 170716108 2133384 1.09 13021692 135325824 4586040 1.74 1178288 147807308 1722952 693240 38542608 19968 48404 306216
12:40:00 PM 7524128 170954416 2165520 1.10 13097896 134570932 4613396 1.75 1189344 147128772 701080 704404 38752384 20144 49080 306312
12:50:00 PM 6051508 170910668 2199872 1.12 13263460 135803304 4612484 1.75 1186232 148528800 373732 705916 38792716 20000 49068 306104
01:00:00 PM 10992332 170726464 2138820 1.09 13288164 130804672 4624612 1.76 1177744 143554556 2956808 697308 38886872 19824 49076 306200
01:10:00 PM 3782976 170738340 2117004 1.08 13314832 137872584 4641200 1.76 1193912 150642700 404072 704116 39023464 20160 50028 306408
01:20:00 PM 6659444 170900236 2167088 1.11 13376480 134926060 4653100 1.77 1184556 147757720 388432 697876 38981788 19888 49888 306248
01:30:00 PM 13172768 170885616 2172216 1.11 13393784 128373088 4653708 1.77 1186008 141224736 4071760 700964 38999004 19840 49880 306120
01:40:00 PM 10682972 170893288 2144464 1.09 13517676 130723148 4701116 1.79 1193700 143698488 1415012 706440 39042600 19952 50844 306296
01:50:00 PM 6085404 170824968 2200596 1.12 13221836 135528552 4736532 1.80 1201632 148203920 2410612 712088 39074472 20176 51992 306344
02:00:00 PM 5555448 170810072 2201640 1.12 13658348 135470552 4758996 1.81 1223860 148582888 307464 731856 39224872 20192 53144 306456
02:10:00 PM 1014880 170836368 2160028 1.10 12728708 140151288 4823616 1.83 1227072 152329884 9645304 731748 40055956 20432 53516 306664
02:20:00 PM 2823068 170828288 2152108 1.10 13199316 137843512 4801392 1.82 1217876 150492036 8488084 721812 40092856 20064 53224 306408
02:30:00 PM 7005404 170678580 2261792 1.15 13358768 133429936 4893344 1.86 1271688 146226924 1625480 752752 40054960 20224 56408 306456
02:40:00 PM 4016676 170697044 2232824 1.14 13727020 136140404 4911972 1.87 1258276 149302784 167584 740892 39993936 20240 54472 306580
02:50:00 PM 9976392 170683428 2192384 1.12 13706924 130450508 4931124 1.87 1261640 143607304 1520456 760304 39784652 20304 55912 304648
03:00:00 PM 8165036 170637924 2235228 1.14 13969932 132128568 4956008 1.88 1249376 145549216 1503472 751636 39612096 20384 55216 304588
Average: 8003274 171354427 1997084 1.02 17978233 128153402 4009864 1.52 2948732 143744234 802698 618928 39978867 18658 35078 303511
03:11:33 PM LINUX RESTART (48 CPU)
12:00:01 AM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
10:20:01 AM bond0 19341.55 16613.52 14852.80 10063.03 0.00 0.00 7.59 0.61
10:30:00 AM bond0 22612.77 23802.93 17064.32 18021.10 0.00 0.00 7.54 0.74
10:40:00 AM bond0 36072.84 28650.68 36436.66 23838.56 0.00 0.00 7.54 1.49
10:50:00 AM bond0 30297.97 26476.98 28896.10 21851.80 0.00 0.00 7.59 1.18
11:00:00 AM bond0 24422.77 22650.70 17771.64 14655.68 0.00 0.00 7.57 0.73
11:10:00 AM bond0 32648.61 55389.48 27350.30 63200.39 0.00 0.00 7.61 2.59
11:20:00 AM bond0 20966.71 20631.12 15049.13 14239.36 0.00 0.00 7.59 0.62
11:30:00 AM bond0 59038.60 85522.45 62065.03 104016.82 0.00 0.00 7.54 4.26
11:40:01 AM bond0 109991.82 97051.54 129799.48 117008.01 0.00 0.00 7.54 5.32
11:50:00 AM bond0 142884.43 45306.77 188287.91 34672.37 0.00 0.00 7.60 7.71
12:00:00 PM bond0 135270.64 29202.31 180105.47 15043.09 0.00 0.00 7.54 7.38
12:10:00 PM bond0 128682.92 38263.76 167859.64 26517.80 0.00 0.00 7.65 6.88
12:20:00 PM bond0 147128.05 41190.26 195974.49 28419.00 0.00 0.00 7.59 8.03
12:30:00 PM bond0 183452.21 59540.64 248497.44 41698.82 0.00 0.00 7.54 10.18
12:40:00 PM bond0 223045.51 49631.56 308994.74 20892.16 0.00 0.00 7.56 12.66
12:50:00 PM bond0 211356.42 40242.54 292794.32 15753.38 0.00 0.00 7.55 11.99
01:00:00 PM bond0 221345.60 43010.18 307861.64 17748.64 0.00 0.00 7.55 12.61
01:10:00 PM bond0 223147.26 40941.92 311071.09 17092.65 0.00 0.00 7.58 12.74
01:20:00 PM bond0 226732.82 43452.27 315370.79 16082.25 0.00 0.00 7.61 12.92
01:30:00 PM bond0 222725.38 72765.59 303529.78 59891.91 0.00 0.00 7.52 12.43
01:40:00 PM bond0 206342.36 40977.06 286164.94 20165.04 0.00 0.00 7.54 11.72
01:50:00 PM bond0 218327.39 49942.46 300638.56 30757.69 0.00 0.00 7.60 12.31
02:00:00 PM bond0 218851.87 54196.47 299367.98 33044.98 0.00 0.00 7.57 12.26
02:10:00 PM bond0 378807.79 53458.22 538611.10 19944.21 0.00 0.00 7.58 22.06
02:20:00 PM bond0 387285.99 55460.02 555147.68 22975.23 0.00 0.00 7.58 22.74
02:30:00 PM bond0 274593.98 37885.29 390578.19 11658.62 0.00 0.00 7.56 16.00
02:40:00 PM bond0 202721.14 41980.65 282541.86 19462.85 0.00 0.00 7.56 11.57
02:50:00 PM bond0 218072.41 52718.98 301602.85 36963.31 0.00 0.00 7.59 12.35
03:00:00 PM bond0 214600.08 51698.15 299890.67 36181.66 0.00 0.00 7.58 12.28
Average: bond0 70892.56 27494.51 89394.02 19260.67 0.00 0.00 7.57 3.66
03:11:33 PM LINUX RESTART (48 CPU)
12:00:01 AM IFACE rxerr/s txerr/s coll/s rxdrop/s txdrop/s txcarr/s rxfram/s rxfifo/s txfifo/s
11:20:00 AM bond0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:30:00 AM bond0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:40:01 AM bond0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:50:00 AM bond0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
12:00:00 PM bond0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
12:10:00 PM bond0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
12:20:00 PM bond0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
12:30:00 PM bond0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
12:40:00 PM bond0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
12:50:00 PM bond0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:00:00 PM bond0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:10:00 PM bond0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:20:00 PM bond0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:30:00 PM bond0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:40:00 PM bond0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:50:00 PM bond0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
02:00:00 PM bond0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
02:10:00 PM bond0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
02:20:00 PM bond0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
02:30:00 PM bond0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
02:40:00 PM bond0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
02:50:00 PM bond0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
03:00:00 PM bond0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: bond0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
03:11:33 PM LINUX RESTART (48 CPU)
