Thanks for your reply, Christian. Sorry for my delay in responding. The kernel logs are silent. Forgot to mention before that ntpd is running and the nodes are sync'd. I'm working on some folks for an updated kernel, but I'm not holding my breath. That said, If I'm seeing this problem by running rados bench on the storage cluster itself, is it fair to say that the kernel code isn't the issue? vm/min_free_kbytes is now set to 512M, though that didn't solve the issue. I also set "filestore_max_sync_interval = 30" (and commented out the journal line) as you suggested, but that didn't seem to change anything, either. Not sure what you mean about the monitors and SSD's... they currently *are* hosted on SSD's, which don't appear to be When rados bench starts, atop (holy crap that's a lot of info) shows that the HDD's go crazy for a little while (busy >85%). The SSD's never get that busy (certainly <50%). I attached a few 'snapshots' of atop taken just after the test starts (~12s), while it was still running (~30s), and after the test was supposed to have ended (~70s), but was essentially waiting for slow-requests. The only thing red-lining at all were the HDD's I wonder how I could test our network. Are you thinking its possible we're losing packets? I'll ping (har!) our network guy... I have to admit that the OSD logs don't mean a whole lot to me. Are OSD log entries like this normal? This is not from during the test, but just before when the system was essentially idle. 2015-01-07 15:38:40.340883 7fa264ff7700 0 -- 39.71.48.8:6800/46686 >> 39.71.48.6:6806/47930 pipe(0x7fa268c14480 sd=111 :40639 s=2 pgs=559 cs=13 l=0 c=0x7fa283060080).fault with nothing to send, going to standby 2015-01-07 15:38:53.573890 7fa2b99f6700 0 -- 39.71.48.8:6800/46686 >> 39.71.48.9:6805/23130 pipe(0x7fa268c55800 sd=127 :6800 s=2 pgs=152 cs=13 l=0 c=0x7fa268c17e00).fault with nothing to send, going to standby 2015-01-07 15:38:55.881934 7fa281bfd700 0 -- 39.71.48.8:6800/46686 >> 39.71.48.9:6809/44433 pipe(0x7fa268c12180 sd=65 :41550 s=2 pgs=599 cs=19 l=0 c=0x7fa28305fc00).fault with nothing to send, going to standby 2015-01-07 15:38:56.360866 7fa29e1f6700 0 -- 39.71.48.8:6800/46686 >> 39.71.48.6:6820/48681 pipe(0x7fa268c14980 sd=145 :6800 s=2 pgs=500 cs=21 l=0 c=0x7fa28305fa80).fault with nothing to send, going to standby 2015-01-07 15:38:58.767181 7fa2a85f6700 0 -- 39.71.48.8:6800/46686 >> 39.71.48.6:6820/48681 pipe(0x7fa268c55d00 sd=52 :6800 s=0 pgs=0 cs=0 l=0 c=0x7fa268c18b80).accept connect_seq 22 vs existing 21 state standby 2015-01-07 15:38:58.943514 7fa253cf0700 0 -- 39.71.48.8:6800/46686 >> 39.71.48.9:6805/23130 pipe(0x7fa268c55f80 sd=49 :6800 s=0 pgs=0 cs=0 l=0 c=0x7fa268c18d00).accept connect_seq 14 vs existing 13 state standby For the OSD complaining about slow requests its logs show something like during the test: 2015-01-07 15:47:28.463470 7fc0714f0700 0 -- 39.7.48.7:6812/16907 >> 39.7.48.4:0/3544514455 pipe(0x7fc08f827a80 sd=153 :6812 s=0 pgs=0 cs=0 l=0 c=0x7fc08f882580).accept peer addr is really 39.7.48.4:0/3544514455 (socket is 39.7.48.4:464 35/0) 2015-01-07 15:48:04.426399 7fc0e9bfd700 0 log [WRN] : 1 slow requests, 1 included below; oldest blocked for > 30.738429 secs 2015-01-07 15:48:04.426416 7fc0e9bfd700 0 log [WRN] : slow request 30.738429 seconds old, received at 2015-01-07 15:47:33.687935: osd_op(client.92886.0:4711 benchmark_data_tvsaq1_29431_object4710 [write 0~4194304] 3.1639422f ack+ondisk+ write e1464) v4 currently waiting for subops from 22,36 2015-01-07 15:48:34.429979 7fc0e9bfd700 0 log [WRN] : 1 slow requests, 1 included below; oldest blocked for > 60.742016 secs 2015-01-07 15:48:34.429997 7fc0e9bfd700 0 log [WRN] : slow request 60.742016 seconds old, received at 2015-01-07 15:47:33.687935: osd_op(client.92886.0:4711 benchmark_data_tvsaq1_29431_object4710 [write 0~4194304] 3.1639422f ack+ondisk+ write e1464) v4 currently waiting for subops from 22,36 ________________________________________ From: Christian Balzer [chibi@xxxxxxx] Sent: Tuesday, January 06, 2015 12:25 AM To: ceph-users@xxxxxxxxxxxxxx Cc: Sanders, Bill Subject: Re: Slow/Hung IOs On Mon, 5 Jan 2015 22:36:29 +0000 Sanders, Bill wrote: > Hi Ceph Users, > > We've got a Ceph cluster we've built, and we're experiencing issues with > slow or hung IO's, even running 'rados bench' on the OSD cluster. > Things start out great, ~600 MB/s, then rapidly drops off as the test > waits for IO's. Nothing seems to be taxed... the system just seems to be > waiting. Any help trying to figure out what could cause the slow IO's > is appreciated. > I assume nothing in the logs of the respective OSDs either? Kernel or other logs equally silent? Watching things with atop (while running the test) not showing anything particular? Looking at the myriad of throttles and other data in http://ceph.com/docs/next/dev/perf_counters/ might be helpful for the affected OSDs. Having this kind of (consistent?) trouble feels like a networking issue of sorts, OSDs not able to reach each other or something massively messed up in the I/O stack. [snip] > Our ceph cluster is 4x Dell R720xd nodes: > 2x1TB spinners configured in RAID for the OS > 10x4TB spinners for OSD's (XFS) > 2x400GB SSD's, each with 5x~50GB OSD journals > 2x Xeon E5-2620 CPU (/proc/cpuinfo reports 24 cores) > 128GB RAM > Two networks (public+cluster), both over infiniband > Usual IB kernel tuning done, network stack stuff and vm/min_free_kbytes to 512MB at least? > Three monitors are configured on the first three nodes, and use a chunk > of one of the SSDs for their data, on an XFS partition > Since you see nothing in the logs probably not your issue, but monitors like the I/O for their leveldb fast, SSD recommended. > Software: > SLES 11SP3, with some in house patching. (3.0.1 kernel, "ceph-client" > backported from 3.10) Ceph version: ceph-0.80.5-0.9.2, packaged by SUSE > Can't get a 3.16 backport for this? > ceph.conf: > fsid = 3e8dbfd8-c3c8-4d30-80e2-cd059619d757 > mon initial members = tvsaq1, tvsaq2, tvsar1 > mon host = 39.7.48.6, 39.7.48.7, 39.7.48.8 > > cluster network = 39.64.0.0/12 > public network = 39.0.0.0/12 > auth cluster required = cephx > auth service required = cephx > auth client required = cephx > osd journal size = 9000 Not sure how this will affect things given that you have 50GB partitions. I'd remove that line and replace it with something like: filestore_max_sync_interval = 30 (I use 10 with 10GB journals) Regards, Christian > filestore xattr use omap = true > osd crush update on start = false > osd pool default size = 3 > osd pool default min size = 1 > osd pool default pg num = 4096 > osd pool default pgp num = 4096 > > mon clock drift allowed = .100 > osd mount options xfs = rw,noatime,inode64 > > > > -- Christian Balzer Network/Systems Engineer chibi@xxxxxxx Global OnLine Japan/Fusion Communications http://www.gol.com/
PRC | sys 7.11s | user 5.73s | | | #proc 374 | | #trun 5 | #tslpi 4307 | #tslpu 10 | | #zombie 0 | clones 4 | | | #exit 4 | CPU | sys 155% | user 142% | | irq 29% | | | idle 1712% | wait 363% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 24% | user 9% | | irq 28% | | | idle 30% | cpu000 w 9% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 21% | user 10% | | irq 0% | | | idle 59% | cpu002 w 11% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 14% | user 12% | | irq 0% | | | idle 70% | cpu012 w 5% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 9% | user 13% | | irq 0% | | | idle 61% | cpu003 w 17% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 8% | user 9% | | irq 0% | | | idle 81% | cpu001 w 3% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 6% | user 8% | | irq 0% | | | idle 32% | cpu010 w 53% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 8% | user 6% | | irq 0% | | | idle 76% | cpu014 w 10% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 8% | user 4% | | irq 0% | | | idle 68% | cpu013 w 20% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 4% | user 6% | | irq 0% | | | idle 76% | cpu007 w 13% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 4% | user 7% | | irq 0% | | | idle 89% | cpu005 w 0% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 5% | user 6% | | irq 0% | | | idle 39% | cpu008 w 50% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 5% | user 5% | | irq 0% | | | idle 52% | cpu004 w 38% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 5% | user 6% | | irq 0% | | | idle 49% | cpu006 w 40% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 4% | user 6% | | irq 0% | | | idle 73% | cpu018 w 17% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 4% | user 6% | | irq 0% | | | idle 90% | cpu009 w 0% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 4% | user 5% | | irq 0% | | | idle 86% | cpu015 w 5% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 5% | user 5% | | irq 0% | | | idle 67% | cpu016 w 23% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 7% | user 2% | | irq 0% | | | idle 91% | cpu017 w 0% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 3% | user 5% | | irq 0% | | | idle 87% | cpu011 w 6% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 3% | user 5% | | irq 0% | | | idle 72% | cpu022 w 21% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 2% | user 3% | | irq 0% | | | idle 92% | cpu019 w 3% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 2% | user 2% | | irq 0% | | | idle 77% | cpu020 w 20% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 1% | user 2% | | irq 0% | | | idle 96% | cpu021 w 0% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 1% | user 2% | | irq 0% | | | idle 97% | cpu023 w 0% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | CPL | avg1 4.91 | | avg5 3.82 | | avg15 2.56 | | | | csw 130531 | | intr 63515 | | | | numcpu 24 | MEM | tot 125.7G | free 14.2G | | cache 92.9G | dirty 565.4M | buff 76.7M | slab 6.8G | | slrec 4.1G | | shmem 0.1M | shrss 0.0M | | shswp 0.0M | | SWP | tot 27.8G | | free 27.8G | | | | | | | | | | vmcom 21.4G | | vmlim 90.6G | PAG | scan 0 | | steal 0 | stall 0 | | | | | | | | | swin 0 | | swout 0 | DSK | sdg | | busy 84% | read 104 | | write 583 | KiB/r 4 | | KiB/w 255 | MBr/s 0.10 | | MBw/s 36.34 | avq 7.90 | | avio 4.86 ms | DSK | sdk | | busy 82% | read 101 | | write 564 | KiB/r 4 | | KiB/w 254 | MBr/s 0.10 | | MBw/s 35.11 | avq 9.32 | | avio 4.89 ms | DSK | sde | | busy 77% | read 104 | | write 483 | KiB/r 4 | | KiB/w 197 | MBr/s 0.10 | | MBw/s 23.24 | avq 13.15 | | avio 5.25 ms | DSK | sdl | | busy 75% | read 83 | | write 511 | KiB/r 4 | | KiB/w 256 | MBr/s 0.08 | | MBw/s 32.03 | avq 8.35 | | avio 5.04 ms | DSK | sdi | | busy 71% | read 74 | | write 478 | KiB/r 4 | | KiB/w 255 | MBr/s 0.07 | | MBw/s 29.81 | avq 8.52 | | avio 5.14 ms | DSK | sdh | | busy 69% | read 54 | | write 635 | KiB/r 4 | | KiB/w 176 | MBr/s 0.05 | | MBw/s 27.36 | avq 40.60 | | avio 4.01 ms | DSK | sdm | | busy 68% | read 73 | | write 490 | KiB/r 4 | | KiB/w 145 | MBr/s 0.07 | | MBw/s 17.42 | avq 29.68 | | avio 4.82 ms | DSK | sdj | | busy 61% | read 51 | | write 323 | KiB/r 4 | | KiB/w 255 | MBr/s 0.05 | | MBw/s 20.15 | avq 8.99 | | avio 6.51 ms | DSK | sdf | | busy 61% | read 74 | | write 465 | KiB/r 4 | | KiB/w 256 | MBr/s 0.07 | | MBw/s 29.09 | avq 7.77 | | avio 4.48 ms | DSK | sdd | | busy 58% | read 63 | | write 441 | KiB/r 4 | | KiB/w 255 | MBr/s 0.06 | | MBw/s 27.46 | avq 8.43 | | avio 4.62 ms | DSK | sdb | | busy 7% | read 0 | | write 2654 | KiB/r 0 | | KiB/w 253 | MBr/s 0.00 | | MBw/s 164.47 | avq 9.31 | | avio 0.11 ms | DSK | sdc | | busy 6% | read 0 | | write 2296 | KiB/r 0 | | KiB/w 255 | MBr/s 0.00 | | MBw/s 143.29 | avq 9.84 | | avio 0.10 ms | DSK | sda | | busy 0% | read 0 | | write 0 | KiB/r 0 | | KiB/w 0 | MBr/s 0.00 | | MBw/s 0.00 | avq 0.00 | | avio 0.00 ms | NET | transport | tcpi 50015 | | tcpo 76871 | udpi 0 | udpo 0 | tcpao 0 | | tcppo 0 | tcprs 31 | tcpie 0 | tcpor 0 | | udpnp 0 | udpip 0 | NET | network | | ipi 50024 | ipo 59717 | | ipfrw 0 | deliv 50021 | | | | | | icmpi 0 | | icmpo 0 | NET | byn0 ---- | pcki 25465 | | pcko 24728 | si 1748 Mbps | | so 1665 Mbps | coll 0 | mlti 0 | | erri 0 | erro 0 | | drpi 0 | drpo 0 | NET | byn1 ---- | pcki 13503 | | pcko 23933 | si 2725 Kbps | | so 2731 Mbps | coll 0 | mlti 0 | | erri 0 | erro 0 | | drpi 0 | drpo 0 |
PRC | sys 7.80s | user 6.79s | | | #proc 374 | | #trun 5 | #tslpi 4301 | #tslpu 16 | | #zombie 0 | clones 4 | | | #exit 5 | CPU | sys 170% | user 170% | | irq 33% | | | idle 1586% | wait 446% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 25% | user 10% | | irq 31% | | | idle 26% | cpu000 w 9% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 18% | user 12% | | irq 0% | | | idle 56% | cpu002 w 15% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 13% | user 8% | | irq 0% | | | idle 50% | cpu014 w 28% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 8% | user 10% | | irq 0% | | | idle 64% | cpu005 w 17% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 8% | user 10% | | irq 0% | | | idle 66% | cpu009 w 14% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 8% | user 10% | | irq 0% | | | idle 49% | cpu011 w 33% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 11% | user 7% | | irq 0% | | | idle 80% | cpu012 w 2% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 8% | user 9% | | irq 0% | | | idle 67% | cpu004 w 16% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 7% | user 8% | | irq 0% | | | idle 52% | cpu006 w 33% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 6% | user 9% | | irq 0% | | | idle 57% | cpu007 w 28% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 6% | user 8% | | irq 0% | | | idle 72% | cpu003 w 13% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 6% | user 8% | | irq 0% | | | idle 80% | cpu001 w 6% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 6% | user 8% | | irq 0% | | | idle 66% | cpu008 w 20% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 6% | user 7% | | irq 0% | | | idle 54% | cpu010 w 33% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 5% | user 7% | | irq 0% | | | idle 88% | cpu017 w 0% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 4% | user 8% | | irq 0% | | | idle 62% | cpu023 w 26% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 3% | user 5% | | irq 0% | | | idle 73% | cpu021 w 18% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 3% | user 5% | | irq 0% | | | idle 76% | cpu013 w 16% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 3% | user 5% | | irq 0% | | | idle 66% | cpu018 w 27% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 4% | user 4% | | irq 0% | | | idle 75% | cpu020 w 17% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 4% | user 3% | | irq 0% | | | idle 53% | cpu016 w 40% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 3% | user 5% | | irq 0% | | | idle 77% | cpu022 w 15% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 3% | user 3% | | irq 0% | | | idle 86% | cpu019 w 8% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 2% | user 2% | | irq 0% | | | idle 86% | cpu015 w 9% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | CPL | avg1 7.71 | | avg5 4.53 | | avg15 2.82 | | | | csw 140083 | | intr 83288 | | | | numcpu 24 | MEM | tot 125.7G | free 8.2G | | cache 98.8G | dirty 755.1M | buff 76.8M | slab 6.9G | | slrec 4.2G | | shmem 0.1M | shrss 0.0M | | shswp 0.0M | | SWP | tot 27.8G | | free 27.8G | | | | | | | | | | vmcom 21.4G | | vmlim 90.6G | PAG | scan 0 | | steal 0 | stall 0 | | | | | | | | | swin 0 | | swout 0 | DSK | sde | | busy 89% | read 101 | | write 607 | KiB/r 4 | | KiB/w 250 | MBr/s 0.10 | | MBw/s 37.15 | avq 9.03 | | avio 5.02 ms | DSK | sdj | | busy 89% | read 79 | | write 687 | KiB/r 4 | | KiB/w 222 | MBr/s 0.08 | | MBw/s 37.24 | avq 10.14 | | avio 4.64 ms | DSK | sdh | | busy 84% | read 100 | | write 620 | KiB/r 4 | | KiB/w 245 | MBr/s 0.10 | | MBw/s 37.23 | avq 9.96 | | avio 4.65 ms | DSK | sdm | | busy 84% | read 90 | | write 640 | KiB/r 4 | | KiB/w 244 | MBr/s 0.09 | | MBw/s 38.22 | avq 9.40 | | avio 4.58 ms | DSK | sdg | | busy 83% | read 77 | | write 565 | KiB/r 4 | | KiB/w 254 | MBr/s 0.08 | | MBw/s 35.11 | avq 9.87 | | avio 5.13 ms | DSK | sdk | | busy 82% | read 93 | | write 645 | KiB/r 4 | | KiB/w 254 | MBr/s 0.09 | | MBw/s 40.14 | avq 7.68 | | avio 4.41 ms | DSK | sdi | | busy 79% | read 94 | | write 627 | KiB/r 4 | | KiB/w 255 | MBr/s 0.09 | | MBw/s 39.13 | avq 8.21 | | avio 4.36 ms | DSK | sdl | | busy 78% | read 101 | | write 595 | KiB/r 4 | | KiB/w 101 | MBr/s 0.10 | | MBw/s 14.78 | avq 44.87 | | avio 4.45 ms | DSK | sdf | | busy 64% | read 72 | | write 516 | KiB/r 4 | | KiB/w 256 | MBr/s 0.07 | | MBw/s 32.31 | avq 7.68 | | avio 4.33 ms | DSK | sdd | | busy 63% | read 70 | | write 442 | KiB/r 4 | | KiB/w 253 | MBr/s 0.07 | | MBw/s 27.39 | avq 7.70 | | avio 4.95 ms | DSK | sdc | | busy 9% | read 0 | | write 3124 | KiB/r 0 | | KiB/w 256 | MBr/s 0.00 | | MBw/s 195.40 | avq 11.89 | | avio 0.12 ms | DSK | sdb | | busy 9% | read 0 | | write 2959 | KiB/r 0 | | KiB/w 254 | MBr/s 0.00 | | MBw/s 183.51 | avq 10.80 | | avio 0.12 ms | DSK | sda | | busy 1% | read 0 | | write 7 | KiB/r 0 | | KiB/w 16 | MBr/s 0.00 | | MBw/s 0.03 | avq 1.17 | | avio 6.86 ms | NET | transport | tcpi 60798 | | tcpo 94293 | udpi 0 | udpo 0 | tcpao 0 | | tcppo 0 | tcprs 41 | tcpie 0 | tcpor 0 | | udpnp 0 | udpip 0 | NET | network | | ipi 60804 | ipo 72065 | | ipfrw 0 | deliv 60800 | | | | | | icmpi 2 | | icmpo 2 | NET | byn0 ---- | pcki 31728 | | pcko 31254 | si 2136 Mbps | | so 2173 Mbps | coll 0 | mlti 0 | | erri 0 | erro 0 | | drpi 0 | drpo 0 | NET | byn1 ---- | pcki 15078 | | pcko 26793 | si 3161 Kbps | | so 3090 Mbps | coll 0 | mlti 0 | | erri 0 | erro 0 | | drpi 0 | drpo 0 |
PRC | sys 1.01s | user 0.38s | | | #proc 373 | | #trun 2 | #tslpi 4317 | #tslpu 0 | | #zombie 0 | clones 4 | | | #exit 4 | CPU | sys 22% | user 10% | | irq 1% | | | idle 2365% | wait 3% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 13% | user 0% | | irq 0% | | | idle 87% | cpu001 w 0% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 5% | user 3% | | irq 0% | | | idle 92% | cpu012 w 0% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 1% | user 2% | | irq 0% | | | idle 97% | cpu005 w 0% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 1% | user 0% | | irq 1% | | | idle 98% | cpu000 w 0% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 0% | user 0% | | irq 0% | | | idle 99% | cpu014 w 0% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 0% | user 0% | | irq 0% | | | idle 97% | cpu002 w 3% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 0% | user 0% | | irq 0% | | | idle 100% | cpu007 w 0% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 0% | user 0% | | irq 0% | | | idle 100% | cpu003 w 0% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 0% | user 0% | | irq 0% | | | idle 100% | cpu004 w 0% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 0% | user 0% | | irq 0% | | | idle 100% | cpu006 w 0% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 0% | user 0% | | irq 0% | | | idle 100% | cpu008 w 0% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 0% | user 0% | | irq 0% | | | idle 100% | cpu009 w 0% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 0% | user 0% | | irq 0% | | | idle 100% | cpu010 w 0% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 0% | user 0% | | irq 0% | | | idle 100% | cpu011 w 0% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 0% | user 0% | | irq 0% | | | idle 100% | cpu013 w 0% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 0% | user 0% | | irq 0% | | | idle 100% | cpu015 w 0% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 0% | user 0% | | irq 0% | | | idle 100% | cpu017 w 0% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 0% | user 0% | | irq 0% | | | idle 100% | cpu018 w 0% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 0% | user 0% | | irq 0% | | | idle 100% | cpu019 w 0% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 0% | user 1% | | irq 0% | | | idle 99% | cpu021 w 0% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 0% | user 0% | | irq 0% | | | idle 100% | cpu016 w 0% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 0% | user 0% | | irq 0% | | | idle 100% | cpu020 w 0% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 0% | user 0% | | irq 0% | | | idle 100% | cpu022 w 0% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | cpu | sys 0% | user 0% | | irq 0% | | | idle 100% | cpu023 w 0% | | | steal 0% | guest 0% | | curf 2.10GHz | curscal ?% | CPL | avg1 5.56 | | avg5 4.65 | | avg15 2.97 | | | | csw 25250 | | intr 13901 | | | | numcpu 24 | MEM | tot 125.7G | free 3.7G | | cache 103.1G | dirty 0.6M | buff 76.8M | slab 7.0G | | slrec 4.4G | | shmem 0.1M | shrss 0.0M | | shswp 0.0M | | SWP | tot 27.8G | | free 27.8G | | | | | | | | | | vmcom 21.4G | | vmlim 90.6G | PAG | scan 0 | | steal 0 | stall 0 | | | | | | | | | swin 0 | | swout 0 | DSK | sdd | | busy 6% | read 0 | | write 37 | KiB/r 0 | | KiB/w 126 | MBr/s 0.00 | | MBw/s 1.14 | avq 5.80 | | avio 5.95 ms | DSK | sdl | | busy 1% | read 0 | | write 5 | KiB/r 0 | | KiB/w 24 | MBr/s 0.00 | | MBw/s 0.03 | avq 2.83 | | avio 9.60 ms | DSK | sda | | busy 1% | read 0 | | write 4 | KiB/r 0 | | KiB/w 14 | MBr/s 0.00 | | MBw/s 0.01 | avq 1.33 | | avio 6.00 ms | DSK | sdb | | busy 0% | read 0 | | write 79 | KiB/r 0 | | KiB/w 109 | MBr/s 0.00 | | MBw/s 2.10 | avq 1.00 | | avio 0.05 ms | DSK | sdc | | busy 0% | read 0 | | write 6 | KiB/r 0 | | KiB/w 2 | MBr/s 0.00 | | MBw/s 0.00 | avq 0.00 | | avio 0.00 ms | DSK | sdf | | busy 0% | read 0 | | write 0 | KiB/r 0 | | KiB/w 0 | MBr/s 0.00 | | MBw/s 0.00 | avq 0.00 | | avio 0.00 ms | DSK | sdg | | busy 0% | read 0 | | write 0 | KiB/r 0 | | KiB/w 0 | MBr/s 0.00 | | MBw/s 0.00 | avq 0.00 | | avio 0.00 ms | DSK | sdh | | busy 0% | read 0 | | write 0 | KiB/r 0 | | KiB/w 0 | MBr/s 0.00 | | MBw/s 0.00 | avq 0.00 | | avio 0.00 ms | DSK | sde | | busy 0% | read 0 | | write 0 | KiB/r 0 | | KiB/w 0 | MBr/s 0.00 | | MBw/s 0.00 | avq 0.00 | | avio 0.00 ms | DSK | sdi | | busy 0% | read 0 | | write 0 | KiB/r 0 | | KiB/w 0 | MBr/s 0.00 | | MBw/s 0.00 | avq 0.00 | | avio 0.00 ms | DSK | sdj | | busy 0% | read 0 | | write 0 | KiB/r 0 | | KiB/w 0 | MBr/s 0.00 | | MBw/s 0.00 | avq 0.00 | | avio 0.00 ms | DSK | sdk | | busy 0% | read 0 | | write 0 | KiB/r 0 | | KiB/w 0 | MBr/s 0.00 | | MBw/s 0.00 | avq 0.00 | | avio 0.00 ms | DSK | sdm | | busy 0% | read 0 | | write 0 | KiB/r 0 | | KiB/w 0 | MBr/s 0.00 | | MBw/s 0.00 | avq 0.00 | | avio 0.00 ms | NET | transport | tcpi 5362 | | tcpo 5317 | udpi 1 | udpo 0 | tcpao 0 | | tcppo 0 | tcprs 0 | tcpie 0 | tcpor 0 | | udpnp 0 | udpip 0 | NET | network | | ipi 5369 | ipo 5319 | | ipfrw 0 | deliv 5365 | | | | | | icmpi 2 | | icmpo 2 | NET | byn1 ---- | pcki 2563 | | pcko 2540 | si 790 Kbps | | so 983 Kbps | coll 0 | mlti 0 | | erri 0 | erro 0 | | drpi 0 | drpo 0 | NET | byn0 ---- | pcki 2453 | | pcko 2431 | si 557 Kbps | | so 992 Kbps | coll 0 | mlti 0 | | erri 0 | erro 0 | | drpi 0 | drpo 0 |
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
