Hi I have a jewel/xenial ceph installation with 61 OSDs mixed sas/sata in hosts with two roots The installation has version jewel 10.2.3-1xenial (and monitors) Two hosts where newly added and the version jewel 10.2.4-1xenial as installed These two hosts’ with the newer packages, ceph-osd processes use 300% cpu usage Second host seems more calm after the weekend but on Friday it was the same Attaching top, tree, config, ceph -s, and logs from osd.54 which is the same for all My next move is to get deb packages from other hosts and install these two hosts with 10.2.3 Any suggestions? Thank you 6242 ceph 20 0 1402008 290444 25196 S 304.3 0.2 13477:31 ceph-osd 7724 ceph 20 0 1446532 317972 24760 S 304.0 0.2 15575:49 ceph-osd 8256 ceph 20 0 1405440 305388 24884 S 302.0 0.2 15584:05 ceph-osd 6998 ceph 20 0 1414428 327496 24804 S 301.3 0.2 15502:35 ceph-osd 6643 ceph 20 0 1391592 256508 24840 S 301.0 0.2 15465:15 ceph-osd 5848 ceph 20 0 1423724 265708 24844 S 206.3 0.2 7649:34 ceph-osd 7456 ceph 20 0 1420584 303252 25408 S 206.0 0.2 13801:49 ceph-osd 8657 ceph 20 0 1242184 307412 28936 S 200.7 0.2 9604:28 ceph-osd 8699 ceph 20 0 1349152 394528 29688 S 200.3 0.3 7553:50 ceph-osd 8786 ceph 20 0 1303448 355868 27148 S 100.7 0.3 3786:07 ceph-osd 7593 ceph 20 0 1274436 272964 27444 S 703.9 0.2 29144:22 ceph-osd 5106 ceph 20 0 1532896 344476 25104 S 111.8 0.3 13437:23 ceph-osd 5222 ceph 20 0 1468644 370156 25340 S 20.6 0.3 2962:27 ceph-osd 5119 ceph 20 0 1500656 342368 24720 S 18.6 0.3 4971:34 ceph-osd 5132 ceph 20 0 1472920 326612 24212 S 18.6 0.2 2793:09 ceph-osd 5153 ceph 20 0 1422372 275816 24420 S 9.8 0.2 2944:55 ceph-osd 5279 ceph 20 0 1418700 299712 24932 S 3.9 0.2 7050:08 ceph-osd ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY -8 3.70183 root host-spin -5 0.76799 host-spin opstck-emc201-spin 8 0.22299 osd.8 up 1.00000 1.00000 15 0.27199 osd.15 up 1.00000 1.00000 24 0.27199 osd.24 up 1.00000 1.00000 -6 0.76799 host-spin opstck-emc202-spin 25 0.22299 osd.25 up 1.00000 1.00000 26 0.27199 osd.26 up 1.00000 1.00000 27 0.27199 osd.27 up 1.00000 1.00000 -7 0 host-spin opstck-emc203-spin -12 0.46696 host-spin opstck-emc209-spin 23 0.22298 osd.23 up 1.00000 1.00000 36 0.24399 osd.36 up 1.00000 1.00000 -10 0.48900 host-spin opstck-emc210-spin 35 0.24399 osd.35 up 1.00000 1.00000 34 0.24399 osd.34 up 1.00000 1.00000 -13 0.76949 host-spin opstck-emc205-spin 50 0.22459 osd.50 up 1.00000 1.00000 51 0.27245 osd.51 up 1.00000 1.00000 52 0.27245 osd.52 up 1.00000 1.00000 -15 0.44040 host-spin opstck-emc211-spin 53 0.19530 osd.53 up 1.00000 1.00000 54 0.24510 osd.54 up 1.00000 1.00000 -1 6.80861 root default -2 1.24399 host opstck-emc201 2 0.17200 osd.2 up 1.00000 1.00000 3 0.17200 osd.3 up 1.00000 1.00000 4 0.18199 osd.4 up 1.00000 1.00000 5 0.18199 osd.5 up 1.00000 1.00000 6 0.18199 osd.6 up 1.00000 1.00000 7 0.18199 osd.7 up 1.00000 1.00000 0 0.17200 osd.0 up 1.00000 1.00000 -3 1.24399 host opstck-emc202 1 0.17200 osd.1 up 1.00000 1.00000 9 0.17200 osd.9 up 1.00000 1.00000 10 0.17200 osd.10 up 1.00000 1.00000 11 0.18199 osd.11 up 1.00000 1.00000 12 0.18199 osd.12 up 1.00000 1.00000 13 0.18199 osd.13 up 1.00000 1.00000 14 0.18199 osd.14 up 1.00000 1.00000 -4 1.24399 host opstck-emc203 16 0.17200 osd.16 up 1.00000 1.00000 17 0.17200 osd.17 up 1.00000 1.00000 18 0.17200 osd.18 up 1.00000 1.00000 19 0.18199 osd.19 up 1.00000 1.00000 20 0.18199 osd.20 up 1.00000 1.00000 21 0.18199 osd.21 up 1.00000 1.00000 22 0.18199 osd.22 up 1.00000 1.00000 -11 0.81573 host opstck-emc209 37 0.18198 osd.37 up 1.00000 1.00000 38 0.18129 osd.38 up 1.00000 1.00000 42 0.09039 osd.42 up 1.00000 1.00000 41 0.09039 osd.41 up 1.00000 1.00000 39 0.18129 osd.39 up 1.00000 1.00000 40 0.09039 osd.40 up 1.00000 1.00000 -9 1.08600 host opstck-emc210 28 0.18100 osd.28 up 1.00000 1.00000 33 0.18100 osd.33 up 1.00000 1.00000 29 0.18100 osd.29 up 1.00000 1.00000 30 0.18100 osd.30 up 1.00000 1.00000 31 0.18100 osd.31 up 1.00000 1.00000 32 0.18100 osd.32 up 1.00000 1.00000 -14 0.63264 host opstck-emc205 43 0.09038 osd.43 up 1.00000 1.00000 44 0.09038 osd.44 up 1.00000 1.00000 45 0.09038 osd.45 up 1.00000 1.00000 46 0.09038 osd.46 up 1.00000 1.00000 47 0.09038 osd.47 up 1.00000 1.00000 48 0.09038 osd.48 up 1.00000 1.00000 49 0.09038 osd.49 up 1.00000 1.00000 -16 0.54227 host opstck-emc211 55 0.09038 osd.55 up 1.00000 1.00000 56 0.09038 osd.56 up 1.00000 1.00000 57 0.09038 osd.57 up 1.00000 1.00000 58 0.09038 osd.58 up 1.00000 1.00000 59 0.09038 osd.59 up 1.00000 1.00000 60 0.09038 osd.60 up 1.00000 1.00000 [global] filestore xattr use omap = true fsid = 238cee7d-8f8b-439c-8ffb-49488e8f1380 mon initial members = opstck-emc201,opstck-emc202,opstck-emc203 mon host = 192.168.124.149,192.168.124.150,192.168.124.151 auth cluster required = cephx auth service required = cephx auth client required = cephx osd journal size = 1024 osd pool default size = 2 osd pool default min size = 1 osd pool default pg num = 333 osd pool default pgp num = 333 osd crush chooseleaf type = 1 public network = 192.168.124.0/22 [mon.opstck-emc201] host = opstck-emc201 mon addr = 192.168.124.149:6789 [mon.opstck-emc202] host = opstck-emc202 mon addr = 192.168.124.150:6789 [mon.opstck-emc203] host = opstck-emc203 mon addr = 192.168.124.151:6789 [osd] osd crush update on start = false cluster 238cee7d-8f8b-439c-8ffb-49488e8f1380 health HEALTH_OK monmap e1: 3 mons at {opstck-emc201=192.168.124.149:6789/0,opstck-emc202=192.168.124.150:6789/0,opstck-emc203=192.168.124.151:6789/0} election epoch 26, quorum 0,1,2 opstck-emc201,opstck-emc202,opstck-emc203 osdmap e5277: 61 osds: 61 up, 61 in flags sortbitwise pgmap v2394621: 2560 pgs, 4 pools, 1660 GB data, 417 kobjects 3254 GB used, 7404 GB / 10658 GB avail 2560 active+clean client io 14126 B/s rd, 26935 kB/s wr, 2 op/s rd, 1582 op/s wr 2016-12-10 03:55:43.262110 7f138e81f700 1 leveldb: Compacting 4@0 + 4@1 files 2016-12-10 03:55:43.315348 7f138e81f700 1 leveldb: Generated table #119: 37467 keys, 2135218 bytes 2016-12-10 03:55:43.382277 7f138e81f700 1 leveldb: Generated table #120: 39570 keys, 2138113 bytes 2016-12-10 03:55:43.457026 7f138e81f700 1 leveldb: Generated table #121: 39776 keys, 2137116 bytes 2016-12-10 03:55:43.505193 7f138e81f700 1 leveldb: Generated table #122: 37782 keys, 2135704 bytes 2016-12-10 03:55:43.506256 7f138e81f700 1 leveldb: Generated table #123: 403 keys, 25351 bytes 2016-12-10 03:55:43.506275 7f138e81f700 1 leveldb: Compacted 4@0 + 4@1 files => 8571502 bytes 2016-12-10 03:55:43.506652 7f138e81f700 1 leveldb: compacted to: files[ 0 6 1 0 0 0 0 ] 2016-12-10 03:55:43.506859 7f138e81f700 1 leveldb: Delete type=2 #106 2016-12-10 03:55:43.507614 7f138e81f700 1 leveldb: Delete type=2 #107 2016-12-10 03:55:43.508325 7f138e81f700 1 leveldb: Delete type=2 #108 2016-12-10 03:55:43.508879 7f138e81f700 1 leveldb: Delete type=2 #109 2016-12-10 03:55:43.509431 7f138e81f700 1 leveldb: Delete type=2 #112 2016-12-10 03:55:43.509791 7f138e81f700 1 leveldb: Delete type=2 #114 2016-12-10 03:55:43.510285 7f138e81f700 1 leveldb: Delete type=2 #116 2016-12-10 03:55:43.510837 7f138e81f700 1 leveldb: Delete type=2 #118 2016-12-10 04:27:53.277922 7f1392259700 0 -- 192.168.124.159:6829/7818 >> 192.168.124.157:6825/4030 pipe(0x560134c83400 sd=26 :39864 s=2 pgs=1133 cs=43 l=0 c=0x5601348fcd80).fault with nothing to send, going to standby 2016-12-10 04:27:54.053701 7f138ff36700 0 -- 192.168.124.159:6829/7818 >> 192.168.124.157:6825/4030 pipe(0x56013925e800 sd=96 :6829 s=0 pgs=0 cs=0 l=0 c=0x560136281c00).accept connect_seq 43 vs existing 43 state standby 2016-12-10 04:27:54.054159 7f138ff36700 0 -- 192.168.124.159:6829/7818 >> 192.168.124.157:6825/4030 pipe(0x56013925e800 sd=96 :6829 s=0 pgs=0 cs=0 l=0 c=0x560136281c00).accept connect_seq 44 vs existing 43 state standby 2016-12-10 04:42:54.082431 7f138ff36700 0 -- 192.168.124.159:6829/7818 >> 192.168.124.157:6825/4030 pipe(0x56013925e800 sd=96 :6829 s=2 pgs=1138 cs=45 l=0 c=0x5601348fcd80).fault with nothing to send, going to standby 2016-12-10 05:16:24.885751 7f138f930700 0 -- 192.168.124.159:6829/7818 >> 192.168.124.150:6833/174449 pipe(0x560136595400 sd=60 :35242 s=2 pgs=1725 cs=19 l=0 c=0x560134bf7180).fault with nothing to send, going to standby 2016-12-10 05:25:16.629798 7f138e81f700 1 leveldb: Level-0 table #125: started 2016-12-10 05:25:16.657153 7f138e81f700 1 leveldb: Level-0 table #125: 1246290 bytes OK 2016-12-10 05:25:16.658440 7f138e81f700 1 leveldb: Delete type=0 #117 2016-12-10 05:42:51.080805 7f138ff36700 0 -- 192.168.124.159:6829/7818 >> 192.168.124.157:6825/4030 pipe(0x56013925e800 sd=96 :40290 s=2 pgs=1143 cs=47 l=0 c=0x5601348fcd80).fault with nothing to send, going to standby 2016-12-10 06:17:17.849436 7f138f72e700 0 -- 192.168.124.159:6829/7818 >> 192.168.124.153:6837/8786 pipe(0x56012bbe5400 sd=136 :59220 s=2 pgs=91 cs=5 l=0 c=0x56013666c580).fault with nothing to send, going to standby
George Kissandrakis Senior Infrastructure Engineer This e-mail is from Mapp Digital, LLC and it's international legal entities and may contain information that is confidential or proprietary. If you are not the intended recipient, do not read, copy or distribute the e-mail or any attachments. Instead, please notify the sender and delete the e-mail and any attachments. Thank you. Please consider the environment before printing. |
Attachment:
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
