OK, I will solve it if free. It's a problem when one connection want to replace existing connection for the same endpoint. Thank you! On Tue, Nov 4, 2014 at 10:41 AM, Alexandre DERUMIER <aderumier@xxxxxxxxx> wrote: >>>could you run "ceph -w --debug-ms=10/10" ? > > Here the output. (monitor ips are 10.11.1.27,10.11.1.28,10.11.1.29) > > # ceph -w --debug-ms=10/10 > 2014-11-04 10:38:16.155461 7fdf4414b700 10 EpollDriver.add_event add event to fd=4 mask=1 > 2014-11-04 10:38:16.155474 7fdf4414b700 10 Event create_file_event create event fd=4 mask=1 now mask is 1 > 2014-11-04 10:38:16.155626 7fdf4414b700 10 EpollDriver.add_event add event to fd=7 mask=1 > 2014-11-04 10:38:16.155630 7fdf4414b700 10 Event create_file_event create event fd=7 mask=1 now mask is 1 > 2014-11-04 10:38:16.155774 7fdf4414b700 10 -- :/0 ready :/0 > 2014-11-04 10:38:16.155783 7fdf4414b700 1 Processor -- start start > 2014-11-04 10:38:16.155785 7fdf4414b700 1 -- :/0 start start > 2014-11-04 10:38:16.155841 7fdf419e2700 10 --entry starting > 2014-11-04 10:38:16.155834 7fdf411e1700 10 --entry starting > 2014-11-04 10:38:16.155883 7fdf419e2700 10 Event process_events wait second 30 usec 0 > 2014-11-04 10:38:16.155899 7fdf411e1700 10 Event process_events wait second 30 usec 0 > 2014-11-04 10:38:16.156711 7fdf4414b700 10 -- :/1009064 create_connect 10.11.1.29:6789/0, creating connection and registering > 2014-11-04 10:38:16.156747 7fdf4414b700 10 -- :/1009064 >> 10.11.1.29:6789/0 conn(0x1f56090 sd=-1 :0 s=STATE_NONE pgs=0 cs=0 l=1)._connect 0 > 2014-11-04 10:38:16.156761 7fdf4414b700 1 Event wakeup > 2014-11-04 10:38:16.156774 7fdf4414b700 10 -- :/1009064 get_connection mon.2 10.11.1.29:6789/0 new 0x1f56090 > 2014-11-04 10:38:16.156793 7fdf4414b700 1 Event wakeup > 2014-11-04 10:38:16.156812 7fdf4414b700 10 -- :/1009064 >> 10.11.1.29:6789/0 conn(0x1f56090 sd=-1 :0 s=STATE_CONNECTING pgs=0 cs=0 l=1).send_message > 2014-11-04 10:38:16.157084 7fdf419e2700 10 EpollDriver.add_event add event to fd=9 mask=1 > 2014-11-04 10:38:16.157106 7fdf419e2700 10 Event create_file_event create event fd=9 mask=1 now mask is 1 > 2014-11-04 10:38:16.157136 7fdf419e2700 10 -- :/1009064 >> 10.11.1.29:6789/0 conn(0x1f56090 sd=9 :0 s=STATE_CONNECTING_WAIT_BANNER pgs=0 cs=0 l=1).handle_write started. > 2014-11-04 10:38:16.157199 7fdf419e2700 10 Event process_events wait second 30 usec 0 > 2014-11-04 10:38:16.157206 7fdf4414b700 1 Event wakeup > 2014-11-04 10:38:16.157259 7fdf419e2700 10 -- :/1009064 >> 10.11.1.29:6789/0 conn(0x1f56090 sd=9 :0 s=STATE_CONNECTING_WAIT_BANNER pgs=0 cs=0 l=1).handle_write started. > 2014-11-04 10:38:16.157284 7fdf419e2700 10 EpollDriver.add_event add event to fd=9 mask=3 > 2014-11-04 10:38:16.157286 7fdf419e2700 10 Event create_file_event create event fd=9 mask=2 now mask is 3 > 2014-11-04 10:38:16.157290 7fdf419e2700 10 Event process_events wait second 30 usec 0 > 2014-11-04 10:38:16.157306 7fdf419e2700 10 -- :/1009064 >> 10.11.1.29:6789/0 conn(0x1f56090 sd=9 :0 s=STATE_CONNECTING_WAIT_BANNER pgs=0 cs=0 l=1)._process_connection get banner, ready to send banner > 2014-11-04 10:38:16.157348 7fdf419e2700 10 -- :/1009064 >> 10.11.1.29:6789/0 conn(0x1f56090 sd=9 :0 s=STATE_CONNECTING_WAIT_IDENTIFY_PEER pgs=0 cs=0 l=1)._process_connection connect write banner done: 10.11.1.29:6789/0 > 2014-11-04 10:38:16.157376 7fdf419e2700 1 -- 10.11.1.27:0/1009064 learned_addr learned my addr 10.11.1.27:0/1009064 > 2014-11-04 10:38:16.157394 7fdf419e2700 10 -- 10.11.1.27:0/1009064 >> 10.11.1.29:6789/0 conn(0x1f56090 sd=9 :0 s=STATE_CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1)._process_connection connect sent my addr 10.11.1.27:0/1009064 > 2014-11-04 10:38:16.157415 7fdf419e2700 10 -- 10.11.1.27:0/1009064 >> 10.11.1.29:6789/0 conn(0x1f56090 sd=9 :0 s=STATE_CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1)._process_connection connect sending gseq=1 cseq=0 proto=15 > 2014-11-04 10:38:16.157434 7fdf419e2700 10 -- 10.11.1.27:0/1009064 >> 10.11.1.29:6789/0 conn(0x1f56090 sd=9 :0 s=STATE_CONNECTING_WAIT_CONNECT_REPLY pgs=0 cs=0 l=1).handle_write started. > 2014-11-04 10:38:16.157442 7fdf419e2700 10 Event process_events wait second 30 usec 0 > 2014-11-04 10:38:16.157446 7fdf419e2700 10 -- 10.11.1.27:0/1009064 >> 10.11.1.29:6789/0 conn(0x1f56090 sd=9 :0 s=STATE_CONNECTING_WAIT_CONNECT_REPLY pgs=0 cs=0 l=1).handle_write started. > 2014-11-04 10:38:16.157451 7fdf419e2700 10 Event process_events wait second 30 usec 0 > 2014-11-04 10:38:16.157560 7fdf419e2700 10 -- 10.11.1.27:0/1009064 >> 10.11.1.29:6789/0 conn(0x1f56090 sd=9 :0 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=1).handle_connect_replygot CEPH_MSGR_TAG_SEQ, reading acked_seq and writing in_seq > 2014-11-04 10:38:16.157580 7fdf419e2700 2 -- 10.11.1.27:0/1009064 >> 10.11.1.29:6789/0 conn(0x1f56090 sd=9 :0 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 18446744073709551615 vs out_seq 0 > 2014-11-04 10:38:16.157591 7fdf419e2700 2 -- 10.11.1.27:0/1009064 >> 10.11.1.29:6789/0 conn(0x1f56090 sd=9 :0 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection discarding previously sent 0 auth(proto 0 34 bytes epoch 0) v1 > msg/async/AsyncConnection.cc: In function 'int AsyncConnection::_process_connection()' thread 7fdf419e2700 time 2014-11-04 10:38:16.157609 > msg/async/AsyncConnection.cc: 1046: FAILED assert(m) > ceph version 0.87-600-g25ca92d (25ca92d7ff5cc326d77c1c3f687ece0718f02db9) > 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x72) [0x7fdf459d0b52] > 2: (AsyncConnection::_process_connection()+0x40e9) [0x7fdf45ae8ac9] > 3: (AsyncConnection::process()+0xcb) [0x7fdf45ae92ab] > 4: (EventCenter::process_events(int)+0x567) [0x7fdf45b02a27] > 5: (Worker::entry()+0x7f) [0x7fdf45af01cf] > 6: (()+0x6b50) [0x7fdf4a1f2b50] > 7: (clone()+0x6d) [0x7fdf4969c7bd] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. > terminate called after throwing an instance of 'ceph::FailedAssertion' > Abandon > ----- Mail original ----- > > De: "Haomai Wang" <haomaiwang@xxxxxxxxx> > À: "Alexandre DERUMIER" <aderumier@xxxxxxxxx> > Cc: "Somnath Roy" <Somnath.Roy@xxxxxxxxxxx>, ceph-devel@xxxxxxxxxxxxxxx, "Sage Weil" <sage@xxxxxxxxxxxx> > Envoyé: Mardi 4 Novembre 2014 10:29:46 > Objet: Re: The Async messenger benchmark with latest master > > could you run "ceph -w --debug-ms=10/10" ? > > On Tue, Nov 4, 2014 at 5:29 AM, Alexandre DERUMIER <aderumier@xxxxxxxxx> wrote: >> Hi, >> >> I'm try to test the async messenger (ms_type = async) with last master, >> and I have an assert on line 1046 of AsyncConnection.cc. >> (I don't have tried yet with previous builds) >> >> no tuning in ceph.conf, only the ms_type = async. >> >> >> >> >> #ceph -w >> msg/async/AsyncConnection.cc: In function 'int AsyncConnection::_process_connection()' thread 7fd396201700 time 2014-11-04 05:22:10.878492 >> msg/async/AsyncConnection.cc: 1046: FAILED assert(m) >> ceph version 0.87-600-g25ca92d (25ca92d7ff5cc326d77c1c3f687ece0718f02db9) >> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x72) [0x7fd39a1efb52] >> 2: (AsyncConnection::_process_connection()+0x40e9) [0x7fd39a307ac9] >> 3: (AsyncConnection::process()+0xcb) [0x7fd39a3082ab] >> 4: (EventCenter::process_events(int)+0x567) [0x7fd39a321a27] >> 5: (Worker::entry()+0x7f) [0x7fd39a30f1cf] >> 6: (()+0x6b50) [0x7fd39ea11b50] >> 7: (clone()+0x6d) [0x7fd39debb7bd] >> >> >> Any idea ? >> >> ----- Mail original ----- >> >> De: "Somnath Roy" <Somnath.Roy@xxxxxxxxxxx> >> À: "Sage Weil" <sage@xxxxxxxxxxxx>, "Haomai Wang" <haomaiwang@xxxxxxxxx> >> Cc: ceph-devel@xxxxxxxxxxxxxxx >> Envoyé: Mercredi 22 Octobre 2014 22:34:34 >> Objet: RE: The Async messenger benchmark with latest master >> >> Sage, >> So, with krbd as client, should I use 'ms_type = random' in the cluster side to test Async messenger in the cluster side? >> BTW, what this option does ? It will pick Async first and fall back to old messenger in case any problem ? >> >> Thanks & Regards >> Somnath >> >> -----Original Message----- >> From: Sage Weil [mailto:sage@xxxxxxxxxxxx] >> Sent: Wednesday, October 22, 2014 10:53 AM >> To: Haomai Wang >> Cc: Somnath Roy; ceph-devel@xxxxxxxxxxxxxxx >> Subject: Re: The Async messenger benchmark with latest master >> >> On Thu, 23 Oct 2014, Haomai Wang wrote: >>> It should be but I'm not convinced of the current impl of >>> AsyncMessenger. It's possible that exists some corner situations which >>> may break >> >> The 'ms_type = random' option we added should give us some confidnece that this is true (by doing QA on mixes old both messenger implementations). >> >> sage >> >> >>> >>> On Thu, Oct 23, 2014 at 1:18 AM, Somnath Roy <Somnath.Roy@xxxxxxxxxxx> wrote: >>> > Thanks Haomai, I will try this branch out. >>> > BTW, will it be compatible with the client which is not using this async messenger for example krbd ? >>> > >>> > Regards >>> > Somnath >>> > >>> > -----Original Message----- >>> > From: Haomai Wang [mailto:haomaiwang@xxxxxxxxx] >>> > Sent: Wednesday, October 22, 2014 10:02 AM >>> > To: Somnath Roy >>> > Cc: ceph-devel@xxxxxxxxxxxxxxx >>> > Subject: Re: The Async messenger benchmark with latest master >>> > >>> > ms_type = async is needed to add [global] section in ceph.conf >>> > >>> > No other options are needed >>> > >>> > On Thu, Oct 23, 2014 at 12:02 AM, Somnath Roy <Somnath.Roy@xxxxxxxxxxx> wrote: >>> >> Hi Haomai/Sage, >>> >> >>> >> I just figured it out that Async messenger is not yet in master. >>> >> So, I didn't really test the async messenger :-( >>> >> >>> >> I saw here is the pull request. >>> >> >>> >> https://github.com/ceph/ceph/pull/2768 >>> >> >>> >> I want to test this out and I have some questions regarding this. >>> >> >>> >> 1. The branch I should test out is the following, right ? >>> >> >>> >> https://github.com/ceph/ceph/tree/wip-msgr >>> >> >>> >> 2. Will Async messenger by default enabled ? OR I need to add some config option for that ? >>> >> >>> >> 3. Other than ms_event_op_threads , is there any tunable parameter I should be playing with ? >>> >> >>> >> Thanks & Regards >>> >> Somnath >>> >> >>> >> -----Original Message----- >>> >> From: Haomai Wang [mailto:haomaiwang@xxxxxxxxx] >>> >> Sent: Saturday, October 18, 2014 10:15 PM >>> >> To: Somnath Roy >>> >> Cc: ceph-devel@xxxxxxxxxxxxxxx >>> >> Subject: Re: The Async messenger benchmark with latest master >>> >> >>> >> Thanks Somnath! >>> >> >>> >> I have another simple performance test for async messenger: >>> >> >>> >> For 4k object read, master branch used 4.46s to complete tests, async Messenger used 3.14s For 4k object write, master branch used 10.6s to complete, async Messenger used 6.6s!! >>> >> >>> >> Detailf results see below, 4k object read is a simple ceph client >>> >> program which will read 5000 objects and 4k object write will write >>> >> 5000 objects. >>> >> I increased "ms_event_op_threads" to 10 compared to the default value is 2. >>> >> Maybe Somnath can do it and tests again, I think we can get more improvements for your tests. >>> >> >>> >> Master Branch(6fa686c8c42937dd069591f16de92e954d8ed34d): >>> >> >>> >> [root@ceph-test src]# for i in `seq 1 3`; do date && >>> >> ~/08.rados_read_4k_for_each_object /etc/ceph/ceph.conf rbd 5000 && >>> >> sleep 3; done >>> >> >>> >> Fri Oct 17 17:10:39 UTC 2014 >>> >> >>> >> Used Time:4.461581 >>> >> >>> >> Fri Oct 17 17:10:44 UTC 2014 >>> >> >>> >> Used Time:4.388572 >>> >> >>> >> Fri Oct 17 17:10:48 UTC 2014 >>> >> >>> >> Used Time:4.448157 >>> >> >>> >> [root@ceph-test src]# >>> >> >>> >> [root@ceph-test src]# for i in `seq 1 3`; do date && >>> >> ~/01.rados_write_4k_for_each_object /etc/ceph/ceph.conf rbd 5000 && >>> >> sleep 3; doneFri Oct 17 17:11:23 UTC 2014 >>> >> >>> >> Used Time:10.638783 >>> >> >>> >> Fri Oct 17 17:11:33 UTC 2014 >>> >> >>> >> Used Time:10.793231 >>> >> >>> >> Fri Oct 17 17:11:44 UTC 2014 >>> >> >>> >> Used Time:10.908003 >>> >> >>> >> >>> >> Master Branch with AsyncMessenger: >>> >> >>> >> [root@ceph-test src]# for i in `seq 1 3`; do date && >>> >> ~/08.rados_read_4k_for_each_object /etc/ceph/ceph.conf rbd 5000 && >>> >> sleep 3; done >>> >> >>> >> Sun Oct 19 06:01:50 UTC 2014 >>> >> >>> >> Used Time:3.155506 >>> >> >>> >> Sun Oct 19 06:01:53 UTC 2014 >>> >> >>> >> Used Time:3.134961 >>> >> >>> >> Sun Oct 19 06:01:56 UTC 2014 >>> >> >>> >> Used Time:3.135814 >>> >> >>> >> [root@ceph-test src]# for i in `seq 1 3`; do date && >>> >> ~/01.rados_write_4k_for_each_object /etc/ceph/ceph.conf rbd 5000 && >>> >> sleep 3; done >>> >> >>> >> Sun Oct 19 06:02:03 UTC 2014 >>> >> >>> >> Used Time:6.536319 >>> >> >>> >> Sun Oct 19 06:02:10 UTC 2014 >>> >> >>> >> Used Time:6.648738 >>> >> >>> >> Sun Oct 19 06:02:16 UTC 2014 >>> >> >>> >> Used Time:6.585156 >>> >> >>> >> On Sat, Oct 18, 2014 at 4:37 AM, Somnath Roy <Somnath.Roy@xxxxxxxxxxx> wrote: >>> >>> Hi Sage/Haomai, >>> >>> >>> >>> I did some 4K Random Read benchmarking with latest master having Async messenger changes and result looks promising. >>> >>> >>> >>> My configuration: >>> >>> --------------------- >>> >>> >>> >>> 1 node, 8 SSDs, 8OSDs, 3 pools , each has 3 images. ~2000 Pg >>> >>> cluster wide Cpu : Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz, dual socket, HT enabled, 40 cores. >>> >>> Used krbd as client. >>> >>> >>> >>> >>> >>> 1 clinet node with 3 rbd images on 3 different pools: >>> >>> ------------------------------------------------------------------ >>> >>> - >>> >>> Master : >>> >>> --------- >>> >>> >>> >>> ~203k IOPS, ~90% latencies within 4msec, total read : ~5.2TB >>> >>> >>> >>> lat (usec) : 4=0.01%, 10=0.01%, 50=0.01%, 100=0.01%, 250=0.03% >>> >>> lat (usec) : 500=0.86%, 750=3.72%, 1000=7.19% >>> >>> lat (msec) : 2=39.17%, 4=41.45%, 10=7.26%, 20=0.24%, 50=0.06% >>> >>> lat (msec) : 100=0.03%, 250=0.01%, 500=0.01% >>> >>> >>> >>> cpu: ~1-2 % idle >>> >>> >>> >>> Giant: >>> >>> ------ >>> >>> >>> >>> ~196k IOPS, ~89-90% latencies within 4msec, total read : ~5.3TB >>> >>> >>> >>> lat (usec) : 250=0.01%, 500=0.51%, 750=2.71%, 1000=6.03% >>> >>> lat (msec) : 2=34.32%, 4=45.74%, 10=10.46%, 20=0.21%, 50=0.02% >>> >>> lat (msec) : 100=0.01%, 250=0.01% >>> >>> >>> >>> cpu: ~2% idle >>> >>> >>> >>> 2 clients with 3 rbd images each on 3 different pools: >>> >>> ------------------------------------------------------------------ >>> >>> - >>> >>> >>> >>> Master : >>> >>> --------- >>> >>> >>> >>> ~207K iops, ~70% latencies within 4msec, Total read: ~5.99 TB >>> >>> >>> >>> lat (usec) : 250=0.03%, 500=0.63%, 750=2.67%, 1000=5.21% >>> >>> lat (msec) : 2=25.12%, 4=36.19%, 10=24.80%, 20=3.16%, 50=1.34% >>> >>> lat (msec) : 100=0.66%, 250=0.18%, 500=0.01% >>> >>> >>> >>> cpu: ~0-1 % idle >>> >>> >>> >>> Giant: >>> >>> -------- >>> >>> ~199K iops, ~64% latencies within 4msec, Total read: ~5.94 TB >>> >>> >>> >>> lat (usec) : 250=0.01%, 500=0.25%, 750=1.47%, 1000=3.45% >>> >>> lat (msec) : 2=21.22%, 4=36.69%, 10=30.63%, 20=4.28%, 50=1.70% >>> >>> lat (msec) : 100=0.28%, 250=0.02%, 500=0.01% >>> >>> >>> >>> cpu: ~1% idle >>> >>> >>> >>> >>> >>> So, in summary the master with Async messenger has improved both in iops and latency. >>> >>> >>> >>> Thanks & Regards >>> >>> Somnath >>> >>> >>> >>> ________________________________ >>> >>> >>> >>> PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies). >>> >>> >>> >>> -- >>> >>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" >>> >>> in the body of a message to majordomo@xxxxxxxxxxxxxxx More >>> >>> majordomo info at http://vger.kernel.org/majordomo-info.html >>> >> >>> >> >>> >> >>> >> -- >>> >> Best Regards, >>> >> >>> >> Wheat >>> > >>> > >>> > >>> > -- >>> > Best Regards, >>> > >>> > Wheat >>> >>> >>> >>> -- >>> Best Regards, >>> >>> Wheat >>> -- >>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" >>> in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo >>> info at http://vger.kernel.org/majordomo-info.html >>> >>> >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> the body of a message to majordomo@xxxxxxxxxxxxxxx >> More majordomo info at http://vger.kernel.org/majordomo-info.html > > > > -- > Best Regards, > > Wheat -- Best Regards, Wheat -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
