If I do a test on a 3 node, 1 osd per node cluster, 2xGbE mode 4 bonded, on a pool with size 1. I see that the first node is sending streams to 2nd and 3rd node using only one of the bonded adapters. This is typical for a 'single line of communication' using lacp. Afaik the streams to the nodes are independent from each other anyway, so maybe it is possible to 'fork' the transmitting process, so linux can detect it as a separate stream and thus use the other link. 1st node rados bench -p data1 30 write --net/eth0----net/eth1- --dsk/sda-----dsk/sdb-----dsk/sdc-- recv send: recv send| read writ: read writ: read writ 809B 119k:1880B 0 | 0 0 : 0 0 : 0 0 1516B 2163B: 895B 0 | 0 0 : 0 0 : 0 0 42k 56M: 64k 0 | 0 0 : 0 44M: 0 0 94k 94M: 73k 0 | 0 20k: 0 88M: 0 0 104k 119M: 100k 0 | 0 32k: 0 76M: 0 0 108k 119M: 96k 0 | 0 0 : 0 80M: 0 0 125k 119M: 102k 0 | 0 38k: 0 72M: 0 0 105k 119M: 117k 0 | 0 60k: 0 94M: 0 0 135k 119M: 86k 0 | 0 105k: 0 71M: 0 0 79k 119M: 122k 0 | 0 33k: 0 84M: 0 0 129k 115M: 57k 0 | 0 0 : 0 64M: 0 0 139k 113M: 71k 0 | 0 69k: 0 60M: 0 0 109k 119M: 106k 0 | 0 74k: 0 100M: 0 0 96k 119M: 120k 0 | 0 77k: 0 84M: 0 0 122k 119M: 71k 0 | 0 33k: 0 84M: 0 0 115k 119M: 83k 0 | 0 0 : 0 92M: 0 0 125k 119M: 98k 0 | 0 61k: 0 92M: 0 0 71k 111M: 123k 0 | 0 73k: 0 104M: 0 0 95k 119M: 115k 0 | 0 81k: 0 84M: 0 0 74k 104M: 96k 0 | 0 33k: 0 98M: 0 0 118k 109M: 57k 0 | 0 0 : 0 88M: 0 0 134k 119M: 81k 0 | 0 57k: 0 87M: 0 0 55k 83M: 92k 124B| 0 196k: 0 56M: 0 0 113k 116M: 96k 0 | 0 72k: 0 76M: 0 0 57k 112M: 128k 0 | 0 32k: 0 96M: 0 0 61k 116M: 129k 0 | 0 0 : 0 89M: 0 0 114k 119M: 108k 0 | 0 57k: 0 92M: 0 0 93k 119M: 121k 0 | 0 72k: 0 92M: 0 0 86k 113M: 118k 0 | 0 76k: 0 98M: 0 0 82k 112M: 105k 0 | 0 32k: 0 96M: 0 0 85k 104M: 85k 0 | 0 0 : 0 87M: 0 0 122k 119M: 99k 0 | 0 61k: 0 102M: 0 0 52k 61M: 65k 0 | 0 76k: 0 39M: 0 0 29k 171k: 46k 0 | 0 76k: 0 280k: 0 0 67k 173k: 54k 0 | 0 32k: 0 648k: 0 0 2node --net/eth0----net/eth1- --dsk/sda-----dsk/sdb-----dsk/sdc-----dsk/sdd-- recv send: recv send| read writ: read writ: read writ: read writ 578B 1568B: 66B 0 | 0 0 : 0 0 : 0 0 : 0 0 1332B 105k: 66B 0 | 0 0 : 0 0 : 0 0 : 0 0 25M 43k: 66B 0 | 0 16k: 0 0 : 0 20M: 0 0 54M 100k: 66B 0 | 0 26k: 0 0 : 0 56M: 0 0 61M 105k: 66B 0 | 0 44k: 0 0 : 0 60M: 0 0 63M 108k: 66B 0 | 0 0 : 0 0 : 0 64M: 0 0 59M 230k: 66B 0 | 0 8192B: 0 0 : 0 56M: 0 0 59M 106k: 66B 0 | 0 85k: 0 0 : 0 60M: 0 0 78M 151k: 66B 0 | 0 148k: 0 0 : 0 78M: 0 0 44M 79k: 66B 124B| 0 40k: 0 0 : 0 47M: 0 0 83M 131k: 66B 0 | 0 0 : 0 0 : 0 76M: 0 0 71M 243k: 66B 0 | 0 76k: 0 0 : 0 76M: 0 0 66M 108k: 66B 0 | 0 100k: 0 0 : 0 64M: 0 0 58M 113k: 66B 0 | 0 104k: 0 0 : 0 56M: 0 0 78M 123k: 66B 0 | 0 40k: 0 0 : 0 76M: 0 0 71M 116k: 66B 0 | 0 0 : 0 0 : 0 72M: 0 0 62M 230k: 66B 0 | 0 84k: 0 0 : 0 60M: 0 0 43M 72k: 66B 0 | 0 100k: 0 0 : 0 44M: 0 0 57M 110k: 66B 0 | 0 104k: 0 0 : 0 56M: 0 0 46M 76k: 66B 0 | 0 44k: 0 0 : 0 48M: 0 0 75M 118k: 190B 0 | 0 0 : 0 0 : 0 72M: 0 0 70M 239k: 66B 0 | 0 81k: 0 0 : 0 68M: 0 0 32M 54k: 66B 0 | 0 97k: 0 0 : 0 36M: 0 0 70M 128k: 66B 0 | 0 101k: 0 0 : 0 68M: 0 0 31M 58k: 66B 0 | 0 40k: 0 0 : 0 32M: 0 0 36M 62k: 66B 0 | 0 0 : 0 0 : 0 32M: 0 0 53M 217k: 66B 0 | 0 77k: 0 0 : 0 56M: 0 0 54M 94k: 66B 0 | 0 235k: 0 0 : 0 53M: 0 0 50M 102k: 66B 0 | 0 101k: 0 0 : 0 48M: 0 0 48M 83k: 66B 0 | 0 40k: 0 0 : 0 48M: 0 0 52M 86k: 66B 0 | 0 0 : 0 0 : 0 48M: 0 0 57M 227k: 132B 0 | 0 88k: 0 0 : 0 60M: 0 0 29M 52k: 66B 0 | 0 100k: 0 0 : 0 32M: 0 0 87k 45k: 66B 0 | 0 100k: 0 0 :4096B 264k: 0 0 94k 68k: 66B 0 | 0 40k: 0 0 : 0 604k: 0 0 ^N 45k 42k: 294B 0 | 0 0 : 0 0 : 0 428k: 0 0 3rd node --net/eth0----net/eth1- --dsk/sda-----dsk/sdb-----dsk/sdc-----dsk/sdd-- recv send: recv send| read writ: read writ: read writ: read writ 856B 1776B: 850B 0 | 0 0 : 0 0 : 0 0 : 0 0 66B 1616B: 118k 0 | 0 0 : 0 0 : 0 0 : 0 0 104k 2567B:1203B 0 | 0 4096B: 0 0 : 0 0 : 0 0 66B 25k:6713k 0 | 0 0 : 0 0 : 0 4096k: 0 0 4821B 84k: 46M 0 | 0 25k: 0 0 : 0 48M: 0 0 724B 95k: 56M 0 | 0 44k: 0 0 : 0 52M: 0 0 66B 97k: 56M 0 | 0 0 : 0 0 : 0 56M: 0 0 103k 104k: 59M 0 | 0 120k: 0 0 : 0 60M: 0 0 2172B 120k: 59M 0 | 0 81k: 0 0 : 0 60M: 0 0 15k 100k: 48M 0 | 0 157k: 0 0 : 0 49M: 0 0 856B 92k: 52M 0 | 0 42k: 0 0 : 0 48M: 0 0 66B 101k: 62M 0 | 0 0 : 0 0 : 0 52M: 0 0 103k 56k: 31M 0 | 0 0 : 0 0 : 0 44M: 0 0 784B 100k: 51M 0 | 0 78k: 0 0 : 0 48M: 0 0 16k 122k: 61M 0 | 0 107k: 0 0 : 0 60M: 0 0 997B 79k: 44M 0 | 0 107k: 0 0 : 0 44M: 0 0 838B 80k: 46M 0 | 0 41k: 0 0 : 0 44M: 0 0 105k 99k: 57M 0 | 0 0 : 0 0 : 0 56M: 0 0 66B 121k: 65M 0 | 0 86k: 0 0 : 0 64M: 0 0 16k 114k: 60M 124B| 0 103k: 0 0 : 0 60M: 0 0 207B 109k: 67M 0 | 0 115k: 0 0 : 0 68M: 0 0 66B 53k: 30M 0 | 0 46k: 0 0 : 0 32M: 0 0 105k 95k: 57M 0 | 0 0 : 0 0 : 0 56M: 0 0 66B 88k: 48M 0 | 0 82k: 0 0 : 0 16M: 0 0 15k 103k: 51M 0 | 0 99k: 0 0 : 0 80M: 0 0 1138B 97k: 58M 0 | 0 111k: 0 0 : 0 57M: 0 0 66B 136k: 85M 0 | 0 42k: 0 0 : 0 83M: 0 0 103k 114k: 69M 0 | 0 0 : 0 0 : 0 72M: 0 0 250B 120k: 65M 0 | 0 80k: 0 0 : 0 64M: 0 0 15k 119k: 63M 0 | 0 103k: 0 0 : 0 63M: 0 0 997B 107k: 64M 0 | 0 119k: 0 0 : 0 65M: 0 0 66B 92k: 56M 0 | 0 43k: 0 0 : 0 52M: 0 0 104k 100k: 61M 0 | 0 0 : 0 0 : 0 60M: 0 0 66B 99k: 52M 0 | 0 86k: 0 0 : 0 35M: 0 0 16k 26k: 58k 0 | 0 106k: 0 0 : 0 21M: 0 0 207B 59k: 116k 0 | 0 103k: 0 0 : 0 596k: 0 0 724B 62k: 205k 0 | 0 61k: 0 0 : 0 616k: 0 0 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -. F1 Outsourcing Development Sp. z o.o. Poland t: +48 (0)124466845 f: +48 (0)124466843 e: marc@xxxxxxxxxxxxxxxxx _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |