On 18 October 2012 08:44, Ling Ho <ling at slac.stanford.edu> wrote: > When you mount using rdma, try running some network tools like iftop to see > if traffics is going through your Ge interface. > > If your volume is created with both tcp and rdma, my experience is rdma does > not work under 3.3.0 and it will always fall back to tcp. > > However ipoib works fine for us. Again you should check where the traffics > go. I used tcpdump and iftop and confirmed the traffic using IPoIB goes through ib0 interface, not eth. When I use rdma transport the traffic doesn't show on neither eth nor ib0 interface - so I guess it's correctly using RDMA. I re-ran the iozone -a tests and they're the same. In addition I did a "dd" test fo read and write on IPoIB and RDMA mounted volume. IPoIB: [root at master gluster3]# dd if=/dev/zero of=test bs=100M count=50 50+0 przeczytanych record?w 50+0 zapisanych record?w skopiowane 5242880000 bajt?w (5,2 GB), 16,997 s, 308 MB/s [root at master gluster3]# dd if=test of=/dev/null 10240000+0 przeczytanych record?w 10240000+0 zapisanych record?w skopiowane 5242880000 bajt?w (5,2 GB), 28,4185 s, 184 MB/s RDMA: [root at master gluster]# dd if=/dev/zero of=test bs=100M count=50 50+0 przeczytanych record?w 50+0 zapisanych record?w skopiowane 5242880000 bajt?w (5,2 GB), 70,3636 s, 74,5 MB/s [root at master gluster]# dd if=test of=/dev/null 10240000+0 przeczytanych record?w 10240000+0 zapisanych record?w skopiowane 5242880000 bajt?w (5,2 GB), 10,8389 s, 484 MB/s I did a "sync" between those tests. Funny isn't it? RDMA is much slower than IPoIB on writing and faster on reading. I think we'll stick to the IPoIB until something's fixed in glusterfs. And still - why the results are so similar to Gigabit Ethernet? Regards -- Bartek Krawczyk network and system administrator
