Thanks, Jens. That works. I did a little more code alignment for more
comfortable reading, and it's attached. (Yahoo mail replaced the
previous patch some white spaces :-)
Here is some sample test result. RDMA memory semantic is one side
operation which means the remote side doesn't know what's the
throughput without extra notification. Now, the client only send the
finish message to the server side after transfer completion, so the
server side statistics for memory semantic are empty.
Memory semantic: RDMA_WRITE
---
Client job file:
# Example rdma client job
[global]
ioengine=rdma
filename=192.168.2.4/18301/RDMA_WRITE
bs=1m
size=100g
[sender]
rw=write
iodepth=1
iodepth_batch_complete=1
$ ./fio examples/rdmaio-client
sender: (g=0): rw=write, bs=1M-1M/1M-1M, ioengine=rdma, iodepth=1
fio 1.57
Starting 1 process
Jobs: 1 (f=1): [W] [100.0% done] [0K/2984M /s] [0 /2914 iops] [eta 00m:00s]
sender: (groupid=0, jobs=1): err= 0: pid=16035
write: io=102400MB, bw=3016.5MB/s, iops=3016 , runt= 33947msec
slat (usec): min=0 , max=9 , avg= 0.65, stdev= 0.50
clat (usec): min=313 , max=36473 , avg=328.97, stdev=533.22
lat (usec): min=316 , max=36474 , avg=329.83, stdev=533.22
bw (KB/s) : min=2693120, max=3190784, per=100.00%, avg=3088837.25,
stdev=122970.35
cpu : usr=1.41%, sys=0.38%, ctx=102435, majf=0, minf=11
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued r/w/d: total=0/102400/0, short=0/0/0
lat (usec): 500=99.96%, 750=0.01%
lat (msec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.03%
Run status group 0 (all jobs):
WRITE: io=102400MB, aggrb=3016.5MB/s, minb=3088.9MB/s,
maxb=3088.9MB/s, mint=33947msec, maxt=33947msec
Server job file:# Example rdma server job
[global]
ioengine=rdma
filename=192.168.2.4/18301
bs=1m
size=100g
[receiver]
rw=read
iodepth=16
$ ./fio examples/rdmaio-server
receiver: (g=0): rw=read, bs=1M-1M/1M-1M, ioengine=rdma, iodepth=16
fio 1.57
Starting 1 process
Jobs: 1 (f=1): [R] [477.8% done] [0K/0K /s] [0 /0 iops] [eta
1158050441d:06h:59m:42s]
receiver: (groupid=0, jobs=1): err= 0: pid=4852
cpu : usr=0.00%, sys=0.00%, ctx=0, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued r/w/d: total=1/0/0, short=0/0/0
Run status group 0 (all jobs):
Memory semantic: RDMA_READ
---
Client job file:
# Example rdma client job
[global]
ioengine=rdma
filename=192.168.2.4/18301/RDMA_READ
bs=1m
size=100g
[sender]
rw=write
iodepth=1
iodepth_batch_complete=1
$ ./fio examples/rdmaio-client
sender: (g=0): rw=write, bs=1M-1M/1M-1M, ioengine=rdma, iodepth=1
fio 1.57
Starting 1 process
Jobs: 1 (f=1): [W] [100.0% done] [0K/3049M /s] [0 /2977 iops] [eta 00m:00s]
sender: (groupid=0, jobs=1): err= 0: pid=28516
write: io=102400MB, bw=2968.2MB/s, iops=2968 , runt= 34500msec
slat (usec): min=0 , max=8 , avg= 0.68, stdev= 0.49
clat (usec): min=314 , max=37021 , avg=334.32, stdev=123.70
lat (usec): min=315 , max=37028 , avg=335.20, stdev=123.72
bw (KB/s) : min=2738176, max=3049472, per=100.03%, avg=3040225.88,
stdev=38989.13
cpu : usr=1.15%, sys=0.43%, ctx=102437, majf=0, minf=11
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued r/w/d: total=0/102400/0, short=0/0/0
lat (usec): 500=99.98%, 750=0.01%, 1000=0.01%
lat (msec): 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
Run status group 0 (all jobs):
WRITE: io=102400MB, aggrb=2968.2MB/s, minb=3039.4MB/s,
maxb=3039.4MB/s, mint=34500msec, maxt=34500msec
Server job file is the same as RDMA_WRITE.
$ ./fio examples/rdmaio-server
receiver: (g=0): rw=read, bs=1M-1M/1M-1M, ioengine=rdma, iodepth=16
fio 1.57
Starting 1 process
Jobs: 1 (f=1): [R] [683.3% done] [0K/0K /s] [0 /0 iops] [eta
1158050441d:06h:59m:41s]
receiver: (groupid=0, jobs=1): err= 0: pid=6036
cpu : usr=0.00%, sys=0.00%, ctx=0, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued r/w/d: total=1/0/0, short=0/0/0
Run status group 0 (all jobs):
Channel Semantic: SEND/RECV
----
Client job file:
# Example rdma client job
[global]
ioengine=rdma
filename=192.168.2.4/18301/SEND
bs=1m
size=100g
[sender]
rw=write
iodepth=1
iodepth_batch_complete=1
$ ./fio examples/rdmaio-client
sender: (g=0): rw=write, bs=1M-1M/1M-1M, ioengine=rdma, iodepth=1
fio 1.57
Starting 1 process
Jobs: 1 (f=1): [W] [100.0% done] [0K/3099M /s] [0 /3025 iops] [eta 00m:00s]
sender: (groupid=0, jobs=1): err= 0: pid=29015
write: io=102400MB, bw=2995.7MB/s, iops=2995 , runt= 34183msec
slat (usec): min=0 , max=6 , avg= 0.54, stdev= 0.51
clat (usec): min=313 , max=36231 , avg=331.38, stdev=573.03
lat (usec): min=317 , max=36232 , avg=332.13, stdev=573.04
bw (KB/s) : min=2536864, max=3178496, per=100.02%, avg=3068169.09,
stdev=146953.37
cpu : usr=1.40%, sys=0.46%, ctx=102446, majf=0, minf=10
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued r/w/d: total=0/102400/0, short=0/0/0
lat (usec): 500=99.95%, 750=0.01%, 1000=0.01%
lat (msec): 2=0.01%, 20=0.01%, 50=0.04%
Run status group 0 (all jobs):
WRITE: io=102400MB, aggrb=2995.7MB/s, minb=3067.6MB/s,
maxb=3067.6MB/s, mint=34183msec, maxt=34183msec
Server job file is the same as RDMA_WRITE.
$ ./fio examples/rdmaio-server
receiver: (g=0): rw=read, bs=1M-1M/1M-1M, ioengine=rdma, iodepth=16
fio 1.57
Starting 1 process
Jobs: 1 (f=1): [R] [100.0% done] [3012M/0K /s] [2941 /0 iops] [eta 00m:00s]
receiver: (groupid=0, jobs=1): err= 0: pid=6065
read : io=102400MB, bw=2996.9MB/s, iops=2996 , runt= 34178msec
slat (usec): min=0 , max=6 , avg= 0.42, stdev= 0.50
clat (usec): min=346 , max=72370 , avg=5337.87, stdev=2383.69
lat (usec): min=354 , max=72370 , avg=5338.50, stdev=2383.69
bw (KB/s) : min=2534552, max=3178496, per=100.02%, avg=3068616.97,
stdev=146695.33
cpu : usr=0.99%, sys=0.39%, ctx=102428, majf=0, minf=11
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued r/w/d: total=102400/0/0, short=0/0/0
lat (usec): 500=0.01%, 750=0.01%
lat (msec): 2=0.01%, 4=0.01%, 10=99.36%, 50=0.61%, 100=0.02%
Run status group 0 (all jobs):
READ: io=102400MB, aggrb=2996.9MB/s, minb=3067.2MB/s,
maxb=3067.2MB/s, mint=34178msec, maxt=34178msec
Yufei
.
----- Original Message -----
From: Jens Axboe <jaxboe@xxxxxxxxxxxx>
To: ren yufei <renyufei83@xxxxxxxxxxxx>
Cc: "fio@xxxxxxxxxxxxxxx" <fio@xxxxxxxxxxxxxxx>
Sent: Monday, August 1, 2011 4:03 AM
Subject: Re: rdma ioengine
On 2011-08-01 09:35, Jens Axboe wrote:
> On 2011-08-01 03:47, ren yufei wrote:
>> Hi Jens,
>>
>> I have hacked an rdma ioengine based on OFED for fio which could test both rdma memory semantic (rdma_write/rdma_read) and channel semantic (send/recv). Would you like to merge this engine into fio?
>>
>> notes
>> 1) RDMA engine works in IB, iWarp and RoCE.
>> 2) RDMA engine is disable by default. To enable it, execute the following before compile:
>> $ export EXTFLAGS="-DFIO_HAVE_RDMA"
>> $ export EXTLIBS="-libverbs -lrdmacm"
>
> Exciting! I'd love to apply this, but somewhere along the way the patch
> has been white space damaged. Can you resend this patch gzip'ed or
> something like that, to ensure that it doesn't get modified along the
> way? Bonus points if you re-indent, fio uses real tabs and not 4 spaces.
> Not a huge problem, I'll just lindent it as-is if need be.
>
> It'd be great to have a small job file included as well for testing, and
> an example of a result using that job file.
And you already did include example job files, excellent. I applied your
patch, the mangling wasn't too bad. I also re-indented it with indent,
so it conforms to the fio coding style. Please double check the result!
Thanks.
--
Jens Axboe
Attachment:
rdma-indent.patch.tar.gz
Description: GNU Zip compressed data
