Hello All,
I’ve been trying to get an fio workload over RDMA working, where a limitation is that there is only ipv6 available for now, and that I have to specify GID index 2 in order to be able to use ipv4. For example, when using ib_write_bw, it looks like this:
softnic@ktu-vm1:~$ ib_write_bw -x 2 10.0.0.4 --port 18515
---------------------------------------------------------------------------------------
RDMA_Write BW Test
Dual-port : OFF Device : roceP30832p0s0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
PCIe relax order: ON
ibv_wr* API : OFF
TX depth : 128
CQ Moderation : 1
Mtu : 1024[B]
Link type : Ethernet
GID index : 2
Max inline data : 0[B]
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0000 QPN 0x0024 PSN 0xd4e107 RKey 0x00cd00 VAddr 0x007240fdfd2000
GID: 38:03:16:176:05:67:00:60:00:00:00:01:00:114:204:90
remote address: LID 0000 QPN 0x0024 PSN 0xb3e770 RKey 0x00cd00 VAddr 0x007bb8d3c51000
GID: 38:03:16:176:05:67:00:58:00:00:00:01:00:59:18:224
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[MiB/sec] BW average[MiB/sec] MsgRate[Mpps]
Conflicting CPU frequency values detected: 2300.000000 != 3599.929000. CPU Frequency is not max.
65536 5000 10828.63 10828.55 0.173257
---------------------------------------------------------------------------------------
Here, -x specifies the GID index.
Now, when using fio, I couldn’t find a similar option in the documentation. The job files I used are:
Server:
[global]
ioengine=rdma
port=18515
bs=1m
size=100g
[receiver]
rw=read
iodepth=16
This appears to work:
receiver: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=rdma, iodepth=16
fio-3.28
Starting 1 process
fio: waiting for connection
On the client:
[global]
ioengine=rdma
hostname=10.0.0.5
port=18515
verb=write
bs=1m
size=100g
[sender]
rw=write
iodepth=1
iodepth_batch_complete=1
This eventually shows:
softnic@ktu-vm2:~$ fio fio_rdma_client.job
sender: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=rdma, iodepth=1
fio-3.28
Starting 1 process
Jobs: 1 (f=0): [W(1)][-.-%][eta 00m:00s]
After a while, I get:
fio: event is RDMA_CM_EVENT_UNREACHABLE instead of RDMA_CM_EVENT_ESTABLISHED
fio: wait for RDMA_CM_EVENT_ESTABLISHED
Given the limitations mentioned above, is it possible to get RDMA working in this scenario?
Rob
