Hi Jens,
Job Files:
Client1:
# cat all_powerpath.fio
[global]
ioengine=libaio
invalidate=1
ramp_time=10
direct=1
refill_buffers=1
time_based
runtime=259200
[readwrite-emcpowera-8k-para]
bs=4k
iodepth=8
filename=/dev/emcpowera
size=10g
rw=randrw
[readwrite-emcpowerb-8k-para]
bs=4k
iodepth=8
filename=/dev/emcpowerb
size=10g
rw=randrw
[readwrite-emcpowerc-8k-para]
bs=4k
iodepth=8
filename=/dev/emcpowerc
size=10g
rw=randrw
[readwrite-emcpowerd-8k-para]
bs=4k
iodepth=8
filename=/dev/emcpowerd
size=10g
rw=randrw
[readwrite-emcpowere-8k-para]
bs=4k
iodepth=8
filename=/dev/emcpowere
size=10g
rw=randrw
[readwrite-emcpowerf-8k-para]
bs=4k
iodepth=8
filename=/dev/emcpowerf
size=10g
rw=randrw
...
... [Extends till /dev/emcpowerx -- 24 devices
client2:
# cat all_multipaths.fio q
[112/2161]
[global]
ioengine=libaio
invalidate=1
ramp_time=10
direct=1
refill_buffers=1
time_based
runtime=259200
norandommap
[readwrite-mpatha-8k-para]
bs=8k
iodepth=8
filename=/dev/mapper/mpatha
size=10g
rw=randrw
[readwrite-mpathb-8k-para]
bs=8k
iodepth=8
filename=/dev/mapper/mpathb
size=10g
rw=randrw
[readwrite-mpathc-8k-para]
bs=8k
iodepth=8
filename=/dev/mapper/mpathc
size=10g
rw=randrw
[readwrite-mpathd-8k-para]
bs=8k
iodepth=8
filename=/dev/mapper/mpathd
size=10g
rw=randrw
[readwrite-mpathe-8k-para]
bs=8k
iodepth=8
filename=/dev/mapper/mpathe
size=10g
rw=randrw
[readwrite-mpathf-8k-para]
bs=8k
iodepth=8
filename=/dev/mapper/mpathf
size=10g
rw=randrw
[readwrite-mpathg-8k-para]
bs=8k
iodepth=8
filename=/dev/mapper/mpathg
size=10g
rw=randrw
...
... [ Extends till /dev/mapper/mpathx --- 24 devices ]
--
Srinivasa R Chamarthy
On Thu, Feb 18, 2016 at 4:28 AM, Jens Axboe <axboe@xxxxxxxxx> wrote:
> On 02/16/2016 01:19 AM, Srinivasa Chamarthy wrote:
>>
>> I started FIO in client/server mode with two clients running 24 jobs
>> each on individual LUNs. After a while, FIO fails with the following
>> assertion, even though the jobs are still running on both the clients.
>>
>> # fio --client=client1 --remote-config=/root/all_powerpath.fio
>> --client=client2 --remote-config=/root/all_multipaths.fio
>> hostname=client1, be=0, 64-bit, os=Linux, arch=x86-64,
>> fio=fio-2.6-13-g8a76, flags=1
>> hostname=client2, be=0, 64-bit, os=Linux, arch=x86-64,
>> fio=fio-2.6-13-g8a76, flags=1
>>
>> <client2> readwrite-mpatha-8k-para: (g=0): rw=randrw, <client1>
>> readwrite-emcpowera-8k-para: (g=0): rw=randrw, bs=8K-8K/8K-8K/8K-8K,
>> bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=8
>> ioengine=libaio, iodepth=8
>> <client2> readwrite-mpathb-8k-para: (g=0): rw=randrw, <client1>
>> readwrite-emcpowerb-8k-para: (g=0): rw=randrw, bs=8K-8K/8K-8K/8K-8K,
>> bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=8
>> ioengine=libaio, iodepth=8
>> <client2> readwrite-mpathc-8k-para: (g=0): rw=randrw, <client1>
>> readwrite-emcpowerc-8k-para: (g=0): rw=randrw, bs=8K-8K/8K-8K/8K-8K,
>> bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=8
>> ioengine=libaio, iodepth=8
>> <client2> readwrite-mpathd-8k-para: (g=0): rw=randrw, <client1>
>> readwrite-emcpowerd-8k-para: (g=0): rw=randrw, bs=8K-8K/8K-8K/8K-8K,
>> bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=8
>> ioengine=libaio, iodepth=8
>>
>> <<< Omit other info >>>
>>
>> <client2> Starting 24 processes
>> <client1> Starting 24 processes
>>
>> client <client1>: timeout on SEND_ETA/405.7M/0K /s] [51.2K/50.7K/0
>> iops] [eta 02d:23h:59m:49s]
>> fio: client: unable to find matching tag (dbebe0)2M/0K /s]
>> [60.8K/58.6K/0 iops] [eta 02d:23h:59m:54s]
>> fio: client.c:1241: handle_eta: Assertion `client->eta_in_flight == eta'
>> failed.
>> Aborted
>>
>> # fio --version
>> fio-2.6-13-g8a76
>>
>> Also sometimes the job information gets mixed up from both the clients
>> as above.
>
>
> Can you include the job file(s) that you used?
>
> --
> Jens Axboe
>
--
To unsubscribe from this list: send the line "unsubscribe fio" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
