Hi Timofey,
On Sun, 2015-12-13 at 15:29 +0300, Timofey Titovets wrote:
> Hi list,
> in my case i proxy rbd to ESXi, by iSCSI fileio plugin
>
> But i see some performance problem and i can't catch where it
>
> Simple case:
> Ceph cluster 15 disks
> On proxy machine with rbd disk in FIO i see 900 IOPs 50/50 rwramdom 64 io depth
>
> Then i create iSCSI target on this RBD and connect to it, by ESXi
> On rbd data store, i place VM with FIO and run this test again, what i
> see by iostat:
> On VM with fio
> iodepth 64, latency ~700 ms
>
> on proxy machine with iSCSI and RBD:
> iodepth ~8 (+- 4) and latency 20-300, and i don't see any problem with CPU load
>
> May be someone already see and fix this problem, as i understand ESXi
> won't offload all commands to Proxy machine
>
> Scheme again:
> Ceph <---> rbd <---> iSCSI target <----> iSCSI Initiator <---> FIO VM
>
> FIO VM Generate load and have 64 commands in queue with crazy latency
>
> On rbd side, by iostat i see only 8 commands in queue with acceptable latency
>
> How i can debug what generate this problem in the iSCSI target <---->
> iSCSI Initiator Stack?
>
> And i don't observe any latency problem with ioping (i.e. if queue
> small enough all works perfect)
It's hard to say based on provided information alone.
A few things to consider though:
FILEIO backends are all using O_DSYNC by default, which effectively
disables Linux/VFS buffer-cache write-back operation. That is, all
FILEIO write become write-through, and no WRITE is acknowledged until
the underlying storage has marked blocks as persisted.
You'll really want to understand what buffered FILEIO operation means
for your setup, and implications it has for data loss of acked but not
flushed WRITEs during a power failure event.
That aside, to use FILEIO backends in buffer-cache mode, you'll need to
be setting this value at creation time in targetcli, or via "buffered
yes" in rtslib v3.x config:
storage fileio disk tmpfile {
buffered yes
path /root/fileio
size 2.0GB
<<<<< SNIP >>>>>
}
and verify it has been enabled with "Mode: Buffered-WCE" in configfs:
root@scsi-mq:~# cat /sys/kernel/config/target/core/fileio_1/tmpfile/info
Status: ACTIVATED Max Queue Depth: 128 SectorSize: 512 HwMaxSectors: 16384
TCM FILEIO ID: 0 File: /root/fileio Size: 2147483648 Mode: Buffered-WCE
That all said, the folks I'm aware of using RBD seriously are using
IBLOCK backends, in order to communicate asynchronously via struct bio
directly into make_request_fn() or blk-mq based queue_rq() callback.
So unless you've got a good reason to use FILEIO buffered IO mode, I'd
recommend considering IBLOCK instead.
--nab
--
To unsubscribe from this list: send the line "unsubscribe target-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
