Re: Poor RBD performance as LIO iSCSI target

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi David,

This is a long shot, but have you checked the Max queue depth on the iscsi side. I've got a feeling that lio might be set at 32 as default.

This would definitely have an effect at the high queue depths you are testing with.

On 8 Dec 2014 16:53, David Moreau Simard <dmsimard@xxxxxxxx> wrote:

Haven't tried other iSCSI implementations (yet). LIO/targetcli makes it very easy to iQuoting David Moreau Simard <dmsimard@xxxxxxxx>

Haven't tried other iSCSI implementations (yet).

LIO/targetcli makes it very easy to implement/integrate/wrap/automate around so I'm really trying to get this right.

PCI-E SSD cache tier in front of spindles-backed erasure coded pool in 10 Gbps across the board yields results slightly better or very similar to two spindles in hardware RAID-0 with writeback caching. With that in mind, the performance is not outright awful by any means, there's just a lot of overhead we have to be reminded about.

What I'd like to further test but am unable to right now is to see what happens if you scale up the cluster. Right now I'm testing on only two nodes. Does the IOPS scale linearly with increasing amount of OSDs/servers ? Or is it more about a capacity thing ?

Perhaps if someone else can chime in, I'm really curious.
--
David Moreau Simard

On Dec 6, 2014, at 11:18 AM, Nick Fisk <nick@xxxxxxxxxx> wrote:

Hi David,

Very strange, but  I'm glad you managed to finally get the cluster working
normally. Thank you for posting the benchmarks figures, it's interesting to
see the overhead of LIO over pure RBD performance.

I should have the hardware for our cluster up and running early next year, I
will be in a better position to test the iSCSI performance then. I will
report back once I have some numbers.

Just out of interest, have you tried any of the other iSCSI implementations
to see if they show the same performance drop?

Nick

-----Original Message-----
From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of
David Moreau Simard
Sent: 05 December 2014 16:03
To: Nick Fisk
Cc: ceph-users@xxxxxxxxxxxxxx
Subject: Re:  Poor RBD performance as LIO iSCSI target

I've flushed everything - data, pools, configs and reconfigured the whole
thing.

I was particularly careful with cache tiering configurations (almost leaving
defaults when possible) and it's not locking anymore.
It looks like the cache tiering configuration I had was causing the problem
? I can't put my finger on exactly what/why and I don't have the luxury of
time to do this lengthy testing again.

Here's what I dumped as far as config goes before wiping:
========
# for var in size min_size pg_num pgp_num crush_ruleset
erasure_code_profile; do ceph osd pool get volumes $var; done
size: 5
min_size: 2
pg_num: 7200
pgp_num: 7200
crush_ruleset: 1
erasure_code_profile: ecvolumes

# for var in size min_size pg_num pgp_num crush_ruleset hit_set_type
hit_set_period hit_set_count target_max_objects target_max_bytes
cache_target_dirty_ratio cache_target_full_ratio cache_min_flush_age
cache_min_evict_age; do ceph osd pool get volumecache $var; done
size: 2
min_size: 1
pg_num: 7200
pgp_num: 7200
crush_ruleset: 4
hit_set_type: bloom
hit_set_period: 3600
hit_set_count: 1
target_max_objects: 0
target_max_bytes: 100000000000
cache_target_dirty_ratio: 0.5
cache_target_full_ratio: 0.8
cache_min_flush_age: 600
cache_min_evict_age: 1800

# ceph osd erasure-code-profile get ecvolumes
directory=/usr/lib/ceph/erasure-code
k=3
m=2
plugin=jerasure
ruleset-failure-domain=osd
technique=reed_sol_van
========

And now:
========
# for var in size min_size pg_num pgp_num crush_ruleset
erasure_code_profile; do ceph osd pool get volumes $var; done
size: 5
min_size: 3
pg_num: 2048
pgp_num: 2048
crush_ruleset: 1
erasure_code_profile: ecvolumes

# for var in size min_size pg_num pgp_num crush_ruleset hit_set_type
hit_set_period hit_set_count target_max_objects target_max_bytes
cache_target_dirty_ratio cache_target_full_ratio cache_min_flush_age
cache_min_evict_age; do ceph osd pool get volumecache $var; done
size: 2
min_size: 1
pg_num: 2048
pgp_num: 2048
crush_ruleset: 4
hit_set_type: bloom
hit_set_period: 3600
hit_set_count: 1
target_max_objects: 0
target_max_bytes: 150000000000
cache_target_dirty_ratio: 0.5
cache_target_full_ratio: 0.8
cache_min_flush_age: 0
cache_min_evict_age: 1800

# ceph osd erasure-code-profile get ecvolumes
directory=/usr/lib/ceph/erasure-code
k=3
m=2
plugin=jerasure
ruleset-failure-domain=osd
technique=reed_sol_van
========

Crush map hasn't really changed before and after.

FWIW, the benchmarks I pulled out of the setup:
https://gist.github.com/dmsimard/2737832d077cfc5eff34
Definite overhead going from krbd to krbd + LIO...
--
David Moreau Simard


On Nov 20, 2014, at 4:14 PM, Nick Fisk <nick@xxxxxxxxxx> wrote:

Here you go:-

Erasure Profile
k=2
m=1
plugin=jerasure
ruleset-failure-domain=osd
ruleset-root=hdd
technique=reed_sol_van

Cache Settings
hit_set_type: bloom
hit_set_period: 3600
hit_set_count: 1
target_max_objects
target_max_objects: 0
target_max_bytes: 1000000000
cache_target_dirty_ratio: 0.4
cache_target_full_ratio: 0.8
cache_min_flush_age: 0
cache_min_evict_age: 0

Crush Dump
# begin crush map
tunable choose_local_tries 0
tunable choose_local_fallback_tries 0
tunable choose_total_tries 50
tunable chooseleaf_descend_once 1

# devices
device 0 osd.0
device 1 osd.1
device 2 osd.2
device 3 osd.3

# types
type 0 osd
type 1 host
type 2 chassis
type 3 rack
type 4 row
type 5 pdu
type 6 pod
type 7 room
type 8 datacenter
type 9 region
type 10 root

# buckets
host ceph-test-hdd {
      id -5           # do not change unnecessarily
      # weight 2.730
      alg straw
      hash 0  # rjenkins1
      item osd.1 weight 0.910
      item osd.2 weight 0.910
      item osd.0 weight 0.910
}
root hdd {
      id -3           # do not change unnecessarily
      # weight 2.730
      alg straw
      hash 0  # rjenkins1
      item ceph-test-hdd weight 2.730 } host ceph-test-ssd {
      id -6           # do not change unnecessarily
      # weight 1.000
      alg straw
      hash 0  # rjenkins1
      item osd.3 weight 1.000
}
root ssd {
      id -4           # do not change unnecessarily
      # weight 1.000
      alg straw
      hash 0  # rjenkins1
      item ceph-test-ssd weight 1.000 }

# rules
rule hdd {
      ruleset 0
      type replicated
      min_size 0
      max_size 10
      step take hdd
      step chooseleaf firstn 0 type osd
      step emit
}
rule ssd {
      ruleset 1
      type replicated
      min_size 0
      max_size 4
      step take ssd
      step chooseleaf firstn 0 type osd
      step emit
}
rule ecpool {
      ruleset 2
      type erasure
      min_size 3
      max_size 20
      step set_chooseleaf_tries 5
      step take hdd
      step chooseleaf indep 0 type osd
      step emit
}



-----Original Message-----
From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf
Of David Moreau Simard
Sent: 20 November 2014 20:03
To: Nick Fisk
Cc: ceph-users@xxxxxxxxxxxxxx
Subject: Re:  Poor RBD performance as LIO iSCSI target

Nick,

Can you share more datails on the configuration you are using ? I'll
try and duplicate those configurations in my environment and see what
happens.
I'm mostly interested in:
- Erasure code profile (k, m, plugin, ruleset-failure-domain)
- Cache tiering pool configuration (ex: hit_set_type, hit_set_period,
hit_set_count, target_max_objects, target_max_bytes,
cache_target_dirty_ratio, cache_target_full_ratio,
cache_min_flush_age,
cache_min_evict_age)

The crush rulesets would also be helpful.

Thanks,
--
David Moreau Simard

On Nov 20, 2014, at 12:43 PM, Nick Fisk <nick@xxxxxxxxxx> wrote:

Hi David,

I've just finished running the 75GB fio test you posted a few days
back on my new test cluster.

The cluster is as follows:-

Single server with 3x hdd and 1 ssd
Ubuntu 14.04 with 3.16.7 kernel
2+1 EC pool on hdds below a 10G ssd cache pool. SSD is also
2+partitioned to
provide journals for hdds.
150G RBD mapped locally

The fio test seemed to run without any problems. I want to run a few
more tests with different settings to see if I can reproduce your
problem. I will let you know if I find anything.

If there is anything you would like me to try, please let me know.

Nick

-----Original Message-----
From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf
Of David Moreau Simard
Sent: 19 November 2014 10:48
To: Ramakrishna Nishtala (rnishtal)
Cc: ceph-users@xxxxxxxxxxxxxx; Nick Fisk
Subject: Re:  Poor RBD performance as LIO iSCSI target

Rama,

Thanks for your reply.

My end goal is to use iSCSI (with LIO/targetcli) to export rbd block
devices.

I was encountering issues with iSCSI which are explained in my
previous emails.
I ended up being able to reproduce the problem at will on various
Kernel and OS combinations, even on raw RBD devices - thus ruling out
the hypothesis that it was a problem with iSCSI but rather with Ceph.
I'm even running 0.88 now and the issue is still there.

I haven't isolated the issue just yet.
My next tests involve disabling the cache tiering.

I do have client krbd cache as well, i'll try to disable it too if
cache tiering isn't enough.
--
David Moreau Simard


On Nov 18, 2014, at 8:10 PM, Ramakrishna Nishtala (rnishtal)
<rnishtal@xxxxxxxxx> wrote:

Hi Dave
Did you say iscsi only? The tracker issue does not say though.
I am on giant, with both client and ceph on RHEL 7 and seems to work
ok,
unless I am missing something here. RBD on baremetal with kmod-rbd
and caching disabled.

[root@compute4 ~]# time fio --name=writefile --size=100G
--filesize=100G --filename=/dev/rbd0 --bs=1M --nrfiles=1 --direct=1
--sync=0 --randrepeat=0 --rw=write --refill_buffers --end_fsync=1
--iodepth=200 --ioengine=libaio
writefile: (g=0): rw=write, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio,
iodepth=200
fio-2.1.11
Starting 1 process
Jobs: 1 (f=1): [W(1)] [100.0% done] [0KB/853.0MB/0KB /s] [0/853/0
iops] [eta 00m:00s] ...
Disk stats (read/write):
rbd0: ios=184/204800, merge=0/0, ticks=70/16164931,
in_queue=16164942, util=99.98%

real    1m56.175s
user    0m18.115s
sys     0m10.430s

Regards,

Rama


-----Original Message-----
From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On
Behalf Of David Moreau Simard
Sent: Tuesday, November 18, 2014 3:49 PM
To: Nick Fisk
Cc: ceph-users@xxxxxxxxxxxxxx
Subject: Re:  Poor RBD performance as LIO iSCSI target

Testing without the cache tiering is the next test I want to do when
I
have time..

When it's hanging, there is no activity at all on the cluster.
Nothing in "ceph -w", nothing in "ceph osd pool stats".

I'll provide an update when I have a chance to test without tiering.
--
David Moreau Simard


On Nov 18, 2014, at 3:28 PM, Nick Fisk <nick@xxxxxxxxxx> wrote:

Hi David,

Have you tried on a normal replicated pool with no cache? I've seen
a number of threads recently where caching is causing various
things to
block/hang.
It would be interesting to see if this still happens without the
caching layer, at least it would rule it out.

Also is there any sign that as the test passes ~50GB that the cache
might start flushing to the backing pool causing slow performance?

I am planning a deployment very similar to yours so I am following
this with great interest. I'm hoping to build a single node test
"cluster" shortly, so I might be in a position to work with you on
this issue and hopefully get it resolved.

Nick

-----Original Message-----
From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On
Behalf Of David Moreau Simard
Sent: 18 November 2014 19:58
To: Mike Christie
Cc: ceph-users@xxxxxxxxxxxxxx; Christopher Spearman
Subject: Re:  Poor RBD performance as LIO iSCSI target

Thanks guys. I looked at http://tracker.ceph.com/issues/8818 and
chatted with "dis" on #ceph-devel.

I ran a LOT of tests on a LOT of comabination of kernels (sometimes
with tunables legacy). I haven't found a magical combination in
which the following test does not hang:
fio --name=writefile --size=100G --filesize=100G
--filename=/dev/rbd0 --bs=1M --nrfiles=1 --direct=1 --sync=0
--randrepeat=0 --rw=write --refill_buffers --end_fsync=1
--iodepth=200 --ioengine=libaio

Either directly on a mapped rbd device, on a mounted filesystem
(over rbd), exported through iSCSI.. nothing.
I guess that rules out a potential issue with iSCSI overhead.

Now, something I noticed out of pure luck is that I am unable to
reproduce the issue if I drop the size of the test to 50GB. Tests
will complete in under 2 minutes.
75GB will hang right at the end and take more than 10 minutes.

TL;DR of tests:
- 3x fio --name=writefile --size=50G --filesize=50G
--filename=/dev/rbd0 --bs=1M --nrfiles=1 --direct=1 --sync=0
--randrepeat=0 --rw=write --refill_buffers --end_fsync=1
--iodepth=200 --ioengine=libaio
-- 1m44s, 1m49s, 1m40s

- 3x fio --name=writefile --size=75G --filesize=75G
--filename=/dev/rbd0 --bs=1M --nrfiles=1 --direct=1 --sync=0
--randrepeat=0 --rw=write --refill_buffers --end_fsync=1
--iodepth=200 --ioengine=libaio
-- 10m12s, 10m11s, 10m13s

Details of tests here: http://pastebin.com/raw.php?i=3v9wMtYP

Does that ring you guys a bell ?

--
David Moreau Simard


On Nov 13, 2014, at 3:31 PM, Mike Christie <mchristi@xxxxxxxxxx>
wrote:

On 11/13/2014 10:17 AM, David Moreau Simard wrote:
Running into weird issues here as well in a test environment. I
don't
have a solution either but perhaps we can find some things in common..

Setup in a nutshell:
- Ceph cluster: Ubuntu 14.04, Kernel 3.16.7, Ceph 0.87-1 (OSDs
with separate public/cluster network in 10 Gbps)
- iSCSI Proxy node (targetcli/LIO): Ubuntu 14.04, Kernel 3.16.7,
Ceph
0.87-1 (10 Gbps)
- Client node: Ubuntu 12.04, Kernel 3.11 (10 Gbps)

Relevant cluster config: Writeback cache tiering with NVME PCI-E
cards (2
replica) in front of a erasure coded pool (k=3,m=2) backed by spindles.

I'm following the instructions here:
http://www.hastexo.com/resources/hints-and-kinks/turning-ceph-rbd
- im a ges-san-storage-devices No issues with creating and
mapping a 100GB RBD image and then creating the target.

I'm interested in finding out the overhead/performance impact of
re-exporting through iSCSI so the idea is to run benchmarks.
Here's a fio test I'm trying to run on the client node on the
mounted
iscsi device:
fio --name=writefile --size=100G --filesize=100G
--filename=/dev/sdu --bs=1M --nrfiles=1 --direct=1 --sync=0
--randrepeat=0 --rw=write --refill_buffers --end_fsync=1
--iodepth=200 --ioengine=libaio

The benchmark will eventually hang towards the end of the test
for some
long seconds before completing.
On the proxy node, the kernel complains with iscsi portal login
timeout: http://pastebin.com/Q49UnTPr and I also see irqbalance
errors in syslog: http://pastebin.com/AiRTWDwR


You are hitting a different issue. German Anders is most likely
correct and you hit the rbd hang. That then caused the iscsi/scsi
command to timeout which caused the scsi error handler to run. In
your logs we see the LIO error handler has received a task abort
from the initiator and that timed out which caused the escalation
(iscsi portal login related messages).

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com





_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com





_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com












_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux