On 03/15/2014 05:40 PM, Karol Kozubal wrote:
Hi Wido,
I will have some new hardware for running tests in the next two weeks or
so and will report my findings once I get a chance to run some tests. I
will disable writeback on the target side as I will be attempting to
configure an ssd caching pool of 24 ssd's with writeback for the main pool
with 360 disks with a 5 osd spinners to 1 ssd journal ratio. I will be
How are the SSDs going to be in writeback? Is that the new caching pool
feature?
running everything through 10Gig SFP+ Ethernet interfaces with a dedicated
cluster network interface, dedicated public ceph interface and a separate
iscsi network also with 10 gig interfaces for the target machines.
That seems like a good network.
I am ideally looking for a 20,000 to 60,000 IOPS from this system if I can
get the caching pool configuration right. The application has a 30ms max
latency requirement for the storage.
20.000 to 60.000 is a big difference. But the only way you are going to
achieve that is by doing a lot of parellel I/O. Ceph doesn't excel in
single threads doing a lot of I/O.
So if you have multiple RBD devices on which you are doing the I/O it
shouldn't be that much of a problem.
Just spread out the I/O. Scale horizontal instead of vertical.
In my current tests I have only spinners with SAS 10K disks, 4.2ms write
latency on the disks with separate journaling on SAS 15K disks with a
3.3ms write latency. With 20 OSDs and 4 Journals I am only concerned with
the overall operation apply latency that I have been seeing (1-6ms idle is
normal, but up to 60-170ms for a moderate workload using rbd bench-write)
however I am on a network where I am bound to 1500 mtu and I will get to
test jumbo frames with the next setup in addition to the ssd¹s. I suspect
the overall performance will be good in the new test setup and I am
curious to see what my tests will yield.
Thanks for the response!
Karol
On 2014-03-15, 12:18 PM, "Wido den Hollander" <wido@xxxxxxxx> wrote:
On 03/15/2014 04:11 PM, Karol Kozubal wrote:
Hi Everyone,
I am just wondering if any of you are running a ceph cluster with an
iSCSI target front end? I know this isn¹t available out of the box,
unfortunately in one particular use case we are looking at providing
iSCSI access and it's a necessity. I am liking the idea of having rbd
devices serving block level storage to the iSCSI Target servers while
providing a unified backed for native rbd access by openstack and
various application servers. On multiple levels this would reduce the
complexity of our SAN environment and move us away from expensive
proprietary solutions that don¹t scale out.
If any of you have deployed any HA iSCSI Targets backed by rbd I would
really appreciate your feedback and any thoughts.
I haven't used it in production, but a couple of things which come to
mind:
- Use TGT so you can run it all in userspace backed by librbd
- Do not use writeback caching on the targets
You could use multipathing if you don't use writeback caching. Use
writeback would also cause data loss/corruption in case of multiple
targets.
It will probably just work with TGT, but I don't know anything about the
performance.
Karol
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Wido den Hollander
42on B.V.
Phone: +31 (0)20 700 9902
Skype: contact42on
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Wido den Hollander
42on B.V.
Phone: +31 (0)20 700 9902
Skype: contact42on
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
