Hi Wido, I will have some new hardware for running tests in the next two weeks or so and will report my findings once I get a chance to run some tests. I will disable writeback on the target side as I will be attempting to configure an ssd caching pool of 24 ssd's with writeback for the main pool with 360 disks with a 5 osd spinners to 1 ssd journal ratio. I will be running everything through 10Gig SFP+ Ethernet interfaces with a dedicated cluster network interface, dedicated public ceph interface and a separate iscsi network also with 10 gig interfaces for the target machines. I am ideally looking for a 20,000 to 60,000 IOPS from this system if I can get the caching pool configuration right. The application has a 30ms max latency requirement for the storage. In my current tests I have only spinners with SAS 10K disks, 4.2ms write latency on the disks with separate journaling on SAS 15K disks with a 3.3ms write latency. With 20 OSDs and 4 Journals I am only concerned with the overall operation apply latency that I have been seeing (1-6ms idle is normal, but up to 60-170ms for a moderate workload using rbd bench-write) however I am on a network where I am bound to 1500 mtu and I will get to test jumbo frames with the next setup in addition to the ssd¹s. I suspect the overall performance will be good in the new test setup and I am curious to see what my tests will yield. Thanks for the response! Karol On 2014-03-15, 12:18 PM, "Wido den Hollander" <wido@xxxxxxxx> wrote: >On 03/15/2014 04:11 PM, Karol Kozubal wrote: >> Hi Everyone, >> >> I am just wondering if any of you are running a ceph cluster with an >> iSCSI target front end? I know this isn¹t available out of the box, >> unfortunately in one particular use case we are looking at providing >> iSCSI access and it's a necessity. I am liking the idea of having rbd >> devices serving block level storage to the iSCSI Target servers while >> providing a unified backed for native rbd access by openstack and >> various application servers. On multiple levels this would reduce the >> complexity of our SAN environment and move us away from expensive >> proprietary solutions that don¹t scale out. >> >> If any of you have deployed any HA iSCSI Targets backed by rbd I would >> really appreciate your feedback and any thoughts. >> > >I haven't used it in production, but a couple of things which come to >mind: > >- Use TGT so you can run it all in userspace backed by librbd >- Do not use writeback caching on the targets > >You could use multipathing if you don't use writeback caching. Use >writeback would also cause data loss/corruption in case of multiple >targets. > >It will probably just work with TGT, but I don't know anything about the >performance. > >> Karol >> >> >> _______________________________________________ >> ceph-users mailing list >> ceph-users@xxxxxxxxxxxxxx >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > > >-- >Wido den Hollander >42on B.V. > >Phone: +31 (0)20 700 9902 >Skype: contact42on >_______________________________________________ >ceph-users mailing list >ceph-users@xxxxxxxxxxxxxx >http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |