Hi, Karol Here is something that I can share. We are running Ceph as an Exchange Backend via iSCSI. We currently host about 2000 mailboxes which is about 7 TB data overall. Our configuration is - Proxy Node (with tgt daemon) x 2 - Ceph Monitor x 3 (virtual machines) - Ceph OSD x 50 (SATA 7200rpm 2T), Replica = 2, Journal on OSD (I know it is bad, but ...) We tested RBD using fio and got a randwrite around 1500 iops. On the living system, I saw the highest op/s around 3.1k. I've benchmarked "tgt with librdb" vs "tgt with kernel rbd" using my virtual machines. It seems that "tgt with librdb" doesn't perform well. It has only 1/5 iops of kernel rbd. We are new to Ceph and still finding ways to improve the performance. I am really looking forward to your benchmark. On Sun 16 Mar 2014 12:40:53 AM CST, Karol Kozubal wrote: > Hi Wido, > I will have some new hardware for running tests in the next two weeks or > so and will report my findings once I get a chance to run some tests. I > will disable writeback on the target side as I will be attempting to > configure an ssd caching pool of 24 ssd's with writeback for the main pool > with 360 disks with a 5 osd spinners to 1 ssd journal ratio. I will be > running everything through 10Gig SFP+ Ethernet interfaces with a dedicated > cluster network interface, dedicated public ceph interface and a separate > iscsi network also with 10 gig interfaces for the target machines. > I am ideally looking for a 20,000 to 60,000 IOPS from this system if I can > get the caching pool configuration right. The application has a 30ms max > latency requirement for the storage. > In my current tests I have only spinners with SAS 10K disks, 4.2ms write > latency on the disks with separate journaling on SAS 15K disks with a > 3.3ms write latency. With 20 OSDs and 4 Journals I am only concerned with > the overall operation apply latency that I have been seeing (1-6ms idle is > normal, but up to 60-170ms for a moderate workload using rbd bench-write) > however I am on a network where I am bound to 1500 mtu and I will get to > test jumbo frames with the next setup in addition to the ssd¹s. I suspect > the overall performance will be good in the new test setup and I am > curious to see what my tests will yield. > Thanks for the response! > Karol > On 2014-03-15, 12:18 PM, "Wido den Hollander" <wido@xxxxxxxx> wrote: > >On 03/15/2014 04:11 PM, Karol Kozubal wrote: > >> Hi Everyone, > >> > >> I am just wondering if any of you are running a ceph cluster with an > >> iSCSI target front end? I know this isn¹t available out of the box, > >> unfortunately in one particular use case we are looking at providing > >> iSCSI access and it's a necessity. I am liking the idea of having rbd > >> devices serving block level storage to the iSCSI Target servers while > >> providing a unified backed for native rbd access by openstack and > >> various application servers. On multiple levels this would reduce the > >> complexity of our SAN environment and move us away from expensive > >> proprietary solutions that don¹t scale out. > >> > >> If any of you have deployed any HA iSCSI Targets backed by rbd I would > >> really appreciate your feedback and any thoughts. > >> > > > >I haven't used it in production, but a couple of things which come to > >mind: > > > >- Use TGT so you can run it all in userspace backed by librbd > >- Do not use writeback caching on the targets > > > >You could use multipathing if you don't use writeback caching. Use > >writeback would also cause data loss/corruption in case of multiple > >targets. > > > >It will probably just work with TGT, but I don't know anything about the > >performance. > > > >> Karol > >> > >> > >> _______________________________________________ > >> ceph-users mailing list > >> ceph-users@xxxxxxxxxxxxxx > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >> > > > > > >-- > >Wido den Hollander > >42on B.V. > > > >Phone: +31 (0)20 700 9902 > >Skype: contact42on > >_______________________________________________ > >ceph-users mailing list > >ceph-users@xxxxxxxxxxxxxx > >http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- _________________________________________ / Save time starting to type a command or \ | file name, then press tab to complete | | Hit tab twice to bring up multiple | \ completion options. / ----------------------------------------- \ \ _____ _______ ____==== ]OO|_n_n__][. | | [________]_|__|________)< | | oo oo 'oo OOOO-| oo\\_ ~~~|~~~ +--+--+--+--+--+--+--+--+--+--+--+--+--+ _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |