Hi Jianing, Sorry for the late reply, I missed your contribution to the thread. Thank you for your response. I am still waiting for some of my hardware and will begin testing the new setup with firefly once it is available as a long term support release. I am looking forward to testing the new setup. I am curious about more details on your proxy node configuration for the tgt deamons? I am interested if your setup tolerates node failure on the iscsi end of things, if so how it is configured? Thanks, Karol On 2014-03-19, 6:58 AM, "Jianing Yang" <jianingy.yang@xxxxxxxxx> wrote: >Hi, Karol > >Here is something that I can share. We are running Ceph as an Exchange >Backend via iSCSI. We currently host about 2000 mailboxes which is about >7 TB data overall. Our configuration is > >- Proxy Node (with tgt daemon) x 2 >- Ceph Monitor x 3 (virtual machines) >- Ceph OSD x 50 (SATA 7200rpm 2T), Replica = 2, Journal on OSD (I know it >is >bad, but ...) > >We tested RBD using fio and got a randwrite around 1500 iops. On the >living system, I saw the highest op/s around 3.1k. > >I've benchmarked "tgt with librdb" vs "tgt with kernel rbd" using my >virtual machines. It seems that "tgt with librdb" doesn't perform >well. It has only 1/5 iops of kernel rbd. > >We are new to Ceph and still finding ways to improve the performance. I >am really looking forward to your benchmark. > >On Sun 16 Mar 2014 12:40:53 AM CST, Karol Kozubal wrote: > > > Hi Wido, > > > I will have some new hardware for running tests in the next two weeks >or > > so and will report my findings once I get a chance to run some tests. I > > will disable writeback on the target side as I will be attempting to > > configure an ssd caching pool of 24 ssd's with writeback for the main >pool > > with 360 disks with a 5 osd spinners to 1 ssd journal ratio. I will be > > running everything through 10Gig SFP+ Ethernet interfaces with a >dedicated > > cluster network interface, dedicated public ceph interface and a >separate > > iscsi network also with 10 gig interfaces for the target machines. > > > I am ideally looking for a 20,000 to 60,000 IOPS from this system if I >can > > get the caching pool configuration right. The application has a 30ms >max > > latency requirement for the storage. > > > In my current tests I have only spinners with SAS 10K disks, 4.2ms >write > > latency on the disks with separate journaling on SAS 15K disks with a > > 3.3ms write latency. With 20 OSDs and 4 Journals I am only concerned >with > > the overall operation apply latency that I have been seeing (1-6ms >idle is > > normal, but up to 60-170ms for a moderate workload using rbd >bench-write) > > however I am on a network where I am bound to 1500 mtu and I will get >to > > test jumbo frames with the next setup in addition to the ssd¹s. I >suspect > > the overall performance will be good in the new test setup and I am > > curious to see what my tests will yield. > > > Thanks for the response! > > > Karol > > > > > On 2014-03-15, 12:18 PM, "Wido den Hollander" <wido@xxxxxxxx> wrote: > > > >On 03/15/2014 04:11 PM, Karol Kozubal wrote: > > >> Hi Everyone, > > >> > > >> I am just wondering if any of you are running a ceph cluster with an > > >> iSCSI target front end? I know this isn¹t available out of the box, > > >> unfortunately in one particular use case we are looking at providing > > >> iSCSI access and it's a necessity. I am liking the idea of having >rbd > > >> devices serving block level storage to the iSCSI Target servers >while > > >> providing a unified backed for native rbd access by openstack and > > >> various application servers. On multiple levels this would reduce >the > > >> complexity of our SAN environment and move us away from expensive > > >> proprietary solutions that don¹t scale out. > > >> > > >> If any of you have deployed any HA iSCSI Targets backed by rbd I >would > > >> really appreciate your feedback and any thoughts. > > >> > > > > > >I haven't used it in production, but a couple of things which come to > > >mind: > > > > > >- Use TGT so you can run it all in userspace backed by librbd > > >- Do not use writeback caching on the targets > > > > > >You could use multipathing if you don't use writeback caching. Use > > >writeback would also cause data loss/corruption in case of multiple > > >targets. > > > > > >It will probably just work with TGT, but I don't know anything about >the > > >performance. > > > > > >> Karol > > >> > > >> > > >> _______________________________________________ > > >> ceph-users mailing list > > >> ceph-users@xxxxxxxxxxxxxx > > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > >> > > > > > > > > >-- > > >Wido den Hollander > > >42on B.V. > > > > > >Phone: +31 (0)20 700 9902 > > >Skype: contact42on > > >_______________________________________________ > > >ceph-users mailing list > > >ceph-users@xxxxxxxxxxxxxx > > >http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > _______________________________________________ > > ceph-users mailing list > > ceph-users@xxxxxxxxxxxxxx > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > >-- > _________________________________________ >/ Save time starting to type a command or \ >| file name, then press tab to complete | >| Hit tab twice to bring up multiple | >\ completion options. / > ----------------------------------------- > \ > \ > _____ _______ > ____==== ]OO|_n_n__][. | | > [________]_|__|________)< | | > oo oo 'oo OOOO-| oo\\_ ~~~|~~~ >+--+--+--+--+--+--+--+--+--+--+--+--+--+ _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |