How are the SSDs going to be in writeback? Is that the new caching pool Feature? I am not sure what version implemented this, but it is documented here (https://ceph.com/docs/master/dev/cache-pool/). I will be using the latest stable release for my next batch of testing, right now I am on 0.67.4 and I will be moving towards the 0.72.x branch. As for the IOPS, it would be a total cluster IO throughput estimate based on an application that would be reading/writing to more than 60 rbd volumes. On 2014-03-15, 1:11 PM, "Wido den Hollander" <wido@xxxxxxxx> wrote: >On 03/15/2014 05:40 PM, Karol Kozubal wrote: >> Hi Wido, >> >> I will have some new hardware for running tests in the next two weeks or >> so and will report my findings once I get a chance to run some tests. I >> will disable writeback on the target side as I will be attempting to >> configure an ssd caching pool of 24 ssd's with writeback for the main >>pool >> with 360 disks with a 5 osd spinners to 1 ssd journal ratio. I will be > >How are the SSDs going to be in writeback? Is that the new caching pool >feature? > >> running everything through 10Gig SFP+ Ethernet interfaces with a >>dedicated >> cluster network interface, dedicated public ceph interface and a >>separate >> iscsi network also with 10 gig interfaces for the target machines. >> > >That seems like a good network. > >> I am ideally looking for a 20,000 to 60,000 IOPS from this system if I >>can >> get the caching pool configuration right. The application has a 30ms max >> latency requirement for the storage. >> > >20.000 to 60.000 is a big difference. But the only way you are going to >achieve that is by doing a lot of parellel I/O. Ceph doesn't excel in >single threads doing a lot of I/O. > >So if you have multiple RBD devices on which you are doing the I/O it >shouldn't be that much of a problem. > >Just spread out the I/O. Scale horizontal instead of vertical. > >> In my current tests I have only spinners with SAS 10K disks, 4.2ms write >> latency on the disks with separate journaling on SAS 15K disks with a >> 3.3ms write latency. With 20 OSDs and 4 Journals I am only concerned >>with >> the overall operation apply latency that I have been seeing (1-6ms idle >>is >> normal, but up to 60-170ms for a moderate workload using rbd >>bench-write) >> however I am on a network where I am bound to 1500 mtu and I will get to >> test jumbo frames with the next setup in addition to the ssd¹s. I >>suspect >> the overall performance will be good in the new test setup and I am >> curious to see what my tests will yield. >> >> Thanks for the response! >> >> Karol >> >> >> >> On 2014-03-15, 12:18 PM, "Wido den Hollander" <wido@xxxxxxxx> wrote: >> >>> On 03/15/2014 04:11 PM, Karol Kozubal wrote: >>>> Hi Everyone, >>>> >>>> I am just wondering if any of you are running a ceph cluster with an >>>> iSCSI target front end? I know this isn¹t available out of the box, >>>> unfortunately in one particular use case we are looking at providing >>>> iSCSI access and it's a necessity. I am liking the idea of having rbd >>>> devices serving block level storage to the iSCSI Target servers while >>>> providing a unified backed for native rbd access by openstack and >>>> various application servers. On multiple levels this would reduce the >>>> complexity of our SAN environment and move us away from expensive >>>> proprietary solutions that don¹t scale out. >>>> >>>> If any of you have deployed any HA iSCSI Targets backed by rbd I would >>>> really appreciate your feedback and any thoughts. >>>> >>> >>> I haven't used it in production, but a couple of things which come to >>> mind: >>> >>> - Use TGT so you can run it all in userspace backed by librbd >>> - Do not use writeback caching on the targets >>> >>> You could use multipathing if you don't use writeback caching. Use >>> writeback would also cause data loss/corruption in case of multiple >>> targets. >>> >>> It will probably just work with TGT, but I don't know anything about >>>the >>> performance. >>> >>>> Karol >>>> >>>> >>>> _______________________________________________ >>>> ceph-users mailing list >>>> ceph-users@xxxxxxxxxxxxxx >>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>> >>> >>> >>> -- >>> Wido den Hollander >>> 42on B.V. >>> >>> Phone: +31 (0)20 700 9902 >>> Skype: contact42on >>> _______________________________________________ >>> ceph-users mailing list >>> ceph-users@xxxxxxxxxxxxxx >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > > >-- >Wido den Hollander >42on B.V. > >Phone: +31 (0)20 700 9902 >Skype: contact42on >_______________________________________________ >ceph-users mailing list >ceph-users@xxxxxxxxxxxxxx >http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |