Re: RADOS Gateway Maturity

Jerry Lam <Jerry.Lam@xxxxxxxxxx> · Mon, 23 Mar 2015 18:26:19 +0000

Hi Chris and Craig,

Thank you for sharing your experience with me about S3 API RADOS gateway!

Jerry Lam
Senior Software Developer, Big Data

Ontario Institute for Cancer Research

MaRS Centre
661 University Avenue
Suite 510 

Toronto, Ontario 
Canada M5G 0A3

Email:   jerry.lam@xxxxxxxxxx
Toll-free: 1-866-678-6427

Twitter: @OICR_news

www.oicr.on.ca

This message and any attachments may contain confidential and/or privileged information for the sole use of the intended recipient. Any review or distribution by anyone other than the person
 for whom it was originally intended is strictly prohibited. If you have received this message in error, please contact the sender and delete all copies. Opinions, conclusions or other information contained in this message may not be that of the organization. 

From: Chris Jones <cjones@xxxxxxxxxxx>

Date: Friday, March 20, 2015 at 7:28 PM

To: Craig Lewis <clewis@xxxxxxxxxxxxxxxxxx>

Cc: Jerry Lam <jerry.lam@xxxxxxxxxx>, "ceph-users@xxxxxxxxxxxxxx" <ceph-users@xxxxxxxxxxxxxx>

Subject: Re:  RADOS Gateway Maturity

Hi Jerry,

We using RGW and RBD in our OpenStack clusters and as stand alone clusters. We have six large clusters and adding more. Most of any issues we have faced have been self inflicted such as not currently supporting bucket names like host names. Some S3 tools
 only work that way which causes some of our developer customers grief. We are addressing that. We have built extensive testing frameworks around S3 RGW testing using a OpenStack or AWS EC2 or Google Cloud Platform to dynamically spin up worker nodes to distribute
 load for stressing and performance monitoring.

I'm actually building a project called IQStack that will be released on github soon that does that plus other OpenStack testing for scalability. Anyway, there may be some incompatibilities depending on the feature set but most can be abstracted away and
 addressed. 

Just as a footnote, I finished running long load testing against AWS S3 and RGW. After tuning our load balancers, firewall rules and a few other tweaks I was able to get parity with AWS S3 up to 10GbE (max size of my load balancer I was testing with).
 We use several CDNs on video clips so my tests were with 2MB byte-range requests.

Thanks,
Chris 

On Fri, Mar 20, 2015 at 6:06 PM, Craig Lewis 
<clewis@xxxxxxxxxxxxxxxxxx> wrote:

I have found a few incompatibilities, but so far they're all on the Ceph side.  One example I remember was having to change the way we delete objects.  The function we originally used fetches a list of object versions, and deletes all versions. 
 Ceph is implementing objects versions now (I believe that'll ship with Hammer), so we had to call a different function to delete the object without iterating over the versions.

AFAIK, that code should work fine if we point it at Amazon.  I haven't tried it though.

I've been using RGW (with replication) in production for 2 years now, although I'm not large.  So far, all of my RGW issues have been Ceph issues.  Most of my issues are caused by my under-powered hardware, or shooting myself in the foot with aggressive
 optimizations.  Things are better with my journals on SSD, but the best thing I did was slow down with my changes.  For example, I have 7 OSD nodes and 72 OSDs.  When I add new OSDs, I add a couple at a time instead of adding all the disks in a node at once. 
 Guess how I learned that lesson. :-)

On Wed, Mar 18, 2015 at 10:03 AM, Jerry Lam 
<Jerry.Lam@xxxxxxxxxx> wrote:

Hi Chris,

Thank you for your reply. 
We are also thinking about using the S3 API but we are concerned about how compatible it is with the real S3. For instance, we would like to design the system using pre-signed URL for storing some objects. I read the ceph documentation, it does not mention
 if it supports it or not. 

My question is do you guys find that the code using the RADOS S3 API can easily run in Amazon S3 without any change? If no, how much effort it is needed to make it compatible?

Best Regards,

Jerry

From: Chris Jones <cjones@xxxxxxxxxxx>

Date: Tuesday, March 17, 2015 at 4:39 PM

To: Jerry Lam <jerry.lam@xxxxxxxxxx>

Cc: "ceph-users@xxxxxxxxxxxxxx" <ceph-users@xxxxxxxxxxxxxx>

Subject: Re:  RADOS Gateway Maturity

Hi Jerry,

I currently work at Bloomberg and we currently have a very large Ceph installation in production and we use the S3 compatible API for rados gateway. We are also re-architecting our new RGW and evaluating a different Apache configuration for a little better
 performance. We only use replicas right now, no erasure coding yet. Actually, you can take a look at our current configuration at
https://github.com/bloomberg/chef-bcpc.

-Chris

On Tue, Mar 17, 2015 at 10:40 AM, Jerry Lam 
<Jerry.Lam@xxxxxxxxxx> wrote:

Hi Ceph user,

I’m new to Ceph but I need to use Ceph as the storage for the Cloud we are building in house. 
Did anyone use RADOS Gateway in production? How mature it is in terms of compatibility with S3 / Swift?
Anyone can share their experience on it?

Best Regards,

Jerry

_______________________________________________

ceph-users mailing list

ceph-users@xxxxxxxxxxxxxx

http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

-- 

Best Regards,
Chris Jones

cjones@xxxxxxxxxxx

(p) 
770.655.0770

This message is intended exclusively for the individual or entity to which it is addressed.  This communication may contain information that is proprietary, privileged or
 confidential or otherwise legally exempt from disclosure.  If you are not the named addressee, you are not authorized to read, print, retain, copy or disseminate this message or any part of it.  If you have received this message in error, please notify the
 sender immediately by e-mail and delete all copies of the message.

_______________________________________________

ceph-users mailing list

ceph-users@xxxxxxxxxxxxxx

http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

-- 

Best Regards,
Chris Jones

cjones@xxxxxxxxxxx

(p) 770.655.0770

This message is intended exclusively for the individual or entity to which it is addressed.  This communication may contain information that is proprietary, privileged or
 confidential or otherwise legally exempt from disclosure.  If you are not the named addressee, you are not authorized to read, print, retain, copy or disseminate this message or any part of it.  If you have received this message in error, please notify the
 sender immediately by e-mail and delete all copies of the message.

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com