The second scenario you mention is exactly what I?m doing? let me try to lay it out: http://i.imgur.com/tUIie1r.jpg - The orange cylinders represent master zones, whereas the white cylinders represent slave zones. - Red diamonds represent data written in the us-east-1 region, with writes at the east coast datacenter. - Yellow diamonds represent data written in the us-west-1 region, with writes at the west coast datacenter. Since ceph is used internally as well as externally, we want to ensure that writes happen to the ?local? geographical site ? so if I?m using S3 storage in the west coast DC, I want to write to pools that physically reside in the west coast DC. So, for example: s3cmd mb s3://testbucket ?bucket-location=us-west-1 I would expect this command to create a bucket in the west coast datacenter, west coast cluster, Zone W1, default placement pool. This command is successful, but creates the bucket in the us-east-1 region, with the master zone in the east coast datacenter. That?s the crux of my problem. Here?s a look at my region map: { "regions": [ { "key": "us-east-1", "val": { "name": "us-east-1", "api_name": "us-east-1", "is_master": "true", "endpoints": [ "https:\/\/us-east-1-s3.mydomain.com" ], "master_zone": "us-east-1.e1", "zones": [ { "name": "us-east-1.e1", "endpoints": [ "https:\/\/e1-01.mydomain.com\/", "https:\/\/e1-02.mydomain.com\/" ], "log_meta": "true", "log_data": "true"}, { "name": "us-east-1.e2", "endpoints": [ "https:\/\/e2-01.mydomain.com\/", "https:\/\/e2-02.mydomain.com\/" ], "log_meta": "true", "log_data": "true"}], "placement_targets": [ { "name": "default-placement", "tags": [] } ], "default_placement": "default-placement"} }, { "key": "us-west-1", "val": { "name": "us-west-1", "api_name": "us-west-1", "is_master": "false", "endpoints": [ "https:\/\/us-west-1-s3.mydomain.com" ], "master_zone": "us-west-1.w1", "zones": [ { "name": "us-west-1.w1", "endpoints": [ "https:\/\/w1-01.mydomain.com\/", "https:\/\/w1-02.mydomain.com\/" ], "log_meta": "true", "log_data": "true"}, { "name": "us-west-1.w2", "endpoints": [ "https:\/\/w2-01.mydomain.com\/", "https:\/\/w2-02.mydomain.com\/" ], "log_meta": "true", "log_data": "true"}], "placement_targets": [ { "name": "default-placement", "tags": [] } ], "default_placement": "default-placement"} }], "master_region": "us-east-1" } From: Craig Lewis [mailto:clewis@xxxxxxxxxxxxxxxxxx] Sent: Monday, July 14, 2014 1:18 PM To: Bachelder, Kurt; Ceph Users Subject: Re: Creating a bucket on a non-master region in a multi-region configuration with unified namespace/replication There's no reason you can't create another set of zones that have a master in us-west, call it us-west-2. Then users that need low latency to us-west write to http://us-west-2.cluster/, and users that need low latency to us-east write to http://us-east-1.cluster/. In general, you want your replication zones geographically separated. In my setup, I have a single region, us. I have the primary zone in California and the secondary zone in Texas. When I create objects in the primary, they appear in the secondary zone in a couple minutes. Then if I need low latency read access in Texas, I read from the secondary. I don't currently need to write from Texas. When I do, I'll create a new set of zones that are primary in Texas and secondary in California. Then Texas can write to that new zone. On Mon, Jul 14, 2014 at 7:50 AM, Bachelder, Kurt <Kurt.Bachelder at sierra-cedar.com<mailto:Kurt.Bachelder at sierra-cedar.com>> wrote: Hi Craig ? Thanks for the info? My understanding is the same as what you?ve laid out below ? I?ve scraped together enough information to understand that Ceph should be working the same as Amazon S3, which is a relief. Here?s my issue ? I have 2 regions, geographically separated. Each region contains 2 zones, which replicate metadata + data ? and that works perfectly ? no issues whatsoever. However, when I add regional replication for a unified namespace, the location constraint SEEMS to be completely ignored. Let?s say ?us-east? is my master region and ?us-west? is my slave ? for latency reasons, I want the actual data to be stored in us-west. So I submit a command to us-east to create a bucket with location_constraint=us-west. However, the radosgw seems to completely ignore the constraint and it creates the bucket in us-east. When I start writing data to the bucket, it physically resides in us-east, not us-west, which doesn?t work for us from a latency standpoint. Creating users and buckets in us-east works fine, and the metadata filters down to all other regions/zones ? so I know the replication working to some extent. Oh, and you?re right about the redirect ? when I try to create a bucket against the ?slave? region us-west, it DOES redirect the request to the master region? but still fails to create the bucket in us-west. It is created in pools in us-east instead. Frustrating ? the documentation is so sparse around this whole multi-region setup ? especially in regards to location constraints. I?m still not sure whether my configuration is off or whether the RGW just isn?t working as expected? Thank you for your reply ? keep in touch if you end up doing some multi-region replication, would love to hear your experience. Kurt From: Craig Lewis [mailto:clewis@xxxxxxxxxxxxxxxxxx<mailto:clewis at centraldesktop.com>] Sent: Saturday, July 12, 2014 7:51 PM To: Bachelder, Kurt Subject: Re: Creating a bucket on a non-master region in a multi-region configuration with unified namespace/replication Disclaimer: I haven't used Amazon S3 yet, so I'm only familiar with regions, zones, and placements as they apply to Ceph. I'm pretty sure Ceph is trying hard to emulate Amazon, but it's obviously missing some features. I was hoping somebody else would chime in, but I'll give it a shot. Bucket creation (really, all writes) should only occur in a master zone. Zones (within a region) are intended to have data+metadata replication, so the primary zone will get the writes, then replicate everything to the secondary zone(s). You can read from the primary or secondary, as long as you remember that replication is asynchronous, and generally takes a few minutes even when it's keeping up. You can have multiple zones in a region that partition instead of replicate as well. For example, us-west-1 replicates to us-east-1, and us-east-2 replicates to us-west-2. Regions can have metadata replication (without data replication), if you want a globally unique list of users and pools. It's up to you if you want that. If you plan to move data between regions, then you probably do. It's not required if you want us-west-1/bucket/ and us-east-2/bucket/ to be different objects (but it's confusing). If you want geo-replication, each zone in a region should have a secondary zone with data replication. zones & data replication are the method of accomplishing geo-replication. Regions are a way to isolate data for latency/legal reasons. Placement targets are setup inside a region. They let the end user select different storage mechanisms that you've configured. For example, you might have default (3x replication on HDD), fast (2x replication on SSD), and durable (4x replication on HDD). The user could pick the most appropriate one when creating a bucket. I haven't tested non-default placement targets and replication yet, so I don't know exactly what happens, or how to set it up. I haven't setup multiple regions yet, just multiple zones in one region. I'm really not sure what the behavior should be if you have metadata replication enabled, and create an object in region1, then try to read it in region2. I would hope it returns a 302 redirection to the primary region and zone. So when you say The issue we?re having is with creating a bucket in the slave region. As we understand it, a bucket is considered ?metadata?, and should be created via the master region, but using a ?location constraint? to ensure the objects within the bucket are physically stored in the secondary region. In Ceph, you should create the bucket in the master zone of your region, and it will be physically stored in the master and slave zones. Does that answer your question? On Thu, Jul 10, 2014 at 7:40 AM, Bachelder, Kurt <Kurt.Bachelder at sierra-cedar.com<mailto:Kurt.Bachelder at sierra-cedar.com>> wrote: OK ? I found this information (http://comments.gmane.org/gmane.comp.file-systems.ceph.user/4992): When creating a bucket users can specify which placement target they want to use with that specific bucket (by using the S3 create bucket location constraints field, format is location_constraints='[region][:placement-target]', e.g., location_constraints=':fast-placement'). This looks like exactly what I want to do ? but HOW is the location_constraints field used to specify a region for bucket creation? Is there any documentation around this? From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx<mailto:ceph-users-bounces at lists.ceph.com>] On Behalf Of Bachelder, Kurt Sent: Tuesday, July 08, 2014 3:22 PM To: ceph-users at ceph.com<mailto:ceph-users at ceph.com> Subject: Creating a bucket on a non-master region in a multi-region configuration with unified namespace/replication Hi ? We have a multi-region configuration with metadata replication between the regions for a unified namespace. Each region has pools in a different cluster. Users created in the master region are replicated to the slave region without any issues ? we can get user info, and everything is consistent. Buckets created in the master region are also indexed in the slave region without any issues. The issue we?re having is with creating a bucket in the slave region. As we understand it, a bucket is considered ?metadata?, and should be created via the master region, but using a ?location constraint? to ensure the objects within the bucket are physically stored in the secondary region. However, there is a lack of documentation on exactly *how* to accomplish this ? we?ve attempted with s3cmd using the ?bucket-location parameter to point to the slave region, but the bucket is still created in the master region. We noticed that the ceph S3 API documentation (http://ceph.com/docs/master/radosgw/s3/#features-support) shows that the Amazon S3 functionality for ?bucket location? is not supported? but I can?t imagine that restriction renders the entire slave region setup useless? does it? So how are buckets created in a secondary region? Thanks in advance! Kurt _______________________________________________ ceph-users mailing list ceph-users at lists.ceph.com<mailto:ceph-users at lists.ceph.com> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140714/70f7807c/attachment.htm>
 |