On Fri, Feb 15, 2013 at 6:29 AM, Ben Rowland <ben.rowland@xxxxxxxxx> wrote:
> Further to my question about reads on a degraded PG, my tests show
> that indeed reads from rgw fail when not all OSDs in a PG are up, even
> when the data is physically available on an up/in OSD.
>
> I have a "size" and "min_size" of 2 on my pool, and 2 hosts with 2
> OSDs on each. Crush map is set to write to 1 OSD on each of 2 hosts.
> After writing a file to successfully to rgw via host 1, I then stop
> all Ceph services on host 2. Attempts to read the file I just wrote
> time out after 30 seconds. Starting Ceph again on host 2 allows reads
> to proceed from host 1 once again.
>
> I see the following in ceph.log after the read times out:
>
> 2013-02-15 12:04:39.162685 osd.0 10.9.64.61:6802/19242 3 : [WRN] slow
> request 30.461867 seconds old, received at 2013-02-15 12:04:08.700630:
> osd_op(client.4345.0:21511
> 4345.365_91bf7acb-8321-494e-bc79-6ab1625162bc [getxattrs,stat,read
> 0~524288] 9.5aaf1592 RETRY) v4 currently reached pg
>
> After stopping Ceph on host 2, "ceph -s" reports:
>
> health HEALTH_WARN 514 pgs degraded; 16 pgs incomplete; 16 pgs
> stuck inactive; 632 pgs stuck unclean; recovery 44/6804 degraded
> (0.647%)
> monmap e1: 1 mons at {a=10.9.64.61:6789/0}, election epoch 1, quorum 0 a
> osdmap e155: 4 osds: 2 up, 2 in
> pgmap v4911: 632 pgs: 102 active+remapped, 514 active+degraded, 16
> incomplete; 844 MB data, 5969 MB used, 2280 MB / 8691 MB avail;
> 44/6804 degraded (0.647%)
> mdsmap e1: 0/0/1 up
>
> OSD tree just in case:
>
> # id weight type name up/down reweight
> -1 2 root default
> -3 2 rack unknownrack
> -2 1 host squeezeceph1
> 0 1 osd.0 up 1
> 2 1 osd.2 up 1
> -4 1 host squeezeceph2
> 1 1 osd.1 down 0
> 3 0 osd.3 down 0
>
> Running "osd map" on both the container and object names say host 1 is
> "acting" for that PG (not sure if I'm looking at the right pools,
> though):
>
> $ ceph osd map .rgw.buckets aa94e84a-e720-45e1-8c85-4afa7d0f6b5c
>
> osdmap e155 pool '.rgw.buckets' (9) object
> 'aa94e84a-e720-45e1-8c85-4afa7d0f6b5c' -> pg 9.494717b9 (9.1) -> up
> [0] acting [0]
>
> $ ceph osd map .rgw 91bf7acb-8321-494e-bc79-6ab1625162bc
>
> osdmap e155 pool '.rgw' (3) object
> '91bf7acb-8321-494e-bc79-6ab1625162bc' -> pg 3.1db18d16 (3.6) -> up
> [2] acting [2]
>
> Any thoughts? It doesn't seem right that taking out a single failure
> domain should cause this degradation.
Hi Ben,
Are you still seeing this? Can you enable osd logging and restart the
osds on host 1?
-sam
>
> Many thanks,
>
> Ben
>
> On Thu, Feb 14, 2013 at 11:53 PM, Ben Rowland <ben.rowland@xxxxxxxxx> wrote:
>>> On 13 Feb 2013 18:16, "Gregory Farnum" <greg@xxxxxxxxxxx> wrote:
>>> >
>>> > On Wed, Feb 13, 2013 at 3:40 AM, Ben Rowland <ben.rowland@xxxxxxxxx> wrote:
>>
>>> So it sounds from the rest of your post like you'd want to, for each
>>> pool that RGW uses (it's not just .rgw), run "ceph osd set .rgw
>>> min_size 2". (and for .rgw.buckets, etc etc)
>>
>> Thanks, that did the trick. When the number of up OSDs is less than
>> min_size, writes block for 30s then return http 500. Ceph honours my
>> crush rule in this case - adding more OSDs to only one of two failure
>> domains continues to block writes - all well and good!
>>
>>> > If this is the expected behaviour of Ceph, then it seems to prefer
>>> > write-availability over read-availability (in this case my data is
>>> > only stored on 1 OSD, thus a SPOF). Is there any way to change this
>>> > trade-off, e.g. as you can in Cassandra with its write quorums?
>>>
>>> I'm not quite sure this is describing it correctly — Ceph guarantees
>>> that anything that's been written to disk will be readable later on,
>>> and placement groups won't go active if they can't retrieve all data.
>>> The sort of flexible policies allowed by Cassandra aren't possible
>>> within Ceph — it is a strictly consistent system.
>>
>> Are objects always readable even if a PG is missing some OSDs, and
>> where it cannot recover? Example: 2 hosts each with 1 osd, pool
>> min_size is 2, with a crush rule saying to write to both hosts. I
>> write a file successfully, then one host goes down, and eventually is
>> marked 'out'. Is the file readable on the 'up' host (say if I'm
>> running rgw there?) What if the up host does not have the primary
>> copy?
>>
>> Furthermore, if Ceph is strictly consistent, how would it resolve
>> possible stale reads? Say, if in the 2 hosts example, the network
>> connection died, but min_size was set to 1. Would it be possible for
>> writes to proceed, say making edits to an existing object? Could
>> readers at the other host see stale data?
>>
>> Thanks again in advance,
>>
>> Ben
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@xxxxxxxxxxxxxxx
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
