On Mon, Dec 1, 2014 at 4:26 PM, Ben <b@benjackson.email> wrote: > On 2014-12-02 11:25, Yehuda Sadeh wrote: >> >> On Mon, Dec 1, 2014 at 4:23 PM, Ben <b@benjackson.email> wrote: ... >>>>>>> How can I tell if the shard has an object in it from the logs? >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Search for a different sequence (e.g., search for rgw.gc_remove). >>>>>> >>>>>> Yehuda >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> 0 Results in the logs for rgw.gc_remove >>>> >>>> >>>> >>>> Well, something is modifying the gc log. Do you happen to have more >>>> than one radosgw running on the same cluster? >>>> >>>> Yehuda >>> >>> >>> >>> We have 2 radosgw servers >>> obj01 and obj02 >> >> >> Are both of them pointing at the same zone? > > > Yes, they are load balanced Well, the gc log show entries, and then it doesn't, so something clears these up. Try reproducing again with logs on, see if you see new entries in the rgw logs. If you don't see these, maybe try turning on 'debug ms = 1' on your osds (ceph tell osd.* injectargs '--debug_ms 1'), and look in your osd logs for such messages. These might give you some hint for their origin. Also, could it be that you ran 'radosgw-admin gc process', instead of waiting for the gc cycle to complete? Yehuda _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com