On Mon, Dec 1, 2014 at 3:47 PM, Ben <b@benjackson.email> wrote: > On 2014-12-02 10:40, Yehuda Sadeh wrote: >> >> On Mon, Dec 1, 2014 at 3:20 PM, Ben <b@benjackson.email> wrote: >>> >>> On 2014-12-02 09:25, Yehuda Sadeh wrote: >>>> >>>> >>>> On Mon, Dec 1, 2014 at 2:10 PM, Ben <b@benjackson.email> wrote: >>>>> >>>>> >>>>> On 2014-12-02 08:39, Yehuda Sadeh wrote: >>>>>> >>>>>> >>>>>> >>>>>> On Sat, Nov 29, 2014 at 2:26 PM, Ben <b@benjackson.email> wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 29/11/14 11:40, Yehuda Sadeh wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Fri, Nov 28, 2014 at 1:38 PM, Ben <b@benjackson.email> wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On 29/11/14 01:50, Yehuda Sadeh wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thu, Nov 27, 2014 at 9:22 PM, Ben <b@benjackson.email> wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 2014-11-28 15:42, Yehuda Sadeh wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Nov 27, 2014 at 2:15 PM, b <b@benjackson.email> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On 2014-11-27 11:36, Yehuda Sadeh wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Wed, Nov 26, 2014 at 3:49 PM, b <b@benjackson.email> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 2014-11-27 10:21, Yehuda Sadeh wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Wed, Nov 26, 2014 at 3:09 PM, b <b@benjackson.email> >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On 2014-11-27 09:38, Yehuda Sadeh wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Wed, Nov 26, 2014 at 2:32 PM, b <b@benjackson.email> >>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I've been deleting a bucket which originally had 60TB of >>>>>>>>>>>>>>>>>>> data >>>>>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>>> it, >>>>>>>>>>>>>>>>>>> with >>>>>>>>>>>>>>>>>>> our cluster doing only 1 replication, the total usage was >>>>>>>>>>>>>>>>>>> 120TB. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I've been deleting the objects slowly using S3 browser, >>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>> I >>>>>>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>>>>> see >>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>> bucket usage is now down to around 2.5TB or 5TB with >>>>>>>>>>>>>>>>>>> duplication, >>>>>>>>>>>>>>>>>>> but >>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>> usage in the cluster has not changed. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I've looked at garbage collection (radosgw-admin gc list >>>>>>>>>>>>>>>>>>> --include >>>>>>>>>>>>>>>>>>> all) >>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>> it just reports square brackets "[]" >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I've run radosgw-admin temp remove --date=2014-11-20, and >>>>>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>>> doesn't >>>>>>>>>>>>>>>>>>> appear >>>>>>>>>>>>>>>>>>> to have any effect. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Is there a way to check where this space is being >>>>>>>>>>>>>>>>>>> consumed? >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Running 'ceph df' the USED space in the buckets pool is >>>>>>>>>>>>>>>>>>> not >>>>>>>>>>>>>>>>>>> showing >>>>>>>>>>>>>>>>>>> any >>>>>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>>> the 57TB that should have been freed up from the deletion >>>>>>>>>>>>>>>>>>> so >>>>>>>>>>>>>>>>>>> far. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Running 'radosgw-admin bucket stats | jshon | grep >>>>>>>>>>>>>>>>>>> size_kb_actual' >>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>> adding up all the buckets usage, this shows that the >>>>>>>>>>>>>>>>>>> space >>>>>>>>>>>>>>>>>>> has >>>>>>>>>>>>>>>>>>> been >>>>>>>>>>>>>>>>>>> freed >>>>>>>>>>>>>>>>>>> from the bucket, but the cluster is all sorts of messed >>>>>>>>>>>>>>>>>>> up. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> ANY IDEAS? What can I look at? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Can you run 'radosgw-admin gc list --include-all'? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Yehuda >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I've done it before, and it just returns square brackets [] >>>>>>>>>>>>>>>>> (see >>>>>>>>>>>>>>>>> below) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> radosgw-admin gc list --include-all >>>>>>>>>>>>>>>>> [] >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Do you know which of the rados pools have all that extra >>>>>>>>>>>>>>>> data? >>>>>>>>>>>>>>>> Try >>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>> list that pool's objects, verify that there are no surprises >>>>>>>>>>>>>>>> there >>>>>>>>>>>>>>>> (e.g., use 'rados -p <pool> ls'). >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Yehuda >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I'm just running that command now, and its taking some time. >>>>>>>>>>>>>>> There >>>>>>>>>>>>>>> is >>>>>>>>>>>>>>> a >>>>>>>>>>>>>>> large number of objects. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Once it has finished, what should I be looking for? >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> I assume the pool in question is the one that holds your >>>>>>>>>>>>>> objects >>>>>>>>>>>>>> data? >>>>>>>>>>>>>> You should be looking for objects that are not expected to >>>>>>>>>>>>>> exist >>>>>>>>>>>>>> anymore, and objects of buckets that don't exist anymore. The >>>>>>>>>>>>>> problem >>>>>>>>>>>>>> here is to identify these. >>>>>>>>>>>>>> I suggest starting by looking at all the existing buckets, >>>>>>>>>>>>>> compose >>>>>>>>>>>>>> a >>>>>>>>>>>>>> list of all the bucket prefixes for the existing buckets, and >>>>>>>>>>>>>> try >>>>>>>>>>>>>> to >>>>>>>>>>>>>> look whether there are objects that have different prefixes. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Yehuda >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Any ideas? I've found the prefix, the number of objects in the >>>>>>>>>>>>> pool >>>>>>>>>>>>> that >>>>>>>>>>>>> match that prefix numbers in the 21 millions >>>>>>>>>>>>> The actual 'radosgw-admin bucket stats' command reports it as >>>>>>>>>>>>> only >>>>>>>>>>>>> having >>>>>>>>>>>>> 1.2 million. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Well, the objects you're seeing are raw objects, and since rgw >>>>>>>>>>>> stripes >>>>>>>>>>>> the data, it is expected to have more raw objects than objects >>>>>>>>>>>> in >>>>>>>>>>>> the >>>>>>>>>>>> bucket. Still, it seems that you have much too many of these. >>>>>>>>>>>> You >>>>>>>>>>>> can >>>>>>>>>>>> try to check whether there are pending multipart uploads that >>>>>>>>>>>> were >>>>>>>>>>>> never completed using the S3 api. >>>>>>>>>>>> At the moment there's no easy way to figure out which raw >>>>>>>>>>>> objects >>>>>>>>>>>> are >>>>>>>>>>>> not supposed to exist. The process would be like this: >>>>>>>>>>>> 1. rados ls -p <data pool> >>>>>>>>>>>> keep the list sorted >>>>>>>>>>>> 2. list objects in the bucket >>>>>>>>>>>> 3. for each object in (2), do: radosgw-admin object stat >>>>>>>>>>>> --bucket=<bucket> --object=<object> --rgw-cache-enabled=false >>>>>>>>>>>> (disabling the cache so that it goes quicker) >>>>>>>>>>>> 4. look at the result of (3), and generate a list of all the >>>>>>>>>>>> parts. >>>>>>>>>>>> 5. sort result of (4), compare it to (1) >>>>>>>>>>>> >>>>>>>>>>>> Note that if you're running firefly or later, the raw objects >>>>>>>>>>>> are >>>>>>>>>>>> not >>>>>>>>>>>> specified explicitly in the command you run at (3), so you might >>>>>>>>>>>> need >>>>>>>>>>>> a different procedure, e.g., find out the raw objects random >>>>>>>>>>>> string >>>>>>>>>>>> that is being used, remove it from the list generated in 1, >>>>>>>>>>>> etc.) >>>>>>>>>>>> >>>>>>>>>>>> That's basically it. >>>>>>>>>>>> I'll be interested to figure out what happened, why the garbage >>>>>>>>>>>> collection didn't work correctly. You could try verifying that >>>>>>>>>>>> it's >>>>>>>>>>>> working by: >>>>>>>>>>>> - create an object (let's say ~10MB in size). >>>>>>>>>>>> - radosgw-admin object stat --bucket=<bucket> >>>>>>>>>>>> --object=<object> >>>>>>>>>>>> (keep this info, see >>>>>>>>>>>> - remove the object >>>>>>>>>>>> - run radosgw-admin gc list --include-all and verify that the >>>>>>>>>>>> raw >>>>>>>>>>>> parts are listed there >>>>>>>>>>>> - wait a few hours, repeat last step, see that the parts >>>>>>>>>>>> don't >>>>>>>>>>>> appear >>>>>>>>>>>> there anymore >>>>>>>>>>>> - run rados -p <pool> ls, check to see if the raw objects >>>>>>>>>>>> still >>>>>>>>>>>> exist >>>>>>>>>>>> >>>>>>>>>>>> Yehuda >>>>>>>>>>>> >>>>>>>>>>>>> Not sure where to go from here, and our cluster is slowly >>>>>>>>>>>>> filling >>>>>>>>>>>>> up >>>>>>>>>>>>> while >>>>>>>>>>>>> not clearing any space. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I did the last section: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I'll be interested to figure out what happened, why the garbage >>>>>>>>>>>> collection didn't work correctly. You could try verifying that >>>>>>>>>>>> it's >>>>>>>>>>>> working by: >>>>>>>>>>>> - create an object (let's say ~10MB in size). >>>>>>>>>>>> - radosgw-admin object stat --bucket=<bucket> >>>>>>>>>>>> --object=<object> >>>>>>>>>>>> (keep this info, see >>>>>>>>>>>> - remove the object >>>>>>>>>>>> - run radosgw-admin gc list --include-all and verify that the >>>>>>>>>>>> raw >>>>>>>>>>>> parts are listed there >>>>>>>>>>>> - wait a few hours, repeat last step, see that the parts >>>>>>>>>>>> don't >>>>>>>>>>>> appear >>>>>>>>>>>> there anymore >>>>>>>>>>>> - run rados -p <pool> ls, check to see if the raw objects >>>>>>>>>>>> still >>>>>>>>>>>> exist >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I added the file, did a stat and it displayed the json output >>>>>>>>>>> I removed the object and then tried to stat the object, this time >>>>>>>>>>> it >>>>>>>>>>> failed >>>>>>>>>>> to stat the object >>>>>>>>>>> After this, I ran the gc list include all command and it >>>>>>>>>>> displayed >>>>>>>>>>> nothing >>>>>>>>>>> but the square brackets [] >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Was the object larger than 512k? Also, did you do it within the >>>>>>>>>> 300 >>>>>>>>>> seconds after removing the object? >>>>>>>>>> >>>>>>>>>> There should exist a garbage collection pool (by default .rgw.gc, >>>>>>>>>> but >>>>>>>>>> it can be something different if you configured your zone >>>>>>>>>> differently), can you verify that you have it, and if so, what >>>>>>>>>> does >>>>>>>>>> it >>>>>>>>>> contain? >>>>>>>>>> >>>>>>>>>> Yehuda >>>>>>>>>> >>>>>>>>> Yes, the object was 10M. As soon as I had deleted it from the >>>>>>>>> bucket, >>>>>>>>> I >>>>>>>>> ran >>>>>>>>> the command to check garbage collection. >>>>>>>>> There is a .rgw.gc pool, we haven't changed it from default. It >>>>>>>>> contains >>>>>>>>> a >>>>>>>>> number of objects ~7800, but the size of the files is 0kb >>>>>>>>> >>>>>>>> They're expected to be 0kb, the data only resides in their omap, and >>>>>>>> that's not reflected in the objects size. You could run 'rados >>>>>>>> listomapkeys' on these. >>>>>>>> >>>>>>>>>>> Maybe garbage collection isn't working properly.. >>>>>>>>>>> >>>>>>>>>>> our gc settings are the following, we have 2 object gateways in >>>>>>>>>>> our >>>>>>>>>>> cluster >>>>>>>>>>> too client.radosgw.obj01 and client.radosgw.obj02 (from >>>>>>>>>>> ceph.conf) >>>>>>>>>>> [client.radosgw.obj01] >>>>>>>>>>> rgw dns name = ceph.###.### >>>>>>>>>>> host = obj01 >>>>>>>>>>> keyring = /etc/ceph/keyring.radosgw.obj01 >>>>>>>>>>> rgw socket path = /tmp/radosgw.sock >>>>>>>>>>> log file = /var/log/ceph/radosgw.log >>>>>>>>>>> rgw data = /var/lib/ceph/radosgw/obj01 >>>>>>>>>>> rgw thread pool size = 128 >>>>>>>>>>> rgw print continue = True >>>>>>>>>>> debug rgw = 0 >>>>>>>>>>> rgw enable ops log = False >>>>>>>>>>> log to stderr = False >>>>>>>>>>> rgw enable usage log = False >>>>>>>>>>> rgw gc max objs = 7877 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> You should put this line (rgw gx max objs) in the global section of >>>>>>>> your ceph.conf. Either that, or run your radosgw-admin command with >>>>>>>> '-n client.radosgw.obj02'. That might change some of the results >>>>>>>> you're seeing (radosgw-admin gc list --include-all, etc.). >>>>>>>> >>>>>>>> Yehuda >>>>>>>> >>>>>>>>>>> rgw gc obj min wait = 300 >>>>>>>>>>> rgw gc processor period = 600 >>>>>>>>>>> rgw init timeout = 180 >>>>>>>>>>> rgw gc processor max time = 600 >>>>>>>>>>> [client.radosgw.obj02] >>>>>>>>>>> rgw dns name = ceph.###.### >>>>>>>>>>> host = obj02 >>>>>>>>>>> keyring = /etc/ceph/keyring.radosgw.obj02 >>>>>>>>>>> rgw socket path = /tmp/radosgw.sock >>>>>>>>>>> log file = /var/log/ceph/radosgw.log >>>>>>>>>>> rgw data = /var/lib/ceph/radosgw/obj02 >>>>>>>>>>> rgw thread pool size = 128 >>>>>>>>>>> rgw print continue = True >>>>>>>>>>> debug rgw = 0 >>>>>>>>>>> rgw enable ops log = False >>>>>>>>>>> log to stderr = False >>>>>>>>>>> rgw enable usage log = False >>>>>>>>>>> rgw gc max objs = 7877 >>>>>>>>>>> rgw gc obj min wait = 300 >>>>>>>>>>> rgw gc processor period = 600 >>>>>>>>>>> rgw init timeout = 180 >>>>>>>>>>> rgw gc processor max time = 600 >>>>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> I've finally deleted the entire bucket. All 60TB cleared from the >>>>>>> bucket, >>>>>>> the bucket no longer exists. >>>>>>> >>>>>>> Yet running rados ls -p .rgw.buckets | grep '4804.14' still lists all >>>>>>> the >>>>>>> _shadow_ files that have that buckets prefix. >>>>>>> >>>>>>> Any ideas why these aren't being deleted/cleared up by garbage >>>>>>> collection? >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Are there any errors in the log? Can you provide a log (debug rgw = >>>>>> 20, debug ms = 1) of the radosgw through the garbage collection stage? >>>>>> >>>>>> Yehuda >>>>> >>>>> >>>>> >>>>> >>>>> Hi Yehuda, >>>>> >>>>> Here is a snapshot of the log. >>>>> >>>>> I created a new bucket, uploaded a 25mb file and then deleted it >>>>> i did a gc list --include all and could see the shadow files >>>>> >>>>> I then forced a gc process, and the log has gone crazy with this stuff >>>>> in >>>>> it, and it is still going. >>>>> >>>>> >>>>> 2014-12-02 09:07:59.634544 7f86256f3700 1 -- 10.150.2.2:0/1009990 --> >>>>> 10.150.2.13:6816/17803 -- ping v1 -- ?+0 0x7f8558000d20 con 0x1e9c470 >>>>> 2014-12-02 09:07:59.634578 7f86256f3700 1 -- 10.150.2.2:0/1009990 --> >>>>> 10.150.2.12:6830/3830 -- ping v1 -- ?+0 0x7f8558001120 con 0x1e93090 >>>>> 2014-12-02 09:07:59.634592 7f86256f3700 1 -- 10.150.2.2:0/1009990 --> >>>>> 10.150.2.14:6818/3517 -- ping v1 -- ?+0 0x7f8558001700 con 0x1e9d230 >>>>> 2014-12-02 09:07:59.634607 7f86256f3700 1 -- 10.150.2.2:0/1009990 --> >>>>> 10.150.2.10:6806/3397 -- ping v1 -- ?+0 0x7f8558006050 con 0x1e9e0e0 >>>>> 2014-12-02 09:07:59.634621 7f86256f3700 1 -- 10.150.2.2:0/1009990 --> >>>>> 10.150.2.19:6830/4000 -- ping v1 -- ?+0 0x7f8558000a10 con 0x1e9fdc0 >>>>> 2014-12-02 09:07:59.634634 7f86256f3700 1 -- 10.150.2.2:0/1009990 --> >>>>> 10.150.2.11:6809/3451 -- ping v1 -- ?+0 0x7f8558001d50 con 0x1e9ef10 >>>>> 2014-12-02 09:07:59.634647 7f86256f3700 1 -- 10.150.2.2:0/1009990 --> >>>>> 10.150.2.19:6803/3449 -- ping v1 -- ?+0 0x7f8558001fe0 con 0x1ea1aa0 >>>>> 2014-12-02 09:07:59.634660 7f86256f3700 1 -- 10.150.2.2:0/1009990 --> >>>>> 10.150.2.17:6827/14001 -- ping v1 -- ?+0 0x7f85580028c0 con 0x1ea0bf0 >>>>> 2014-12-02 09:07:59.637762 7f86277f8700 1 -- 10.150.2.2:0/1009990 <== >>>>> osd.108 10.150.2.18:6827/13867 158 ==== osd_op_reply(14761 gc.1283 >>>>> [call] >>>>> v0'0 uv98856 ondisk = 0) v6 ==== 174+0+11 (1634423764 0 1993775135) >>>>> 0x7f8570006050 con 0x1f1e4d0 >>>>> 2014-12-02 09:07:59.637937 7f8630ba57c0 1 -- 10.150.2.2:0/1009990 --> >>>>> 10.150.2.18:6827/13867 -- osd_op(client.6359480.0:14762 gc.1283 [call >>>>> lock.unlock] 6.10bb9719 ondisk+write e19074) v4 -- ?+0 0x1f30a80 con >>>>> 0x1f1e4d0 >>>>> 2014-12-02 09:07:59.652166 7f86277f8700 1 -- 10.150.2.2:0/1009990 <== >>>>> osd.108 10.150.2.18:6827/13867 159 ==== osd_op_reply(14762 gc.1283 >>>>> [call] >>>>> v19074'98857 uv98857 ondisk = 0) v6 ==== 174+0+0 (1358419719 0 0) >>>>> 0x7f8570006050 con 0x1f1e4d0 >>>>> 2014-12-02 09:07:59.652293 7f8630ba57c0 1 -- 10.150.2.2:0/1009990 --> >>>>> 10.150.2.15:6805/1032 -- osd_op(client.6359480.0:14763 gc.1284 [call >>>>> lock.lock] 6.4a5f5aa9 ondisk+write e19074) v4 -- ?+0 0x1f30a80 con >>>>> 0x1f290a0 >>>>> 2014-12-02 09:07:59.670108 7f86277f8700 1 -- 10.150.2.2:0/1009990 <== >>>>> osd.53 10.150.2.15:6805/1032 130 ==== osd_op_reply(14763 gc.1284 [call] >>>>> v19074'100398 uv100398 ondisk = 0) v6 ==== 174+0+0 (3807569510 0 0) >>>>> 0x7f85b0000b60 con 0x1f290a0 >>>>> 2014-12-02 09:07:59.670239 7f8630ba57c0 1 -- 10.150.2.2:0/1009990 --> >>>>> 10.150.2.15:6805/1032 -- osd_op(client.6359480.0:14764 gc.1284 [call >>>>> rgw.gc_list] 6.4a5f5aa9 ack+read e19074) v4 -- ?+0 0x1f30a80 con >>>>> 0x1f290a0 >>>>> 2014-12-02 09:07:59.673871 7f86277f8700 1 -- 10.150.2.2:0/1009990 <== >>>>> osd.53 10.150.2.15:6805/1032 131 ==== osd_op_reply(14764 gc.1284 [call] >>>>> v0'0 >>>>> uv100398 ondisk = 0) v6 ==== 174+0+11 (4101371043 0 1993775135) >>>>> 0x7f85b0000b60 con 0x1f290a0 >>>>> 2014-12-02 09:07:59.674042 7f8630ba57c0 1 -- 10.150.2.2:0/1009990 --> >>>>> 10.150.2.15:6805/1032 -- osd_op(client.6359480.0:14765 gc.1284 [call >>>>> lock.unlock] 6.4a5f5aa9 ondisk+write e19074) v4 -- ?+0 0x1f30a80 con >>>>> 0x1f290a0 >>>>> 2014-12-02 09:07:59.686255 7f86277f8700 1 -- 10.150.2.2:0/1009990 <== >>>>> osd.53 10.150.2.15:6805/1032 132 ==== osd_op_reply(14765 gc.1284 [call] >>>>> v19074'100399 uv100399 ondisk = 0) v6 ==== 174+0+0 (2220917153 0 0) >>>>> 0x7f85b0000b60 con 0x1f290a0 >>>>> 2014-12-02 09:07:59.686376 7f8630ba57c0 1 -- 10.150.2.2:0/1009990 --> >>>>> 10.150.2.19:6812/3767 -- osd_op(client.6359480.0:14766 gc.1285 [call >>>>> lock.lock] 6.9f912e07 ondisk+write e19074) v4 -- ?+0 0x1f1f560 con >>>>> 0x1f0ea90 >>>>> 2014-12-02 09:07:59.721432 7f86277f8700 1 -- 10.150.2.2:0/1009990 <== >>>>> osd.83 10.150.2.19:6812/3767 163 ==== osd_op_reply(14766 gc.1285 [call] >>>>> v19074'302537 uv302537 ondisk = 0) v6 ==== 174+0+0 (2463619648 0 0) >>>>> 0x7f85b8000900 con 0x1f0ea90 >>>>> 2014-12-02 09:07:59.721591 7f8630ba57c0 1 -- 10.150.2.2:0/1009990 --> >>>>> 10.150.2.19:6812/3767 -- osd_op(client.6359480.0:14767 gc.1285 [call >>>>> rgw.gc_list] 6.9f912e07 ack+read e19074) v4 -- ?+0 0x1f30a80 con >>>>> 0x1f0ea90 >>>>> 2014-12-02 09:07:59.725699 7f86277f8700 1 -- 10.150.2.2:0/1009990 <== >>>>> osd.83 10.150.2.19:6812/3767 164 ==== osd_op_reply(14767 gc.1285 [call] >>>>> v0'0 >>>>> uv302537 ondisk = 0) v6 ==== 174+0+11 (3924946656 0 1993775135) >>>>> 0x7f85b8000900 con 0x1f0ea90 >>>>> 2014-12-02 09:07:59.725868 7f8630ba57c0 1 -- 10.150.2.2:0/1009990 --> >>>>> 10.150.2.19:6812/3767 -- osd_op(client.6359480.0:14768 gc.1285 [call >>>>> lock.unlock] 6.9f912e07 ondisk+write e19074) v4 -- ?+0 0x1f1f560 con >>>>> 0x1f0ea90 >>>>> 2014-12-02 09:07:59.738287 7f86277f8700 1 -- 10.150.2.2:0/1009990 <== >>>>> osd.83 10.150.2.19:6812/3767 165 ==== osd_op_reply(14768 gc.1285 [call] >>>>> v19074'302538 uv302538 ondisk = 0) v6 ==== 174+0+0 (314638285 0 0) >>>>> 0x7f85b8000900 con 0x1f0ea90 >>>>> 2014-12-02 09:07:59.738415 7f8630ba57c0 1 -- 10.150.2.2:0/1009990 --> >>>>> 10.150.2.10:6803/3331 -- osd_op(client.6359480.0:14769 gc.1286 [call >>>>> lock.lock] 6.61beb660 ondisk+write e19074) v4 -- ?+0 0x1f30a80 con >>>>> 0x1f1ce50 >>>>> 2014-12-02 09:07:59.764523 7f86277f8700 1 -- 10.150.2.2:0/1009990 <== >>>>> osd.1 >>>>> 10.150.2.10:6803/3331 145 ==== osd_op_reply(14769 gc.1286 [call] >>>>> v19074'136677 uv136677 ondisk = 0) v6 ==== 174+0+0 (3559159328 0 0) >>>>> 0x7f8540002dd0 con 0x1f1ce50 >>>>> 2014-12-02 09:07:59.764670 7f8630ba57c0 1 -- 10.150.2.2:0/1009990 --> >>>>> 10.150.2.10:6803/3331 -- osd_op(client.6359480.0:14770 gc.1286 [call >>>>> rgw.gc_list] 6.61beb660 ack+read e19074) v4 -- ?+0 0x1f1f560 con >>>>> 0x1f1ce50 >>>>> 2014-12-02 09:07:59.767836 7f86277f8700 1 -- 10.150.2.2:0/1009990 <== >>>>> osd.1 >>>>> 10.150.2.10:6803/3331 146 ==== osd_op_reply(14770 gc.1286 [call] v0'0 >>>>> uv136677 ondisk = 0) v6 ==== 174+0+11 (1532209036 0 1993775135) >>>>> 0x7f8540002dd0 con 0x1f1ce50 >>>>> 2014-12-02 09:07:59.768009 7f8630ba57c0 1 -- 10.150.2.2:0/1009990 --> >>>>> 10.150.2.10:6803/3331 -- osd_op(client.6359480.0:14771 gc.1286 [call >>>>> lock.unlock] 6.61beb660 ondisk+write e19074) v4 -- ?+0 0x1f30a80 con >>>>> 0x1f1ce50 >>>>> 2014-12-02 09:07:59.780792 7f86277f8700 1 -- 10.150.2.2:0/1009990 <== >>>>> osd.1 >>>>> 10.150.2.10:6803/3331 147 ==== osd_op_reply(14771 gc.1286 [call] >>>>> v19074'136678 uv136678 ondisk = 0) v6 ==== 174+0+0 (1412649901 0 0) >>>>> 0x7f8540002dd0 con 0x1f1ce50 >>>>> 2014-12-02 09:07:59.780968 7f8630ba57c0 1 -- 10.150.2.2:0/1009990 --> >>>>> 10.150.2.12:6833/4016 -- osd_op(client.6359480.0:14772 gc.1287 [call >>>>> lock.lock] 6.9bfdc387 ondisk+write e19074) v4 -- ?+0 0x1f1f560 con >>>>> 0x1f3b2f0 >>>>> 2014-12-02 09:07:59.804154 7f86277f8700 1 -- 10.150.2.2:0/1009990 <== >>>>> osd.65 10.150.2.12:6833/4016 118 ==== osd_op_reply(14772 gc.1287 [call] >>>>> v19074'279063 uv279063 ondisk = 0) v6 ==== 174+0+0 (4117862336 0 0) >>>>> 0x7f85d80024b0 con 0x1f3b2f0 >>>>> 2014-12-02 09:07:59.804279 7f8630ba57c0 1 -- 10.150.2.2:0/1009990 --> >>>>> 10.150.2.12:6833/4016 -- osd_op(client.6359480.0:14773 gc.1287 [call >>>>> rgw.gc_list] 6.9bfdc387 ack+read e19074) v4 -- ?+0 0x1f30a80 con >>>>> 0x1f3b2f0 >>>>> >>>> >>>> >>>> Can you find the part of the log where it's handling a shard that >>>> actually has an object inside? These are all just iterating over empty >>>> shards. >>>> >>>> Yehuda >>> >>> >>> >>> How can I tell if the shard has an object in it from the logs? >> >> >> Search for a different sequence (e.g., search for rgw.gc_remove). >> >> Yehuda > > > > 0 Results in the logs for rgw.gc_remove Well, something is modifying the gc log. Do you happen to have more than one radosgw running on the same cluster? Yehuda _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |