On Fri, Jan 15, 2016 at 5:04 PM, seapasulli@xxxxxxxxxxxx
<seapasulli@xxxxxxxxxxxx> wrote:
> I have looked all over and I do not see any explicit mention of
> "NWS_NEXRAD_NXL2DP_PAKC_20150101110000_20150101115959" in the logs nor do I
> see a timestamp from November 4th although I do see log rotations dating
> back to october 15th. I don't think it's possible it wasn't logged so I am
> going through the bucket logs from the 'radosgw-admin log show --object'
> side and I found the following::
>
> 4604932 {
> 4604933 "bucket": "noaa-nexrad-l2",
> 4604934 "time": "2015-11-04 21:29:27.346509Z",
> 4604935 "time_local": "2015-11-04 15:29:27.346509",
> 4604936 "remote_addr": "",
> 4604937 "object_owner": "b05f707271774dbd89674a0736c9406e",
> 4604938 "user": "b05f707271774dbd89674a0736c9406e",
> 4604939 "operation": "PUT",
I'd expect a multipart upload completion to be done with a POST, not a PUT.
> 4604940 "uri":
> "\/noaa-nexrad-l2\/2015\/01\/01\/PAKC\/NWS_NEXRAD_NXL2DP_PAKC_20150101110000_20150101115959.tar",
> 4604941 "http_status": "200",
> 4604942 "error_code": "",
> 4604943 "bytes_sent": 19,
> 4604944 "bytes_received": 0,
> 4604945 "object_size": 0,
Do you see a zero object_size for other multipart uploads?
Yehuda
> 4604946 "total_time": 1426404000000,
> 4604947 "user_agent": "Boto\/2.38.0 Python\/2.7.7
> Linux\/2.6.32-573.7.1.el6.x86_64",
> 4604948 "referrer": ""
> 4604949 }
>
> Does this help at all. The total time seems exceptionally high. Would it be
> possible that there is a timeout issue where the put request started a
> multipart upload with the correct header and then timed out but the radosgw
> took the data anyway?
>
> I am surprised the radosgw returned a 200 let alone placed the key in the
> bucket listing.
>
>
> That said here is another object (different object) that 404s:
> 1650873 {
> 1650874 "bucket": "noaa-nexrad-l2",
> 1650875 "time": "2015-11-05 04:50:42.606838Z",
> 1650876 "time_local": "2015-11-04 22:50:42.606838",
> 1650877 "remote_addr": "",
> 1650878 "object_owner": "b05f707271774dbd89674a0736c9406e",
> 1650879 "user": "b05f707271774dbd89674a0736c9406e",
> 1650880 "operation": "PUT",
> 1650881 "uri":
> "\/noaa-nexrad-l2\/2015\/02\/25\/KVBX\/NWS_NEXRAD_NXL2DP_KVBX_20150225160000_20150225165959.tar",
> 1650882 "http_status": "200",
> 1650883 "error_code": "",
> 1650884 "bytes_sent": 19,
> 1650885 "bytes_received": 0,
> 1650886 "object_size": 0,
> 1650887 "total_time": 0,
> 1650888 "user_agent": "Boto\/2.38.0 Python\/2.7.7
> Linux\/2.6.32-573.7.1.el6.x86_64",
> 1650889 "referrer": ""
> 1650890 }
>
> And this one fails with a 404 as well. Does this help at all? Here is a
> successful object (different object) log entry as well just in case::
>
> 17462367 {
> 17462368 "bucket": "noaa-nexrad-l2",
> 17462369 "time": "2015-11-04 21:16:44.148603Z",
> 17462370 "time_local": "2015-11-04 15:16:44.148603",
> 17462371 "remote_addr": "",
> 17462372 "object_owner": "b05f707271774dbd89674a0736c9406e",
> 17462373 "user": "b05f707271774dbd89674a0736c9406e",
> 17462374 "operation": "PUT",
> 17462375 "uri":
> "\/noaa-nexrad-l2\/2015\/01\/01\/KAKQ\/NWS_NEXRAD_NXL2DP_KAKQ_20150101080000_20150101085959.tar",
> 17462376 "http_status": "200",
> 17462377 "error_code": "",
> 17462378 "bytes_sent": 19,
> 17462379 "bytes_received": 0,
> 17462380 "object_size": 0,
> 17462381 "total_time": 0,
> 17462382 "user_agent": "Boto\/2.38.0 Python\/2.7.7
> Linux\/2.6.32-573.7.1.el6.x86_64",
> 17462383 "referrer": ""
> 17462384 }
>
> So I am guessing these are not pertinent as they look nearly identical.
> Unfortunately I do not have any client.radosgw.logs to show for the failed
> files for some reason. Is there anything else I can do to troubleshoot this
> issue. In the end the radosgw should have never list these files as they
> never completed successfully, right?
>
>
>
>
>
>
> On 1/15/16 4:36 PM, Yehuda Sadeh-Weinraub wrote:
>>
>> The head object of a multipart object has 0 size, so it's expected.
>> What's missing is the tail of the object. I don't assume you have any
>> logs from when the object was uploaded?
>>
>> Yehuda
>>
>> On Fri, Jan 15, 2016 at 2:12 PM, seapasulli@xxxxxxxxxxxx
>> <seapasulli@xxxxxxxxxxxx> wrote:
>>>
>>> Sorry for the confusion::
>>>
>>> When I grepped for the prefix of the missing object::
>>>
>>> "2015\/01\/01\/PAKC\/NWS_NEXRAD_NXL2DP_PAKC_20150101110000_20150101115959.tar.2~pcu5Hz6foFXjlSxBat22D8YMcHlQOBD"
>>>
>>> I am not able to find any chunks of the object::
>>>
>>> lacadmin@kh28-10:~$ rados -p .rgw.buckets ls | grep 'pcu5Hz6'
>>> lacadmin@kh28-10:~$
>>>
>>> The only piece of the object that I can seem to find is the original one
>>> I
>>> posted::
>>> lacadmin@kh28-10:~$ rados -p .rgw.buckets ls | grep
>>> 'NWS_NEXRAD_NXL2DP_PAKC_20150101110000_20150101115959'
>>>
>>> default.384153.1_2015/01/01/PAKC/NWS_NEXRAD_NXL2DP_PAKC_20150101110000_20150101115959.tar
>>>
>>> And when we stat this object is is 0 bytes as shown earlier::
>>> lacadmin@kh28-10:~$ rados -p .rgw.buckets stat
>>>
>>> 'default.384153.1_2015/01/01/PAKC/NWS_NEXRAD_NXL2DP_PAKC_20150101110000_20150101115959.tar'
>>>
>>> .rgw.buckets/default.384153.1_2015/01/01/PAKC/NWS_NEXRAD_NXL2DP_PAKC_20150101110000_20150101115959.tar
>>> mtime 2015-11-04 15:29:30.000000, size 0
>>>
>>> Sorry again for the confusion.
>>>
>>>
>>>
>>> On 1/15/16 3:58 PM, Yehuda Sadeh-Weinraub wrote:
>>>>
>>>> Ah, I see. Misread that and the object names were very similar. No,
>>>> don't copy it. You can try to grep for the specific object name and
>>>> see if there are pieces of it lying around under a different upload
>>>> id.
>>>>
>>>> Yehuda
>>>>
>>>> On Fri, Jan 15, 2016 at 1:44 PM, seapasulli@xxxxxxxxxxxx
>>>> <seapasulli@xxxxxxxxxxxx> wrote:
>>>>>
>>>>> Sorry I am a bit confused. The successful list that I provided is from
>>>>> a
>>>>> different object of the same size to show that I could indeed get a
>>>>> list.
>>>>> Are you saying to copy the working object to the missing object? Sorry
>>>>> for
>>>>> the confusion.
>>>>>
>>>>>
>>>>> On 1/15/16 3:20 PM, Yehuda Sadeh-Weinraub wrote:
>>>>>>
>>>>>> That's interesting, and might point at the underlying issue that
>>>>>> caused it. Could be a racing upload that somehow ended up with the
>>>>>> wrong object head. The 'multipart' object should be 4M in size, and
>>>>>> the 'shadow' one should have the remainder of the data. You can run
>>>>>> 'rados stat -p .rgw.buckets <oid>' to validate that. If that's the
>>>>>> case, you can copy these to the expected object names:
>>>>>>
>>>>>> $ src_uploadid=wksHvto9gRgHUJbhm_TZPXJTZUPXLT2
>>>>>> $ dest_uploadid=pcu5Hz6foFXjlSxBat22D8YMcHlQOBD
>>>>>>
>>>>>> $ rados -p .rgw.buckets cp
>>>>>>
>>>>>>
>>>>>>
>>>>>> default.384153.1__multipart_2015/01/01/KABR/NWS_NEXRAD_NXL2DP_KABR_20150101130000_20150101135959.tar.2~${src_uploadid}.1
>>>>>>
>>>>>>
>>>>>>
>>>>>> default.384153.1__multipart_2015/01/01/KABR/NWS_NEXRAD_NXL2DP_KABR_20150101130000_20150101135959.tar.2~${dest_uploadid}.1
>>>>>>
>>>>>> $ rados -p .rgw.buckets cp
>>>>>>
>>>>>>
>>>>>>
>>>>>> default.384153.1__shadow_2015/01/01/KABR/NWS_NEXRAD_NXL2DP_KABR_20150101130000_20150101135959.tar.2~${src_upload_id}.1_1
>>>>>>
>>>>>>
>>>>>>
>>>>>> default.384153.1__shadow_2015/01/01/KABR/NWS_NEXRAD_NXL2DP_KABR_20150101130000_20150101135959.tar.2~${dest_upload_id}.1_1
>>>>>>
>>>>>> Yehuda
>>>>>>
>>>>>>
>>>>>> On Fri, Jan 15, 2016 at 1:02 PM, seapasulli@xxxxxxxxxxxx
>>>>>> <seapasulli@xxxxxxxxxxxx> wrote:
>>>>>>>
>>>>>>> lacadmin@kh28-10:~$ rados -p .rgw.buckets ls | grep 'pcu5Hz6'
>>>>>>> lacadmin@kh28-10:~$
>>>>>>>
>>>>>>> Nothing was found. That said when I run the command with another
>>>>>>> prefix
>>>>>>> snippet::
>>>>>>> lacadmin@kh28-10:~$ rados -p .rgw.buckets ls | grep 'wksHvto'
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> default.384153.1__shadow_2015/01/01/KABR/NWS_NEXRAD_NXL2DP_KABR_20150101130000_20150101135959.tar.2~wksHvto9gRgHUJbhm_TZPXJTZUPXLT2.1_1
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> default.384153.1__multipart_2015/01/01/KABR/NWS_NEXRAD_NXL2DP_KABR_20150101130000_20150101135959.tar.2~wksHvto9gRgHUJbhm_TZPXJTZUPXLT2.1
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 1/15/16 12:05 PM, Yehuda Sadeh-Weinraub wrote:
>>>>>>>>
>>>>>>>> On Fri, Jan 15, 2016 at 9:36 AM, seapasulli@xxxxxxxxxxxx
>>>>>>>> <seapasulli@xxxxxxxxxxxx> wrote:
>>>>>>>>>
>>>>>>>>> Hello Yehuda,
>>>>>>>>>
>>>>>>>>> Here it is::
>>>>>>>>>
>>>>>>>>> radosgw-admin object stat --bucket="noaa-nexrad-l2"
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --object="2015/01/01/PAKC/NWS_NEXRAD_NXL2DP_PAKC_20150101110000_20150101115959.tar"
>>>>>>>>> {
>>>>>>>>> "name":
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> "2015\/01\/01\/PAKC\/NWS_NEXRAD_NXL2DP_PAKC_20150101110000_20150101115959.tar",
>>>>>>>>> "size": 7147520,
>>>>>>>>> "policy": {
>>>>>>>>> "acl": {
>>>>>>>>> "acl_user_map": [
>>>>>>>>> {
>>>>>>>>> "user": "b05f707271774dbd89674a0736c9406e",
>>>>>>>>> "acl": 15
>>>>>>>>> }
>>>>>>>>> ],
>>>>>>>>> "acl_group_map": [
>>>>>>>>> {
>>>>>>>>> "group": 1,
>>>>>>>>> "acl": 1
>>>>>>>>> }
>>>>>>>>> ],
>>>>>>>>> "grant_map": [
>>>>>>>>> {
>>>>>>>>> "id": "",
>>>>>>>>> "grant": {
>>>>>>>>> "type": {
>>>>>>>>> "type": 2
>>>>>>>>> },
>>>>>>>>> "id": "",
>>>>>>>>> "email": "",
>>>>>>>>> "permission": {
>>>>>>>>> "flags": 1
>>>>>>>>> },
>>>>>>>>> "name": "",
>>>>>>>>> "group": 1
>>>>>>>>> }
>>>>>>>>> },
>>>>>>>>> {
>>>>>>>>> "id": "b05f707271774dbd89674a0736c9406e",
>>>>>>>>> "grant": {
>>>>>>>>> "type": {
>>>>>>>>> "type": 0
>>>>>>>>> },
>>>>>>>>> "id":
>>>>>>>>> "b05f707271774dbd89674a0736c9406e",
>>>>>>>>> "email": "",
>>>>>>>>> "permission": {
>>>>>>>>> "flags": 15
>>>>>>>>> },
>>>>>>>>> "name": "noaa-commons",
>>>>>>>>> "group": 0
>>>>>>>>> }
>>>>>>>>> }
>>>>>>>>> ]
>>>>>>>>> },
>>>>>>>>> "owner": {
>>>>>>>>> "id": "b05f707271774dbd89674a0736c9406e",
>>>>>>>>> "display_name": "noaa-commons"
>>>>>>>>> }
>>>>>>>>> },
>>>>>>>>> "etag": "b91b6f1650350965c5434c547b3c38ff-1\u0000",
>>>>>>>>> "tag": "_cWrvEa914Gy1AeyzIhRlUdp1wJnek3E\u0000",
>>>>>>>>> "manifest": {
>>>>>>>>> "objs": [],
>>>>>>>>> "obj_size": 7147520,
>>>>>>>>> "explicit_objs": "false",
>>>>>>>>> "head_obj": {
>>>>>>>>> "bucket": {
>>>>>>>>> "name": "noaa-nexrad-l2",
>>>>>>>>> "pool": ".rgw.buckets",
>>>>>>>>> "data_extra_pool": ".rgw.buckets.extra",
>>>>>>>>> "index_pool": ".rgw.buckets.index",
>>>>>>>>> "marker": "default.384153.1",
>>>>>>>>> "bucket_id": "default.384153.1"
>>>>>>>>> },
>>>>>>>>> "key": "",
>>>>>>>>> "ns": "",
>>>>>>>>> "object":
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> "2015\/01\/01\/PAKC\/NWS_NEXRAD_NXL2DP_PAKC_20150101110000_20150101115959.tar",
>>>>>>>>> "instance": ""
>>>>>>>>> },
>>>>>>>>> "head_size": 0,
>>>>>>>>> "max_head_size": 0,
>>>>>>>>> "prefix":
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> "2015\/01\/01\/PAKC\/NWS_NEXRAD_NXL2DP_PAKC_20150101110000_20150101115959.tar.2~pcu5Hz6foFXjlSxBat22D8YMcHlQOBD",
>>>>>>>>
>>>>>>>> Try running:
>>>>>>>> $ rados -p .rgw.buckets ls | grep pcu5Hz6
>>>>>>>>
>>>>>>>> Yehuda
>>>>>>>>
>>>>>>>>
>>>>>>>>> "tail_bucket": {
>>>>>>>>> "name": "noaa-nexrad-l2",
>>>>>>>>> "pool": ".rgw.buckets",
>>>>>>>>> "data_extra_pool": ".rgw.buckets.extra",
>>>>>>>>> "index_pool": ".rgw.buckets.index",
>>>>>>>>> "marker": "default.384153.1",
>>>>>>>>> "bucket_id": "default.384153.1"
>>>>>>>>> },
>>>>>>>>> "rules": [
>>>>>>>>> {
>>>>>>>>> "key": 0,
>>>>>>>>> "val": {
>>>>>>>>> "start_part_num": 1,
>>>>>>>>> "start_ofs": 0,
>>>>>>>>> "part_size": 0,
>>>>>>>>> "stripe_max_size": 4194304,
>>>>>>>>> "override_prefix": ""
>>>>>>>>> }
>>>>>>>>> }
>>>>>>>>> ]
>>>>>>>>> },
>>>>>>>>> "attrs": {}
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> On 1/15/16 11:17 AM, Yehuda Sadeh-Weinraub wrote:
>>>>>>>>>>
>>>>>>>>>> radosgw-admin object stat --bucket=<bucket> --object=<object>'
>>>>>>>>>
>>>>>>>>>
>
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
