I have looked all over and I do not see any explicit mention of
"NWS_NEXRAD_NXL2DP_PAKC_20150101110000_20150101115959" in the logs nor
do I see a timestamp from November 4th although I do see log rotations
dating back to october 15th. I don't think it's possible it wasn't
logged so I am going through the bucket logs from the 'radosgw-admin log
show --object' side and I found the following::
4604932 {
4604933 "bucket": "noaa-nexrad-l2",
4604934 "time": "2015-11-04 21:29:27.346509Z",
4604935 "time_local": "2015-11-04 15:29:27.346509",
4604936 "remote_addr": "",
4604937 "object_owner": "b05f707271774dbd89674a0736c9406e",
4604938 "user": "b05f707271774dbd89674a0736c9406e",
4604939 "operation": "PUT",
4604940 "uri":
"\/noaa-nexrad-l2\/2015\/01\/01\/PAKC\/NWS_NEXRAD_NXL2DP_PAKC_20150101110000_20150101115959.tar",
4604941 "http_status": "200",
4604942 "error_code": "",
4604943 "bytes_sent": 19,
4604944 "bytes_received": 0,
4604945 "object_size": 0,
4604946 "total_time": 1426404000000,
4604947 "user_agent": "Boto\/2.38.0 Python\/2.7.7
Linux\/2.6.32-573.7.1.el6.x86_64",
4604948 "referrer": ""
4604949 }
Does this help at all. The total time seems exceptionally high. Would it
be possible that there is a timeout issue where the put request started
a multipart upload with the correct header and then timed out but the
radosgw took the data anyway?
I am surprised the radosgw returned a 200 let alone placed the key in
the bucket listing.
That said here is another object (different object) that 404s:
1650873 {
1650874 "bucket": "noaa-nexrad-l2",
1650875 "time": "2015-11-05 04:50:42.606838Z",
1650876 "time_local": "2015-11-04 22:50:42.606838",
1650877 "remote_addr": "",
1650878 "object_owner": "b05f707271774dbd89674a0736c9406e",
1650879 "user": "b05f707271774dbd89674a0736c9406e",
1650880 "operation": "PUT",
1650881 "uri":
"\/noaa-nexrad-l2\/2015\/02\/25\/KVBX\/NWS_NEXRAD_NXL2DP_KVBX_20150225160000_20150225165959.tar",
1650882 "http_status": "200",
1650883 "error_code": "",
1650884 "bytes_sent": 19,
1650885 "bytes_received": 0,
1650886 "object_size": 0,
1650887 "total_time": 0,
1650888 "user_agent": "Boto\/2.38.0 Python\/2.7.7
Linux\/2.6.32-573.7.1.el6.x86_64",
1650889 "referrer": ""
1650890 }
And this one fails with a 404 as well. Does this help at all? Here is a
successful object (different object) log entry as well just in case::
17462367 {
17462368 "bucket": "noaa-nexrad-l2",
17462369 "time": "2015-11-04 21:16:44.148603Z",
17462370 "time_local": "2015-11-04 15:16:44.148603",
17462371 "remote_addr": "",
17462372 "object_owner": "b05f707271774dbd89674a0736c9406e",
17462373 "user": "b05f707271774dbd89674a0736c9406e",
17462374 "operation": "PUT",
17462375 "uri":
"\/noaa-nexrad-l2\/2015\/01\/01\/KAKQ\/NWS_NEXRAD_NXL2DP_KAKQ_20150101080000_20150101085959.tar",
17462376 "http_status": "200",
17462377 "error_code": "",
17462378 "bytes_sent": 19,
17462379 "bytes_received": 0,
17462380 "object_size": 0,
17462381 "total_time": 0,
17462382 "user_agent": "Boto\/2.38.0 Python\/2.7.7
Linux\/2.6.32-573.7.1.el6.x86_64",
17462383 "referrer": ""
17462384 }
So I am guessing these are not pertinent as they look nearly identical.
Unfortunately I do not have any client.radosgw.logs to show for the
failed files for some reason. Is there anything else I can do to
troubleshoot this issue. In the end the radosgw should have never list
these files as they never completed successfully, right?
On 1/15/16 4:36 PM, Yehuda Sadeh-Weinraub wrote:
The head object of a multipart object has 0 size, so it's expected.
What's missing is the tail of the object. I don't assume you have any
logs from when the object was uploaded?
Yehuda
On Fri, Jan 15, 2016 at 2:12 PM, seapasulli@xxxxxxxxxxxx
<seapasulli@xxxxxxxxxxxx> wrote:
Sorry for the confusion::
When I grepped for the prefix of the missing object::
"2015\/01\/01\/PAKC\/NWS_NEXRAD_NXL2DP_PAKC_20150101110000_20150101115959.tar.2~pcu5Hz6foFXjlSxBat22D8YMcHlQOBD"
I am not able to find any chunks of the object::
lacadmin@kh28-10:~$ rados -p .rgw.buckets ls | grep 'pcu5Hz6'
lacadmin@kh28-10:~$
The only piece of the object that I can seem to find is the original one I
posted::
lacadmin@kh28-10:~$ rados -p .rgw.buckets ls | grep
'NWS_NEXRAD_NXL2DP_PAKC_20150101110000_20150101115959'
default.384153.1_2015/01/01/PAKC/NWS_NEXRAD_NXL2DP_PAKC_20150101110000_20150101115959.tar
And when we stat this object is is 0 bytes as shown earlier::
lacadmin@kh28-10:~$ rados -p .rgw.buckets stat
'default.384153.1_2015/01/01/PAKC/NWS_NEXRAD_NXL2DP_PAKC_20150101110000_20150101115959.tar'
.rgw.buckets/default.384153.1_2015/01/01/PAKC/NWS_NEXRAD_NXL2DP_PAKC_20150101110000_20150101115959.tar
mtime 2015-11-04 15:29:30.000000, size 0
Sorry again for the confusion.
On 1/15/16 3:58 PM, Yehuda Sadeh-Weinraub wrote:
Ah, I see. Misread that and the object names were very similar. No,
don't copy it. You can try to grep for the specific object name and
see if there are pieces of it lying around under a different upload
id.
Yehuda
On Fri, Jan 15, 2016 at 1:44 PM, seapasulli@xxxxxxxxxxxx
<seapasulli@xxxxxxxxxxxx> wrote:
Sorry I am a bit confused. The successful list that I provided is from a
different object of the same size to show that I could indeed get a list.
Are you saying to copy the working object to the missing object? Sorry
for
the confusion.
On 1/15/16 3:20 PM, Yehuda Sadeh-Weinraub wrote:
That's interesting, and might point at the underlying issue that
caused it. Could be a racing upload that somehow ended up with the
wrong object head. The 'multipart' object should be 4M in size, and
the 'shadow' one should have the remainder of the data. You can run
'rados stat -p .rgw.buckets <oid>' to validate that. If that's the
case, you can copy these to the expected object names:
$ src_uploadid=wksHvto9gRgHUJbhm_TZPXJTZUPXLT2
$ dest_uploadid=pcu5Hz6foFXjlSxBat22D8YMcHlQOBD
$ rados -p .rgw.buckets cp
default.384153.1__multipart_2015/01/01/KABR/NWS_NEXRAD_NXL2DP_KABR_20150101130000_20150101135959.tar.2~${src_uploadid}.1
default.384153.1__multipart_2015/01/01/KABR/NWS_NEXRAD_NXL2DP_KABR_20150101130000_20150101135959.tar.2~${dest_uploadid}.1
$ rados -p .rgw.buckets cp
default.384153.1__shadow_2015/01/01/KABR/NWS_NEXRAD_NXL2DP_KABR_20150101130000_20150101135959.tar.2~${src_upload_id}.1_1
default.384153.1__shadow_2015/01/01/KABR/NWS_NEXRAD_NXL2DP_KABR_20150101130000_20150101135959.tar.2~${dest_upload_id}.1_1
Yehuda
On Fri, Jan 15, 2016 at 1:02 PM, seapasulli@xxxxxxxxxxxx
<seapasulli@xxxxxxxxxxxx> wrote:
lacadmin@kh28-10:~$ rados -p .rgw.buckets ls | grep 'pcu5Hz6'
lacadmin@kh28-10:~$
Nothing was found. That said when I run the command with another prefix
snippet::
lacadmin@kh28-10:~$ rados -p .rgw.buckets ls | grep 'wksHvto'
default.384153.1__shadow_2015/01/01/KABR/NWS_NEXRAD_NXL2DP_KABR_20150101130000_20150101135959.tar.2~wksHvto9gRgHUJbhm_TZPXJTZUPXLT2.1_1
default.384153.1__multipart_2015/01/01/KABR/NWS_NEXRAD_NXL2DP_KABR_20150101130000_20150101135959.tar.2~wksHvto9gRgHUJbhm_TZPXJTZUPXLT2.1
On 1/15/16 12:05 PM, Yehuda Sadeh-Weinraub wrote:
On Fri, Jan 15, 2016 at 9:36 AM, seapasulli@xxxxxxxxxxxx
<seapasulli@xxxxxxxxxxxx> wrote:
Hello Yehuda,
Here it is::
radosgw-admin object stat --bucket="noaa-nexrad-l2"
--object="2015/01/01/PAKC/NWS_NEXRAD_NXL2DP_PAKC_20150101110000_20150101115959.tar"
{
"name":
"2015\/01\/01\/PAKC\/NWS_NEXRAD_NXL2DP_PAKC_20150101110000_20150101115959.tar",
"size": 7147520,
"policy": {
"acl": {
"acl_user_map": [
{
"user": "b05f707271774dbd89674a0736c9406e",
"acl": 15
}
],
"acl_group_map": [
{
"group": 1,
"acl": 1
}
],
"grant_map": [
{
"id": "",
"grant": {
"type": {
"type": 2
},
"id": "",
"email": "",
"permission": {
"flags": 1
},
"name": "",
"group": 1
}
},
{
"id": "b05f707271774dbd89674a0736c9406e",
"grant": {
"type": {
"type": 0
},
"id": "b05f707271774dbd89674a0736c9406e",
"email": "",
"permission": {
"flags": 15
},
"name": "noaa-commons",
"group": 0
}
}
]
},
"owner": {
"id": "b05f707271774dbd89674a0736c9406e",
"display_name": "noaa-commons"
}
},
"etag": "b91b6f1650350965c5434c547b3c38ff-1\u0000",
"tag": "_cWrvEa914Gy1AeyzIhRlUdp1wJnek3E\u0000",
"manifest": {
"objs": [],
"obj_size": 7147520,
"explicit_objs": "false",
"head_obj": {
"bucket": {
"name": "noaa-nexrad-l2",
"pool": ".rgw.buckets",
"data_extra_pool": ".rgw.buckets.extra",
"index_pool": ".rgw.buckets.index",
"marker": "default.384153.1",
"bucket_id": "default.384153.1"
},
"key": "",
"ns": "",
"object":
"2015\/01\/01\/PAKC\/NWS_NEXRAD_NXL2DP_PAKC_20150101110000_20150101115959.tar",
"instance": ""
},
"head_size": 0,
"max_head_size": 0,
"prefix":
"2015\/01\/01\/PAKC\/NWS_NEXRAD_NXL2DP_PAKC_20150101110000_20150101115959.tar.2~pcu5Hz6foFXjlSxBat22D8YMcHlQOBD",
Try running:
$ rados -p .rgw.buckets ls | grep pcu5Hz6
Yehuda
"tail_bucket": {
"name": "noaa-nexrad-l2",
"pool": ".rgw.buckets",
"data_extra_pool": ".rgw.buckets.extra",
"index_pool": ".rgw.buckets.index",
"marker": "default.384153.1",
"bucket_id": "default.384153.1"
},
"rules": [
{
"key": 0,
"val": {
"start_part_num": 1,
"start_ofs": 0,
"part_size": 0,
"stripe_max_size": 4194304,
"override_prefix": ""
}
}
]
},
"attrs": {}
}
On 1/15/16 11:17 AM, Yehuda Sadeh-Weinraub wrote:
radosgw-admin object stat --bucket=<bucket> --object=<object>'
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
