Hi Mariusz, all
We have seen this issue as well, on redhat ceph 4 (I have an unresolved
case open). In our case, `radosgw-admin stat` is not a sufficient check to
guarantee that there are rados objects. You have to do a `rados stat` to
know that.
In your case, the object is ~48M in size, appears to also use S3 multipart.
This means, when uploaded, S3 will slice it up into parts based on what S3
multipart size you use (5M default, i think 8M here). After that, rados
further slices any incoming (multipart size objects) into rados
object objects of 4Mib size (default).
The end result is you have a bunch of rados objects labelled with the
'prefix' from the `radosgw-admin stat` you ran, as well as a head object
(named the same as the S3 object you uploaded) that contains the metadata
so rgw knows how to put the S3 object back together. In our case, the head
object is there but the other rados pieces that hold the actual data seem
to be gone, so `radosgw-admin stat` returns fine, but we get NoSuchKey when
trying to download.
Try `rados -p {rgw buckets pool} stat
255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4`, it
will show you the rados stat of the head object, which will be much smaller
than the S3 object.
To check if you actually have all rados objects for this 48M S3 object, try
searching for parts of the prefix or the whole prefix on a list of all
rados objects in buckets pool.
FYI, the `rados ls` will list every rados object in the bucket, so it may
be very large and take a long time if you have many objects.
rados -p {rgw buckets pool} ls > {tmpfile}
grep '2~NTy88SkDkXR9ifSrrRcw5WPDxqN3PO2' {tmpfile}
grep 'juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4' {tmpfile}
The first grep is actually the S3 multipart ID string added to the prefix
by rgw.
Rafael
On Tue, 10 Nov 2020 at 01:04, Janek Bevendorff <
janek.bevendorff@xxxxxxxxxxxxx> wrote:
> We are having the exact same problem (also Octopus). The object is
> listed by s3cmd, but trying to download it results in a 404 error.
> radosgw-admin object stat shows that the object still exists. Any
> further ideas how I can restore access to this object?
>
> (Sorry if this is a duplicate, but it seems like the mailing list hasn't
> accepted my original mail).
>
>
> > Mariusz Gronczewski wrote:
> >
> >
> >> Dnia 2020-07-27, o godz. 21:31:33
> >> "Robin H. Johnson" <robbat2@xxxxxxxxxx
> >> <mailto:robbat2@xxxxxxxxxx>> napisał(a):
> >>
> >>
> >>> On Mon, Jul 27, 2020 at 08:02:23PM +0200, Mariusz Gronczewski wrote:
> >>>
> >>>> Hi,
> >>>> I've got a problem on Octopus (15.2.3, debian packages) install,
> >>>> bucket S3 index shows a file:
> >>>> s3cmd ls s3://upvid/255/38355 --recursive
> >>>> 2020-07-27 17:48 50584342
> >>>>
> s3://upvid/255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4
> >>>> radosgw-admin bi list also shows it
> >>>> {
> >>>> "type": "plain",
> >>>> "idx":
> >>>>
> "255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4",
> >>>> "entry": { "name":
> >>>>
> "255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4",
> >>>> "instance": "", "ver": {
> >>>> "pool": 11,
> >>>> "epoch": 853842
> >>>> },
> >>>> "locator": "",
> >>>> "exists": "true",
> >>>> "meta": {
> >>>> "category": 1,
> >>>> "size": 50584342,
> >>>> "mtime": "2020-07-27T17:48:27.203008Z",
> >>>> "etag": "2b31cc8ce8b1fb92a5f65034f2d12581-7",
> >>>> "storage_class": "",
> >>>> "owner": "filmweb-app",
> >>>> "owner_display_name": "filmweb app user",
> >>>> "content_type": "",
> >>>> "accounted_size": 50584342,
> >>>> "user_data": "",
> >>>> "appendable": "false"
> >>>> },
> >>>> "tag": "_3ubjaztglHXfZr05wZCFCPzebQf-ZFP",
> >>>> "flags": 0,
> >>>> "pending_map": [],
> >>>> "versioned_epoch": 0
> >>>> }
> >>>> },
> >>>> but trying to download it via curl (I've set permissions to public0
> >>>>
> >>> only gets me
> >>> Does the RADOS object for this still exist?
> >>>
> >>> try:
> >>> radosgw-admin object stat --bucket ... --object
> >>> '255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4'
> >>>
> >>> If that doesn't return, then the backing object is gone, and you have
> >>> a stale index entry that can be cleaned up in most cases with check
> >>> bucket.
> >>> For cases where that doesn't fix it, my recommended way to fix it is
> >>> write a new 0-byte object to the same name, then delete it.
> >>>
> >>
> >>
> >>
> >> it does exist:
> >>
> >> {
> >> "name":
> >> "255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4",
> >> "size": 50584342, "policy": {
> >> "acl": {...},
> >> "owner": {...}
> >> },
> >> "etag": "2b31cc8ce8b1fb92a5f65034f2d12581-7",
> >> "tag": "_3ubjaztglHXfZr05wZCFCPzebQf-ZFP",
> >> "manifest": {
> >> "objs": [],
> >> "obj_size": 50584342,
> >> "explicit_objs": "false",
> >> "head_size": 0,
> >> "max_head_size": 0,
> >> "prefix":
> >>
> "255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4.2~NTy88SkDkXR9ifSrrRcw5WPDxqN3PO2",
> >> "rules": [ {
> >> "key": 0,
> >> "val": {
> >> "start_part_num": 1,
> >> "start_ofs": 0,
> >> "part_size": 8388608,
> >> "stripe_max_size": 4194304,
> >> "override_prefix": ""
> >> }
> >> },
> >> {
> >> "key": 50331648,
> >> "val": {
> >> "start_part_num": 7,
> >> "start_ofs": 50331648,
> >> "part_size": 252694,
> >> "stripe_max_size": 4194304,
> >> "override_prefix": ""
> >> }
> >> }
> >> ],
> >> "tail_instance": "",
> >> "tail_placement": {
> >> "bucket": {
> >> "name": "upvid",
> >> "marker":
> >> "88d4f221-0da5-444d-81a8-517771278350.665933.2", "bucket_id":
> >> "88d4f221-0da5-444d-81a8-517771278350.665933.2", "tenant": "",
> >> "explicit_placement": {
> >> "data_pool": "",
> >> "data_extra_pool": "",
> >> "index_pool": ""
> >> }
> >> },
> >> "placement_rule": "default-placement"
> >> },
> >> "begin_iter": {
> >> "part_ofs": 0,
> >> "stripe_ofs": 0,
> >> "ofs": 0,
> >> "stripe_size": 4194304,
> >> "cur_part_id": 1,
> >> "cur_stripe": 0,
> >> "cur_override_prefix": "",
> >> "location": {
> >> "placement_rule": "default-placement",
> >> "obj": {
> >> "bucket": {
> >> "name": "upvid",
> >> "marker":
> >> "88d4f221-0da5-444d-81a8-517771278350.665933.2", "bucket_id":
> >> "88d4f221-0da5-444d-81a8-517771278350.665933.2", "tenant": "",
> >> "explicit_placement": {
> >> "data_pool": "",
> >> "data_extra_pool": "",
> >> "index_pool": ""
> >> }
> >> },
> >> "key": {
> >> "name":
> >>
> "255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4.2~NTy88SkDkXR9ifSrrRcw5WPDxqN3PO2.1",
> >> "instance": "", "ns": "multipart"
> >> }
> >> },
> >> "raw_obj": {
> >> "pool": "",
> >> "oid": "",
> >> "loc": ""
> >> },
> >> "is_raw": false
> >> }
> >> },
> >> "end_iter": {
> >> "part_ofs": 50584342,
> >> "stripe_ofs": 50584342,
> >> "ofs": 50584342,
> >> "stripe_size": 252694,
> >> "cur_part_id": 8,
> >> "cur_stripe": 0,
> >> "cur_override_prefix": "",
> >> "location": {
> >> "placement_rule": "default-placement",
> >> "obj": {
> >> "bucket": {
> >> "name": "upvid",
> >> "marker":
> >> "88d4f221-0da5-444d-81a8-517771278350.665933.2", "bucket_id":
> >> "88d4f221-0da5-444d-81a8-517771278350.665933.2", "tenant": "",
> >> "explicit_placement": {
> >> "data_pool": "",
> >> "data_extra_pool": "",
> >> "index_pool": ""
> >> }
> >> },
> >> "key": {
> >> "name":
> >>
> "255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4.2~NTy88SkDkXR9ifSrrRcw5WPDxqN3PO2.8",
> >> "instance": "", "ns": "multipart"
> >> }
> >> },
> >> "raw_obj": {
> >> "pool": "",
> >> "oid": "",
> >> "loc": ""
> >> },
> >> "is_raw": false
> >> }
> >> }
> >> },
> >> "attrs": {
> >> "user.rgw.pg_ver": "/u0005u",
> >> "user.rgw.source_zone": "�I}�",
> >> "user.rgw.tail_tag":
> >> "88d4f221-0da5-444d-81a8-517771278350.658638.5538809",
> >> "user.rgw.x-amz-acl": "private", "user.rgw.x-amz-content-sha256":
> >> "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
> >> "user.rgw.x-amz-date": "20200619T194517Z" }
> >> }
> >>
> >>
> >>
> >>
> >>
> >> --
> >> Mariusz Gronczewski, Administrator
> >>
> >> Efigence S. A.
> >> ul. Wołoska 9a, 02-583 Warszawa
> >> T: [+48] 22 380 13 13
> >> NOC: [+48] 22 380 10 20
> >> E: admin@xxxxxxxxxxxx <mailto:admin@xxxxxxxxxxxx>
> >> _______________________________________________
> >> ceph-users mailing list -- ceph-users@xxxxxxx <mailto:
> ceph-users@xxxxxxx>
> >> To unsubscribe send an email to ceph-users-leave@xxxxxxx
> >> <mailto:ceph-users-leave@xxxxxxx>
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>
--
*Rafael Lopez*
Devops Systems Engineer
Monash University eResearch Centre
T: +61 3 9905 9118
E: rafael.lopez@xxxxxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
