Re: [EXTERNAL] Re: RGW Bucket Notifications and MultiPart Uploads

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



I have not tested with Quincy/17.x yet so I do not know which notifications are sent for Multipart uploads in this release set.

I know that for Pacific.16.x I needed to add some code/logic to only act on notifications that represented the end state of an Object creation.

My tests show that when a multipart upload is in progress if you perform a head on the object before the final part is uploaded that you will get back a 200 with the size that the object eventually will be. The multiple noitifications that occur with the Multipart upload have size set to the chunk that is being uploaded, not the total size.

In order to get the behavior that I wanted, act on an object after it has been uploaded I had to code the following:

- get a notification and get it's size
- head the object in RGW and get it's size
- if the sizes do not match then do nothing.
- When the size in the notification matches the size on the head request the event type is ObjectCreated:CompleteMultipartUpload and we know the upload is complete.

This is a bunch of extra code and round trips that.I wish that I did not have to make.

Hopefully Quincy only sends 2 notifications for a multipart upload (1) The initial Post and (2) The find Put.



-- 


Mark Selby
Sr Linux Administrator, The Voleon Group
mselby@xxxxxxxxxx 
 
 This email is subject to important conditions and disclosures that are listed on this web page: https://voleon.com/disclaimer/.
 

On 7/20/22, 5:57 AM, "Daniel Gryniewicz" <dang@xxxxxxxxxx> wrote:

    Seems like the notification for a multipart upload should look different 
    to a normal upload?

    Daniel

    On 7/20/22 08:53, Yehuda Sadeh-Weinraub wrote:
    > Can maybe leverage one of the other calls to check for upload completion:
    > list multipart uploads and/or list parts. The latter should work if you
    > have the upload id at hand.
    > 
    > Yehuda
    > 
    > On Wed, Jul 20, 2022, 8:40 AM Casey Bodley <cbodley@xxxxxxxxxx> wrote:
    > 
    >> On Wed, Jul 20, 2022 at 12:57 AM Yuval Lifshitz <ylifshit@xxxxxxxxxx>
    >> wrote:
    >>>
    >>> yes, that would work. you would get a "404" until the object is fully
    >>> uploaded.
    >>
    >> just note that you won't always get 404 before multipart complete,
    >> because multipart uploads can overwrite existing objects
    >>
    >> _______________________________________________
    >> ceph-users mailing list -- ceph-users@xxxxxxx
    >> To unsubscribe send an email to ceph-users-leave@xxxxxxx
    >>
    >>
    > _______________________________________________
    > ceph-users mailing list -- ceph-users@xxxxxxx
    > To unsubscribe send an email to ceph-users-leave@xxxxxxx
    > 

    _______________________________________________
    ceph-users mailing list -- ceph-users@xxxxxxx
    To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux