Re: Luminous 12.2.1 - RadosGW Multisite doesnt replicate multipart uploads

[Casey Bodley <cbodley@xxxxxxxxxx>]

Thanks Enrico. I wrote a test case that reproduces the issue, and opened http://tracker.ceph.com/issues/21772 to track the bug. It sounds like this is a regression in luminous.

On 10/11/2017 06:41 PM, Enrico Kern wrote:

or this:

   {

        "shard_id": 22,

        "entries": [

            {

                "id": "1_1507761448.758184_10459.1",

                "section": "data",

                "name": "testbucket:6a9448d2-bdba-4bec-aad6-aba72cd8eac6.21344646.3/Wireshark-win64-2.2.7.exe",

                "timestamp": "2017-10-11 22:37:28.758184Z",

                "info": {

                    "source_zone": "6a9448d2-bdba-4bec-aad6-aba72cd8eac6",

                    "error_code": 5,

                    "message": "failed to sync object"

                }

            }

        ]

    },

Virenfrei. www.avg.com

On Thu, Oct 12, 2017 at 12:39 AM, Enrico Kern <enrico.kern@xxxxxxxxxxxxxxx> wrote:

its 45MB, but it happens with all multipart uploads.

sync error list shows

   {

        "shard_id": 31,

        "entries": [

            {

                "id": "1_1507761459.607008_8197.1",

                "section": "data",

                "name": "testbucket:6a9448d2-bdba-4bec-aad6-aba72cd8eac6.21344646.3",

                "timestamp": "2017-10-11 22:37:39.607008Z",

                "info": {

                    "source_zone": "6a9448d2-bdba-4bec-aad6-aba72cd8eac6",

                    "error_code": 5,

                    "message": "failed to sync bucket instance: (5) Input/output error"

                }

            }

        ]

    }

for multiple shards not just this one

On Thu, Oct 12, 2017 at 12:31 AM, Yehuda Sadeh-Weinraub <yehuda@xxxxxxxxxx> wrote:

What is the size of the object? Is it only this one?

Try this command: 'radosgw-admin sync error list'. Does it show anything related to that object?

Thanks,

Yehuda

On Wed, Oct 11, 2017 at 3:26 PM, Enrico Kern <enrico.kern@xxxxxxxxxxxxxxx> wrote:

if i change permissions the sync status shows that it is syncing 1 shard, but no files ends up in the pool (testing with empty data pool). after a while it shows that data is back in sync but there is no file

On Wed, Oct 11, 2017 at 11:26 PM, Yehuda Sadeh-Weinraub <yehuda@xxxxxxxxxx> wrote:

Thanks for your report. We're looking into it. You can try to see if touching the object (e.g., modifying its permissions) triggers the sync.

Yehuda

On Wed, Oct 11, 2017 at 1:36 PM, Enrico Kern <enrico.kern@xxxxxxxxxxxxxxx> wrote:

Hi David,

yeah seems you are right, they are stored as different filenames in the data bucket when using multisite upload. But anyway it stil doesnt get replicated. As example i have files like

6a9448d2-bdba-4bec-aad6-aba72cd8eac6.21344646.1__multipart_Wireshark-win64-2.2.7.exe.2~0LAfq93OMdk7hrijvyzW_EBRkVQLX37.6

in the data pool on one zone. But its not replicated to the other zone. naming is not relevant, the other data bucket doesnt have any file multipart or not.

im really missing the file on the other zone.

Virenfrei. www.avg.com

On Wed, Oct 11, 2017 at 10:25 PM, David Turner <drakonstein@xxxxxxxxx> wrote:

Multipart is a client side setting when uploading.  Multisite in and of itself is a client and it doesn't use multipart (at least not by default).  I have a Jewel RGW Multisite cluster and one site has the object as multi-part while the second site just has it as a single object.  I had to change from looking at the objects in the pool for monitoring to looking at an ls of the buckets to see if they were in sync.

I don't know if multisite has the option to match if an object is multipart between sites, but it definitely doesn't seem to be the default behavior.

On Wed, Oct 11, 2017 at 3:56 PM Enrico Kern <enrico.kern@xxxxxxxxxxxxxxx> wrote:

Hi all,

i just setup multisite replication according to the docs from http://docs.ceph.com/docs/master/radosgw/multisite/ and everything works except that if a client uploads via multipart the files dont get replicated.

If i in one zone rename a file that was uploaded via multipart it gets replicated, but not if i left it untouched. Any ideas why? I remember there was a similar bug with jewel a while back.

On the slave node i also permanently get this error (unrelated to the replication) in the radosgw log:

 meta sync: ERROR: failed to read mdlog info with (2) No such file or directory

we didnt run radosgw before the luminous upgrade of our clusters.

after a finished multipart upload which is only visible at one zone "radosgw-admin sync status" just shows that metadata and data is caught up with the source.

--

Enrico Kern

Lead System Engineer

T +49 (0) 30 555713017  | M +49 (0)152 26814501

E  enrico.kern@xxxxxxxxxx |  Skype flyersa |  LinkedIn View my Profile 

Glispa GmbH - Berlin Office

Sonnenburger Str. 73 10437 Berlin, Germany 

Managing Director: David Brown, Registered in Berlin, AG Charlottenburg HRB 114678B

            

Virenfrei. www.avg.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

--

Enrico Kern

Lead System Engineer

T +49 (0) 30 555713017  | M +49 (0)152 26814501

E  enrico.kern@xxxxxxxxxx |  Skype flyersa |  LinkedIn View my Profile 

Glispa GmbH - Berlin Office

Sonnenburger Str. 73 10437 Berlin, Germany 

Managing Director: Dina Karol-Gavish, Registered in Berlin, AG Charlottenburg HRB 114678B

            

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

--

Enrico Kern

Lead System Engineer

T +49 (0) 30 555713017  | M +49 (0)152 26814501

E  enrico.kern@xxxxxxxxxx |  Skype flyersa |  LinkedIn View my Profile 

Glispa GmbH - Berlin Office

Sonnenburger Str. 73 10437 Berlin, Germany 

Managing Director: Dina Karol-Gavish, Registered in Berlin, AG Charlottenburg HRB 114678B

            

--

Enrico Kern

Lead System Engineer

T +49 (0) 30 555713017  | M +49 (0)152 26814501

E  enrico.kern@xxxxxxxxxx |  Skype flyersa |  LinkedIn View my Profile 

Glispa GmbH - Berlin Office

Sonnenburger Str. 73 10437 Berlin, Germany 

Managing Director: Dina Karol-Gavish, Registered in Berlin, AG Charlottenburg HRB 114678B

            

--

Enrico Kern

Lead System Engineer

T +49 (0) 30 555713017  | M +49 (0)152 26814501

E  enrico.kern@xxxxxxxxxx |  Skype flyersa |  LinkedIn View my Profile 

Glispa GmbH - Berlin Office

Sonnenburger Str. 73 10437 Berlin, Germany 

Managing Director: Dina Karol-Gavish, Registered in Berlin, AG Charlottenburg HRB 114678B

            

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com