Very slow S3 sync with big number of object.

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi,

I'm testing S3 and I created a test where I sync a big part of my
homedirectory, about 4GB of data in a lot of small objects, towards a S3
bucket.
The first part of the sync was very fast but after some time it became a
lot slower.

What I basically see is this for every file:

The file gets transferred.
The S3 gateway returns a 200 for the transfer
s3cmd seems to think that something failed (probably the acl???)
It starts a new transfer of the same file with ?acl behind the call and
inserts a 3 second delay for this transfer.

With 40.000 files this really takes a very long time.

I have created debug logfiles for this but they are really big and I
probably need to filter out the parts related to one set of calls to
make it usable.

The user that I use to do the transfer has full controll on the bucket.

Has anyone seen this before?

-- 
Met vriendelijke groet / Best regards,

Jan Hugo Prins
Infra and Isilon storage consultant

Better.be B.V.
Auke Vleerstraat 140 E | 7547 AN Enschede | KvK 08097527
T +31 (0) 53 48 00 694 | M +31 (0)6 26 358 951
jprins@xxxxxxxxxxxx | www.betterbe.com

This e-mail is intended exclusively for the addressee(s), and may not
be passed on to, or made available for use by any person other than 
the addressee(s). Better.be B.V. rules out any and every liability 
resulting from any electronic transmission.

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux