Re: CompleteMultipartUpload takes a long time to finish

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]


On Mon, Feb 05, 2024 at 07:51:34PM +0100, Ondřej Kukla wrote:
> Hello,
> For some time now I’m struggling with the time it takes to CompleteMultipartUpload on one of my rgw clusters.
> I have a customer with ~8M objects in one bucket uploading quite a large files. From 100GB to like 800GB.
> I’ve noticed when they are uploading ~200GB files that the requests started timeouting on a LB we have infront of the rgw.
> When I’ve started going through the logs I’ve noticed that the CompleteMultipartUpload request took like 700s to finish. Which seemed ok-ish, but the number seem quite large.
> However, when they started uploading 750GB files the time to complete the multipart upload ended around 2500s -> more than 40minutes which seems like a way to much.
> Do you have a similar experience? Is there anything we can do to improve this? How much time does the CompleteMultipartUpload takes on your clusters?
> The cluster is running on version 17.2.6.
How many incomplete MPUs and MPU parts exist in that bucket?

Is it 8M objects based on ListObjects, or the number of objects reported
by radosgw-admin bucket stats?

If there are LOTS of incomplete objects, that can cause extreme cases in
the listing used by both ListObjects & CompleteMultipartUpload.

This is esp. likely if they the huge files used many tiny parts (some
tooling defaults to 5MB parts).

Easy way to test this is ask them to do a single 200GB upload to a brand
new bucket (with no objects).

If *that* case is fast, then it's something about the index entries in
the existing bucket; likely a high proportion of incomplete parts.

CompleteMultipartUpload for 50GB => single-digit seconds.

Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation President & Treasurer
E-Mail   : robbat2@xxxxxxxxxx
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136

Attachment: signature.asc
Description: PGP signature

ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]

  Powered by Linux