Hi,
On 31/07/2019 19:02, Paul Emmerich wrote:
Some interesting points here, thanks for raising them :)
From our experience: buckets with tens of million objects work just fine with
no big problems usually. Buckets with hundreds of million objects require some
attention. Buckets with billions of objects? "How about indexless buckets?" -
"No, we need to list them".
We've had some problems with large buckets (from around the 70Mobject
mark).
One you don't mention is that multipart uploads break during resharding
- so if our users are filling up a bucket with many writers uploading
multipart objects, some of these will fail (rather than blocking) when
the bucket is resharded.
1. The recommended number of objects per shard is 100k. Why? How was this
default configuration derived?
I don't know what a good number is, but by the time you get into O(10M)
objects, some sharding does seem to help - we've found a particular OSD
getting really hammered by heavy updates on large buckets (in Jewel,
before we had online resharding).
3. Deleting large buckets
Someone accidentaly put 450 million small objects into a bucket and only noticed
when the cluster ran full. The bucket isn't needed, so just delete it and case
closed?
Deleting is unfortunately far slower than adding objects, also
radosgw-admin leaks
memory during deletion:
We've also seen bucket deletion via radosgw-admin failing because of
oddities in the bucket itself (e.g. missing shadow objects, omap objects
that still exist when the related object is gone); sorting that was a
bit fiddly (with some help from Canonical, who I think are working on
patches).
Increasing --max-concurrent-ios helps with deletion speed (option does effect
deletion concurrency, documentation says it's only for other specific commands).
Yes, we found increasing max-concurrent-ios helped.
Regards,
Matthew
--
The Wellcome Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.
_______________________________________________
Dev mailing list -- dev@xxxxxxx
To unsubscribe send an email to dev-leave@xxxxxxx
