Hello, On Tue, 14 Jun 2016 12:20:44 +0300 Nmz wrote: > > > > ----- Original Message ----- > From: Wido den Hollander <wido@xxxxxxxx> > To: Василий Ангапов <angapov@xxxxxxxxx> > Date: Tuesday, June 14, 2016, 12:05:51 PM > Subject: 40Mil objects in S3 rados pool / how calculate PGs > > > >> Op 14 juni 2016 om 11:00 schreef Василий Ангапов <angapov@xxxxxxxxx>: > >> > >> > >> Is it a good idea to disable scrub and deep-scrub for bucket.index > >> pool? What negative consequences it may cause? > >> > > > No, I would not do that. Scrubbing is essential to detect (silent) > > data corruption. > > > You should really scrub all your data. > > Ceph do not protect from silent data corruption at all. > > You can read this thread > http://lists.ceph.com/pipermail/ceph-users-ceph.com/2016-February/007680.html > While that is unfortunately very true, Ceph does at least allow you to detect it (after the fact) and if you're lucky it is a replica and not the primary object that's corrupted. So it's better than Ext4 or XFS, but worse than ZFS or BTRFS. Bluestore is supposed to address this, but currently lacks live checksums as well. Now with a storage that large (40million 300KB objects...) the statistical chances of bitrot do of course increase. I've run a cluster with a few TB of data for more than a year w/o deep scrubs and unsurprisingly nothing bad was found when I turned it back on. But your millage may vary, caveat emptor, etc. Christan > >> 2016-06-14 11:51 GMT+03:00 Wido den Hollander <wido@xxxxxxxx>: > >> > > >> >> Op 14 juni 2016 om 10:10 schreef Ansgar Jazdzewski > >> >> <a.jazdzewski@xxxxxxxxxxxxxx>: > >> >> > >> >> > >> >> Hi, > >> >> > >> >> we are using ceph and radosGW to store images (~300kb each) in S3, > >> >> when in comes to deep-scrubbing we facing task timeouts (> 30s ...) > >> >> > >> >> my questions is: > >> >> > >> >> in case of that amount of objects/files is it better to calculate > >> >> the PGs on a object-bases instant of the volume size? and how it > >> >> should be done? > >> >> > >> > > >> > Do you have bucket sharding enabled? > >> > > >> > And how many objects do you have in a single bucket? > >> > > >> > If sharding is not enabled for the bucket index you might have > >> > large RADOS objects with bucket indexes which are hard to scrub. > >> > > >> > Wido > >> > > >> >> thanks > >> >> Ansgar > >> >> _______________________________________________ > >> >> ceph-users mailing list > >> >> ceph-users@xxxxxxxxxxxxxx > >> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >> > _______________________________________________ > >> > ceph-users mailing list > >> > ceph-users@xxxxxxxxxxxxxx > >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > _______________________________________________ > > ceph-users mailing list > > ceph-users@xxxxxxxxxxxxxx > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Christian Balzer Network/Systems Engineer chibi@xxxxxxx Global OnLine Japan/Rakuten Communications http://www.gol.com/ _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |