Got it. Thanks, Mark!
Kind regards,
Charles Alva
Sent from Gmail Mobile
On Fri, Apr 12, 2019 at 10:53 PM Mark Nelson <mnelson@xxxxxxxxxx> wrote:
They have the same issue, but depending on the SSD may be better at
absorbing the extra IO if network or CPU are bigger bottlenecks. That's
one of the reasons that a lot of folks like to put the DB on flash for
HDD based clusters. It's still possible to oversubscribe them, but
you've got more headroom.
Mark
On 4/12/19 10:25 AM, Charles Alva wrote:
> Thanks Mark,
>
> This is interesting. I'll take a look at the links you provided.
>
> Does rocksdb compacting issue only affect HDDs? Or SSDs are having
> same issue?
>
> Kind regards,
>
> Charles Alva
> Sent from Gmail Mobile
>
> On Fri, Apr 12, 2019, 9:01 PM Mark Nelson <mnelson@xxxxxxxxxx
> <mailto:mnelson@xxxxxxxxxx>> wrote:
>
> Hi Charles,
>
>
> Basically the goal is to reduce write-amplification as much as
> possible. The deeper that the rocksdb hierarchy gets, the worse the
> write-amplifcation for compaction is going to be. If you look at the
> OSD logs you'll see the write-amp factors for compaction in the
> rocksdb
> compaction summary sections that periodically pop up. There's a
> couple
> of things we are trying to see if we can improve things on our end:
>
>
> 1) Adam has been working on experimenting with sharding data across
> multiple column families. The idea here is that it might be
> better to
> hav multiple L0 and L1 levels rather than L0, L1, L2 and L3. I'm not
> sure if this will pan out of not, but that was one of the goals
> behind
> trying this.
>
>
> 2) Toshiba recently released trocksdb which could have a really big
> impact on compaction write amplification:
>
>
> Code: https://github.com/ToshibaMemoryAmerica/trocksdb/tree/TRocksRel
>
> Wiki: https://github.com/ToshibaMemoryAmerica/trocksdb/wiki
>
>
> I recently took a look to see if our key/value size distribution
> would
> work well with the approach that trocksdb is taking to reduce
> write-amplification:
>
>
> https://docs.google.com/spreadsheets/d/1fNFI8U-JRkU5uaRJzgg5rNxqhgRJFlDB4TsTAVsuYkk/edit?usp=sharing
>
>
> The good news is that it sounds like the "Trocks Ratio" for the
> data we
> put in rocksdb is sufficiently high that we'd see some benefit
> since it
> should greatly reduce write-amplification during compaction for data
> (but not keys). This doesn't help your immediate problem, but I
> wanted
> you to know that you aren't the only one and we are thinking about
> ways
> to reduce the compaction impact.
>
>
> Mark
>
>
> On 4/10/19 2:07 AM, Charles Alva wrote:
> > Hi Ceph Users,
> >
> > Is there a way around to minimize rocksdb compacting event so
> that it
> > won't use all the spinning disk IO utilization and avoid it being
> > marked as down due to fail to send heartbeat to others?
> >
> > Right now we have frequent high IO disk utilization for every 20-25
> > minutes where the rocksdb reaches level 4 with 67GB data to compact.
> >
> >
> > Kind regards,
> >
> > Charles Alva
> > Sent from Gmail Mobile
> >
> > _______________________________________________
> > ceph-users mailing list
> > ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx>
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx>
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
