Re: DB/WALL and RGW index on the same NVME

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



>
> My understanding is that omap and EC are incompatible, though.

 Reason why multipart upload is using a non-EC pool to save metadata to an
omap database?




On Mon, 8 Apr 2024 at 20:21, Anthony D'Atri <anthony.datri@xxxxxxxxx> wrote:

> My understanding is that omap and EC are incompatible, though.
>
> > On Apr 8, 2024, at 09:46, David Orman <ormandj@xxxxxxxxxxxx> wrote:
> >
> > I would suggest that you might consider EC vs. replication for index
> data, and the latency implications. There's more than just the nvme vs.
> rotational discussion to entertain, especially if using the more widely
> spread EC modes like 8+3. It would be worth testing for your particular
> workload.
> >
> > Also make sure to factor in storage utilization if you expect to see
> versioning/object lock in use. This can be the source of a significant
> amount of additional consumption that isn't planned for initially.
> >
> > On Mon, Apr 8, 2024, at 01:42, Daniel Parkes wrote:
> >> Hi Lukasz,
> >>
> >> RGW uses Omap objects for the index pool; Omaps are stored in Rocksdb
> >> database of each osd, not on the actual index pool, so by putting
> DB/WALL
> >> on an NVMe as you mentioned, you are already configuring the index pool
> on
> >> a non-rotational drive, you don't need to do anything else.
> >>
> >> You just need to size your DB/WALL partition accordingly. For RGW/object
> >> storage, a good starting point for the DB/Wall sizing is 4%.
> >>
> >> Example of Omap entries in the index pool using 0 bytes, as they are
> stored
> >> in Rocksdb:
> >>
> >> # rados -p default.rgw.buckets.index listomapkeys
> >> .dir.7fb0a3df-9553-4a76-938d-d23711e67677.34162.1.2
> >> file1
> >> file2
> >> file4
> >> file10
> >>
> >> rados df -p default.rgw.buckets.index
> >> POOL_NAME                  USED  OBJECTS  CLONES  COPIES
> >> MISSING_ON_PRIMARY  UNFOUND  DEGRADED  RD_OPS       RD  WR_OPS      WR
> >> USED COMPR  UNDER COMPR
> >> default.rgw.buckets.index   0 B       11       0      33
> >>    0        0         0     208  207 KiB      41  20 KiB         0 B
> >>        0 B
> >>
> >> # rados -p default.rgw.buckets.index stat
> >> .dir.7fb0a3df-9553-4a76-938d-d23711e67677.34162.1.2
> >>
> default.rgw.buckets.index/.dir.7fb0a3df-9553-4a76-938d-d23711e67677.34162.1.2
> >> mtime 2022-12-20T07:32:11.000000-0500, size 0
> >>
> >>
> >> On Sun, Apr 7, 2024 at 10:06 PM Lukasz Borek <lukasz@xxxxxxxxxxxx>
> wrote:
> >>
> >>> Hi!
> >>>
> >>> I'm working on a POC cluster setup dedicated to backup app writing
> objects
> >>> via s3 (large objects, up to 1TB transferred via multipart upload
> process).
> >>>
> >>> Initial setup is 18 storage nodes (12HDDs + 1 NVME card for DB/WALL) +
> EC
> >>> pool.  Plan is to use cephadm.
> >>>
> >>> I'd like to follow good practice and put the RGW index pool on a
> >>> no-rotation drive. Question is how to do it?
> >>>
> >>>   - replace a few HDDs (1 per node) with a SSD (how many? 4-6-8?)
> >>>   - reserve space on NVME drive on each node, create lv based OSD and
> let
> >>>   rgb index use the same NVME drive as DB/WALL
> >>>
> >>> Thoughts?
> >>>
> >>> --
> >>> Lukasz
> >>> _______________________________________________
> >>> ceph-users mailing list -- ceph-users@xxxxxxx
> >>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
> >>>
> >>>
> >> _______________________________________________
> >> ceph-users mailing list -- ceph-users@xxxxxxx
> >> To unsubscribe send an email to ceph-users-leave@xxxxxxx
> > _______________________________________________
> > ceph-users mailing list -- ceph-users@xxxxxxx
> > To unsubscribe send an email to ceph-users-leave@xxxxxxx
>
>

-- 
Łukasz Borek
lukasz@xxxxxxxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux