Re: Suggestion to build ceph storage

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Sun, 19 Jun 2022 at 02:29, Satish Patel <satish.txt@xxxxxxxxx> wrote:

> Greeting folks,
>
> We are planning to build Ceph storage for mostly cephFS for HPC workload
> and in future we are planning to expand to S3 style but that is yet to be
> decided. Because we need mass storage, we bought the following HW.
>
> 15 Total servers and each server has a 12x18TB HDD (spinning disk) . We
> understand SSD/NvME would be best fit but it's way out of budget.
>
> I hope you have extra HW on hand for Monitor and MDS  servers


> Ceph recommends using a faster disk for wal/db if the data disk is slow and
> in my case I do have a slower disk for data.
>
> Question:
> 1. Let's say if i want to put a NvME disk for wal/db then what size i
> should buy.
>

The official recommendation is to budget 4% of OSD size for WAL/DB - so in
your case that would be 720GB per OSD. Especially if you want to go to S3
later you should stick closer to that limit since RGW is a heavy meta data
user.
Also with 12 OSD per node you should have at least 2 NVME - so 2x4TB might
do or maybe 3x3TB
The WAL/DB device is a Single Point of Failure for all OSDs attached (in
other words - if the WAL/DB device fails then all OSDs that have their
WAL/DB located there need to be rebuilt)
Make sure you budget for good number of DWPD (I assume in HPC scenario
you'll have a lot of scratch data) and test it with O_DIRECT and F_SYNC and
QD=1 and BS=4K to find one that can reliably handle high IOPS under that
condition


> 2. Do I need to partition wal/db for each OSD or just a single
> partition can share for all OSDs?
>

You need one partition per OSD


> 3. Can I put the OS on the same disk where the wal/db is going to sit ?
> (This way i don't need to spend extra money for extra disk)
>

Yes you can but in your case that would mean putting the WAL/DB on the HDD
- I would predict your HPC users not being very impressed with the
resulting performance but YMMV


> Any suggestions you have for this kind of storage would be much
> appreciated.
>

Budget plenty of RAM to deal with  recovery scenarios - I'd say in your
case 256GB minimum.
Normally you build a POC and test the heck out of it to cover your usage
scenarios but you already bought the HW so not a lot you can change now -
but you should test and tune your setup before you put production data on
it to ensure that you have a good idea how the system is going to behave
when it get s under load. Make sure you test failure scenarios (failing
OSDs,  failing nodes, network cuts, failing MDS etc.) so you know what to
expect and how to handle them

Another bottleneck in CephFS setups tends to be the MDS - again in your
setup you probably want at least 2 MDS in active-active (i.e. shared load)
plus 1 or 2 on standby as failover but others on this list have more
experience with that.


_______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux