Hello, I'd like some advices about the setup of a new ceph cluster. Here the use case : RadowGW (S3 and maybe swift for hadoop/spark) will be the main usage. Most of the access will be in read only mode. Write access will only be done by the admin to update the datasets. We might use rbd some time to sync data as temp storage (when POSIX is needed) but performance will not be an issue here. We might use cephfs in the futur if that can replace a filesystem on rdb. We gonna start with 16 nodes (up to 24). The configuration of each node is : CPU : 2 x Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz (12c/48t) Memory : 128GB OS Storage : 2 x SSD 240GB Intel S3500 DC (raid 1) Journal or cache Storage : 2 x SSD 400GB Intel S3300 DC (no Raid) OSD Disk : 10 x HGST ultrastar-7k6000 6TB Public Network : 1 x 10Gb/s Private Network : 1 x 10Gb/s OS : Ubuntu 16.04 Ceph version : Jewel The question is : journal or cache tier (read only) on the SD 400GB Intel S3300 DC ? Each disk is able to write sequentially at 220MB/s. SSDs can write at ~500MB/s. if we set 5 journals on each SSDs, SSD will still be the bottleneck (1GB/s vs 2GB/s). If we set the journal on OSDs, we can expect a good throughput in read on the disk (in case of data not in the cache) and write shouldn't be so bad too, even if we have random read on the OSD during the write ? SSDs as cache tier seem to be a better usage than only 5 journal on each ? Is that correct ? We gonna use an EC pool for big files (jerasure 8+2 I think) and a replicated pool for small files. If I check on http://ceph.com/pgcalc/, in this use case replicated pool: pg_num = 8192 for 160 OSDs but 16384 for 240 OSDs Ec pool : pg_num = 4096 and pgp_num = pg_num Should I set the pg_num to 8192 or 16384 ? what is the impact on the cluster if we set the pg_num to 16384 at the beginning ? 16384 is high, isn't it ? Thanks for your help -- Yoann Moulin EPFL IC-IT _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |