Den lör 24 sep. 2022 kl 23:38 skrev Murilo Morais <murilo@xxxxxxxxxxxxxx>: > I'm relatively new to Ceph. I set up a small cluster with two hosts with 12 > disks each host, all 3 TB SAS 7500 RPM and two 10 Gigabit interfaces. I > created a pool in replicated mode and configured it to use two replicas. > > What I'm finding strange is that, with these configurations, using the NFS > or iSCSI protocol, I'm getting approximately 500 Megabytes per second in > writing and 250 Megabytes per second in reading. Is this correct? Just one > disk can achieve rates of approximately 200 Megabytes per second. According > to Grafana's graphics, disks are not used extensively. This is very dependent on how you are testing reads. If your program* sends a request to read X MB, then waits for this to arrive, and then asks for X MB more and repeats until all of the data is received, you should get single-drive-speed values for reads, as long as the data is not in any cache. In order to make all 24 drives help out, you would need to make sure there is 24+ threads separately asking for data in different places to be read at the same time, so that the requests have a decent chance to go to all drives (this is slightly dependant on how data is randomized on the drives and hosts of course), otherwise you are mostly reading from one drive at a time and getting performance accordingly. Writes get more chances to "lie" a bit and tell you it is almost-certainly-written so that the writer thread can move on to next piece, and this can be done at many levels in the client, in the network protocols, server device drivers, drive caches and all that, adding up to more perf than reads would give where uncached data request can't be finished early. Then ceph has some overhead, and translating it to iSCSI or NFS would add some more overhead. *) Like you would get when running "dd if=./big-piece-of-data of=/dev/null bs=1M" -- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |