Thanks Mark & Marc We will do more testing inc kernel client as well as testing the block storage performance first. We just did some direct raw performance test on a single spinning disk (format as ext4) and it could delivery 200-300MB/s throughput in various writing and mix testings. But FUSE client could only give ~50MB/s. Cheers, D On Thu, Jun 4, 2020 at 1:27 PM Mark Nelson <mnelson@xxxxxxxxxx> wrote: > Try using the kernel client instead of the FUSE client. The FUSE client > is known to be slow for a variety of reasons and I suspect you may see > faster performance with the kernel client. > > > Thanks, > > Mark > > > On 6/2/20 8:00 PM, Derrick Lin wrote: > > Hi guys, > > > > We just deployed a CEPH 14.2.9 cluster with the following hardware: > > > > MDSS x 1 > > Xeon Gold 5122 3.6Ghz > > 192GB > > Mellanox ConnectX-4 Lx 25GbE > > > > > > MON x 3 > > Xeon Bronze 3103 1.7Ghz > > 48GB > > Mellanox ConnectX-4 Lx 25GbE > > 6 x 600GB 10K SAS > > > > OSD x 5 > > Xeon Silver 4110 2.1Ghz x 2 > > 192GB > > Mellanox ConnectX-4 Lx 25GbE > > 16 x 10TB 7.2K NLSAS (block) > > 2 x 2TB Intel P4600 NVMe (block.db) > > > > Network is all Mellanox SN2410/SN2700 configured at 25GbE for both front > > and back network. > > > > Just for POC at this stage, the cluster was deployed by ceph_ansible > > without much customization and the initial test on its cephFS FUSE mount > > performance seems to be very low. We did some test with iozone the result > > as follow: > > > > ]# /opt/iozone/bin/iozone -i 0 -i 1-r 128k -s 5G -t 20 > > Iozone: Performance Test of File I/O > > Version $Revision: 3.465 $ > > Compiled for 64 bit mode. > > Build: linux-AMD64 > > > > Contributors:William Norcott, Don Capps, Isom Crawford, Kirby > > Collins > > Al Slater, Scott Rhine, Mike Wisner, Ken Goss > > Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain > CYR, > > Randy Dunlap, Mark Montague, Dan Million, Gavin > > Brebner, > > Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, > Dave > > Boone, > > Erik Habbinga, Kris Strecker, Walter Wong, Joshua > Root, > > Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren > Sawyer, > > Vangel Bojaxhi, Ben England, Vikentsi Lapa, > > Alexey Skidanov. > > > > Run began: Tue Jun 2 16:40:53 2020 > > > > File size set to 5242880 kB > > Command line used: /opt/iozone/bin/iozone -i 0 -i 1-r -s 5G -t > 20 > > 128k > > Output is in kBytes/sec > > Time Resolution = 0.000001 seconds. > > Processor cache size set to 1024 kBytes. > > Processor cache line size set to 32 bytes. > > File stride size set to 17 * record size. > > Throughput test with 20 processes > > Each process writes a 5242880 kByte file in 4 kByte records > > > > Children see throughput for 20 initial writers = 35001.12 > kB/sec > > Parent sees throughput for 20 initial writers = 34967.65 > kB/sec > > Min throughput per process = 1748.22 > kB/sec > > Max throughput per process = 1751.62 > kB/sec > > Avg throughput per process = 1750.06 > kB/sec > > Min xfer = 5232724.00 kB > > > > Children see throughput for 20 rewriters = 35704.79 > kB/sec > > Parent sees throughput for 20 rewriters = 35704.30 > kB/sec > > Min throughput per process = 1783.44 > kB/sec > > Max throughput per process = 1786.29 > kB/sec > > Avg throughput per process = 1785.24 > kB/sec > > Min xfer = 5234532.00 kB > > > > Children see throughput for 20 readers = 49368539.50 > kB/sec > > Parent sees throughput for 20 readers = 49317231.38 > kB/sec > > Min throughput per process = 2414424.00 > kB/sec > > Max throughput per process = 2599996.25 > kB/sec > > Avg throughput per process = 2468426.98 > kB/sec > > Min xfer = 4868708.00 kB > > > > Children see throughput for 20 re-readers = 48675891.50 > kB/sec > > Parent sees throughput for 20 re-readers = 48617335.67 > kB/sec > > Min throughput per process = 2316395.25 > kB/sec > > Max throughput per process = 2703868.75 > kB/sec > > Avg throughput per process = 2433794.58 > kB/sec > > Min xfer = 4491704.00 kB > > > > We also did some dd tests, the write speed on a single test on our > standard > > server is ~50MB/s but on a very big memory server, the speed is double ~ > > 80-90MB/s. > > > > We have zero experience on ceph and as said we haven't done more tuning > at > > this stage. But if this sort of performance is way too low from those > > hardware spec? > > > > Any hints will be appreciated. > > > > Cheers > > D > > _______________________________________________ > > ceph-users mailing list -- ceph-users@xxxxxxx > > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |