On Mon, 23 Aug 2021 at 00:59, Kai Börnert <kai.boernert@xxxxxxxxx> wrote: > > As far as i understand, more important factor (for the ssds) is if they > have power loss protections (so they can use their ondevice write cache) > and how many iops they have when using direct writes with queue depth 1 So what you're saying is that where the WAL is stored is supercritical, since it could kill performance completely? > I just did a test for a hdd with block.db on ssd cluster using extra > cheap consumer ssds, adding the ssds reduced! the performance by about > 1-2 magnitudes > > While it is running the benchmark ssds are at 100%io according to > iostat, the hdds are below 10%, the performance is an absolute joke > > pinksupervisor:~$ sudo rados bench -p scbench 5 write --no-cleanup > hints = 1 > Maintaining 16 concurrent writes of 4194304 bytes to objects of size > 4194304 for up to 5 seconds or 0 objects > Total time run: 15.5223 > Total writes made: 21 > Write size: 4194304 > Object size: 4194304 > Bandwidth (MB/sec): 5.41157 > Stddev Bandwidth: 3.19595 > Max bandwidth (MB/sec): 12 > Min bandwidth (MB/sec): 0 > Average IOPS: 1 > Stddev IOPS: 0.798809 > Max IOPS: 3 > Min IOPS: 0 > Average Latency(s): 11.1352 > Stddev Latency(s): 4.79918 > Max latency(s): 15.4896 > Min latency(s): 1.13759 > > tl;dr the interface is not that important, a good sata drive can easily > beat a sas drive > > On 8/21/21 10:34 PM, Teoman Onay wrote: > > You seem to focus only on the controller bandwith while you should also > > consider disk rpms. Most SATA drives runs at 7200rpm while SAS ones goes > > from 10k to 15k rpm which increases the number of iops. > > > > Sata 80 iops > > Sas 10k 120iops > > Sas 15k 180iops > > > > MBTF of SAS drives is also higher than SATA ones. > > > > What is your use case ? RGW ? Small or large files ? RBD ? > > > > > > > > On Sat, 21 Aug 2021, 19:47 Roland Giesler, <roland@xxxxxxxxxxxxxx> wrote: > > > >> Hi all, > >> > >> (I asked this on the Proxmox forums, but I think it may be more > >> appropriate here.) > >> > >> In your practical experience, when I choose new hardware for a > >> cluster, is there any noticable difference between using SATA or SAS > >> drives. I know SAS drives can have a 12Gb/s interface and I think SATA > >> can only do 6Gb/s, but in my experience the drives themselves can't > >> write at 12Gb/s anyway, so it makes little if any difference. > >> > >> I use a combination of SSD's and SAS drives in my current cluster (in > >> different ceph pools), but I suspect that if I choose SATA enterprise > >> class drives for this project, it will get the same level of > >> performance. > >> > >> I think with ceph the hard error rate of drives becomes less relevant > >> that if I had used some level of RAID. > >> > >> Also, if I go with SATA, I can use AMD Epyc processors (and I don't > >> want to use a different supplier), which gives me a lot of extra cores > >> per unit at a lesser price, which of course all adds up to a better > >> deal in the end. > >> > >> I'd like to specifically hear from you what your experience is in this > >> regard. > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users@xxxxxxx > >> To unsubscribe send an email to ceph-users-leave@xxxxxxx > >> > >> > > _______________________________________________ > > ceph-users mailing list -- ceph-users@xxxxxxx > > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |