Hi Marc,
thank you very much for your feedback, actually that is what I am looking for (Design advices and feedback). I also wanted to get in touch with the community because right now I am on my own with this project with no experience at all. But I also wanted to get into it first and learn the basics and setup a system before bothering other people with it and to be able to even hold a conversation about it.
So we are a 3D Content Production Company with around 10-12 Workstations and 6 Render stations. We work with software like Autodesk Maya or Adobe After effects. All our Projects are stored on the NAS and we work directly with those files on the NAS.
This is our current System:
Synology RS4017xs+
Intel Xeon D-1541 2,1 Ghz, 8 Cores
32GB Ram
16x 8TB WD Red Drives
2 TB SSD Cache
2 x 10 Gb SFP+ Nics
About the Bcache I just did tests with it as I found the idea very interesting and also performance was better. If this project is ever getting into real-life production I will stick to the most common setups for sure.
Here are some Results of Diskmark on one Computer. (I Had only 5 OSDs per Node up and running at this test because I am still waiting for the new Seagate drives) For Sequential Reads and Writes I sometimes maxed out the 10Gbe host connection so the data is not very useful but the other numbers are quite interesting:
For Comparison:
Hosts Local NVME Drive (Samsung 970 EVO Pro 1TB):
Read: Write:
Sequential 1MiB (Q= 8, T= 1): 3507.700 MB/s [ 3345.2 IOPS] < 2389.61 us> Sequential 1MiB (Q= 8, T= 1): 2548.063 MB/s [ 2430.0 IOPS] < 3284.79 us>
Sequential 1MiB (Q= 1, T= 1): 2368.308 MB/s [ 2258.6 IOPS] < 442.07 us> Sequential 1MiB (Q= 1, T= 1): 2456.471 MB/s [ 2342.7 IOPS] < 426.10 us>
Random 4KiB (Q= 32, T=16): 1555.565 MB/s [ 379776.6 IOPS] < 1312.45 us> Random 4KiB (Q= 32, T=16): 16.580 MB/s [ 4047.9 IOPS] <124267.88 us>
Random 4KiB (Q= 1, T= 1): 51.666 MB/s [ 12613.8 IOPS] < 78.82 us> Random 4KiB (Q= 1, T= 1): 108.983 MB/s [ 26607.2 IOPS] < 37.13 us>
Current Synology NAS (SMB):
Read: Write:
Sequential 1MiB (Q= 8, T= 1): 1045.594 MB/s [ 997.2 IOPS] < 7990.33 us> Sequential 1MiB (Q= 8, T= 1): 1101.007 MB/s [ 1050.0 IOPS] < 7588.25 us>
Sequential 1MiB (Q= 1, T= 1): 953.709 MB/s [ 909.5 IOPS] < 1098.57 us> Sequential 1MiB (Q= 1, T= 1): 847.847 MB/s [ 808.6 IOPS] < 1235.26 us>
Random 4KiB (Q= 32, T=16): 4.198 MB/s [ 1024.9 IOPS] <380158.65 us> Random 4KiB (Q= 32, T=16): 188.827 MB/s [ 46100.3 IOPS] < 11076.14 us>
Random 4KiB (Q= 1, T= 1): 2.486 MB/s [ 606.9 IOPS] < 1637.08 us> Random 4KiB (Q= 1, T= 1): 7.177 MB/s [ 1752.2 IOPS] < 570.16 us>
Ceph With WAL/DB on NVME total of 5 old SATA HDD OSDs (SMB):
Read: Write:
Sequential 1MiB (Q= 8, T= 1): 534.050 MB/s [ 509.3 IOPS] < 15628.63 us> Sequential 1MiB (Q= 8, T= 1): 198.420 MB/s [ 189.2 IOPS] < 42020.67 us>
Sequential 1MiB (Q= 1, T= 1): 340.580 MB/s [ 324.8 IOPS] < 2921.17 us> Sequential 1MiB (Q= 1, T= 1): 184.329 MB/s [ 175.8 IOPS] < 5603.99 us>
Random 4KiB (Q= 32, T=16): 3.172 MB/s [ 774.4 IOPS] <398622.73 us> Random 4KiB (Q= 32, T=16): 8.639 MB/s [ 2109.1 IOPS] <222699.43 us>
Random 4KiB (Q= 1, T= 1): 1.907 MB/s [ 465.6 IOPS] < 2139.08 us> Random 4KiB (Q= 1, T= 1): 7.294 MB/s [ 1780.8 IOPS] < 560.91 us>
Ceph With Bcache total of 5 old SATA HDD OSDs (SMB):
Sequential 1MiB (Q= 8, T= 1): 967.386 MB/s [ 922.6 IOPS] < 8660.59 us> Sequential 1MiB (Q= 8, T= 1): 882.882 MB/s [ 842.0 IOPS] < 9444.23 us>
Sequential 1MiB (Q= 1, T= 1): 418.157 MB/s [ 398.8 IOPS] < 2448.00 us> Sequential 1MiB (Q= 1, T= 1): 793.417 MB/s [ 756.7 IOPS] < 1320.37 us>
Random 4KiB (Q= 32, T=16): 39.756 MB/s [ 9706.1 IOPS] < 51788.10 us> Random 4KiB (Q= 32, T=16): 128.303 MB/s [ 31324.0 IOPS] < 16298.94 us>
Random 4KiB (Q= 1, T= 1): 29.527 MB/s [ 7208.7 IOPS] < 138.36 us> Random 4KiB (Q= 1, T= 1): 32.334 MB/s [ 7894.0 IOPS] < 126.25 us>
I will do an Update as soon as I have all the 18 OSDs running.
I could also need some advice on how to do proper testing.
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
