Hello Moritz, drop the EVO disk and use a SSD that is working with Ceph. For example just use PM883 / PM983 from the same vendor and you will have a huge performance increase. -- Martin Verges Managing director Mobile: +49 174 9335695 E-Mail: martin.verges@xxxxxxxx Chat: https://t.me/MartinVerges croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io YouTube: https://goo.gl/PGE1Bx Am So., 17. Mai 2020 um 15:12 Uhr schrieb Moritz Wilhelm <moritz@xxxxxxxxxxx >: > Hi Marc, > > thank you very much for your feedback, actually that is what I am looking > for (Design advices and feedback). I also wanted to get in touch with the > community because right now I am on my own with this project with no > experience at all. But I also wanted to get into it first and learn the > basics and setup a system before bothering other people with it and to be > able to even hold a conversation about it. > > So we are a 3D Content Production Company with around 10-12 Workstations > and 6 Render stations. We work with software like Autodesk Maya or Adobe > After effects. All our Projects are stored on the NAS and we work directly > with those files on the NAS. > > This is our current System: > > Synology RS4017xs+ > Intel Xeon D-1541 2,1 Ghz, 8 Cores > 32GB Ram > 16x 8TB WD Red Drives > 2 TB SSD Cache > 2 x 10 Gb SFP+ Nics > > > About the Bcache I just did tests with it as I found the idea very > interesting and also performance was better. If this project is ever > getting into real-life production I will stick to the most common setups > for sure. > > Here are some Results of Diskmark on one Computer. (I Had only 5 OSDs per > Node up and running at this test because I am still waiting for the new > Seagate drives) For Sequential Reads and Writes I sometimes maxed out the > 10Gbe host connection so the data is not very useful but the other numbers > are quite interesting: > > For Comparison: > > Hosts Local NVME Drive (Samsung 970 EVO Pro 1TB): > Read: > Write: > Sequential 1MiB (Q= 8, T= 1): 3507.700 MB/s [ 3345.2 IOPS] < > 2389.61 us> Sequential 1MiB (Q= 8, T= 1): 2548.063 MB/s [ > 2430.0 IOPS] < 3284.79 us> > Sequential 1MiB (Q= 1, T= 1): 2368.308 MB/s [ 2258.6 IOPS] < > 442.07 us> Sequential 1MiB (Q= 1, T= 1): 2456.471 MB/s [ > 2342.7 IOPS] < 426.10 us> > Random 4KiB (Q= 32, T=16): 1555.565 MB/s [ 379776.6 IOPS] < > 1312.45 us> Random 4KiB (Q= 32, T=16): 16.580 MB/s [ > 4047.9 IOPS] <124267.88 us> > Random 4KiB (Q= 1, T= 1): 51.666 MB/s [ 12613.8 IOPS] < > 78.82 us> Random 4KiB (Q= 1, T= 1): 108.983 MB/s [ > 26607.2 IOPS] < 37.13 us> > > Current Synology NAS (SMB): > Read: > Write: > Sequential 1MiB (Q= 8, T= 1): 1045.594 MB/s [ 997.2 IOPS] < > 7990.33 us> Sequential 1MiB (Q= 8, T= 1): 1101.007 MB/s [ > 1050.0 IOPS] < 7588.25 us> > Sequential 1MiB (Q= 1, T= 1): 953.709 MB/s [ 909.5 IOPS] < > 1098.57 us> Sequential 1MiB (Q= 1, T= 1): 847.847 MB/s [ > 808.6 IOPS] < 1235.26 us> > Random 4KiB (Q= 32, T=16): 4.198 MB/s [ 1024.9 IOPS] > <380158.65 us> Random 4KiB (Q= 32, T=16): 188.827 MB/s [ > 46100.3 IOPS] < 11076.14 us> > Random 4KiB (Q= 1, T= 1): 2.486 MB/s [ 606.9 IOPS] < > 1637.08 us> Random 4KiB (Q= 1, T= 1): 7.177 MB/s > [ 1752.2 IOPS] < 570.16 us> > > Ceph With WAL/DB on NVME total of 5 old SATA HDD OSDs (SMB): > Read: > Write: > Sequential 1MiB (Q= 8, T= 1): 534.050 MB/s [ 509.3 IOPS] < > 15628.63 us> Sequential 1MiB (Q= 8, T= 1): 198.420 MB/s [ > 189.2 IOPS] < 42020.67 us> > Sequential 1MiB (Q= 1, T= 1): 340.580 MB/s [ 324.8 IOPS] < > 2921.17 us> Sequential 1MiB (Q= 1, T= 1): 184.329 MB/s [ > 175.8 IOPS] < 5603.99 us> > Random 4KiB (Q= 32, T=16): 3.172 MB/s [ 774.4 IOPS] > <398622.73 us> Random 4KiB (Q= 32, T=16): 8.639 MB/s [ > 2109.1 IOPS] <222699.43 us> > Random 4KiB (Q= 1, T= 1): 1.907 MB/s [ 465.6 IOPS] < > 2139.08 us> Random 4KiB (Q= 1, T= 1): 7.294 MB/s > [ 1780.8 IOPS] < 560.91 us> > > Ceph With Bcache total of 5 old SATA HDD OSDs (SMB): > Sequential 1MiB (Q= 8, T= 1): 967.386 MB/s [ 922.6 IOPS] < > 8660.59 us> Sequential 1MiB (Q= 8, T= 1): 882.882 MB/s [ > 842.0 IOPS] < 9444.23 us> > Sequential 1MiB (Q= 1, T= 1): 418.157 MB/s [ 398.8 IOPS] < > 2448.00 us> Sequential 1MiB (Q= 1, T= 1): 793.417 MB/s [ > 756.7 IOPS] < 1320.37 us> > Random 4KiB (Q= 32, T=16): 39.756 MB/s [ 9706.1 IOPS] < > 51788.10 us> Random 4KiB (Q= 32, T=16): 128.303 MB/s [ > 31324.0 IOPS] < 16298.94 us> > Random 4KiB (Q= 1, T= 1): 29.527 MB/s [ 7208.7 IOPS] < > 138.36 us> Random 4KiB (Q= 1, T= 1): 32.334 MB/s [ > 7894.0 IOPS] < 126.25 us> > > I will do an Update as soon as I have all the 18 OSDs running. > > I could also need some advice on how to do proper testing. > > > > > > __________________________________ > > EventArt GmbH > Kirchenstraße 13, > 3243 St. Leonhard am Forst > mobile: +43(0)6641951196 > mail: office@xxxxxxxxxxx > UID: ATU 64417269 > > http://www.eventart.at II http://www.clamlive.at > > -----Ursprüngliche Nachricht----- > Von: Marc Roos <M.Roos@xxxxxxxxxxxxxxxxx> > Gesendet: Sonntag, 17. Mai 2020 14:33 > An: ceph-users <ceph-users@xxxxxxx>; Moritz Wilhelm <moritz@xxxxxxxxxxx> > Betreff: RE: Ceph as a Fileserver for 3D Content Production > > > > I was just reading your post, and started wondering why you posted it. I > do not see clear question, and you also do not share test results (from > your nas vs cephfs smb). So maybe you like some attention in this covid > social distancing time? ;) > > Anyway, I have been 'testing' with ceph for 2,5 years or so. (I am telling > you so, you know how to value my info) If you start playing with ceph keep > it simple and stick to what is recommended. Do not start looking for > unconventional ways making ceph faster, because you are not able to do it > better than the developers. You do not know the ins and outs, and you are > more likely to shoot yourself in the foot. At least ask first. Eg this > bcache, I am not 100% sure what it is, but if it is sitting between the osd > process and the disk, it could be getting nasty with a reset/power outage, > when ceph thinks data is written to disk, while it is not. > > I would start by identifying what your minimum performance requirements > are, maybe post them, and ask if someone has realized them with a ceph > setup. > > > > > -----Original Message----- > Sent: 17 May 2020 13:42 > To: ceph-users@xxxxxxx > Subject: Ceph as a Fileserver for 3D Content Production > > Hi, > > my Name is Moritz and I am working for a 3D production company. Because of > the corona virus I have too much time left and also to much unused > hardware. That is why I started playing around with Ceph as a fileserver > for us. Here I want to share my experience for all those who are > interested. To start of here is my actual running test system. I am > interested in the thoughts of the community and also on more suggestions on > what to try out with my available Hardware. I don’t know how to test it > right now because I am a newbie to ceph and our production file server is a > super user-friendly but high performance Synology NAS 😉. > All I have done so far was running Crystal disk benchmark on 1 Windows > machine on the SMB Share. > > 3 Nodes: (original those where render workstations that are not in use > right now) > > Each Node is MON MGM OSD > > Mainboard: ASRock TRX40 Creator > CPU: AMD Ryzen Threadripper 3960X, 24 Cores, 3.8Ghz > RAM: 2 x Samsung 32 GB 2 x 8 DDR4 2666 MHz 288-pin DIMM, Unregistered, ECC > (64 GB Total) NIC Public: OnBoard Aquantia 107, 10Gbit NIC Ceph: > Intel XXV710-DA2, 2x SFP28, 25Gbit System Drive: 2x Samsung SSD 860 PRO > 256GB, SATA, ZFS Raid 1 > System: Proxmox VE 6.2, Debian Buster, Ceph Nautilus > HBA: Broadcom SAS 9305-16i > > OSDs: > 6x Seagate Exos, 16TB, 7.200 rpm, 12Gb SAS > > Cache: > 1x Micron 9300 MAX 3.2TB U.2 NVME > > I Played around with setting it up as a WAL/DB Device. Right now I have > configured the Micron NVME as a BCache Infront of the 6 Seagate Drives in > writeback mode. > Because in this configuration BCache takes care of translating random > writes to sequential ones for the HDDs I turned the Ceph WAL LOG off. I > think Bcache gives more options to tune the System for my use case instead > of just putting WAL/DB on the NVME. And also I can easily add cache drives > or remove them without touching osds. > > I set up SMB Shares with the vfs_ceph module. I still have to add CTDB to > distribute Samba to all nodes. > > My Next steps are to keep playing around in tuning the system and testing > stability and performance. After that I want to put the Ceph cluster > infront of our production NAS. Because our data is not super critical I > thought of setting the replicas to 2 and running Rsync overnight to our > NAS. That way I can switch to the old NAS at any time and wouldn’t loose > more than 1 Day of work which is acceptable for us. > This is how I could compare the two solutions side by side with real-life > workload. > > I know that ceph might not be the best solution right now but if I am able > to get at least similar performance to our Synology HDD NAS out of it, it > would give a super scalable Solution in size and performance to grow with > our needs. And who knows what performance improvements we get with ceph in > the next 3 years. > > I am happy to hear your thoughts and ideas. And please I know this might > be kind of a crazy setup but I have fun with it and I learned a lot the > last few weeks. If my experiment fails I will go back to my original > plan: Put FreeNas on two of the Nodes with overnight replication and put > the third Node back to his render-friends. 😃 > > By the way I also have a spare Dell Server: 2x Xeon E5-2630 v3 2,40GHz, > 128G Ram. I just don’t have an idea on how to utilize it. Maybe as extra > OSD Node or as a separate Samba Server to get the SMB traffic away from the > Public Ceph Network. > > > > > Moritz Wilhelm > > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an > email to ceph-users-leave@xxxxxxx > > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |