Hey Angelo, Ya, we are using the RBD driver for quite a few customers in production, and it is working quite good! Hahahaha, I am familiar with the bug you are talking about I think, I believe that may be resolved by now. I believe the driver is either out of beta now/soon to be out of beta? I recall watching a talk by the developers where they mentioned it should be out of beta soon or already is, in addition I want to say they are also/would also eventually be offering a paid support for it, but can't say for certain on that. Otherwise though like I said we are using it for quite a few customers with some really good success. Also, if you have any further questions re: CephFS and SMB please feel free to ask! I am sure other people might also find the same questions and answers helpful 😊 Regards, Bailey >-----Original Message----- >From: Angelo Hongens <angelo@xxxxxxxxxx> >Sent: April 29, 2023 11:21 PM >To: Bailey Allison <ballison@xxxxxxxxxxxx>; ceph-users@xxxxxxx >Subject: Re: architecture help (iscsi, rbd, backups?) > >Bailey, > >Thanks for your extensive reply, you got me down the wormhole of CephFS and SMB (and looking at a lot of 45drives videos and knowledge base, Houston dashboard, >reading up on CTDB, etc), and this is a really interesting option as well! Thanks for the write-up. > > >By the way, are you using the RBD driver in Windows in production with your customers? > >The binaries are still called beta, and last time I tried it in a proof of concept setup (a while back), it would never connect and always crash out on me. After reporting >an issue, I did not get a response for almost three months before a dev responded that it was an unsupported ipv6 issue. Not a problem, and all very understandable, >it's open source software written mostly by volunteers, but I got a bit cautious about deploying this to production ;) > >Angelo. >On 27/04/2023 18:20, Bailey Allison wrote: > Hey Angelo, > > Just to make sure I'm understanding correctly, the main idea for the > use case is to be able to present Ceph storage to windows clients as SMB? > > If so, you can absolutely use CephFS to get that done. This is > something we do all the time with our cluster configurations, if we're > looking to present ceph storage to windows clients for the use case of > a file server is our standard choice, and to your point of > security/ACLs we can make use of joining the samba server that to an > existing active directory, and then assigning permissions through Windows. > > I will provide a high level overview of an average setup to hopefully > explain it better, and of course if you have any questions please let > me know. I understand that this is way different of a setup of what > you currently have planned, but it's a different choice that could > prove useful in your case. > > Essentially how it works is we have ceph cluster with CephFS > configured, of which we map CephFS kernel mounts onto some gateway > nodes, at which point we expose to clients via CTDB with SMB shares (CTDB for high availability). > > i.e > > ceph cluster > ceph fs > map cephfs kernel mount on linux client > > create smb share on top of cephfs kernel mount > connect to samba > share with windows clients. > > The SMB gateway nodes hosting samba also can be joined to an Active > Directory to allow setting Windows ACL permissions to allow more in > depth control of ACLs. > > Also I will say +1 for the RBD driver on Windows, something we also > make use of a lot and have a lot of success with. > > Again, please let me know if you need any insight or clarification, or > have any further questions. Hope this is of assistance. > > Regards, > > Bailey > > -----Original Message----- >> From: Angelo Höngens <angelo@xxxxxxxxxx> >> Sent: April 27, 2023 6:06 PM >> To: ceph-users@xxxxxxx >> Subject: architecture help (iscsi, rbd, backups?) >> >> Hey guys and girls, >> >> I'm working on a project to build storage for one of our departments, >> and I > want to ask you guys and girls for input on the high-level overview part. > It's a long one, I hope you read along and comment. >> >> SUMMARY >> >> I made a plan last year to build a 'storage solution' including ceph >> and > some windows VM's to expose the data over SMB to clients. A year later > I finally have the hardware, built a ceph cluster, and I'm doing > tests. Ceph itself runs great, but when I wanted to start exposing the > data using iscsi to our VMware farm, I ran into some issues. I know > the iscsi gateways will introduce some new performance bottlenecks, > but I'm seeing really slow performance, still working on that. >> >> But then I ran into the warning on the iscsi gateway page: "The iSCSI > gateway is in maintenance as of November 2022. This means that it is > no longer in active development and will not be updated to add new features.". > Wait, what? Why!? What does this mean? Does this mean that iSCSI is > now 'feature complete' and will still be supported the next 5 years, > or will it be deprecated in the future? I tried searching, but > couldn't find any info on the decision and the roadmap. >> >> My goal is to build a future-proof setup, and using deprecated >> components > should not be part of that of course. >> >> If the iscsi gateway will still be supported the next few years and I >> can > iron out the performance issues, I can still go on with my original > plan. If not, I have to go back to the drawing board. And maybe you > guys would advise me to take another route anyway. >> >> GOALS >> >> My goals/considerations are: >> >> - we want >1PB of storage capacity for cheap (on a tight budget) for > research data. Most of it is 'store once, read sometimes'. <1% of the > data is 'hot'. >> - focus is on capacity, but it would be nice to have > 200MB/s of > sequential write/read performance and not 'totally suck' on random > i/o. Yes, not very well quantified, but ah. Sequential writes are most important. >> - end users all run Windows computers (mostly VDI's) and a lot of > applications require SMB shares. >> - security is a big thing, we want really tight ACL's, specific >> monitoring > agents, etc. >> - our data is incredibly important to us, we still want the 3-2-1 >> backup > rule. Primary storage solution, a second storage solution in a > different place, and some of the data that is not reproducible is also > written to tape. We also want to be protected from ransomware or user > errors (so no direct replication to the second storage). >> - I like open source, reliability, no fork-lift upgrades, no vendor > lock-in, blah, well, I'm on the ceph list here, no need to convince > you guys > ;) >> - We're hiring a commercial company to do ceph maintenance and >> support for > when I'm on leave or leaving the company, but they won't support > clients, backup software, etc, so I want something as simple as > possible. We do have multiple Windows/VMware admins, but no other real linux guru's. >> >> THE INITIAL PLAN >> >> Given these considerations, I ordered two identical clusters, each > consisting of 3 monitor nodes and 8 osd nodes, Each osd node has 2 > ssd's and > 10 capacity disks (EC 4:2 for the data), and each node is connected > using a 2x25Gbps bond. Ceph is running like a charm. Now I just have > to think about exposing the data to end users, and I've been testing different setups. >> >> My original plan was to expose for example 10x100TB rbd images using >> iSCSI > to our VMware farm, formatting the luns with VMFS6, and run for > example 2 Windows file servers per datastore on that with a single DFS > namespace to end users. Then backup the file servers using our > existing Veeam infrastructure to RGW running on the second cluster > with an immutable bucket. This way we would have easily defined > security boundaries: the clients can only reach the file servers, the > file servers only see their local VMDK's, ESX only sees the luns on > the iSCSI target, etc. When a file server would be compromised, it > would have no access to ceph. We have easy incremental backups, > immutability for ransomware protection, etc. And the best part is that > the ceph admin can worry about ceph, the vmware admin can focus on > ESX, VMFS and all the vmware stuff, and the Windows admins can focus > on the Windows boxes, Windows-specific ACLS and tools and Veeam backups and stuff. >> >> CURRENT SITUATION >> >> I'm building out this plan now, but I'm running into issues with >> iSCSI. Are > any of you doing something similar? What is your iscsi performance > compared to direct rbd? >> >> In regard to performance: If I take 2 test windows VM's, I put one on >> an > iSCSI datastore and another with direct rbd access using the windows > rbd driver, I create a share on those boxes and push data to it, I see > different results (of course). Copying some iso images over SMB to the > 'windows vm running direct rbd' I see around 800MB/s write, and > 200MB/s read, which is pretty okay. When I send data to the 'windows > vm running on top of iscsi' it starts writing at around 350MB/s, but > after like 10-20 seconds drops to 100MB/s and won't go faster. Reads > are anywhere from 40MB/s to 80MB/s, which is not really acceptable. >> >> Another really viable and performant scenario would be to have the >> Windows > file servers connect to rbd directly with the windows rbd driver. It > seems to work well, it's fast, and you don't have the bottleneck that > the iscsi gateway creates. But I see this driver is still in beta. Is > anyone using this in production? What are your experiences? We would > miss out on the separation of layers and thus have less security, but > at the same time, it really increases efficiency and performance. >> >> And if I use rbd, then vmware won't see the storage, and I cannot do >> an > image backup using veeam. I could of course do backups of the rbd > images, using tools like restic or backy to rgw running on the second > cluster with immutable buckets. What are your experiences? Is it easy > to do differential backups of lots of 50TB rbd images? Change rate is > usually like 0.005% per day or something ;) >> >> By the way, we also thought about CephFS, but we have some complex >> stuff > going on with extended ACL's that I don't think will play nice with > CephFS, and I think it's a lot more complex to backup CephFS than block images. >> >> If you made it here, thank you for your time! I hope you can share >> thoughts > on my questions! >> >> Angelo. >> _______________________________________________ >> ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an >> email > to ceph-users-leave@xxxxxxx > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |