On 01/04/2019 07:32 PM, Peter Woodman wrote: > not to mention that the current released version of mimic (.2) has a > bug that is potentially catastrophic to cephfs, known about for > months, yet it's not in the release notes. would have upgraded and > destroyed data had i not caught a thread on this list. indeed. we're a big cephfs user here for HPC. everytime I get asked about it by my peers, sadly I have to tell them that they should not use it for production, that it's not stable and has serious stability bugs (eventhough it was declared "stable" upstream some time ago). (e.g. doing an rsync on, from or to a cephfs, just like someone wrote a couple of days again on the list, reliably kills it, everytime - we reproduce it with every kernel release and every ceph release since february 2015 on several independent clusters. even more catastropic is that single inconsistent files stopps the whole cephfs which then cannot be restored unless the affected cephfs is unmounted on all(!) machines that have it mounted, etc. we can use cephfs only in our sort-of-stable setup with 12.2.5 because we have mostly non-malicious users that usualy behave nicely. but it's to brittle in the end and apparently no silver lining ahead. because of that, during our scaling up of our cephfs cluster from 300tb to 1.2pb this spring, we'll be moving away from cephfs entirely and switch to mounting RBDs and export them with samba instead. we have good experiences with RBDs on other clusters. but using RBDs that way is quite painful when knowing that cephfs exists, it's slower, and not really HA anymore, but it's overall more reliable than cephfs) as much as I like ceph, I unfortunatly can't say the same for cephfs :( Regards, Daniel _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |