Hi, Le 18/03/2016 20:58, Mark Nelson a écrit : > FWIW, from purely a performance perspective Ceph usually looks pretty > fantastic on a fresh BTRFS filesystem. In fact it will probably > continue to look great until you do small random writes to large > objects (like say to blocks in an RBD volume). Then COW starts > fragmenting the objects into oblivion. I've seen sequential read > performance drop by 300% after 5 minutes of 4K random writes to the > same RBD blocks. > > Autodefrag might help. With 3.19 it wasn't enough for our workload and we had to develop our own defragmentation, see scheduler https://github.com/jtek/ceph-utils. We tried autodefrag again with a 4.0.5 kernel but it wasn't good enough yet (and based on my reading of the linux-btrfs list I don't think there is any work done on it currently). > A long time ago I recall Josef told me it was dangerous to use (I > think it could run the node out of memory and corrupt the FS), but it > may be that it's safer now. No problem here (as long as we use our defragmentation scheduler, otherwise the performance degrades over time/amount of rewrites). > In any event we don't really do a lot of testing with BTRFS these > days as bluestore is indeed the next gen OSD backend. Will bluestore provide the same protection against bitrot than BTRFS? Ie: with BTRFS the deep-scrubs detect inconsistencies *and* the OSD(s) with invalid data get IO errors when trying to read corrupted data and as such can't be used as the source for repairs even if they are primary OSD(s). So with BTRFS you get a pretty good overall protection against bitrot in Ceph (it allowed us to automate the repair process in the most common cases). With XFS IIRC unless you override the default behavior the primary OSD is always the source for repairs (even if all the secondaries agree on another version of the data). Best regards, Lionel _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |