On 7/29/12 10:31 AM, Vladimir Bashkirtsev wrote:
On 21/07/12 02:12, Tommi Virtanen wrote:
But it leaves me with very final question: should we rely on btrfs at
this
point given it is having such major faults? What if I will use well
tested
by time ext4?
You might want to try xfs. We hear/see problems with all three, but
xfs currently seems to have the best long-term performance and
reliability.
I'm not sure if anyone's run detailed tests with ext4 after the
xattrs-in-leveldb feature; before that, we ran into fs limitations.
Just reporting back what was going on for last week. I have rebuilt all
OSDs with fresh btrfs and leaf size of 64K. Straight after rebuild
everything was flying! But mysql processing I wrote about continued and
whole cluster was brought again to a stand still in a week. I have done
some investigation as to causes and it appears that fragmentation went
ballistic. Reading somewhere on the net I have seen suggestion that if
cow is not really needed then btrfs mounted with nocow option less
likely to get overly fragmented. Haven't tried it actually but wondering
will ceph cope well with nocow? ie does it rely on cow feature?
Something tells me that as ceph can run on FS which does not have cow we
actually can mount nocow. Just need some confirmation from devs.
Hi Vladimir,
I haven't tried nocow, but we did try with autodefrag which didn't do
much to improve the situation. So far most of the degradation I've seen
was also with small writes.
In the mean time I opted to convert all OSDs to xfs. Even after
rebuilding only two OSDs performance boost is apparent again. So it
appears that btrfs as it currently is in 3.4.6 is not up to prime time
and good number of random writes consistently bring it to a halt.
As xfs apparently have its own share of problems when ageing I think
that periodic online defragmentation may bring xfs back to reasonable
condition. Have anyone tried xfs defragmentation while ceph uses it?
I haven't tried doing xfs defragmentation while ceph is running, though
we did test performance degradation on XFS. XFS started out slower than
btrfs but degraded more slowly than btrfs, so overall ended up faster by
the end of the test. It would be interesting to try doing periodic
defragmentation and see if that brings the performance back up.
Mark
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
