On Thu 01 Nov 2012 11:22:59 AM CDT, Nathan Howell wrote:
We have a small (3 node) Ceph cluster that occasionally has issues. It
loses files and directories, truncates them or fills the contents with
NULL bytes. So far we haven't been able to build a repro case but it
seems to happen when bulk loading data into the cluster, a process
that is run each evening by a cron job. We've gone about a month
without any issues but had it happen again yesterday during a larger
bulk load. The data is backed up outside of ceph and can be reloaded
but finding the corrupt files takes quite a while.
Has anyone heard of similar issues before? Should I try upgrading to
0.48.2 or a newer kernel?
Hi Nathan,
Do the writes succeed? I.e. the programs creating the files don't get
errors back? Are you seeing any problems with the ceph mds or osd
processes crashing? Can you describe your I/O workload during these
bulk loads? How many files, how much data, multiple clients writing,
etc.
As far as I know, there haven't been any fixes to 0.48.2 to resolve
problems like yours. You might try the ceph fuse client to see if you
get the same behavior. If not, then at least we have narrowed down the
problem to the ceph kernel client.
Thanks,
-sam
ceph version 0.48.1argonaut (commit:a7ad701b9bd479f20429f19e6fea7373ca6bba7c)
Linux _ 3.4.4-gentoo #2 SMP Sun Jul 1 18:28:16 UTC 2012 x86_64
Intel(R) Xeon(R) CPU E31240 @ 3.30GHz GenuineIntel GNU/Linux
I'm using the kernel provided cephfs, mounted with these options:
10.0.2.2:6789:/ on /ceph type ceph (rw,noatime,nodiratime)
thanks,
-n
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
