I think I hit the same issue.
I have corrupted data on cephfs and I don't remember the same issue
before Luminous (i did the same tests before).
It is on my test 1 node cluster with lower memory then recommended (so
server is swapping) but it shouldn't lose data (it never did before).
So slow requests may appear in the log like Florent B mentioned.
My test is to take some bigger files (few GB) and copy it to cephfs or
from cephfs to cephfs and stress the cluster so data copying stall for a
while. It will resume in few seconds/minutes and everything looks ok (no
error on copying). But copied file may be corrupted silently.
I checked wiles with MD5SUM and compared some corrupted files in detail.
There were missing some 4MB blocks of data (cephfs object size) -
corrupted file had that block of data filled with zeroes.
My idea is, that there happen something wrong when cluster is under
pressure and client want to save the block. Client gets OK and continues
with another block so data is lost and corrupted block is filled with zeros.
I tried kernel client 4.x and ceph-fuse client with same result.
I'm using erasure for cephfs data pool, cache tier and my storage is
bluestore and filestore mixed.
How can I help to debug or what should I do to help to find the problem?
With regards
Jan Pekar
On 14.12.2017 15:41, Yan, Zheng wrote:
On Thu, Dec 14, 2017 at 8:52 PM, Florent B <florent@xxxxxxxxxxx> wrote:
On 14/12/2017 03:38, Yan, Zheng wrote:
On Thu, Dec 14, 2017 at 12:49 AM, Florent B <florent@xxxxxxxxxxx> wrote:
Systems are on Debian Jessie : kernel 3.16.0-4-amd64 & libfuse 2.9.3-15.
I don't know pattern of corruption, but according to error message in
Dovecot, it seems to expect data to read but reach EOF.
All seems fine using fuse_disable_pagecache (no more corruption, and
performance increased : no more MDS slow requests on filelock requests).
I checked ceph-fuse changes since kraken, didn't find any clue. I
would be helpful if you can try recent version kernel.
Regards
Yan, Zheng
Problem occurred this morning even with fuse_disable_pagecache=true.
It seems to be a lock issue between imap & lmtp processes.
Dovecot uses fcntl as locking method. Is there any change about it in
Luminous ? I switched to flock to see if problem is still there...
I don't remenber there is any change.
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
============
Ing. Jan Pekař
jan.pekar@xxxxxxxxx | +420603811737
----
Imatic | Jagellonská 14 | Praha 3 | 130 00
http://www.imatic.cz
============
--
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
