On 12/30/2014 09:40 AM, Chen, Xiaoxi wrote: > Hi, > First of all, the data is safe since it's persistent in journal, if error occurs on OSD data partition, replay the journal will get the data back. Agree. Data are safe in journal. But when journal is flushed data are moved to a filestore and not flushed to disk immediately. > And, there is a wbthrottle there, you can config how much data(ios, bytes, inodes) you wants to remain in memory. A background thread will start to flush data into disk when any of the value exceeds "filestore_wbthrottle_[xfs,btrfs]_[bytes,ios,inodes]_start_flusher", and will block the filestore op thread when hard limit exceeds. You could set these value to something smaller if you still not feeling comfortable:) I assume that you are talking about WBThrottle::entry() (src/os/WBThrottle.cc). There is fsync/fdatasync there but it's return value isn't checked at all. So if you call *write* you have data in dirty buffer. Then you flush that data to disk by calling *fsync* without checking it's return value. If there was an IO error *fsync* will return -1 meaning data has been lost. OSD will not be aware of this. > Xiaoxi > > -----Original Message----- > From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of Pawe? Sadowski > Sent: Tuesday, December 30, 2014 4:10 PM > To: ceph-users > Subject: Ceph data consistency > > Hi, > > On our Ceph cluster from time to time we have some inconsistent PGs (after deep-scrub). We have some issues with disk/sata cables/lsi controller causing IO errors from time to time (but that's not the point in this case). > > When IO error occurs on OSD journal partition everything works as is should -> OSD is crashed and that's ok - Ceph will handle that. > > But when IO error occurs on OSD data partition during journal flush OSD continue to work. After calling *writev* (in buffer::list::write_fd) OSD does check return code from this call but does NOT verify if write has been successful to disk (data are still only in memory and there is no fsync). That way OSD thinks that data has been stored on disk but it might be discarded (during sync dirty page will be reclaimed and you'll see "lost page write due to I/O error" in dmesg). > > Since there is no checksumming of data I just wanted to make sure that this is by design. Maybe there is a way to tell OSD to call fsync after write and have data consistent? > -- PS _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |