probably caused by http://tracker.ceph.com/issues/12551 On Wed, Sep 2, 2015 at 7:57 PM, Arthur Liu <arthurhsliu@xxxxxxxxx> wrote: > Hi, > > I am experiencing an issue with CephFS with cache tiering where the kernel > clients are reading files filled entirely with 0s. > > The setup: > ceph 0.94.3 > create cephfs_metadata replicated pool > create cephfs_data replicated pool > cephfs was created on the above two pools, populated with files, then: > create cephfs_ssd_cache replicated pool, > then adding the tiers: > ceph osd tier add cephfs_data cephfs_ssd_cache > ceph osd tier cache-mode cephfs_ssd_cache writeback > ceph osd tier set-overlay cephfs_data cephfs_ssd_cache > > While the cephfs_ssd_cache pool is empty, multiple kernel clients on > different hosts open the same file (the size of the file is small, <10k) at > approximately the same time. A number of the clients from the OS level see > the entire file being empty. I can do a rados -p {cache pool} ls for the > list of files cached, and do a rados -p {cache pool} get {object} /tmp/file > and see the complete contents of the file. > I can repeat this by setting cache-mode to forward, rados -p {cache pool} > cache-flush-evict-all, checking no more objects in cache with rados -p > {cache pool} ls, resetting cache-mode to writeback with an empty pool, and > doing the multiple same file opens. > > Has anyone seen this issue? It seems like what may be a race condition where > the object is not yet completely loaded into the cache pool so the cache > pool serves out an incomplete object. > If anyone can shed some light or any suggestions to help debug this issue, > that would be very helpful. > > Thanks, > Arthur > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |