Re: Delay Between Writing Data and that Data being available for reading?

Thomas Sumpter <thomas.sumpter@xxxxxxxxxx> · Wed, 19 Sep 2018 14:30:54 +0000

Linux version 4.18.4-1.el7.elrepo.x86_64 (mockbuild@Build64R7) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC))
CentOS 7

From: Gregory Farnum <gfarnum@xxxxxxxxxx> 

Sent: Wednesday, September 19, 2018 4:27 PM

To: Thomas Sumpter <thomas.sumpter@xxxxxxxxxx>

Cc: ceph-users@xxxxxxxxxxxxxx

Subject: Re: [ceph-users] Delay Between Writing Data and that Data being available for reading?

Okay, so you’re using the kernel client. What kernel version is it? I think this was one of a few known bugs there a while ago that have since been fixed.

On Wed, Sep 19, 2018 at 7:24 AM Thomas Sumpter <thomas.sumpter@xxxxxxxxxx> wrote:

Hi Gregory,

Thanks for your reply.

Yes, the file is stored on CephFS.

Accessed using ceph client
Everything is a basic install following the ceph-deploy guide

Note sure what details would be helpful…
The file is written to by a webserver (apache)
The file is accessed by the webserver on request of some specific data within the file. The webserver will fetch the file from the local OS i.e.
file://some_file, which in turn is fetched from the cephfs.
An observation, if I tell the webserver to make a web request to itself to fetch the file (http://localhost/etc),
 then I don’t have the problem.

# mount | grep ceph
mon-1:6789,mon-2:6789,mon-3:6789:/ on /mnt/ceph type ceph (rw,noatime,name=admin,secret=<hidden>,acl)

# rpm -qa | grep ceph
libcephfs2-13.2.1-0.el7.x86_64
python-cephfs-13.2.1-0.el7.x86_64
ceph-common-13.2.1-0.el7.x86_64

# ceph --version
ceph version 13.2.1 (5533ecdc0fda920179d7ad84e0aa65a127b20d77) mimic (stable)

Regards,

Tom

From: Gregory Farnum <gfarnum@xxxxxxxxxx>

Sent: Wednesday, September 19, 2018 4:04 PM

To: Thomas Sumpter <thomas.sumpter@xxxxxxxxxx>

Cc: ceph-users@xxxxxxxxxxxxxx

Subject: Re: [ceph-users] Delay Between Writing Data and that Data being available for reading?

You're going to need to tell us *exactly* what you're doing. I presume this uses CephFS somehow? Are you accessing via NFS or something? Using what client versions?

CephFS certainly isn't supposed to allow this, and I don't think there are any currently known bugs which could leak it. But there are lots of things you can stack on top of it
 which won't provide the same guarantees.

On Wed, Sep 19, 2018 at 6:45 AM Thomas Sumpter <thomas.sumpter@xxxxxxxxxx> wrote:

Hello,

We have Mimic version 13.2.1 using Bluestore. OSDs are using NVMe disks for data storage (in AWS).
Four OSDs are active in replicated mode.
Further information on request, since there are so many config options I am not sure where to focus my attention yet. Assume we have default options.

We have a scenario where one file is continuously been written to and read from.

Very occasionally the write operation is completed but then the subsequent read op on that file does not contain this new data for a brief period.

Does anyone know a reason for the delay between write operations being completed and the new data that was written to be present in the file?

It is not very easily reproducible, but possible with fast scripted attempts. We do not have this problem when using other filesystems.

Note: there is one client writing data and 2 clients reading data to/from this file.

Many Thanks!
Tom

_______________________________________________

ceph-users mailing list

ceph-users@xxxxxxxxxxxxxx

http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com