Re: Delay Between Writing Data and that Data being available for reading?

[Thomas Sumpter <thomas.sumpter@xxxxxxxxxx>]

In case you or anyone else reading is interested, I tried using the latest fuse client instead of the kernel client and my problem seems to be gone.

I think our kernel is recent enough that it should include the bug fix you mentioned? So maybe some else going on there...

 

Regards,

Tom

 

From: Thomas Sumpter
Sent: Wednesday, September 19, 2018 4:31 PM
To: 'Gregory Farnum' <gfarnum@xxxxxxxxxx>
Cc: ceph-users@xxxxxxxxxxxxxx
Subject: RE: [ceph-users] Delay Between Writing Data and that Data being available for reading?

 

Linux version 4.18.4-1.el7.elrepo.x86_64 (mockbuild@Build64R7) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC))

CentOS 7

 

From: Gregory Farnum <gfarnum@xxxxxxxxxx>
Sent: Wednesday, September 19, 2018 4:27 PM
To: Thomas Sumpter <thomas.sumpter@xxxxxxxxxx>
Cc: ceph-users@xxxxxxxxxxxxxx
Subject: Re: [ceph-users] Delay Between Writing Data and that Data being available for reading?

 

Okay, so you’re using the kernel client. What kernel version is it? I think this was one of a few known bugs there a while ago that have since been fixed.

On Wed, Sep 19, 2018 at 7:24 AM Thomas Sumpter <thomas.sumpter@xxxxxxxxxx> wrote:

Hi Gregory,

 

Thanks for your reply.

 

Yes, the file is stored on CephFS.

Accessed using ceph client

Everything is a basic install following the ceph-deploy guide

 

Note sure what details would be helpful…

The file is written to by a webserver (apache)

The file is accessed by the webserver on request of some specific data within the file. The webserver will fetch the file from the local OS i.e. file://some_file, which in turn is fetched from the cephfs.

An observation, if I tell the webserver to make a web request to itself to fetch the file (http://localhost/etc), then I don’t have the problem.

 

# mount | grep ceph

mon-1:6789,mon-2:6789,mon-3:6789:/ on /mnt/ceph type ceph (rw,noatime,name=admin,secret=<hidden>,acl)

 

# rpm -qa | grep ceph

libcephfs2-13.2.1-0.el7.x86_64

python-cephfs-13.2.1-0.el7.x86_64

ceph-common-13.2.1-0.el7.x86_64

 

# ceph --version

ceph version 13.2.1 (5533ecdc0fda920179d7ad84e0aa65a127b20d77) mimic (stable)

 

Regards,

Tom

 

From: Gregory Farnum <gfarnum@xxxxxxxxxx>
Sent: Wednesday, September 19, 2018 4:04 PM
To: Thomas Sumpter <thomas.sumpter@xxxxxxxxxx>
Cc: ceph-users@xxxxxxxxxxxxxx
Subject: Re: [ceph-users] Delay Between Writing Data and that Data being available for reading?

 

You're going to need to tell us *exactly* what you're doing. I presume this uses CephFS somehow? Are you accessing via NFS or something? Using what client versions?

 

CephFS certainly isn't supposed to allow this, and I don't think there are any currently known bugs which could leak it. But there are lots of things you can stack on top of it which won't provide the same guarantees.

 

On Wed, Sep 19, 2018 at 6:45 AM Thomas Sumpter <thomas.sumpter@xxxxxxxxxx> wrote:

Hello,

 

We have Mimic version 13.2.1 using Bluestore. OSDs are using NVMe disks for data storage (in AWS).

Four OSDs are active in replicated mode.

Further information on request, since there are so many config options I am not sure where to focus my attention yet. Assume we have default options.

 

We have a scenario where one file is continuously been written to and read from.

Very occasionally the write operation is completed but then the subsequent read op on that file does not contain this new data for a brief period.

Does anyone know a reason for the delay between write operations being completed and the new data that was written to be present in the file?

It is not very easily reproducible, but possible with fast scripted attempts. We do not have this problem when using other filesystems.

 

Note: there is one client writing data and 2 clients reading data to/from this file.

 

Many Thanks!

Tom

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com