Hi,
Thanks for looking into this, this seems to mitigate the problem.
Do you think this is just related to httpd or is going to impact other services such as nginx and be a wider point to know about / document?
Thanks
On Mon, Oct 24, 2016 at 2:28 PM, Yan, Zheng <ukernel@xxxxxxxxx> wrote:
I finally reproduced this issue. Adding following lines to httpd.conf
can workaround this issue.
EnableMMAP off
EnableSendfile off
On Sat, Sep 3, 2016 at 11:07 AM, Yan, Zheng <ukernel@xxxxxxxxx> wrote:
> On Fri, Sep 2, 2016 at 5:10 PM, Sean Redmond <sean.redmond1@xxxxxxxxx> wrote:
>> I have checked all the servers in scope running 'dmesg | grep -i stale' and
>> it does not yield any results.
>>
>> As a test I have rebooted the servers in scope and I can still replicate the
>> behavior 100% of the time.
>>
>
> Can you reproduce this bug manually? (updating file on one server and
> reading the file on another server). If you do, please enable
> debug_mds=10 , repeat the steps that reproduce this and send log to
> us.
>
> Regards
> Yan, Zheng
>
>
>> On Fri, Sep 2, 2016 at 4:37 AM, Yan, Zheng <ukernel@xxxxxxxxx> wrote:
>>>
>>> I think about this again. This issue could be caused by stale session.
>>> Could you check kernel logs of your servers. Are there any ceph
>>> related kernel message (such as "ceph: mds0 caps stale")
>>>
>>> Regards
>>> Yan, Zheng
>>>
>>>
>>> On Thu, Sep 1, 2016 at 11:02 PM, Sean Redmond <sean.redmond1@xxxxxxxxx>
>>> wrote:
>>> > Hi,
>>> >
>>> > It seems to be using syscall mmap() from what I read this indicates it
>>> > is
>>> > using memory-mapped IO.
>>> >
>>> > Please see a strace here: http://pastebin.com/6wjhSNrP
>>> >
>>> > Thanks
>>> >
>>> > On Wed, Aug 31, 2016 at 5:51 PM, Sean Redmond <sean.redmond1@xxxxxxxxx>
>>> > wrote:
>>> >>
>>> >> I am not sure how to tell?
>>> >>
>>> >> Server1 and Server2 mount the ceph file system using kernel client
>>> >> 4.7.2
>>> >> and I can replicate the problem using '/usr/bin/sum' to read the file
>>> >> or a
>>> >> http GET request via a web server (apache).
>>> >>
>>> >> On Wed, Aug 31, 2016 at 2:38 PM, Yan, Zheng <ukernel@xxxxxxxxx> wrote:
>>> >>>
>>> >>> On Wed, Aug 31, 2016 at 12:49 AM, Sean Redmond
>>> >>> <sean.redmond1@xxxxxxxxx>
>>> >>> wrote:
>>> >>> > Hi,
>>> >>> >
>>> >>> > I have been able to pick through the process a little further and
>>> >>> > replicate
>>> >>> > it via the command line. The flow seems looks like this:
>>> >>> >
>>> >>> > 1) The user uploads an image to webserver server 'uploader01' it
>>> >>> > gets
>>> >>> > written to a path such as
>>> >>> > '/cephfs/webdata/static/456/JHL/66448H-755h.jpg'
>>> >>> > on cephfs
>>> >>> >
>>> >>> > 2) The MDS makes the file meta data available for this new file
>>> >>> > immediately
>>> >>> > to all clients.
>>> >>> >
>>> >>> > 3) The 'uploader01' server asynchronously commits the file contents
>>> >>> > to
>>> >>> > disk
>>> >>> > as sync is not explicitly called during the upload.
>>> >>> >
>>> >>> > 4) Before step 3 is done the visitor requests the file via one of
>>> >>> > two
>>> >>> > web
>>> >>> > servers server1 or server2 - the MDS provides the meta data but the
>>> >>> > contents
>>> >>> > of the file is not committed to disk yet so the data read returns
>>> >>> > 0's -
>>> >>> > This
>>> >>> > is then cached by the file system page cache until it expires or is
>>> >>> > flushed
>>> >>> > manually.
>>> >>>
>>> >>> do server1 or server2 use memory-mapped IO to read the file?
>>> >>>
>>> >>> Regards
>>> >>> Yan, Zheng
>>> >>>
>>> >>> >
>>> >>> > 5) As step 4 typically only happens on one of the two web servers
>>> >>> > before
>>> >>> > step 3 is complete we get the mismatch between server1 and server2
>>> >>> > file
>>> >>> > system page cache.
>>> >>> >
>>> >>> > The below demonstrates how to reproduce this issue
>>> >>> >
>>> >>> > http://pastebin.com/QK8AemAb
>>> >>> >
>>> >>> > As we can see the checksum of the file returned by the web server is
>>> >>> > 0
>>> >>> > as
>>> >>> > the file contents has not been flushed to disk from server
>>> >>> > uploader01
>>> >>> >
>>> >>> > If however we call ‘sync’ as shown below the checksum is correct:
>>> >>> >
>>> >>> > http://pastebin.com/p4CfhEFt
>>> >>> >
>>> >>> > If we also wait for 10 seconds for the kernel to flush the dirty
>>> >>> > pages,
>>> >>> > we
>>> >>> > can also see the checksum is valid:
>>> >>> >
>>> >>> > http://pastebin.com/1w6UZzNQ
>>> >>> >
>>> >>> > It looks it maybe a race between the time it takes the uploader01
>>> >>> > server to
>>> >>> > commit the file to the file system and the fast incoming read
>>> >>> > request
>>> >>> > from
>>> >>> > the visiting user to server1 or server2.
>>> >>> >
>>> >>> > Thanks
>>> >>> >
>>> >>> >
>>> >>> > On Tue, Aug 30, 2016 at 10:21 AM, Sean Redmond
>>> >>> > <sean.redmond1@xxxxxxxxx>
>>> >>> > wrote:
>>> >>> >>
>>> >>> >> You are correct it only seems to impact recently modified files.
>>> >>> >>
>>> >>> >> On Tue, Aug 30, 2016 at 3:36 AM, Yan, Zheng <ukernel@xxxxxxxxx>
>>> >>> >> wrote:
>>> >>> >>>
>>> >>> >>> On Tue, Aug 30, 2016 at 2:11 AM, Gregory Farnum
>>> >>> >>> <gfarnum@xxxxxxxxxx>
>>> >>> >>> wrote:
>>> >>> >>> > On Mon, Aug 29, 2016 at 7:14 AM, Sean Redmond
>>> >>> >>> > <sean.redmond1@xxxxxxxxx>
>>> >>> >>> > wrote:
>>> >>> >>> >> Hi,
>>> >>> >>> >>
>>> >>> >>> >> I am running cephfs (10.2.2) with kernel 4.7.0-1. I have
>>> >>> >>> >> noticed
>>> >>> >>> >> that
>>> >>> >>> >> frequently static files are showing empty when serviced via a
>>> >>> >>> >> web
>>> >>> >>> >> server
>>> >>> >>> >> (apache). I have tracked this down further and can see when
>>> >>> >>> >> running a
>>> >>> >>> >> checksum against the file on the cephfs file system on the node
>>> >>> >>> >> serving the
>>> >>> >>> >> empty http response the checksum is '00000'
>>> >>> >>> >>
>>> >>> >>> >> The below shows the checksum on a defective node.
>>> >>> >>> >>
>>> >>> >>> >> [root@server2]# ls -al
>>> >>> >>> >> /cephfs/webdata/static/456/JHL/66448H-755h.jpg
>>> >>> >>> >> -rw-r--r-- 1 apache apache 53317 Aug 28 23:46
>>> >>> >>> >> /cephfs/webdata/static/456/JHL/66448H-755h.jpg
>>> >>> >>>
>>> >>> >>> It seems this file was modified recently. Maybe the web server
>>> >>> >>> silently modifies the files. Please check if this issue happens on
>>> >>> >>> older files.
>>> >>> >>>
>>> >>> >>> Regards
>>> >>> >>> Yan, Zheng
>>> >>> >>>
>>> >>> >>> >>
>>> >>> >>> >> [root@server2]# sum
>>> >>> >>> >> /cephfs/webdata/static/456/JHL/66448H-755h.jpg
>>> >>> >>> >> 00000 53
>>> >>> >>> >
>>> >>> >>> > So can we presume there are no file contents, and it's just 53
>>> >>> >>> > blocks
>>> >>> >>> > of zeros?
>>> >>> >>> >
>>> >>> >>> > This doesn't sound familiar to me; Zheng, do you have any ideas?
>>> >>> >>> > Anyway, ceph-fuse shouldn't be susceptible to this bug even with
>>> >>> >>> > the
>>> >>> >>> > page cache enabled; if you're just serving stuff via the web
>>> >>> >>> > it's
>>> >>> >>> > probably a better idea anyway (harder to break, easier to
>>> >>> >>> > update,
>>> >>> >>> > etc).
>>> >>> >>> > -Greg
>>> >>> >>> >
>>> >>> >>> >>
>>> >>> >>> >> The below shows the checksum on a working node.
>>> >>> >>> >>
>>> >>> >>> >> [root@server1]# ls -al
>>> >>> >>> >> /cephfs/webdata/static/456/JHL/66448H-755h.jpg
>>> >>> >>> >> -rw-r--r-- 1 apache apache 53317 Aug 28 23:46
>>> >>> >>> >> /cephfs/webdata/static/456/JHL/66448H-755h.jpg
>>> >>> >>> >>
>>> >>> >>> >> [root@server1]# sum
>>> >>> >>> >> /cephfs/webdata/static/456/JHL/66448H-755h.jpg
>>> >>> >>> >> 03620 53
>>> >>> >>> >> [root@server1]#
>>> >>> >>> >>
>>> >>> >>> >> If I flush the cache as shown below the checksum returns as
>>> >>> >>> >> expected
>>> >>> >>> >> and the
>>> >>> >>> >> web server serves up valid content.
>>> >>> >>> >>
>>> >>> >>> >> [root@server2]# echo 3 > /proc/sys/vm/drop_caches
>>> >>> >>> >> [root@server2]# sum
>>> >>> >>> >> /cephfs/webdata/static/456/JHL/66448H-755h.jpg
>>> >>> >>> >> 03620 53
>>> >>> >>> >>
>>> >>> >>> >> After some time typically less than 1hr the issue repeats, It
>>> >>> >>> >> seems to
>>> >>> >>> >> not
>>> >>> >>> >> repeat if I take any one of the servers out of the LB and only
>>> >>> >>> >> serve
>>> >>> >>> >> requests from one of the servers.
>>> >>> >>> >>
>>> >>> >>> >> I may try and use the FUSE client has has a mount option
>>> >>> >>> >> direct_io
>>> >>> >>> >> that
>>> >>> >>> >> looks to disable page cache.
>>> >>> >>> >>
>>> >>> >>> >> I have been hunting in the ML and tracker but could not see
>>> >>> >>> >> anything
>>> >>> >>> >> really
>>> >>> >>> >> close to this issue, Any input or feedback on similar
>>> >>> >>> >> experiences
>>> >>> >>> >> is
>>> >>> >>> >> welcome.
>>> >>> >>> >>
>>> >>> >>> >> Thanks
>>> >>> >>> >>
>>> >>> >>> >>
>>> >>> >>> >> _______________________________________________
>>> >>> >>> >> ceph-users mailing list
>>> >>> >>> >> ceph-users@xxxxxxxxxxxxxx
>>> >>> >>> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph. com
>>> >>> >>> >>
>>> >>> >>> > _______________________________________________
>>> >>> >>> > ceph-users mailing list
>>> >>> >>> > ceph-users@xxxxxxxxxxxxxx
>>> >>> >>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph. com
>>> >>> >>
>>> >>> >>
>>> >>> >
>>> >>
>>> >>
>>> >
>>
>>
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
