I did a bit more testing.
1. I tried on a newer kernel and was not able to recreate the problem, maybe it is that kernel bug you mentioned. Although its not an exact replica of the load.
1. I tried on a newer kernel and was not able to recreate the problem, maybe it is that kernel bug you mentioned. Although its not an exact replica of the load.
2. I haven't tried the debug yet since I have to wait for the right moment.
One thing I realized and maybe it is not an issue is we are using a symlink to a folder in the ceph mount.
ceph-fuse on /mnt/ceph type fuse.ceph-fuse (rw,nosuid,nodev,noatime,user_id=0,group_id=0,default_permissions,allow_other)
lrwxrwxrwx 1 root root metadata -> /mnt/ceph/DataCenter/metadata
Not sure if that would create any issues.
Anyway we are going to update the machine soon so, I can report if we keep having the issue.
Thanks for your support,
Scott
On Mon, Mar 2, 2015 at 4:07 PM Scottix <scottix@xxxxxxxxx> wrote:
I'll try the following things and report back to you.1. I can get a new kernel on another machine and mount to the CephFS and see if I get the following errors.2. I'll run the debug and see if anything comes up.I'll report back to you when I can do these things.Thanks,ScottieOn Mon, Mar 2, 2015 at 4:04 PM Gregory Farnum <greg@xxxxxxxxxxx> wrote:I bet it's that permission issue combined with a minor bug in FUSE on
that kernel, or maybe in the ceph-fuse code (but I've not seen it
reported before, so I kind of doubt it). If you run ceph-fuse with
"debug client = 20" it will output (a whole lot of) logging to the
client's log file and you could see what requests are getting
processed by the Ceph code and how it's responding. That might let you
narrow things down. It's certainly not any kind of timeout.
-Greg
On Mon, Mar 2, 2015 at 3:57 PM, Scottix <scottix@xxxxxxxxx> wrote:
> 3 Ceph servers on Ubuntu 12.04.5 - kernel 3.13.0-29-generic
>
> We have an old server that we compiled the ceph-fuse client on
> Suse11.4 - kernel 2.6.37.6-0.11
> This is the only mount we have right now.
>
> We don't have any problems reading the files and the directory shows full
> 775 permissions and doing a second ls fixes the problem.
>
> On Mon, Mar 2, 2015 at 3:51 PM Bill Sanders <billysanders@xxxxxxxxx> wrote:
>>
>> Forgive me if this is unhelpful, but could it be something to do with
>> permissions of the directory and not Ceph at all?
>>
>> http://superuser.com/a/528467
>>
>> Bill
>>
>> On Mon, Mar 2, 2015 at 3:47 PM, Gregory Farnum <greg@xxxxxxxxxxx> wrote:
>>>
>>> On Mon, Mar 2, 2015 at 3:39 PM, Scottix <scottix@xxxxxxxxx> wrote:
>>> > We have a file system running CephFS and for a while we had this issue
>>> > when
>>> > doing an ls -la we get question marks in the response.
>>> >
>>> > -rw-r--r-- 1 wwwrun root 14761 Feb 9 16:06
>>> > data.2015-02-08_00-00-00.csv.bz2
>>> > -????????? ? ? ? ? ?
>>> > data.2015-02-09_00-00-00.csv.bz2
>>> >
>>> > If we do another directory listing it show up fine.
>>> >
>>> > -rw-r--r-- 1 wwwrun root 14761 Feb 9 16:06
>>> > data.2015-02-08_00-00-00.csv.bz2
>>> > -rw-r--r-- 1 wwwrun root 13675 Feb 10 15:21
>>> > data.2015-02-09_00-00-00.csv.bz2
>>> >
>>> > It hasn't been a problem but just wanted to see if this is an issue,
>>> > could
>>> > the attributes be timing out? We do have a lot of files in the
>>> > filesystem so
>>> > that could be a possible bottleneck.
>>>
>>> Huh, that's not something I've seen before. Are the systems you're
>>> doing this on the same? What distro and kernel version? Is it reliably
>>> one of them showing the question marks, or does it jump between
>>> systems?
>>> -Greg
>>>
>>> >
>>> > We are using the ceph-fuse mount.
>>> > ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578)
>>> > We are planning to do the update soon to 87.1
>>> >
>>> > Thanks
>>> > Scottie
>>> >
>>> >
>>> > _______________________________________________
>>> > ceph-users mailing list
>>> > ceph-users@xxxxxxxxxxxxxx
>>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>> >
>>> _______________________________________________
>>> ceph-users mailing list
>>> ceph-users@xxxxxxxxxxxxxx
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
>
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
