I was testing a little bit more and decided to run the cephfs-journal-tool
I ran across some errors
$ cephfs-journal-tool journal inspect
2015-03-03 14:18:54.453981 7f8e29f86780 -1 Bad entry start ptr (0x2aeb0000f6) at 0x2aeb32279b
2015-03-03 14:18:54.539060 7f8e29f86780 -1 Bad entry start ptr (0x2aeb000733) at 0x2aeb322dd8
2015-03-03 14:18:54.584539 7f8e29f86780 -1 Bad entry start ptr (0x2aeb000d70) at 0x2aeb323415
2015-03-03 14:18:54.669991 7f8e29f86780 -1 Bad entry start ptr (0x2aeb0013ad) at 0x2aeb323a52
2015-03-03 14:18:54.707724 7f8e29f86780 -1 Bad entry start ptr (0x2aeb0019ea) at 0x2aeb32408f
Overall journal integrity: DAMAGED
Corrupt regions:
0x2aeb3226a5-2aeb32279b
0x2aeb32279b-2aeb322dd8
0x2aeb322dd8-2aeb323415
0x2aeb323415-2aeb323a52
0x2aeb323a52-2aeb32408f
0x2aeb32408f-2aeb3246cc
$ cephfs-journal-tool header get
{ "magic": "ceph fs volume v011",
"write_pos": 184430420380,
"expire_pos": 184389995327,
"trimmed_pos": 184389992448,
"stream_format": 1,
"layout": { "stripe_unit": 4194304,
"stripe_count": 4194304,
"object_size": 4194304,
"cas_hash": 4194304,
"object_stripe_unit": 4194304,
"pg_pool": 4194304}}
$ cephfs-journal-tool event get summary
2015-03-03 14:32:50.102863 7f47c3006780 -1 Bad entry start ptr (0x2aee8000e6) at 0x2aee800c25
2015-03-03 14:32:50.242576 7f47c3006780 -1 Bad entry start ptr (0x2aee800b3f) at 0x2aee80167e
2015-03-03 14:32:50.486354 7f47c3006780 -1 Bad entry start ptr (0x2aee800e4f) at 0x2aee80198e
2015-03-03 14:32:50.577443 7f47c3006780 -1 Bad entry start ptr (0x2aee801f65) at 0x2aee802aa4
Events by type:
<no output here>
On Tue, Mar 3, 2015 at 12:01 PM Scottix <scottix@xxxxxxxxx> wrote:
I did a bit more testing.
1. I tried on a newer kernel and was not able to recreate the problem, maybe it is that kernel bug you mentioned. Although its not an exact replica of the load.2. I haven't tried the debug yet since I have to wait for the right moment.One thing I realized and maybe it is not an issue is we are using a symlink to a folder in the ceph mount.ceph-fuse on /mnt/ceph type fuse.ceph-fuse (rw,nosuid,nodev,noatime,user_id=0,group_id=0,default_permissions,allow_other)lrwxrwxrwx 1 root root metadata -> /mnt/ceph/DataCenter/metadataNot sure if that would create any issues.Anyway we are going to update the machine soon so, I can report if we keep having the issue.Thanks for your support,ScottOn Mon, Mar 2, 2015 at 4:07 PM Scottix <scottix@xxxxxxxxx> wrote:I'll try the following things and report back to you.1. I can get a new kernel on another machine and mount to the CephFS and see if I get the following errors.2. I'll run the debug and see if anything comes up.I'll report back to you when I can do these things.Thanks,ScottieOn Mon, Mar 2, 2015 at 4:04 PM Gregory Farnum <greg@xxxxxxxxxxx> wrote:I bet it's that permission issue combined with a minor bug in FUSE on
that kernel, or maybe in the ceph-fuse code (but I've not seen it
reported before, so I kind of doubt it). If you run ceph-fuse with
"debug client = 20" it will output (a whole lot of) logging to the
client's log file and you could see what requests are getting
processed by the Ceph code and how it's responding. That might let you
narrow things down. It's certainly not any kind of timeout.
-Greg
On Mon, Mar 2, 2015 at 3:57 PM, Scottix <scottix@xxxxxxxxx> wrote:
> 3 Ceph servers on Ubuntu 12.04.5 - kernel 3.13.0-29-generic
>
> We have an old server that we compiled the ceph-fuse client on
> Suse11.4 - kernel 2.6.37.6-0.11
> This is the only mount we have right now.
>
> We don't have any problems reading the files and the directory shows full
> 775 permissions and doing a second ls fixes the problem.
>
> On Mon, Mar 2, 2015 at 3:51 PM Bill Sanders <billysanders@xxxxxxxxx> wrote:
>>
>> Forgive me if this is unhelpful, but could it be something to do with
>> permissions of the directory and not Ceph at all?
>>
>> http://superuser.com/a/528467
>>
>> Bill
>>
>> On Mon, Mar 2, 2015 at 3:47 PM, Gregory Farnum <greg@xxxxxxxxxxx> wrote:
>>>
>>> On Mon, Mar 2, 2015 at 3:39 PM, Scottix <scottix@xxxxxxxxx> wrote:
>>> > We have a file system running CephFS and for a while we had this issue
>>> > when
>>> > doing an ls -la we get question marks in the response.
>>> >
>>> > -rw-r--r-- 1 wwwrun root 14761 Feb 9 16:06
>>> > data.2015-02-08_00-00-00.csv.bz2
>>> > -????????? ? ? ? ? ?
>>> > data.2015-02-09_00-00-00.csv.bz2
>>> >
>>> > If we do another directory listing it show up fine.
>>> >
>>> > -rw-r--r-- 1 wwwrun root 14761 Feb 9 16:06
>>> > data.2015-02-08_00-00-00.csv.bz2
>>> > -rw-r--r-- 1 wwwrun root 13675 Feb 10 15:21
>>> > data.2015-02-09_00-00-00.csv.bz2
>>> >
>>> > It hasn't been a problem but just wanted to see if this is an issue,
>>> > could
>>> > the attributes be timing out? We do have a lot of files in the
>>> > filesystem so
>>> > that could be a possible bottleneck.
>>>
>>> Huh, that's not something I've seen before. Are the systems you're
>>> doing this on the same? What distro and kernel version? Is it reliably
>>> one of them showing the question marks, or does it jump between
>>> systems?
>>> -Greg
>>>
>>> >
>>> > We are using the ceph-fuse mount.
>>> > ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578)
>>> > We are planning to do the update soon to 87.1
>>> >
>>> > Thanks
>>> > Scottie
>>> >
>>> >
>>> > _______________________________________________
>>> > ceph-users mailing list
>>> > ceph-users@xxxxxxxxxxxxxx
>>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>> >
>>> _______________________________________________
>>> ceph-users mailing list
>>> ceph-users@xxxxxxxxxxxxxx
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
>
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
