Re: OSD commits suicide

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Tue, Nov 18, 2014 at 10:04 PM, Craig Lewis <clewis@xxxxxxxxxxxxxxxxxx> wrote:
> That would probably have helped.  The XFS deadlocks would only occur when
> there was relatively little free memory.  Kernel 3.18 is supposed to have a
> fix for that, but I haven't tried it yet.
>
> Looking at my actual usage, I don't even need 64k inodes.  64k inodes should
> make things a bit faster when you have a large number of files in a
> directory.  Ceph will automatically split directories with too many files
> into multiple sub-directories, so it's kinda pointless.
>
> I may try the experiment again, but probably not.  It took several weeks to
> reformat all of the OSDS.  Even on a single node, it takes 4-5 days to
> drain, format, and backfill.  That was months ago, and I'm still dealing
> with the side effects.  I'm not eager to try again.
>
>
> On Mon, Nov 17, 2014 at 2:04 PM, Andrey Korolyov <andrey@xxxxxxx> wrote:
>>
>> On Tue, Nov 18, 2014 at 12:54 AM, Craig Lewis <clewis@xxxxxxxxxxxxxxxxxx>
>> wrote:
>> > I did have a problem in my secondary cluster that sounds similar to
>> > yours.
>> > I was using XFS, and traced my problem back to 64 kB inodes (osd mkfs
>> > options xfs = -i size=64k).   This showed up with a lot of "XFS:
>> > possible
>> > memory allocation deadlock in kmem_alloc" in the kernel logs.  I was
>> > able to
>> > keep things limping along by flushing the cache frequently, but I
>> > eventually
>> > re-formatted every OSD to get rid of the 64k inodes.
>> >
>> > After I finished the reformat, I had problems because of deep-scrubbing.
>> > While reformatting, I disabled deep-scrubbing.  Once I re-enabled it,
>> > Ceph
>> > wanted to deep-scrub the whole cluster, and sometimes 90% of my OSDs
>> > would
>> > be doing a deep-scrub.  I'm manually deep-scrubbing now, trying to
>> > spread
>> > out the schedule a bit.  Once this finishes in a few day, I should be
>> > able
>> > to re-enable deep-scrubbing and keep my HEALTH_OK.
>> >
>> >
>>
>> Would you mind to check suggestions by following mine hints or hints
>> from mentioned URLs from there
>> http://marc.info/?l=linux-mm&m=141607712831090&w=2 with 64k again? As
>> for me, I am not observing lock loop after setting min_free_kbytes for
>> a half of gigabyte per OSD. Even if your locks has a different nature,
>> it may be worthy to try anyway.
>
>

Thanks, I perfectly understand this. But, if you have low enough
OSD/node ratio, it can be possible to check the problem at a node
scale. By the way, I do not see real reasons behind using lower
allocsize NOT for object storage-designed cluster.
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux