For the record: I've created issue #12671 to improve our memory management in this type of situation. John http://tracker.ceph.com/issues/12671 On Tue, Aug 11, 2015 at 10:25 PM, John Spray <jspray@xxxxxxxxxx> wrote: > On Tue, Aug 11, 2015 at 6:23 PM, Bob Ababurko <bob@xxxxxxxxxxxx> wrote: >> Here is the backtrace from the core dump. >> >> (gdb) bt >> #0 0x00007f71f5404ffb in raise () from /lib64/libpthread.so.0 >> #1 0x000000000087065d in reraise_fatal (signum=6) at >> global/signal_handler.cc:59 >> #2 handle_fatal_signal (signum=6) at global/signal_handler.cc:109 >> #3 <signal handler called> >> #4 0x00007f71f40235d7 in raise () from /lib64/libc.so.6 >> #5 0x00007f71f4024cc8 in abort () from /lib64/libc.so.6 >> #6 0x00007f71f49279b5 in __gnu_cxx::__verbose_terminate_handler() () from >> /lib64/libstdc++.so.6 >> #7 0x00007f71f4925926 in ?? () from /lib64/libstdc++.so.6 >> #8 0x00007f71f4925953 in std::terminate() () from /lib64/libstdc++.so.6 >> #9 0x00007f71f4925b73 in __cxa_throw () from /lib64/libstdc++.so.6 >> #10 0x000000000077d0fc in operator new (num_bytes=2408) at mds/CInode.h:120 >> Python Exception <type 'exceptions.IndexError'> list index out of range: >> #11 CDir::_omap_fetched (this=0x90af04f8, hdrbl=..., omap=std::map with >> 65536 elements, want_dn="", r=<optimized out>) at mds/CDir.cc:1700 >> #12 0x00000000007d7d44 in complete (r=0, this=0x502b000) at >> include/Context.h:65 >> #13 MDSIOContextBase::complete (this=0x502b000, r=0) at mds/MDSContext.cc:59 >> #14 0x0000000000894818 in Finisher::finisher_thread_entry (this=0x5108698) >> at common/Finisher.cc:59 >> #15 0x00007f71f53fddf5 in start_thread () from /lib64/libpthread.so.0 >> #16 0x00007f71f40e41ad in clone () from /lib64/libc.so.6 > > If we believe the line numbers here, then it's a malloc failure. Are > you running out of memory? > > The MDS is loading a bunch of these 64k file directories (presumably a > characteristic of your workload), and ending up with an unusually > large number of inodes in cache (this is all happening during the > "rejoin" phase so no trimming of the cache is done and we merrily > exceed the default mds_cache_size limit of 100k inodes). > > The thing triggering the load of the dirs is clients replaying > requests that refer to inodes by inode number, and the MDS's procedure > for handling that involves fully loading the relevant dirs. That > might be something we can improve, it doesn't seem obviously necessary > to load all the dentrys in a dirfrag during this phase. > > Anyway, you can hopefully recover from this state by forcibly > unmounting your clients. Since you're using the kernel client it may > be easiest to hard reset the client boxes. When you next restart your > MDS, the clients won't be present, so the MDS will be able to make it > all the way up without trying to load a bunch of directory fragments. > If you've got some more RAM for the MDS box that wouldn't hurt either. > > One of the less well tested (but relevant here) features we have is > directory fragmentation, where large dirs like these are internally > split up (partly to avoid memory management issues like this). It > might be a risky business on a system that you've already got real > data on, but once your MDS is back up and running you can try enabling > the mds_bal_frag setting. > > This is not a use case we have particularly strong coverage of in our > automated tests, so thanks for your experimentation and persistence. > > John > >> >> I have also gotten a log file w / debug mds = 20. It was 1.2GB, so I >> bzip2'd it w max compression and got it down to 75MB. I wasn't sure where >> to upload it so if there is a better place to put it, please let me know. >> >> https://mega.nz/#!5V4z3A7K!0METjVs5t3DAQAts8_TYXWrLh2FhGHcb7oC4uuhr2T8 >> >> thanks, >> Bob >> >> >> On Mon, Aug 10, 2015 at 8:05 PM, Yan, Zheng <ukernel@xxxxxxxxx> wrote: >>> >>> On Tue, Aug 11, 2015 at 9:21 AM, Bob Ababurko <bob@xxxxxxxxxxxx> wrote: >>> > I had a dual mds server configuration and have been copying data via >>> > cephfs >>> > kernel module to my cluster for the past 3 weeks and just had a MDS >>> > crash >>> > halting all IO. Leading up to the crash, I ran a test dd that increased >>> > the >>> > throughput by about 2x and stopped it but about 10 minutes later, the >>> > MDS >>> > server crashed and did not fail over to the standby properly. I have >>> > using >>> > an active/standby mds configuration but neither of the mds servers will >>> > stay >>> > running at this point and crash after starting them. >>> > >>> > [bababurko@cephmon01 ~]$ sudo ceph -s >>> > cluster f25cb23f-2293-4682-bad2-4b0d8ad10e79 >>> > health HEALTH_WARN >>> > mds cluster is degraded >>> > mds cephmds02 is laggy >>> > noscrub,nodeep-scrub flag(s) set >>> > monmap e1: 3 mons at >>> > >>> > {cephmon01=10.15.24.71:6789/0,cephmon02=10.15.24.80:6789/0,cephmon03=10.15.24.135:6789/0} >>> > election epoch 4, quorum 0,1,2 cephmon01,cephmon02,cephmon03 >>> > mdsmap e2760: 1/1/1 up {0=cephmds02=up:rejoin(laggy or crashed)} >>> > osdmap e324: 30 osds: 30 up, 30 in >>> > flags noscrub,nodeep-scrub >>> > pgmap v1555346: 2112 pgs, 3 pools, 4993 GB data, 246 Mobjects >>> > 14051 GB used, 13880 GB / 27931 GB avail >>> > 2112 active+clean >>> > >>> > >>> > I am not sure what information is relevant so I will try to cover what I >>> > think is relevant based on posts I have read through: >>> > >>> > Cluster: >>> > running ceph-0.94.1 on CenttOS 7.1 >>> > [root@mdstest02 bababurko]$ uname -r >>> > 3.10.0-229.el7.x86_64 >>> > >>> > Here is my ceph-mds log with 'debug objector = 10' : >>> > >>> > >>> > https://www.zerobin.net/?179a6789dfc9eb86#AHAS3YEkpHTj6CSQg8u4hk+jHBasejQNLDc9/KYkYVQ= >>> >>> >>> could you use gdb to check where the crash happened. (gdb >>> /usr/local/bin/ceph-mds /core.xxxxx. maybe you need re-compile mds >>> with debuginfo) >>> >>> Yan, Zheng >>> >>> > >>> > cat /sys/kernel/debug/ceph/*/mdsc output: >>> > >>> > >>> > https://www.zerobin.net/?ed238ce77b20583d#CK7Yt6yC1VgHfDee7y/CGkFh5bfyLkhwZB6i5R6N/8g= >>> > >>> > ceph.conf : >>> > >>> > >>> > https://www.zerobin.net/?62a125349aa43c92#5VH3XRR4P7zjhBHNWmTHrFYmwE0TZEig6r2EU6X1q/U= >>> > >>> > I have copied almost 5TB of small files to this cluster which has taken >>> > the >>> > better part of three weeks, so I am really hoping that there is a way to >>> > recover from this. This is ourPOC cluster >>> > >>> > I'm sure I have missed something relevant as i'm just getting my mind >>> > back >>> > after nearly losing it, so feel free to ask for anything to assist. >>> > >>> > Any help would be greatly appreciated. >>> > >>> > thanks, >>> > Bob >>> > >>> > >>> > >>> > _______________________________________________ >>> > ceph-users mailing list >>> > ceph-users@xxxxxxxxxxxxxx >>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> > >> >> >> >> _______________________________________________ >> ceph-users mailing list >> ceph-users@xxxxxxxxxxxxxx >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |