On Thu, Apr 14, 2016 at 12:19 PM, Lomayani S. Laizer <lomlaizer@xxxxxxxxx> wrote: > Hello, > Upgraded the cluster but still seeing the same issue. Is the cluster not > recoverable? > > ceph --version > ceph version 10.1.2-64-ge657ecf (e657ecf8e437047b827aa89fb9c10be82643300c) > > root@mon-b:~# ceph -w > 2016-04-14 22:17:56.766169 7f5da3fff700 0 -- 10.10.200.3:0/1828342317 >> > 10.10.200.3:6789/0 pipe(0x7f5da8000c80 sd=4 :0 s=1 pgs=0 cs=0 l=1 > c=0x7f5da8001f90).fault > 2016-04-14 22:18:02.766859 7f5db8215700 0 -- 10.10.200.3:0/1828342317 >> > 10.10.200.2:6789/0 pipe(0x7f5da8007790 sd=3 :0 s=1 pgs=0 cs=0 l=1 > c=0x7f5da8002410).fault > 2016-04-14 22:18:05.767017 7f5da3fff700 0 -- 10.10.200.3:0/1828342317 >> > 10.10.200.3:6789/0 pipe(0x7f5da80051a0 sd=4 :0 s=1 pgs=0 cs=0 l=1 > c=0x7f5da8002bc0).fault Please check the state of the actual daemon — this just means that the ceph cli client couldn't set up a session with a monitor, which can happen for an infinite number of reasons. If the monitor has actually crashed again, please install the debug packages and start up the monitor with "debug mon = 20" and "debug mds = 20" in its config file, then post the log. If it hasn't crashed, you probably don't have a quorum running. You'll need to upgrade each of them to that gitbuilder version of the code for them to be happy. -Greg > > -- > Lomayani > > > On Thu, Apr 14, 2016 at 8:46 PM, Gregory Farnum <gfarnum@xxxxxxxxxx> wrote: >> >> Yep! This is fixed in the jewel and master branches now, but we're >> going to wait until the next rc (or final release!) to push official >> packages for it. >> >> In the meantime, you can install those from our gitbuilders following >> the instructions at >> >> http://docs.ceph.com/docs/master/install/get-packages/#add-ceph-development >> >> Other CephFS users may want to hold off until the next release happens >> — although I hope anybody using RCs is comfortable with dev packages >> when needed. :) >> -Greg >> >> On Thu, Apr 14, 2016 at 10:41 AM, Lomayani S. Laizer >> <lomlaizer@xxxxxxxxx> wrote: >> > Hello Gregory, >> > Thanks for your reply. I think am hitting the same bug. Below is the >> > link >> > for log just after an upgrade >> > >> > https://justpaste.it/ta16 >> > >> > -- >> > Lomayani >> > >> > On Thu, Apr 14, 2016 at 6:24 PM, Gregory Farnum <gfarnum@xxxxxxxxxx> >> > wrote: >> >> >> >> On Thu, Apr 14, 2016 at 7:05 AM, Lomayani S. Laizer >> >> <lomlaizer@xxxxxxxxx> >> >> wrote: >> >> > Hello, >> >> > I upgraded from 10.1.0 to 10.1.2 with ceph-deploy and my cluster is >> >> > down >> >> > now. getting below errors >> >> > >> >> > ceph -s >> >> > >> >> > 2016-04-14 17:04:58.909894 7f14686e4700 0 -- :/2590574876 >> >> >> > 10.10.200.4:6789/0 pipe(0x7f146405adf0 sd=3 :0 s=1 pgs=0 cs=0 l=1 >> >> > c=0x7f146405c0b0).fault >> >> > 2016-04-14 17:05:01.909949 7f14685e3700 0 -- :/2590574876 >> >> >> > 10.10.200.3:6789/0 pipe(0x7f1458000c80 sd=4 :0 s=1 pgs=0 cs=0 l=1 >> >> > c=0x7f1458001f90).fault >> >> > 2016-04-14 17:05:04.910416 7f14686e4700 0 -- :/2590574876 >> >> >> > 10.10.200.4:6789/0 pipe(0x7f1458005120 sd=4 :0 s=1 pgs=0 cs=0 l=1 >> >> > c=0x7f14580063e0).fault >> >> > 2016-04-14 17:05:07.910697 7f14685e3700 0 -- :/2590574876 >> >> >> > 10.10.200.2:6789/0 pipe(0x7f1458000c80 sd=4 :0 s=1 pgs=0 cs=0 l=1 >> >> > c=0x7f1458002410).fault >> >> >> >> Assuming your monitors aren't running and have crashed, can you get >> >> the backtrace out of their log files? >> >> >> >> We just discovered an issue with the new FSMap encoding in 10.1.2, if >> >> you had already run an rc and had a filesystem. Patch is building and >> >> being tested now. >> >> -Greg >> >> >> >> > >> >> > -- >> >> > Lomayani >> >> > >> >> > >> >> > _______________________________________________ >> >> > ceph-users mailing list >> >> > ceph-users@xxxxxxxxxxxxxx >> >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> > >> > >> > > > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |