thanks John Tuite Corporate Global Infrastructure Services Pittsburgh Manager, Information Technology Global Hosting Services Thermo Fisher Scientific 600 Business Center Drive Pittsburgh, Pennsylvania 15205 Office 412-490-7292 Mobile 412-897-3401 Fax 412-490-9401 john.tuite at thermofisher.com http://www.thermofisher.com -----Original Message----- From: Tregaron Bayly [mailto:tbayly@xxxxxxxxxxxx] Sent: Friday, July 11, 2014 1:53 PM To: Tuite, John E. Cc: ceph-users at lists.ceph.com Subject: Re: ceph mount not working anymore "Ceph is short of Cephalopod, a class of mollusks that includes the octopus, squid, and cuttlefish" http://community.redhat.com/blog/2014/06/ceph-turns-10-a-look-back/ On Fri, 2014-07-11 at 10:48 -0700, Tuite, John E. wrote: > Is Ceph an acronym? If yes, what? > > > > John Tuite > > Corporate Global Infrastructure Services Pittsburgh > > Manager, Information Technology > Global Hosting Services > > Thermo Fisher Scientific > > 600 Business Center Drive > > Pittsburgh, Pennsylvania 15205 > > Office 412-490-7292 > > Mobile 412-897-3401 > > Fax 412-490-9401 > > john.tuite at thermofisher.com > > http://www.thermofisher.com > > > > From: ceph-users [mailto:ceph-users-bounces at lists.ceph.com] On Behalf > Of Joshua McClintock > Sent: Friday, July 11, 2014 1:44 PM > To: Alfredo Deza > Cc: ceph-users at lists.ceph.com > Subject: Re: [ceph-users] ceph mount not working anymore > > > > > Hello Alfredo, isn't this what the 'ceph-release-1-0.el6.noarch' > package is for in my rpm -qa list? Here are the yum repo files I have > in /etc/yum.repos.d. I don't see any priorities in the ceph one which > is where libcephfs1 comes from I think. I tried to 'yum reinstall > ceph-release', but the file still doesn't include any priority lines. > > > > > > > > ceph.repo > > > [Ceph] > > name=Ceph packages for $basearch > > baseurl=http://ceph.com/rpm-firefly/el6/$basearch > > enabled=1 > > gpgcheck=1 > > type=rpm-md > > gpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.as > c > > > > [Ceph-noarch] > > name=Ceph noarch packages > > baseurl=http://ceph.com/rpm-firefly/el6/noarch > > enabled=1 > > gpgcheck=1 > > type=rpm-md > > gpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.as > c > > > > [ceph-source] > > name=Ceph source packages > > baseurl=http://ceph.com/rpm-firefly/el6/SRPMS > > enabled=1 > > gpgcheck=1 > > type=rpm-md > > gpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.as > c > > > > Here is the one for apache > > [apache2-ceph-noarch] > > name=Apache noarch packages for Ceph > > baseurl=http://gitbuilder.ceph.com/apache2-rpm-centos6-x86_64-basic/re > f/master > > enabled=1 > > priority=2 > > gpgcheck=1 > > type=rpm-md > > gpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild. > asc > > > > [apache2-ceph-source] > > name=Apache source packages for Ceph > > baseurl=http://gitbuilder.ceph.com/apache2-rpm-centos6-x86_64-basic/re > f/master > > enabled=0 > > priority=2 > > gpgcheck=1 > > type=rpm-md > > gpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild. > asc > > > > Here's the one for fastcgi > > [fastcgi-ceph-basearch] > > name=FastCGI basearch packages for Ceph > > baseurl=http://gitbuilder.ceph.com/mod_fastcgi-rpm-centos6-x86_64-basi > c/ref/master > > enabled=1 > > priority=2 > > gpgcheck=1 > > type=rpm-md > > gpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild. > asc > > > > [fastcgi-ceph-noarch] > > name=FastCGI noarch packages for Ceph > > baseurl=http://gitbuilder.ceph.com/mod_fastcgi-rpm-centos6-x86_64-basi > c/ref/master > > enabled=1 > > priority=2 > > gpgcheck=1 > > type=rpm-md > > gpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild. > asc > > > > [fastcgi-ceph-source] > > name=FastCGI source packages for Ceph > > baseurl=http://gitbuilder.ceph.com/mod_fastcgi-rpm-centos6-x86_64-basi > c/ref/master > > enabled=0 > > priority=2 > > gpgcheck=1 > > type=rpm-md > > gpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild. > asc > > > > > On Fri, Jul 11, 2014 at 5:44 AM, Alfredo Deza > <alfredo.deza at inktank.com> wrote: > > Joshua it looks like you got Ceph from EPEL (that version has the '-2' > slapped on it). And it is why you are seeing this for ceph: > > ceph-0.80.1-2.el6.x86_64 > > And this for others: > > libcephfs1-0.80.1-0.el6.x86_64 > > Make sure that you do get Ceph from our repos. Newer versions of > ceph-deploy fix this by installing the priority plugin and making sure > the ceph.repo file will have higher priority than EPEL. > > > > > On Fri, Jul 11, 2014 at 1:31 AM, Joshua McClintock > <joshua at gravityedge.com> wrote: > > [root at chefwks01 ~]# ceph --cluster us-west01 osd crush dump > > > > { "devices": [ > > > > { "id": 0, > > > > "name": "osd.0"}, > > > > { "id": 1, > > > > "name": "osd.1"}, > > > > { "id": 2, > > > > "name": "osd.2"}, > > > > { "id": 3, > > > > "name": "osd.3"}, > > > > { "id": 4, > > > > "name": "osd.4"}], > > > > "types": [ > > > > { "type_id": 0, > > > > "name": "osd"}, > > > > { "type_id": 1, > > > > "name": "host"}, > > > > { "type_id": 2, > > > > "name": "chassis"}, > > > > { "type_id": 3, > > > > "name": "rack"}, > > > > { "type_id": 4, > > > > "name": "row"}, > > > > { "type_id": 5, > > > > "name": "pdu"}, > > > > { "type_id": 6, > > > > "name": "pod"}, > > > > { "type_id": 7, > > > > "name": "room"}, > > > > { "type_id": 8, > > > > "name": "datacenter"}, > > > > { "type_id": 9, > > > > "name": "region"}, > > > > { "type_id": 10, > > > > "name": "root"}], > > > > "buckets": [ > > > > { "id": -1, > > > > "name": "default", > > > > "type_id": 10, > > > > "type_name": "root", > > > > "weight": 147455, > > > > "alg": "straw", > > > > "hash": "rjenkins1", > > > > "items": [ > > > > { "id": -2, > > > > "weight": 29491, > > > > "pos": 0}, > > > > { "id": -3, > > > > "weight": 29491, > > > > "pos": 1}, > > > > { "id": -4, > > > > "weight": 29491, > > > > "pos": 2}, > > > > { "id": -5, > > > > "weight": 29491, > > > > "pos": 3}, > > > > { "id": -6, > > > > "weight": 29491, > > > > "pos": 4}]}, > > > > { "id": -2, > > > > "name": "ceph-node20", > > > > "type_id": 1, > > > > "type_name": "host", > > > > "weight": 29491, > > > > "alg": "straw", > > > > "hash": "rjenkins1", > > > > "items": [ > > > > { "id": 0, > > > > "weight": 29491, > > > > "pos": 0}]}, > > > > { "id": -3, > > > > "name": "ceph-node22", > > > > "type_id": 1, > > > > "type_name": "host", > > > > "weight": 29491, > > > > "alg": "straw", > > > > "hash": "rjenkins1", > > > > "items": [ > > > > { "id": 2, > > > > "weight": 29491, > > > > "pos": 0}]}, > > > > { "id": -4, > > > > "name": "ceph-node24", > > > > "type_id": 1, > > > > "type_name": "host", > > > > "weight": 29491, > > > > "alg": "straw", > > > > "hash": "rjenkins1", > > > > "items": [ > > > > { "id": 4, > > > > "weight": 29491, > > > > "pos": 0}]}, > > > > { "id": -5, > > > > "name": "ceph-node21", > > > > "type_id": 1, > > > > "type_name": "host", > > > > "weight": 29491, > > > > "alg": "straw", > > > > "hash": "rjenkins1", > > > > "items": [ > > > > { "id": 1, > > > > "weight": 29491, > > > > "pos": 0}]}, > > > > { "id": -6, > > > > "name": "ceph-node23", > > > > "type_id": 1, > > > > "type_name": "host", > > > > "weight": 29491, > > > > "alg": "straw", > > > > "hash": "rjenkins1", > > > > "items": [ > > > > { "id": 3, > > > > "weight": 29491, > > > > "pos": 0}]}], > > > > "rules": [ > > > > { "rule_id": 0, > > > > "rule_name": "replicated_ruleset", > > > > "ruleset": 0, > > > > "type": 1, > > > > "min_size": 1, > > > > "max_size": 10, > > > > "steps": [ > > > > { "op": "take", > > > > "item": -1, > > > > "item_name": "default"}, > > > > { "op": "chooseleaf_firstn", > > > > "num": 0, > > > > "type": "host"}, > > > > { "op": "emit"}]}, > > > > { "rule_id": 1, > > > > "rule_name": "erasure-code", > > > > "ruleset": 1, > > > > "type": 3, > > > > "min_size": 3, > > > > "max_size": 20, > > > > "steps": [ > > > > { "op": "set_chooseleaf_tries", > > > > "num": 5}, > > > > { "op": "take", > > > > "item": -1, > > > > "item_name": "default"}, > > > > { "op": "chooseleaf_indep", > > > > "num": 0, > > > > "type": "host"}, > > > > { "op": "emit"}]}, > > > > { "rule_id": 2, > > > > "rule_name": "ecpool", > > > > "ruleset": 2, > > > > "type": 3, > > > > "min_size": 3, > > > > "max_size": 20, > > > > "steps": [ > > > > { "op": "set_chooseleaf_tries", > > > > "num": 5}, > > > > { "op": "take", > > > > "item": -1, > > > > "item_name": "default"}, > > > > { "op": "choose_indep", > > > > "num": 0, > > > > "type": "osd"}, > > > > { "op": "emit"}]}], > > > > "tunables": { "choose_local_tries": 0, > > > > "choose_local_fallback_tries": 0, > > > > "choose_total_tries": 50, > > > > "chooseleaf_descend_once": 1, > > > > "profile": "bobtail", > > > > "optimal_tunables": 0, > > > > "legacy_tunables": 0, > > > > "require_feature_tunables": 1, > > > > "require_feature_tunables2": 1}} > > > > > > > > On Thu, Jul 10, 2014 at 8:16 PM, Sage Weil <sweil at redhat.com> wrote: > >> > >> That is CEPH_FEATURE_CRUSH_V2. Can you attach teh output of > >> > >> ceph osd crush dump > >> > >> Thanks! > >> sage > >> > >> > >> On Thu, 10 Jul 2014, Joshua McClintock wrote: > >> > >> > Yes, I change some of the mount options on my osds (xfs mount > options), > >> > but > >> > I think this may be the answer from dmesg, sorta looks like a > version > >> > mismatch: > >> > > >> > libceph: loaded (mon/osd proto 15/24) > >> > > >> > ceph: loaded (mds proto 32) > >> > > >> > libceph: mon0 192.168.0.14:6789 feature set mismatch, my 4a042aca > < > >> > server's > >> > 104a042aca, missing 1000000000 > >> > > >> > libceph: mon0 192.168.0.14:6789 socket error on read > >> > > >> > libceph: mon2 192.168.0.16:6789 feature set mismatch, my 4a042aca > < > >> > server's > >> > 104a042aca, missing 1000000000 > >> > > >> > libceph: mon2 192.168.0.16:6789 socket error on read > >> > > >> > libceph: mon1 192.168.0.15:6789 feature set mismatch, my 4a042aca > < > >> > server's > >> > 104a042aca, missing 1000000000 > >> > > >> > libceph: mon1 192.168.0.15:6789 socket error on read > >> > > >> > libceph: mon0 192.168.0.14:6789 feature set mismatch, my 4a042aca > < > >> > server's > >> > 104a042aca, missing 1000000000 > >> > > >> > libceph: mon0 192.168.0.14:6789 socket error on read > >> > > >> > libceph: mon2 192.168.0.16:6789 feature set mismatch, my 4a042aca > < > >> > server's > >> > 104a042aca, missing 1000000000 > >> > > >> > libceph: mon2 192.168.0.16:6789 socket error on read > >> > > >> > libceph: mon1 192.168.0.15:6789 feature set mismatch, my 4a042aca > < > >> > server's > >> > 104a042aca, missing 1000000000 > >> > > >> > libceph: mon1 192.168.0.15:6789 socket error on read > >> > > >> > > >> > I maybe I didn't update as well as I thought it did. I did hit > every > >> > mon, > >> > but I remember I couldn't upgrade to the new 'ceph' package > because it > >> > conflicted with 'python-ceph', so I uninstalled it (python-ceph), > and > >> > then > >> > upgraded to .80.1-2. Maybe there's a subcomponent I missed? > >> > > >> > > >> > Here's rpm -qa from the client: > >> > > >> > > >> > [root at chefwks01 ~]# rpm -qa|grep ceph > >> > > >> > ceph-deploy-1.5.2-0.noarch > >> > > >> > ceph-release-1-0.el6.noarch > >> > > >> > ceph-0.80.1-2.el6.x86_64 > >> > > >> > libcephfs1-0.80.1-0.el6.x86_64 > >> > > >> > > >> > Here's rpm -qa from the mons: > >> > > >> > > >> > [root at ceph-mon01 ~]# rpm -qa|grep ceph > >> > > >> > ceph-0.80.1-2.el6.x86_64 > >> > > >> > ceph-release-1-0.el6.noarch > >> > > >> > libcephfs1-0.80.1-0.el6.x86_64 > >> > > >> > [root at ceph-mon01 ~]# > >> > > >> > > >> > [root at ceph-mon02 ~]# rpm -qa|grep ceph > >> > > >> > libcephfs1-0.80.1-0.el6.x86_64 > >> > > >> > ceph-0.80.1-2.el6.x86_64 > >> > > >> > ceph-release-1-0.el6.noarch > >> > > >> > [root at ceph-mon02 ~]# > >> > > >> > > >> > [root at ceph-mon03 ~]# rpm -qa|grep ceph > >> > > >> > libcephfs1-0.80.1-0.el6.x86_64 > >> > > >> > ceph-0.80.1-2.el6.x86_64 > >> > > >> > ceph-release-1-0.el6.noarch > >> > > >> > [root at ceph-mon03 ~]# > >> > > >> > > >> > Joshua > >> > > >> > > >> > > >> > On Thu, Jul 10, 2014 at 6:04 PM, Sage Weil <sweil at redhat.com> > wrote: > >> > Have you made any other changes after the upgrade? (Like > >> > adjusting > >> > tunables, or creating EC pools?) > >> > > >> > See if there is anything in 'dmesg' output. > >> > > >> > sage > >> > > >> > On Thu, 10 Jul 2014, Joshua McClintock wrote: > >> > > >> > > I upgraded my cluster to .80.1-2 (CentOS). My mount > command > >> > just freezes > >> > > and outputs an error: > >> > > > >> > > mount.ceph > 192.168.0.14,192.168.0.15,192.168.0.16:/ /us-west01 > >> > -o > >> > > name=chefwks01,secret=`ceph-authtool -p -n client.admin > >> > > /etc/ceph/us-west01.client.admin.keyring` > >> > > > >> > > mount error 5 = Input/output error > >> > > > >> > > > >> > > Here's the output from 'ceph -s' > >> > > > >> > > > >> > > cluster xxxxxxxxxxxxxxxxxxxxxx > >> > > > >> > > health HEALTH_OK > >> > > > >> > > monmap e1: 3 > >> > > monsat{ceph-mon01=192.168.0.14:6789/0,ceph-mon02=192.168.0.15:6789/0,c > eph-mon03 > >> > =1 > >> > > 92.168.0.16:6789/0}, election epoch 88, quorum 0,1,2 > >> > > ceph-mon01,ceph-mon02,ceph-mon03 > >> > > > >> > > mdsmap e26: 1/1/1 up {0=0=up:active} > >> > > > >> > > osdmap e1371: 5 osds: 5 up, 5 in > >> > > > >> > > pgmap v49431: 192 pgs, 3 pools, 135 GB data, 34733 > >> > objects > >> > > > >> > > 406 GB used, 1874 GB / 2281 GB avail > >> > > > >> > > 192 active+clean > >> > > > >> > > > >> > > I can see some packets being exchanged between the client > and > >> > the mon, but > >> > > it's a pretty short exchange. > >> > > > >> > > Any ideas where to look next? > >> > > > >> > > Joshua > >> > > > >> > > > >> > > > >> > > >> > > >> > > >> > > > > > > > > > > > _______________________________________________ > > ceph-users mailing list > > ceph-users at lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > > > > _______________________________________________ > ceph-users mailing list > ceph-users at lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |