Hello,
It appears that something is horribly wrong with the cluster itself. I
can't create or add any new osds to it at all.
On Mon, Aug 20, 2018 at 11:04 AM Daznis <daznis@xxxxxxxxx> wrote:
>
> Hello,
>
>
> Zapping the journal didn't help. I tried to create the journal after
> zapping it. Also failed. I'm not really sure why this happens.
>
> Looking at the monitor logs with 20/20 debug I'm seeing these errors:
>
> 2018-08-20 08:57:58.753 7f9d85934700 0 mon.mon02@1(peon) e4
> handle_command mon_command({"prefix": "osd crush set-device-class",
> "class": "ssd", "ids": ["48"]} v 0) v1
> 2018-08-20 08:57:58.753 7f9d85934700 20 is_capable service=osd
> command=osd crush set-device-class read write on cap allow profile osd
> 2018-08-20 08:57:58.753 7f9d85934700 20 allow so far , doing grant
> allow profile osd
> 2018-08-20 08:57:58.753 7f9d85934700 20 match
> 2018-08-20 08:57:58.753 7f9d85934700 10 mon.mon02@1(peon) e4
> _allowed_command capable
> 2018-08-20 08:57:58.753 7f9d85934700 0 log_channel(audit) log [INF] :
> from='osd.48 10.24.52.17:6800/153683' entity='osd.48' cmd=[{"prefix":
> "osd crush set-device-class", "class": "ssd", "ids": ["48"]}]:
> dispatch
> 2018-08-20 08:57:58.753 7f9d85934700 10 mon.mon02@1(peon).osd e46327
> preprocess_query mon_command({"prefix": "osd crush set-device-class",
> "class": "ssd", "ids": ["48"]} v 0) v1 from osd.48
> 10.24.52.17:6800/153683
> 2018-08-20 08:57:58.753 7f9d85934700 10 mon.mon02@1(peon) e4
> forward_request 4 request mon_command({"prefix": "osd crush
> set-device-class", "class": "ssd", "ids": ["48"]} v 0) v1 features
> 4611087854031142907
> 2018-08-20 08:57:58.753 7f9d85934700 20 mon.mon02@1(peon) e4
> _ms_dispatch existing session 0x55b4ec482a80 for mon.1
> 10.24.52.11:6789/0
> 2018-08-20 08:57:58.753 7f9d85934700 20 mon.mon02@1(peon) e4 caps allow *
> 2018-08-20 08:57:58.753 7f9d85934700 10 mon.mon02@1(peon).log
> v10758065 preprocess_query log(1 entries from seq 4 at 2018-08-20
> 08:57:58.755306) v1 from mon.1 10.24.52.11:6789/0
> 2018-08-20 08:57:58.753 7f9d85934700 10 mon.mon02@1(peon).log
> v10758065 preprocess_log log(1 entries from seq 4 at 2018-08-20
> 08:57:58.755306) v1 from mon.1
> 2018-08-20 08:57:58.753 7f9d85934700 20 is_capable service=log
> command= write on cap allow *
> 2018-08-20 08:57:58.753 7f9d85934700 20 allow so far , doing grant allow *
> 2018-08-20 08:57:58.753 7f9d85934700 20 allow all
> 2018-08-20 08:57:58.753 7f9d85934700 10 mon.mon02@1(peon) e4
> forward_request 5 request log(1 entries from seq 4 at 2018-08-20
> 08:57:58.755306) v1 features 4611087854031142907
> 2018-08-20 08:57:58.754 7f9d85934700 20 mon.mon02@1(peon) e4
> _ms_dispatch existing session 0x55b4ec4828c0 for mon.0
> 10.24.52.10:6789/0
> 2018-08-20 08:57:58.754 7f9d85934700 20 mon.mon02@1(peon) e4 caps allow *
> 2018-08-20 08:57:58.754 7f9d85934700 20 is_capable service=mon
> command= read on cap allow *
> 2018-08-20 08:57:58.754 7f9d85934700 20 allow so far , doing grant allow *
> 2018-08-20 08:57:58.754 7f9d85934700 20 allow all
> 2018-08-20 08:57:58.754 7f9d85934700 20 is_capable service=mon
> command= exec on cap allow *
> 2018-08-20 08:57:58.754 7f9d85934700 20 allow so far , doing grant allow *
> 2018-08-20 08:57:58.754 7f9d85934700 20 allow all
> 2018-08-20 08:57:58.754 7f9d85934700 10 mon.mon02@1(peon) e4
> handle_route mon_command_ack([{"prefix": "osd crush set-device-class",
> "class": "ssd", "ids": ["48"]}]=-22 (22) Invalid argument v46327) v1
> to unknown.0 -
> 2018-08-20 08:57:58.785 7f9d85934700 10 mon.mon02@1(peon) e4
> ms_handle_reset 0x55b4ecf4b200 10.24.52.17:6800/153683
> 2018-08-20 08:57:58.785 7f9d85934700 10 mon.mon02@1(peon) e4
> reset/close on session osd.48 10.24.52.17:6800/153683
> 2018-08-20 08:57:58.785 7f9d85934700 10 mon.mon02@1(peon) e4
> remove_session 0x55b4ecf86380 osd.48 10.24.52.17:6800/153683 features
> 0x3ffddff8ffa4fffb
> 2018-08-20 08:57:58.828 7f9d85934700 20 mon.mon02@1(peon) e4
> _ms_dispatch existing session 0x55b4ec4828c0 for mon.0
> 10.24.52.10:6789/0
> On Sat, Aug 18, 2018 at 7:54 PM Daznis <daznis@xxxxxxxxx> wrote:
> >
> > Hello,
> >
> > not sure about it. I assumed ceph-deploy would do it with the
> > "--zap-disk" flag defined. I will try it on Monday and report the
> > progress.
> > On Sat, Aug 18, 2018 at 3:02 PM Alfredo Deza <adeza@xxxxxxxxxx> wrote:
> > >
> > > On Fri, Aug 17, 2018 at 7:05 PM, Daznis <daznis@xxxxxxxxx> wrote:
> > > > Hello,
> > > >
> > > >
> > > > I have replace one of our failed OSD drives and recreated a new osd
> > > > with ceph-deploy and it failes to start.
> > >
> > > Is it possible you haven't zapped the journal on nvme0n1p13 ?
> > >
> > >
> > >
> > > >
> > > > Command: ceph-deploy --overwrite-conf osd create --filestore
> > > > --zap-disk --data /dev/bcache0 --journal /dev/nvme0n1p13 <Hostname>
> > > >
> > > > Output off ceph-deploy:
> > > > [ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
> > > > [ceph_deploy.cli][INFO ] Invoked (2.0.1): /usr/bin/ceph-deploy
> > > > --overwrite-conf osd create --filestore --zap-disk --data /dev/bcache0
> > > > --journal /dev/nvme0n1p13 <Hostname>
> > > > [ceph_deploy.cli][INFO ] ceph-deploy options:
> > > > [ceph_deploy.cli][INFO ] verbose : False
> > > > [ceph_deploy.cli][INFO ] bluestore : None
> > > > [ceph_deploy.cli][INFO ] cd_conf :
> > > > <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f8622160bd8>
> > > > [ceph_deploy.cli][INFO ] cluster : ceph
> > > > [ceph_deploy.cli][INFO ] fs_type : xfs
> > > > [ceph_deploy.cli][INFO ] block_wal : None
> > > > [ceph_deploy.cli][INFO ] default_release : False
> > > > [ceph_deploy.cli][INFO ] username : None
> > > > [ceph_deploy.cli][INFO ] journal : /dev/nvme0n1p13
> > > > [ceph_deploy.cli][INFO ] subcommand : create
> > > > [ceph_deploy.cli][INFO ] host : <Hostname>
> > > > [ceph_deploy.cli][INFO ] filestore : True
> > > > [ceph_deploy.cli][INFO ] func : <function
> > > > osd at 0x7f8622194848>
> > > > [ceph_deploy.cli][INFO ] ceph_conf : None
> > > > [ceph_deploy.cli][INFO ] zap_disk : True
> > > > [ceph_deploy.cli][INFO ] data : /dev/bcache0
> > > > [ceph_deploy.cli][INFO ] block_db : None
> > > > [ceph_deploy.cli][INFO ] dmcrypt : False
> > > > [ceph_deploy.cli][INFO ] overwrite_conf : True
> > > > [ceph_deploy.cli][INFO ] dmcrypt_key_dir :
> > > > /etc/ceph/dmcrypt-keys
> > > > [ceph_deploy.cli][INFO ] quiet : False
> > > > [ceph_deploy.cli][INFO ] debug : False
> > > > [ceph_deploy.osd][DEBUG ] Creating OSD on cluster ceph with data
> > > > device /dev/bcache0
> > > > [<Hostname>][DEBUG ] connected to host: <Hostname>
> > > > [<Hostname>][DEBUG ] detect platform information from remote host
> > > > [<Hostname>][DEBUG ] detect machine type
> > > > [<Hostname>][DEBUG ] find the location of an executable
> > > > [ceph_deploy.osd][INFO ] Distro info: CentOS Linux 7.5.1804 Core
> > > > [ceph_deploy.osd][DEBUG ] Deploying osd to <Hostname>
> > > > [<Hostname>][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
> > > > [<Hostname>][DEBUG ] find the location of an executable
> > > > [ceph_deploy.osd][WARNIN] zapping is no longer supported when preparing
> > > > [<Hostname>][INFO ] Running command: /usr/sbin/ceph-volume --cluster
> > > > ceph lvm create --filestore --data /dev/bcache0 --journal
> > > > /dev/nvme0n1p13
> > > > [<Hostname>][DEBUG ] Running command: /bin/ceph-authtool --gen-print-key
> > > > [<Hostname>][DEBUG ] Running command: /bin/ceph --cluster ceph --name
> > > > client.bootstrap-osd --keyring
> > > > /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new
> > > > a503ae5e-b5b9-40d7-b8b3-194f15e52082
> > > > [<Hostname>][DEBUG ] Running command: /usr/sbin/vgcreate --force --yes
> > > > ceph-a1ffe5bb-6f06-49c6-8aec-e3eb3a311162 /dev/bcache0
> > > > [<Hostname>][DEBUG ] stdout: Physical volume "/dev/bcache0"
> > > > successfully created.
> > > > [<Hostname>][DEBUG ] stdout: Volume group
> > > > "ceph-a1ffe5bb-6f06-49c6-8aec-e3eb3a311162" successfully created
> > > > [<Hostname>][DEBUG ] Running command: /usr/sbin/lvcreate --yes -l
> > > > 100%FREE -n osd-data-a503ae5e-b5b9-40d7-b8b3-194f15e52082
> > > > ceph-a1ffe5bb-6f06-49c6-8aec-e3eb3a311162
> > > > [<Hostname>][DEBUG ] stdout: Logical volume
> > > > "osd-data-a503ae5e-b5b9-40d7-b8b3-194f15e52082" created.
> > > > [<Hostname>][DEBUG ] Running command: /bin/ceph-authtool --gen-print-key
> > > > [<Hostname>][DEBUG ] Running command: /usr/sbin/mkfs -t xfs -f -i
> > > > size=2048 /dev/ceph-a1ffe5bb-6f06-49c6-8aec-e3eb3a311162/osd-data-a503ae5e-b5b9-40d7-b8b3-194f15e52082
> > > > [<Hostname>][DEBUG ] stdout:
> > > > meta-data=/dev/ceph-a1ffe5bb-6f06-49c6-8aec-e3eb3a311162/osd-data-a503ae5e-b5b9-40d7-b8b3-194f15e52082
> > > > isize=2048 agcount=4, agsize=244154112 blks
> > > > [<Hostname>][DEBUG ] = sectsz=512
> > > > attr=2, projid32bit=1
> > > > [<Hostname>][DEBUG ] = crc=1
> > > > finobt=0, sparse=0
> > > > [<Hostname>][DEBUG ] data = bsize=4096
> > > > blocks=976616448, imaxpct=5
> > > > [<Hostname>][DEBUG ] = sunit=0 swidth=0 blks
> > > > [<Hostname>][DEBUG ] naming =version 2 bsize=4096
> > > > ascii-ci=0 ftype=1
> > > > [<Hostname>][DEBUG ] log =internal log bsize=4096
> > > > blocks=476863, version=2
> > > > [<Hostname>][DEBUG ] = sectsz=512
> > > > sunit=0 blks, lazy-count=1
> > > > [<Hostname>][DEBUG ] realtime =none extsz=4096
> > > > blocks=0, rtextents=0
> > > > [<Hostname>][DEBUG ] Running command: /bin/mount -t xfs -o
> > > > rw,noatime,inode64,noquota,nodiratime,logbufs=8,logbsize=256k,attr2
> > > > /dev/ceph-a1ffe5bb-6f06-49c6-8aec-e3eb3a311162/osd-data-a503ae5e-b5b9-40d7-b8b3-194f15e52082
> > > > /var/lib/ceph/osd/ceph-48
> > > > [<Hostname>][DEBUG ] Running command: /bin/chown -R ceph:ceph /dev/nvme0n1p13
> > > > [<Hostname>][DEBUG ] Running command: /bin/ln -s /dev/nvme0n1p13
> > > > /var/lib/ceph/osd/ceph-48/journal
> > > > [<Hostname>][DEBUG ] Running command: /bin/ceph --cluster ceph --name
> > > > client.bootstrap-osd --keyring
> > > > /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o
> > > > /var/lib/ceph/osd/ceph-48/activate.monmap
> > > > [<Hostname>][DEBUG ] stderr: got monmap epoch 4
> > > > [<Hostname>][DEBUG ] Running command: /bin/chown -R ceph:ceph /dev/nvme0n1p13
> > > > [<Hostname>][DEBUG ] Running command: /bin/chown -R ceph:ceph
> > > > /var/lib/ceph/osd/ceph-48/
> > > > [<Hostname>][DEBUG ] Running command: /bin/ceph-osd --cluster ceph
> > > > --osd-objectstore filestore --mkfs -i 48 --monmap
> > > > /var/lib/ceph/osd/ceph-48/activate.monmap --keyfile - --osd-data
> > > > /var/lib/ceph/osd/ceph-48/ --osd-journal
> > > > /var/lib/ceph/osd/ceph-48/journal --osd-uuid
> > > > a503ae5e-b5b9-40d7-b8b3-194f15e52082 --setuser ceph --setgroup ceph
> > > > [<Hostname>][DEBUG ] stderr: 2018-08-17 18:23:26.067 7f1f62c471c0 -1
> > > > auth: unable to find a keyring on /var/lib/ceph/osd/ceph-48//keyring:
> > > > (2) No such file or directory
> > > > [<Hostname>][DEBUG ] stderr: 2018-08-17 18:23:26.188 7f1f62c471c0 -1
> > > > journal read_header error decoding journal header
> > > > [<Hostname>][DEBUG ] stderr: 2018-08-17 18:23:26.198 7f1f62c471c0 -1
> > > > journal do_read_entry(4096): bad header magic
> > > > [<Hostname>][DEBUG ] stderr: 2018-08-17 18:23:26.198 7f1f62c471c0 -1
> > > > journal do_read_entry(4096): bad header magic
> > > > [<Hostname>][DEBUG ] Running command: /bin/ceph-authtool
> > > > /var/lib/ceph/osd/ceph-48/keyring --create-keyring --name osd.48
> > > > --add-key AQB39nZbiXJBMBAAOb9cxepxJrflhSNADuVNSQ==
> > > > [<Hostname>][DEBUG ] stdout: creating /var/lib/ceph/osd/ceph-48/keyring
> > > > [<Hostname>][DEBUG ] added entity osd.48 auth auth(auid =
> > > > 18446744073709551615 key=AQB39nZbiXJBMBAAOb9cxepxJrflhSNADuVNSQ== with
> > > > 0 caps)
> > > > [<Hostname>][DEBUG ] Running command: /bin/chown -R ceph:ceph
> > > > /var/lib/ceph/osd/ceph-48/keyring
> > > > [<Hostname>][DEBUG ] --> ceph-volume lvm prepare successful for: /dev/bcache0
> > > > [<Hostname>][DEBUG ] Running command: /bin/ln -snf /dev/nvme0n1p13
> > > > /var/lib/ceph/osd/ceph-48/journal
> > > > [<Hostname>][DEBUG ] Running command: /bin/chown -R ceph:ceph /dev/nvme0n1p13
> > > > [<Hostname>][DEBUG ] Running command: /bin/systemctl enable
> > > > ceph-volume@lvm-48-a503ae5e-b5b9-40d7-b8b3-194f15e52082
> > > > [<Hostname>][DEBUG ] stderr: Created symlink from
> > > > /etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-48-a503ae5e-b5b9-40d7-b8b3-194f15e52082.service
> > > > to /usr/lib/systemd/system/ceph-volume@.service.
> > > > [<Hostname>][DEBUG ] Running command: /bin/systemctl start ceph-osd@48
> > > > [<Hostname>][DEBUG ] --> ceph-volume lvm activate successful for osd ID: 48
> > > > [<Hostname>][DEBUG ] --> ceph-volume lvm create successful for: /dev/bcache0
> > > > [<Hostname>][INFO ] checking OSD status...
> > > > [<Hostname>][DEBUG ] find the location of an executable
> > > > [<Hostname>][INFO ] Running command: /bin/ceph --cluster=ceph osd
> > > > stat --format=json
> > > >
> > > >
> > > > When OSD service is starting I'm getting these errors:
> > > > 2018-08-17 19:12:02.998 7fd06e5a91c0 0 _get_class not permitted to load lua
> > > > 2018-08-17 19:12:02.998 7fd06e5a91c0 1 osd.48 0 warning: got an error
> > > > loading one or more classes: (1) Operation not permitted
> > > > 2018-08-17 19:12:02.998 7fd06e5a91c0 0 osd.48 0 crush map has
> > > > features 288232575208783872, adjusting msgr requires for clients
> > > > 2018-08-17 19:12:02.998 7fd06e5a91c0 0 osd.48 0 crush map has
> > > > features 288232575208783872 was 8705, adjusting msgr requires for mons
> > > > 2018-08-17 19:12:02.998 7fd06e5a91c0 0 osd.48 0 crush map has
> > > > features 288232575208783872, adjusting msgr requires for osds
> > > > 2018-08-17 19:12:02.998 7fd06e5a91c0 0 osd.48 0 load_pgs
> > > > 2018-08-17 19:12:02.998 7fd06e5a91c0 0 osd.48 0 load_pgs opened 0 pgs
> > > > 2018-08-17 19:12:02.998 7fd06e5a91c0 0 osd.48 0 using
> > > > weightedpriority op queue with priority op cut off at 64.
> > > > 2018-08-17 19:12:02.999 7fd06e5a91c0 -1 osd.48 0 log_to_monitors {default=true}
> > > > 2018-08-17 19:12:03.004 7fd06e5a91c0 -1 osd.48 0
> > > > mon_cmd_maybe_osd_create fail: '(22) Invalid argument': (22) Invalid
> > > > argument
> > > > 2018-08-17 19:12:03.004 7fd06e5a91c0 -1 osd.48 0 init unable to
> > > > update_crush_device_class: (22) Invalid argument
> > > >
> > > >
> > > > So i tried to add the osd into crushmap with "ceph osd crush add
> > > > osd.48 4.0 host=<Hostname>" same 22 error appears: Error EINVAL: (22)
> > > > Invalid argument. Trying to set device class it also fails with the
> > > > same error.
> > > >
> > > > If I manually add the OSD to crushmap. Crustool fails to compile the
> > > > map with errors.
> > > > _______________________________________________
> > > > ceph-users mailing list
> > > > ceph-users@xxxxxxxxxxxxxx
> > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
