Hi!
I changed the partitioning scheme to use a "real" primary partition instead of a logical volume. Ceph-deploy seems run fine now, but the OSD does not start.
I see lots of these in the journal:
Jul 06 13:53:42 sh[9768]: 0> 2017-07-06 13:53:42.794027 7fcf9918fb80 -1 *** Caught signal (Aborted) **
Jul 06 13:53:42 sh[9768]: in thread 7fcf9918fb80 thread_name:ceph-osd
Jul 06 13:53:42 sh[9768]: ceph version 12.1.0 (262617c9f16c55e863693258061c5b25dea5b086) luminous (dev)
Jul 06 13:53:42 sh[9768]: 1: (()+0x9cd6af) [0x7fcf99b776af]
Jul 06 13:53:42 sh[9768]: 2: (()+0xf370) [0x7fcf967d9370]
Jul 06 13:53:42 sh[9768]: 3: (gsignal()+0x37) [0x7fcf958031d7]
Jul 06 13:53:42 sh[9768]: 4: (abort()+0x148) [0x7fcf958048c8]
Jul 06 13:53:42 sh[9768]: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x284) [0x7fcf99bb5394]
Jul 06 13:53:42 sh[9768]: 6: (BitMapAreaIN::reserve_blocks(long)+0xb6) [0x7fcf99b6c486]
Jul 06 13:53:42 sh[9768]: 7: (BitMapAllocator::reserve(unsigned long)+0x80) [0x7fcf99b6a240]
Jul 06 13:53:42 sh[9768]: 8: (BlueFS::_allocate(unsigned char, unsigned long, std::vector<bluefs_extent_t, mempool::pool_allocator<(mempool::pool_index_t)9, bluefs_extent_t> >*)+0xee) [0x7fcf99b31c\
0e]
Jul 06 13:53:42 sh[9768]: 9: (BlueFS::_flush_and_sync_log(std::unique_lock<std::mutex>&, unsigned long, unsigned long)+0xbc4) [0x7fcf99b38be4]
Jul 06 13:53:42 sh[9768]: 10: (BlueFS::sync_metadata()+0x215) [0x7fcf99b3d725]
Jul 06 13:53:42 sh[9768]: 11: (BlueFS::umount()+0x74) [0x7fcf99b3dc44]
Jul 06 13:53:42 sh[9768]: 12: (BlueStore::_open_db(bool)+0x579) [0x7fcf99a62859]
Jul 06 13:53:42 sh[9768]: 13: (BlueStore::fsck(bool)+0x39b) [0x7fcf99a9581b]
Jul 06 13:53:42 sh[9768]: 14: (BlueStore::mkfs()+0x1168) [0x7fcf99a6d118]
Jul 06 13:53:42 sh[9768]: 15: (OSD::mkfs(CephContext*, ObjectStore*, std::string const&, uuid_d, int)+0x29b) [0x7fcf9964b75b]
Jul 06 13:53:42 sh[9768]: 16: (main()+0xf83) [0x7fcf99590573]
Jul 06 13:53:42 sh[9768]: 17: (__libc_start_main()+0xf5) [0x7fcf957efb35]
Jul 06 13:53:42 sh[9768]: 18: (()+0x4826e6) [0x7fcf9962c6e6]
Jul 06 13:53:42 sh[9768]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Jul 06 13:53:42 sh[9768]: Traceback (most recent call last):
Jul 06 13:53:42 sh[9768]: File "/usr/sbin/ceph-disk", line 9, in <module>
Jul 06 13:53:42 sh[9768]: load_entry_point('ceph-disk==1.0.0', 'console_scripts', 'ceph-disk')()
Also interesting is the message "-1 rocksdb: Invalid argument: db: does not exist (create_if_missing is false)"... Looks to me as if ceph-deploy did not create the RocksDB?
So still no success with bluestore :(
Thanks,
Martin
-----Ursprüngliche Nachricht-----
Von: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] Im Auftrag von Martin Emrich
Gesendet: Dienstag, 4. Juli 2017 22:02
An: Loris Cuoghi <loris.cuoghi@xxxxxxxxxxxxxxx>; ceph-users@xxxxxxxxxxxxxx
Betreff: Re: How to set up bluestore manually?
Hi!
After getting some other stuff done, I finally got around to continuing here.
I set up a whole new cluster with ceph-deploy, but adding the first OSD fails:
ceph-deploy osd create --bluestore ${HOST}:/dev/sdc --block-wal /dev/cl/ceph-waldb-sdc --block-db /dev/cl/ceph-waldb-sdc .
.
.
[WARNIN] get_partition_dev: Try 9/10 : partition 1 for /dev/cl/ceph-waldb-sdc does not exist in /sys/block/dm-2 [WARNIN] get_dm_uuid: get_dm_uuid /dev/cl/ceph-waldb-sdc uuid path is /sys/dev/block/253:2/dm/uuid [WARNIN] get_dm_uuid: get_dm_uuid /dev/cl/ceph-waldb-sdc uuid is LVM-2r0bGcoyMB0VnWeGGS77eOD5IOu8wAPN3wPX4OWSS1XGkYZYoziXhfAFMjJf4FJR
[WARNIN]
[WARNIN] get_partition_dev: Try 10/10 : partition 1 for /dev/cl/ceph-waldb-sdc does not exist in /sys/block/dm-2 [WARNIN] get_dm_uuid: get_dm_uuid /dev/cl/ceph-waldb-sdc uuid path is /sys/dev/block/253:2/dm/uuid [WARNIN] get_dm_uuid: get_dm_uuid /dev/cl/ceph-waldb-sdc uuid is LVM-2r0bGcoyMB0VnWeGGS77eOD5IOu8wAPN3wPX4OWSS1XGkYZYoziXhfAFMjJf4FJR
[WARNIN]
[WARNIN] Traceback (most recent call last):
[WARNIN] File "/usr/sbin/ceph-disk", line 9, in <module>
[WARNIN] load_entry_point('ceph-disk==1.0.0', 'console_scripts', 'ceph-disk')()
[WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 5687, in run
[WARNIN] main(sys.argv[1:])
[WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 5638, in main
[WARNIN] args.func(args)
[WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2004, in main
[WARNIN] Prepare.factory(args).prepare()
[WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1993, in prepare
[WARNIN] self._prepare()
[WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2074, in _prepare
[WARNIN] self.data.prepare(*to_prepare_list)
[WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2807, in prepare
[WARNIN] self.prepare_device(*to_prepare_list)
[WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2983, in prepare_device
[WARNIN] to_prepare.prepare()
[WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2216, in prepare
[WARNIN] self.prepare_device()
[WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2310, in prepare_device
[WARNIN] partition = device.get_partition(num)
[WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1714, in get_partition
[WARNIN] dev = get_partition_dev(self.path, num)
[WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 717, in get_partition_dev
[WARNIN] (pnum, dev, error_msg))
[WARNIN] ceph_disk.main.Error: Error: partition 1 for /dev/cl/ceph-waldb-sdc does not appear to exist in /sys/block/dm-2 [ERROR ] RuntimeError: command returned non-zero exit status: 1 [ceph_deploy.osd][ERROR ] Failed to execute command: /usr/sbin/ceph-disk -v prepare --block.wal /dev/cl/ceph-waldb-sdc --block.db /dev/cl/ceph-waldb-sdc --bluestore --cluster ceph --fs-type xfs -- /dev/sdc [ceph_deploy][ERROR ] GenericError: Failed to create 1 OSDs
I found some open issues about this: http://tracker.ceph.com/issues/6042, and http://tracker.ceph.com/issues/5461.. Could this be related?
Cheers,
Martin
-----Ursprüngliche Nachricht-----
Von: Loris Cuoghi [mailto:loris.cuoghi@xxxxxxxxxxxxxxx]
Gesendet: Montag, 3. Juli 2017 15:48
An: Martin Emrich <martin.emrich@xxxxxxxxxxx>
Cc: ceph-users@xxxxxxxxxxxxxx; Vasu Kulkarni <vakulkar@xxxxxxxxxx>
Betreff: Re: How to set up bluestore manually?
Le Mon, 3 Jul 2017 12:32:20 +0000,
Martin Emrich <martin.emrich@xxxxxxxxxxx> a écrit :
> Hi!
>
> Thanks for the super-fast response!
>
> That did work somehow... Here's my commandline (As Bluestore seems to
> still require a Journal,
No, it doesn't. :D
> I repurposed the SSD partitions for it and put the DB/WAL on the
> spinning disk):
On the contrary, Bluestore's DB/WAL are good candidates for low-latency storage like an SSD.
>
> ceph-deploy osd create --bluestore
> <hostname>:/dev/sdc:/dev/mapper/cl-ceph_journal_sdc
Just
ceph-deploy osd create --bluestore ${hostname}:/device/path
should be sufficent to create a device composed of:
* 1 small (~100 MB) XFS partition
* 1 big (remaining space) partition formatted as bluestore
Additional options like:
--block-wal /path/to/ssd/partition
--block-db /path/to/ssd/partition
allow having SSD-backed WAL and DB.
> But it created two (!) new OSDs instead of one, and placed them under
> the default CRUSH rule (thus making my cluster doing stuff; they
> should be under a different rule)...
Default stuff is applied... by default :P
Take a good read:
http://docs.ceph.com/docs/master/rados/operations/crush-map/
In particular, on how editing an existing CRUSH map:
http://docs.ceph.com/docs/master/rados/operations/crush-map/#editing-a-crush-map
-- Loris
> Did I do something wrong or are
> the two OSDs part of the bluestore concept? If yes, how to handle them
> in the CRUSH map (I have different categories of OSD hosts for
> different use cases, split by appropriate CRUSH rules).
>
> Thanks
>
> Martin
>
> -----Ursprüngliche Nachricht-----
> Von: Loris Cuoghi [mailto:loris.cuoghi@xxxxxxxxxxxxxxx]
> Gesendet: Montag, 3. Juli 2017 13:39
> An: Martin Emrich <martin.emrich@xxxxxxxxxxx>
> Cc: Vasu Kulkarni <vakulkar@xxxxxxxxxx>; ceph-users@xxxxxxxxxxxxxx
> Betreff: Re: How to set up bluestore manually?
>
> Le Mon, 3 Jul 2017 11:30:04 +0000,
> Martin Emrich <martin.emrich@xxxxxxxxxxx> a écrit :
>
> > Hi!
> >
> > Thanks for the hint, but I get this error:
> >
> > [ceph_deploy][ERROR ] ConfigError: Cannot load config: [Errno 2] No
> > such file or directory: 'ceph.conf'; has `ceph-deploy new` been run
> > in this directory?
> >
> > Obviously, ceph-deploy only works if the cluster has been managed
> > with ceph-deply all along (and I won’t risk messing with my cluster
> > by attempting to “retrofit” ceph-deply to it)… I’ll setup a single
> > VM cluster to squeeze out the necessary commands and check back…
>
> No need to start with ceph-deploy in order to use it :)
>
> You can:
>
> create a working directory (e.g. ~/ceph-deploy)
> - cd ~/ceph-deploy
> - copy the ceph.conf from /etc/ceph/ceph.conf in the working directory
> - execute ceph-deploy gatherkeys to obtain the necessary keys
>
> et voilà ;)
>
> Give it a try!
>
> -- Loris
>
> >
> > Regards,
> >
> > Martin
> >
> > Von: Vasu Kulkarni [mailto:vakulkar@xxxxxxxxxx]
> > Gesendet: Freitag, 30. Juni 2017 17:58
> > An: Martin Emrich <martin.emrich@xxxxxxxxxxx>
> > Cc: ceph-users@xxxxxxxxxxxxxx
> > Betreff: Re: How to set up bluestore manually?
> >
> >
> >
> > On Fri, Jun 30, 2017 at 8:31 AM, Martin Emrich
> > <martin.emrich@xxxxxxxxxxx<mailto:martin.emrich@xxxxxxxxxxx>> wrote:
> > Hi!
> >
> > I’d like to set up new OSDs with bluestore: the real data (“block”)
> > on a spinning disk, and DB+WAL on a SSD partition.
> >
> > But I do not use ceph-deploy, and never used ceph-disk (I set up the
> > filestore OSDs manually). Google tells me that ceph-disk does not
> > (yet) support splitting the components across multiple block
> > devices, I also had no luck while attempting it anyways. (using Ceph
> > 12.1 RC1)
> >
> > why not give ceph-deploy a chance, It has options to specify for db
> > and wal in command line. you dont have to worry about what options
> > to pass to ceph-disk as it encapsulates that
> >
> > ex: ceph-deploy osd create --bluestore --block-wal /dev/nvme0n1
> > --block-db /dev/nvme0n1 p30:sdb
> >
> >
> >
> > I just can’t find documentation on how to set up a bluestore OSD
> > manually:
> >
> >
> > * How do I “prepare” the block, block.wal and block.db
> > blockdevices? Just directing ceph-osd to the block devices via
> > ceph.conf does not seem to be enough.
> > * Do bluestore OSDs still use/need a separate journal
> > file/device? Or ist that replaced by the WAL?
> >
> > http://docs.ceph.com/docs/master/ also has not very much information
> > on using bluestore, is this documentation deprecated?
> >
> > The document has fallen behind for manual deployment and ceph-deploy
> > usage for bluestore as well, but if you use the above command, it
> > will clearly throw out what it is doing and how ceph-disk is being
> > called, those are essentially manual steps that should go in
> > document. I think we have a tracker for update which is pending for
> > sometime.
> >
> >
> > Thanks for any hints,
> >
> > Martin
> >
> >
> > _______________________________________________
> > ceph-users mailing list
> > ceph-users@xxxxxxxxxxxxxx<mailto:ceph-users@xxxxxxxxxxxxxx>
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
