Re: Filestore to Bluestore migration question

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Sorry my mixup.

Therefore you shouldn’t be running ZAP against /dev/sda as this will wipe the whole SSD.

I Guess currently in its setup it’s using a partition on /dev/sda? Like /dev/sda2 for example.

,Ashley

On Wed, 7 Nov 2018 at 11:30 PM, Hayashida, Mami <mami.hayashida@xxxxxxx> wrote:
Yes, that was indeed a copy-and-paste mistake.  I am trying to use /dev/sdh (hdd) for data and a part of /dev/sda (ssd)  for the journal.  That's how the Filestore is set-up.  So, for the Bluestore, data on /dev/sdh,  wal and db on /dev/sda. 


On Wed, Nov 7, 2018 at 10:26 AM, Ashley Merrick <singapore@xxxxxxxxxxxxxx> wrote:
ceph osd destroy 70  --yes-i-really-mean-it

I am guessing that’s a copy and paste mistake and should say 120.

Is the SSD @ /dev/sdh fully for the OSD120 is a partition on this SSD the journal and other partitions are for other SSD’s?

On Wed, 7 Nov 2018 at 11:21 PM, Hayashida, Mami <mami.hayashida@xxxxxxx> wrote:
I would agree with that.  So, here is what I am planning on doing today.  I will try this from scratch on a different OSD node from the very first step and log input and output for every step.  Here is the outline of what I think (based on all the email exchanges so far) should happen.  

*******
Trying to convert osd.120 to Bluestore.  Data is on /sda/sdh.   Filestore Journal is on a partition drive (40GB) on /dev/sda.  

#Mark those OSDs out
ceph osd out 120

# Stop the OSDs
systemctl kill ceph-osd@120

# Unmount the filesystem
sudo umount /var/lib/ceph/osd/ceph-120

# Destroy the data
ceph-volume lvm zap /dev/sdh --destroy   # data disk
ceph-volume lvm zap /dev/sda --destroy   # ssd for wal and db

# Inform the cluster
ceph osd destroy 70  --yes-i-really-mean-it

# Check all the /etc/fstab and /etc/systemd/system to make sure that all the references to the filesystem is gone. Run
ln -sf /dev/null /etc/systemd/system/ceph-disk@70.service 

# Create PVs, VGs, LVs 
pvcreate /dev/sda # for wal and db
pvcreate /dev/sdh # for data

vgcreate ssd0 /dev/sda
vgcreate hdd120  /dev/sdh

lvcreate -L 40G -n db120 ssd0
lvcreate -l 100%VG data120 hdd120

# Run ceph-volume
ceph-volume lvm prepare --bluestore --data hdd120/data120 --block.db ssd0/db120  --osd-id 120 

# Activate
ceph-volume lvm activate 120 <osd fsid>

******
Does this sound right?

On Tue, Nov 6, 2018 at 4:32 PM, Alfredo Deza <adeza@xxxxxxxxxx> wrote:
It is pretty difficult to know what step you are missing if we are
getting the `activate --all` command.

Maybe if you try one by one, capturing each command, throughout the
process, with output. In the filestore-to-bluestore guides we never
advertise `activate --all` for example.

Something is missing here, and I can't tell what it is.
On Tue, Nov 6, 2018 at 4:13 PM Hayashida, Mami <mami.hayashida@xxxxxxx> wrote:
>
> This is becoming even more confusing. I got rid of those ceph-disk@6[0-9].service (which had been symlinked to /dev/null).  Moved /var/lib/ceph/osd/ceph-6[0-9] to  /var/...../osd_old/.  Then, I ran  `ceph-volume lvm activate --all`.  I got once again
>
> root@osd1:~# ceph-volume lvm activate --all
> --> Activating OSD ID 67 FSID 17cd6755-76f9-4160-906c-1bf13d09fb3d
> Running command: mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-67
> --> Absolute path not found for executable: restorecon
> --> Ensure $PATH environment variable contains common executable locations
> Running command: ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/hdd67/data67 --path /var/lib/ceph/osd/ceph-67
>  stderr: failed to read label for /dev/hdd67/data67: (2) No such file or directory
> -->  RuntimeError: command returned non-zero exit status: 1
>
> But when I ran `df` and `mount` ceph-67 is the only one that exists. (and in  /var/lib/ceph/osd/)
>
> root@osd1:~# df -h | grep ceph-6
> tmpfs           126G     0  126G   0% /var/lib/ceph/osd/ceph-67
>
> root@osd1:~# mount | grep ceph-6
> tmpfs on /var/lib/ceph/osd/ceph-67 type tmpfs (rw,relatime)
>
> root@osd1:~# ls /var/lib/ceph/osd/ | grep ceph-6
> ceph-67
>
> But in I cannot restart any of these 10 daemons (`systemctl start ceph-osd@6[0-9]`).
>
> I am wondering if I should zap these 10 osds and start over although at this point I am afraid even zapping may not be a simple task....
>
>
>
> On Tue, Nov 6, 2018 at 3:44 PM, Hector Martin <hector@xxxxxxxxxxxxxx> wrote:
>>
>> On 11/7/18 5:27 AM, Hayashida, Mami wrote:
>> > 1. Stopped osd.60-69:  no problem
>> > 2. Skipped this and went to #3 to check first
>> > 3. Here, `find /etc/systemd/system | grep ceph-volume` returned
>> > nothing.  I see in that directory
>> >
>> > /etc/systemd/system/ceph-disk@60.service    # and 61 - 69.
>> >
>> > No ceph-volume entries.
>>
>> Get rid of those, they also shouldn't be there. Then `systemctl
>> daemon-reload` and continue, see if you get into a good state. basically
>> feel free to nuke anything in there related to OSD 60-69, since whatever
>> is needed should be taken care of by the ceph-volume activation.
>>
>>
>> --
>> Hector Martin (hector@xxxxxxxxxxxxxx)
>> Public Key: https://mrcn.st/pub
>
>
>
>
> --
> Mami Hayashida
> Research Computing Associate
>
> Research Computing Infrastructure
> University of Kentucky Information Technology Services
> 301 Rose Street | 102 James F. Hardymon Building
> Lexington, KY 40506-0495
> mami.hayashida@xxxxxxx
> (859)323-7521



--
Mami Hayashida
Research Computing Associate

Research Computing Infrastructure
University of Kentucky Information Technology Services
301 Rose Street | 102 James F. Hardymon Building
Lexington, KY 40506-0495
mami.hayashida@xxxxxxx
(859)323-7521
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



--
Mami Hayashida
Research Computing Associate

Research Computing Infrastructure
University of Kentucky Information Technology Services
301 Rose Street | 102 James F. Hardymon Building
Lexington, KY 40506-0495
mami.hayashida@xxxxxxx
(859)323-7521
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux