Hi all,
I'm trying to migrate my 5-node cluster to Nautilus… Debian Stretch (I
intend to update this once I'm onto Nautilus) as the OS, all 5 machines
are combined OSD/MON/MDS nodes.
Nodes were originally deployed with an earlier version of Ceph (can't
recall now) using filestore on btrfs. Later, after an abortive attempt
to move to Bluestore (it ran… veeeeerrryyy ssslllooowwwlllyyy), I moved
back to filestore on xfs, which whilst no speed demon, worked well enough.
Journal is on the OSD due to space constraints elsewhere in the nodes.
So step 1:
> If you are unsure whether or not your Luminous cluster has completed a full scrub of all PGs, you can check your cluster’s state by running:
>
> # ceph osd dump | grep ^flags
>
> In order to be able to proceed to Nautilus, your OSD map must include the recovery_deletes and purged_snapdirs flags.
Tick to that, I see both flags.
Step 2.
> Make sure your cluster is stable and healthy (no down or recovering OSDs). (Optional, but recommended.)
Tick, all healthy.
Step 3.
> Set the noout flag for the duration of the upgrade. (Optional, but recommended.):
Done
Step 4.
> Upgrade monitors by installing the new packages and restarting the monitor daemons.
Okay, so I update /etc/apt/sources.list.d/ceph.list to point to the
`nautilus` repository, `apt-get update`, `apt-get dist-upgrade -y`.
This goes without a hitch, I now have Ceph 12 binaries on my nodes.
> systemctl restart ceph-mon.target
Ran that, on all of them (and yes, they are all in the quorum)…
> # ceph mon dump | grep min_mon_release
>
> should report:
>
> min_mon_release 14 (nautilus)
I get this:> root@helium:~# ceph mon dump | grep min_mon_release
> dumped monmap epoch 4
No `min_mon_release` mentioned anywhere. I tried re-starting the MON
daemons on all 5 nodes, even doing it in parallel using Ansible… no dice.
Nothing says what to do at this point. It's not reporting an earlier
release, it's just not reporting full stop.
Figuring that, well, I *did* update all the monitors, and re-start them,
I press on.
Step 5.
> Upgrade ceph-mgr daemons by installing the new packages and restarting all manager daemons.
Done, no issues.
> Verify the ceph-mgr daemons are running by checking ceph -s:
I get:
> root@helium:~# ceph -s
> cluster:
> id: 45b532b7-aa3d-4754-9906-d4a70b57630c
> health: HEALTH_WARN
> noout flag(s) set
>
> services:
> mon: 5 daemons, quorum hydrogen,helium,carbon,nitrogen,boron
> mgr: hydrogen(active), standbys: carbon, helium
> mds: cephfs-1/1/1 up {0=helium=up:active}, 2 up:standby
> osd: 5 osds: 5 up, 5 in
> flags noout
>
> data:
> pools: 4 pools, 324 pgs
> objects: 302.75k objects, 1.13TiB
> usage: 3.52TiB used, 5.53TiB / 9.05TiB avail
> pgs: 323 active+clean
> 1 active+clean+scrubbing+deep
>
> io:
> client: 45.0KiB/s rd, 1.10MiB/s wr, 1op/s rd, 38op/s wr
So far so good.
Step 6.
> Upgrade all OSDs by installing the new packages and restarting the ceph-osd daemons on all OSD hosts:
> systemctl restart ceph-osd.target
I actually missed this step at first and went on to step 7, but doubled
back and did it.
Step 7.
> If there are any OSDs in the cluster deployed with ceph-disk (e.g., almost any OSDs that were created before the Mimic release), you need to tell ceph-volume to adopt responsibility for starting the daemons.
Okay, seems simple enough. Now I did this by mistake earlier, so I'll
force it to ensure everything is up to scratch.
> root@hydrogen:~# ceph-volume simple scan --force
> stderr: lsblk: /var/lib/ceph/osd/ceph-5: not a block device
> stderr: Unknown device, --name=, --path=, or absolute path in /dev/ or /sys expected.
> Running command: /sbin/cryptsetup status /dev/sdc1
> --> OSD 5 got scanned and metadata persisted to file: /etc/ceph/osd/5-2a1e6a2b-7742-4cd9-9a39-e4a35ebe74fe.json
> --> To take over managment of this scanned OSD, and disable ceph-disk and udev, run:
> --> ceph-volume simple activate 5 2a1e6a2b-7742-4cd9-9a39-e4a35ebe74fe
> root@hydrogen:~# ceph-volume simple activate --all
> --> activating OSD specified in /etc/ceph/osd/5-2a1e6a2b-7742-4cd9-9a39-e4a35ebe74fe.json
> --> Required devices (data, and journal) not present for filestore
> --> filestore devices found: [u'data']
> --> RuntimeError: Unable to activate filestore OSD due to missing devices
Uhh ohh? So, my journal is on the OSD itself:
> root@hydrogen:~# ls -l /var/lib/ceph/osd/ceph-5/journal
> -rw-r--r-- 1 ceph ceph 21474836480 Aug 29 14:16 /var/lib/ceph/osd/ceph-5/journal
> root@hydrogen:~# cat /etc/ceph/osd/5-2a1e6a2b-7742-4cd9-9a39-e4a35ebe74fe.json
> {
> "active": "ok",
> "ceph_fsid": "45b532b7-aa3d-4754-9906-d4a70b57630c",
> "cluster_name": "ceph",
> "data": {
> "path": "/dev/sdc1",
> "uuid": "2a1e6a2b-7742-4cd9-9a39-e4a35ebe74fe"
> },
> "fsid": "2a1e6a2b-7742-4cd9-9a39-e4a35ebe74fe",
> "keyring": "…censored…",
> "magic": "ceph osd volume v026",
> "ready": "ready",
> "require_osd_release": "",
> "systemd": "",
> "type": "filestore",
> "whoami": 5
> }
How do I tell it that the journal is on `/dev/sdc1` as a file within the
XFS file store?
--
Stuart Longland (aka Redhatter, VK4MSL)
I haven't lost my mind...
...it's backed up on a tape somewhere.
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
