Hi,
Ceph prevents clients from performing I/O operations on full OSD nodes to
avoid losing data. It returns the HEALTH_ERR full osds message when the
cluster reaches the capacity set by the mon_osd_full_ratio parameter. By
default, this parameter is set to 0.95 which means 95% of the cluster
capacity.
If % RAW USED is above 70-75%, there are two options:
1. Delete unnecessary data, but this is short time solution.
2. OR,Scale the cluster by adding a new OSD node.
On Wed, Mar 23, 2022 at 11:44 PM Rodrigo Werle <rodrigo.werle@xxxxxxxxx>
wrote:
> Thanks Eugen!
> Actually, as it is a nvme disk and I thought that the weight could be
> greater. I tried changing it to 0.9 but it is still saying that osd is
> full.
> This osd got full yesterday. I wiped it out and re added. After some time,
> the "osd full" message happened again. It is like the ceph thinks the osd
> is still full, as it was before...
>
>
> Em qua., 23 de mar. de 2022 às 14:38, Eugen Block <eblock@xxxxxx>
> escreveu:
>
> > Without having an answer to the question why the OSD is full I'm
> > wondering why the OSD has a crush weight of 1.29999 while its size is
> > only 1 TB. Was that changed on purpose? I'm not sure if that would
> > explain the OSD full message, though.
> >
> >
> > Zitat von Rodrigo Werle <rodrigo.werle@xxxxxxxxx>:
> >
> > > Hi everyone!
> > > I'm trying to understand why Ceph changed the state of one osd as full
> if
> > > it is 73% used and full_ratio is 0.97.
> > >
> > > Follow some information:
> > >
> > > # ceph health detail
> > > (...)
> > > [ERR] OSD_FULL: 1 full osd(s)
> > > osd.11 is full
> > > (...)
> > >
> > > # ceph osd metadata
> > > (...)
> > >
> > >> {
> > >> "id": 11,
> > >> "arch": "x86_64",
> > >> "back_iface": "",
> > >> "bluefs": "1",
> > >> "bluefs_dedicated_db": "0",
> > >> "bluefs_dedicated_wal": "0",
> > >> "bluefs_single_shared_device": "1",
> > >> "bluestore_bdev_access_mode": "blk",
> > >> "bluestore_bdev_block_size": "4096",
> > >> "bluestore_bdev_dev_node": "/dev/dm-41",
> > >> "bluestore_bdev_devices": "nvme0n1",
> > >> "bluestore_bdev_driver": "KernelDevice",
> > >> "bluestore_bdev_partition_path": "/dev/dm-41",
> > >> "bluestore_bdev_rotational": "0",
> > >> "bluestore_bdev_size": "1000200994816",
> > >> "bluestore_bdev_support_discard": "1",
> > >> "bluestore_bdev_type": "ssd",
> > >> "ceph_release": "octopus",
> > >> "ceph_version": "ceph version 15.2.16
> > >> (d46a73d6d0a67a79558054a3a5a72cb561724974) octopus (stable)",
> > >> "ceph_version_short": "15.2.16",
> > >> "cpu": "Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz",
> > >> "default_device_class": "ssd",
> > >> "device_ids":
> > >> "nvme0n1=Samsung_SSD_970_EVO_Plus_1TB_S59ANJ0N308050F",
> > >> "device_paths":
> > >> "nvme0n1=/dev/disk/by-path/pci-0000:01:00.0-nvme-1",
> > >> "devices": "nvme0n1",
> > >> "distro": "ubuntu",
> > >> "distro_description": "Ubuntu 18.04.6 LTS",
> > >> "distro_version": "18.04",
> > >> "front_iface": "",
> > >> "hostname": "pf-us1-dfs6",
> > >> "journal_rotational": "0",
> > >> "kernel_description": "#182-Ubuntu SMP Fri Mar 18 15:53:46 UTC
> > >> 2022",
> > >> "kernel_version": "4.15.0-173-generic",
> > >> "mem_swap_kb": "117440508",
> > >> "mem_total_kb": "181456152",
> > >> "network_numa_unknown_ifaces": "back_iface,front_iface",
> > >> "objectstore_numa_node": "0",
> > >> "objectstore_numa_nodes": "0",
> > >> "os": "Linux",
> > >> "osd_data": "/var/lib/ceph/osd/ceph-11",
> > >> "osd_objectstore": "bluestore",
> > >> "osdspec_affinity": "",
> > >> "rotational": "0"
> > >> } (...)
> > >
> > >
> > > # cat /var/lib/ceph/osd/ceph-11/type
> > >
> > >> bluestore
> > >
> > >
> > > # ls -l /var/lib/ceph/osd/ceph-11 | grep block
> > >
> > >> lrwxrwxrwx 1 ceph ceph 50 Mar 22 21:44 block ->
> > >> /dev/mapper/YpB2cx-HlyU-VPqT-Abaz-Dutx-iMrz-Tty6o1
> > >
> > >
> > > # lsblk
> > >
> > >> nvme0n1 259:0
> 0
> > >> 931.5G 0 disk
> > >>
> >
> └─ceph--7ef8f83a--d055--4a59--8d6b--c564544c5a55-osd--block--c7c03dfd--a1b1--4182--9e61--bf264f293f2b
> > >> 253:0 0
> > >> 931.5G 0 lvm
> > >> └─YpB2cx-HlyU-VPqT-Abaz-Dutx-iMrz-Tty6o1 253:41
> 0
> > >> 931.5G 0 crypt
> > >
> > >
> > > # ceph osd dump | grep full_ratio
> > >
> > >> full_ratio 0.97
> > >> backfillfull_ratio 0.95
> > >> nearfull_ratio 0.9
> > >
> > >
> > > # ceph osd df | head
> > >>
> > >> ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP
> > >> META AVAIL %USE VAR PGS STATUS
> > >> 11 nvme 1.29999 1.00000 932 GiB 694 GiB 319 GiB 301
> > >> GiB 74 GiB 238 GiB 74.49 1.09 96 up
> > >
> > >
> > > Any ideas?
> > > Thanks!
> > > --
> > >
> > > Rodrigo Werle
> > > _______________________________________________
> > > ceph-users mailing list -- ceph-users@xxxxxxx
> > > To unsubscribe send an email to ceph-users-leave@xxxxxxx
> >
> >
> >
> > _______________________________________________
> > ceph-users mailing list -- ceph-users@xxxxxxx
> > To unsubscribe send an email to ceph-users-leave@xxxxxxx
> >
>
>
> --
>
> Rodrigo Magalhães Werle
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>
--
*Thanks and Regards!!*
*Neeraj Pratap Singh*
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
