Re: [Bluestore] Some of my osd's uses BlueFS slow storage for db - why?

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi Sage, Thanks for chiming in.  I can't image how busy you are.

Sorry guys.  I reprovisioned the offending osd right after this email and a conversation on #ceph.  I do have the output from '/usr/bin/ceph daemon osd.5 perf dump | /usr/bin/jq .' saved.  I'll be happy to add it to the issue tracker.

/C


On Fri, Mar 22, 2019 at 7:01 PM Sage Weil <sage@xxxxxxxxxxxx> wrote:
I have a ticket open for this:

        http://tracker.ceph.com/issues/38745

Please comment there with the health warning you're seeing and any other
details so we can figure out why it's happening.

I wouldn't reprovision those OSDs yet, until we know why it happens. 
Also, it's likely that ceph-bluestore-tool can sort it out be
adding/removing the db volume.

Thanks!
sage


On Fri, 22 Mar 2019, Mazzystr wrote:

> I am also seeing BlueFS spill since updating to Nautilus.  I also see high
> slow_used_bytes and slow_total_bytes metrics.  It sure looks to me that the
> only solution is to zap and rebuilt the osd.  I had to manually check 36
> osds some of them traditional processes and some containerized.  The lack
> of tooling here is underwhelming...  As soon as I rebuilt the osd the
> "BlueFS spill..." warning went away.
>
> I use 50Gb db partitions on an nvme with 3 or 6 Tb spinning disks.  I don't
> understand the spillover.
>
>
> On Fri, Feb 15, 2019 at 12:33 PM David Turner <drakonstein@xxxxxxxxx> wrote:
>
> > The answer is probably going to be in how big your DB partition is vs how
> > big your HDD disk is.  From your output it looks like you have a 6TB HDD
> > with a 28GB Blocks.DB partition.  Even though the DB used size isn't
> > currently full, I would guess that at some point since this OSD was created
> > that it did fill up and what you're seeing is the part of the DB that
> > spilled over to the data disk.  This is why the official recommendation
> > (that is quite cautious, but cautious because some use cases will use this
> > up) for a blocks.db partition is 4% of the data drive.  For your 6TB disks
> > that's a recommendation of 240GB per DB partition.  Of course the actual
> > size of the DB needed is dependent on your use case.  But pretty much every
> > use case for a 6TB disk needs a bigger partition than 28GB.
> >
> > On Thu, Feb 14, 2019 at 11:58 PM Konstantin Shalygin <k0ste@xxxxxxxx>
> > wrote:
> >
> >> Wrong metadata paste of osd.73 in previous message.
> >>
> >>
> >> {
> >>
> >>      "id": 73,
> >>      "arch": "x86_64",
> >>      "back_addr": "10.10.10.6:6804/175338",
> >>      "back_iface": "vlan3",
> >>      "bluefs": "1",
> >>      "bluefs_db_access_mode": "blk",
> >>      "bluefs_db_block_size": "4096",
> >>      "bluefs_db_dev": "259:22",
> >>      "bluefs_db_dev_node": "nvme2n1",
> >>      "bluefs_db_driver": "KernelDevice",
> >>      "bluefs_db_model": "INTEL SSDPEDMD400G4                     ",
> >>      "bluefs_db_partition_path": "/dev/nvme2n1p11",
> >>      "bluefs_db_rotational": "0",
> >>      "bluefs_db_serial": "CVFT4324002Q400BGN  ",
> >>      "bluefs_db_size": "30064771072",
> >>      "bluefs_db_type": "nvme",
> >>      "bluefs_single_shared_device": "0",
> >>      "bluefs_slow_access_mode": "blk",
> >>      "bluefs_slow_block_size": "4096",
> >>      "bluefs_slow_dev": "8:176",
> >>      "bluefs_slow_dev_node": "sdl",
> >>      "bluefs_slow_driver": "KernelDevice",
> >>      "bluefs_slow_model": "TOSHIBA HDWE160 ",
> >>      "bluefs_slow_partition_path": "/dev/sdl2",
> >>      "bluefs_slow_rotational": "1",
> >>      "bluefs_slow_size": "6001069199360",
> >>      "bluefs_slow_type": "hdd",
> >>      "bluefs_wal_access_mode": "blk",
> >>      "bluefs_wal_block_size": "4096",
> >>      "bluefs_wal_dev": "259:22",
> >>      "bluefs_wal_dev_node": "nvme2n1",
> >>      "bluefs_wal_driver": "KernelDevice",
> >>      "bluefs_wal_model": "INTEL SSDPEDMD400G4                     ",
> >>      "bluefs_wal_partition_path": "/dev/nvme2n1p12",
> >>      "bluefs_wal_rotational": "0",
> >>      "bluefs_wal_serial": "CVFT4324002Q400BGN  ",
> >>      "bluefs_wal_size": "1073741824",
> >>      "bluefs_wal_type": "nvme",
> >>      "bluestore_bdev_access_mode": "blk",
> >>      "bluestore_bdev_block_size": "4096",
> >>      "bluestore_bdev_dev": "8:176",
> >>      "bluestore_bdev_dev_node": "sdl",
> >>      "bluestore_bdev_driver": "KernelDevice",
> >>      "bluestore_bdev_model": "TOSHIBA HDWE160 ",
> >>      "bluestore_bdev_partition_path": "/dev/sdl2",
> >>      "bluestore_bdev_rotational": "1",
> >>      "bluestore_bdev_size": "6001069199360",
> >>      "bluestore_bdev_type": "hdd",
> >>      "ceph_version": "ceph version 12.2.10
> >> (177915764b752804194937482a39e95e0ca3de94) luminous (stable)",
> >>      "cpu": "Intel(R) Xeon(R) CPU E5-2609 v4 @ 1.70GHz",
> >>      "default_device_class": "hdd",
> >>      "distro": "centos",
> >>      "distro_description": "CentOS Linux 7 (Core)",
> >>      "distro_version": "7",
> >>      "front_addr": "172.16.16.16:6803/175338",
> >>      "front_iface": "vlan4",
> >>      "hb_back_addr": "10.10.10.6:6805/175338",
> >>      "hb_front_addr": "172.16.16.16:6805/175338",
> >>      "hostname": "ceph-osd5",
> >>      "journal_rotational": "0",
> >>      "kernel_description": "#1 SMP Tue Aug 14 21:49:04 UTC 2018",
> >>      "kernel_version": "3.10.0-862.11.6.el7.x86_64",
> >>      "mem_swap_kb": "0",
> >>      "mem_total_kb": "65724256",
> >>      "os": "Linux",
> >>      "osd_data": "/var/lib/ceph/osd/ceph-73",
> >>      "osd_objectstore": "bluestore",
> >>      "rotational": "1"
> >> }
> >> _______________________________________________
> >> ceph-users mailing list
> >> ceph-users@xxxxxxxxxxxxxx
> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >>
> > _______________________________________________
> > ceph-users mailing list
> > ceph-users@xxxxxxxxxxxxxx
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux