Hi Dan,
I can't speak for those specific Toshiba drives, but we have absolutely
seen very strange behavior (sometimes with cache enabled and sometimes
not) with different drives and firmwares over the years from various
manufacturers. There was one especially bad case from back in the
Inktank days, but my memory is a bit fuzzy. I think we were seeing
weird periodic commit latency spikes that grew worse over time. That
one might have been cache related. I believe we ended up doing a lot of
tests with blktrace and iowatcher to show the manufacturer what we were
seeing, but I don't recall if anything ever got fixed.
Mark
On 11/4/21 5:33 AM, Dan van der Ster wrote:
Hello Benoît, (and others in this great thread),
Apologies for replying to this ancient thread.
We have been debugging similar issues during an ongoing migration to
new servers with TOSHIBA MG07ACA14TE hdds.
We see a similar commit_latency_ms issue on the new drives (~60ms in
our env vs ~20ms for some old 6TB Seagates).
However, disabling the write cache (hdparm -W 0) made absolutely no
difference for us.
So we're wondering:
* Are we running the same firmware as you? (We have 0104). I wonder if
Toshiba has changed the implementation of the cache in the meantime...
* Is anyone aware of some HBA or other setting in the middle that
might be masking this setting from reaching the drive?
Best Regards,
Dan
On Wed, Jun 24, 2020 at 9:44 AM Benoît Knecht <bknecht@xxxxxxxxxxxxx> wrote:
Hi,
We have a Nautilus (14.2.9) Ceph cluster with two types of HDDs:
- TOSHIBA MG07ACA14TE [1]
- HGST HUH721212ALE604 [2]
They're all bluestore OSDs with no separate DB+WAL and part of the same pool.
We noticed that while the HGST OSDs have a commit latency of about 15ms, the Toshiba OSDs hover around 150ms (these values come from the `ceph_osd_commit_latency_ms` metric in Prometheus).
On paper, it seems like those drives have very similar specs, so it's not clear to me why we're seeing such a large difference when it comes to commit latency.
Has anyone had any experience with those Toshiba drives? Or looking at the specs, do you spot anything suspicious?
And if you're running a Ceph cluster with various disk brands/models, have you ever noticed some of them standing out when looking at `ceph_osd_commit_latency_ms`?
Thanks in advance for your feedback.
Cheers,
--
Ben
[1]: https://toshiba.semicon-storage.com/content/dam/toshiba-ss/asia-pacific/docs/product/storage/product-manual/eHDD-MG07ACA-Product-Manual.pdf
[2]: https://documents.westerndigital.com/content/dam/doc-library/en_us/assets/public/western-digital/product/data-center-drives/ultrastar-dc-hc500-series/data-sheet-ultrastar-dc-hc520.pdf
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
