I recommend that people check their disk controller caches/batteries as well as checking for subfolder splitting on filestore (which is the only option on Jewel). The former leads to high await, the later contributes to blocked requests.
On Sun, Mar 25, 2018, 3:36 AM Sam Huracan <nowitzki.sammy@xxxxxxxxx> wrote:
Thank you all.1. Here is my ceph.conf file:2. Here is result from ceph -s:root@ceph1:/etc/ceph# ceph -scluster 31154d30-b0d3-4411-9178-0bbe367a5578health HEALTH_OKmonmap e3: 3 mons at {ceph1=10.0.30.51:6789/0,ceph2=10.0.30.52:6789/0,ceph3=10.0.30.53:6789/0}election epoch 18, quorum 0,1,2 ceph1,ceph2,ceph3osdmap e2473: 63 osds: 63 up, 63 inflags sortbitwise,require_jewel_osdspgmap v34069952: 4096 pgs, 6 pools, 21534 GB data, 5696 kobjects59762 GB used, 135 TB / 194 TB avail4092 active+clean2 active+clean+scrubbing2 active+clean+scrubbing+deepclient io 36096 kB/s rd, 41611 kB/s wr, 1643 op/s rd, 1634 op/s wr3. We use 1 SSD for journaling 7 HDD (/dev/sdi), I set 16GB for each journal, here is result from ceph-disk list command:/dev/sda :/dev/sda1 ceph data, active, cluster ceph, osd.0, journal /dev/sdi1/dev/sdb :/dev/sdb1 ceph data, active, cluster ceph, osd.1, journal /dev/sdi2/dev/sdc :/dev/sdc1 ceph data, active, cluster ceph, osd.2, journal /dev/sdi3/dev/sdd :/dev/sdd1 ceph data, active, cluster ceph, osd.3, journal /dev/sdi4/dev/sde :/dev/sde1 ceph data, active, cluster ceph, osd.4, journal /dev/sdi5/dev/sdf :/dev/sdf1 ceph data, active, cluster ceph, osd.5, journal /dev/sdi6/dev/sdg :/dev/sdg1 ceph data, active, cluster ceph, osd.6, journal /dev/sdi7/dev/sdh :/dev/sdh3 other, LVM2_member/dev/sdh1 other, vfat, mounted on /boot/efi/dev/sdi :/dev/sdi1 ceph journal, for /dev/sda1/dev/sdi2 ceph journal, for /dev/sdb1/dev/sdi3 ceph journal, for /dev/sdc1/dev/sdi4 ceph journal, for /dev/sdd1/dev/sdi5 ceph journal, for /dev/sde1/dev/sdi6 ceph journal, for /dev/sdf1/dev/sdi7 ceph journal, for /dev/sdg14. With iostat, we just run "iostat -x 2", /dev/sdi is journal SSD, /dev/sdh is OS Disk, and the rest is OSD Disks.root@ceph1:/etc/ceph# lsblkNAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTsda 8:0 0 3.7T 0 disk└─sda1 8:1 0 3.7T 0 part /var/lib/ceph/osd/ceph-0sdb 8:16 0 3.7T 0 disk└─sdb1 8:17 0 3.7T 0 part /var/lib/ceph/osd/ceph-1sdc 8:32 0 3.7T 0 disk└─sdc1 8:33 0 3.7T 0 part /var/lib/ceph/osd/ceph-2sdd 8:48 0 3.7T 0 disk└─sdd1 8:49 0 3.7T 0 part /var/lib/ceph/osd/ceph-3sde 8:64 0 3.7T 0 disk└─sde1 8:65 0 3.7T 0 part /var/lib/ceph/osd/ceph-4sdf 8:80 0 3.7T 0 disk└─sdf1 8:81 0 3.7T 0 part /var/lib/ceph/osd/ceph-5sdg 8:96 0 3.7T 0 disk└─sdg1 8:97 0 3.7T 0 part /var/lib/ceph/osd/ceph-6sdh 8:112 0 278.9G 0 disk├─sdh1 8:113 0 512M 0 part /boot/efi└─sdh3 8:115 0 278.1G 0 part├─hnceph--hdd1--vg-swap (dm-0) 252:0 0 59.6G 0 lvm [SWAP]└─hnceph--hdd1--vg-root (dm-1) 252:1 0 218.5G 0 lvm /sdi 8:128 0 185.8G 0 disk├─sdi1 8:129 0 16.6G 0 part├─sdi2 8:130 0 16.6G 0 part├─sdi3 8:131 0 16.6G 0 part├─sdi4 8:132 0 16.6G 0 part├─sdi5 8:133 0 16.6G 0 part├─sdi6 8:134 0 16.6G 0 part└─sdi7 8:135 0 16.6G 0 partCould you give me some idea to continue check?_______________________________________________2018-03-25 12:25 GMT+07:00 Budai Laszlo <laszlo.budai@xxxxxxxxx>:could you post the result of "ceph -s" ? besides the health status there are other details that could help, like the status of your PGs., also the result of "ceph-disk list" would be useful to understand how your disks are organized. For instance with 1 SSD for 7 HDD the SSD could be the bottleneck.
>From the outputs you gave us we don't know which are the spinning disks and which is the ssd (looking at the numbers I suspect that sdi is your SSD). we also don't kow what parameters were you using when you've ran the iostat command.
Unfortunately it's difficult to help you without knowing more about your system.
Kind regards,
Laszlo
On 24.03.2018 20:19, Sam Huracan wrote:
> This is from iostat:
>
> I'm using Ceph jewel, has no HW error.
> Ceph health OK, we've just use 50% total volume.
>
>
> 2018-03-24 22:20 GMT+07:00 <ceph@xxxxxxxxxx <mailto:ceph@xxxxxxxxxx>>:
>
> I would Check with Tools like atop the utilization of your Disks also. Perhaps something Related in dmesg or dorthin?
>
> - Mehmet
>
> Am 24. März 2018 08:17:44 MEZ schrieb Sam Huracan <nowitzki.sammy@xxxxxxxxx <mailto:nowitzki.sammy@xxxxxxxxx>>:
>
>
> Hi guys,
> We are running a production OpenStack backend by Ceph.
>
> At present, we are meeting an issue relating to high iowait in VM, in some MySQL VM, we see sometime IOwait reaches abnormal high peaks which lead to slow queries increase, despite load is stable (we test with script simulate real load), you can see in graph.
> https://prnt.sc/ivndni
>
> MySQL VM are place on Ceph HDD Cluster, with 1 SSD journal for 7 HDD. In this cluster, IOwait on each ceph host is about 20%.
> https://prnt.sc/ivne08
>
>
> Can you guy help me find the root cause of this issue, and how to eliminate this high iowait?
>
> Thanks in advance.
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx>
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com>
>
>
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
