Hi, I have a odd case with SMR disks in a Ceph cluster. Before I continue, yes, I am fully aware of SMR and Ceph not playing along well, but there is something happening which I'm not able to fully explain. On a 2x replica cluster with 8TB Seagate SMR disks I can write with about 30MB/sec to each disk using a simple RADOS bench: $ rados bench -t 1 $ time rados put 1GB.bin Both ways I found out that the disk can write at that rate. Now, when I start a benchmark with 32 threads it writes fine. Not super fast, but it works. After 15 minutes or so various disks go to 100% busy and just stay there. These OSDs are being marked as down and some even commit suicide due to threads timing out. Stopping the RADOS bench and starting the OSDs again resolves the situation. I am trying to explain what's happening. I'm aware that SMR isn't very good at Random Writes. To partially overcome this there are Intel DC 3510s in there as Journal SSDs. Can anybody explain why this 100% busy pops up after 15 minutes or so? Obviously it would the best if BlueStore had SMR support, but for now it's just Filestore with XFS on there. Wido _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |