Hi Christian,
On 8 September 2015 at 14:02, Christian Balzer <chibi@xxxxxxx> wrote:
Indeed. But first a word about the setup where I'm seeing this.
These are 2 mailbox server clusters (2 nodes each), replicating via DRBD
over Infiniband (IPoIB at this time), LSI 3008 controller. One cluster
with the Samsung DC SSDs, one with the Intel S3610.
2 of these chassis to be precise:
https://www.supermicro.com/products/system/2U/2028/SYS-2028TP-DC0FR.cfm
We are using the same box, but DC0R (no infiniband) so I guess not surprising we're seeing the same thing happening.
Of course latest firmware and I tried this with any kernel from Debian
3.16 to stock 4.1.6.
With nobarrier I managed to trigger the error only once yesterday on the
DRBD replication target, not the machine that actual has the FS mounted.
Usually I'd be able to trigger quite a bit more often during those tests.
So this morning I updated the firmware of all S3610s on one node and
removed the nobarrier flag. It took a lot of punishment, but eventually
this happened:
---
Sep 8 10:43:47 mbx09 kernel: [ 1743.358329] sd 0:0:1:0: attempting task abort! scmd(ffff880fdc85b680)
Sep 8 10:43:47 mbx09 kernel: [ 1743.358339] sd 0:0:1:0: [sdb] CDB: Write(10) 2a 00 0e 9a fb b8 00 00 08 00
Sep 8 10:43:47 mbx09 kernel: [ 1743.358345] scsi target0:0:1: handle(0x000a), sas_address(0x4433221101000000), phy(1)
Sep 8 10:43:47 mbx09 kernel: [ 1743.358348] scsi target0:0:1: enclosure_logical_id(0x5003048019e98d00), slot(1)
Sep 8 10:43:47 mbx09 kernel: [ 1743.387951] sd 0:0:1:0: task abort: SUCCESS scmd(ffff880fdc85b680)
---
Note that on the un-patched node (DRBD replication target) I managed to
trigger this bug 3 times in the same period.
So unless Intel has something to say (and given that this happens with
Samsungs as well), I'd still look beady eyed at LSI/Avago...
Yes, I think there may be more than one issue here. The reduction in occurrences seems to prove there is an issue fixed by the Intel firmware, but something is still happening.
Once I have updated the firmware on the drives on one of our hosts tonight, hopefully I can get some more statistics and pinpoint if there is another issue specifically with the LSI3008.
I'd be interested to know if the combination of nobarriers and the updated firmware fixes the issue.
Regards,
Richard
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
