On 12/17/2014 07:42 PM, Gregory Farnum wrote: > On Wed, Dec 17, 2014 at 8:35 AM, Wido den Hollander <wido@xxxxxxxx> wrote: >> Hi, >> >> Today I've been playing with CephFS and the morning started great with >> CephFS playing along just fine. >> >> Some information first: >> - Ceph 0.89 >> - Linux kernel 3.18 >> - Ceph fuse 0.89 >> - One Active MDS, one Standby >> >> This morning I could write a 10GB file like this using the kclient: >> $ dd if=/dev/zero of=10GB.bin bs=1M count=10240 conv=fsync >> >> That gave me 850MB/sec (all 10G network) and I could read the same file >> again with 610MB/sec. >> >> After writing to it multiple times it suddenly started to hang. >> >> No real evidence on the MDS (debug mds set to 20) or anything on the >> client. That specific operation just blocked, but I could still 'ls' the >> filesystem in a second terminal. >> >> The MDS was showing in it's log that it was checking active sessions of >> clients. It showed the active session of my single client. >> >> The client renewed it's caps and proceeded. > > Can you clarify this? I'm not quite sure what you mean. > I currently don't have the logs available. That was my problem when typing the original e-mail. >> I currently don't have any logs, but I'm just looking for a direction to >> be pointed towards. >> >> Any ideas? > > Well, now that you're on v0.89 you should explore the admin > socket...there are commands on the MDS to dump ops in flight (and > maybe to look at session states? I don't remember when that merged). Sage's pointer towards the kernel debugging and the new admin socket showed me that it were RADOS calls which were hanging. I investigated even further and it seems that this is not a CephFS problem, but a local TCP issue which is only triggered when using CephFS. At some point, which is still unclear to me, data transfer becomes very slow. The MDS doesn't seem to be able to update the journal and the client can't write to the OSDs anymore. It happened after I did some very basic TCP tuning (timestamp, rmem, wmem, sack, fastopen). Reverting back to the Ubuntu 14.04 defaults resolved it all and CephFS is running happily now. I'll dig some deeper to see why this system was affected by those changes. I applied these settings earlier on a RBD-only cluster without any problems. > -Greg > -- Wido den Hollander 42on B.V. Ceph trainer and consultant Phone: +31 (0)20 700 9902 Skype: contact42on -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
