On Mon, Jul 13, 2015 at 11:00 PM, Simion Rad <Simion.Rad@xxxxxxxxx> wrote: > Hi , > > I'm running a small cephFS ( 21 TB , 16 OSDs having different sizes between > 400G and 3.5 TB ) cluster that is used as a file warehouse (both small and > big files). > Every day there are times when a lot of processes running on the client > servers ( using either fuse of kernel client) become stuck in D state and > when I run a strace of them I see them waiting in FUTEX_WAIT syscall. > The same issue I'm able to see on all OSD demons. > The ceph version I'm running is Firefly 0.80.10 both on clients and on > server daemons. > I use ext4 as osd filesystem. > Operating system on servers : Ubuntu 14.04 and kernel 3.13. > Operaing system on clients : Ubuntu 12.04 LTS with HWE option kernel 3.13 > The osd daemons are using RAID5 virtual disks (6 x 300 GB 10K RPM disks on > RAID controller Dell PERC H700 with 512MB BBU using write-back mode). > The servers which the ceph daemons are running on are also hosting KVM VMs ( > OpenStack Nova ). > Because of this unfortunate setup the performance is really bad, but at > least I shouldn't see as many locking issues (or shoud I ? ). > The only thing which temporarily improves the performance is restarting > every osd. After such a restart I see some processes on client machines > resume I/O but only for a couple of > hours, then the whole process must be repeated. > I cannot afford to run a setup without RAID because there isn't enough RAM > left for a couple of osd daemons. > > The ceph.conf settings I use : > > auth cluster required = cephx > auth service required = cephx > auth client required = cephx > filestore xattr use omap = true > osd pool default size = 2 > osd pool default min size = 1 > osd pool default pg num = 128 > osd pool default pgp num = 128 > public network = 10.71.13.0/24 > cluster network = 10.71.12.0/24 > > Did someone else experienced this kind of behaviour (stuck processes in > FUTEX_WAIT syscall) when running firefly release on Ubuntu 14.04 ? What's the output of "ceph -s" on your cluster? When your clients get stuck, is the cluster complaining about stuck ops on the OSDs? Are you running kernel clients on the same boxes as your OSDs? If I were to guess I'd imagine that you might just have overloaded your cluster and the FUTEX_WAIT is the clients waiting for writes to get acknowledged, but if restarting the OSDs brings everything back up for a few hours that might not be the case. -Greg _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |