Re: Slow clients after git pull

David Turner <drakonstein@xxxxxxxxx> · Thu, 01 Mar 2018 14:29:21 +0000

Using CephFS for something like this is about the last thing I would do.  Does it need to be on a networked posix filesystem that can be mounted on multiple machines at the same time?  If so, then you're kinda stuck and we can start looking at your MDS hardware and see if there are any MDS settings that need to be configured differently for this to work.
If you don't NEED CephFS, then I would recommend utilizing an RBD for something like this.  Its limitation is only being able to be mapped to 1 server at a time, but that's decent enough for most failover scenarios for build setups.  If you need to failover, unmap it from the primary and map it to another server to resume workloads.
Hosting websites out of CephFS also seems counter-intuitive.  Have you looked at S3 websites?  RGW supports configuring websites out of a bucket that might be of interest.  Your RGW daemon configuration could easily become an HA website with an LB in front of them.

I'm biased here a bit, but I don't like to use networked filesystems unless nothing else can be worked out or the software using it is 3rd party and just doesn't support anything else.

On Thu, Mar 1, 2018 at 9:05 AM Daniel Carrasco <d.carrasco@xxxxxxxxx> wrote:
Hello,

I've tried to change a lot of things on configuration and use ceph-fuse but nothing makes it work better... When I deploy the git repository it becomes much slower until I remount the FS (just executing systemctl stop nginx && umount /mnt/ceph && mount -a && systemctl start nginx). It happen when the FS gets a lot of IO because when I execute Rsync I got the same problem.
I'm thinking about to downgrade to a lower version of ceph like for example jewel to see if works better. I know that will be deprecated soon, but I don't know what other tests I can do...

Greetings!!

2018-02-28 17:11 GMT+01:00 Daniel Carrasco <d.carrasco@xxxxxxxxx>:
Hello,
I've created a Ceph cluster with 3 nodes and a FS to serve a webpage. The webpage speed is good enough (near to NFS speed), and have HA if one FS die.
My problem comes when I deploy a git repository on that FS. The server makes a lot of IOPS to check the files that have to update and then all clients starts to have problems to use the FS (it becomes much slower).
In a normal usage the web takes about 400ms to load, and when the problem start it takes more than 3s. To fix the problem I just have to remount the FS on clients, but I can't remount the FS on every deploy...

While is deploying I see how the CPU on MDS is a bit higher, but when it ends the CPU usage goes down again, so look like is not a problem of CPU.

My config file is:
[global]
fsid = bf56854......e611c08
mon_initial_members = fs-01, fs-02, fs-03
mon_host = 10.50.0.94,10.50.1.216,10.50.2.52
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx

public network = 10.50.0.0/22
osd pool default size = 3

##
### OSD
##
[osd]
  osd_mon_heartbeat_interval = 5
  osd_mon_report_interval_max = 10
  osd_heartbeat_grace = 15
  osd_fast_fail_on_connection_refused = True
  osd_pool_default_pg_num = 128
  osd_pool_default_pgp_num = 128
  osd_pool_default_size = 2
  osd_pool_default_min_size = 2

##
### Monitores
##
[mon]
  mon_osd_min_down_reporters = 1

##
### MDS
##
[mds]
  mds_cache_memory_limit = 792723456
  mds_bal_mode = 1  

##
### Client
##
[client]
  client_cache_size = 32768
  client_mount_timeout = 30
  client_oc_max_objects = 2000
  client_oc_size = 629145600
  client_permissions = false
  rbd_cache = true
  rbd_cache_size = 671088640

My cluster and clients uses Debian 9 with latest ceph version (12.2.4). The clients uses kernel modules to mount the share, because are a bit faster than fuse modules. The deploy is done on one of the Ceph nodes, that have the FS mounted by kernel module too.
My cluster is not a high usage cluster, so have all daemons on one machine (3 machines with OSD, MON, MGR and MDS). All OSD has a copy of the data, only one MGR is active and two of the MDS are active with one on standby. The clients mount the FS using the three MDS IP addresses and just now don't have any request because is not published.

Someone knows what can be happening?, because all works fine (even on other cluster I did with an high load), but just deploy the git repository and all start to work very slow.

Thanks!!

-- 
_________________________________________

      Daniel Carrasco Marín
      Ingeniería para la Innovación i2TIC, S.L.
      Tlf:  +34 911 12 32 84 Ext: 223
      www.i2tic.com
_________________________________________

-- 
_________________________________________

      Daniel Carrasco Marín
      Ingeniería para la Innovación i2TIC, S.L.
      Tlf:  +34 911 12 32 84 Ext: 223
      www.i2tic.com
_________________________________________

_______________________________________________

ceph-users mailing list

ceph-users@xxxxxxxxxxxxxx

http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com