On 07/02/2021 22:19, Marc wrote:
I was wondering if someone could post a config for haproxy. Is there something specific to configure? Like binding clients to a specific backend server, client timeouts, security specific to rgw etc.
Ours is templated out by ceph-ansible; to try and condense out just the
interesting bits:
global
nbthread 24
#this plus 8 rados handles and 600 civetweb threads lets us use all the
#CPU on our RGW systems
defaults
timeout connect 60s
timeout client 2m
timeout server 2m
# give clients chance to benefit from keepalive; but don't
# let idle connections linger
timeout http-keep-alive 1s
frontend listen_https
mode http
option forwardfor
bind :443 ssl crt /etc/ceph/rgwtls.pem
stick-table type ip size 1m expire 1h store conn_cur
tcp-request content track-sc0 src
# tcp-request is processed before http-request
# these soft and hard limits templated
tcp-request content reject if { sc_conn_cur(0) gt 170 }
http-request set-nice 1000 if { sc_conn_cur(0) gt 113 }
default_backend rgw_servers
backend rgw_servers
balance roundrobin
#Use our server if it's got connections spare
use-server sto-rgw-1 if { srv_conn(sto-rgw-1) le 341 }
server sto-rgw-1 172.27.50.8:8443 check ssl verifyhost
cog.sanger.ac.uk ca-file ca-certificates.crt fall 5 inter 2000 rise 2
maxconn 341 weight 0
#Otherwise, prefer the two network-local servers
server sto-rgw-2 172.27.50.9:8443 check ssl verifyhost
cog.sanger.ac.uk ca-file ca-certificates.crt fall 5 inter 2000 rise 2
maxconn 341 weight 100
server sto-rgw-3 172.27.50.10:8443 check ssl verifyhost
cog.sanger.ac.uk ca-file ca-certificates.crt fall 5 inter 2000 rise 2
maxconn 341 weight 100
#Finally, the more remote options
server sto-rgw-4 172.27.50.136:8443 check ssl verifyhost
cog.sanger.ac.uk ca-file ca-certificates.crt fall 5 inter 2000 rise 2
maxconn 341 weight 5
server sto-rgw-5 172.27.50.137:8443 check ssl verifyhost
cog.sanger.ac.uk ca-file ca-certificates.crt fall 5 inter 2000 rise 2
maxconn 341 weight 5
server sto-rgw-6 172.27.50.138:8443 check ssl verifyhost
cog.sanger.ac.uk ca-file ca-certificates.crt fall 5 inter 2000 rise 2
maxconn 341 weight 5
The aim is to use all available CPU on the RGWs at peak load, but to
also try and prevent one user overwhelming the service for everyone else
- hence the dropping of idle connections and soft (and then hard) limits
on per-IP connections.
Regards,
Matthew
--
The Wellcome Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
