Quick top-post reply.
We host websites on Gluster. It took us a bit of doing to get working quickly but here's roughly how we do it. I'm not saying this is *the* way to do, just a way that works for us:
1. Gluster volume with sharding enabled for storing filesystem files as a backing store for iSCSI.
2. Gluster servers also provide TGT iSCSI LUNS for multipath access.
3. Cluster of NFS servers connecting to iSCSI LUNS with
multipathd. We run this using Corosync to manage
redundancy/failover. Make sure you get this bit right. It's
probably the hardest bit. It's probably also the bit most likely
ripe for improvement.
4. VMs running web servers access the NFS data for websites. Each server runs an FS-Cache volume for the NFS mounted volumes.
5. OPCache/FPM for the dynamic stuff (assuming PHP here).
Your underlying storage needs to be relatively fast/capable of course.
We have our VM volumes on sharded Gluster too.
My theory in setting it up this way was to avoid the small-file scenario that Gluster is not very good at for one it is good at, sharded big files.
Anyway, hope that helps a little.
Ronny
Hi All,
We've got a glusterfs cluster that houses some php web sites.
This is generally considered a bad idea and we can see why.
With performance.nl-cache on it actually turns out to be very reasonable, however, with this turned of performance is roughly 5x worse. meaning a request that would take sub 500ms now takes 2500ms. In other cases we see far, far worse cases, eg, with nl-cache takes ~1500ms, without takes ~30s (20x worse).
So why not use nl-cache? Well, it results in readdir reporting files which then fails to open with ENOENT. The cache also never clears even though the configuration says nl-cache entries should only be cached for 60s. Even for "ls -lah" in affected folders you'll notice ???? mark entries for attributes on files. If this recovers in a reasonable time (say, a few seconds, sure).
# gluster volume info
Type: Replicate
Volume ID: cbe08331-8b83-41ac-b56d-88ef30c0f5c7
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Options Reconfigured:
performance.nl-cache: on
cluster.readdir-optimize: on
config.client-threads: 2
config.brick-threads: 4
config.global-threading: on
performance.iot-pass-through: on
storage.fips-mode-rchecksum: on
cluster.granular-entry-heal: enable
cluster.data-self-heal-algorithm: full
cluster.locking-scheme: granular
client.event-threads: 2
server.event-threads: 2
transport.address-family: inet
nfs.disable: on
cluster.metadata-self-heal: off
cluster.entry-self-heal: off
cluster.data-self-heal: off
cluster.self-heal-daemon: on
server.allow-insecure: on
features.ctime: off
performance.io-cache: on
performance.cache-invalidation: on
features.cache-invalidation: on
performance.qr-cache-timeout: 600
features.cache-invalidation-timeout: 600
performance.io-cache-size: 128MB
performance.cache-size: 128MB
Are there any other recommendations short of abandon all hope of redundancy and to revert to a single-server setup (for the web code at least). Currently the cost of the redundancy seems to outweigh the benefit.
Glusterfs version 10.2. With patch for --inode-table-size, mounts happen with:
/usr/sbin/glusterfs --acl --reader-thread-count=2 --lru-limit=524288 --inode-table-size=524288 --invalidate-limit=16 --background-qlen=32 --fuse-mountopts=nodev,nosuid,noexec,noatime --process-name fuse --volfile-server=127.0.0.1 --volfile-id=gv_home --fuse-mountopts=nodev,nosuid,noexec,noatime /home
Kind Regards,
Jaco
________
Community Meeting Calendar:
Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
Ronny Adsetts Technical Director Amazing Internet Ltd, London t: +44 20 8977 8943 w: www.amazinginternet.com Registered office: 85 Waldegrave Park, Twickenham, TW1 4TJ Registered in England. Company No. 4042957
________ Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://meet.google.com/cpu-eiue-hvk Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
