A few thoughts from another ZFS backend user: use arcstats to look at your cache use over time and consider: Don’t mirror your cache drives, use them as 2x cache volumes to increase available cache. Add more RAM. Lots more RAM (if I’m reading that right and you have 32Gb ram per zfs server). Adjust ZFS’s max arc caching upwards if you have lots of RAM. Try more metadata caching & less content caching if you’re find heavy. compression on these volumes could help improve IO on the raidZ2s, but you’ll have to copy the data on with compression enabled if you didn’t already have it enabled. Different zStd levels are worth evaluating here. Read up on recordsize and consider if you would get any performance benefits from 64K or maybe something larger for your large data, depends on where the reads are being done. Use relatime or no atime tracking. Upgrade to ZFS 2.0.6 if you aren’t already at 2 or 2.1 For gluster, sounds like gluster 10 would be good for your use case. Without knowing what your workload is (VMs, gluster mounts, nfs mounts?), I don’t have much else on that level, but you can probably play with the cluster.read-hash-mode (try 3) to spread the read load out amongst your servers. Search the list archives for general performance hints too, server & client .event-threads are probably good targets, and the various performance.*threads may/may not help depending on how the volumes are being used. More details (zfs version, gluster version, volume options currently applied, more details on the workload) may help if others use similar setups. You may be getting into the area where you just need to get your environment setup to try some A/B testing with different options though. Good luck! -Darrell
|
________ Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://meet.google.com/cpu-eiue-hvk Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
