Hi, On Tue, 2011-08-30 at 16:25 +0200, Davide Brunato wrote: > Hi, > > Steven Whitehouse wrote: > > Hi, > > > > On Tue, 2011-08-30 at 12:51 +0200, Davide Brunato wrote: > >> Hello, > >> > >> I've a Red Hat 5.7 2-node cluster for electronic mail services where the mailboxes (maildir format) > >> are stored on GFS2 volume. The volume contains about 7500000 files for ~740 GB of disk space > >> occupation. Previously the mailboxes were on a GFS1 volume, and I migrated to GFS2 when we changed > >> the SAN storage system. > >> > >> Due to incremental backups that have become extremely slow (about 41-42H) after the migration from > >> GFS to GFS2, I checked the configuration/tuning of the cluster and volume mount options, with the > >> help of Red Hat support, but the optimizations (<gfs_controld plock_rate_limit="0"/>, mount with > >> noatime and nodiratime) don't have significantly accelerated the incremental backups. > >> > > You don't mention how fast the backups were before... > > > > Full: about 17-18 H > Incremental: about 7-8 H > > but it was 15th month ago, with a less number of e-mails (4.5-5 millions). > So from that, it seems that the current times have not suddenly got worse, but have just been extended due to a greater amount of data to be processed. > > The issue is most likely just that GFS2 caches more data (on average) > > than GFS does. If you access that data from the node where that data is > > cached, then its faster, if you try to access that same data from > > another node then it will be slower. > > > > The issue therefore is ensuring that you divide your backup amoung nodes > > in such a way as the backup will mostly be working only with the working > > set of files on that node. > > > > Either that, or as you've mentioned below, use your array's snapshot > > capability to avoid this issue. > > > > > >> So I tried another backup strategy, using the snaphot feature of our SAN storage system, doing > >> backups outside the cluster environment. I use the snapshots of the GSF2 on another server (also > >> with RHEL 5.7) mounting the volume as a local (not clustered) filesystem: > >> > >> /var/mailboxes type gfs2 (rw,noatime,nodiratime,lockproto=lock_nolock,localflocks,localcaching) > >> > >> The duration of full backups are slightly better (from 24-25H to 21-22H of duration) and the > >> incremental backup are "acceptable" (about 9H). But the speed is still low in comparison to backups > >> of Ext3 filesystems, particularly for incremental backups. > >> > > It is bound to be a bit slower, ext3 can make some optimisations which > > are just not possible in a clustered environment. On the other hand, if > > it is taking that length of time to snapshot the GFS2 volume on the > > array, then that seems to be to be an issue with the array rather than > > the filesystem. > > > >> I've notice that the glocks are still used, also when I mount a snapshot of the mailbox GFS2 > >> partition as a local filesystem: > >> > > The glocks are pretty low overhead, when clustering is not involved. > >> > >> # mount -t gfs2 /dev/mapper/posta_mbox_disk_vg-posta_mbox_disk_lvol1 /var/mailboxes -o > >> lockproto=lock_nolock,noatime,nodiratime > >> # time cp -Rp /var/mailboxes/prova* /var/tmp/test/ > >> > >> real 2m5.648s > >> user 0m0.311s > >> sys 0m13.243s > >> # rm -Rf /var/tmp/test/* > >> # time cp -Rp /var/mailboxes/prova* /var/tmp/test/ > >> > >> real 0m10.946s > >> user 0m0.254s > >> sys 0m10.634s > >> > > This is a nice demonstration of the effects of accessing cached vs. > > uncached data. > > > >> # cat /proc/slabinfo | grep gloc > >> gfs2_glock 35056 35064 424 9 1 : tunables 54 27 8 : slabdata 3896 3896 > >> 0 > >> > > That is a pretty small number of glocks. > > > > Yes, because they are related only to the copy of my test files. > > >> Is there a way to exclude the use of the glocks, or them are necessary to access the partition, even > >> if mounted as local filesystem? > >> > >> Thanks > >> > >> Davide Brunato > >> > > Yes, they are required, but the overhead is pretty small, so I doubt > > that is the real issue here, > > > > Steve. > > > > I tried other tests with local GFS2 partitions (on new test LUNs), both in a cluster node and in the > external server (the server that I use for backups from snapshots, but in this case I used a test > LUN assigned directly to this system). > > The overhead is still heavy. For example: > > # mkfs.gfs2 -j 4 -p lock_nolock /dev/mapper/posta_test_disk_vg-posta_test_disk_lvol1 > # mount -t gfs2 /dev/mapper/posta_test_disk_vg-posta_test_disk_lvol1 /mnt -o noatime,nodiratime > # time cp -Rp /mnt/* /var/tmp/test > > real 1m9.386s > user 0m0.265s > sys 0m10.647s > # rm -Rf /var/tmp/test/* > # time cp -Rp /mnt/* /var/tmp/test > > real 0m9.397s > user 0m0.250s > sys 0m9.036s > > Excluding issues on the storage system (we have other TBs in Ext3 for filesystems without issues, > and the storage is an enterprise class system), the overhead appears greater than expected and not > correlated with the snapshot mechanism of the storage system. Similar results are obtained if I > create a local GFS2 test volume on a node of the cluster. > > Is this behaviour of the GFS2 local volumes an anomaly? > > Thank you > > Davide > No, you appear to be measuring the difference between reading all the data off disk and reading all the data from the cache, assuming that you didn't flush the caches or umount between the two tests. So thats basically what I'd expect to see - that cached data is read much faster, Steve. > > > > -- > > Linux-cluster mailing list > > Linux-cluster@xxxxxxxxxx > > https://www.redhat.com/mailman/listinfo/linux-cluster > -- Linux-cluster mailing list Linux-cluster@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/linux-cluster
