We have some users of our software who have had a good experience with postgresql on zfs/zol. Two features which have proved useful are the native encryption (less fiddly than luks) and compression. Interestingly, many of our users are stuck with quite old and slow disks. Using compression (even together with encryption) on the slow disks gives quite a significant performance boost. Trading cpu for disk bandwidth. Also they often dont have infinite access to more disk, so the storage efficiency is welcomed.
We are interested in the snapshots but a little wary of potential data integrity issues.
We have a disturbance in our database structure (usually nightly) where large tables are dropped and recreated. snapshots of a gradually increasing size database probably work very well. I think these massive deletions probably make the snapshots quite heavy. Also creating a challenge for incremental backups, replication etc but that is another (not quite unrelated) issue.
Regards
Bob
On Tue, 26 Oct 2021 at 01:18, Benedict Holland <benedict.m.holland@xxxxxxxxx> wrote:
In my opinion, ext4 will solve any and all problems without a very deep understanding of file system architecture. In short, i would stick with ext4 unless you have a good reason not to. Maybe there is one. I have done this a long time and never thought twice about which file system should support my servers.On Mon, Oct 25, 2021, 6:01 PM Robert L Mathews <lists@xxxxxxxxxxxxx> wrote:On 10/25/21 1:40 PM, Mladen Gogala wrote:
> This is probably not the place
> to discuss the inner workings of snapshots, but it is worth knowing that
> snapshots drastically increase the IO rate on the file system - for
> every snapshot. That's where the slowness comes from.
I have recent anecdotal experience of this. I experiment with using
Btrfs for a 32 TB backup system that has five 8 TB spinning disks.
There's an average of 8 MBps of writes scattered around the disks, which
isn't super high, obviously.
The results were vaguely acceptable until I created a snapshot of it, at
which point it became completely unusable. Even having one snapshot
present caused hundreds of btrfs-related kernel threads to thrash in the
"D" state almost constantly, and it never stopped doing that even when
left for many hours.
I then experimented with adding a bcache layer on top of Btrfs to see if
it would help. I added a 2 TB SSD using bcache, partitioned as 1900 GB
read cache and 100 GB write cache. It made very little difference and
was still unusable as soon as a snapshot was taken.
I did play with the various btrfs and bcache tuning knobs quite a bit
and couldn't improve it.
Since that test was a failure, I then decided to try the same setup with
OpenZFS on a lark, with the same set of disks in a "raidz" array, with
the 2 TB SSD as an l2arc read cache (no write cache). It easily handles
the same load, even with 72 hourly snapshots present, with the default
settings. I'm actually quite impressed with it.
I'm sure that the RAID, snapshots and copy-on-write reduce the maximum
performance considerably, compared to ext4. But on the other hand, it
did provide the performance I expected to be possible given the setup.
Btrfs *definitely* didn't; I was surprised at how badly it performed.
--
Robert L Mathews, Tiger Technologies, http://www.tigertech.net/
