On 05/06/15 19:51, Lionel Bouton wrote: > During normal operation Btrfs OSD volumes continue to behave in the same > way XFS ones do on the same system (sometimes faster/sometimes slower). > What is really slow though it the OSD process startup. I've yet to make > serious tests (umounting the filesystems to clear caches), but I've > already seen 3 minutes of delay reading the pgs. Example : > > 2015-05-05 16:01:24.854504 7f57c518b780 0 osd.17 22428 load_pgs > 2015-05-05 16:01:24.936111 7f57ae7fc700 0 > btrfsfilestorebackend(/var/lib/ceph/osd/ceph-17) destroy_checkpoint: > ioctl SNAP_DESTROY got (2) No such file or directory > 2015-05-05 16:01:24.936137 7f57ae7fc700 -1 > filestore(/var/lib/ceph/osd/ceph-17) unable to destroy snap > 'snap_1671188' got (2) No such file or directory > 2015-05-05 16:01:24.991629 7f57ae7fc700 0 > btrfsfilestorebackend(/var/lib/ceph/osd/ceph-17) destroy_checkpoint: > ioctl SNAP_DESTROY got (2) No such file or directory > 2015-05-05 16:01:24.991654 7f57ae7fc700 -1 > filestore(/var/lib/ceph/osd/ceph-17) unable to destroy snap > 'snap_1671189' got (2) No such file or directory > 2015-05-05 16:04:25.413110 7f57c518b780 0 osd.17 22428 load_pgs opened > 160 pgs > > The filesystem might not have reached its balance between fragmentation > and defragmentation rate at this time (so this may change) but mirrors > our initial experience with Btrfs where this was the first symptom of > bad performance. We've seen progress on this front. Unfortunately for us we had 2 power outages and they seem to have damaged the disk controller of the system we are testing Btrfs on: we just had a system crash. On the positive side this gives us an update on the OSD boot time. With a freshly booted system without anything in cache : - the first Btrfs OSD we installed loaded the pgs in ~1mn30s which is half of the previous time, - the second Btrfs OSD where defragmentation was disabled for some time and was considered more fragmented by our tool took nearly 10 minutes to load its pgs (and even spent 1mn before starting to load them). - the third Btrfs OSD which was always defragmented took 4mn30 seconds to load its pgs (it was considered more fragmented than the first and less than the second). My current assumption is that the defragmentation process we use can't handle large spikes of writes (at least when originally populating the OSD with data through backfills) but then can repair the damage on performance they cause at least partially (it's still slower to boot than the 3 XFS OSDs on the same system where loading pgs took 6-9 seconds). In the current setup the defragmentation is very slow to process because I set it up to generate very little load on the filesystems it processes : there may be room to improve. Best regards, Lionel _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |