There are 300M files right I am not counting wrong? With that file profile I would never use EC in first place. Maybe you can pack the files into tar archives or similar before migrating to gluster? It will take ages to heal a drive with that file count... On Mon, May 8, 2017 at 3:59 PM, Ingard Mevåg <ingard@xxxxxxxx> wrote: > With attachments :) > > 2017-05-08 14:57 GMT+02:00 Ingard Mevåg <ingard@xxxxxxxx>: >> >> Hi >> >> We've got 3 servers with 60 drives each setup with an EC volume running on >> gluster 3.10.0 >> The servers are connected via 10gigE. >> >> We've done the changes recommended here : >> https://bugzilla.redhat.com/show_bug.cgi?id=1349953#c17 and we're able to >> max out the network with the iozone tests referenced in the same ticket. >> >> However for small files we are getting 3-5 MB/s with the smallfile_cli.py >> tool. For instance: >> python smallfile_cli.py --operation create --threads 32 --file-size 100 >> --files 1000 --top /tmp/dfs-archive-001/ >> . >> . >> total threads = 32 >> total files = 31294 >> total data = 2.984 GB >> 97.79% of requested files processed, minimum is 90.00 >> 785.542908 sec elapsed time >> 39.837416 files/sec >> 39.837416 IOPS >> 3.890373 MB/sec >> . >> >> We're going to use these servers for archive purposes, so the files will >> be moved there and accessed very little. After noticing our migration tool >> performing very badly we did some analyses on the data actually being moved >> : >> >> Bucket 31808791 (16.27 GB) :: 0 bytes - 1.00 KB >> Bucket 49448258 (122.89 GB) :: 1.00 KB - 5.00 KB >> Bucket 13382242 (96.92 GB) :: 5.00 KB - 10.00 KB >> Bucket 13557684 (195.15 GB) :: 10.00 KB - 20.00 KB >> Bucket 22735245 (764.96 GB) :: 20.00 KB - 50.00 KB >> Bucket 15101878 (1041.56 GB) :: 50.00 KB - 100.00 KB >> Bucket 10734103 (1558.35 GB) :: 100.00 KB - 200.00 KB >> Bucket 17695285 (5773.74 GB) :: 200.00 KB - 500.00 KB >> Bucket 13632394 (10039.92 GB) :: 500.00 KB - 1.00 MB >> Bucket 21815815 (32641.81 GB) :: 1.00 MB - 2.00 MB >> Bucket 36940815 (117683.33 GB) :: 2.00 MB - 5.00 MB >> Bucket 13580667 (91899.10 GB) :: 5.00 MB - 10.00 MB >> Bucket 10945768 (232316.33 GB) :: 10.00 MB - 50.00 MB >> Bucket 1723848 (542581.89 GB) :: 50.00 MB - 9223372036.85 GB >> >> So it turns out we've got a very large number of very small files being >> written to this volume. >> I've attached the volume config and 2 profiling runs so if someone wants >> to take a look and maybe give us some hints in terms of what volume settings >> will be best for writing a lot of small files that would be much >> appreciated. >> >> kind regards >> ingard > > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users@xxxxxxxxxxx > http://lists.gluster.org/mailman/listinfo/gluster-users _______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://lists.gluster.org/mailman/listinfo/gluster-users
