Hello, Thanks to "partner" on IRC who told me about this (quite big) problem. Apparently in a distributed setup once a brick fills up you start getting write failures. Is there a way to work around this? I would have thought gluster would check for free space before writing to a brick. It's very easy to test, I created a distributed volume from 2 uneven bricks and started to write to it; as one of them got full I started getting write failures. This is how I tested: ======================= [root at localhost gluster1]# for i in `seq 1 20`; do df -h /mnt/gluster1/; dd if=/dev/zero of=16_$i bs=16M count=1; done Filesystem Size Used Avail Use% Mounted on 192.168.192.5:/test 291M 16M 276M 6% /mnt/gluster1 1+0 records in 1+0 records out 16777216 bytes (17 MB) copied, 0.115813 s, 145 MB/s Filesystem Size Used Avail Use% Mounted on 192.168.192.5:/test 291M 32M 260M 11% /mnt/gluster1 1+0 records in 1+0 records out 16777216 bytes (17 MB) copied, 0.140746 s, 119 MB/s Filesystem Size Used Avail Use% Mounted on 192.168.192.5:/test 291M 48M 243M 17% /mnt/gluster1 1+0 records in 1+0 records out 16777216 bytes (17 MB) copied, 0.0905644 s, 185 MB/s Filesystem Size Used Avail Use% Mounted on 192.168.192.5:/test 291M 64M 227M 22% /mnt/gluster1 1+0 records in 1+0 records out 16777216 bytes (17 MB) copied, 0.088424 s, 190 MB/s Filesystem Size Used Avail Use% Mounted on 192.168.192.5:/test 291M 80M 211M 28% /mnt/gluster1 1+0 records in 1+0 records out 16777216 bytes (17 MB) copied, 0.0876373 s, 191 MB/s Filesystem Size Used Avail Use% Mounted on 192.168.192.5:/test 291M 96M 195M 33% /mnt/gluster1 1+0 records in 1+0 records out 16777216 bytes (17 MB) copied, 0.0890243 s, 188 MB/s Filesystem Size Used Avail Use% Mounted on 192.168.192.5:/test 291M 112M 179M 39% /mnt/gluster1 1+0 records in 1+0 records out 16777216 bytes (17 MB) copied, 0.0853196 s, 197 MB/s Filesystem Size Used Avail Use% Mounted on 192.168.192.5:/test 291M 128M 163M 45% /mnt/gluster1 1+0 records in 1+0 records out 16777216 bytes (17 MB) copied, 0.0923682 s, 182 MB/s Filesystem Size Used Avail Use% Mounted on 192.168.192.5:/test 291M 145M 147M 50% /mnt/gluster1 1+0 records in 1+0 records out 16777216 bytes (17 MB) copied, 0.0861475 s, 195 MB/s Filesystem Size Used Avail Use% Mounted on 192.168.192.5:/test 291M 161M 131M 56% /mnt/gluster1 dd: writing `16_10': No space left on device dd: closing output file `16_10': No space left on device Filesystem Size Used Avail Use% Mounted on 192.168.192.5:/test 291M 170M 121M 59% /mnt/gluster1 dd: writing `16_11': No space left on device dd: closing output file `16_11': No space left on device Filesystem Size Used Avail Use% Mounted on 192.168.192.5:/test 291M 170M 121M 59% /mnt/gluster1 dd: opening `16_12': No space left on device Filesystem Size Used Avail Use% Mounted on 192.168.192.5:/test 291M 170M 121M 59% /mnt/gluster1 1+0 records in 1+0 records out 16777216 bytes (17 MB) copied, 0.0842241 s, 199 MB/s Filesystem Size Used Avail Use% Mounted on 192.168.192.5:/test 291M 186M 105M 64% /mnt/gluster1 1+0 records in 1+0 records out 16777216 bytes (17 MB) copied, 0.102602 s, 164 MB/s Filesystem Size Used Avail Use% Mounted on 192.168.192.5:/test 291M 202M 89M 70% /mnt/gluster1 dd: opening `16_15': No space left on device Filesystem Size Used Avail Use% Mounted on 192.168.192.5:/test 291M 202M 89M 70% /mnt/gluster1 1+0 records in 1+0 records out 16777216 bytes (17 MB) copied, 0.0866302 s, 194 MB/s Filesystem Size Used Avail Use% Mounted on 192.168.192.5:/test 291M 219M 73M 76% /mnt/gluster1 1+0 records in 1+0 records out 16777216 bytes (17 MB) copied, 0.0898677 s, 187 MB/s Filesystem Size Used Avail Use% Mounted on 192.168.192.5:/test 291M 235M 57M 81% /mnt/gluster1 dd: opening `16_18': No space left on device Filesystem Size Used Avail Use% Mounted on 192.168.192.5:/test 291M 235M 57M 81% /mnt/gluster1 1+0 records in 1+0 records out 16777216 bytes (17 MB) copied, 0.126375 s, 133 MB/s Filesystem Size Used Avail Use% Mounted on 192.168.192.5:/test 291M 251M 41M 87% /mnt/gluster1 dd: opening `16_20': No space left on device ======================= -- Sent from the Delta quadrant using Borg technology! Nux! www.nux.ro
