looks like you're using glusterfs ufo, for performance issue you'd better set object only to yes(/etc/swift/fs.conf), still you'd better not put too many files under one container, since for glusterfs ufo the container actually is a folder and put huge number of files under one dir obviously impact performance. (Not like orignal swift, the openstack swift hash files to a flat architecture) I have the following suggestions and hope it helps: 1 turn object only on 2 better to hash the files to several subfolders(if the object path contains / it will map to a folder in file system) 2012/12/4, Peter Portante <pportant at redhat.com>: > Hi Andrew, > > We have seen this kind of scaling issues with RHS 2.0 code base, which is > what appears to be the version you are using. We are currently in the > process of some changes that we think might alleviate this problem, or fix > it entirely, but we have not reached a solid footing to verify it yet. > > -peter > > > ----- Original Message ----- >> From: "Andrew Holway" <a.holway at syseleven.de> >> To: "Peter Portante" <pportant at redhat.com> >> Cc: gluster-users at gluster.org, kaleb at keithley.org >> Sent: Friday, November 30, 2012 4:09:44 PM >> Subject: Re: Gluster swift >> >> :D >> >> Maybe this is more useful. >> >> [root at cheese25 ~]# yum list installed | grep gluster >> glusterfs.x86_64 3.3.1-3.el6 >> @epel-glusterfs >> glusterfs-fuse.x86_64 3.3.1-3.el6 >> @epel-glusterfs >> glusterfs-server.x86_64 3.3.1-3.el6 >> @epel-glusterfs >> glusterfs-swift.noarch 3.3.1-3.el6 >> @epel-glusterfs-swift >> glusterfs-swift-account.noarch 3.3.1-3.el6 >> @epel-glusterfs-swift >> glusterfs-swift-container.noarch 3.3.1-3.el6 >> @epel-glusterfs-swift >> glusterfs-swift-object.noarch 3.3.1-3.el6 >> @epel-glusterfs-swift >> glusterfs-swift-plugin.noarch 3.3.1-3.el6 >> @epel-glusterfs-swift >> glusterfs-swift-proxy.noarch 3.3.1-3.el6 >> @epel-glusterfs-swift >> >> >> On Nov 29, 2012, at 9:03 PM, Andrew Holway wrote: >> >> > Hi, >> > >> > glusterfs.x86_64 3.3.1-3.el6 >> > @epel-glusterfs >> > glusterfs-fuse.x86_64 3.3.1-3.el6 >> > @epel-glusterfs >> > glusterfs-server.x86_64 3.3.1-3.el6 >> > @epel-glusterfs >> > glusterfs-swift.noarch 3.3.1-3.el6 >> > @epel-glusterfs-swift >> > glusterfs-swift-account.noarch 3.3.1-3.el6 >> > @epel-glusterfs-swift >> > glusterfs-swift-container.noarch 3.3.1-3.el6 >> > @epel-glusterfs-swift >> > glusterfs-swift-object.noarch 3.3.1-3.el6 >> > @epel-glusterfs-swift >> > glusterfs-swift-plugin.noarch 3.3.1-3.el6 >> > @epel-glusterfs-swift >> > glusterfs-swift-proxy.noarch 3.3.1-3.el6 >> > @epel-glusterfs-swift >> > glusterfs.i686 3.2.7-1.el6 >> > epel >> > glusterfs-debuginfo.x86_64 3.3.1-3.el6 >> > epel-glusterfs >> > glusterfs-devel.i686 3.2.7-1.el6 >> > epel >> > glusterfs-devel.x86_64 3.3.1-3.el6 >> > epel-glusterfs >> > glusterfs-geo-replication.x86_64 3.3.1-3.el6 >> > epel-glusterfs >> > glusterfs-rdma.x86_64 3.3.1-3.el6 >> > epel-glusterfs >> > glusterfs-swift-doc.noarch 3.3.1-3.el6 >> > epel-glusterfs-swift >> > glusterfs-vim.x86_64 3.2.7-1.el6 >> > epel >> > >> > Ta, >> > >> > Andrew >> > >> > On Nov 29, 2012, at 8:52 PM, Peter Portante wrote: >> > >> >> Hi Andrew, >> >> >> >> What version of Gluster are you using? >> >> >> >> -peter >> >> >> >> >> >> ----- Original Message ----- >> >>> From: "Andrew Holway" <a.holway at syseleven.de> >> >>> To: gluster-users at gluster.org >> >>> Cc: kaleb at keithley.org >> >>> Sent: Thursday, November 29, 2012 1:02:15 PM >> >>> Subject: Re: Gluster swift >> >>> >> >>> In addition, >> >>> >> >>> Requests to view the contents of containers that have been filled >> >>> in >> >>> this manner fail. >> >>> >> >>> [root at cheese25 free]# ls -l | wc >> >>> 3654 32879 193695 >> >>> [root at cheese25 free]# >> >>> >> >>> [root at bright60 lots_of_little_files]# curl --verbose -H >> >>> 'X-Auth-Token: AUTH_tk289d8ebe3ff44c97a9721970a4251f02' >> >>> https://cheese25:443/v1/AUTH_gv0/free/ -k >> >>> * About to connect() to cheese25 port 443 (#0) >> >>> * Trying 10.141.105.25... connected >> >>> * Connected to cheese25 (10.141.105.25) port 443 (#0) >> >>> * Initializing NSS with certpath: sql:/etc/pki/nssdb >> >>> * warning: ignoring value of ssl.verifyhost >> >>> * skipping SSL peer certificate verification >> >>> * SSL connection using TLS_RSA_WITH_AES_256_CBC_SHA >> >>> * Server certificate: >> >>> * subject: CN=cheese25,O=Default Company Ltd,L=Default City,C=XX >> >>> * start date: Nov 29 16:27:49 2012 GMT >> >>> * expire date: Dec 29 16:27:49 2012 GMT >> >>> * common name: cheese25 >> >>> * issuer: CN=cheese25,O=Default Company Ltd,L=Default City,C=XX >> >>>> GET /v1/AUTH_gv0/free/ HTTP/1.1 >> >>>> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 >> >>>> NSS/3.13.1.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2 >> >>>> Host: cheese25 >> >>>> Accept: */* >> >>>> X-Auth-Token: AUTH_tk289d8ebe3ff44c97a9721970a4251f02 >> >>>> >> >>> < HTTP/1.1 503 Internal Server Error >> >>> < Content-Type: text/html; charset=UTF-8 >> >>> < Content-Length: 0 >> >>> < Date: Thu, 29 Nov 2012 18:00:04 GMT >> >>> < >> >>> * Connection #0 to host cheese25 left intact >> >>> * Closing connection #0 >> >>> >> >>> But, non full volumes are ok :) >> >>> >> >>> [root at bright60 lots_of_little_files]# curl --verbose -H >> >>> 'X-Auth-Token: AUTH_tk289d8ebe3ff44c97a9721970a4251f02' >> >>> https://cheese25:443/v1/AUTH_gv0/stuff/ -k >> >>> * About to connect() to cheese25 port 443 (#0) >> >>> * Trying 10.141.105.25... connected >> >>> * Connected to cheese25 (10.141.105.25) port 443 (#0) >> >>> * Initializing NSS with certpath: sql:/etc/pki/nssdb >> >>> * warning: ignoring value of ssl.verifyhost >> >>> * skipping SSL peer certificate verification >> >>> * SSL connection using TLS_RSA_WITH_AES_256_CBC_SHA >> >>> * Server certificate: >> >>> * subject: CN=cheese25,O=Default Company Ltd,L=Default City,C=XX >> >>> * start date: Nov 29 16:27:49 2012 GMT >> >>> * expire date: Dec 29 16:27:49 2012 GMT >> >>> * common name: cheese25 >> >>> * issuer: CN=cheese25,O=Default Company Ltd,L=Default City,C=XX >> >>>> GET /v1/AUTH_gv0/stuff/ HTTP/1.1 >> >>>> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 >> >>>> NSS/3.13.1.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2 >> >>>> Host: cheese25 >> >>>> Accept: */* >> >>>> X-Auth-Token: AUTH_tk289d8ebe3ff44c97a9721970a4251f02 >> >>>> >> >>> < HTTP/1.1 200 OK >> >>> < X-Container-Object-Count: 15 >> >>> < X-Container-Bytes-Used: 0 >> >>> < Accept-Ranges: bytes >> >>> < Content-Length: 84 >> >>> < Content-Type: text/plain; charset=utf-8 >> >>> < Date: Thu, 29 Nov 2012 18:00:49 GMT >> >>> < >> >>> a1 >> >>> a10 >> >>> a100 >> >>> a1000 >> >>> a10000 >> >>> a1001 >> >>> a1002 >> >>> a1003 >> >>> a1004 >> >>> a1005 >> >>> a1006 >> >>> a1007 >> >>> a1008 >> >>> a1009 >> >>> a101 >> >>> * Connection #0 to host cheese25 left intact >> >>> * Closing connection #0 >> >>> >> >>> Thanks, >> >>> >> >>> Andrew >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> On Nov 29, 2012, at 6:38 PM, Andrew Holway wrote: >> >>> >> >>>> Hi, >> >>>> >> >>>> Im mooperd on irc. >> >>>> >> >>>> After lots of swearing and learning I think I am getting the >> >>>> hang >> >>>> of it. Today, I created 10000 files and squirted them into UFO. >> >>>> >> >>>> for file in $(ls); do curl -X PUT -T $file -H 'X-Auth-Token: >> >>>> AUTH_tk289d8ebe3ff44c97a9721970a4251f02' >> >>>> https://cheese25:443/v1/AUTH_gv0/new_container/ -k; done >> >>>> >> >>>> It works perfectly until you get to about 3500 files....then. >> >>>> >> >>>> </body> >> >>>> </html><html> >> >>>> <head> >> >>>> <title>201 Created</title> >> >>>> </head> >> >>>> <body> >> >>>> <h1>201 Created</h1> >> >>>> <br /><br /> >> >>>> >> >>>> >> >>>> >> >>>> </body> >> >>>> </html><html> >> >>>> <head> >> >>>> <title>201 Created</title> >> >>>> </head> >> >>>> <body> >> >>>> <h1>201 Created</h1> >> >>>> <br /><br /> >> >>>> >> >>>> >> >>>> >> >>>> </body> >> >>>> </html><html> >> >>>> <head> >> >>>> <title>404 Not Found</title> >> >>>> </head> >> >>>> <body> >> >>>> <h1>404 Not Found</h1> >> >>>> The resource could not be found.<br /><br /> >> >>>> >> >>>> >> >>>> This seems to be a hard limit for the number of files in a dir. >> >>>> Any >> >>>> ideas? >> >>>> >> >>>> Thanks, >> >>>> >> >>>> Andrew >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> >>> >> >>> >> >>> _______________________________________________ >> >>> Gluster-users mailing list >> >>> Gluster-users at gluster.org >> >>> http://supercolony.gluster.org/mailman/listinfo/gluster-users >> >>> >> > >> > >> > _______________________________________________ >> > Gluster-users mailing list >> > Gluster-users at gluster.org >> > http://supercolony.gluster.org/mailman/listinfo/gluster-users >> >> >> > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://supercolony.gluster.org/mailman/listinfo/gluster-users > -- ???
