On Wed, Jul 01, 2009 at 05:50:12PM -0700, Toshio Kuratomi wrote: > On 07/01/2009 05:39 PM, Mike McGrath wrote: > > On Wed, 1 Jul 2009, Stephen John Smoogen wrote: > > > >> On Wed, Jul 1, 2009 at 6:08 PM, Mike McGrath<mmcgrath@xxxxxxxxxx> wrote: > >>> On Wed, 1 Jul 2009, Stephen John Smoogen wrote: > >>> > >>>> On Wed, Jul 1, 2009 at 3:33 PM, Mike McGrath<mmcgrath@xxxxxxxxxx> wrote: > >>>>> I'm not sure if we have any memcached experience on the list but I thought > >>>>> I'd ask. Can anyone explain this: > >>>>> > >>>>> http://pastebin.ca/1481219 > >>>>> > >>>>> Notice how memcached1 has a much higher hit rate and memcached2 has a much > >>>>> lower hit rate? > >>>>> > >>>> > >>>> The time for memcached1 is 5x less than memcached2 being up. That can > >>>> have an effect on caching as right after a system comes up its rates > >>>> are usually much higher and then over time fall off (iirc). I think it > >>>> would take bringing both up at the same time to figure out if there is > >>>> a true disparity over caching. > >>>> > >>> > >>> I thought that exact same thing :) > >>> > >>> memcahed1: > >>> STAT uptime 9143 > >>> STAT get_hits 311736 > >>> STAT get_misses 11255 > >>> > >>> memcached2: > >>> STAT uptime 9144 > >>> STAT get_hits 49679 > >>> STAT get_misses 11116 > >>> > >> > >> Now that shows something not kosher. My guess is some app is not > >> talking to both? What apps use memcached for what? > >> > > > > I was just talking to ricky about this a bit in IRC. So here's the scoop. > > > > We've got mediawiki using memcached for a couple of things, including > > session data (which is weird and wrong but fast). > > > > The recent addition to the group is Fedora community, specifically in it's > > implementation of beaker. I'm going to get ahold of luke tomorrow to > > verify and test some stuff but I think this line in the config: > > > > beaker.cache.url = memcached1;memcached > > > > I'm not sure how beaker reads that, but I suspect it might be only sending > > information to memcached1 and ignoring memcached2 altogether. If this > > theory holds it'd explain why memcached1 not only has a higher request > > rate but also a higher hit rate because I suspect fedoracommunity requests > > some of the same info over and over again compared to the wiki which > > probably has a broader data pool it pulls from. > > > easy test: reverse that: > > beaker.cache.url = memcached2;memcached1 > > and see if the cache hit ratio reverses itself. > > -Toshio So, I've been poking at this a little bit lately, and I'm thinking there are some problems with the way Fedora Community is utilizing it's Beaker cache & memcached. The fact that something is wrong with the caching setup becomes obvious when playing with the Bugzilla grid *should* cache the first 5 pages, and be very snappy, as it is in my local instance. However, that is not the case, which makes me think it's not hitting our memcached servers. I wrote up a little test script on app1:: from beaker.cache import CacheManager memcached1 = CacheManager(type='ext:memcached', url='memcached1', lock_dir='.') memcached2 = CacheManager(type='ext:memcached', url='memcached2', lock_dir='.') def createfunc(*args): print "createfunc(%s)" % (args,) def get_value(cache, value): cache_1 = memcached1.get_cache(cache) cache_2 = memcached2.get_cache(cache) result1 = cache_1.get_value(key=value, createfunc=createfunc) result2 = cache_2.get_value(key=value, createfunc=createfunc) print "memcached1[%s][%s] = %s" % (cache, value, result1) print "memcached2[%s][%s] = %s" % (cache, value, result2) return result1, result2 get_value('fedoracommunity_alerts_global', 'today') get_value('bodhi', 'dashboard_None') Which produces:: memcached1[fedoracommunity_alerts_global][today] = None memcached2[fedoracommunity_alerts_global][today] = None memcached1[bodhi][dashboard_None] = None memcached2[bodhi][dashboard_None] = None I also tried using netcat to query for these by hand, to no avail. So, it looks like we need to look a bit deeper into what is going on here. Either I'm Doing It Wrong with Beaker, or we're hitting a bug somewhere. luke _______________________________________________ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
