On Wed, Sep 09, 2009 at 04:33:19PM -0400, Luke Macken wrote: > On Wed, Jul 01, 2009 at 05:50:12PM -0700, Toshio Kuratomi wrote: > > On 07/01/2009 05:39 PM, Mike McGrath wrote: > > > On Wed, 1 Jul 2009, Stephen John Smoogen wrote: > > > > > >> On Wed, Jul 1, 2009 at 6:08 PM, Mike McGrath<mmcgrath@xxxxxxxxxx> wrote: > > >>> On Wed, 1 Jul 2009, Stephen John Smoogen wrote: > > >>> > > >>>> On Wed, Jul 1, 2009 at 3:33 PM, Mike McGrath<mmcgrath@xxxxxxxxxx> wrote: > > >>>>> I'm not sure if we have any memcached experience on the list but I thought > > >>>>> I'd ask. Can anyone explain this: > > >>>>> > > >>>>> http://pastebin.ca/1481219 > > >>>>> > > >>>>> Notice how memcached1 has a much higher hit rate and memcached2 has a much > > >>>>> lower hit rate? > > >>>>> > > >>>> > > >>>> The time for memcached1 is 5x less than memcached2 being up. That can > > >>>> have an effect on caching as right after a system comes up its rates > > >>>> are usually much higher and then over time fall off (iirc). I think it > > >>>> would take bringing both up at the same time to figure out if there is > > >>>> a true disparity over caching. > > >>>> > > >>> > > >>> I thought that exact same thing :) > > >>> > > >>> memcahed1: > > >>> STAT uptime 9143 > > >>> STAT get_hits 311736 > > >>> STAT get_misses 11255 > > >>> > > >>> memcached2: > > >>> STAT uptime 9144 > > >>> STAT get_hits 49679 > > >>> STAT get_misses 11116 > > >>> > > >> > > >> Now that shows something not kosher. My guess is some app is not > > >> talking to both? What apps use memcached for what? > > >> > > > > > > I was just talking to ricky about this a bit in IRC. So here's the scoop. > > > > > > We've got mediawiki using memcached for a couple of things, including > > > session data (which is weird and wrong but fast). > > > > > > The recent addition to the group is Fedora community, specifically in it's > > > implementation of beaker. I'm going to get ahold of luke tomorrow to > > > verify and test some stuff but I think this line in the config: > > > > > > beaker.cache.url = memcached1;memcached > > > > > > I'm not sure how beaker reads that, but I suspect it might be only sending > > > information to memcached1 and ignoring memcached2 altogether. If this > > > theory holds it'd explain why memcached1 not only has a higher request > > > rate but also a higher hit rate because I suspect fedoracommunity requests > > > some of the same info over and over again compared to the wiki which > > > probably has a broader data pool it pulls from. > > > > > easy test: reverse that: > > > > beaker.cache.url = memcached2;memcached1 > > > > and see if the cache hit ratio reverses itself. > > > > -Toshio > > So, I've been poking at this a little bit lately, and I'm thinking there are > some problems with the way Fedora Community is utilizing it's Beaker cache & > memcached. > > The fact that something is wrong with the caching setup becomes obvious when > playing with the Bugzilla grid *should* cache the first 5 pages, and be very > snappy, as it is in my local instance. However, that is not the case, which > makes me think it's not hitting our memcached servers. > > I wrote up a little test script on app1:: > > from beaker.cache import CacheManager > > memcached1 = CacheManager(type='ext:memcached', url='memcached1', lock_dir='.') > memcached2 = CacheManager(type='ext:memcached', url='memcached2', lock_dir='.') > > def createfunc(*args): > print "createfunc(%s)" % (args,) > > def get_value(cache, value): > cache_1 = memcached1.get_cache(cache) > cache_2 = memcached2.get_cache(cache) > result1 = cache_1.get_value(key=value, createfunc=createfunc) > result2 = cache_2.get_value(key=value, createfunc=createfunc) > print "memcached1[%s][%s] = %s" % (cache, value, result1) > print "memcached2[%s][%s] = %s" % (cache, value, result2) > return result1, result2 > > get_value('fedoracommunity_alerts_global', 'today') > get_value('bodhi', 'dashboard_None') > > Which produces:: > > memcached1[fedoracommunity_alerts_global][today] = None > memcached2[fedoracommunity_alerts_global][today] = None > memcached1[bodhi][dashboard_None] = None > memcached2[bodhi][dashboard_None] = None > > I also tried using netcat to query for these by hand, to no avail. > > So, it looks like we need to look a bit deeper into what is going on here. > Either I'm Doing It Wrong with Beaker, or we're hitting a bug somewhere. I did a little more testing with this today. On app1 I was unable to get fedoracommunity to put any values in memcached, and I was not seeing any errors either. I tried this with both the mod_wsgi deployment, and running paster locally. I then ran `curl http://localhost:8080/apps/fedoracommunity.alerts/` to trigger the cache, and then ran the script I mentioned earlier to see if I could retrieve those values. I was able to reproduce this failure on a local RHEL5 VM. However, after upgrading to python-beaker-1.4 *and* restarting memcached, my script *worked*. (the app just silently returned None for all requests if I didn't restart memcached after upgrading Beaker) Looking at the ChangeLog for Beaker 1.4, I'm thinking this could have done the trick: * Fixed bug with CacheMiddleware overwriting configuration with default arguments despite prior setting. It'll be tough to test this new package in staging, as earlier today I couldn't get any of the cached apps to load due to koji/fas problems in staging. When we deploy this production, we'll need to make sure to restart our memcache daemons. luke _______________________________________________ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
