seth vidal wrote: > >>I think I have most of it covered. I've added two arguments to >>retrygrab/urlgrab for user and password. I've created the >>HTTPBasicAuthHandler(), and made an optional opener.add_handler along >>the lines of the Keepalive stuff that is already in there. >> >>Anyway, I think I have all of the back-end bits. But now I'm looking at >>the way arguments are passed around (and more importantly, how >>infrequently the serveridlist is being passed around), and I don't see >>how to get the user and password into the places where I need it (i.e. >>everywhere a retrygrab and urlgrab happen). It looks like I'm going to >>have to re-write large chunks of code to haul extra stuff around, when >>it seems like it ought to be very simple. > > it's actually not hard Somehow, I've managed to make it hard. ;-) > you can store the user/pass as a per-server variable in the config file > and they get hauled around in the config class. Got that part. > pretty much anything in the config class can get pushed in. > > Here is what I would suggest: > urlgrab inherits from retrygrab so just do it in one place > - urlgrab - add the user and pass there > - add the basic auth there > - we would be wise to make it smart enough to build up the url > correctly for ftp users so there is one interface > - check to see if conf.serveruser[serverid] and serverpass are not None > - if not then pass them into the grab() call and let grab() handle it > internally. So, I have access to conf.serveruser[serverid] within urlgrabber? Forgive my ignorance, but I don't see it coming in anywhere, or being used anywhere. This is where my confusion comes from I think. > internally I think urlgrab should look to see if it is http, ftp or file > urls > > if http - build up the basicauthhandler > if ftp put them inline ftp://user:pass@xxxxxxx/path/bar/ > if file ignore them and maybe even complain that the user is a moron. This I can handle. > Then that should be able to deal with it nicely and not require dragging > a lot of data around. Ok, so am I missing where this is being used in urlgrabber? As far as I can tell, everything that urlgrab uses comes in with the function arguments--and the server config information isn't among them. > Joe - you might be wise to smack Jack Neely in the head and see if you > two can merge your patch for this into his failover patches. It would > make my life easier b/c y'all are working on more or less the same > section(s) of code. I can do that, thought my changes are only going to be about 10 lines or so in urlgrabber.py and 4 in config.py, once I've figured out how to get the information I need into urlgrabber. (I feel like this is an absolute boneheaded thing to be confused about but, well, sometimes boneheaded is the best I can do.) BTW for anyone who understands Python better than me: Since opener may or may not already exist, because of this 'try' bit at the top of urlgrabber: opener = urllib2.build_opener(keepalive_handler) I need to do an 'if opener' check before I choose to either add_handler or build_opener with my auth data, correct? Thanks! -- Joe Cooper <joe@xxxxxxxxxxxxx> Web caching appliances and support. http://www.swelltech.com
