-----Forwarded Message-----
From: Ryan Tomayko <rtomayko at naeblis.cx>
To: Seth Vidal <skvidal at phy.duke.edu>
Cc: Michael Stenner <mstenner at ece.arizona.edu>
Subject: Yum usage of URLGrabber
Date: Sat, 14 Feb 2004 23:58:48 -0500
Seth,
I wanted to respond to a couple of questions you've asked about
urlgrabber wrt usage in yum. I figure now would be a good time to ramble
a bit on usage. Maybe we can fold some of this into a doc for urlgrabber
somewhere.
There have been a couple of questions like "How do I set XXX option?"
and "Is XXX option a module level option or instance level option?".
Basically, the way things are working right now, all kwargs can be set
and overridden at the module level, URLGrabber instance level, and/or
request (urlXXX function call) level. The kwargs are documented in the
module level __doc__ in grabber.py and cover pretty much everything that
is configurable. I'm going to run through various usage scenarios and
then suggest what I think might be the best option for yum.
First, the grabber.URLGrabber class is pretty much the center of the
urlgrabber universe now. The module level urlgrab, urlopen, and urlread
are one line functions that call a default/global instance of URLGrabber
(default_grabber). The default values for all kwargs can be found in the
grabber.URLGrabberOptions class (look in __init__). Here are the current
values for your convenience:
self.progress_obj = None
self.throttle = 1.0
self.bandwidth = 0
self.retry = None
self.retrycodes = [-1,2,4,5,6,7]
self.checkfunc = None
self.copy_local = 0
self.close_connection = 0
self.range = None
self.user_agent = 'urlgrabber/%s' % VERSION
self.keepalive = 1
self.proxies = None
To set any of these at the module level--meaning for module level urlXXX
functions--you would use something like this:
<<
import URLGrabber.grabber as urlgrabber
urlgrabber.default_grabber.opts.throttle = 0.3
urlgrabber.default_grabber.opts.retry = 10
urlgrabber.default_grabber.opts.user_agent = 'Yum!'
>>
At this point, calling the following types of functions will use the
defaults you set. You can override any of the defaults using kwargs to
the method calls.
# use all defaults
urlgrabber.urlgrab('http://bla.com/file', 'localfile')
# override some defaults
urlgrabber.urlgrab('http://bla.com/file', 'localfile', throttle=0.5,
retry=7)
One thing to note is that setting options on the default_grabber
instance does not effect new instances of the URLGrabber class. New
instances always start with the defaults listed above.
So, now forget all that about module level functions and defaults
because I think the best thing to do in yum would be to create your own
URLGrabber instance and use it in place of the module level functions.
What you want to do here is have config.py or yummain.py or some other
module that is loaded really early create a URLGrabber instance, set
defaults, and then use that from the rest of yum. For example, there are
some urlgrabber related options set in yum.conf so let's say we create
this in config.py as follows:
<<
import URLGrabber.grabber as urlgrabber
# create the URLGrabber
yumgrabber = urlgrabber.URLGrabber(user_agent='Yum', progress_obj=yumpo)
# read some values from config file:
yumgrabber.opts.throttle = config.get('throttle', 1.0)
yumgrabber.opts.retry = config.get('retries', 1.0)
yumgrabber.opts.keepalive = config.get('keepalive', 1.0)
>>
Next, you want to import this "yumgrabber" instance into other modules
instead of using the URLGrabber module directly. So, if yumgrabber was a
global in config.py, from yummain.py you would do something like:
<<
from config import yumgrabber
yumgrabber.urlgrab(url, filename)
# or
yumgrabber.urlgrab(url, filename, range=(200,500))
>>
The benefit of this approach, IMO, is that you have your own instance of
URLGrabber that does not effect (and is not effected by) the module
level options.
Also, I wanted to note after running through some of those examples that
these are all hypothetical wrt yum code. i.e. config.py probably isn't
the right place to create the URLGrabber instance, I used it as an
first-guess example. I still need to take a peek at what you have
cooking on the HEAD for yum. I might be able to make some real
suggestions at that point.
Sorry if this was a little disconnected. I wanted to get that out before
you got to deep into integrating this stuff into yum. Also, some of this
might change based on what Michael wants to do after he sweeps the code.
I think the interface is pretty close to what was envisioned but I don't
want to make any promises.
Hope this helps,
- Ryan
