Spinning Yarns

Suppose you wanted to inject a new function into a Python module, based on the value of an existing variable in that module. If that statement made no sense to you, or if you can imagine no reason why you might want to do such a thing, then you can safely skip the remainder of this post. Suffice it to say, then, that you want to do this thing: How might you go about it?

My first attempt looked something like this:

m = __import__("modulename")
 
expr = getattr(m, 'SPECIAL_VALUE')
def new_func(x):
  return stuff_involving(x, expr)
 
setattr(m, 'new_func', new_func)

This seems to be fine; the definition of new_func closes over the binding of expr visible at the time of its definition, and all is well. Unfortunately, a problem arises when you attempt to move this code into a loop, so that you can apply the same operation to a bunch of modules, e.g.,

for mod_name in module_list:
  m = __import__(mod_name)
 
  expr = getattr(m, 'SPECIAL_VALUE')
  def new_func(x):
    return stuff_involving(x, expr)
 
  setattr(m, 'new_func', new_func)

Now, the problem here is that each time the new_func definition is encountered in the loop, it captures the same binding of expr that all its predecessors had. The value changes from one iteration to the next, but the binding remains the same, with the result that all of the closures wind up with whatever value is left in that binding at the end of the loop. In general, this is not what you want?

So, how can you get around this? You could probably reach inside the function object and manually twiddle its environment, but I do not know of any reliable way to do that in Python, and besides, I consider that kind of behaviour to be morally dubious. In the end, the easiest solution I could find was to force each iteration to create a new binding for expr, by wrapping the definition inside another closure, e.g.,

for mod_name in module_list:
  m = __import__(mod_name)
 
  def t():
    expr = getattr(m, 'SPECIAL_VALUE') # A fresh binding each time
 
    def new_func(x):
      return stuff_involving(x, expr)
 
    return new_func
 
  setattr(m, 'new_func', t())

This now works as intended. Solutions like this fall under the category of “sleazy, yet effective”. The only thing that is really objectionable here is that there isn’t any way to make a first-class anonymous function in expression position. Python’s lambda would work in this simplified example, but if the body of new_func isn’t just a single expression, that won’t help. As far as I can tell, this is about the simplest solution to this problem, given that we assume the body of new_func doesn’t need to access other globals from inside the namespace of the target module.

If anyone knows a simpler way to do this, though, I’m all ears!

Filed by Michael at 18:19 under Programming, Technology
No Comments