Comments on: Memoization in Python: easier than what it should be

By: pkoch

pkoch — Mon, 25 Aug 2008 01:16:50 +0000

Illustrative example:

@memoize
def prt(*args,**kwargs):
    print "Calculated!"
    return (args,kwargs)

a = [1,2,3]
print prt(a,one=10, c='cee', d2='two')

a.append(1)
print prt(a, one=10, c='cee', d2='two')
print prt(a, one=10, d2='two', c='cee')

a.pop()
print prt(a, c='cee',one=10, d2='two')

By: pkoch

pkoch — Mon, 25 Aug 2008 00:05:31 +0000

I’ve updated my post to reflect many of the improvements presented on reddit comments.

Thank you, all! =)

By: pkoch

pkoch — Sat, 23 Aug 2008 01:49:20 +0000

John (et al.): Bug fixed! Thank you!

Jonas Gorauskas: I came up with this memoization code to cache some DB calls, as mentioned in the post. I had to run through 40k+ rows in Excel and do some auxiliary queries to construct insert statements. Those auxiliary queries would always return the same result for the same arguments. Memoization just popped into my mind.

By: Nobody In Particular

Nobody In Particular — Fri, 22 Aug 2008 21:44:14 +0000

list(kwargs.iteritems()).sort()
does not return a list. Instead .sort() sorts the list in place and returns None.

By: Andrew

Andrew — Fri, 22 Aug 2008 20:03:28 +0000

Isn’t list(kwargs.iteritems()).sort() going to have a value of None ?

By: Jonas Gorauskas

Jonas Gorauskas — Fri, 22 Aug 2008 19:36:43 +0000

I am new to python and in general very interested in how the concept of memoization applies to parsers and other areas. Could you provide a working example of the function above for context? Thanks in advance!

By: Peter Jones

Peter Jones — Fri, 22 Aug 2008 18:52:07 +0000

I went ahead and ran this out of curiosity as soon as I got my hands on a Python interpreter and it does indeed work recursively. The subtlety is that using memoize as a decorator redefines what the function refers to, so the recursive calls are indeed to the memoized version. Disregard my previous comment; your version is ten times more elegant than mine!

By: John

John — Fri, 22 Aug 2008 18:13:41 +0000

There is a bug on the 4th line:
>>> print list.sort.__doc__
L.sort(cmp=None, key=None, reverse=False) — stable sort *IN PLACE*;
cmp(x, y) -> -1, 0, 1

list.sort sorts in place, so the second item in your haxh tuple will always be None. Fix it by changing that line to:
haxh = tuple([args,sorted(kwargs.iteritems())])

sorted will return an iterator (which when put in to the tuple() is changed in to a list)

>>> args = (1,2,3,)
>>> kwargs={‘a’:5,’b':6,’aa’:9}
>>> tuple([args,sorted(kwargs.iteritems())])
((1, 2, 3), [('a', 5), ('aa', 9), ('b', 6)])

By: Peter Jones

Peter Jones — Fri, 22 Aug 2008 17:57:46 +0000

Very nice. This is almost a mirror image of the memoize function I wrote a few days ago, except I think yours is a bit more robust (mine has no kwargs support, for example).

Where mine shines is with recursive functions. With my first effort, if I tried to memoize the fibonacci function, it would calculate fib(n) the inefficient way and take a million years, then memoize the topmost result and return it. Unfortunately, memoizing a recursive function in this way is impossible, since the function body contains a reference to the non-memoized function.

My solution (which is not exactly great) required a change to the type of input that memoize takes. This is hard to explain and would certainly have been beyond my understanding a week ago so I’ll let my mediocre code speak for me: http://pcj600.googlepages.com/memoize.py

As you can see, the function fibonacci_constructor MAKES the fibonacci function USING a function passed as a parameter for its recursive calls. This way, memoize can step in and tell the function to do recursion using a different function – namely, the memoized version.

It’s a bit ugly and fails to opaque its functionality since it demands a second order function that you’d never write normally. What would be really nice is if we could do a find + replace on the function body, replacing all references to itself with references to the memoized version, so that there’d be no need to pass in the recursive function as a parameter. I bet this is possible with Lisp macros, but I’m still a beginner with Lisp …