IMPORTANT: Strawberry Perl October 2009 Beta 2 Portable has a big problem…

It turns out that Portable and the “vendor” changes butted heads and broke horns against each other, so that @INC was messed up and the portability code never got ran. At any rate, I’m making a patching .zip available that replaces 2 files in the Str…

Isotoma: Of Python, memcached and decorators: easy-peasy function caching

Following on from channam’s Memcached in 2 minutes, I’ve been working on a decorator to make life even simpler than this, memorised.
The most popular use-case for using memcached in Python apps is to cache the return value of a function or method. Over and over again you’ll find yourself doing something like:

mc = memcached.Client(['localhost:11211'])
def get_something(mc=mc):
    value = mc.get('something')
    if value is not None:
        return value
    else:
        # Do something else ...
        return 'hello world'

So in the interests of DRY

, why not reduce that down to a reusable pattern?
We can do this using a decorator, to replace our function with a function that instead checks for the existance of some sort of key, returns the value, otherwise delegates to the actual function for output (and then caches this output back into memcache).

The simplest way to add reusable implementations of patterns such as these onto existing functions in Python is to use a decorator, effectively a function wrapper that replaces (or returns a replacement) function in place of the original call.
How does this help? Well using this we can intercept a call to the decorated function, generate a signature for use as a key in memcache, check if the item is available in the cache, if so return that, otherwise grab the output from the function, pop it into memcache using our generated key, and finally return this value.

The biggest challenge we face is generating the unique (at least to that particularly function call) key to reference in memcache. The way I’ve found best to do this is to use a combination of module, class name and key attributes (if it’s a method of an instance or decorated with @classmethod), function name, and call arguments, in this form: <module>.<class>[<key attributes>]::(<arguments>)

In order to do this we need information about both the arguments, and if it’s a method we also need information about the class the method is bound to. Ordinarily these are both easy tasks using the inspect module, and the im_self attribute (funnily enough, referencing self) that bound methods contain.
However, a quick explanation of how the @<decorator> shortcut tag works, and indeed how decorators work, reveals a slight kink in this assumption. For example take the following:

class Test:
    @testfunc
    def test(self, arg1):
        pass

This simple bit of syntactic sugar is actually equal in functionality to the following (which is how you had to do it before Python 2.4, see PEP-318):

class Test:
    def test(self, arg1):
        pass
    test = testfunc(test)

To explain further, as I said before, decorators are just function which take other function instances and wrap around or replace them, and they are applied at the time of definition, and not as you might think at first calltime.
This is the reason the decorator function returns a function, as it is first called and instanced at this point, but any arguments to the function it is wrapping are passed at call time as a call to the function returned by the decorator.

If this confuses you, don’t worry, it’s not actually that important right now, except for the fact that because the function instance is not passed in at calltime, it means it is not bound, and loses it’s frame (e.g. how it is called, from which instance etc.).
This means we can no longer use im_self and several of the class functions in the inspect module. What we can do however is cheat and use the fact that bound methods always pass in their bound object instance or class instance as the first argument of a call, the ’self’ argument.

memorised.decorators.memorise() uses the following trick, to first check if ’self’ or ‘cls’ (the standard first parameter of an @classmethod) is there, and then using the *args list of passed in arguments to access the first parameter and grab either .__class__._name for an object instance’s class name or .__name__ for class instances:

# Get the list of arg names from func_code
argnames = fn.func_code.co_varnames[:fn.func_code.co_argcount]

. . .

if classmethod:
    # Get the class name from the cls argument
    class_name = args[0].__name__
else:
    # Get the class name from the self argument
    class_name = args[0].__class__.__name__

By then merging *args and **kwargs, we can build a hash key of this particular function call. Next just create a handy MD5 hash of this string using hashlib.md5, and then do our memcache, checks pretty much as above in the first example.

Using memorise() to replace the first example, we get:

mc = memcached.Client(['localhost:11211'])
@memorise(mc=mc)
def get_something():
    return 'hello world'

Notice I’m still defining the mc variable to be a memcached.Client instance, memorise() does handle do this itself, either by using the default localhost:11211 server setting or by accepting a list of servers (via an argument named ‘mc_servers’). However, this isn’t ideal as the memcached.Client instance would be created every time a function definition is decorated with memorise() (which could be lots), so best to pass an instance in each time.
Not to mention the fact using dependency injection like this over any other way of keeping the instance (e.g. singleton) is much cleaner.

Another point of interest is that we need to always include the call parenthesis even when not passing in any arguments to memorise(), e.g. @memorise() and not @memorise, as you would expect from decorators such as @classmethod. This is best of the way arguments are passed to both a decorator, and then to the function being decorated. There are workarounds for this problem, but up until now I haven’t seen one that can be used with class-based decorators (which memorise() is). I hope to solve this in a future release, so expect a follow up post on using optional arguments with class-based decorators sometime in the near future.

Finally I’ll finish with a more realistic example of using this to decorate methods on a Django Model:

class BlogUser(User):
    objects = UserManager()

    def __unicode__(self):
        return u'%s' self.get_full_name();

    @property
    @memorise(parent_keys=['id'], mc=mc)
    def posts(self):
        Post.objects.filter(creator=self)

And there you have it, any posts by that user will be cached for as long as memcached’s cachetime is set, or until memorised.utils.uncache() is used to clear down the cache for that method.

Geert Vanderkelen: Getting illegal dates from MySQL with Connector/Python

Today we received a bug report saying that we shouldn’t throw an exception but instead return what MySQLdb is returning. Bit research and MySQLdb is actually returning None for illegal dates: good!
There is now a fix (showing up soon) on Launchpad which will return dates as None where they are inserted as '0000-00-00'.
A few lines of Python:

..data = [   (datetime.now().date(),datetime.now()),   ('0000-00-00','0000-00-00 00:00:00'),   ('1000-00-00','9999-00-00 00:00:00'),   ]

for d in data:   stmt_insert = "INSERT INTO %s (c1,c2) VALUES (%%s,%%s)" % (tbl)   try:       cursor.execute(stmt_insert, d)   except (mysql.connector.errors.InterfaceError, TypeError) as e:       print "Failed inserting %s\nError: %s\n" % (d,e)

   if cursor.warnings:       print cursor.warnings..

The script outputs the following data, and notice also the warnings (SQL Mode set to NO_ZERO_IN_DATE,NO_ZERO_DATE):

[(u'Warning', 1265L, u"Data truncated for column 'c1' at row 1"), (u'Warning', 1264L, u"Out of range value for column 'c2' at row 1")][(u'Warning', 1265L, u"Data truncated for column 'c1' at row 1"), (u'Warning', 1264L, u"Out of range value for column 'c2' at row 1")](datetime.date(2009, 9, 30), datetime.datetime(2009, 9, 30, 15, 12, 23))(None, None)(None, None)

Another change we did today was returning a row as tuple, and rows as list of tuples.

Tip: use STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE SQL modes in MySQL 5.0 and later for new projects to make sure no illegal dates are inserted, or fix your application.

js-test-driver Phing task – Jakob Westhoff

In my need of a cross-browser javascript unit-testing framework, which can be run from the commandline I discovered js-test-driver. I soon realized to be able to productively use this framework I needed to integrate the test run into the used build process. As Phing is used for building I needed a simple Phing task to run js-test-driver tests.

Building Web Sites with Perl

Over on my other blog last night I wrote a piece about how building simple web sites has never been easier. I talked about how it’s really simple to use something like Wordpress or Drupal to build a web site that will suit the needs of many organisatio…

Greg Wilson: Net-Generation Instructors

The University of Michigan’s Carl Berger did some statistical analyses a while back of the faculty at his school. Long story short, we’re now seeing net-generation instructors as well as students. The future always arrives too soon, but in the wrong order.

Calvin Spealman: How To Invest in Poor Decision Makers

I couldn’t think of a better title that fit my “How To …” pattern. The point is, I wanted to make a response to the 37signals post that I found a little harsh. Sure, if you were able to build your company up without investors, that’s a great thing! It doesn’t make it a terrible thing to get a boost in the early stages or give you a license to insult people trying to pay the bills and put children through college.

Making great products is something a lot of us aspire to. Frankly, that simply isn’t all of us and there really are good developers out there who are still only in it for the money. I don’t know that is the case with Mint.com, but neither does anyone over at 37signals. Belittling them for taking a quick-cash option assumes a lot that may just be completely wrong about the intentions.

Now with a chunk of change, maybe the founders are planning to jump ship in a couple years and self-fund their real dreams.

On the matter of start-up investment itself, I do want to make some comments. Full disclosure: I’ve never been involved in a venture backed startup and I’m completely making this up from my own opinions about the world!

Pretend I’m from your bank and call you back after a loan application. You’re taking out a small business loan to build an additional room in your home for a new child. Everything looks good, and I’ve got a few questions to go over before approving the loan.

“I’d like to make you an offer for 10% ownership in exchange for this investment in your new venture,” I begin.

“What the hell are you talking about?” you quizzically respond.

“We’re talking about a significant investment in a potentially very profitable new enterprise. This child may well become a doctor or lawyer and if we’re going to help with the initial costs of raising this from the ground up, we all feel it is a reasonable request to share part ownership and benefit from that share over the lifetime of its profitability.”

“Umm… I thought I’d pay the loan back. Plus interest, even. I don’t even think I would own the child myself, technically. This is very strange…”

“Pay us back? A guarantee of interest accumulated as profit on our contribution? We’d rather take a chance of nothing or you paying us regularly for the rest of the child’s entire lifespan. Oh, and all of it’s children, of course.”

*click*

If we look at everything in our world from neutral eyes that aren’t used to our ways, things look weird. Does our investment model make sense, in this industry or any other? Why are any initial investments not setup as high-risk, high-interest loans, most likely with some initial grace period to await profitability? Of course, we could make some comments about the predatory loans and paying a cut of income for the rest of one’s life, but at least banks pretend that isn’t the deal upfront.

It isn’t like this isn’t an unusual idea. People get small business loans all the time. The tech sector seems to have skewed expectations that lead to dangerous and strange arrangements for funding. Still, I can’t help but wonder if there are independent investors who would or do take such a (relatively) altruistic route. I imagine something like a traditional investment round, mandating some grace period of 1-2 years, a repayment schedule requirement full reimbursement, and interest accumulation that tapers off after repayment of the initial investment.

The basic foundation could be extended to view all initial players as investors, be the investors of time or money. Invest your time to get a business started, helped by monetary investments from others, and after repaying yourself and those individuals the company becomes its own entity. It is not burdened with paying out profit shares to you or anyone else. Yes, you’ll still make your salary and you’ll still run the company, but it might be a better one for it.

Kiwi PyCon: Reminder To Register For Kiwi PyCon 2009

Reminder To Register For Kiwi PyCon 2009

Chris Miles: BlastOff 0.2 released

BlastOff 0.2 has now been released. BlastOff is a quickstart template for Pylons. It is a substitute for the simple boilerplate template that Pylons provides out-of-the-box. Use BlastOff to generate a Pylons skeleton configured with SQLAlchemy, mako, repoze.who, ToscaWidgets, TurboMail, WebFlash and (optionally) SchemaBot. The generated app is pre-configured with authentication, login and registration forms, and (optionally) email confirmation.

The great thing about Pylons is that it provides a basic web framework stack that assumes very little about what you want to do. The default application template contains no authentication, for example. It is up to the user to choose an authentication middleware that best fits their needs. Unfortunately, for newbies, it may not be obvious which middleware to choose. Pre-configured templates like BlastOff can help users get started with the author’s recommended choices of middleware components and configuration. BlastOff contains my own choice of best-of-breed components that I use in my own Pylons applications.

Not every template will be best for every developer, so I encourage others to create Pylons template packages. A few others already exist, I found out recently, so I created a Pylons Project Templates page in the Pylons wiki to list them. Take a look if you want to see how others are configuring their Pylons apps or just want to accelerate your Pylons development.

Cheers,
Chris Miles

Image by lecates @ flickr / CC BY-NC 2.0

Looking for a Perl related job?

If you are looking for a Perl related job I am sure you already know about
jobs.perl.org. You might say there
are not many job posts there or that there are no posts there from your
country. So here is a suggestions that might help both you and the ot…

« go back — keep looking »

Enterprise LAMP