Perl on Android, progressing…

First, the meeting:

I arrived a bit late but hadn’t missed much. Gabor was in the middle of explaining the benefits of getting Perl on Android. He went over a lot of API examples, showing code and explaining and doing a damn goo job at it!

After that…

Sean Reifschneider: Python syslog patch to log exceptions

I’ve just completed a patch to the Python syslog module to add the
method enable_exception_logging(). It sets up a sys.excepthook so that
unhandled exceptions are logged to syslog. By default, it chains to the
existing excepthook.

So, once this code gets accepted, you will be able to have exceptions
logged by doing: “import syslog; syslog.enable_exception_logging()”.

For Python software that runs from cron or init or Apache, it can be
very useful to capture the exceptions in a persistent location.

I’d appreciate some reviews of the code, it’s in Issue8214.

April 2010 Strawberry Perl … finally!

Been a long few days, but Strawberry Perl April 2010 (why do I want to keep typing 2009?) is out.Get it at http://strawberryperl.com/Two major things have been added:1) There is a version of Strawberry Perl that can relocate on installation – the 5.12….

PyCon Podcast: rapid multi-purpose testing (#81) (PyCon 2010)

Want to rapidly test code, javascript or documents on different Python Interpreters and versions? This talk gives an overview on new py.test features of the last year. I’ll discuss the minimal-boilerplate testing and new plugin model and give an overview on which testing needs are covered so far, for example:

* run tests written for nose or unittest
* ad-hoc distribute tests to local or remote Python environments
* test compatibility with Python3
* run javascript unit-tests in real browsers
* coverage testing
* django-specific testing

I’ll conclude with upcoming features regarding testing-in-the-cloud and plans on more test tool convergence. (URL: http://pytest.org)

Chris Leary: Efficiency of list comprehensions

I’m psyched about the awesome comments on my previous entry, Python by example: list comprehensions. Originally this entry was just a response to those comments, but people who stumbled across this entry on the interwebz found the response format too confusing, so I’ve restructured it for posterity.

Efficiency of the more common usage

Let’s look at the efficiency of list comprehensions in the more common usage, where the comprehension’s list result is actually relevant (or, in compiler-speak, live-out).

Using the following program, you can see the time spent in each implementation and the corresponding bytecode sequence:

import dis
import inspect
import timeit
 
 
programs = dict(
    loop="""
result = []
for i in range(20):
    result.append(i * 2)
""",
   loop_faster="""
result = []
add = result.append
for i in range(20):
    add(i * 2)
""",
    comprehension='result = [i * 2 for i in range(20)]',
)
 
 
for name, text in programs.iteritems():
    print name, timeit.Timer(stmt=text).timeit()
    code = compile(text, '<string>', 'exec')
    dis.disassemble(code)

loop 11.1495118141
  2           0 BUILD_LIST               0
              3 STORE_NAME               0 (result)
 
  3           6 SETUP_LOOP              37 (to 46)
              9 LOAD_NAME                1 (range)
             12 LOAD_CONST               0 (20)
             15 CALL_FUNCTION            1
             18 GET_ITER
        >>   19 FOR_ITER                23 (to 45)
             22 STORE_NAME               2 (i)
 
  4          25 LOAD_NAME                0 (result)
             28 LOAD_ATTR                3 (append)
             31 LOAD_NAME                2 (i)
             34 LOAD_CONST               1 (2)
             37 BINARY_MULTIPLY
             38 CALL_FUNCTION            1
             41 POP_TOP
             42 JUMP_ABSOLUTE           19
        >>   45 POP_BLOCK
        >>   46 LOAD_CONST               2 (None)
             49 RETURN_VALUE
loop_faster 8.36096310616
  2           0 BUILD_LIST               0
              3 STORE_NAME               0 (result)
 
  3           6 LOAD_NAME                0 (result)
              9 LOAD_ATTR                1 (append)
             12 STORE_NAME               2 (add)
 
  4          15 SETUP_LOOP              34 (to 52)
             18 LOAD_NAME                3 (range)
             21 LOAD_CONST               0 (20)
             24 CALL_FUNCTION            1
             27 GET_ITER
        >>   28 FOR_ITER                20 (to 51)
             31 STORE_NAME               4 (i)
 
  5          34 LOAD_NAME                2 (add)
             37 LOAD_NAME                4 (i)
             40 LOAD_CONST               1 (2)
             43 BINARY_MULTIPLY
             44 CALL_FUNCTION            1
             47 POP_TOP
             48 JUMP_ABSOLUTE           28
        >>   51 POP_BLOCK
        >>   52 LOAD_CONST               2 (None)
             55 RETURN_VALUE
comprehension 7.08145213127
  1           0 BUILD_LIST               0
              3 DUP_TOP
              4 STORE_NAME               0 (_[1])
              7 LOAD_NAME                1 (range)
             10 LOAD_CONST               0 (20)
             13 CALL_FUNCTION            1
             16 GET_ITER
        >>   17 FOR_ITER                17 (to 37)
             20 STORE_NAME               2 (i)
             23 LOAD_NAME                0 (_[1])
             26 LOAD_NAME                2 (i)
             29 LOAD_CONST               1 (2)
             32 BINARY_MULTIPLY
             33 LIST_APPEND
             34 JUMP_ABSOLUTE           17
        >>   37 DELETE_NAME              0 (_[1])
             40 STORE_NAME               3 (result)
             43 LOAD_CONST               2 (None)
             46 RETURN_VALUE

List comprehensions perform better here because you don’t need to load the append attribute off of the list (loop program, bytecode 28) and call it as a function (loop program, bytecode 38). Instead, in a comprehension, a specialized LIST_APPEND bytecode is generated for a fast append onto the result list (comprehension program, bytecode 33).

In the loop_faster program, you avoid the overhead of the append attribute lookup by hoisting it out of the loop and placing the result in a fastlocal (bytecode 9-12), so it loops more quickly; however, the comprehension uses a specialized LIST_APPEND bytecode instead of incurring the overhead of a function call, so it still trumps.

Using list comprehensions for side effects

I want to address a point that was brought up in the previous entry as to the efficiency of for loops versus list comprehensions when used purely for side effects, but I’ll discuss the subjective bit first, since that’s the least sciency part.

Readability

Simple test – if you did need the result would the comprehension be easily understood? If the answer is yes then removing the assignment on the left hand side doesn’t magically make it less readable…

—Michael Foord

First of all, thanks to Michael for his excellent and thought provoking comment!

My response is that removing the use of the result does indeed make it less readable, precisely because you’re using a result-producing control flow construct where the result is not needed. I suppose I’m positing that it’s inherently confusing to do that with your syntax: there’s a looping form that doesn’t produce a result, so that should be used instead. It’s expressing your semantic intention via syntax.

For advanced Pythonistas it’s easy for figure out what’s going on at a glance, but comprehension-as-loop definitely has a "there’s more than one way to do it" smell about it, which also makes it less amenable to people learning the language.

With a viable comprehension-as-loop option, every time a user goes to write a loop that doesn’t require a result they now ask themselves, "Can I fit this into the list comprehension form?" Those mental branches are, to me, what "one way to do it" is designed to avoid. When I read Perl code, I take "mental exceptions" all the time because the author didn’t use the construct that I would have used in the same situation. Minimizing that is a good thing, so I maintain that "no result needed" should automatically imply a loop construct.

Efficiency

Consider two functions, comprehension and loop:

def loop():
    accum = []
    for i in range(20):
        accum.append(i)
    return accum
 
 
def comprehension():
    accum = []
    [accum.append(i) for i in range(20)]
    return accum

N.B. This example is comparing the efficiency of a list comprehension where the result of the comprehension is ignored to a for loop that produces no result, as is discussed in the referenced entry, Python by example: list comprehensions.

Michael Foord comments:

Your alternative for the single line, easily readable, list comprehension is four lines that are less efficient because the loop happens in the interpreter rather than in C.

However, the disassembly, obtained via dis.dis(func) looks like the following for the loop:

2           0 BUILD_LIST               0
            3 STORE_FAST               0 (accum)
 
3           6 SETUP_LOOP              33 (to 42)
            9 LOAD_GLOBAL              0 (range)
           12 LOAD_CONST               1 (20)
           15 CALL_FUNCTION            1
           18 GET_ITER
      >>   19 FOR_ITER                19 (to 41)
           22 STORE_FAST               1 (i)
 
4          25 LOAD_FAST                0 (accum)
           28 LOAD_ATTR                1 (append)
           31 LOAD_FAST                1 (i)
           34 CALL_FUNCTION            1
           37 POP_TOP
           38 JUMP_ABSOLUTE           19
      >>   41 POP_BLOCK
 
5     >>   42 LOAD_FAST                0 (accum)
           45 RETURN_VALUE

And it looks like the following for the comprehension:

2           0 BUILD_LIST               0
            3 STORE_FAST               0 (accum)
 
3           6 BUILD_LIST               0
            9 DUP_TOP
           10 STORE_FAST               1 (_[1])
           13 LOAD_GLOBAL              0 (range)
           16 LOAD_CONST               1 (20)
           19 CALL_FUNCTION            1
           22 GET_ITER
      >>   23 FOR_ITER                22 (to 48)
           26 STORE_FAST               2 (i)
           29 LOAD_FAST                1 (_[1])
           32 LOAD_FAST                0 (accum)
           35 LOAD_ATTR                1 (append)
           38 LOAD_FAST                2 (i)
           41 CALL_FUNCTION            1
           44 LIST_APPEND
           45 JUMP_ABSOLUTE           23
      >>   48 DELETE_FAST              1 (_[1])
           51 POP_TOP
 
4          52 LOAD_FAST                0 (accum)
           55 RETURN_VALUE

By looking at the bytecode instructions, we see that the list comprehension is, at a language level, actually just "syntactic sugar" for the for loop, as mentioned by nes — they both lower down into the same control flow construct at a virtual machine level, at least in CPython.

The primary difference between the two disassemblies is that a superfluous list comprehension result is stored into fastlocal 1, which is loaded (bytecode 29) and appended to (bytecode 44) each iteration, creating some additional overhead — it’s simply deleted in bytecode 48. Unless the POP_BLOCK operation (bytecode 41) of the loop disassembly is very expensive (I haven’t looked into its implementation), the comprehension disassembly is guaranteed to be less efficient.

Because of this, I believe that Michael was mistaken in referring to an overhead that results from use of a for loop versus a list comprehension for CPython. It would be interesting to perform a survey of the list comprehension optimization techniques used in various Python implementations, but optimization seems difficult outside of something like a special Cython construct, because LOAD_GLOBAL range could potentially be changed from the builtin range function. Various issues of this kind are discussed in the (very interesting) paper The effect of unrolling and inlining for Python bytecode optimizations.

Brian Jones: Nose Hates Me

I easy_install’d nose on my iMac some time in the last month, and tried to use it with options for the first time today, and I’ve found that a good number of the ones shown in ‘nosetests –help’ are actually not recognized when I run nosetests. Meanwhile, running nosetests with no options still works fine. This is 0.11.3 on OS X. Google gives me only references to plugins not being found. These are allegedly “built in”! Nobody replied on Twitter either, which is pretty odd in my experience. So here I am. Wtf is going on here?

For sure, -x and -v aren’t recognized, and -p *is* recognized. Using ‘-w’ with ‘.’ as an argument results in ‘no option -w’, but feeding it a non-existent directory results in a Python ValueError (/foo not found, or not a directory). Wtf?

In checking out /Library/Python/2.6/site-packages/nose-0.11.3-py2.6.egg/nose/config.py, I can see that these options are defined, so I’m a bit confused. Here’s a small sampling of output:

Brian-Joness-iMac:tests bjones$ nosetests -x
Usage: nosetests [options]

nosetests: error: no such option: -x
Brian-Joness-iMac:tests bjones$ nosetests -v
Usage: nosetests [options]

nosetests: error: no such option: -v
Brian-Joness-iMac:tests bjones$ nosetests
..
----------------------------------------------------------------------
Ran 2 tests in 0.360s

OK

Clues hereby solicited.

Jared Forsyth: Embed youtube videos in RestructuredText

I write this blog almost exclusively in RestructuredText and I recently wanted to embed a youtube video on the Baby Tux project page. Fortunately someone’s already done the work of making a directive, so you can just save the code here to youtube_rst.py and put

import youtube_rst

in the __init__.py of your django project. Using the directive is dead simple:

.. youtube:: 9ZWyv9icIfI

Cheers.

Almost Done – Sebastian Bergmann

We are almost done with the manuscript for the German edition (preorder at Amazon.de) of the book. In a couple of days we can focus on getting the manuscript for the English edition (preorder at Amazon.com) ready for the publisher.

I will be in the US from the 12th to the 23rd of May

On the 13th and 14th of May I will be in Redmond, Washington at the 2 day kick off workshop for Microsoft-funded “Common Opensource Application Publishing Platform”.CoApp is an attempt to replicate something similar to Debian’s dpkg/apt within the nati…

PyCon Podcast: Dealing with unsightly data in the real world. (#156) (PyCon 2010)

Drawing on experiences writing http://m.ox.ac.uk/, we’ll explore the art of getting data out of unhelpful systems. We’ll start with working out how to interact with a system, move on to techniques for parsing the data it gives you, and round off by implementing a shinier interface over the top.

keep looking »

Enterprise LAMP