Andy Todd: Weird easy_install Behaviour

Dear lazyweb, I unsubscribed from the distutils-sig mailing list a while back and consequently I’m not up to date with the latest to-ings and fro-ings. But, I have a problem. As reported by someone today Gerald eggs won’t install on Windows.

Everything is fine on my Ubuntu virtual machine, but on my shiny new work laptop I have Python 2.6 and today I downloaded and installed setuptools version 06.c11. When I try and install Gerald I get an error complaining about a lack of a setup.py file;

(TEST) C:\Work\virtualenvs\TEST>easy_install gerald
Searching for gerald
Reading http://pypi.python.org/simple/gerald/
Reading http://halfcooked.com/code/gerald/
Reading http://sourceforge.net/project/showfiles.php?group_id=53184&package_id=109623
Reading http://sourceforge.net/projects/halfcooked/files
Best match: gerald 0.3.5
Downloading http://sourceforge.net/projects/halfcooked/files/gerald/0.3.5/gerald-0.3.5-py2.6.egg/download
Processing download
error: Couldn't find a setup script in c:\docume~1\andy~1.tod\locals~1\temp\easy_install-woqly0\download
(TEST) C:\Work\virtualenvs\TEST>

The only thing that I can find different is that my Ubuntu virtual machine is running version 0.6c9 of setuptools. Has the function changed between two release candidates?

Needless to say this means that Gerald won’t install under Windows using easy_install until I figure this out. All help and suggestions warmly received.

Test Automation using Perl classes

The conference season is warming up and so I’ll start offering my
Test Automation using Perl training class so I can have an excuse to
go to the workshops and conferences. Here is the schedule:

March 8-11, Berlin, Germany, after CeBIT where we ha…

Richard Tew: Patching through code modification

Previous post: Tracking class instantiations

As I have been exploring patching __init__ of classes loaded by my code reloading framework so that I can track creation of instances, I’ve been considering other approaches.

In the previous post, where there was an existing __init__ method, I renamed it and had my replacement __init__ call it before it registered the freshly created instance. But I can do better, if I modified the bytecode of the existing method, I could inject my registration call directly into it. As an optimisation, in this case it does not add much. But it is interesting to look into, and there is the possibility that this sort of functionality can be added in a more general way within the code reloading framework.

I found three commonly mentioned bytecode manipulating frameworks:

bytecodehacks: No longer maintained and out of date for 2.6.
BytecodeAssembler: Lots of dependencies and it only allows creation of bytecode, not modification of existing bytecode.
byteplay: One file, allows modification of existing code, works out of the box.

byteplay looks like the only suitable candidate that I can just pick up and use.

Code to be modified:

>>> class Test:...     def __init__(self):...             if f():...                     print 1...                     return...             if g():...                     print 2...                     return...             print 3...

I want to make my injected call after the logic in the function has been executed, but before it returns. In this function, there are multiple return points.

Passing the code into byteplay:

>>> import byteplay>>> c = byteplay.Code.from_code(Test.__init__.func_code)>>> print c.code

  3           1 LOAD_GLOBAL          f              2 CALL_FUNCTION        0              3 JUMP_IF_FALSE        to 13              4 POP_TOP

  4           6 LOAD_CONST           1              7 PRINT_ITEM              8 PRINT_NEWLINE

  5          10 LOAD_CONST           None             11 RETURN_VALUE        >>   13 POP_TOP

  6          15 LOAD_GLOBAL          g             16 CALL_FUNCTION        0             17 JUMP_IF_FALSE        to 27             18 POP_TOP

  7          20 LOAD_CONST           2             21 PRINT_ITEM             22 PRINT_NEWLINE

  8          24 LOAD_CONST           None             25 RETURN_VALUE        >>   27 POP_TOP

  9          29 LOAD_CONST           3             30 PRINT_ITEM             31 PRINT_NEWLINE             32 LOAD_CONST           None             33 RETURN_VALUE

Basically I want to inject my call before each LOAD_CONST None/RETURN_VALUE pair.

Code to inject:

>>> def f(self):...     events.Register(self)

Passing the code into byteplay:

>>> c2 = byteplay.Code.from_code(f.func_code)>>> print c2.code

  2           1 LOAD_GLOBAL          events              2 LOAD_ATTR            Register              3 LOAD_FAST            self              4 CALL_FUNCTION        1              5 POP_TOP

              6 LOAD_CONST           None              7 RETURN_VALUE

Basically I want to select the bytecode entries matching displayed lines 1 through 7 and insert them in place of any existing pairs as described above. But something these bytecode listings do not show, is that line numbers are also marked up with bytecode entries. So I need to make sure I do not obliterate existing line numbers in the code I am modifying, or copy over line numbers in the code I am injecting.

Injecting the call before the returns:

offset = len(c.code) - 1lastInstruction = Nonewhile offset >= 0:    instruction, value = c.code[offset]    if lastInstruction == byteplay.RETURN_VALUE and \       instruction == byteplay.LOAD_CONST:        c.code[offset:offset+2] = c2.code[1:]    lastInstruction = instruction    offset -= 1

The resulting bytecode:

>>> print c.code

  3           1 LOAD_GLOBAL          f              2 CALL_FUNCTION        0              3 JUMP_IF_FALSE        to 18              4 POP_TOP

  4           6 LOAD_CONST           1              7 PRINT_ITEM              8 PRINT_NEWLINE

  5          10 LOAD_GLOBAL          events             11 LOAD_ATTR            Register             12 LOAD_FAST            self             13 CALL_FUNCTION        1             14 POP_TOP             15 LOAD_CONST           None             16 RETURN_VALUE        >>   18 POP_TOP

  6          20 LOAD_GLOBAL          g             21 CALL_FUNCTION        0             22 JUMP_IF_FALSE        to 37             23 POP_TOP

  7          25 LOAD_CONST           2             26 PRINT_ITEM             27 PRINT_NEWLINE

  8          29 LOAD_GLOBAL          events             30 LOAD_ATTR            Register             31 LOAD_FAST            self             32 CALL_FUNCTION        1             33 POP_TOP             34 LOAD_CONST           None             35 RETURN_VALUE        >>   37 POP_TOP

  9          39 LOAD_CONST           3             40 PRINT_ITEM             41 PRINT_NEWLINE             42 LOAD_GLOBAL          events             43 LOAD_ATTR            Register             44 LOAD_FAST            self             45 CALL_FUNCTION        1             46 POP_TOP             47 LOAD_CONST           None

The next step is to make f, g and events, and to execute the modified bytecode.

Testing the bytecode:

>>> def f(): return False...>>> def g(): return False...>>> Test.__init__.im_func.func_code  = c.to_code()>>> class Events:...     def Register(self, instance):...             print "REGISTERED", instance...>>> events = Events()>>> t = Test()3REGISTERED <__main__.Test instance at 0x01D2DAD0>

Excellent. I’ll have to think about the possibilities for this. It has potential to allow the creation of all sorts of interesting features in a code reloading framework.

Neil Schemenauer: Quixote 2.7b2 beta released

I released another Quixote beta about a week ago. I think it finally fixes the problem cased by Python 2.6’s breakage of the ihooks module. What happened was that the when relative imports were implemented, the ihooks module got forgotten. If you use the 2.6 ihooks module and any package uses a relative import, you lose.

I’ve fixed the ihooks module in the SVN trunk and it will be fixed in Python 2.7. Quixote 2.7b2 works around the problem by shipping with it’s own ihooks module.

Carl Trachte: OpenBSD and Python

Last time we covered FreeBSD’s third party module, freebsd; this time we’ll take a quick look at the equivalent openbsd package for the OpenBSD operating system.

$ python2.5
Python 2.5.4 (r254:67916, Jul 1 2009, 11:37:21)
[GCC 3.3.5 (propolice)] on openbsd4
Type “help”, “copyright”, “credits” or “license” for more information.
>>> import openbsd
>>> dir(openbsd)
['__builtins__', '__doc__', '__file__', '__name__', '__path__', '_ifconfig', '_netstat', '_packetDescriptors', '_pcap', '_sysvar', 'arc4random', 'ifconfig', 'netstat', 'packet', 'pcap', 'utils']

Let’s see what all is hidden in that utils item:

>>> dir(openbsd.utils)
['DoubleAssociation', '__builtins__', '__doc__', '__file__', '__name__', 'cksum16', 'ethToBytes', 'ethToStr', 'findLongestSubsequence', 'getBlocks', 'ip6FromPrefix', 'ip6ToBytes', 'ip6ToStr', 'ipFromPrefix', 'ipToBytes', 'ipToStr', 'isIP6Addr', 'isIPAddr', 'isStringLike', 'multichar', 'multiord']

OK, a fair number of network addressing related functions.

help(openbsd.utils.ipFromPrefix)

ipFromPrefix(prefix)
Produce an IPv4 address (netmask) from a prefix length.

That sounds handy. Let’s give it a shot:

>>> openbsd.utils.ipFromPrefix(24)
‘255.255.255.0′

>>> help(openbsd.utils.DoubleAssociation)

Help on class DoubleAssociation in module openbsd.utils:

class DoubleAssociation(__builtin__.dict)
| A double-association is a broadminded dictionary – it goes both ways.
|
| The rather simple implementation below requires the keys and values to
| be two disjoint sets. That is, if a given value is both a key and a
| value in a DoubleAssociation, you get unexpected behaviour.
|
| Method resolution order:
|      DoubleAssociation
|      __builtin__.dict
|      __builtin__.object
|
| Methods defined here:
|
| __init__(self, idict=None)
|      # FIXME:
|      #   While DoubleAssociation is adequate for our use, it is not entirely complete:
|      #       – Deletion should delete both associations
|      #       – Other dict methods that set values (eg. setdefault) will need to be over-ridden.

This one is kind of interesting – let’s have a look:

>>> d = {1:’a', 2:’b', 3:’c'}
>>> d.get(1)
‘a’
>>> print d.get(’a')
None
>>> da = openbsd.utils.DoubleAssociation(d)
>>> da.get(1)
‘a’
>>> da.get(’a')
1

Just like the doc described it. Both the keys and the values are keys, if that makes sense.

Back up to the main modules of the openbsd package:

>>> help(openbsd.arc4random)

NAME
openbsd.arc4random

FILE
/usr/local/lib/python2.5/site-packages/openbsd/arc4random.so

FUNCTIONS
getbytes(…)
Get some random bytes.

And the result -

>>> bytesx = openbsd.arc4random.getbytes(10)
>>> [bytex for bytex in bytesx]
['\xb4', '\xd1', '\x86', '\xb7', 'g', '8', '\x10', '}', '\x8b', '\xe5']

One last module on a more common theme:

NAME
openbsd.ifconfig – A Python module for querying and manipulating network interfaces.

FILE
/usr/local/lib/python2.5/site-packages/openbsd/ifconfig.py

CLASSES
    __builtin__.int(__builtin__.object)
        FlagVal
    __builtin__.object
        Flags
        IFConfig
        Interface
        MTU
        Media
        Metric
    exceptions.Exception(exceptions.BaseException)
        _ifconfig.IfConfigError

    class FlagVal(__builtin__.int)
     | Method resolution order:
(etc.)

>>> intx = openbsd.ifconfig.Interface(’rl0′)
>>> print intx
rl0: flags=8843 mtu 1500
         media: Ethernet autoselect
         link: 00:30:bd:72:6a:a0
         inet6: fe80:2::230:bdff:fe72:6aa0
         inet: 192.168.100.100
>>> dir(intx)
['Iftype', 'Name', '__class__', '__delattr__', '__dict__', '__doc__', '__getattribute__', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__str__', '__weakref__', '_addrToStr', '_addrTypeLookup', '_getAddresses', '_getinfo', '_setflags', '_setmetric', '_setmtu', 'addAddress', 'addresses', 'delAddress', 'flags', 'media', 'metric', 'mtu', 'setAddress']
>>> intx.media
media: Ethernet autoselect
>>> intx.addresses
[{'address': {'sa_family': 18L, 'iftype': 'ETHER', 'address': '00:30:bd:72:6a:a0'}}, {'netmask': {'sa_family': 24L, 'address': 'ffff:ffff:ffff:ffff::'}, 'address': {'sa_family': 24L, 'address': 'fe80:2::230:bdff:fe72:6aa0'}}, {'netmask': {'sa_family': 0L, 'address': None}, 'dstaddr': {'sa_family': 2L, 'address': '192.168.100.255'}, 'address': {'sa_family': 2L, 'address': '192.168.100.100'}}]
>>>

ifconfig available within Python – sweet. rl0 is the ethernet device on my old Dell tower.

Examination of the openbsd package shows that it has quite a bit to offer. If you’re using OpenBSD, there’s nothing stopping you from doing routine sysadmin tasks with Python. If not, now you’ve got a reason to check it out.

Carl Trachte: Python Modules for the BSD’s

Well, for FreeBSD and OpenBSD, at least. I can’t yet vouch for NetBSD and Dragonfly BSD.

First, FreeBSD – the port is named py-freebsd. Once built, the module can be imported with “import freebsd”.

[carl@pcbsd]/usr/local/lib/python2.6/site-packages(158)% python
Python 2.6.2 (r262:71600, Jun 24 2009, 23:31:28)
[GCC 4.2.1 20070719 [FreeBSD]] on freebsd7
Type “help”, “copyright”, “credits” or “license” for more information.
>>> import freebsd
>>> dir(freebsd)
['__doc__', '__file__', '__name__', '__package__', '__version__', 'chflags', 'const', 'fchflags', 'fstatfs', 'geom_getxml', 'getfsent', 'getfsfile', 'getfsspec', 'getfsstat', 'gethostname', 'getloadavg', 'getlogin', 'getosreldate', 'getpriority', 'getprogname', 'getpwent', 'getpwnam', 'getpwuid', 'getquota', 'getrlimit', 'getrusage', 'ifstats', 'ipstats', 'jail', 'kevent', 'kqueue', 'ktrace', 'lchflags', 'quotaoff', 'quotaon', 'quotasync', 'reboot', 'sendfile', 'sethostname', 'setlogin', 'setpriority', 'setproctitle', 'setprogname', 'setquota', 'setrlimit', 'statfs', 'sysctl', 'sysctldescr', 'sysctlmibtoname', 'sysctlnametomib', 'tcpstats', 'udpstats']

Not a bad collection of utilities. Let’s take a couple for a test drive:

>>> freebsd.gethostname()
‘pcbsd’

>>> freebsd.getprogname()
‘python’
>>> help(freebsd.jail)
Help on built-in function jail in module freebsd:

jail(…)
jail(path, hostname, ip_number):
The jail() system call sets up a jail and locks the current process
in it. The “path” should be set to the directory which is to be
the root of the prison. The “hostname” can be set to the hostname
of the prison. This can be changed from the inside of the prison.
The “ip_number” can be set to the IP number assigned to the prison.

>>> # wow, you can set up a jail with python

>>> freebsd.ifstats()
>>> >>> import pprint
>>> pprint.pprint(_)
{’bge0′: {’addrlen’: 6,
‘baudrate’: 100000000L,
‘collisions’: 0L,
‘flags’: 34883,
‘hdrlen’: 14,
‘hwassist’: 7L,
‘ibytes’: 19222590L,
‘ierrors’: 0L,
‘imcasts’: 577L,
‘ipackets’: 19728L,
‘iqdrops’: 0L,
‘metric’: 0L,
‘mtu’: 1500L,
‘name’: ‘bge0′,
‘noproto’: 0L,
‘obytes’: 2009038L,
‘oerrors’: 0L,
‘omcasts’: 0L,
‘opackets’: 13285L,
‘pcount’: 0,
‘physical’: 0,
’snd_drops’: 0,
’snd_len’: 0,
’snd_maxlen’: 511,
‘type’: 6},

bge0 is the ethernet device on my Thinkpad.

>>> freebsd.getlogin()
‘carl’
>>> freebsd.tcpstats()
>>> pprint.pprint(_)
{’accepts’: 0L,
‘badsyn’: 0L,
‘cachedrtt’: 147L,
‘cachedrttvar’: 150L,
‘cachedssthresh’: 4L,
‘closed’: 495L,
‘connattempt’: 360L,
‘conndrops’: 20L,
‘connects’: 340L,
‘delack’: 277L,
‘drops’: 22L,
‘keepdrops’: 0L,
‘keepprobe’: 0L,
‘keeptimeo’: 0L,
‘listendrop’: 0L,
‘mturesent’: 0L,
‘pawsdrop’: 0L,
‘persistdrop’: 0L,
‘persisttimeo’: 0L,
‘predack’: 0L,
‘preddat’: 15226L,
‘rcvackbyte’: 1093284L,
‘rcvackpack’: 1848L,
‘rcvacktoomuch’: 0L,
‘rcvafterclose’: 7L,
‘rcvbadoff’: 0L,
‘rcvbadsum’: 0L,
‘rcvbyte’: 16595286L,
‘rcvbyteafterwin’: 0L,
‘rcvdupack’: 232L,
‘rcvdupbyte’: 88723L,
‘rcvduppack’: 77L,
‘rcvoobyte’: 1015050L,
‘rcvoopack’: 919L,
‘rcvpack’: 15882L,
‘rcvpackafterwin’: 0L,
‘rcvpartdupbyte’: 525L,
‘rcvpartduppack’: 2L,
‘rcvshort’: 0L,
‘rcvtotal’: 18489L,
‘rcvwinprobe’: 0L,
‘rcvwinupd’: 3L,
‘rexmttimeo’: 118L,
‘rttupdated’: 1817L,
’sc_aborted’: 0L,
’sc_added’: 0L,
’sc_badack’: 0L,
’sc_bucketoverflow’: 0L,
’sc_cacheoverflow’: 0L,
’sc_completed’: 0L,
’sc_dropped’: 0L,
’sc_dupsyn’: 0L,
’sc_recvcookie’: 0L,
’sc_reset’: 0L,
’sc_retransmitted’: 0L,
’sc_sendcookie’: 0L,
’sc_stale’: 0L,
’sc_unreach’: 0L,
’sc_zonefail’: 0L,
’segstimed’: 1688L,
’sndacks’: 9261L,
’sndbyte’: 1098259L,
’sndctrl’: 697L,
’sndpack’: 1252L,
’sndprobe’: 0L,
’sndrexmitbyte’: 2252L,
’sndrexmitpack’: 2L,
’sndtotal’: 12381L,
’sndurg’: 0L,
’sndwinup’: 1169L,
‘timeoutdrop’: 9L}

22 drops, 9 of them timeouts, and a bunch of other stuff too.

Enough for today. Next time we’ll take a quick look at the Python module for OpenBSD.

Hany Fahim: Helpful Django View Decorator Pattern

One of Python’s (2.5+) most useful shorthands is the concept of decorators. Here is an example of using Decorators with Django to make view functions a little more “DRY”.
I probably found the original version of this on www.djangosnippets.org, but I’ve made a couple of little “enhancements”. First of all, the decorator takes two argument, the first the main template to render, but the second optional argument is the name of the template to render in logged-in mode. This is a useful pattern if the same view has two different templates depending on whether the user is logged-in or not.
I’ve also added an automatic “body_class” context variable to the rendered template that can be used to specify a custom class for the HTML body:

<body class="{{ body_class }}" ...

This might not be the cleanest way to do things, but I find it very useful to automatically have a CSS class added to my body tag. Here is how you apply it in your views

@render_with('mytemplates/template.html')def my_view(request):  return {'context_var': value}

def render_with(template, logged_in_template=None):   """   Decorator for Django views that sends returned dict to render_to_response function   with given template and RequestContext as context instance.   """    def renderer(func):        def wrapper(request, *args, **kw):           template_to_render = logged_in_template if logged_in_template and request.user.is_authenticated() else template

           output = func(request, *args, **kw)           if request.META['PATH_INFO'] == '/':               body_class = 'home'           else:               body_class = ' '.join('body-'+slugify(p) for p in request.META['PATH_INFO'].split('/') if p)

           if isinstance(output, dict):             if 'body_class' not in output:               output.update({'body_class': body_class})

             return render_to_response(               template_to_render, output,               CustomRequestContext(request, output))

           return output       return wrapper   return renderer

phpUnderControl 0.5.1 released – Manuel Pichler

Today I have released phpUnderControl version 0.5.1. It’s a bug fix release that closes several issues open since a long time. First of all I would like to thank Sebastian Marek who was the main contributor to this releases, so a big thankyou to you.

Now phpUnderControl should work with CruiseControl 2.8.3. Thanks to Mike van Riel who provided some hints on this issue in a blog comment.
Fixed #983: Graph unitests throw fatal error when ezComponents not available.
Fixed #966: phpcs-details.xsl not showing file name.
Closed #863: Destination option is now deprecated.
Fixed #862: Command line switches without parameter don’t work.
Fixed #861: Password is used as username in check outs. This patch was supplied by Thorsten Daners via e-mail.
Fixed #734: Now the build dropdown redirects to the correct build uri.
Implemented #703: PHPUnit test results are now the first entry on the project overview page.
Fixed #700: Throw an exception when the specified project does not exist.
Implemented #675: Use “php -l” for lint checking and not PHPUnit.
Implemented #625: Integrate PHP_Depend results.

Truncated by Planet PHP, read more at the original (another 2141 bytes)

TechniqueNW 10 – Stuart Herbert

Whilst everyone else was over at PHP Benelux 10 (which sounded like a great conference according to the Twitter feedback!), I was up in Morecambe, at the Technique|NorthWest training event organised by Northwest Vision and Media and run by The White Room. A huge thanks to Paul Collins for inviting me up at the last minute to run the PHP workshop on the Saturday, and I’d love to be involved in further events like this.

I had a great time at the event, and I was delighted to see how the North West of England is trying to build and support a digital economy, instead of simply leaving it to chance. If only South Wales had such an initiative!

Perhaps the most interesting thing I took from the weekend was the large disconnect between the people who attended and many of my friends on Twitter. If you listen to the Twitterarti, you’d think that Adobe Flash is a technology that has run its course and is now in terminal decline (mostly because the iPhone and iPad do not support it, plus Adobe not seen as exactly a bastion of innovation these days). And yet, by far the most popular workshop at Technique|NorthWest was the Flash workshop. To these people, Flash is not only still relevant, but in their industry it is still the only real option for delivering online advertising campaigns.

Food for thought.

PS: I also took some photos of Morecambe before the Saturday workshops started.

Live exploiting aus Angreifersicht (XSS / CSRF), Vortrag@Mayflower-Würzburg – ThinkPHP /dev/blog – PHP

Am kommenden Donnerstag, den 04.02.2010 findet wieder ein öffentlicher Vortrag im Mayflower Büro in Würzburg statt (Pleichertorstrasse 2, 97070 Würzburg, Straßenbahn und die Haltestelle Congress Centrum).
Beginn ist um 18:00 Uhr, Thema des Vortrags ist “Live exploiting aus Angreifersicht (XSS / CSRF)“.

Anhand von interaktiven Beispielen erklärt Frank Ruske die Sicherheitsprobleme XSS (Cross-Site Scripting) & CSRF (Cross-Site Request Forgery). Welche Gefahren gibt es und wie werden diese Lücken ausgenutzt – ist das zentrale Thema dieses Vortrages?.

Die “Donnerstags-Vorträge” werden sowohl in Würzburg als auch in München gehalten. Bei Interesse einfach das Blog beobachten, um auf dem Laufenden zu bleiben!
Wir freuen uns auf viele Teilnehmer!

Größere Kartenansicht

keep looking »

Enterprise LAMP