Enterprise LAMP

Downloading RT patches from the command line

I love it when people post patches to RT, but I don’t really like the work required to download them from the web interface. I know there’s an rt command line client somewhere, but the last time I tried it, it didn’t work very well for me. Thankfully…

The next challenge for Perl on Windows (et al)

Even at this early stage, before any actual installers have appeared, things are looking pretty good for Strawberry Professional Alpha 2.Padre is starting to firm up as a usable editor, Frozen Bubble works quite well and is completely playable, and I t…

Gotcha on Scraping .NET Applications with PHP and cURL – Matthew Turland

Obligatory pitch: Many other useful tidbits like this can be yours by purchasing my book, php|architect’s Guide to Web Scraping with PHP.

I recently wrote a PHP script to scrape data from a .NET application. In the process of developing this script, I noticed something interesting that I thought I’d share. In this case, I was using the cURL extension, but the tip isn’t necessarily specific to that. One thing my script did was submit a POST request to simulate a form submission. The code looked something like the sample below.

$ch = curl_init();
curl_setopt_array($ch, array(
    CURLOPT_URL => 'http://...',
    CURLOPT_POST => true,
    CURLOPT_POSTFIELDS => array(
        'field1' => 'value1',
        // ...
    ),
    // ...
));

The issue I ran into had to do with a behavior of the CURLOPT_POSTFIELDS setting that’s easy to overlook. This is a segment of its description from the PHP manual page for the curl_setopt() function.

If value is an array, the Content-Type header will be set to multipart/form-data.

If the form being submitted is not set to have an enctype attribute value of multipart/form-data in the form’s markup, .NET returns a 500-level HTTP response with no further information on what causes the error (for security purposes). This presumably happens because it’s expecting one value for the Content-Type request header and getting another.

Setting CURLOPT_HEADER and CURLOPT_VERBOSE to true helped to reveal that this was the issue. The fix is pretty simple: instead of passing the array itself for CURLOPT_POSTFIELDS, pass the result of wrapping it in a call to the  http_build_query() function (see its PHP manual page). This converts it to a properly formatted query string, which causes cURL to use the default Content-Type header value of application/x-www-form-urlencoded instead.

Tools like Firebug can help you to examine requests made by a browser. Together with these settings for cURL, you can modify your script’s requests to match those of your browser as closely as possible, making gotchas like this less likely to trip you up.

Rakudo Perl 6 in Your PostgreSQL Database!

It has been a very exciting few weeks in the Perl 6 world with regard to database access.
mberends++ just wrote
a nice blog post about how Perl 6
support for DBI is ramping up with work on MiniDBI (formerly FakeDBI). Multiple developers recently mad…

Michael Foord: ContextDecorator: creating APIs that work as decorators and context managers

Two of the best additions to Python in recent years are the with statement and decorators. Both context managers (objects used in with statements) and decorators can be used for similar purposes: performing an action before and after executing the decorated function or the code inside the with block. … [443 words]

Change Blindness and Zooming Out – Chris Shiflett

Change Blindness Experiment at the Dutch PHP Conference

Two weeks ago, I had the great honor of giving a keynote at the Dutch PHP Conference. Because I had never been to Amsterdam or to the Dutch PHP Conference, I was really excited to have a chance to speak there. It was also an opportunity to give my favorite talk to a new audience.

On the morning of the keynote, I followed along with conference organizer Lorna Mitchell to the RAI Center where the conference was being held. As soon as I saw the stage, I smiled. Not only would I be able to stand on a stage unobstructed by a podium or any other obstacle, the seats were arranged like a theater, making it easier to connect with the audience. (Conference organizers, please take note.)

It was lucky that I arrived early to test the video and audio. If you own a MacBook Pro, you may or may not know that it puts the sound card to sleep if it has been unused for a few minutes. (If you use headphones, you can hear when it sleeps and awakes.) When you’re connected to a massive sound system for a theater, this behavior creates a really horrible noise. The solution I came up with was to play iTunes the entire time with the iTunes volume control turned all the way down. Hopefully this trick will save someone else a lot of trouble. :-)

As I began speaking, I noted that PHP had just turned 15 years old. Years ago, the community was energized by all of the misinformation being spread about PHP. It doesn’t scale. It’s insecure. It’s not maintainable. When I began speaking about security, it was partly in response to some of this. I wanted to educate developers, so that we would not only take responsibility for the security of our apps, but also so that we could avoid the most common and dangerous security problems.

These days, petty insults probably continue in the comments on Digg or Hacker News, but no one takes them too seriously. Can PHP scale? Well, the biggest and most popular sites on the Web all use PHP, so I guess so. With no misinformation to energize us, it can easily seem like the PHP community has lost its luster. Not so.

In my talk, Security-Centered Design, I suggest we take a lesson from the design community, where user experience takes priority. We must evolve, or as Aral Balkan puts it:

The age of features is dead; welcome to the age of user experience.

My talk revolves loosely around security, but it’s really a call to arms for my developer peers to take a step back and consider the bigger picture. We need to zoom out. Sometimes, even with a subject as technical as security, the social elements of the problems we face are just as important as the technical. If we can’t empathize with users, we can’t be great developers.

I want to thank everyone who took the time to say nice things about my talk. Hearing it described as the “best keynote I have ever seen” and the “highlight of the event” is really encouraging and makes it all worthwhile. I can’t possibly thank you enough.

I’ll leave you with a little taste of the talk where I invite everyone to participate in a change blindness experiment. I may discuss change blindness and how it applies to the Web in more detail later, but for now, see if you can spot the difference in the two photos I used in the change blindness video I created for this talk.

Thanks again to everyone who woke up early after a late night at the conference social to see me speak, and I hope to see you again sometime. :-)

Wed, 30 Jun 2010 22:12 GMT — Chris Shiflett’s Blog Chris Shiflett’s Feed

Links for 2010-06-30

Date::Manip Error in Ubuntu 10.04
: “XMLTV requires a Date::Manip timezone of +0000 to work properly” — caused by incompatibility between Date::Manip version 6.00 and XMLTV
(tags: date-manip xmltv breakage mythtv ubuntu lucid 10.04)

Misco.ie
: …

WinCache Extension 1.1 for PHP – Release to Web – Ruslan Yakushev

Today IIS team has published the final release of WinCache Extension 1.1 for PHP. This is the latest stable and production ready version of the extension. The v1.1 has all the features available in version 1.0 plus the following features.

  • User Cache API’s can be used by PHP scripts to store PHP objects and variables in shared memory. This way PHP scripts may improve the execution speed by storing processed data in the cache and then using it in subsequent requests instead of re-creating the data on every request.
  • WinCache Session Handler can be used to configure PHP to store the session data in shared memory cache. Using shared memory instead of the default file session storage helps improve performance of PHP applications that store large amount of data in session objects. The content of the WinCache session cache is persisted on disk so that it is not lost during IIS worker process recycling.
  • File Change Notifications – the entries in the opcode and file caches are now updated as soon as the corresponding PHP files are modified on a file system. This is very useful for PHP applications that store its configuration in PHP files – for example Joomla!. Now the configuration changes for those applications take effect right away instead of a 30 seconds delay (default cache refresh interval).
  • Lock/Unlock API’s – these API’s can be used to obtain/release an exclusive lock on a key in the cache.

Install the Windows Cache Extension 1.1 for PHP – RTW

To install the WinCache Extension 1.1 for PHP 5.2 and PHP 5.3, use the download links at the extension’s home page at http://www.iis.net/expand/WinCacheForPhp.

The installation with Web Platform Installer is the easiest as it will automatically place the extension binary into proper location and will update the PHP configuration to enable the extension. Also, if you have the previous version of the extension installed, then Web PI will upgrade it. If you install any PHP application by using Web PI then the WinCache Extension 1.1 for PHP will be offered as an optional component.

If you install the extension manually, then follow the instructions at Installing/Configuring WinCache for PHP.

Getting the extension source code

The source code for the extension is available at http://pecl.php.net/package/WinCache/1.1.0stable. For the instructions on how to build the extension yourself refer to Building WinCache Extension.

Getting support

The WinCache extension is not supported by Microsoft Product Support Services. Instead it is supported as any other community PHP extension via the forums, mailing lists and the PECL bugs database. Use the following resources:

Will McGugan: What to do with Locidesktop?

So what to do with locidesktop.com? It’s a desktop-like website bookmarking tool – if you haven’t seen it, take a quick look at this example desktop.

I built Loci Desktop a few months ago and promoted it on a few geek sites. It’s been running ever since, with no maintenance from myself, happily serving up start pages to a small number of regular users. There was a buzz when I promoted it, people were largely impressed, some were indifferent, but few ended up using it regularly. So now I’m left with a quandary.

I could try and promote it. But to what end? It’s not like I need a certain number of visitors to cover the hosting. I’m using the same VPS as I am for my blog, and I designed Locidesktop to be ultra-low bandwidth anyway – so it effectively costs me nothing to run.

One option would be to sell the entire site outright, as the domain and technology rights. But there is currently no way of monetizing it and I doubt anyone would be interested as a commercial venture.

I could try and license it as a b2b service. A few people have commented that it would be a useful intranet service. I’m not sure about this, but it sounds plausible.

Alternatively, I could give back to the Django community and release it as open source, which I imagine would be the popular thing to do. Money isn’t my primary motivator (a close second perhaps), so I wouldn’t be averse to doing this. Thing is though, it would require work on my part to document it and maintain it, and I have other open source projects I would prefer to concentrate on. But I can’t deny that it would be cool to see Locidesktop sites popping up over the interwebs.

Finally, I could just leave it as is. I’m pleased with how it turned out, and I have a few loyal users. Maybe I should just be satisfied.

Any options I haven’t considered? Leave a comment…

Eli Bendersky: Python internals: adding a new statement to Python

This article is an attempt to better understand how the front-end of Python works. Just reading documentation and source code may be a bit boring, so I’m taking a hands-on approach here: I’m going to add an until statement to Python.

All the coding for this article was done against the cutting-edge Py3k branch in the Python Mercurial repository mirror.

The until statement

Some languages, like Ruby, have an until statement, which is the complement to while (until num == 0 is equivalent to while num != 0). In Ruby, I can write:

num = 3
until num == 0 do
  puts num
  num -= 1
end

And it will print:

3
2
1

So, I want to add a similar capability to Python. That is, being able to write:

num = 3
until num == 0:
  print(num)
  num -= 1

A language-advocacy digression

This article doesn’t attempt to suggest the addition of an until statement to Python. Although I think such a statement would make some code clearer, and this article displays how easy it is to add, I completely respect Python’s philosophy of minimalism. All I’m trying to do here, really, is gain some insight into the inner workings of Python.

Modifying the grammar

Python uses a custom parser generator named pgen. This is a LL(1) parser that converts Python source code into a parse tree. The input to the parser generator is the file Grammar/Grammar [1]. This is a simple text file that specifies the grammar of Python.

Two modifications have to be made to the grammar file. The first is to add a definition for the until statement. I found where the while statement was defined (while_stmt), and added until_stmt below [2]:

compound_stmt: if_stmt | while_stmt | until_stmt | for_stmt | try_stmt | with_stmt | funcdef | classdef | decorated
if_stmt: 'if' test ':' suite ('elif' test ':' suite)* ['else' ':' suite]
while_stmt: 'while' test ':' suite ['else' ':' suite]
until_stmt: 'until' test ':' suite

Note that I’ve decided to exclude the else clause from my definition of until, just to make it a little bit different (and because frankly I dislike the else clause of loops and don’t think it fits well with the Zen of Python).

The second change is to modify the rule for compound_stmt to include until_stmt, as you can see in the snippet above. It’s right after while_stmt, again.

When you run make after modifying Grammar/Grammar, notice that the pgen program is run to re-generate Include/graminit.h and Python/graminit.c, and then several files get re-compiled.

Modifying the AST generation code

After the Python parser has created a parse tree, this tree is converted into an AST, since ASTs are much simpler to work with in subsequent stages of the compilation process.

So, we’re going to visit Parser/Python.asdl which defines the structure of Python’s ASTs and add an AST node for our new until statement, again right below the while:

| While(expr test, stmt* body, stmt* orelse)
| Until(expr test, stmt* body)

If you now run make, notice that before compiling a bunch of files, Parser/asdl_c.py is run to generate C code from the AST definition file. This (like Grammar/Grammar) is another example of the Python source-code using a mini-language (in other words, a DSL) to simplify programming. Also note that since Parser/asdl_c.py is a Python script, this is a kind of bootstrapping – to build Python from scratch, Python already has to be available.

While Parser/asdl_c.py generated the code to manage our newly defined AST node (into the files Include/Python-ast.h and Python/Python-ast.c), we still have to write the code that converts a relevant parse-tree node into it by hand. This is done in the file Python/ast.c. There, a function named ast_for_stmt converts parse tree nodes for statements into AST nodes. Again, guided by our old friend while, we jump right into the big switch for handling compound statements and add a clause for until_stmt:

case while_stmt:
    return ast_for_while_stmt(c, ch);
case until_stmt:
    return ast_for_until_stmt(c, ch);

Now we should implement ast_for_until_stmt. Here it is:

static stmt_ty
ast_for_until_stmt(struct compiling *c, const node *n)
{
    /* until_stmt: 'until' test ':' suite */
    REQ(n, until_stmt);

    if (NCH(n) == 4) {
        expr_ty expression;
        asdl_seq *suite_seq;

        expression = ast_for_expr(c, CHILD(n, 1));
        if (!expression)
            return NULL;
        suite_seq = ast_for_suite(c, CHILD(n, 3));
        if (!suite_seq)
            return NULL;
        return Until(expression, suite_seq, LINENO(n), n->n_col_offset, c->c_arena);
    }

    PyErr_Format(PyExc_SystemError,
                 "wrong number of tokens for 'until' statement: %d",
                 NCH(n));
    return NULL;
}

Again, this was coded while closely looking at the equivalent ast_for_while_stmt, with the difference that for until I’ve decided not to support the else clause. As expected, the AST is created recursively, using other AST creating functions like ast_for_expr for the condition expression and ast_for_suite for the body of the until statement. Finally, a new node named Until is returned.

Note that we access the parse-tree node n using some macros like NCH and CHILD. These are worth understanding – their code is in Include/node.h.

Digression: AST composition

I chose to create a new type of AST for the until statement, but actually this isn’t necessary. I could’ve saved some work and implemented the new functionality using composition of existing AST nodes, since:

until condition:
   # do stuff

Is functionally equivalent to:

while not condition:
  # do stuff

Instead of creating the Until node in ast_for_until_stmt, I could have created a Not node with an While node as a child. Since the AST compiler already knows how to handle these nodes, the next steps of the process could be skipped.

Compiling ASTs into bytecode

The next step is compiling the AST into Python bytecode. The compilation has an intermediate result which is a CFG (Control Flow Graph), but since the same code handles it I will ignore this detail for now and leave it for another article.

The code we will look at next is Python/compile.c. Following the lead of while, we find the function compiler_visit_stmt, which is responsible for compiling statements into bytecode. We add a clause for Until:

case While_kind:
    return compiler_while(c, s);
case Until_kind:
    return compiler_until(c, s);

If you wonder what Until_kind is, it’s a constant (actually a value of the _stmt_kind enumeration) automatically generated from the AST definition file into Include/Python-ast.h. Anyway, we call compiler_until which, of course, still doesn’t exist. I’ll get to it an a moment.

If you’re curious like me, you’ll notice that compiler_visit_stmt is peculiar. No amount of grep-ping the source tree reveals where it is called. When this is the case, only one option remains – C macro-fu. Indeed, a short investigation leads us to the VISIT macro defined in Python/compile.c:

#define VISIT(C, TYPE, V) {\
    if (!compiler_visit_ ## TYPE((C), (V))) \
        return 0; \

It’s used to invoke compiler_visit_stmt in compiler_body. Back to our business, however…

As promised, here’s compiler_until:

static int
compiler_until(struct compiler *c, stmt_ty s)
{
    basicblock *loop, *end, *anchor = NULL;
    int constant = expr_constant(s->v.Until.test);

    if (constant == 1) {
        return 1;
    }
    loop = compiler_new_block(c);
    end = compiler_new_block(c);
    if (constant == -1) {
        anchor = compiler_new_block(c);
        if (anchor == NULL)
            return 0;
    }
    if (loop == NULL || end == NULL)
        return 0;

    ADDOP_JREL(c, SETUP_LOOP, end);
    compiler_use_next_block(c, loop);
    if (!compiler_push_fblock(c, LOOP, loop))
        return 0;
    if (constant == -1) {
        VISIT(c, expr, s->v.Until.test);
        ADDOP_JABS(c, POP_JUMP_IF_TRUE, anchor);
    }
    VISIT_SEQ(c, stmt, s->v.Until.body);
    ADDOP_JABS(c, JUMP_ABSOLUTE, loop);

    if (constant == -1) {
        compiler_use_next_block(c, anchor);
        ADDOP(c, POP_BLOCK);
    }
    compiler_pop_fblock(c, LOOP, loop);
    compiler_use_next_block(c, end);

    return 1;
}

I have a confession to make: this code wasn’t written based on a deep understanding of Python bytecode. Like the rest of the article, it was done in imitation of the kin compiler_while function. By reading it carefully, however, keeping in mind that the Python VM is stack-based, and glancing into the documentation of the dis module, which has a list of Python bytecodes with descriptions, it’s possible to understand what’s going on.

That’s it, we’re done… Aren’t we?

After making all the changes and running make, we can run the newly compiled Python and try our new until statement:

>>> until num == 0:
...   print(num)
...   num -= 1
...
3
2
1

Voila, it works! Let’s see the bytecode created for the new statement by using the dis module as follows:

import dis

def myfoo(num):
    until num == 0:
        print(num)
        num -= 1

dis.dis(myfoo)

Here’s the result:

4           0 SETUP_LOOP              36 (to 39)
      >>    3 LOAD_FAST                0 (num)
            6 LOAD_CONST               1 (0)
            9 COMPARE_OP               2 (==)
           12 POP_JUMP_IF_TRUE        38

5          15 LOAD_NAME                0 (print)
           18 LOAD_FAST                0 (num)
           21 CALL_FUNCTION            1
           24 POP_TOP

6          25 LOAD_FAST                0 (num)
           28 LOAD_CONST               2 (1)
           31 INPLACE_SUBTRACT
           32 STORE_FAST               0 (num)
           35 JUMP_ABSOLUTE            3
      >>   38 POP_BLOCK
      >>   39 LOAD_CONST               0 (None)
           42 RETURN_VALUE

The most interesting operation is number 12: if the condition is true, we jump to after the loop. This is correct semantics for until. If the jump isn’t executed, the loop body keeps running until it jumps back to the condition at operation 35.

Feeling good about my change, I then tried running the function (executing myfoo(3)) instead of showing its bytecode. The result was less than encouraging:

Traceback (most recent call last):
  File "zy.py", line 9, in <module>
    myfoo(3)
  File "zy.py", line 5, in myfoo
    print(num)
SystemError: no locals when loading 'print'

Whoa… this can’t be good. So what went wrong?

The case of the missing symbol table

One of the steps the Python compiler performs when compiling the AST is create a symbol table for the code it compiles. The call to PySymtable_Build in PyAST_Compile calls into the symbol table module (Python/symtable.c), which walks the AST in a manner similar to the code generation functions. Having a symbol table for each scope helps the compiler figure out some key information, such as which variables are global and which are local to a scope.

To fix the problem, we have to modify the symtable_visit_stmt function in Python/symtable.c, adding code for handling until statements, after the similar code for while statements [3]:

case While_kind:
    VISIT(st, expr, s->v.While.test);
    VISIT_SEQ(st, stmt, s->v.While.body);
    if (s->v.While.orelse)
        VISIT_SEQ(st, stmt, s->v.While.orelse);
    break;
case Until_kind:
    VISIT(st, expr, s->v.Until.test);
    VISIT_SEQ(st, stmt, s->v.Until.body);
    break;

And now we really are done. Compiling the source after this change makes the execution of myfoo(3) work as expected.

Conclusion

In this article I’ve demonstrated how to add a new statement to Python. Albeit requiring quite a bit of tinkering in the code of the Python compiler, the change wasn’t difficult to implement, because I used a similar and existing statement as a guideline.

The Python compiler is a sophisticated chunk of software, and I don’t claim being an expert in it. However, I am really interested in the internals of Python, and particularly its front-end. Therefore, I found this exercise a very useful companion to theoretical study of the compiler’s principles and source code. It will serve as a base for future articles that will get deeper into the compiler.

References

I used a few excellent references for the construction of this article. Here they are, in no particular order:

  • PEP 339: Design of the CPython compiler – probably the most important and comprehensive piece of official documentation for the Python compiler. Being very short, it painfully displays the scarcity of good documentation of the internals of Python.
  • "Python Compiler Internals" – an article by Thomas Lee
  • "Python: Design and Implementation" – a presentation by Guido van Rossum
  • Python (2.5) Virtual Machine, A guided tour – a presentation by Peter Tröger
http://eli.thegreenplace.net/wp-content/uploads/hline.jpg
[1] From here on, references to files in the Python source are given relatively to the root of the source tree, which is the directory where you run configure and make to build Python.
[2] This demonstrates a common technique I use when modifying source code I’m not familiar with: work by similarity. This principle won’t solve all your problems, but it can definitely ease the process. Since everything that has to be done for while also has to be done for until, it serves as a pretty good guideline.
[3] By the way, without this code there’s a compiler warning for Python/symtable.c. The compiler notices that the Until_kind enumeration value isn’t handled in the switch statement of symtable_visit_stmt and complains. It’s always important to check for compiler warnings!

Related posts:

  1. Python internals: Working with Python ASTs Starting with Python 2.5, the Python compiler (the part…
  2. Abstract vs. Concrete Syntax Trees CSTs – Concrete Syntax Trees (a.k.a. Parse Trees) and ASTs…
  3. ASTs for analyzing C As I wrote here, I’ve commonly found myself in the…
keep looking »

Warning: include(/home/remarkwit/enterpriselamp.org/wp-content/themes/Enterprise_LAMP/r_sidebar.php) [function.include]: failed to open stream: No such file or directory in /home/remarkwit/enterpriselamp.org/wp-content/themes/Enterprise_LAMP/archive.php on line 23

Warning: include() [function.include]: Failed opening '/home/remarkwit/enterpriselamp.org/wp-content/themes/Enterprise_LAMP/r_sidebar.php' for inclusion (include_path='.:/usr/local/lib/php:/usr/local/php5/lib/pear') in /home/remarkwit/enterpriselamp.org/wp-content/themes/Enterprise_LAMP/archive.php on line 23