Ian Bicking: Throw out your frameworks! (forms included)

No, I should say forms particularly.

I have lots of things to blog about, but nothing makes me want to blog like code. Ideas are hard, code is easy. So when I saw Jacob’s writeup about dynamic Django form generation I felt a desire to respond. I didn’t see the form panel at PyCon (I intended to but I hardly saw any talks at PyCon, and yet still didn’t even see a good number of the people I wanted to see), but as the author of an ungenerator and as a general form library skeptic I have a somewhat different perspective on the topic.

The example created for the panel might display that perspective. You should go read Jacob’s description; but basically it’s a simple registration form with a dynamic set of questions to ask.

I have created a complete example, because I wanted to be sure I wasn’t skipping anything, but I’ll present a trimmed-down version.

First, the basic control logic:

from webob.dec import wsgify
from webob import exc
from formencode import htmlfill

@wsgify
def questioner(req):
    questions = get_questions(req) # This is provided as part of the example
    if req.method == 'POST':
        errors = validate(req, questions)
        if not errors:
            ... save response ...
            return exc.HTTPFound(location='/thanks')
    else:
        errors = {}
    ## Here's the "form generation":
    page = page_template.substitute(
        action=req.url,
        questions=questions)
    page = htmlfill.render(
        page,
        defaults=req.POST,
        errors=errors)
    return Response(page)

def validate(req, questions):
    # All manual, but do it however you want:
    errors = {}
    form = req.POST
    if (form.get('password')
        and form['password'] != form.get('password_confirm')):
        errors['password_confirm'] = 'Passwords do not match'
    fields = questions + ['username', 'password']
    for field in fields:
        if not form.get(field):
            errors[field] = 'Please enter a value'
    return errors

I’ve just manually handled validation here. I don’t feel like doing it with FormEncode. Manual validation isn’t that big a deal; FormEncode would just produce the same errors dictionary anyway. In this case (as in many form validation cases) you can’t do better than hand-written validation code: it’s shorter, more self-contained, and easier to tweak.

After validation the template is rendered:

page = page_template.substitute(
    action=req.url,
    questions=questions)

I’m using Tempita, but it really doesn’t matter. The template looks like this:

<form action="{{action}}" method="POST">
New Username: <input type="text" name="username"><br />
Password: <input type="password" name="password"><br />
Repeat Password:
  <input type="password" name="password_confirm"><br />
{{for question in questions}}
  {{question}}: <input type="text" name="{{question}}"><br />
{{endfor}}
<input type="submit">
</form>

Note that the only "logic" here is to render the form to include fields for all the questions. Obviously this produces an ugly form, but it’s very obvious how you make this form pretty, and how to tweak it in any way you might want. Also if you have deeper dynamicism (e.g., get_questions start returning the type of response required, or weird validation, or whatever) it’s very obvious where that change would go: display logic goes in the form, validation logic goes in that validate function.

This just gives you the raw form. You wouldn’t need a template at all if it wasn’t for the dynamicism. Everything else is added when the form is "filled":

page = htmlfill.render(
    page,
    defaults=req.POST,
    errors=errors)

How exactly you want to calculate defaults is up to the application; you might want query string variables to be able to pre-fill the form (use req.params), you might want the form bare to start (like here with req.POST), you can easily implement wizards by stuffing req.POST into the session to repeat a form, you might read the defaults out of a user object to make this an edit form. And errors are just handled automatically, inserted into the HTML with appropriate CSS classes.

A great aspect of this pattern if you use it (I’m not even sure it deserves the moniker library): when HTML 5 Forms finally come around and we can all stop doing this stupid server-side overthought nonsense, you won’t have overthought your forms. Your mind will be free and ready to accept that the world has actually become simpler, not more complicated, and that there is knowledge worth forgetting (forms are so freakin’ stupid!) If at all possible, dodging complexity is far better than cleverly responding to complexity.

Tim Knapp: Christchurch NZPUG Meetup This Friday

Just to let you all know that we’ve got the following event this Friday. Again we ask that you send an email to meeting-christchurch AT nzpug dot org so we have an idea of those who are coming along.

Again we’ll order some pizzas so please bring along some money to contribute to this (usually works out at about $7-8 each).

Details are as follows:

Date: 5 March 2010
Time: 5:30-7:30pm
URL: http://nzpug.org/MeetingsChristchurch/Mar2010
Talks:

Darryl Cousins: “Demo: Building a web application using repoze.bfg“

Look forward to seeing you all there!

See the live Ignite talk on our http://geo2gov.com.au/

This is just a short promotional post.If you are interested in the Perl/PostGIS http://geo2gov.com.au/ application we took second prize with during the Mashup Australia competition (or the subject of open government data in general) my “Nerds for Democ…

Simon Willison: Johnny Cache

Johnny Cache. Clever twist on ORM-level caching for Django. Johnny Cache (great name) monkey-patches Django’s QuerySet classes and caches the result of every single SELECT query in memcached with an infinite expiry time. The cache key includes a “generation” ID for each dependent database table, and the generation is changed every single time a table is updated. For apps with infrequent writes, this strategy should work really well—but if a popular table is being updated constantly the cache will be all but useless. Impressively, the system is transaction-aware—cache entries created during a transaction are held in local memory and only pushed to memcached should the transaction complete successfully.

codeboje: pySVG meets Trundle, the turtle

My python pySVG just got a mate named Trundle the turtle (named after a turtle in my kids english school book).
For a while now I haven’t coded much for pySVG.
I had a lot of requests and emails though and one of those involves resurrecting parts of my old abandoned lib for representing natural phenomena.

I can’t (yet) write much about the project yet but i thought the addition of some turtle graphics might be helpful for some people using pySVG.

Basically Trundle supports the normal turtle commands (via method calls) like
– forward(distance)
– backward(distance)
– right(angle)
– left(angle)
– moveTo(vector)
– penDown
– penUp

So you just initialize an instance of the Turtle class, along with the initial position, orientation and the style for drawing (filling, stroke, strokewidth).
You move it to the position you want to start drawing in.
Lower the pen.
Draw.

Whenever you lower or raise the pen the path so far is stored into a polyline object and a new one will be instantiated. All such polylines are added to a list.
The only thing you have to keep in mind is that you need to call finish() when you completed drawing or else the last path won’t be added.
And if you create an instance of svg and call addTurtlePathToSVG you will have the turtles drawings nicely added into the svg container which you can save as usual.

I haven’t finished or polished the api yet, so you need to get it from here (login might be required).

I just give you some example code and the result ok?

from pysvg.turtle import Turtle, Vector
from pysvg.structure import svg

def testLindenMayer():
    s=svg(0, 0, 2000, 2000)
    commands='F+F-F-FF+F+F-F+F+F-F-FF+F+F-F+F+F-F-FF+F+F-F+F+F-F-FF+F+F-F'
    t=Turtle()
    t.moveTo(Vector(500,250))
    t.penDown()
    angle=90
    distance=40
    for cmd in commands:
        print(cmd)
        if cmd=='F':
            t.forward(distance)
        elif cmd=='+':
            t.right(angle)
        elif cmd=='-':
            t.left(angle)
        print(t.getPosition())
    t.penDown()
    print (t.getXML())
    s=t.addTurtlePathToSVG(s)
    s.save('./testoutput/testTurtle.svg')

if __name__ == '__main__':
    testLindenMayer()

This produces the following image:

SimpleTurtleExample

Mike Driscoll: PyCon 2010 and Volunteering

This year, I decided to volunteer at PyCon. At both my previous PyCons, I had planned to help, but wasn’t sure how to join in. The evening before the tutorials started in 2009, I wandered all over the hotel looking for PyCon staff and found no one. Once the tutorials started, I felt pretty drained in the evenings because I was taking the maximum number of tutorials and that’s a lot of information to soak up. But back to this year. I volunteered to be a Session Chair for one session. It was an interesting experience. I got to meet a fair number of cool Python people, although the only ones that I really saw after the session were Stani and Nadia.

The first evening I was at PyCon, I was approached by some guy who wanted to know if I would do some data entry. He had a bunch of tutorial surveys that needed to be entered for Greg Lindstrom, founder (I think) of Elegant Stitches (AKA: WearPython). So I ended up doing some of that once I actually found a location where the PyCon wireless worked.

My next act of good will occurred Friday morning. I had just found a place a sit near an access point when this fellow came up to me and asked if I could help put the power strips under the tables. This was in the main auditorium. I ended up doing about half the room. Sometime during that, another fellow started helping me. The objective was to plug them all together from the front to the back, usually in two series. The biggest drawback to this method is that if anyone in the back bumped their power strip’s switch, then all the people who were feeding off that one would lose power. I never heard if this was an issue or not.

My final volunteering activity was being a Session Chair. I actually talked about that in another post, so I won’t repeat myself here. I’ll just say that it was a mostly good experience other than some minor audio issues and some butterflies.

I was kind of hoping to get an awesome staff tee, but the main staff tees were lame. They’re just black with some white lettering and the PyCon logo. If you were paying attention at the conference, then you’d know that the only differences between the staff and the conference tees were slightly different verbiage and the fact that one was black and the other blue. Oh well. I do like the PyCon logo, but it would have been cooler if they had a large logo on the back too or something.

Next year, I encourage you all to get out there and volunteer to make PyCon 2011 even better!

Steven Klass: Development strategy for google-app-engine

In developing my site for the cloud I found it a bit of a challenge. There is very little documentation in terms of infrastructure strategy and a lot of sites assume you have a clear methodology. I always strive for a clean infrastructure — having developed code for the quite awhile I have come to know that if you have a solid scalable strategy up front it makes things just work easier later on. That is why I was really turned onto google-app-engine. It gives you a clear way of building and testing your code and then pushing it to the web. It is a great place to start but it isn’t the “everything” I’m looking for.
Because some of the limitations/restrictions with using google-app-engine (GAE) you need to understand that if you want to use it (i.e. python package) you need to include it in your application. That is where the benefits of a virtualenv come into play. virtualenv (when combined with pip) let you know exactly what packages you have because throughout the application development process if you need it you ‘pip-it’. Then pip gives you a clean manifest on what you have so you always know. But.. (you knew this was coming) GAE doesn’t like a virtual env. In fact if you try it you almost surely run into problems. For me it was a error along this line…

ImportError: No module named unittest

So what can I do? For me the reasons to use a virtualenv/pip combination is too compelling to simply not use it. I really like knowing what packages I willbe deploying up to the cloud and since I need to package them with my application why not? Furthermore if you are used to running in a virtualenv/pip environment or you have a somewhat stock python environment this is just a good way to keep things straight. Understanding the obvious limitation that you can’t run in a virtualenv we are going to take advantage of everything else this environment has to offer.

General Development Strategy

Use Django. While I understand that GAE now support 1.1 I want that control left to me.
Keep the development tree streamlined for the entire site. Guido recommends (as I understand him in this video ~11:45) to use a single app-engine application per independent application. This doesn’t make much sense to me as the ability to share information between different models is really extending the multi-dimensionality of the site. So my preference is to keep it all under one hood.
Use revision control (for me it’s perforce – I don’t have anything against the others (hg/git/svn) it’s just what I like)
Use eclipse/pydev as my development environment

General Directory Structure

I am developing for $SITE ( i.e export SITE=“foobar.com”)

/dev
        /<$SITE>
                /bin
                /lib
                /include
                /src
                        /django
                /www
                        app.yaml
                        /appengine_django
                        manage.py
                        main.py
                        /apps
                                /app_1
                                /app_2
                                /app_3
                                /app_n
                        /settings
                                __init__.py
                        urls.py
                        django symlink to ../src/django/django

Getting started

Setup your virtualenv

 cd dev

Build up a baseline virtualenv

 virtualenv --python=python2.5 --no-site-packages ${SITE}

Install some needed apps.. (make sure you are in your dev directory..)

# pip install -E ${SITE} -e svn+http://code.djangoproject.com/svn/django/trunk#egg=Django# pip install -E ${SITE} -e svn+http://code.djangoproject.com/svn/django/tags/releases/1.1.1/#egg=Django pip install -E ${SITE} -e svn+http://code.djangoproject.com/svn/django/tags/releases/1.1/#egg=Django pip install -E ${SITE} yolk

*Notes: 1.1.1 failed 2 tests and trunk failed more.. 1.1 passes all tests

Activate your virtualenv

 source ${SITE}/bin/activate

Create your www tree (this is where we house our app). Use the google-app-engine-django helper stuff. It’s well worth it!!

 cd ${SITE} svn export http://google-app-engine-django.googlecode.com/svn/trunk/ www

Link in your django tree

 cd www ln -s ../src/django/django

At this point you should be able to deactivate yourself and simply verify everything works.

 deactivate python2.5 manage.py test  #Ran 61 tests in 8.058s python2.5 manage.py runserver

Now open up a web browser to http://127.0.0.1:8000/ and you should get the famous “It works!!”. At this point you should also configure the Google App Engine Launcher to point to your site directory and “www” will be your application.

Build up a quick index and push it to the web.

This is not for the faint. I am going to quickly build up a basic django index page. I want to show it works and we con complete the process.

Shuffle around my tree (personal preference)

Like I indicated above I like to have my apps all under a single apps tree (keep it clean). I also like to put settings.py under a settings tree – this allows me to mess with the __init__.py

So with that hang on..

 cd ${SITE}/www mkdir settings mv settings.py settings/__init__.py rm settings.pyc

 mkdir apps python2.5 manage.py startapp core mv core apps/

Build your simple app

Edit your ${SITE}/www/urls.py

 urlpatterns = patterns('',      url(r'^', include('apps.core.urls')),  )

Add in your app to the ${SITE}/settings/__init__.py

 INSTALLED_APPS = (       'appengine_django',       'apps.core', )

Create your apps/core/views.py

 from django.http import HttpResponse  def index(request):    return HttpResponse('Hello World -- Django rocks!!')

Get your apps/core/urls.py ready to accept the index view

 from django.conf.urls.defaults import * import views urlpatterns = patterns('',    url(r'^$', views.index), )

Verify it works.. Voila – pretty simple eh??

Next up publish it..

References:

Guido’s screencast – it was awesome..
Rapid Development with Python, Django, and Google App Engine (2008 Google I/O Session Videos and Slides)

Simon Willison: Unit Testing Achievements

Unit Testing Achievements. A plugin for Python’s nose test runner that adds achievements—“Night Shift: Make a failing suite pass between 12am and 5am.”

Peter Bengtsson: Massive improvement on sorting a fat list

IssueTrackerMassContainer is a simple Zope product that is used to put a bunch of IssueTrackerProduct instances into. It doesn’t add much apart from a nice looking dashboard that lists all recent issues and then with an AJAX poll it keeps updating automatically.

But what it was doing was it recursively put together all issues across all issue trackers, sorting them and then returning only the first 20. Fine, but once the numbers start to add up it can become a vast sort operation to deal with.

In my local development copy of 814 issues, by the use of pympler and time() I was able to go from 7 Mb taking 2 seconds down to using only 8 Kb and taking 0.05 seconds.

[409 more words]

Stijn Debrouwere: Coder happiness in Drupal and Django, part I: hacking away

I’m currently working on a Drupal project, and it did well to remind me as to why I’ve switched most of my development over to Django some time ago. I have a comprehensive comparison in the works between these two frameworks and a few others as well, but for now I’d like to focus on just one thing and what it means for development in Drupal and Django: coder happiness.

Configuration over coding

Most people who have a career in software development actually, surprisingly, really like coding a lot. We code after work, perhaps hacking together a widget for personal use or contributing to an open-source project. A lot of coders write about coding as well.

One of the more frustrating things about Drupal is that most of your time spent hacking together a project does not go towards actually coding, but is spent configuring and tweaking settings in a graphical user interface. Another big lump of your time will go towards searching through the gargantuan directory listing of Drupal modules that now contains about 5.000 of those, to see if any of these modules do something you need. And the coding that does happen is split 50/50 between actually writing code yourself and getting familiar with the code of the existing modules that you’ve just installed, because you’re not happy with how this or that works and need to patch things up.

There are tremendous benefits to the reuse of existing software. The Not Invented Here syndrome is an ailment that has crippled many a good developer. It led Steve Yelvington to coin the First Rule of Coding for Drupal, namely: “We do not write code for Drupal.”

Any code you or someone else on your team writes has to be documented and maintained, and can break in unexpected ways. Coding things yourself has subtle hidden costs. So Steve raises a valid point and it’s an issue that I hope to write about in the foreseeable future. But this post is about coder happiness. And, being a coder, having my work reduced to that of an administrative clerk, searching for and configuring modules, just sucks.

Yelvington is in the news biz, and he talks about how building news websites in Drupal actually goes about:

configuring and tailoring the platform to the needs of a news site is a huge creative challenge that involves fairly little in the way of heavy technology (i.e., developing new modules) but a lot of what I regard as “configuration” work, including interface design and theming.

The description of the work is accurate. But I disagree with his evaluation. There’s nothing creative about configuring a Drupal site at all. It offers very little satisfaction. It requires a lot of experience but hardly any skill.

Take, for example, how you access your database in Drupal. You use the Views module. It’s incredibly flexible. It also means that instead of writing a simple line of Django ORM code like

and processing that in your view with

you get to fill out a gigantic form that generates your view, after which you’ll have to fill out a similarly large form in the Panels module to get that list displayed where you want it to. And if want to customize the html that Views produces (because, being geared towards end-users, Views is both a query builder and an html generator) you’ll have to override a bunch of theme files as well.

Could you give me the latest comments?" — "Sure, just fill in this small form, file it in triplicate, and we’ll get back to you."

Easier for them is harder for me

Configuration over coding sucks. I’ll take Django’s approach any day. It doesn’t work for non-technical users, true enough. You can expect a non-technical user to complete a form but you can’t expect him or her to handle an object-relational mapper. However, I’m not a non-technical user and I don’t want to be treated like one. Easier for them is harder for me.

Dries Buytaert, the project lead for Drupal, is adamant about lowering the barriers to participation on the web for non-techies. That’s really cool. But I can write code perfectly well, thank you, so that’s just not a selling point for me.

The developers behind Drupal understand that user experience (UX) and developer experience (DX) are two different things and that they can work against each other. But because they have to serve both communities, they have to compromise. Django is geared towards developers and developers only, so it can skip all these delicate discussions about how to balance the needs of developers and those of end-users and just do whatever makes the framework more productive or comfortable for coders. And it shows.

The Sharpened Knife

Clicking in menus isn’t a very satisfying way to spend your day, but that’s not the only reason why I dislike it. Coding-by-configuring takes you away from the skills that make you so valuable as a coder. You’re not advancing as a developer because you’re not learning any skills that remain valuable outside of the CMS.

I often felt that I was coding in Drupal, not in PHP. Drupal has modules and wrappers for just about anything. You can’t fault Drupal for providing so many conveniences, but it does mean that you never come into contact with e.g. the plain Google Maps API, which means that you can’t transfer that experience to other languages or frameworks.

After working with Drupal for more than three years, I started to feel that I was handicapping myself.

In Django, you’re mostly writing plain Python and using general Python libraries. Django does a lot of work for you, but it’s nonetheless fairly compact. The four-part introductory tutorial teaches about half there is to coding in Django, and can be completed in a few hours. That gives me a warm and fuzzy feeling. If I ever feel the inclination to try out, say, Ruby on Rails or Symfony, I know I have the basics down of MVC-style frameworks and can re-use my knowledge of external API’s, so I shouldn’t find it hard to adapt.

It goes deeper than just skills. The Python and Django communities talk about programming techniques, about continuous integration, about agile methodology. Contrast that with Newspapers on Drupal, a group where I used to spend a lot of time, where almost no best practices are being shared or discussions being held about the future of online news, but rather is filled with questions about what modules to use for this or that.

Building rather than modifying

Because Drupal is still at heart a CMS (although it’s closer to a framework than most other CMSes), it’s all-encompassing. In Django you build things, which is easy. In Drupal you modify how the existing code works, so you need a good working knowledge of what can be modified and how to modify it. There are hooks, theme layer overrides, helper functions, existing modules that you can leverage and so on. Getting familiar with Drupal is not easy.

Coding in Drupal, using hooks, feels somewhat similar to aspect-oriented programming, and the same benefits and caveats apply.

If you haven’t tried out aspect-oriented programming techniques before, you should. It’s very doable in Python with some metaprogramming, and it’s a fun exercise. If you have, you know that it’s wonderfully flexible and that you can modify the behavior of your code in any way that you could possibly want. Kudos to the Drupal devs for pulling that off in a crummy language like PHP.

The trouble, though: if you’re not careful with aspect-oriented coding, the flow of code becomes unbearably opaque. While it does not often come to that point in Drupal, it does make for code that is tougher to write and maintain than it would be if you could simply change how Drupal behaves e.g. by subclassing, as you would in Django.

Regardless of which is more productive, the ‘construction’ approach just feels better to me. It’s not that big of an issue when what you need is close to what Drupal provides out of the box or through modules, but the more custom development you have to do, the more cumbersome tweaking existing behavior becomes, and the more it starts to feel like you’re working around Drupal rather than with Drupal.

Frameworks should enable you to get to the cool stuff quicker, not after you’ve worked around a bunch of default behavior that is sensible for a plain CMS but not so much for the kind of websites people have come to expect in 2010.

Documentation

Because of the added complexity of steering an existing system in the direction you want versus just building something, you’d expect that Drupal would be very well documented. Some parts of it are. The docs for the basic API’s are pretty good. But those docs only work if you know full well what you want and what you’re doing.

Drupal lacks documentation on how all of its API’s fit together, which is what makes the learning curve so tough on developers new to Drupal. And, as you’d expect, the documentation for user-contributed modules (which do most of the work for you) is mostly non-existent.

Contrast that with Django, which has a simpler architecture to begin with — it’s a framework, not a CMS — and has really good documentation on top of that. Django’s documentation is made up of step-by-step tutorials, topical guides and low-level reference material (cf. what Jacob Kaplan-Moss has to say about that) so people new to Django can get up to speed quickly, and gradually delve deeper without facing a wall.

Good documentation is probably the biggest contributor to programmer happiness. If the docs are good, you spend more time coding and less time scouring the web. Good docs feel empowering. Bad docs aren’t just frustrating, they also make you feel like an idiot.

Drupal made me feel like an idiot, Django doesn’t. I’m happy to have made the switch.

Stay tuned for a (shorter) second part about coder happiness in Drupal and Django, which looks at the issue from a bigger perspective: how running a project in both systems feels like.

keep looking »

Enterprise LAMP