Edward K. Ream: SQLite is serious about testing!

The How SQLite is Tested page is the most interesting discussion of software testing I have ever seen.

Mike Driscoll: Python 101: How to Open a File or Program

When I began learning Python, one of the first things I needed to know how to do was open a file. Now, the term “open a file” can mean different things depending on the context. Sometimes it means to actually open the file with Python and read from it, like with a text file. Other times, it means “to open the file in its default program”; and sometimes it means, “open the file in the program I specify”. So, when you go looking for how to do the latter two, you need to know how to ask Google just the right question or all you’ll end up with is learning how to open and read a text file.

In this article, we’re going to cover all three and we’ll also show how to open (or run) programs that are already installed on your PC. Why? Because that topic is also one of the first things I needed to learn and it uses some of the same techniques.

How to Open a Text File

Let’s start by learning how to open a file with Python. In this case, what we mean is to actually use Python to open it and not some other program. For that, we have two choices (in Python 2.x): open or file. Let’s take a look and see how it’s done!

# the open keyword opens a file in read-only mode by default
f = open("path/to/file.txt")
 
# read all the lines in the file and return them in a list
lines = f.readlines()
 
f.close()

As you can see, it’s really quite easy to open and read a text file. You can replace the “open” keyword with the “file” keyword and it will work the same. If you want to be extra explicit, you can write the open command like this instead:

f = open("path/to/file.txt", mode="r")

The “r” means to just read the file. You can also open a file in “rb” (read binary), “w” (write), “a” (append), or “wb” (write binary). Note that if you use either “w” or “wb”, Python will overwrite the file, if it exists already or create it if the file doesn’t exist.

If you want to read the file, you can use the following methods:

read - reads the whole file and returns the whole thing in a string
readline - reads the first line of the file and returns it as a string
readlines – reads the entire file and returns it as a list of strings

You can also read a file with a loop, like this:

f = open("path/to/file.txt")
for line in f:
    print line
f.close()

Pretty cool, huh? Python rocks! Now it’s time to take a look at how to open a file with another program.

Open a file with its own program

Python has a simple method for opening a file with its default program. It goes something like this:

import os
os.startfile(path)

Yes, it’s that easy, if you’re on Windows. If you’re on Unix or Mac, you’ll need the subprocess module or “os.system”. Of course, if you’re a real geek, then you probably have multiple programs that you might want to use to open a specific file. For example, I might want to edit my JPEG file with Picasa, Paint Shop Pro, Lightroom, Paint.NET or a plethora of other programs, but I don’t want to change my default JPEG editing program. How do we solve this with Python? We use Python’s subprocess module! Note: If you want to go old school, you can also use os.popen* or os.system, but subprocess is supposed to supersede them.

import subprocess
 
pdf = "path/to/pdf"
acrobatPath = r'C:\Program Files\Adobe\Reader 9.0\Reader\AcroRd32.exe'
subprocess.Popen("%s %s" % (acrobatPath, pdf))

You can also write the last line like this: subprocess.Popen([acrobatPath, pdf]). Suffice it to say, using the subprocess module is also a breeze. There are many other ways to use the subprocess module, but this is one of its primary tasks. I usually use it to open a specific file (like above) or to open a program for me with specific arguments applied. I also use subprocess’s “call” method, which causes the Python script to wait for the “called” application to complete before continuing. You can also communicate with the processes that subprocess launches, if you have the know how.

Wrapping Up

As usual, Python comes through with easy ways to accomplish the tasks thrown at it. I have found very little that Python can’t handle eloquently and in an easy-to-understand way. I hope you find this helpful when you’re starting out and need to know how to open a file or program.

recent work with zend framework and doctrine2 – Michael Kimsal

I recently began working with an organization and we are in the midst of making some changes. They were not far enough along that any changes now make that much of a difference, but it was an odd set of changes nonetheless.

Initially, the client had started with CodeIgniter (during a phone call earlier, there was interest in ZF2, but when I started they’d gone with CI). One of the first things I tried to do was to get unit testing set up with CI, and the CIUnit failed out of the box. We’re using PHP5.3, and apparently CIUnit doesn’t work well with PHP5.3. I was on the fence about whether we should try to use a vanilla phpunit setup, but I counseled against it (was that wise?). CI was not hurting anyone, but it’s also PHP4 compatible (still), which strikes me as a drag. I can’t quite put my finger on it, and I know that ‘our’ code can still take advantage of PHP5 stuff. However, with things like split() being deprecated in 5.3, I think we might start seeing more warnings if/when we started exercising more of CI. I might be wholly wrong though. I had a bad experience (multiple) with Kohana, so don’t go there.

I’ve had zfkit in place for a while now, and suggested we look at that. However, zfkit has doctrine1 integrated, and the code the client had was already built around doctrine2. That took a little bit of cajoling to get working – once I’m a bit more comfortable with it, I plan to roll in doctrine2 support in to zfkit. I was a bit surprised to see some of the more useful aspects of doctrine1 were removed – the ‘acts as timestampable’, for example. The authors disagreed that something like that is fundamental, and offer some ways around it, but those workarounds are fairly inelegant, imo. The more I work with doctrine2, the clumsier this sort of stuff feels in PHP. Yes, I know, the language itself is the impediment, and that’s why many of my recent projects have been in Grails instead of PHP. For basic work, GORM is a lifesaver compared to doctrine and other PHP ORM tools. That said, phpunit is miles faster than unit testing in Grails, so there’s always tradeoffs. Also, this client has chosen to do their doctrine definitions in php docblock annotations. They irk me.

Still, we’ll have managed to port over the base of the existing app, have a replicatable(?) data story (drop/update schema, load test data) and the start of unit tests (possibly automated later with phpundercontrol) in less than a week’s time. This doesn’t feel too bad, all in all.

I did run in to a ‘problem’ (well, for me anyway) in that the client’s Doctrine models are namespaced. Referencing PHP’s DateTime object from inside a namespaced class requires it to be referenced as \DateTime().

<?php
namespace foo;
 
class bar {
  public function demo() {
    $now = new \DateTime("now");
  }
}

That was a bit new to me, and I didn’t see any specific references to that online, but I may have missed it somewhere. Would have been nice for PHP to drop up to a global scope, or understand references to its own built-in objects, rather than having to do this workaround.

Note: for some reason, an earlier posted tagged ‘business’ made it in to planet-php’s feed. It should only be pulling ‘PHP’ tags. Apologies if it seemed I was spamming the aggregator.

Gael Varoquaux: Scikit Learn coding sprint

We have been really crap at communicating the next scikit-learn coding sprint. It’s next week!

The coding sprint will take place the 8 and 9 September at INRIA Saclay, near Paris, in the room K110 (building K).

For those who cannot make it, it will be possible to participate using the IRC chan (#scikit-learn on irc.freenode.net).

We will start at 9am (Paris time), and a sketch of the planning can be found here. In particular:

More docs! we still need tutorials: features selection, model selection, cross-validation, etc..
Make the pipeline object really work + illustration in different contexts.
Clean up and doc for bayesian approaches.
Implementation of PCA (fit + transform).
FastICA (adapt the CanICA code)
LDA : Covariance estimators (Ledoit-Wolf) and add transform.
Preprocessing routines (center, standardize) with fit transform.
Anything that you have a particular interest in.

Do not hesitate to send on the mailing list some advices on this (incomplete…) list, and see you next week!

scikit-learn is a Python module for efficient and easy machine learning using scipy and numpy.

The Central Tension Of Programming – Paul M. Jones

The central tension in the software process comes from the fact that we must go from an informally identified need that exists in-the-world to a formal model that operates in-the-computer.

From “Beyond Programming” by Bruce Blum, as quoted in “The Design of Design” by Frederick P. Brooks Jr.

Local video stores going away? Hopefully not. – Michael Kimsal

Just saw a comment on a forum that ‘Blockbuster has a dead business model’, and I’ve noticed that one branch of our local video store chain closed. The other store, while still big, is feeling empty.

The rise of Netflix predicted the end of video stores years ago, but it didn’t happen. At least, not quickly. The rise of ‘video on demand’ also was seen as a nail in the coffin for video stores, but it didn’t happen right away either. ‘Video on demand’ is something we’ve had via Tivo for 3 years from Amazon, but I’ve only used it once. It’s not that convenient, perhaps due to a slow internet connection?

Cable services’ “on demand” options have probably hurt video stores too, but those have been around for a long time, and it didn’t seem to hurt video stores that much.

The advent of redbox and similar services, allowing quick, cheap rental of popular videos in convenient places – this is likely hurting video stores too.

It does seem the local video store may be in danger, and I’ll offer a few suggestions for local video stores who want to stay in business, and perhaps thrive.

1. Offer dropoff service with various local merchants. If I could drop off video-store X videos at the local grocery, subway, dry cleaners, gym, etc., even if I couldn’t rent another one, it would save me a trip and add to the convenience factor of dealing with your store.

2. Offer in-store systems to give me recommendations. I’ve been surprised that in 10 years, I’ve never seen a kiosk in any store that would let me hit IMDB or something similar. I can turn to the net and get movie recommendations, reviews, etc, but I’ve never been able to do that *at the moment I’m trying to make a decision*. Yes, many people come get the new releases and that’s it, but many stores have a large vast back catalog of items that just sit unwatched because people don’t know anything about them. Yes… it might be that videos people find on the web aren’t in stock. So… keep track of the movies people are looking up and… *get them* to rent.

3. Offer meal/movie deals with local eateries. Our local video store has a Burger King, Dairy Queen, PIzza Hut, Subway and private Greek and Chinese restaurants in the same shopping plaza – 1 minute walk from store to store. Yet none of them have ever paired up. I sometimes order my Chinese, walk down to the video store, rent a movie, come back and pick up the Chinese. I can’t be the only one doing this, but I bet if more people were offered $1 off a movie rental with a Chinese takeaway order, they’d be back. Or just a $20 “dinner/movie” package – here’s your coupon, go pick up the movie, and dinner’ll be ready in 10 minutes.

Eventually – years from now, even more people will live a digital lifestyle, and most movies will be available at the click of a button. But… I suspect it’ll be newer releases for a long time. Local video stores can survive for many years by offering a better shopping experience and catering to the niche interests.

Dave Beazley: Using telnet to access my Superboard II (via Python and cassette ports)

Welcome to part 3 of my “Superboard II” trilogy. For the first two parts, see these posts:

Dave’s Superboard II

First, a brief digression.

Why Bother?

Aside from the obvious nostalgia (the Superboard II being my first computer), why bother messing around with something like this? After all, we’re talking about a long-since-dead 1970s technology. Any sort of practical application certainly seems far-fetched.

The simple answer is that doing this sort of thing is fun–fun for the same reasons I got into programming in the first place. When my family first got the Superboard, it was this magical device–a device where you could command it to do anything you wanted. You could write programs to make it play games. Or, more importantly, you could command it to do your math homework. Not only that, everything about the machine was open. It came with electrical schematics and memory maps. You could directly input hex 6502 opcodes. There were no rules at all. Although writing a game or doing your homework might be an end goal, the real fun was the process of figuring out how to do those things (to be honest, I think I learned much more about math by writing programs to do my math homework than I ever did by actually doing the homework, but that’s a different story).

Flash forward about 30 years and I’m now doing most of my coding in Python. However, Python (and most other dynamic languages) embody everything that was great about my old Superboard II. For instance, the instant gratification of using the interactive interpreter to try things out. Or, the complete freedom to do almost anything you want in a program (first-class functions, duck-typing, metaprogramming, etc.). Or, the ability to dig deep into the bowels of your system (ctypes, Swig, etc.). Frankly, it’s all great fun. It’s what programming should be about. Clearly the designers of more “serious” languages (especially those designed for the “enterprise”) never had anything like a Superboard.

Anyways, getting back to my motivations, I don’t really have any urgent need to access my Superboard from my Mac. I’m mostly just interested in the problem of how I would do it. The fun is all in the process of figuring it out.

Back to the Superboard Cassette Ports

Getting back to topic, you will recall that in my prior posts, I was interested in the problem of encoding and decoding the audio stream transmitted from the cassette input and output ports on my Superboard II. In part, this was due to the fact that those are the only available I/O ports–forget about USB, Firewire, Ethernet, RS-232, or a parallel port. Nope, cassette audio is all there is.

From the two parts, I wrote some Python scripts that encode and decode the cassette audio data to and from WAV files. Although that is somewhat interesting, working with WAV files was never my real goal. Instead, what I really wanted to do was to set up a real-time bidirectional data communication channel between my Mac and the Superboard II. Simply stated, I wanted to create the equivalent of a network connection using the cassette ports. Would it even be possible? Who knows?

So far as I know, the cassette ports on the Superboard were never intended for this purpose. Although there are commands to save a program and to load a program, driving both the cassette input and output simultaneously isn’t something you would do. It didn’t even make any sense. There certainly weren’t any Superboard commands to do that.

Building a Soft-Modem Using PyAudio

To perform real-time communications, the Superboard needs to be connected to both the audio line-out and line-in ports of my Mac. Using those connections, I would then need to write a program that operates as a soft-modem. This program would simultaneously read and transmit audio data by encoding or decoding it as appropriate (see my past posts).

I’ve never written a program for manipulating audio on my Mac, but after some searching, I found the PyAudio extension that seemed to provide the exact set of features I needed.

To create a soft-modem, I defined reader and writer threads as follows:

# Note : This is Python 2 due to the PyAudio dependency.
import pyaudio
import kcs_decode      # See prior posts
import kcs_encode      # See prior posts
from Queue import Queue

FORMAT    = pyaudio.paInt8
CHANNELS  = 1
RATE      = 9600
CHUNKSIZE = 1024

# Buffered data received and waiting to transmit
audio_write_buffer = Queue()
audio_read_buffer = Queue()

# Generate a sequence representing sign change bits on the real-time
# audio stream (needed as input for decoding)
def generate_sign_change_bits(stream):
    previous = 0
    while True:
        frames = stream.read(CHUNKSIZE)
        if not frames:
            break
        msbytes = bytearray(frames)
        # Emit a stream of sign-change bits
        for byte in msbytes:
            signbit = byte & 0x80
            yield 1 if (signbit ^ previous) else 0
            previous = signbit

# Thread that reads and decodes KCS audio input
def audio_reader():
    print("Reader starting")
    p = pyaudio.PyAudio()
    stream = p.open(format = FORMAT,
                    channels = CHANNELS,
                    rate = RATE,
                    input=True,
                    frames_per_buffer=CHUNKSIZE)

    bits = generate_sign_change_bits(stream)
    byte_stream = kcs_decode.generate_bytes(bits, RATE)
    for b in byte_stream:
        audio_read_buffer.put(chr(b))

# Thread that writes KCS audio data
def audio_writer():
    print("Writer starting")
    p = pyaudio.PyAudio()
    stream = p.open(format = FORMAT,
                    channels = CHANNELS,
                    rate = RATE,
                    output=True)
    while True:
        if not audio_write_buffer.empty():
            msg = kcs_encode.kcs_encode_byte(ord(audio_write_buffer.get()))
            stream.write(buffer(msg))
        else:
            stream.write(buffer(kcs_encode.one_pulse))

if __name__ == '__main__':
    import threading

    # Launch the reader/writer threads
    reader_thr = threading.Thread(target=audio_reader)
    reader_thr.daemon = True
    reader_thr.name = "Reader"
    reader_thr.start()

    writer_thr = threading.Thread(target=audio_writer)
    writer_thr.daemon = True
    writer_thr.name = "Writer"
    writer_thr.start()

The operation of this code is relatively straightforward. There is a reader thread that constantly samples audio on the line-in port and decodes it into bytes which are stored in a
queue for later consumption. There is a writer thread that encodes and transmits outgoing data (if any). If there is no data, the writer transmits a constant carrier tone on the line out (a 2400 Hz wave).

These threads operate entirely in the background. To read data from the Superboard, you simply check the contents of the audio read buffer. To send data to the Superboard, you simply append outgoing data to the audio write buffer.

Creating a Network Server

To tie all of this together, you can now write a network server that connects the real-time audio streams to a network socket. This can be done by defining a third thread like this:

import socket
import time

def server(addr):
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR,1)
    s.bind(addr)
    s.listen(1)
    print("Server running on", addr)
    # Wait for the client to connect
    while True:
        c,a = s.accept()
        print("Got connection",a)
        c.setblocking(False)
        try:
            # Enter a loop where we try to transmit data back and forth between the client and the audio stream
            while True:
                # Check for incoming data
                try:
                    indata = c.recv(8192)
                    if not indata:
                        raise EOFError()
                    indata = indata.replace(b'\r',b'\r' + b'\x00'*20)
                    for b in indata:
                        audio_write_buffer.put(b)
                except socket.error:
                    pass
                # Check if there is any outgoing data to transmit (try to send it all)
                if not audio_read_buffer.empty():
                    while not audio_read_buffer.empty():
                        b = audio_read_buffer.get()
                        c.send(b)
                else:
                    # Sleep briefly if nothing is going on.  This is fine, the max
                    # data transfer rate of the Superboard is 300 baud
                    time.sleep(0.01)
        except EOFError:
            print("Connection closed")
            c.close()

if __name__ == '__main__':
    import threading

    # Launch the reader/writer threads
    ... see above code ..

    # Launch the network server
    server_thr = threading.Thread(target=server,args=(("",15000),))
    server_thr.daemon = True
    server_thr.name = "Server"
    server_thr.start()

    # Have the main thread do something (so Ctrl-C works)
    while True:
            time.sleep(1)

This server operates as a simple polling loop over a client socket and the incoming audio data stream. Data received on the socket is placed in the write buffer used by the audio writer thread. Data received by the audio reader is send back to the client. This code could probably be cleaned up through the use of the select() call, but I frankly don’t know if select() works with PyAudio and didn’t investigate it. Given that the maximum data rate of the Superboard is 300 baud, a “good enough” solution seemed to be just that.

Putting it to the Test

Now, the ultimate test–does it actually work? To try it out, you first have to launch the above audio server process. For example:

bash % python audioserv.py
Reader starting
Writer starting
Server running on ('', 15000)

Next, make sure the Superboard II is plugged into the line-in and line-out ports on my Mac. On the Superboard, I had to manually type two POKE statements to make it send all output to the cassette output and to read all keyboard input from the cassette input.

POKE 517, 128
POKE 515, 128

Finally, use the telnet command to connect to the audio server.

bash $ telnet localhost 15000
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
^]
telnet> mode character
LIST

OK
PRINT "HELLO WORLD"
HELLO WORLD

OK

Excellent! It seems to be working. It’s a little hard to appreciate with just a screenshot. Therefore, you can check out the following movie that shows it all in action:

Again, it’s important to emphasize that there is no other connection between the two machines other than a pair of audio cables.

That is all (for now)

Well, there you have it–using Python to implement a soft-modem that encodes/decodes cassette audio data in real-time, allowing me to remotely access my old Superboard using telnet. At last, I can write old Microsoft Basic 1.0 programs from the comfort of my Aeron chair and a 23-inch LCD display–and there’s nothing old-school about that!

Hope you enjoyed this series of posts. Sadly, it’s now time to get back to some “real work.” Of course, if you’d like to see all of this in person, you should sign up for one of my Python courses.

Michele Simionato: What’s new in plac 0.7

plac is much more than a command-line arguments parser. You can use it to implement interactive interpreters (both on a local machine on a remote server) as well as batch interpreters. It features a doctest-like mode, the ability to launch commands in parallel, and more. And it is easy to use too!

Highlight source code lines in LaTeX – Tobias Schlitt

I love LaTeX for any kind of text writing (actually typesetting), simply because it creates so nice looking and consistent layouts. And, of course, because I can write it in my favorite text editor. We use LaTeX especially for presentation slides at Qafoo, since the beamer package provides such a convenient environment. Combined with listings package, presenting source code snippets with nice syntax highlighting has never been easier. However, there was one problem we did not solve, yet, until some days ago: Highlighting certain source code lines of a listing on different slides.

Writing a perl read-eval-print loop (REPL) – part 1

=== update, April 25 For the impatient, part 2 and part 3 are already out and I’m aiming to publish a part per week until I run out of ideas and change to a different topic. === end update, original post follows So, I need to sort out my p…

« go back — keep looking »

Enterprise LAMP