Sometimes, in the course of one’s life, one is left with a one-off task. In this case, I needed to call a binary a whole lot of times, and do something with the output each time. The details aren’t important; I just needed to write a wrapper script for this binary, do a modest amount of processing on it, and then output the result to a file. This is a pretty common task, one to which the command line in general and *nix in particular is well-suited.

Now, for various reasons, I prefer not to do this via shell scripts. I don’t have a hard and fast rule for when or why. Unless you routinely write shell scripts (I don’t), you’ll inevitably spend a bunch of time looking stuff up that you looked up, oh, six months ago. Or at least I do. I don’t enjoy it, so if I can’t do it with a few commands, maybe one loop or so, then I prefer to use a scripting language. I have the advantage of better/clearer failure modes and simpler syntax and I feel like I’m learning something more powerful and more widely applicable in the process.

One of my “favorite” things to forget and look up again is the difference between [[ ]] and [ ] in tests. Another one is how you loop over a file or output from a command (while read line; do bar --baz $line; done <foo and foo | while read line; do bar $line; done, respectively).

If you have no trouble remembering this stuff or you write scripts often enough, I suppose this is moot. There was a time when I was scripting enough that I didn’t have to look all this stuff up every time, but that time is past. YMMV and all that.

Anyway, the point is that I probably could’ve done this with something like this (which is simplified somewhat):

ARGS="--bar=baz"
CMD="~/bin/foo"
KEY="quux"
MIN=0
MAX=10000
 
for $(seq 0 1000) num do;
  $CMD $num $ARGS | while read line;
    if egrep -q $KEY <<<$line; then
      value=$(egrep -o '[1-9]+$')
        echo "$num,$value"
    fi
  done
done

I’m sure I have a bajillion bugs in there. For example, I am not at all confident about the outer for loop syntax. And while it was easy to write, I’m not doing anything with command line args, doing any error handling, or anything like that. (On the other other hand, I uh underestimated how easy this was. That’s partially why I’m convinced there are bugs in there.)

I still don’t know how to write idiomatic (typo: idiotmatic) Ruby, so this is going to be very rough. Still, it feels natural to me, minus one or two hitches:

ARGS = "--bar=baz"
CMD = "~/bin/foo"
KEY = /quux/
MIN = 0
MAX = 10000
 
def main
  (MIN..MAX).each do |i|
    out = %x[ #{CMD} #{i} #{ARGS} ]
    out.split('\n').each do |line|
      KEY.match(line) do |m|
        value = line.split(' ')[1]
        print "#{i},#{value}\n"
      end
    end
  end
end

I’m not confident about the outer loop once again, and I’m not sure the call to match() will do what I want, let alone whether it’s elegant. Still, I feel pretty good about it. I love Ruby’s pattern of passing in blocks.

Fortunately or unfortunately, I had to use Python. And don’t get me wrong: I love Python. It is through Python that I learned to love scripting languages. Processing a file line by line was, I think, the real epiphany. And list comprehensions are wonderfully expressive.

But man alive is it awful for this sort of thing. I’m not even going to bother writing it out in Python. I’ll just excerpt, from the 2.7 subprocess docs. Here’s what they say should replace backticks output = \mycmd myarg``:

output = Popen(["mycmd", "myarg"], stdout=PIPE).communicate()[0]

Mind you, this is if you did from subprocess import *. Generally I don’t, which means it ends up looking like this:

output = \
  subprocess.Popen(["mycmd", "myarg"], subprocess.stdout=PIPE).\
  communicate()[0]

Yes, once you have it written, sequestered in its own function that you never touch again, it’s not so bad. However, this is firmly in the category of something I will not be able to (and have not been able to) remember months later.

It is also not discoverable in the sense that it’s highly particular— stdout=PIPE? Really? Compare and contrast opening a file (for line in open('somefile'): print foo(line) or [foo(line) for line in open('somefile')) with this monster. Even my Ruby example could have used backticks if I hadn’t remembered the %x syntax. The best I can say about subprocess is that it’s a) better than Popen in Python 2.4 (or whatever), and b) it’s easier to search the web for than %x.

The kicker of course is that rest of the Python script was very easy to write! However, since calling the binary was changing some external state, though, I had to make sure it was doing the right thing. In the end building and testing the call to subprocess.Popen() took longer than the rest of the script. In an otherwise elegant, no-bullshit, batteries included language, the subprocess module is a terrible blemish. It doesn’t look any better in Python 3.0, either, unfortunately.

Even more unfortunately, this is ultimately why it would be my preference to use either Ruby or shell. Ruby seems to work well for a variety of tasks, from writing a full-fledged webapp to some grungy text manipulation. You don’t have to compromise because it’s quite easy to use it for Python-y things as well as Perl-y things. It’s just a shame that Python seems to treat this case with a bizarre kind of fussiness incongruent with the rest of the language and standard library.

Once I found a WordPress plugin and a Vim syntax file, I was all set.

What’s more, Ubuntu has a package called python-markdown. I’m not sure whether or not I’ll use Markdown for anything other than blog posting, but on the other hand, it’s awfully tempting to take advantage of this kind of intuitive, powerful formatting.

Yes, I’ve taken a step down a dark path: I’m writing blog posts in Vim, in Linux. Although I use it for coding as often as possible, I’ve thus far avoided it for anything like actual writing.

If you’ve tried to get IPython working on Mac OS X Leopard (that’d be 10.5, and 10.5.7 in particular) with an Intel based CPU, you’ve probably had some problems. I know I did! Let me cut to the chase, with a more technical explanation to follow. Oh, and some mild bitching, too.

I’m following the instructions I originally saw here, on IPython’s Launchpad site.  Yes, my instructions are merely a reprint of someone else’s bug report. More on that after I explain.

Quick instructions

I’m going to assume you haven’t downloaded anything, although if, like me, you wrestled with this for a while, you can still do these steps and it’ll work. It worked for me, anyway. :/

1. Download the .egg for readline, presumably the latest version. Get IPython if you haven’t already. As of this writing, you can get it here at the IPython download site. You can s/i386/fat/ if you like.
2. Copy it to a directory, such as ~/python.
3. You can try to install readline and IPython: sudo easy_install readline ipython. One or both will fail.
4. Open /usr/local/bin/ipython in your favorite text editor, and replace the contents of the file as described at the IPython Launchpad bug linked above. The key is to hardcode readline into the Python system path immediately after sys, and then to launch IPython explicitly.
5. Launch ipython.

You should be OK now, tabbing and all. If you haven’t checked this shit out, using IPython for your system shell, I highly recommend that you do. It’s none too shabby. There’s a book on this, too.

Technical stuff after the jump.

wtf, man

So, here’s the skinny. It took me an hour or two to figure this out, which is way too long. I’m writing this post so that hopefully I can nail some of the search terms that weren’t around when I looked for it on the Googles. I’m mentioning a whole bunch of keywords and specific stuff in the hopes that the next person who has this problem gets here or to the Launchpad site and manages to solve their problem.

Anyway, the technical background: IPython wants GNU’s readline library (tabbing!), but Mac OS X ships with libedit. If you somehow manage to get IPython to work, tabbing won’t because IPython is using libedit. It sucks and practically defeats the purpose of IPython. I had this working briefly, and then I broke it somehow. Whee!

You can download the readline egg and try to install it with easy_install. It will try to compile, then error out. In particular, it errors out on Modules/readline.c:681 a bunch of times. I’m not 100% sure why it does this, and I’m too lazy to remember/figure it out again. I think it’s trying to compile the readline lib specifically for Python and failing because of some mismatch. It sucks and it makes me sad inside.

When I tried to run IPython anyway, I got a bunch of errors about unable to find an entry point. I suspect this is some failure related to  easy_install, egg, or pkg_resources and readline being missing, although it’s vastly unintuitive if so. You’ll get an ImportError related to console_scripts and ipython. It’ll say entry point not found.

The fix appears to be skipping the egg, easy_install, pkg_resource shenanigans by explicitly adding readline to your path and then explicitly invoking IPython instead of giving something else a chance to error out. Sounds good to me!

readline + IPython + Fink?

I didn’t try doing anything with Fink. Fink does have both a version of IPython and readline, so it’s possible that Fink would be a better choice overall. Honestly, I’d try that before anything else. I was too stubborn to give up halfway through.

Of course, not having tried it, I don’t know that it’ll work. It’s gotta be less painful than the hackery above, though, right? Right? I mean, what could possibly go wrong?