When I was introduced to Python, suddenly programming was fun again. It was hard to imagine going back to anything like C++ when I could become so productive so quickly with Python.

I held to this view for a very, very long time. I appreciated how languages like C++ or Java would be useful for writing “serious” applications, but I didn’t write that stuff. That was for other people.

In the last year or so, I’ve gradually come to believe that a strong type-checking system AND a high-productivity language ought not and are not mutually exclusive. The corollary is that I will consider carefully before I use a scripting language for anything more than a thousand or so lines.

The first sign of trouble

The first time I ran into serious trouble with a scripting language was a few years ago (yikes). I wrote a script in Python which would monitor some servers for updated software and kick off a job when they were updated. It was less than 1000 lines, but despite knowing better, I did a poor job keeping it clean. And because it was such a mess, of course I had no tests.

Without unit tests, I had to test it manually. The cycle was something like: write a bunch of code, and run the whole dang script. This was time-consuming and error-prone. I’d miss things. Basic things. The script itself wasn’t mission critical but I found these bugs and the script itself personally embarrassing; I knew better, and I knew I knew better.

Still, I was otherwise quite productive in Python. Around the same time, using the Django framework, I wrote a server which slurped XML-formatted test results and allowed the user to view results grouped by various properties. It was very fast to write, and I could hardly imagine doing it in C++ or Java.

I had three experiences in the last year or two which rearranged my prejudices.

JavaScript and the Closure Compiler

The first was working in JavaScript land, with the Closure Compiler. JavaScript was far less restrictive and, in my opinion, more expressive than Java. The compiler caught some of my dumb mistakes and I grudgingly came to value it. However, it didn’t catch all of my mistakes and the ones that it did not catch were sometimes difficult to track down.

The lesson I learned was that even the slightest type-checking can really help save you from yourself. It also makes refactoring (and thus maintenance) easier; adding or removing a parameter, or passing in the wrong type can be difficult to debug in a language like JavaScript.

Java

The second was working in Java land. I wrote a framework which would test a remote API. I had to do a lot of fighting with the language, with a lot of boilerplate and book-keeping. I missed closures and first class functions dearly; contrary to what I thought at first, Java does have a very limited form of closures, in the form of anonymous inner classes. The lengths I had to go to get certain pieces working was comical; at one point, I wrote a FactoryFactory. I’m not kidding! And I’m not proud of it, but the language and API I was using drove me to it.

However, up to this point in my career, I don’t think I’d spent as much time with a decent IDE on an extended project. Several aspects were revelatory. I’ll mention two.

1) The type system gave me substantially more confidence that I was doing the right thing most of the time. I spent more time fixing incorrect assumptions about the API, adding error-recovery, or simply writing code than chasing down basic mistakes. I do that in any language, but the difference here was that the IDE and language eliminated an entire class of mistakes as I introduced them. By the time I ran my program, I had substantially more confidence about its correctness than I ever did in Python.

2) If you haven’t yet, read Martin Fowler’s Refactoring. The short version is that there are a set of highly mechanical techniques you can use to facilitate refactoring. Furthermore, many Java IDEs support many of these operations. Even something as basic as renaming method calls or variables within a specific context is massively more helpful than dumb S&R.

Lest I seem insane

Despite all that, I still wouldn’t program in Java again by choice. I want closures. I want first class functions. I want built-in higher order functions, for that matter. I want syntax and/or a syntactic paradigm (think variables or methods like fooByBarWithBazAsQuux) that’s substantially less verbose.

It’s the difference between

numbers = [1, 2, 3, 4, 5, 6]
evens = numbers.select {|x| x % 2 == 0}

and

// I'm sure this is still invalid for many reasons.
ArrayList<Integer> intList = new ArrayList<Integer>() {
  1, 2, 3, 4, 5, 6
};
ArrayList<Integer> evenIntsList = new ArrayList<Integer>();
for (Integer i : intList) {
  if (i % 2 == 0) {
    evenIntsList.add(i)
  }
}

That? That up there? It hurts me inside. The compiler makes it harder for you to screw it up, but you still have to do the whole song and dance every. Single. Time. You spend ~6 lines on an operation which, in the grand scheme of your program, ought to be small and trivial.

Python

Speaking of Python, I started relatively recently on refactoring a set of Python scripts responsible for kicking off tests. Nobody owned these scripts and consequently they languished as multiple people had patched or hacked on them to fix bugs or add features. The scripts were a mess.

Furthermore, there are few thousand lines of these scripts, plus thousands of lines more in the form of libraries which the scripts use. There were and are tests, but obviously nobody ran them because they were all broken.

After having spent so much time in the Land of Type-checking, reality came rushing back. It scary to make changes without having any confidence that they were correct. I wrote tests to find basic mistakes which would not manifest until runtime.

I would say that this was the biggest shock and what did the most to change my views. A scripting language is very good for getting something running quickly, but in terms of refactoring and maintenance, it took two or three times as long as I thought it ought to.

A word about tests

You could make a strong case that the scenario above— writing tests to the point of boredom— is Working As Intended. I am strongly sympathetic to this idea.

You ought to write unit tests with good coverage no matter the language. Tools like autonose allow you to get feedback from those tests rather quickly. This workflow arguably gives you faster, more comprehensive feedback than one with just a compiler and manually run unit tests.

One hitch is that getting a good error message from a unit test requires an amount of diligence multiplied by the number of unit tests. A compiler does only catch a small subset of the problems a unit test would, but any decent one ought to catch them more reliably and precisely.

Anyway, I haven’t delved too much into larger-scale projects which are based on scripting languages, but my guess is that their approach is similar to this. Write testable code. Use frameworks which make it easy to write tests (with a nice DSL, flexible mocking, or what have you). Use frameworks which make it easy to run tests (such as speed and parallelism). Minimize the inherent friction and you’ll get something richer than a compiler alone would offer you.

The answer?

I’m still writing tests in Java. I’m still refactoring Python scripts. For obvious reasons it would be silly to introduce a new technology into an existing stack chiefly or even partially for my own personal satisfaction. Rewriting is a dubious proposition anyway, one that must be considered very carefully. So for practical reasons, the situation won’t change.

Even so, I still want to have some go-to language. What could I use for a new project, personal or otherwise which would have good odds of making me happy?

C++ is not memory safe. C is simpler than C++ but is also memory unsafe and has more trade-offs. In both C and C++, it can be very difficult to write basic functionality.

Java is memory safe and has a rich set of libraries, but as I’ve mentioned, I’m not interested.

So what’s the answer? What language or technology do I feel is most promising, offering a combination of strong typing and high productivity? I have one prospect, or even two. But I’ll save those for the next post.

Sometimes, in the course of one’s life, one is left with a one-off task. In this case, I needed to call a binary a whole lot of times, and do something with the output each time. The details aren’t important; I just needed to write a wrapper script for this binary, do a modest amount of processing on it, and then output the result to a file. This is a pretty common task, one to which the command line in general and *nix in particular is well-suited.

Now, for various reasons, I prefer not to do this via shell scripts. I don’t have a hard and fast rule for when or why. Unless you routinely write shell scripts (I don’t), you’ll inevitably spend a bunch of time looking stuff up that you looked up, oh, six months ago. Or at least I do. I don’t enjoy it, so if I can’t do it with a few commands, maybe one loop or so, then I prefer to use a scripting language. I have the advantage of better/clearer failure modes and simpler syntax and I feel like I’m learning something more powerful and more widely applicable in the process.

One of my “favorite” things to forget and look up again is the difference between [[ ]] and [ ] in tests. Another one is how you loop over a file or output from a command (while read line; do bar --baz $line; done <foo and foo | while read line; do bar $line; done, respectively).

If you have no trouble remembering this stuff or you write scripts often enough, I suppose this is moot. There was a time when I was scripting enough that I didn’t have to look all this stuff up every time, but that time is past. YMMV and all that.

Anyway, the point is that I probably could’ve done this with something like this (which is simplified somewhat):

ARGS="--bar=baz"
CMD="~/bin/foo"
KEY="quux"
MIN=0
MAX=10000
 
for $(seq 0 1000) num do;
  $CMD $num $ARGS | while read line;
    if egrep -q $KEY <<<$line; then
      value=$(egrep -o '[1-9]+$')
        echo "$num,$value"
    fi
  done
done

I’m sure I have a bajillion bugs in there. For example, I am not at all confident about the outer for loop syntax. And while it was easy to write, I’m not doing anything with command line args, doing any error handling, or anything like that. (On the other other hand, I uh underestimated how easy this was. That’s partially why I’m convinced there are bugs in there.)

I still don’t know how to write idiomatic (typo: idiotmatic) Ruby, so this is going to be very rough. Still, it feels natural to me, minus one or two hitches:

ARGS = "--bar=baz"
CMD = "~/bin/foo"
KEY = /quux/
MIN = 0
MAX = 10000
 
def main
  (MIN..MAX).each do |i|
    out = %x[ #{CMD} #{i} #{ARGS} ]
    out.split('\n').each do |line|
      KEY.match(line) do |m|
        value = line.split(' ')[1]
        print "#{i},#{value}\n"
      end
    end
  end
end

I’m not confident about the outer loop once again, and I’m not sure the call to match() will do what I want, let alone whether it’s elegant. Still, I feel pretty good about it. I love Ruby’s pattern of passing in blocks.

Fortunately or unfortunately, I had to use Python. And don’t get me wrong: I love Python. It is through Python that I learned to love scripting languages. Processing a file line by line was, I think, the real epiphany. And list comprehensions are wonderfully expressive.

But man alive is it awful for this sort of thing. I’m not even going to bother writing it out in Python. I’ll just excerpt, from the 2.7 subprocess docs. Here’s what they say should replace backticks output = \mycmd myarg``:

output = Popen(["mycmd", "myarg"], stdout=PIPE).communicate()[0]

Mind you, this is if you did from subprocess import *. Generally I don’t, which means it ends up looking like this:

output = \
  subprocess.Popen(["mycmd", "myarg"], subprocess.stdout=PIPE).\
  communicate()[0]

Yes, once you have it written, sequestered in its own function that you never touch again, it’s not so bad. However, this is firmly in the category of something I will not be able to (and have not been able to) remember months later.

It is also not discoverable in the sense that it’s highly particular— stdout=PIPE? Really? Compare and contrast opening a file (for line in open('somefile'): print foo(line) or [foo(line) for line in open('somefile')) with this monster. Even my Ruby example could have used backticks if I hadn’t remembered the %x syntax. The best I can say about subprocess is that it’s a) better than Popen in Python 2.4 (or whatever), and b) it’s easier to search the web for than %x.

The kicker of course is that rest of the Python script was very easy to write! However, since calling the binary was changing some external state, though, I had to make sure it was doing the right thing. In the end building and testing the call to subprocess.Popen() took longer than the rest of the script. In an otherwise elegant, no-bullshit, batteries included language, the subprocess module is a terrible blemish. It doesn’t look any better in Python 3.0, either, unfortunately.

Even more unfortunately, this is ultimately why it would be my preference to use either Ruby or shell. Ruby seems to work well for a variety of tasks, from writing a full-fledged webapp to some grungy text manipulation. You don’t have to compromise because it’s quite easy to use it for Python-y things as well as Perl-y things. It’s just a shame that Python seems to treat this case with a bizarre kind of fussiness incongruent with the rest of the language and standard library.

It’s JavaScript and Ruby that’ve influenced me most lately. Functional patterns in particular have attracted my attention. I really dig stuff like this in Ruby:

open('somefile').readlines.each do |x|
  print x
end

And then I spent a whole boatload of time in JavaScript, where you get to do stuff like this:

var someArray = [1, 4, 2, 3, 5];
goog.array.forEach(arr, function (e, i, a) {
  console.log("Here's a number: " + e);
});

What I really dig about this pattern is passing in a function where I can do whatever I like. The factory pattern becomes much simpler. We can pass this around to whoever:

var url = "/foo";
var reqFactory = function() {
  var req = new Request();
  req.setUrl(url);
  req.addHeader("baz", "quux");
  return req;
};

The current conundrum involves implementing some kind of retry logic. In one case, I need to build a new request each time the request fails. In another case, I just want to retry for a certain subset of exceptions. In yet another case, if I make the request and it doesn’t have what I want, I want to re-issue it.

In Java, it’s— it’s messy. I don’t know what the Java-esque way of doing it is, quite frankly. I find myself wanting to create a Retrier interface which defines an onSuccess, onFailure, and a doRetry method.

I’m not sure what each method should return; what if I want to be able to return whatever it is that we got from the request? Do I create a wrapper class w/generics? I think this is how it works in the Kingdom of Nouns. The nouns just keep proliferating.

The one thing I think Java has going for it is anonymous inner classes. I feel like the way files and classes proliferate in Java is nuts, and being able to keep the implementation of such as a Retrier close to where it’s used seems like a win. It’s a one-off, right? A whole class & implementation & file devoted to it feels like it’s belaboring the point.

I’m sure at some point I’ll figure it out and/or get used to it. In the meantime, every time I want to add a layer of abstraction, it feels like massive overkill. I almost wrote up code samples for the above but it turned out to be too much effort. I can’t be as glib with it, at least not yet. Oh well.