Our Blog

The how and why of Code Reviews

By Daniel Browning
August 7, 2008

Everyone believes that code reviews are highly beneficial to software and web site quality. Yet many of those who agree in principle don’t follow through with them in practice, at least not consistently or thoroughly. To find ways to improve real-world practice, I attended Code Reviews for Fun and Profit, given by Alex Martelli, Über Tech Lead at Google, during OSCON 2008.

One barrier to good reviews is when developers are reluctant to point out flaws in the code of more experienced programmers, perhaps due to culture or personal dynamics. In Open Source projects, and at End Point, the reverse is often true: corrections earn Nerd Cred. But if it is an issue, one good workaround is to ask questions. Instead of “If you use a value of zero, it crashes,” say “What happens if you use a value of zero?”

There are several prerequisites that should be taken care of before code reviews are started. First, a version control system is required (we prefer Git at End Point). Second, a minimal amount of process should be in place to ensure reviews occur, so that some commits do not fall through the cracks. Third, automatable tasks, such as style, test coverage, and smoke tests, should be …

Testing Concurrency Control

By Ethan Rowe
August 6, 2008

When dealing with complex systems, unit testing sometimes poses a bigger implementation challenge than does the system itself.

For most of us working in the web application development space, we tend to deal with classic serial programming scenarios: everything happens in a certain order, within a single logical process, be it a Unix(-like) process or a thread. While testing serial programs/modules can certainly get tricky, particularly if the interface of your test target is especially rich or involves interaction with lots of other widgets, it at least does not involve multiple logical lines of execution. Once concurrency is brought into the mix, testing can become inordinately complex. If you want to test the independent units that will operate in parallel, you can of course test each in isolation and presumably have simple “standard” serial-minded tests that are limited to the basic behaviors of the independent units in question. If you need to test the interaction of these units when run in parallel, however, you will do well to expect Pain.

One simple pattern that has helped me a few times in the past:

• identify what it is that you need to verify
• in your test script, fork at …

Perl on Google App Engine

By Jon Jensen
August 2, 2008

People are working on getting Perl support for Google App Engine, led by Brad Fitzpatrick (of Livejournal, memcached, etc. fame) at Google.

They’ve created a new module, Sys::Protect, to simulate the restricted Perl interpreter that would have to exist for Google App Engine. There’s some discussion of why they didn’t use Safe, but it sounds like it’s based only on rumors of Safe problems, not anything concrete.

Safe is built on Opcode, and Sys::Protect appears to work the same way Safe + Opcode do, by blocking certain Perl opcodes. All the problems I’ve heard of and personally experienced with Safe were because it was working just fine—​but being terribly annoying because many common Perl modules do things a typical Safe compartment disallows. That’s because most Perl module writers don’t use Safe and thus never encounter such problems. It seems likely that Sys::Protect and a hardened Perl Google App Engine environment will have the same problem and will have to modify many common modules if they’re to be used.

Moving on, posters are talking about having support for Moose, Catalyst, CGI::Application, POE, Template::Toolkit, HTML::Template … well, a lot. I guess that makes …

Switching from Sendmail to Postfix on OpenBSD

By Jon Jensen
August 1, 2008

It’s easy to pick on Sendmail, and with good reason. A poor security record, baroque configuration, slowness, painful configuration, monolithic design, and arcane configuration. Once you know Sendmail it’s bearable, and long-time experts aren’t always eager to give it up, but I wouldn’t recommend anyone deploy it for a serious mail server these days. But for a send-only mail daemon or a private, internal mail server, it works fine. Since it’s the default mailer for OpenBSD, and I haven’t been using OpenBSD as a heavy-traffic mail server, I’ve usually just left Sendmail in place.

A few years ago some of our clients’ internal mail servers running Sendmail were getting heavy amounts of automated output from cron jobs, batch job output, transaction notifications, etc., and they bogged down and sometimes even stopped working entirely under the load. It wasn’t that much email, though—​the machines should’ve been able to handle it.

After trying to tune Sendmail to be more tolerant of heavy load and having little success, I finally switched to Postfix (which we had long used elsewhere) and the CPU load immediately dropped from 30+ down to below 1, and mail delivery worked without …

Code Debt-Free

By Ethan Rowe
July 31, 2008

Every now and then, the opportunity arises to write debt-free code (meaning free of technical debt). When such opportunities come, we must seize them.

I recently had the distinct pleasure of cranking out some Perl modules in the following order:

1. Write documentation for the forthcoming functionality

2. Implement unit tests for the aforementioned forthcoming functionality

3. Verify that the unit tests fail

4. Implement the awaited functionality

5. Verify (jumping back to step 4 as necessary) that the unit tests work.

Timelines, interruptions, and other pressures often get in the way of this short-term development cycle. The cycle can feel tedious; it makes the task of implementing even simple functions seem unpleasantly large and drawn out. When an implementation approach flashes into the engineer’s mind, leaping to step 4 (implementation) feels natural and immediately gratifying. The best-intentioned of us can fall into this out of habit, out of inertia, out of raw enthusiasm.

Documentation, though, demonstrates that you know what you’re trying to achieve. It is not a nicety, it is proof that you understand the problem at hand. Unit tests, as hard as they can sometimes be to …

MySQL vs. PostgreSQL mailing list activity

By Jon Jensen
July 31, 2008

My co-worker, Greg Sabino Mullane, noted this writeup on the MarkMail blog comparing the amount of traffic on the various MySQL and PostgreSQL mailing lists.

I suppose you could pessimistically say that PostgreSQL users need more community help than MySQL users do, but reviewing the content of the traffic (and going from years of personal experience) doesn’t support such a view. The PostgreSQL community seems to have more long-term, deeply involved users who are also contributors.

But let’s hope the competition in the free database world picks up. It looks like the new Drizzle project has a good chance of growing a new community around MySQL.

In any case, the MarkMail mailing list archive and search service is an excellent resource. Thanks, MarkMail folks!

Git push: know your refspecs

By Ethan Rowe
July 30, 2008

The ability to push and pull commits to/from remote repositories is obviously one of the great aspects of Git. However, if you’re not careful with how you use git-push, you may find yourself in an embarrassing situation.

When you have multiple remote tracking branches within a Git repository, any bare git push invocation will attempt to push to all of those remote branches out. If you have commits stacked up that you weren’t quite ready to push out, this can be somewhat unfortunate.

There are a variety of ways to accommodate this:

• use local branches for your commits, only merging those commits into your remote tracking branches when you’re ready to push them out;
• push remote tracking branches out whenever you have something worth committing.

However, even with sensible branch management practices, it’s worthwhile to know exactly what it is you’re pushing. Therefore, if you want to have a sense of what you’re potentially doing in calling a bare git push, always call it with the –dry-run option first. This will show you what a the push will send out, where the conflicts are, and so on, all without actually performing the push.

It is ultimately best, though, to understand the …

Building Perl on 64-bit RHEL/Fedora/CentOS

By Jon Jensen
July 28, 2008

When building Perl from source on 64-bit Red Hat Enterprise Linux, Fedora, CentOS, or derivatives, Perl’s Configure command needs to be told about the “multilib” setup Red Hat uses.

The multilib arrangement allows both 32-bit and 64-bit libraries to exist on the same system, and leaves the “non-native” 32-bit libraries in /lib and /usr/lib while the “native” 64-bit libraries go in /lib64 and /usr/lib64. That allows the same 32-bit RPMs to be used on either i386 or x86_64 systems. The downside of this is that 64-bit applications have to be told where to look for, and put, libraries, or they usually won’t work.

For Perl, to compile from a source tarball with the defaults:

./Configure -des -Dlibpth="/usr/local/lib64 /lib64 /usr/lib64"

Then build as normal:

make && make test && sudo make install

I hope this information will come in handy for someone. I believe I learned it from Red Hat’s source RPM for Perl.