Our Blog

Best practices for cron

By Greg Sabino Mullane
December 8, 2008

Cron is a wonderful tool, and a standard part of all sysadmins toolkit. Not only does it allow for precise timing of unattended events, but it has a straightforward syntax, and by default emails all output. What follows are some best practices for writing crontabs I’ve learned over the years. In the following discussion, “cron” indicates the program itself, “crontab” indicates the file changed by “crontab -e”, and “entry” begin a single timed action specified inside the crontab file. Cron best practices:

Version control

This rule is number one for a reason. Always version control everything you do. It provides an instant backup, accountability, easy rollbacks, and a history. Keeping your crontabs in version control is slightly more work than normal files, but all you have to do is pick a standard place for the file, then export it with crontab -l > crontab.postgres.txt. I prefer RCS for quick little version control jobs like this: no setup required, and everything is in one place. Just run: ci -l crontab.postgres.txt and you are done. The name of the file should be something like the example shown, indicating what it is (a crontab file), which one it is (belongs to the user …

Creating a PL/Perl RPM linked against a custom Perl build (updated)

By Jon Jensen
November 29, 2008

I recently needed to refer to a post I made on March 7, 2007, showing how to build a PL/Perl RPM linked against a custom Perl build. A few things have changed since that time, so I’ve reworked it here, updated for local Perl 5.10.0 built into RPMs:

We sometimes have to install a custom Perl build without thread support, and to have some specific newer and/or older versions of CPAN modules, and we don’t want to affect the standard distribution Perl that lives in /usr/bin/perl and /usr/lib/perl5. We use standard PGDG RPMs to install PostgreSQL. We also use PL/Perl, and want PL/Perl to link against our custom Perl build in /usr/local/bin and /usr/local/lib/perl5.

It’s easy to achieve this with a small patch to the source RPM spec file:

--- postgresql-8.3.spec 2008-10-31 17:34:34.000000000 +0000
+++ postgresql-8.3.custom.spec  2008-11-30 02:10:09.000000000 +0000
@@ -315,6 +315,7 @@
 CFLAGS=`echo $CFLAGS|xargs -n 1|grep -v ffast-math|xargs -n 100`

 export LIBNAME=%{_lib}
+export PATH=/usr/local/bin:$PATH
 %configure --disable-rpath \
 %if %beta
    --enable-debug \
@@ -322,6 +323,7 @@
 %endif
 %if %plperl
    --with-perl \
+ …

Multiple reverse DNS pointers per IP address

By Jon Jensen
November 28, 2008

I recently ran across an IP address that had two PTR (reverse DNS) records in DNS. I’ve always thought that each IP address is limited to only a single PTR record, and I’ve seen this rule enforced by many ISPs, but I don’t remember ever seeing it conclusively stated.

I was going to note the problem to the responsible person but thought it’d be good to test my assumption first. Lo and behold, it’s not true. The Wikipedia “Reverse DNS lookup” page and a source it cites, an IETF draft on reverse DNS, note that multiple PTR records per IP address have always been allowed.

There is apparently plenty of software out there that can’t properly deal with more than one PTR record per IP address, and with too many PTR records, a DNS query response will no longer fit inside a single UDP packet, forcing a TCP response instead, which can cause trouble of its own. And as I noted, many ISPs won’t allow more than one PTR record, so in those cases it’s an academic question.

But it’s not invalid, and I saved myself and someone else a bit of wasted time by doing a quick bit of research. It was a good reminder of the value of checking assumptions.

OpenSQL Camp 2008

By Greg Sabino Mullane
November 19, 2008

I attended the OpenSQL Camp last weekend, which ran Friday night to Sunday, November 14-16th. This was the first “unconference” I had been to, and Baron Schwartz did a great job in pulling this all together. I drove down with Bruce Momjian who said that this is the first cross-database conference of any kind since at least the year 2000.

The conference was slated to start at 6 pm, and Bruce and I arrived at our hotel a few minutes before then. Our hotel was at one end of the Charlottesville Downtown Mall, and the conference was at the other end, so we got a quick walking tour of the mall. Seems like a great place—​lots of shops, people walking, temporary booths set out, outdoor seating for the restaurants. It reminded me a lot of Las Ramblas, but without the “human statue” performance artists. Having a hotel within walking distance of a conference is a big plus in my book, and I’ll go out of my way to find one.

The first night was simply mingling with other people and designing the next day’s sessions. There was a grid of talk slots on a wall, with large sticky notes stuck to some of them to indicate already-scheduled sessions. Next to the grid were two sections, where people …

Google Sponsored AFS Hack-A-Thon

By Max Cohan
November 13, 2008

Day One:

Woke up an hour early, due to having had a bit of confusion as to the start time (the initial email was a bit optimistic as to what time AFS developers wanted to wake up for the conference).

Met up with Mike Meffie (an AFS Developer of Sine Nomine) and got a shuttle from the hotel to the ‘Visitors Lobby’; only to find out that each building has a visitors lobby. One neat thing, Google provides free bikes (beach cruisers) to anyone who needs them. According to the receptionist, any bike that isn’t locked down is considered public property at Google. However, it’s hard to pedal a bike and hold a briefcase; so off we went hiking several blocks to the correct building. Mike was smart enough to use a backpack, but hiked with me regardless.

The food was quite good, a reasonably healthy breakfast including fresh fruit (very ripe kiwi, and a good assortment). The coffee was decent as well! After much discussion, it was decided that Mike & I would work towards migrating the community CVS repository over to Git. Because Git sees the world as ‘patch sets’ instead of just individual file changes, migrating it from a view of the ‘Deltas’ makes the most sense. The new Git repo. …

10,000 databases on a PostgreSQL cluster

By Jon Jensen
November 10, 2008

One of our clients wanted confirmation that PostgreSQL will have no problem handling 1000 or 2000 databases on a single database cluster. I remember testing some years ago, probably on Postgres 7.2 or 7.3, creating 1000 or so databases and finding that it worked fine. But that was a long time ago, the software has changed, and I thought I should make sure my old experiment results still stand.

There’s a PostgreSQL FAQ question, “What is the maximum size for a row, a table, and a database?” but no mention of the maximum number (or more importantly, maximum practical number) of databases per cluster. So I threw together a test script to create 10,000 databases, each with between (randomly) 1-5 tables with 2 columns each (INTEGER and TEXT), each getting randomly between 1-10 inserts with random data up to 100 or so characters in the TEXT field.

I ran the test on PostgreSQL 8.1, the default that ships with Red Hat Enterprise Linux 5 (x86_64). The hardware was a desktop-class HP with an Intel Core 2 @ 1.86 GHz that wasn’t always idle.

The short answer: Postgres 8.1 handles 10,000 databases just fine. \l in psql generates a long list of databases, of course, but returns quickly enough. …

Varnish, Radiant, etc.

By Ethan Rowe
October 26, 2008

As my colleague Jon mentioned, the Presidential Youth Debates launched its full debate content this week. And, as Jon also mentioned, the mix of tools involved was fairly interesting:

• Radiant, a simple Rails-based CMS
• Varnish, a high-performance reverse proxy server
• Amazon Simple Storage Service, a scalable storage layer
• PostgreSQL, the truly marvelous open-source relational database management system that End Point loves so well

Our use of Postgres for this project was not particularly special, and is simply a reflection of our using Postgres by default. So I won’t discuss the Postgres usage further (though it pains me to ignore my favorite piece of the software stack).

Radiant

Dan Collis-Puro, who has done a fair amount of CMS-focused work throughout his career, was the initial engineer on this project and chose Radiant as the backbone of the site. He organized the content within Radiant, configured the Page Attachments extension for use with Amazon’s S3 (Simple Storage Service), and designed the organization of videos and thumbnails for easy administration through the standard Radiant admin interface. Furthermore, prior to the release of the debate videos, Dan built a question …

Ohio Linux Fest AFS Hackathon

By Steven Jenkins
October 24, 2008

The one-day Ohio Linux Fest AFS Hackathon flew by in a hurry. Those new to OpenAFS got started converting some commented source code into Doxygen-usable format to both improve the code documentation as well as get a feel for some of the subsystems in OpenAFS. Several of the developers took advantage of the time to analyze some outstanding issues in Demand Attach (DAFS). We also worked on some vldb issues and had several good conversations about AFS roadmaps, Rx OSD, the migration from CVS to Git, and the upcoming Google-sponsored AFS hackathon.

The Doxygen gave those new to OpenAFS code a chance to look under the covers of OpenAFS. Doxygen produces pretty nice output from simple formatting commands, so it’s really just a matter of making comments follow some basic rules. Sample Doxygen output (from some previous work) can be seen here, and some of the new Doxygen changes made to OpenAFS are already.

The Demand Attach work focused on the interprocess communications pieces, namely the FSSYNC & SALVSYNC channels, specifying requirements and outlining the approaches for implementing bi-directional communications so that the failure of one process would not leave a volume in an …