Our Blog

Downstream Implications of Data Page Checksums

By Josh Williams
October 30, 2013

Now that Postgres 9.3 is all the rage, page checksums are starting to see use in production. It’s not enabled by default during initdb, so you may want to double check the options used when you upgraded.

What? You have already upgraded to 9.3, right? No? Oh well, when you do get around to updating, keep an eye out for initdb’s –data-checksums option, or just -k. To give the feature a try on my development desktop, after the initdb I created a table and loaded in some text data. Small text strings are being cast from integers so we can more easily see it in the on-disk structure. You’ll see why in a moment. The table was loaded with a good amount of data, at least more than my shared_buffers setting:

postgres=# CREATE TABLE filler (txt TEXT PRIMARY KEY);
CREATE TABLE
postgres=# INSERT INTO filler SELECT generate_series::text FROM generate_series(-10000000,10000000);
INSERT 0 20000001
postgres=# \dt+
List of relations
 Schema |  Name  | Type  |  Owner   |  Size  | Description
--------+--------+-------+----------+--------+-------------
 public | filler | table | postgres | 761 MB |
(1 row)

There. Maybe a little more than I needed, but it works. My storage (on this desktop) is …

SSL Certificate SANs and Multi-level Wildcards

By Cas Rusnov
October 29, 2013

Some bits of advice for those that run their own Certificate Authorities or use self-signed certificates, related to multiple matches and wildcard domains.

In some circumstances it’s desirable to match multiple levels of wildcards in an SSL certificate. One example of this is in our Camp development system (whose domain names are in the format n.camp.foo.com, where n is a numeric identifier of the camp), where having a certificate which matches something like fr.0.camp.foo.com and also en.91.camp.foo.com would be needed.

The most obvious way to do this is to create a certificate whose commonName is ..camp.foo.com; unfortunately this is also not a working solution, as it is unsupported with current-day browsers. The alternative is to create a subjectAltName (which is an alias within the certificate for the subject of the certificate, abbreviated SAN) for each subdomain which we want to wildcard. For Camps this works well because the subdomains are in an extremely regular format so we can create a SAN for each [0..99].camp.foo.com. One caveat is that if SANs are in use they must also contain the commonName (CN) as an alternate name, since the browser will ignore the CN in that case …

Changing Postgres pg_dump warnings into errors with sed

By Greg Sabino Mullane
October 28, 2013

Turning off warnings when loading a PostgreSQL data dump can be a trickier task than it may first appear. One of the goals of a program I was working on was to automatically load the schema of a production database into a development system. The program needed to stay as quiet as possible - the users running it did not need to be distracted by annoying messages generated by Postgres as the schema was loaded.

The problem occurred when a text-based dump of the database (created via pg_dump with the –schema-only flag) was loaded into an existing database with the help of the the psql program. To load the file “test.sql” into the database wilber, one usually does:

$ psql wilber -U postgres -f test.sql

There are two general classes of “noise” that are generated by such an action. I’ll show how to make both kinds quiet.

The first problem is that psql by default is very verbose and every single command gives an echoing confirmation of what just occurred. Thus, a typical schema load outputs a lot of things like this:

$ psql wilber -U postgres -f test.sql
SET
SET
SET
SET
SET
CREATE EXTENSION
COMMENT
SET
SET
SET
CREATE TABLE
ALTER TABLE
ALTER TABLE
CREATE SEQUENCE

These types of …

Full Page Caching in Interchange 5

By Mark Johnson
October 28, 2013

I recently attended the eCommerce Innovation Conference 2013 with fellow End Pointer Richard Templet and presented on the work End Point has done to develop full-page caching in Interchange 5. The work is in final review currently for inclusion into core Interchange and should provide a roadmap for cache management in Nitesi/Interchange 6.

Parameters of a Caching Solution

In order to identify the full scope of what one means when one says an application uses caching, there are (at least) 3 aspects to define:

• Where in the application stack is the cache being applied.
• How long until the cache is expired.
• What level of user state must it support.

The issue of cache duration is not addressed here as any such formulation will be specific to business requirements and tolerance of stale data. I will state, however, that even very brief cache durations can have an enormous impact on performance, particularly for sites that have a relatively small number of resources that absorb the bulk of the traffic. Cache durations of several minutes to several hours can drop the traffic that makes it to Interchange to a small fraction of the overall requests.

Caching and the Application Stack

Let’s …

How to DRY out your Active Record queries with Squeel

By Kamil Ciemniewski
October 24, 2013

Active Record alone isn’t enough when it comes to having a data-access code-base that is DRY and clean in real world Rails applications. You need a tool like the squeel gem and some good practices in place.

Using JOIN with Active Record is cumbersome

Having JOINs in composable data-access Rails code, quickly makes the code ugly and hard to read.

Consider this snippet of code taken directly from Rails guides, that gives developer a full control over the joining part of the query:

Client.joins('LEFT OUTER JOIN addresses ON addresses.client_id = clients.id')

That’s 77 characters for something as simple as a left outer join.

Now—​when using Rails, we almost always have properly defined associations in our models, which allows us to put in code something like this:

Category.joins(:posts)

That’s much better—​there’s not much code and you immediately know what it does. It isn’t “left outer join” but is useful nevertheless. You could do more with that feature. For example you could join with multiple associations at once:

Post.joins(:category, :comments)

Also—​join with nested associations:

Post.joins(comments: :guest)

Which produces:

SELECT posts.* FROM posts
INNER JOIN …

Controlling interactive programs with pexpect-u

By Miguel Alatorre
October 23, 2013

A client I am working with requires that various machines have Ubuntu 10.04.4 installed along with certain software dependencies prior to installation of their own software.

In order to have our client avoid the tedious task of spinning up a new machine for each new client of theirs, I decided to attempt to automate the process (minus the OS installation) in Python.

A couple of the software installations require the user to interact with a console application. For example, this is the Matlab Runtime Environment installer:

and the Passenger installer:

Here I used the Python package pexpect-u which allows you to spawn child applications and control them automatically.

To spawn the Matlab installer I run:

import pexpect
child = pexpect.spawn("sudo ./MCRInstaller.bin -console")

Now we tell pexpect what to expect:

child.expect("Press 1 for Next, 3 to Cancel or 5 to Redisplay \[1\]")

And we send a command with:

child.sendline("1")

The package can be found here and the source code includes many more examples, one of which might be of use for this very client: hive.py

This client has various installations of a Rails application and each sends requests to the …

Ecommerce Innovation 2013

By Richard Templet
October 16, 2013

Mark Johnson and I went to the Ecommerce Innovation 2013 conference in beautiful Hancock, NY. The event was hosted by Sam Batschelet of West Branch Resort. The conference was spread out over three days and was very well planned. We had plenty of time in between talks to mingle with the other people. All of the talks were very insightful and informative. I found the mixture of technology and marketing talks beneficial. I have already discussed some things with my clients that I learned.

A brief overview of the talks

• Jure Kodzoman of Informa had two different subjects.

  • His first talk was about Template::Flute which is a Perl based template system which is the default template for Interchange 6. It utilizes the use of html classes to figure out where to parse in the data returned from your Perl code. Overall it seems pretty straight forward to use.
  • His second talk was about the current state of the new Interchange 6 demo store.

• Ana Kozole of Informa had a talk named “Remarketing with Banners” that was really informative.The base of this is to have the ability to show specific banners to visitors on different websites. She discussed different remarketing techniques including …

PostgreSQL Debian apt repository GnuPG key update

By Jon Jensen
October 15, 2013

The excellent PGDG (PostgreSQL Global Development Group) apt repositories provide current point releases of supported PostgreSQL versions for Debian and Ubuntu LTS Linux. If you’d like to use a newer version of PostgreSQL than ships with your Linux distribution, or need to use an older Postgres release, you should take a look at http://wiki.postgresql.org/wiki/Apt.

A minor housekeeping matter arose just a few days ago: The GnuPG key used to sign the PostgreSQL packages expired on October 13. During a run of apt-get update && apt-get upgrade that leads to errors such as those seen here:

Get:1 http://security.debian.org wheezy/updates Release.gpg [836 B]
Get:2 http://security.debian.org wheezy/updates Release [102 kB]
Get:3 http://apt.postgresql.org wheezy-pgdg Release.gpg [836 B]
Get:4 http://ftp.de.debian.org wheezy Release.gpg [1,672 B]
Get:5 http://apt.postgresql.org wheezy-pgdg Release [29.2 kB]
Get:6 http://ftp.de.debian.org wheezy-updates Release.gpg [836 B]
Get:7 http://ftp.de.debian.org wheezy Release [168 kB]
Err http://apt.postgresql.org wheezy-pgdg Release

Get:8 http://security.debian.org wheezy/updates/main Sources [61.2 kB]
Get:9 http://security.debian.org …