Our Blog

Changing Postgres pg_dump warnings into errors with sed

By Greg Sabino Mullane
October 28, 2013

Turning off warnings when loading a PostgreSQL data dump can be a trickier task than it may first appear. One of the goals of a program I was working on was to automatically load the schema of a production database into a development system. The program needed to stay as quiet as possible - the users running it did not need to be distracted by annoying messages generated by Postgres as the schema was loaded.

The problem occurred when a text-based dump of the database (created via pg_dump with the –schema-only flag) was loaded into an existing database with the help of the the psql program. To load the file “test.sql” into the database wilber, one usually does:

$ psql wilber -U postgres -f test.sql

There are two general classes of “noise” that are generated by such an action. I’ll show how to make both kinds quiet.

The first problem is that psql by default is very verbose and every single command gives an echoing confirmation of what just occurred. Thus, a typical schema load outputs a lot of things like this:

$ psql wilber -U postgres -f test.sql
SET
SET
SET
SET
SET
CREATE EXTENSION
COMMENT
SET
SET
SET
CREATE TABLE
ALTER TABLE
ALTER TABLE
CREATE SEQUENCE

These types of …

Full Page Caching in Interchange 5

By Mark Johnson
October 28, 2013

I recently attended the eCommerce Innovation Conference 2013 with fellow End Pointer Richard Templet and presented on the work End Point has done to develop full-page caching in Interchange 5. The work is in final review currently for inclusion into core Interchange and should provide a roadmap for cache management in Nitesi/Interchange 6.

Parameters of a Caching Solution

In order to identify the full scope of what one means when one says an application uses caching, there are (at least) 3 aspects to define:

• Where in the application stack is the cache being applied.
• How long until the cache is expired.
• What level of user state must it support.

The issue of cache duration is not addressed here as any such formulation will be specific to business requirements and tolerance of stale data. I will state, however, that even very brief cache durations can have an enormous impact on performance, particularly for sites that have a relatively small number of resources that absorb the bulk of the traffic. Cache durations of several minutes to several hours can drop the traffic that makes it to Interchange to a small fraction of the overall requests.

Caching and the Application Stack

Let’s …

How to DRY out your Active Record queries with Squeel

By Kamil Ciemniewski
October 24, 2013

Active Record alone isn’t enough when it comes to having a data-access code-base that is DRY and clean in real world Rails applications. You need a tool like the squeel gem and some good practices in place.

Using JOIN with Active Record is cumbersome

Having JOINs in composable data-access Rails code, quickly makes the code ugly and hard to read.

Consider this snippet of code taken directly from Rails guides, that gives developer a full control over the joining part of the query:

Client.joins('LEFT OUTER JOIN addresses ON addresses.client_id = clients.id')

That’s 77 characters for something as simple as a left outer join.

Now—​when using Rails, we almost always have properly defined associations in our models, which allows us to put in code something like this:

Category.joins(:posts)

That’s much better—​there’s not much code and you immediately know what it does. It isn’t “left outer join” but is useful nevertheless. You could do more with that feature. For example you could join with multiple associations at once:

Post.joins(:category, :comments)

Also—​join with nested associations:

Post.joins(comments: :guest)

Which produces:

SELECT posts.* FROM posts
INNER JOIN …

Controlling interactive programs with pexpect-u

By Miguel Alatorre
October 23, 2013

A client I am working with requires that various machines have Ubuntu 10.04.4 installed along with certain software dependencies prior to installation of their own software.

In order to have our client avoid the tedious task of spinning up a new machine for each new client of theirs, I decided to attempt to automate the process (minus the OS installation) in Python.

A couple of the software installations require the user to interact with a console application. For example, this is the Matlab Runtime Environment installer:

and the Passenger installer:

Here I used the Python package pexpect-u which allows you to spawn child applications and control them automatically.

To spawn the Matlab installer I run:

import pexpect
child = pexpect.spawn("sudo ./MCRInstaller.bin -console")

Now we tell pexpect what to expect:

child.expect("Press 1 for Next, 3 to Cancel or 5 to Redisplay \[1\]")

And we send a command with:

child.sendline("1")

The package can be found here and the source code includes many more examples, one of which might be of use for this very client: hive.py

This client has various installations of a Rails application and each sends requests to the …

Ecommerce Innovation 2013

By Richard Templet
October 16, 2013

Mark Johnson and I went to the Ecommerce Innovation 2013 conference in beautiful Hancock, NY. The event was hosted by Sam Batschelet of West Branch Resort. The conference was spread out over three days and was very well planned. We had plenty of time in between talks to mingle with the other people. All of the talks were very insightful and informative. I found the mixture of technology and marketing talks beneficial. I have already discussed some things with my clients that I learned.

A brief overview of the talks

• Jure Kodzoman of Informa had two different subjects.

  • His first talk was about Template::Flute which is a Perl based template system which is the default template for Interchange 6. It utilizes the use of html classes to figure out where to parse in the data returned from your Perl code. Overall it seems pretty straight forward to use.
  • His second talk was about the current state of the new Interchange 6 demo store.

• Ana Kozole of Informa had a talk named “Remarketing with Banners” that was really informative.The base of this is to have the ability to show specific banners to visitors on different websites. She discussed different remarketing techniques including …

PostgreSQL Debian apt repository GnuPG key update

By Jon Jensen
October 15, 2013

The excellent PGDG (PostgreSQL Global Development Group) apt repositories provide current point releases of supported PostgreSQL versions for Debian and Ubuntu LTS Linux. If you’d like to use a newer version of PostgreSQL than ships with your Linux distribution, or need to use an older Postgres release, you should take a look at http://wiki.postgresql.org/wiki/Apt.

A minor housekeeping matter arose just a few days ago: The GnuPG key used to sign the PostgreSQL packages expired on October 13. During a run of apt-get update && apt-get upgrade that leads to errors such as those seen here:

Get:1 http://security.debian.org wheezy/updates Release.gpg [836 B]
Get:2 http://security.debian.org wheezy/updates Release [102 kB]
Get:3 http://apt.postgresql.org wheezy-pgdg Release.gpg [836 B]
Get:4 http://ftp.de.debian.org wheezy Release.gpg [1,672 B]
Get:5 http://apt.postgresql.org wheezy-pgdg Release [29.2 kB]
Get:6 http://ftp.de.debian.org wheezy-updates Release.gpg [836 B]
Get:7 http://ftp.de.debian.org wheezy Release [168 kB]
Err http://apt.postgresql.org wheezy-pgdg Release

Get:8 http://security.debian.org wheezy/updates/main Sources [61.2 kB]
Get:9 http://security.debian.org …

First Dance: a Gentle Introduction to Dancer.pm for Web Services

By Jeff Boes
October 7, 2013

I’ve been dabbling in Dancer (Version One, not so much Version Two). Our first opportunity to create a production-worthy Dancer application just rolled out with encouraging results, so I thought I would outline some things we tried, and things we learned. Don’t worry if you don’t know a thing about Dancer; I’ll educate you a little as we go along.

First, some background. This application was a re-host and rewrite of an existing application written in JavaScript and plain old CGI Perl scripts, returning either HTML or fairly simple JSON objects. Even the database it connected to was ancient DBM files. Those files were created by extracting data from the main website’s database, so that part was easy to replace—​we just had to use the extraction queries (mostly) to connect directly.

We chose Dancer as the platform for this effort because of its purely Perl footprint, and its relatively easy deployment.

Since I didn’t really want to rewrite the front-end JavaScript code, I left that mostly as-is and concentrated on replacing the CGI scripts. The first step was to move all their code into a non-Dancer Perl module, which communicated with the outside world entirely in the form of …

Newest European Liquid Galaxy installation: Google Hamburg shop-in-shop

By Bianca Rodrigues
October 2, 2013

In September 2013, Google opened a store-in-store within the large electronics retailer Saturn in Hamburg. Dominating the visuals of the store is a large Liquid Galaxy on the back wall.

We partnered with system integrator Pixel Projects this time, and instead of our usual 7 screen display, worked the Liquid Galaxy into an unique 3x3 video wall array. The Liquid Galaxy shows Google Earth and will be showing Google Street View starting later this month, further highlighting the rich set of services offered by Google. It is also integrated with traditional video wall software configured by Pixel Projects such that it it can be switched to displaying Google services such as YouTube, Google Drive, Hangouts, and more. As always, the Liquid Galaxy is a huge customer draw. Customers who come through the store can explore any point on the globe while they browse the Google products.

End Point worked under the direction of the retail experience agency, Make Retail, to design a fully integrated experience. Once the unique configuration of the Liquid Galaxy was set, we sent along one of our top engineers, Matt Vollrath, to oversee the final hardware and software integration and …