Our Blog

Surge 2010 wrap-up

By Jon Jensen
October 5, 2010

Following up on my earlier post about day 1 of the conference, here is an unsorted collection of what I felt were noteworthy observations made in talks at Surge 2010:

Web engineering as a separate discipline from computer science or software development started around 1999. It is interdisciplinary, involving human factors engineering, systems engineering, operations research, fault-tolerant design, and control systems engineering. (John Allspaw)

A real-time system is one in which the correctness of a system is tied to its timeliness. Eventual consistency is an oxymoron if timeliness is part of the data itself. Caching by CDNs can’t solve our problems here. (Bryan Cantrill)

Pre-fab metrics are worth less (and maybe worthless) when not tied to something in your business. Message queues enable lots of new uses because of the ability to have multiple observers. See Esper (Java, GPL) for live ongoing SQL-like queries of messages from AMQP sources, etc. (Theo Schlossnagle)

On scaling up vs. out: If your numbers show “up” is enough, be happy you can keep your system simpler. (Theo Schlossnagle)

Anyone can only ever know the past in a distributed system. There’s no such thing as global …

Two Cool Things about Liquid Galaxy

By Benjamin Goldstein
October 4, 2010

I. It uses COTS Hardware.

Liquid Galaxy is suitable for using with COTS (Commodity Off The Shelf) hardware. Yes, Google Earth itself is rather resource intensive, so it helps performance to use feisty computers (including ones with SSDs) but it’s still COTS hardware. Of course the very cool thing about using COTS hardware is that the price is right and gets better all the time.

II. A simple, elegant and powerful master/​slave configuration and communication approach

Liquid Galaxy works by configuring its “slave” systems to have offsets from the point of view of the master system that the system’s user navigates on. The slave systems “know” their locations relative to the master system. The master system broadcasts its location to the slaves via UDP packets. It’s then up to the slave systems to figure out what portion of a Google Earth globe they need to retrieve themselves relative to the coordinates broadcast from the master system.

With this approach it’s easy to scale to a large number of slave systems. An interesting extension to this configuration and communication approach that the Google engineering team for the project provided for is the ability to configure one or more …

Surge 2010 day 1

By Jon Jensen
October 1, 2010

Today (technically, yesterday) was the first day of the Surge 2010 conference in Baltimore, Maryland. The Tremont Grand venue is perfect for a conference. The old Masonic lodge makes for great meeting rooms, and having a hallway connect it to the hotel was nice to avoid the heavy rain today. The conference organization and scheduling and Internet have all been solid. Well done!

There were a lot of great talks, but I wanted to focus on just one that was very interesting to me: Artur Bergman’s on scaling Wikia. Some points of interest:

• They (ab)use Google Analytics to track other things besides the typical pages viewed by whom, when. For example, page load time as measured by JavaScript, with data sent to a separate GA profile for analysis separately from normal traffic. That is then correlated with exit rates to give an idea of the benefit of page delivery speed in terms of user stickiness.

• They use the excellent Varnish reverse proxy cache.

• 500 errors from the origin result in a static page served by Varnish, with error data hitting a separate Google Analytics profile.

• They have both geographically distributed servers and team.

• They’ve found SSDs (solid state disks) to be …

Google Liquid Galaxy Support

By Benjamin Goldstein
September 30, 2010

End Point is tremendously excited to announce our turnkey installation and support service for Google’s Liquid Galaxy. Google has just today announced the release of Liquid Galaxy to the general public and to its Earth Enterprise customers. Liquid Galaxy is an astounding 3D immersive environment for Google Earth. End Point’s turnkey installation service includes on-site booth construction, precise installation of compute nodes and displays, plus expert configuration of operating system, networking, and Google Earth client software. End Point also provides support for upgrades and troubleshooting.

End Point has extensive hands-on experience with installing and configuring Liquid Galaxy. We’ve installed a number of Galaxies for Google itself in the US and abroad. We also recently installed the Liquid Galaxy unveiled this month at the San Jose Tech Museum’s spectacular new Silicon Valley Innovation Gallery. This is the first Liquid Galaxy deployed outside of a Google office or trade show.

Google’s Liquid Galaxy is 3-dimensional without your having to wear funny glasses. Google’s reference implementation specifies a circular booth 12’4” (3.75m) in diameter with 8 large high definition …

Gift Certificates: Easy to Say, Hard to Do

By Carl Bailey
September 29, 2010

A recent blog article by Steph goes through the steps taken to handle implementation of gift certificates using Spree’s coupon/discount model. At the end of the article, she rightly concludes that gift certificates are not that well suited to the discount model and need to be handled separately. Gift certificates are rightly handled as a method of payment—​they should have no effect on the amount of the sale, sales tax or shipping charges. This is an attempt to further that discussion.

Some people want to put rules on to how gift certificates can be used: Minimum purchase, expiration dates, etc. However, these are rules that apply to qualification: Does the person or the sale qualify to get a lower price? To my mind, these are not rules that can be applied to true gift certificates. A gift certificate is a business liability. The merchant has the money, and the customer should be allowed to spend it as she sees fit, without having to pass any qualifications separate from those that apply to everyone else. So gift certificates should be treated by the system as a form of payment, and customers should not have to qualify just to use them.

To implement a gift certificate system, we …

Debugging jQuery

By Ron Phipps
September 28, 2010

A recent reskin project for a client requires that we convert some of their old Prototype code to jQuery as well as create new jQuery code. I have not done much with converting Prototype to jQuery and I felt like my debugging tools for JavaScript were under par. So this morning I set out to find what was available for jQuery and I found this article on the subject.

I’ve used Firebug for some time now, but was unaware of some of the supporting plugins that would certainly help with debugging JavaScript. Some of the plugins and methods found in the article that I found immediately helpful were:

• FireFinder: Makes it quite easy to verify that the selector values in your code are correct and that the proper elements are returned. I was able to immediately pinpoint problems with my selectors and this brought to light why certain events were not firing.

• Firebug Console: Using the console.log function allowed me to check values without littering my code with alert statements.

• FireQuery: At a glance this plugin for Firebug shows which elements have event handlers bound to them.

• Firebug Breakpoints: Setting breakpoints and watch statements in your code makes it easier to see what is …

PostgreSQL 8.4 in RHEL/CentOS 5.5

By Jon Jensen
September 28, 2010

The announcement of end of support coming soon for PostgreSQL 7.4, 8.0, and 8.1 means that people who’ve put off upgrading their Postgres systems are running out of time before they’re in the danger zone where critical bugfixes won’t be available.

Given that PostgreSQL 7.4 was released in November 2003, that’s nearly 7 years of support, quite a long time for free community support of an open-source project.

Many of our systems run Red Hat Enterprise Linux 5, which shipped with PostgreSQL 8.1. All indications are that Red Hat will continue to support that version of Postgres as it does all parts of a given version of RHEL during its support lifetime. But of course it would be nice to get those systems upgraded to a newer version of Postgres to get the performance and feature benefits of newer versions.

For any developers or DBAs familiar with Postgres, upgrading to a new version with RPMs from the PGDG or other custom Yum repository is not a big deal, but occasionally we’ve had a client worry that using a packages other than the ones supplied by Red Hat is riskier.

For those holdouts still on PostgreSQL 8.1 because it’s the “norm” on RHEL 5, Red Hat gave us a gift in their RHEL 5.5 …

Postgres configuration best practices

By Greg Sabino Mullane
September 27, 2010

This is the first in an occasional series of articles about configuring PostgreSQL. The main way to do this, of course, is the postgresql.conf file, which is read by the Postgres daemon on startup and contains a large number of parameters that affect the database’s performance and behavior. Later posts will address specific settings inside this file, but before we do that, there are some global best practices to address.

Version Control

The single most important thing you can do is to put your postgresql.conf file into version control. I care not which one you use, but go do it right now. If you don’t already have a version control system on your database box, git is a good choice to use. Barring that, RCS. Doing so is extremely easy. Just change to the directory postgresql.conf is in. The process for git:

• Install git if not there already (e.g. “sudo yum install git”)
• Run: git init
• Run: git add postgresql.conf pg_hba.conf
• Run: git commit -a -m “Initial commit”

For RCS:

• Install as needed (e.g. “sudo apt-get install rcs”)
• Run: mkdir RCS
• Run: ci -l postgresql.conf pg_hba.conf

Note that we also checked in pg_hba.conf as well. You want to check in any file in that directory you may …