Our Blog

Perl, UTF-8, and binmode on filehandles

By Greg Sabino Mullane
February 21, 2012

Original image by avlxyz

I recently ran into a Perl quirk involving UTF-8, standard filehandles, and the built-in Perl die() and warn() functions. Someone reported a bug in the check_postgres program in which the French output was displaying incorrectly. That is, when the locale was set to FR_fr, the French accented characters generated by the program were coming out as “byte soup” instead of proper UTF-8. Some other languages, English and Japanese among them, seemed to be fine. For example:

## English: "sorry, too many clients already"
## Japanese: "現在クライアント数が多すぎます"
## French expected: "désolé, trop de clients sont déjà connectés"
## French actual: "d�sol�, trop de clients sont d�j� connect�s"

That last line should be very familiar to anyone who has struggled with Unicode on a command line, with those question marks on an inverted background. Our problem was that the output of the script looked like the last line, rather than the one before it. The Japanese output, despite being chock full of Unicode, does have the same problem! More on that later.

I was able to duplicate the problem easy enough by setting my locale to FR_fr and having …

World IPv6 Launch: 6 June 2012

By Jon Jensen
February 17, 2012

For any of our readers who don’t know: The classic Internet Protocol (IPv4) supported around 4 billion IP addresses, but it recently ran out of free addresses. That the addresses would eventually all be used was no surprise. For more than a decade, a replacement called IPv6 has been under development and allows practically unlimited addresses.

Last year there was a one-day trial run called World IPv6 Day. Our own Josh Williams wrote about it here. It was the first major attempt for mainstream websites to enable dual-stack IPv4 and IPv6 networking, so that both the “old” and “new” Internet could access the same site. It was intended to bring to the surface any problems, and it went very well – most people never knew it was happening, which was the goal.

This year there’s a much bigger event planned: World IPv6 Launch, and this time IPv6 is meant to stay on. Google, Facebook, Yahoo!, Bing, and many other major sites are participating. A big advance over last year’s event is that many ISPs and vendors of home networking gear are also participating. That means it won’t just be a test that classic IPv4 still works for people alongside IPv6, but that for some customers, native IPv6 …

Tracking down PostgreSQL XYZ error: tablespace, database, and relfilnode

By Greg Sabino Mullane
February 14, 2012

One of our Postgres clients recently had this error show up in their logs:

ERROR: could not read block 3 of relation 1663/18421/31582:
read only 0 of 8192 bytes

Because we were using the tail_n_mail program, the above error was actually mailed to us within a minute of it occurring. The message is fairly cryptic, but it basically means that Postgres could not read data from a physical file that represented a table or index. This is generally caused by corruption or a missing file. In this case, the “read only 0 of 8192” indicates this was most likely a missing file.

When presented with an error like this, it’s nice to be able to figure out which relation the message is referring to. The word “relation” is Postgres database-speak for a generic object in the database: in this case, it is almost certainly going to be a table or an index. Both of those are, of course, represented by actual files on disk, usually inside of your data_directory. The number given, 1663/18421/31582, is in the standard X/Y/Z format Postgres uses to identify a file, where X represents the tablespace, Y is the database, and Z is the file.

The first number, X, indicates which tablespace this relation …

Django and Virtual Environments

By Szymon Lipiński
February 13, 2012

When you have to work with a bunch of different Python applications, the usual problem is that you have to deal with plenty of different packages in different versions. Each application needs its own set of libraries. Usually the versions of the libraries vary between the applications.

To solve all the problems you could create Python virtual environments. There is a great tool: virtualenv. It can create virtual environments for Python. Using it is not too nice. However there is a wrapper to it, called virtualenvwrapper. It wraps all the virtualenv commands into a couple of shell commands.

Let’s assume that I need to work on two applications written in Django 1.2 and Django 1.3. Each of the applications needs different set of packages in different versions. I will create two virtual environments.

Installing virtualenvwrapper on Ubuntu is pretty easy:

$ sudo apt-get install virtualenvwrapper

After the installation there is a couple of new commands. The basic one is: mkvirtualenv which creates a new environment. Let’s create one.

$ mkvirtualenv django_demo_12

This command automatically switches to the new environment, so you might notice that the prompt changed. The prompt …

DevCamps setup with Ruby 1.9.3, rbenv, Nginx and Unicorn

By Richard Templet
February 10, 2012

I was working with Steph Skardal on the setup of a new DevCamps installation that was going to need to use Ruby 1.9.3, Rails 3, Unicorn and Nginx. This setup was going to be much different than a standard setup due to the different application stack that was required.

The first trick for this was going to get Ruby 1.9.3 on the server. We were using Debian Squeeze but that still only comes with Ruby 1.9.1. We wanted Ruby 1.9.3 for the increased overall speed and significant speed increase with Rails 3. We decided on using rbenv for this task. It’s a very easy to setup utility that allows you to maintain multiple version of Ruby in your system user account without the headache of adjusting anything but the PATH environment variable. It takes advantage of another easy to setup utility called ruby build to handle the actual installation of the Ruby source code.

A quick and easy version for setting up a user with this is as follows:

Ensure you are in the home directory. Then, clone the repository into a .rbenv directory

git clone git://github.com/sstephenson/rbenv.git .rbenv

Adjust your users path to find the newly installed commands

echo 'export …

Lock up your keys

By Jeff Boes
February 9, 2012

Locking hash keys with Hash::Util

It’s a given that you shouldn’t write Perl without “use strict”; it prevents all kinds of silent bugs involving misspelled and uninitialized variables. A similar aid for misspelled and uninitialized hash keys exists in the module “Hash::Util”.

By way of background: I was working on a long chunk of code that prepares an e-commerce order for storage in a database. Many of the incoming fields map directly to the table, but others do not. The interface between this code and the page which submits a large JSON structure was in flux for a while, so from time to time I had to chase bugs involving “missing” or “extra” fields. I settled on a restricted hash to help me squash these and future bugs.

The idea of a restricted hash is to clamp down on Perl’s rather loose “record” structure (by which I mean the common practice of using a hash to represent a record with named fields), which is great in some circumstances. While in most programming languages you must pre-declare a structure and live with it, in Perl hashes you can add new keys on the fly, misspellings and all. A restricted hash can only have a particular set of keys, but is still a hash for all …

Download Functionality for Rails Ecommerce

By Steph Skardal
February 8, 2012

I recently had to build out downloadable product support for a client project running on Piggybak (a Ruby on Rails Ecommerce engine) with extensive use of RailsAdmin. Piggybak’s core functionality does not support downloadable products, but it was not difficult to extend. Here are some steps I went through to add this functionality. While the code examples apply specifically to a Ruby on Rails application using paperclip for managing attachments, the general steps here would apply across languages and frameworks.

Data Migration

Piggybak is a pluggable ecommerce engine. To make any models inside your application “sellable”, the class method acts_as_variant must be called for any class. This provides a nice flexibility in defining various sellable models throughout the application. Given that I will sell tracks in this example, my first step to supporting downloadable content is adding an is_downloadable boolean and attached file fields to the migration for a sellable item. The migration looks like this:

class CreateTracks < ActiveRecord::Migration
  def change
    create_table :tracks do |t|
      # a bunch of fields specific to tracks

      t.boolean :is_downloadable, :nil …

RailsAdmin Import: Part 2

By Steph Skardal
February 1, 2012

I recently wrote about importing data in RailsAdmin. RailsAdmin is a Rails engine that provides a nice admin interface for managing your data, which comes packed with configuration options.

In a recent Ruby on Rails ecommerce project, I’ve been using RailsAdmin, Piggybak (a Rails ecommerce gem supported by End Point), and have been building out custom front-end features such as advanced search and downloadable product support. When this client came to End Point with the project, we offered several options for handling data migration from a legacy system to the new Rails application:

1. Create a standard migration file, which migrates data from the existing legacy database to the new data architecture. The advantage with this method is that it requires virtually no manual interaction for the migration process. The disadvantage with this is that it’s basically a one-off solution and would never be useful again.
2. Have the client manually enter data. This was a reasonable solution for several of the models that required 10 or less entries, but not feasible for the tables containing thousands of entries.
3. Develop import functionality to plug into RailsAdmin which imports from CSV files. The …