Our Blog

PostgreSQL at LinuxFest Northwest

By Selena Deckelmann
April 23, 2010

This is my third year driving up to Bellingham for LinuxFest Northwest, and I’m excited to be presenting two talks about PostgreSQL there. Adrian Klaver is one of the organizers of the conference, and has always been a huge supporter of PostgreSQL. He has gone out of his way to have a track of content about our favorite database.

I’ll be presenting an introduction to Bucardo and co-hosting a talk about new features in version 9.0 of PostgreSQL with Gabrielle Roth.

Talking about Bucardo and replication is always a blast. The last time I gave this talk to a packed house in Seattle, so I’m hoping for another lively discussion about the state of replication in PostgreSQL.

Using charge tag in Interchange’s profiles, and trickiness with logic and tag interpolation order

By Ron Phipps
April 22, 2010

One of the standard ways to charging in older versions of the Interchange demo was to do the charging from a profile using the &charge command. New versions of the demo store do the charging from log_transaction once the order profiles have finished, so it is not an issue there. I’ve come across quite a few catalogs where the &charge command is replaced with the [charge] tag wrapped in if-then-else blocks in an order profile. It had been so long since I had used &charge so I needed to lookup how options are passed to it, which may be why people tend to use the tag version instead of the &charge command. The problem here is that Interchange tags interpolate before any of the profile specifications execute, so if you have a [charge] tag in an order profile, it executes before any of the other checks, such as validation of fields.

Here’s a stripped down example of where a profile will have tags executed before the other profile checks:

lname=required Last name required
fname=required First name required
&fatal=yes
&credit_card=standard keep

[charge route="[var MV_PAYMENT_MODE]" amount="[scratch some_total_calculation]"]

&final=yes

In this …

Restoring individual table data from a Postgres dump

By Greg Sabino Mullane
April 20, 2010

Recently, one of our clients needed to restore the data in a specific table from the previous night’s PostgreSQL dump file. Basically, there was a UPDATE query that did not do what it was supposed to, and some of the columns in the table were irreversibly changed. So, the challenge was to quickly restore the contents of that table.

The SQL dump file was generated by the pg_dumpall command, and thus there was no easy way to extract individual tables. If you are using the pg_dump command, you can specify a “custom” dump format by adding the -Fc option. Then, pulling out the data from a single table becomes as simple as adding a few flags to the pg_restore command like so:

$ pg_restore --data-only --table=alpha large.custom.dumpfile.pg > alpha.data.pg

One of the drawbacks of using the custom format is that it is only available on a per-database basis; you cannot use it with pg_dumpall. That was the case here, so we needed to extract the data of that one table from within the large dump file. If you know me well, you might suspect at this point that I’ve written yet another handy perl script to tackle the problem. As tempting as that may have been, time was of the essence, and the …

Authlogic and RESTful Authentication Encryption

By Steph Skardal
April 19, 2010

I recently did a bit of digging around for the migration of user data from RESTful authentication to Authlogic in Rails. My task was to implement changes required to move the application and data from RESTful Authentication to Authlogic user authentication.

I was given a subset of the database dump for new and old users in addition to sample user login data for testing. I didn’t necessarily want to use the application to test login functionality, so I examined the repositories here and here and came up with the two blocks of code shown below to replicate and verify encryption methods and data for both plugins.

RESTful Authentication

user = User.find_by_email('test@endpoint.com')

key = REST_AUTH_SITE_KEY
actual_password = "password"
digest = key

REST_AUTH_DIGEST_STRETCHES.times { digest = Digest::SHA1.hexdigest([digest, user.salt, actual_password, key].join('--')) }

# compare digest and user.crypted_password here to verify password, REST_AUTH_SITE_KEY, and REST_AUTH_DIGEST_STRETCHES

Note that the stretches value for RESTful authentication defaults to 10, but it can be adjusted. If no REST_AUTH_SITE_KEY is provided, the value defaults to an empty string. …

A decade of change in our work

By Jon Jensen
April 16, 2010

Lately I’ve been looking back a bit at how things have changed in our work during the past decade. Maybe I’m a little behind the times since this is a little like a new year’s reflection.

Since 2000, in our world of open source/free software ecommerce and other Internet application development, many things have stayed the same or become more standard.

The Internet is a completely normal part of people’s lives now, in the way TV and phones long have been.

Open source and free software is a widely-used and accepted part of the software ecosystem, used on both servers and desktops, at home and in companies of all sizes. Software licensing differences are still somewhat arcane, but the more popular options are now fairly widely-known.

Many of today’s major open source software systems key to Internet infrastructure and application development were already familiar in 2000:

• the GNU toolset, Linux (including Red Hat and Debian), FreeBSD, OpenBSD

• Apache, mod_ssl (the RSA patent expired in September 2000, making it freely usable)

• PostgreSQL, MySQL, *DBM

• Perl and CPAN, Python, PHP, Ruby (though little known then)

• Interchange (just changed from MiniVend)

• JavaScript

• Sendmail, …

Tip: Find all non-UTF-8 files

By David Christensen
April 9, 2010

Here’s an easy way to find all non-UTF-8 files for later perusal:

find . -type f | xargs -I {} bash -c "iconv -f utf-8 -t utf-16 {} &>/dev/null || echo {}" > utf8_fail

I’ve needed this before for converting projects over into UTF-8; obviously certain files are going to be binary and will show up in this list, so manual vetting will need to be done before converting all your images over into UTF-8.

Modifying Models in Rails Migrations

By Sonny Cook
April 8, 2010

One problem that has haunted me in the past was making modifications to the model in migrations. Specifically, stuff like removing or changing associations. At the time the migration is expected to run, the file for the model class will have been updated already, so it is hard use that in the migration itself, even though it would be useful.

In this case I found myself with an even slightly trickier example. I have a model that contains some address info. Part of that is an association to an external table that lists the states. So part of the class definition was like so:

Class Contact {
 belongs_to :state
 ...
}

What I needed to do in the migration was to remove the association and introduce another field called “state” which would just be a varchar field representing the state part of the address. The two problems the migration would encounter are:

1. the state association would not exist at the time it ran
2. and even if it did, there would be a name conflict between it and the new column I wanted

to get around these restrictions I did this in my migration:

Contact.class_eval {
      belongs_to :orig_state,
                 :class_name => "State", …

Git Submodules: What is the Ideal Workflow?

By Steph Skardal
April 5, 2010

Last week, I asked some coworkers at End Point about the normal workflow for using git submodules. Brian responded and the discussion turned into an overview on git submodules. I reorganized the content to be presented in a FAQ format:

How do you get started with git submodules?

You should use git submodule add to add a new submodule. So for example you would issue the commands:

git submodule add git://github.com/stephskardal/extension1.git extension
git submodule init

Then you would git add extension (the path of the submodule installation), git commit.

What does the initial setup of a submodule look like?

The super project repo stores a .gitmodules file. A sample:

[submodule "extension1"]
        path = extension
        url = git://github.com/stephskardal/extension1.git
[submodule "extension2"]
        path = extension_two
        url = git://github.com/stephskardal/extension2.git

When you have submodules in a project, do you have to separately clone them from the master project, or does the initial checkout take care of that recursively for you?

Generally, you will issue the commands below when you clone a super project repository. These commands will “install” …