Our Blog

MountainWest RubyConf 2010 — Steph’s Notes

By Steph Skardal
March 16, 2010

Last Thursday and Friday, I attended MountainWest RubyConf in Salt Lake City. As usual with any conference, my notebook is full of various tools and tips jotted down for later investigation. I thought I’d summarize some of the topics that grabbed my interest and go more in depth on a couple of topics in the next couple of weeks.

• Lambda: In a talk early in the conference, lambda in Ruby came up. I had a hard time coming up with use cases for its use in Rails or a web app in another server side language with equivalent functionality (Python, Perl), but I’d like to look into it. An example was presented using Ruby’s lambda to calculate Google’s PageRank value, which is particularly appealing to my interest in SEO.

• Chef: I’ve heard of Chef since I started working with Rails, but have yet to use it. After my recent adventures with Spree on Heroku, I see the value in becoming more familiar with Chef or another configuration management software tool such as Puppet. I’m particularly interested in creating some Chef recipes for Spree.

• RVM: RVM, or Ruby Version Manager, is a nice tool to work with multiple Ruby environments from interpreters to sets of gems. For a couple of …

NoSQL Live: The Dynamo Derivatives (Cassandra, Voldemort, Riak)

By Ethan Rowe
March 12, 2010

For me, one of the big parts of attending the NoSQL Live conference was to hear more about the differences between the various Dynamo-inspired open software projects. The Cassandra, Voldemort, and Riak projects were all represented, and while they differ in various ways at the logical layer (how one models data) and various features, they all share a similar conceptual foundation as outlined in Amazon’s seminal Dynamo paper. So what differentiates these projects? What production deployments exist and what kind of stories do people have from running these systems in production?

Some of these questions can be at least partially answered by combing the interwebs, looking over the respective project sites, etc. Yet that’s not quite the same thing as having community players in the same room, talking about stuff.

Of the three projects mentioned, Cassandra clearly has the “momentum” (a highly accurate indicator of future dominance). To me, this felt like the case even before Twitter started getting involved with it, but the Twitter effect was pretty evident based on the number of people sticking around for the Cassandra break-out session with Jonathan Ellis, compared to the break-out …

Quick Thoughts on NoSQL Live Boston Conference

By Ethan Rowe
March 11, 2010

I’m back home now from the Boston “NoSQL Live” conference organized by 10gen.com (the MongoDB folks). It was a good event. A lot of stuff covered, a broad range of topics. I have a fair amount to say, but need to digest, review notes, etc. In any case, thanks to 10gen and the various sponsors that made it happen.

Some quick, random thoughts:

• Picking a good table at lunch is key: we ended up sitting with four different presenters, including Jonathan Ellis for Cassandra and Alex Feinberg for Voldemort, which happen to be two of the systems I’m personally most interested in using at the moment.
• There is an undeniable latest-thing-fan(boy|girl)ism aura surrounding the “NoSQL” brand/meme/whatever, but the various presenters and leading lights in various projects appear to be reasonable and fact-based; don’t let the breathless silliness of fans fool you.
• I went in feeling convinced of the desirability of non-relational datastores for specific modeling situations (graphs) and for scalability/availability/volume concerns (Dynamo and BigTable derivatives), while feeling relatively skeptical of “document datastores”. I left feeling basically the same way, though decidedly less skeptical of …

PostgreSQL UTF-8 Conversion

By Jon Jensen
March 9, 2010

It’s becoming increasingly common for me to be involved in conversion of an old version of PostgreSQL to a new one, and at the same time, from an old “SQL_ASCII” encoding (that is, undeclared, unvalidated byte soup) to UTF-8.

Common ways to do this are to run pg_dumpall and then pipe the output through iconv or recode. When your source encoding is all pure ASCII, you don’t need to do even that. When it’s really all Windows-1252 (a superset of Latin-1 aka ISO-8859-1) it’s easy.

But often, the data is stored in various unknown encodings from several sources over the course of years, including some that’s already in UTF-8. When you convert with iconv, it dies with an error at the first problem, whereas recode will let you ignore encoding problems, but that leaves you with junk in your output.

The case I’m often encountering is fairly easy, but not perfect: Lots of ASCII, some Windows-1252, and some UTF-8. Since both pure ASCII and UTF-8 can be mechanistically detected, I put together this script to do the detection. It’s Perl and uses the nice IsUTF8 module to do its character encoding detection:

Pipe input to the script. It handles one line at a time. When run with any arguments …

PostgreSQL tip: arbitrary serialized rows

By David Christensen
March 8, 2010

Sometimes when using PostgreSQL, you want to deal with a record in its serialized form. If you’re dealing with a specific table, you can accomplish this using the table name itself:

psql # CREATE TABLE foo (bar text, baz int);
CREATE TABLE

psql # INSERT INTO foo VALUES ('test 1', 1), ('test 2', 2);
INSERT 0 2

psql # SELECT foo FROM foo;
     foo      
--------------
 ("test 1",1)
 ("test 2",2)
(2 rows)

This works fine for defined tables, but how to go about this for arbitrary SELECTs? The answer is simple: wrap in a subselect and alias as so:

psql # SELECT q FROM (SELECT 1, 2) q;
   q   
-------
 (1,2)
(1 row)

Spree on Heroku for Development

By Steph Skardal
March 8, 2010

Yesterday, I worked through some issues to setup and run Spree on Heroku. One of End Point’s clients is using Spree for a multi-store solution. We are using the the recently released Spree 0.10.0.beta gem, which includes some significant Spree template and hook changes discussed here and here in addition to other substantial updates and fixes. Our client will be using Heroku for their production server, but our first goal was to work through deployment issues to use Heroku for development.

Since Heroku includes a free offering to be used for development, it’s a great option for a quick and dirty setup to run Spree non-locally. I experienced several problems and summarized them below.

Application Changes

1. After a failed attempt to setup the basic Heroku installation described here because of a RubyGems 1.3.6 requirement, I discovered the need for Heroku’s bamboo deployment stack, which requires you to declare the gems required for your application. I also found a Spree Heroku extension and reviewed the code, but I wanted to take a more simple approach initially since the extension offers several features that I didn’t need. After some testing, I created a .gems file in the main …

Riak Install on Debian Lenny

By Ethan Rowe
March 4, 2010

I’m doing some comparative analysis of various distributed non-relational databases and consequently wrestled with the installation of Riak on a server running Debian Lenny.

I relied upon the standard “erlang” debian package, which installs cleanly on a basically bare system without a hitch (as one would expect). However, the latest Riak’s “make” tasks fail to run; this is because the rebar script on which the make tasks rely chokes on various bad characters:

riak@nosql-01:~/riak$ make all rel
./rebar compile
./rebar:2: syntax error before: PK
./rebar:11: illegal atom
./rebar:30: illegal atom
./rebar:72: illegal atom
./rebar:76: syntax error before: ��n16
./rebar:79: syntax error before: ','
./rebar:91: illegal integer
./rebar:149: illegal atom
./rebar:160: syntax error before: Za��ze
./rebar:172: illegal atom
./rebar:176: illegal atom
escript: There were compilation errors.
make: *** [compile] Error 127

Delicious.

Ultimately, I came across this article describing issues getting Riak to install on Ubuntu 9.04, and ultimately determined that the Erlang version mentioned seemed to apply here. Following the article’s instructions for building Erlang from source worked out …

PostgreSQL EC2/EBS/RAID 0 snapshot backup

By Jon Jensen
February 23, 2010

One of our clients uses Amazon Web Services to host their production application and database servers on EC2 with EBS (Elastic Block Store) storage volumes. Their main database is PostgreSQL.

A big benefit of Amazon’s cloud services is that you can easily add and remove virtual server instances, storage space, etc. and pay as you go. One known problem with Amazon’s EBS storage is that it is much more I/O limited than, say, a nice SAN.

To partially mitigate the I/O limitations, they’re using 4 EBS volumes to back a Linux software RAID 0 block device. On top of that is the xfs filesystem. This gives roughly 4x the I/O throughput and has been effective so far.

They ship WAL files to a secondary server that serves as warm standby in case the primary server fails. That’s working fine.

They also do nightly backups using pg_dumpall on the master so that there’s a separate portable (SQL) backup not dependent on the server architecture. The problem that led to this article is that extra I/O caused by pg_dumpall pushes the system beyond its I/O limits. It adds both reads (from the PostgreSQL database) and writes (to the SQL output file).

There are several solutions we are considering so that …