Our Blog

Replicate only parts of my table

By Josh Tolley
February 19, 2009

A day or two ago in #slony, someone asked if Slony would replicate only selected columns of a table. A natural response might be to create a view containing only the columns you’re interested in, and have Slony replicate that. But Slony is trigger-based—​the only reason it knows there’s something to replicate is because a trigger has told it so—​and you can’t have a trigger on a view. So that won’t work. Greg chimed in to say that Bucardo could do it, and mentioned a Bucardo feature I’d not yet noticed.

Bucardo is trigger-based, like Slony, so defining a view won’t work. But it allows you to specify a special query string for each table you’re replicating. This query is called a “customselect”, and can serve to limit the columns you replicate, transform the rows as they’re being replicated, etc., and probably a bunch of other stuff I haven’t thought of yet. A simple example:

1. Create a table in one database as follows:

CREATE TABLE synctest (
   id INTEGER PRIMARY KEY,
   field1 TEXT,
   field2 TEXT,
   field3 TEXT
);

Also create this table in the replication destination database; Bucardo won’t replicate schema changes or database structure.

1. Tell Bucardo about the table. I won’t …

Announcing Release of PostgreSQL System Impact (PGSI) Log Analyzer

By Mark Johnson
February 14, 2009

The PostgreSQL System Impact (PGSI) log analyzer is now available at https://bucardo.org/Pgsi/.

System Impact (SI) is a measure of the overall load a given query imposes on a server. It is expressed as a percentage of a query’s average duration over the its average interval between successive calls.

Queries are collected into canonical form with respect to literals and bind params; further, IN lists of varying cardinality are collapsed. Thus, queries that differ only in argument composition will be collected together in the evaluation. However, logically equivalent queries that differ in any other manner of structure (say two comparisons between and that are transposed) will be seen as distinct.

The goal of SI is to identify those queries most likely to cause performance degradation on the database during heaviest traffic periods. Focusing exclusively on the least efficient queries can hide relatively fast-running queries that saturate the system more because they are called far more frequently. By contrast, focusing only on the most-frequently called queries will tend to emphasize small, highly optimized queries at the expense of slightly less popular queries that spend much more …

Testing in the Web Environment

By Jeff Boes
February 13, 2009

Introduction

Testing is an important part of good software engineering practices. In fact, it can be said that it is at once the most important, and yet most neglected part of software engineering. Testing methodology for software engineering developed out of its hardware engineering roots: software was defined in terms of its inputs and outputs, and testing was similarly defined in terms of applied inputs and expected outputs.

However, software testing is more complex than that: this is because software almost always incorporates “state” or memory that affects subsequent operations. For instance, the following pseudocode:

if (VALUE is not defined)
then
VALUE := 1.0
fi
FRACTION := 1.0 / VALUE

In this simple case, the code fragment will always operate correctly on the first execution, but subsequent executions may fail if VALUE is zero.

Testing web applications involves planning for this kind of memory, because in essence a web application runs within a larger program (the web server and perhaps the application server) and may inherit state from the environment, or indeed may preserve its own state from one page reference to the next.

In addition, web applications involve human …

Hot topic: user groups!

By Selena Deckelmann
February 10, 2009

Theodore T’so, Linux filesystem hacker, recently asked:

Do you attend or participate in a LUG? How active is it? What do you get out of participating in a LUG?

I attend and participate in my local LUG, PLUG. I’ve attended meetings since about 2000, and presented once to a group of about 45 people. They meet every month, and have an additional “advanced topics” meeting every month I’ve presented at as well.

What I get out of PLUG has changed over the years. First I was attending to meet other Linux enthusiasts and learn things. Now I go to meetings to see my friends, swap war stories, and to present on topics of interest to me. My goals are less about general networking, and more about directly connecting to, sharing information with and learning from people I already know, who are for the most part my friends and colleagues.

The comment thread on Ted’s post is kind of long, and there were lots of interesting comments. I decided to respond to some requests for howtos and weigh in with my opinion on the rise of user groups:

My experience has been that user groups (not just Linux User Groups) are exploding in number, but that the total number of people in each group is …

Test::Database Postgres support

By Greg Sabino Mullane
February 8, 2009

At our recent company meeting, we organized a ‘hackathon’ at which the company was split into small groups to work on specific projects. My group was Postgres-focused and we chose to add Postgres support to the new Perl module Test::Database.

This turned out to be a decent sized task for the few hours we had to accomplish it. The team consisted of myself (Greg Sabino Mullane), Mark Johnson, Selena Deckelmann, and Josh Tolley. While I undertook the task of downloading the latest version and putting it into a local git repository, others were assigned to get an overview of how it worked, examine the API, and start writing some unit tests.

In a nutshell, the Test::Database module allows an easy interface to creating and destroying test databases. This can be a non-trivial task on some systems, so putting it all into a module make sense (as well as the benefits of preventing everyone from reinventing this particular wheel). Once we had a basic understanding of how it worked, we were off.

While all of our tasks overlapped to some degree, we managed to get the job done without too much trouble, and in a fairly efficient manner. We made a new file for Postgres, added in all the required …

David Mamet on software development

By Jon Jensen
February 7, 2009

I recently read the book True and False: Heresy and Common Sense for the Actor (1997) by David Mamet. (A gift from Uncle Steve. Thanks!) Mamet’s film The Spanish Prisoner is one of my favorites, and I liked Oleanna, Things Change, and Glengarry Glen Ross a lot too. Wait, looking over his writing credits at IMDb, I see I’d forgotten The Winslow Boy, State and Main, and Wag the Dog, which were good too.

Enough of the fanboyism. This little book of his is packed with what sounds like good advice for actors. Not having been an actor in anything more than a school play or two, I can only imagine, but I was surprised to find myself nodding in agreement during many passages because they applied so well to my field, software development.

I’ve selected a few quotations that seemed especially apt:

I was once at a marriage ceremony where the parties swore “to try to be faithful, to try to be considerate …” That marriage was, of course, doomed. Any worthwhile goal is difficult to accomplish. To say of it “I’ll try” is to excuse oneself in advance. Those who respond to our requests with “I’ll try” intend to deny us, and call on us to join in the hypocrisy—as if there were some merit in …

NY Puppet Users Group meeting

By Steven Jenkins
February 4, 2009

Several of us from End Point attended the first NY Puppet Users Group meeting last night, and we had a great time. Brian Gupta and Larry Ludwig did an excellent time coordinating, even though the weather and the venue didn’t cooperate. After a little shuffling, we found ourselves at The Perfect Pint, where we had a large table in a fairly quiet room where we could meet each other and talk about Puppet.

The group was split fairly evenly between those who have used Puppet and those who are considering it: everyone there understood the need for solid configuration management (CM), so much of the discussions (at least at my end of the table) centered around

• comparisons of different CM systems

• challenges in getting Puppet on legacy systems (e.g., that may not even have Ruby)

• features people would like to see in Puppet

• how to make the business case for Puppet

All in all it was a great time, and I really appreciated Brian and Larry’s leadership, as well as their flexibility. Not only did they move the venue at the last minute to better accommodate the group, but they were super hosts in coordinating the date and time: several of us were only in NY for a short time, but they …

Slony1-2.0.0 + PostgreSQL 8.4devel

By Josh Tolley
February 2, 2009

Many people use Slony to replicate PostgreSQL databases in various interesting ways. Slony is a bit tough to get used to, but works very well, and can be found in important places at a number of high-load, high-profile sites. A few weeks back I set up Slony1-2.0.0 (the latest release) with a development version of PostgreSQL 8.4, and kept track of the play-by-play, as follows:

Starting Environment

On this machine, PostgreSQL is installed from the CVS tree. I updated the tree and reinstalled just to have a well-known starting platform (output of each command has been removed for brevity).

jtolley@uber:~/devel/pgsql$ make distclean
jtolley@uber:~/devel/pgsql$ cvs update -Ad
jtolley@uber:~/devel/pgsql$ ./configure --prefix=/home/jtolley/devel/pgdb
jtolley@uber:~/devel/pgsql$ make
jtolley@uber:~/devel/pgsql$ make install
jtolley@uber:~/devel/pgsql$ cd ../pgdb
jtolley@uber:~/devel/pgdb$ initdb data
jtolley@uber:~/devel/pgdb$ pg_ctl -l ~/logfile -D data start

The –prefix option in ./configure tells PostgreSQL where to install itself.

Slony uses a daemon called slon to do its work, and slon connects to a database over TCP, so I needed to configure PostgreSQL to allow TCP …