Testing user-defined functions with pgTAP

2023-09-25T00:00:00+00:00

For a great introduction to the pgTAP protocol, please read over my colleague Josh Tolley’s article, using pgTAP to automate database testing. Also check out Edgar Mlowe’s article on how to set up pgTAP for writing PostgreSQL database unit tests.

As a protocol, pgTAP provides a great list of functions and commands to easily and clearly test all aspects of a database. As I’ve been tasked with more implementations of unit tests for user-defined functions, I thought it helpful to share some useful methods.

First we’ll go over a few basic pgTAP functions that are useful in testing the existence of functions and procedures. Then we’ll use the PostgreSQL port of the Sakila sample database for MySQL, called Pagila. We’ll construct two functions and go over a single test case. Using basic Postgres tools, I’ll walk through a few methods of calling and testing our functions. Finally, we’ll go over a more concise way of testing all cases for said functions.

Feel free to work alongside this article to hopefully leave with a comfortable idea of how to construct unit test files for your database.

Functions:

For this first example, remember to have the pgTAP extension in your schema and to perform it in a transaction (I’ve omitted BEGIN and ROLLBACK for brevity). Also use SELECT plan(<number of tests you’ll run>) when running the pgTAP functions.

Starting off, let’s create a basic addition function that returns an integer value:

CREATE OR REPLACE FUNCTION add_numbers(
    num1 integer,
    num2 integer
)
RETURNS integer
LANGUAGE plpgsql AS $$
BEGIN
    RETURN num1 + num2;
END;
$$;

To test this user-defined function, a few of the most relevant functions in pgTAP we can use are:

`has_function()`

Variations:

SELECT has_function( :schema, :function, :args, :description );
SELECT has_function( :schema, :function, :args );
SELECT has_function( :schema, :function, :description );
SELECT has_function( :schema, :function );
SELECT has_function( :function, :args, :description );
SELECT has_function( :function, :args );
SELECT has_function( :function, :description );
SELECT has_function( :function );

Parameters:

:schema
    Name of a schema in which to find the function.
:function
    Name of a function.
:args
    Array of data types of the function arguments.
:description
    A short description of the test.

Example:

SELECT has_function('your_schema', 'add_numbers', ARRAY['integer','integer']); -- default description used

Results:

ok 1 - Function add_numbers(integer, integer) should exist

has_function() is useful in checking the existence of a specific function. If you want to parse an entire schema, functions_are() is your command.

`functions_are()`

Variations:

SELECT functions_are( :schema, :functions, :description );
SELECT functions_are( :schema, :functions );
SELECT functions_are( :functions, :description );
SELECT functions_are( :functions );

Parameters:

:schema
    Name of a schema in which to find functions.
:functions
    An array of function names.
:description
    A short description of the test.

Example:

SELECT functions_are('your_schema', ARRAY['add_numbers', <All other functions need to be listed as well>]); -- default description used

`ok()` and `is()`

ok() and is() have similar functionality with slightly different parameters:

SELECT ok( :boolean, :description );

Example:

SELECT ok(add_numbers(5,5)=10, 'add_numbers correctly works');

Parameters:

:boolean
    A boolean value indicating success or failure.
:description
    A short description of the test.

While ok() is used for passing a test returning a boolean, is() takes two values to compare:

SELECT is( :have, :want, :description );

Example:

SELECT is(add_numbers(5,5), 10, 'add_numbers correctly works');

You can find functions that cover pretty much anything in PostgreSQL here in pgTAP’s documentation.

Testing more complex user-defined stored functions:

A client product we created utilizes a function that calls several other functions depending on its given argument. This function is called by the application and is passed an integer as an argument from which the function bases an initial SELECT query’s WHERE condition on, in this case WHERE id = [argument]. Variables are then assigned column values from this queried table. These variables are then used as the conditions to PERFORM functions that change parts of the database.

When testing this sort of scenario, it became apparent that we needed a set of test data, as well as the ability to track that test data, run the function with the data, and test to see if the changes are performed correctly.

The nice thing about pgTAP is that unit tests are meant to be rolled back leaving the database as you found it. When faced with a more complex function that nests other functions which loop through multiple tables, it may make more sense to fill those tables temporarily with the data required for the tests.

Let’s take a nostalgia trip and imagine we’re operating a DVD rental store. The Pagila sample database, which you can download on GitHub, represents our scenario. It comes populated and we’ll construct a function which utilizes other functions to perform changes on multiple tables.

Say the DVD rental shop gives extensions of rentals for special occasions, one being that a DVD was rented on Christmas week. Suppose we have an initial function that checks if a new DVD is rented on one of these special occasions. It then calls another function that will extend the return date, based on the specific occasion:

CREATE OR REPLACE FUNCTION extension_check(_rental_id integer)
    RETURNS void
    LANGUAGE plpgsql
AS $function$
DECLARE
    _rental_date date;
    _inventory_id integer;
    _customer_id integer;
    _return_date date;
    christmas_week date;
BEGIN

    SELECT INTO _rental_date
    rental_date FROM rental
    WHERE rental_id = _rental_id;

    IF EXTRACT(YEAR FROM _rental_date) = EXTRACT(YEAR FROM CURRENT_DATE) AND
       _rental_date <= (EXTRACT(YEAR FROM CURRENT_DATE)::text || '-12-25')::date AND
       _rental_date >  (EXTRACT(YEAR FROM CURRENT_DATE)::text || '-12-25')::date - INTERVAL '7 days' THEN
        PERFORM christmas_extension(_rental_id);
    END IF;
END;
$function$;

CREATE OR REPLACE FUNCTION christmas_extension(_rental_id integer)
    RETURNS void
    LANGUAGE plpgsql
AS $function$
DECLARE
    _rental_date date;
    _return_date date;
BEGIN
    SELECT return_date FROM rental WHERE rental_id = _rental_id INTO _return_date;

    IF _return_date IS NULL THEN
        UPDATE rental SET return_date = rental_date + INTERVAL '10 days' WHERE rental_id=_rental_id;
    ELSE
        UPDATE rental SET return_date = return_date + INTERVAL '10 days' WHERE rental_id=_rental_id;
    END IF;
END;
$function$;

The initial function, extension_check, simply checks if the rental was checked out on the week prior to and on Christmas (It’s a really dedicated DVD shop). We can also assume a function like this has other scenarios where it would extend rental dates but we’ll just show one for this exercise; christmas_extension is our one extension function that extends the return date by ten days.

To hold our unit test, we’ll create a file fittingly named extension_unit_tests.sql. A problem with the current sample database is that we can’t really test out the function directly since the dates don’t extend to the current year. Rather even if they did, we don’t want to directly affect the existing data especially if the dates were updated and the database was in production use.

Relatively basic but handy Postgres tools let us create test cases which we can pass to the functions. Afterwards, we can use pgTAP functions to confirm that our user-defined functions extension_check and christmas_extension worked properly.

For test cases in our new file, extension_unit_tests.sql, we’ll need a way to create and track them. For this, we’ll use a temporary table to track the test IDs and a PROCEDURE which can be called at any point to create the test cases. After that, we’ll simply call the procedure and run the user-defined function extension_check. Finally test its existence, test its result, and roll back our changes.

BEGIN;

CREATE TEMPORARY TABLE test_instance_references (
    test_rental_id integer PRIMARY KEY,
    test_number integer
);

CREATE OR REPLACE PROCEDURE fill_test_data (
    _rental_date date,
    _test_number integer
)
LANGUAGE PLPGSQL
AS $procedure$
DECLARE
    _address_id integer;
    _store_id integer;
    _staff_id integer;
    _inventory_id integer;
    _customer_id integer;
    _rental_id integer;
BEGIN

    SELECT nextval('customer_customer_id_seq1'::regclass) INTO _customer_id;
    SELECT nextval('rental_rental_id_seq'::regclass) INTO _rental_id;
    SELECT nextval('inventory_inventory_id_seq'::regclass) INTO _inventory_id;
    SELECT store_id FROM store LIMIT 1 INTO _store_id; -- any store will do
    SELECT address_id FROM address LIMIT 1 INTO _address_id; -- any address will do

    INSERT INTO customer (customer_id, store_id, first_name, last_name, address_id)
    VALUES (
        _customer_id,
        _store_id,
        'Test First Name',
        'Test Last Name',
        _address_id
    );

    INSERT INTO inventory (inventory_id, film_id, store_id)
    VALUES (
        _inventory_id,
        (SELECT film_id FROM film LIMIT 1), -- arbitrary film
        _store_id
    );

    INSERT INTO rental (rental_id, rental_date, inventory_id, customer_id, staff_id)
    VALUES (
        _rental_id,
        _rental_date,
        _inventory_id,
        _customer_id,
        (SELECT staff_id FROM staff WHERE store_id = _store_id LIMIT 1) -- select arbitrary staff member from store
    );

    INSERT INTO test_instance_references (test_rental_id, test_number)
    values (
        _rental_id,
        _test_number
    );

END;
$procedure$;

CALL fill_test_data('2023-12-25'::date, 1); -- Fill the db with our test data

SELECT extension_check((SELECT test_rental_id FROM test_instance_references WHERE test_number = 1)); -- Call our user-defined function

SELECT plan(2); -- State the number of tests we'll be performing

SELECT has_function(
    'extension_check',
    ARRAY['int']
);

SELECT is(
    (SELECT return_date FROM rental WHERE rental_id in (SELECT test_rental_id FROM test_instance_references WHERE test_number = 1))::date,
    ('2023-12-25'::date + INTERVAL '10 DAYS')::date,
    'Checking test data 1, week of Christmas'
);


ROLLBACK;

By design, pgTAP tests are performed in a transaction block with a ROLLBACK at the end to reverse all changes. Our first transaction is the creation of a temporary table called test_instance_references which will hold the IDs of the sample data we insert specifically in the rental table. This way we keep track of our specific test cases through their primary identifier.

Our PROCEDURE works similarly to a function, but where a function may have issues being called in a transaction block through pgTAP’s command line pg_prove, the use of a procedure and its CALL command is far more reliable.

In our example, the procedure simply adds in all the required test data. Note the insertion of test data into other tables that are referenced in rental, as well as an insert into the temporary table for tracking. To avoid using any existing data in the database, references are also test samples, like for the customer. Some, however, are selected from existing tables. It’s important to make sure you’re not affecting real data in the database in case a transaction fails. The example is obviously lax when it comes to that, specifically with the references to store and address, so please take note in your own unit tests to be careful when referring to existing data.

After we’ve established our procedure, we’re ready to run the tests. The pgTAP function plan() defines how many unit tests we are going to run. For the first example we run two tests.

Before running tests, we call the procedure fill_test_data which populates the database with test data and the temporary table to track that data.

Our first test is to check if the function exists. pgTAP’s has_function function is perfect for this.

Our procedure is then called with a date for the test rental. This test rental’s ID is saved into the temporary test_instance_references table which is then passed to the main function we want to test, extension_check. We call this function with the SELECT command, and immediately test the outcome with pgTAP’s is(). Our :have parameter holds the return_date of our test rental which is tested to see if it’s been correctly extended by 10 days. Our final parameter is the description which will be returned in the outcome of pg_prove if the test fails.

The ROLLBACK is meant to clean the database of all test data.

Let’s run this unit test on the command line, with pg_prove:

pg_prove -d <your_database> <path_to_your_unit_test>/extension_unit_tests.sql

We should see these results:

extension_unit_tests.sql .. ok
All tests successful.
Files=1, Tests=2,  0 wallclock secs ( 0.01 usr +  0.00 sys =  0.01 CPU)
Result: PASS

Congrats! We’ve tested out a single case for a function.

However, what if we want to test all possible cases for this function to be sure it performs correctly?

To do this, we can alter the procedure fill_test_data to loop through a date range given in the arguments _start_date and _end_date. For our example we’ll provide it with the week of Christmas, but it’ll work for any date range provided the rental store gives extensions for DVDs rented on other dates.

Instead of passing a test number as an argument, we can simply initialize it as a variable that’s incremented with every loop. Inside the WHILE loop, every iteration will populate the database with the test data, perform extension_check with the test rental ID, and increment the date and test number.

We’ll end the transaction a little differently by setting our pgTAP plan equal to the number of tests we have in our temporary table, to avoid hard-coding our plan count. We’ll also add one to check its existence with has_function.

Finally, our is() will also iterate through our temporary table and perform a check in the rental table to confirm that the return date has been correctly extended by ten days.

BEGIN;

CREATE TEMPORARY TABLE test_instance_references (
    test_rental_id integer PRIMARY KEY,
    test_number integer,
    start_date date
);


CREATE OR REPLACE PROCEDURE fill_test_data (
        _start_date date,
        _end_date date
    )
    LANGUAGE PLPGSQL
    AS $procedure$
    DECLARE
        _address_id smallint;
        _store_id smallint;
        _inventory_id integer;
        _customer_id integer;
        _rental_id integer;
        _test_number integer;
    BEGIN
        _test_number := 1;

        WHILE _start_date <= _end_date LOOP
            SELECT nextval('customer_customer_id_seq1'::regclass) INTO _customer_id;
            SELECT nextval('rental_rental_id_seq'::regclass) INTO _rental_id;
            SELECT nextval('inventory_inventory_id_seq'::regclass) INTO _inventory_id;
            SELECT store_id FROM store LIMIT 1 INTO _store_id; -- any store will do
            SELECT address_id FROM address LIMIT 1 INTO _address_id; -- any address will do

            INSERT INTO customer (customer_id, store_id, first_name, last_name, address_id)
            VALUES (
                _customer_id,
                _store_id,
                'Test First Name',
                'Test Last Name',
                _address_id
            );

            INSERT INTO inventory (inventory_id, film_id, store_id)
            VALUES (
                _inventory_id,
                (SELECT film_id FROM film LIMIT 1), -- arbitrary film
                _store_id
            );

            INSERT INTO rental (rental_id, rental_date, inventory_id, customer_id, staff_id)
            VALUES (
                _rental_id,
                _start_date,
                _inventory_id,
                _customer_id::smallint,
                (SELECT staff_id FROM staff WHERE store_id = _store_id LIMIT 1) -- select arbitrary staff member from store
            );

            INSERT INTO test_instance_references (test_rental_id, test_number, start_date)
            VALUES (
                _rental_id,
                _test_number,
                _start_date
            );

            PERFORM extension_check(_rental_id);

            _start_date := _start_date + INTERVAL '1 day';
            _test_number := _test_number + 1;

        END LOOP;
    END;
$procedure$;

CALL fill_test_data(('2023-12-25'::date - INTERVAL '6 days')::date, '2023-12-25'::date);

SELECT plan(count(*)::integer + 1) from test_instance_references;

SELECT * from test_instance_references;

SELECT has_function(
    'extension_check',
    ARRAY['int']
    -- Description optional - default description usually substantial
);

SELECT is(
    (SELECT return_date FROM rental WHERE rental_id = tf.test_rental_id)::date,
    (tf.start_date + INTERVAL '10 DAYS')::date
)
FROM test_instance_references as tf;


ROLLBACK;

Our results should be:

$ pg_prove -d <your_database> <path_to_your_unit_test>/extension_unit_tests.sql
extension_unit_tests.sql .. ok
All tests successful.
Files=1, Tests=8,  0 wallclock secs ( 0.02 usr +  0.00 sys =  0.02 CPU)
Result: PASS

As someone who found pgTAP fairly straightforward the challenge of intertwining those functions in a repeatable way was what drove me to write this article. Hopefully this makes it easier for the next coder to play around and get the most out of this simple but intuitive “Test Anything Protocol”.

Feel free to share your thoughts and comments below, or to share any more useful tips on pgTAP or general Postgres unit testing.

How To Set Up pgTAP for Writing PostgreSQL Database Unit Tests

2023-04-12T00:00:00+00:00

In a previous blog post, my colleague Josh Tolley introduced the concept of using pgTAP, a set of Postgres functions designed for writing unit tests within the database. This post will serve as a supplement to Josh’s post, focusing on the process of setting up pgTAP for testing a Postgres database.

Installing & setting up pgTAP

Note: make sure you have Postgres installed on your system. If you don’t have it, you can follow the Postgres documentation to learn how to install it.

To install pgTAP for Postgres, you will need to follow these steps:

Download the pgTAP source code from its GitHub repository.
```
git clone https://github.com/pgtap/pgtap.git
```
Navigate to the pgtap directory.
```
cd pgtap
```
Install with make.
```
make && make install
```
Finally, connect to your Postgres database using psql or any other Postgres client and run the following SQL command to create the pgTAP extension.
```
CREATE EXTENSION pgtap;
```

Now you should have pgTAP installed and ready to use in your Postgres database. If you face issues with installation, visit the pgTAP documentation for further help.

Writing simple database tests with pgTAP

Test to check if a table exists:

SELECT plan(1);
SELECT has_table('public', 'your_table', 'Table your_table should exist');
SELECT finish();

Test to check if a column exists in a table:

SELECT plan(1);
SELECT has_column('public', 'your_table', 'your_column', 'Column your_column should exist in your_table');
SELECT finish();

Test to check if a function returns the expected result:

SELECT plan(1);
SELECT is(your_function(), expected_result, 'your_function() should return expected_result');
SELECT finish();

More pgTAP tests

Test to check if a trigger is triggered

Consider the following example. We have a table named employees and a table named audit_log. When a new employee is added to the employees table, a trigger named insert_employee_trigger fires and logs the new employee’s ID and creation timestamp in the audit_log table. Let’s test if the trigger gets fired and produces the expected outcome.

Create the tables.

CREATE TABLE employees (
  id SERIAL PRIMARY KEY,
  name VARCHAR(255) NOT NULL
);

CREATE TABLE audit_log (
  id SERIAL PRIMARY KEY,
  employee_id INTEGER NOT NULL,
  created_at TIMESTAMP NOT NULL
);

Create the trigger and trigger function.

CREATE OR REPLACE FUNCTION insert_employee_trigger_function()
RETURNS TRIGGER AS $$
BEGIN
  INSERT INTO audit_log (employee_id, created_at) VALUES (NEW.id, NOW());
  RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER insert_employee_trigger
  AFTER INSERT ON employees
  FOR EACH ROW
  EXECUTE FUNCTION insert_employee_trigger_function();

Now, write a test using pgTAP to test the trigger.

-- Set the test plan
SELECT plan(1);

-- Prepare test data and perform operations that should fire the trigger
INSERT INTO employees (name) VALUES ('John Doe');


-- Check if the desired outcome of the trigger has occurred
SELECT is(
  (SELECT name FROM employees WHERE id = (SELECT employee_id FROM audit_log ORDER BY id DESC LIMIT 1)),
  'John Doe',
  'The insert_employee_trigger should have been fired and produced the expected outcome'
);

Test to check if a constraint is enforced

For this example, we’ll use the following two tables:

customers table with columns id and name.
orders table with columns id, customer_id, and amount
- There is a foreign key constraint between the orders table and the customers table on the customer_id column.

First, create the two tables and add the foreign key constraint.

CREATE TABLE customers (
  id SERIAL PRIMARY KEY,
  name VARCHAR(255) NOT NULL
);

CREATE TABLE orders (
  id SERIAL PRIMARY KEY,
  customer_id INTEGER REFERENCES customers(id),
  amount INTEGER
);

Now, create a test script named fk_test.sql with the following content.

BEGIN;

-- Create test data in customers table
INSERT INTO customers (id, name) VALUES (1, 'Customer 1');

-- Declare the number of tests
SELECT plan(1);

-- Attempt to insert an order with an invalid customer_id
-- This should fail due to the foreign key constraint
SELECT throws_ok(
    'INSERT INTO orders (id, customer_id, amount) VALUES (1, 969, 100);',
    '23503',
    'insert or update on table "orders" violates foreign key constraint "orders_customer_id_fkey"'
);

-- Conclude the test and discard changes
SELECT finish();
ROLLBACK;

Organizing and naming your test files based on Test Anything Protocol (TAP)

Test Anything Protocol (TAP) is a convention for naming test files and organizing them within a project directory structure. The naming convention involves adding a numeric prefix to the test file name to indicate the order in which the tests should be run. For example, 00-test.sql is the first test file to be executed, followed by 01-another-test.sql, and so on.

The project directory structure usually includes a directory called t (short for “test”) under the project root directory. This t directory contains all the test files following the TAP naming convention.

Using this structure makes it easier to maintain and organize tests, as well as to run them in the correct order. This is especially important when tests depend on the results of previous tests, or when you want to ensure that tests are executed in a specific sequence.

Here’s an example of a project directory structure using the TAP-related naming convention:

my_project/
│
├─ src/
│   ├── function1.sql
│   ├── function2.sql
│   └── function3.sql
│
└─ t/
    ├── 00-test.sql
    ├── 01-another-test.sql
    └── 02-yet-another-test.sql

In this example, the project contains a src directory for the source code (database functions) and a t directory for the test files following the TAP naming convention.

Running your tests using pg_prove

Now that we have written some tests using pgTAP, let’s see how we can run them. We’ll use pg_prove, a utility included with pgTAP, to automate the process and get a summary of the results.

To run your tests with pg_prove, you’ll need to specify the name of the database you want to connect to and the path to your test file. For example, if your test file is located at t/00-test.sql and your database is named epitrax_db, you can run the following command:

pg_prove -d epitrax_db t/00-test.sql

When you run this command, pg_prove will execute all the tests in the specified file and print a summary of the results. The output will include the number of tests run, the number of tests passed, the number of tests failed, and any diagnostic messages or errors that were encountered.

Here’s an example of what the output might look like:

t/00-test.sql .. ok
All tests successful.
Files=1, Tests=5,  0 wallclock secs ( 0.01 usr  0.00 sys +  0.02 cusr  0.00 csys =  0.03 CPU)
Result: PASS

In this example, pg_prove ran one test file (00-test.sql) and executed 5 tests. All of the tests passed, so the output shows that the result was PASS.

If any of the tests fail, pg_prove will print a detailed report of the failures, including any error messages or diagnostic information. This can help you quickly identify and fix any issues in your code.

Using pg_prove to automate your tests can save you a lot of time and effort, especially if you have a large number of tests to run. It also makes it easy to integrate your tests into your build process or continuous integration pipeline.

pgTAP in EpiTrax

EpiTrax relies on Postgres and its functions, making pgTAP an ideal tool for testing the platform’s database functions, ensuring data integrity, and maintaining the robustness of the system. By utilizing pgTAP, EpiTrax efficiently validates various aspects of its database structure and behavior.

Conclusion

pgTAP database unit tests provide numerous benefits, including ensuring code quality and simplifying the testing process. This guide should serve as a starting point for setting up pgTAP, and we encourage you to explore the pgTAP documentation and read the aforementioned blog post on automated database testing with pgTAP for more in-depth information on writing and running tests with pgTAP. By integrating pgTAP in projects like EpiTrax, it’s possible to improve overall database performance, stability, and security.

Using pgTAP to automate database testing

2022-03-16T00:00:00+00:00

Photo from PxHere

Recently I started learning to tune pianos. There are many techniques and variations, but the traditional method, and the one apparently most accepted by ardent piano tuning purists, involves tuning one note to a reference, tuning several other notes in relation to the first, and testing the results by listening closely to different combinations of notes.

The tuner adjusts each new note in relation to several previously tuned notes. Physics being what it is, no piano can play all its tones perfectly, and one of the tricks of it all is adjusting each note to minimize audible imperfections. The tuner achieves this with an exacting series of musical intervals tested against each other.

Databases need tests too

One of our customers needed to add security policies to their PostgreSQL database, to limit data visibility for certain new users. This can quickly become complicated and ticklish, ensuring that the rules work properly for the affected users while leaving other users unmolested.

This struck me as an excellent opportunity to create some unit tests, not that there’s any short supply of good opportunities to add unit tests! This is not just because it helps prove that these security policies really do work properly, but because (confession time) I recently did a similar project for a different database without the help of unit tests, and it wasn’t much fun.

So this seemed like a good time to use pgTAP, a set of database functions designed to allow writing unit tests within the database itself. They produce “Test Anything Protocol” (TAP) output, a simple protocol that displays unit test results in an easily understood report.

What to test?

A good first step in writing unit tests is deciding on something to test. In my case, I figured I should make sure row-level security policies were turned on for the tables I was interested in, which is available from the rowsecurity field in the pg_tables view:

select ok(rowsecurity, tablename || ' has row security enabled') from pg_tables
    where schemaname = 'public'
        and (
            tablename in (
                -- ... some hard coded table names
            ) or tablename like 'some_other_tables_%'
        );

The ok() function comes from pgTAP. It counts as one test each time it’s called; the test passes when the first argument is true, and fails when the argument is something else. The second argument is an optional comment describing what’s being tested. Following a pretty common TAP-related naming convention, I put this in a file called 00-test.sql in a directory under the root of my project, simply called t.

A more complicated set of tests could include several different files, where the numeric part of the name helps sort the tests in the desired run order, and the rest of the filename describes the subject of the tests within. But this will do just to get started. I can run it with pg_prove, included with the pgTAP package:

pg_prove -d mydatabase t/00-test.sql

Iteratively improving

This fails, for several reasons.

First, I haven’t yet installed the pgTAP extension in my database, with CREATE EXTENSION pgtap.

I also haven’t actually done anything in my test to run the code I’m testing. The actual code in this project consists of some database functions, which we need to run to create the database security policies, and I haven’t run any of them yet.

And finally, pgTAP requires me to “plan” my tests first, or in other words, I need to inform pgTAP how many tests I plan to run, before I run them. It’s also nice to call finish() so pgTAP can clean up after itself.

I installed the pgTAP extension in my database, and modified the test as follows:

begin;

\i create_policy.sql

select plan(1);  -- plan for a single test

select ok(rowsecurity, tablename || ' has row security enabled') from pg_tables
    where schemaname = 'public'
        and (
            tablename in (
                -- ... some hard coded table names
            ) or tablename like 'some_other_tables_%'
        );

select finish();

rollback;

This wraps my test in a transaction, so that I can roll everything back to leave the database essentially as I found it. It also calls the actual code I’m testing, in create_policy.sql, and plans one test. And it gives me this new failure:

t/m.sql .. All 1 subtests passed

Test Summary Report
-------------------
t/m.sql (Wstat: 0 Tests: 226 Failed: 225)
  Failed tests:  2-226
Parse errors: Bad plan.  You planned 1 tests but ran 226.
Files=1, Tests=226,  1 wallclock secs ( 0.04 usr  0.00 sys +  0.03 cusr  0.01 csys =  0.08 CPU)
Result: FAIL

The problem here is that each call to ok() counts as one test, and my test apparently found 226 tables to check for row-level security. I can improve the planning like this:

select plan(count(*)::integer)
    from pg_tables where schemaname = 'public'
        and (
            tablename in (
                -- ... some hard coded table names
            ) or tablename like 'some_other_tables_%'
        );

count() returns a bigint, and plan() expects integer, so this requires a typecast, but is otherwise pretty simple. And now my tests pass:

josh@here:~dw$ pg_prove -d nedss t/00-test.sql
t/00-test.sql .. ok
All tests successful.
Files=1, Tests=226,  1 wallclock secs ( 0.03 usr  0.01 sys +  0.03 cusr  0.01 csys =  0.08 CPU)
Result: PASS

Looking back and ahead

Suffice it to say that pgTAP includes many functions similar to ok(), to test various aspects of the database, its structure, and its behavior, and I’d recommend interested users review the documentation for more details. I intended this post only as an introduction.

In its completed state, my test suite comprised several tests ensuring various required preliminaries were in place, a few tests like the one above that check for necessary table-specific settings, others that ensure the affected roles were created, and finally some which create some sample data and use SET ROLE to test the data visibility directly for roles with various policies applied.

And to be honest, I was surprised at the sense of security that came over me with this completed test suite. As I mentioned, I’d done similar work previously, and knew that although I was confident in the code when it was written, that confidence came only through fairly extensive manual testing. I know very well the struggles of bit rot, and I knew it would be at least as hard to repeat that testing regimen by hand sometime down the road after a year or two.

I also recognized that if I ever needed to set up similar policies again, I could use these tests themselves as a reference, because they show exactly how to run the code in question. Though of course I included that information in the project’s associated README file as well … right?

Let us know if you’ve used pgTAP, and what effect it has had on your database development.

Database Design: Using Documents

2022-03-09T00:00:00+00:00

Using documents in relational databases is increasingly popular. This technique can be practical and efficient when used in fitting circumstances.

Example

Let’s start with an example. Imagine we are scraping web sites for external URLs and store them in a table. We’ll have the web sites table to store the scrape timestamp and another table to store all of the references.

CREATE TABLE web_sites (
    web_site_domain text NOT NULL,
    last_scraped_at timestamptz,

    PRIMARY KEY (web_site_domain)
);

CREATE TABLE refs (
    web_site_domain text NOT NULL,
    ref_location text NOT NULL,
    link_url text NOT NULL,

    PRIMARY KEY (web_site_domain, ref_location, link_url),
    FOREIGN KEY (web_site_domain) REFERENCES web_sites
);

We do not need to bother adding an id to the web_sites table, because we assume there won’t be too many of them. The domain is small and more practical to use as an identifier. If you are curious about advantages of using natural keys, see my previous article.

Normalized Tables

There may be many thousands of unique URLs for a single web site and other web sites may refer to the same URLs. To try to minimize the storage, we can keep the locations and the URLs in separate tables, give them integer identifiers, and have another table for the many-to-many relations.

CREATE TABLE locations (
    location_id bigint NOT NULL GENERATED ALWAYS AS IDENTITY,
    web_site_domain text NOT NULL,
    ref_location text NOT NULL,

    PRIMARY KEY (location_id),
    UNIQUE (web_site_domain, ref_location),
    FOREIGN KEY (web_site_domain) REFERENCES web_sites
);

CREATE TABLE links (
    link_id bigint NOT NULL GENERATED ALWAYS AS IDENTITY,
    link_url text NOT NULL,

    PRIMARY KEY (link_id),
    UNIQUE (link_url)
);

CREATE TABLE locations_links (
    location_id bigint NOT NULL,
    link_id bigint NOT NULL,

    PRIMARY KEY (location_id, link_id),
    FOREIGN KEY (location_id) REFERENCES locations,
    FOREIGN KEY (link_id) REFERENCES links
);

The idea in here is to keep our biggest table narrow. It’d pay off to refer to the locations and links just with integer identifiers when we’ll have very many of them.

Table Sizes

We’ll have many web sites and many URLs, so our lookup tables would be big and the relation table would be even bigger. That is going to be a major problem because of many reasons, one of them being space efficiency. Narrow tables with many rows are not very space efficient especially on Postgres where there’s 24 bytes per-row overhead. To demonstrate it being significant, let’s add one row to each of the tables to see the sizes compared with the overhead.

INSERT INTO web_sites (web_site_domain)
VALUES ('example.com');

INSERT INTO refs (web_site_domain, ref_location, link_url)
VALUES ('example.com', '/source', 'http://example.net/target.html');

INSERT INTO locations (web_site_domain, ref_location)
VALUES ('example.com', '/source');

INSERT INTO links (link_url)
VALUES ('http://example.net/example.html');

INSERT INTO locations_links
VALUES (1, 1);

SELECT table_name, tuple_len, (100.0 * 24 / tuple_len)::int AS overhead_perc
FROM information_schema.tables, LATERAL pgstattuple(table_name)
WHERE table_schema = 'public';

   table_name    | tuple_len | overhead_perc
-----------------+-----------+---------------
 web_sites       |        36 |            67
 refs            |        75 |            32
 locations       |        52 |            46
 links           |        64 |            38
 locations_links |        40 |            60

With just one row and such short domains and URLs, we wouldn’t be gaining anything by having the links on a lookup table. In fact, it’s a lot less efficient. Although, the situation may change with many rows.

Composite Types

We can combine multiple fields in a column to avoid the overhead of many rows in narrow tables. Postgres has good support for composite types and arrays over them. They are useful if you need strict data type checking.

CREATE TYPE web_site_page AS (
    ref_location text,
    link_urls text[]
);

ALTER TABLE web_sites ADD COLUMN pages web_site_page[];

UPDATE web_sites
SET pages = ARRAY[ROW(ref_location, ARRAY[link_url])::web_site_page]
FROM refs;

SELECT pg_column_size(pages) FROM web_sites;

 pg_column_size
----------------
            117

As you see, the new column is not exactly small because composite types and arrays come with their own overheads. Still it could be much better when a web site has many references.

JSON Document Column

The composite types are not very easy to work with. We can store the pages in another format. JSON is the most popular one nowadays. There are multiple options to store JSON in a column in Postgres. Let’s start with the text-based data type.

ALTER TABLE web_sites ADD COLUMN pages_json json;

UPDATE web_sites SET pages_json = to_json(pages);

SELECT pg_column_size(pages_json) FROM web_sites;

 pg_column_size
----------------
             76

What is perhaps also surprising is the JSON being smaller than the array over composite type. This is because Postgres data types are a bit wasteful because of alignment and storing oids of the enclosed data types repeatedly.

Binary JSON

Another JSON data type, jsonb is available in Postgres. It is a binary structure that allows faster access to the values inside the document.

ALTER TABLE web_sites ADD COLUMN pages_jsonb jsonb;

UPDATE web_sites SET pages_jsonb = to_jsonb(pages);

SELECT pg_column_size(pages_jsonb) FROM web_sites;

 pg_column_size
----------------
             98

As you see the text-based JSON is still smaller. This is due to the binary offsets stored in the binary structure for faster access to the fields. The text-based JSON also compresses better.

Size Comparison

Very simple single row data shows the text-based json as the smallest, binary jsonb a bit larger, and the composite type to be the largest. However, the differences would be a lot smaller with more realistic sample data with many items inside the documents. I generated some data and gathered these results (the sizes excluding the indexes).

    model         |   size
------------------|---------
single table      |  436 MiB
normalized tables |  387 MiB
composite type    |  324 MiB
json              |  318 MiB
jsonb             |  320 MiB

You may expect the normalized tables to be the smallest as otherwise the metadata is stored repeatedly in the columns. Though, the results reveal it’s the other way around. This is because of the per-row overhead in Postgres. It becomes noticeable in narrow tables as in this example.

It’s also useful to notice that the most of the size is occupied by the TOAST tables when the data is stored in a single column. TOAST is a mechanism in Postgres that kicks in when a column is large enough which would often be the case when you design your tables this way. In this case, the table excluding the TOASTed part is pretty small which would help any query that doesn’t touch the large column.

Usability

What matters more than size is the usability of having it all in a single column. It is really practical to send and receive this column as a whole all the way from the database to the frontend. Compare this with dealing with lots of small objects used with the normalized tables.

It’s also possible in Postgres to index the document columns to allow efficient searches in them. There are many options to do so with advantages and disadvantages over each other. Though this post is getting long. I’ll leave indexing to another one.

String Processing in SQL, the Hard Way

2022-02-28T00:00:00+00:00

Photo by Kristy Lee

When I was a kid, my family visited a prehistoric cliff-dweller settlement, reconstructed and preserved as a museum exhibit. The homes were built in caves in high sandstone cliffs, and though I followed a well-marked path to visit them, the original inhabitants climbed in and out via holes cut into the rock face. Apparently to deter unwanted visitors, the builders carefully spaced the foot holds so that climbers needed to start their ascent on the correct foot. Someone who didn’t know the secret of the path, or who carelessly forgot, might get halfway up the cliff and discover he or she couldn’t stretch to the next foot hold and had to turn back.

The other day I found myself confronted with several comma-separated lists of strings, and a problem: I needed the longest possible substring from each, starting from the beginning, ending in a comma, and shorter than a certain length, and I needed to do it in the database. “Easy peasy,” you may say to yourself, if you have a little SQL programming experience and a penchant for cheesy clichés, and you’d be right except for one thing: I let my mind wander a little too far down into the uncharted mists of imagination and like the cliff dweller, I got off on the wrong foot. What followed was a thankfully brief exploration of the Hard Way to Do It.

The unsettling method on which I settled had me dividing each list into its component parts, reconstructing candidate results element by element, and finding the longest one less than the maximum overall length. Fortunately the problem didn’t allow me to reorder the elements in my input lists, or I’d probably still be working on this query. Here’s the monstrosity I eventually built:

select
    length(c), c
from (
    select
        concat_ws(', ', variadic array_agg(f) over (rows between unbounded preceding and current row)) c
    from (
        select regexp_split_to_table('alpha, bravo, charlie, delta, echo, fox trot', ',\s+') as f
    ) f
) c
where length(c) < 30
order by length(c) desc nulls last
limit 1

The innermost subquery here, based on regexp_split_to_table, divides the input string into an ordered list of its component elements:

select regexp_split_to_table('alpha, bravo, charlie, delta, echo, fox trot', ',\s+');
 regexp_split_to_table
-----------------------
 alpha
 bravo
 charlie
 delta
 echo
 fox trot
(6 rows)

I’m never one to bypass a good opportunity to shoehorn a window function into my SQL, and here we get to use the frame clause. Recall that for each row of a query, a window function processes a selection of rows, called a “partition”, and in fact processes only a “frame”, a subset of that partition. Window frames are fairly complicated subjects; I’ll try to present here a brief summary of the bits relevant to this query, and recommend the PostgreSQL documentation for further details.

Our particular window frame clause, rows between unbounded preceding and current row, means that the window frame for a given row should include all the rows from the first row of the partition through the current row. Given that we have no order by clause in our window definition, terms like “first row” refer to the first row in the order the rows were provided to the WindowAgg executor node. Whether it’s safe to trust that order to remain stable without a specific order by clause is an awfully good question I didn’t try to answer but for my purposes I needed to preserve the order of elements as returned by regexp_split_to_table. The default window frame clause is range unbounded preceding, which the documentation notes is a shortened form of range between unbounded preceding and current row, and without a specific order by clause, this means the window frame includes all rows in the partition. As a fun exercise, compare these two results, which differ only in the presence of an order by clause:

select array_agg(f) over () c
    from (
        select regexp_split_to_table('alpha, bravo, charlie, delta, echo, fox trot', ',\s+') as word
    ) f
;
                              c
-------------------------------------------------------------
 {(alpha),(bravo),(charlie),(delta),(echo),"(\"fox trot\")"}
 {(alpha),(bravo),(charlie),(delta),(echo),"(\"fox trot\")"}
 {(alpha),(bravo),(charlie),(delta),(echo),"(\"fox trot\")"}
 {(alpha),(bravo),(charlie),(delta),(echo),"(\"fox trot\")"}
 {(alpha),(bravo),(charlie),(delta),(echo),"(\"fox trot\")"}
 {(alpha),(bravo),(charlie),(delta),(echo),"(\"fox trot\")"}
(6 rows)

select array_agg(f) over (order by word) c
    from (
        select regexp_split_to_table('alpha, bravo, charlie, delta, echo, fox trot', ',\s+') as word
    ) f
;
                              c
-------------------------------------------------------------
 {(alpha)}
 {(alpha),(bravo)}
 {(alpha),(bravo),(charlie)}
 {(alpha),(bravo),(charlie),(delta)}
 {(alpha),(bravo),(charlie),(delta),(echo)}
 {(alpha),(bravo),(charlie),(delta),(echo),"(\"fox trot\")"}
(6 rows)

Anyway, that’s a tangent, but I found it a helpful one, as the default window frame is a well-known trap for young players, to which I’ve fallen prey in the past at least once. Back to my over-complicated query.

I’m aiming to produce a series of strings, each containing one more element than the last one. Now that I have a set of arrays with the elements I want inside, I turned to the concat_ws function to turn them into strings. concat_ws concatenates each of its arguments with a user-specified separator string, and the variadic keyword lets me pass an array of values and have them treated as individual arguments:

select
    concat_ws(', ', variadic array_agg(f) over (rows between unbounded preceding and current row)) c
from (
    select regexp_split_to_table('alpha, bravo, charlie, delta, echo, fox trot', ',\s+') as f
) f
;
                      c
----------------------------------------------
 alpha
 alpha, bravo
 alpha, bravo, charlie
 alpha, bravo, charlie, delta
 alpha, bravo, charlie, delta, echo
 alpha, bravo, charlie, delta, echo, fox trot

Now we’re getting close to the answer. My final steps are to filter out all resultant strings longer than the acceptable length and pick the longest one:

select
    length(c), c
from (
    select
        concat_ws(', ', variadic array_agg(f) over (rows between unbounded preceding and current row)) c
    from (
        select regexp_split_to_table('alpha, bravo, charlie, delta, echo, fox trot', ',\s+') as f
    ) f
) c
where length(c) < 30
order by length(c) desc nulls last
limit 1
;
 length |              c
--------+------------------------------
     28 | alpha, bravo, charlie, delta
(1 row)

Success! It only took about a hundred more steps than were really necessary. In my defense, it wasn’t too long after I’d finished composing this query that I realized I was making it much harder than I should have.

I posted it to our internal company chat, and within a minute or two my colleague Kürşat had answered with his own version:

SELECT regexp_matches(substring('alpha, bravo, charlie, delta, echo, fox trot' from 1 for 30), '(.*),');
          regexp_matches
----------------------------------
 {"alpha, bravo, charlie, delta"}
(1 row)

This simply finds a substring of acceptable length from the input text, and removes everything from the final comma onward. Not only did this take about 30 seconds to compose, compared to the half hour or so I’d spent on my first version, but on my laptop it completes in about half the time.

That fact notwithstanding, it’s often true that we learn rather a lot from our mistakes — and others’ mistakes — and I hope that’s true for you in this case.

How to use regular expression group quantifiers in PostgreSQL

2022-01-27T00:00:00+00:00

I recently encountered a situation where it was necessary to extract address content from text in HL7 V2 format from a PostgreSQL table’s column. The following example is representative:

||121212^^^2^ID 1|676767||SELVA^KUMAR^^^^|19480203|M||B||123456 SAMPLE ROAD^^New York City^NY^12345^USA^H^^New York||123456-7890|||M|NON|4000|

In order to manipulate our example, the address section needs to be extracted from the HL7 V2 message PID segment for patient demographic information. The segments have delimiters for fields (|), components (^), subcomponents (&) and repetition (~).

Our example has only fields and components delimited by pipe (|) and caret (^). The address contains nine components delimited by ^.

I hoped to do this by applying a regular expression (regex) because the address is in a standard format that regex can match with alphanumeric and caret repetition.

Here is my journey figuring out how to match the data I wanted.

Regex pattern in grep

As a test, I got this regex working with the grep command, which successfully extracts the address section from the content:

$ content='||121212^^^2^ID 1|676767||SELVA^KUMAR^^^^|19480203|M||B||123456 SAMPLE ROAD^^New York City^NY^12345^USA^H^^New York||123456-7890|||M|NON|4000|'
$ echo "$content" | grep -Eo "([A-Za-z0-9 #'.,/-]*\^){8}[A-Za-z0-9 ]*"
123456 SAMPLE ROAD^^New York City^NY^12345^USA^H^^New York

PostgreSQL regex attempt

The PostgreSQL regexp_matches function supports extraction by pattern-matching data from the content. But when I used the same regex pattern with the regexp_matches function, instead of all eight groups, only the eighth value was returned:

=# SELECT regexp_matches('||121212^^^2^ID 1|676767||SELVA^KUMAR^^^^|19480203|M||B||123456 SAMPLE ROAD^^New York City^NY^12345^USA^H^^New York||123456-7890|||M|NON|4000|', '([A-Za-z0-9 #''.,/-]*\^){8}[A-Za-z0-9 ]*', 'g');
 regexp_matches 
----------------
 {^}
(1 row)

More generally, our query is returning the Nth matching group instead of returning all matching groups until the Nth regex group. So if we try to fetch the text matching with 3 groups, the quantifier will return the third field of all match sections instead of the third group itself.

=# SELECT regexp_matches('||121212^^^2^ID 1|676767||SELVA^KUMAR^^^^|19480203|M||B||123456 SAMPLE ROAD^^New York City^NY^12345^USA^H^^New York||123456-7890|||M|NON|4000|', '([A-Za-z0-9 #''.,/-]*\^){3}[A-Za-z0-9 ]*', 'g');
   regexp_matches   
--------------------
 {^}
 {^}
 {"New York City^"}
 {USA^}
(4 rows)

PL/Perl function

Since that doesn’t satisfy our requirements, I tried using a Perl regex through my own PL/Perl function, and got the expected answer:

=# CREATE OR REPLACE FUNCTION perl_regexp_matches (IN str text, IN pattern text) RETURNS text AS $$
    my ($input, $pattern) = @_;
    $output = [$input =~ m/($pattern)/];
    return $output->[0]
$$ LANGUAGE plperl;

=# SELECT perl_regexp_matches('||121212^^^2^ID 1|676767||SELVA^KUMAR^^^^|19480203|M||B||123456 SAMPLE ROAD^^New York City^NY^12345^USA^H^^New York||123456-7890|||M|NON|4000|', '([A-Za-z0-9 #''.,/-]*\^){8}[A-Za-z0-9 ]*');
                    perl_regexp_matches
------------------------------------------------------------
 123456 SAMPLE ROAD^^New York City^NY^12345^USA^H^^New York
(1 row)

But I researched further for a simple solution to achieve the result without using a custom function and the PL/Perl extension.

PostgreSQL regex solution

In Postgres regex syntax, parentheses () create a numbered capture group which leads to returning the contained matching results.

To get the entire matching data, the regex should have a question mark and a colon (?:) added at the beginning of the regex pattern to create a non-capturing group. Then because no group is capturing, instead the complete match is returned:

=> SELECT regexp_matches('||121212^^^2^ID 1|676767||SELVA^KUMAR^^^^|19480203|M||B||123456 SAMPLE ROAD^^New York City^NY^12345^USA^H^^New York||123456-7890|||M|NON|4000|', '(?:[A-Za-z0-9 #''.,/-]*\^){8}[A-Za-z0-9 ]*');
                         regexp_matches                         
----------------------------------------------------------------
 {"123456 SAMPLE ROAD^^New York City^NY^12345^USA^H^^New York"}
(1 row)

That turns out to be what was happening with my PL/Perl function where m/($pattern)/ captures the entire match, and what grep was doing because of its option -o or --only-matching, which prints the matching part of the lines rather than its default of printing the entire line.

And we can also use the Postgres function substring to return the bare text itself instead of arrays as regexp_matches does:

=# SELECT substring('||121212^^^2^ID 1|676767||SELVA^KUMAR^^^^|19480203|M||B||123456 SAMPLE ROAD^^New York City^NY^12345^USA^H^^New York||123456-7890|||M|NON|4000|' FROM '(?:[A-Za-z0-9 #''.,/-]*\^){8}[A-Za-z0-9 ]*');
                         substring
------------------------------------------------------------
 123456 SAMPLE ROAD^^New York City^NY^12345^USA^H^^New York
(1 row)

Reference

Ecommerce customer names with interesting Unicode characters

2021-12-29T00:00:00+00:00

One of our clients with a busy ecommerce site sees a lot of orders, and among those, sometimes there are unusual customer name and address submissions.

We first noticed in 2015 that they had a customer order come in with an emoji in the name field of the order. The emoji was 😏 and we half-jokingly wondered if that was a new sign of fraud.

Over the following 3 years only a few more orders came in with various emoji in customer’s names, but in mid-2018 emoji started to appear increasingly frequently until now one now appears on average every day or two.

Why the sudden appearance of emoji in 2015? It correlates with the rapid shift to browsing the web and shopping on mobile devices. Mobile visits now represent more than half of this client’s ecommerce traffic.

Most people in 2015 didn’t even know how to type emoji on a desktop or laptop computer, but mobile touchscreen keyboards began showing emoji choices around that time, so the mobile explanation makes sense. And mobile keyboard autocorrect also sometimes offers emoji in addition to words, making them even more common in the past few years.

Just for fun I wanted to automatically find all such “interesting” names, so I wrote a simple report that uses SQL to query their PostgreSQL ecommerce database.

To preserve customer privacy, names shown here have been changed and limited to a few names that are common in the United States.

Real names with bonuses

First let’s look at the apparently real names with emoji and other non-alphabetical Unicode characters mixed in:

Amy 🩺
Amy💕
Amy👑
👸Amy
Anna 💫
Anna 😎
Anna ❤️
Anna💊💉
Bob☯️
Bob😍😘😜👑💞
Bob🐞
Brenda 🌙
Brenda 🥳🤪
Brenda☠️
B R E N D A 💕💅🏾💃🏾👜
Cameron 🌻
Cameron 😎
Cameron 💁🏻‍♀️
Doug 💕
Doug 🅱️
Doug 🌸🤩
Doug 👦🏻❤️
Emily 🌷
Emily 😊🇨🇺
Emily😚
Emily 🎰👑❤️
Frank 🥰
Frank ⚗️
Frank🔆
Frank💞💰
Jane 💕
Jane⁸
Jane ❤️❤️
Jane🔆
Jill 👑🎀
Jill 👼🏽✨🌫
Jill🍓🍒🍒
Jill👸🏽💖
Jim 🐰
Jim, 💕
Jim’s iPhone✨
Joe 🐯
Joe 🇦🇺
Joe😏
Joe🏀🎸‼️
John 👪
John⁵
John🎭
Karen 🌻🌹
Karen 👑✨
Karen 🔒❤️
Karen🎀👑
Kate💙💚
Kate🤪🤞🏽💙
Kate.💘
K$ate💉
Liz🌺
Liz❣️
Liz❤️🙃
Liz Mama💙💙
Mary 🎀
Mary💘
Mary👼🏼💓
Mary⁹
Mike 👑
Mike 👑🌸
Mike 👩🏻‍🌾
Mike⁶
Sarah 👑
Sarah👑A.
✨Sarah✨
💛🌻 Sarah
Steve🐑
Steve 🦁
Steve♓️💓
Victoria 🥴
Victoria🌻
V𝚒𝚌𝚝𝚘𝚛𝚒𝚊
V I C T O R I A 🤍

Would you have expected all that in ecommerce orders? I didn’t!

Fake names

Next let’s look at placeholder names with people’s role or self-description or similar:

Amor ⚽️
Babe ❤️
C𝚒𝚝𝚒𝚣𝚎𝚗
Daddy🥴😏
Daddy😘👴👨🙏👨👩👧🙇🏾
Fly High 🕊
Forever 💍💜
Granny 👵🏽
HOME ❤️
Home🏠
Home🏠💜
Hubby🥰
💦Juicy🍑
me!! 💛
mi amor ❤️
Mom💗
Mom♥️
Mom 🐥💛
MOMMY 💗
Myself 😘
Princess👑
princess❤️
Queen 😍💖🔓
Queen💘
The Husband💍❤️
Wifey 😈✌🏽👅

Perhaps the occurrence of “home” several times reflects a mobile address book auto-fill function for billing or shipping address fields?

Not names at all!

Then there are those customers who didn’t provide any kind of name at all, just emoji and other special characters:

🦋
💙
🤡🎪
💗☁️
🍌
♥️
🌙
⁵
∅

I guess only one or two of those per year doesn’t amount to much, but they’re interesting to see.

Strange addresses

In addition to the name fields we also checked the address fields for unusual characters and found (again, details changed to preserve privacy):

125 E 27😎
227 W 24 Circle ⭕️
3 Blvd. George Washington™

Simply odd

The prizewinner for oddity, which seems like some kind of copy-and-paste mistake, is this in the city field of an address:

Indian® Roadmaster™ Classic

Maybe at least one motorcycle has achieved sentience and needed to do some online shopping!

“Interesting” Unicode ranges

When searching for interesting Unicode ranges, we could just look for characters in the Unicode emoji ranges. That would be fairly straightforward since there are just a few ranges to match.

But we were curious what other unusual characters were getting used aside from emoji, so we wanted to include other classes of characters. So perhaps we should include everything to start and then exclude the entire class of Unicode “word characters”? That covers the world’s standard characters used for names and addresses, including not just Roman/Latin with optional diacritics, but also other character sets such as Cyrillic, Arabic, Hebrew, Korean, Chinese, Japanese, Devanagari, Thai, and many others.

PostgreSQL POSIX regular expressions include the class “word character” represented by either [[:word:]] or the Perl shorthand \w. I started with that, but found it covered too many things I did want to see, such as the visually double-width Latin characters that are part of the Chinese word character range, and some special numbers.

So I switched back to matching what I want, rather than excluding what I don’t want, and I manually went through the Unicode code charts and noted the ranges to include.

The list of Unicode code ranges I came up with, in hexadecimal, is:

250-2ba
2bc-2c5
2cc-2dc
2de-2ff
58d-58e
fd5-fd8
1d00-1dbf
2070-2079
207b-209f
20d0-2104
2106-2115
2117-215f
2163-218b
2190-2211
2213-266e
2670-2bff
2e00-2e7f
2ff0-2fff
3004
3012-3013
3020
3200-33ff
4dc0-4dff
a000-abf9
fff0-fffc
fffe-1d35f
1d360-1d37f
1d400-1d7ff
1ec70-1ecbf
1ed00-1ed4f
1ee00-1eeff
1f000-10ffff

Those ranges exclude several fairly common characters that people (or their software’s autocorrect) used in their address fields, which we wanted to ignore, such as:

Music sharp sign: ♯ (instead of # before a number)
numero: №
care of: ℅
replacement character: � (though this could be interesting if it reveals unknown encoding errors)

The SQL query

PostgreSQL allows us to represent Unicode characters in hexadecimal numbers as either \u plus 4 digits or \U plus 8 digits. So the character 2ba is written in a Postgres string as \u02ba and the range fffe-1d35f is written in a Postgres regex range as [\ufffe-\U0001d35f].

With a little scripting to put it all together, I came up with:

SELECT order_number, order_timestamp::date AS order_date,
    fname, lname, company, address1, address2, city, state, b_fname, b_lname, b_company, b_address1, b_address2, b_city, b_state, phone
FROM orders
WHERE concat(fname, lname, company, address1, address2, city, state, b_fname, b_lname, b_company, b_address1, b_address2, b_city, b_state, phone)
    ~ '[\u0250-\u02ba\u02bc-\u02c5\u02cc-\u02dc\u02de-\u02ff\u058d-\u058e\u0fd5-\u0fd8\u1d00-\u1dbf\u2070-\u2079\u207b-\u209f\u20d0-\u2104\u2106-\u2115\u2117-\u215f\u2163-\u218b\u2190-\u2211\u2213-\u266e\u2670-\u2bff\u2e00-\u2e7f\u2ff0-\u2fff\u3004\u3012-\u3013\u3020\u3200-\u33ff\u4dc0-\u4dff\ua000-\uabf9\ufff0-\ufffc\ufffe-\U0001d35f\U0001d360-\U0001d37f\U0001d400-\U0001d7ff\U0001ec70-\U0001ecbf\U0001ed00-\U0001ed4f\U0001ee00-\U0001eeff\U0001f000-\U0010ffff]'
    -- limit how many years back to go
    AND order_timestamp > (SELECT CURRENT_TIMESTAMP - interval '3 years')
    -- exclude any order that had PII expunged for GDPR
    AND expunged_at IS NULL
ORDER BY order_timestamp DESC

Try a similar query on databases you have access to, and see what interesting user submissions you discover. I always find surprises and in addition to being fun, sometimes we find things that help us improve our input validation and user guidance so that more mistakes are caught when they’re easy for the customer to correct.

Happy holidays!

Database Design: Using Composite Keys

2021-05-20T00:00:00+00:00

Photo by Chuttersnap

Whether to use single-column or composite keys is another long-debated subject of database design. I previously wrote to support using natural keys and here I want to make good arguments for using composite keys.

Single-column vs. Composite

Single-column keys are widely used nowadays. I wouldn’t be surprised if many developers today don’t even think database design with composite keys is possible, even though they were essential in the beginning. Relational databases make no assumption that the keys must be composed of a single column.

Let’s see the composite keys with the corporate database example again. First, we’d need departments and employees:

CREATE TABLE departments (
  department_id text NOT NULL,
  department_location text NOT NULL,

  PRIMARY KEY (department_id)
);

CREATE TABLE employees (
  username text NOT NULL,
  department_id text NOT NULL,

  PRIMARY KEY (username),
  FOREIGN KEY (department_id) REFERENCES departments
);

Then our database grows, and we need to split the departments into multiple teams. Here’s what they’ll look like:

| id | department  | team           | members |
| -- | ----------- | -------------- | ------- |
|  1 | sysadmin    | infrastructure |       5 |
|  2 | sysadmin    | internal_tools |       3 |
|  3 | development | internal_tools |       4 |
|  4 | development | web_site       |       8 |

As you noticed there are 2 teams named internal_tools, so we cannot use this as the primary key column. We can add a surrogate auto-increment column to use as the primary key, or make the department and team the primary key. Let’s go with the surrogate key option first to demonstrate the problem:

CREATE TABLE teams (
  team_id int NOT NULL GENERATED ALWAYS AS IDENTITY,
  department_id text NOT NULL,
  team_name text NOT NULL
  team_members int NOT NULL,

  PRIMARY KEY (team_id),
  UNIQUE (department_id, team_name),
  FOREIGN KEY (department_id) REFERENCES departments
);

As you noticed, we used the surrogate column as the primary key, and added an additional unique index to ensure the team name to be unique in with the department. Now, let’s relate the employees with the teams:

ALTER TABLE employees
  ADD COLUMN team_id int NOT NULL,
  ADD FOREIGN KEY (team_id) REFERENCES teams;

Now, we know both the department and the team of an employee, but the problem is that they can point to inconsistent rows. For example, I can INSERT myself as an employee:

| username | department_id | team_id |
| -------- | ------------- | ------- |
| hasegeli | sysadmin      |       3 |

team_id 3 is in development department, so now you’d never know if I am in the sysadmin or development department. This is a very common source of data integrity problems in the databases. Applications have no good option to handle this. They would typically crash or sometimes show the employee in one department and sometimes in the other.

You cannot easily add a constraint to the database to prevent this. The best option would be to remove the department_id when the team_id is added as we know the department of the team anyway, but this option is expensive and not always possible, for example when the team_id can be NULL.

Relations with Composite Keys

Now, let’s create the teams table with a composite key:

DROP TABLE teams CASCADE;

CREATE TABLE teams (
  department_id text NOT NULL,
  team_id text NOT NULL,
  team_members int NOT NULL,

  PRIMARY KEY (department_id, team_id),
  FOREIGN KEY (department_id) REFERENCES departments
);

ALTER TABLE employees
  ALTER COLUMN team_id TYPE text,
  ADD FOREIGN KEY (department_id, team_id) REFERENCES teams;

With this method, we ensure data integrity and don’t need to disturb the existing users of the employees table while adding the teams. They can still reliably use the department_id column. We can also set the team_id as NULL and still maintain the data integrity.

Ease of Change

As we have already seen, one of the benefits of using composite keys is to respond to database model changes easier and without compromising data integrity. This becomes relevant in many real world scenarios. For example, let’s add the employees to rooms which must belong to the same department:

CREATE TABLE rooms (
  department_id text NOT NULL,
  room_id text NOT NULL,
  room_location text NOT NULL,

  PRIMARY KEY (department_id, room_id),
  FOREIGN KEY (department_id) REFERENCES departments
);

ALTER TABLE employees
  ADD COLUMN room_id text,
  ADD FOREIGN KEY (department_id, room_id) REFERENCES rooms;

Nothing special needed to be done in here, and it is often easy to respond to more complicated change requests. Composite keys play well with database constraints. For example, let’s add an employee rank which needs to be unique in every department:

ALTER TABLE employees
  ADD COLUMN employee_rank int,
  ADD CHECK (employee_rank > 0),
  ADD UNIQUE (department_id, employee_rank);

This was so easy because we have the department_id in here. Now, let’s imagine a more complicated change request. We need to divide the departments into sections:

CREATE TABLE sections (
  department_id text NOT NULL,
  section_id text NOT NULL,
  section_location text NOT NULL,

  PRIMARY KEY (department_id, section_id),
  FOREIGN KEY (department_id) REFERENCES departments
);

ALTER TABLE teams
  ADD COLUMN section_id text NOT NULL,
  DROP CONSTRAINT teams_pkey CASCADE,
  ADD PRIMARY KEY (department_id, section_id, team_id),
  ADD FOREIGN KEY (epartment_id, section_id) REFERENCES sections;

ALTER TABLE employees
  ADD COLUMN section_id text NOT NULL,
  ADD FOREIGN KEY (department_id, section_id, team_id) REFERENCES teams;

As you see, we can make this happen with minimal impact to the users of the database. As a final practice, let’s reserve employee ranks between 1 and 10 for the main section of every department:

ALTER TABLE employees
  ADD CHECK (section_id = 'main' OR employee_rank > 10);

Querying

Another advantage of using composite keys is to have more possibility for joining of tables. For example using the tables we created we can join employees to sections without using the teams tables. This would not be possible if we had used single-column keys everywhere.

Join conditions get complicated with composite keys. The USING clause helps. To demonstrate, let’s join all of the tables we created so far:

SELECT *
  FROM employees
    JOIN deparments USING (department_id)
    JOIN sections USING (department_id, section_id)
    JOIN rooms USING (department_id, room_id)
    JOIN teams USING (department_id, section_id, team_id);

The USING clause only works if you name the columns the same on all tables. I recommend doing so.

Another advantage of USING clause is to eliminate duplicate columns on the result set. If you run this query, you would not see the department_id column repeated 5 times.

Performance Considerations

One disadvantage of using composite keys is to store more data on tables as references. You would also need more space for the indexes as the reference columns often need to be indexed. However storage is the cheapest of resources, and the performance advantages easily outweigh the extra storage.

The main performance advantage of using composite keys is eliminating the need for many joins as mentioned before. However, when you do need to join many tables, the query planner would have many different paths. It’s the query planners’ strong suit to find the best join order. Composite keys allow them to come up with better plans in many scenarios. To demonstrate this, let’s get our join-all-tables query and add some WHERE conditions:

SELECT *
  FROM employees
    JOIN deparments USING (department_id)
    JOIN sections USING (department_id, section_id)
    JOIN rooms USING (department_id, room_id);
    JOIN teams USING (department_id, section_id, team_id)
  WHERE username LIKE 'a%' AND
    department_location LIKE 'b%' AND
    section_location LIKE 'c%' AND
    room_location LIKE 'd%';
    team_members > 3;

Now we added 5 conditions using 5 columns on 5 different tables. The query planner can estimate which conditions are more selective and plan to join the tables from the smaller one to bigger one.

Database Design: Using Natural Keys

2021-03-15T00:00:00+00:00

Whether to use natural or surrogate keys is a long-debated subject of database design. I am a fan of using natural keys. I think there are even more compelling reasons to use natural keys in databases as the systems grow more complex and interdependent.

Natural or Surrogate

Let’s start by what we mean by natural. It’s not trivial to define this. In today’s world of APIs, someone’s surrogate key is another’s natural key. Wikipedia defines natural keys as “a type of unique key in a database formed of attributes that exist and are used in the external world outside the database”. This makes it clear that the keys we get from APIs are our natural keys. But how about the ones generated by us to be used in the external world?

When applications expose the keys on the URLs and APIs, others start relying on them. This is where our choices become important. When all those different applications generate their own keys instead of using the keys they got from other places, life becomes difficult for no reason.

Let’s elaborate with an example corporate database where employees are identified by their usernames and the departments with their domains. So our data would look like this:

| department        | username  | job                     |
| ----------------- | --------- | ----------------------- |
| sysadm.corp-x.com | hasegeli  | Database Administrator  |
| sysadm.corp-x.com | john      | System Administrator    |
| dep1.corp-x.com   | jane      | Developer               |

When we design this using surrogate keys, it’d look like this:

CREATE TABLE departments (
  id int NOT NULL GENERATED ALWAYS AS IDENTITY,
  domain text NOT NULL,

  PRIMARY KEY (id),
  UNIQUE (domain)
);

CREATE TABLE employees (
  id int NOT NULL GENERATED ALWAYS AS IDENTITY,
  username text NOT NULL,

  PRIMARY KEY (id),
  UNIQUE (username)
);

CREATE TABLE department_employees (
  id int NOT NULL GENERATED ALWAYS AS IDENTITY,
  department_id int NOT NULL,
  employee_id int NOT NULL,
  job text NOT NULL,

  PRIMARY KEY (id),
  UNIQUE (department_id, employee_id),
  FOREIGN KEY (department_id) REFERENCES departments (id),
  FOREIGN KEY (employee_id) REFERENCES employees (id)
);

Then, we would have to make a query joining them all together:

SELECT d.domain, e.username, de.job
FROM department_employees de
JOIN departments d ON de.department_id = d.id
JOIN employees e ON de.employee_id = e.id;

Instead, we could just have it as we need it on a single table:

CREATE TABLE department_employees (
  domain text NOT NULL,
  username text NOT NULL,
  job text NOT NULL,

  PRIMARY KEY (domain, username)
);

And, we could just query it:

SELECT domain, username, job
FROM department_employees;

Good Examples of Natural Keys

A primary key must be always available (NOT NULL) and unique. It should also be compact and easy to represent. Here are some good examples of natural keys:

Keys acquired from various APIs
Usernames
Domains
Email addresses
ISO country codes
IATA airport codes
Dates

I think easy representation matters for unchanged use among many systems. For that reason, I don’t think names with spaces, symbols, or non-ASCII Unicode letters are good keys.

UUIDs and hashes/digests like MD5 and SHA are also popular as primary keys. I don’t think they are a good choice as surrogate keys. Auto-incrementing IDs are easier to use, generate, store, predict — although, when they are defined in the external world and we have to store them, there’s nothing wrong with using them as the primary keys.

Another consideration when selecting keys is their stability. Things that can change are not good keys. This is not only because it is more work to change the keys everywhere, but because different parts of the system cannot rely on them.

The context matters to evaluate a key’s stability. For example, a phone number is not a good key for the users table, because users may change their phone number. However, it is perfect for the phone numbers table.

Also, I don’t think we should discard some useful keys just because they can change in an edge case. It is a good trade-off to allow the rare change rather than to complicate the system with a surrogate key.

Enterprise Architecture

I don’t know what you understand from the word “enterprise” but I have heard people advocating streamlined, unpractical solutions in the name of “enterprise” for a long time. I have seen people using layers of layers in complex frameworks and ORMs with caches and buffers, managers and supervisors, factories and workers, locks and semaphores, queues and consumers, all in the name of “enterprise architecture”.

Many of those frameworks have a rule like requiring all database tables to have a single-column auto-incremented primary key named “id”. Such a rule is easy to enforce and follow but it’s limiting. It would not be practical, maybe not even possible to use natural keys as the primary keys within such framework.

Even though I am writing this post to support using natural keys, strong conventions built around surrogate keys do certainly help developing complex systems in the “enterprise” environments. They help to keep systems consistent when the developers are more motivated to write code than to read what’s there, while working until they get a better offer from another company.

Performance Consideration

Performance is often an argument in favour of surrogate keys. Surrogate keys can be small auto-incrementing integers which would keep the indexes small, next branches easy to predict, so well performing.

Surrogate keys would take a bit of extra space since otherwise we don’t need to have them at all, but the space savings of using them as foreign keys would easily outweigh this. However, using surrogate keys may require the queries to have more tables joined together.

Some relational databases maintain the tables indexed by the primary key. Such systems are more affected by the size of the primary key. On PostgreSQL all indexes are secondary including the primary key, therefore the primary key is not any more important than the other indexes on the table.

Composite Keys

Natural keys are often unique together with other keys. This forces the database designer to use multiple columns as keys. Such design has many advantages, which I may cover in future blog posts.

Using CTEs to do a binary search of large tables with non-indexed correlated data

2020-10-02T00:00:00+00:00

Photo by Colin Knowles, used under CC BY-SA 2.0.

Query optimization can take different forms depending on the data represented and the required needs. In a recent case, we had a large table that we had to query for some non-indexed criteria. This table was on an appliance that we were unable to modify, so we had to find a way to query efficiently without indexes that would have made it easier.

The straightforward approach for this query was something along these lines:

SELECT accounts.*
FROM accounts
JOIN logs
    ON accounts.id = logs.account_id
WHERE
    logs.created_at BETWEEN $1 - 'interval 1 minute' AND $1 + 'interval 1 minute' AND
    logs.field1 = $2 AND
    logs.field2 = $3
FETCH FIRST ROW ONLY

Unfortunately, none of the fields involved in this query were indexed, nor could they be, due to our access level on this database system. This lack of indexes means that our query against those fields would end up doing a sequential scan of the whole table which made things unacceptably slow. This specific table held time-series data with ~ 100k records per 1 minute period over a period of several weeks, which meant we were dealing with a lot of data.

While we could not create any additional indexes to help us with this query, we could use some specific properties to help us:

There was a primary key field, id, which was unique and monotonic, i.e., always increasing in value.
This table was append only; no updates or deletes, so once data existed in the table it was always the same.
The field we actually care about (created_at) also ends up being monotonic: subsequent records would always have the same or later values.
Since records were created sequentially and the id field was always increasing, id values and created_at fields would be together be generally monotonic; this means there is an indexed field which we can use as a surrogate stand-in for the target field that we want to treat as an index. While due to the nature of logging ingest it is possible the created_at and id values are not strictly monotonic (for instance if there are multiple logging records being created by separate ingest processes, of which ids get assigned in chunks), for our purposes this was close enough, since we were looking around in a time window which was wider than we were actually expecting the message to appear.
Since we are looking for fields matching a specific window of time, we can substitute the non-indexed clause created_at BETWEEN <timestamp_min> AND <timestamp_max> with an expression matching the indexed statement id BETWEEN <id of first id gt timestamp_min> AND <id of first id gt timestamp_max> to get the same effective approach.
In order to find the specific id fields which match the created_at time ranges we are interested in, we would need to find the first id value which matched the criteria created_at > 'timestamp'::timestamp, as all subsequent id values would match that condition as well. This would effectively require a binary search of the table to check which records match the criteria, and return the smallest id value for which this criteria held.

So now that we have identified how we can use an indexed surrogate key to substitute for the non-indexed expression, we need to figure out how to calculate the ranges in question.

Based on some recent discoveries about optimizing simple-looking but poorly-performing queries using more complicated queries¹, I had instincts that this could be solved with a WITH RECURSIVE Common Table Expression. After toying around for a while and not coming up with the exact solution, I ended up visiting the #postgresql channel on FreeNode IRC Network. There, I presented the problem and got some interested responses, as this is the exact kind of question that database ~~nerds~~ experts love². The solution that user xocolatl (Vik Fearing) came up with for a basic binary search is the basis for the rest of my solution:

CREATE TABLE test_table (id integer PRIMARY KEY, ts timestamptz);

INSERT INTO test_table
SELECT g, date 'today' + interval '1s' * g
FROM generate_series(1, 1000000) AS g;

WITH RECURSIVE
search (min, max, middle, level) AS (
    SELECT min(id), max(id), (min(id)+max(id))/2, 0
    FROM test_table
        UNION ALL
    SELECT v.min, v.max, (v.min + v.max)/2, s.level+1
    FROM search AS s
    CROSS JOIN LATERAL (
        SELECT *
        FROM test_table AS e
        WHERE e.id >= s.middle
        ORDER BY e.id
        FETCH FIRST ROW ONLY
    ) AS e
    CROSS JOIN LATERAL (VALUES (
        CASE WHEN e.ts < now() THEN e.id ELSE s.min END,
        CASE WHEN e.ts < now() THEN s.max ELSE e.id END
    )) AS v (min, max)
    WHERE (v.min + v.max)/2 NOT IN (v.min, v.max)
)
SELECT *
FROM search AS s
JOIN test_table AS e ON e.id = s.middle
ORDER BY s.level DESC
FETCH FIRST ROW ONLY;

As expected, the solution involved a WITH RECURSIVE CTE.

The basic explanation here is that the search expression first starts with the min, max, and middle (mean) values of id for the table (the initialization expression), then iteratively adds additional rows to the results depending on if the table row with id >= middle matches our specific test criteria, then continues until one of the boundaries of the region is hit. (Since we are using integer division in our terminal expression (v.min + v.max)/2 NOT IN (v.min, v.max) we are guaranteed to hit one of the boundary conditions eventually in our search.)

A few other things worthy of note:

This approach uses the check e.ts < now() as the condition we are testing, which means in this specific example, the answer to this “closest id” query would actually change depend on when you run this query relative to when the initial test data was populated. However, we can replace that condition with whatever condition we want to use to test for our surrogate non-indexed data.
This approach will work whether or not there are gaps in the sequence. In order to properly handle gaps, we are selecting the first row with id >= middle ... FETCH FIRST ROW ONLY rather than just selecting id = middle, which you could do in a gapless sequence.
In addition to not caring about gaps, we are also not trying to make sure this is using a balanced binary search; it would be not worth the computing effort to find the middlest existing row in an index, as we’d need to know the number of rows in the segment we’re searching. Since in PostgreSQL this would entail a COUNT(*) from a subselect, this would be quite slow and not worth the trouble.

Considering my specific use case was trying to limit the records considered based on two created_at values I needed to calculate a search_min and search_max to find the start/end ids for each side in the interval.

Given this, I just modified the CTE to calculate those and add the additional conditions we were wanting to consider. I also had to turn the result query from a join against the found id value to a range; the final query is as follows:

WITH RECURSIVE
search_min (min, max, middle, level) AS (
    SELECT min(id), max(id), (min(id)+max(id))/2, 0
    FROM logs
        UNION ALL
    SELECT v.min, v.max, (v.min + v.max)/2, s.level+1
    FROM search_min AS s
    CROSS JOIN LATERAL (
        SELECT *
        FROM logs AS e
        WHERE e.id >= s.middle
        ORDER BY e.id
        FETCH FIRST ROW ONLY
    ) AS e
    CROSS JOIN LATERAL (VALUES (
        CASE WHEN extract(epoch FROM e.created_at)::integer < $1 THEN e.id ELSE s.min END,
        CASE WHEN extract(epoch FROM e.created_at)::integer < $1 THEN s.max ELSE e.id END
    )) AS v (min, max)
    WHERE (v.min + v.max)/2 NOT IN (v.min, v.max)
),
search_max (min, max, middle, level) AS (
    SELECT min(id), max(id), (min(id)+max(id))/2, 0
    FROM logs
        UNION ALL
    SELECT v.min, v.max, (v.min + v.max)/2, s.level+1
    FROM search_max AS s
    CROSS JOIN LATERAL (
        SELECT *
        FROM logs AS e
        WHERE e.id >= s.middle
        ORDER BY e.id
        FETCH FIRST ROW ONLY
    ) AS e
    CROSS JOIN LATERAL (VALUES (
        CASE WHEN extract(epoch FROM e.created_at)::integer < $2 THEN e.id ELSE s.min END,
        CASE WHEN extract(epoch FROM e.created_at)::integer < $2 THEN s.max ELSE e.id END
    )) AS v (min, max)
    WHERE (v.min + v.max)/2 NOT IN (v.min, v.max)
)
SELECT accounts.*
FROM accounts
JOIN logs
    ON logs.account_id = accounts.id
WHERE
    logs.field1 = $3 AND
    logs.field2 = $4 AND
    logs.id >= (SELECT middle FROM search_min ORDER BY level DESC FETCH FIRST ROW ONLY) AND
    logs.id <= (SELECT middle FROM search_max ORDER BY level DESC FETCH FIRST ROW ONLY)
ORDER BY logs.id
FETCH FIRST ROW ONLY

The final results were drastically improved. The initial query went from timing out in the webservice in question to returning results in a fraction of a second. Clearly this technique, while not as useful as directly indexing data we care about, can come in handy in some circumstances.

Another tool for the toolbag!

1 (back): Improving max() performance in PostgreSQL: GROUP BY vs. CTE

2 (back): See nerdsniping

How to Migrate from Microsoft SQL Server to PostgreSQL

2019-01-23T00:00:00+00:00

One of our clients had a Java-based application stack on Linux that connected to a pretty old version of SQL Server on Windows. We wanted to migrate the entire system to a more consistent unified stack that developers are efficient with, and that is current so it receives regular updates.

We decided to migrate the database from SQL Server to PostgreSQL on Linux because porting the database, while not entirely quick or simple, was still much simpler than porting the app to .NET/C# would have been. Rewriting the application would have taken far longer, been much riskier to the business, and cost a lot more.

I experimented with a few approaches to the migration and decided to go with the process of schema migration and then the data migration approach which is referred to on the Postgres wiki. Let’s walk through the process of migration step by step.

Schema Migration

A schema of the SQL Server database tables and views needs to be exported to perform schema conversion. The following steps will show you how to export the schema.

Export SQL Server Database Schema

In SQL Management Studio, right click on the database and select Tasks → Generate Scripts.

Choose “Select specific database objects” and check only your application schema Tables (untick dbo schema objects and others if any).

Ensure that “Types of data to script” in advanced options is set to “Schema only”.

Review and save the database tables schema file tables.sql. Use WinSCP and public key auth to transfer tables.sql to the Linux server.

Convert Schema from SQL Server to Postgres

sqlserver2pgsql is a good migration tool written in Perl to convert SQL Server schemas to Postgres schemas. Clone it from GitHub to your database server and execute the following commands to convert the tables schema:

$ git clone https://github.com/dalibo/sqlserver2pgsql.git
$ cd sqlserver2pgsql
$ perl sqlserver2pgsql.pl -f tables.sql -b tables-before.sql -a tables-after.sql -u tables-unsure.sql

The converted schema will be available in tables-before.sql and statements to create constraints will be in tables-after.sql to execute after data migration. Just review tables-unsure.sql and do what’s needed if there are any tables not converted by the tool. If you want to change any schema names in Postgres, you can rename them now in the SQL file, e.g.:

$ sed -i ‘s/sql_server_schema/public/g’ *.sql

Setup Postgres Database:

Hopefully, you have your Postgres database ready in your server. If not, install the latest version of Postgres on your server. Then create a Postgres user and database with permissions granted to the user.

CREATE USER <user_name> WITH
  LOGIN
  NOSUPERUSER
  INHERIT
  NOCREATEDB
  NOCREATEROLE
  NOREPLICATION;

CREATE DATABASE <database_name>
    WITH
    OWNER = <user_name>
    ENCODING = 'UTF8'
    LC_COLLATE = 'en_US.UTF-8'
    LC_CTYPE = 'en_US.UTF-8'
    TABLESPACE = pg_default
    CONNECTION LIMIT = -1;

Once the database and user account are ready, load the converted tables-before.sql script into your database to create tables. Then we can move forward with data migration.

psql -U user_name -p 5432 -h localhost -d database_name -f tables-before.sql
#	or
database_name=# \i /path/to/tables-before.sql

Data Migration

Data migration through data dump and restore makes the process cumbersome, with a huge manual process, data type mismatches, date formats, etc. It is wise to use existing stable systems instead of investing more time into it.

Pentaho Data Integration

Pentaho offers various stable data-centric products. Pentaho Data Integration (PDI) is an ETL tool which provides great support for migrating data between different databases without manual intervention. The community edition of PDI is good enough to perform our task here. It needs to establish a connection to both the source and destination databases. Then it will do the rest of work on migrating data from SQL server to Postgres database by executing a PDI job.

Download Pentaho Data Integration Community Edition and extract the tarball in your local environment. See:

Run spoon.sh to open the GUI application in your local environment.

$ cd pentaho/data-integration
$ ./spoon.sh

Create connections to both SQL Server and Postgres databases in PDI.

1. Create a New Job.

File → New → Job

2. Create Source Database Connection.

Click View in left sidebar → Right Click ‘Database Connections’ → Choose New → Provide SQL Server connection details

3. Create Destination Database Connection.

Click View in left sidebar → Right Click ‘Database Connections’ → Choose New → Provide Postgres connection details

4. From Wizard menu, choose Copy Tables Wizard.

Tools → Wizard → Copy Tables

5. Choose Source and Destination databases.

6. Select the list of tables to migrate.

7. Move forward to choose the path for job and transformation files.

8. The transformations were created to copy data from source to destination database.

9. Copy the Pentaho Data Integration tarball and job with transformations to the Postgres server to avoid network latency on data migration.

$ tar -cvzf pdi-migration-job.tar.gz pdi-migration-job/
$ rsync -azP data-integration.zip pdi-migration-job.tar.gz user@server:

10. Check Postgres server to SQL Server database access (based on architecture design at your end).

Establish SSH tunneling to connect to SQL Server through the application server.

$ cat config
host tunnel-server
    Hostname <application_server>
    Port 22
    User root
    LocalForward 1433 <ipaddress>:1433

$ ssh tunnel-server
$ telnet localhost 1433

PDI Job Execution

Execute PDI job using Pentaho kitchen utility on the database server. Add a kettle configuration to avoid Pentaho considering empty values as NULL values which affects NOT NULL constraints.

$ cat .kettle/kettle.properties
KETTLE_EMPTY_STRING_DIFFERS_FROM_NULL=Y

Confirm IP address for database servers’ connections and execute the job using kitchen.sh, a command-line utility to execute PDI jobs.

$ ./kitchen.sh -file="/path/to/pdi-migration-job.kjb" -level=Basic | tee pdi-migration-job.log

The data migration process took around 30 minutes to copy 10GB of data from SQL Server to Postgres database over network.

Output:

2018/08/23 11:16:52 - SQLServerToPostgres20180823 - Job execution finished
2018/08/23 11:16:52 - Kitchen - Finished!
2018/08/23 11:16:52 - Kitchen - Start=2018/08/23 10:51:00.701, Stop=2018/08/23 11:16:52.951
2018/08/23 11:16:52 - Kitchen - Processing ended after 25 minutes and 52 seconds (1552 seconds total).

We ran into a space problem while copying a large table and handled it by reducing the number of rows size in the corresponding table transformation file. Here’s a discussion of the problem.

Fix:

vi **table_name**.ktr
    <size_rowset>100</size_rowset>

sed 's/<size_rowset>10000<\/size_rowset>/<size_rowset>100<\/size_rowset>/' *table_name*

Once the data migration is completed, execute the tables-after.sql schema-migration script to apply constraints.

Migrate Views

Follow the same steps to convert the views.sql schema. Right click on the database and click on Tasks → Generate Script → Export only views.

$ perl sqlserver2pgsql.pl -f views.sql -b views-before.sql -a views-after.sql -u views-unsure.sql

Migrate Functions

The functions migration requires skill and syntax awareness of both SQL Server and Postgres. The functions need to validated and tested properly after rewriting for Postgres. Place all rewritten functions into postgres-function.sql and load into the database as part of the migration process.

Most Common Errors

ERROR: syntax error at or near "["
Remove SQL Server square brackets quoting around column and table names.
SELECT [user_id] → SELECT user_id
ERROR: operator does not exist: character varying = integer
Type casting on comparison
AND column_name in (1, -1) → AND column_name in (‘1’, ‘-1’)
ERROR: syntax error at or near ","
Datatype conversion
convert(varchar(10),column_name) → column_name::text
ERROR: column "user_id" of relation "user" does not exist
No double quotes on mixed-case column names (unless used everywhere)
INSERT INTO user (“user_id”) VALUES (1) → INSERT INTO user (user_id) VALUES (1)
ERROR: column "mm" does not exist
DATEADD → INTERVAL
DATEADD(mm,-6,GETDATE()) → CURRENT_DATE - INTERVAL ‘6 months’
ERROR: column "varchar" does not exist
Convert date to USA format
convert(varchar, tc_agreed_dt, 101) → to_char(CURRENT_TIMESTAMP, ‘MM/DD/YYYY’)

Result

After migration, we performed extensive testing on the application and optimized the queries in various modules. We also performed a benchmark test on speed and performance between the new and old systems. The migration paid off with a promising result of 40% average gain on overall performance of the application. That reflects the newer server hardware as well as the query optimization work, not solely the move to Postgres.

The entire migration was automated with a script to test multiple times and reduce manual intervention to perform error-free process. The automated script is available to customize based on your scenario: sqlserver-postgres-migration-automation.sh

Building Rasters in PostGIS

2018-09-12T00:00:00+00:00

In a past blog post I described a method I’d used to digest raw statistics from the Mexican government statistics body, INEGI, quantifying the relative educational level of residents in Mexico City. In the post, I divided these data into squares geographically, and created a KML visualization consisting of polygons, where each polygon’s color and height reflected the educational level of residents in the corresponding area. Where the original data proved slow to render and difficult to interpret, the result after processing was visually appealing, intuitively meaningful, and considerably more performant. Plus I got to revisit a favorite TV show, examine SQL’s Common Table Expressions, and demonstrate their use in building complex queries.

But the post left a few loose ends untied. For instance, the blog post built the visualization using just one large query. Although its CTE-based design rendered it fairly readable, the query remained far too complex, and far too slow, at least for general use. Doing everything in one query makes for a sometimes enjoyable mental exercise, but it also means the query has to start from zero, every time it runs, so iterative development bogs down quickly as the query must recalculate at every new iteration all the expensive stuff it already got worked out in previous runs. Further, the query in that post calculated its grid coordinates all on its own; there’s an easier method I promised to introduce, which I’d like to demonstrate here.

Rasters

GIS databases contain information of (at least) two different types: vector data, and raster data. Vector data describes points, connected with lines and curves, and various objects derived from them, such as polygons and linestrings. Rasters, on the other hand, represent a two dimensional array of pixels, where each pixel corresponds to a certain geographic area. Each pixel in the raster will have two sets of associated coordinates: first, latitude and longitude values describing the geographic square this pixel represents, and second, the pixel coordinates within the raster, which in PostGIS begin with (1,1) at the origin. Rasters organize their data in “bands”, collections of boolean or numeric values, with at most one value per pixel. One band might represent the elevation above sea level for each pixel, or the surface temperature. Aerial imagery is often delivered as rasters: a full-color image might have one band for the red channel, another for green, and a third for blue. Bands may also contain no value for certain pixels, and we’ll make use of this ability later to make some parts of our final image transparent.

The Data

The INEGI data I introduced in my previous blog post contains points representing every school in Mexico, and continuing on the theme of education, I thought it would be an interesting exercise to make an image of the country, where the color varies depending on the concentration of schools in the area. This resembles my previous work in many ways: I’m dividing the area under consideration into small parts, generating a value for each of those parts, and representing the result visually. Here, though, rather than drawing and extruding polygons, I’ll simply color a raster pixel. This simpler rendering, where the display system must simply project a static image, rather than an array of polygons, means I can support a much higher resolution grid than in my previous blog post, without overtaxing display hardware.

Creating the Image

Having learned my lesson from the work with Mexico City data, I elected to create this image with multiple queries. In the end, I wrapped these queries in a Perl script, building sufficient intelligence therein to recreate any bits of the database I might have simply dropped, as I developed the script, and to skip reprocessing any steps for which it finds current results lying around.

PostGIS supports a data type called raster, and anticipating that I might want multiple rasters in my database each using a variety of settings and script modifications, I created a table to store these rasters and some extra data about them.

CREATE TABLE rasters (id SERIAL PRIMARY KEY, pixels INTEGER, r RASTER, comment TEXT);

The pixels field holds the number of pixels along one edge of the raster, which the script presumes is a square. This is redundant, as I can extract that number from the raster itself, but having it stored here made things easier for my purposes.

To calculate a raster, I need to find a value for each of its pixels, or to confirm that the pixel should have no value, and for that I wanted another table. This table contains both geographic and Euclidean coordinates for each pixel in each raster, the id of the associated entry in the rasters table, a field for the number of schools I’ve counted in this square, fields for the RGB color assigned to the pixel, and a boolean flag telling me whether or not I’ve finished calculating everything for this pixel or not.

CREATE TABLE polys (
    x INTEGER,
    y INTEGER,
    geom GEOMETRY,
    count INTEGER,
    r_val INTEGER,
    b_val INTEGER,
    g_val INTEGER,
    processed BOOLEAN DEFAULT 'f',
    raster_id INTEGER NOT NULL
);

The script’s first step is to create an empty raster, which it does with PostGIS’s ST_MakeEmptyRaster() function, and to add four bands to it, one each for the pixels’ red, green, blue, and alpha values. The data type for each band is given as 8BUI, PostGIS parlance for an 8-bit unsigned integer. The various color bands default to NULL values, and the alpha band to zeroes. The code looks something like this:

INSERT INTO rasters (pixels, r, comment)
SELECT
    ST_AddBand(
    ST_AddBand(
    ST_AddBand(
    ST_AddBand(
        ST_MakeEmptyRaster(
            numpixels,  -- The number of pixels along one edge of the raster
            numpixels,
            st_xmin,    -- latitude and longitude coordinates of the corner of the raster
            st_ymax,
            (st_xmax - st_xmin)/numpixels::float,   -- width and height of the raster
            -(st_ymax - st_ymin)/numpixels::float,
            0,
            0,
            4326        -- We'll use the WGS 84 coordinate system
        ),
        1, '8BUI'::TEXT, NULL::DOUBLE PRECISION, NULL::DOUBLE PRECISION),
        2, '8BUI'::TEXT, NULL::DOUBLE PRECISION, NULL::DOUBLE PRECISION),
        3, '8BUI'::TEXT, NULL::DOUBLE PRECISION, NULL::DOUBLE PRECISION),
        4, '8BUI'::TEXT, 0, 0
    ) AS r
FROM limits, params;

The query refers to the limits and params tables (actually, they’re table expressions from elsewhere in the query from which this snippet is extracted). These tables provide values for numpixels, the number of pixels along one edge of the raster, provided by the user on the script’s command line, and for st_xmin, st_xmax, st_ymin, and st_ymax, which are latitude and longitude values describing a bounding box around the Mexican territory.

Before we can do too much experimentation and query development, we need a way to inspect our results, by rendering the raster from the database to an actual image file. This is one place where having a Perl wrapper script comes in handy, because doing this in pure SQL is actually something of a pain..

print "Creating image\n";
my $res = $dbh->selectall_arrayref(qq{
    SELECT ST_AsPng(r, ARRAY[1,2,3,4]::INT[])
    FROM rasters WHERE id = $id
});

my $png = $res->[0][0];

open (my $fh, '>', $filename) || die "Couldn't open handle to $filename: $!";
binmode $fh;

print $fh $png;

close $fh;

This uses ST_AsPng to create an image file under whatever path is stored in $filename, and it’s helpful for reviewing incremental results as the script develops, and of course for rendering the final product. Here we’ve told it to use band 1 for the red values, band 2 for green, 3 for blue, and 4 for alpha. The alpha channel is important: eventually we’ll want to feed the result to Google Earth, and the transparency will mean we can see the ocean surrounding Mexico, without any unimportant raster data getting int he way.

Without any values in the bands, there’s nothing to look at yet—so let’s make some data.

Finding Polygons

We need to find the polygon corresponding to each pixel in the raster, and PostGIS gives us two ways to do it. The first is ST_PixelAsPolygons(), which returns a set of values, including the geographic rectangle the pixel represents, called geom, and its x and y coordinates. The other function, ST_PixelAsCentroids(), also accepts a single raster as its sole argument and returns a set of similar results, except that instead of a geographic rectangle, it returns only the point at the geographic center of the pixel. I used both as I experimented to make this visualization. The simplicity of these functions makes them far superior to the manual calculation I did in the previous blog post I’ve referred to.

This query fills the polys table with polygons for a particular raster:

INSERT INTO polys (x, y, geom, raster_id)
SELECT (ST_PixelAsPolygons(r)).*, $id FROM rasters
-- Having code for both functions here made it easy to switch back and forth
-- SELECT (ST_PixelAsCentroids(r)).*, $id FROM rasters
WHERE id = $id

One step that proved critical for decent performance was to filter the set of polygons so that I calculate values only for those pixels that actually intersect some part of Mexico. PostGIS rasters are regular rectangles, so a substantial portion of this raster lies over open ocean. Since we’re counting schools of people, and not of fish, we can skip all those ocean pixels, and though filtering them out takes considerable processing, we earn back that cost and more in savings later when calculating schools within each pixel. This modified version of the query above does that filtering.

WITH polygons AS (
    SELECT (ST_PixelAsPolygons(r)).* FROM rasters
    -- Having code for both functions here made it easy to switch back and forth
    -- SELECT (ST_PixelAsCentroids(r)).* FROM rasters
    WHERE id = $id
),
filtered_polygons AS (
    SELECT x, y, p.geom
    FROM polygons p
        JOIN mexico m
            ON ST_Intersects(m.geom, p.geom)
    GROUP BY 1, 2, 3
)
INSERT INTO polys (x, y, geom, raster_id)
SELECT x, y, geom, $id FROM filtered_polygons

This query refers to a table called mexico, which contains geographic representations of each of Mexico’s 31 states and the region of Mexico City, which I didn’t previously know wasn’t a state. It’s this table that gave us st_xmin, st_xmax, etc., when we made the original raster, and now we use it to filter out all pixels from our raster that don’t intersect some portion of Mexico.

At this point it would be nice to see a visual representation of our raster, to make sure the filtering worked the way we intended. We still haven’t filled the raster with any data, but we could fill it with dummy values and render the image, just to get a look at our progress. This raster modification we achieve with the ST_SetValues() function, which accepts a raster as input and returns the raster, with updated values in one of its bands. The function comes in several forms; the one we’ll use accepts an array of geomval objects, each of which contains a PostGIS geometry, and a single double precision value. Internally, the function finds the pixel or pixels corresponding to the geometry objects provided, and sets those pixels’ values in the given band accordingly. We make sure to provide POINT values here— the center point for each pixel’s corresponding polygon—gc as ST_SetValues is considerably faster that way than when given POLYGON values themselves.

-- Update the original raster
UPDATE rasters
    SET r = ST_SetValues(ST_SetValues(ST_SetValues(ST_SetValues(r, 1, r_geomvalset), 2, g_geomvalset), 3, b_geomvalset), 4, a_geomvalset)
FROM (
    SELECT
        ARRAY_AGG((geom, 255)::GEOMVAL) AS r_geomvalset,
        ARRAY_AGG((geom, 255)::GEOMVAL) AS g_geomvalset,
        ARRAY_AGG((geom, 255)::GEOMVAL) AS b_geomvalset,
        ARRAY_AGG((ST_Centroid(geom), 255)::GEOMVAL) AS a_geomvalset
    FROM polys
    WHERE raster_id = $id
) foo
WHERE id = $id

For a raster with many pixels, this takes a bit of time to accomplish. For my initial testing, I used rasters with only 50 pixels per side, which my wimpy laptop can churn out fairly rapidly. Here’s the first visual confirmation that we’re headed down the right path. Because of the low resolution, the image is small, but it’s obvious that our result follows the basic shape of Mexico, which means our polygon generation and filtering has worked.

I mentioned transparency above. Note that in the default format of this blog, it would be impossible to tell that the transparency in this image was correct, so I’ve post-processed all the rendered raster images in this article to replace the transparency with a gray checkerboard pattern.

Counting schools

Now all that remains is to count the schools in each pixel, and assign corresponding color values.

WITH polygons AS (
    SELECT x, y, geom FROM polys
    WHERE
        processed = 'f'
        AND raster_id = $id
    LIMIT 100
),
schoolcounts AS (
    SELECT x AS sx, y AS sy, COUNT(*) AS schoolcount
    FROM polygons p
        JOIN sip ON (ST_Within(sip.geom, p.geom) AND sip.geografico = 'Escuela')
    GROUP BY x, y, p.geom
)
UPDATE polys SET count = COALESCE(s.schoolcount, 0), processed = 't'
FROM schoolcounts s
    RIGHT JOIN polygons p
        ON (sx = p.x AND sy = p.y)
WHERE
    polys.x = p.x AND polys.y = p.y AND raster_id = $id

This query finally refers to the actual INEGI data set I started with, a table called sip, in which each record includes a POINT object and a label. We’re interested in labels marked “Escuela”, Spanish for “School”. I tried a few different techniques here, as shown by the two different JOIN variants in the schoolcounts portion of the query. The version shown here simply counts schools within each square pixel. Later I’ll try counting schools within a certain radius of the center of the pixel. The two give different results, of course; which one is more useful depends on the end goal for these data.

Note the use of the processed field, and the 100 item limit in the polygons section of the query. Together with some logic in the surrounding Perl, this allows me to process pixels in small batches, interrupt processing when necessary, and later, resume more or less where I left off.

Color assignment

After counting schools, I need to assign colors to the pixels. Initially I imagined this would be fairly easy, but in fact it took some detective work to get it right. My plan was to get a gradient of colors, divide the school counts into histogram-like buckets according to school density, and assign a color to each bucket. Building your own decent color gradient is surprisingly difficult, and I got mine from an online service. As with the last blog post, I figured a relatively small number of colors would be easiest. These I encoded as a PostgreSQL array, and I used PostgreSQL’s ntile window function to return the right bucket for each polygon. The results surprised me, shown here in a raster with 500 pixels on each side:

It’s easy to see in that image that some pixels are colored to represent schools, but where did all the vertical bands come from? Much of Mexico is very sparsely populated, so vast swaths of our pixels have no schools at all. This means that the pixels with a school count of zero fill most of the available buckets. Pixels with schools in them are so much in the minority, for any raster of sufficient resolution, as to place all non-zero pixels in a single bucket. So I modified my approach slightly: I made my histogram only from those pixels with non-zero school counts. That query is below.

WITH colors AS (
    SELECT ARRAY[
        ARRAY[x'd4'::INT, x'ea'::INT, x'd7'::INT],
        ARRAY[x'a9'::INT, x'd6'::INT, x'af'::INT],
        ARRAY[x'7e'::INT, x'c1'::INT, x'88'::INT],
        ARRAY[x'53'::INT, x'ad'::INT, x'60'::INT],
        ARRAY[x'29'::INT, x'99'::INT, x'39'::INT]] AS colors
),
p AS (
    SELECT x, y, geom, raster_id, count
    FROM polys WHERE raster_id = $id AND count IS NOT NULL AND count != 0
),
colorval AS (
    SELECT
        x, y, geom, raster_id,
        NTILE(ARRAY_LENGTH(colors, 1)) OVER (ORDER BY count ASC) AS ntile,
        colors[NTILE(ARRAY_LENGTH(colors, 1)) OVER (ORDER BY count ASC)][1] AS r,
        colors[NTILE(ARRAY_LENGTH(colors, 1)) OVER (ORDER BY count ASC)][2] AS g,
        colors[NTILE(ARRAY_LENGTH(colors, 1)) OVER (ORDER BY count ASC)][3] AS b,
        255 AS a
    FROM colors, p
    WHERE raster_id = $id
        AND count != 0
        AND count IS NOT NULL
)
UPDATE polys
    SET r_val = r, g_val = g, b_val = b
FROM colorval c
WHERE c.x = polys.x AND c.y = polys.y AND c.raster_id = polys.raster_id

A second, very simple query not included here found any pixels with zero schools and colored them white.

Packaging the result

The final bit of required scripting to generate a usable image is to use ST_SetValues again, in a somewhat more complex form than last time, creating an array of geomval objects and using them to update the original raster.

UPDATE rasters
    SET r = ST_SetValues(ST_SetValues(ST_SetValues(ST_SetValues(r, 1, r_geomvalset), 2, g_geomvalset), 3, b_geomvalset), 4, a_geomvalset)
FROM (
    SELECT
        ARRAY_AGG((ST_Centroid(geom), r_val)::GEOMVAL) FILTER (WHERE r_val IS NOT NULL) AS r_geomvalset,
        ARRAY_AGG((ST_Centroid(geom), g_val)::GEOMVAL) FILTER (WHERE g_val IS NOT NULL) AS g_geomvalset,
        ARRAY_AGG((ST_Centroid(geom), b_val)::GEOMVAL) FILTER (WHERE b_val IS NOT NULL) AS b_geomvalset,
        ARRAY_AGG((ST_Centroid(geom), 255)::GEOMVAL) AS a_geomvalset
    FROM polys
    WHERE raster_id = $id
) foo
WHERE id = $id

This gives me something like the image below, with 500 pixels on each edge of the raster:

This image does the job, but contains an awful lot of empty space. I mentioned above that instead of calculating the exact number of schools within a pixel’s polygon, I thought I might try calculating the number of schools within a specific distance of the pixel’s center; perhaps that technique would lead to a more visually appealing result. This is simple enough, using the ST_PixelAsCentroids() function to get the centroids of the pixels’ polygons, and then modifying the schoolcounts CTE as shown here. The key modification—in fact, the only modification from the earlier version shown above—is in the JOIN clause, which uses ST_Buffer to create a circle surrounding a point, in this case, with an arbitrarily chosen radius, and counts the number of schools found within that circle.

schoolcounts AS (
    SELECT x AS sx, y AS sy, COUNT(*) AS schoolcount
    FROM polygons p
        JOIN sip ON (ST_Within(sip.geom, ST_Buffer(p.geom, 0.4)) AND sip.geografico = 'Escuela')
    GROUP BY x, y, p.geom
)

The resulting image, again with 500x500 pixel resolution, shows evidence of its circle-based counting heritage.

Obviously at this point, a person could mess with any number of variables to get all sorts of results. But for me it was enough to embed this as a GroundOverlay object within a simple KML file, to see the result in Google Earth:

Systematic Query Building with Common Table Expressions

2018-06-12T00:00:00+00:00

The first time I got paid for doing PostgreSQL work on the side, I spent most of the proceeds on the mortgage (boring, I know), but I did get myself one little treat: a boxed set of DVDs from a favorite old television show. They became part of my evening ritual, watching an episode while cleaning the kitchen before bed. The show features three military draftees, one of whom, Frank, is universally disliked. In one episode, we learn that Frank has been unexpectedly transferred away, leaving his two roommates the unenviable responsibility of collecting Frank’s belongings and sending them to his new assignment. After some grumbling, they settle into the job, and one of them picks a pair of shorts off the clothesline, saying, “One pair of shorts, perfect condition: mine,” and he throws the shorts onto his own bed. Picking up another pair, he says, “One pair of shorts. Holes, buttons missing: Frank’s.”

The other starts on the socks: “One pair of socks, perfect condition: mine. One pair socks, holes: Frank’s. You know, this is going to be a lot easier than I thought.”

“A matter of having a system,” responds the first.

I find most things go better when I have a system, as a recent query writing task made clear. It involved data from the Instituto Nacional de Estadística y Geografía, or INEGI, an organization of the Mexican government tasked with collecting and managing country-wide statistics and geographical information. The data set contained the geographic outline of each city block in Mexico City, along with demographic and statistical data for each block: total population, a numeric score representing average educational level, how much of the block had sidewalks and landscaping, whether the homes had access to the municipal sewer and water systems, etc. We wanted to display the data on a Liquid Galaxy in some meaningful way, so I loaded it all in a PostGIS database and built a simple visualization showing each city block as a polygon extruded from the earth, with the height and color of the polygon proportional to the educational score for that block, compared to the city-wide average.

It wasn’t entirely a surprise that with so many polygons, rendering performance suffered a bit, but most of all the display was just plain confusing. This image is just one of Mexico City’s 16 boroughs.

With so much going on in the image, it’s difficult for the user to extract any meaningful information. So I turned to a technique we’d used in the past: reprocess the geographical area into grid squares, extrapolate the statistic of interest over the area of the square, and plot it again as a set of squares. The result is essentially a three dimensional heat map, much easier to comprehend, and, incidentally, to render.

As with most programming tasks, it’s helpful to have a system, so I started by sketching out how exactly to produce the desired result. I planned to overlay the features in the data set with a grid, and then for each square in the grid, find all intersecting city blocks, a number representing the educational level of residents of that block, and what percentage of the block’s total area intersects each grid square. From that information I can extrapolate that block’s contribution to the grid square’s total educational score. The precise derivation of the score isn’t important for our purposes here; suffice it to say it’s a numeric value with no particular associated unit, whose value lies in its relation to the scores of other blocks. Residents of a block with a high score are, on average, probably more educated than residents of a lower-scoring block. For this query, a block with an average educational level of, say, 100 “points”, would contribute all 100 points to a grid square if the entire block lay within that square, 60 points if only 60% of it was within the square, and so on. In the end, I should be able to add up all the scores for each grid square, rank them against all other grid squares, and produce a visualization.

I suffer from the decidedly masochistic habit of doing whatever I can in a single query, while maintaining a desire for readable and maintainable code. Cramming everything into one query isn’t always a good technique, as I hope to illustrate in a future blog post, but it worked well enough in this instance, and provides a good example I wanted to share, of one way to use Common Table Expressions. They do for SQL what subroutines do for other languages, separating tasks into distinct units. A common table expression looks like this:

WITH alias AS (
    SELECT something FROM wherever
),
another_alias AS (
    SELECT another_thing FROM alias LEFT JOIN something_else
)
SELECT an, assortment, of, fields
FROM another_alias;

As shown by this example query, the WITH keyword precedes a list of named, parenthesized queries, each of which functions throughout the life of the query as though it were a full-fledged table. These pseudo-tables are called Common Table Expressions, and they allow me to make one table for each distinct function in what will prove to be a fairly complicated query.

Let’s go through the elements of our new query piece by piece. First, I want to store the results of the query, so I can create different visualizations without recalculating everything. So I’ll make this query create a new table filled with its results. In this case, I called the table grid_mza_vals.

Now, for my first CTE. This query involves a few user-selected parameters, and I want an easy way to adjust these settings as I experiment to get the best results. I’ll want to fiddle with the number of grid squares in the overall result, as well as the coefficients used later on to calculate the height of each polygon. So my first CTE is called simply params, and returns a single row, composed of these parameters.

CREATE TABLE grid_mza_vals AS
WITH params AS (
    SELECT
        25 AS numsq,
        3800 AS alt_bias,
        500 AS alt_percfactor
),

The numsq value represents the number of grid squares along one edge of my overall grid; we’ll discuss the other values later. I’ve chosen a relatively small number of total grid squares for faster processing while building the rest of the query. I can make it more detailed, if I want, after everything else works.

The next thing I want is a sequence of numbers from 1 to numsq:

range AS (
    SELECT GENERATE_SERIES(0, numsq - 1) AS rng
    FROM params
),

Now I can join the range CTE with itself, to get the coordinates for each grid square:

gridix AS (
    SELECT
        x.rng AS x_ix,
        y.rng AS y_ix
    FROM range x, range y
),

Occasionally I like to check my progress, running whatever bits of the query I’ve already written to review its behavior. CTEs make this convenient, because I can adjust the final clause of the query to select data from whichever of the CTEs I’m currently interested in. Here’s the query thus far:

inegi=# WITH params AS (
    SELECT
        25 AS numsq,
        3800 AS alt_bias,
        500 AS alt_percfactor
),
range AS (
    SELECT GENERATE_SERIES(0, numsq - 1) AS rng
    FROM params
),
gridix AS (
    SELECT
        x.rng AS x_ix,
        y.rng AS y_ix
    FROM range x, range y
)
SELECT * FROM gridix;

 x_ix | y_ix
------+------
    0 |    0
    0 |    1
    0 |    2
    0 |    3
    0 |    4
    0 |    5
    0 |    6
    0 |    7
    0 |    8
    0 |    9
    0 |   10
    0 |   11
    0 |   12
    0 |   13
...

So far, so good. The gridix CTE returns coordinates for each cell in the grid, from zero to the numsq value from my params CTE. From those coordinates, if I know the geographic boundaries of the data set and the number of squares in each edge, I can calculate the latitude and longitude of the four corners of each grid square. First, I need to find the geographic boundaries of the data. My dataset lives in a table called manzanas, Spanish for “city block” (and also “apple”). Each row contains one geographic attribute containing a polygon defining the boundaries of the block, and several other attributes such as the education score I mentioned. In PostGIS there are several different ways to find the bounding box I want; here’s the one I used.

limits AS (
    SELECT
        MIN(ST_XMin(geom)) AS xmin,
        MAX(ST_XMax(geom)) AS xmax,
        MIN(ST_YMin(geom)) AS ymin,
        MAX(ST_YMax(geom)) AS ymax
    FROM
        manzanas
),

And, just to check my results:

inegi=#     SELECT
        MIN(ST_XMin(geom)) AS xmin,
        MAX(ST_XMax(geom)) AS xmax,
        MIN(ST_YMin(geom)) AS ymin,
        MAX(ST_YMax(geom)) AS ymax
    FROM
        manzanas;
     xmin      |     xmax     |       ymin       |       ymax
---------------+--------------+------------------+------------------
 -99.349658451 | -98.94668802 | 19.1241898199991 | 19.5863775499992
(1 row)

So the data in question extend from about 99.35 to 98.95 west longitude, and 19.12 to 19.59 north latitude. Now I’ll calculate the boundaries of each grid square, compose a text representation for the square in Well-Known Text format, and convert the text to a PostGIS geometry object. There’s another way I could do this, much simpler and probably much faster, which I hope to detail in a future blog post, but this will do for now.

gridcoords AS (
    SELECT
        (xmax - xmin) / numsq * x_ix + xmin AS x0,
        (xmax - xmin) / numsq * (x_ix + 1) + xmin AS x1,
        (ymax - ymin) / numsq * y_ix + ymin AS y0,
        (ymax - ymin) / numsq * (y_ix + 1) + ymin AS y1,
        x_ix || ',' || y_ix AS grid_ix
    FROM limits, params, gridix
),
gridewkt AS (
    SELECT
        grid_ix,
        'POLYGON((' ||
        x0 || ' ' || y0 || ',' ||
        x1 || ' ' || y0 || ',' ||
        x1 || ' ' || y1 || ',' ||
        x0 || ' ' || y1 || ',' ||
        x0 || ' ' || y0 || '))' AS ewkt
    FROM gridcoords
),
gridgeom AS (
    SELECT
        grid_ix, ewkt,
        ST_GeomFromText(ewkt, 4326) AS geom
    FROM gridewkt
),

And again, I’ll check the result by SELECTing grid_ix, ewkt, and geom, from the first row of the gridgeom CTE.

-[ RECORD 1 ]-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
grid_ix | 0,0
ewkt    | POLYGON((-99.349658451 19.1241898199991,-99.34562874669 19.1241898199991,-99.34562874669 19.1288116972991,-99.349658451 19.1288116972991,-99.349658451 19.1241898199991))
geom    | 0103000020E6100000010000000500000029F4D6CD60D658C06B646FE7CA1F3340D3E508C81ED658C06B646FE7CA1F3340D3E508C81ED658C0EC3DABCDF920334029F4D6CD60D658C0EC3DABCDF920334029F4D6CD60D658C06B646FE7CA1F3340

I won’t claim an ability to translate the geometry object as represented above, but the ewkt value looks correct, so let’s keep going. Next I need to cut the blocks into pieces, corresponding to the parts of each block that belong in a single grid square. So a block lying entirely in one square will return one row in this next query; a block that intersects two squares will return two rows. Each row will include the geometries of the block, the square, and the intersection of the two, the total area of the block, the area of the intersection, and the total educational score for the block.

block_part AS (
    SELECT
        grid_ix,                     -- Grid square coordinates, e.g. 0,0
        m.gid AS manz_gid,           -- Manzana identifier
        ggm.geom,                    -- Grid square geometry
        m.graproes::FLOAT,           -- Educational score
        ST_Intersection(m.geom, ggm.geom)
            AS manz_bg_int,          -- Geometric intersection between grid square and city block
        ST_Area(m.geom)
            AS manz_area,            -- Area of the city block
        ST_Area(ST_Intersection(m.geom, ggm.geom)) / ST_Area(m.geom)
            AS manz_area_perc        -- Area of the intersection
    from
        gridgeom ggm
        JOIN manzanas m
            ON (ST_Intersects(m.geom, ggm.geom))
    WHERE
        m.graproes IS NOT NULL       -- Skip null educational scores
),

Here’s a sample of those results:

-[ RECORD 1 ]--+---------------------
grid_ix        | 15,16
manz_gid       | 61853
geom           | 0103000020E610000...
graproes       | 11
manz_bg_int    | 0103000020E610000...
manz_area      | 3.26175061395103e-06
manz_area_perc | 0.149256808754461
-[ RECORD 2 ]--+---------------------
grid_ix        | 15,17
manz_gid       | 61853
geom           | 0103000020E610000...
graproes       | 11
manz_bg_int    | 0103000020E610000...
manz_area      | 3.26175061395103e-06
manz_area_perc | 0.850743191246843

These results, which I admit to having selected with some care, show a single block, number 61853, which lies across the border between two grid squares. Now we’ll calculate the education score for each block fragment, and then divide the fragments into groups based on the grid square in which they belong, and aggregate the results. I did this in two separate CTEs.

grid_calc AS (
    SELECT
        grid_ix,
        geom,
        graproes * manz_area_perc AS grid_graproes,
        manz_area_perc
    FROM
        block_part
),
grid_accum AS (
    SELECT
        grid_ix,
        geom,
        SUM(grid_graproes) AS sum_graproes
    FROM grid_calc
    GROUP BY grid_ix, geom
),

This latest CTE gives results such as these:

-[ RECORD 1 ]+-----------------
grid_ix      | 12,14
geom         | 0103000020E61...
sum_graproes | 3888.53630440106

Now we’re left with turning these results into a visualization. I’d like to assign each polygon a height, and a color. To make the visualization easier to understand, I’ll divide the results into a handful of classes, and assign a height and color to each class.

Using an online color palette generator, I came up with a sequence of six colors, which progress from white to a green similar to the green bar on the Mexican flag. Another CTE will return these colors as an array, and yet another will assign the grid squares to groups based on their calculated score. Finally, a third will select the proper color from the array using that group value. At this point, readers are probably thinking “enough already; quit dividing everything into ever smaller CTEs”, to which I can only say, “Yeah, you may be right.”

colors AS (
    SELECT ARRAY['ffffff','d4ead7','a9d6af','7ec188','53ad60','299939'] AS colors
),
colorix AS (
    SELECT *,
        NTILE(ARRAY_LENGTH(colors, 1)) OVER (ORDER BY sum_graproes asc) AS edu_colorix
    FROM grid_accum, colors
),
color AS (
    SELECT *,
        colors[edu_colorix] as edu_color
    from colorix
)

The ntile() window function is useful for this kind of thing. It divides the given partition into buckets, and returns the number of the bucket for each row. Here, the partition consists of the whole data set; we sort it by educational score to ensure low-scoring grid squares get low-numbered buckets. Note also that I can change the colors, adding or removing groups, simply by adjusting the colors CTE. This could theoretically prove handy, if I decided I didn’t like the number of levels or the color scheme, but it’s a feature I never used for this visualization.

We’re on the home stretch, at last, and I should clarify how I plan to turn the database objects into KML, usable on a Liquid Galaxy. I used ogr2ogr from the GDAL toolkit. It converts between a number of different GIS data sources, including PostGIS to KML. I need to feed it the geometry I want to draw, as well as styling instructions and, in this case, a custom KML altitudeMode.

Styling is an involved topic; for our purposes it’s enough to say that I’ll tell ogr2ogr to use our selected color both to draw the lines of our polygons, and to fill them in. But moving the grid square’s geometry to an altitude corresponding to its educational score is fairly easy, using PostGIS’s ST_Force3DZ() function to add to the hitherto two-dimensional polygon a zero-valued third dimension, and ST_Translate() to move it above the surface of the earth a ways. So I can probably finish this with one final query:

-- Insert all previous CTEs here
SELECT
    ST_Translate(
        ST_Force3DZ(geom), 0, 0,
        alt_bias + edu_colorix * alt_percfactor
    ) AS edu_geom,
    'BRUSH(fc:#' || edu_color || 'ff);PEN(c:#' || edu_color || 'ff)' AS edu_style,
    'absolute' AS "altitudeMode"
FROM color, params;

You may remember alt_bias and alt_percfactor, the oddly named and thus far unexplained values in my first params CTE. These I used to control how far apart in altitude one group of polygons is from another, and to bias them all far enough above the ground to avoid the problem of them being obscured by terrain features. You may also remember that this query began with the CREATE TABLE grid_mza_vals AS... command, meaning that we’ll store the results of all this processing in a table, so ogr2ogr can get to it. We call ogr2ogr like this:

ogr2ogr \
    -f LIBKML education.kml \
    PG:"dbname=inegi user=josh password=<redacted>" \
    -sql "SELECT grid_ix, edu_geom, edu_style as \"OGR_STYLE\", \"altitudeMode\" FROM grid_mza_vals"

OGR’s LIBKML driver knows an attribute called “OGR_STYLE” is a style string, and one called “altitudeMode” is, predictably, the feature’s altitude mode. So this will create a KML file, containing one placemark for each row in our grid_mza_tables table. Each placemark consists of a colored square, floating in the air above Mexico City. The squares come in six different levels and six different colors, corresponding to our original education data. Something like this:

Here’s one of the placemarks from the KML.

      <Placemark id="sql_statement.1">
        <Style>
          <LineStyle>
            <color>ffffffff</color>
            <width>1</width>
          </LineStyle>
          <PolyStyle>
            <color>ffffffff</color>
          </PolyStyle>
        </Style>
        <ExtendedData>
          <SchemaData schemaUrl="#sql_statement.schema">
            <SimpleData name="grid_ix">11,10</SimpleData>
            <SimpleData name="OGR_STYLE">BRUSH(fc:#ffffffff);PEN(c:#ffffffff)</SimpleData>
          </SchemaData>
        </ExtendedData>
        <Polygon>
          <altitudeMode>absolute</altitudeMode>
          <outerBoundaryIs>
            <LinearRing>
              <coordinates>
                -99.17235146136,19.3090649119991,4300
                -99.15623264412,19.3090649119991,4300
                -99.15623264412,19.3275524211991,4300
                -99.17235146136,19.3275524211991,4300
                -99.17235146136,19.3090649119991,4300
              </coordinates>
            </LinearRing>
          </outerBoundaryIs>
        </Polygon>
      </Placemark>

This may be sufficient, but it gets confusing when viewed from a low angle, because it’s hard to tell which square belongs to which part of the map. I prefer having the polygons “extruded” from the ground, as KML terms it. I used a simple Perl script to add the extrude element to each polygon in the KML, resulting in this:

This query works, but leaves a few things to be desired. For instance, doing everything in one query is probably not the best option when lots of processing is involved. Ideally we would calculate the grid geometries once, and save them in a table somewhere, for quicker processing as we build the rest of the query and experiment with visulization options. Second, PostGIS provides an arguably more elegant way of finding grid squares in the first place, a method which also affords other options for interesting visualizations. Stay tuned for a future blog post discussing these issues. Meanwhile, CTEs proved a valuable way to systematize and modularize a very complicated query.

Temporary tables in SQL query optimization

2015-02-25T00:00:00+00:00

SQL queries can get complex in a big hurry. If you are querying multiple tables, and in particular if your query involves operations like UNION and INTERSECT, then you can find yourself in a big, messy pile of SQL. Worse, that big pile will often run slowly, sometimes to the point where a web application times out!

I won’t inflict a real-world example on you; that would be cruel. So let’s look at a “toy” problem, keeping in mind that this won’t illustrate any time-savings, just the technique involved.

Here’s the original SQL:

SELECT p.* FROM products p
JOIN (SELECT * FROM inventory WHERE /* complex clause here */) i USING (sku)
UNION ALL
SELECT p.* FROM clearance_products p
JOIN (SELECT * FROM inventory WHERE /* complex clause here */) i USING (sku)

Bonus hint: using “UNION ALL“ instead of just “UNION” will allow the query processor to skip an unnecessary step here. “UNION ALL” says you know the rows on either side of the clause are unique. “UNION” means the results will be post-processed to remove duplicates. This might save you more than a smidgen of time, depending on how large the two sub-queries get.

Now, many times the query optimizer will just do the right thing here. But sometimes (cough, cough-MySQL), your database isn’t quite up to the task. So you have to shoulder the burden and help out. That’s where we can apply a temporary table.

Temporary tables are created for the length of the database session; that’s different than a transaction. For a web application, that’s usually (not always) the length of the request (i.e., from the time your web application opens a database connection, until it explicitly closes it, or until it returns control to the web server, usually by passing it a completed page). For a script, it’s a similar duration, e.g. until the script exits.

CREATE TEMPORARY TABLE cross_inventory AS
SELECT * FROM inventory WHERE /* complex clause here */;

CREATE INDEX cross_inv_sku ON cross_inventory(sku);

There’s no significant difference for our purposes between a “permanent” and a “temporary” table. However, you do have to keep in mind that these tables are created without indexes, so if your goal is to improve the speed of queries involving the data here, adding an index after creating the table is usually desirable.

With all this in place, now we can:

SELECT p.* FROM products p
JOIN cross_inventory i USING (sku)
UNION
SELECT p.* FROM clearance_products p
JOIN cross_inventory i USING (sku)

Sometimes your temporary table will be built up not by a straightforward “CREATE … AS SELECT …”, but by your application:

CREATE TEMPORARY TABLE tmp_inventory AS SELECT * FROM inventory WHERE false;
CREATE INDEX tmp_inv_sku ON tmp_inventory(sku);

And then within the application:

# Pseudocode
while (more_data) {
  row = build_inv_record(more_data);
  sql_do('INSERT INTO tmp_inventory VALUES (?,?,...)', row);
}

Here, we are creating an empty “inventory” table template as a temporary table (“SELECT * FROM inventory WHERE false”), then adding rows to it from the application, and finally running our query. Note that in a practical application of this, it’s not likely to be a lot faster, because the individual INSERT statements will take time. But this approach may have some utility where the existing “inventory” table doesn’t have the data we want to JOIN against, or has the data, but not in a way we can easily filter.

I’ve used temporary tables (in a MySQL/Interchange/Perl environment) to speed up a query by a factor of two or more. It’s usually in those cases where you have a complex JOIN that appears in two or more parts of the query (again, usually a UNION). I’ve even had big-win situations where the same temporary table was used in two different queries during the same session.

A similar approach is the Common Table Expression (CTE) found in PostgreSQL starting with version 8.4. This allows you to identify the rows you would be pouring into your temporary table as a named result-set, then reference it in your query. Our “toy” example would become:

WITH cross_inventory AS
(SELECT * FROM inventory WHERE /* complex clause here */)
SELECT p.* FROM products p
JOIN cross_inventory i USING (sku)
UNION
SELECT p.* FROM clearance_products p
JOIN cross_inventory i USING (sku)

I’ve not had an opportunity to use CTEs yet, and of course they aren’t available in MySQL, so the temporary-table technique will still have a lot of value for me in the foreseeable future.

How to DRY out your Active Record queries with Squeel

2013-10-24T00:00:00+00:00

Active Record alone isn’t enough when it comes to having a data-access code-base that is DRY and clean in real world Rails applications. You need a tool like the squeel gem and some good practices in place.

Using JOIN with Active Record is cumbersome

Having JOINs in composable data-access Rails code, quickly makes the code ugly and hard to read.

Consider this snippet of code taken directly from Rails guides, that gives developer a full control over the joining part of the query:

Client.joins('LEFT OUTER JOIN addresses ON addresses.client_id = clients.id')

That’s 77 characters for something as simple as a left outer join.

Now—when using Rails, we almost always have properly defined associations in our models, which allows us to put in code something like this:

Category.joins(:posts)

That’s much better—there’s not much code and you immediately know what it does. It isn’t “left outer join” but is useful nevertheless. You could do more with that feature. For example you could join with multiple associations at once:

Post.joins(:category, :comments)

Also—join with nested associations:

Post.joins(comments: :guest)

Which produces:

SELECT posts.* FROM posts
INNER JOIN comments ON comments.post_id = posts.id
INNER JOIN guests ON guests.comment_id = comments.id

Another example from the Rails guides shows that you can join multiple nested associations as well:

Category.joins(posts: [{comments: :guest}, :tags])

Producing:

SELECT categories.* FROM categories
INNER JOIN posts ON posts.category_id = categories.id
INNER JOIN comments ON comments.post_id = posts.id
INNER JOIN guests ON guests.comment_id = comments.id
INNER JOIN tags ON tags.post_id = posts.id

Specifying conditions on nested associations works nicely according to the guides too:

time_range = (Time.now.midnight - 1.day)..Time.now.midnight
Client.joins(:orders).where(orders: {created_at: time_range})

And indeed this looks pretty sweet.

The ugly side of the Active Record JOIN mechanism

If the real world would consist only of the use cases found in docs—we wouldn’t need to look for any better solutions.

Take a look at the following example that might occur:

@tags = params[:tags]
Post.joins(comments: [{guest: :tags}]).
     where(comments: {guest: {tags: {code: @tags}}})

Now try to look at this code and answer the question:

“what does this code do?”.

Squeel—The missing query building DSL

The last example could be easily stated using squeel as:

@tags = params[:tags]
Post.joins{comments.guest.tags}.
     where{comments.guest.tags.code << my{@tags}}

Isn’t that simpler and easier to read and comprehend?

What about queries you wouldn’t do even in Active Record?

If you were to produce a query with e.g. left outer join—you’d have to resort to writing the joining part in a string. With squeel you can just say:

Person.joins{articles.outer}

Can I do the same with subqueries?

Surely you can, take a look here:

awesome_people = Person.where{awesome == true}
Article.where{author_id.in(awesome_people.select{id})}

Which produces the following SQL query:

SELECT "articles".* FROM "articles"
WHERE "articles"."author_id" IN 
  (SELECT "people"."id" FROM "people"  WHERE "people"."awesome" = 't')

But if I were to use SQL functions—I’d have to write queries in strings, no?

No! Take a look here:

Person.joins{articles}.group{articles.title}.
       having{{articles => {max(id) => id}}}

Producing:

SELECT "people".* FROM "people"
INNER JOIN "articles" ON "articles"."person_id" = "people"."id"
GROUP BY "articles"."title"
HAVING max("articles"."id") = "articles"."id"

Why use a DSL to generate SQL?—Composability

At this point many hard-core database experts sit with a wry face, frowning at the idea of having an ORM and DSL for accessing the database.

I won’t go into reasons as to why their concerns are valid or not. I’ll just say that I completely understand and feel most of them.

Here though, is an argument for using those DSLs: you simply can’t compose SQL strings easily, mixing and matching many of them in order to have a DRY and maintainable code base.

You can’t compose:

SELECT "people".* FROM "people"
WHERE "people"."active" IS TRUE

With:

SELECT "people".* FROM "people"
WHERE "people"."sex" = 'female'

But with a DSL like squeel you can easily define:

def self.active
  where{active == true}
end

def self.females
  where{sex == 'female'}
end

And then in your controller say:

def index
  @people = Person.active.females
end

And nothing would stop you from reusing those methods to build nice looking and easy to comprehend queries.

Speedier column additions in PostgreSQL

2013-07-17T00:00:00+00:00

Say you want to add a column to a large table, e.g.,

ALTER TABLE transactions ADD COLUMN email_sent BOOLEAN DEFAULT FALSE;

You really do want new rows to start out with the column “false” (if no value is supplied when the row is created). However, you also want all existing rows to be “true”, so innocently:

UPDATE transactions SET email_sent = TRUE;

This is a great time for a coffee break, or a trip to the post office, or maybe (if you’re a telecommuter like me), a stroll out to weed the garden. Unfortunately for all those side activities, you really didn’t have to take the long way.

BEGIN;
ALTER TABLE transactions ADD COLUMN email_sent BOOLEAN DEFAULT TRUE;
ALTER TABLE transactions ALTER TABLE email_sent SET DEFAULT FALSE;
COMMIT;

This is a lot faster; create all the columns with the appropriate value, then set the default for new rows, and all inside a transaction so you know it gets done atomically.

Window functions in action

2013-06-05T00:00:00+00:00

Image by Wikimedia user Ardfern

Yesterday I ran on to a nice practical application of a number of slightly unusual SQL features, in particular, window functions. PostgreSQL has had window functions for quite a while now (since version 8.4, in fact, the oldest version still officially supported), but even though they’re part of the SQL standard, window functions aren’t necessarily a feature people use every day. As a bonus, I also threw in some common table expressions (also known as CTEs, also a SQL standard feature), to help break up what could have been a more confusing, complex query.

A client of ours noticed a problem in some new code they were working on. It was possible for users to submit duplicate orders to the system in quick succession, by double-clicking or something similar. This was fixed in the code easily enough, but we needed to clean up the duplicate orders in the database. Which meant we had to find them. We defined a group of duplicates as all orders involving the same line items, with one of a set of possible status codes, created in an interval of less than five minutes by the same user.

This discussion of the time interval between two different records should immediately signal “window functions” (or possibly a self-join, but window functions are much easier in this case). A window function takes a set of rows and lets you chop them up into subsets, processing the subsets in various ways. In this case, we want to take all the rows in the orders table with a particular status value, group them by the customer who placed the order as well as by the items in the order, and then evaluate each of those groups.

As might be expected, the items associated with an order are in a different table from the orders themselves. There are probably several different ways I could have compared the items; I chose to accumulate all the items in an order into an array, and compare the resulting arrays (I imagine this would be awfully slow if orders had many different items attached to them, but it wasn’t a problem in this instance). An item consists of its ID value, and an integer quantity; I created a composite type, so I could compare these values as an array.

CREATE TYPE order_item AS (
   item_id integer,
   quantity integer
);

Now I need a query that will gather all the items on an order into an array. I’ll use the array_agg() aggregate to do that. While I’m at it, I’ll also filter out order status codes I don’t want. One important note here is that later, I’ll be comparing the results of array_agg() with each other, and array element ordering will matter. So I need to sort the rows as they’re being aggregated. Fortunately we’ve been able to do that easily since version 9.0, with ordered aggregates (NB! many of the features in use in this post are part of the SQL standard; ordered aggregates are a PostgreSQL-specific extension).

SELECT
    o.id, o.created_at, o.user_id,
    array_agg((item_id, quantity)::order_item ORDER BY item_id, quantity) AS order_items
FROM
    orders o
    JOIN order_items oi
        ON (o.id = oi.order_id)
WHERE o.status >= 200 AND o.status <= 400
GROUP BY o.id

The result, taken from a sample data set I created for demonstration purposes, is this:

 id |         created_at         | user_id |        order_items         
----+----------------------------+---------+----------------------------
  4 | 2013-06-04 17:56:22.857817 |       3 | {"(2,3)"}
  5 | 2013-06-04 17:57:11.099472 |       1 | {"(1,11)","(2,1)","(3,2)"}
  1 | 2013-06-04 17:56:16.017938 |       1 | {"(1,10)","(2,1)","(3,2)"}
  2 | 2013-06-04 17:56:19.27393  |       1 | {"(1,10)","(2,1)","(3,2)"}
  3 | 2013-06-04 17:56:21.137858 |       2 | {"(1,1)"}
(5 rows)

Now I need to compare various rows in this list, and that’s where window functions come in. A call to a window function looks like any other function call, except that it is followed by a “window definition”, which describes the window of rows the function will operate on: how rows are grouped into windows, and optionally, how rows within a window are sorted. Here a query I used to get started.

SELECT
    id, created_at,
    first_value(created_at) OVER (PARTITION BY user_id ORDER BY created_at)
FROM orders

The OVER and subsequent parenthetical expression is the window clause. This one tells PostgreSQL to group all the orders by the user ID that created them, and sort them in ascending order of their creation time. In future iterations I’ll need to partition by the items in the order as well, but we’re keeping it simple for this query. The first_value() function is the actual window function, and returns the expression it is passed, evaluated in terms of the first row in the window. In this case the sort order is important; without it, there is no guarantee which of the window’s rows is considered “first”. There are lots of other window functions available, documented here.

It’s time to start combining these queries together. For this, I like to use common table expressions, which essentially let me define a named view for purposes of just this query. My first such expression will gather the items associated with an order into an array. The second part of the query will use window functions to compare the results of the first part with each other.

WITH order_items_array AS (
    SELECT
        o.id, o.created_at, o.user_id,
        array_agg((item_id, quantity)::order_item ORDER BY item_id, quantity) AS order_items
    FROM
        orders o
        JOIN order_items oi
            ON (o.id = oi.order_id)
    WHERE o.status >= 200 AND o.status <= 400
    GROUP BY o.id
)
SELECT
    id, user_id, created_at,
    first_value(id) OVER user_order_partition AS first_id,
    first_value(created_at) OVER user_order_partition AS first_created
FROM order_items_array
WINDOW user_order_partition AS (PARTITION BY user_id, order_items ORDER BY created_at);

This gives these results:

 id | user_id |         created_at         | first_id |       first_created        
----+---------+----------------------------+----------+----------------------------
  1 |       1 | 2013-06-04 17:56:16.017938 |        1 | 2013-06-04 17:56:16.017938
  2 |       1 | 2013-06-04 17:56:19.27393  |        1 | 2013-06-04 17:56:16.017938
  5 |       1 | 2013-06-04 17:57:11.099472 |        5 | 2013-06-04 17:57:11.099472
  3 |       2 | 2013-06-04 17:56:21.137858 |        3 | 2013-06-04 17:56:21.137858
  4 |       3 | 2013-06-04 17:56:22.857817 |        4 | 2013-06-04 17:56:22.857817
(5 rows)

This shows the order ID, user ID, and creation timestamp, as well as the ID and creation timestamp of the first order in the window each row belongs to. Note that in order to avoid having to retype the same long window definition twice, I used an alternate syntax whereby I created a named window definition, and referred to that name. Anyway, you can see that the first two rows have the same first_id value; this means they’re duplicates, and we want to get rid of one of them. You’ll have to trust me that that’s the only duplicated order in my sample database; suffice it to say that these results are, in fact, correct. I’ll decide (because it turns out to be easier this way) to keep the earliest of the duplicate orders, so from the results above, I can see that I want to remove order ID 2. It would be nice, though, to have a list of just the order IDs I need to remove without any other information. Even better, a list of SQL commands to run to remove them. Like this:

WITH order_items_array AS (
    SELECT
        o.id, o.created_at, o.user_id,
        array_agg((item_id, quantity)::order_item ORDER BY item_id, quantity) AS order_items
    FROM
        orders o
        JOIN order_items oi
            ON (o.id = oi.order_id)
    WHERE o.status >= 200 AND o.status <= 400
    GROUP BY o.id
), order_duplicates AS (
    SELECT
        id, user_id, created_at,
        first_value(id) OVER user_order_partition AS first_id,
        first_value(created_at) OVER user_order_partition AS first_created
    FROM order_items_array
    WINDOW user_order_partition AS (PARTITION BY user_id, order_items ORDER BY created_at)
)
    SELECT 'DELETE FROM orders WHERE id = ' || id || ';' FROM order_duplicates
    WHERE first_id != id;

…which returns this one result:

 ?column?
----------------------------------
 DELETE FROM orders WHERE id = 2;
(1 row)

So by combining common table expressions, ordered aggregates, composite types, arrays, and window functions, we’ve successfully cleaned up this database. Until we find another application bug…

Foreign Data Wrappers

2013-05-13T00:00:00+00:00

Original images from Flickr user jenniferwilliams

One of our clients, for various historical reasons, runs both MySQL and PostgreSQL to support their website. Information for user login lives in one database, but their customer activity lives in the other. The eventual plan is to consolidate these databases, but thus far, other concerns have been more pressing. So when they needed a report combining user account information and customer activity, the involvement of two separate databases became a significant complicating factor.

In similar situations in the past, using earlier versions of PostgreSQL, we’ve written scripts to pull data from MySQL and dump it into PostgreSQL. This works well enough, but we’ve updated PostgreSQL fairly recently, and can use the SQL/MED features added in version 9.1. SQL/MED (“MED” stands for “Management of External Data”) is a decade-old standard designed to allow databases to make external data sources, such as text files, web services, and even other databases look like normal database tables, and access them with the usual SQL commands. PostgreSQL has supported some of the SQL/MED standard since version 9.1, with a feature called Foreign Data Wrappers, and among other things, it means we can now access MySQL through PostgreSQL seamlessly.

The first step is to install the right software, called mysql_fdw. It comes to us via Dave Page, PostgreSQL core team member and contributor to many projects. It’s worth noting Dave’s warning that he considers this experimental code. For our purposes it works fine, but as will be seen in this post, we didn’t push it too hard. We opted to download the source and build it, but installing using pgxn works as well:

$ env USE_PGXS=1 pgxnclient install mysql_fdw
INFO: best version: mysql_fdw 1.0.1
INFO: saving /tmp/tmpjrznTj/mysql_fdw-1.0.1.zip
INFO: unpacking: /tmp/tmpjrznTj/mysql_fdw-1.0.1.zip
INFO: building extension
gcc -O2 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -g -fpic -I/usr/include/mysql -I. -I. -I/home/josh/devel/pg91/include/postgresql/server -I/home/josh/devel/pg91/include/postgresql/internal -D_GNU_SOURCE -I/usr/include/libxml2   -c -o mysql_fdw.o mysql_fdw.c
mysql_fdw.c: In function ‘mysqlPlanForeignScan’:
mysql_fdw.c:466:8: warning: ‘rows’ may be used uninitialized in this function [-Wmaybe-uninitialized]
gcc -O2 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -g -fpic -shared -o mysql_fdw.so mysql_fdw.o -L/home/josh/devel/pg91/lib -L/usr/lib  -Wl,--as-needed -Wl,-rpath,'/home/josh/devel/pg91/lib',--enable-new-dtags  -L/usr/lib/x86_64-linux-gnu -lmysqlclient -lpthread -lz -lm -lrt -ldl
INFO: installing extension
< ... snip ... >

Here I’ll refer to the documentation provided in mysql_fdw’s README. The first step in using a foreign data wrapper, once the software is installed, is to create the foreign server, and the user mapping. The foreign server tells PostgreSQL how to connect to MySQL, and the user mapping covers what credentials to use. This is an interesting detail; it means the foreign data wrapper system can authenticate with external data sources in different ways depending on the PostgreSQL user involved. You’ll note the pattern in creating these objects: each simply takes a series of options that can mean whatever the FDW needs them to mean. This allows the flexibility to support all sorts of different data sources with one interface.

The final step in setting things up is to create a foreign table. In MySQL’s case, this is sort of like a view, in that it creates a PostgreSQL table from the results of a MySQL query. For our purposes, we needed access to several thousand structurally identical MySQL tables (I mentioned the goal is to move off of this one day, right?), so I automated the creation of each table with a simple bash script, which I piped into psql:

for i in `cat mysql_tables`; do
    echo "CREATE FOREIGN TABLE mysql_schema.$i ( ... table definition ...)
        SERVER mysql_server OPTIONS (
            database 'mysqldb',
            query 'SELECT ... some fields ... FROM $i'
        );"
done

In a step not shown above, this script also consolidates the data from each table into one, native PostgreSQL table, to simplify later reporting. In our case, pulling the data once and reporting on the results is perfectly acceptable; in other words, data a few seconds old wasn’t a concern. We also didn’t need to write back to MySQL, which presumably could complicate things somewhat. We did, however, run into the same data validation problems PostgreSQL users habitually complain about when working with MySQL. Here’s an example, in my own test database:

mysql> create table bad_dates (mydate date);
Query OK, 0 rows affected (0.07 sec)

mysql> insert into bad_dates values ('2013-02-30'), ('0000-00-00');
Query OK, 2 rows affected (0.02 sec)
Records: 2  Duplicates: 0  Warnings: 0

Note that MySQL silently transformed ‘2013-02-30’ into ‘0000-00-00’. Sigh. Then, in psql we do this:

josh=# create extension mysql_fdw;
CREATE EXTENSION

josh=# create server mysql_svr foreign data wrapper mysql_fdw options (address '127.0.0.1', port '3306');
CREATE SERVER

josh=# create user mapping for public server mysql_svr options (username 'josh', password '');
CREATE USER MAPPING

josh=# create foreign table bad_dates (mydate date) server mysql_svr options (query 'select * from test.bad_dates');
CREATE FOREIGN TABLE

josh=# select * from bad_dates ;
ERROR:  date/time field value out of range: "0000-00-00"

We’ve told PostgreSQL we’ll be feeding it valid dates, but MySQL’s idea of a valid date differs from PostgreSQL’s, and the latter complains when the dates don’t meet its stricter requirements. Several different workarounds exist, including admitting that ‘0000-00-00’ really is wrong and cleaning up MySQL, but in this case, we modified the query underlying the foreign table to fix the dates on the fly:

SELECT CASE disabled WHEN '0000-00-00' THEN NULL ELSE disabled END,
    -- various other fields
    FROM some_table

Fortunately this is the only bit of MySQL / PostgreSQL impedance mismatch that has tripped us up thus far; we’d have to deal with any others we found individually, just as we did this one.

Postgres alter column problems and solutions

2012-11-09T00:00:00+00:00

Image from Flickr user ell brown

A common situation for database-backed applications is the need to change the attributes of a column. One can change the data type, or more commonly, only the size limitation, e.g. VARCHAR(32) gets changed to VARCHAR(42). There are a few ways to accomplish this in PostgreSQL, from a straightforward ALTER COLUMN, to replacing VARCHAR with TEXT (plus a table constraint), to some advanced system catalog hacking.

The most common example of such a change is expanding a VARCHAR declaration to allow more characters. For example, your “checksum” column was based on MD5 (at 32 characters), and now needs to be based on Keccak (Keccak is pronounced “catch-ack”) (at 64 characters) In other words, you need a column in your table to change from VARCHAR(32) to VARCHAR(64). The canonical approach is to do this:

ALTER TABLE foobar ALTER COLUMN checksum TYPE VARCHAR(64);

This approach works fine, but it has two huge and interrelated problems: locking and time. This approach locks the table for as long as the command takes to run. And by lock, we are talking a heavy “access exclusive” lock which shuts everything else out of the table. If your table is small, this is not an issue. If your table has a lot of data, however, this brings us to the second issue: table rewrite. The above command will cause Postgres to rewrite every single row of the table, which can be a very expensive operation (both in terms of disk I/O and wall clock time). So, a simple ALTER COLUMN solution usually comes at a very high cost for large tables. Luckily, there are workarounds for this problem.

First, some good news: as of version 9.2, there are many operations that will no longer require a full table rewrite. Going from VARCHAR(32) to VARCHAR(64) is one of those operations! Thus, if you are lucky enough to be using version 9.2 or higher of Postgres, you can simply run the ALTER TABLE and have it return almost instantly. From the release notes:

Reduce need to rebuild tables and indexes for certain ALTER TABLE … ALTER COLUMN TYPE operations (Noah Misch)*

Increasing the length limit for a varchar or varbit column, or removing the limit altogether, no longer requires a table rewrite. Similarly, increasing the allowable precision of a numeric column, or changing a column from constrained numeric to unconstrained numeric, no longer requires a table rewrite. Table rewrites are also avoided in similar cases involving the interval, timestamp, and timestamptz types.

However, if you are not yet on version 9.2, or are making an operation not covered above (such as shrinking the size limit of a VARCHAR), your only option to avoid a full table rewrite is the system catalog change below. However, before you jump down there, consider a different option: abandoning VARCHAR altogether.

In the Postgres world, there are few differences between the VARCHAR and TEXT data types. The latter can be thought of as an unbounded VARCHAR, or if you like, a VARCHAR(999999999999). You may also add a check constraint to a table to emulate the limit of a VARCHAR. For example, to convert a VARCHAR(32) column named “checksum” to a TEXT column:

ALTER TABLE foobar ALTER COLUMN checksum TYPE text;
ALTER TABLE foobar ADD CONSTRAINT checksum_length
  CHECK (LENGTH(checksum) <= 32);

The data type change suffers from the same full table rewrite problem as before, but if you are using version 9.1 or newer of Postgres, the change from VARCHAR to TEXT does not do a table rewrite. The creation of the check constraint, however, will scan all of the existing table rows to make sure they meet the condition. While not as costly as a full table rewrite, scanning every single row in a large table will still be expensive. Luckily, version 9.2 of Postgres comes to the rescue again with the addition of the NOT VALID phrase to the check constraint clause. Thus, in newer versions you can avoid the scan entirely by writing:

ALTER TABLE foobar ADD CONSTRAINT checksum_length
  CHECK (LENGTH(checksum) <= 32) NOT VALID;

This is a one-time exception for the constraint, and only applies as the constraint is being created. In other words, despite the name, the constraint is very much valid after it is created. If you want to validate all the rows that you skipped at a later time, you can use the ALTER TABLE .. VALIDATE CONSTRAINT command. This has the double advantage of allowing the check to be delayed until a better time, and taking a much lighter lock on the table than the ALTER TABLE .. ADD CONSTRAINT does.

So why would you go through the trouble of switching from your VARCHAR(32) to a TEXT column with a CHECK constraint? There are at least three good reasons.

First, if you are running Postgres 9.2 or better, this means you can change the constraint requirements on the fly, without a table scan—even for the “non-optimal” situations such as going from 64 characters down to 32. Just drop the old constraint, and add a new one with the NOT VALID clause thrown on it.

Second, the check constraint gives a better error message, and a clearer indication that the limitation was constructed with some thought behind it. Compare these messages:

postgres=# CREATE TABLE river( checksum VARCHAR(4) );
CREATE TABLE

postgres=# INSERT INTO river VALUES ('abcde');
ERROR:  value too long for type character varying(4)

postgres=# CREATE TABLE river( checksum TEXT,
postgres-#   CONSTRAINT checksum_length CHECK (LENGTH(checksum) <= 4) );
CREATE TABLE

postgres=# INSERT INTO river VALUES ('abcde');
ERROR:  new row for relation "river" violates check constraint "checksum_length"
DETAIL:  Failing row contains (abcde).

Third, and most important, you are no longer limited to a single column attribute (maximum length). You can use the constraint to check for many other things as well: minimum size, actual content, regex matching, you name it. As a good example, if we are are truly storing checksums, we probably want the hexadecimal Keccak checksums to be exactly 64 characters, and not just a maximum length of 64 characters. So, to illustrate the above point about switching constraints on the fly, you could change the VARCHAR(32) to a TEXT and enforce a strict 64 character limit with:

BEGIN;

ALTER TABLE foobar DROP CONSTRAINT checksum_length;

ALTER TABLE foobar ADD CONSTRAINT checksum_length
  CHECK (LENGTH(checksum) = 64) NOT VALID;

COMMIT;

We just introduced a minimum and a maximum, something old VARCHAR could not do. We can constrain it further, as we should only be allowing hexadecimal characters to be stored. Thus, we can also reject and characters other than 0123456789abcdef from being added:

BEGIN;

ALTER TABLE foobar DROP CONSTRAINT checksum_length;

ALTER TABLE foobar ADD CONSTRAINT checksum_valid
  CHECK ( LENGTH(checksum) = 64 AND checksum ~ '^[a-f0-9]*$' ) NOT VALID;

COMMIT;

Since we have already added a regex check, we can reduce the size of the CHECK with a small hit in clarity like so:

BEGIN;

ALTER TABLE foobar DROP CONSTRAINT checksum_length;

ALTER TABLE foobar ADD CONSTRAINT checksum_valid
  CHECK ( checksum ~ '^[a-f0-9]{64}$' ) NOT VALID;

COMMIT;

Image from Flickr user loozrboy

Back to the other problem, however: how can we avoid a table rewrite when going from VARCHAR(64) to VARCHAR(32), or when stuck on an older version of Postgres that always insists on a table rewrite? The answer is the system catalogs. Please note that any updating to the system catalogs should be done very, very carefully. This is one of the few types of update I will publicly mention and condone. Do not apply this lesson to any other system table or column, as there may be serious unintended consequences.

So, what does it mean to have VARCHAR(32) vs. VARCHAR(64)? As it turns out, there is no difference in the way the actual table data is written. The length limit of a VARCHAR is simply an implicit check constraint, after all, and as such, it is quite easy to change.

Let’s create a table and look at some of the important fields in the system table pg_attribute. In these examples we will use Postgres 8.4, but other versions should look very similar—this part of the system catalog rarely changes.

postgres=# CREATE TABLE foobar ( checksum VARCHAR(32) );
CREATE TABLE

postgres=# \x
Expanded display is on.

postgres=# SELECT attname, atttypid::regtype, atttypmod FROM pg_attribute 
postgres=#  WHERE attrelid = 'foobar'::regclass AND attname = 'checksum';
-[ RECORD 1 ]----------------
attname   | checksum
atttypid  | character varying
atttypmod | 36

The important column is atttypmod. It indicates the legal length of this varchar column (whose full legal name is “character varying”, but everyone calls it varchar). In the case of Postgres, there is also 4 characters of overhead. So VARCHAR(32) shows up as 36 in the atttypmod column. Thus, if we want to change it to a VARCHAR(64), we add 4 to 64 and get a number of 68. Before we do this change, however, we need to make sure that nothing else will be affected. There are other dependencies to consider, such as views and foreign keys, that you need to keep in mind before making this change. What you should do is carefully check all the dependencies this table has:

postgres=# SELECT c.relname||':'||objid AS dependency, deptype
postgres-#   FROM pg_depend d JOIN pg_class c ON (c.oid=d.classid)
postgres-#   WHERE refobjid = 'foobar'::regclass;
  dependency   | deptype 
---------------+---------
 pg_type:16419 | i

We can see in the above that the only dependency is an entry in the pg_type table—which is a normal thing for all tables and will not cause any issues. Any other entries, however, should give you pause before doing a manual update of pg_attribute. You can use the information returned by the first column of the above query to see exactly what is referencing the table. For example, let’s make that column unique, as well as adding a view that uses the table, and then see the effects on the pg_depend table:

postgres=# CREATE UNIQUE INDEX jack ON foobar(checksum);
CREATE INDEX

postgres=# CREATE VIEW martha AS SELECT * FROM foobar;
CREATE VIEW

postgres=# SELECT c.relname||':'||objid AS dependency, deptype
postgres-#   FROM pg_depend d JOIN pg_class c ON (c.oid=d.classid)
postgres-#   WHERE refobjid = 'foobar'::regclass;
   dependency     | deptype 
------------------+---------
 pg_type:16419    | i
 pg_class:16420   | a
 pg_rewrite:16424 | n

The “i”, “a”, and “n” stand for internal, auto, and normal. They are not too important in this context, but more details can be found in the docs on the pg_depend table. The first column shows us the system table and oid of the dependency, so we can look them up and see what they are:

postgres=# SELECT typname FROM pg_type WHERE oid = 16419;
 typname 
---------
 foobar

postgres=# SELECT relname, relkind FROM pg_class WHERE oid = 16420;
 relname | relkind 
---------+---------
 jack    | i

-- Views require a little redirection as they are implemented via the rules system
postgres=# SELECT relname,relkind FROM pg_class WHERE oid = 
postgres-#   (SELECT ev_class FROM pg_rewrite WHERE oid = 16424);
 relname | relkind 
---------+---------
 martha  | v

postgres=# \d martha
              View "public.martha"
  Column  |         Type          | Modifiers 
----------+-----------------------+-----------
 checksum | character varying(32) | 
View definition:
 SELECT foobar.checksum
   FROM foobar;

So what does all that tell us? It tells us we should look carefully at the index and the view to make sure they will not be affected by the change. In this case, a simple index on the column will not be affected by changing the length, so it (along with the pg_type entry) can be ignored. The view, however, should be recreated so that it records the actual column size.

We are now ready to make the actual change. This would be an excellent time to make a backup of your database. This procedure should be done very carefully—if you are unsure about any of the entries in pg_depend, do not proceed.

First, we are going to start a transaction, lock the table, and drop the view. Then we are going to change the length of the varchar directly, recreate the view, and commit! Here we go:

postgres=# SELECT c.relname||':'||objid AS dependency, deptype
postgres-#   FROM pg_depend d JOIN pg_class c ON (c.oid=d.classid)
postgres-#   WHERE refobjid = 'foobar'::regclass;
   dependency     | deptype 
------------------+---------
 pg_type:16419    | i
 pg_class:16420   | a
 pg_rewrite:16424 | n

postgres=# \d foobar
            Table "public.foobar"
  Column  |         Type          | Modifiers 
----------+-----------------------+-----------
 checksum | character varying(32) | 
Indexes:
    "jack" UNIQUE, btree (checksum)

postgres=# \d martha
            View "public.martha"
  Column  |         Type          | Modifiers 
----------+-----------------------+-----------
 checksum | character varying(32) | 
View definition:
 SELECT foobar.checksum
   FROM foobar;
postgres=# BEGIN;
BEGIN

postgres=# DROP VIEW martha;
DROP VIEW

postgres=# LOCK TABLE pg_attribute IN EXCLUSIVE MODE;
LOCK TABLE

postgres=# UPDATE pg_attribute SET atttypmod = 68
postgres-#   WHERE attrelid = 'foobar'::regclass AND attname = 'checksum';
UPDATE 1

postgres=# COMMIT;
COMMIT

Verify the changes and check out the pg_depend entries:

postgres=# \d foobar
            Table "public.foobar"
  Column  |         Type          | Modifiers 
----------+-----------------------+-----------
 checksum | character varying(64) | 
Indexes:
    "jack" UNIQUE, btree (checksum)

postgres=# CREATE VIEW martha AS SELECT * FROM foobar;
CREATE VIEW

postgres=# \d martha
            View "public.martha"
  Column  |         Type          | Modifiers 
----------+-----------------------+-----------
 checksum | character varying(64) | 
View definition:
 SELECT foobar.checksum
   FROM foobar;

postgres=# SELECT c.relname||':'||objid AS dependency, deptype
postgres-#   FROM pg_depend d JOIN pg_class c ON (c.oid=d.classid)
postgres-#   WHERE refobjid = 'foobar'::regclass;
    dependency    | deptype 
------------------+---------
 pg_type:16419    | i
 pg_class:16420   | a
 pg_rewrite:16428 | n

Success. Both the table and the view are showing the new VARCHAR size, but the data in the table was not rewritten. Note how the final row returned by the pg_depend query changed: we dropped the view and created a new one, resulting in a new row in both pg_class and pg_rewrite, and thus a new OID shown in the pg_rewrite table.

Hopefully this is not something you ever have to perform. The new features of 9.1 and 9.2 that prevent table rewrites and table scanning should go a long way towards that.

You shall not pass! Preventing SQL injection

2012-06-14T00:00:00+00:00

Greg Sabino Mullane presented a few extremely useful techniques for preventing SQL injection. His advice was mostly based on his recent real-world experience.

The chunk of simple code was causing a potentially very dangerous security breach to the system:

[query … where order_number='[scratch order_number] and username='[session username]']

This code can generate this SQL query:

select * from orders where order_number = '12345' and username = 'alice';

Or this SQL query:

select * from orders where order_number=' ';  delete from orders where id IS NOT NULL;

This is a vulnerability, and you certainly do not want any random stranger to delete records from the “orders” table in your database.

The problem was solved in no time by escaping user input.

Here is Greg’s list of recommendations to make SQL injection impossible:

Escape all user input passed to the database.
Log extensively. If this system hadn’t logged SQL queries, they would have never noticed anything strange. They used tail_n_mail that tracks PostgreSQL logs and sends out emails whenever SQL exception occurs.
Introduce fine-grained control for accessing and manipulating the database. Split responsibilities between a lot of database users and selectively grant permissions to them. Run your code as the appropriate database user with the most restrictive set of permissions possible.
Database triggers can become very handy. In Greg’s case it was impossible to delete the already shipped order because of the triggers assigned to the record.
Have a lot of eyes on the code to eliminate the obvious mistakes.
And finally, if SQL injection is happening, consider shutting down the database server until you find the cause. This is an emergency!

SQL errors in Interchange

2011-09-07T00:00:00+00:00

Interchange has a little feature whereby errors in a [query] tag are reported back to the session just like form validation errors. That is, given the intentional syntax error here:

[query ... sql="select 1 from foo where 1="]

Interchange will paste the error from your database in

  $Session->{errors}{'table foo'}

That’s great, but it comes with a price: now you have a potential for a page with SQL in it, which site security services like McAfee will flag as “SQL injection failures”. Sometimes you just don’t want your SQL failures plastered all over for the world to see.

Simple solution:

  DatabaseDefault LOG_SESSION_ERROR 0

in your Interchange configuration file, possibly constrained so it only affects production (because you’d love to see your SQL errors when you are testing, right?).

Bucardo multi-master for PostgreSQL

2011-06-06T00:00:00+00:00

The original Bucardo

The next version of Bucardo, a replication system for Postgres, is almost complete. The scope of the changes required a major version bump, so this Bucardo will start at version 5.0.0. Much of the innards was rewritten, with the following goals:

Multi-master support

Where “multi” means “as many as you want”! There are no more pushdelta (master to slaves) or swap (master to master) syncs: there is simply one sync where you tell it which databases to use, and what role they play. See examples below.

Ease of use

The bucardo program (previously known as ‘bucardo_ctl’) has been greatly improved, making all the administrative tasks such as adding tables, creating syncs, etc. much easier.

Performance

Much of the underlying architecture was improved, and sometimes rewritten, to make things go much faster. Most striking is the difference between the old multi-master “swap syncs” and the new method, which has been described as “orders of magnitudes” faster by early testers. We use async database calls whenever possible, and no longer have the bottleneck of a single large bucardo_delta table.

Improved logging

Not only are more details provided, there is now the ability to control how verbose the logs are. Just set the log_level parameter to terse, normal, verbose, or debug. Those who had busy systems, which was the equivalent of a ‘debug’ firehose, will really appreciate this.

Different targets

Who says your slave (target) databases need to be Postgres? In addition to the ability to write text SQL files (for say, shipping to a different system), you can have Bucardo push to other systems as well. Stay tuned for more details on this. (Update: there is a blog post about using MongoDB as a target)

This new version is not quite at beta yet, but you can try out a demo of multi-master on Postgres quie easily. Let’s see if we can do it in ten steps.

I. Download all prerequisites

To run Bucardo, you will need a Postgres database (obviously), the DBIx::Safe module, the DBI and DBD::Pg modules, and (for the purposes of this demo) the pgbench utility. Systems vary, but on aptitude-based systems, one can grab all of the above like this:

aptitude install postgresql-server \
perl-DBIx-Safe \
perl-DBD-Pg \
postgresql-contrib

II. Grab the latest Bucardo

git clone git://bucardo.org/bucardo.git

III. Install the program

cd bucardo
perl Makefile.PL
make
sudo make install

You can ignore any errors that come up about ExtUtils::MakeMaker not being recent.

IV. Setup an instance of Bucardo

This step assumes there is a running Postgres available to connect to.

sudo mkdir /var/run/bucardo
sudo chown $USER /var/run/bucardo
bucardo install

V. Use the pgbench program to create some test tables

psql -c 'CREATE DATABASE btest1'
pgbench -i btest1
psql -c 'CREATE DATABASE btest2 TEMPLATE btest1'
psql -c 'CREATE DATABASE btest3 TEMPLATE btest1'
psql -c 'CREATE DATABASE btest4 TEMPLATE btest1'
psql -c 'CREATE DATABASE btest5 TEMPLATE btest1'

VI. Tell Bucardo about the databases and tables you are going to use

bucardo add db t1 dbname=btest1
bucardo add db t2 dbname=btest2
bucardo add db t3 dbname=btest3
bucardo add db t4 dbname=btest4
bucardo add db t5 dbname=btest5
bucardo list dbs

bucardo add table pgbench_accounts pgbench_branches pgbench_tellers herd=therd
bucardo list tables

A herd is simply a logical grouping of tables. We did not add the other pgbench table, pgbench_history, because it has no primary key or unique index.

VII. Group the databases together and set their roles

bucardo add dbgroup tgroup t1:source t2:source t3:source t4:source t5:target

We’ve grouped all five databases together, and made four of them masters (aka source), and one of them a slave (aka target). You can any combination of master and slaves you want, as long as there is at least one master.

VII. Create the Bucardo sync

bucardo add sync foobar herd=therd dbs=tgroup ping=false

Here we simply create a new sync, which is a controllable replication event, telling it which tables we want to replicate, and which databases we are going to use. We also set ping to false, which means that we will not create triggers to automatically fire off replication on any changes, but will do it manually. In a real world scenario, you generally do want those triggers, or want to set Bucardo to check periodically.

VIII. Start up Bucardo

bucardo start

If all went well, you should see some information in the log.bucardo file in the current directory.

IX. Make a bunch of changes on all the source databases.

pgbench -t 10000 btest1
pgbench -t 10000 btest2
pgbench -t 10000 btest3
pgbench -t 10000 btest4

Here, we’ve told pgbench to run ten thousand transactions against each of the first four databases. Triggers on these tables have captured the changes.

X. Kick off the sync and watch the fun.

bucardo kick foobar

You can now tail the log.bucardo file to see the fun, or simply run:

bucardo status

…to see what it is doing, and the final counts when we are done. Don’t forget to stop Bucardo when you are done testing:

bucardo stop

The output of bucardo status, after the sync has completed, should look like this:

bucardo status
Name     State    Last good    Time    Last I/D/C           Last bad    Time
========+========+============+=======+====================+===========+=======
foobar | Good   | 17:58:37   | 3m2s  | 131836/131836/4785 | none      |

Here we see that this syncs has never failed (“Last bad”), the time of day of the last good run, how long ago it was from right now (3 minutes and 2 seconds), as well as details of the last successful run. Last I/D/C stands for number of inserts, deletes, and collisions across all databases for this syncs. This is just an overview of all syncs at a high level, but we can also give status an argument of a sync name to see more details like so:

bucardo status foobar
Last good                       : Jun 02, 2011 17:57:47 (time to run: 42s)
Rows deleted/inserted/conflicts : 131,836 / 131,836 / 4,785
Sync name                       : foobar
Current state                   : Good
Source herd/database            : therd / t1
Tables in sync                  : 3
Status                          : active
Check time                      : none
Overdue time                    : 00:00:00
Expired time                    : 00:00:00
Stayalive/Kidsalive             : yes / yes
Rebuild index                   : 0
Ping                            : no
Onetimecopy                     : 0
Post-copy analyze               : Yes
Last error:                     :

This gives us a little more information about the sync itself, as well as another important metric, how long the sync itself took to run, in this case, 42 seconds. That particular metric might make its way back to the overall “status” view above. Try things out and help us find bugs and improve Bucardo!

Saving time with generate_series()

2011-05-31T00:00:00+00:00

I was giving a presentation once on various SQL constructs, and, borrowing an analogy I’d seen elsewhere, described PostgreSQL’s generate_series() function as something you might use in places where, in some other language, you’d use a FOR loop. One attendee asked, “So, why would you ever want a FOR loop in a SQL query?” A fair question, and one that I answered using examples later in the presentation. Another such example showed up recently on a client’s system where the ORM was trying to be helpful, and chose a really bad query to do it.

The application in question was trying to display a list of records, and allow the user to search through them, modify them, filter them, etc. Since the ORM knew users might filter on a date-based field, it wanted to present a list of years containing valid records. So it did this:

SELECT DISTINCT DATE_TRUNC('year', some_date_field) FROM some_table;

In fairness to the ORM, this query wouldn’t be so bad if some_table only had a few hundred or thousand rows. But in our case it has several tens of millions. This query results in a sequential scan of each of those records, in order to build a list of, as it turns out, about fifty total years. There must be a better way…

The better way we chose turns out to be, in essence, this: find the years of the maximum and minimum date values in the date field, construct a list of all years between the minimum and maximum, inclusive, and see which ones exist in the table. This date field is indexed, so finding its maximum and minimum is very fast:

SELECT
    DATE_TRUNC('year', MIN(some_date_field)) AS mymin,
    DATE_TRUNC('year', MAX(some_date_field)) AS mymax
FROM some_table

Here’s where the FOR loop idea comes in, though it’s probably better described as an “iterator” rather than a FOR loop specifically: for each year between mymin and mymax inclusive, I want a database row. The analogy may not hold terribly well, but the technique is very useful, because it will create a list of all the possible years I might be interested in, and it will do it with just two scans of the some_date_field index, rather than a sequential scan of millions of rows.

SELECT
    generate_series(mymin::INTEGER, mymax::INTEGER) AS yearnum
FROM (
    SELECT
        DATE_TRUNC('year', MIN(some_date_field)) AS mymin,
        DATE_TRUNC('year', MAX(some_date_field)) AS mymax
    FROM some_table
) minmax_tbl

Now I simply have to convert these values to years, and see which ones exist in the underlying table:

SELECT
    yearbegin::timestamptz
FROM
    (
        SELECT
            yearnum * INTERVAL '1 year' + '0000-01-01'::date AS yearbegin
        FROM (
            SELECT
                generate_series(mymin::INTEGER, mymax::INTEGER) AS yearnum
            FROM (
                SELECT
                    DATE_TRUNC('year', MIN(some_date_field)) AS mymin,
                    DATE_TRUNC('year', MAX(some_date_field)) AS mymax
                FROM some_table
        ) yearnum_tbl
    ) beginend_tbl
WHERE
    EXISTS (
        SELECT 1 FROM some_table
        WHERE
            some_date_field BETWEEN yearbegin AND yearbegin + INTERVAL '1 year'
    )
ORDER BY yearbegin ASC
;

As expected, this probes the some_date_field index twice, to get the maximum and minimum date values, and then once for each year between those values. Because of some strangely-dated data in there, that means nearly 10,000 index probes, but that’s still much faster than scanning the entire table.

Fun with SQL

2009-10-10T00:00:00+00:00

Many programmers, I expect, have a favorite obscure language or two they’d like to see in wider use. Haskell has quite a following, though it sees relatively little use; the same can be said for most pure functional languages. Prolog seemed like a neat idea when I first read about it, but I’ve never heard of anyone using it for something serious (caveat: there are lots of things I’ve never heard of).

My own favorite underused language is SQL. Although most programmers have at least a passing familiarity with SQL, and many use it daily, few seem to achieve real SQL fluency. This is unfortunate; SQL databases are powerful and ubiquitous tools, and even the least among them can generally manage a great deal more than the fairly simple uses to which they are commonly put.

I recently had the opportunity to trot out this curmudgeonly opinion of mine at the Utah Open Source Conference. Now in its third year, this annual conference continues to surprise me. Utah is home to an extraordinarily vibrant open source community, and it shows, in the attendance (over 400 expected), the number of presentations (92 by my count of the schedule, perhaps minus a few that aren’t really presentations, such as the Geek Dinner), and the overall quality and diversity of information presented.

Conference presentations were all recorded, and will eventually be available from the UTOSC website. Most speakers, in the meantime, have made their slides available. Mine are here.

Testing user-defined functions with pgTAP

Functions:

has_function()

functions_are()

ok() and is()

Testing more complex user-defined stored functions:

How To Set Up pgTAP for Writing PostgreSQL Database Unit Tests

Installing & setting up pgTAP

Writing simple database tests with pgTAP

More pgTAP tests

Test to check if a trigger is triggered

Test to check if a constraint is enforced

Organizing and naming your test files based on Test Anything Protocol (TAP)

Running your tests using pg_prove

pgTAP in EpiTrax

Conclusion

Using pgTAP to automate database testing

Databases need tests too

What to test?

Iteratively improving

Looking back and ahead

Database Design: Using Documents

Example

Normalized Tables

Table Sizes

Composite Types

JSON Document Column

Binary JSON

Size Comparison

Usability

String Processing in SQL, the Hard Way

How to use regular expression group quantifiers in PostgreSQL

Regex pattern in grep

PostgreSQL regex attempt

PL/Perl function

PostgreSQL regex solution

Reference

Ecommerce customer names with interesting Unicode characters

Real names with bonuses

Fake names

Not names at all!

Strange addresses

Simply odd

“Interesting” Unicode ranges

The SQL query

Database Design: Using Composite Keys

Single-column vs. Composite

Relations with Composite Keys

Ease of Change

Querying

Performance Considerations

Database Design: Using Natural Keys

Natural or Surrogate

Good Examples of Natural Keys

Enterprise Architecture

Performance Consideration

Composite Keys

Using CTEs to do a binary search of large tables with non-indexed correlated data

How to Migrate from Microsoft SQL Server to PostgreSQL

Schema Migration

Export SQL Server Database Schema

Convert Schema from SQL Server to Postgres

Setup Postgres Database:

Data Migration

Pentaho Data Integration

PDI Job Execution

Migrate Views

Migrate Functions

Most Common Errors

Result

Building Rasters in PostGIS

Rasters

The Data

Creating the Image

Finding Polygons

Counting schools

Color assignment

Packaging the result

Systematic Query Building with Common Table Expressions

Temporary tables in SQL query optimization

`has_function()`

`functions_are()`

`ok()` and `is()`

Squeel—The missing query building DSL

But if I were to use SQL functions—I’d have to write queries in strings, no?

Why use a DSL to generate SQL?—Composability