Creating API for Invoice Generator Using C# and Minimal APIs

2025-07-02T00:00:00+00:00

In this blog post, we will explore how to create a robust invoice generator API using .NET 9 and minimal APIs, with a focus on integrating it with an existing Vue frontend whose implementation we covered in a previous blog post.

We will use the minimal APIs framework to create a RESTful API that can be easily used by our Vue application. Minimal APIs require less boilerplate code and configuration compared to traditional controller-based approaches. It is suited for smaller APIs, microservices, or serverless functions. You can learn more about choosing between controller-based APIs and minimal APIs here.

Prerequisites

Before going further into the code, we should make sure to have the following tools and technologies installed:

.NET 9 (the latest version of the .NET runtime)

Additionally, we already have an existing Vue frontend implementation that lacks API connectivity from our previous post. We will guide you through the process of integrating the Invoice Generator API with Vue application.

Setting up the Project

We will use the dotnet command to create a new .NET 9 project and set up the minimal APIs framework using the steps below:

Create a new .NET Core Web API project using the following command:

dotnet new webapi -n invoice-generator-api

Navigate to the newly created project directory:

cd invoice-generator-api

Open the Program.cs file which should look like below.

var builder = WebApplication.CreateBuilder(args);

// Add services to the container.
// Learn more about configuring OpenAPI at https://aka.ms/aspnet/openapi
builder.Services.AddOpenApi();

var app = builder.Build();

// Configure the HTTP request pipeline.
if (app.Environment.IsDevelopment())
{
    app.MapOpenApi();
}

app.UseHttpsRedirection();

var summaries = new[]
{
    "Freezing", "Bracing", "Chilly", "Cool", "Mild", "Warm", "Balmy", "Hot", "Sweltering", "Scorching"
};

app.MapGet("/weatherforecast", () =>
{
    var forecast =  Enumerable.Range(1, 5).Select(index =>
        new WeatherForecast
        (
            DateOnly.FromDateTime(DateTime.Now.AddDays(index)),
            Random.Shared.Next(-20, 55),
            summaries[Random.Shared.Next(summaries.Length)]
        ))
        .ToArray();
    return forecast;
})
.WithName("GetWeatherForecast");

app.Run();

record WeatherForecast(DateOnly Date, int TemperatureC, string? Summary)
{
    public int TemperatureF => 32 + (int)(TemperatureC / 0.5556);
}

We will also add the following packages for Swagger UI, Entity Framework, and the Entity Framework Core provider for PostgreSQL.

dotnet add package Microsoft.Extensions.ApiDescription.Server
dotnet add package Swashbuckle.AspNetCore.SwaggerUi
dotnet add package Microsoft.EntityFrameworkCore
dotnet add package Npgsql.EntityFrameworkCore.PostgreSQL
dotnet add package Microsoft.EntityFrameworkCore.Design

Setting up the Swagger UI

Update Program.cs and configure Swagger UI to point to the OpenAPI JSON file.

 if (app.Environment.IsDevelopment())
 {
     app.MapOpenApi();
+    app.UseSwaggerUI(config =>
+    {
+        config.SwaggerEndpoint("/openapi/v1.json", "v1");
+    });
 }

 app.UseHttpsRedirection();

Once the application is running using dotnet run, you can access Swagger UI at http://localhost:<PORT>/swagger on your browser, which should look like below.

Setting up the Database Context with Entity Framework

Create a file, InvoiceDbContext.cs, in the root of the project and add the following code.

using Microsoft.EntityFrameworkCore;

public class InvoiceDbContext : DbContext
{
    public InvoiceDbContext(DbContextOptions<InvoiceDbContext> options)
     : base(options)
    {

    }
}

Update the Program.cs file to include the configuration for connecting to the PostgreSQL database using Entity Framework.

+using Microsoft.EntityFrameworkCore;
+
 var builder = WebApplication.CreateBuilder(args);

+builder.Services.AddDbContext<InvoiceDbContext>(options =>
+    options.UseNpgsql(builder.Configuration.GetConnectionString("InvoiceDbcontext")));
+
+

Add the connection string InvoiceDbcontext in the appsettings.json file as shown below:

       "Microsoft.AspNetCore": "Warning"
     }
   },
-  "AllowedHosts": "*"
+  "AllowedHosts": "*",
+  "ConnectionStrings": {
+    "InvoiceDbcontext": "Host=<HOSTNAME>;Port=<PORT>;Database=<DATABASE>;Username=<USERNAME>;Password=<PASSWORD>;"
+  }
 }

Designing the Invoice Generator API

The next step is to design the API endpoints required for invoice generation. We will need the following endpoints:

GET /api/invoice — Get list of invoices
POST /api/invoice — Create a new invoice
GET /api/invoice/{id} — Get an invoice using ID
PUT /api/invoice/{id} — Update an invoice using ID
DELETE /api/invoice/{id} — Soft delete an invoice using ID
PUT /api/invoice/{id}/upload-logo — upload logo to saved invoice using ID
DELETE /api/invoice/{invoiceId}/detail/{id} — Soft delete invoice line item using line item ID and invoice ID

We will also need data models and entities to represent invoices and invoice line items. Let’s create the following classes in the model folder:

public class Base
{
    public int Id { get; set; }
    public DateTime CreatedAt { get; set; } = DateTime.UtcNow;
    public DateTime? ModifiedAt { get; set; }
    public bool IsDeleted { get; set; } = false;
    public DateTime? DeletedAt { get; set; }
}

public class Invoice : Base
{
    public string? Name { get; set; }
    public string? logo { get; set; }
    public string? Number { get; set; }
    public string? PONumber { get; set; }
    public virtual List<InvoiceItem> InvoiceItems { get; set; } = new();
    public UsageAmount? DiscountAmount { get; set; }
    public UsageAmount? TaxAmount { get; set; }
    public UsageAmount? ShippingAmount { get; set; }
    public decimal TotalAmount { get; set; }
    public decimal PaidAmount { get; set; }
    public string? Sender { get; set; }
    public string? Buyer { get; set; }
    public Status? Status { get; set; }
    public string? Notes { get; set; }
    public string? Terms { get; set; }
    public DateTime? Date { get; set; }
    public DateTime? DueDate { get; set; }
    public bool SentToBuyer { get; set; } = false;
}

public class InvoiceItem : Base
{
    public string Description { get; set; } = string.Empty;
    public int Quantity { get; set; }
    public decimal Rate { get; set; }
    public decimal Amount { get => this.Quantity * this.Rate; }
}

public class UsageAmount : Base
{
    public bool IsUsed { get; set; } = false;
    public bool IsPercentage { get; set; } = false;
    public decimal Value { get; set; } = 0;
}

Will will now use these classes to create tables in the database and then define a DBSet for each class in our InvoiceDbContext.cs.

 public InvoiceDbContext(DbContextOptions<InvoiceDbContext> options)
  : base(options)
 {

 }
+    public DbSet<Invoice> Invoices => Set<Invoice>();
+    public DbSet<InvoiceItem> InvoiceItems => Set<InvoiceItem>();
+    public DbSet<UsageAmount> UsageAmounts => Set<UsageAmount>();
 }

Since we will be using the soft delete feature in our API, we will use Global Query Filters to filter out the deleted records from our table. So, we configure the query filters in OnModelCreating using the HasQueryFilter API in InvoiceDbContext.cs. After which, the file should look like below.

 public InvoiceDbContext(DbContextOptions<InvoiceDbContext> options)
  : base(options)
 {

 }
+    protected override void OnModelCreating(ModelBuilder modelBuilder)
+    {
+        modelBuilder.Entity<Invoice>().HasQueryFilter(p => !p.IsDeleted);
+        modelBuilder.Entity<InvoiceItem>().HasQueryFilter(p => !p.IsDeleted);
+        modelBuilder.Entity<UsageAmount>().HasQueryFilter(p => !p.IsDeleted);
+    }
+
+    public DbSet<Invoice> Invoices => Set<Invoice>();
+    public DbSet<InvoiceItem> InvoiceItems => Set<InvoiceItem>();
+    public DbSet<UsageAmount> UsageAmounts => Set<UsageAmount>();
 }

Creating and running database migrations

We have defined models for our data models, but we need to create a database and tables for them. We can do this using Entity Framework tool dotnet ef. On the command prompt, in the root path of project, run the following command:

dotnet ef migrations add "create table and columns"

This command will create migration files in the Migrations folder. We can run this migration using the following command:

dotnet ef database update

Implementing API Endpoints

Now that we have designed the API endpoints and data models, let’s implement them using minimal APIs. We will create a new endpoints folder and add an InvoiceEndpoints.cs file in it. This file will contain the implementation for the Invoice API endpoints.

// InvoiceEndopints.cs

using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.Hosting.Internal;

public static class InvoiceEndpoints
{
    public static List<Invoice> Invoices = new();
    public static RouteGroupBuilder MapInvoiceEndpoints(this RouteGroupBuilder group)
    {
      // Return list of invoices
        group.MapGet("/", async (InvoiceDbContext db) =>
        {
            return await db.Invoices.Include(x => x.InvoiceItems).ToListAsync();
        })
            .WithSummary("Get list of invoices");

        // Return an existing invoice
        group.MapGet("/{id}", async (int id, InvoiceDbContext db) =>
        {
            return await db.Invoices.Include(x => x.InvoiceItems).FirstOrDefaultAsync(x => x.Id == id)
            is Invoice invoice
                ? Results.Ok(invoice)
                : Results.NotFound();
        })
            .WithSummary("Get an invoice using id");

        // Create an invoice
        group.MapPost("/", async (Invoice invoice, InvoiceDbContext db) =>
        {
            invoice.DueDate = invoice.DueDate.HasValue ? invoice.DueDate.Value.ToUniversalTime() : null;
            invoice.Date = invoice.Date.HasValue ? invoice.Date.Value.ToUniversalTime() : null;
            db.Invoices.Add(invoice);
            await db.SaveChangesAsync();
            return Results.Created($"/api/invoice/{invoice.Id}", invoice);
        })
            .WithSummary("Create a new invoice");

        // Create a new line item for an existing invoice
        group.MapPost("/{id}/items", async (int id, List<InvoiceItem> invoiceItems, InvoiceDbContext db) =>
        {
            if (await db.Invoices.FindAsync(id)
            is not Invoice invoice)
            {
                return Results.NotFound();
            }
            invoice.InvoiceItems.AddRange(invoiceItems);
            await db.SaveChangesAsync();
            return Results.Created($"/api/invoice/{invoice.Id}", invoice);
        })
            .WithSummary("Create new line items for invoice");

        // Update an existing invoice
        group.MapPut("/{id}", async (int id, Invoice inputInvoice, InvoiceDbContext db) =>
        {
            var invoice = await db.Invoices.AsNoTracking().FirstOrDefaultAsync(x => x.Id == id);

            if (invoice is null) return Results.NotFound();

            invoice.Id = id;
            db.Invoices.Update(inputInvoice);
            await db.SaveChangesAsync();
            return Results.NoContent();
        })
            .WithSummary("Update invoice");

        // Upload invoice logo image file for an existing invoice
        group.MapPut("/{id}/upload-logo", async (int id, IFormFile uploadImageFile, InvoiceDbContext db) =>
        {
            var invoice = await db.Invoices.AsNoTracking().FirstOrDefaultAsync(x => x.Id == id);

            if (invoice is null) return Results.NotFound();

            if (uploadImageFile == null)
            {
                throw new ArgumentNullException(nameof(uploadImageFile));
            }

            var contentPath = Environment.CurrentDirectory;
            var uploadPath = Path.Combine("Uploads", id.ToString());
            var rootPath = Path.Combine(contentPath, uploadPath);

            if (!Directory.Exists(rootPath))
            {
                Directory.CreateDirectory(rootPath);
            }
            string[] allowedFileExtensions = [".jpg", ".jpeg", ".png"];
            var ext = Path.GetExtension(uploadImageFile.FileName);
            if (!allowedFileExtensions.Contains(ext))
            {
                throw new ArgumentException($"Please upload only .jpg, .jpeg, and .png images.");
            }

            // generate a unique filename
            var fileName = $"{Guid.NewGuid().ToString()}{ext}";
            var fileNameWithPath = Path.Combine(uploadPath, fileName);
            var rootPathWithFileName = Path.Combine(rootPath, fileName);
            using var stream = new FileStream(rootPathWithFileName, FileMode.Create);
            await uploadImageFile.CopyToAsync(stream);

            invoice.logo = fileNameWithPath;
            db.Update(invoice);
            await db.SaveChangesAsync();
            return Results.NoContent();
          })
          .DisableAntiforgery()
          .WithSummary("upload logo");

        // soft delete an invoice
        group.MapDelete("/{id}", async (int id, InvoiceDbContext db) =>
        {
            if (await db.Invoices.FindAsync(id) is Invoice todo)
            {
                todo.IsDeleted = true;
                todo.DeletedAt = DateTime.Now.ToUniversalTime();
                await db.SaveChangesAsync();
                return Results.NoContent();
            }
            return Results.NotFound();
        })
            .WithSummary("Soft delete an invoice");

        return group;
    }
}

Then we will also implement InvoiceDetailEndpoint.cs. We will only define a delete endpoint for soft deleting the line items. We created this also to demonstrate how to use multiple files in the Program.cs to map the endpoints to a certain URL.

// InvoiceDetailEndpoint.cs

public static class InvoiceDetailEndpoints
{
    public static RouteGroupBuilder MapInvoiceDetailEndpoints(this RouteGroupBuilder group)
    {
        group.MapDelete("/{id}", async (int id, InvoiceDbContext db) =>
        {
            if (await db.InvoiceItems.FindAsync(id) is InvoiceItem invoiceItem)
            {
                invoiceItem.IsDeleted = true;
                invoiceItem.DeletedAt = DateTime.Now.ToUniversalTime();
                await db.SaveChangesAsync();
                return Results.NoContent();
            }
            return Results.NotFound();
        }).WithSummary("Soft delete invoice line item");

        return group;
    }
}

Similar to how traditional controller APIs are separated in their own files, we now have the implementation for the endpoints in two files. We will now register and map the endpoints to a preferred URL using MapGroup in the Program.cs file.

 // Program.cs

+var root = app.MapGroup("api");
+var invoice = root.MapGroup("invoice").MapInvoiceEndpoints();
+var invoiceDetail = root.MapGroup("invoice/{invoiceId}/detail").MapInvoiceDetailEndpoints();
+
 app.Run();

We also need to add a config to expose the Uploads folder so the images can be loaded from the running API server. Also, make sure to create the Uploads folder before running the application, otherwise the application wouldn’t run.

 // Program.cs

 var invoice = root.MapGroup("invoice").MapInvoiceEndpoints();
 var invoiceDetail = root.MapGroup("invoice/{invoiceId}/detail").MapInvoiceDetailEndpoints();

+app.UseStaticFiles(new StaticFileOptions
+{
+    FileProvider = new PhysicalFileProvider(
+        Path.Combine(builder.Environment.ContentRootPath, "Uploads")),
+    RequestPath = "/Uploads"
+});
+
 app.Run();

Adding CORS for development

While working locally we usually run the frontend application and the API application in different ports, so CORS is required for the frontend application to access the API. To enable CORS in development mode, we can add the following code snippet to our Program.cs file:

 // Program.cs

 using Microsoft.EntityFrameworkCore;
 using Microsoft.Extensions.FileProviders;

 var builder = WebApplication.CreateBuilder(args);
+var MyAllowSpecificOrigins = "_myAllowSpecificOrigins";
+builder.Services.AddCors(options =>
+{
+    options.AddPolicy(name: MyAllowSpecificOrigins,
+                      policy =>
+                      {
+                          policy.AllowAnyOrigin();
+                          policy.AllowAnyHeader();
+                          policy.AllowAnyMethod();
+                      });
+});

 builder.Services.AddDbContext<InvoiceDbContext>(options =>
     options.UseNpgsql(builder.Configuration.GetConnectionString("InvoiceDbcontext")));

 app.MapGet("/weatherforecast", () =>
 var root = app.MapGroup("api");
 var invoice = root.MapGroup("invoice").MapInvoiceEndpoints();
 var invoiceDetail = root.MapGroup("invoice/{invoiceId}/detail").MapInvoiceDetailEndpoints();

+app.UseCors(MyAllowSpecificOrigins);
 app.UseStaticFiles(new StaticFileOptions
 {
     FileProvider = new PhysicalFileProvider(

Now, we can start the application and test our API. When we go to /swagger, we can see our API endpoints.

The source code for the API is available on GitHub.

Integrating with Vue 3 Application

We already created a Vue 3 application in the previous blog. Now, we will integrate our API with the Vue application.

To integrate the Invoice Generator API with our Vue 3 application, we will need to configure API calls from the frontend to interact with the API. We will follow the following steps:

Install the axios package in our Vue 3 project:

npm install axios

Create a new file called index.ts in the services folder of our Vue 3 project and add the following code. Also, add .env file with key VITE_API_URL and value of the API URL:

// services/index.ts

import axios from 'axios'
export default axios.create({
    baseURL: import.meta.env.VITE_API_URL,
    timeout: 5000,
});

# .env
VITE_API_URL=https://localhost:7041/api/
VITE_BACKEND_URL=https://localhost:7041/

Create a new file called invoiceService.ts in the services folder of your Vue 3 project. Add the methods that calls the API endpoints:

import { Invoice, InvoiceItem } from "../types/index.type"
import axios from "./"

export default {
    getInvoices: async () => {
        return await axios.get('invoice')
    },
    getInvoice: async (id) => {
        return await axios.get(`invoice/${id}`)
    },
    saveInvoice: async (data: Invoice) => {
        return await axios.post('invoice', data)
    },
    deleteInvoice: async (id: number) => {
        return await axios.delete(`invoice/${id}`)
    },
    updateInvoice: async (id: number, data: Invoice) => {
        return await axios.put(`invoice/${id}`, data)
    },
    uploadLogo: async (invoiceId, file) => {
        const formData = new FormData();
        formData.append("imageFile", file);
        return axios.put(`invoice/${invoiceId}/upload-logo`, formData, { headers: { "Content-Type": "multipart/form-data" } });
    },
    addLineItem: async (id: number, data: InvoiceItem) => {
        return await axios.post(`invoice/${id}/detail/`, data);
    },
    updateLineItem: async (id: number, lineId: number, data: InvoiceItem) => {
        return await axios.put(`invoice/${id}/detail/${lineId}`, data)
    },
    deleteLineItem: async (id: number, lineId: number) => {
        return await axios.delete(`invoice/${id}/detail/${lineId}`)
    },
}

We will use invoiceService.ts to make API calls in our application. Normally, the API calls are usually made from Vue 3 components, but in this case we will use it to make API calls from our composable useInvoiceStorage.ts where all the business logic related to invoice are handled. Below are the code changes done to integrate the API calls from invoiceService.ts.

 // composables/useInvoiceStorage.ts

 import { useStorage } from '@vueuse/core'
-import { reactive } from 'vue';
+import { computed, ref } from 'vue';
 import { Invoice, InvoiceStore } from '../types/index.type';
+import invoiceService from '../services/invoiceService';

 const storage = useStorage<InvoiceStore>('invoice-store', {
-    currentInvoiceNumber: 10000,
-    invoices: {}
+    currentInvoiceNumber: "10000",
 }, localStorage);

+const state = ref({
+    ...storage.value,
+    invoices: {},
+});
 export default function useState() {
-    const state = reactive(storage);

     function saveInvoice(invoice: Invoice) {
         state.value.invoices[invoice.number] = { ...invoice };
-        state.value.currentInvoiceNumber = invoice.number + 1;
+        state.value.currentInvoiceNumber = `${+invoice.number + 1}`;
+        invoiceService.saveInvoice(invoice);
     }

-    function updateInvoice(invoice: Invoice) {
+    function updateInvoice(id: number, invoice: Invoice) {
         state.value.invoices[invoice.number] = { ...invoice };
+        invoiceService.updateInvoice(id, invoice);
     }

-    function deleteInvoice(invoiceId: number) {
+    async function deleteInvoice(invoiceId: number) {
         // delete state.value.invoices[invoiceId];
+        await invoiceService.deleteInvoice(invoiceId);
         state.value.invoices = Object.keys(state.value.invoices).filter(x => x != invoiceId.toString()).reduce((acc, curr) => {
             return {
                 ...acc,
@@ -40,15 +45,33 @@ export default function useState() {
         })
             .map(x => ({ ...state.value.invoices[x] }))

+
+    }
+
+    async function fetchInvoices() {
+        const result = await invoiceService.getInvoices();
+        state.value.invoices = result.data.reduce((acc, curr) => {
+            return { ...acc, [curr.id]: curr }
+        }, {})
+    }
+    async function fetchInvoice(id) {
+        const result = await invoiceService.getInvoice(id);
+        console.log(result);
+        state.value.invoices[result.data.id] = result.data;
+        console.log(state.value.invoices);
+        return result.data;
     }
     return {
         // variables
         storage: state,
+        invoices: computed(() => state.value.invoices),

         // methods
         saveInvoice,
         updateInvoice,
         deleteInvoice,
-        findInvoices
+        findInvoices,
+        fetchInvoices,
+        fetchInvoice
     }
 }

The above fetchInvoices method is then used by Invoices.vue to fetch the invoices. Similarly, all other endpoints are used in the application for CRUD operations.

The source code for the Vue app is available on GitHub.

Deployment

To deploy our API to Azure App Service, we can follow the documentation provided by Microsoft here.

Conclusion

In this blog post, we explored how to create a robust invoice generator API using .NET 9 and minimal APIs. We designed the API endpoints, implemented them using minimal APIs, and integrated the API with an existing Vue 3 application. This approach allowed us to leverage the power of minimal APIs while maintaining a clean and modular codebase.

Testing user-defined functions with pgTAP

2023-09-25T00:00:00+00:00

For a great introduction to the pgTAP protocol, please read over my colleague Josh Tolley’s article, using pgTAP to automate database testing. Also check out Edgar Mlowe’s article on how to set up pgTAP for writing PostgreSQL database unit tests.

As a protocol, pgTAP provides a great list of functions and commands to easily and clearly test all aspects of a database. As I’ve been tasked with more implementations of unit tests for user-defined functions, I thought it helpful to share some useful methods.

First we’ll go over a few basic pgTAP functions that are useful in testing the existence of functions and procedures. Then we’ll use the PostgreSQL port of the Sakila sample database for MySQL, called Pagila. We’ll construct two functions and go over a single test case. Using basic Postgres tools, I’ll walk through a few methods of calling and testing our functions. Finally, we’ll go over a more concise way of testing all cases for said functions.

Feel free to work alongside this article to hopefully leave with a comfortable idea of how to construct unit test files for your database.

Functions:

For this first example, remember to have the pgTAP extension in your schema and to perform it in a transaction (I’ve omitted BEGIN and ROLLBACK for brevity). Also use SELECT plan(<number of tests you’ll run>) when running the pgTAP functions.

Starting off, let’s create a basic addition function that returns an integer value:

CREATE OR REPLACE FUNCTION add_numbers(
    num1 integer,
    num2 integer
)
RETURNS integer
LANGUAGE plpgsql AS $$
BEGIN
    RETURN num1 + num2;
END;
$$;

To test this user-defined function, a few of the most relevant functions in pgTAP we can use are:

`has_function()`

Variations:

SELECT has_function( :schema, :function, :args, :description );
SELECT has_function( :schema, :function, :args );
SELECT has_function( :schema, :function, :description );
SELECT has_function( :schema, :function );
SELECT has_function( :function, :args, :description );
SELECT has_function( :function, :args );
SELECT has_function( :function, :description );
SELECT has_function( :function );

Parameters:

:schema
    Name of a schema in which to find the function.
:function
    Name of a function.
:args
    Array of data types of the function arguments.
:description
    A short description of the test.

Example:

SELECT has_function('your_schema', 'add_numbers', ARRAY['integer','integer']); -- default description used

Results:

ok 1 - Function add_numbers(integer, integer) should exist

has_function() is useful in checking the existence of a specific function. If you want to parse an entire schema, functions_are() is your command.

`functions_are()`

Variations:

SELECT functions_are( :schema, :functions, :description );
SELECT functions_are( :schema, :functions );
SELECT functions_are( :functions, :description );
SELECT functions_are( :functions );

Parameters:

:schema
    Name of a schema in which to find functions.
:functions
    An array of function names.
:description
    A short description of the test.

Example:

SELECT functions_are('your_schema', ARRAY['add_numbers', <All other functions need to be listed as well>]); -- default description used

`ok()` and `is()`

ok() and is() have similar functionality with slightly different parameters:

SELECT ok( :boolean, :description );

Example:

SELECT ok(add_numbers(5,5)=10, 'add_numbers correctly works');

Parameters:

:boolean
    A boolean value indicating success or failure.
:description
    A short description of the test.

While ok() is used for passing a test returning a boolean, is() takes two values to compare:

SELECT is( :have, :want, :description );

Example:

SELECT is(add_numbers(5,5), 10, 'add_numbers correctly works');

You can find functions that cover pretty much anything in PostgreSQL here in pgTAP’s documentation.

Testing more complex user-defined stored functions:

A client product we created utilizes a function that calls several other functions depending on its given argument. This function is called by the application and is passed an integer as an argument from which the function bases an initial SELECT query’s WHERE condition on, in this case WHERE id = [argument]. Variables are then assigned column values from this queried table. These variables are then used as the conditions to PERFORM functions that change parts of the database.

When testing this sort of scenario, it became apparent that we needed a set of test data, as well as the ability to track that test data, run the function with the data, and test to see if the changes are performed correctly.

The nice thing about pgTAP is that unit tests are meant to be rolled back leaving the database as you found it. When faced with a more complex function that nests other functions which loop through multiple tables, it may make more sense to fill those tables temporarily with the data required for the tests.

Let’s take a nostalgia trip and imagine we’re operating a DVD rental store. The Pagila sample database, which you can download on GitHub, represents our scenario. It comes populated and we’ll construct a function which utilizes other functions to perform changes on multiple tables.

Say the DVD rental shop gives extensions of rentals for special occasions, one being that a DVD was rented on Christmas week. Suppose we have an initial function that checks if a new DVD is rented on one of these special occasions. It then calls another function that will extend the return date, based on the specific occasion:

CREATE OR REPLACE FUNCTION extension_check(_rental_id integer)
    RETURNS void
    LANGUAGE plpgsql
AS $function$
DECLARE
    _rental_date date;
    _inventory_id integer;
    _customer_id integer;
    _return_date date;
    christmas_week date;
BEGIN

    SELECT INTO _rental_date
    rental_date FROM rental
    WHERE rental_id = _rental_id;

    IF EXTRACT(YEAR FROM _rental_date) = EXTRACT(YEAR FROM CURRENT_DATE) AND
       _rental_date <= (EXTRACT(YEAR FROM CURRENT_DATE)::text || '-12-25')::date AND
       _rental_date >  (EXTRACT(YEAR FROM CURRENT_DATE)::text || '-12-25')::date - INTERVAL '7 days' THEN
        PERFORM christmas_extension(_rental_id);
    END IF;
END;
$function$;

CREATE OR REPLACE FUNCTION christmas_extension(_rental_id integer)
    RETURNS void
    LANGUAGE plpgsql
AS $function$
DECLARE
    _rental_date date;
    _return_date date;
BEGIN
    SELECT return_date FROM rental WHERE rental_id = _rental_id INTO _return_date;

    IF _return_date IS NULL THEN
        UPDATE rental SET return_date = rental_date + INTERVAL '10 days' WHERE rental_id=_rental_id;
    ELSE
        UPDATE rental SET return_date = return_date + INTERVAL '10 days' WHERE rental_id=_rental_id;
    END IF;
END;
$function$;

The initial function, extension_check, simply checks if the rental was checked out on the week prior to and on Christmas (It’s a really dedicated DVD shop). We can also assume a function like this has other scenarios where it would extend rental dates but we’ll just show one for this exercise; christmas_extension is our one extension function that extends the return date by ten days.

To hold our unit test, we’ll create a file fittingly named extension_unit_tests.sql. A problem with the current sample database is that we can’t really test out the function directly since the dates don’t extend to the current year. Rather even if they did, we don’t want to directly affect the existing data especially if the dates were updated and the database was in production use.

Relatively basic but handy Postgres tools let us create test cases which we can pass to the functions. Afterwards, we can use pgTAP functions to confirm that our user-defined functions extension_check and christmas_extension worked properly.

For test cases in our new file, extension_unit_tests.sql, we’ll need a way to create and track them. For this, we’ll use a temporary table to track the test IDs and a PROCEDURE which can be called at any point to create the test cases. After that, we’ll simply call the procedure and run the user-defined function extension_check. Finally test its existence, test its result, and roll back our changes.

BEGIN;

CREATE TEMPORARY TABLE test_instance_references (
    test_rental_id integer PRIMARY KEY,
    test_number integer
);

CREATE OR REPLACE PROCEDURE fill_test_data (
    _rental_date date,
    _test_number integer
)
LANGUAGE PLPGSQL
AS $procedure$
DECLARE
    _address_id integer;
    _store_id integer;
    _staff_id integer;
    _inventory_id integer;
    _customer_id integer;
    _rental_id integer;
BEGIN

    SELECT nextval('customer_customer_id_seq1'::regclass) INTO _customer_id;
    SELECT nextval('rental_rental_id_seq'::regclass) INTO _rental_id;
    SELECT nextval('inventory_inventory_id_seq'::regclass) INTO _inventory_id;
    SELECT store_id FROM store LIMIT 1 INTO _store_id; -- any store will do
    SELECT address_id FROM address LIMIT 1 INTO _address_id; -- any address will do

    INSERT INTO customer (customer_id, store_id, first_name, last_name, address_id)
    VALUES (
        _customer_id,
        _store_id,
        'Test First Name',
        'Test Last Name',
        _address_id
    );

    INSERT INTO inventory (inventory_id, film_id, store_id)
    VALUES (
        _inventory_id,
        (SELECT film_id FROM film LIMIT 1), -- arbitrary film
        _store_id
    );

    INSERT INTO rental (rental_id, rental_date, inventory_id, customer_id, staff_id)
    VALUES (
        _rental_id,
        _rental_date,
        _inventory_id,
        _customer_id,
        (SELECT staff_id FROM staff WHERE store_id = _store_id LIMIT 1) -- select arbitrary staff member from store
    );

    INSERT INTO test_instance_references (test_rental_id, test_number)
    values (
        _rental_id,
        _test_number
    );

END;
$procedure$;

CALL fill_test_data('2023-12-25'::date, 1); -- Fill the db with our test data

SELECT extension_check((SELECT test_rental_id FROM test_instance_references WHERE test_number = 1)); -- Call our user-defined function

SELECT plan(2); -- State the number of tests we'll be performing

SELECT has_function(
    'extension_check',
    ARRAY['int']
);

SELECT is(
    (SELECT return_date FROM rental WHERE rental_id in (SELECT test_rental_id FROM test_instance_references WHERE test_number = 1))::date,
    ('2023-12-25'::date + INTERVAL '10 DAYS')::date,
    'Checking test data 1, week of Christmas'
);


ROLLBACK;

By design, pgTAP tests are performed in a transaction block with a ROLLBACK at the end to reverse all changes. Our first transaction is the creation of a temporary table called test_instance_references which will hold the IDs of the sample data we insert specifically in the rental table. This way we keep track of our specific test cases through their primary identifier.

Our PROCEDURE works similarly to a function, but where a function may have issues being called in a transaction block through pgTAP’s command line pg_prove, the use of a procedure and its CALL command is far more reliable.

In our example, the procedure simply adds in all the required test data. Note the insertion of test data into other tables that are referenced in rental, as well as an insert into the temporary table for tracking. To avoid using any existing data in the database, references are also test samples, like for the customer. Some, however, are selected from existing tables. It’s important to make sure you’re not affecting real data in the database in case a transaction fails. The example is obviously lax when it comes to that, specifically with the references to store and address, so please take note in your own unit tests to be careful when referring to existing data.

After we’ve established our procedure, we’re ready to run the tests. The pgTAP function plan() defines how many unit tests we are going to run. For the first example we run two tests.

Before running tests, we call the procedure fill_test_data which populates the database with test data and the temporary table to track that data.

Our first test is to check if the function exists. pgTAP’s has_function function is perfect for this.

Our procedure is then called with a date for the test rental. This test rental’s ID is saved into the temporary test_instance_references table which is then passed to the main function we want to test, extension_check. We call this function with the SELECT command, and immediately test the outcome with pgTAP’s is(). Our :have parameter holds the return_date of our test rental which is tested to see if it’s been correctly extended by 10 days. Our final parameter is the description which will be returned in the outcome of pg_prove if the test fails.

The ROLLBACK is meant to clean the database of all test data.

Let’s run this unit test on the command line, with pg_prove:

pg_prove -d <your_database> <path_to_your_unit_test>/extension_unit_tests.sql

We should see these results:

extension_unit_tests.sql .. ok
All tests successful.
Files=1, Tests=2,  0 wallclock secs ( 0.01 usr +  0.00 sys =  0.01 CPU)
Result: PASS

Congrats! We’ve tested out a single case for a function.

However, what if we want to test all possible cases for this function to be sure it performs correctly?

To do this, we can alter the procedure fill_test_data to loop through a date range given in the arguments _start_date and _end_date. For our example we’ll provide it with the week of Christmas, but it’ll work for any date range provided the rental store gives extensions for DVDs rented on other dates.

Instead of passing a test number as an argument, we can simply initialize it as a variable that’s incremented with every loop. Inside the WHILE loop, every iteration will populate the database with the test data, perform extension_check with the test rental ID, and increment the date and test number.

We’ll end the transaction a little differently by setting our pgTAP plan equal to the number of tests we have in our temporary table, to avoid hard-coding our plan count. We’ll also add one to check its existence with has_function.

Finally, our is() will also iterate through our temporary table and perform a check in the rental table to confirm that the return date has been correctly extended by ten days.

BEGIN;

CREATE TEMPORARY TABLE test_instance_references (
    test_rental_id integer PRIMARY KEY,
    test_number integer,
    start_date date
);


CREATE OR REPLACE PROCEDURE fill_test_data (
        _start_date date,
        _end_date date
    )
    LANGUAGE PLPGSQL
    AS $procedure$
    DECLARE
        _address_id smallint;
        _store_id smallint;
        _inventory_id integer;
        _customer_id integer;
        _rental_id integer;
        _test_number integer;
    BEGIN
        _test_number := 1;

        WHILE _start_date <= _end_date LOOP
            SELECT nextval('customer_customer_id_seq1'::regclass) INTO _customer_id;
            SELECT nextval('rental_rental_id_seq'::regclass) INTO _rental_id;
            SELECT nextval('inventory_inventory_id_seq'::regclass) INTO _inventory_id;
            SELECT store_id FROM store LIMIT 1 INTO _store_id; -- any store will do
            SELECT address_id FROM address LIMIT 1 INTO _address_id; -- any address will do

            INSERT INTO customer (customer_id, store_id, first_name, last_name, address_id)
            VALUES (
                _customer_id,
                _store_id,
                'Test First Name',
                'Test Last Name',
                _address_id
            );

            INSERT INTO inventory (inventory_id, film_id, store_id)
            VALUES (
                _inventory_id,
                (SELECT film_id FROM film LIMIT 1), -- arbitrary film
                _store_id
            );

            INSERT INTO rental (rental_id, rental_date, inventory_id, customer_id, staff_id)
            VALUES (
                _rental_id,
                _start_date,
                _inventory_id,
                _customer_id::smallint,
                (SELECT staff_id FROM staff WHERE store_id = _store_id LIMIT 1) -- select arbitrary staff member from store
            );

            INSERT INTO test_instance_references (test_rental_id, test_number, start_date)
            VALUES (
                _rental_id,
                _test_number,
                _start_date
            );

            PERFORM extension_check(_rental_id);

            _start_date := _start_date + INTERVAL '1 day';
            _test_number := _test_number + 1;

        END LOOP;
    END;
$procedure$;

CALL fill_test_data(('2023-12-25'::date - INTERVAL '6 days')::date, '2023-12-25'::date);

SELECT plan(count(*)::integer + 1) from test_instance_references;

SELECT * from test_instance_references;

SELECT has_function(
    'extension_check',
    ARRAY['int']
    -- Description optional - default description usually substantial
);

SELECT is(
    (SELECT return_date FROM rental WHERE rental_id = tf.test_rental_id)::date,
    (tf.start_date + INTERVAL '10 DAYS')::date
)
FROM test_instance_references as tf;


ROLLBACK;

Our results should be:

$ pg_prove -d <your_database> <path_to_your_unit_test>/extension_unit_tests.sql
extension_unit_tests.sql .. ok
All tests successful.
Files=1, Tests=8,  0 wallclock secs ( 0.02 usr +  0.00 sys =  0.02 CPU)
Result: PASS

As someone who found pgTAP fairly straightforward the challenge of intertwining those functions in a repeatable way was what drove me to write this article. Hopefully this makes it easier for the next coder to play around and get the most out of this simple but intuitive “Test Anything Protocol”.

Feel free to share your thoughts and comments below, or to share any more useful tips on pgTAP or general Postgres unit testing.

How To Set Up pgTAP for Writing PostgreSQL Database Unit Tests

2023-04-12T00:00:00+00:00

In a previous blog post, my colleague Josh Tolley introduced the concept of using pgTAP, a set of Postgres functions designed for writing unit tests within the database. This post will serve as a supplement to Josh’s post, focusing on the process of setting up pgTAP for testing a Postgres database.

Installing & setting up pgTAP

Note: make sure you have Postgres installed on your system. If you don’t have it, you can follow the Postgres documentation to learn how to install it.

To install pgTAP for Postgres, you will need to follow these steps:

Download the pgTAP source code from its GitHub repository.
```
git clone https://github.com/pgtap/pgtap.git
```
Navigate to the pgtap directory.
```
cd pgtap
```
Install with make.
```
make && make install
```
Finally, connect to your Postgres database using psql or any other Postgres client and run the following SQL command to create the pgTAP extension.
```
CREATE EXTENSION pgtap;
```

Now you should have pgTAP installed and ready to use in your Postgres database. If you face issues with installation, visit the pgTAP documentation for further help.

Writing simple database tests with pgTAP

Test to check if a table exists:

SELECT plan(1);
SELECT has_table('public', 'your_table', 'Table your_table should exist');
SELECT finish();

Test to check if a column exists in a table:

SELECT plan(1);
SELECT has_column('public', 'your_table', 'your_column', 'Column your_column should exist in your_table');
SELECT finish();

Test to check if a function returns the expected result:

SELECT plan(1);
SELECT is(your_function(), expected_result, 'your_function() should return expected_result');
SELECT finish();

More pgTAP tests

Test to check if a trigger is triggered

Consider the following example. We have a table named employees and a table named audit_log. When a new employee is added to the employees table, a trigger named insert_employee_trigger fires and logs the new employee’s ID and creation timestamp in the audit_log table. Let’s test if the trigger gets fired and produces the expected outcome.

Create the tables.

CREATE TABLE employees (
  id SERIAL PRIMARY KEY,
  name VARCHAR(255) NOT NULL
);

CREATE TABLE audit_log (
  id SERIAL PRIMARY KEY,
  employee_id INTEGER NOT NULL,
  created_at TIMESTAMP NOT NULL
);

Create the trigger and trigger function.

CREATE OR REPLACE FUNCTION insert_employee_trigger_function()
RETURNS TRIGGER AS $$
BEGIN
  INSERT INTO audit_log (employee_id, created_at) VALUES (NEW.id, NOW());
  RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER insert_employee_trigger
  AFTER INSERT ON employees
  FOR EACH ROW
  EXECUTE FUNCTION insert_employee_trigger_function();

Now, write a test using pgTAP to test the trigger.

-- Set the test plan
SELECT plan(1);

-- Prepare test data and perform operations that should fire the trigger
INSERT INTO employees (name) VALUES ('John Doe');


-- Check if the desired outcome of the trigger has occurred
SELECT is(
  (SELECT name FROM employees WHERE id = (SELECT employee_id FROM audit_log ORDER BY id DESC LIMIT 1)),
  'John Doe',
  'The insert_employee_trigger should have been fired and produced the expected outcome'
);

Test to check if a constraint is enforced

For this example, we’ll use the following two tables:

customers table with columns id and name.
orders table with columns id, customer_id, and amount
- There is a foreign key constraint between the orders table and the customers table on the customer_id column.

First, create the two tables and add the foreign key constraint.

CREATE TABLE customers (
  id SERIAL PRIMARY KEY,
  name VARCHAR(255) NOT NULL
);

CREATE TABLE orders (
  id SERIAL PRIMARY KEY,
  customer_id INTEGER REFERENCES customers(id),
  amount INTEGER
);

Now, create a test script named fk_test.sql with the following content.

BEGIN;

-- Create test data in customers table
INSERT INTO customers (id, name) VALUES (1, 'Customer 1');

-- Declare the number of tests
SELECT plan(1);

-- Attempt to insert an order with an invalid customer_id
-- This should fail due to the foreign key constraint
SELECT throws_ok(
    'INSERT INTO orders (id, customer_id, amount) VALUES (1, 969, 100);',
    '23503',
    'insert or update on table "orders" violates foreign key constraint "orders_customer_id_fkey"'
);

-- Conclude the test and discard changes
SELECT finish();
ROLLBACK;

Organizing and naming your test files based on Test Anything Protocol (TAP)

Test Anything Protocol (TAP) is a convention for naming test files and organizing them within a project directory structure. The naming convention involves adding a numeric prefix to the test file name to indicate the order in which the tests should be run. For example, 00-test.sql is the first test file to be executed, followed by 01-another-test.sql, and so on.

The project directory structure usually includes a directory called t (short for “test”) under the project root directory. This t directory contains all the test files following the TAP naming convention.

Using this structure makes it easier to maintain and organize tests, as well as to run them in the correct order. This is especially important when tests depend on the results of previous tests, or when you want to ensure that tests are executed in a specific sequence.

Here’s an example of a project directory structure using the TAP-related naming convention:

my_project/
│
├─ src/
│   ├── function1.sql
│   ├── function2.sql
│   └── function3.sql
│
└─ t/
    ├── 00-test.sql
    ├── 01-another-test.sql
    └── 02-yet-another-test.sql

In this example, the project contains a src directory for the source code (database functions) and a t directory for the test files following the TAP naming convention.

Running your tests using pg_prove

Now that we have written some tests using pgTAP, let’s see how we can run them. We’ll use pg_prove, a utility included with pgTAP, to automate the process and get a summary of the results.

To run your tests with pg_prove, you’ll need to specify the name of the database you want to connect to and the path to your test file. For example, if your test file is located at t/00-test.sql and your database is named epitrax_db, you can run the following command:

pg_prove -d epitrax_db t/00-test.sql

When you run this command, pg_prove will execute all the tests in the specified file and print a summary of the results. The output will include the number of tests run, the number of tests passed, the number of tests failed, and any diagnostic messages or errors that were encountered.

Here’s an example of what the output might look like:

t/00-test.sql .. ok
All tests successful.
Files=1, Tests=5,  0 wallclock secs ( 0.01 usr  0.00 sys +  0.02 cusr  0.00 csys =  0.03 CPU)
Result: PASS

In this example, pg_prove ran one test file (00-test.sql) and executed 5 tests. All of the tests passed, so the output shows that the result was PASS.

If any of the tests fail, pg_prove will print a detailed report of the failures, including any error messages or diagnostic information. This can help you quickly identify and fix any issues in your code.

Using pg_prove to automate your tests can save you a lot of time and effort, especially if you have a large number of tests to run. It also makes it easy to integrate your tests into your build process or continuous integration pipeline.

pgTAP in EpiTrax

EpiTrax relies on Postgres and its functions, making pgTAP an ideal tool for testing the platform’s database functions, ensuring data integrity, and maintaining the robustness of the system. By utilizing pgTAP, EpiTrax efficiently validates various aspects of its database structure and behavior.

Conclusion

pgTAP database unit tests provide numerous benefits, including ensuring code quality and simplifying the testing process. This guide should serve as a starting point for setting up pgTAP, and we encourage you to explore the pgTAP documentation and read the aforementioned blog post on automated database testing with pgTAP for more in-depth information on writing and running tests with pgTAP. By integrating pgTAP in projects like EpiTrax, it’s possible to improve overall database performance, stability, and security.

Find Text in Any Column of a PostgreSQL Table

2023-02-06T00:00:00+00:00

It’s not uncommon for me to want to find a particular text snippet in a PostgreSQL database. Easy enough, you might say. After all, that’s what databases are for: You feed them a bunch of information, ask them questions in the form of a query, and they give you the answer. So just write a query, right?

Well, maybe.

SQL stands for “Structured Query Language”, and the fact that it’s “structured” means not only that the database abides by some defined structure, but that your queries do, too, which implies that you know at the time you’re writing the query where in the structure you want to look. And that’s where the problem arises. What if I know “Kilroy” is somewhere in a table, but I don’t know what column to look in to find him? How do I write that query?

The first answer I came up with to that question depends on pg_dump: dump the contents of a table, search the results with grep, and there you have it.

$ pg_dump -t person mydb | grep -i kilroy
633132  F       NH      \N      Cristen212      J    Kilroy44        1983-09-28 00:00:00     \N      t       \N      \N      \N      \N      F       USA  \N       \N      \N      \N      \N      \N      \N      \N

Here I knew I wanted to look in the person table, so I specified that in the call to pg_dump, but this works as well when I want to search the entire database:

$ pg_dump  mydb | grep -i kilroy
716565  public  person  18185   {"session_user": "josh"}        {"id": 633132}  {"last_name": "Feeney44"}       {"last_name": "Kilroy44"}       U       123349  2023-02-03 11:19:40.587883-062023-02-03 11:19:40.587883-06    2023-02-03 11:19:40.592103-06   josh    \N      \N
633132  F       NH      \N      Cristen212      J    Kilroy44        1983-09-28 00:00:00     \N      t       \N      \N      \N      \N      F       USA  \N       \N      \N      \N      \N      \N      \N      \N

I can switch pg_dump to --inserts mode and thus see the table these entries were found in:

$ pg_dump --inserts mydb | grep -i kilroy
INSERT INTO audit.modification_log VALUES (716565, 'public', 'person', 18185, '{"session_user": "josh"}', '{"id": 633132}', '{"last_name": "Feeney44"}', '{"last_name": "Kilroy44"}', 'U', 123349, '2023-02-03 11:19:40.587883-06', '2023-02-03 11:19:40.587883-06', '2023-02-03 11:19:40.592103-06', 'josh', NULL, NULL);
INSERT INTO public.person VALUES (633132, 'F', 'NH', NULL, 'Cristen212', 'J', 'Kilroy44', '1983-09-28 00:00:00', NULL, true, NULL, NULL, NULL, NULL, 'F', 'USA', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL);

But pg_dump is considerably slower with --inserts turned on (about 200% slower, in my rudimentary testing on this one database), so perhaps I can get fancier with grep and achieve the same thing:

$ pg_dump mydb | grep -ie kilroy -e '^COPY' | grep -B1 -i kilroy
COPY audit.modification_log (id, table_schema, table_name, table_relid, app_user_info, id_columns, old_data, new_data, operation, transaction_id, ts_transaction, ts_statement, ts_clock, session_user_name, client_addr, query_text) FROM stdin;
716565  public  person  18185   {"session_user": "josh"}        {"id": 633132}  {"last_name": "Feeney44"}       {"last_name": "Kilroy44"}       U       123349  2023-02-03 11:19:40.587883-062023-02-03 11:19:40.587883-06    2023-02-03 11:19:40.592103-06   josh    \N      \N
--
COPY public.person (id, birth_gender, ethnicity, language, first_name, middle_name, last_name, birth_date, date_of_death, live, days_old_no_birthday, from_pregnant_id, fatal_condition_id, death_status, current_gender, country_of_birth, people_id, entity_id, birth_gender_id, ethnicity_id, primary_language_id, age_type_id, approximate_age_no_birthday, updated_at) FROM stdin;
633132  F       NH      \N      Cristen212      J    Kilroy44        1983-09-28 00:00:00     \N      t       \N      \N      \N      \N      F       USA  \N       \N      \N      \N      \N      \N      \N      \N

But the other day I realized I could easily search every a single table without leaving the database, thanks to \copy:

mydb=# \copy person to program 'grep -i kilroy';
COPY 117
633132  F       NH      \N      Cristen212      J    Kilroy44        1983-09-28 00:00:00     \N      t       \N      \N      \N      \N      F       USA  \N       \N      \N      \N      \N      \N      \N      \N

\copy requires copying the entire table to the client side, which wasn’t a concern in my instance but could be a problem for some folks, and you need to be a superuser to use server-side COPY ... TO PROGRAM, so there are some limitations to this approach.

I started thinking about this problem fresh off an encounter with row types, and realized we could do the whole thing in SQL:

mydb=# select * from person where person::text ~* 'kilroy';
   id   | birth_gender | ethnicity | language | first_name |             middle_name              | last_name |     birth_date      | date_of_death | live | days_old_no_birthday | from_pregnant_id | fatal_condition_id | death_status | current_gender | country_of_birth | people_id | entity_id | birth_gender_id | ethnicity_id | primary_language_id | age_type_id | approximate_age_no_birthday | updated_at
--------+--------------+-----------+----------+------------+--------------------------------------+-----------+---------------------+---------------+------+----------------------+------------------+--------------------+--------------+----------------+------------------+-----------+-----------+-----------------+--------------+---------------------+-------------+-----------------------------+------------
 633132 | F            | NH        |          | Cristen212 | J | Kilroy44  | 1983-09-28 00:00:00 |               | t    |                      |                  |                    |              | F              | USA              |           |           |                 |              |                     |             |                             |
(1 row)

This is my new favorite way of solving this problem, when I know what table I’m looking in but don’t know the column. In cases where I don’t know either, there’s always this monstrosity:

mydb=# select 'select ' || quote_literal(relname) || ' as relname, f.*
from ' || oid::regclass || ' f where f::text ~* ''kilroy''' from pg_class
where relkind = 'r' and relnamespace = 'public'::regnamespace; \gexec

This will work only in psql, as it depends on the \gexec command. It composes one query for each table, and then executes them in turn. It produces rather a lot of probably useless output you’ll need to sort through, and if you have (as I do) some tables with a large number of columns, I recommend trying it with \x turned on, to avoid having to page through quite so many results. But it did work, to find the entry I was looking for:

... Lots of output
(0 rows)

-[ RECORD 1 ]---------------+-------------------------------------
relname                     | person
id                          | 633132
birth_gender                | F
ethnicity                   | NH
language                    |
first_name                  | Cristen212
middle_name                 | J
last_name                   | Kilroy44
birth_date                  | 1983-09-28 00:00:00
date_of_death               |
live                        | t
days_old_no_birthday        |
from_pregnant_id            |
fatal_condition_id          |
death_status                |
current_gender              | F
country_of_birth            | USA
people_id                   |
entity_id                   |
birth_gender_id             |
ethnicity_id                |
primary_language_id         |
age_type_id                 |
approximate_age_no_birthday |
updated_at                  |

(0 rows)
... Lots more output

If I’m willing to expand beyond one-liner solutions, I can get psql to filter that useless output for me:

mydb=# \o | grep -i kilroy | grep -v select
mydb=# select 'select ' || quote_literal(relname) || ' as rname, f.* from ' || oid::regclass || ' f where f::text ~* ''kilroy'''
from pg_class where relkind = 'r' and relnamespace = 'public'::regnamespace; \gexec
mydb=# \o
 person | 633132 | F            | NH        |          | Cristen212 | J | Kilroy44  | 1983-09-28 00:00:00 |               | t    |                      |                  |                    |              | F              | USA              |           |           |                 |              |                     |             |                             |

When I did this, I had to use a final \o to set the output mode back to normal, before I saw results.

Do you have a different technique to solve this problem? I’m sure there are many other possibilities out there! Comment below!

Data Migration Tips

2023-02-04T00:00:00+00:00

When you’re in the business of selling software to people, you tend to get a few chances to migrate data from their legacy software to your shiny new system. Most recently for me that has involved public health data exported from legacy disease surveillance systems into PostgreSQL databases for use by the open source EpiTrax system and its companion EMSA.

We have collected a few tips that may help you learn from our successes, as well as our ~~mistakes~~particularly educational experiences.

Customer Management

Your job is to satisfy your customers, and your customers want to know how the migration is progressing. Give them an answer, even if it’s just a generalization. This may be a burndown chart, a calculated percentage, a nifty graphic, or whatever, but something your project managers can show to their managers, to know more or less how far along things are.

Your job is also to know your system; that’s not the customer’s job. They shouldn’t have to get their data into a specific format for you to make use of it. Be as flexible as possible in the data format and structure you’ll accept. In theory, so long as your customer can provide the legacy data in a machine-readable format, you should be able to use it. Let them focus on getting the data out of their legacy system — which is sometimes quite an effort in itself! Real data is almost always messy data, and your customer will probably want to take the opportunity to clean things up; make it as easy as possible for them to do that, while still ensuring the migration proceeds quickly and smoothly.

Be careful about the vocabulary you use with your customer. Your system and the legacy system probably deal with the same kinds of data, and do generally the same kinds of things. Of course, your software does it better than the old ’n’ busted mess you’re replacing, but in order to be better, your software has to be different from what it’s replacing. Different software means different methods, different processes, and different concepts. You and your customer might use the same words to mean totally different things, and unless you’re careful, those differences can remain hidden well into the implementation process, coming to light just in time to highlight some horrible mistake you’ve made. Try to avoid that. Talk with your customer to make sure your team and their team share a common vocabulary. If your system’s core function is to track widgets, and their legacy system’s function is also to track widgets, make sure you understand the differences between your software’s concept of a widget and their legacy system’s concept of a widget.

Migration Scripting

For our products and our customers, we’ve found pretty much every migration process is different, and each one is a custom programming job. Your team should decide at the beginning what steps the migration needs to include, and what technologies it will use in each step. Here at End Point, we often like to work directly in the database, in SQL. Migrations are all about manipulating data, and SQL is well suited for that task. With whatever technology you use, decide how you’ll use it, to manage the considerations given here. You can change your mind later and refactor accordingly, but always have a plan you’re following.

Design the migration as a sequence of processes. That is, first you might import one type of record, next another type of record that depends on the previous one, followed by several further steps to import data from a third source, clean it, map values from the legacy system to the new system, validate the results, and create records in the destination database. Of course the steps will vary from project to project, but the point is your migration will probably include several steps which need to be run in a specific order, so plan your development conventions accordingly. At End Point, we often like to put each step in a SQL file, and name each file beginning with a number, so you can run each script in order sorted by filename, and achieve the correct result. We might have files called:

01_import_products.sql
02_import_customers.sql
03_import_order_history.sql

It’s also common to implement each step one at a time, and to need to run each step several times as it’s being developed. We find it very helpful to wrap each step in a transaction. Often that means each SQL file begins with a BEGIN; statement, and ends with COMMIT;. Often I’ll leave out the COMMIT until I’m finished working on a file. That way to work on the code I can open a database session and run the migration script I’m working on, and when it completes, it will leave me inside the open transaction where I can inspect the results of my work. I make changes to the script, roll back the transaction, and run the script again, for as many iterations as it takes. I only add a COMMIT when I’ve tested the whole file and think it’s ready for the next step in testing.

I mentioned above that the customer may want to use this opportunity to clean their data. You should want this, too. Make sure the data you’re feeding your new system is as clean and well-structured as possible. You may find, as we do, that most of the work in your migrations is in validating the input data, and that actually creating new records in your application is almost an afterthought. That’s OK. You may also find there are places where your application, wonderful though it may be, could stand to be more strict about the data it accepts. I have often discovered my application’s database needs a uniqueness constraint, or a foreign key, thanks to a migration I was working on.

Data Migration History

I wish I could truthfully claim all our migrations go off flawlessly, but that would be a lie. It’s not unheard of to run into some corner case, a few weeks or even months after the migration goes live, where data wasn’t migrated correctly.

On the other hand, it’s certainly not uncommon for a customer or coworker to spot something that strikes them as odd, after the migration goes live, only to find later that everything was in fact correct. In either case, it’s important to preserve a history of the migration to investigate these concerns. We accomplish this with a few specific steps:

Create a database schema for the migration, and a table within that schema for each data file or object type we’re importing. I call these tables “staging tables”, where the incoming data is “staged” as it’s cleaned and validated. Having a separate schema means these tables can remain in the production database long after the migration is complete, generally without interfering with anything.
These staging tables should generally use text fields, to be as forgiving and flexible as possible with the incoming data. We can clean, parse, and reformat the data after it’s imported.
Don’t change the data in these staging tables; add to the data instead. In other words, if you need to map a value from the legacy system to a different value for your new system, don’t change the column you imported into the staging table; instead, add a new column to the staging table where you’ll store the re-mapped value. If you need to parse a text field into a date (because you followed the instruction to use text fields!), don’t change the type of an imported column; instead, add a new column of date or timestamp type, to store the parsed value. That way, when three months down the road someone discovers that some of the imported records have weird dates, you have all the information you need to determine whether the fault lies with the imported data or some step of the migration progress. Knowing exactly where the fault crept in leaves you that much more empowered to fix it.
Keep track of your migrated records’ primary keys, in the legacy system and the new system. Imagine you’ve just imported your client’s legacy customer list into a staging table. This data includes the legacy system’s primary key. Add a new column to the table for your new system’s primary key, and populate it. Many of our systems use an integer sequence as a primary key, so we’d add a new integer column to the staging table, and fill it with the next values from the sequence. Following this principle will give you several important abilities:
- You can always connect a record in the legacy system with its corresponding record(s) in the new system. If you’ve imported a customer list in this fashion, then when you’re importing the order data later, and each order points to a customer using a legacy customer primary key, you can easily find the correct customer primary key to use in your system.
- You can easily know if a record in your system comes from the migration, or from normal day-to-day business. You will probably use this every time you try to debug something with your migration.
- If you need to remove all imported records and re-import them, you can identify exactly which records those are. This should be only rarely needed.

Finally, document the decision making process, in comments directly in your code. For instance, if you have a table of mappings from one value to another, chances are good you arrived at the final version of that mapping table only after some discussion with the customer. Chances are also good someone’s going to question it later on. It’s helpful to keep a comment around, something like, # Joe Rogers verified this is the correct mapping in the daily standup meeting, 23 Nov 2022. This is especially common if you eventually decide to ignore a certain class of records. /* Rebecca says ignore all records with type = "ARCHIVED", via group email 9 Jan 2023 */ is a very helpful clue when someone comes around wondering where those records went.

Teamwork

My remaining tips apply to almost any programming project. First, use source control, and commit your code to it often. I can’t count how often I’ve been grateful the Git repository had a backup of my work, or made my work accessible to fill some unexpected need on some other system, nor can I count how many times I’ve been stuck because someone else didn’t commit their code so I couldn’t get at it when I needed it. Let’s not talk about how many times I’ve caused someone else to get stuck in the same way… Of course, don’t commit your customer’s data. But you should commit your code, and commit it often.

Finally, where possible, work with someone else. Two programmers reviewing each other’s code and collaborating on solutions are often far better than two programmers working alone, or one programmer working twice as much.

Speaking of working out solutions together, I’d love your help improving this list. What keys have you found are important for data migration projects? I welcome your comments. And if you’re looking for someone to handle a data migration project for you, give us a call!

Using pgTAP to automate database testing

2022-03-16T00:00:00+00:00

Photo from PxHere

Recently I started learning to tune pianos. There are many techniques and variations, but the traditional method, and the one apparently most accepted by ardent piano tuning purists, involves tuning one note to a reference, tuning several other notes in relation to the first, and testing the results by listening closely to different combinations of notes.

The tuner adjusts each new note in relation to several previously tuned notes. Physics being what it is, no piano can play all its tones perfectly, and one of the tricks of it all is adjusting each note to minimize audible imperfections. The tuner achieves this with an exacting series of musical intervals tested against each other.

Databases need tests too

One of our customers needed to add security policies to their PostgreSQL database, to limit data visibility for certain new users. This can quickly become complicated and ticklish, ensuring that the rules work properly for the affected users while leaving other users unmolested.

This struck me as an excellent opportunity to create some unit tests, not that there’s any short supply of good opportunities to add unit tests! This is not just because it helps prove that these security policies really do work properly, but because (confession time) I recently did a similar project for a different database without the help of unit tests, and it wasn’t much fun.

So this seemed like a good time to use pgTAP, a set of database functions designed to allow writing unit tests within the database itself. They produce “Test Anything Protocol” (TAP) output, a simple protocol that displays unit test results in an easily understood report.

What to test?

A good first step in writing unit tests is deciding on something to test. In my case, I figured I should make sure row-level security policies were turned on for the tables I was interested in, which is available from the rowsecurity field in the pg_tables view:

select ok(rowsecurity, tablename || ' has row security enabled') from pg_tables
    where schemaname = 'public'
        and (
            tablename in (
                -- ... some hard coded table names
            ) or tablename like 'some_other_tables_%'
        );

The ok() function comes from pgTAP. It counts as one test each time it’s called; the test passes when the first argument is true, and fails when the argument is something else. The second argument is an optional comment describing what’s being tested. Following a pretty common TAP-related naming convention, I put this in a file called 00-test.sql in a directory under the root of my project, simply called t.

A more complicated set of tests could include several different files, where the numeric part of the name helps sort the tests in the desired run order, and the rest of the filename describes the subject of the tests within. But this will do just to get started. I can run it with pg_prove, included with the pgTAP package:

pg_prove -d mydatabase t/00-test.sql

Iteratively improving

This fails, for several reasons.

First, I haven’t yet installed the pgTAP extension in my database, with CREATE EXTENSION pgtap.

I also haven’t actually done anything in my test to run the code I’m testing. The actual code in this project consists of some database functions, which we need to run to create the database security policies, and I haven’t run any of them yet.

And finally, pgTAP requires me to “plan” my tests first, or in other words, I need to inform pgTAP how many tests I plan to run, before I run them. It’s also nice to call finish() so pgTAP can clean up after itself.

I installed the pgTAP extension in my database, and modified the test as follows:

begin;

\i create_policy.sql

select plan(1);  -- plan for a single test

select ok(rowsecurity, tablename || ' has row security enabled') from pg_tables
    where schemaname = 'public'
        and (
            tablename in (
                -- ... some hard coded table names
            ) or tablename like 'some_other_tables_%'
        );

select finish();

rollback;

This wraps my test in a transaction, so that I can roll everything back to leave the database essentially as I found it. It also calls the actual code I’m testing, in create_policy.sql, and plans one test. And it gives me this new failure:

t/m.sql .. All 1 subtests passed

Test Summary Report
-------------------
t/m.sql (Wstat: 0 Tests: 226 Failed: 225)
  Failed tests:  2-226
Parse errors: Bad plan.  You planned 1 tests but ran 226.
Files=1, Tests=226,  1 wallclock secs ( 0.04 usr  0.00 sys +  0.03 cusr  0.01 csys =  0.08 CPU)
Result: FAIL

The problem here is that each call to ok() counts as one test, and my test apparently found 226 tables to check for row-level security. I can improve the planning like this:

select plan(count(*)::integer)
    from pg_tables where schemaname = 'public'
        and (
            tablename in (
                -- ... some hard coded table names
            ) or tablename like 'some_other_tables_%'
        );

count() returns a bigint, and plan() expects integer, so this requires a typecast, but is otherwise pretty simple. And now my tests pass:

josh@here:~dw$ pg_prove -d nedss t/00-test.sql
t/00-test.sql .. ok
All tests successful.
Files=1, Tests=226,  1 wallclock secs ( 0.03 usr  0.01 sys +  0.03 cusr  0.01 csys =  0.08 CPU)
Result: PASS

Looking back and ahead

Suffice it to say that pgTAP includes many functions similar to ok(), to test various aspects of the database, its structure, and its behavior, and I’d recommend interested users review the documentation for more details. I intended this post only as an introduction.

In its completed state, my test suite comprised several tests ensuring various required preliminaries were in place, a few tests like the one above that check for necessary table-specific settings, others that ensure the affected roles were created, and finally some which create some sample data and use SET ROLE to test the data visibility directly for roles with various policies applied.

And to be honest, I was surprised at the sense of security that came over me with this completed test suite. As I mentioned, I’d done similar work previously, and knew that although I was confident in the code when it was written, that confidence came only through fairly extensive manual testing. I know very well the struggles of bit rot, and I knew it would be at least as hard to repeat that testing regimen by hand sometime down the road after a year or two.

I also recognized that if I ever needed to set up similar policies again, I could use these tests themselves as a reference, because they show exactly how to run the code in question. Though of course I included that information in the project’s associated README file as well … right?

Let us know if you’ve used pgTAP, and what effect it has had on your database development.

Database Design: Using Documents

2022-03-09T00:00:00+00:00

Using documents in relational databases is increasingly popular. This technique can be practical and efficient when used in fitting circumstances.

Example

Let’s start with an example. Imagine we are scraping web sites for external URLs and store them in a table. We’ll have the web sites table to store the scrape timestamp and another table to store all of the references.

CREATE TABLE web_sites (
    web_site_domain text NOT NULL,
    last_scraped_at timestamptz,

    PRIMARY KEY (web_site_domain)
);

CREATE TABLE refs (
    web_site_domain text NOT NULL,
    ref_location text NOT NULL,
    link_url text NOT NULL,

    PRIMARY KEY (web_site_domain, ref_location, link_url),
    FOREIGN KEY (web_site_domain) REFERENCES web_sites
);

We do not need to bother adding an id to the web_sites table, because we assume there won’t be too many of them. The domain is small and more practical to use as an identifier. If you are curious about advantages of using natural keys, see my previous article.

Normalized Tables

There may be many thousands of unique URLs for a single web site and other web sites may refer to the same URLs. To try to minimize the storage, we can keep the locations and the URLs in separate tables, give them integer identifiers, and have another table for the many-to-many relations.

CREATE TABLE locations (
    location_id bigint NOT NULL GENERATED ALWAYS AS IDENTITY,
    web_site_domain text NOT NULL,
    ref_location text NOT NULL,

    PRIMARY KEY (location_id),
    UNIQUE (web_site_domain, ref_location),
    FOREIGN KEY (web_site_domain) REFERENCES web_sites
);

CREATE TABLE links (
    link_id bigint NOT NULL GENERATED ALWAYS AS IDENTITY,
    link_url text NOT NULL,

    PRIMARY KEY (link_id),
    UNIQUE (link_url)
);

CREATE TABLE locations_links (
    location_id bigint NOT NULL,
    link_id bigint NOT NULL,

    PRIMARY KEY (location_id, link_id),
    FOREIGN KEY (location_id) REFERENCES locations,
    FOREIGN KEY (link_id) REFERENCES links
);

The idea in here is to keep our biggest table narrow. It’d pay off to refer to the locations and links just with integer identifiers when we’ll have very many of them.

Table Sizes

We’ll have many web sites and many URLs, so our lookup tables would be big and the relation table would be even bigger. That is going to be a major problem because of many reasons, one of them being space efficiency. Narrow tables with many rows are not very space efficient especially on Postgres where there’s 24 bytes per-row overhead. To demonstrate it being significant, let’s add one row to each of the tables to see the sizes compared with the overhead.

INSERT INTO web_sites (web_site_domain)
VALUES ('example.com');

INSERT INTO refs (web_site_domain, ref_location, link_url)
VALUES ('example.com', '/source', 'http://example.net/target.html');

INSERT INTO locations (web_site_domain, ref_location)
VALUES ('example.com', '/source');

INSERT INTO links (link_url)
VALUES ('http://example.net/example.html');

INSERT INTO locations_links
VALUES (1, 1);

SELECT table_name, tuple_len, (100.0 * 24 / tuple_len)::int AS overhead_perc
FROM information_schema.tables, LATERAL pgstattuple(table_name)
WHERE table_schema = 'public';

   table_name    | tuple_len | overhead_perc
-----------------+-----------+---------------
 web_sites       |        36 |            67
 refs            |        75 |            32
 locations       |        52 |            46
 links           |        64 |            38
 locations_links |        40 |            60

With just one row and such short domains and URLs, we wouldn’t be gaining anything by having the links on a lookup table. In fact, it’s a lot less efficient. Although, the situation may change with many rows.

Composite Types

We can combine multiple fields in a column to avoid the overhead of many rows in narrow tables. Postgres has good support for composite types and arrays over them. They are useful if you need strict data type checking.

CREATE TYPE web_site_page AS (
    ref_location text,
    link_urls text[]
);

ALTER TABLE web_sites ADD COLUMN pages web_site_page[];

UPDATE web_sites
SET pages = ARRAY[ROW(ref_location, ARRAY[link_url])::web_site_page]
FROM refs;

SELECT pg_column_size(pages) FROM web_sites;

 pg_column_size
----------------
            117

As you see, the new column is not exactly small because composite types and arrays come with their own overheads. Still it could be much better when a web site has many references.

JSON Document Column

The composite types are not very easy to work with. We can store the pages in another format. JSON is the most popular one nowadays. There are multiple options to store JSON in a column in Postgres. Let’s start with the text-based data type.

ALTER TABLE web_sites ADD COLUMN pages_json json;

UPDATE web_sites SET pages_json = to_json(pages);

SELECT pg_column_size(pages_json) FROM web_sites;

 pg_column_size
----------------
             76

What is perhaps also surprising is the JSON being smaller than the array over composite type. This is because Postgres data types are a bit wasteful because of alignment and storing oids of the enclosed data types repeatedly.

Binary JSON

Another JSON data type, jsonb is available in Postgres. It is a binary structure that allows faster access to the values inside the document.

ALTER TABLE web_sites ADD COLUMN pages_jsonb jsonb;

UPDATE web_sites SET pages_jsonb = to_jsonb(pages);

SELECT pg_column_size(pages_jsonb) FROM web_sites;

 pg_column_size
----------------
             98

As you see the text-based JSON is still smaller. This is due to the binary offsets stored in the binary structure for faster access to the fields. The text-based JSON also compresses better.

Size Comparison

Very simple single row data shows the text-based json as the smallest, binary jsonb a bit larger, and the composite type to be the largest. However, the differences would be a lot smaller with more realistic sample data with many items inside the documents. I generated some data and gathered these results (the sizes excluding the indexes).

    model         |   size
------------------|---------
single table      |  436 MiB
normalized tables |  387 MiB
composite type    |  324 MiB
json              |  318 MiB
jsonb             |  320 MiB

You may expect the normalized tables to be the smallest as otherwise the metadata is stored repeatedly in the columns. Though, the results reveal it’s the other way around. This is because of the per-row overhead in Postgres. It becomes noticeable in narrow tables as in this example.

It’s also useful to notice that the most of the size is occupied by the TOAST tables when the data is stored in a single column. TOAST is a mechanism in Postgres that kicks in when a column is large enough which would often be the case when you design your tables this way. In this case, the table excluding the TOASTed part is pretty small which would help any query that doesn’t touch the large column.

Usability

What matters more than size is the usability of having it all in a single column. It is really practical to send and receive this column as a whole all the way from the database to the frontend. Compare this with dealing with lots of small objects used with the normalized tables.

It’s also possible in Postgres to index the document columns to allow efficient searches in them. There are many options to do so with advantages and disadvantages over each other. Though this post is getting long. I’ll leave indexing to another one.

String Processing in SQL, the Hard Way

2022-02-28T00:00:00+00:00

Photo by Kristy Lee

When I was a kid, my family visited a prehistoric cliff-dweller settlement, reconstructed and preserved as a museum exhibit. The homes were built in caves in high sandstone cliffs, and though I followed a well-marked path to visit them, the original inhabitants climbed in and out via holes cut into the rock face. Apparently to deter unwanted visitors, the builders carefully spaced the foot holds so that climbers needed to start their ascent on the correct foot. Someone who didn’t know the secret of the path, or who carelessly forgot, might get halfway up the cliff and discover he or she couldn’t stretch to the next foot hold and had to turn back.

The other day I found myself confronted with several comma-separated lists of strings, and a problem: I needed the longest possible substring from each, starting from the beginning, ending in a comma, and shorter than a certain length, and I needed to do it in the database. “Easy peasy,” you may say to yourself, if you have a little SQL programming experience and a penchant for cheesy clichés, and you’d be right except for one thing: I let my mind wander a little too far down into the uncharted mists of imagination and like the cliff dweller, I got off on the wrong foot. What followed was a thankfully brief exploration of the Hard Way to Do It.

The unsettling method on which I settled had me dividing each list into its component parts, reconstructing candidate results element by element, and finding the longest one less than the maximum overall length. Fortunately the problem didn’t allow me to reorder the elements in my input lists, or I’d probably still be working on this query. Here’s the monstrosity I eventually built:

select
    length(c), c
from (
    select
        concat_ws(', ', variadic array_agg(f) over (rows between unbounded preceding and current row)) c
    from (
        select regexp_split_to_table('alpha, bravo, charlie, delta, echo, fox trot', ',\s+') as f
    ) f
) c
where length(c) < 30
order by length(c) desc nulls last
limit 1

The innermost subquery here, based on regexp_split_to_table, divides the input string into an ordered list of its component elements:

select regexp_split_to_table('alpha, bravo, charlie, delta, echo, fox trot', ',\s+');
 regexp_split_to_table
-----------------------
 alpha
 bravo
 charlie
 delta
 echo
 fox trot
(6 rows)

I’m never one to bypass a good opportunity to shoehorn a window function into my SQL, and here we get to use the frame clause. Recall that for each row of a query, a window function processes a selection of rows, called a “partition”, and in fact processes only a “frame”, a subset of that partition. Window frames are fairly complicated subjects; I’ll try to present here a brief summary of the bits relevant to this query, and recommend the PostgreSQL documentation for further details.

Our particular window frame clause, rows between unbounded preceding and current row, means that the window frame for a given row should include all the rows from the first row of the partition through the current row. Given that we have no order by clause in our window definition, terms like “first row” refer to the first row in the order the rows were provided to the WindowAgg executor node. Whether it’s safe to trust that order to remain stable without a specific order by clause is an awfully good question I didn’t try to answer but for my purposes I needed to preserve the order of elements as returned by regexp_split_to_table. The default window frame clause is range unbounded preceding, which the documentation notes is a shortened form of range between unbounded preceding and current row, and without a specific order by clause, this means the window frame includes all rows in the partition. As a fun exercise, compare these two results, which differ only in the presence of an order by clause:

select array_agg(f) over () c
    from (
        select regexp_split_to_table('alpha, bravo, charlie, delta, echo, fox trot', ',\s+') as word
    ) f
;
                              c
-------------------------------------------------------------
 {(alpha),(bravo),(charlie),(delta),(echo),"(\"fox trot\")"}
 {(alpha),(bravo),(charlie),(delta),(echo),"(\"fox trot\")"}
 {(alpha),(bravo),(charlie),(delta),(echo),"(\"fox trot\")"}
 {(alpha),(bravo),(charlie),(delta),(echo),"(\"fox trot\")"}
 {(alpha),(bravo),(charlie),(delta),(echo),"(\"fox trot\")"}
 {(alpha),(bravo),(charlie),(delta),(echo),"(\"fox trot\")"}
(6 rows)

select array_agg(f) over (order by word) c
    from (
        select regexp_split_to_table('alpha, bravo, charlie, delta, echo, fox trot', ',\s+') as word
    ) f
;
                              c
-------------------------------------------------------------
 {(alpha)}
 {(alpha),(bravo)}
 {(alpha),(bravo),(charlie)}
 {(alpha),(bravo),(charlie),(delta)}
 {(alpha),(bravo),(charlie),(delta),(echo)}
 {(alpha),(bravo),(charlie),(delta),(echo),"(\"fox trot\")"}
(6 rows)

Anyway, that’s a tangent, but I found it a helpful one, as the default window frame is a well-known trap for young players, to which I’ve fallen prey in the past at least once. Back to my over-complicated query.

I’m aiming to produce a series of strings, each containing one more element than the last one. Now that I have a set of arrays with the elements I want inside, I turned to the concat_ws function to turn them into strings. concat_ws concatenates each of its arguments with a user-specified separator string, and the variadic keyword lets me pass an array of values and have them treated as individual arguments:

select
    concat_ws(', ', variadic array_agg(f) over (rows between unbounded preceding and current row)) c
from (
    select regexp_split_to_table('alpha, bravo, charlie, delta, echo, fox trot', ',\s+') as f
) f
;
                      c
----------------------------------------------
 alpha
 alpha, bravo
 alpha, bravo, charlie
 alpha, bravo, charlie, delta
 alpha, bravo, charlie, delta, echo
 alpha, bravo, charlie, delta, echo, fox trot

Now we’re getting close to the answer. My final steps are to filter out all resultant strings longer than the acceptable length and pick the longest one:

select
    length(c), c
from (
    select
        concat_ws(', ', variadic array_agg(f) over (rows between unbounded preceding and current row)) c
    from (
        select regexp_split_to_table('alpha, bravo, charlie, delta, echo, fox trot', ',\s+') as f
    ) f
) c
where length(c) < 30
order by length(c) desc nulls last
limit 1
;
 length |              c
--------+------------------------------
     28 | alpha, bravo, charlie, delta
(1 row)

Success! It only took about a hundred more steps than were really necessary. In my defense, it wasn’t too long after I’d finished composing this query that I realized I was making it much harder than I should have.

I posted it to our internal company chat, and within a minute or two my colleague Kürşat had answered with his own version:

SELECT regexp_matches(substring('alpha, bravo, charlie, delta, echo, fox trot' from 1 for 30), '(.*),');
          regexp_matches
----------------------------------
 {"alpha, bravo, charlie, delta"}
(1 row)

This simply finds a substring of acceptable length from the input text, and removes everything from the final comma onward. Not only did this take about 30 seconds to compose, compared to the half hour or so I’d spent on my first version, but on my laptop it completes in about half the time.

That fact notwithstanding, it’s often true that we learn rather a lot from our mistakes — and others’ mistakes — and I hope that’s true for you in this case.

Migrating a Node.js app database from MongoDB to PostgreSQL

2022-02-25T00:00:00+00:00

Recently I worked on a web app designed for tracking, editing, and reporting on archaeological survey data. When I joined the project, it used React on the front-end, Node.js on the back-end, GraphQL as the API interface, and MongoDB as the database. For the most part, this architecture and choice of technology worked well, and I was quickly able to make meaningful contributions to the app both in the user interface and in the backend. However, after a while, things started to get difficult and we began to consider why we were using MongoDB, and what might be better.

MongoDB vs. PostgreSQL

First, why was MongoDB difficult to use in our case? The biggest difficulty was using their query language for certain aspects of our database. Part of the database involved representing artifacts found at a site using a specific type hierarchy defined by the state government. Representing that hierarchy within Mongo was difficult, and querying it even more so. It was hard to find how to query a recursive structure using the MongoDB query language.

As I worked on this, scouring the documentation and Q&A websites to find ways to query this data efficiently, I began to realize that it was a problem that could be easily solved with PostgreSQL’s recursive queries.

Now, a single recursive table isn’t really a good reason to completely switch the database used by an application, but it got us thinking: Why were we using MongoDB? What benefits was it offering?

I found a relevant article published on the official MongoDB website entitled Comparing MongoDB vs PostgreSQL. That article is generally very positive in its portrayal of PostgreSQL, so I was interested to hear what they had to say for themselves. Here are some of the main points in favor of MongoDB that the article discusses:

MongoDB is horizontally scalable by default: This seems cool, but we were working on a new app that was not expected to need to scale much, and a single PostgreSQL instance has been shown to scale more than enough for our anticipated needs.
MongoDB supports fancy new serverless and mobile paradigms: Again, cool, but this was a fairly small-scale project using a pretty normal Linux/Node.js stack. No need for serverless support.
MongoDB’s document structure makes it easy for developers to change the structure of data on the fly: While this can certainly be nice, it can also bring its own issues. If data is expected to fit a certain structure and one developer changes that structure, this will likely cause problems for other developers whether or not there is a schema. Additionally, our project had a small team of 1-3 developers working on it and changing the schema as needed just wasn’t a problem.

The MongoDB vs. PostgreSQL article also mentions “resilience” as a strength of MongoDB, citing its ability to easily be replicated and broken into shards across datacenters and regions. However, research regarding the historical reliability of PostgreSQL and MongoDB showed that MongoDB’s fairly short history is littered with data loss problems. Anecdotally, a Google search for “MongoDB data loss” returns 1.26 million results (an average of 0.14 million results per year since its initial release) while “PostgreSQL data loss” returns 1.57 million (0.06 million per year since release). More specifically, there are fascinating in-depth reviews of MongoDB’s reliability, such as those conducted by Jepsen as recently as 2020 (see #1, #2, #3, #4, #5).

PostgreSQL hasn’t been perfect in the 25 years since its initial release, and MongoDB has improved a lot in recent years, but it’s clear that Postgres is the winner when it comes to data reliability.

After reviewing all of these pros and cons of MongoDB, it became clear that it wasn’t really offering anything that we needed, and our data, which was strictly organized and relational by nature, would fit much better in a relational database. We decided that it would be better to switch to PostgreSQL.

Mongoose and Sequelize

Our database was only being used within the backend GraphQL server, using Mongoose, an object modeling library for Node.js. Mongoose uses a schema/model system, where every MongoDB collection contains documents fitting a schema defined within Mongoose model definitions, for example:

import mongoose from "mongoose";
const { Schema } = mongoose;

const blogPostSchema = new Schema({
  title: String, // String is shorthand for {type: String}
  author: String,
  body: String,
  comments: [{ body: String, date: Date }],
  date: { type: Date, default: Date.now },
  hidden: Boolean,
  meta: {
    votes: Number,
    favs: Number,
  },
});

const BlogPost = mongoose.model("BlogPost", blogPostSchema);

Once a model is defined it can easily be used to query and create documents:

const new_post = new BlogPost({
  title: "On the international epidemic of counterfeit at-home brogal treatments",
  author: "Ronald McClure",
  body: "...",
  // ...
});
await new_post.save();

const posts = await BlogPost.find({ author: "Ronald McClure" });

Mongoose provides a lot of functions for querying and editing documents in a collection on the Model class, as well as a rich Query class to make manual queries, among other things. Their documentation provides helpful guides and a great API reference.

Because we generally liked the Mongoose interface and didn’t want to have to change our existing code too drastically, we searched for a similar library that could work with PostgreSQL and settled upon Sequelize.

Sequelize has a very similar interface to Mongoose, where you start by defining tables (instead of “collections”) as models in JavaScript:

const { Sequelize, DataTypes } = require('sequelize');
const sequelize = new Sequelize('postgres://user:pass@example.com:5432/dbname');

const BlogPost = sequelize.define('BlogPost', {
  // Model attributes are defined here
  title: DataTypes.STRING,
  author: DataTypes.STRING,
  body: DataTypes.STRING,
  // This will need to be modeled differently
  // comments: [{ body: DataTypes.STRING, date: Date }],
  date: {
    type: DataTypes.DATETIME,
    defaultValue: DataTypes.NOW
  },
  hidden: DataTypes.BOOLEAN,
  votes: DataTypes.INTEGER,
  favs: DataTypes.INTEGER
}, {
  // Other model options go here
});

// Object creation is similar
const new_post = BlogPost.build({
  title: "On the international epidemic of counterfeit at-home brogal treatments",
  author: "Ronald McClure",
  body: "...",
  // ...
});
await new_post.save();

Note that this model, while mimicking the Mongoose schema as much as possible, still requires some changes to fit the relational model. First, the comments field, which was an array of documents in the Mongoose object will need to be represented in another table, which can then be linked via foreign key.

While this seems annoying in some ways — creating a new table will require creating another Sequelize model, setting up the relations correctly, etc. — it makes a lot of sense for this example (a blog) and most other situations. We don’t just want blog posts to have associated comments, but we likely also want those comments to be tied to an author and to have their own IDs, which would allow lookup by author (for example so that on a user’s page they can see all comments they’ve made on all posts) and by individual comment (so it’s easier to make links directly to a single comment).

So, rather than having comments be a subfield of a blog post, we want them to be defined by some kind of relation. In MongoDB this can be done in a few ways, most obviously by creating a comments collection making the comments list contain ObjectIDs:

const mongoose = require("mongoose");
const { Schema } = mongoose;

const Comment = mongoose.model(
  "Comment",
  new Schema({
    body: String,
    date: Date,
  })
);

const blogPostSchema = new Schema({
  // ...
  comments: [{ type: Schema.Types.ObjectId, ref: "Comment" }],
  // ...
});

const BlogPost = mongoose.model("BlogPost", blogPostSchema);

Mongoose also has a way to automatically fill in those referenced objects with the real thing as needed, using their populate functionality:

BlogPost.findOne({ author: "Ronald McClure" })
  .populate("comments")
  .exec(function (err, post) {
    if (err) return handleError(err);
    post.comments.forEach((comment) => console.log(comment));
  });

With the call to .populate('comments'), the comments array is filled with objects instead of just the object IDs. Sequelize has similar functionality, but uses the concept of associations which model the relational structure of SQL more closely:

const { Sequelize, DataTypes } = require('sequelize');
const sequelize = new Sequelize('postgres://user:pass@example.com:5432/dbname');

const BlogPost = sequelize.define('BlogPost', {
  // ... fields here
}, {});

const Comment = sequelize.define('Comment', {
  body: DataTypes.TEXT,
  date: DataTypes.DATE,
}, {});

Comment.belongsTo(BlogPost;
BlogPost.hasMany(Comment);

These belongsTo and hasMany associations use foreign key/primary key references on the models, and Sequelize also provides a similar interface for filling in the referenced data:

const post = await BlogPost.findOne({
  where: {
    author: "Ronald McClure",
  },
  include: Comment,
});

Both libraries also have ways to lazy-load related objects, only fetching them as necessary. For MongoDB, this involves writing your own queries on referenced fields or using MongoDB’s $lookup operator, while Sequelize allows custom queries using the foreign key fields, as well as a set of nice methods to load associations on demand.

Modeling and Migrating Data

Having weighed for this situation the pros and cons of PostgreSQL and MongoDB as well as Mongoose and Sequelize, the question then comes: How do we actually move the data from one system to another? Unfortunately, as far as I know there is no one tool that can be used to migrate data between MongoDB and PostgreSQL. However, using a combination of mongoexport and some basic SQL files, it was not too difficult to migrate data.

The first step and perhaps one of the more difficult ones is to create an SQL schema that works for your data. If your MongoDB data is already in a normalized and relational structure, it might not be too hard, but if you have a lot of subdocuments and arrays that need to be expanded into rows, things might be more difficult. My recommendation for how to approach this is:

Model your data in SQL. If you’re doing a migration from MongoDB, you likely have an idea of what the application needs are and how data should be organized.
Test your model with some dummy data. This will probably be a bit of a back-and-forth iterative process as you iron out issues with relations, decide what kind of constraints will be necessary, etc.
Use mongoexport to export your data as appropriate. For many collections, it may be easy to simply export it as CSV and use Postgres’s COPY command to import it into the appropriate table. However, many things won’t be so simple and might require writing your own script that reads the exported CSV or JSON and writes it to the database, altering data as needed. This might be necessary for example to change MongoDB’s object IDs to match a datatype in PostgreSQL, such as an identity column (defined in CREATE TABLE), serial, or UUID.
Use your set of import commands and scripts to import all of your data into a test database.
Modify your application to use Sequelize instead of Mongoose.
Find and update all cases in your code where a Mongoose model or instance was used and update it to use Sequelize instead.

Caveats and Gotchas

One of the biggest problems we came across after our migration was the differences in query syntax between Mongoose and Sequelize. Compare the following queries:

// Mongoose
post = await BlogPost.findOne({ author: 'Ronald McClure' });

// Sequelize
post = await BlogPost.findOne({ author: 'Ronald McClure' });

They look the same, they both run with no errors, and they both return the one post we created. But say we change which author we are querying for:

// Mongoose
post = await BlogPost.findOne({ author: 'Darlene Roberts' });

// Sequelize
post = await BlogPost.findOne({ author: 'Darlene Roberts' });

Mongoose’s query returns nothing, as we expect, but Sequelize… returns the same post as the “Ronald McClure” query returned? Astute readers might notice what the issue is from earlier examples: Sequelize’s find methods take an options object which doesn’t use its fields as WHERE parameters by default. The correct way to write this query with Sequelize is:

post = await BlogPost.findOne({ where: { author: "Darlene Roberts" } });

It’s a bit of a silly mistake, but one that caused pain several times in the process of migration. Some queries seemed to work just fine, but just wouldn’t return the right results!

Database integration testing with .NET

2022-01-12T00:00:00+00:00

Ruby on Rails is great. We use it at End Point for many projects with great success. One of Rails’ cool features is how easy it is to write database integration tests. Out of the box, Rails projects come with all the configuration necessary to set up a database that’s exclusive for the automated test suite. This database includes all the tables and other objects that exist within the regular database that the app uses during its normal execution. So, it is very easy to write automated tests that cover the application components that interact with the database.

ASP.NET Core is also great! However, it doesn’t have this feature out of the box. Let’s see if we can’t do it ourselves.

The sample project

As a sample project we will use a REST API that I wrote for another article. Check it out if you want to learn more about the ins and outs of developing REST APIs with .NET. You can find the source code on GitHub.

The API is very straightforward. It provides a few endpoints for CRUDing some database tables. It also provides an endpoint which, when given some vehicle information, will calculate a monetary value for that vehicle. That’s a feature that would be interesting for us to cover with some tests.

The logic for that feature is backed by a specific class and it depends heavily on database interactions. As such, that class is a great candidate for writing a few automated integration tests against. The class in question is QuotesService which is defined in Services/QuoteService.cs. The class provides features for fetching records from the database (the GetAllQuotes method) as well as creating new records based on data from the incoming request and a set of rules stored in the database itself (the CalculateQuote method).

In order to add automated tests, the first step is to organize our project so that it supports them. Let’s do that next.

Organizing the source code to allow for automated testing

In general, the source code of most real world .NET applications is organized as one or more “projects” under one “solution”. A solution is a collection of related projects, and a project is something that produces a deployment artifact. An artifact is a library (i.e. a *.dll file) or something that can be executed like a console or web app.

Our sample app is a stand-alone “webapi” project, meaning that it’s not within a solution. For automated tests, however, we need to create a new project for tests, parallel to our main one. Now that we have two projects instead of one, we need to reorganize the sample app’s source code to comply with the “projects in a solution” structure I mentioned earlier.

Let’s start by moving all the files in the root directory into a new VehicleQuotes directory. That’s one project. Then, we create a new automated tests project by running the following, still from the root directory:

dotnet new xunit -o VehicleQuotes.Tests

That creates a new automated tests project named VehicleQuotes.Tests (under a new aptly-named VehicleQuotes.Tests directory) which uses the xUnit.net test framework. There are other options when it comes to test frameworks in .NET, such as MSTest and NUnit. We’re going to use xUnit.net, but the others should work just as well for our purposes.

Now, we need to create a new solution to contain those two projects. Solutions come in the form of *.sln files and we can create ours like so:

dotnet new sln -o vehicle-quotes

That should’ve created a new vehicle-quotes.sln file for us. We should now have a file structure like this:

.
├── vehicle-quotes.sln
├── VehicleQuotes
│   ├── VehicleQuotes.csproj
│   └── ...
└── VehicleQuotes.Tests
    ├── VehicleQuotes.Tests.csproj
    └── ...

Like I said, the *.sln file indicates that this is a solution. The *.csproj files identify the individual projects that make up the solution.

Now, we need to tell dotnet that those two projects belong in the same solution. These commands do that:

dotnet sln add ./VehicleQuotes/VehicleQuotes.csproj
dotnet sln add ./VehicleQuotes.Tests/VehicleQuotes.Tests.csproj

Finally, we update the VehicleQuotes.Tests project so that it references the VehicleQuotes project. That way, the test suite will have access to all the classes defined in the REST API. Here’s the command for that:

dotnet add ./VehicleQuotes.Tests/VehicleQuotes.Tests.csproj reference ./VehicleQuotes/VehicleQuotes.csproj

With all that setup out of the way, we can now start writing some tests.

You can learn more about project organization in the official online documentation.

Creating a DbContext instance to talk to the database

The VehicleQuotes.Tests automated tests project got created with a default test file named UnitTest1.cs. You can delete it or ignore it, since we will not use it.

In general, it’s a good idea for the test project to mimic the directory structure of the project that it will be testing. Also, we already decided that we would focus our test efforts on the QuoteService class from the VehicleQuotes project. That class is defined in VehicleQuotes/Services/QuoteService.cs, so let’s create a similarly located file within the test project which will contain the test cases for that class. Here: VehicleQuotes.Tests/Services/QuoteServiceTests.cs. These would be the contents:

// VehicleQuotes.Tests/Services/QuoteServiceTests.cs

using System;
using Xunit;

namespace VehicleQuotes.Tests.Services
{
    public class QuoteServiceTests
    {
        [Fact]
        public void GetAllQuotesReturnsEmptyWhenThereIsNoDataStored()
        {
            // Given

            // When

            // Then
        }
    }
}

This is the basic structure for tests using xUnit.net. Any method annotated with a [Fact] attribute will be picked up and run by the test framework. In this case, I’ve created one such method called GetAllQuotesReturnsEmptyWhenThereIsNoDataStored which should give away its intention. This test case will validate that QuoteService’s GetAllQuotes method returns an empty set when called with no data in the database.

Before we can write this test case, though, the suite needs access to the test database. Our app uses Entity Framework Core for database interaction, which means that the database is accessed via a DbContext class. Looking at the source code of our sample app, we can see that the DbContext being used is VehicleQuotesContext, defined in VehicleQuotes/Data/VehicleQuotesContext.cs. Let’s add a utility method to the QuoteServiceTests class which can be used to create new instances of VehicleQuotesContext:

// VehicleQuotes.Tests/Services/QuoteServiceTests.cs

// ...
using Microsoft.EntityFrameworkCore;
using VehicleQuotes.Services;

namespace VehicleQuotes.Tests.Services
{
    public class QuoteServiceTests
    {
        private VehicleQuotesContext CreateDbContext()
        {
            var options = new DbContextOptionsBuilder<VehicleQuotesContext>()
                .UseNpgsql("Host=db;Database=vehicle_quotes_test;Username=vehicle_quotes;Password=password")
                .UseSnakeCaseNamingConvention()
                .Options;

            var context = new VehicleQuotesContext(options);

            context.Database.EnsureCreated();

            return context;
        }

        // ...
    }
}

As you can see, we need to go through three steps to create the VehicleQuotesContext instance and get a database that’s ready for testing:

First, we create a DbContextOptionsBuilder and use that to obtain the options object that the VehicleQuotesContext needs as a constructor parameter. We needed to include the Microsoft.EntityFrameworkCore namespace in order to have access to the DbContextOptionsBuilder. For this, I just copied and slightly modified this statement from the ConfigureServices method in the REST API’s VehicleQuotes/Startup.cs file:

// VehicleQuotes/Startup.cs

public void ConfigureServices(IServiceCollection services)
{
    // ...

    services.AddDbContext<VehicleQuotesContext>(options =>
        options
            .UseNpgsql(Configuration.GetConnectionString("VehicleQuotesContext"))
            .UseSnakeCaseNamingConvention()
            .UseLoggerFactory(LoggerFactory.Create(builder => builder.AddConsole()))
            .EnableSensitiveDataLogging()
    );

    // ...
}

This is a method that runs when the application is starting up to set up all the services that the app uses to work. Here, it’s setting up the DbContext to enable database interaction. For the test suite, I took this statement as a starting point and removed the logging configurations and specified a hardcoded connection string that specifically points to a new vehicle_quotes_test database that will be used for testing.

If you’re following along, then you need a PostgreSQL instance that you can use to run the tests. In my case, I have one running that is reachable via the connection string I specified: Host=db;Database=vehicle_quotes_test;Username=vehicle_quotes;Password=password.

If you have Docker, a quick way to get a Postgres database up and running is with this command:

docker run -d \
    --name vehicle-quotes-db \
    -p 5432:5432 \
    --network host \
    -e POSTGRES_DB=vehicle_quotes \
    -e POSTGRES_USER=vehicle_quotes \
    -e POSTGRES_PASSWORD=password \
    postgres

That’ll spin up a new Postgres instance that’s reachable via localhost.

Secondly, now that we have the options parameter ready, we can quite simply instantiate a new VehicleQuotesContext:

var context = new VehicleQuotesContext(options);

Finally, we call the EnsureCreated method so that the database that we specified in the connection string is actually created.

context.Database.EnsureCreated();

This is the database that our test suite will use.

Defining the test database connection string in the appsettings.json file

One quick improvement that we can do to the code we’ve written so far is move the connection string for the test database into a separate configuration file, instead of having it hardcoded. Let’s do that next.

We need to create a new appsettings.json file under the VehicleQuotes.Tests directory. Then we have to add the connection string like so:

{
  "ConnectionStrings": {
    "VehicleQuotesContext": "Host=db;Database=vehicle_quotes_test;Username=vehicle_quotes;Password=password"
  }
}

This is the standard way of configuring connection strings in .NET. Now, to actually fetch this value from within our test suite code, we make the following changes:

// ...
+using Microsoft.Extensions.Hosting;
+using Microsoft.Extensions.Configuration;
+using Microsoft.Extensions.DependencyInjection;

namespace VehicleQuotes.Tests.Services
{
    public class QuoteServiceTests
    {
        private VehicleQuotesContext CreateDbContext()
        {
+           var host = Host.CreateDefaultBuilder().Build();
+           var config = host.Services.GetRequiredService<IConfiguration>();

            var options = new DbContextOptionsBuilder<VehicleQuotesContext>()
-               .UseNpgsql("Host=db;Database=vehicle_quotes_test;Username=vehicle_quotes;Password=password")
+               .UseNpgsql(config.GetConnectionString("VehicleQuotesContext"))
                .UseSnakeCaseNamingConvention()
                .Options;

            var context = new VehicleQuotesContext(options);

            context.Database.EnsureCreated();

            return context;
        }

        // ...
    }
}

First we add a few using statements. We need Microsoft.Extensions.Hosting so that we can have access to the Host class through which we obtain access to the application’s execution context. This allows us to access the built-in configuration service. We also need Microsoft.Extensions.Configuration to have access to the IConfiguration interface which is how we reference the configuration service which allows us access to the appsettings.json config file. And we also need the Microsoft.Extensions.DependencyInjection namespace which allows us to tap into the built-in dependency injection mechanism, through which we can access the default configuration service I mentioned before. Specifically, that namespace is where the GetRequiredService extension method lives.

All this translates into the few code changes that you see in the previous diff: first getting the app’s host, then getting the configuration service, then using that to fetch our connection string.

You can refer to the official documentation to learn more about configuration in .NET.

Writing a simple test case that fetches data

Now that we have a way to access the database from within the test suite, we can finally write an actual test case. Here’s the GetAllQuotesReturnsEmptyWhenThereIsNoDataStored one that I alluded to earlier:

// ...

namespace VehicleQuotes.Tests.Services
{
    public class QuoteServiceTests
    {
        // ...

        [Fact]
        public async void GetAllQuotesReturnsEmptyWhenThereIsNoDataStored()
        {
            // Given
            var dbContext = CreateDbContext();
            var service = new QuoteService(dbContext, null);

            // When
            var result = await service.GetAllQuotes();

            // Then
            Assert.Empty(result);
        }
    }
}

This one is a very simple test. We obtain a new VehicleQuotesContext instance that we can use to pass as a parameter when instantiating the component that we want to test: the QuoteService. We then call the GetAllQuotes method and assert that it returned an empty set. The test database was just created, so there should be no data in it, hence the empty resource set.

To run this test, we do dotnet test. I personally like a more verbose output so I like to use this variant of the command: dotnet test --logger "console;verbosity=detailed". Here’s what the output looks like.

$ dotnet test --logger "console;verbosity=detailed"
  Determining projects to restore...
  All projects are up-to-date for restore.
  VehicleQuotes -> /app/VehicleQuotes/bin/Debug/net5.0/VehicleQuotes.dll
  VehicleQuotes.Tests -> /app/VehicleQuotes.Tests/bin/Debug/net5.0/VehicleQuotes.Tests.dll
Test run for /app/VehicleQuotes.Tests/bin/Debug/net5.0/VehicleQuotes.Tests.dll (.NETCoreApp,Version=v5.0)
Microsoft (R) Test Execution Command Line Tool Version 16.11.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
/app/VehicleQuotes.Tests/bin/Debug/net5.0/VehicleQuotes.Tests.dll
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.3+1b45f5407b (64-bit .NET 5.0.12)
[xUnit.net 00:00:01.03]   Discovering: VehicleQuotes.Tests
[xUnit.net 00:00:01.06]   Discovered:  VehicleQuotes.Tests
[xUnit.net 00:00:01.06]   Starting:    VehicleQuotes.Tests
[xUnit.net 00:00:03.25]   Finished:    VehicleQuotes.Tests
  Passed VehicleQuotes.Tests.Services.QuoteServiceTests.GetAllQuotesReturnsEmptyWhenThereIsNoDataStored [209 ms]

Test Run Successful.
Total tests: 1
     Passed: 1
 Total time: 3.7762 Seconds

Resetting the state of the database after each test

Now we need to write a test that actually writes data into the database. However, every test case needs to start with the database in its original state. In other words, the changes that one test case does to the test database should not be seen, affect, or be expected by any subsequent test. That will make it so our test cases are isolated and repeatable. That’s not possible with our current implementation, though.

You can read more about the FIRST principles of testing here.

Luckily, that’s a problem that’s easily solved with Entity Framework Core. All we need to do is call a method that ensures that the database is deleted just before it ensures that it is created. Here’s what it looks like:

 private VehicleQuotesContext CreateDbContext()
 {
     var host = Host.CreateDefaultBuilder().Build();
     var config = host.Services.GetRequiredService<IConfiguration>();

     var options = new DbContextOptionsBuilder<VehicleQuotesContext>()
         .UseNpgsql(config.GetConnectionString("VehicleQuotesContext"))
         .UseSnakeCaseNamingConvention()
         .Options;

     var context = new VehicleQuotesContext(options);

+    context.Database.EnsureDeleted();
     context.Database.EnsureCreated();

     return context;
 }

And that’s all. Now every test case that calls CreateDbContext in order to obtain a DbContext instance will effectively trigger a database reset. Feel free to dotnet test again to validate that the test suite is still working.

Now, depending on the size of the database, this can be quite expensive. For integration tests, performance is not as big of a concern as for unit tests. This is because integration tests should be fewer in number and less frequently run.

We can make it better though. Instead of deleting and recreating the database before each test case, we’ll take a page out of Ruby on Rails’ book and run each test case within a database transaction which gets rolled back after the test is done. For now though, let’s write another test case: this time, one where we insert new records into the database.

If you want to hear a more in-depth discussion about automated testing in general, I go into further detail on the topic in this article: An introduction to automated testing for web applications with Symfony.

Writing another simple test case that stores data

Now let’s write another test that exercises QuoteService’s GetAllQuotes method. This time though, let’s add a new record to the database before calling it so that the method’s result is not empty. Here’s what the test looks like:

// ...
using VehicleQuotes.Models;
using System.Linq;

namespace VehicleQuotes.Tests.Services
{
    public class QuoteServiceTests
    {
        // ...

        [Fact]
        public async void GetAllQuotesReturnsTheStoredData()
        {
            // Given
            var dbContext = CreateDbContext();

            var quote = new Quote
            {
                OfferedQuote = 100,
                Message = "test_quote_message",

                Year = "2000",
                Make = "Toyota",
                Model = "Corolla",
                BodyTypeID = dbContext.BodyTypes.Single(bt => bt.Name == "Sedan").ID,
                SizeID = dbContext.Sizes.Single(s => s.Name == "Compact").ID,

                ItMoves = true,
                HasAllWheels = true,
                HasAlloyWheels = true,
                HasAllTires = true,
                HasKey = true,
                HasTitle = true,
                RequiresPickup = true,
                HasEngine = true,
                HasTransmission = true,
                HasCompleteInterior = true,

                CreatedAt = DateTime.Now
            };

            dbContext.Quotes.Add(quote);

            dbContext.SaveChanges();

            var service = new QuoteService(dbContext, null);

            // When
            var result = await service.GetAllQuotes();

            // Then
            Assert.NotEmpty(result);
            Assert.Single(result);
            Assert.Equal(quote.ID, result.First().ID);
            Assert.Equal(quote.OfferedQuote, result.First().OfferedQuote);
            Assert.Equal(quote.Message, result.First().Message);
        }
    }
}

First we include the VehicleQuotes.Models namespace so that we can use the Quotes model class. In our REST API, this is the class that represents the data from the quotes table. This is the main table that GetAllQuotes queries. We also include the System.Linq namespace, which allows us to use various collection extension methods (like Single and First) which we leverage throughout the test case to query lookup tables and assert on the test results.

Other than that, the test case itself is pretty self-explanatory. We start by obtaining an instance of VehicleQuotesContext via the CreateDbContext method. Remember that this also resets the whole database so that the test case can run over a clean slate. Then, we create a new Quote object and use our VehicleQuotesContext to insert it as a record into the database. We do this so that the later call to QuoteService’s GetAllQuotes method actually finds some data to return this time. Finally, the test case validates that the result contains a record and that its data is identical to what we set manually.

Neat! At this point we have what I think is the bare minimum infrastructure when it comes to serviceable and effective database integration tests, namely, access to a test database. We can take it one step further, though, and make things more reusable and a little bit better performing.

Refactoring into a fixture for reusability

We can use the test fixture functionality offered by xUnit.net in order to make the database interactivity aspect of our test suite into a reusable component. That way, if we had other test classes focused on other components that interact with the database, we could just plug that code in. We can define a fixture by creating a new file called, for example, VehicleQuotes.Tests/Fixtures/DatabaseFixture.cs with these contents:

using System;
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;

namespace VehicleQuotes.Tests.Fixtures
{
    public class DatabaseFixture : IDisposable
    {
        public VehicleQuotesContext DbContext { get; private set; }

        public DatabaseFixture()
        {
            DbContext = CreateDbContext();
        }

        public void Dispose()
        {
            DbContext.Dispose();
        }

        private VehicleQuotesContext CreateDbContext()
        {
            var host = Host.CreateDefaultBuilder().Build();
            var config = host.Services.GetRequiredService<IConfiguration>();

            var options = new DbContextOptionsBuilder<VehicleQuotesContext>()
                .UseNpgsql(config.GetConnectionString("VehicleQuotesContext"))
                .UseSnakeCaseNamingConvention()
                .Options;

            var context = new VehicleQuotesContext(options);

            context.Database.EnsureDeleted();
            context.Database.EnsureCreated();

            return context;
        }
    }
}

All this class does is define the CreateDbContext method that we’re already familiar with but puts it in a nice reusable package. Upon instantiation, as seen in the constructor, it stores a reference to the VehicleQuotesContext in its DbContext property.

With that, our QuoteServiceTests test class can use it if we make the following changes to it:

 using System;
 using Xunit;
-using Microsoft.EntityFrameworkCore;
 using VehicleQuotes.Services;
-using Microsoft.Extensions.Hosting;
-using Microsoft.Extensions.Configuration;
-using Microsoft.Extensions.DependencyInjection;
 using VehicleQuotes.Models;
 using System.Linq;
+using VehicleQuotes.Tests.Fixtures;

 namespace VehicleQuotes.Tests.Services
 {
-    public class QuoteServiceTests
+    public class QuoteServiceTests : IClassFixture<DatabaseFixture>
     {
+        private VehicleQuotesContext dbContext;

+        public QuoteServiceTests(DatabaseFixture fixture)
+        {
+            dbContext = fixture.DbContext;
+        }

-        private VehicleQuotesContext CreateDbContext()
-        {
-            var host = Host.CreateDefaultBuilder().Build();
-            var config = host.Services.GetRequiredService<IConfiguration>();

-            var options = new DbContextOptionsBuilder<VehicleQuotesContext>()
-                .UseNpgsql(config.GetConnectionString("VehicleQuotesContext"))
-                .UseSnakeCaseNamingConvention()
-                .Options;

-            var context = new VehicleQuotesContext(options);

-            context.Database.EnsureDeleted();
-            context.Database.EnsureCreated();

-            return context;
-        }

         [Fact]
         public async void GetAllQuotesReturnsEmptyWhenThereIsNoDataStored()
         {
             // Given
-            var dbContext = CreateDbContext();

             // ...
         }

         [Fact]
         public async void GetAllQuotesReturnsTheStoredData()
         {
             // Given
-            var dbContext = CreateDbContext();

             // ...
         }
     }
 }

Here we’ve updated the QuoteServiceTests class definition so that it inherits from IClassFixture<DatabaseFixture>. This is how we tell xUnit.net that our tests use the new fixture that we created. Next, we define a constructor that receives a DatabaseFixture object as a parameter. That’s how xUnit.net allows our test class to access the capabilities provided by the fixture. In this case, we take the fixture’s DbContext instance, and store it for later use in all of the test cases that need database interaction. We also removed the CreateDbContext method because now that’s defined within the fixture. We also removed a few using statements that became unnecessary.

One important aspect to note about this fixture is that it is initialized once per whole test suite run, not once per test case. Specifically, the code within the DatabaseFixture’s constructor gets executed once, before all of the test cases. Similarly, the code in DatabaseFixture’s Dispose method get executed once at the end, after all test cases have been run.

This means that our test database deletion and recreation step now happens only once for the entire test suite. This is not good with our current implementation because that means that individual test cases no longer run with a fresh, empty database. This can be good for performance though, as long as we update our test cases to run within database transactions. Let’s do just that.

Using transactions to reset the state of the database

Here’s how we update out test class so that each test case runs within a transaction:

 // ...

 namespace VehicleQuotes.Tests.Services
 {
-    public class QuoteServiceTests : IClassFixture<DatabaseFixture>
+    public class QuoteServiceTests : IClassFixture<DatabaseFixture>, IDisposable
     {
         private VehicleQuotesContext dbContext;

         public QuoteServiceTests(DatabaseFixture fixture)
         {
             dbContext = fixture.DbContext;

+            dbContext.Database.BeginTransaction();
         }

+        public void Dispose()
+        {
+            dbContext.Database.RollbackTransaction();
+        }

         # ...
     }
 }

The first thing to note here is that we added a call to BeginTransaction in the test class constructor. xUnit.net creates a new instance of the test class for each test case. This means that this constructor is run before each and every test case. We use that opportunity to begin a database transaction.

The other interesting point is that we’ve updated the class to implement the IDisposable interface’s Dispose method. xUnit.net will run this code after each test case, so we rollback the transaction.

Put those two together and we’ve updated our test suite so that every test case runs within the context of its own database transaction. Try it out with dotnet test and see what happens.

To learn more about database transactions with Entity Framework Core, you can look at the official docs.

You can learn more about xUnit.net’s test class fixtures in the samples repository.

Alright, that’s all for now. It is great to see that implementing automated database integration tests is actually fairly straightforward using .NET, xUnit.net, and Entity Framework. Even if it isn’t quite as easy as it is in Rails, it is perfectly doable.

Full-text search on a budget: Xapian

2021-08-19T00:00:00+00:00

Over the years I’ve seen and implemented different full-text search applications using various technologies: plain SQL, PostgreSQL, Elasticsearch, Solr, and most recently Xapian.

While Solr and Elasticsearch are very well known, Xapian, despite the fact that it’s available and packaged in all the major GNU/Linux distributions, doesn’t seem to be so popular, at least not among project managers.

But Xapian is fast, advanced, can be configured to do faceted searches (so the user can filter the search results), and my favorite, is fast to build and has virtually no maintenance overhead.

Its main feature is that it’s not a stand-alone application, like Solr or Elasticsearch, but instead it’s a library written in C++ which has bindings for all the major languages (as advertised on its homepage). It has also great documentation.

Now, being in the e-commerce business, my typical use-case is that the client’s shop needs something faster and better than a search using a SQL query against the products table. And beware, even implementing a non-trivial SQL-based search can burn more hours than setting up Xapian.

With Xapian you can prototype very quickly, without losing hours wading through obscure options, setting up services and configuring firewalls. And yet, the prototype will allow you to build more advanced features once you need them.

I’m a Perl guy, so I will show you some Perl code, but the procedure is the same for the other languages. Even the documentation can be built specifically for your language!

Typically, to add a search engine to your site you need two pieces: an indexer to which you feed the data (from static files or databases or even fetching remote pages or whatever you need) and the search itself in the site.

Both the indexer and the search code need to load the Xapian library and point to the same Xapian database, which is usually a directory (or a file pointing to a directory).

Indexing

Now, stripped down to the minimum, this is what a typical indexer’s code looks like:

#!/usr/bin/env perl
use utf8;
use strict;
use warnings;
use Search::Xapian (':all');
use JSON qw/encode_json/;

my $dblocation = "xapiandb";
my $xapian = Search::Xapian::WritableDatabase->new($dblocation, DB_CREATE_OR_OPEN);
my $indexer = Search::Xapian::TermGenerator->new;
$indexer->set_database($xapian);
$indexer->set_stemmer(Search::Xapian::Stem->new("english"));
my @entries = ({ uri => '/blog/1', title => 'T1', text => 'Marco loves pizza' },
               { uri => '/blog/2', title => 'T2', text => 'They love chapati' });
foreach my $data (@entries) {
    my $doc = Search::Xapian::Document->new;
    my $qterm = 'Q' . $data->{uri};
    $doc->add_term($qterm);
    $doc->set_data(encode_json({ uri => $data->{uri}, title => $data->{title} }));
    $indexer->set_document($doc);
    $indexer->index_text($data->{text});
    $xapian->replace_document_by_term($qterm, $doc);
}

This code will create a xapiandb directory with the Xapian database, indexing the blog posts in the @entries array. In a real script, they would come from the database.

Still, there are a couple of things worth noting in this minimal code.

We set the stemmer for the given language, so the text passed to the indexer via the index_text call is parsed.

Then we store the data structure we want to retrieve later with set_data. The best thing to do is probably to serialize it with JSON, in this case I’m excluding the full text, which we don’t need in the output (but it would be wise to add a teaser).

Also, we use a Q prefix to produce an unique term to update the entry when it already exists.

Of course the indexer will need to grow if you need more power and more structured data (like filtering or searching a specific field), but at this point we want just to show something to our hypothetical client.

The database can be inspected very easily. Xapian comes with a tool called delve (or xapian-delve):

$ xapian-delve xapiandb -a -v -1
All terms in database (termfreq):
Q/blog/1 1
Q/blog/2 1
Zchapati 1
Zlove 2
Zmarco 1
Zpizza 1
Zthey 1
chapati 1
love 1
loves 1
marco 1
pizza 1
they 1

And you can also try a search from the command line with quest:

$ quest -d xapiandb "loves NOT chapati"
Parsed Query: Query((Zlove@1 AND_NOT Zchapati@2))
Exactly 1 matches
MSet:
1: [0.0953102]
{"title":"T1","uri":"/blog/1"}

$ quest -d xapiandb "pizza OR chapati"
Parsed Query: Query((Zpizza@1 OR Zchapati@2))
Exactly 2 matches
MSet:
1: [0.405465]
{"title":"T1","uri":"/blog/1"}
2: [0.405465]
{"uri":"/blog/2","title":"T2"}

As the example above shows, it should be clear that:

the search works as you would expect (with logical operators) out of the box
the stemming works, searching for “loves” and “love” is the same.
the results give us back the JSON we stored in the index.

Searching

So let’s call it done and move to the next part, the searcher.

Now, while the indexer is a single script, the search needs to be plugged into the live code of your site. For the purposes of this article, I will provide a script instead, which does basically the same thing as quest. Plugging it into the web application is left as an exercise for the reader. I would also suggest to put both the indexing and searching code in a single shared module, keeping the logic in a single location.

#!/usr/bin/env perl
use utf8;
use strict;
use warnings;
use Search::Xapian ':all';
use JSON;

my ($cgi) = join(' ', @ARGV);
my $dblocation = "xapiandb";
my $database = Search::Xapian::Database->new($dblocation);
my $enquire = Search::Xapian::Enquire->new($database);
my $qp = Search::Xapian::QueryParser->new;
$qp->set_database($database);
$qp->set_stemmer(Search::Xapian::Stem->new("english"));
$qp->set_stemming_strategy(STEM_SOME);
$qp->set_default_op(OP_AND);
my $query = $qp->parse_query($cgi, FLAG_PHRASE|FLAG_BOOLEAN|FLAG_WILDCARD);
$enquire->set_query($query);

# fetch the first 50 results
my $mset = $enquire->get_mset(0, 50);
print "Total results: " . $mset->get_matches_estimated . "\n";

my $json_pretty = JSON->new->pretty(1)->utf8(1)->canonical(1);
foreach my $m ($mset->items) {
    my $data = decode_json($m->get_document->get_data);
    # decode and reencode the json in a human-readable fashion
    print $json_pretty->encode($data);
}

If you’re wondering what those constants are and where to look for more, they are in the module’s documentation, in plain sight (we asked for them when loading the module with the :all argument).

Most of the code shown here is boilerplate, but that could change once you build up. Notably we set the stemmer for the current language and the query parser options, so we can use wildcard (e.g. piz*), the AND/OR operators, and quoting.

Let’s see the script in action.

Wildcard:

$ ./search.pl 'piz'
Total results: 0
$ ./search.pl 'piz*'
Total results: 1
{
   "title" : "T1",
   "uri" : "/blog/1"
}

Operators:

$ ./search.pl 'pizza OR chapati'
Total results: 2
{
   "title" : "T1",
   "uri" : "/blog/1"
}
{
   "title" : "T2",
   "uri" : "/blog/2"
}
$ ./search.pl 'pizza AND chapati'
Total results: 0

Quoting (beware here the double quotes to escape the shell):

$ ./search.pl '"loves chapati"'
Total results: 0
$ ./search.pl '"love chapati"'
Total results: 1
{
   "title" : "T2",
   "uri" : "/blog/2"
}

The whole thing already looks pretty good. Way better (and way faster to code and to execute) than a home-baked SQL search.

As already noted, this is just scratching the surface. Xapian can do much more: filtering, range queries, facets, sorting, even spelling corrections!

I don’t doubt that Solr & co. have their use-cases, but for the common scenario of a small/mid-sized e-shop or site, I think that this solution is more affordable and maintainable than having a whole separate application (like a Solr server) to maintain, upgrade and secure. Don’t forget that here we haven’t done a single HTTP request. We didn’t have to manage daemons, opening/closing ports, and the like. We didn’t have to configure a schema and a tokenizer in a separate application (and keep that aligned with the handling code). It’s all there in our (Perl) code in two files (as already noted, the logic should live in a single module).

We just installed a library (there is a very good chance that it’s already installed) and a Perl module.

The Xapian database lives on the disk and your code has full control over it. Also it’s normally your GNU/Linux distribution taking care of the security upgrades.

If your client is on a budget, building a full-text search Xapian can be the right choice, and you can scale it up on the go, as more features are required.

Database Design: Using Composite Keys

2021-05-20T00:00:00+00:00

Photo by Chuttersnap

Whether to use single-column or composite keys is another long-debated subject of database design. I previously wrote to support using natural keys and here I want to make good arguments for using composite keys.

Single-column vs. Composite

Single-column keys are widely used nowadays. I wouldn’t be surprised if many developers today don’t even think database design with composite keys is possible, even though they were essential in the beginning. Relational databases make no assumption that the keys must be composed of a single column.

Let’s see the composite keys with the corporate database example again. First, we’d need departments and employees:

CREATE TABLE departments (
  department_id text NOT NULL,
  department_location text NOT NULL,

  PRIMARY KEY (department_id)
);

CREATE TABLE employees (
  username text NOT NULL,
  department_id text NOT NULL,

  PRIMARY KEY (username),
  FOREIGN KEY (department_id) REFERENCES departments
);

Then our database grows, and we need to split the departments into multiple teams. Here’s what they’ll look like:

| id | department  | team           | members |
| -- | ----------- | -------------- | ------- |
|  1 | sysadmin    | infrastructure |       5 |
|  2 | sysadmin    | internal_tools |       3 |
|  3 | development | internal_tools |       4 |
|  4 | development | web_site       |       8 |

As you noticed there are 2 teams named internal_tools, so we cannot use this as the primary key column. We can add a surrogate auto-increment column to use as the primary key, or make the department and team the primary key. Let’s go with the surrogate key option first to demonstrate the problem:

CREATE TABLE teams (
  team_id int NOT NULL GENERATED ALWAYS AS IDENTITY,
  department_id text NOT NULL,
  team_name text NOT NULL
  team_members int NOT NULL,

  PRIMARY KEY (team_id),
  UNIQUE (department_id, team_name),
  FOREIGN KEY (department_id) REFERENCES departments
);

As you noticed, we used the surrogate column as the primary key, and added an additional unique index to ensure the team name to be unique in with the department. Now, let’s relate the employees with the teams:

ALTER TABLE employees
  ADD COLUMN team_id int NOT NULL,
  ADD FOREIGN KEY (team_id) REFERENCES teams;

Now, we know both the department and the team of an employee, but the problem is that they can point to inconsistent rows. For example, I can INSERT myself as an employee:

| username | department_id | team_id |
| -------- | ------------- | ------- |
| hasegeli | sysadmin      |       3 |

team_id 3 is in development department, so now you’d never know if I am in the sysadmin or development department. This is a very common source of data integrity problems in the databases. Applications have no good option to handle this. They would typically crash or sometimes show the employee in one department and sometimes in the other.

You cannot easily add a constraint to the database to prevent this. The best option would be to remove the department_id when the team_id is added as we know the department of the team anyway, but this option is expensive and not always possible, for example when the team_id can be NULL.

Relations with Composite Keys

Now, let’s create the teams table with a composite key:

DROP TABLE teams CASCADE;

CREATE TABLE teams (
  department_id text NOT NULL,
  team_id text NOT NULL,
  team_members int NOT NULL,

  PRIMARY KEY (department_id, team_id),
  FOREIGN KEY (department_id) REFERENCES departments
);

ALTER TABLE employees
  ALTER COLUMN team_id TYPE text,
  ADD FOREIGN KEY (department_id, team_id) REFERENCES teams;

With this method, we ensure data integrity and don’t need to disturb the existing users of the employees table while adding the teams. They can still reliably use the department_id column. We can also set the team_id as NULL and still maintain the data integrity.

Ease of Change

As we have already seen, one of the benefits of using composite keys is to respond to database model changes easier and without compromising data integrity. This becomes relevant in many real world scenarios. For example, let’s add the employees to rooms which must belong to the same department:

CREATE TABLE rooms (
  department_id text NOT NULL,
  room_id text NOT NULL,
  room_location text NOT NULL,

  PRIMARY KEY (department_id, room_id),
  FOREIGN KEY (department_id) REFERENCES departments
);

ALTER TABLE employees
  ADD COLUMN room_id text,
  ADD FOREIGN KEY (department_id, room_id) REFERENCES rooms;

Nothing special needed to be done in here, and it is often easy to respond to more complicated change requests. Composite keys play well with database constraints. For example, let’s add an employee rank which needs to be unique in every department:

ALTER TABLE employees
  ADD COLUMN employee_rank int,
  ADD CHECK (employee_rank > 0),
  ADD UNIQUE (department_id, employee_rank);

This was so easy because we have the department_id in here. Now, let’s imagine a more complicated change request. We need to divide the departments into sections:

CREATE TABLE sections (
  department_id text NOT NULL,
  section_id text NOT NULL,
  section_location text NOT NULL,

  PRIMARY KEY (department_id, section_id),
  FOREIGN KEY (department_id) REFERENCES departments
);

ALTER TABLE teams
  ADD COLUMN section_id text NOT NULL,
  DROP CONSTRAINT teams_pkey CASCADE,
  ADD PRIMARY KEY (department_id, section_id, team_id),
  ADD FOREIGN KEY (epartment_id, section_id) REFERENCES sections;

ALTER TABLE employees
  ADD COLUMN section_id text NOT NULL,
  ADD FOREIGN KEY (department_id, section_id, team_id) REFERENCES teams;

As you see, we can make this happen with minimal impact to the users of the database. As a final practice, let’s reserve employee ranks between 1 and 10 for the main section of every department:

ALTER TABLE employees
  ADD CHECK (section_id = 'main' OR employee_rank > 10);

Querying

Another advantage of using composite keys is to have more possibility for joining of tables. For example using the tables we created we can join employees to sections without using the teams tables. This would not be possible if we had used single-column keys everywhere.

Join conditions get complicated with composite keys. The USING clause helps. To demonstrate, let’s join all of the tables we created so far:

SELECT *
  FROM employees
    JOIN deparments USING (department_id)
    JOIN sections USING (department_id, section_id)
    JOIN rooms USING (department_id, room_id)
    JOIN teams USING (department_id, section_id, team_id);

The USING clause only works if you name the columns the same on all tables. I recommend doing so.

Another advantage of USING clause is to eliminate duplicate columns on the result set. If you run this query, you would not see the department_id column repeated 5 times.

Performance Considerations

One disadvantage of using composite keys is to store more data on tables as references. You would also need more space for the indexes as the reference columns often need to be indexed. However storage is the cheapest of resources, and the performance advantages easily outweigh the extra storage.

The main performance advantage of using composite keys is eliminating the need for many joins as mentioned before. However, when you do need to join many tables, the query planner would have many different paths. It’s the query planners’ strong suit to find the best join order. Composite keys allow them to come up with better plans in many scenarios. To demonstrate this, let’s get our join-all-tables query and add some WHERE conditions:

SELECT *
  FROM employees
    JOIN deparments USING (department_id)
    JOIN sections USING (department_id, section_id)
    JOIN rooms USING (department_id, room_id);
    JOIN teams USING (department_id, section_id, team_id)
  WHERE username LIKE 'a%' AND
    department_location LIKE 'b%' AND
    section_location LIKE 'c%' AND
    room_location LIKE 'd%';
    team_members > 3;

Now we added 5 conditions using 5 columns on 5 different tables. The query planner can estimate which conditions are more selective and plan to join the tables from the smaller one to bigger one.

Spatial queries with MySQL

2021-03-24T00:00:00+00:00

Photo by Francois Powell, CC BY 2.0, cropped

MySQL is one of the most widely used relational databases. Most PHP websites rely on MySQL for persisting their information, which makes it one of the DB-Engines top four most popular databases along with Oracle, SQL Server, and PostgreSQL.

One of its capabilities that is not very well known is that the engine supports working with spatial data, allowing you to save different shapes (points, lines, polygons) and querying information based on intersections, distances, or overlaps. This capability was included in MySQL a long time ago, but it became easier to use starting in version 5.6, when the distance and point intersection functions were added.

Spatial data can be useful for many needs, including:

Searching for places based on latitude/longitude coordinates
Displaying information and areas as layers on maps
Architecture or home design applications

My first experience with spatial queries was for a weather website I developed that displays local alerts/warnings on a map using MySQL spatial functions to return active weather alerts for a given location, or to inform if lightning has been observed near the user’s current coordinates. So far, MySQL has given me all the resources I need to do such operations with relatively good performance and without needing to write lots of custom code.

Adding spatial information

There are many resources available to import spatial information into our database. From the United States Census Bureau we can find a set of shapefiles with all US states and counties. The Back4App social database platform also has many datasets available to download for free.

Of course, we can also create a table ourselves that contains any kind of spatial information. In the example below, we will create a table named restaurants which will have a name and location (lat/long) geometry for each row.

CREATE TABLE restaurants (
  name VARCHAR(100),
  location GEOMETRY NOT NULL,
  SPATIAL INDEX(location)
);

Note that we are adding an index of type SPATIAL to the location field. This is mainly for performance reasons. Once the table is created, we can populate it with some data using the ST_GeomFromText() function that will convert a string representation of any shape into a geometry:

INSERT INTO restaurants VALUES ('Restaurant 1', ST_GeomFromText('POINT(-26.66115 40.95858)'));
INSERT INTO restaurants VALUES ('Restaurant 2', ST_GeomFromText('POINT(-26.68685 40.93992)'));
INSERT INTO restaurants VALUES ('Restaurant 3', ST_GeomFromText('POINT(-31.11924 42.39557)'));

Querying and filtering with spatial data

Let’s suppose we imported a list of US states from the website above, along with a shape field that holds the geometry associated with each state into a states table. Then, we will be able to get the geometry by just running a SELECT statement:

SELECT state_name, ST_AsText(shape) FROM states ORDER BY state_name;

*-----------------------------*
| state_name | shape          |
*-----------------------------*
| Alabama    | POLYGON((...)) |
| Alaska     | POLYGON((...)) |
| Arizona    | POLYGON((...)) |
| ...                         |
*-----------------------------*

The ST_AsText() function will convert the shape contents into a string representation for us to read and parse, doing the inverse process of the ST_GeomFromText() function we used above. In our application, we can then parse that string and process it the way we need.

Intersecting

For example, we can get a list of states that intersects with a given shape. To do that, we will use the ST_Intersects() function, which will return a boolean value indicating if two shapes intersect or not.

SELECT s.state_name FROM states s WHERE ST_Intersects(s.shape, ST_GeomFromText(?)) ORDER BY s.state_name;

We can define our geometry to be any shape we need. For example, if we want to know which state contains a given point, we can set our parameter to something like 'POINT(40.69 -74.25)' which should return the state of New York.

If we have a polygon, we can set the parameter to specify a polygon shape, i.e. 'POLYGON((40.69 -74.25, 41.10 -74.25, 41.10 -76.11, 40.69 -76.11, 40.69 -74.25))' will mean a rectangle that intersects the states of New York and Pennsylvania. In that case, our query will return two rows with both state names.

Spherical distance

Let’s get back to our restaurants table. If we want to get a list of restaurants that are close to our location, we can use the ST_Distance_Sphere() function that will return a distance in meters between two points in a sphere (defaulting to Earth’s radius):

SELECT * FROM restaurants WHERE ST_Distance_Sphere(location, ST_GeomFromText(?)) <= 10 * 1000 ORDER BY name;

This query will give us a list of restaurants, sorted by name, that are within 10 kilometers from the point we pass as a parameter. We should set the parameter as 'POINT(lat long)' where lat and long represent our current geolocation.

We can use the distance function for many other purposes, like finding people that are close to each other or planning a road trip for the user.

Conclusion

We can avoid writing our own routines to handle spatial data by using MySQL’s built-in spatial types and methods. For most websites, MySQL offers a set of functions that will provide support for most common scenarios. And starting with version 8.0 MySQL includes extended support for geographic and ellipsoid computations.

We should consider that, if we need a complex solution for a robust spatial-based enterprise application, there are alternatives like PostgreSQL’s PostGIS that might be better suited to our purpose. MySQL lacks some advanced features like transformations, custom topology handling, or BRIN indexes, that might have an impact on our development process, depending on our needs.

The image below shows several weather alerts rendered on a Leaflet.js map that were fetched from a field of type GEOMETRY in MySQL:

Resources

There are many other functions we can use to work with our spatial data, from getting the centroid associated to a shape in order to automatically center a map view, to simplifying/optimizing existing geometry. I recommend reviewing the following links to start trying things out:

Database Design: Using Natural Keys

2021-03-15T00:00:00+00:00

Whether to use natural or surrogate keys is a long-debated subject of database design. I am a fan of using natural keys. I think there are even more compelling reasons to use natural keys in databases as the systems grow more complex and interdependent.

Natural or Surrogate

Let’s start by what we mean by natural. It’s not trivial to define this. In today’s world of APIs, someone’s surrogate key is another’s natural key. Wikipedia defines natural keys as “a type of unique key in a database formed of attributes that exist and are used in the external world outside the database”. This makes it clear that the keys we get from APIs are our natural keys. But how about the ones generated by us to be used in the external world?

When applications expose the keys on the URLs and APIs, others start relying on them. This is where our choices become important. When all those different applications generate their own keys instead of using the keys they got from other places, life becomes difficult for no reason.

Let’s elaborate with an example corporate database where employees are identified by their usernames and the departments with their domains. So our data would look like this:

| department        | username  | job                     |
| ----------------- | --------- | ----------------------- |
| sysadm.corp-x.com | hasegeli  | Database Administrator  |
| sysadm.corp-x.com | john      | System Administrator    |
| dep1.corp-x.com   | jane      | Developer               |

When we design this using surrogate keys, it’d look like this:

CREATE TABLE departments (
  id int NOT NULL GENERATED ALWAYS AS IDENTITY,
  domain text NOT NULL,

  PRIMARY KEY (id),
  UNIQUE (domain)
);

CREATE TABLE employees (
  id int NOT NULL GENERATED ALWAYS AS IDENTITY,
  username text NOT NULL,

  PRIMARY KEY (id),
  UNIQUE (username)
);

CREATE TABLE department_employees (
  id int NOT NULL GENERATED ALWAYS AS IDENTITY,
  department_id int NOT NULL,
  employee_id int NOT NULL,
  job text NOT NULL,

  PRIMARY KEY (id),
  UNIQUE (department_id, employee_id),
  FOREIGN KEY (department_id) REFERENCES departments (id),
  FOREIGN KEY (employee_id) REFERENCES employees (id)
);

Then, we would have to make a query joining them all together:

SELECT d.domain, e.username, de.job
FROM department_employees de
JOIN departments d ON de.department_id = d.id
JOIN employees e ON de.employee_id = e.id;

Instead, we could just have it as we need it on a single table:

CREATE TABLE department_employees (
  domain text NOT NULL,
  username text NOT NULL,
  job text NOT NULL,

  PRIMARY KEY (domain, username)
);

And, we could just query it:

SELECT domain, username, job
FROM department_employees;

Good Examples of Natural Keys

A primary key must be always available (NOT NULL) and unique. It should also be compact and easy to represent. Here are some good examples of natural keys:

Keys acquired from various APIs
Usernames
Domains
Email addresses
ISO country codes
IATA airport codes
Dates

I think easy representation matters for unchanged use among many systems. For that reason, I don’t think names with spaces, symbols, or non-ASCII Unicode letters are good keys.

UUIDs and hashes/digests like MD5 and SHA are also popular as primary keys. I don’t think they are a good choice as surrogate keys. Auto-incrementing IDs are easier to use, generate, store, predict — although, when they are defined in the external world and we have to store them, there’s nothing wrong with using them as the primary keys.

Another consideration when selecting keys is their stability. Things that can change are not good keys. This is not only because it is more work to change the keys everywhere, but because different parts of the system cannot rely on them.

The context matters to evaluate a key’s stability. For example, a phone number is not a good key for the users table, because users may change their phone number. However, it is perfect for the phone numbers table.

Also, I don’t think we should discard some useful keys just because they can change in an edge case. It is a good trade-off to allow the rare change rather than to complicate the system with a surrogate key.

Enterprise Architecture

I don’t know what you understand from the word “enterprise” but I have heard people advocating streamlined, unpractical solutions in the name of “enterprise” for a long time. I have seen people using layers of layers in complex frameworks and ORMs with caches and buffers, managers and supervisors, factories and workers, locks and semaphores, queues and consumers, all in the name of “enterprise architecture”.

Many of those frameworks have a rule like requiring all database tables to have a single-column auto-incremented primary key named “id”. Such a rule is easy to enforce and follow but it’s limiting. It would not be practical, maybe not even possible to use natural keys as the primary keys within such framework.

Even though I am writing this post to support using natural keys, strong conventions built around surrogate keys do certainly help developing complex systems in the “enterprise” environments. They help to keep systems consistent when the developers are more motivated to write code than to read what’s there, while working until they get a better offer from another company.

Performance Consideration

Performance is often an argument in favour of surrogate keys. Surrogate keys can be small auto-incrementing integers which would keep the indexes small, next branches easy to predict, so well performing.

Surrogate keys would take a bit of extra space since otherwise we don’t need to have them at all, but the space savings of using them as foreign keys would easily outweigh this. However, using surrogate keys may require the queries to have more tables joined together.

Some relational databases maintain the tables indexed by the primary key. Such systems are more affected by the size of the primary key. On PostgreSQL all indexes are secondary including the primary key, therefore the primary key is not any more important than the other indexes on the table.

Composite Keys

Natural keys are often unique together with other keys. This forces the database designer to use multiple columns as keys. Such design has many advantages, which I may cover in future blog posts.

Job opening: PostgreSQL database administrator & developer

2020-11-13T00:00:00+00:00

Photo by manhhai, cropped & filtered, CC BY 2.0

We are seeking a full-time PostgreSQL database administrator and developer to support and consult with our clients. For this role we prefer those living in the Western Hemisphere.

End Point is an Internet technology consulting company based in New York City, with 50 employees serving many clients ranging from small family businesses to large corporations. The company turned 25 years old this year!

Even before the pandemic most of us worked remotely from home offices. We collaborate using SSH, Git, project tracking tools, Zulip chat, video conferencing, and of course email and phones.

What you bring:

5+ years of hands-on professional experience and technical expertise with:

Database design and systems architecture
Advanced SQL and query optimization
Programming for web applications, stored procedures, and scripting, in Ruby, Java, Python, PHP, Perl, JavaScript, C#, Go, PL/pgSQL, bash, etc.
Performance tuning, scalability, connection pooling
Cloud database hosting services, especially at AWS, Google Cloud, and Azure
Security and encryption
Replication, PITR, backup, monitoring, and relevant tools
HA, failover, and disaster recovery
Database upgrades and migrations
Programming language interfaces to the database
Linux system administration and observation
DevOps and “configuration as code”
Troubleshooting problems and handling emergencies
Other databases such as MySQL, MongoDB, Oracle, SQL Server, Redis, Solr, Elasticsearch, etc.
Git version control
Bonus for familiarity with running PostgreSQL on Windows, and with the Bucardo replication system

These work traits are just as important:

Strong verbal and written communication skills
An eye for detail
Tenacity in solving problems and focusing on customer needs
A feeling of ownership of your projects
Work both independently and as part of a team
Ability to pass a criminal background check or get a security clearance when clients require
A good remote work environment

What work here offers:

Collaborate with knowledgeable, friendly, helpful, and diligent co-workers around the world
Flexible, sane work hours
Paid holidays and vacation
Annual bonus opportunity
For U.S. employees: health insurance subsidy and 401(k) retirement savings plan
Freedom from being tied to an office location
Use your desktop OS of choice: Linux, macOS, Windows
Use open source tools and contribute back as opportunity arises

Get in touch with us:

~~Please email us an introduction to jobs@endpointdev.com to apply.~~ (This job has been filled.) Include your location, a resume/CV, your Git repository or LinkedIn URLs, and whatever else may help us get to know you.

We look forward to hearing from you! Direct work seekers only, please—this role is not for agencies or subcontractors.

We are an equal opportunity employer and value diversity at our company. We do not discriminate on the basis of sex/gender, race, religion, color, national origin, sexual orientation, age, marital status, veteran status, or disability status.

Detecting gaps in time-series data in PostgreSQL

2020-10-26T00:00:00+00:00

Photo by Phoebe Baker

A client has a number of data feeds that are supposed to update at regular intervals. Like most things in the universe (thanks, entropy!) this does not always end up working out the way we want. We recently discovered that some of the data had not loaded as expected, and were brought in to assess the extent of the issue.

There were 2 main feeds that had issues, with different types of time data. The first had date-based data with batches and the second had full-range timestamps. We will examine each type individually, since they have similar — but not the same — characteristics.

Date-based, with batches

In the first data feed there was a table which tracked which files had been loaded by file date and batch number (there was a morning/evening batch designated by a batch field with either A or B). Since the files that had not loaded successfully did not have entries in this table we could find that one or both of the day’s batch files would be missing, but since the table tracked those which did get loaded, we needed to turn this list into something useful.

Any time I find myself considering sequences of data, I remember my friend generate_series(). While generate_series() is often called with numeric data, it can also generate date ranges, which can come in very handy:

postgres=# SELECT * FROM generate_series('2020-01-01'::date, '2020-10-01'::date, interval '1 day') LIMIT 10;
    generate_series
------------------------
 2020-01-01 00:00:00-06
 2020-01-02 00:00:00-06
 2020-01-03 00:00:00-06
 2020-01-04 00:00:00-06
 2020-01-05 00:00:00-06
 2020-01-06 00:00:00-06
 2020-01-07 00:00:00-06
 2020-01-08 00:00:00-06
 2020-01-09 00:00:00-06
 2020-01-10 00:00:00-06
(10 rows)

Let’s create our sample table and populate with data to simulate the situation we ran into:

CREATE TABLE loader_manifest (filename text, batch char, status char, processed_at timestamptz);

INSERT INTO loader_manifest (filename, batch, status, processed_at)
SELECT
    to_char(generate_series('2020-01-01'::date, '2020-10-22'::date, '1 day'), 'YYYYMMDD'),
    batch,
    'L',
    now()
FROM (VALUES ('A'::char), ('B')) batches(batch)
ORDER BY 1,2
;

-- create some gaps to later detect

DELETE FROM loader_manifest WHERE filename = '20200218' AND batch = 'B';
DELETE FROM loader_manifest WHERE filename = '20200426' AND batch = 'A';
DELETE FROM loader_manifest WHERE filename = '20201019';

The data that we were trying to match against looked similar to the following:

postgres=# SELECT * FROM loader_manifest LIMIT 5;
 filename | batch | status |         processed_at
----------+-------+--------+-------------------------------
 20200101 | A     | L      | 2020-10-18 11:25:05.878229-05
 20200101 | B     | L      | 2020-10-18 11:25:05.878229-05
 20200102 | A     | L      | 2020-10-18 11:25:05.878229-05
 20200102 | B     | L      | 2020-10-18 11:25:05.878229-05
 20200103 | A     | L      | 2020-10-18 11:25:05.878229-05
(5 rows)

Since either the batch or entire days could have been missing we basically needed to generate the list of expected combinations and use a LEFT JOIN to find which values in this table were missing. While this example has only 2 batch options, this solution would work for additional numbers of expected batches.

Because generate_series() is a set-returning function we can use it as a source in our FROM clause, in combination with the explicit values we want for our batch field. We also will need to extract the date pieces out in a way that we can match the expected filename format. We use something similar to the following:

postgres=# SELECT to_char(filename, 'YYYYMMDD') AS filename, batch FROM generate_series('2020-01-01'::date, '2020-10-01'::date, interval '1 day') AS filename, (VALUES ('A'), ('B')) AS batches(batch) LIMIT 10;
 filename  | batch
----------+-------
 20200101  | A
 20200101  | B
 20200102  | A
 20200102  | B
 20200103  | A
 20200103  | B
 20200104  | A
 20200104  | B
 20200105  | A
 20200105  | B
(10 rows)

So as we can see, this will generate the list of all expected filename/batch combinations that we can then check the original table against for missing values:

SELECT
    candidates.*
FROM (
    SELECT
        to_char(filename,'YYYYMMDD') AS filename,
        batch
    FROM
        generate_series('2020-01-01'::date, '2020-10-01'::date, interval '1 day') AS filename,
        (values ('A'), ('B')) AS batches(batch)
) candidates
NATURAL LEFT JOIN
    loader_manifest
WHERE loader_manifest.filename IS NULL

Here candidates is the generated list of all expected file/batch combinations.

And the results from our sample dataset:

 filename | batch
----------+-------
 20200218 | B
 20200426 | A
 20201019 | A
 20201019 | B
(4 rows)

As expected, this has identified the rows that we know are missing from the table. Success!

Non-date-based

The second type of query had to identify similar gaps in loaded data, but used a different approach. This data feed had a processing_time_utc field which would vary, but we expected this to have fairly regular updates. This data feed could be processed somewhat erratically, but if the data was complete we would expect to see records with processing_time_utc approximately every 10 minutes. If we found that there were gaps much larger than every 10 minutes we could suspect missing data and would need to locate/reprocess the underlying data files.

This approach also uses generate_series(), but since we do not have a single date that we can check existence of, the JOIN approach does not work directly. This means that in order to find the gaps we care about, the need to check that at least some data exists within each time period in question.

Here’s the query:

SELECT
    day
FROM generate_series('2020-07-01'::date, '2020-08-01'::date, interval '10 minutes') AS day
LEFT JOIN LATERAL
    (SELECT day FROM the_table WHERE processing_time_utc >= day AND processing_time_utc < day + interval '10 minutes' LIMIT 1) gap ON day = gap.day
WHERE gap.day IS NULL;

The key here is that we use generate_series() with an interval of the window size we care about, and then use that same interval when checking for any row’s existence that fits that criteria. With the LIMIT 1 on the gap subquery, we stop at the first record we find in that range, and thus we can exclude in the antijoin to find our results.

I should note that this approach works regardless of the size of the logical date partitions, and can be used to identify all gaps of at least the interval size we are looking at; the same query can check for gaps every 10 minutes as for hourly, daily, weekly, etc. It is purely a matter of selecting the proper interval size in the query in question.

Also note that the processing_time_utc field is indexed, or this would be painfully slow. We use a LATERAL query to pull in one specific row that matches the time interval we care about and ignore any others.

Be aware that this technique finds only the cases where no data is present for the time interval, not less than expected. You could certainly adapt this approach to that situation by using an appropriate query.

Summary

These approaches definitely come in handy when trying to isolate missing data based on time; which tack you take will depend on the expected data present, as well as the granularity in which you identify your gaps.

Using CTEs to do a binary search of large tables with non-indexed correlated data

2020-10-02T00:00:00+00:00

Photo by Colin Knowles, used under CC BY-SA 2.0.

Query optimization can take different forms depending on the data represented and the required needs. In a recent case, we had a large table that we had to query for some non-indexed criteria. This table was on an appliance that we were unable to modify, so we had to find a way to query efficiently without indexes that would have made it easier.

The straightforward approach for this query was something along these lines:

SELECT accounts.*
FROM accounts
JOIN logs
    ON accounts.id = logs.account_id
WHERE
    logs.created_at BETWEEN $1 - 'interval 1 minute' AND $1 + 'interval 1 minute' AND
    logs.field1 = $2 AND
    logs.field2 = $3
FETCH FIRST ROW ONLY

Unfortunately, none of the fields involved in this query were indexed, nor could they be, due to our access level on this database system. This lack of indexes means that our query against those fields would end up doing a sequential scan of the whole table which made things unacceptably slow. This specific table held time-series data with ~ 100k records per 1 minute period over a period of several weeks, which meant we were dealing with a lot of data.

While we could not create any additional indexes to help us with this query, we could use some specific properties to help us:

There was a primary key field, id, which was unique and monotonic, i.e., always increasing in value.
This table was append only; no updates or deletes, so once data existed in the table it was always the same.
The field we actually care about (created_at) also ends up being monotonic: subsequent records would always have the same or later values.
Since records were created sequentially and the id field was always increasing, id values and created_at fields would be together be generally monotonic; this means there is an indexed field which we can use as a surrogate stand-in for the target field that we want to treat as an index. While due to the nature of logging ingest it is possible the created_at and id values are not strictly monotonic (for instance if there are multiple logging records being created by separate ingest processes, of which ids get assigned in chunks), for our purposes this was close enough, since we were looking around in a time window which was wider than we were actually expecting the message to appear.
Since we are looking for fields matching a specific window of time, we can substitute the non-indexed clause created_at BETWEEN <timestamp_min> AND <timestamp_max> with an expression matching the indexed statement id BETWEEN <id of first id gt timestamp_min> AND <id of first id gt timestamp_max> to get the same effective approach.
In order to find the specific id fields which match the created_at time ranges we are interested in, we would need to find the first id value which matched the criteria created_at > 'timestamp'::timestamp, as all subsequent id values would match that condition as well. This would effectively require a binary search of the table to check which records match the criteria, and return the smallest id value for which this criteria held.

So now that we have identified how we can use an indexed surrogate key to substitute for the non-indexed expression, we need to figure out how to calculate the ranges in question.

Based on some recent discoveries about optimizing simple-looking but poorly-performing queries using more complicated queries¹, I had instincts that this could be solved with a WITH RECURSIVE Common Table Expression. After toying around for a while and not coming up with the exact solution, I ended up visiting the #postgresql channel on FreeNode IRC Network. There, I presented the problem and got some interested responses, as this is the exact kind of question that database ~~nerds~~ experts love². The solution that user xocolatl (Vik Fearing) came up with for a basic binary search is the basis for the rest of my solution:

CREATE TABLE test_table (id integer PRIMARY KEY, ts timestamptz);

INSERT INTO test_table
SELECT g, date 'today' + interval '1s' * g
FROM generate_series(1, 1000000) AS g;

WITH RECURSIVE
search (min, max, middle, level) AS (
    SELECT min(id), max(id), (min(id)+max(id))/2, 0
    FROM test_table
        UNION ALL
    SELECT v.min, v.max, (v.min + v.max)/2, s.level+1
    FROM search AS s
    CROSS JOIN LATERAL (
        SELECT *
        FROM test_table AS e
        WHERE e.id >= s.middle
        ORDER BY e.id
        FETCH FIRST ROW ONLY
    ) AS e
    CROSS JOIN LATERAL (VALUES (
        CASE WHEN e.ts < now() THEN e.id ELSE s.min END,
        CASE WHEN e.ts < now() THEN s.max ELSE e.id END
    )) AS v (min, max)
    WHERE (v.min + v.max)/2 NOT IN (v.min, v.max)
)
SELECT *
FROM search AS s
JOIN test_table AS e ON e.id = s.middle
ORDER BY s.level DESC
FETCH FIRST ROW ONLY;

As expected, the solution involved a WITH RECURSIVE CTE.

The basic explanation here is that the search expression first starts with the min, max, and middle (mean) values of id for the table (the initialization expression), then iteratively adds additional rows to the results depending on if the table row with id >= middle matches our specific test criteria, then continues until one of the boundaries of the region is hit. (Since we are using integer division in our terminal expression (v.min + v.max)/2 NOT IN (v.min, v.max) we are guaranteed to hit one of the boundary conditions eventually in our search.)

A few other things worthy of note:

This approach uses the check e.ts < now() as the condition we are testing, which means in this specific example, the answer to this “closest id” query would actually change depend on when you run this query relative to when the initial test data was populated. However, we can replace that condition with whatever condition we want to use to test for our surrogate non-indexed data.
This approach will work whether or not there are gaps in the sequence. In order to properly handle gaps, we are selecting the first row with id >= middle ... FETCH FIRST ROW ONLY rather than just selecting id = middle, which you could do in a gapless sequence.
In addition to not caring about gaps, we are also not trying to make sure this is using a balanced binary search; it would be not worth the computing effort to find the middlest existing row in an index, as we’d need to know the number of rows in the segment we’re searching. Since in PostgreSQL this would entail a COUNT(*) from a subselect, this would be quite slow and not worth the trouble.

Considering my specific use case was trying to limit the records considered based on two created_at values I needed to calculate a search_min and search_max to find the start/end ids for each side in the interval.

Given this, I just modified the CTE to calculate those and add the additional conditions we were wanting to consider. I also had to turn the result query from a join against the found id value to a range; the final query is as follows:

WITH RECURSIVE
search_min (min, max, middle, level) AS (
    SELECT min(id), max(id), (min(id)+max(id))/2, 0
    FROM logs
        UNION ALL
    SELECT v.min, v.max, (v.min + v.max)/2, s.level+1
    FROM search_min AS s
    CROSS JOIN LATERAL (
        SELECT *
        FROM logs AS e
        WHERE e.id >= s.middle
        ORDER BY e.id
        FETCH FIRST ROW ONLY
    ) AS e
    CROSS JOIN LATERAL (VALUES (
        CASE WHEN extract(epoch FROM e.created_at)::integer < $1 THEN e.id ELSE s.min END,
        CASE WHEN extract(epoch FROM e.created_at)::integer < $1 THEN s.max ELSE e.id END
    )) AS v (min, max)
    WHERE (v.min + v.max)/2 NOT IN (v.min, v.max)
),
search_max (min, max, middle, level) AS (
    SELECT min(id), max(id), (min(id)+max(id))/2, 0
    FROM logs
        UNION ALL
    SELECT v.min, v.max, (v.min + v.max)/2, s.level+1
    FROM search_max AS s
    CROSS JOIN LATERAL (
        SELECT *
        FROM logs AS e
        WHERE e.id >= s.middle
        ORDER BY e.id
        FETCH FIRST ROW ONLY
    ) AS e
    CROSS JOIN LATERAL (VALUES (
        CASE WHEN extract(epoch FROM e.created_at)::integer < $2 THEN e.id ELSE s.min END,
        CASE WHEN extract(epoch FROM e.created_at)::integer < $2 THEN s.max ELSE e.id END
    )) AS v (min, max)
    WHERE (v.min + v.max)/2 NOT IN (v.min, v.max)
)
SELECT accounts.*
FROM accounts
JOIN logs
    ON logs.account_id = accounts.id
WHERE
    logs.field1 = $3 AND
    logs.field2 = $4 AND
    logs.id >= (SELECT middle FROM search_min ORDER BY level DESC FETCH FIRST ROW ONLY) AND
    logs.id <= (SELECT middle FROM search_max ORDER BY level DESC FETCH FIRST ROW ONLY)
ORDER BY logs.id
FETCH FIRST ROW ONLY

The final results were drastically improved. The initial query went from timing out in the webservice in question to returning results in a fraction of a second. Clearly this technique, while not as useful as directly indexing data we care about, can come in handy in some circumstances.

Another tool for the toolbag!

1 (back): Improving max() performance in PostgreSQL: GROUP BY vs. CTE

2 (back): See nerdsniping

Improving max() performance in PostgreSQL: GROUP BY vs. CTE

2020-06-30T00:00:00+00:00

Photo by Maxpax, used under CC BY-SA 2.0, cropped from original.

When working with large tables, even simple actions can have high costs to complete. What queries are acceptable for smaller tables can often be less than ideal when applied to large tables, so your specific choice of approach to a given problem becomes more important.

Note: We are using PostgreSQL 12, which supports some nice features like parallel btree index building, which can speed up parts of this process compared to earlier versions. We are using the default settings for this, which lets PostgreSQL use up to 2 parallel backend workers to speed up some operations.

Say you have a table table_a with multiple grouping fields field_a and field_b and you want to find the maximum value of another field field_c for each group.

The direct approach is to do something like the following:

SELECT field_a, field_b, max(field_c) FROM table_a GROUP BY 1,2;

This is functional and very straightforward. However, even if you have an index on (field_a, field_b, field_c), this can end up taking quite a long time if the tables are large. Let’s look at an actual example and the numbers we use.

First, let’s create our table:

CREATE TABLE table_a (field_a varchar, field_b integer, field_c date);

And populate it with some data:

INSERT INTO table_a
SELECT field_a, field_b, now()::date + (random() * 100)::int AS field_c
FROM unnest(array['AAA','BBB','CCC','DDD','EEE','FFF']) field_a,
    generate_series(1, 10000) field_b,
    generate_series(1, 1000);

This statement will populate this table with 60 million rows, consisting of 1000 random dates per each field_a, field_b pair; our task will now be to see how to efficiently find the max value for field_c for each grouping.

Let’s now create an index on all 3 fields:

CREATE INDEX ON table_a (field_a, field_b, field_c);

For the purposes of sanity/clarity when testing approaches, let’s VACUUM and ANALYZE that table:

VACUUM ANALYZE table_a;

And let’s check out the plan:

postgres=# EXPLAIN SELECT field_a, field_b, max(field_c) FROM table_a GROUP BY 1,2;
                                             QUERY PLAN
-----------------------------------------------------------------------------------------------------
 Finalize GroupAggregate  (cost=767659.40..781742.03 rows=54510 width=12)
   Group Key: field_a, field_b
   ->  Gather Merge  (cost=767659.40..780379.28 rows=109020 width=12)
         Workers Planned: 2
         ->  Sort  (cost=766659.37..766795.65 rows=54510 width=12)
               Sort Key: field_a, field_b
               ->  Partial HashAggregate  (cost=761825.91..762371.01 rows=54510 width=12)
                     Group Key: field_a, field_b
                     ->  Parallel Seq Scan on table_a  (cost=0.00..574325.52 rows=25000052 width=12)
(9 rows)

We can see that the plan is using an index-only scan for our table, which is good, but it also is using a separate GroupAggregate group/gather in order to get the grouping done, which still has to iterate through all the rows in the results to find the final answers.

Best of 3 timings: 5.60s

Hypothetically, PostgreSQL could detect that we’re asking for a max() value from an index with the individual keys as group values and just derive the max() value directly from the index, but it does not currently have this sort of capability, so instead we will rewrite our query to include more of the smarts.

Since we have a btree index on all of the fields, we know that the max value is easy to find, so let’s consider the conditions in which we can find this:

For one of the grouping (field_a, field_b), we can find the maximum value for that group by using an ORDER BY clause and LIMIT 1, so if we knew the (field_a,field_b) pair the max could be found easily in the index with:

postgres=# SELECT field_c FROM table_a WHERE field_a = 'AAA' and field_b = 1 ORDER BY field_c DESC LIMIT 1;
  field_c
------------
 2020-10-01
(1 row)

Plan:

postgres=# EXPLAIN SELECT field_c FROM table_a WHERE field_a = 'AAA' AND field_b = 1 ORDER BY field_c DESC LIMIT 1;
                                                        QUERY PLAN
---------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=0.43..4.43 rows=1 width=4)
   ->  Index Only Scan Backward using table_a_field_a_field_b_field_c_idx on table_a  (cost=0.43..404.15 rows=101 width=4)
         Index Cond: ((field_a = 'AAA'::text) AND (field_b = 1))
(3 rows)

Since we want to find all the values in this table, we effectively want to do this query for each (field_a, field_b) pairing, but how can we iterate over this in an efficient way?

The keyword “iterate” should bring to mind a WITH RECURSIVE CTE, and here is where the trick comes in. Since we want all the values, we can use any of them to start with and end up checking against the last known value to find the next.

An important thing to know is that for a btree index over (a,b,c), there is a well-defined index ordering of all leading subsets of columns, so we can compare (a,b) against (a0,b0) in an indexed way and use this to our advantage. This means that ('AAA',1) < ('AAA',2) < ('BBB',1) just by virtue of how the row indexing works.

Using this property, we can then construct the following query:

WITH RECURSIVE t AS (
    (SELECT field_a, field_b, field_c FROM table_a ORDER BY field_a DESC, field_b DESC, field_c DESC LIMIT 1)
    UNION ALL
    SELECT s.field_a, s.field_b, s.field_c FROM t,
    LATERAL (
        SELECT field_a, field_b, field_c
        FROM table_a
        WHERE (table_a.field_a, table_a.field_b) < (t.field_a, t.field_b)
        ORDER BY field_a DESC, field_b DESC, field_c DESC LIMIT 1
    ) s
) SELECT * FROM t;

Wow, pretty different, eh? Breaking it down, we basically start with the most extreme value in the index, then recursively add the next row for fields (field_a, field_b), which is the next lowest value in the index.

Let’s see the plan:

                                                                            QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------------------------
 CTE Scan on t  (cost=66.41..68.43 rows=101 width=40)
   CTE t
     ->  Recursive Union  (cost=0.56..66.41 rows=101 width=12)
           ->  Limit  (cost=0.56..0.60 rows=1 width=12)
                 ->  Index Only Scan Backward using table_a_field_a_field_b_field_c_idx on table_a  (cost=0.56..1825278.43 rows=60000124 width=12)
           ->  Nested Loop  (cost=0.56..6.38 rows=10 width=12)
                 ->  WorkTable Scan on t t_1  (cost=0.00..0.20 rows=10 width=36)
                 ->  Limit  (cost=0.56..0.60 rows=1 width=12)
                       ->  Index Only Scan Backward using table_a_field_a_field_b_field_c_idx on table_a table_a_1  (cost=0.56..658429.28 rows=20000041 width=12)
                             Index Cond: (ROW(field_a, field_b) < ROW((t_1.field_a)::text, t_1.field_b))
(10 rows)

Again, taking the best of 3 timings, we get 0.86s.

As you can see, the CTE approach is 6–7 times faster over the same data for the same results.

This same approach can work for finding the min() value, by just changing the ORDER clauses and comparison operator:

Compare:

WITH RECURSIVE t AS (
    (SELECT field_a, field_b, field_c FROM table_a ORDER BY field_a, field_b, field_c LIMIT 1)
    UNION ALL
    SELECT s.field_a, s.field_b, s.field_c FROM t,
    LATERAL (
        SELECT field_a, field_b, field_c
        FROM table_a
        WHERE (table_a.field_a, table_a.field_b) > (t.field_a, t.field_b)
        ORDER BY field_a, field_b, field_c LIMIT 1
    ) s
) SELECT * FROM t;

Timing: 0.81s

vs. GROUP BY:

SELECT field_a, field_b, min(field_c) FROM table_a GROUP BY 1,2;

Timing: 5.45s

This technique is generalizable to any number of fields in the table, it just relies on using the same index as you would want for GROUP BY. This is another nice tool to have in the DBA toolbox.

Migrating large PostgreSQL databases

2020-04-21T00:00:00+00:00

Photo by Harshil Gudka on Unsplash

The challenge

One of our clients has a large and important health-related application. It’s built on an end-of-life Ruby on Rails-based open source framework, heavily customized over the years. They wanted to upgrade to a newer, supported, Java-based open source application a partner organization had developed as a replacement. Both organizations used the old system previously. To do that we would need to migrate all their existing PostgreSQL data from the old system to the new one, retaining important customizations while adapting to the new database schema.

Although there were many similarities between the old system and the new, the differences were significant enough to require careful study of the database schemas and the migration scripts designed to move the data:

There were schema-level differences between our old database and the partner organization’s old database.
Even where the two old databases were similar there were differences on the data level, such as different standards, different values in table records, different representation, etc.
We had different content, so if a script was working well for their data, it was not necessarily correct for us.
There were dynamically generated tables for both old databases and we had to find out how we can convert our current schema elements along with its records to the planned schema elements along with its records.

We had to understand the differences between our and their old databases. Due to the number of tables and average number of columns, manual comparison between databases was not really an option. We knew that the algorithm of handling the scripts would look like below:

For each S in Scripts
    Analyze S and understand the intent behind it
    Compute a read-only version of S to avoid write operations
    Execute the read-only version of S
    Analyze the results and find out whether they are different from the expected results
    Convert our read-only version of S to S′, where S′ is compatible with our expectations
    While there are technical issues do
        Fix it
    While end
    Execute S′
For end

Understanding these differences was easier said than done. The main problem is that we found it difficult to define our exact expectations. For that purpose we needed a deeper understanding about the databases.

Entity Relationship Diagram

We first needed to see an Entity Relationship Diagram (ER diagram or ERD). We used DbVisualizer for this.

We imported the database into our local RDBMS and then created a database connection by right-clicking Connections in the left menu tree

then clicking on Create Database Connection and selecting No Wizard

and filling in the data.

After that, we double-clicked on the schema we wanted to generate an ER diagram for and clicked on Open Object.

Finally, we clicked on the References tab and a bird’s-eye view of the ER diagram was generated.

Then we right-clicked on the diagram and clicked on Export.

We chose SVG and saved it. After opening the SVG diagram, we saw that the schema was too big to easily analyze, so we dropped the tables we were not specifically interested in in our local copy and generated a new ER diagram. It was super easy and cool. Finally, we were able to see which parts of the other team’s database was old and which was new. We were also able to compare our old database with the new database we were implementing.

Comparing our databases against their counterparts

Next we needed to understand what the schema differences were between their old database and our old database to determine what selections in the scripts will not work properly and to determine how we needed to modify it to fit our technical nuances.

We used Liquibase for this purpose. See Selva’s article on comparing PostgreSQL database schema versions.

The actual command we used was diff.

So, we needed to make sure we had a proper setup and then we could run the command. The example output the documentation gives is this:

Diff Results:
Reference Database: MYSCHEMA2 @ jdbc:oracle:thin:@localhost:1521:ORCL (Default Schema: MYSCHEMA2)
Comparison Database: MYSCHEMA @ jdbc:oracle:thin:@localhost:1521:ORCL (Default Schema: MYSCHEMA)
Compared Schemas: MYSCHEMA2 -> MYSCHEMA
Product Name: EQUAL
Product Version: EQUAL
Missing Catalog(s): NONE
Unexpected Catalog(s): NONE
Changed Catalog(s): NONE
Missing Check Constraint(s): NONE
Unexpected Check Constraint(s): NONE
Changed Check Constraint(s): NONE
Missing Column(s): NONE
Unexpected Column(s):
     MYSCHEMA.DEPARTMENT.ACTIVE
     MYSCHEMA.SERVICETECH.ACTIVE
     MYSCHEMA.SERVICETECH2.ACTIVE
     MYSCHEMA.SERVICETECH3.ACTIVE
     MYSCHEMA.VIEW1.ACTIVE
     MYSCHEMA.DATABASECHANGELOG.AUTHOR
     MYSCHEMA.DATABASECHANGELOG.COMMENTS
     MYSCHEMA.DATABASECHANGELOG.CONTEXTS
     MYSCHEMA.DATABASECHANGELOG.DATEEXECUTED
     MYSCHEMA.DATABASECHANGELOG.DEPLOYMENT_ID
     MYSCHEMA.DATABASECHANGELOG.DESCRIPTION
     MYSCHEMA.DATABASECHANGELOG.EXECTYPE
     MYSCHEMA.DATABASECHANGELOG.FILENAME
     MYSCHEMA.DATABASECHANGELOG.ID
     MYSCHEMA.DATABASECHANGELOGLOCK.ID
     MYSCHEMA.DEPARTMENT.ID
     MYSCHEMA.SERVICETECH.ID
     MYSCHEMA.SERVICETECH2.ID
     MYSCHEMA.SERVICETECH3.ID
     MYSCHEMA.VIEW1.ID
     MYSCHEMA.DATABASECHANGELOG.LABELS
     MYSCHEMA.DATABASECHANGELOG.LIQUIBASE
     MYSCHEMA.DATABASECHANGELOGLOCK.LOCKED
     MYSCHEMA.DATABASECHANGELOGLOCK.LOCKEDBY
     MYSCHEMA.DATABASECHANGELOGLOCK.LOCKGRANTED
     MYSCHEMA.DATABASECHANGELOG.MD5SUM
     MYSCHEMA.DEPARTMENT.NAME
     MYSCHEMA.SERVICETECH.NAME
     MYSCHEMA.SERVICETECH2.NAME
     MYSCHEMA.SERVICETECH3.NAME
     MYSCHEMA.VIEW1.NAME
     MYSCHEMA.DATABASECHANGELOG.ORDEREXECUTED
     MYSCHEMA.DATABASECHANGELOG.TAG
Changed Column(s): NONE
Missing Database Package(s): NONE
Unexpected Database Package(s): NONE
Changed Database Package(s): NONE
Missing Database Package Body(s): NONE
Unexpected Database Package Body(s): NONE
Changed Database Package Body(s): NONE
Missing Foreign Key(s): NONE
Unexpected Foreign Key(s): NONE
Changed Foreign Key(s): NONE
Missing Function(s): NONE
Unexpected Function(s): NONE
Changed Function(s): NONE
Missing Index(s): NONE
Unexpected Index(s):
     PK_DATABASECHANGELOGLOCK UNIQUE  ON MYSCHEMA.DATABASECHANGELOGLOCK(ID)
     PK_DEPARTMENT UNIQUE  ON MYSCHEMA.DEPARTMENT(ID)
     PK_SERVICETECH UNIQUE  ON MYSCHEMA.SERVICETECH(ID)
     PK_SERVICETECH2 UNIQUE  ON MYSCHEMA.SERVICETECH2(ID)
     PK_SERVICETECH3 UNIQUE  ON MYSCHEMA.SERVICETECH3(ID)
Changed Index(s): NONE
Missing Java Class(s): NONE
Unexpected Java Class(s): NONE
Changed Java Class(s): NONE
Missing Java Source(s): NONE
Unexpected Java Source(s): NONE
Changed Java Source(s): NONE
Missing Primary Key(s): NONE
Unexpected Primary Key(s):
     PK_DATABASECHANGELOGLOCK on MYSCHEMA.DATABASECHANGELOGLOCK(ID)
     PK_DEPARTMENT on MYSCHEMA.DEPARTMENT(ID)
     PK_SERVICETECH on MYSCHEMA.SERVICETECH(ID)
     PK_SERVICETECH2 on MYSCHEMA.SERVICETECH2(ID)
     PK_SERVICETECH3 on MYSCHEMA.SERVICETECH3(ID)
Changed Primary Key(s): NONE
Missing Sequence(s): NONE
Unexpected Sequence(s): NONE
Changed Sequence(s): NONE
Missing Stored Procedure(s): NONE
Unexpected Stored Procedure(s): NONE
Changed Stored Procedure(s): NONE
Missing Synonym(s): NONE
Unexpected Synonym(s): NONE
Changed Synonym(s): NONE
Missing Table(s): NONE
Unexpected Table(s):
     DATABASECHANGELOG
     DATABASECHANGELOGLOCK
     DEPARTMENT
     SERVICETECH
     SERVICETECH2
     SERVICETECH3
Changed Table(s): NONE
Missing Trigger(s): NONE
Unexpected Trigger(s): NONE
Changed Trigger(s): NONE
Missing Unique Constraint(s): NONE
Unexpected Unique Constraint(s): NONE
Changed Unique Constraint(s): NONE
Missing View(s): NONE
Unexpected View(s):
     VIEW1
Changed View(s): NONE
Liquibase command 'diff' was executed successfully.

Of course, we could do this job manually by listing all the tables with psql’s \dt and then checking each of them individually with \d tablename, but if there are many tables, this would take forever.

Yes, we can write software for this purpose, implementing an algorithm along the lines of

tables = <execute \dt>
For each (tables as table) do
    Differences[table] = difference(<execute \d table at db1>, <execute \d table at db2>)
End For

however, the algorithm above won’t handle special cases, like tables existing in db1 and not in db2 or vice versa. The algorithm above graciously outsources the arduous task of splitting the rows in both cases by identifying whether a row is a column name, an index, a foreign key, etc. and by identifying the subject of the line (e.g. column name) and finding the matches between the two to a function called difference.

It is of course implementable, but it would add a considerable amount of work. We should also mention that such a newly developed piece of code would not be well tested yet and we would have to watch out for possible bugs, create unit tests, create a nice UI or file export to ensure that we can analyse the results, and so on. All this work is unnecessary due to the availability of Liquibase and we are only talking about a single command compared to the many here.

Dynamically generated tables

In practical terms this means the data our software must manage does not fit a pre-established schema; users create and update new data collection forms regularly. These forms consist of sets of uniquely named questions and text-based answers to those questions. The PostgreSQL JSON data type may seem like a natural fit for such data. However, the original version of the software predates PostgreSQL’s now extensive JSON support. The software version from which we were upgrading stored these data in an Entity-Attribute-Value schema, a database pattern often maligned (justly) by database designers.

In this version, a single table stored all the answers given for every user-defined question for every case in the system, along with a pointer to the associated question and case. As one might expect, this table grew fairly large, though its principal drawbacks were not its size but rather the large number of joins necessary to process data it contained, and the lack of sufficient data validation. It is possible that the hstore data type would have been a better fit, however, neither programming language support for hstore data nor developer familiarity with it made it an obvious choice at the time. We did use hstore widely in the backend for data manipulation functions that could be contained entirely in SQL.

Fast forward to newer versions, where this schema has been redesigned. We weren’t involved in the design process and can’t comment on the justification behind this design decision, but the new version creates new tables within the database as needed for each data entry form, and text fields for each question on the form. This reduces the number of joins or aggregations necessary to compile all the data for one form for a single case, but it means creating SQL queries dynamically to create, and later to find, the tables and columns containing data of interest.

We’ve run our fingers through the data several times, both during and after the migration, and found neither schema variant satisfies our every wish. Both versions store users’ data as text fields, whatever data type they may represent. Some form of data validation at the database level would be very nice, and in the new version where each field has its own column in the database, this is entirely possible, though of course, it would have required more work in the development process. In particular, many questions expect answers taken from a predefined set, for which enumerated types could be a good fit. Of course, stored procedures could conceivably ensure valid data no matter its data type in the schema, but this doesn’t seem like a plausible option in practice. As a further drawback to the new approach, column and table names derive from user-defined data, meaning we need to sanitize user input to create valid PostgreSQL identifiers. This is a tricky process, and difficult to separate entirely into its own module to avoid reimplementing the same intricate logic multiple times.

JSON data types provide one possible schema alternative, with all entries for one data entry form for a single case stored in a single JSON field, and indeed the PostgreSQL documentation proposes its use in such situations. It’s not entirely clear, though, that this would be a win. We could define new keys within the JSON structure without needing to modify the database schema itself, and with JSON we’d always know exactly what table and field we needed, to find the data we were after, but we’d still need to write queries dynamically in order to pull the desired fields. We could avoid some of the data sanitization necessary to create field names, as the rules for JSON key names are far more permissive than for column names in a proper database table. But, again barring extensive stored procedures, we would still have very limited ability to validate data within the database itself, as JSON supports only a small set of primitive types.

Putting it all together

After we acquired the understanding that we needed we were able to work out the migration script according to the algorithm that we outlined at the start of this article.

This was still a long, labor-intensive task which was done by repeated pair-programming sessions but we were able to reach high enough accuracy. So high that to our great surprise we were able to start the application after the migration process was done.

Release

We were able to do the release on a weekend and the three of us moved on to solving problems submitted by beta testers. We called this process “dragon hunting”.

(Written with help from Selvakumar Arumugam and Joshua Tolley.)

A Tool to Compare PostgreSQL Database Schema Versions

2020-02-11T00:00:00+00:00

Photo by @kelvyn on Unsplash

The End Point development team has completed a major application migration from one stack to another. Many years ago, the vendor maintaining the old stack abandoned support and development. This led to a stack evolution riddled with independent custom changes and new features in the following years.

The new application was developed by a consortium that created migration scripts to transfer data to a fresh stack resulting in a completely restructured database schema. While we could not directly use those consortium migration scripts to our client application, attempting to create migration scripts from scratch would be tedious due to the many labor-intensive and time-consuming tasks. We looked to reuse and customize the scripts in order to ensure an exact match of the customized changes to the client’s application.

Liquibase: A Schema Comparison Tool

After an arduous hunt for a suitable solution, we came across Liquibase, an open-source schema comparison tool that utilizes the diff command to assess missing, changed, and unexpected objects.

Installation and Usage

Let’s see how to use Liquibase and review the insights and results offered by the diff command.

Before beginning, download the latest version of Liquibase. As the default package doesn’t have its own driver, it would be wise to add the PostgreSQL driver to the Liquibase lib folder. (You’ll need to do this with any other database types and their necessary libraries and drivers.)

$ wget https://github.com/liquibase/liquibase/releases/download/v3.8.5/liquibase-3.8.5.tar.gz
$ tar xvzf liquibase-3.8.5.tar.gz
$ wget https://repo1.maven.org/maven2/org/postgresql/postgresql/42.2.5/postgresql-42.2.5.jar -P lib/

$ ./liquibase \
--classpath="/path/to/home/apps/liquidiff/lib" \
--outputFile=liquibase_output.txt \
--driver=org.postgresql.Driver \
--url=jdbc:postgresql://localhost:5432/schema_two \
--username=postgres \
--password=CHANGEME \
--defaultSchemaName=public \
Diff \
--referenceUrl=jdbc:postgresql://localhost:5432/schema_one \
--referenceUsername=postgres \
--referencePassword=CHANGEME \
--referenceDefaultSchemaName=public

Comparison Results

The following output shows the list of all sections with missing, changed, and newly added objects to each section.

$ cat liquibase_output.txt

Reference Database: postgres @ jdbc:postgresql://localhost:5432/schema_one (Default Schema: public)
Comparison Database: postgres @ jdbc:postgresql://localhost:5432/schema_two (Default Schema: public)
Compared Schemas: public
Product Name: EQUAL
Product Version: EQUAL
Missing Catalog(s): NONE
Unexpected Catalog(s): NONE
Changed Catalog(s): 
     schema_one
          name changed from 'schema_one' to 'schema_two'
Missing Column(s): 
     public.users.settings
     ...
Changed Column(s): 
     public.table_one.unique_no
          type changed from 'varchar(20 BYTE)' to 'varchar(255 BYTE)'     
     public.table_two.created_at
          defaultValue changed from 'null' to 'now()'
          nullable changed from 'false' to 'true'
          order changed from '4' to '22'
    ...
Missing Foreign Key(s): 
     one_two_id_fkey(one[two_id] -> two[id])
    ...
Unexpected Foreign Key(s): 
    ...
Changed Foreign Key(s): 
    ...
Missing Index(s): 
    ...
Unexpected Index(s): 
     table_pkey UNIQUE  ON public.table(id)
    ...
Changed Index(s): 
     index_events_on_record_number ON public.table(record_number)
          unique changed from 'false' to 'true'
    ...
Missing Primary Key(s): ...
Unexpected Primary Key(s): ...
Changed Primary Key(s): ...
Missing Schema(s): NONE
Unexpected Schema(s): NONE
Changed Schema(s): NONE
Missing Sequence(s): ...
Unexpected Sequence(s): NONE
Changed Sequence(s): NONE
Missing Stored Procedure(s): NONE
Unexpected Stored Procedure(s): NONE
Changed Stored Procedure(s): NONE
Missing Table(s): ...
Unexpected Table(s): ...
Changed Table(s): NONE
Missing Unique Constraint(s): NONE
Unexpected Unique Constraint(s): NONE
Changed Unique Constraint(s): NONE
Missing View(s): ...
Unexpected View(s): NONE
Changed View(s): NONE
Liquibase command 'Diff' was executed successfully.

Conclusion

Although comparing and contrasting the 100+ tables from the old application was beneficial, the data migration was still challenging due to volume and variety of data. However, with the help of Liquibase, we became more familiar with differences in the schema, including table level, columns, references, indexes, views, etc.

This led to an increase in accuracy which was very helpful during the migration process. We hope that by sharing our findings, others will also benefit from this tool and all that it offers.

GraphQL — An Alternative to REST

2019-05-11T00:00:00+00:00

GraphQL has become more and more popular recently as an alternative to traditional RESTful APIs since it was released as open source by Facebook in 2015. According to the GraphQL website, it is “a query language for APIs and a runtime for fulfilling those queries with your existing data”. In this blog post, I’ll go over some of what makes GraphQL different from other API solutions, and then show how to get a GraphQL API up and running so you can try it out yourself!

GraphQL is designed to fit on top of your database layer. With the help of libraries like Apollo GraphQL, it can be used with many different databases. Some of the main differences between GraphQL and more traditional RESTful APIs include:

GraphQL uses one endpoint. Most traditional APIs use an endpoint for each type of data; in my example, you’d probably have one each for users (/user), posts (/post) and comments (/comment). Each of these would return some JSON with the data you want. GraphQL, on the other hand, lives at one endpoint (usually /graphql) and changes what it returns based on what you ask for, as detailed in the next point.
You can get multiple types in one request. For instance, if you want to get information about an author plus all of their posts, instead of making a request for the author and a request for posts, you do just one request for the author and specify that you’d like their posts as well:

query {
  user(id: "12345") {
    name
    posts {
      title
      body
    }
  }
}

You decide which parts of the data you want. Traditional REST APIs give you data based on which endpoint you’re querying (/post/:id, /user/:id, etc.), and the format of the data is generally the same. For instance, no matter which id you ask for at /posts/:id, you’ll always get something that looks like this back:

{
  "id":"123",
  "name":"Smash Mouth",
  "joined":"1994"
}

But what if we don’t need to know when they joined right now? Another example that better illustrates this problem (and GraphQL’s solution) is getting a list of blog posts. The home page of a blog usually shows several of the most recent posts, but you can also view a list of just post names that will go farther back. With a traditional REST API, you would have to create separate endpoints for each of these scenarios, for example /first_posts and /posts, or add GET parameters or similar. With GraphQL, you can just specify exactly what you want. The query to replace /first_posts might look like this:

query {
  posts {
    title
    body
    author {
      name
    }
  }
}

The data returned might look like:

{
  "data": {
    "posts": [
      {
        "title": "All Star",
        "body": "Somebody once told me the world is gonna roll me\n I ain't the sharpest tool in the shed\n She was looking kind of dumb with her finger and her thumb\n In the shape of an \"L\" on her forehead\n Well the years start coming and they don't stop coming\n Fed to the rules and I hit the ground running\n Didn't make sense not to live for fun\n Your brain gets smart but your head gets dumb\n So much to do, so much to see\n So what's wrong with taking the back streets?\n You'll never know if you don't go\n You'll never shine if you don't glow\n Hey now, you're an all-star, get your game on, go play\n Hey now, you're a rock star, get the show on, get paid\n And all that glitters is gold\n Only shooting stars break the mold\n Only shooting stars break the mold",
        "author": {
          "name": "Smash Mouth"
        }
      },
      {
        "title": "Less-interesting Stuff",
        "body": "Sed ut perspiciatis, unde omnis iste natus error sit voluptatem\n accusantium doloremque laudantium, totam rem aperiam eaque ipsa, quae ab illo inventore\n veritatis et quasi architecto beatae vitae dicta sunt, explicabo. Nemo enim ipsam\n voluptatem, quia voluptas sit, aspernatur aut odit aut fugit, sed quia consequuntur\n magni dolores eos, qui ratione voluptatem sequi nesciunt, neque porro quisquam est, qui\n dolorem ipsum, quia dolor sit amet consectetur adipisci[ng] velit, sed quia non-numquam\n [do] eius modi tempora inci[di]dunt, ut labore et dolore magnam aliquam quaerat\n voluptatem. Ut enim ad minima veniam, quis nostrum[d] exercitationem ullam corporis\n suscipit laboriosam, nisi ut aliquid ex ea commodi consequatur? Quis autem vel eum iure\n reprehenderit, qui in ea voluptate velit esse, quam nihil molestiae consequatur, vel\n illum, qui dolorem eum fugiat, quo voluptas nulla pariatur?",
        "author": {
          "name": "Some Latin-speaking Guy"
        }
      }
    ]
  }
}

That’s great for /first_posts, but we probably don’t need to bloat the response with the entire post body if we’re just using the names for a list, right? Let’s try again, but this time we’ll remove body from the request:

query {
  posts {
    title
    author {
      name
    }
  }
}

Now we just get the title and author, exactly like we want. Notice also that these queries are pulling in data from a related model, the User model, via the author property.

{
  "data": {
    "posts": [
      {
        "title": "All Star",
        "author": {
          "name": "Smash Mouth"
        }
      },
      {
        "title": "Less-interesting Stuff",
        "author": {
          "name": "Some Latin-speaking Guy"
        }
      }
    ]
  }
}

Although you can get similar functionality by using GET parameters or creating extra endpoints, with GraphQL, you can just make that one simple change without needing to write extra code.

Try it!

OK, so there are some of the things that sets GraphQL apart. However, reading about it is only so helpful—now I’ll show how to get a GraphQL backend up and running quickly using Graph.cool, a Node.js backend service that does most of the work of setting up a server, including setting up a database based on your schema. (Graphcool was retired in mid-2020, so you should instead check out their more flexible server option called Prisma.)

First, you need to install graphcool by running:

npm install -g graphcool

You’ll also need to sign up for a (free) account, but you can do that later. Once graphcool is installed, create a directory to use for your code and initialize it.

$ mkdir gql-blog
$ cd gql-blog
$ graphcool init

Open the file types.graphql and uncomment the lines describing a model called Post, as well as the related line in User; aftewards, it should look something like this:

type User @model {
  # id is of type ID, must be not null (!), and must be unique
  id: ID! @isUnique
  name: String
  dateOfBirth: DateTime

  # posts contains an array of Post objects
  posts: [Post!]! @relation(name: "UserPosts")
}


type Post @model {
  id: ID! @isUnique
  title: String!

  # Graphcool relations have to be defined both ways so the service knows
  # whether it's a one-to-many, many-to-many, or many-to-one relation
  author: User! @relation(name: "UserPosts")
}

Note that there are a few extra descriptors here that aren’t part of the core GraphQL schema; @isUnique, @relation, @model, and ID are all part of the Graphcool service’s additions. However, they are allowed within GraphQL’s spec and provide some extra functionality to the API. GraphQL’s website has more information about the type system.

Now that you have a basic schema, it’s time to run your server. In the directory where you ran graphcool init, run graphcool deploy. First, it’ll open a browser window and ask you to log in or create an account. Once you do that, you should see a prompt asking you which server cluster you want to deploy to. Pick any of them (but note that local requires a bit more setup before working). Push enter on the next few prompts to use the default options and you’ll get a big wall of text telling you that your server is up and running! To try it out, find the line near the end that looks like this:

Here are your GraphQL Endpoints:

  Simple API:        https://api.graph.cool/simple/v1/bbd987478jjhhds902k2l

Copy and paste that URL into a browser and you’ll have access to the GraphQL Playground, where you can run queries and mutations (like queries, but for creating/updating data). You need some data to play with, so copy the following into the Playground one at a time and run them to create a User and a Post. Note that you’ll need to look at the ID returned by the first mutation and paste it into the second mutation so that the two are linked.

mutation {
  createUser(
    name: "SmashMouth"
  ) {
    id
    name
  }
}

mutation {
  createPost(
    title: "All Star",
    authorId: "USER_ID_HERE"
  ) {
    id
    title
  }
}

Once those are finished, you can run queries to get the data you want. Here’s an example of a query to return the author you created earlier plus any posts associated with it:

query {
  User(id: "USER_ID_HERE") {
    id
    posts {
      id
    }
  }
}

You’ll notice that this query only returns the ID of the user and their posts. That’s because we only asked for the ID—if you want to get the name of the user and the title of the post, you’ll have to add that to the query:

query {
  User(id: "USER_ID_HERE") {
    id
    name
    posts {
      id
      title
    }
  }
}

Pretty cool, huh? You get exactly what you ask for. Next, try getting your post (you’ll need its ID) and asking for its author along with it..

On the far right of the GraphQL Playground, there’s a tab labeled Schema that shows the entire schema of your app. If you look in there, you’ll see that there are far more types defined than were in the types.graphql file we edited earlier—this is because graphcool automatically generates them for you. That’s the main reason why it’s easier to use than other GraphQL libraries. In particular, you might try getting all posts or all users.

If you want to keep messing around with GraphQL, I’d suggest trying to add your own new Comment type to types.graphql, emulating the way User and Post are related to each other. Keep in mind that you’ll have to run graphcool deploy and reload your browser to see your changes show up in the Playground.

For more information and thorough documentation, see the GraphQL website.

If you want to try GraphQL without installing anything, visit graphql.nodaljs.com which lets you try it on their own server.

How to Migrate from Microsoft SQL Server to PostgreSQL

2019-01-23T00:00:00+00:00

One of our clients had a Java-based application stack on Linux that connected to a pretty old version of SQL Server on Windows. We wanted to migrate the entire system to a more consistent unified stack that developers are efficient with, and that is current so it receives regular updates.

We decided to migrate the database from SQL Server to PostgreSQL on Linux because porting the database, while not entirely quick or simple, was still much simpler than porting the app to .NET/C# would have been. Rewriting the application would have taken far longer, been much riskier to the business, and cost a lot more.

I experimented with a few approaches to the migration and decided to go with the process of schema migration and then the data migration approach which is referred to on the Postgres wiki. Let’s walk through the process of migration step by step.

Schema Migration

A schema of the SQL Server database tables and views needs to be exported to perform schema conversion. The following steps will show you how to export the schema.

Export SQL Server Database Schema

In SQL Management Studio, right click on the database and select Tasks → Generate Scripts.

Choose “Select specific database objects” and check only your application schema Tables (untick dbo schema objects and others if any).

Ensure that “Types of data to script” in advanced options is set to “Schema only”.

Review and save the database tables schema file tables.sql. Use WinSCP and public key auth to transfer tables.sql to the Linux server.

Convert Schema from SQL Server to Postgres

sqlserver2pgsql is a good migration tool written in Perl to convert SQL Server schemas to Postgres schemas. Clone it from GitHub to your database server and execute the following commands to convert the tables schema:

$ git clone https://github.com/dalibo/sqlserver2pgsql.git
$ cd sqlserver2pgsql
$ perl sqlserver2pgsql.pl -f tables.sql -b tables-before.sql -a tables-after.sql -u tables-unsure.sql

The converted schema will be available in tables-before.sql and statements to create constraints will be in tables-after.sql to execute after data migration. Just review tables-unsure.sql and do what’s needed if there are any tables not converted by the tool. If you want to change any schema names in Postgres, you can rename them now in the SQL file, e.g.:

$ sed -i ‘s/sql_server_schema/public/g’ *.sql

Setup Postgres Database:

Hopefully, you have your Postgres database ready in your server. If not, install the latest version of Postgres on your server. Then create a Postgres user and database with permissions granted to the user.

CREATE USER <user_name> WITH
  LOGIN
  NOSUPERUSER
  INHERIT
  NOCREATEDB
  NOCREATEROLE
  NOREPLICATION;

CREATE DATABASE <database_name>
    WITH
    OWNER = <user_name>
    ENCODING = 'UTF8'
    LC_COLLATE = 'en_US.UTF-8'
    LC_CTYPE = 'en_US.UTF-8'
    TABLESPACE = pg_default
    CONNECTION LIMIT = -1;

Once the database and user account are ready, load the converted tables-before.sql script into your database to create tables. Then we can move forward with data migration.

psql -U user_name -p 5432 -h localhost -d database_name -f tables-before.sql
#	or
database_name=# \i /path/to/tables-before.sql

Data Migration

Data migration through data dump and restore makes the process cumbersome, with a huge manual process, data type mismatches, date formats, etc. It is wise to use existing stable systems instead of investing more time into it.

Pentaho Data Integration

Pentaho offers various stable data-centric products. Pentaho Data Integration (PDI) is an ETL tool which provides great support for migrating data between different databases without manual intervention. The community edition of PDI is good enough to perform our task here. It needs to establish a connection to both the source and destination databases. Then it will do the rest of work on migrating data from SQL server to Postgres database by executing a PDI job.

Download Pentaho Data Integration Community Edition and extract the tarball in your local environment. See:

Run spoon.sh to open the GUI application in your local environment.

$ cd pentaho/data-integration
$ ./spoon.sh

Create connections to both SQL Server and Postgres databases in PDI.

1. Create a New Job.

File → New → Job

2. Create Source Database Connection.

Click View in left sidebar → Right Click ‘Database Connections’ → Choose New → Provide SQL Server connection details

3. Create Destination Database Connection.

Click View in left sidebar → Right Click ‘Database Connections’ → Choose New → Provide Postgres connection details

4. From Wizard menu, choose Copy Tables Wizard.

Tools → Wizard → Copy Tables

5. Choose Source and Destination databases.

6. Select the list of tables to migrate.

7. Move forward to choose the path for job and transformation files.

8. The transformations were created to copy data from source to destination database.

9. Copy the Pentaho Data Integration tarball and job with transformations to the Postgres server to avoid network latency on data migration.

$ tar -cvzf pdi-migration-job.tar.gz pdi-migration-job/
$ rsync -azP data-integration.zip pdi-migration-job.tar.gz user@server:

10. Check Postgres server to SQL Server database access (based on architecture design at your end).

Establish SSH tunneling to connect to SQL Server through the application server.

$ cat config
host tunnel-server
    Hostname <application_server>
    Port 22
    User root
    LocalForward 1433 <ipaddress>:1433

$ ssh tunnel-server
$ telnet localhost 1433

PDI Job Execution

Execute PDI job using Pentaho kitchen utility on the database server. Add a kettle configuration to avoid Pentaho considering empty values as NULL values which affects NOT NULL constraints.

$ cat .kettle/kettle.properties
KETTLE_EMPTY_STRING_DIFFERS_FROM_NULL=Y

Confirm IP address for database servers’ connections and execute the job using kitchen.sh, a command-line utility to execute PDI jobs.

$ ./kitchen.sh -file="/path/to/pdi-migration-job.kjb" -level=Basic | tee pdi-migration-job.log

The data migration process took around 30 minutes to copy 10GB of data from SQL Server to Postgres database over network.

Output:

2018/08/23 11:16:52 - SQLServerToPostgres20180823 - Job execution finished
2018/08/23 11:16:52 - Kitchen - Finished!
2018/08/23 11:16:52 - Kitchen - Start=2018/08/23 10:51:00.701, Stop=2018/08/23 11:16:52.951
2018/08/23 11:16:52 - Kitchen - Processing ended after 25 minutes and 52 seconds (1552 seconds total).

We ran into a space problem while copying a large table and handled it by reducing the number of rows size in the corresponding table transformation file. Here’s a discussion of the problem.

Fix:

vi **table_name**.ktr
    <size_rowset>100</size_rowset>

sed 's/<size_rowset>10000<\/size_rowset>/<size_rowset>100<\/size_rowset>/' *table_name*

Once the data migration is completed, execute the tables-after.sql schema-migration script to apply constraints.

Migrate Views

Follow the same steps to convert the views.sql schema. Right click on the database and click on Tasks → Generate Script → Export only views.

$ perl sqlserver2pgsql.pl -f views.sql -b views-before.sql -a views-after.sql -u views-unsure.sql

Migrate Functions

The functions migration requires skill and syntax awareness of both SQL Server and Postgres. The functions need to validated and tested properly after rewriting for Postgres. Place all rewritten functions into postgres-function.sql and load into the database as part of the migration process.

Most Common Errors

ERROR: syntax error at or near "["
Remove SQL Server square brackets quoting around column and table names.
SELECT [user_id] → SELECT user_id
ERROR: operator does not exist: character varying = integer
Type casting on comparison
AND column_name in (1, -1) → AND column_name in (‘1’, ‘-1’)
ERROR: syntax error at or near ","
Datatype conversion
convert(varchar(10),column_name) → column_name::text
ERROR: column "user_id" of relation "user" does not exist
No double quotes on mixed-case column names (unless used everywhere)
INSERT INTO user (“user_id”) VALUES (1) → INSERT INTO user (user_id) VALUES (1)
ERROR: column "mm" does not exist
DATEADD → INTERVAL
DATEADD(mm,-6,GETDATE()) → CURRENT_DATE - INTERVAL ‘6 months’
ERROR: column "varchar" does not exist
Convert date to USA format
convert(varchar, tc_agreed_dt, 101) → to_char(CURRENT_TIMESTAMP, ‘MM/DD/YYYY’)

Result

After migration, we performed extensive testing on the application and optimized the queries in various modules. We also performed a benchmark test on speed and performance between the new and old systems. The migration paid off with a promising result of 40% average gain on overall performance of the application. That reflects the newer server hardware as well as the query optimization work, not solely the move to Postgres.

The entire migration was automated with a script to test multiple times and reduce manual intervention to perform error-free process. The automated script is available to customize based on your scenario: sqlserver-postgres-migration-automation.sh

Switching PostgreSQL WAL-based Backup Options

2019-01-03T00:00:00+00:00

Photo by Paul Hudson · CC BY 2.0, modified

I was woken up this morning. It happens every morning, true, but not usually by a phone call requesting for help with a PostgreSQL database server that was running out of disk space.

It turns out that one of the scripts we’re in the process of retiring, but still had in place, got stuck in a loop and filled most of the available space with partial, incomplete base backups. So, since I’m awake, I’d might as well talk about Postgres backup options. I don’t mean for it to be a gripe session, but I’m tired and it kind of is.

For this particular app, since it resides partially on AWS we looked specifically at options that are able to work natively with S3. We’ve currently settled on pgBackRest. There’s a bunch of options out there, which doesn’t make the choice easy. But I suppose that’s the nature of things these days.

At first we’d tried out pghoard. It looks pretty good on the tin, especially with its ability to connect to multiple cloud storage services beyond S3: Azure, Google, Swift, etc. Having options is always nice. And for the most part it works well, apart from a couple idiosyncrasies.

We had the most trouble with the encryption feature. It didn’t have any problem on the encryption side. But for some reason on the restore the process would hang and eventually fail out without unpacking any data. Having a backup solution is a pretty important thing, but it doesn’t mean anything unless we can get the data back from it. So this was a bit of a sticking point. We probably could have figured out how to get it functioning, and at least been a good citizen and reported it upstream to get it resolved in the source. But we kind of just needed it working, and giving something else a shot is a quicker path to that goal. Sorry, pghoard devs.

The other idiosyncratic behaviors that are probably worth mentioning are that it does its own scheduling. The base backups, for instance, happen at a fixed hour interval in the configuration file, starting from when the service is first spun up. So if I set it to 24 hours, and then service pghoard start at 1 PM, well, that’s the schedule now. So much for having it run overnight. Nor can tell it “create a base backup right now” at other times. And, so far as I can find, there’s no way to override that.

Also rather than using an archive_command, it connects as a streaming replica and pulls WAL files that way. Which is great from a perspective of it not being invasive. But during a load test we generated activity faster than it could write to S3, and hit the classic streaming replica problem of it needing WAL files that had already been removed from the master. It doesn’t handle this condition, though, and just retried and retries after the error, even after another base backup happened. I don’t know that I’d let it run long enough to see if it would eventually give up when the base backup that needed those WAL files had been pushed out by the retention policies. But I eventually killed it and had it make a fresh start. There’s some chatter out there about having it use replication slots, which would be the natural solution for this case, so maybe that’s fixed by now.

So, we switched to something else, and elected pgBackRest next. It’d since been used in another project or two, so it had some bonus points for recently acquired institutional knowledge. It supports storage in S3 (and only S3, but whatevs). And this time we didn’t have to build and distribute deb’s; it’s already available in the Postgres apt repo.

I like things neat and tidy. A base backup I’ve always seen as essentially one unit; it’s the database cluster as a whole (even though I know it’s in a potentially inconsistent state by being spread over an interval of time.) Stored and delivered as a compressed tar file or some such. pgBackRest takes a different view, and instead stores its base backup as separate relation files, essentially as they are on disk, and individually compressed. Which, given it’s something I don’t have to touch and move around manually, is a format that’s growing on me.

Because it does this, pgBackRest brings back a concept I haven’t thought about in a long time: “incremental” or “differential” backups. If a relation (or part of a really big relation, since it’s chopped up into 1 GB sections) doesn’t change between base backups, pgBackRest can skip those and point back to an earlier copy of the file. In most databases (at least I’d argue) this won’t matter so much, as between user activity, background tasks like vacuum, and such, things are frequently changing in those relation files. But in this case, the client has a large table of file blobs that mostly gets appended to. So the data has been loaded, we ran a VACUUM FREEZE on it, and now the daily differential backups avoid a little bit of S3 cost.

Between the entries we have to add to cron and the edit of the archive_command parameter to save WAL pgBackRest takes a couple more steps to get fully in place. But it’s all in configuration management now, so, meh. Besides, the fixed schedule does make it possible to run at a specific time. And the WAL archival is guaranteed not to fall behind and get stuck.

Like pghoard, pgBackRest does encryption. pghoard’s is based on public/private keys, which I might have preferred because the base backups are used to seed development systems, and only distributing the public key to those would be neat. pgBackRest’s is symmetric with a passphrase. Both also do automatic retention maintenance; we have it maintain two weekly full base backups and daily differentials in between. WAL retention is similarly automatic.

The cool thing was we could run both side by side for a while, and confirm that pgBackRest would shake out before disabling pghoard. At least I thought that was cool, until that phone call came through. I kind of wish I knew what got it there in the first place; we found pghoard in a base backup attempt loop, where it’d ended up mostly filling the disk, and just ended up killing it and manually cleaned up its storage to get monitoring back to green.

So I suppose the morals of the story are:

Keep your options open. There are lots of projects out there that do similar things a little differently. Try them out, and don’t be afraid to pivot if something is a little more to your liking.
The things you do try, test out fully. Restore the backups. Let them run for a while to see how they fare long-term and make sure all the assumptions hold up.
I need a second cup of coffee. BRB.

Building Rasters in PostGIS

2018-09-12T00:00:00+00:00

In a past blog post I described a method I’d used to digest raw statistics from the Mexican government statistics body, INEGI, quantifying the relative educational level of residents in Mexico City. In the post, I divided these data into squares geographically, and created a KML visualization consisting of polygons, where each polygon’s color and height reflected the educational level of residents in the corresponding area. Where the original data proved slow to render and difficult to interpret, the result after processing was visually appealing, intuitively meaningful, and considerably more performant. Plus I got to revisit a favorite TV show, examine SQL’s Common Table Expressions, and demonstrate their use in building complex queries.

But the post left a few loose ends untied. For instance, the blog post built the visualization using just one large query. Although its CTE-based design rendered it fairly readable, the query remained far too complex, and far too slow, at least for general use. Doing everything in one query makes for a sometimes enjoyable mental exercise, but it also means the query has to start from zero, every time it runs, so iterative development bogs down quickly as the query must recalculate at every new iteration all the expensive stuff it already got worked out in previous runs. Further, the query in that post calculated its grid coordinates all on its own; there’s an easier method I promised to introduce, which I’d like to demonstrate here.

Rasters

GIS databases contain information of (at least) two different types: vector data, and raster data. Vector data describes points, connected with lines and curves, and various objects derived from them, such as polygons and linestrings. Rasters, on the other hand, represent a two dimensional array of pixels, where each pixel corresponds to a certain geographic area. Each pixel in the raster will have two sets of associated coordinates: first, latitude and longitude values describing the geographic square this pixel represents, and second, the pixel coordinates within the raster, which in PostGIS begin with (1,1) at the origin. Rasters organize their data in “bands”, collections of boolean or numeric values, with at most one value per pixel. One band might represent the elevation above sea level for each pixel, or the surface temperature. Aerial imagery is often delivered as rasters: a full-color image might have one band for the red channel, another for green, and a third for blue. Bands may also contain no value for certain pixels, and we’ll make use of this ability later to make some parts of our final image transparent.

The Data

The INEGI data I introduced in my previous blog post contains points representing every school in Mexico, and continuing on the theme of education, I thought it would be an interesting exercise to make an image of the country, where the color varies depending on the concentration of schools in the area. This resembles my previous work in many ways: I’m dividing the area under consideration into small parts, generating a value for each of those parts, and representing the result visually. Here, though, rather than drawing and extruding polygons, I’ll simply color a raster pixel. This simpler rendering, where the display system must simply project a static image, rather than an array of polygons, means I can support a much higher resolution grid than in my previous blog post, without overtaxing display hardware.

Creating the Image

Having learned my lesson from the work with Mexico City data, I elected to create this image with multiple queries. In the end, I wrapped these queries in a Perl script, building sufficient intelligence therein to recreate any bits of the database I might have simply dropped, as I developed the script, and to skip reprocessing any steps for which it finds current results lying around.

PostGIS supports a data type called raster, and anticipating that I might want multiple rasters in my database each using a variety of settings and script modifications, I created a table to store these rasters and some extra data about them.

CREATE TABLE rasters (id SERIAL PRIMARY KEY, pixels INTEGER, r RASTER, comment TEXT);

The pixels field holds the number of pixels along one edge of the raster, which the script presumes is a square. This is redundant, as I can extract that number from the raster itself, but having it stored here made things easier for my purposes.

To calculate a raster, I need to find a value for each of its pixels, or to confirm that the pixel should have no value, and for that I wanted another table. This table contains both geographic and Euclidean coordinates for each pixel in each raster, the id of the associated entry in the rasters table, a field for the number of schools I’ve counted in this square, fields for the RGB color assigned to the pixel, and a boolean flag telling me whether or not I’ve finished calculating everything for this pixel or not.

CREATE TABLE polys (
    x INTEGER,
    y INTEGER,
    geom GEOMETRY,
    count INTEGER,
    r_val INTEGER,
    b_val INTEGER,
    g_val INTEGER,
    processed BOOLEAN DEFAULT 'f',
    raster_id INTEGER NOT NULL
);

The script’s first step is to create an empty raster, which it does with PostGIS’s ST_MakeEmptyRaster() function, and to add four bands to it, one each for the pixels’ red, green, blue, and alpha values. The data type for each band is given as 8BUI, PostGIS parlance for an 8-bit unsigned integer. The various color bands default to NULL values, and the alpha band to zeroes. The code looks something like this:

INSERT INTO rasters (pixels, r, comment)
SELECT
    ST_AddBand(
    ST_AddBand(
    ST_AddBand(
    ST_AddBand(
        ST_MakeEmptyRaster(
            numpixels,  -- The number of pixels along one edge of the raster
            numpixels,
            st_xmin,    -- latitude and longitude coordinates of the corner of the raster
            st_ymax,
            (st_xmax - st_xmin)/numpixels::float,   -- width and height of the raster
            -(st_ymax - st_ymin)/numpixels::float,
            0,
            0,
            4326        -- We'll use the WGS 84 coordinate system
        ),
        1, '8BUI'::TEXT, NULL::DOUBLE PRECISION, NULL::DOUBLE PRECISION),
        2, '8BUI'::TEXT, NULL::DOUBLE PRECISION, NULL::DOUBLE PRECISION),
        3, '8BUI'::TEXT, NULL::DOUBLE PRECISION, NULL::DOUBLE PRECISION),
        4, '8BUI'::TEXT, 0, 0
    ) AS r
FROM limits, params;

The query refers to the limits and params tables (actually, they’re table expressions from elsewhere in the query from which this snippet is extracted). These tables provide values for numpixels, the number of pixels along one edge of the raster, provided by the user on the script’s command line, and for st_xmin, st_xmax, st_ymin, and st_ymax, which are latitude and longitude values describing a bounding box around the Mexican territory.

Before we can do too much experimentation and query development, we need a way to inspect our results, by rendering the raster from the database to an actual image file. This is one place where having a Perl wrapper script comes in handy, because doing this in pure SQL is actually something of a pain..

print "Creating image\n";
my $res = $dbh->selectall_arrayref(qq{
    SELECT ST_AsPng(r, ARRAY[1,2,3,4]::INT[])
    FROM rasters WHERE id = $id
});

my $png = $res->[0][0];

open (my $fh, '>', $filename) || die "Couldn't open handle to $filename: $!";
binmode $fh;

print $fh $png;

close $fh;

This uses ST_AsPng to create an image file under whatever path is stored in $filename, and it’s helpful for reviewing incremental results as the script develops, and of course for rendering the final product. Here we’ve told it to use band 1 for the red values, band 2 for green, 3 for blue, and 4 for alpha. The alpha channel is important: eventually we’ll want to feed the result to Google Earth, and the transparency will mean we can see the ocean surrounding Mexico, without any unimportant raster data getting int he way.

Without any values in the bands, there’s nothing to look at yet—so let’s make some data.

Finding Polygons

We need to find the polygon corresponding to each pixel in the raster, and PostGIS gives us two ways to do it. The first is ST_PixelAsPolygons(), which returns a set of values, including the geographic rectangle the pixel represents, called geom, and its x and y coordinates. The other function, ST_PixelAsCentroids(), also accepts a single raster as its sole argument and returns a set of similar results, except that instead of a geographic rectangle, it returns only the point at the geographic center of the pixel. I used both as I experimented to make this visualization. The simplicity of these functions makes them far superior to the manual calculation I did in the previous blog post I’ve referred to.

This query fills the polys table with polygons for a particular raster:

INSERT INTO polys (x, y, geom, raster_id)
SELECT (ST_PixelAsPolygons(r)).*, $id FROM rasters
-- Having code for both functions here made it easy to switch back and forth
-- SELECT (ST_PixelAsCentroids(r)).*, $id FROM rasters
WHERE id = $id

One step that proved critical for decent performance was to filter the set of polygons so that I calculate values only for those pixels that actually intersect some part of Mexico. PostGIS rasters are regular rectangles, so a substantial portion of this raster lies over open ocean. Since we’re counting schools of people, and not of fish, we can skip all those ocean pixels, and though filtering them out takes considerable processing, we earn back that cost and more in savings later when calculating schools within each pixel. This modified version of the query above does that filtering.

WITH polygons AS (
    SELECT (ST_PixelAsPolygons(r)).* FROM rasters
    -- Having code for both functions here made it easy to switch back and forth
    -- SELECT (ST_PixelAsCentroids(r)).* FROM rasters
    WHERE id = $id
),
filtered_polygons AS (
    SELECT x, y, p.geom
    FROM polygons p
        JOIN mexico m
            ON ST_Intersects(m.geom, p.geom)
    GROUP BY 1, 2, 3
)
INSERT INTO polys (x, y, geom, raster_id)
SELECT x, y, geom, $id FROM filtered_polygons

This query refers to a table called mexico, which contains geographic representations of each of Mexico’s 31 states and the region of Mexico City, which I didn’t previously know wasn’t a state. It’s this table that gave us st_xmin, st_xmax, etc., when we made the original raster, and now we use it to filter out all pixels from our raster that don’t intersect some portion of Mexico.

At this point it would be nice to see a visual representation of our raster, to make sure the filtering worked the way we intended. We still haven’t filled the raster with any data, but we could fill it with dummy values and render the image, just to get a look at our progress. This raster modification we achieve with the ST_SetValues() function, which accepts a raster as input and returns the raster, with updated values in one of its bands. The function comes in several forms; the one we’ll use accepts an array of geomval objects, each of which contains a PostGIS geometry, and a single double precision value. Internally, the function finds the pixel or pixels corresponding to the geometry objects provided, and sets those pixels’ values in the given band accordingly. We make sure to provide POINT values here— the center point for each pixel’s corresponding polygon—gc as ST_SetValues is considerably faster that way than when given POLYGON values themselves.

-- Update the original raster
UPDATE rasters
    SET r = ST_SetValues(ST_SetValues(ST_SetValues(ST_SetValues(r, 1, r_geomvalset), 2, g_geomvalset), 3, b_geomvalset), 4, a_geomvalset)
FROM (
    SELECT
        ARRAY_AGG((geom, 255)::GEOMVAL) AS r_geomvalset,
        ARRAY_AGG((geom, 255)::GEOMVAL) AS g_geomvalset,
        ARRAY_AGG((geom, 255)::GEOMVAL) AS b_geomvalset,
        ARRAY_AGG((ST_Centroid(geom), 255)::GEOMVAL) AS a_geomvalset
    FROM polys
    WHERE raster_id = $id
) foo
WHERE id = $id

For a raster with many pixels, this takes a bit of time to accomplish. For my initial testing, I used rasters with only 50 pixels per side, which my wimpy laptop can churn out fairly rapidly. Here’s the first visual confirmation that we’re headed down the right path. Because of the low resolution, the image is small, but it’s obvious that our result follows the basic shape of Mexico, which means our polygon generation and filtering has worked.

I mentioned transparency above. Note that in the default format of this blog, it would be impossible to tell that the transparency in this image was correct, so I’ve post-processed all the rendered raster images in this article to replace the transparency with a gray checkerboard pattern.

Counting schools

Now all that remains is to count the schools in each pixel, and assign corresponding color values.

WITH polygons AS (
    SELECT x, y, geom FROM polys
    WHERE
        processed = 'f'
        AND raster_id = $id
    LIMIT 100
),
schoolcounts AS (
    SELECT x AS sx, y AS sy, COUNT(*) AS schoolcount
    FROM polygons p
        JOIN sip ON (ST_Within(sip.geom, p.geom) AND sip.geografico = 'Escuela')
    GROUP BY x, y, p.geom
)
UPDATE polys SET count = COALESCE(s.schoolcount, 0), processed = 't'
FROM schoolcounts s
    RIGHT JOIN polygons p
        ON (sx = p.x AND sy = p.y)
WHERE
    polys.x = p.x AND polys.y = p.y AND raster_id = $id

This query finally refers to the actual INEGI data set I started with, a table called sip, in which each record includes a POINT object and a label. We’re interested in labels marked “Escuela”, Spanish for “School”. I tried a few different techniques here, as shown by the two different JOIN variants in the schoolcounts portion of the query. The version shown here simply counts schools within each square pixel. Later I’ll try counting schools within a certain radius of the center of the pixel. The two give different results, of course; which one is more useful depends on the end goal for these data.

Note the use of the processed field, and the 100 item limit in the polygons section of the query. Together with some logic in the surrounding Perl, this allows me to process pixels in small batches, interrupt processing when necessary, and later, resume more or less where I left off.

Color assignment

After counting schools, I need to assign colors to the pixels. Initially I imagined this would be fairly easy, but in fact it took some detective work to get it right. My plan was to get a gradient of colors, divide the school counts into histogram-like buckets according to school density, and assign a color to each bucket. Building your own decent color gradient is surprisingly difficult, and I got mine from an online service. As with the last blog post, I figured a relatively small number of colors would be easiest. These I encoded as a PostgreSQL array, and I used PostgreSQL’s ntile window function to return the right bucket for each polygon. The results surprised me, shown here in a raster with 500 pixels on each side:

It’s easy to see in that image that some pixels are colored to represent schools, but where did all the vertical bands come from? Much of Mexico is very sparsely populated, so vast swaths of our pixels have no schools at all. This means that the pixels with a school count of zero fill most of the available buckets. Pixels with schools in them are so much in the minority, for any raster of sufficient resolution, as to place all non-zero pixels in a single bucket. So I modified my approach slightly: I made my histogram only from those pixels with non-zero school counts. That query is below.

WITH colors AS (
    SELECT ARRAY[
        ARRAY[x'd4'::INT, x'ea'::INT, x'd7'::INT],
        ARRAY[x'a9'::INT, x'd6'::INT, x'af'::INT],
        ARRAY[x'7e'::INT, x'c1'::INT, x'88'::INT],
        ARRAY[x'53'::INT, x'ad'::INT, x'60'::INT],
        ARRAY[x'29'::INT, x'99'::INT, x'39'::INT]] AS colors
),
p AS (
    SELECT x, y, geom, raster_id, count
    FROM polys WHERE raster_id = $id AND count IS NOT NULL AND count != 0
),
colorval AS (
    SELECT
        x, y, geom, raster_id,
        NTILE(ARRAY_LENGTH(colors, 1)) OVER (ORDER BY count ASC) AS ntile,
        colors[NTILE(ARRAY_LENGTH(colors, 1)) OVER (ORDER BY count ASC)][1] AS r,
        colors[NTILE(ARRAY_LENGTH(colors, 1)) OVER (ORDER BY count ASC)][2] AS g,
        colors[NTILE(ARRAY_LENGTH(colors, 1)) OVER (ORDER BY count ASC)][3] AS b,
        255 AS a
    FROM colors, p
    WHERE raster_id = $id
        AND count != 0
        AND count IS NOT NULL
)
UPDATE polys
    SET r_val = r, g_val = g, b_val = b
FROM colorval c
WHERE c.x = polys.x AND c.y = polys.y AND c.raster_id = polys.raster_id

A second, very simple query not included here found any pixels with zero schools and colored them white.

Packaging the result

The final bit of required scripting to generate a usable image is to use ST_SetValues again, in a somewhat more complex form than last time, creating an array of geomval objects and using them to update the original raster.

UPDATE rasters
    SET r = ST_SetValues(ST_SetValues(ST_SetValues(ST_SetValues(r, 1, r_geomvalset), 2, g_geomvalset), 3, b_geomvalset), 4, a_geomvalset)
FROM (
    SELECT
        ARRAY_AGG((ST_Centroid(geom), r_val)::GEOMVAL) FILTER (WHERE r_val IS NOT NULL) AS r_geomvalset,
        ARRAY_AGG((ST_Centroid(geom), g_val)::GEOMVAL) FILTER (WHERE g_val IS NOT NULL) AS g_geomvalset,
        ARRAY_AGG((ST_Centroid(geom), b_val)::GEOMVAL) FILTER (WHERE b_val IS NOT NULL) AS b_geomvalset,
        ARRAY_AGG((ST_Centroid(geom), 255)::GEOMVAL) AS a_geomvalset
    FROM polys
    WHERE raster_id = $id
) foo
WHERE id = $id

This gives me something like the image below, with 500 pixels on each edge of the raster:

This image does the job, but contains an awful lot of empty space. I mentioned above that instead of calculating the exact number of schools within a pixel’s polygon, I thought I might try calculating the number of schools within a specific distance of the pixel’s center; perhaps that technique would lead to a more visually appealing result. This is simple enough, using the ST_PixelAsCentroids() function to get the centroids of the pixels’ polygons, and then modifying the schoolcounts CTE as shown here. The key modification—in fact, the only modification from the earlier version shown above—is in the JOIN clause, which uses ST_Buffer to create a circle surrounding a point, in this case, with an arbitrarily chosen radius, and counts the number of schools found within that circle.

schoolcounts AS (
    SELECT x AS sx, y AS sy, COUNT(*) AS schoolcount
    FROM polygons p
        JOIN sip ON (ST_Within(sip.geom, ST_Buffer(p.geom, 0.4)) AND sip.geografico = 'Escuela')
    GROUP BY x, y, p.geom
)

The resulting image, again with 500x500 pixel resolution, shows evidence of its circle-based counting heritage.

Obviously at this point, a person could mess with any number of variables to get all sorts of results. But for me it was enough to embed this as a GroundOverlay object within a simple KML file, to see the result in Google Earth:

Systematic Query Building with Common Table Expressions

2018-06-12T00:00:00+00:00

The first time I got paid for doing PostgreSQL work on the side, I spent most of the proceeds on the mortgage (boring, I know), but I did get myself one little treat: a boxed set of DVDs from a favorite old television show. They became part of my evening ritual, watching an episode while cleaning the kitchen before bed. The show features three military draftees, one of whom, Frank, is universally disliked. In one episode, we learn that Frank has been unexpectedly transferred away, leaving his two roommates the unenviable responsibility of collecting Frank’s belongings and sending them to his new assignment. After some grumbling, they settle into the job, and one of them picks a pair of shorts off the clothesline, saying, “One pair of shorts, perfect condition: mine,” and he throws the shorts onto his own bed. Picking up another pair, he says, “One pair of shorts. Holes, buttons missing: Frank’s.”

The other starts on the socks: “One pair of socks, perfect condition: mine. One pair socks, holes: Frank’s. You know, this is going to be a lot easier than I thought.”

“A matter of having a system,” responds the first.

I find most things go better when I have a system, as a recent query writing task made clear. It involved data from the Instituto Nacional de Estadística y Geografía, or INEGI, an organization of the Mexican government tasked with collecting and managing country-wide statistics and geographical information. The data set contained the geographic outline of each city block in Mexico City, along with demographic and statistical data for each block: total population, a numeric score representing average educational level, how much of the block had sidewalks and landscaping, whether the homes had access to the municipal sewer and water systems, etc. We wanted to display the data on a Liquid Galaxy in some meaningful way, so I loaded it all in a PostGIS database and built a simple visualization showing each city block as a polygon extruded from the earth, with the height and color of the polygon proportional to the educational score for that block, compared to the city-wide average.

It wasn’t entirely a surprise that with so many polygons, rendering performance suffered a bit, but most of all the display was just plain confusing. This image is just one of Mexico City’s 16 boroughs.

With so much going on in the image, it’s difficult for the user to extract any meaningful information. So I turned to a technique we’d used in the past: reprocess the geographical area into grid squares, extrapolate the statistic of interest over the area of the square, and plot it again as a set of squares. The result is essentially a three dimensional heat map, much easier to comprehend, and, incidentally, to render.

As with most programming tasks, it’s helpful to have a system, so I started by sketching out how exactly to produce the desired result. I planned to overlay the features in the data set with a grid, and then for each square in the grid, find all intersecting city blocks, a number representing the educational level of residents of that block, and what percentage of the block’s total area intersects each grid square. From that information I can extrapolate that block’s contribution to the grid square’s total educational score. The precise derivation of the score isn’t important for our purposes here; suffice it to say it’s a numeric value with no particular associated unit, whose value lies in its relation to the scores of other blocks. Residents of a block with a high score are, on average, probably more educated than residents of a lower-scoring block. For this query, a block with an average educational level of, say, 100 “points”, would contribute all 100 points to a grid square if the entire block lay within that square, 60 points if only 60% of it was within the square, and so on. In the end, I should be able to add up all the scores for each grid square, rank them against all other grid squares, and produce a visualization.

I suffer from the decidedly masochistic habit of doing whatever I can in a single query, while maintaining a desire for readable and maintainable code. Cramming everything into one query isn’t always a good technique, as I hope to illustrate in a future blog post, but it worked well enough in this instance, and provides a good example I wanted to share, of one way to use Common Table Expressions. They do for SQL what subroutines do for other languages, separating tasks into distinct units. A common table expression looks like this:

WITH alias AS (
    SELECT something FROM wherever
),
another_alias AS (
    SELECT another_thing FROM alias LEFT JOIN something_else
)
SELECT an, assortment, of, fields
FROM another_alias;

As shown by this example query, the WITH keyword precedes a list of named, parenthesized queries, each of which functions throughout the life of the query as though it were a full-fledged table. These pseudo-tables are called Common Table Expressions, and they allow me to make one table for each distinct function in what will prove to be a fairly complicated query.

Let’s go through the elements of our new query piece by piece. First, I want to store the results of the query, so I can create different visualizations without recalculating everything. So I’ll make this query create a new table filled with its results. In this case, I called the table grid_mza_vals.

Now, for my first CTE. This query involves a few user-selected parameters, and I want an easy way to adjust these settings as I experiment to get the best results. I’ll want to fiddle with the number of grid squares in the overall result, as well as the coefficients used later on to calculate the height of each polygon. So my first CTE is called simply params, and returns a single row, composed of these parameters.

CREATE TABLE grid_mza_vals AS
WITH params AS (
    SELECT
        25 AS numsq,
        3800 AS alt_bias,
        500 AS alt_percfactor
),

The numsq value represents the number of grid squares along one edge of my overall grid; we’ll discuss the other values later. I’ve chosen a relatively small number of total grid squares for faster processing while building the rest of the query. I can make it more detailed, if I want, after everything else works.

The next thing I want is a sequence of numbers from 1 to numsq:

range AS (
    SELECT GENERATE_SERIES(0, numsq - 1) AS rng
    FROM params
),

Now I can join the range CTE with itself, to get the coordinates for each grid square:

gridix AS (
    SELECT
        x.rng AS x_ix,
        y.rng AS y_ix
    FROM range x, range y
),

Occasionally I like to check my progress, running whatever bits of the query I’ve already written to review its behavior. CTEs make this convenient, because I can adjust the final clause of the query to select data from whichever of the CTEs I’m currently interested in. Here’s the query thus far:

inegi=# WITH params AS (
    SELECT
        25 AS numsq,
        3800 AS alt_bias,
        500 AS alt_percfactor
),
range AS (
    SELECT GENERATE_SERIES(0, numsq - 1) AS rng
    FROM params
),
gridix AS (
    SELECT
        x.rng AS x_ix,
        y.rng AS y_ix
    FROM range x, range y
)
SELECT * FROM gridix;

 x_ix | y_ix
------+------
    0 |    0
    0 |    1
    0 |    2
    0 |    3
    0 |    4
    0 |    5
    0 |    6
    0 |    7
    0 |    8
    0 |    9
    0 |   10
    0 |   11
    0 |   12
    0 |   13
...

So far, so good. The gridix CTE returns coordinates for each cell in the grid, from zero to the numsq value from my params CTE. From those coordinates, if I know the geographic boundaries of the data set and the number of squares in each edge, I can calculate the latitude and longitude of the four corners of each grid square. First, I need to find the geographic boundaries of the data. My dataset lives in a table called manzanas, Spanish for “city block” (and also “apple”). Each row contains one geographic attribute containing a polygon defining the boundaries of the block, and several other attributes such as the education score I mentioned. In PostGIS there are several different ways to find the bounding box I want; here’s the one I used.

limits AS (
    SELECT
        MIN(ST_XMin(geom)) AS xmin,
        MAX(ST_XMax(geom)) AS xmax,
        MIN(ST_YMin(geom)) AS ymin,
        MAX(ST_YMax(geom)) AS ymax
    FROM
        manzanas
),

And, just to check my results:

inegi=#     SELECT
        MIN(ST_XMin(geom)) AS xmin,
        MAX(ST_XMax(geom)) AS xmax,
        MIN(ST_YMin(geom)) AS ymin,
        MAX(ST_YMax(geom)) AS ymax
    FROM
        manzanas;
     xmin      |     xmax     |       ymin       |       ymax
---------------+--------------+------------------+------------------
 -99.349658451 | -98.94668802 | 19.1241898199991 | 19.5863775499992
(1 row)

So the data in question extend from about 99.35 to 98.95 west longitude, and 19.12 to 19.59 north latitude. Now I’ll calculate the boundaries of each grid square, compose a text representation for the square in Well-Known Text format, and convert the text to a PostGIS geometry object. There’s another way I could do this, much simpler and probably much faster, which I hope to detail in a future blog post, but this will do for now.

gridcoords AS (
    SELECT
        (xmax - xmin) / numsq * x_ix + xmin AS x0,
        (xmax - xmin) / numsq * (x_ix + 1) + xmin AS x1,
        (ymax - ymin) / numsq * y_ix + ymin AS y0,
        (ymax - ymin) / numsq * (y_ix + 1) + ymin AS y1,
        x_ix || ',' || y_ix AS grid_ix
    FROM limits, params, gridix
),
gridewkt AS (
    SELECT
        grid_ix,
        'POLYGON((' ||
        x0 || ' ' || y0 || ',' ||
        x1 || ' ' || y0 || ',' ||
        x1 || ' ' || y1 || ',' ||
        x0 || ' ' || y1 || ',' ||
        x0 || ' ' || y0 || '))' AS ewkt
    FROM gridcoords
),
gridgeom AS (
    SELECT
        grid_ix, ewkt,
        ST_GeomFromText(ewkt, 4326) AS geom
    FROM gridewkt
),

And again, I’ll check the result by SELECTing grid_ix, ewkt, and geom, from the first row of the gridgeom CTE.

-[ RECORD 1 ]-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
grid_ix | 0,0
ewkt    | POLYGON((-99.349658451 19.1241898199991,-99.34562874669 19.1241898199991,-99.34562874669 19.1288116972991,-99.349658451 19.1288116972991,-99.349658451 19.1241898199991))
geom    | 0103000020E6100000010000000500000029F4D6CD60D658C06B646FE7CA1F3340D3E508C81ED658C06B646FE7CA1F3340D3E508C81ED658C0EC3DABCDF920334029F4D6CD60D658C0EC3DABCDF920334029F4D6CD60D658C06B646FE7CA1F3340

I won’t claim an ability to translate the geometry object as represented above, but the ewkt value looks correct, so let’s keep going. Next I need to cut the blocks into pieces, corresponding to the parts of each block that belong in a single grid square. So a block lying entirely in one square will return one row in this next query; a block that intersects two squares will return two rows. Each row will include the geometries of the block, the square, and the intersection of the two, the total area of the block, the area of the intersection, and the total educational score for the block.

block_part AS (
    SELECT
        grid_ix,                     -- Grid square coordinates, e.g. 0,0
        m.gid AS manz_gid,           -- Manzana identifier
        ggm.geom,                    -- Grid square geometry
        m.graproes::FLOAT,           -- Educational score
        ST_Intersection(m.geom, ggm.geom)
            AS manz_bg_int,          -- Geometric intersection between grid square and city block
        ST_Area(m.geom)
            AS manz_area,            -- Area of the city block
        ST_Area(ST_Intersection(m.geom, ggm.geom)) / ST_Area(m.geom)
            AS manz_area_perc        -- Area of the intersection
    from
        gridgeom ggm
        JOIN manzanas m
            ON (ST_Intersects(m.geom, ggm.geom))
    WHERE
        m.graproes IS NOT NULL       -- Skip null educational scores
),

Here’s a sample of those results:

-[ RECORD 1 ]--+---------------------
grid_ix        | 15,16
manz_gid       | 61853
geom           | 0103000020E610000...
graproes       | 11
manz_bg_int    | 0103000020E610000...
manz_area      | 3.26175061395103e-06
manz_area_perc | 0.149256808754461
-[ RECORD 2 ]--+---------------------
grid_ix        | 15,17
manz_gid       | 61853
geom           | 0103000020E610000...
graproes       | 11
manz_bg_int    | 0103000020E610000...
manz_area      | 3.26175061395103e-06
manz_area_perc | 0.850743191246843

These results, which I admit to having selected with some care, show a single block, number 61853, which lies across the border between two grid squares. Now we’ll calculate the education score for each block fragment, and then divide the fragments into groups based on the grid square in which they belong, and aggregate the results. I did this in two separate CTEs.

grid_calc AS (
    SELECT
        grid_ix,
        geom,
        graproes * manz_area_perc AS grid_graproes,
        manz_area_perc
    FROM
        block_part
),
grid_accum AS (
    SELECT
        grid_ix,
        geom,
        SUM(grid_graproes) AS sum_graproes
    FROM grid_calc
    GROUP BY grid_ix, geom
),

This latest CTE gives results such as these:

-[ RECORD 1 ]+-----------------
grid_ix      | 12,14
geom         | 0103000020E61...
sum_graproes | 3888.53630440106

Now we’re left with turning these results into a visualization. I’d like to assign each polygon a height, and a color. To make the visualization easier to understand, I’ll divide the results into a handful of classes, and assign a height and color to each class.

Using an online color palette generator, I came up with a sequence of six colors, which progress from white to a green similar to the green bar on the Mexican flag. Another CTE will return these colors as an array, and yet another will assign the grid squares to groups based on their calculated score. Finally, a third will select the proper color from the array using that group value. At this point, readers are probably thinking “enough already; quit dividing everything into ever smaller CTEs”, to which I can only say, “Yeah, you may be right.”

colors AS (
    SELECT ARRAY['ffffff','d4ead7','a9d6af','7ec188','53ad60','299939'] AS colors
),
colorix AS (
    SELECT *,
        NTILE(ARRAY_LENGTH(colors, 1)) OVER (ORDER BY sum_graproes asc) AS edu_colorix
    FROM grid_accum, colors
),
color AS (
    SELECT *,
        colors[edu_colorix] as edu_color
    from colorix
)

The ntile() window function is useful for this kind of thing. It divides the given partition into buckets, and returns the number of the bucket for each row. Here, the partition consists of the whole data set; we sort it by educational score to ensure low-scoring grid squares get low-numbered buckets. Note also that I can change the colors, adding or removing groups, simply by adjusting the colors CTE. This could theoretically prove handy, if I decided I didn’t like the number of levels or the color scheme, but it’s a feature I never used for this visualization.

We’re on the home stretch, at last, and I should clarify how I plan to turn the database objects into KML, usable on a Liquid Galaxy. I used ogr2ogr from the GDAL toolkit. It converts between a number of different GIS data sources, including PostGIS to KML. I need to feed it the geometry I want to draw, as well as styling instructions and, in this case, a custom KML altitudeMode.

Styling is an involved topic; for our purposes it’s enough to say that I’ll tell ogr2ogr to use our selected color both to draw the lines of our polygons, and to fill them in. But moving the grid square’s geometry to an altitude corresponding to its educational score is fairly easy, using PostGIS’s ST_Force3DZ() function to add to the hitherto two-dimensional polygon a zero-valued third dimension, and ST_Translate() to move it above the surface of the earth a ways. So I can probably finish this with one final query:

-- Insert all previous CTEs here
SELECT
    ST_Translate(
        ST_Force3DZ(geom), 0, 0,
        alt_bias + edu_colorix * alt_percfactor
    ) AS edu_geom,
    'BRUSH(fc:#' || edu_color || 'ff);PEN(c:#' || edu_color || 'ff)' AS edu_style,
    'absolute' AS "altitudeMode"
FROM color, params;

You may remember alt_bias and alt_percfactor, the oddly named and thus far unexplained values in my first params CTE. These I used to control how far apart in altitude one group of polygons is from another, and to bias them all far enough above the ground to avoid the problem of them being obscured by terrain features. You may also remember that this query began with the CREATE TABLE grid_mza_vals AS... command, meaning that we’ll store the results of all this processing in a table, so ogr2ogr can get to it. We call ogr2ogr like this:

ogr2ogr \
    -f LIBKML education.kml \
    PG:"dbname=inegi user=josh password=<redacted>" \
    -sql "SELECT grid_ix, edu_geom, edu_style as \"OGR_STYLE\", \"altitudeMode\" FROM grid_mza_vals"

OGR’s LIBKML driver knows an attribute called “OGR_STYLE” is a style string, and one called “altitudeMode” is, predictably, the feature’s altitude mode. So this will create a KML file, containing one placemark for each row in our grid_mza_tables table. Each placemark consists of a colored square, floating in the air above Mexico City. The squares come in six different levels and six different colors, corresponding to our original education data. Something like this:

Here’s one of the placemarks from the KML.

      <Placemark id="sql_statement.1">
        <Style>
          <LineStyle>
            <color>ffffffff</color>
            <width>1</width>
          </LineStyle>
          <PolyStyle>
            <color>ffffffff</color>
          </PolyStyle>
        </Style>
        <ExtendedData>
          <SchemaData schemaUrl="#sql_statement.schema">
            <SimpleData name="grid_ix">11,10</SimpleData>
            <SimpleData name="OGR_STYLE">BRUSH(fc:#ffffffff);PEN(c:#ffffffff)</SimpleData>
          </SchemaData>
        </ExtendedData>
        <Polygon>
          <altitudeMode>absolute</altitudeMode>
          <outerBoundaryIs>
            <LinearRing>
              <coordinates>
                -99.17235146136,19.3090649119991,4300
                -99.15623264412,19.3090649119991,4300
                -99.15623264412,19.3275524211991,4300
                -99.17235146136,19.3275524211991,4300
                -99.17235146136,19.3090649119991,4300
              </coordinates>
            </LinearRing>
          </outerBoundaryIs>
        </Polygon>
      </Placemark>

This may be sufficient, but it gets confusing when viewed from a low angle, because it’s hard to tell which square belongs to which part of the map. I prefer having the polygons “extruded” from the ground, as KML terms it. I used a simple Perl script to add the extrude element to each polygon in the KML, resulting in this:

This query works, but leaves a few things to be desired. For instance, doing everything in one query is probably not the best option when lots of processing is involved. Ideally we would calculate the grid geometries once, and save them in a table somewhere, for quicker processing as we build the rest of the query and experiment with visulization options. Second, PostGIS provides an arguably more elegant way of finding grid squares in the first place, a method which also affords other options for interesting visualizations. Stay tuned for a future blog post discussing these issues. Meanwhile, CTEs proved a valuable way to systematize and modularize a very complicated query.

Creating API for Invoice Generator Using C# and Minimal APIs

Prerequisites

Setting up the Project

Setting up the Swagger UI

Setting up the Database Context with Entity Framework

Designing the Invoice Generator API

Creating and running database migrations

Implementing API Endpoints

Adding CORS for development

Integrating with Vue 3 Application

Deployment

Conclusion

Testing user-defined functions with pgTAP

Functions:

has_function()

functions_are()

ok() and is()

Testing more complex user-defined stored functions:

How To Set Up pgTAP for Writing PostgreSQL Database Unit Tests

Installing & setting up pgTAP

Writing simple database tests with pgTAP

More pgTAP tests

Test to check if a trigger is triggered

Test to check if a constraint is enforced

Organizing and naming your test files based on Test Anything Protocol (TAP)

Running your tests using pg_prove

pgTAP in EpiTrax

Conclusion

Find Text in Any Column of a PostgreSQL Table

Data Migration Tips

Customer Management

Migration Scripting

Data Migration History

Teamwork

Using pgTAP to automate database testing

Databases need tests too

What to test?

Iteratively improving

Looking back and ahead

Database Design: Using Documents

Example

Normalized Tables

Table Sizes

Composite Types

JSON Document Column

Binary JSON

Size Comparison

Usability

String Processing in SQL, the Hard Way

Migrating a Node.js app database from MongoDB to PostgreSQL

MongoDB vs. PostgreSQL

Mongoose and Sequelize

Modeling and Migrating Data

Caveats and Gotchas

Database integration testing with .NET

The sample project

Organizing the source code to allow for automated testing

Creating a DbContext instance to talk to the database

Defining the test database connection string in the appsettings.json file

Writing a simple test case that fetches data

Resetting the state of the database after each test

Writing another simple test case that stores data

Refactoring into a fixture for reusability

Using transactions to reset the state of the database

Full-text search on a budget: Xapian

Indexing

Searching

Database Design: Using Composite Keys

Single-column vs. Composite

Relations with Composite Keys

Ease of Change

Querying

Performance Considerations

Spatial queries with MySQL

Adding spatial information

Querying and filtering with spatial data

Intersecting

Spherical distance

Conclusion

Resources

`has_function()`

`functions_are()`

`ok()` and `is()`