Testing user-defined functions with pgTAP

2023-09-25T00:00:00+00:00

For a great introduction to the pgTAP protocol, please read over my colleague Josh Tolley’s article, using pgTAP to automate database testing. Also check out Edgar Mlowe’s article on how to set up pgTAP for writing PostgreSQL database unit tests.

As a protocol, pgTAP provides a great list of functions and commands to easily and clearly test all aspects of a database. As I’ve been tasked with more implementations of unit tests for user-defined functions, I thought it helpful to share some useful methods.

First we’ll go over a few basic pgTAP functions that are useful in testing the existence of functions and procedures. Then we’ll use the PostgreSQL port of the Sakila sample database for MySQL, called Pagila. We’ll construct two functions and go over a single test case. Using basic Postgres tools, I’ll walk through a few methods of calling and testing our functions. Finally, we’ll go over a more concise way of testing all cases for said functions.

Feel free to work alongside this article to hopefully leave with a comfortable idea of how to construct unit test files for your database.

Functions:

For this first example, remember to have the pgTAP extension in your schema and to perform it in a transaction (I’ve omitted BEGIN and ROLLBACK for brevity). Also use SELECT plan(<number of tests you’ll run>) when running the pgTAP functions.

Starting off, let’s create a basic addition function that returns an integer value:

CREATE OR REPLACE FUNCTION add_numbers(
    num1 integer,
    num2 integer
)
RETURNS integer
LANGUAGE plpgsql AS $$
BEGIN
    RETURN num1 + num2;
END;
$$;

To test this user-defined function, a few of the most relevant functions in pgTAP we can use are:

`has_function()`

Variations:

SELECT has_function( :schema, :function, :args, :description );
SELECT has_function( :schema, :function, :args );
SELECT has_function( :schema, :function, :description );
SELECT has_function( :schema, :function );
SELECT has_function( :function, :args, :description );
SELECT has_function( :function, :args );
SELECT has_function( :function, :description );
SELECT has_function( :function );

Parameters:

:schema
    Name of a schema in which to find the function.
:function
    Name of a function.
:args
    Array of data types of the function arguments.
:description
    A short description of the test.

Example:

SELECT has_function('your_schema', 'add_numbers', ARRAY['integer','integer']); -- default description used

Results:

ok 1 - Function add_numbers(integer, integer) should exist

has_function() is useful in checking the existence of a specific function. If you want to parse an entire schema, functions_are() is your command.

`functions_are()`

Variations:

SELECT functions_are( :schema, :functions, :description );
SELECT functions_are( :schema, :functions );
SELECT functions_are( :functions, :description );
SELECT functions_are( :functions );

Parameters:

:schema
    Name of a schema in which to find functions.
:functions
    An array of function names.
:description
    A short description of the test.

Example:

SELECT functions_are('your_schema', ARRAY['add_numbers', <All other functions need to be listed as well>]); -- default description used

`ok()` and `is()`

ok() and is() have similar functionality with slightly different parameters:

SELECT ok( :boolean, :description );

Example:

SELECT ok(add_numbers(5,5)=10, 'add_numbers correctly works');

Parameters:

:boolean
    A boolean value indicating success or failure.
:description
    A short description of the test.

While ok() is used for passing a test returning a boolean, is() takes two values to compare:

SELECT is( :have, :want, :description );

Example:

SELECT is(add_numbers(5,5), 10, 'add_numbers correctly works');

You can find functions that cover pretty much anything in PostgreSQL here in pgTAP’s documentation.

Testing more complex user-defined stored functions:

A client product we created utilizes a function that calls several other functions depending on its given argument. This function is called by the application and is passed an integer as an argument from which the function bases an initial SELECT query’s WHERE condition on, in this case WHERE id = [argument]. Variables are then assigned column values from this queried table. These variables are then used as the conditions to PERFORM functions that change parts of the database.

When testing this sort of scenario, it became apparent that we needed a set of test data, as well as the ability to track that test data, run the function with the data, and test to see if the changes are performed correctly.

The nice thing about pgTAP is that unit tests are meant to be rolled back leaving the database as you found it. When faced with a more complex function that nests other functions which loop through multiple tables, it may make more sense to fill those tables temporarily with the data required for the tests.

Let’s take a nostalgia trip and imagine we’re operating a DVD rental store. The Pagila sample database, which you can download on GitHub, represents our scenario. It comes populated and we’ll construct a function which utilizes other functions to perform changes on multiple tables.

Say the DVD rental shop gives extensions of rentals for special occasions, one being that a DVD was rented on Christmas week. Suppose we have an initial function that checks if a new DVD is rented on one of these special occasions. It then calls another function that will extend the return date, based on the specific occasion:

CREATE OR REPLACE FUNCTION extension_check(_rental_id integer)
    RETURNS void
    LANGUAGE plpgsql
AS $function$
DECLARE
    _rental_date date;
    _inventory_id integer;
    _customer_id integer;
    _return_date date;
    christmas_week date;
BEGIN

    SELECT INTO _rental_date
    rental_date FROM rental
    WHERE rental_id = _rental_id;

    IF EXTRACT(YEAR FROM _rental_date) = EXTRACT(YEAR FROM CURRENT_DATE) AND
       _rental_date <= (EXTRACT(YEAR FROM CURRENT_DATE)::text || '-12-25')::date AND
       _rental_date >  (EXTRACT(YEAR FROM CURRENT_DATE)::text || '-12-25')::date - INTERVAL '7 days' THEN
        PERFORM christmas_extension(_rental_id);
    END IF;
END;
$function$;

CREATE OR REPLACE FUNCTION christmas_extension(_rental_id integer)
    RETURNS void
    LANGUAGE plpgsql
AS $function$
DECLARE
    _rental_date date;
    _return_date date;
BEGIN
    SELECT return_date FROM rental WHERE rental_id = _rental_id INTO _return_date;

    IF _return_date IS NULL THEN
        UPDATE rental SET return_date = rental_date + INTERVAL '10 days' WHERE rental_id=_rental_id;
    ELSE
        UPDATE rental SET return_date = return_date + INTERVAL '10 days' WHERE rental_id=_rental_id;
    END IF;
END;
$function$;

The initial function, extension_check, simply checks if the rental was checked out on the week prior to and on Christmas (It’s a really dedicated DVD shop). We can also assume a function like this has other scenarios where it would extend rental dates but we’ll just show one for this exercise; christmas_extension is our one extension function that extends the return date by ten days.

To hold our unit test, we’ll create a file fittingly named extension_unit_tests.sql. A problem with the current sample database is that we can’t really test out the function directly since the dates don’t extend to the current year. Rather even if they did, we don’t want to directly affect the existing data especially if the dates were updated and the database was in production use.

Relatively basic but handy Postgres tools let us create test cases which we can pass to the functions. Afterwards, we can use pgTAP functions to confirm that our user-defined functions extension_check and christmas_extension worked properly.

For test cases in our new file, extension_unit_tests.sql, we’ll need a way to create and track them. For this, we’ll use a temporary table to track the test IDs and a PROCEDURE which can be called at any point to create the test cases. After that, we’ll simply call the procedure and run the user-defined function extension_check. Finally test its existence, test its result, and roll back our changes.

BEGIN;

CREATE TEMPORARY TABLE test_instance_references (
    test_rental_id integer PRIMARY KEY,
    test_number integer
);

CREATE OR REPLACE PROCEDURE fill_test_data (
    _rental_date date,
    _test_number integer
)
LANGUAGE PLPGSQL
AS $procedure$
DECLARE
    _address_id integer;
    _store_id integer;
    _staff_id integer;
    _inventory_id integer;
    _customer_id integer;
    _rental_id integer;
BEGIN

    SELECT nextval('customer_customer_id_seq1'::regclass) INTO _customer_id;
    SELECT nextval('rental_rental_id_seq'::regclass) INTO _rental_id;
    SELECT nextval('inventory_inventory_id_seq'::regclass) INTO _inventory_id;
    SELECT store_id FROM store LIMIT 1 INTO _store_id; -- any store will do
    SELECT address_id FROM address LIMIT 1 INTO _address_id; -- any address will do

    INSERT INTO customer (customer_id, store_id, first_name, last_name, address_id)
    VALUES (
        _customer_id,
        _store_id,
        'Test First Name',
        'Test Last Name',
        _address_id
    );

    INSERT INTO inventory (inventory_id, film_id, store_id)
    VALUES (
        _inventory_id,
        (SELECT film_id FROM film LIMIT 1), -- arbitrary film
        _store_id
    );

    INSERT INTO rental (rental_id, rental_date, inventory_id, customer_id, staff_id)
    VALUES (
        _rental_id,
        _rental_date,
        _inventory_id,
        _customer_id,
        (SELECT staff_id FROM staff WHERE store_id = _store_id LIMIT 1) -- select arbitrary staff member from store
    );

    INSERT INTO test_instance_references (test_rental_id, test_number)
    values (
        _rental_id,
        _test_number
    );

END;
$procedure$;

CALL fill_test_data('2023-12-25'::date, 1); -- Fill the db with our test data

SELECT extension_check((SELECT test_rental_id FROM test_instance_references WHERE test_number = 1)); -- Call our user-defined function

SELECT plan(2); -- State the number of tests we'll be performing

SELECT has_function(
    'extension_check',
    ARRAY['int']
);

SELECT is(
    (SELECT return_date FROM rental WHERE rental_id in (SELECT test_rental_id FROM test_instance_references WHERE test_number = 1))::date,
    ('2023-12-25'::date + INTERVAL '10 DAYS')::date,
    'Checking test data 1, week of Christmas'
);


ROLLBACK;

By design, pgTAP tests are performed in a transaction block with a ROLLBACK at the end to reverse all changes. Our first transaction is the creation of a temporary table called test_instance_references which will hold the IDs of the sample data we insert specifically in the rental table. This way we keep track of our specific test cases through their primary identifier.

Our PROCEDURE works similarly to a function, but where a function may have issues being called in a transaction block through pgTAP’s command line pg_prove, the use of a procedure and its CALL command is far more reliable.

In our example, the procedure simply adds in all the required test data. Note the insertion of test data into other tables that are referenced in rental, as well as an insert into the temporary table for tracking. To avoid using any existing data in the database, references are also test samples, like for the customer. Some, however, are selected from existing tables. It’s important to make sure you’re not affecting real data in the database in case a transaction fails. The example is obviously lax when it comes to that, specifically with the references to store and address, so please take note in your own unit tests to be careful when referring to existing data.

After we’ve established our procedure, we’re ready to run the tests. The pgTAP function plan() defines how many unit tests we are going to run. For the first example we run two tests.

Before running tests, we call the procedure fill_test_data which populates the database with test data and the temporary table to track that data.

Our first test is to check if the function exists. pgTAP’s has_function function is perfect for this.

Our procedure is then called with a date for the test rental. This test rental’s ID is saved into the temporary test_instance_references table which is then passed to the main function we want to test, extension_check. We call this function with the SELECT command, and immediately test the outcome with pgTAP’s is(). Our :have parameter holds the return_date of our test rental which is tested to see if it’s been correctly extended by 10 days. Our final parameter is the description which will be returned in the outcome of pg_prove if the test fails.

The ROLLBACK is meant to clean the database of all test data.

Let’s run this unit test on the command line, with pg_prove:

pg_prove -d <your_database> <path_to_your_unit_test>/extension_unit_tests.sql

We should see these results:

extension_unit_tests.sql .. ok
All tests successful.
Files=1, Tests=2,  0 wallclock secs ( 0.01 usr +  0.00 sys =  0.01 CPU)
Result: PASS

Congrats! We’ve tested out a single case for a function.

However, what if we want to test all possible cases for this function to be sure it performs correctly?

To do this, we can alter the procedure fill_test_data to loop through a date range given in the arguments _start_date and _end_date. For our example we’ll provide it with the week of Christmas, but it’ll work for any date range provided the rental store gives extensions for DVDs rented on other dates.

Instead of passing a test number as an argument, we can simply initialize it as a variable that’s incremented with every loop. Inside the WHILE loop, every iteration will populate the database with the test data, perform extension_check with the test rental ID, and increment the date and test number.

We’ll end the transaction a little differently by setting our pgTAP plan equal to the number of tests we have in our temporary table, to avoid hard-coding our plan count. We’ll also add one to check its existence with has_function.

Finally, our is() will also iterate through our temporary table and perform a check in the rental table to confirm that the return date has been correctly extended by ten days.

BEGIN;

CREATE TEMPORARY TABLE test_instance_references (
    test_rental_id integer PRIMARY KEY,
    test_number integer,
    start_date date
);


CREATE OR REPLACE PROCEDURE fill_test_data (
        _start_date date,
        _end_date date
    )
    LANGUAGE PLPGSQL
    AS $procedure$
    DECLARE
        _address_id smallint;
        _store_id smallint;
        _inventory_id integer;
        _customer_id integer;
        _rental_id integer;
        _test_number integer;
    BEGIN
        _test_number := 1;

        WHILE _start_date <= _end_date LOOP
            SELECT nextval('customer_customer_id_seq1'::regclass) INTO _customer_id;
            SELECT nextval('rental_rental_id_seq'::regclass) INTO _rental_id;
            SELECT nextval('inventory_inventory_id_seq'::regclass) INTO _inventory_id;
            SELECT store_id FROM store LIMIT 1 INTO _store_id; -- any store will do
            SELECT address_id FROM address LIMIT 1 INTO _address_id; -- any address will do

            INSERT INTO customer (customer_id, store_id, first_name, last_name, address_id)
            VALUES (
                _customer_id,
                _store_id,
                'Test First Name',
                'Test Last Name',
                _address_id
            );

            INSERT INTO inventory (inventory_id, film_id, store_id)
            VALUES (
                _inventory_id,
                (SELECT film_id FROM film LIMIT 1), -- arbitrary film
                _store_id
            );

            INSERT INTO rental (rental_id, rental_date, inventory_id, customer_id, staff_id)
            VALUES (
                _rental_id,
                _start_date,
                _inventory_id,
                _customer_id::smallint,
                (SELECT staff_id FROM staff WHERE store_id = _store_id LIMIT 1) -- select arbitrary staff member from store
            );

            INSERT INTO test_instance_references (test_rental_id, test_number, start_date)
            VALUES (
                _rental_id,
                _test_number,
                _start_date
            );

            PERFORM extension_check(_rental_id);

            _start_date := _start_date + INTERVAL '1 day';
            _test_number := _test_number + 1;

        END LOOP;
    END;
$procedure$;

CALL fill_test_data(('2023-12-25'::date - INTERVAL '6 days')::date, '2023-12-25'::date);

SELECT plan(count(*)::integer + 1) from test_instance_references;

SELECT * from test_instance_references;

SELECT has_function(
    'extension_check',
    ARRAY['int']
    -- Description optional - default description usually substantial
);

SELECT is(
    (SELECT return_date FROM rental WHERE rental_id = tf.test_rental_id)::date,
    (tf.start_date + INTERVAL '10 DAYS')::date
)
FROM test_instance_references as tf;


ROLLBACK;

Our results should be:

$ pg_prove -d <your_database> <path_to_your_unit_test>/extension_unit_tests.sql
extension_unit_tests.sql .. ok
All tests successful.
Files=1, Tests=8,  0 wallclock secs ( 0.02 usr +  0.00 sys =  0.02 CPU)
Result: PASS

As someone who found pgTAP fairly straightforward the challenge of intertwining those functions in a repeatable way was what drove me to write this article. Hopefully this makes it easier for the next coder to play around and get the most out of this simple but intuitive “Test Anything Protocol”.

Feel free to share your thoughts and comments below, or to share any more useful tips on pgTAP or general Postgres unit testing.

How To Set Up pgTAP for Writing PostgreSQL Database Unit Tests

2023-04-12T00:00:00+00:00

In a previous blog post, my colleague Josh Tolley introduced the concept of using pgTAP, a set of Postgres functions designed for writing unit tests within the database. This post will serve as a supplement to Josh’s post, focusing on the process of setting up pgTAP for testing a Postgres database.

Installing & setting up pgTAP

Note: make sure you have Postgres installed on your system. If you don’t have it, you can follow the Postgres documentation to learn how to install it.

To install pgTAP for Postgres, you will need to follow these steps:

Download the pgTAP source code from its GitHub repository.
```
git clone https://github.com/pgtap/pgtap.git
```
Navigate to the pgtap directory.
```
cd pgtap
```
Install with make.
```
make && make install
```
Finally, connect to your Postgres database using psql or any other Postgres client and run the following SQL command to create the pgTAP extension.
```
CREATE EXTENSION pgtap;
```

Now you should have pgTAP installed and ready to use in your Postgres database. If you face issues with installation, visit the pgTAP documentation for further help.

Writing simple database tests with pgTAP

Test to check if a table exists:

SELECT plan(1);
SELECT has_table('public', 'your_table', 'Table your_table should exist');
SELECT finish();

Test to check if a column exists in a table:

SELECT plan(1);
SELECT has_column('public', 'your_table', 'your_column', 'Column your_column should exist in your_table');
SELECT finish();

Test to check if a function returns the expected result:

SELECT plan(1);
SELECT is(your_function(), expected_result, 'your_function() should return expected_result');
SELECT finish();

More pgTAP tests

Test to check if a trigger is triggered

Consider the following example. We have a table named employees and a table named audit_log. When a new employee is added to the employees table, a trigger named insert_employee_trigger fires and logs the new employee’s ID and creation timestamp in the audit_log table. Let’s test if the trigger gets fired and produces the expected outcome.

Create the tables.

CREATE TABLE employees (
  id SERIAL PRIMARY KEY,
  name VARCHAR(255) NOT NULL
);

CREATE TABLE audit_log (
  id SERIAL PRIMARY KEY,
  employee_id INTEGER NOT NULL,
  created_at TIMESTAMP NOT NULL
);

Create the trigger and trigger function.

CREATE OR REPLACE FUNCTION insert_employee_trigger_function()
RETURNS TRIGGER AS $$
BEGIN
  INSERT INTO audit_log (employee_id, created_at) VALUES (NEW.id, NOW());
  RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER insert_employee_trigger
  AFTER INSERT ON employees
  FOR EACH ROW
  EXECUTE FUNCTION insert_employee_trigger_function();

Now, write a test using pgTAP to test the trigger.

-- Set the test plan
SELECT plan(1);

-- Prepare test data and perform operations that should fire the trigger
INSERT INTO employees (name) VALUES ('John Doe');


-- Check if the desired outcome of the trigger has occurred
SELECT is(
  (SELECT name FROM employees WHERE id = (SELECT employee_id FROM audit_log ORDER BY id DESC LIMIT 1)),
  'John Doe',
  'The insert_employee_trigger should have been fired and produced the expected outcome'
);

Test to check if a constraint is enforced

For this example, we’ll use the following two tables:

customers table with columns id and name.
orders table with columns id, customer_id, and amount
- There is a foreign key constraint between the orders table and the customers table on the customer_id column.

First, create the two tables and add the foreign key constraint.

CREATE TABLE customers (
  id SERIAL PRIMARY KEY,
  name VARCHAR(255) NOT NULL
);

CREATE TABLE orders (
  id SERIAL PRIMARY KEY,
  customer_id INTEGER REFERENCES customers(id),
  amount INTEGER
);

Now, create a test script named fk_test.sql with the following content.

BEGIN;

-- Create test data in customers table
INSERT INTO customers (id, name) VALUES (1, 'Customer 1');

-- Declare the number of tests
SELECT plan(1);

-- Attempt to insert an order with an invalid customer_id
-- This should fail due to the foreign key constraint
SELECT throws_ok(
    'INSERT INTO orders (id, customer_id, amount) VALUES (1, 969, 100);',
    '23503',
    'insert or update on table "orders" violates foreign key constraint "orders_customer_id_fkey"'
);

-- Conclude the test and discard changes
SELECT finish();
ROLLBACK;

Organizing and naming your test files based on Test Anything Protocol (TAP)

Test Anything Protocol (TAP) is a convention for naming test files and organizing them within a project directory structure. The naming convention involves adding a numeric prefix to the test file name to indicate the order in which the tests should be run. For example, 00-test.sql is the first test file to be executed, followed by 01-another-test.sql, and so on.

The project directory structure usually includes a directory called t (short for “test”) under the project root directory. This t directory contains all the test files following the TAP naming convention.

Using this structure makes it easier to maintain and organize tests, as well as to run them in the correct order. This is especially important when tests depend on the results of previous tests, or when you want to ensure that tests are executed in a specific sequence.

Here’s an example of a project directory structure using the TAP-related naming convention:

my_project/
│
├─ src/
│   ├── function1.sql
│   ├── function2.sql
│   └── function3.sql
│
└─ t/
    ├── 00-test.sql
    ├── 01-another-test.sql
    └── 02-yet-another-test.sql

In this example, the project contains a src directory for the source code (database functions) and a t directory for the test files following the TAP naming convention.

Running your tests using pg_prove

Now that we have written some tests using pgTAP, let’s see how we can run them. We’ll use pg_prove, a utility included with pgTAP, to automate the process and get a summary of the results.

To run your tests with pg_prove, you’ll need to specify the name of the database you want to connect to and the path to your test file. For example, if your test file is located at t/00-test.sql and your database is named epitrax_db, you can run the following command:

pg_prove -d epitrax_db t/00-test.sql

When you run this command, pg_prove will execute all the tests in the specified file and print a summary of the results. The output will include the number of tests run, the number of tests passed, the number of tests failed, and any diagnostic messages or errors that were encountered.

Here’s an example of what the output might look like:

t/00-test.sql .. ok
All tests successful.
Files=1, Tests=5,  0 wallclock secs ( 0.01 usr  0.00 sys +  0.02 cusr  0.00 csys =  0.03 CPU)
Result: PASS

In this example, pg_prove ran one test file (00-test.sql) and executed 5 tests. All of the tests passed, so the output shows that the result was PASS.

If any of the tests fail, pg_prove will print a detailed report of the failures, including any error messages or diagnostic information. This can help you quickly identify and fix any issues in your code.

Using pg_prove to automate your tests can save you a lot of time and effort, especially if you have a large number of tests to run. It also makes it easy to integrate your tests into your build process or continuous integration pipeline.

pgTAP in EpiTrax

EpiTrax relies on Postgres and its functions, making pgTAP an ideal tool for testing the platform’s database functions, ensuring data integrity, and maintaining the robustness of the system. By utilizing pgTAP, EpiTrax efficiently validates various aspects of its database structure and behavior.

Conclusion

pgTAP database unit tests provide numerous benefits, including ensuring code quality and simplifying the testing process. This guide should serve as a starting point for setting up pgTAP, and we encourage you to explore the pgTAP documentation and read the aforementioned blog post on automated database testing with pgTAP for more in-depth information on writing and running tests with pgTAP. By integrating pgTAP in projects like EpiTrax, it’s possible to improve overall database performance, stability, and security.

Find Text in Any Column of a PostgreSQL Table

2023-02-06T00:00:00+00:00

It’s not uncommon for me to want to find a particular text snippet in a PostgreSQL database. Easy enough, you might say. After all, that’s what databases are for: You feed them a bunch of information, ask them questions in the form of a query, and they give you the answer. So just write a query, right?

Well, maybe.

SQL stands for “Structured Query Language”, and the fact that it’s “structured” means not only that the database abides by some defined structure, but that your queries do, too, which implies that you know at the time you’re writing the query where in the structure you want to look. And that’s where the problem arises. What if I know “Kilroy” is somewhere in a table, but I don’t know what column to look in to find him? How do I write that query?

The first answer I came up with to that question depends on pg_dump: dump the contents of a table, search the results with grep, and there you have it.

$ pg_dump -t person mydb | grep -i kilroy
633132  F       NH      \N      Cristen212      J    Kilroy44        1983-09-28 00:00:00     \N      t       \N      \N      \N      \N      F       USA  \N       \N      \N      \N      \N      \N      \N      \N

Here I knew I wanted to look in the person table, so I specified that in the call to pg_dump, but this works as well when I want to search the entire database:

$ pg_dump  mydb | grep -i kilroy
716565  public  person  18185   {"session_user": "josh"}        {"id": 633132}  {"last_name": "Feeney44"}       {"last_name": "Kilroy44"}       U       123349  2023-02-03 11:19:40.587883-062023-02-03 11:19:40.587883-06    2023-02-03 11:19:40.592103-06   josh    \N      \N
633132  F       NH      \N      Cristen212      J    Kilroy44        1983-09-28 00:00:00     \N      t       \N      \N      \N      \N      F       USA  \N       \N      \N      \N      \N      \N      \N      \N

I can switch pg_dump to --inserts mode and thus see the table these entries were found in:

$ pg_dump --inserts mydb | grep -i kilroy
INSERT INTO audit.modification_log VALUES (716565, 'public', 'person', 18185, '{"session_user": "josh"}', '{"id": 633132}', '{"last_name": "Feeney44"}', '{"last_name": "Kilroy44"}', 'U', 123349, '2023-02-03 11:19:40.587883-06', '2023-02-03 11:19:40.587883-06', '2023-02-03 11:19:40.592103-06', 'josh', NULL, NULL);
INSERT INTO public.person VALUES (633132, 'F', 'NH', NULL, 'Cristen212', 'J', 'Kilroy44', '1983-09-28 00:00:00', NULL, true, NULL, NULL, NULL, NULL, 'F', 'USA', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL);

But pg_dump is considerably slower with --inserts turned on (about 200% slower, in my rudimentary testing on this one database), so perhaps I can get fancier with grep and achieve the same thing:

$ pg_dump mydb | grep -ie kilroy -e '^COPY' | grep -B1 -i kilroy
COPY audit.modification_log (id, table_schema, table_name, table_relid, app_user_info, id_columns, old_data, new_data, operation, transaction_id, ts_transaction, ts_statement, ts_clock, session_user_name, client_addr, query_text) FROM stdin;
716565  public  person  18185   {"session_user": "josh"}        {"id": 633132}  {"last_name": "Feeney44"}       {"last_name": "Kilroy44"}       U       123349  2023-02-03 11:19:40.587883-062023-02-03 11:19:40.587883-06    2023-02-03 11:19:40.592103-06   josh    \N      \N
--
COPY public.person (id, birth_gender, ethnicity, language, first_name, middle_name, last_name, birth_date, date_of_death, live, days_old_no_birthday, from_pregnant_id, fatal_condition_id, death_status, current_gender, country_of_birth, people_id, entity_id, birth_gender_id, ethnicity_id, primary_language_id, age_type_id, approximate_age_no_birthday, updated_at) FROM stdin;
633132  F       NH      \N      Cristen212      J    Kilroy44        1983-09-28 00:00:00     \N      t       \N      \N      \N      \N      F       USA  \N       \N      \N      \N      \N      \N      \N      \N

But the other day I realized I could easily search every a single table without leaving the database, thanks to \copy:

mydb=# \copy person to program 'grep -i kilroy';
COPY 117
633132  F       NH      \N      Cristen212      J    Kilroy44        1983-09-28 00:00:00     \N      t       \N      \N      \N      \N      F       USA  \N       \N      \N      \N      \N      \N      \N      \N

\copy requires copying the entire table to the client side, which wasn’t a concern in my instance but could be a problem for some folks, and you need to be a superuser to use server-side COPY ... TO PROGRAM, so there are some limitations to this approach.

I started thinking about this problem fresh off an encounter with row types, and realized we could do the whole thing in SQL:

mydb=# select * from person where person::text ~* 'kilroy';
   id   | birth_gender | ethnicity | language | first_name |             middle_name              | last_name |     birth_date      | date_of_death | live | days_old_no_birthday | from_pregnant_id | fatal_condition_id | death_status | current_gender | country_of_birth | people_id | entity_id | birth_gender_id | ethnicity_id | primary_language_id | age_type_id | approximate_age_no_birthday | updated_at
--------+--------------+-----------+----------+------------+--------------------------------------+-----------+---------------------+---------------+------+----------------------+------------------+--------------------+--------------+----------------+------------------+-----------+-----------+-----------------+--------------+---------------------+-------------+-----------------------------+------------
 633132 | F            | NH        |          | Cristen212 | J | Kilroy44  | 1983-09-28 00:00:00 |               | t    |                      |                  |                    |              | F              | USA              |           |           |                 |              |                     |             |                             |
(1 row)

This is my new favorite way of solving this problem, when I know what table I’m looking in but don’t know the column. In cases where I don’t know either, there’s always this monstrosity:

mydb=# select 'select ' || quote_literal(relname) || ' as relname, f.*
from ' || oid::regclass || ' f where f::text ~* ''kilroy''' from pg_class
where relkind = 'r' and relnamespace = 'public'::regnamespace; \gexec

This will work only in psql, as it depends on the \gexec command. It composes one query for each table, and then executes them in turn. It produces rather a lot of probably useless output you’ll need to sort through, and if you have (as I do) some tables with a large number of columns, I recommend trying it with \x turned on, to avoid having to page through quite so many results. But it did work, to find the entry I was looking for:

... Lots of output
(0 rows)

-[ RECORD 1 ]---------------+-------------------------------------
relname                     | person
id                          | 633132
birth_gender                | F
ethnicity                   | NH
language                    |
first_name                  | Cristen212
middle_name                 | J
last_name                   | Kilroy44
birth_date                  | 1983-09-28 00:00:00
date_of_death               |
live                        | t
days_old_no_birthday        |
from_pregnant_id            |
fatal_condition_id          |
death_status                |
current_gender              | F
country_of_birth            | USA
people_id                   |
entity_id                   |
birth_gender_id             |
ethnicity_id                |
primary_language_id         |
age_type_id                 |
approximate_age_no_birthday |
updated_at                  |

(0 rows)
... Lots more output

If I’m willing to expand beyond one-liner solutions, I can get psql to filter that useless output for me:

mydb=# \o | grep -i kilroy | grep -v select
mydb=# select 'select ' || quote_literal(relname) || ' as rname, f.* from ' || oid::regclass || ' f where f::text ~* ''kilroy'''
from pg_class where relkind = 'r' and relnamespace = 'public'::regnamespace; \gexec
mydb=# \o
 person | 633132 | F            | NH        |          | Cristen212 | J | Kilroy44  | 1983-09-28 00:00:00 |               | t    |                      |                  |                    |              | F              | USA              |           |           |                 |              |                     |             |                             |

When I did this, I had to use a final \o to set the output mode back to normal, before I saw results.

Do you have a different technique to solve this problem? I’m sure there are many other possibilities out there! Comment below!

Data Migration Tips

2023-02-04T00:00:00+00:00

When you’re in the business of selling software to people, you tend to get a few chances to migrate data from their legacy software to your shiny new system. Most recently for me that has involved public health data exported from legacy disease surveillance systems into PostgreSQL databases for use by the open source EpiTrax system and its companion EMSA.

We have collected a few tips that may help you learn from our successes, as well as our ~~mistakes~~particularly educational experiences.

Customer Management

Your job is to satisfy your customers, and your customers want to know how the migration is progressing. Give them an answer, even if it’s just a generalization. This may be a burndown chart, a calculated percentage, a nifty graphic, or whatever, but something your project managers can show to their managers, to know more or less how far along things are.

Your job is also to know your system; that’s not the customer’s job. They shouldn’t have to get their data into a specific format for you to make use of it. Be as flexible as possible in the data format and structure you’ll accept. In theory, so long as your customer can provide the legacy data in a machine-readable format, you should be able to use it. Let them focus on getting the data out of their legacy system — which is sometimes quite an effort in itself! Real data is almost always messy data, and your customer will probably want to take the opportunity to clean things up; make it as easy as possible for them to do that, while still ensuring the migration proceeds quickly and smoothly.

Be careful about the vocabulary you use with your customer. Your system and the legacy system probably deal with the same kinds of data, and do generally the same kinds of things. Of course, your software does it better than the old ’n’ busted mess you’re replacing, but in order to be better, your software has to be different from what it’s replacing. Different software means different methods, different processes, and different concepts. You and your customer might use the same words to mean totally different things, and unless you’re careful, those differences can remain hidden well into the implementation process, coming to light just in time to highlight some horrible mistake you’ve made. Try to avoid that. Talk with your customer to make sure your team and their team share a common vocabulary. If your system’s core function is to track widgets, and their legacy system’s function is also to track widgets, make sure you understand the differences between your software’s concept of a widget and their legacy system’s concept of a widget.

Migration Scripting

For our products and our customers, we’ve found pretty much every migration process is different, and each one is a custom programming job. Your team should decide at the beginning what steps the migration needs to include, and what technologies it will use in each step. Here at End Point, we often like to work directly in the database, in SQL. Migrations are all about manipulating data, and SQL is well suited for that task. With whatever technology you use, decide how you’ll use it, to manage the considerations given here. You can change your mind later and refactor accordingly, but always have a plan you’re following.

Design the migration as a sequence of processes. That is, first you might import one type of record, next another type of record that depends on the previous one, followed by several further steps to import data from a third source, clean it, map values from the legacy system to the new system, validate the results, and create records in the destination database. Of course the steps will vary from project to project, but the point is your migration will probably include several steps which need to be run in a specific order, so plan your development conventions accordingly. At End Point, we often like to put each step in a SQL file, and name each file beginning with a number, so you can run each script in order sorted by filename, and achieve the correct result. We might have files called:

01_import_products.sql
02_import_customers.sql
03_import_order_history.sql

It’s also common to implement each step one at a time, and to need to run each step several times as it’s being developed. We find it very helpful to wrap each step in a transaction. Often that means each SQL file begins with a BEGIN; statement, and ends with COMMIT;. Often I’ll leave out the COMMIT until I’m finished working on a file. That way to work on the code I can open a database session and run the migration script I’m working on, and when it completes, it will leave me inside the open transaction where I can inspect the results of my work. I make changes to the script, roll back the transaction, and run the script again, for as many iterations as it takes. I only add a COMMIT when I’ve tested the whole file and think it’s ready for the next step in testing.

I mentioned above that the customer may want to use this opportunity to clean their data. You should want this, too. Make sure the data you’re feeding your new system is as clean and well-structured as possible. You may find, as we do, that most of the work in your migrations is in validating the input data, and that actually creating new records in your application is almost an afterthought. That’s OK. You may also find there are places where your application, wonderful though it may be, could stand to be more strict about the data it accepts. I have often discovered my application’s database needs a uniqueness constraint, or a foreign key, thanks to a migration I was working on.

Data Migration History

I wish I could truthfully claim all our migrations go off flawlessly, but that would be a lie. It’s not unheard of to run into some corner case, a few weeks or even months after the migration goes live, where data wasn’t migrated correctly.

On the other hand, it’s certainly not uncommon for a customer or coworker to spot something that strikes them as odd, after the migration goes live, only to find later that everything was in fact correct. In either case, it’s important to preserve a history of the migration to investigate these concerns. We accomplish this with a few specific steps:

Create a database schema for the migration, and a table within that schema for each data file or object type we’re importing. I call these tables “staging tables”, where the incoming data is “staged” as it’s cleaned and validated. Having a separate schema means these tables can remain in the production database long after the migration is complete, generally without interfering with anything.
These staging tables should generally use text fields, to be as forgiving and flexible as possible with the incoming data. We can clean, parse, and reformat the data after it’s imported.
Don’t change the data in these staging tables; add to the data instead. In other words, if you need to map a value from the legacy system to a different value for your new system, don’t change the column you imported into the staging table; instead, add a new column to the staging table where you’ll store the re-mapped value. If you need to parse a text field into a date (because you followed the instruction to use text fields!), don’t change the type of an imported column; instead, add a new column of date or timestamp type, to store the parsed value. That way, when three months down the road someone discovers that some of the imported records have weird dates, you have all the information you need to determine whether the fault lies with the imported data or some step of the migration progress. Knowing exactly where the fault crept in leaves you that much more empowered to fix it.
Keep track of your migrated records’ primary keys, in the legacy system and the new system. Imagine you’ve just imported your client’s legacy customer list into a staging table. This data includes the legacy system’s primary key. Add a new column to the table for your new system’s primary key, and populate it. Many of our systems use an integer sequence as a primary key, so we’d add a new integer column to the staging table, and fill it with the next values from the sequence. Following this principle will give you several important abilities:
- You can always connect a record in the legacy system with its corresponding record(s) in the new system. If you’ve imported a customer list in this fashion, then when you’re importing the order data later, and each order points to a customer using a legacy customer primary key, you can easily find the correct customer primary key to use in your system.
- You can easily know if a record in your system comes from the migration, or from normal day-to-day business. You will probably use this every time you try to debug something with your migration.
- If you need to remove all imported records and re-import them, you can identify exactly which records those are. This should be only rarely needed.

Finally, document the decision making process, in comments directly in your code. For instance, if you have a table of mappings from one value to another, chances are good you arrived at the final version of that mapping table only after some discussion with the customer. Chances are also good someone’s going to question it later on. It’s helpful to keep a comment around, something like, # Joe Rogers verified this is the correct mapping in the daily standup meeting, 23 Nov 2022. This is especially common if you eventually decide to ignore a certain class of records. /* Rebecca says ignore all records with type = "ARCHIVED", via group email 9 Jan 2023 */ is a very helpful clue when someone comes around wondering where those records went.

Teamwork

My remaining tips apply to almost any programming project. First, use source control, and commit your code to it often. I can’t count how often I’ve been grateful the Git repository had a backup of my work, or made my work accessible to fill some unexpected need on some other system, nor can I count how many times I’ve been stuck because someone else didn’t commit their code so I couldn’t get at it when I needed it. Let’s not talk about how many times I’ve caused someone else to get stuck in the same way… Of course, don’t commit your customer’s data. But you should commit your code, and commit it often.

Finally, where possible, work with someone else. Two programmers reviewing each other’s code and collaborating on solutions are often far better than two programmers working alone, or one programmer working twice as much.

Speaking of working out solutions together, I’d love your help improving this list. What keys have you found are important for data migration projects? I welcome your comments. And if you’re looking for someone to handle a data migration project for you, give us a call!

Building and Hosting a Web App with .NET 6, Postgres and Linux

2022-11-03T00:00:00+00:00

For well over a decade, working with the .NET framework meant running Windows. With the release of .NET Core in 2016, developers were granted the freedom to choose their OS, including Linux; no longer were we bound to Windows. However, few took the plunge, at least in my experience. Why? Well, we are comfortable with what we know, and afraid of what we don’t.

The truth is that building a .NET application on Linux is not that hard, once you get over a few minor bumps in the road. And there are many advantages to this approach, including flexibility, simplicity, and lower costs.

To demonstrate this, we will create a simple .NET MVC web application that connects to Postgres. Then, we will host the app on Linux with Nginx. Shall we start?

Preparing the database

First, you’ll want to install Postgres locally. If you’re using a Mac, this step is very easy. Simply install Postgres.app and you’ll be ready to go.

If you’re using Windows, check out the Windows Installers page on the Postgres website to download the latest installer.

Creating the projects

To develop .NET 6 apps, you will need to install Visual Studio 2022. Check out the Visual Studio downloads page for options for both Windows and Mac.

Start by opening up Visual Studio and creating a new Web Application (MVC) project, and choosing .NET 6.0 as the target framework. I’ve named my project “DotNetSix.Demo”. Here are the steps as they look in Visual Studio on my Mac.

On the final screen, go ahead and give your solution a name, and then click Create. Visual Studio should create a new solution for you, with a single web project. It will automatically create the necessary MVC folders, including Controllers, Models, and Views.

Setting up the database connection

For this demo, we’ll create a simple app that tracks books that you’ve recently read. Let’s go ahead and add a few simple models for the database: Author and Book. You can create these files in the pre-existing Models folder.

Book Model:

using System.ComponentModel.DataAnnotations;
using System.Text.Json.Serialization;

namespace DotNetSix.Demo.Models
{
    public class Book
    {
        public int Id { get; set; }

        public string Title { get; set; }

        [Display(Name = "Publish Date")]
        [DisplayFormat(DataFormatString = "{0:d}")]
        public DateTime PublishDate { get; set; }

        public Author? Author { get; set; }
    }
}

Author Model:

using System.ComponentModel.DataAnnotations;
using System.Text.Json.Serialization;

namespace DotNetSix.Demo.Models
{
    public class Author
    {
        public int Id { get; set; }

        public string FirstName { get; set; }

        public string LastName { get; set; }

        public ICollection<Book>? Books { get; set; }
    }
}

Next, add a folder named “Data” in your project. This will hold a few classes necessary for connecting to Postgres and creating the database. Create a new class file in the folder called AppDBContext.cs. This class will use EF Core to setup a database connection.

AppDBContext.cs:

using Microsoft.EntityFrameworkCore;
using DotNetSix.Demo.Models;

namespace DotNetSix.Demo.Data
{
	public class AppDBContext : DbContext
	{
		public AppDBContext(DbContextOptions<AppDBContext> options) : base(options)
		{
			Database.EnsureCreated();
			DBInitializer.Initialize(this);
		}

		public DbSet<Book> Books { get; set; }
		public DbSet<Author> Authors { get; set; }
	}
}

Then create another class file in the folder called DBInitializer.cs. This class will initialize the Postgres database with test data.

DBInitializer.cs:

using DotNetSix.Demo.Models;

namespace DotNetSix.Demo.Data
{
    public static class DBInitializer
    {
        public static void Initialize(AppDBContext context)
        {
            // Look for any existing Book records.
            if (context.Books.Any())
            {
                return;   // DB has been seeded
            }

            var books = new Book[]
            {
                new Book
                {
                    Title="Blood Meridian",
                    PublishDate=DateTime.Parse("04-01-1985"),
                    Author=new Author{FirstName="Cormac",LastName="McCarthy"}
                },
                new Book
                {
                    Title="The Dog of the South",
                    PublishDate=DateTime.Parse("01-31-1979"),
                    Author=new Author{FirstName="Charles",LastName="Portis"}
                },
                new Book
                {
                    Title="Outline",
                    PublishDate=DateTime.Parse("05-15-2014"),
                    Author=new Author{FirstName="Rachel",LastName="Cusk"}
                },
            };

            context.Books.AddRange(books);
            context.SaveChanges();
        }
    }
}

At this point, you may notice some errors in your IDE. This is because we need to add two important Nuget packages: Microsoft.EntityFrameworkCore and Npgsql.EntityFrameworkCore.PostgreSQL. You can add these by right-clicking on the Dependencies folder and clicking “Manage Nuget Packages…”. Here’s how it looks in Visual Studio.

To round out the database connection, you’ll want to update your Program.cs file, adding the DB Context and the initializing the database. Also, you may encounter an error when you first run your application, citing incompatible dates between .NET and Postgres. To fix this, we will set the Npgsql.EnableLegacyTimestampBehavior to true.

Here is the complete Program.cs for your reference. Lines 7–8, 14, and 24–34 are what was added to the default Program.cs that is created as part of the web project.

Program.cs:

using DotNetSix.Demo.Data;
using DBContext = DotNetSix.Demo.Data.AppDBContext;
using Microsoft.EntityFrameworkCore;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddDbContext<DBContext>(options =>
    options.UseNpgsql(builder.Configuration.GetConnectionString("DemoDbContext")));

// Add services to the container.
builder.Services.AddControllersWithViews();

var app = builder.Build();
AppContext.SetSwitch("Npgsql.EnableLegacyTimestampBehavior", true);

// Configure the HTTP request pipeline.
if (!app.Environment.IsDevelopment())
{
    app.UseExceptionHandler("/Home/Error");
    // The default HSTS value is 30 days. You may want to change this for production scenarios, see https://aka.ms/aspnetcore-hsts.
    app.UseHsts();
}

using (var scope = app.Services.CreateScope())
{
    var services = scope.ServiceProvider;

    var context = services.GetRequiredService<DBContext>();
    // Note: if you're having trouble with EF, database schema, etc.,
    // uncomment the line below to re-create the database upon each run.
    //context.Database.EnsureDeleted();
    context.Database.EnsureCreated();
    DBInitializer.Initialize(context);
}

app.UseHttpsRedirection();
app.UseStaticFiles();

app.UseRouting();

app.UseAuthorization();

app.MapControllerRoute(
    name: "default",
    pattern: "{controller=Books}/{action=Index}/{id?}");

app.Run();

Note that the pattern in MapControllerRoute points to a new controller and action that we will create in the next section.

Updating the config file

The final task to connect to Postgres is to update the appsettings.json file with a connection string. Since I used Postgres.app to install Postgres, my connection string is simple. The username is the same as my Mac, and there is no password. Here is the full file:

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning"
    }
  },
  "AllowedHosts": "*",
  "ConnectionStrings": {
    "DemoDbContext": "Host=localhost;Database=demo;Username=dylan"
  }
}

Displaying the test data

Now that we have the database connection setup, let’s make some small changes to the controllers and views in order to see the data in Postgres.

Add a new file in the Controllers folder called BookController.cs. This file provides an Index controller action that queries the book data from Postgres using EFCore.

Controllers/BooksController.cs:

using Microsoft.EntityFrameworkCore;
using Microsoft.AspNetCore.Mvc;
using DotNetSix.Demo.Data;

namespace DotNetSix.Demo.Controllers
{
    public class BooksController : Controller
    {
        private readonly AppDBContext _context;
        private readonly IConfiguration _config;

        public BooksController(AppDBContext context, IConfiguration config)
        {
            _context = context;
            _config = config;
        }
        // GET: Papers
        public async Task<IActionResult> Index()
        {
            var appDBContext = _context.Books.Include(b => b.Author);
            return View(await appDBContext.ToListAsync());
        }
    }
}

Now create an accompanying view. Add a “Books” folder under Views, and then add a new file to the Books folder called Index.cshtml. This view will receive the data from the controller and display a simple table of recently read books.

Books/Index.cshtml:

@model IEnumerable<DotNetSix.Demo.Models.Book>

<head>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <link rel="stylesheet" href="./css/CreateStyleSheet.css" asp-append-version="true" />
    <link rel="stylesheet" href="~/css/site.css" asp-append-version="true" />
</head>

@{
    ViewData["Title"] = "Index";
}

<h1>Books Recently Read</h1>

<div class="table-wrapper">
    <div class="table-container">
        <table class="table">
            <thead>
                <tr>
                    <th>
                        @Html.DisplayNameFor(model => model.Title)
                    </th>
                    <th>
                        @Html.DisplayNameFor(model => model.Author)
                    </th>
                    <th>
                        @Html.DisplayNameFor(model => model.PublishDate)
                    </th>
                </tr>
            </thead>
            <tbody>
                @foreach (var item in Model)
                    {
                    <tr>
                        <td>
                            @Html.DisplayFor(modelItem => item.Title)
                        </td>
                        <td>
                            @{
                                var authorFullName = item.Author.FirstName + " " + item.Author.LastName;
                                @Html.DisplayFor(modelItem => authorFullName);
                            }
                        </td>
                        <td>
                            @Html.DisplayFor(modelItem => item.PublishDate)
                        </td>
                    </tr>
                }
            </tbody>
        </table>
    </div>
</div>

With the above files in place, you are now ready to run your app. Go ahead and run the project in Visual Studio. You should then see the Books index page in your browser. Hopefully it looks like this:

Installing and configuring Nginx

Just a heads up that the following sections borrow from the MSDN article on hosting ASP.NET Core on Linux. Be sure to check out the article if you’re in need of more info or additional help!

Now that we have the application running, we can prepare to deploy it to a Linux server. The first step in preparing your server to host the application is to install Nginx. Nginx will act as a reverse proxy to your .NET application running on localhost. To install Nginx, use the appropriate package manager for your Linux distro. For example, if you’re running Debian, use apt-get to install Nginx:

sudo apt-get update
sudo apt-get install nginx

You can verify the installation by running sudo nginx -v, which should display the version. Finally, start Nginx using the command sudo service nginx start.

Once Nginx is installed, you’ll need to configure it to host the .NET Core application. Go ahead and create a new Nginx configuration file in /etc/nginx/conf.d. In our case, we’ll name it dotnetsixdemo.conf. Within this configuration file, we’ll do a few things:

Redirect HTTP traffic to HTTPS.
Use HTTPS with an SSL cert installed by Let’s Encrypt certbot.
Configure Nginx to forward HTTP requests to the .NET application, which by default will run locally at http://127.0.0.1:5000.

Here is what our configuration file ends up looking like.

server {
    if ($host = dotnetsixdemo.org) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

    listen        80;
    listen       [::]:80;
    server_name  dotnetsixdemo.org;
    return 404; # managed by Certbot
}

server {
    server_name  dotnetsixdemo.org;

    location / {
        proxy_pass         http://127.0.0.1:5000;
        proxy_http_version 1.1;
        proxy_set_header   Upgrade $http_upgrade;
        proxy_set_header   Connection keep-alive;
        proxy_set_header   Host $host;
        proxy_cache_bypass $http_upgrade;
        proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto $scheme;
    }

    listen [::]:443 ssl; # managed by Certbot
    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/dotnetsixdemo.org/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/dotnetsixdemo.org/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}

Final application adjustments

Before we publish the app, we need to make a small change to the Program.cs file. This will allow redirect and security policies to work correctly given the Nginx reverse proxy setup.

using Microsoft.AspNetCore.HttpOverrides;

...

app.UseForwardedHeaders(new ForwardedHeadersOptions
{
    ForwardedHeaders = ForwardedHeaders.XForwardedFor | ForwardedHeaders.XForwardedProto
});

Additionally, you’ll want to update your appsettings.json file to reflect your target environment, in particular the connection string for Postgres.

Time to publish!

We are now ready to publish the app to the server. In the command prompt, navigate to the root directory of the project, and run the following command.

dotnet publish --configuration release

This will build the app and create a new output folder at /your-project-root/bin/Release/net6.0/publish.

At this point, all that’s left to do is to copy the contents of the publish folder to the server. You can do this using a variety of tools including SCP, SFTP, etc.

Running the app on the server

Once you’ve copied the app to the server, you can start the app by navigating to the directory where the app was copied, and then running the command dotnet [app_assembly].dll. For our demo app, the target DLL would be DotNetSix.Demo.dll.

This will run the app at http://127.0.0.1:5000, and it should now be accessible via the URL that you configured using Nginx. Based on the Nginx configuration provided above, that would be https://dotnetsixdemo.org. Go ahead and test your site in the browser to make sure it is accessible and the reverse proxy is working properly.

Using systemd to run the app as a service

As you might have noticed, there are a few problems with running the app directly using the dotnet command above. The app could easily stop running if the server encounters an issue, and the app is not monitorable. To fix this, let’s use systemd to run and monitor the app process.

Create a new service definition file by running sudo nano /etc/systemd/system/dot-net-six-demo.service and entering the following.

[Unit]
Description=Dotnet Six Demo App

[Service]
WorkingDirectory=/home/dotnetsix/publish
ExecStart=/usr/bin/dotnet /home/dotnetsix/publish/DotNetSix.Demo.dll
Restart=always
# Restart service after 10 seconds if the dotnet service crashes:
RestartSec=10
KillSignal=SIGINT
SyslogIdentifier=dotnetsixdemo
User=dotnetsix
Environment=ASPNETCORE_ENVIRONMENT=Production
Environment=DOTNET_PRINT_TELEMETRY_MESSAGE=false

[Install]
WantedBy=multi-user.target

Note that the paths in WorkingDirectory and ExecStart should match where you copied the application build files on the server, as part of the publish step above.

Also, the User option specifies the user that manages the service and runs the process. In our example, this is the dotnetsix user. You’ll want to create your own user, and importantly grant that user proper ownership of the application files.

To finalize the new service, save the service file and then start the service with sudo systemctl enable --now dot-net-six-demo.service. This will run the app now in the background via systemd and ensure it also starts up after the next reboot.

Viewing logs

Since we are now running the app via systemd, we can use the journalctl to view logs. To view the logs for the demo app, you would run the command sudo journalctl -fu dot-net-six-demo.service.

Wrapping up

That is all! We now have a .NET application running on our Linux server, hosted with Nginx via a reverse proxy, and connecting to a local PostgreSQL database. As we can see, there are several steps to take into account, but the process itself is not particularly complex.

See also my co-worker Kevin Campusano’s blog post for more tips on using .NET with PostgreSQL.

Have you had problems working with .NET and Linux? Do you have any alternative solutions to the ones proposed here? We await your comment.

Upgrading PostgreSQL 14 to 15 on Fedora, RHEL, CentOS, Rocky, Alma Linux with PGDG RPM packages

2022-10-14T00:00:00+00:00

PostgreSQL 15 changes

Yesterday PostgreSQL 15 was released! It includes a number of headline features since version 14 that make it worth upgrading your databases running earlier versions of PostgreSQL:

Improved sort performance
In-memory statistics collection (removing the statistics collector process)
SQL-standard MERGE syntax that can include INSERT, UPDATE, and DELETE actions in a single statement
Logical replication publisher options to include all tables in a schema (including those added in the future), row filtering and column lists, and simplified conflict management
JSON format log output (to files only, not syslog)
Optional LZ4 and Zstandard compression for WAL (write-ahead log) files, especially useful for those not using btrfs or zfs filesystem built-in compression
New regular expression functions regexp_count, regexp_instr, regexp_like, and regexp_substr
And many other performance improvements and feature enhancements

Thanks to the PostgreSQL developers for the continuing amazing work!

Prerequisites for upgrading

This article shows how to upgrade to PostgreSQL 15 if you:

are running one of the Red Hat family of Linux operating systems, including Fedora, Red Hat Enterprise Linux (RHEL), Rocky Linux, AlmaLinux, Oracle Linux, or CentOS
have PostgreSQL 14 installed
are not using dnf modularity or the stock distribution postgresql RPMs, but rather the PostgreSQL Global Development Group (PGDG) postgresql14 RPMs that allow simultaneous coexistence of multiple major Postgres versions

My examples are done on the Fedora 36 x86_64 OS. Things will be very similar on other Red Hat family members.

We are doing system administration work here, so let’s act as system administrators and work as the root user rather than using sudo repetitively:

$ su -
Password:
[root@yourhost ~]#

Stop PostgreSQL 14

Stop your existing PostgreSQL 14 database server, if it is running, and prevent it from starting automatically at boot in the future:

[root@yourhost ~]# systemctl disable --now postgresql-14

Update your system

Make sure your packages are fully updated:

[root@yourhost ~]# dnf upgrade
Last metadata expiration check: 2:28:40 ago on Fri 14 Oct 2022 09:01:38 AM MDT.
Dependencies resolved.
Nothing to do.
Complete!

If there were updates pending, apply them. If the Linux kernel, glibc, systemd, or other core packages were updated, reboot to load the latest of all parts of your system.

Install PostgreSQL 15

We will here use a basic set of PostgreSQL client, server, development, and extra contributed packages.

You may also want some of the other available packages for procedural languages such as PL/Perl and PL/Python, PostGIS, or other packages. See the Yum repository index.

Now install PostgreSQL 15, which happily coexists alongside PostgreSQL 14 (and possibly other versions) thanks to the way the PGDG RPMs are designed and where they install files:

[root@yourhost ~]# dnf install postgresql15-server postgresql15-devel postgresql15-contrib
Last metadata expiration check: 2:29:14 ago on Fri 14 Oct 2022 09:01:38 AM MDT.
Dependencies resolved.
==========================================================================
 Package                  Architecture  Version         Repository   Size
==========================================================================
Installing:
 postgresql15-contrib     x86_64        15.0-1PGDG.f36  pgdg15      708 k
 postgresql15-devel       x86_64        15.0-1PGDG.f36  pgdg15      5.2 M
 postgresql15-server      x86_64        15.0-1PGDG.f36  pgdg15      5.9 M
Installing dependencies:
 postgresql15             x86_64        15.0-1PGDG.f36  pgdg15      1.5 M
 postgresql15-libs        x86_64        15.0-1PGDG.f36  pgdg15      290 k

Transaction Summary
==========================================================================
Install  5 Packages

Total download size: 14 M
Installed size: 54 M
Is this ok [y/N]: y
Downloading Packages:
(1/5): postgresql15-contrib-15.0-1PGDG.f36.x86_64.rpm  412 kB/s | 708 kB     00:01
(2/5): postgresql15-libs-15.0-1PGDG.f36.x86_64.rpm     171 kB/s | 290 kB     00:01
(3/5): postgresql15-15.0-1PGDG.f36.x86_64.rpm          435 kB/s | 1.5 MB     00:03
(4/5): postgresql15-devel-15.0-1PGDG.f36.x86_64.rpm    1.0 MB/s | 5.2 MB     00:05
(5/5): postgresql15-server-15.0-1PGDG.f36.x86_64.rpm   1.7 MB/s | 5.9 MB     00:03
-----------------------------------------------------------------------------------
Total                                                  2.0 MB/s |  14 MB     00:06
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                             1/1
  Installing       : postgresql15-libs-15.0-1PGDG.f36.x86_64     1/5
  Running scriptlet: postgresql15-libs-15.0-1PGDG.f36.x86_64     1/5
  Installing       : postgresql15-15.0-1PGDG.f36.x86_64          2/5
  Running scriptlet: postgresql15-15.0-1PGDG.f36.x86_64          2/5
  Running scriptlet: postgresql15-server-15.0-1PGDG.f36.x86_64   3/5
  Installing       : postgresql15-server-15.0-1PGDG.f36.x86_64   3/5
  Running scriptlet: postgresql15-server-15.0-1PGDG.f36.x86_64   3/5
  Installing       : postgresql15-contrib-15.0-1PGDG.f36.x86_64  4/5
  Installing       : postgresql15-devel-15.0-1PGDG.f36.x86_64    5/5
  Running scriptlet: postgresql15-devel-15.0-1PGDG.f36.x86_64    5/5
  Verifying        : postgresql15-15.0-1PGDG.f36.x86_64          1/5
  Verifying        : postgresql15-contrib-15.0-1PGDG.f36.x86_64  2/5
  Verifying        : postgresql15-devel-15.0-1PGDG.f36.x86_64    3/5
  Verifying        : postgresql15-libs-15.0-1PGDG.f36.x86_64     4/5
  Verifying        : postgresql15-server-15.0-1PGDG.f36.x86_64   5/5

Installed:
  postgresql15-15.0-1PGDG.f36.x86_64         postgresql15-contrib-15.0-1PGDG.f36.x86_64
  postgresql15-devel-15.0-1PGDG.f36.x86_64   postgresql15-libs-15.0-1PGDG.f36.x86_64
  postgresql15-server-15.0-1PGDG.f36.x86_64

Complete!

Create the new PostgreSQL 15 database cluster

Now that we have the new PostgreSQL version installed, we can create the new database cluster.

You need to use the same Postgres initdb options that were used when you ran initdb for Postgres 14.

For example, if you used the helpful data checksums feature, you need to pass the -k option to initdb, which is done indirectly via the RPM-specific postgresql-15-setup script like this:

[root@yourhost ~]# PGSETUP_INITDB_OPTIONS=-k /usr/pgsql-15/bin/postgresql-15-setup initdb
Initializing database ... OK

If you used the default initdb options, just omit the PGSETUP_INITDB_OPTIONS environment variable:

[root@yourhost ~]# /usr/pgsql-15/bin/postgresql-15-setup initdb
Initializing database ... OK

Migrate your data with pg_upgrade

At this point you could start the PostgreSQL 15 server and use psql to connect to it, if you want to start with an empty database or otherwise load your data by the usual means such as with psql or pg_restore using output from pg_dump.

But if you want to convert all of your existing PostgreSQL 14 cluster’s data so everything comes over as is, you can now run pg_upgrade.

For any important system, back up your data before doing anything else, and read through the whole pg_upgrade manual. It contains several important points to consider:

Use compatible initdb flags that match the old cluster. (Already discussed.)
Install extension shared object files. (Some may be contained in the contrib package, while others are separate, such as PostGIS.)
Set authentication to peer in pg_hba.conf for both old and new Postgres versions. This is the default for the new cluster, but your old PostgreSQL 14 cluster may need to be adjusted so pg_update can access it.

We will run pg_upgrade as the postgres OS user and specify the standard PGDG RPM locations for Postgres 14 & 15:

[root@yourhost ~]# su - postgres
[postgres@yourhost ~]$ /usr/pgsql-15/bin/pg_upgrade -b /usr/pgsql-14/bin -B /usr/pgsql-15/bin -d /var/lib/pgsql/14/data -D /var/lib/pgsql/15/data -j 4
Performing Consistency Checks
-----------------------------
Checking cluster versions                                   ok

old cluster does not use data checksums but the new one does
Failure, exiting

Oops! pg_upgrade can’t convert an old cluster that didn’t have data checksums into a new one that did, so in this case we need to delete our new cluster and recreate it without the initdb -k option for checksums.

Exit back to your root shell and re-run a suitable initdb command like those shown above:

[postgres@yourhost ~]$ exit
logout
[root@yourhost ~]# rm -rf ~postgres/15/data/*
[root@yourhost ~]# /usr/pgsql-15/bin/postgresql-15-setup initdb
Initializing database ... OK

Now let’s try running pg_upgrade again:

[root@yourhost ~]# su - postgres
[postgres@yourhost ~]$ /usr/pgsql-15/bin/pg_upgrade -b /usr/pgsql-14/bin -B /usr/pgsql-15/bin -d /var/lib/pgsql/14/data -D /var/lib/pgsql/15/data -j 4
Performing Consistency Checks
-----------------------------
Checking cluster versions                                   ok
Checking database user is the install user                  ok
Checking database connection settings                       ok
Checking for prepared transactions                          ok
Checking for system-defined composite types in user tables  ok
Checking for reg* data types in user tables                 ok
Checking for contrib/isn with bigint-passing mismatch       ok
Creating dump of global objects                             ok
Creating dump of database schemas
                                                            ok
Checking for presence of required libraries                 ok
Checking database user is the install user                  ok
Checking for prepared transactions                          ok
Checking for new cluster tablespace directories             ok

If pg_upgrade fails after this point, you must re-initdb the
new cluster before continuing.

Performing Upgrade
------------------
Analyzing all rows in the new cluster                       ok
Freezing all rows in the new cluster                        ok
Deleting files from new pg_xact                             ok
Copying old pg_xact to new server                           ok
Setting oldest XID for new cluster                          ok
Setting next transaction ID and epoch for new cluster       ok
Deleting files from new pg_multixact/offsets                ok
Copying old pg_multixact/offsets to new server              ok
Deleting files from new pg_multixact/members                ok
Copying old pg_multixact/members to new server              ok
Setting next multixact ID and offset for new cluster        ok
Resetting WAL archives                                      ok
Setting frozenxid and minmxid counters in new cluster       ok
Restoring global objects in the new cluster                 ok
Restoring database schemas in the new cluster
                                                            ok
Copying user relation files
                                                            ok
Setting next OID for new cluster                            ok
Sync data directory to disk                                 ok
Creating script to delete old cluster                       ok
Checking for extension updates                              ok

Upgrade Complete
----------------
Optimizer statistics are not transferred by pg_upgrade.
Once you start the new server, consider running:
    /usr/pgsql-15/bin/vacuumdb --all --analyze-in-stages


Running this script will delete the old cluster's data files:
    ./delete_old_cluster.sh

That’s better!

Start PostgreSQL 15

You may want to manually migrate over any configuration in postgresql.conf and pg_hba.conf from your old PostgreSQL 14 cluster to your new PostgreSQL 15 cluster, using diff or similar tools.

Once you’re ready, exit back to your root shell, then start your new PostgreSQL 15 database and set it to start automatically at boot:

[postgres@yourhost ~]$ exit
logout
[root@yourhost ~]# systemctl enable --now postgresql-15
Created symlink /etc/systemd/system/multi-user.target.wants/postgresql-15.service → /usr/lib/systemd/system/postgresql-15.service.

Let’s take the advice of pg_upgrade to analyze our freshly imported databases so the query planner has statistics it needs to plan wisely. We aren’t in any hurry to get the database back online, so we won’t bother analyzing in stages, and will just do it all at once:

[root@yourhost ~]# su - postgres
[postgres@yourhost ~]$ /usr/pgsql-15/bin/vacuumdb -a -Z
vacuumdb: vacuuming database "funtimes"
vacuumdb: vacuuming database "postgres"
vacuumdb: vacuuming database "template1"

Try it out

Now we can try our new installation, and check that some of those neat new PostgreSQL 15 features are really available:

[postgres@yourhost ~]$ psql
psql (15.0)
Type "help" for help.

postgres=# select version();
                                                 version
----------------------------------------------------------------------------------------------------------
 PostgreSQL 15.0 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 12.2.1 20220819 (Red Hat 12.2.1-2), 64-bit
(1 row)

postgres=# select regexp_count('the quick brown fox jumped over the lazy', ' ');
 regexp_count
--------------
            7
(1 row)
postgres=# \h merge
Command:     MERGE
Description: conditionally insert, update, or delete rows of a table
Syntax:
[ WITH with_query [, ...] ]
MERGE INTO target_table_name [ [ AS ] target_alias ]
USING data_source ON join_condition
when_clause [...]

where data_source is:

{ source_table_name | ( source_query ) } [ [ AS ] source_alias ]

and when_clause is:

{ WHEN MATCHED [ AND condition ] THEN { merge_update | merge_delete | DO NOTHING } |
  WHEN NOT MATCHED [ AND condition ] THEN { merge_insert | DO NOTHING } }

and merge_insert is:

INSERT [( column_name [, ...] )]
[ OVERRIDING { SYSTEM | USER } VALUE ]
{ VALUES ( { expression | DEFAULT } [, ...] ) | DEFAULT VALUES }

and merge_update is:

UPDATE SET { column_name = { expression | DEFAULT } |
             ( column_name [, ...] ) = ( { expression | DEFAULT } [, ...] ) } [, ...]

and merge_delete is:

DELETE

URL: https://www.postgresql.org/docs/15/sql-merge.html

Remove PostgreSQL 14

There is likely no urgency for you to do this, but later after you’re convinced PostgreSQL 15 is working well for you, you can remove the old PostgreSQL 14 packages and data.

First we’ll run this command without agreeing, to see what the package manager plans to do:

[root@yourhost ~]# rpm -qa postgresql14\* | xargs dnf erase
Dependencies resolved.
======================================================================
 Package              Architecture  Version         Repository   Size
======================================================================
Removing:
 postgresql14         x86_64        14.5-1PGDG.f36  @pgdg14     7.7 M
 postgresql14-devel   x86_64        14.5-1PGDG.f36  @pgdg14      19 M
 postgresql14-libs    x86_64        14.5-1PGDG.f36  @pgdg14     935 k
 postgresql14-server  x86_64        14.5-1PGDG.f36  @pgdg14      24 M

Transaction Summary
======================================================================
Remove  4 Packages

Freed space: 51 M
Is this ok [y/N]: Operation aborted.

If it proposes to remove only what you want, and not any other dependencies you want to keep, you can give dnf the go-ahead with the -y (yes) option:

[root@yourhost ~]# rpm -qa postgresql14\* | xargs dnf erase -y

And you can run the script that pg_upgrade left you to delete the old cluster’s data files:

[root@yourhost ~]# su - postgres
[postgres@yourhost ~]$ ./delete_old_cluster.sh

Enjoy!

Reference

Running PostgreSQL on Docker

2022-07-27T00:00:00+00:00

Introduction

PostgreSQL, or Postgres, is an open-source relational database. It is officially supported on all the major operating systems: Windows, Linux, BSD, MacOS, and others.

Besides running as an executable binary in an operating system, Postgres is able to run as a containerized application on Docker! In this article we are going to walk through the Postgres implementation on Docker.

Prerequisites

Docker or Docker Desktop. Please refer to my previous article for help with Docker installation.
Internet access is required to pull or download the Postgres container image from the Docker Hub.
A decent text editor, such as Vim or Notepad++, to create the configuration YAML files.

Get to know the official Postgres Image

Go to Docker Hub and search for “postgres”.

There are a lot of images for PostgreSQL at Docker Hub. If you don’t have any special requirements, it is best to select the official image. This is the image maintained by the Docker PostgreSQL Community.

The page that search result links to describes the Postgres image, how it was made and how to use it. From this page we know the image name and the required parameters. This is essential information for starting a Docker image, as we will see in the following steps.

Run the Postgres image as a basic Postgres container

The following command is the bare minimum for running Postgres on Docker:

docker run --name basic-postgres -e POSTGRES_PASSWORD=mysecretpassword -d postgres

Where:

--name basic-postgres sets the container name to basic-postgres,
-e POSTGRES_PASSWORD=mysecretpassword sets the password of the default user postgres,
-d runs the container in detached mode or in other words in the background, and
postgres uses the postgres image. By default it will get the image from https://hub.docker.com.

Execute docker ps to check on running Docker containers. We should see our basic-postgres container running. docker ps is like ps -ef on Linux/Unix, which lists all running applications.

Sample output:

Working with the Postgres container

Just as Postgres running natively on an operating system, Postgres on Docker comes with the psql front-end client for accessing the Postgres database. To access psql in the Postgres container execute the following command:

docker exec -it basic-postgres psql -U postgres

Where:

exec -it executes something interactive (-i) with a TTY (-t),
basic-postgres specifies the container, and
psql -U postgres is the psql command with its switch to specify the Postgres user.

Now we are able to execute any psql command.

Let’s try a few Postgres commands and import the famous “dvdrental” sample database to our Postgres installation.

List all available databases:

\l

Create a database named dvdrental:

create database dvdrental;

List all available databases. We should now see the created dvdrental database.

\l

Quit from psql:

\q

Download the dvdrental database backup from postgresqltutorial.com and after it succeeds, unzip it:

curl -O https://www.postgresqltutorial.com/wp-content/uploads/2019/05/dvdrental.zip
unzip dvdrental.zip

Execute the following command to import the data. It will restore the dvdrental.tar backup to our Postgres database.

docker exec -i basic-postgres pg_restore -U postgres -v -d dvdrental < dvdrental.tar

Where:

exec -i executes something interactive,
basic-postgres specifies which container,
pg_restore -U postgres -v -d dvdrental is the pg_restore command with its own arguments:
- -U postgres says to connect as the postgres user,
- -v enables verbose mode,
- -d dvdrental specifies the database to connect to, and
< dvdrental.tar says which file’s data the outside shell should pass into the container to pg_restore.

docker exec -it basic-postgres psql -U postgres -d dvdrental

Where:

exec -it executes something interactive with a terminal,
basic-postgres specifies which container, and
psql -U postgres -d dvdrental is the psql command with the postgres user and the dvdrental database specified.

List all tables by describing the tables in the dvdrental database:

\dt

List the first 10 actors from the actor table:

select * from actor limit 10;

Quit from psql:

\q

Gracefully stop the Docker container:

docker stop basic-postgres

If you don’t need it anymore you can delete the container:

docker rm basic-postgres

Sample output:

And later:

Run Postgres image as a “real world” Postgres container

The basic Postgres container is only good for learning or testing. It requires more features to be able to serve as a working database for a real world application. We will add 2 more features to make it useable:

Persistent storage: By default the container filesystem is ephemeral. What this means is whenever we restart a terminated or deleted container, it will get an all-new, fresh filesystem and all previous data will be wiped clean. This is not suitable for database systems. To be a working database, we need to add a persistent filesystem to the container.
Port forwarding from host to container: The container network is isolated, making it inaccessible from the outside world. A database is no use if it can’t be accessed. To make it accessible we need to forward a host operating system port to the container port.

Let’s start building a “real world” Postgres container. Firstly we need to create the persistent storage. In Docker this is known as a volume.

Execute the following command to create a volume named pg-data:

docker volume create pg-data

List all Docker volumes and ensure that pg-data was created:

docker volume ls | grep pg-data

Run a Postgres container with persistent storage and port forwarding:

docker run --name real-postgres \
-e POSTGRES_PASSWORD=mysecretpassword \
-v pg-data:/var/lib/postgresql/data \
-p 5432:5432 \
-d \
postgres

Where:

--name real-postgres sets the container name,
-e POSTGRES_PASSWORD=mysecretpassword sets the password of the default user postgres
-v pg-data:/var/lib/postgresql/data mounts the pg-data volume as the postgres data directory,
-p 5432:5432 forwards from port 5432 of host operating system to port 5432 of container,
-d runs the container in detached mode or, in other words, in the background, and
postgres use the postgres image. By default it will get the image from https://hub.docker.com

Execute docker ps to check on running containers on Docker. Take note that the real-postgres container has port forwarding information.

Now we are going to try to access the Postgres container with psql from the host operating system.

psql -h localhost -p 5432 -U postgres

Sample output:

Cleaning up the running container

Stop the container:

docker stop real-postgres

Delete the container:

docker rm real-postgres

Delete the volume:

docker volume rm pg-data

Managing Postgres container with Docker Compose

Managing a container with a long list of arguments to Docker is tedious and error prone. Instead of the Docker CLI command we could use Docker Compose, which is a tool for managing containers from a YAML manifest file.

Create the following file named docker-compose.yaml:

version: '3.1'
services:
  db:
    container_name: real-postgres-2
    image: postgres
    restart: always
    ports:
      - "5432:5432"
    environment:
      POSTGRES_PASSWORD: mysecretpassword
    volumes:
      - pg-data-2:/var/lib/postgresql/data
volumes:
  pg-data-2:
    external: false

To start the Postgres container with Docker Compose, execute the following command in the same location as docker-compose.yaml:

docker-compose up -d

Where -d runs the container in detached mode.

Execute docker ps to check on running Docker containers. Take note that real-postgres-2 was created by Docker Compose.

To stop Postgres container with Docker Compose, execute the following command in the same location as docker-compose.yaml:

docker-compose down

Sample output:

Conclusion

That’s all, folks. We have successfully deployed PostgreSQL on Docker.

Now we are able to reap the benefits of container technology for PostgreSQL, including portability, agility, and better management.

How to deploy a Django App with Aurora Serverless and AWS Copilot

2022-06-26T00:00:00+00:00

Photo by Виктор Куликов

AWS Copilot has the capability to provision an external database for its containerized work load. The database options are DynamoDB (NoSQL), Aurora Serverless (SQL), and S3 Buckets. For this blog we are going to provision and use Aurora Serverless with a containerized Django app. Aurora Serverless comes with 2 options for its engine: MySQL or PostgreSQL.

Watch Amazon’s 2-minute introduction video to get the basic idea of Aurora Serverless.

We are going to work with the same Django application from my last article on AWS Copilot.

In my last article, the Django application was deployed with SQLite as the database. The application’s data is stored in SQLite which resides internally inside the container. The problem with this setup is the data is not persistent. Whenever we redeploy the application, the container will get a new filesystem. Thus all old data will be removed automatically.

Now we are moving away the application’s data externally so that the life of the data does not depend on the container. We are going to put the data on the Aurora Serverless with PostgreSQL as the engine.

Django with SQLite as the internal database

Django with Aurora Serverless as the external database

The Prerequisites

Docker, AWS CLI, and AWS Copilot CLI are required. Please refer to my last article for how to install them.

The Django Project

Create a Django project by using a Python Docker Image. You can clone my Git project to get the Dockerfile, docker-compose.yaml and requirements.txt that I’m using:

$ git clone https://github.com/aburayyanjeffry/django-copilot.git django-aurora

Go to the django-aurora directory and execute docker-compose to create a Django project named “mydjango”.

$ cd django-aurora
$ docker-compose run web django-admin startproject mydjango

The Deployment with AWS Copilot

Execute the following command to create a AWS Copilot application with the name of “mydjango”, a load balancer container with the service name “django-web” which is made from the Dockerfile in the current directory.

$ copilot init \
-a mydjango \
-t "Load Balanced Web Service" -n django-web \
-d ./Dockerfile

Answer N to the following question. We want to defer the deployment until we have set up the database.

All right, you're all set for local development.

 Would you like to deploy a test environment? [? for help] (y/N) N

We need to create an environment for our application. Execute the following to create an environment named test for the “mydjango” application with the default configuration.

$ copilot env init \
--name test \
--profile default \
--app mydjango \
--default-config

Now we are going to generate a config for our Aurora Serverless database. Basically this is the CloudFormation template that will be used for Aurora Serverless.

Execute the following to generate the configuration for an Aurora cluster named “mydjango-db” that we will use for the “django-web” application. The Aurora cluster will be using the PostgreSQL engine and the database name will be “mydb”.

$ copilot storage init \
-n mydjango-db \
-t Aurora -w \
django-web \
--engine PostgreSQL \
--initial-db mydb

Take note of the injected environment variable name. This is where the database info and credentials are stored, and we will use this variable in later steps.

✔ Wrote CloudFormation template at copilot/django-web/addons/mydjango-db.yml

Recommended follow-up actions:
 - Update django-web's code to leverage the injected environment variable MYDJANGODB_SECRET.

Edit mydjango/settings.py to include the following. We will pass the injected environment variable we got previously to the function for getting the DBINFO variables.

from pathlib import Path
import os
import json
...
ALLOWED_HOSTS = ['*']
...
DBINFO = json.loads(os.environ.get('MYDJANGODB_SECRET', '{}'))
DATABASES = {
   'default': {
       'ENGINE': 'django.db.backends.postgresql',
       'HOST': DBINFO['host'],
       'PORT': DBINFO['port'],
       'NAME': DBINFO['dbname'],
       'USER': DBINFO['username'],
       'PASSWORD': DBINFO['password'],
   }
}

Deploy the application:

$ copilot deploy --name django-web

Open the terminal of the service:

$ copilot svc exec

Execute the following commands to migrate the initial database and to create a superuser account:

$ python manage.py migrate
$ python manage.py createsuperuser

Execute the following command to check on the environment variable. Take note of the MYDJANGODB_SECRET variable. It is the variable that holds the database information.

$ env | grep MYDJANGODB_SECRET

How to query Aurora Serverless

We can use the Query Editor at AWS Console for RDS to query Aurora Serverless.

Click the DB base on the DB identifier from the injected environment variable and click Modify.

Click the check box for Data API.

Select Apply Immediately.

Click Query Editor and fill in the Database information from the injected environment variable.

Now you may use the Query Editor to query the database. Execute the following query to list all tables in the database:

The End

That’s all, folks. We have deployed a containerized Django application and an Aurora Serverless with AWS Copilot. For further info on AWS Copilot visit its website.

Formatting SQL code with pgFormatter within Vim

2022-04-26T00:00:00+00:00

Photo by Garrett Skinner

Sometimes a little, seemingly simple tip can make a world of difference. I’ve got enough gray hair these days that it would be pretty easy for me to start thinking I’d seen an awful lot, yet quite frequently when I watch a colleague working in a meeting or a tmux session or somewhere, I learn some new and simple thing that makes my life demonstrably easier.

Luca Ferrari recently authored a post about using pgFormatter in Emacs; essentially the same thing works in Vim, my editor of choice, and it’s one of my favorite quick tips when working with complicated queries. I don’t especially want to get involved an editor war, and offer the following only in the spirit of friendly cooperation for the Vim users out there.

As Luca mentioned, pgFormatter is a convenient way to make SQL queries readable, automatically. It’s easy enough to feed it some SQL, and get a nice-looking result as output:

$ pg_format < create_outbreaks.sql
INSERT INTO outbreak                      
SELECT                              
    nextval('outbreak_id'::regclass),
    extract('year' FROM now())::text || '-' || nextval('outbreak_number_seq')::text, --number
    (                        
        SELECT  
            first_name
        FROM person TABLESAMPLE BERNOULLI (10)
LIMIT 1), -- name
    NOW() - interval '1 day' * random() * 100, (
        SELECT
            id
        FROM "user" TABLESAMPLE BERNOULLI (10)
...

In my perfect world I might quibble with some of its formatting decisions, such as the lack of indent on the LIMIT 1 line above. But in practice the results are good enough for my tastes that I haven’t bothered to investigate whether I can improve them. I just use it, and it’s good enough for me.

And because Vim lets me highlight a region, pipe it through an external program, and replace the region with that program’s output, it’s easy to use it simply by selecting a section of code and typing :!pg_format like this:

On Shapefiles and PostGIS

2022-04-02T00:00:00+00:00

Partial map of the voyage of the Endurance, from “South”, by Ernest Shackleton

The shapefile format is commonly used in geospatial vector data interchange, but as it’s managed by a commercial entity, Esri, and as GIS is a fairly specialized field, and perhaps because the format specification is only “mostly open”, these files can sometimes be confusing to the newcomer. Perhaps these notes can help clarify things.

Though the name “shapefile” would suggest a single file in filesystem parlance, a shapefile requires at least three different files, including filename extensions .shp, .shx, and .dbf, stored in the same directory, and the term “shapefile” often refers to that directory, or to an archive such as a zipfile or tarball containing that directory.

QGIS

QGIS is an open-source package to create, view, and process GIS data. One good first step with any shapefile, or indeed any GIS data, is often to take a look at it. Simply tell QGIS to open the shapefile directory. It may help to add other layers, such as one of the world map layers QGIS provides by default, to see the shapefile data in context.

GDAL

Though QGIS can convert between GIS formats itself, I prefer working in a command-line environment. The GDAL software suite aims to translate GIS data between many available formats, including shapefiles. I most commonly use its ogr2ogr command-line utility, along with the excellent accompanying manpage.

In short, a typical ogr2ogr command tells the utility where to find the input data and where to put the converted output, optionally with various reformatting and processing options. You’ll find some examples below.

PostGIS

Much of our (ok, my) GIS work has involved PostGIS, an extension to the PostgreSQL database for handling GIS data. It’s been convenient for me to process GIS data using the same language and tools I use to process other data. It uses GDAL’s libraries internally.

Examples

Import Shapefile data into PostGIS

The example below comes from a customer’s project we recently worked on. They provided us a set of several shapefiles, which I first arranged in a directory structure. This code imports each of them into a PostGIS database, in the shapefiles schema.

The other arguments to ogr2ogr specify the output format (“PostgreSQL”), the destination database name, and the directory which stores the shapefile. ogr2ogr expects the destination and source arguments in that order, as two positional arguments, so here the destination is PG:dbname=destdb, and the source file name comes from the the $i script variable.

for i in `find . -name "*shp"`; do
    j=$(basename $i)
    k=${j/.shp/}
    ogr2ogr -f PostgreSQL -nln shapefiles.${k} PG:dbname=destdb $i
done

Export PostGIS data as KML

This example creates a KML file from PostGIS query results. The arguments provide the query to use to fetch the data, the output format (“KML”), the output file name, and the source database. This will create a KML file containing a set of unstyled placemarks, with names from the property_code column, and geometry data from the outline_geom column in the properties table of our database.

In this project, outline_geom contained GIS “linestrings”, data types consisting of a series of lines, which ogr2ogr translated into KML polygons. Had outline_geom contained points, for instance, the KML result would also have been points. In other words, ogr2ogr automatically chooses the correct KML object type based on the GIS object type in the input data.

ogr2ogr -sql "select property_code, outline_geom from properties" -f KML outlines.kml PG:dbname=properties

Note that though the examples above use PostGIS, ogr2ogr can take shapefile input and produce KML output directly without the PostGIS intermediary. We used PostGIS in these cases for other purposes, such as to filter the output and limit the attributes stored in the KML result.

By default, ogr2ogr puts all the attributes from the shapefile into ExtendedData elements in the KML, but in our case we didn’t want those. We also didn’t want all the entries in the shapefile in our resulting KML. To skip the PostGIS step, we might do something like this:

ogr2ogr -f kml output.kml shapefile_directory/

What tools do you use for shapefile processing? Please let us know!

Using pgTAP to automate database testing

2022-03-16T00:00:00+00:00

Photo from PxHere

Recently I started learning to tune pianos. There are many techniques and variations, but the traditional method, and the one apparently most accepted by ardent piano tuning purists, involves tuning one note to a reference, tuning several other notes in relation to the first, and testing the results by listening closely to different combinations of notes.

The tuner adjusts each new note in relation to several previously tuned notes. Physics being what it is, no piano can play all its tones perfectly, and one of the tricks of it all is adjusting each note to minimize audible imperfections. The tuner achieves this with an exacting series of musical intervals tested against each other.

Databases need tests too

One of our customers needed to add security policies to their PostgreSQL database, to limit data visibility for certain new users. This can quickly become complicated and ticklish, ensuring that the rules work properly for the affected users while leaving other users unmolested.

This struck me as an excellent opportunity to create some unit tests, not that there’s any short supply of good opportunities to add unit tests! This is not just because it helps prove that these security policies really do work properly, but because (confession time) I recently did a similar project for a different database without the help of unit tests, and it wasn’t much fun.

So this seemed like a good time to use pgTAP, a set of database functions designed to allow writing unit tests within the database itself. They produce “Test Anything Protocol” (TAP) output, a simple protocol that displays unit test results in an easily understood report.

What to test?

A good first step in writing unit tests is deciding on something to test. In my case, I figured I should make sure row-level security policies were turned on for the tables I was interested in, which is available from the rowsecurity field in the pg_tables view:

select ok(rowsecurity, tablename || ' has row security enabled') from pg_tables
    where schemaname = 'public'
        and (
            tablename in (
                -- ... some hard coded table names
            ) or tablename like 'some_other_tables_%'
        );

The ok() function comes from pgTAP. It counts as one test each time it’s called; the test passes when the first argument is true, and fails when the argument is something else. The second argument is an optional comment describing what’s being tested. Following a pretty common TAP-related naming convention, I put this in a file called 00-test.sql in a directory under the root of my project, simply called t.

A more complicated set of tests could include several different files, where the numeric part of the name helps sort the tests in the desired run order, and the rest of the filename describes the subject of the tests within. But this will do just to get started. I can run it with pg_prove, included with the pgTAP package:

pg_prove -d mydatabase t/00-test.sql

Iteratively improving

This fails, for several reasons.

First, I haven’t yet installed the pgTAP extension in my database, with CREATE EXTENSION pgtap.

I also haven’t actually done anything in my test to run the code I’m testing. The actual code in this project consists of some database functions, which we need to run to create the database security policies, and I haven’t run any of them yet.

And finally, pgTAP requires me to “plan” my tests first, or in other words, I need to inform pgTAP how many tests I plan to run, before I run them. It’s also nice to call finish() so pgTAP can clean up after itself.

I installed the pgTAP extension in my database, and modified the test as follows:

begin;

\i create_policy.sql

select plan(1);  -- plan for a single test

select ok(rowsecurity, tablename || ' has row security enabled') from pg_tables
    where schemaname = 'public'
        and (
            tablename in (
                -- ... some hard coded table names
            ) or tablename like 'some_other_tables_%'
        );

select finish();

rollback;

This wraps my test in a transaction, so that I can roll everything back to leave the database essentially as I found it. It also calls the actual code I’m testing, in create_policy.sql, and plans one test. And it gives me this new failure:

t/m.sql .. All 1 subtests passed

Test Summary Report
-------------------
t/m.sql (Wstat: 0 Tests: 226 Failed: 225)
  Failed tests:  2-226
Parse errors: Bad plan.  You planned 1 tests but ran 226.
Files=1, Tests=226,  1 wallclock secs ( 0.04 usr  0.00 sys +  0.03 cusr  0.01 csys =  0.08 CPU)
Result: FAIL

The problem here is that each call to ok() counts as one test, and my test apparently found 226 tables to check for row-level security. I can improve the planning like this:

select plan(count(*)::integer)
    from pg_tables where schemaname = 'public'
        and (
            tablename in (
                -- ... some hard coded table names
            ) or tablename like 'some_other_tables_%'
        );

count() returns a bigint, and plan() expects integer, so this requires a typecast, but is otherwise pretty simple. And now my tests pass:

josh@here:~dw$ pg_prove -d nedss t/00-test.sql
t/00-test.sql .. ok
All tests successful.
Files=1, Tests=226,  1 wallclock secs ( 0.03 usr  0.01 sys +  0.03 cusr  0.01 csys =  0.08 CPU)
Result: PASS

Looking back and ahead

Suffice it to say that pgTAP includes many functions similar to ok(), to test various aspects of the database, its structure, and its behavior, and I’d recommend interested users review the documentation for more details. I intended this post only as an introduction.

In its completed state, my test suite comprised several tests ensuring various required preliminaries were in place, a few tests like the one above that check for necessary table-specific settings, others that ensure the affected roles were created, and finally some which create some sample data and use SET ROLE to test the data visibility directly for roles with various policies applied.

And to be honest, I was surprised at the sense of security that came over me with this completed test suite. As I mentioned, I’d done similar work previously, and knew that although I was confident in the code when it was written, that confidence came only through fairly extensive manual testing. I know very well the struggles of bit rot, and I knew it would be at least as hard to repeat that testing regimen by hand sometime down the road after a year or two.

I also recognized that if I ever needed to set up similar policies again, I could use these tests themselves as a reference, because they show exactly how to run the code in question. Though of course I included that information in the project’s associated README file as well … right?

Let us know if you’ve used pgTAP, and what effect it has had on your database development.

Database Design: Using Documents

2022-03-09T00:00:00+00:00

Using documents in relational databases is increasingly popular. This technique can be practical and efficient when used in fitting circumstances.

Example

Let’s start with an example. Imagine we are scraping web sites for external URLs and store them in a table. We’ll have the web sites table to store the scrape timestamp and another table to store all of the references.

CREATE TABLE web_sites (
    web_site_domain text NOT NULL,
    last_scraped_at timestamptz,

    PRIMARY KEY (web_site_domain)
);

CREATE TABLE refs (
    web_site_domain text NOT NULL,
    ref_location text NOT NULL,
    link_url text NOT NULL,

    PRIMARY KEY (web_site_domain, ref_location, link_url),
    FOREIGN KEY (web_site_domain) REFERENCES web_sites
);

We do not need to bother adding an id to the web_sites table, because we assume there won’t be too many of them. The domain is small and more practical to use as an identifier. If you are curious about advantages of using natural keys, see my previous article.

Normalized Tables

There may be many thousands of unique URLs for a single web site and other web sites may refer to the same URLs. To try to minimize the storage, we can keep the locations and the URLs in separate tables, give them integer identifiers, and have another table for the many-to-many relations.

CREATE TABLE locations (
    location_id bigint NOT NULL GENERATED ALWAYS AS IDENTITY,
    web_site_domain text NOT NULL,
    ref_location text NOT NULL,

    PRIMARY KEY (location_id),
    UNIQUE (web_site_domain, ref_location),
    FOREIGN KEY (web_site_domain) REFERENCES web_sites
);

CREATE TABLE links (
    link_id bigint NOT NULL GENERATED ALWAYS AS IDENTITY,
    link_url text NOT NULL,

    PRIMARY KEY (link_id),
    UNIQUE (link_url)
);

CREATE TABLE locations_links (
    location_id bigint NOT NULL,
    link_id bigint NOT NULL,

    PRIMARY KEY (location_id, link_id),
    FOREIGN KEY (location_id) REFERENCES locations,
    FOREIGN KEY (link_id) REFERENCES links
);

The idea in here is to keep our biggest table narrow. It’d pay off to refer to the locations and links just with integer identifiers when we’ll have very many of them.

Table Sizes

We’ll have many web sites and many URLs, so our lookup tables would be big and the relation table would be even bigger. That is going to be a major problem because of many reasons, one of them being space efficiency. Narrow tables with many rows are not very space efficient especially on Postgres where there’s 24 bytes per-row overhead. To demonstrate it being significant, let’s add one row to each of the tables to see the sizes compared with the overhead.

INSERT INTO web_sites (web_site_domain)
VALUES ('example.com');

INSERT INTO refs (web_site_domain, ref_location, link_url)
VALUES ('example.com', '/source', 'http://example.net/target.html');

INSERT INTO locations (web_site_domain, ref_location)
VALUES ('example.com', '/source');

INSERT INTO links (link_url)
VALUES ('http://example.net/example.html');

INSERT INTO locations_links
VALUES (1, 1);

SELECT table_name, tuple_len, (100.0 * 24 / tuple_len)::int AS overhead_perc
FROM information_schema.tables, LATERAL pgstattuple(table_name)
WHERE table_schema = 'public';

   table_name    | tuple_len | overhead_perc
-----------------+-----------+---------------
 web_sites       |        36 |            67
 refs            |        75 |            32
 locations       |        52 |            46
 links           |        64 |            38
 locations_links |        40 |            60

With just one row and such short domains and URLs, we wouldn’t be gaining anything by having the links on a lookup table. In fact, it’s a lot less efficient. Although, the situation may change with many rows.

Composite Types

We can combine multiple fields in a column to avoid the overhead of many rows in narrow tables. Postgres has good support for composite types and arrays over them. They are useful if you need strict data type checking.

CREATE TYPE web_site_page AS (
    ref_location text,
    link_urls text[]
);

ALTER TABLE web_sites ADD COLUMN pages web_site_page[];

UPDATE web_sites
SET pages = ARRAY[ROW(ref_location, ARRAY[link_url])::web_site_page]
FROM refs;

SELECT pg_column_size(pages) FROM web_sites;

 pg_column_size
----------------
            117

As you see, the new column is not exactly small because composite types and arrays come with their own overheads. Still it could be much better when a web site has many references.

JSON Document Column

The composite types are not very easy to work with. We can store the pages in another format. JSON is the most popular one nowadays. There are multiple options to store JSON in a column in Postgres. Let’s start with the text-based data type.

ALTER TABLE web_sites ADD COLUMN pages_json json;

UPDATE web_sites SET pages_json = to_json(pages);

SELECT pg_column_size(pages_json) FROM web_sites;

 pg_column_size
----------------
             76

What is perhaps also surprising is the JSON being smaller than the array over composite type. This is because Postgres data types are a bit wasteful because of alignment and storing oids of the enclosed data types repeatedly.

Binary JSON

Another JSON data type, jsonb is available in Postgres. It is a binary structure that allows faster access to the values inside the document.

ALTER TABLE web_sites ADD COLUMN pages_jsonb jsonb;

UPDATE web_sites SET pages_jsonb = to_jsonb(pages);

SELECT pg_column_size(pages_jsonb) FROM web_sites;

 pg_column_size
----------------
             98

As you see the text-based JSON is still smaller. This is due to the binary offsets stored in the binary structure for faster access to the fields. The text-based JSON also compresses better.

Size Comparison

Very simple single row data shows the text-based json as the smallest, binary jsonb a bit larger, and the composite type to be the largest. However, the differences would be a lot smaller with more realistic sample data with many items inside the documents. I generated some data and gathered these results (the sizes excluding the indexes).

    model         |   size
------------------|---------
single table      |  436 MiB
normalized tables |  387 MiB
composite type    |  324 MiB
json              |  318 MiB
jsonb             |  320 MiB

You may expect the normalized tables to be the smallest as otherwise the metadata is stored repeatedly in the columns. Though, the results reveal it’s the other way around. This is because of the per-row overhead in Postgres. It becomes noticeable in narrow tables as in this example.

It’s also useful to notice that the most of the size is occupied by the TOAST tables when the data is stored in a single column. TOAST is a mechanism in Postgres that kicks in when a column is large enough which would often be the case when you design your tables this way. In this case, the table excluding the TOASTed part is pretty small which would help any query that doesn’t touch the large column.

Usability

What matters more than size is the usability of having it all in a single column. It is really practical to send and receive this column as a whole all the way from the database to the frontend. Compare this with dealing with lots of small objects used with the normalized tables.

It’s also possible in Postgres to index the document columns to allow efficient searches in them. There are many options to do so with advantages and disadvantages over each other. Though this post is getting long. I’ll leave indexing to another one.

String Processing in SQL, the Hard Way

2022-02-28T00:00:00+00:00

Photo by Kristy Lee

When I was a kid, my family visited a prehistoric cliff-dweller settlement, reconstructed and preserved as a museum exhibit. The homes were built in caves in high sandstone cliffs, and though I followed a well-marked path to visit them, the original inhabitants climbed in and out via holes cut into the rock face. Apparently to deter unwanted visitors, the builders carefully spaced the foot holds so that climbers needed to start their ascent on the correct foot. Someone who didn’t know the secret of the path, or who carelessly forgot, might get halfway up the cliff and discover he or she couldn’t stretch to the next foot hold and had to turn back.

The other day I found myself confronted with several comma-separated lists of strings, and a problem: I needed the longest possible substring from each, starting from the beginning, ending in a comma, and shorter than a certain length, and I needed to do it in the database. “Easy peasy,” you may say to yourself, if you have a little SQL programming experience and a penchant for cheesy clichés, and you’d be right except for one thing: I let my mind wander a little too far down into the uncharted mists of imagination and like the cliff dweller, I got off on the wrong foot. What followed was a thankfully brief exploration of the Hard Way to Do It.

The unsettling method on which I settled had me dividing each list into its component parts, reconstructing candidate results element by element, and finding the longest one less than the maximum overall length. Fortunately the problem didn’t allow me to reorder the elements in my input lists, or I’d probably still be working on this query. Here’s the monstrosity I eventually built:

select
    length(c), c
from (
    select
        concat_ws(', ', variadic array_agg(f) over (rows between unbounded preceding and current row)) c
    from (
        select regexp_split_to_table('alpha, bravo, charlie, delta, echo, fox trot', ',\s+') as f
    ) f
) c
where length(c) < 30
order by length(c) desc nulls last
limit 1

The innermost subquery here, based on regexp_split_to_table, divides the input string into an ordered list of its component elements:

select regexp_split_to_table('alpha, bravo, charlie, delta, echo, fox trot', ',\s+');
 regexp_split_to_table
-----------------------
 alpha
 bravo
 charlie
 delta
 echo
 fox trot
(6 rows)

I’m never one to bypass a good opportunity to shoehorn a window function into my SQL, and here we get to use the frame clause. Recall that for each row of a query, a window function processes a selection of rows, called a “partition”, and in fact processes only a “frame”, a subset of that partition. Window frames are fairly complicated subjects; I’ll try to present here a brief summary of the bits relevant to this query, and recommend the PostgreSQL documentation for further details.

Our particular window frame clause, rows between unbounded preceding and current row, means that the window frame for a given row should include all the rows from the first row of the partition through the current row. Given that we have no order by clause in our window definition, terms like “first row” refer to the first row in the order the rows were provided to the WindowAgg executor node. Whether it’s safe to trust that order to remain stable without a specific order by clause is an awfully good question I didn’t try to answer but for my purposes I needed to preserve the order of elements as returned by regexp_split_to_table. The default window frame clause is range unbounded preceding, which the documentation notes is a shortened form of range between unbounded preceding and current row, and without a specific order by clause, this means the window frame includes all rows in the partition. As a fun exercise, compare these two results, which differ only in the presence of an order by clause:

select array_agg(f) over () c
    from (
        select regexp_split_to_table('alpha, bravo, charlie, delta, echo, fox trot', ',\s+') as word
    ) f
;
                              c
-------------------------------------------------------------
 {(alpha),(bravo),(charlie),(delta),(echo),"(\"fox trot\")"}
 {(alpha),(bravo),(charlie),(delta),(echo),"(\"fox trot\")"}
 {(alpha),(bravo),(charlie),(delta),(echo),"(\"fox trot\")"}
 {(alpha),(bravo),(charlie),(delta),(echo),"(\"fox trot\")"}
 {(alpha),(bravo),(charlie),(delta),(echo),"(\"fox trot\")"}
 {(alpha),(bravo),(charlie),(delta),(echo),"(\"fox trot\")"}
(6 rows)

select array_agg(f) over (order by word) c
    from (
        select regexp_split_to_table('alpha, bravo, charlie, delta, echo, fox trot', ',\s+') as word
    ) f
;
                              c
-------------------------------------------------------------
 {(alpha)}
 {(alpha),(bravo)}
 {(alpha),(bravo),(charlie)}
 {(alpha),(bravo),(charlie),(delta)}
 {(alpha),(bravo),(charlie),(delta),(echo)}
 {(alpha),(bravo),(charlie),(delta),(echo),"(\"fox trot\")"}
(6 rows)

Anyway, that’s a tangent, but I found it a helpful one, as the default window frame is a well-known trap for young players, to which I’ve fallen prey in the past at least once. Back to my over-complicated query.

I’m aiming to produce a series of strings, each containing one more element than the last one. Now that I have a set of arrays with the elements I want inside, I turned to the concat_ws function to turn them into strings. concat_ws concatenates each of its arguments with a user-specified separator string, and the variadic keyword lets me pass an array of values and have them treated as individual arguments:

select
    concat_ws(', ', variadic array_agg(f) over (rows between unbounded preceding and current row)) c
from (
    select regexp_split_to_table('alpha, bravo, charlie, delta, echo, fox trot', ',\s+') as f
) f
;
                      c
----------------------------------------------
 alpha
 alpha, bravo
 alpha, bravo, charlie
 alpha, bravo, charlie, delta
 alpha, bravo, charlie, delta, echo
 alpha, bravo, charlie, delta, echo, fox trot

Now we’re getting close to the answer. My final steps are to filter out all resultant strings longer than the acceptable length and pick the longest one:

select
    length(c), c
from (
    select
        concat_ws(', ', variadic array_agg(f) over (rows between unbounded preceding and current row)) c
    from (
        select regexp_split_to_table('alpha, bravo, charlie, delta, echo, fox trot', ',\s+') as f
    ) f
) c
where length(c) < 30
order by length(c) desc nulls last
limit 1
;
 length |              c
--------+------------------------------
     28 | alpha, bravo, charlie, delta
(1 row)

Success! It only took about a hundred more steps than were really necessary. In my defense, it wasn’t too long after I’d finished composing this query that I realized I was making it much harder than I should have.

I posted it to our internal company chat, and within a minute or two my colleague Kürşat had answered with his own version:

SELECT regexp_matches(substring('alpha, bravo, charlie, delta, echo, fox trot' from 1 for 30), '(.*),');
          regexp_matches
----------------------------------
 {"alpha, bravo, charlie, delta"}
(1 row)

This simply finds a substring of acceptable length from the input text, and removes everything from the final comma onward. Not only did this take about 30 seconds to compose, compared to the half hour or so I’d spent on my first version, but on my laptop it completes in about half the time.

That fact notwithstanding, it’s often true that we learn rather a lot from our mistakes — and others’ mistakes — and I hope that’s true for you in this case.

Migrating a Node.js app database from MongoDB to PostgreSQL

2022-02-25T00:00:00+00:00

Recently I worked on a web app designed for tracking, editing, and reporting on archaeological survey data. When I joined the project, it used React on the front-end, Node.js on the back-end, GraphQL as the API interface, and MongoDB as the database. For the most part, this architecture and choice of technology worked well, and I was quickly able to make meaningful contributions to the app both in the user interface and in the backend. However, after a while, things started to get difficult and we began to consider why we were using MongoDB, and what might be better.

MongoDB vs. PostgreSQL

First, why was MongoDB difficult to use in our case? The biggest difficulty was using their query language for certain aspects of our database. Part of the database involved representing artifacts found at a site using a specific type hierarchy defined by the state government. Representing that hierarchy within Mongo was difficult, and querying it even more so. It was hard to find how to query a recursive structure using the MongoDB query language.

As I worked on this, scouring the documentation and Q&A websites to find ways to query this data efficiently, I began to realize that it was a problem that could be easily solved with PostgreSQL’s recursive queries.

Now, a single recursive table isn’t really a good reason to completely switch the database used by an application, but it got us thinking: Why were we using MongoDB? What benefits was it offering?

I found a relevant article published on the official MongoDB website entitled Comparing MongoDB vs PostgreSQL. That article is generally very positive in its portrayal of PostgreSQL, so I was interested to hear what they had to say for themselves. Here are some of the main points in favor of MongoDB that the article discusses:

MongoDB is horizontally scalable by default: This seems cool, but we were working on a new app that was not expected to need to scale much, and a single PostgreSQL instance has been shown to scale more than enough for our anticipated needs.
MongoDB supports fancy new serverless and mobile paradigms: Again, cool, but this was a fairly small-scale project using a pretty normal Linux/Node.js stack. No need for serverless support.
MongoDB’s document structure makes it easy for developers to change the structure of data on the fly: While this can certainly be nice, it can also bring its own issues. If data is expected to fit a certain structure and one developer changes that structure, this will likely cause problems for other developers whether or not there is a schema. Additionally, our project had a small team of 1-3 developers working on it and changing the schema as needed just wasn’t a problem.

The MongoDB vs. PostgreSQL article also mentions “resilience” as a strength of MongoDB, citing its ability to easily be replicated and broken into shards across datacenters and regions. However, research regarding the historical reliability of PostgreSQL and MongoDB showed that MongoDB’s fairly short history is littered with data loss problems. Anecdotally, a Google search for “MongoDB data loss” returns 1.26 million results (an average of 0.14 million results per year since its initial release) while “PostgreSQL data loss” returns 1.57 million (0.06 million per year since release). More specifically, there are fascinating in-depth reviews of MongoDB’s reliability, such as those conducted by Jepsen as recently as 2020 (see #1, #2, #3, #4, #5).

PostgreSQL hasn’t been perfect in the 25 years since its initial release, and MongoDB has improved a lot in recent years, but it’s clear that Postgres is the winner when it comes to data reliability.

After reviewing all of these pros and cons of MongoDB, it became clear that it wasn’t really offering anything that we needed, and our data, which was strictly organized and relational by nature, would fit much better in a relational database. We decided that it would be better to switch to PostgreSQL.

Mongoose and Sequelize

Our database was only being used within the backend GraphQL server, using Mongoose, an object modeling library for Node.js. Mongoose uses a schema/model system, where every MongoDB collection contains documents fitting a schema defined within Mongoose model definitions, for example:

import mongoose from "mongoose";
const { Schema } = mongoose;

const blogPostSchema = new Schema({
  title: String, // String is shorthand for {type: String}
  author: String,
  body: String,
  comments: [{ body: String, date: Date }],
  date: { type: Date, default: Date.now },
  hidden: Boolean,
  meta: {
    votes: Number,
    favs: Number,
  },
});

const BlogPost = mongoose.model("BlogPost", blogPostSchema);

Once a model is defined it can easily be used to query and create documents:

const new_post = new BlogPost({
  title: "On the international epidemic of counterfeit at-home brogal treatments",
  author: "Ronald McClure",
  body: "...",
  // ...
});
await new_post.save();

const posts = await BlogPost.find({ author: "Ronald McClure" });

Mongoose provides a lot of functions for querying and editing documents in a collection on the Model class, as well as a rich Query class to make manual queries, among other things. Their documentation provides helpful guides and a great API reference.

Because we generally liked the Mongoose interface and didn’t want to have to change our existing code too drastically, we searched for a similar library that could work with PostgreSQL and settled upon Sequelize.

Sequelize has a very similar interface to Mongoose, where you start by defining tables (instead of “collections”) as models in JavaScript:

const { Sequelize, DataTypes } = require('sequelize');
const sequelize = new Sequelize('postgres://user:pass@example.com:5432/dbname');

const BlogPost = sequelize.define('BlogPost', {
  // Model attributes are defined here
  title: DataTypes.STRING,
  author: DataTypes.STRING,
  body: DataTypes.STRING,
  // This will need to be modeled differently
  // comments: [{ body: DataTypes.STRING, date: Date }],
  date: {
    type: DataTypes.DATETIME,
    defaultValue: DataTypes.NOW
  },
  hidden: DataTypes.BOOLEAN,
  votes: DataTypes.INTEGER,
  favs: DataTypes.INTEGER
}, {
  // Other model options go here
});

// Object creation is similar
const new_post = BlogPost.build({
  title: "On the international epidemic of counterfeit at-home brogal treatments",
  author: "Ronald McClure",
  body: "...",
  // ...
});
await new_post.save();

Note that this model, while mimicking the Mongoose schema as much as possible, still requires some changes to fit the relational model. First, the comments field, which was an array of documents in the Mongoose object will need to be represented in another table, which can then be linked via foreign key.

While this seems annoying in some ways — creating a new table will require creating another Sequelize model, setting up the relations correctly, etc. — it makes a lot of sense for this example (a blog) and most other situations. We don’t just want blog posts to have associated comments, but we likely also want those comments to be tied to an author and to have their own IDs, which would allow lookup by author (for example so that on a user’s page they can see all comments they’ve made on all posts) and by individual comment (so it’s easier to make links directly to a single comment).

So, rather than having comments be a subfield of a blog post, we want them to be defined by some kind of relation. In MongoDB this can be done in a few ways, most obviously by creating a comments collection making the comments list contain ObjectIDs:

const mongoose = require("mongoose");
const { Schema } = mongoose;

const Comment = mongoose.model(
  "Comment",
  new Schema({
    body: String,
    date: Date,
  })
);

const blogPostSchema = new Schema({
  // ...
  comments: [{ type: Schema.Types.ObjectId, ref: "Comment" }],
  // ...
});

const BlogPost = mongoose.model("BlogPost", blogPostSchema);

Mongoose also has a way to automatically fill in those referenced objects with the real thing as needed, using their populate functionality:

BlogPost.findOne({ author: "Ronald McClure" })
  .populate("comments")
  .exec(function (err, post) {
    if (err) return handleError(err);
    post.comments.forEach((comment) => console.log(comment));
  });

With the call to .populate('comments'), the comments array is filled with objects instead of just the object IDs. Sequelize has similar functionality, but uses the concept of associations which model the relational structure of SQL more closely:

const { Sequelize, DataTypes } = require('sequelize');
const sequelize = new Sequelize('postgres://user:pass@example.com:5432/dbname');

const BlogPost = sequelize.define('BlogPost', {
  // ... fields here
}, {});

const Comment = sequelize.define('Comment', {
  body: DataTypes.TEXT,
  date: DataTypes.DATE,
}, {});

Comment.belongsTo(BlogPost;
BlogPost.hasMany(Comment);

These belongsTo and hasMany associations use foreign key/primary key references on the models, and Sequelize also provides a similar interface for filling in the referenced data:

const post = await BlogPost.findOne({
  where: {
    author: "Ronald McClure",
  },
  include: Comment,
});

Both libraries also have ways to lazy-load related objects, only fetching them as necessary. For MongoDB, this involves writing your own queries on referenced fields or using MongoDB’s $lookup operator, while Sequelize allows custom queries using the foreign key fields, as well as a set of nice methods to load associations on demand.

Modeling and Migrating Data

Having weighed for this situation the pros and cons of PostgreSQL and MongoDB as well as Mongoose and Sequelize, the question then comes: How do we actually move the data from one system to another? Unfortunately, as far as I know there is no one tool that can be used to migrate data between MongoDB and PostgreSQL. However, using a combination of mongoexport and some basic SQL files, it was not too difficult to migrate data.

The first step and perhaps one of the more difficult ones is to create an SQL schema that works for your data. If your MongoDB data is already in a normalized and relational structure, it might not be too hard, but if you have a lot of subdocuments and arrays that need to be expanded into rows, things might be more difficult. My recommendation for how to approach this is:

Model your data in SQL. If you’re doing a migration from MongoDB, you likely have an idea of what the application needs are and how data should be organized.
Test your model with some dummy data. This will probably be a bit of a back-and-forth iterative process as you iron out issues with relations, decide what kind of constraints will be necessary, etc.
Use mongoexport to export your data as appropriate. For many collections, it may be easy to simply export it as CSV and use Postgres’s COPY command to import it into the appropriate table. However, many things won’t be so simple and might require writing your own script that reads the exported CSV or JSON and writes it to the database, altering data as needed. This might be necessary for example to change MongoDB’s object IDs to match a datatype in PostgreSQL, such as an identity column (defined in CREATE TABLE), serial, or UUID.
Use your set of import commands and scripts to import all of your data into a test database.
Modify your application to use Sequelize instead of Mongoose.
Find and update all cases in your code where a Mongoose model or instance was used and update it to use Sequelize instead.

Caveats and Gotchas

One of the biggest problems we came across after our migration was the differences in query syntax between Mongoose and Sequelize. Compare the following queries:

// Mongoose
post = await BlogPost.findOne({ author: 'Ronald McClure' });

// Sequelize
post = await BlogPost.findOne({ author: 'Ronald McClure' });

They look the same, they both run with no errors, and they both return the one post we created. But say we change which author we are querying for:

// Mongoose
post = await BlogPost.findOne({ author: 'Darlene Roberts' });

// Sequelize
post = await BlogPost.findOne({ author: 'Darlene Roberts' });

Mongoose’s query returns nothing, as we expect, but Sequelize… returns the same post as the “Ronald McClure” query returned? Astute readers might notice what the issue is from earlier examples: Sequelize’s find methods take an options object which doesn’t use its fields as WHERE parameters by default. The correct way to write this query with Sequelize is:

post = await BlogPost.findOne({ where: { author: "Darlene Roberts" } });

It’s a bit of a silly mistake, but one that caused pain several times in the process of migration. Some queries seemed to work just fine, but just wouldn’t return the right results!

How to use regular expression group quantifiers in PostgreSQL

2022-01-27T00:00:00+00:00

I recently encountered a situation where it was necessary to extract address content from text in HL7 V2 format from a PostgreSQL table’s column. The following example is representative:

||121212^^^2^ID 1|676767||SELVA^KUMAR^^^^|19480203|M||B||123456 SAMPLE ROAD^^New York City^NY^12345^USA^H^^New York||123456-7890|||M|NON|4000|

In order to manipulate our example, the address section needs to be extracted from the HL7 V2 message PID segment for patient demographic information. The segments have delimiters for fields (|), components (^), subcomponents (&) and repetition (~).

Our example has only fields and components delimited by pipe (|) and caret (^). The address contains nine components delimited by ^.

I hoped to do this by applying a regular expression (regex) because the address is in a standard format that regex can match with alphanumeric and caret repetition.

Here is my journey figuring out how to match the data I wanted.

Regex pattern in grep

As a test, I got this regex working with the grep command, which successfully extracts the address section from the content:

$ content='||121212^^^2^ID 1|676767||SELVA^KUMAR^^^^|19480203|M||B||123456 SAMPLE ROAD^^New York City^NY^12345^USA^H^^New York||123456-7890|||M|NON|4000|'
$ echo "$content" | grep -Eo "([A-Za-z0-9 #'.,/-]*\^){8}[A-Za-z0-9 ]*"
123456 SAMPLE ROAD^^New York City^NY^12345^USA^H^^New York

PostgreSQL regex attempt

The PostgreSQL regexp_matches function supports extraction by pattern-matching data from the content. But when I used the same regex pattern with the regexp_matches function, instead of all eight groups, only the eighth value was returned:

=# SELECT regexp_matches('||121212^^^2^ID 1|676767||SELVA^KUMAR^^^^|19480203|M||B||123456 SAMPLE ROAD^^New York City^NY^12345^USA^H^^New York||123456-7890|||M|NON|4000|', '([A-Za-z0-9 #''.,/-]*\^){8}[A-Za-z0-9 ]*', 'g');
 regexp_matches 
----------------
 {^}
(1 row)

More generally, our query is returning the Nth matching group instead of returning all matching groups until the Nth regex group. So if we try to fetch the text matching with 3 groups, the quantifier will return the third field of all match sections instead of the third group itself.

=# SELECT regexp_matches('||121212^^^2^ID 1|676767||SELVA^KUMAR^^^^|19480203|M||B||123456 SAMPLE ROAD^^New York City^NY^12345^USA^H^^New York||123456-7890|||M|NON|4000|', '([A-Za-z0-9 #''.,/-]*\^){3}[A-Za-z0-9 ]*', 'g');
   regexp_matches   
--------------------
 {^}
 {^}
 {"New York City^"}
 {USA^}
(4 rows)

PL/Perl function

Since that doesn’t satisfy our requirements, I tried using a Perl regex through my own PL/Perl function, and got the expected answer:

=# CREATE OR REPLACE FUNCTION perl_regexp_matches (IN str text, IN pattern text) RETURNS text AS $$
    my ($input, $pattern) = @_;
    $output = [$input =~ m/($pattern)/];
    return $output->[0]
$$ LANGUAGE plperl;

=# SELECT perl_regexp_matches('||121212^^^2^ID 1|676767||SELVA^KUMAR^^^^|19480203|M||B||123456 SAMPLE ROAD^^New York City^NY^12345^USA^H^^New York||123456-7890|||M|NON|4000|', '([A-Za-z0-9 #''.,/-]*\^){8}[A-Za-z0-9 ]*');
                    perl_regexp_matches
------------------------------------------------------------
 123456 SAMPLE ROAD^^New York City^NY^12345^USA^H^^New York
(1 row)

But I researched further for a simple solution to achieve the result without using a custom function and the PL/Perl extension.

PostgreSQL regex solution

In Postgres regex syntax, parentheses () create a numbered capture group which leads to returning the contained matching results.

To get the entire matching data, the regex should have a question mark and a colon (?:) added at the beginning of the regex pattern to create a non-capturing group. Then because no group is capturing, instead the complete match is returned:

=> SELECT regexp_matches('||121212^^^2^ID 1|676767||SELVA^KUMAR^^^^|19480203|M||B||123456 SAMPLE ROAD^^New York City^NY^12345^USA^H^^New York||123456-7890|||M|NON|4000|', '(?:[A-Za-z0-9 #''.,/-]*\^){8}[A-Za-z0-9 ]*');
                         regexp_matches                         
----------------------------------------------------------------
 {"123456 SAMPLE ROAD^^New York City^NY^12345^USA^H^^New York"}
(1 row)

That turns out to be what was happening with my PL/Perl function where m/($pattern)/ captures the entire match, and what grep was doing because of its option -o or --only-matching, which prints the matching part of the lines rather than its default of printing the entire line.

And we can also use the Postgres function substring to return the bare text itself instead of arrays as regexp_matches does:

=# SELECT substring('||121212^^^2^ID 1|676767||SELVA^KUMAR^^^^|19480203|M||B||123456 SAMPLE ROAD^^New York City^NY^12345^USA^H^^New York||123456-7890|||M|NON|4000|' FROM '(?:[A-Za-z0-9 #''.,/-]*\^){8}[A-Za-z0-9 ]*');
                         substring
------------------------------------------------------------
 123456 SAMPLE ROAD^^New York City^NY^12345^USA^H^^New York
(1 row)

Reference

Kubernetes 101: Deploying a web application and database

2022-01-08T00:00:00+00:00

The DevOps world seems to have been taken over by Kubernetes during the past few years. And rightfully so, I believe, as it is a great piece of software that promises and delivers when it comes to managing deployments of complex systems.

Kubernetes is hard though. But it’s all good, I’m not a DevOps engineer. As a software developer, I shouldn’t care about any of that. Or should I? Well… Yes. I know that very well after being thrown head first into a project that heavily involves Kubernetes, without knowing the first thing about it.

Even if I wasn’t in the role of a DevOps engineer, as a software developer, I had to work with it in order to set up dev environments, troubleshoot system issues, and make sound design and architectural decisions.

After a healthy amount of struggle, I eventually gained some understanding on the subject. In this blog post I’ll share what I learned. My hope is to put out there the things I wish I knew when I first encountered and had to work with Kubernetes.

So, I’m going to introduce the basic concepts and building blocks of Kubernetes. Then, I’m going to walk you through the process of containerizing a sample application, developing all the Kubernetes configuration files necessary for deploying it into a Kubernetes cluster, and actually deploying it into a local development cluster. We will end up with an application and its associated database running completely on and being managed by Kubernetes.

In short: If you know nothing about Kubernetes, and are interested in learning, read on. This post is for you.

What is Kubernetes?

Simply put, Kubernetes (or K8s) is software for managing computer clusters. That is, groups of computers that are working together in order to process some workload or offer a service. Kubernetes does this by leveraging application containers. Kubernetes will help you out in automating the deployment, scaling, and management of containerized applications.

Once you’ve designed an application’s complete execution environment and associated components, using Kubernetes you can specify all that declaratively via configuration files. Then, you’ll be able to deploy that application with a single command. Once deployed, Kubernetes will give you tools to check on the health of your application, recover from issues, keep it running, scale it, etc.

There are a few basic concepts that we need to be familiar with in order to effectively work with Kubernetes. I think the official documentation does a great job in explaining this, but I’ll try to summarize.

Nodes, pods, and containers

First up are containers. If you’re interested in Kubernetes, chances are that you’ve already been exposed to some sort of container technology like Docker. If not, no worries. For our purposes here, we can think of a container as an isolated process with its own resources and file system in which an application can run.

A container has all the software dependencies that an application needs to run, including the application itself. From the application’s perspective, the container is its execution environment: the “machine” in which it’s running. In more practical terms, a container is a form of packaging, delivering, and executing an application. What’s the advantage? Instead of installing the application and its dependencies directly into the machine that’s going to run it, having it containerized allows for a container runtime (like Docker) to just run it as a self-contained unit. This makes it possible for the application to run anywhere that has the container runtime installed, with minimal configuration.

Something very closely related to containers is the concept of images. You can think of images as the blueprint for containers. An image is the spec, and the container is the instance that’s actually running.

When deploying applications into Kubernetes, this is how it runs them: via containers. In other words, for Kubernetes to be able to run an application, it needs to be delivered within a container.

Next is the concept of a node. This is very straightforward and not even specific to Kubernetes. A node is a computer within the cluster. That’s it. Like I said before, Kubernetes is built to manage computer clusters. A node is just one computer, either virtual or physical, within that cluster.

Then there are pods. Pods are the main executable units in Kubernetes. When we deploy an application or service into a Kubernetes cluster, it runs within a pod. Kubernetes works with containerized applications though, so it is the pods that take care of running said containers within them.

These three work very closely together within Kubernetes. To summarize: containers run within pods which in turn exist within nodes in the cluster.

There are other key components to talk about like deployments, services, replica sets, and persistent volumes. But I think that’s enough theory for now. We’ll learn more about all these as we get our hands dirty working though our example. So let’s get started with our demo and we’ll be discovering and discussing them organically as we go through it.

Installing and setting up Kubernetes

The first thing we need is a Kubernetes environment. There are many Kubernetes implementations out there. Google, Microsoft, and Amazon offer Kubernetes solutions on their respective cloud platforms, for example. There are also implementations that one can install and run on their own, like kind, minikube, and MicroK8s. We are going to use MicroK8s for our demo, for no particular reason other than “this is the one I know”.

When done installing, MicroK8s will have set up a whole Kubernetes cluster, with your machine as its one and only node.

Installing MicroK8s

So, if you’re in Ubuntu and have snapd, installing MicroK8s is easy. The official documentation explains it best. You install it with a command like this:

$ sudo snap install microk8s --classic --channel=1.21

MicroK8s will create a user group which is best to add your user account to so you can execute commands that would otherwise require admin privileges. You can do so with:

$ sudo usermod -a -G microk8s $USER
$ sudo chown -f -R $USER ~/.kube

With that, our very own Kubernetes cluster, courtesy of MicroK8s, should be ready to go. Check its status with:

$ microk8s status --wait-ready

You should see a “MicroK8s is running” message along with some specifications on your cluster. Including the available add-ons, which ones are enabled and which ones are disabled.

You can also shut down your cluster anytime with microk8s stop. Use microk8s start to bring it back up.

Introducing kubectl

MicroK8s also comes with kubectl. This is our gateway into Kubernetes, as this is the command line tool that we use to interact with it. By default, MicroK8s makes it so we can call it using microk8s kubectl .... That is, namespaced. This is useful if you have multiple Kubernetes implementations running at the same time, or another, separate kubectl. I don’t, so I like to create an alias for it, so that I can call it without having to use the microk8s prefix. You can do it like so:

$ sudo snap alias microk8s.kubectl kubectl

Now that all that’s done, we can start talking to our Kubernetes cluster. We can ask it for example to tell us which are the nodes in the cluster with this command:

$ kubectl get nodes

That will result in something like:

NAME     STATUS   ROLES    AGE   VERSION
pop-os   Ready    <none>   67d   v1.21.4-3+e5758f73ed2a04

The only node in the cluster is your own machine. In my case, my machine is called “pop-os” so that’s what shows up. You can get more information out of this command by using kubectl get nodes -o wide.

Installing add-ons

MicroK8s supports many add-ons that we can use to enhance our Kubernetes installation. We are going to need a few of them so let’s install them now. They are:

The dashboard, which gives us a nice web GUI which serves as a window into our cluster. In there we can see everything that’s running, read logs, run commands, etc.
dns, which sets up DNS for within the cluster. In general it’s a good idea to enable this one because other add-ons use it.
storage, which allows the cluster to access the host machine’s disk for storage. The application that we will deploy needs a persistent database so we need this plugin to make it happen.
registry, which sets up a container image registry that Kubernetes can access. Kubernetes runs containerized applications, containers are based on images. So, having this add-on allows us to define an image for our application and make it available to Kubernetes.

To install these, just run the following commands:

$ microk8s enable dashboard
$ microk8s enable dns
$ microk8s enable storage
$ microk8s enable registry

Those are all the add-ons that we’ll use.

Introducing the Dashboard

The dashboard is one we can play with right now. In order to access it, first run this:

$ microk8s dashboard-proxy

That will start up a proxy into the dashboard. The command will give you a URL and login token that you can use to access the dashboard. It results in an output like this:

Checking if Dashboard is running.
Dashboard will be available at https://127.0.0.1:10443
Use the following token to login:
<YOUR LOGIN TOKEN>

Now you can navigate to that URL in your browser and you’ll find a screen like this:

Make sure the “Token” option is selected and take the login token generated by the microk8s dashboard-proxy command from before and paste it in the field in the page. Click the “Sign In” button and you’ll be able to see the dashboard, allowing you access to many aspects of your cluster. It should look like this:

Feel free to play around with it a little bit. You don’t have to understand everything yet. As we work through our example, we’ll see how the dashboard and the other add-ons come into play.

There’s also a very useful command line tool called K9s, which helps in interacting with our cluster. We will not be discussing it further in this article but feel free to explore it if you need or want a command line alternative to the built-in dashboard.

Deploying applications into a Kubernetes cluster

With all that setup out of the way, we can start using our K8s cluster for what it was designed: running applications.

Deployments

Pods are very much the stars of the show when it comes to Kubernetes. However, most of the time we don’t create them directly. We usually do so through “deployments”. Deployments are a more abstract concept in Kubernetes. They basically control pods and make sure they behave as specified. You can think of them as wrappers for pods which make our lives easier than if we had to handle pods directly. Let’s go ahead and create a deployment so things will be clearer.

In Kubernetes, there are various ways of managing objects like deployments. For this post, I’m going to focus exclusively on the configuration-file-driven declarative approach as that’s the one better suited for real world scenarios.

You can learn more about the different ways of interacting with Kubernetes objects in the official documentation.

So, simply put, if we want to create a deployment, then we need to author a file that defines it. A simple deployment specification looks like this:

# nginx-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

This example is taken straight from the official documentation.

Don’t worry if most of that doesn’t make sense at this point. I’ll explain it in detail later. First, let’s actually do something with it.

Save that in a new file. You can call it nginx-deployment.yaml. Once that’s done, you can actually create the deployment (and its associated objects) in your K8s cluster with this command:

$ kubectl apply -f nginx-deployment.yaml

Which should result in the following message:

deployment.apps/nginx-deployment created

And that’s it for creating deployments! (Or any other type of object in Kubernetes for that matter.) We define the object in a file and then invoke kubectl’s apply command. Pretty simple.

If you want to delete the deployment, then this command will do it:
$ kubectl delete -f nginx-deployment.yaml
deployment.apps "nginx-deployment" deleted

Using kubectl to explore a deployment

Now, let’s inspect our cluster to see what this command has done for us.

First, we can ask it directly for the deployment with:

$ kubectl get deployments

Which outputs:

NAME               READY   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   3/3     3            3           2m54s

Now you can see that the deployment that we just created is right there with the name that we gave it.

As I said earlier, deployments are used to manage pods, and that’s just what the READY, UP-TO-DATE and AVAILABLE columns allude to with those values of 3. This deployment has three pods because, in our YAML file, we specified we wanted three replicas with the replicas: 3 line. Each “replica” is a pod. For our example, that means that we will have three instances of NGINX running side by side.

We can see the pods that have been created for us with this command:

$ kubectl get pods

Which gives us something like this:

NAME                                READY   STATUS    RESTARTS   AGE
nginx-deployment-66b6c48dd5-fs5rq   1/1     Running   0          55m
nginx-deployment-66b6c48dd5-xmnl2   1/1     Running   0          55m
nginx-deployment-66b6c48dd5-sfzxm   1/1     Running   0          55m

The exact names will vary, as the IDs are auto-generated. But as you can see, this command gives us some basic information about our pods. Remember that pods are the ones that actually run our workloads via containers. The READY field is particularly interesting in this sense because it tells us how many containers are running in the pods vs how many are supposed to run. So, 1/1 means that the pod has one container ready out of 1. In other words, the pod is fully ready.

Using the dashboard to explore a deployment

Like I said before, the dashboard offers us a window into our cluster. Let’s see how we can use it to see the information that we just saw via kubectl. Navigate into the dashboard via your browser and you should now see that some new things have appeared:

We now have new “CPU Usage” and “Memory Usage” sections that give us insight into the utilization of our machine’s resources.

There’s also “Workload status” that has some nice graphs giving us a glance at the status of our deployments, pods, and replica sets.

Don’t worry too much about replica sets right now, as we seldom interact with them directly. Suffice it to say, replica sets are objects that deployments rely on to make sure that the number of specified replica pods is maintained. As always, there’s more info in the official documentation.

Scroll down a little bit more and you’ll find the “Deployments” and “Pods” sections, which contain the information that we’ve already seen via kubectl before.

Feel free to click around and explore the capabilities of the dashboard.

Dissecting the deployment configuration file

Now that we have a basic understanding of deployments and pods and how to create them, let’s look more closely into the configuration file that defines it. This is what we had:

# nginx-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

This example is very simple, but it touches on the key aspects of deployment configuration. We will be building more complex deployments as we work through this article, but this is a great start. Let’s start at the top:

apiVersion: Under the hood, a Kubernetes cluster exposes its functionality via a REST API. We seldom interact with this API directly because we have kubectl that takes care of it for us. kubectl takes our commands, translates them into HTTP requests that the K8s REST API can understand, sends them, and gives us back the results. So, this apiVersion field specifies which version of the K8s REST API we are expecting to talk to.
kind: It represents the type of object that the configuration file defines. All objects in Kubernetes can be managed via YAML configuration files and kubectl apply. So, this field specifies which one we are managing at any given time.
metadata.name: Quite simply, the name of the object. It’s how we and Kubernetes refer to it.
metadata.labels: These help us further categorize cluster objects. These have no real effect in the system so they are useful for user help more than anything else.
spec: This contains the actual functional specification for the behavior of the deployment. More details below.
spec.replicas: The number of replica pods that the deployment should create. We already talked a bit about this before.
spec.selector.labels: This is one case when labels are actually important. Remember that when we create deployments, replica sets and pods are created with it. Within the K8s cluster, they each are their own individual objects though. This field is the mechanism that K8s uses to associate a given deployment with its replica set and pods. In practice, that means that whatever labels are in this field need to match the labels in spec.template.metadata.labels. More on that one below.
spec.template: Specifies the configuration of the pods that will be part of the deployment.
spec.template.metadata.labels: Very similar to metadata.labels. The only difference is that those labels are added to the deployment while these ones are added to the pods. The only notable thing is that these labels are key for the deployment to know which pods it should care about (as explained in above in spec.selector.labels).
spec.template.spec: This section specifies the actual functional configuration of the pods.
spec.template.spec.containers: This section specifies the configuration of the containers that will be running inside the pods. It’s an array so there can be many. In our example we have only one.
spec.template.spec.containers[0].name: The name of the container.
spec.template.spec.containers[0].image: The image that will be used to build the container.
spec.template.spec.containers[0].ports[0].containerPort: A port through which the container will accept traffic from the outside. In this case, 80.

You can find a detailed description of all the fields supported by deployment configuration files in the official API reference documentation. And much more!

Connecting to the containers in the pods

Kubernetes allows us to connect to the containers running inside a pod. This is pretty easy to do with kubectl. All we need to know is the name of the pod and the container that we want to connect to. If the pod is running only one container (like our NGINX one does) then we don’t need the container name. We can find out the names of our pods with:

$ kubectl get pods
NAME                                READY   STATUS    RESTARTS   AGE
nginx-deployment-66b6c48dd5-85nwq   1/1     Running   0          25s
nginx-deployment-66b6c48dd5-x5b4x   1/1     Running   0          25s
nginx-deployment-66b6c48dd5-wvkhc   1/1     Running   0          25s

Pick one of those and we can open a bash session in it with:

$ kubectl exec -it nginx-deployment-66b6c48dd5-85nwq -- bash

Which results in a prompt like this:

root@nginx-deployment-66b6c48dd5-85nwq:/#

We’re now connected to the container in one of our NGINX pods. There isn’t a lot to do with this right now, but feel free to explore it. It’s got its own processes and file system which are isolated from the other replica pods and your actual machine.

We can also connect to containers via the dashboard. Go back to the dashboard in your browser, log in again if the session expired, and scroll down to the “Pods” section. Each pod in the list has an action menu with an “Exec” command. See it here:

Click it and you’ll be taken to a screen with a console just like the one we obtained via kubectl exec:

The dashboard is quite useful, right?

Services

So far, we’ve learned quite a bit about deployments. How to specify and create them, how to explore them via command line and the dashboard, how to interact with the pods, etc. We haven’t seen a very important part yet, though: actually accessing the application that has been deployed. That’s where services come in. We use services to expose an application running in a set of pods to the world outside the cluster.

Here’s what a configuration file for a service that exposes access to our NGINX deployment could look like:

# nginx-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  type: NodePort
  selector:
    app: nginx
  ports:
    - name: "http"
      port: 80
      targetPort: 80
      nodePort: 30080

Same as with the deployment’s configuration file, this one also has a kind field that specifies what it is; and a name given to it via the metadata.name field. The spec section is where things get interesting.

spec.type specifies, well… The type of the service. Kubernetes supports many types of services. For now, we want a NodePort. This type of service makes sure to expose itself as a static port (given by spec.ports[0].nodePort) on every node in the cluster. In our setup, we only have one node, which is our own machine.
spec.ports defines which ports of the pods’ containers the service will expose.
spec.ports[0].name: The name of the port. To be used elsewhere to reference the specific port.
spec.ports[0].port: The port that will be exposed by the service.
spec.ports[0].targetPort: The port that the service will target in the container.
spec.ports[0].nodePort: The port that the service will expose in all the nodes of the cluster.

Same as with deployments, we can create such a service with the kubectl apply command. If you save the contents from the YAML above into a nginx-service.yaml file, you can run the following to create it:

$ kubectl apply -f nginx-service.yaml

And to inspect it and validate that it was in fact created:

$ kubectl get services
NAME            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes      ClusterIP   10.152.183.1    <none>        443/TCP        68d
nginx-service   NodePort    10.152.183.22   <none>        80:30080/TCP   27s

The dashboard also has a section for services. It looks like this:

Accessing an application via a service

We can access our service in a few different ways. We can use its “cluster IP” which we obtain from the output of the kubectl get services command. As given by the example above, that would be 10.152.183.22 in my case. Browsing to that IP gives us the familiar NGINX default welcome page:

Another way is by using the “NodePort”. Remember that the “NodePort” specifies the port in which the service will be available on every node of the cluster. With our current MicroK8s setup, our own machine is a node in the cluster, so we can also access the NGINX that’s running in our Kubernetes cluster using localhost:30080. 30080 is given by the spec.ports[0].nodePort field in the service configuration file from before. Try it out:

How cool is that? We have identical, replicated NGINX instances running in a Kubernetes cluster that’s installed locally in our machine.

Deploying our own custom application

Alright, by deploying NGINX, we’ve learned a lot about nodes, pods, deployments, services, and how they all work together to run and serve an application from a Kubernetes cluster. Now, let’s take all that knowledge and try to do the same for a completely custom application of our own.

What are we building

The application that we are going to deploy into our cluster is a simple one with only two components: a REST API written with .NET 5 and a Postgres database. You can find the source code in GitHub. It’s an API for supporting a hypothetical front end application for capturing used vehicle information and calculating their value in dollars.

If you’re interested in learning more about the process of actually writing that app, it’s all documented in another blog post: Building REST APIs with .NET 5, ASP.NET Core, and PostgreSQL.

If you’re following along, now would be a good time to download the source code of the web application that we’re going to be playing with. You can find it on GitHub. From now on, we’ll use that as the root directory of all the files we create and modify.

Also, be sure to delete or put aside the k8s directory. We’ll be building that throughout the rest of this post.

Deploying the database

Let’s begin with the Postgres database. Similar as before, we start by setting up a deployment with one pod and one container. We can do so with a deployment configuration YAML file like this:

# db-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vehicle-quotes-db
spec:
  selector:
    matchLabels:
      app: vehicle-quotes-db
  replicas: 1
  template:
    metadata:
      labels:
        app: vehicle-quotes-db
    spec:
      containers:
        - name: vehicle-quotes-db
          image: postgres:13
          ports:
            - containerPort: 5432
              name: "postgres"
          env:
            - name: POSTGRES_DB
              value: vehicle_quotes
            - name: POSTGRES_USER
              value: vehicle_quotes
            - name: POSTGRES_PASSWORD
              value: password
          resources:
            limits:
              memory: 4Gi
              cpu: "2"

This deployment configuration YAML file is similar to the one we used for NGINX before, but it introduces a few new elements:

spec.template.spec.containers[0].ports[0].name: We can give specific names to ports which we can reference later, elsewhere in the K8s configurations, which is what this field is for.
spec.template.spec.containers[0].env: This is a list of environment variables that will be defined in the container inside the pod. In this case, we’ve specified a few variables that are necessary to configure the Postgres instance that will be running. We’re using the official Postgres image from Dockerhub, and it calls for these variables. Their purpose is straightforward: they specify database name, username, and password.
spec.template.spec.containers[0].resources: This field defines the hardware resources that the container needs in order to function. We can specify upper limits with limits and lower ones with requests. You can learn more about resource management in the official documentation. In our case, we’ve kept it simple and used limits to prevent the container from using more than 4Gi of memory and 2 CPU cores.

Now, let’s save that YAML into a new file called db-deployment.yaml and run the following:

$ kubectl apply -f db-deployment.yaml

Which should output:

deployment.apps/vehicle-quotes-db created

After a few seconds, you should be able to see the new deployment and pod via kubectl:

$ kubectl get deployment -A
NAMESPACE            NAME                        READY   UP-TO-DATE   AVAILABLE   AGE
...
default              vehicle-quotes-db           1/1     1            1           9m20s

$ kubectl get pods -A
NAMESPACE            NAME                                         READY   STATUS    RESTARTS   AGE
...
default              vehicle-quotes-db-5fb576778-gx7j6            1/1     Running   0          9m22s

Remember you can also see them in the dashboard:

Connecting to the database

Let’s try connecting to the Postgres instance that we just deployed. Take note of the pod’s name and try:

$ kubectl exec -it <DB_POD_NAME> -- bash

You’ll get a bash session on the container that’s running the database. For me, given the pod’s auto-generated name, it looks like this:

root@vehicle-quotes-db-5fb576778-gx7j6:/#

From here, you can connect to the database using the psql command line client. Remember that we told the Postgres instance to create a vehicle_quotes user. We set it up via the container environment variables on our deployment configuration. As a result, we can do psql -U vehicle_quotes to connect to the database. Put together, it all looks like this:

$ kubectl exec -it vehicle-quotes-db-5fb576778-gx7j6 -- bash
root@vehicle-quotes-db-5fb576778-gx7j6:/# psql -U vehicle_quotes
psql (13.3 (Debian 13.3-1.pgdg100+1))
Type "help" for help.

vehicle_quotes=# \l
                                            List of databases
      Name      |     Owner      | Encoding |  Collate   |   Ctype    |         Access privileges
----------------+----------------+----------+------------+------------+-----------------------------------
 postgres       | vehicle_quotes | UTF8     | en_US.utf8 | en_US.utf8 |
 template0      | vehicle_quotes | UTF8     | en_US.utf8 | en_US.utf8 | =c/vehicle_quotes                +
                |                |          |            |            | vehicle_quotes=CTc/vehicle_quotes
 template1      | vehicle_quotes | UTF8     | en_US.utf8 | en_US.utf8 | =c/vehicle_quotes                +
                |                |          |            |            | vehicle_quotes=CTc/vehicle_quotes
 vehicle_quotes | vehicle_quotes | UTF8     | en_US.utf8 | en_US.utf8 |
(4 rows)

Pretty cool, don’t you think? We have a database running on our cluster now with minimal effort. There’s a slight problem though…

Persistent volumes and claims

The problem in our database is that any changes are lost if the pod or container were to shut down or reset for some reason. This is because all the database files live inside the container’s file system, so when the container is gone, the data is also gone.

In Kubernetes, pods are supposed to be treated as ephemeral entities. The idea is that pods should easily be brought down and replaced by new pods and users and clients shouldn’t even notice. This is all Kubernetes working as expected. That is to say, pods should be as stateless as possible to work well with this behavior.

However, a database is, by definition, not stateless. So what we need to do to solve this problem is have some available disk space from outside the cluster that can be used by our database to store its files. Something persistent that won’t go away if the pod or container goes away. That’s where persistent volumes and persistent volume claims come in.

We will use a persistent volume (PV) to define a directory in our host machine that we will allow our Postgres container to use to store data files. Then, a persistent volume claim (PVC) is used to define a “request” for some of that available disk space that a specific container can make. In short, a persistent volume says to K8s “here’s some storage that the cluster can use” and a persistent volume claim says “here’s a portion of that storage that’s available for containers to use”.

Configuration files for the PV and PVC

Start by tearing down our currently broken Postgres deployment:

$ kubectl delete -f db-deployment.yaml

Now let’s add two new YAML configuration files. One is for the persistent volume:

# db-persistent-volume.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: vehicle-quotes-postgres-data-persisent-volume
  labels:
    type: local
spec:
  claimRef:
    namespace: default
    name: vehicle-quotes-postgres-data-persisent-volume-claim
  storageClassName: manual
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/home/kevin/projects/vehicle-quotes-postgres-data"

In this config file, we already know about the kind and metadata fields. A few of the other elements are interesting though:

spec.claimRef: Contains identifying information about the claim that’s associated with the PV. Used to bind the PVC with a specific PVC. Notice how it matches the name defined in the PVC config file from below.
spec.capacity.storage: Is pretty straightforward in that it specifies the size of the persistent volume.
spec.accessModes: Defines how the PV can be accessed. In this case, we’re using ReadWriteOnce so that it can only be used by a single node in the cluster which is allowed to read from and write into the PV.
spec.hostPath.path: Specifies the directory in the host machine’s file system where the PV will be mounted. Simply put, the containers in the cluster will have access to the specific directory defined here. I’ve used /home/kevin/projects/vehicle-quotes-postgres-data because that makes sense on my own machine. If you’re following along, make sure to set it to something that makes sense in your environment.

hostPath is just one type of persistent volume which works well for development deployments. Managed Kubernetes implementations like the ones from Google or Amazon have their own types which are more appropriate for production.

We also need another config file for the persistent volume claim:

# db-persistent-volume-claim.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: vehicle-quotes-postgres-data-persisent-volume-claim
spec:
  volumeName: vehicle-quotes-postgres-data-persisent-volume
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi

Like I said, PVCs are essentially usage requests for PVs. So, the config file is simple in that it’s mostly specified to match the PV.

spec.volumeName: The name of the PV that this PVC is going to access. Notice how it matches the name that we defined in the PV’s config file.
spec.resources.requests: Defines how much space this PVC requests from the PV. In this case, we’re just requesting all the space that the PV has available to it, as given by its config file: 5Gi.

Configuring the deployment to use the PVC

After saving those files, all that’s left is to update the database deployment configuration to use the PVC. Here’s what the updated config file would look like:

 apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: vehicle-quotes-db
 spec:
   selector:
     matchLabels:
       app: vehicle-quotes-db
   replicas: 1
   template:
     metadata:
       labels:
         app: vehicle-quotes-db
     spec:
       containers:
         - name: vehicle-quotes-db
           image: postgres:13
           ports:
             - containerPort: 5432
               name: "postgres"
+          volumeMounts:
+            - mountPath: "/var/lib/postgresql/data"
+              name: vehicle-quotes-postgres-data-storage
           env:
             - name: POSTGRES_DB
               value: vehicle_quotes
             - name: POSTGRES_USER
               value: vehicle_quotes
             - name: POSTGRES_PASSWORD
               value: password
           resources:
             limits:
               memory: 4Gi
               cpu: "2"
+      volumes:
+        - name: vehicle-quotes-postgres-data-storage
+          persistentVolumeClaim:
+            claimName: vehicle-quotes-postgres-data-persisent-volume-claim

First, notice the volumes section at the bottom of the file. Here’s where we define the volume that will be available to the container, give it a name and specify which PVC it will use. The spec.template.volumes[0].persistentVolumeClaim.claimName needs to match the name of the PVC that we defined in db-persistent-volume-claim.yaml.

Then, up in the containers section, we define a volumeMounts element. We use that to specify which directory within the container will map to our PV. In this case, we’ve set the container’s /var/lib/postgresql/data directory to use the volume that we defined at the bottom of the file. That volume is backed by our persistent volume claim, which is in turn backed by our persistent volume. The significance of the /var/lib/postgresql/data directory is that this is where Postgres stores database files by default.

In summary: We created a persistent volume that defines some disk space in our machine that’s available to the cluster; then we defined a persistent volume claim that represents a request of some of that space that a container can have access to; after that we defined a volume within our pod configuration in our deployment to point to that persistent volume claim; and finally we defined a volume mount in our container that uses that volume to store the Postgres database files.

By setting it up this way, we’ve made it so that regardless of how many Postgres pods come and go, the database files will always be persisted, because the files now live outside of the container. They are stored in our host machine instead.

There’s another limitation that’s important to note. Just using the approach that we discussed, it’s not possible to deploy multiple replicas of Postgres which work in tandem and operate on the same data. Even though the data files can be defined outside of the cluster and persisted that way, there can only be one single Postgres instance running against it at any given time.

In production, the high availability problem is better solved leveraging the features provided by the database software itself. Postgres offers various options in that area. Or, if you are deploying to the cloud, the best strategy may be to use a relational database service managed by your cloud provider. Examples are Amazon’s RDS and Microsoft’s Azure SQL Database.

Applying changes

Now let’s see it in action. Run the following three commands to create the objects:

$ kubectl apply -f db-persistent-volume.yaml
$ kubectl apply -f db-persistent-volume-claim.yaml
$ kubectl apply -f db-deployment.yaml

After a while, they will show up in the dashboard. You already know how to look for deployments and pods. For persistent volumes, click the “Persistent Volumes” option under the “Cluster” section in the sidebar:

Persistent volume claims can be found in the “Persistent Volume Claims” option under the “Config and Storage” section in the sidebar:

Now, try connecting to the database (using kubectl exec -it <VEHICLE_QUOTES_DB_POD_NAME> -- bash and then psql -U vehicle_quotes) and creating some tables. Something simple like this would work:

CREATE TABLE test (test_field varchar);

Now, close psql and the bash in the pod and delete the objects:

$ kubectl delete -f db-deployment.yaml
$ kubectl delete -f db-persistent-volume-claim.yaml
$ kubectl delete -f db-persistent-volume.yaml

Create them again:

$ kubectl apply -f db-persistent-volume.yaml
$ kubectl apply -f db-persistent-volume-claim.yaml
$ kubectl apply -f db-deployment.yaml

Connect to the database again and you should see that the table is still there:

vehicle_quotes=# \c vehicle_quotes
You are now connected to database "vehicle_quotes" as user "vehicle_quotes".
vehicle_quotes=# \dt
           List of relations
 Schema | Name | Type  |     Owner
--------+------+-------+----------------
 public | test | table | vehicle_quotes
(1 row)

That’s just what we wanted: the database is persisting independently of what happens to the pods and containers.

Exposing the database as a service

Lastly, we need to expose the database as a service so that the rest of the cluster can access it without having to use explicit pod names. We don’t need this for our testing, but we do need it for later when we deploy our web app, so that it can reach the database. As you’ve seen, services are easy to create. Here’s the YAML config file:

# db-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: vehicle-quotes-db-service
spec:
  type: NodePort
  selector:
    app: vehicle-quotes-db
  ports:
    - name: "postgres"
      protocol: TCP
      port: 5432
      targetPort: 5432
      nodePort: 30432

Save that into a new db-service.yaml file and don’t forget to kubectl apply -f db-service.yaml.

Deploying the web application

Now that we’ve got the database sorted out, let’s turn our attention to the app itself. As you’ve seen, Kubernetes runs apps as containers. That means that we need images to build those containers. A custom web application is no exception. We need to build a custom image that contains our application so that it can be deployed into Kubernetes.

Building the web application image

The first step for building a container image is writing a Dockerfile. Since our application is a Web API built using .NET 5, I’m going to use a slightly modified version of the Dockerfile used by Visual Studio Code’s development container demo for .NET. These development containers are excellent for, well… development. You can see the original in the link above, but here’s mine:

# [Choice] .NET version: 5.0, 3.1, 2.1
ARG VARIANT="5.0"
FROM mcr.microsoft.com/vscode/devcontainers/dotnet:0-${VARIANT}

# [Option] Install Node.js
ARG INSTALL_NODE="false"

# [Option] Install Azure CLI
ARG INSTALL_AZURE_CLI="false"

# Install additional OS packages.
RUN apt-get update && export DEBIAN_FRONTEND=noninteractive \
    && apt-get -y install --no-install-recommends postgresql-client-common postgresql-client

# Run the remaining commands as the "vscode" user
USER vscode

# Install EF and code generator development tools
RUN dotnet tool install --global dotnet-ef
RUN dotnet tool install --global dotnet-aspnet-codegenerator
RUN echo 'export PATH="$PATH:/home/vscode/.dotnet/tools"' >> /home/vscode/.bashrc

WORKDIR /app

# Prevent the container from closing automatically
ENTRYPOINT ["tail", "-f", "/dev/null"]

There are many other development containers for other languages and frameworks. Take a look at the microsoft/vscode-dev-containers GitHub repo to learn more.

An interesting thing about this Dockerfile is that we install the psql command line client so that we can connect to our Postgres database from within the web application container. The rest is stuff specific to .NET and the particular image we’re basing this Dockerfile on, so don’t sweat it too much.

If you’ve downloaded the source code, this Dockerfile should already be there as Dockerfile.dev.

Making the image accessible to Kubernetes

Now that we have a Dockerfile, we can use it to build an image that Kubernetes is able to use to build a container to deploy. So that Kubernetes can see it, we need to build it in a specific way and push it into a registry that’s accessible to Kubernetes. Remember how we ran microk8s enable registry to install the registry add-on when we were setting up MicroK8s? That will pay off now, as that’s the registry to which we’ll push our image.

First, we build the image:

$ docker build . -f Dockerfile.dev -t localhost:32000/vehicle-quotes-dev:registry

That will take some time to download and set up everything. Once that’s done, we push the image to the registry:

$ docker push localhost:32000/vehicle-quotes-dev:registry

That will also take a little while.

Deploying the web application

The next step is to create a deployment for the web app. Like usual, we start with a deployment YAML configuration file. Let’s call it web-deployment.yaml:

# web-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vehicle-quotes-web
spec:
  selector:
    matchLabels:
      app: vehicle-quotes-web
  replicas: 1
  template:
    metadata:
      labels:
        app: vehicle-quotes-web
    spec:
      containers:
        - name: vehicle-quotes-web
          image: localhost:32000/vehicle-quotes-dev:registry
          ports:
            - containerPort: 5000
              name: "http"
            - containerPort: 5001
              name: "https"
          volumeMounts:
            - mountPath: "/app"
              name: vehicle-quotes-source-code-storage
          env:
            - name: POSTGRES_DB
              value: vehicle_quotes
            - name: POSTGRES_USER
              value: vehicle_quotes
            - name: POSTGRES_PASSWORD
              value: password
            - name: CUSTOMCONNSTR_VehicleQuotesContext
              value: Host=$(VEHICLE_QUOTES_DB_SERVICE_SERVICE_HOST);Database=$(POSTGRES_DB);Username=$(POSTGRES_USER);Password=$(POSTGRES_PASSWORD)
          resources:
            limits:
              memory: 2Gi
              cpu: "1"
      volumes:
        - name: vehicle-quotes-source-code-storage
          persistentVolumeClaim:
            claimName: vehicle-quotes-source-code-persisent-volume-claim

This deployment configuration should look very familiar to you by now as it is very similar to the ones we’ve already seen. There are a few notable elements though:

Notice how we specified localhost:32000/vehicle-quotes-dev:registry as the container image. This is the exact same name of the image that we built and pushed into the registry before.
In the environment variables section, the one named CUSTOMCONNSTR_VehicleQuotesContext is interesting for a couple of reasons:
- First, the value is a Postgres connection string being built off of other environment variables using the following format: $(ENV_VAR_NAME). That’s a neat feature of Kubernetes config files that allows us to reference variables to build other ones.
- Second, the VEHICLE_QUOTES_DB_SERVICE_SERVICE_HOST environment variable used within that connection string is not defined anywhere in our configuration files. That’s an automatic environment variable that Kubernetes injects on all containers when there are services available. In this case, it contains the hostname of the vehicle-quotes-db-service that we created a few sections ago. The automatic injection of this *_SERVICE_HOST variable always happens as long as the service is already created by the time that the pod gets created. We have already created the service so we should be fine using the variable here. As usual, there’s more info in the official documentation.

As you may have noticed, this deployment has a persistent volume. That’s to store the application’s source code. Or, more accurately, to make the source code, which lives in our machine, available to the container. This is a development setup after all, so we want to be able to edit the code from the comfort of our own file system, and have the container inside the cluster be aware of that.

Anyway, let’s create the associated persistent volume and persistent volume claim. Here’s the PV (save it as web-persistent-volume.yaml):

# web-persistent-volume.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: vehicle-quotes-source-code-persisent-volume
  labels:
    type: local
spec:
  claimRef:
    namespace: default
    name: vehicle-quotes-source-code-persisent-volume-claim
  storageClassName: manual
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/home/kevin/projects/vehicle-quotes"

And here’s the PVC (save it as web-persistent-volume-claim.yaml):

# web-persistent-volume-claim.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: vehicle-quotes-source-code-persisent-volume-claim
spec:
  volumeName: vehicle-quotes-source-code-persisent-volume
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

The only notable element here is the PV’s hostPath. I have it pointing to the path where I downloaded the app’s source code from GitHub. Make sure to do the same on your end.

Finally, tie it all up with a service that will expose the development build of our REST API. Here’s the config file:

# web-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: vehicle-quotes-web-service
spec:
  type: NodePort
  selector:
    app: vehicle-quotes-web
  ports:
    - name: "http"
      protocol: TCP
      port: 5000
      targetPort: 5000
      nodePort: 30000
    - name: "https"
      protocol: TCP
      port: 5001
      targetPort: 5001
      nodePort: 30001

Should be pretty self-explanatory at this point. In this case, we expose two ports, one for HTTP and another for HTTPS. Our .NET 5 Web API works with both so that’s why we specify them here. This configuration says that the service should expose port 30000 and send traffic that comes into that port from the outside world into port 5000 on the container. Likewise, outside traffic coming to port 30001 will be sent to port 5001 in the container.

Save that file as web-service.yaml and we’re ready to apply the changes:

$ kubectl apply -f web-persistent-volume.yaml
$ kubectl apply -f web-persistent-volume-claim.yaml
$ kubectl apply -f web-deployment.yaml
$ kubectl apply -f web-service.yaml

Feel free to explore the dashboard’s “Deployments”, “Pods”, “Services”, “Persistent Volumes”, and “Persistent Volume Claims” sections to see the fruits of our labor.

Starting the application

Let’s now do some final setup and start up our application. Start by connecting to the web application pod:

$ kubectl exec -it vehicle-quotes-web-86cbc65c7f-5cpg8 -- bash

Remember that the pod name will be different for you, so copy it from the dashboard or kubectl get pods -A.

You’ll get a prompt like this:

vscode ➜ /app (master ✗) $

Try ls to see all of the app’s source code files courtesy of the PV that we set up before:

vscode ➜ /app (master ✗) $ ls
Controllers     Dockerfile.prod  Models      README.md       Startup.cs            appsettings.Development.json  k8s      queries.sql
Data            K8S_README.md    Program.cs  ResourceModels  Validations           appsettings.json              k8s_wip
Dockerfile.dev  Migrations       Properties  Services        VehicleQuotes.csproj  database.dbml                 obj

Now it’s just a few .NET commands to get the app up and running. First, compile and download packages:

$ dotnet build

That will take a while. Once done, let’s build the database schema:

$ dotnet ef database update

And finally, run the development web server:

$ dotnet run

If you get the error message “System.InvalidOperationException: Unable to configure HTTPS endpoint.” while trying dotnet run, follow the error message’s instructions and run dotnet dev-certs https --trust. This will generate a development certificate so that the dev server can serve HTTPS.

As a result, you should see this:

vscode ➜ /app (master ✗) $ dotnet run
Building...
info: Microsoft.Hosting.Lifetime[0]
      Now listening on: https://0.0.0.0:5001
info: Microsoft.Hosting.Lifetime[0]
      Now listening on: http://0.0.0.0:5000
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Development
info: Microsoft.Hosting.Lifetime[0]
      Content root path: /app

It indicates that the application is up and running. Now, navigate to http://localhost:30000 in your browser of choice and you should see our REST API’s Swagger UI:

Notice that 30000 is the port we specified in the web-service.yaml’s nodePort for the http port. That’s the port that the service exposes to the world outside the cluster. Notice also how our .NET web app’s development server listens to traffic coming from ports 5000 and 5001 for HTTP and HTTPS respectively. That’s why we configured web-service.yaml like we did.

Outstanding! All our hard work has paid off and we have a full-fledged web application running in our Kubernetes cluster. This is quite a momentous occasion. We’ve built a custom image that can be used to create containers to run a .NET web application, pushed that image into our local registry so that K8s could use it, and deployed a functioning application. As a cherry on top, we made it so the source code is super easy to edit, as it lives within our own machine’s file system and the container in the cluster accesses it directly from there. Quite an accomplishment.

Now it’s time to go the extra mile and organize things a bit. Let’s talk about Kustomize next.

Putting it all together with Kustomize

Kustomize is a tool that helps us improve Kubernetes’ declarative object management with configuration files (which is what we’ve been doing throughout this post). Kustomize has useful features that help with better organizing configuration files, managing configuration variables, and support for deployment variants (for things like dev vs. test vs. prod environments). Let’s explore what Kustomize has to offer.

First, be sure to tear down all the objects that we have created so far as we will be replacing them later once we have a setup with Kustomize. This will work for that:

$ kubectl delete -f db-service.yaml
$ kubectl delete -f db-deployment.yaml
$ kubectl delete -f db-persistent-volume-claim.yaml
$ kubectl delete -f db-persistent-volume.yaml

$ kubectl delete -f web-service.yaml
$ kubectl delete -f web-deployment.yaml
$ kubectl delete -f web-persistent-volume-claim.yaml
$ kubectl delete -f web-persistent-volume.yaml

Next, let’s reorganize our db-* and web-* YAML files like this:

k8s
├── db
│   ├── db-deployment.yaml
│   ├── db-persistent-volume-claim.yaml
│   ├── db-persistent-volume.yaml
│   └── db-service.yaml
└── web
    ├── web-deployment.yaml
    ├── web-persistent-volume-claim.yaml
    ├── web-persistent-volume.yaml
    └── web-service.yaml

As you can see, we’ve put them all inside a new k8s directory, and further divided them into db and web sub-directories. web-* files went into the web directory and db-* files went into db. At this point, the prefixes on the files are a bit redundant so we can remove them. After all, we know what component they belong to because of the name of their respective sub-directories.

There’s already a k8s directory in the repo. Feel free to get rid of it as we will build it back up from scratch now.

So it should end up looking like this:

k8s
├── db
│   ├── deployment.yaml
│   ├── persistent-volume-claim.yaml
│   ├── persistent-volume.yaml
│   └── service.yaml
└── web
    ├── deployment.yaml
    ├── persistent-volume-claim.yaml
    ├── persistent-volume.yaml
    └── service.yaml

kubectl’s apply and delete commands support directories as well, not only individual files. That means that, at this point, to build up all of our objects you could simply do kubectl apply -f k8s/db and kubectl apply -f k8s/web. This is much better than what we’ve been doing until now where we had to specify every single file. Still, with Kustomize, we can do better than that…

The Kustomization file

We can bring everything together with a kustomization.yaml file. For our setup, here’s what it could look like:

# k8s/kustomization.yaml
kind: Kustomization

resources:
  - db/persistent-volume.yaml
  - db/persistent-volume-claim.yaml
  - db/service.yaml
  - db/deployment.yaml
  - web/persistent-volume.yaml
  - web/persistent-volume-claim.yaml
  - web/service.yaml
  - web/deployment.yaml

This first iteration of the Kustomization file is simple. It just lists all of our other config files in the resources section in their relative locations. Save that as k8s/kustomization.yaml and you can apply it with the following:

$ kubectl apply -k k8s

The -k option tells kubectl apply to look for a Kustomization within the given directory and use that to build the cluster objects. After running it, you should see familiar output:

service/vehicle-quotes-db-service created
service/vehicle-quotes-web-service created
persistentvolume/vehicle-quotes-postgres-data-persisent-volume created
persistentvolume/vehicle-quotes-source-code-persisent-volume created
persistentvolumeclaim/vehicle-quotes-postgres-data-persisent-volume-claim created
persistentvolumeclaim/vehicle-quotes-source-code-persisent-volume-claim created
deployment.apps/vehicle-quotes-db created
deployment.apps/vehicle-quotes-web created

Feel free to explore the dashboard or kubectl get commands to see the objects that got created. You can connect to pods, run the app, query the database, everything. Just like we did before. The only difference is that now everything is neatly organized and there’s a single file that serves as bootstrap for the whole setup. All thanks to Kustomize and the -k option.

kubectl delete -k k8s can be used to tear everything down.

Defining reusable configuration values with ConfigMaps

Another useful feature of Kustomize is ConfigMaps. These allow us to specify configuration variables in the Kustomization and use them throughout the rest of the resource config files. A good candidate to demonstrate their use are the environment variables that configure our Postgres database and the connection string in our web application.

We’re going to make changes to the config so be sure to tear everything down with kubectl delete -k k8s.

We can start by adding the following to the kustomization.yaml file:

 # k8s/kustomization.yaml
 kind: Kustomization

 resources:
   - db/persistent-volume.yaml
   - db/persistent-volume-claim.yaml
   - db/service.yaml
   - db/deployment.yaml
   - web/persistent-volume.yaml
   - web/persistent-volume-claim.yaml
   - web/service.yaml
   - web/deployment.yaml

+configMapGenerator:
+  - name: postgres-config
+    literals:
+      - POSTGRES_DB=vehicle_quotes
+      - POSTGRES_USER=vehicle_quotes
+      - POSTGRES_PASSWORD=password

The configMapGenerator section is where the magic happens. We’ve kept it simple and defined the variables as literals. configMapGenerator is much more flexible than that though, accepting external configuration files. The official documentation has more details.

Now, let’s see what we have to do to actually use those values in our configuration.

First up is the database deployment configuration file, k8s/db/deployment.yaml. Update its env section like so:

# k8s/db/deployment.yaml
# ...
env:
-  - name: POSTGRES_DB
-    value: vehicle_quotes
-  - name: POSTGRES_USER
-    value: vehicle_quotes
-  - name: POSTGRES_PASSWORD
-    value: password
+  - name: POSTGRES_DB
+    valueFrom:
+      configMapKeyRef:
+        name: postgres-config
+        key: POSTGRES_DB
+  - name: POSTGRES_USER
+    valueFrom:
+      configMapKeyRef:
+        name: postgres-config
+        key: POSTGRES_USER
+  - name: POSTGRES_PASSWORD
+    valueFrom:
+      configMapKeyRef:
+        name: postgres-config
+        key: POSTGRES_PASSWORD
# ...

Notice how we’ve replaced the simple key-value pairs with new, more complex objects. Their names are still the same, they have to be because that’s what the Postgres database container expects. But instead of a literal, hard-coded value, we have changed them to these valueFrom.configMapKeyRef objects. Their names match the name of the configMapGenerator we configured in the Kustomization. Their keys match the keys of the literal values that we specified in the configMapGenerator’s literals field. That’s how it all ties together.

Similarly, we can update the web application deployment configuration file, k8s/web/deployment.yaml. Its env section would look like this:

# k8s/web/deployment.yaml
# ...
env:
-  - name: POSTGRES_DB
-    value: vehicle_quotes
-  - name: POSTGRES_USER
-    value: vehicle_quotes
-  - name: POSTGRES_PASSWORD
-    value: password
+  - name: POSTGRES_DB
+    valueFrom:
+      configMapKeyRef:
+        name: postgres-config
+        key: POSTGRES_DB
+  - name: POSTGRES_USER
+    valueFrom:
+      configMapKeyRef:
+        name: postgres-config
+        key: POSTGRES_USER
+  - name: POSTGRES_PASSWORD
+    valueFrom:
+      configMapKeyRef:
+        name: postgres-config
+        key: POSTGRES_PASSWORD
  - name: CUSTOMCONNSTR_VehicleQuotesContext
    value: Host=$(VEHICLE_QUOTES_DB_SERVICE_SERVICE_HOST);Database=$(POSTGRES_DB);Username=$(POSTGRES_USER);Password=$(POSTGRES_PASSWORD)
# ...

This is the exact same change as with the database deployment. Out with the hard coded values and in with the new ConfigMap-driven ones.

Try kubectl apply -k k8s and you’ll see that things are still working well. Try to connect to the web application pod and build and run the app.

For data that’s important to secure like passwords, tokens, and keys, Kubernetes and Kustomize also offer Secrets and secretGenerator. Secrets are very similar to ConfigMaps in how they work, but are tailored specifically for handling secret data. You can learn more about them in the official documentation.

Creating variants for production and development environments

The crowning achievement of Kustomize is its ability to facilitate multiple deployment variants. Variants, as the name suggests, are variations of deployment configurations that are ideal for setting up various execution environments for an application. Think development, staging, production, etc., all based on a common set of reusable configurations to avoid superfluous repetition.

Kustomize does this by introducing the concepts of bases and overlays. A base is a set of configs that can be reused but not deployed on its own, and overlays are the actual configurations that use and extend the base and can be deployed.

To demonstrate this, let’s build two variants: one for development and another for production. Let’s consider the one we’ve already built to be the development variant and work towards properly specifying it as so, and then building a new production variant.

Note that the so-called “production” variant we’ll build is not actually meant to be production worthy. It’s just an example to illustrate the concepts and process of building bases and overlays. It does not meet the rigors of a proper production system.

Creating the base and overlays

The strategy I like to use is to just copy everything over from one variant to another, implement the differences, identify the common elements, and extract them into a base that both use.

Let’s begin by creating a new k8s/dev directory and move all of our YAML files into it. That will be our “development overlay”. Then, make a copy the k8s/dev directory and all of its contents and call it k8s/prod. That will be our “production overlay”. Let’s also create a k8s/base directory to store the common files. That will be our “base”. It should be like this:

k8s
├── base
├── dev
│   ├── kustomization.yaml
│   ├── db
│   │   ├── deployment.yaml
│   │   ├── persistent-volume-claim.yaml
│   │   ├── persistent-volume.yaml
│   │   └── service.yaml
│   └── web
│       ├── deployment.yaml
│       ├── persistent-volume-claim.yaml
│       ├── persistent-volume.yaml
│       └── service.yaml
└── prod
    ├── kustomization.yaml
    ├── db
    │   ├── deployment.yaml
    │   ├── persistent-volume-claim.yaml
    │   ├── persistent-volume.yaml
    │   └── service.yaml
    └── web
        ├── deployment.yaml
        ├── persistent-volume-claim.yaml
        ├── persistent-volume.yaml
        └── service.yaml

Now we have two variants, but they don’t do us any good because they aren’t any different. We’ll now go through each file one by one and identify which aspects need to be the same and which need to be different between our two variants:

db/deployment.yaml: I want the same database instance configuration for both our variants. So we copy the file into base/db/deployment.yaml and delete dev/db/deployment.yaml and prod/db/deployment.yaml.
db/persistent-volume-claim.yaml: This one is also the same for both variants. So we copy the file into base/db/persistent-volume-claim.yaml and delete dev/db/persistent-volume-claim.yaml and prod/db/persistent-volume-claim.yaml.
db/persistent-volume.yaml: This file defines the location in the host machine that will be available for the Postgres instance that’s running in the cluster to store its data files. I do want this path to be different between variants. So let’s leave them where they are and do the following changes to them: For dev/db/persistent-volume.yaml, change its spec.hostPath.path to "/path/to/vehicle-quotes-postgres-data-dev". For prod/db/persistent-volume.yaml, change its spec.hostPath.path to "/path/to/vehicle-quotes-postgres-data-prod". Of course, adjust the paths to something that makes sense in your environment.
db/service.yaml: There doesn’t need to be any difference in this file between the variants so we copy it into base/db/service.yaml and delete dev/db/service.yaml and prod/db/service.yaml.
web/deployment.yaml: There are going to be quite a few differences between the dev and prod deployments of the web application. So we leave them as they are. Later we’ll see the differences in detail.
web/persistent-volume-claim.yaml: This is also going to be different. Let’s leave it be now and we’ll come back to it later.
web/persistent-volume.yaml: Same as web/persistent-volume-claim.yaml. Leave it be for now.
wev/service.yaml: This one is going to be the same for both dev and prod so let’s do the usual and copy it into base/web/service.yaml and remove dev/web/service.yaml and prod/web/service.yaml

The decisions made when designing these overlays and the base may seem arbitrary. That’s because they totally are. The purpose of this article is to demonstrate Kustomize’s features, not produce a real-world, production-worthy setup.

Once all those changes are done, you should have the following file structure:

k8s
├── base
│   ├── db
│   │   ├── deployment.yaml
│   │   ├── persistent-volume-claim.yaml
│   │   └── service.yaml
│   └── web
│       └── service.yaml
├── dev
│   ├── kustomization.yaml
│   ├── db
│   │   └── persistent-volume.yaml
│   └── web
│       ├── deployment.yaml
│       ├── persistent-volume-claim.yaml
│       └── persistent-volume.yaml
└── prod
    ├── kustomization.yaml
    ├── db
    │   └── persistent-volume.yaml
    └── web
        ├── deployment.yaml
        ├── persistent-volume-claim.yaml
        └── persistent-volume.yaml

Much better, huh? We’ve gotten rid of quite a bit of repetition. But we’re not done just yet. The base also needs a Kustomization file. Let’s create it as k8s/base/kustomization.yaml and add these contents:

# k8s/base/kustomization.yaml
kind: Kustomization

resources:
  - db/persistent-volume-claim.yaml
  - db/service.yaml
  - db/deployment.yaml
  - web/service.yaml

configMapGenerator:
  - name: postgres-config
    literals:
      - POSTGRES_DB=vehicle_quotes
      - POSTGRES_USER=vehicle_quotes
      - POSTGRES_PASSWORD=password

As you can see, the file is very similar to the other one we created. We just list the resources that we moved into the base directory and define the database environment variables via the configMapGenerator. We need to define the configMapGenerator here because we’ve moved all the other files that use them into here.

Now that we have the base defined, we need to update the kustomization.yaml files of the overlays to use it. We also need to update them so that they only point to the resources that they need to.

Here’s how the changes to the “dev” overlay’s kustomization.yaml file look:

 # dev/kustomization.yaml
 kind: Kustomization

+bases:
+  - ../base

 resources:
   - db/persistent-volume.yaml
-  - db/persistent-volume-claim.yaml
-  - db/service.yaml
-  - db/deployment.yaml
   - web/persistent-volume.yaml
   - web/persistent-volume-claim.yaml
-  - web/service.yaml
   - web/deployment.yaml

-configMapGenerator:
-  - name: postgres-config
-    literals:
-      - POSTGRES_DB=vehicle_quotes
-      - POSTGRES_USER=vehicle_quotes
-      - POSTGRES_PASSWORD=password

As you can see we removed the configMapGenerator and the individual resources that were already defined in the base. Most importantly, we’ve added a bases element that indicates that our Kustomization over on the base directory is this overlay’s base.

The changes to the “prod” overlay’s kustomization.yaml file are identical. Go ahead and make them.

At this point, you can run kubectl apply -k k8s/dev or kubectl apply -k k8s/prod and things should work just like before.

Don’t forget to also do kubectl delete -k k8s/dev or kubectl delete -k k8s/prod when you’re done testing the previous commands, as we’ll continue doing changes to the configs. Keep in mind also that both variants can’t be deployed at the same time. So be sure delete one before applying the other.

Developing the production variant

I want our production variant to use a different image for the web application. That means a new Dockerfile. If you downloaded the source code from the GitHub repo, you should see the production Dockerfile in the root directory of the repo. It’s called Dockerfile.prod.

Here’s what it looks like:

# Dockerfile.prod
ARG VARIANT="5.0"
FROM mcr.microsoft.com/dotnet/sdk:${VARIANT}

RUN apt-get update && export DEBIAN_FRONTEND=noninteractive \
    && apt-get -y install --no-install-recommends postgresql-client-common postgresql-client

RUN dotnet tool install --global dotnet-ef
ENV PATH $PATH:/root/.dotnet/tools

RUN dotnet dev-certs https

WORKDIR /source

COPY . .

ENTRYPOINT ["tail", "-f", "/dev/null"]

The first takeaway from this production Dockerfile is that it is simpler, when compared to the development one. The image here is based on the official dotnet/sdk instead of the dev-ready one from vscode/devcontainers/dotnet. Also, this Dockerfile just copies all the source code into a /source directory within the image. This is because we want to “ship” the image with everything it needs to work without too much manual intervention. Also, we won’t be editing code live on the container, as opposed to how we set up the dev variant which allowed that. So we just copy the files over instead of leaving them out to provision them later via volumes. We’ll see how that pans out later.

Now that we have our production Dockerfile, we can build an image with it and push it to the registry so that Kubernetes can use it. So, save that file as Dockerfile.prod (or just use the one that’s already in the repo), and run the following commands:

Build the image with:

$ docker build . -f Dockerfile.prod -t localhost:32000/vehicle-quotes-prod:registry

And push it to the registry with:

$ docker push localhost:32000/vehicle-quotes-prod:registry

Now, we need to modify our prod variant’s deployment configuration so that it can work well with this new prod image. Here’s how the new k8s/prod/web/deployment.yaml should look:

# k8s/prod/web/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vehicle-quotes-web
spec:
  selector:
    matchLabels:
      app: vehicle-quotes-web
  replicas: 1
  template:
    metadata:
      labels:
        app: vehicle-quotes-web
    spec:
      initContainers:
        - name: await-db-ready
          image: postgres:13
          command: ["/bin/sh"]
          args: ["-c", "until pg_isready -h $(VEHICLE_QUOTES_DB_SERVICE_SERVICE_HOST) -p 5432; do echo waiting for database; sleep 2; done;"]
        - name: build
          image: localhost:32000/vehicle-quotes-dev:registry
          workingDir: "/source"
          command: ["/bin/sh"]
          args: ["-c", "dotnet restore -v n && dotnet ef database update && dotnet publish -c release -o /app --no-restore"]
          volumeMounts:
            - mountPath: "/app"
              name: vehicle-quotes-source-code-storage
          env:
            - name: POSTGRES_DB
              valueFrom:
                configMapKeyRef:
                  name: postgres-config
                  key: POSTGRES_DB
            - name: POSTGRES_USER
              valueFrom:
                configMapKeyRef:
                  name: postgres-config
                  key: POSTGRES_USER
            - name: POSTGRES_PASSWORD
              valueFrom:
                configMapKeyRef:
                  name: postgres-config
                  key: POSTGRES_PASSWORD
            - name: CUSTOMCONNSTR_VehicleQuotesContext
              value: Host=$(VEHICLE_QUOTES_DB_SERVICE_SERVICE_HOST);Database=$(POSTGRES_DB);Username=$(POSTGRES_USER);Password=$(POSTGRES_PASSWORD)
      containers:
        - name: vehicle-quotes-web
          image: localhost:32000/vehicle-quotes-dev:registry
          workingDir: "/app"
          command: ["/bin/sh"]
          args: ["-c", "dotnet VehicleQuotes.dll --urls=https://0.0.0.0:5001/"]
          ports:
            - containerPort: 5001
              name: "https"
          volumeMounts:
            - mountPath: "/app"
              name: vehicle-quotes-source-code-storage
          env:
            - name: POSTGRES_DB
              valueFrom:
                configMapKeyRef:
                  name: postgres-config
                  key: POSTGRES_DB
            - name: POSTGRES_USER
              valueFrom:
                configMapKeyRef:
                  name: postgres-config
                  key: POSTGRES_USER
            - name: POSTGRES_PASSWORD
              valueFrom:
                configMapKeyRef:
                  name: postgres-config
                  key: POSTGRES_PASSWORD
            - name: CUSTOMCONNSTR_VehicleQuotesContext
              value: Host=$(VEHICLE_QUOTES_DB_SERVICE_SERVICE_HOST);Database=$(POSTGRES_DB);Username=$(POSTGRES_USER);Password=$(POSTGRES_PASSWORD)
          resources:
            limits:
              memory: 2Gi
              cpu: "1"
      volumes:
        - name: vehicle-quotes-source-code-storage
          emptyDir: {}

This deployment config is similar to the one from the dev variant, but we’ve changed a few elements on it.

Init containers

The most notable change is that we added an initContainers section. Init Containers are one-and-done containers that run specific processes during pod initialization. They are good for doing any sort of initialization tasks that need to be run once, before a pod is ready to work. After they’ve done their task, they go away, and the pod is left with the containers specified in the containers section, like usual. In this case, we’ve added two init containers.

First is the await-db-ready one. This is a simple container based on the postgres:13 image that just sits there waiting for the database to become available. This is thanks to its command and args, which make up a simple shell script that leverages the pg_isready tool to continuously check if connections can be made to our database:

command: ["/bin/sh"]
args: ["-c", "until pg_isready -h $(VEHICLE_QUOTES_DB_SERVICE_SERVICE_HOST) -p 5432; do echo waiting for database; sleep 2; done;"]

This will cause pod initialization to stop until the database is ready.

Thanks to this blog post for the very useful recipe.

We need to wait for the database to be ready before continuing because of what the second init container does. Among other things, the build init container sets up the database. The database needs to be available for it to be able to to do that. The init container also downloads dependencies, builds the app, produces the deployable artifacts and copies them over to the directory from which the app will run: /app. You can see that all that is specified in the command and args elements, which define a few shell commands to do those tasks.

command: ["/bin/sh"]
args: ["-c", "dotnet restore -v n && dotnet ef database update && dotnet publish -c release -o /app --no-restore"]

Another interesting aspect of this deployment is the volume that we’ve defined. It’s at the bottom of the file, take a quick look:

volumes:
  - name: vehicle-quotes-source-code-storage
    emptyDir: {}

This one is different from the ones we’ve seen before which relied on persistent volumes and persistent volume claims. This one uses emptyDir. This means that this volume will provide storage that is persistent throughout the lifetime of the pod. That is, not tied to any specific container. In other words, even when the container goes away, the files in this volume will stay. This is a mechanism that’s useful when we want one container to produce some files that another container will use. In our case, the build init container produces the artifacts/binaries that the main vehicle-quotes-web container will use to actually run the web app.

The only other notable difference of this deployment is how its containers use the new prod image that we built before, instead of the dev one. That is, it uses localhost:32000/vehicle-quotes-prod:registry instead of localhost:32000/vehicle-quotes-dev:registry.

The rest of the deployment doesn’t have anything we haven’t already seen. Feel free to explore it.

As you saw, this prod variant doesn’t need to access the source code via a persistent volume. So, we don’t need PV and PVC definitions for it. So feel free to delete k8s/prod/web/persistent-volume.yaml and k8s/prod/web/persistent-volume-claim.yaml. Remember to also remove them from the resources section in k8s/prod/kustomization.yaml.

With those changes done, we can fire up our prod variants with:

$ kubectl apply -k k8s/prod

The web pod will take quite a while to properly start up because it’s downloading a lot of dependencies. Remember that you can use kubectl get pods -A to see their current status. Take note of their names and you would also be able to see container specific logs.

Use kubectl logs -f <vehicle-quotes-web-pod-name> await-db-ready to see the logs from the await-db-ready init container.
Use kubectl logs -f <vehicle-quotes-web-pod-name> build to see the logs from the build init container.

If this were an actual production setup, and we were worried about pod startup time, there’s one way we could make it faster. We could perform the “download dependencies” step when building the production image instead of when deploying the pods. So, we could have our Dockerfile.prod call dotnet restore -v n, instead of the build init container. That way building the image would take more time, but it would have all dependencies already baked in by the time Kubernetes tries to use it to build containers. Then the web pod would start up much faster.

This deployment automatically starts the web app, so after the pods are in the “Running” status (or green in the dashboard!), we can just navigate to the app via a web browser. We’ve configured the deployment to only work over HTTPS (as given by the ports section in the vehicle-quotes-web container), so this is the only URL that’s available to us: https://localhost:30001. We can navigate to it and see the familiar screen:

At this point, we finally have fully working, distinct variants. However, we can take our configuration a few steps further by leveraging some additional Kustomize features.

Using patches for small, precise changes

Right now, the persistent volume configurations for the databases of both variants are pretty much identical. The only difference is the hostPath. With patches, we can focus in on that property and vary it specifically.

To do it, we first copy either of the variant’s db/persistent-volume.yaml into k8s/base/db/persistent-volume.yaml. We also need to add it under resources on k8s/base/kustomization.yaml:

 # k8s/base/kustomization.yaml
 # ...
 resources:
+  - db/persistent-volume.yaml
   - db/persistent-volume-claim.yaml
 # ...

That will serve as the common ground for overlays to “patch over”. Now we can create the patches. First, the one for the one for the dev variant:

# k8s/dev/db/persistent-volume-host-path.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: vehicle-quotes-postgres-data-persisent-volume
spec:
  hostPath:
    path: "/home/kevin/projects/vehicle-quotes-postgres-data-dev"

And then the one for the prod variant:

# k8s/prod/db/persistent-volume-host-path.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: vehicle-quotes-postgres-data-persisent-volume
spec:
  hostPath:
    path: "/home/kevin/projects/vehicle-quotes-postgres-data-prod"

As you can see, these patches are sort of truncated persistent volume configs which only include the kind, metadata.name, and the value that actually changes: the hostPath.

Once those are saved, we need to include them in their respective kustomization.yaml. It’s the same modification to both k8s/dev/kustomization.yaml and k8s/prod/kustomization.yaml. Just remove the db/persistent-volume.yaml item from their resources sections and add the following to both of them:

patches:
  - db/persistent-volume-host-path.yaml

Right now, k8s/dev/kustomization.yaml should be:

# k8s/dev/kustomization.yaml
kind: Kustomization

bases:
  - ../base

resources:
  - web/persistent-volume.yaml
  - web/persistent-volume-claim.yaml
  - web/deployment.yaml

patches:
  - db/persistent-volume-host-path.yaml

And k8s/prod/kustomization.yaml should be:

# k8s/prod/kustomization.yaml
kind: Kustomization

bases:
  - ../base

resources:
  - web/deployment.yaml

patches:
  - db/persistent-volume-host-path.yaml

Overriding container images

Another improvement we can make is to use the images element in the kustomization.yaml files to control the web app images used by the deployments in the variants. This is easier for maintenance as it’s defined in one single, expected place. Also, the full name of the image doesn’t have to be used throughout the configs so it reduces repetition.

To put it in practice, add the following at the end of the k8s/dev/kustomization.yaml file:

images:
  - name: vehicle-quotes-web
    newName: localhost:32000/vehicle-quotes-dev
    newTag: registry

Similar thing with k8s/prod/kustomization.yaml, only use the prod image for this one:

images:
  - name: vehicle-quotes-web
    newName: localhost:32000/vehicle-quotes-prod
    newTag: registry

Now, we can replace any mention of localhost:32000/vehicle-quotes-dev in the dev variant, and any mention of localhost:32000/vehicle-quotes-prod in the prod variant with vehicle-quotes-web. Which is simpler.

In k8s/dev/web/deployment.yaml:

 # ...
     spec:
       containers:
         - name: vehicle-quotes-web
-          image: localhost:32000/vehicle-quotes-dev:registry
+          image: vehicle-quotes-web
           ports:
 # ...

And in k8s/prod/web/deployment.yaml:

 # ...

         - name: build
-          image: localhost:32000/vehicle-quotes-prod:registry
+          image: vehicle-quotes-web
           workingDir: "/source"
           command: ["/bin/sh"]
 # ...

       containers:
         - name: vehicle-quotes-web
-          image: localhost:32000/vehicle-quotes-prod:registry
+          image: vehicle-quotes-web
           workingDir: "/app"
 # ...

Once that’s all done, you should be able to kubectl apply -k k8s/dev or kubectl apply -k k8s/prod and everything should work fine. Be sure to kubectl delete before kubectl applying a different variant though, as both of them cannot coexist in the same cluster, due to many objects having the same name.

Closing thoughts

Wow! That was a good one. In this post I’ve captured all the knowledge that I wish I had when I first encountered Kubernetes. We went from knowing nothing to being able to put together a competent environment. We figured out how to install Kubernetes locally via MicroK8s, along with a few useful add-ons. We learned about the main concepts in Kubernetes like nodes, pods, images, containers, deployments, services, and persistent volumes. Most importantly, we learned how to define and create them using a declarative configuration file approach.

Then, we learned about Kustomize and how to use it to implement variants of our configurations. And we did all that by actually getting our hands dirty and, step by step, deploying a real web application and its backing database system. When all was said and done, a simple kubeclt apply -k <kustomization> was all it took to get the app up and running fully. Not bad, eh?

Useful commands

Start up MicroK8s: microk8s start
Shut down MicroK8s: microk8s stop
Check MicroK8s info: microk8s status
Start up the K8s dashboard: microk8s dashboard-proxy
Get available pods on all namespaces: kubectl get pods -A
Watch and follow the logs on a specific container in a pod: kubectl logs -f <POD_NAME> <CONTAINER_NAME>
Open a shell into the default container in a pod: kubectl exec -it <POD_NAME> -- bash
Create a K8s resource given a YAML file or directory: kubectl apply -f <FILE_OR_DIRECTORY_NAME>
Delete a K8s resource given a YAML file or directory: kubectl delete -f <FILE_OR_DIRECTORY_NAME>
Create K8s resources with Kustomize: kubectl apply -k <KUSTOMIZATION_DIR>
Delete K8s resources with Kustomize: kubectl delete -k <KUSTOMIZATION_DIR>
Build custom images for the K8s registry: docker build . -f <DOCKERFILE> -t localhost:32000/<IMAGE_NAME>:registry
Push custom images to the K8s registry: docker push localhost:32000/<IMAGE_NAME>:registry

Fixing a PostgreSQL cluster that has no superuser

2022-01-07T00:00:00+00:00

Normally in a newly-created PostgreSQL database cluster there is a single all-powerful administrative role (user) with “superuser” privileges, conventionally named postgres, though it can have any name.

After the initial cluster setup you can create other roles as needed. You may optionally grant one or more of your new roles the superuser privilege, but it is best to avoid granting superuser to any other roles because you and your applications should generally connect as roles with lower privilege to reduce the risk of accidental or malicious damage to your database.

Let’s break something 😎

Imagine you have a cluster with two or more superuser roles. If you accidentally remove superuser privilege from one role, you can simply connect as the other superuser and re-grant it.

But if you have a cluster where only the single postgres role is a superuser, what happens if you connect as that role and try to remove its superuser privilege?

$ psql -U postgres postgres
psql (14.1)
Type "help" for help.

postgres=# \du
                       List of roles
 Role name |            Attributes             | Member of
-----------+-----------------------------------+-----------
 postgres  | Superuser, Create role, Create DB | {}
 somebody  | Create DB                         | {}

postgres=> \conninfo
You are connected to database "postgres" as user "postgres" via socket in "/tmp" at port "5432".
postgres=# ALTER ROLE postgres NOSUPERUSER;
ALTER ROLE
postgres=# \du
                 List of roles
 Role name |       Attributes       | Member of
-----------+------------------------+-----------
 postgres  | Create role, Create DB | {}
 somebody  | Create DB              | {}

postgres=# ALTER ROLE postgres SUPERUSER;
ERROR:  must be superuser to alter superuser roles or change superuser attribute
postgres=# \q

PostgreSQL happily lets us do that, and now we have no superuser, and so we cannot re-grant the privilege to that role or any other!

Homebrew PostgreSQL problem

Aside from such a severe operator error, there are other situations where you may find no superuser exists. One happened to me recently while experimenting with PostgreSQL installed by Homebrew on macOS.

I used Homebrew to install postgresql@14 and later noticed that it left me with a single role named after my OS user, and it was not a superuser. It couldn’t even create other roles. I’m not sure how that happened, perhaps somehow caused by an earlier installation of postgresql on the same computer, but so it was.

Since there wasn’t any data in there yet, I could have simply deleted the existing PostgreSQL cluster and created a new one. But in other circumstances there could have been data in there that I needed to preserve, which wasn’t accessible to my one less-privileged user, and which caused errors in pg_dumpall.

So how can we solve this problem the right way?

First, stop the server

We need to get lower-level access to our database. To do that, first we stop the running database server.

On a typical modern Linux server running systemd, that looks like:

# systemctl stop postgresql-14

On macOS using Homebrew services, that could be:

$ brew services stop postgresql

Or in my experimental case with Homebrew using a temporary Postgres server which I’m showing here:

$ pg_ctl -D /opt/homebrew/var/postgresql@14 stop
waiting for server to shut down.... done
server stopped

Next, start the PostgreSQL stand-alone backend

Next we start the “stand-alone backend” which a single user can interact with directly, not using a separate client.

No privilege checks are done here, so we can re-grant the superuser privilege to our postgres role.

Interestingly, SQL statements entered here end with a newline, no ; needed, though adding one doesn’t hurt. And statements here can’t span multiple lines without being continued with \ at the end of each intermediate line.

In the command below, note that the --single option must come first, and the postgres at the end of the command is the name of the database we want to connect to:

$ postgres --single -D /opt/homebrew/var/postgresql@14 postgres

PostgreSQL stand-alone backend 14.1
backend> ALTER ROLE postgres SUPERUSER
2022-01-07 20:32:51.321 MST [27246] LOG:  statement: ALTER ROLE postgres SUPERUSER

2022-01-07 20:32:51.322 MST [27246] LOG:  duration: 1.242 ms
backend>

The postgres server stand-alone prompt does not have all the niceties of psql including line editing features, history, and backslash metacommands such as \q to quit, so we type control-D there to mark “end of file” on our input stream and exit the program.

Back to normal

Now we can again start the normal multi-user client/server PostgreSQL service:

$ pg_ctl -D /opt/homebrew/var/postgresql@14 start
waiting for server to start.... done
server started

And finally we can connect to the server and confirm our change persisted:

$ psql -U postgres postgres
psql (14.1)
Type "help" for help.

postgres=# \du
                       List of roles
 Role name |            Attributes             | Member of
-----------+-----------------------------------+-----------
 postgres  | Superuser, Create role, Create DB | {}
 somebody  | Create DB                         | {}

postgres=#

Reference

Ecommerce customer names with interesting Unicode characters

2021-12-29T00:00:00+00:00

One of our clients with a busy ecommerce site sees a lot of orders, and among those, sometimes there are unusual customer name and address submissions.

We first noticed in 2015 that they had a customer order come in with an emoji in the name field of the order. The emoji was 😏 and we half-jokingly wondered if that was a new sign of fraud.

Over the following 3 years only a few more orders came in with various emoji in customer’s names, but in mid-2018 emoji started to appear increasingly frequently until now one now appears on average every day or two.

Why the sudden appearance of emoji in 2015? It correlates with the rapid shift to browsing the web and shopping on mobile devices. Mobile visits now represent more than half of this client’s ecommerce traffic.

Most people in 2015 didn’t even know how to type emoji on a desktop or laptop computer, but mobile touchscreen keyboards began showing emoji choices around that time, so the mobile explanation makes sense. And mobile keyboard autocorrect also sometimes offers emoji in addition to words, making them even more common in the past few years.

Just for fun I wanted to automatically find all such “interesting” names, so I wrote a simple report that uses SQL to query their PostgreSQL ecommerce database.

To preserve customer privacy, names shown here have been changed and limited to a few names that are common in the United States.

Real names with bonuses

First let’s look at the apparently real names with emoji and other non-alphabetical Unicode characters mixed in:

Amy 🩺
Amy💕
Amy👑
👸Amy
Anna 💫
Anna 😎
Anna ❤️
Anna💊💉
Bob☯️
Bob😍😘😜👑💞
Bob🐞
Brenda 🌙
Brenda 🥳🤪
Brenda☠️
B R E N D A 💕💅🏾💃🏾👜
Cameron 🌻
Cameron 😎
Cameron 💁🏻‍♀️
Doug 💕
Doug 🅱️
Doug 🌸🤩
Doug 👦🏻❤️
Emily 🌷
Emily 😊🇨🇺
Emily😚
Emily 🎰👑❤️
Frank 🥰
Frank ⚗️
Frank🔆
Frank💞💰
Jane 💕
Jane⁸
Jane ❤️❤️
Jane🔆
Jill 👑🎀
Jill 👼🏽✨🌫
Jill🍓🍒🍒
Jill👸🏽💖
Jim 🐰
Jim, 💕
Jim’s iPhone✨
Joe 🐯
Joe 🇦🇺
Joe😏
Joe🏀🎸‼️
John 👪
John⁵
John🎭
Karen 🌻🌹
Karen 👑✨
Karen 🔒❤️
Karen🎀👑
Kate💙💚
Kate🤪🤞🏽💙
Kate.💘
K$ate💉
Liz🌺
Liz❣️
Liz❤️🙃
Liz Mama💙💙
Mary 🎀
Mary💘
Mary👼🏼💓
Mary⁹
Mike 👑
Mike 👑🌸
Mike 👩🏻‍🌾
Mike⁶
Sarah 👑
Sarah👑A.
✨Sarah✨
💛🌻 Sarah
Steve🐑
Steve 🦁
Steve♓️💓
Victoria 🥴
Victoria🌻
V𝚒𝚌𝚝𝚘𝚛𝚒𝚊
V I C T O R I A 🤍

Would you have expected all that in ecommerce orders? I didn’t!

Fake names

Next let’s look at placeholder names with people’s role or self-description or similar:

Amor ⚽️
Babe ❤️
C𝚒𝚝𝚒𝚣𝚎𝚗
Daddy🥴😏
Daddy😘👴👨🙏👨👩👧🙇🏾
Fly High 🕊
Forever 💍💜
Granny 👵🏽
HOME ❤️
Home🏠
Home🏠💜
Hubby🥰
💦Juicy🍑
me!! 💛
mi amor ❤️
Mom💗
Mom♥️
Mom 🐥💛
MOMMY 💗
Myself 😘
Princess👑
princess❤️
Queen 😍💖🔓
Queen💘
The Husband💍❤️
Wifey 😈✌🏽👅

Perhaps the occurrence of “home” several times reflects a mobile address book auto-fill function for billing or shipping address fields?

Not names at all!

Then there are those customers who didn’t provide any kind of name at all, just emoji and other special characters:

🦋
💙
🤡🎪
💗☁️
🍌
♥️
🌙
⁵
∅

I guess only one or two of those per year doesn’t amount to much, but they’re interesting to see.

Strange addresses

In addition to the name fields we also checked the address fields for unusual characters and found (again, details changed to preserve privacy):

125 E 27😎
227 W 24 Circle ⭕️
3 Blvd. George Washington™

Simply odd

The prizewinner for oddity, which seems like some kind of copy-and-paste mistake, is this in the city field of an address:

Indian® Roadmaster™ Classic

Maybe at least one motorcycle has achieved sentience and needed to do some online shopping!

“Interesting” Unicode ranges

When searching for interesting Unicode ranges, we could just look for characters in the Unicode emoji ranges. That would be fairly straightforward since there are just a few ranges to match.

But we were curious what other unusual characters were getting used aside from emoji, so we wanted to include other classes of characters. So perhaps we should include everything to start and then exclude the entire class of Unicode “word characters”? That covers the world’s standard characters used for names and addresses, including not just Roman/Latin with optional diacritics, but also other character sets such as Cyrillic, Arabic, Hebrew, Korean, Chinese, Japanese, Devanagari, Thai, and many others.

PostgreSQL POSIX regular expressions include the class “word character” represented by either [[:word:]] or the Perl shorthand \w. I started with that, but found it covered too many things I did want to see, such as the visually double-width Latin characters that are part of the Chinese word character range, and some special numbers.

So I switched back to matching what I want, rather than excluding what I don’t want, and I manually went through the Unicode code charts and noted the ranges to include.

The list of Unicode code ranges I came up with, in hexadecimal, is:

250-2ba
2bc-2c5
2cc-2dc
2de-2ff
58d-58e
fd5-fd8
1d00-1dbf
2070-2079
207b-209f
20d0-2104
2106-2115
2117-215f
2163-218b
2190-2211
2213-266e
2670-2bff
2e00-2e7f
2ff0-2fff
3004
3012-3013
3020
3200-33ff
4dc0-4dff
a000-abf9
fff0-fffc
fffe-1d35f
1d360-1d37f
1d400-1d7ff
1ec70-1ecbf
1ed00-1ed4f
1ee00-1eeff
1f000-10ffff

Those ranges exclude several fairly common characters that people (or their software’s autocorrect) used in their address fields, which we wanted to ignore, such as:

Music sharp sign: ♯ (instead of # before a number)
numero: №
care of: ℅
replacement character: � (though this could be interesting if it reveals unknown encoding errors)

The SQL query

PostgreSQL allows us to represent Unicode characters in hexadecimal numbers as either \u plus 4 digits or \U plus 8 digits. So the character 2ba is written in a Postgres string as \u02ba and the range fffe-1d35f is written in a Postgres regex range as [\ufffe-\U0001d35f].

With a little scripting to put it all together, I came up with:

SELECT order_number, order_timestamp::date AS order_date,
    fname, lname, company, address1, address2, city, state, b_fname, b_lname, b_company, b_address1, b_address2, b_city, b_state, phone
FROM orders
WHERE concat(fname, lname, company, address1, address2, city, state, b_fname, b_lname, b_company, b_address1, b_address2, b_city, b_state, phone)
    ~ '[\u0250-\u02ba\u02bc-\u02c5\u02cc-\u02dc\u02de-\u02ff\u058d-\u058e\u0fd5-\u0fd8\u1d00-\u1dbf\u2070-\u2079\u207b-\u209f\u20d0-\u2104\u2106-\u2115\u2117-\u215f\u2163-\u218b\u2190-\u2211\u2213-\u266e\u2670-\u2bff\u2e00-\u2e7f\u2ff0-\u2fff\u3004\u3012-\u3013\u3020\u3200-\u33ff\u4dc0-\u4dff\ua000-\uabf9\ufff0-\ufffc\ufffe-\U0001d35f\U0001d360-\U0001d37f\U0001d400-\U0001d7ff\U0001ec70-\U0001ecbf\U0001ed00-\U0001ed4f\U0001ee00-\U0001eeff\U0001f000-\U0010ffff]'
    -- limit how many years back to go
    AND order_timestamp > (SELECT CURRENT_TIMESTAMP - interval '3 years')
    -- exclude any order that had PII expunged for GDPR
    AND expunged_at IS NULL
ORDER BY order_timestamp DESC

Try a similar query on databases you have access to, and see what interesting user submissions you discover. I always find surprises and in addition to being fun, sometimes we find things that help us improve our input validation and user guidance so that more mistakes are caught when they’re easy for the customer to correct.

Happy holidays!

Building REST APIs with .NET 5, ASP.NET Core, and PostgreSQL

2021-07-09T00:00:00+00:00

Photo by Sam Beasley

This is old news by now, but I’m still amazed by the fact that nowadays .NET is open source and can run on Linux. I truly believe that this new direction can help the technology realize its true potential, since it’s no longer shackled to Windows-based environments. I’ve personally been outside the .NET game for a good while, but with the milestone release that is .NET 5, I think now is a great time to dive back in.

So I thought of taking some time to do just that, really dive in, see what’s new, and get a sense of the general developer experience that the current incarnation of .NET offers. So in this blog post, I’m going to chronicle my experience developing a simple but complete REST API application. Along the way, I’ll touch on the most common problems that one runs into when developing such applications and how they are solved in the .NET world. So think of this piece as a sort of tutorial or overview of the most common framework features when it comes to developing REST APIs.

There’s a table of contents at the bottom.

First, let’s get familiar with what we’re building.

What we’re building

The demo application

You can find the finished product on my GitHub.

The application that we’ll be building throughout this article will address a request from a hypothetical car junker business. Our client wants to automate the process of calculating how much money to offer their customers for their vehicles, given certain information about them. And they want an app to do that. We are building the back-end component that will support that app. It is a REST API that allows users to provide vehicle information (year, make, model, condition, etc.) and will produce a quote of how much money our hypothetical client would be willing to pay for it.

Here’s a short list of features that we need to implement in order to fulfill that requirement:

Given a vehicle model and condition, calculate a price.
Store and manage rules that are used to calculate vehicle prices.
Store and manage pricing overrides on a vehicle model basis. Price overrides are used regardless of the current rules.
CRUD vehicle models so that overrides can be specified for them.

The data model

Here’s what our data model looks like:

The main table in our model is the quotes table. It stores all the requests for quotes received from our client’s customers. It captures all the relevant vehicle information in terms of model and condition. It also captures the offered quote; that is, the money value that our system calculates for their vehicle.

The quotes table includes all the fields that identify a vehicle: year, make, model, body style and size. It also includes a model_style_year_id field which is an optional foreign key to another table. This FK points to the model_style_years table which contains specific vehicle models that our system can store explicitly.

The idea of this is that, when a customer submits a request for a quote, if we have their vehicle registered in our database, then we can populate this foreign key and link the quote with the specific vehicle that it’s quoting. If we don’t have their vehicle registered, then we leave that field unpopulated. Either way, we can offer a quote. The only difference is the level or certainty of the quote.

The records in the model_style_years table represent specific vehicles. That whole hierarchy works like this: A vehicle make (e.g. Honda, Toyota, etc. in the makes table) has many models (e.g. Civic, Corolla, etc. in the models table), each model has many styles (the model_styles table). Styles are combinations of body types (the body_types table) and sizes (e.g. Mid-size Sedan, Compact Coupe, etc. in the sizes table). And finally, each model style has many years in which they were being produced (via the model_style_years table).

This model allows us very fine-grained differentiation between vehicles. For example, we can have a “2008 Honda Civic Hatchback which is a Compact car” and also a “1990 Honda Civic Hatchback which is a Sub-compact”. That is, same model, different year, size or body type.

We also have a quote_rules table which stores the rules that are applied when it comes to calculating a vehicle quote. The rules are pairs of key-values with an associated monetary value. So for example, rules like “a vehicle that has alloy wheels is worth $10 more” can be expressed in the table with a record where feature_type is “has_alloy_wheels”, feature_value is “true” and price_modifier is “10”.

Finally, we have a quote_overrides table which specifies a flat, static price for specific vehicles (via the link to the model_style_years table). The idea here is that if some customer requests a quote for a vehicle for which we have an override, no price calculation rules are applied and they are offered what is specified in the override record.

The development environment

Setting up the PostgreSQL database with Docker

For this project, our database of choice is PostgreSQL. Luckily for us, getting a PostgreSQL instance up and running is very easy thanks to Docker.

If you want to learn more about dockerizing a typical web application, take a look at this article that explains the process in detail.

Once you have Docker installed in your machine, getting a PostgreSQL instance is as simple as running the following command:

$ docker run -d \
    --name vehicle-quote-postgres \
    -p 5432:5432 \
    --network host \
    -e POSTGRES_DB=vehicle_quote \
    -e POSTGRES_USER=vehicle_quote \
    -e POSTGRES_PASSWORD=password \
    postgres

Here we’re asking Docker to run a new container based on the latest postgres image from DockerHub, name it vehicle-quote-postgres, specify the port to use the default PostgreSQL one, make it accessible to the local network (with the --network host option) and finally, specify a few environment variables that the postgres image uses when building our new instance to set up the default database name, user and password (with the three -e options).

After Docker is done working its magic, you should be able to access the database with something like this:

$ docker exec -it vehicle-quote-postgres psql -U vehicle_quote
psql (13.2 (Debian 13.2-1.pgdg100+1))
Type "help" for help.

vehicle_quote=#

This command is connecting to our new vehicle-quote-postgres container and then, from within the container, using the command line client psql in order to connect to the database.

If you have psql installed on your own machine, you can use it directly to connect to the PostgreSQL instance running inside the container:

$ psql -h localhost -U vehicle_quote

This is possible because we specified in our docker run command that the container would be accepting traffic over port 5432 (-p 5432:5432) and that it would be accesible within the same network as our actual machine (--network host).

Installing the .NET 5 SDK

Ok, with that out of the way, let’s install .NET 5.

.NET 5 truly is multi-platform, so whatever environment you prefer to work with, they’ve got you covered. You can go to the .NET 5 download page and pick your desired flavor of the SDK.

On Ubuntu 20.10, which is what I’m running, installation is painless. It’s your typical process with APT and this page from the official docs has all the details.

First step is to add the Microsoft package repository:

$ wget https://packages.microsoft.com/config/ubuntu/20.10/packages-microsoft-prod.deb -O packages-microsoft-prod.deb

$ sudo dpkg -i packages-microsoft-prod.deb

Then, install .NET 5 with APT like one would any other software package:

$ sudo apt-get update; \
  sudo apt-get install -y apt-transport-https && \
  sudo apt-get update && \
  sudo apt-get install -y dotnet-sdk-5.0

Run dotnet --version in your console and you should see something like this:

$ dotnet --version
5.0.301

Setting up the project

Creating our ASP.NET Core REST API project

Ok now that we have our requirements, database and SDK, let’s start setting up our project. We do so with the following command:

$ dotnet new webapi -o VehicleQuotes

This instructs the dotnet command line tool to create a new REST API web application project for us in a new VehicleQuotes directory.

As a result, dotnet will give you some messages, including this:

The template "ASP.NET Core Web API" was created successfully.

A new directory was created with our web application files. The newly created VehicleQuotes project looks like this:

.
├── appsettings.Development.json
├── appsettings.json
├── Controllers
│   └── WeatherForecastController.cs
├── obj
│   ├── project.assets.json
│   ├── project.nuget.cache
│   ├── VehicleQuotes.csproj.nuget.dgspec.json
│   ├── VehicleQuotes.csproj.nuget.g.props
│   └── VehicleQuotes.csproj.nuget.g.targets
├── Program.cs
├── Properties
│   └── launchSettings.json
├── Startup.cs
├── VehicleQuotes.csproj
└── WeatherForecast.cs

Important things to note here are the appsettings.json and appsettings.Development.json files which contain environment specific configuration values; the Controllers directory where we define our application controllers and action methods (i.e. our REST API endpoints); the Program.cs and Startup.cs files that contain our application’s entry point and bootstrapping logic; and finally VehicleQuotes.csproj which is the file that contains project-wide configuration that the framework cares about like references, compilation targets, and other options. Feel free to explore.

The dotnet new command has given us quite a bit. These files make up a fully working application that we can run and play around with. It even has a Swagger UI, as I’ll demonstrate shortly. It’s a great place to get started from.

You can also get a pretty comprehensive .gitignore file by running the dotnet new gitignore command.

From inside the VehicleQuotes directory, you can run the application with:

$ dotnet run

Which will start up a development server and give out the following output:

$ dotnet run
Building...
info: Microsoft.Hosting.Lifetime[0]
      Now listening on: https://localhost:5001
info: Microsoft.Hosting.Lifetime[0]
      Now listening on: http://localhost:5000
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Development
info: Microsoft.Hosting.Lifetime[0]
      Content root path: /home/kevin/projects/endpoint/blog/VehicleQuotes

Open up a browser window and go to https://localhost:5001/swagger to find a Swagger UI listing our API’s endpoints:

As you can see we’ve got a GET /WeatherForecast endpoint in our app. This is included by default in the webapi project template that we specified in our call to dotnet new. You can see it defined in the Controllers/WeatherForecastController.cs file.

Installing packages we’ll need

Now let’s install all the tools and libraries we will need for our application. First, we install the ASP.NET Code Generator tool which we’ll use later for scaffolding controllers:

$ dotnet tool install --global dotnet-aspnet-codegenerator

We also need to install the Entity Framework command line tools which help us with creating and applying database migrations:

$ dotnet tool install --global dotnet-ef

Now, we need to install a few libraries that we’ll use in our project. First are all the packages that allow us to use Entity Framework Core, provide scaffolding support and give us a detailed debugging page for database errors:

$ dotnet add package Microsoft.VisualStudio.Web.CodeGeneration.Design
$ dotnet add package Microsoft.EntityFrameworkCore.Design
$ dotnet add package Microsoft.EntityFrameworkCore.SqlServer
$ dotnet add package Microsoft.AspNetCore.Diagnostics.EntityFrameworkCore

We also need the EF Core driver for PostgreSQL which will allow us to interact with our database:

$ dotnet add package Npgsql.EntityFrameworkCore.PostgreSQL

Finally, we need another package that will allow us to use the snake case naming convention for our database tables, fields, etc. We need this because EF Core uses capitalized camel case by default, which is not very common in the PostgreSQL world, so this will allow us to play nice. This is the package:

$ dotnet add package EFCore.NamingConventions

Connecting to the database and performing initial app configuration

In order to connect to, query, and modify a database using EF Core, we need to create a DbContext. This is a class that serves as the entry point into the database. Create a new directory called Data in the project root and add this new VehicleQuotesContext.cs file to it:

using Microsoft.EntityFrameworkCore;

namespace VehicleQuotes
{
    public class VehicleQuotesContext : DbContext
    {
        public VehicleQuotesContext (DbContextOptions<VehicleQuotesContext> options)
            : base(options)
        {
        }
    }
}

As you can see this is just a simple class that inherits from EF Core’s DbContext class. That’s all we need for now. We will continue building on this class as we add new tables and configurations.

Now, we need to add this class into ASP.NET Core’s built-in IoC (inversion of control) container so that it’s available to controllers and other classes via Dependency Injection, and tell it how to find our database. Go to Startup.cs and add the following using statement near the top of the file:

using Microsoft.EntityFrameworkCore;

That will allow us to do the following change in the ConfigureServices method:

 public void ConfigureServices(IServiceCollection services)
 {

     // ...

+    services.AddDbContext<VehicleQuotesContext>(options =>
+        options
+            .UseNpgsql(Configuration.GetConnectionString("VehicleQuotesContext"))
+    );
 }

UseNpgsql is an extension method made available to us by the Npgsql.EntityFrameworkCore.PostgreSQL package that we installed in the previous step.

The services variable contains all the objects (known as “services”) that are available in the app for Dependency Injection. So here, we’re adding our newly created DbContext to it, specifying that it will connect to a PostgreSQL database (via the options.UseNpgsql call), and that it will use a connection string named VehicleQuotesContext from the app’s default configuration file. So let’s add the connection string then. To do so, change the appsettings.json like so:

 {
   "Logging": {
     "LogLevel": {
       "Default": "Information",
       "Microsoft": "Warning",
       "Microsoft.Hosting.Lifetime": "Information"
     }
   },
-  "AllowedHosts": "*"
+  "AllowedHosts": "*",
+  "ConnectionStrings": {
+      "VehicleQuotesContext": "Host=localhost;Database=vehicle_quote;Username=vehicle_quote;Password=password"
+  }
 }

This is your typical PostgreSQL connection string. The only gotcha is that it needs to be specified under the ConnectionStrings -> VehicleQuotesContext section so that our call to Configuration.GetConnectionString can find it.

Now let’s put the EFCore.NamingConventions package to good use and configure EF Core to use snake case when naming database objects. Add the following to the ConfigureServices method in Startup.cs:

public void ConfigureServices(IServiceCollection services)
{
    // ...

    services.AddDbContext<VehicleQuotesContext>(options =>
        options
            .UseNpgsql(Configuration.GetConnectionString("VehicleQuotesContext"))
+           .UseSnakeCaseNamingConvention()
    );
}

UseSnakeCaseNamingConvention is an extension method made available to us by the EFCore.NamingConventions package that we installed in the previous step.

Now let’s make logging a little bit more verbose with:

public void ConfigureServices(IServiceCollection services)
{
    // ...

    services.AddDbContext<VehicleQuotesContext>(options =>
        options
            .UseNpgsql(Configuration.GetConnectionString("VehicleQuotesContext"))
            .UseSnakeCaseNamingConvention()
+           .UseLoggerFactory(LoggerFactory.Create(builder => builder.AddConsole()))
+           .EnableSensitiveDataLogging()
    );
}

This will make sure full database queries appear in the log in the console, including parameter values. This could expose sensitive data so be careful when using EnableSensitiveDataLogging in production.

We can also add the following service configuration to have the app display detailed error pages when something related to the database or migrations goes wrong:

public void ConfigureServices(IServiceCollection services)
{
    // ...

+   services.AddDatabaseDeveloperPageExceptionFilter();
}

AddDatabaseDeveloperPageExceptionFilter is an extension method made available to us by the Microsoft.AspNetCore.Diagnostics.EntityFrameworkCore package that we installed in the previous step.

Finally, one last configuration I like to do is have the Swagger UI show up at the root URL, so that instead of using https://localhost:5001/swagger, we’re able to just use https://localhost:5001. We do so by by updating the Configure method this time, in the same Startup.cs file that we’ve been working on:

public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
    if (env.IsDevelopment())
    {
        app.UseDeveloperExceptionPage();
        app.UseSwagger();
-       app.UseSwaggerUI(c => c.SwaggerEndpoint("/swagger/v1/swagger.json", "VehicleQuotes v1"));
+       app.UseSwaggerUI(c => {
+           c.SwaggerEndpoint("/swagger/v1/swagger.json", "VehicleQuotes v1");
+           c.RoutePrefix = "";
+       });
    }

The magic is done by the c.RoutePrefix = ""; line which makes it so there’s no need to put any prefix in order to access the auto generated Swagger UI.

Try it out. Do dotnet run and navigate to https://localhost:5001 and you should see the Swagger UI there.

Building the application

Creating model entities, migrations and updating the database

Alright, with all that configuration out of the way, let’s implement some of our actual application logic now. Refer back to our data model. We’ll start by defining our three simplest tables: makes, sizes and body_types. With EF Core, we define tables via so-called POCO entities, which are simple C# classes with some properties. The classes become tables and the properties become the tables’ fields. Instances of these classes represent records in the database.

So, create a new Models directory in our project’s root and add these three files:

// Models/BodyType.cs
namespace VehicleQuotes.Models
{
    public class BodyType
    {
        public int ID { get; set; }
        public string Name { get; set; }
    }
}

// Models/Make.cs
namespace VehicleQuotes.Models
{
    public class Make
    {
        public int ID { get; set; }
        public string Name { get; set; }
    }
}

// Models/Size.cs
namespace VehicleQuotes.Models
{
    public class Size
    {
        public int ID { get; set; }
        public string Name { get; set; }
    }
}

Now, we add three corresponding DbSets to our DbContext in Data/VehicleQuoteContext.cs. Here’s the diff:

using Microsoft.EntityFrameworkCore;
+using VehicleQuotes.Models;

namespace VehicleQuotes
{
    public class VehicleQuotesContext : DbContext
    {
        public VehicleQuotesContext (DbContextOptions<VehicleQuotesContext> options)
            : base(options)
        {
        }

+       public DbSet<Make> Makes { get; set; }
+       public DbSet<Size> Sizes { get; set; }
+       public DbSet<BodyType> BodyTypes { get; set; }
    }
}

This is how we tell EF Core to build tables in our database for our entities. You’ll see later how we use those DbSets to access the data in those tables. For now, let’s create a migration script that we can later run to apply changes to our database. Run the following to have EF Core create it for us:

$ dotnet ef migrations add AddLookupTables

Now take a look at the newly created Migrations directory. It contains a few new files, but the one we care about right now is Migrations/{TIMESTAMP}_AddLookupTables.cs. In its Up method, it’s got some code that will modify the database structure when run. The EF Core tooling has inspected our project, identified the new entities, and automatically generated a migration script for us that creates tables for them. Notice also how the tables and fields use the snake case naming convention, just as we specified with the call to UseSnakeCaseNamingConvention in Startup.cs.

Now, to actually run the migration script and apply the changes to the database, we do:

$ dotnet ef database update

That command inspects our project to find any migrations that haven’t been run yet, and applies them. In this case, we only have one, so that’s what it runs. Look at the output in the console to see it working its magic step by step:

$ dotnet ef database update
Build started...
Build succeeded.
warn: Microsoft.EntityFrameworkCore.Model.Validation[10400]
      Sensitive data logging is enabled. Log entries and exception messages may include sensitive application data; this mode should only be enabled during development.

...

info: Microsoft.EntityFrameworkCore.Database.Command[20101]
      Executed DbCommand (4ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
      CREATE TABLE sizes (
          id integer GENERATED BY DEFAULT AS IDENTITY,
          name text NULL,
          CONSTRAINT pk_sizes PRIMARY KEY (id)
      );
info: Microsoft.EntityFrameworkCore.Database.Command[20101]
      Executed DbCommand (1ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
      INSERT INTO "__EFMigrationsHistory" (migration_id, product_version)
      VALUES ('20210625212939_AddLookupTables', '5.0.7');
Done.

Notice how it warns us about potential exposure of sensitive data because of that EnableSensitiveDataLogging option we opted into in Startup.cs. Also, EF Core related logs are extra verbose showing all database operations because of another configuration option that we applied there: the UseLoggerFactory(LoggerFactory.Create(builder => builder.AddConsole())) one.

You can connect to the database with the psql command line client and see that the changes took effect:

$ psql -h localhost -U vehicle_quote

...

vehicle_quote=# \c vehicle_quote 
psql (12.7 (Ubuntu 12.7-0ubuntu0.20.10.1), server 13.2 (Debian 13.2-1.pgdg100+1))
You are now connected to database "vehicle_quote" as user "vehicle_quote".
vehicle_quote=# \dt
                   List of relations
 Schema |         Name          | Type  |     Owner     
--------+-----------------------+-------+---------------
 public | __EFMigrationsHistory | table | vehicle_quote
 public | body_types            | table | vehicle_quote
 public | makes                 | table | vehicle_quote
 public | sizes                 | table | vehicle_quote
(4 rows)

vehicle_quote=# \d makes
                            Table "public.makes"
 Column |  Type   | Collation | Nullable |             Default              
--------+---------+-----------+----------+----------------------------------
 id     | integer |           | not null | generated by default as identity
 name   | text    |           |          | 
Indexes:
    "pk_makes" PRIMARY KEY, btree (id)

There are our tables in all their normalized, snake-cased glory. The __EFMigrationsHistory table is used internally by EF Core to keep track of which migrations have been applied.

Creating controllers for CRUDing our tables

Now that we have that, let’s add a few endpoints to support basic CRUD of those tables. We can use the dotnet-aspnet-codegenerator scaffolding tool that we installed earlier. For the three tables that we have, we would do:

$ dotnet aspnet-codegenerator controller \
    -name MakesController \
    -m Make \
    -dc VehicleQuotesContext \
    -async \
    -api \
    -outDir Controllers

$ dotnet aspnet-codegenerator controller \
    -name BodyTypesController \
    -m BodyType \
    -dc VehicleQuotesContext \
    -async \
    -api \
    -outDir Controllers

$ dotnet aspnet-codegenerator controller \
    -name SizesController \
    -m Size \
    -dc VehicleQuotesContext \
    -async \
    -api \
    -outDir Controllers

Those commands tell the scaffolding tool to create new controllers that:

Are named as given by the -name option.
Use the model class specified in the -m option.
Use our VehicleQuotesContext to talk to the database. As per the -dc option.
Define the methods using async/await syntax. Given by the -async option.
Are API controllers. Specified by the -api option.
Are created in the Controllers directory. Via the -outDir option.

Explore the new files that got created in the Controllers directory: MakesController.cs, BodyTypesController.cd and SizesController.cs. The controllers have been generated with the necessary Action Methods to fetch, create, update and delete their corresponding entities. Try dotnet run and navigate to https://localhost:5001 to see the new endpoints in the Swagger UI:

Try it out! You can interact with each of the endpoints from the Swagger UI and it all works as you’d expect.

Adding unique constraints via indexes

Ok, our app is coming along well. Right now though, there’s an issue with the tables that we’ve created. It’s possible to create vehicle makes with the same name. The same is true for body types and sizes. This doesn’t make much sense for these tables. So let’s fix that by adding a uniqueness constraint. We can do it by creating a unique database index using EF Core’s Index attribute. For example, we can modify our Models/Make.cs like so:

+using Microsoft.EntityFrameworkCore;

namespace VehicleQuotes.Models
{
+   [Index(nameof(Name), IsUnique = true)]
    public class Make
    {
        public int ID { get; set; }
        public string Name { get; set; }
    }
}

In fact, do the same for our other entities in Models/BodyType.cs and Models/Size.cs. Don’t forget the using Microsoft.EntityFrameworkCore statement.

With that, we can create a new migration:

$ dotnet ef migrations add AddUniqueIndexesToLookupTables

That will result in a new migration script in Migrations/{TIMESTAMP}_AddUniqueIndexesToLookupTables.cs. Its Up method looks like this:

protected override void Up(MigrationBuilder migrationBuilder)
{
    migrationBuilder.CreateIndex(
        name: "ix_sizes_name",
        table: "sizes",
        column: "name",
        unique: true);

    migrationBuilder.CreateIndex(
        name: "ix_makes_name",
        table: "makes",
        column: "name",
        unique: true);

    migrationBuilder.CreateIndex(
        name: "ix_body_types_name",
        table: "body_types",
        column: "name",
        unique: true);
}

As you can see, new unique indexes are being created on the tables and fields that we specified. Like before, apply the changes to the database structure with:

$ dotnet ef database update

Now if you try to create, for example, a vehicle make with a repeated name, you’ll get an error. Try doing so by POSTing to /api/Makes via the Swagger UI:

Responding with specific HTTP error codes (409 Conflict)

The fact that we can now enforce unique constraints is all well and good. But the error scenario is not very user friendly. Instead of returning a “500 Internal Server Error” status code with a wall of text, we should be responding with something more sensible. Maybe a “409 Conflict” would be more appropriate for this kind of error. We can easily update our controllers to handle that scenario. What we need to do is update the methods that handle the POST and PUT endpoints so that they catch the Microsoft.EntityFrameworkCore.DbUpdateException exception and return the proper response. Here’s how we would do it for the MakesController:

// ...
namespace VehicleQuotes.Controllers
{
    [Route("api/[controller]")]
    [ApiController]
    public class MakesController : ControllerBase
    {
        // ...

        [HttpPut("{id}")]
        public async Task<IActionResult> PutMake(int id, Make make)
        {
            // ...

            try
            {
                await _context.SaveChangesAsync();
            }
            // ...
+           catch (Microsoft.EntityFrameworkCore.DbUpdateException)
+           {
+               return Conflict();
+           }

            return NoContent();
        }

        [HttpPost]
        public async Task<ActionResult<Make>> PostMake(Make make)
        {
            _context.Makes.Add(make);
-           await _context.SaveChangesAsync();

+           try
+           {
+               await _context.SaveChangesAsync();
+           }
+           catch (Microsoft.EntityFrameworkCore.DbUpdateException)
+           {
+               return Conflict();
+           }

            return CreatedAtAction("GetMake", new { id = make.ID }, make);
        }

        // ...
    }
}

Go ahead and do the same for the other two controllers, and try again to POST a repeated make name via the Swagger UI. You should see this now instead:

Much better now, don’t you think?

Adding a more complex entity to the model

Now let’s work on an entity that’s a little bit more complex: the one we will use to represent vehicle models.

For this entity, we don’t want our API to be as low level as the one for the other three, where it was basically a thin wrapper over database tables. We want it to be a little bit more abstract and not expose the entire database structure verbatim.

Refer back to the data model. We’ll add models, model_styles and model_style_years. Let’s start by adding the following classes:

// Models/Model.cs
using System.Collections.Generic;

namespace VehicleQuotes.Models
{
    public class Model
    {
        public int ID { get; set; }
        public string Name { get; set; }
        public int MakeID { get; set; }

        public Make Make { get; set; }

        public ICollection<ModelStyle> ModelStyles { get; set; }
    }
}

// Models/ModelStyle.cs
using System.Collections.Generic;

namespace VehicleQuotes.Models
{
    public class ModelStyle
    {
        public int ID { get; set; }
        public int ModelID { get; set; }
        public int BodyTypeID { get; set; }
        public int SizeID { get; set; }

        public Model Model { get; set; }
        public BodyType BodyType { get; set; }
        public Size Size { get; set; }

        public ICollection<ModelStyleYear> ModelStyleYears { get; set; }
    }
}

// Models/ModelStyleYear.cs
namespace VehicleQuotes.Models
{
    public class ModelStyleYear
    {
        public int ID { get; set; }
        public string Year { get; set; }
        public int ModelStyleID { get; set; }

        public ModelStyle ModelStyle { get; set; }
    }
}

Notice how some of these entities now include properties whose types are other entities. Some of them are collections even. These are called Navigation Properties and are how we tell EF Core that our entities are related to one another. These will result in foreign keys being created in the database.

Take the Model entity for example. It has a property Make of type Make. It also has a MakeID property of type int. EF Core sees this and figures out that there’s a relation between the makes and models tables. Specifically, that models have a make. A many-to-one relation where the models table stores a foreign key to the makes table.

Similarly, the Model entity has a ModelStyles property of type ICollection<ModelStyleYear>. This tells EF Core that models have many model_styles. This one is a one-to-many relation from the perspective of the models table. The foreign key lives in the model_styles table and points back to models.

The official documentation is a great resource to learn more details about how relationships work in EF Core.

After that, same as before, we have to add the corresponding DbSets to our DbContext:

// Data/VehicleQuotesContext.cs
// ...

namespace VehicleQuotes
{
    public class VehicleQuotesContext : DbContext
    {
        // ...

+       public DbSet<Model> Models { get; set; }
+       public DbSet<ModelStyle> ModelStyles { get; set; }
+       public DbSet<ModelStyleYear> ModelStyleYears { get; set; }
    }
}

Don’t forget the migration script. First create it:

$ dotnet ef migrations add AddVehicleModelTables

And then apply it:

$ dotnet ef database update

Adding composite unique indexes

These vehicle model related tables also need some uniqueness enforcement. This time, however, the unique keys are composite. Meaning that they involve multiple fields. For vehicle models, for example, it makes no sense to have multiple records with the same make and name. But it does make sense to have multiple models with the same name, as long as they belong to different makes. We can solve for that with a composite index. Here’s how we create one of those in Model with EF Core:

using System.Collections.Generic;
+using Microsoft.EntityFrameworkCore;

namespace VehicleQuotes.Models
{
+   [Index(nameof(Name), nameof(MakeID), IsUnique = true)]
    public class Model
    {
        // ...
    }
}

Very similar to what we did with the Make, BodyType, and Size entities. The only difference is that this time we included multiple fields in the parameters for the Index attribute.

We should do the same for ModelStyle and ModelStyleYear:

using System.Collections.Generic;
+using Microsoft.EntityFrameworkCore;

namespace VehicleQuotes.Models
{
+   [Index(nameof(ModelID), nameof(BodyTypeID), nameof(SizeID), IsUnique = true)]
    public class ModelStyle
    {
        // ...
    }
}

+using Microsoft.EntityFrameworkCore;

namespace VehicleQuotes.Models
{
+   [Index(nameof(Year), nameof(ModelStyleID), IsUnique = true)]
    public class ModelStyleYear
    {
        // ...
    }
}

Don’t forget the migrations:

$ dotnet ef migrations add AddUniqueIndexesForVehicleModelTables

$ dotnet ef database update

Adding controllers with custom routes

Our data model dictates that vehicle models belong in a make. In other words, a vehicle model has no meaning by itself. It only has meaning within the context of a make. Ideally, we want our API routes to reflect this concept. In other words, instead of URLs for models to look like this: /api/Models/{id}; we’d rather them look like this: /api/Makes/{makeId}/Models/{modelId}. Let’s go ahead and scaffold a controller for this entity:

$ dotnet aspnet-codegenerator controller \
    -name ModelsController \
    -m Model \
    -dc VehicleQuotesContext \
    -async \
    -api \
    -outDir Controllers

Now let’s change the resulting Controllers/ModelsController.cs to use the URL structure that we want. To do so, we modify the Route attribute that’s applied to the ModelsController class to this:

[Route("api/Makes/{makeId}/[controller]/")]

Do a dotnet run and take a peek at the Swagger UI on https://localhost:5001 to see what the Models endpoint routes look like now:

The vehicle model routes are now nested within makes, just like we wanted.

Of course, this is just eye candy for now. We need to actually use this new makeId parameter for the logic in the endpoints. For example, one would expect a GET to /api/Makes/1/Models to return all the vehicle models that belong to the make with id 1. But right now, all vehicle models are returned regardless. All other endpoints behave similarly, there’s no limit to the operations on the vehicle models. The given makeId is not taken into consideration at all.

Let’s update the ModelsController’s GetModels method (which is the one that handles the GET /api/Makes/{makeId}/Models endpoint) to behave like one would expect. It should look like this:

[HttpGet]
public async Task<ActionResult<IEnumerable<Model>>> GetModels([FromRoute] int makeId)
{
    var make = await _context.Makes.FindAsync(makeId);

    if (make == null)
    {
        return NotFound();
    }

    return await _context.Models.Where(m => m.MakeID == makeId).ToListAsync();
}

See how we’ve included a new parameter to the method: [FromRoute] int makeId. This [FromRoute] attribute is how we tell ASP.NET Core that this endpoint will use that makeId parameter coming from the URL route. Then, we use our DbContext to try to find the make that corresponds to the given identifier. This is done in _context.Makes.FindAsync(makeId). Then, if we can’t find the given make, we return a 404 Not Found HTTP status code with the return NotFound(); line. Finally, we query the models table for all the records whose make_id matches the given parameter. That’s done in the last line of the method.

We have access to the DbContext because it has been injected as a dependency into the controller via its constructor by the framework.

The official documentation is a great resource to learn about all the possibilities when querying data with EF Core.

Let’s update the GetModel method, which handles the GET /api/Makes/{makeId}/Models/{id} endpoint, similarly.

[HttpGet("{id}")]
-public async Task<ActionResult<Model>> GetModel(int id)
+public async Task<ActionResult<Model>> GetModel([FromRoute] int makeId, int id)
{
-   var model = await _context.Models.FindAsync(id);
+   var model = await _context.Models.FirstOrDefaultAsync(m =>
+       m.MakeID == makeId && m.ID == id
+   );

    if (model == null)
    {
        return NotFound();
    }

    return model;
}

We’ve once again included the makeId as a parameter to the method and modified the EF Core query to use both the make ID and the vehicle model ID when looking for the record.

And that’s the gist of it. Other methods would need to be updated similarly. The next section will include these methods in their final form, so I won’t go through each one of them here.

Using resource models as DTOs for controllers

Now, I did say at the beginning that we wanted the vehicle model endpoint to be a bit more abstract. Right now it’s operating directly over the EF Core entities and our table. As a result, creating new vehicle models via the POST /api/Makes/{makeId}/Models endpoint is a pain. Take a look at the Swagger UI request schema for that endpoint:

This is way too much. Let’s make it a little bit more user friendly by making it more abstract.

To do that, we will introduce what I like to call a Resource Model (or DTO, Data Transfer Object, or View Model). This is a class whose only purpose is to streamline the API contract of the endpoint by defining a set of fields that clients will use to make requests and interpret responses. Something that’s simpler than our actual database structure, but still captures all the information that’s important for our application. We will update the ModelsController so that it’s able to receive objects of this new class as requests, operate on them, translate them to our EF Core entities and actual database records, and return them as a response. The hope is that, by hiding the details of our database structure, we make it easier for clients to interact with our API.

So let’s create a new ResourceModels directory in our project’s root and add these two classes:

// ResourceModels/ModelSpecification.cs
namespace VehicleQuotes.ResourceModels
{
    public class ModelSpecification
    {
        public int ID { get; set; }
        public string Name { get; set; }

        public ModelSpecificationStyle[] Styles { get; set; }
    }
}

// ResourceModels/ModelSpecificationStyle.cs
namespace VehicleQuotes.ResourceModels
{
    public class ModelSpecificationStyle
    {
        public string BodyType { get; set; }
        public string Size { get; set; }

        public string[] Years { get; set; }
    }
}

Thanks to these two, instead of that mess from above, clients POSTing to /api/Makes/{makeId}/Models will be able to use a request body like this:

{
  "name": "string",
  "styles": [
    {
      "bodyType": "string",
      "size": "string",
      "years": [
        "string"
      ]
    }
  ]
}

Which is much simpler. We have the vehicle model name and an array of styles. Each style has a body type and a size, which we can specify by their names because those are unique keys. We don’t need their integer IDs (i.e. primary keys) in order to find to them. Then, each style has an array of strings that contain the years in which those styles are available for that model. The make is part of the URL already, so we don’t need to also specify it in the request payload.

Let’s update our ModelsController to use these Resource Models instead of the Model EF Core entity. Be sure to include the namespace where the Resource Models are defined by adding the following using statement: using VehicleQuotes.ResourceModels;. Now, let’s update the GetModels method (which handles the GET /api/Makes/{makeId}/Models endpoint) so that it looks like this:

[HttpGet]
// Return a collection of `ModelSpecification`s and expect a `makeId` from the URL.
public async Task<ActionResult<IEnumerable<ModelSpecification>>> GetModels([FromRoute] int makeId)
{
    // Look for the make identified by `makeId`.
    var make = await _context.Makes.FindAsync(makeId);

    // If we can't find the make, then we return a 404.
    if (make == null)
    {
        return NotFound();
    }

    // Build a query to fetch the relevant records from the `models` table and
    // build `ModelSpecification` with the data.
    var modelsToReturn = _context.Models
        .Where(m => m.MakeID == makeId)
        .Select(m => new ModelSpecification {
            ID = m.ID,
            Name = m.Name,
            Styles = m.ModelStyles.Select(ms => new ModelSpecificationStyle {
                BodyType = ms.BodyType.Name,
                Size = ms.Size.Name,
                Years = ms.ModelStyleYears.Select(msy => msy.Year).ToArray()
            }).ToArray()
        });

    // Execute the query and respond with the results.
    return await modelsToReturn.ToListAsync();
}

The first thing that we changed was the return type. Instead of Task<ActionResult<IEnumerable<Model>>>, the method now returns Task<ActionResult<IEnumerable<ModelSpecification>>>. We’re going to use our new Resource Models as these endpoints’ contract, so we need to make sure we are returning those. Next, we considerably changed the LINQ expression that searches the database for the vehicle model records we want. The filtering logic (given by the Where) is the same. That is, we’re still searching for vehicle models within the given make ID. What we changed was the projection logic in the Select. Our Action Method now returns a collection of ModelSpecification objects, so we updated the Select to produce such objects, based on the records from the models table that match our search criteria. We build ModelSpecifications using the data coming from models records and their related model_styles and model_style_years. Finally, we asynchronously execute the query to fetch the data from the database and return it.

Next, let’s move on to the GetModel method, which handles the GET /api/Makes/{makeId}/Models/{id} endpoint. This is what it should look like:

[HttpGet("{id}")]
// Return a `ModelSpecification`s and expect `makeId` and `id` from the URL.
public async Task<ActionResult<ModelSpecification>> GetModel([FromRoute] int makeId, [FromRoute] int id)
{
    // Look for the model specified by the given identifiers and also load
    // all related data that we care about for this method.
    var model = await _context.Models
        .Include(m => m.ModelStyles).ThenInclude(ms => ms.BodyType)
        .Include(m => m.ModelStyles).ThenInclude(ms => ms.Size)
        .Include(m => m.ModelStyles).ThenInclude(ms => ms.ModelStyleYears)
        .FirstOrDefaultAsync(m => m.MakeID == makeId && m.ID == id);

    // If we couldn't find it, respond with a 404.
    if (model == null)
    {
        return NotFound();
    }

    // Use the fetched data to construct a `ModelSpecification` to use in the response.
    return new ModelSpecification {
        ID = model.ID,
        Name = model.Name,
        Styles = model.ModelStyles.Select(ms => new ModelSpecificationStyle {
            BodyType = ms.BodyType.Name,
            Size = ms.Size.Name,
            Years = ms.ModelStyleYears.Select(msy => msy.Year).ToArray()
        }).ToArray()
    };
}

Same as before, we changed the return type of the method to be ModelSpecification. Then, we modified the query so that it loads all the related data for the Model entity via its navigation properties. That’s what the Include and ThenInclude calls do. We need this data loaded because we use it in the method’s return statement to build the ModelSpecification that will be included in the response. The logic to build it is very similar to that of the previous method.

You can learn more about the various available approaches for loading data with EF Core in the official documentation.

Next is the PUT /api/Makes/{makeId}/Models/{id} endpoint, handled by the PutModel method:

[HttpPut("{id}")]
// Expect `makeId` and `id` from the URL and a `ModelSpecification` from the request payload.
public async Task<IActionResult> PutModel([FromRoute] int makeId, int id, ModelSpecification model)
{
    // If the id in the URL and the request payload are different, return a 400.
    if (id != model.ID)
    {
        return BadRequest();
    }

    // Obtain the `models` record that we want to update. Include any related
    // data that we want to update as well.
    var modelToUpdate = await _context.Models
        .Include(m => m.ModelStyles)
        .FirstOrDefaultAsync(m => m.MakeID == makeId && m.ID == id);

    // If we can't find the record, then return a 404.
    if (modelToUpdate == null)
    {
        return NotFound();
    }

    // Update the record with what came in the request payload.
    modelToUpdate.Name = model.Name;

    // Build EF Core entities based on the incoming Resource Model object.
    modelToUpdate.ModelStyles = model.Styles.Select(style => new ModelStyle {
        BodyType = _context.BodyTypes.Single(bodyType => bodyType.Name == style.BodyType),
        Size = _context.Sizes.Single(size => size.Name == style.Size),

        ModelStyleYears = style.Years.Select(year => new ModelStyleYear {
            Year = year
        }).ToList()
    }).ToList();

    try
    {
        // Try saving the changes. This will run the UPDATE statement in the database.
        await _context.SaveChangesAsync();
    }
    catch (Microsoft.EntityFrameworkCore.DbUpdateException)
    {
        // If there's an error updating, respond accordingly.
        return Conflict();
    }

    // Finally return a 204 if everything went well.
    return NoContent();
}

The purpose of this endpoint is to update existing resources. So, it receives a representation of said resource as a parameter that comes from the request body. Before, it expected an instance of the Model entity, but now, we’ve changed it to receive a ModelSpecification. The rest of the method is your usual structure of first obtaining the record to update by the given IDs, then changing its values according to what came in as a parameter, and finally, saving the changes.

You probably get the idea by now: since the API is using the Resource Model, we need to change input and output values for the methods and run some logic to translate between Resource Model objects and Data Model objects that EF Core can understand so that it can perform its database operations.

That said, here’s what the PostModel Action Method, handler of the POST /api/Makes/{makeId}/Models endpoint, should look like:

[HttpPost]
// Return a `ModelSpecification`s and expect `makeId` from the URL and a `ModelSpecification` from the request payload.
public async Task<ActionResult<ModelSpecification>> PostModel([FromRoute] int makeId, ModelSpecification model)
{
    // First, try to find the make specified by the incoming `makeId`.
    var make = await _context.Makes.FindAsync(makeId);

    // Respond with 404 if not found.
    if (make == null)
    {
        return NotFound();
    }

    // Build out a new `Model` entity, complete with all related data, based on
    // the `ModelSpecification` parameter.
    var modelToCreate = new Model {
        Make = make,
        Name = model.Name,

        ModelStyles = model.Styles.Select(style => new ModelStyle {
            // Notice how we search both body type and size by their name field.
            // We can do that because their names are unique.
            BodyType = _context.BodyTypes.Single(bodyType => bodyType.Name == style.BodyType),
            Size = _context.Sizes.Single(size => size.Name == style.Size),

            ModelStyleYears = style.Years.Select(year => new ModelStyleYear {
                Year = year
            }).ToArray()
        }).ToArray()
    };

    // Add it to the DbContext.
    _context.Add(modelToCreate);

    try
    {
        // Try running the INSERTs.
        await _context.SaveChangesAsync();
    }
    catch (Microsoft.EntityFrameworkCore.DbUpdateException)
    {
        // Return accordingly if an error happens.
        return Conflict();
    }

    // Get back the autogenerated ID of the record we just INSERTed.
    model.ID = modelToCreate.ID;

    // Finally, return a 201 including a location header containing the newly
    // created resource's URL and the resource itself in the response payload.
    return CreatedAtAction(
        nameof(GetModel),
        new { makeId = makeId, id = model.ID },
        model
    );
}

All that should be pretty self explanatory by now. Moving on to the DeleteModel method which handles the DELETE /api/Makes/{makeId}/Models/{id} endpoint:

[HttpDelete("{id}")]
// Expect `makeId` and `id` from the URL.
public async Task<IActionResult> DeleteModel([FromRoute] int makeId, int id)
{
    // Try to find the record identified by the ids from the URL.
    var model = await _context.Models.FirstOrDefaultAsync(m => m.MakeID == makeId && m.ID == id);

    // Respond with a 404 if we can't find it.
    if (model == null)
    {
        return NotFound();
    }

    // Mark the entity for removal and run the DELETE.
    _context.Models.Remove(model);
    await _context.SaveChangesAsync();

    // Respond with a 204.
    return NoContent();
}

And that’s all for that controller. Hopefully that demonstrated what it looks like to have endpoints that operate using objects other than the EF Core entities. Fire up the app with dotnet run and explore the Swagger UI and you’ll see the changes that we’ve made reflected in there. Try it out. Try CRUDing some vehicle models. And don’t forget to take a look at our POST endpoint specification which looks much more manageable now:

Which means that you can send in something like this, for example:

{
    "name": "Corolla",
    "styles": [
        {
            "bodyType": "Sedan",
            "size": "Compact",
            "years": [ "2000", "2001" ]
        }
    ]
}

This will work assuming you’ve created at least one make to add the vehicle model to, as well as a body type whose name is Sedan and a size whose name is Compact.

There’s also a ModelExists method in that controller which we don’t need anymore. You can delete it.

Validation using built-in Data Annotations

Depending on how “creative” you were in the previous section when trying to CRUD models, you may have run into an issue or two regarding the data that’s allowed into our database. We solve that by implementing input validation. In ASP.NET Core, the easiest way to implement validation is via Data Annotation attributes on the entities or other objects that controllers receive as request payloads. So let’s see about adding some validation to our app. Since our ModelsController uses the ModelSpecification and ModelSpecificationStyle Resource Models to talk to clients, let’s start there. Here’s the diff:

+using System.ComponentModel.DataAnnotations;

namespace VehicleQuotes.ResourceModels
{
    public class ModelSpecification
    {
        public int ID { get; set; }
+       [Required]
        public string Name { get; set; }

+       [Required]
        public ModelSpecificationStyle[] Styles { get; set; }
    }
}

+using System.ComponentModel.DataAnnotations;

namespace VehicleQuotes.ResourceModels
{
    public class ModelSpecificationStyle
    {
+       [Required]
        public string BodyType { get; set; }
+       [Required]
        public string Size { get; set; }

+       [Required]
+       [MinLength(1)]
        public string[] Years { get; set; }
    }
}

And just like that, we get a good amount of functionality. We use the Required and MinLength attributes from the System.ComponentModel.DataAnnotations namespace to specify that some fields are required, and that our Years array needs to contain at least one element. When the app receives a request to the PUT or POST endpoints — which are the ones that expect a ModelSpecification as the payload — validation kicks in. If it fails, the action method is never executed and a 400 status code is returned as a response. Try POSTing to /api/Makes/{makeId}/Models with a payload that violates some of these rules to see for yourself. I tried for example sending this:

{
  "name": null,
  "styles": [
    {
      "bodyType": "Sedan",
      "size": "Full size",
      "years": []
    }
  ]
}

And I got back a 400 response with this payload:

{
  "type": "https://tools.ietf.org/html/rfc7231#section-6.5.1",
  "title": "One or more validation errors occurred.",
  "status": 400,
  "traceId": "00-0fd4f00eeb9f2f458ccefc180fcfba1c-79a618f13218394b-00",
  "errors": {
    "Name": [
      "The Name field is required."
    ],
    "Styles[0].Years": [
      "The field Years must be a string or array type with a minimum length of '1'."
    ]
  }
}

Pretty neat, huh? With minimal effort, we have some basic validation rules in place and a pretty usable response for when errors occur.

To learn more about model validation, including all the various validation attributes included in the framework, check the official documentation: Model validation and System.ComponentModel.DataAnnotations Namespace.

Validation using custom attributes

Of course, the framework is never going to cover all possible validation scenarios with the built-in attributes. Case in point, it’d be great to validate that the Years array contains values that look like actual years. That is, four-character, digit-only strings. There are no validation attributes for that. So, we need to create our own. Let’s add this file into a new Validations directory:

// Validations/ContainsYearsAttribute.cs
using System;
using System.ComponentModel.DataAnnotations;
using System.Linq;
using System.Runtime.CompilerServices;

namespace VehicleQuotes.Validation
{
    // In the .NET Framework, attribute classes need to have their name suffixed with the word "Attribute".
    // Validation attributes need to inherit from `System.ComponentModel.DataAnnotations`'s `ValidationAttribute` class
    // and override the `IsValid` method.
    public class ContainsYearsAttribute : ValidationAttribute
    {
        private string propertyName;

        // This constructor is called by the framework when the attribute is applied to some member. In this specific
        // case, we define a `propertyName` parameter annotated with a `CallerMemberName` attribute. This makes it so
        // the framework sends in the name of the member to which our `ContainsYears` attribute is applied to.
        // We store the value to use it later when constructing our validation error message.
        // Check https://docs.microsoft.com/en-us/dotnet/api/system.runtime.compilerservices.callermembernameattribute?view=net-5.0
        // for more info on `CallerMemberName`.
        public ContainsYearsAttribute([CallerMemberName] string propertyName = null)
        {
            this.propertyName = propertyName;
        }

        // This method is called by the framework during validation. `value` is the actual value of the field that this
        // attribute will validate.
        protected override ValidationResult IsValid(object value, ValidationContext validationContext)
        {
            // By only applying the validation checks when the value is not null, we make it possible for this
            // attribute to work on optional fields. In other words, this attribute will skip validation if there is no
            // value to validate.
            if (value != null)
            {
                // Check if all the elements of the string array are valid years. Check the `IsValidYear` method below
                // to see what checks are applied for each of the array elements.
                var isValid = (value as string[]).All(IsValidYear);

                if (!isValid)
                {
                    // If not, return an error.
                    return new ValidationResult(GetErrorMessage());
                }
            }

            // Return a successful validation result if no errors were detected.
            return ValidationResult.Success;
        }

        // Determines if a given value is valid by making sure it's not null, nor empty, that its length is 4 and that
        // all its characters are digits.
        private bool IsValidYear(string value) =>
            !String.IsNullOrEmpty(value) && value.Length == 4 && value.All(Char.IsDigit);

        // Builds a user friendly error message which includes the name of the field that this validation attribute has
        // been applied to.
        private string GetErrorMessage() =>
            $"The {propertyName} field must be an array of strings containing four digits.";
    }
}

Check the comments in the code for more details into how that class works. Then, we apply our custom attribute to our ModelSpecificationStyle class in the same way that we applied the built in ones:

using System.ComponentModel.DataAnnotations;
+using VehicleQuotes.Validation;

namespace VehicleQuotes.ResourceModels
{
    public class ModelSpecificationStyle
    {
        // ...

        [Required]
        [MinLength(1)]
+       [ContainsYears]
        public string[] Years { get; set; }
    }
}

Now do a dotnet run and try to POST to /api/Makes/{makeId}/Models this payload:

{
  "name": "Rav4",
  "styles": [
    {
      "bodyType": "SUV",
      "size": "Mid size",
      "years": [ "not_a_year" ]
    }
  ]
}

That should make the API respond with this:

{
  "type": "https://tools.ietf.org/html/rfc7231#section-6.5.1",
  "title": "One or more validation errors occurred.",
  "status": 400,
  "traceId": "00-9980325f3e388f48a5975ef382d5b137-2d55da1bb9613e4f-00",
  "errors": {
    "Styles[0].Years": [
      "The Years field must be an array of strings containing four numbers."
    ]
  }
}

That’s our custom validation attribute doing its job.

There’s another aspect that we could validate using a custom validation attribute. What happens if we try to POST a payload with a body type or size that doesn’t exist? These queries from the PostModel method would throw an InvalidOperationException:

BodyType = _context.BodyTypes.Single(bodyType => bodyType.Name == style.BodyType)

and

Size = _context.Sizes.Single(size => size.Name == style.Size)

They do so because we used the Single method, which is designed like that. It tries to find a body type or size whose name is the given value, can’t find it, and thus, throws an exception.

If, for example, we wanted not-founds to return null, we could have used SingleOrDefault instead.

This unhandled exception results in a response that’s quite unbecoming:

So, to prevent that exception and control the error messaging, we need a couple of new validation attributes that go into the body_types and sizes tables and check if the given values exist. Here’s what one would look like:

// Validations/VehicleBodyTypeAttribute.cs
using System;
using System.ComponentModel.DataAnnotations;
using Microsoft.EntityFrameworkCore;
using System.Linq;

namespace VehicleQuotes.Validation
{
    public class VehicleBodyTypeAttribute : ValidationAttribute
    {
        protected override ValidationResult IsValid(object value, ValidationContext validationContext)
        {
            if (value == null) return ValidationResult.Success;

            var dbContext = validationContext.GetService(typeof(VehicleQuotesContext)) as VehicleQuotesContext;

            var bodyTypes = dbContext.BodyTypes.Select(bt => bt.Name).ToList();

            if (!bodyTypes.Contains(value))
            {
                var allowed = String.Join(", ", bodyTypes);
                return new ValidationResult(
                    $"Invalid vehicle body type {value}. Allowed values are {allowed}."
                );
            }

            return ValidationResult.Success;
        }
    }
}

You can find the other one here: Validations/VehicleSizeAttribute.cs.

These two are very similar to one another. The most interesting part is how we use the IsValid method’s second parameter (ValidationContext) to obtain an instance of VehicleQuotesContext that we can use to query the database. The rest should be pretty self-explanatory. These attributes are classes that inherit from System.ComponentModel.DataAnnotations’s ValidationAttribute and implement the IsValid method. The method then checks that the value under scrutiny exists in the corresponding table and if it does not, raises a validation error. The validation error includes a list of all allowed values. They can be applied to our ModelSpecificationStyle class like so:

// ...
namespace VehicleQuotes.ResourceModels
{
    public class ModelSpecificationStyle
    {
        [Required]
+       [VehicleBodyType]
        public string BodyType { get; set; }

        [Required]
+       [VehicleSize]
        public string Size { get; set; }

        //...
    }
}

Now, a request like this:

{
  "name": "Rav4",
  "styles": [
    {
      "bodyType": "not_a_body_type",
      "size": "Mid size",
      "years": [ "2000" ]
    }
  ]
}

Produces a response like this:

{
  "type": "https://tools.ietf.org/html/rfc7231#section-6.5.1",
  "title": "One or more validation errors occurred.",
  "status": 400,
  "traceId": "00-9ad59a7aff60944ab54c19a73be73cc7-eeabafe03df74e40-00",
  "errors": {
    "Styles[0].BodyType": [
      "Invalid vehicle body type not_a_body_type. Allowed values are Coupe, Sedan, Convertible, Hatchback, SUV, Truck."
    ]
  }
}

Implementing endpoints for quote rules and overrides

At this point we’ve explored many of the most common features available to us for developing Web APIs. So much so that implementing the next two pieces of functionality for our app doesn’t really introduce any new concepts. So, I wont discuss that here in great detail.

Feel free to browse the source code on GitHub if you want though. These are the relevant files:

The FeatureTypeAttribute class is interesting in that it provides another example of a validation attribute. This time is one that makes sure the value being validated is included in an array of strings that’s defined literally in the code.

Other than that, it’s all stuff we’ve already covered: models, migrations, scaffolding controllers, custom routes, resource models, etc.

If you are following along, be sure to add those files and run a dotnet ef database update to apply the migration.

Implementing the quote model

Let’s now start implementing the main capability of our app: calculating quotes for vehicles. Let’s start with the Quote entity. This is what the new Models/Quote.cs file containing the entity class will look like:

// Models/Quote.cs
using System;
using System.ComponentModel.DataAnnotations.Schema;

namespace VehicleQuotes.Models
{
    public class Quote
    {
        public int ID { get; set; }

        // Directly tie this quote record to a specific vehicle that we have
        // registered in our db, if we have it.
        public int? ModelStyleYearID { get; set; }

        // If we don't have the specific vehicle in our db, then store the
        // vehicle model details independently.
        public string Year { get; set; }
        public string Make { get; set; }
        public string Model { get; set; }
        public int BodyTypeID { get; set; }
        public int SizeID { get; set; }

        public bool ItMoves { get; set; }
        public bool HasAllWheels { get; set; }
        public bool HasAlloyWheels { get; set; }
        public bool HasAllTires { get; set; }
        public bool HasKey { get; set; }
        public bool HasTitle { get; set; }
        public bool RequiresPickup { get; set; }
        public bool HasEngine { get; set; }
        public bool HasTransmission { get; set; }
        public bool HasCompleteInterior { get; set; }

        public int OfferedQuote { get; set; }
        public string Message { get; set; }
        public DateTime CreatedAt { get; set; }

        public ModelStyleYear ModelStyleYear { get; set; }

        public BodyType BodyType { get; set; }
        public Size Size { get; set; }
    }
}

This should be pretty familiar by now. It’s a plain old class that defines a number of properties, one for each of the fields in the resulting table and a few navigation properties that serve to access related data.

The only aspect worth noting is that we’ve defined the ModelStyleYearID property as a nullable integer (with int?). This is because, like we discussed at the beginning, the foreign key from quotes to vehicle_style_years is actually optional. The reason being that we may receive a quote request for a vehicle that we don’t have registered in our database. We need to be able to support quoting those vehicles too, so if we don’t have the requested vehicle registered, then that foreign key will stay unpopulated and we’ll rely on the other fields (i.e. Year, Make, Model, BodyTypeID and SizeID) to identify the vehicle and calculate the quote for it.

Using Dependency Injection

So far we’ve been putting a lot of logic in our controllers. That’s generally not ideal, but fine as long as the logic is simple. The problem is that a design like that can quickly become a hindrance for maintainability and testing as our application grows more complex. For the logic that calculates a quote, we’d be better served by implementing it in its own class, outside of the controller that should only care about defining endpoints and handling HTTP concerns. Then, the controller can be given access to that class and delegate to it all the quote calculation logic. Thankfully, ASP.NET Core includes an IoC container by default, which allows us to use Dependency Injection to solve these kinds of problems. Let’s see what that looks like.

For working with quotes, we want to offer two endpoints:

A POST api/Quotes that captures the vehicle information, calculates the quote, keeps record of the request, and responds with the calculated value.
A GET api/Quotes that returns all the currently registered quotes in the system.

Using the Dependency Injection capabilities, a controller that implements those two could look like this:

using System.Collections.Generic;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Mvc;
using VehicleQuotes.ResourceModels;
using VehicleQuotes.Services;

namespace VehicleQuotes.Controllers
{
    [Route("api/[controller]")]
    [ApiController]
    public class QuotesController : ControllerBase
    {
        private readonly QuoteService _service;

        // When intiating the request processing logic, the framework recognizes
        // that this controller has a dependency on QuoteService and expects an
        // instance of it to be injected via the constructor. The framework then
        // does what it needs to do in order to provide that dependency.
        public QuotesController(QuoteService service)
        {
            _service = service;
        }

        // GET: api/Quotes
        [HttpGet]
        // This method returns a collection of a new resource model instead of just the `Quote` entity directly.
        public async Task<ActionResult<IEnumerable<SubmittedQuoteRequest>>> GetAll()
        {
            // Instead of directly implementing the logic in this method, we call on
            // the service class and let it take care of the rest.
            return await _service.GetAllQuotes();
        }

        // POST: api/Quotes
        [HttpPost]
        // This method receives as a paramater a `QuoteRequest` of just the `Quote` entity directly.
        // That way callers of this endpoint don't need to be exposed to the details of our data model implementation.
        public async Task<ActionResult<SubmittedQuoteRequest>> Post(QuoteRequest request)
        {
            // Instead of directly implementing the logic in this method, we call on
            // the service class and let it take care of the rest.
            return await _service.CalculateQuote(request);
        }
    }
}

As you can see, we’ve once again opted to abstract away clients from the implementation details of our data model and used Resource Models for the API contract instead of the Quote entity directly. We have one for input data that’s called QuoteRequest and another one for output: SubmittedQuoteRequest. Not very remarkable by themselves, but feel free to explore the source code in the GitHub repo.

This controller has a dependency on QuoteService, which it uses to perform all of the necessary logic. This class is not defined yet so let’s do that next:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using Microsoft.EntityFrameworkCore;
using VehicleQuotes.Models;
using VehicleQuotes.ResourceModels;

namespace VehicleQuotes.Services
{
    public class QuoteService
    {
        private readonly VehicleQuotesContext _context;

        // This constructor defines a dependency on VehicleQuotesContext, similar to most of our controllers.
        // Via the built in dependency injection features, the framework makes sure to provide this parameter when
        // creating new instances of this class.
        public QuoteService(VehicleQuotesContext context)
        {
            _context = context;
        }

        // This method takes all the records from the `quotes` table and constructs `SubmittedQuoteRequest`s with them.
        // Then returns that as a list.
        public async Task<List<SubmittedQuoteRequest>> GetAllQuotes()
        {
            var quotesToReturn = _context.Quotes.Select(q => new SubmittedQuoteRequest
            {
                ID = q.ID,
                CreatedAt = q.CreatedAt,
                OfferedQuote = q.OfferedQuote,
                Message = q.Message,

                Year = q.Year,
                Make = q.Make,
                Model = q.Model,
                BodyType = q.BodyType.Name,
                Size = q.Size.Name,

                ItMoves = q.ItMoves,
                HasAllWheels = q.HasAllWheels,
                HasAlloyWheels = q.HasAlloyWheels,
                HasAllTires = q.HasAllTires,
                HasKey = q.HasKey,
                HasTitle = q.HasTitle,
                RequiresPickup = q.RequiresPickup,
                HasEngine = q.HasEngine,
                HasTransmission = q.HasTransmission,
                HasCompleteInterior = q.HasCompleteInterior,
            });

            return await quotesToReturn.ToListAsync();
        }

        // This method takes an incoming `QuoteRequest` and calculates a quote based on the vehicle described by it.
        // To calculate this quote, it looks for any overrides before trying to use the currently existing rules defined
        // in the `quote_rules` table. It also stores a record on the `quotes` table with all the incoming data and the
        // quote calculation result. It returns back the quote value as well as a message explaining the conditions of
        // the quote.
        public async Task<SubmittedQuoteRequest> CalculateQuote(QuoteRequest request)
        {
            var response = this.CreateResponse(request);
            var quoteToStore = await this.CreateQuote(request);
            var requestedModelStyleYear = await this.FindModelStyleYear(request);
            QuoteOverride quoteOverride = null;

            if (requestedModelStyleYear != null)
            {
                quoteToStore.ModelStyleYear = requestedModelStyleYear;

                quoteOverride = await this.FindQuoteOverride(requestedModelStyleYear);

                if (quoteOverride != null)
                {
                    response.OfferedQuote = quoteOverride.Price;
                }
            }

            if (quoteOverride == null)
            {
                response.OfferedQuote = await this.CalculateOfferedQuote(request);
            }

            if (requestedModelStyleYear == null)
            {
                response.Message = "Offer subject to change upon vehicle inspection.";
            }

            quoteToStore.OfferedQuote = response.OfferedQuote;
            quoteToStore.Message = response.Message;

            _context.Quotes.Add(quoteToStore);
            await _context.SaveChangesAsync();

            response.ID = quoteToStore.ID;
            response.CreatedAt = quoteToStore.CreatedAt;

            return response;
        }

        // Creates a `SubmittedQuoteRequest`, initialized with default values, using the data from the incoming
        // `QuoteRequest`. `SubmittedQuoteRequest` is what gets returned in the response payload of the quote endpoints.
        private SubmittedQuoteRequest CreateResponse(QuoteRequest request)
        {
            return new SubmittedQuoteRequest
            {
                OfferedQuote = 0,
                Message = "This is our final offer.",

                Year = request.Year,
                Make = request.Make,
                Model = request.Model,
                BodyType = request.BodyType,
                Size = request.Size,

                ItMoves = request.ItMoves,
                HasAllWheels = request.HasAllWheels,
                HasAlloyWheels = request.HasAlloyWheels,
                HasAllTires = request.HasAllTires,
                HasKey = request.HasKey,
                HasTitle = request.HasTitle,
                RequiresPickup = request.RequiresPickup,
                HasEngine = request.HasEngine,
                HasTransmission = request.HasTransmission,
                HasCompleteInterior = request.HasCompleteInterior,
            };
        }

        // Creates a `Quote` based on the data from the incoming `QuoteRequest`. This is the object that gets eventually
        // stored in the database.
        private async Task<Quote> CreateQuote(QuoteRequest request)
        {
            return new Quote
            {
                Year = request.Year,
                Make = request.Make,
                Model = request.Model,
                BodyTypeID = (await _context.BodyTypes.SingleAsync(bt => bt.Name == request.BodyType)).ID,
                SizeID = (await _context.Sizes.SingleAsync(s => s.Name == request.Size)).ID,

                ItMoves = request.ItMoves,
                HasAllWheels = request.HasAllWheels,
                HasAlloyWheels = request.HasAlloyWheels,
                HasAllTires = request.HasAllTires,
                HasKey = request.HasKey,
                HasTitle = request.HasTitle,
                RequiresPickup = request.RequiresPickup,
                HasEngine = request.HasEngine,
                HasTransmission = request.HasTransmission,
                HasCompleteInterior = request.HasCompleteInterior,

                CreatedAt = DateTime.Now
            };
        }

        // Tries to find a registered vehicle that matches the one for which the quote is currently being requested.
        private async Task<ModelStyleYear> FindModelStyleYear(QuoteRequest request)
        {
            return await _context.ModelStyleYears.FirstOrDefaultAsync(msy =>
                msy.Year == request.Year &&
                msy.ModelStyle.Model.Make.Name == request.Make &&
                msy.ModelStyle.Model.Name == request.Model &&
                msy.ModelStyle.BodyType.Name == request.BodyType &&
                msy.ModelStyle.Size.Name == request.Size
            );
        }

        // Tries to find an override for the vehicle for which the quote is currently being requested.
        private async Task<QuoteOverride> FindQuoteOverride(ModelStyleYear modelStyleYear)
        {
            return await _context.QuoteOverides
                .FirstOrDefaultAsync(qo => qo.ModelStyleYear == modelStyleYear);
        }

        // Uses the rules stored in the `quote_rules` table to calculate how much money to offer for the vehicle
        // described in the incoming `QuoteRequest`.
        private async Task<int> CalculateOfferedQuote(QuoteRequest request)
        {
            var rules = await _context.QuoteRules.ToListAsync();

            // Given a vehicle feature type, find a rule that applies to that feature type and has the value that
            // matches the condition of the incoming vehicle being quoted.
            Func<string, QuoteRule> theMatchingRule = featureType =>
                rules.FirstOrDefault(r =>
                    r.FeatureType == featureType &&
                    r.FeatureValue == request[featureType]
                );

            // For each vehicle feature that we care about, sum up the the monetary values of all the rules that match
            // the given vehicle condition.
            return QuoteRule.FeatureTypes.All
                .Select(theMatchingRule)
                .Where(r => r != null)
                .Sum(r => r.PriceModifier);
        }
    }
}

Finally, we need to tell the framework that this class is available for Dependency Injection. Similarly to how we did with our VehicleQuotesContext, we do so in the Startup.cs file’s ConfigureServices method. Just add this line at the top:

services.AddScoped<Services.QuoteService>();

The core tenet of Inversion of Control is to depend on abstractions, not on implementations. So ideally, we would not have our controller directly call for a QuoteService instance. Instead, we would have it reference an abstraction, e.g. an interface like IQuoteService. The statement on Startup.cs would then look like this instead: services.AddScoped<Services.IQuoteService, Services.QuoteService>();.

This is important because it would allow us to unit test the component that depends on our service class (i.e. the controller in this case) by passing it a mock object — one that also implements IQuoteService but does not really implement all the functionality of the actual QuoteService class. Since the controller only knows about the interface (that is, it “depends on an abstraction”), the actual object that we give it as a dependency doesn’t matter to it, as long as it implements that interface. This ability to inject mocks as dependencies is invaluable during testing. Testing is beyond the scope of this article though, so I’ll stick with the simpler approach with a static dependency on a concrete class. Know that this is not a good practice when it comes to actual production systems.

And that’s all it takes. Once you add a few rules via POST /api/QuoteRules, you should be able to get some vehicles quoted with POST /api/Quotes. And also see what the system has stored via GET /api/Quotes.

And that’s all the functionality that we set out to build into our REST API! There are a few other neat things that I thought I’d include though.

Adding seed data for lookup tables

Our vehicle size and body type data isn’t meant to really change much. In fact, we could even preload that data when our application starts. EF Core provides a data seeding feature that we can access via configurations on the DbContext itself. For our case, we could add this method to our VehicleQuotesContext:

protected override void OnModelCreating(ModelBuilder modelBuilder)
{
    modelBuilder.Entity<Size>().HasData(
        new Size { ID = 1, Name = "Subcompact" },
        new Size { ID = 2, Name = "Compact" },
        new Size { ID = 3, Name = "Mid Size" },
        new Size { ID = 5, Name = "Full Size" }
    );

    modelBuilder.Entity<BodyType>().HasData(
        new BodyType { ID = 1, Name = "Coupe" },
        new BodyType { ID = 2, Name = "Sedan" },
        new BodyType { ID = 3, Name = "Hatchback" },
        new BodyType { ID = 4, Name = "Wagon" },
        new BodyType { ID = 5, Name = "Convertible" },
        new BodyType { ID = 6, Name = "SUV" },
        new BodyType { ID = 7, Name = "Truck" },
        new BodyType { ID = 8, Name = "Mini Van" },
        new BodyType { ID = 9, Name = "Roadster" }
    );
}

OnModelCreating is a hook that we can define to run some code at the time the model is being created for the first time. Here, we’re using it to seed some data. In order to apply that, a migration needs to be created and executed. If you’ve added some data to the database, be sure to wipe it before running the migration so that we don’t run into unique constraint violations. Here are the migrations:

$ dotnet ef migrations add AddSeedDataForSizesAndBodyTypes

$ dotnet ef database update

After that’s done, it no longer makes sense to allow creating, updating, deleting and fetching individual sizes and body types, so I would delete those endpoints from the respective controllers.

There are other options for data seeding in EF Core. Take a look: Data Seeding.

Improving the Swagger UI via XML comments

Our current auto-generated Swagger UI is pretty awesome. Especially considering that we got it for free. It’s a little lacking when it comes to more documentation about specific endpoint summaries or expected responses. The good news is that there’s a way to leverage C# XML Comments in order to improve the Swagger UI.

We can add support for that by configuring our project to produce, at build time, an XML file with the docs that we write. In order to do so, we need to update the VehicleQuotes.csproj like this:

<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <!-- ... -->
+   <GenerateDocumentationFile>true</GenerateDocumentationFile>
+   <NoWarn>$(NoWarn);1591</NoWarn>
  </PropertyGroup>

  <!-- ... -->
</Project>

GenerateDocumentationFile is the flag that tells the .NET 5 build tools to generate the documentation file. The NoWarn element prevents our build output from getting cluttered with a lot of warnings saying that some classes and methods are not properly documented. We don’t want that because we just want to write enough documentation for the Swagger UI. And that includes only the controllers.

You can run dotnet build and look for the new file in bin/Debug/net5.0/VehicleQuotes.xml.

Then, we need to update Startup.cs. First we need to add the following using statements:

using System.IO;
using System.Reflection;

And add the following code to the ConfigureServices method on Startup.cs:

public void ConfigureServices(IServiceCollection services)
{
    // ...

    services.AddSwaggerGen(c =>
    {
        c.SwaggerDoc("v1", new OpenApiInfo { Title = "VehicleQuotes", Version = "v1" });

+       c.IncludeXmlComments(
+           Path.Combine(
+               AppContext.BaseDirectory,
+               $"{Assembly.GetExecutingAssembly().GetName().Name}.xml"
+           )
        );
    });

    // ...
}

This makes it so the SwaggerGen service knows to look for the XML documentation file when building up the Open API specification file used for generating the Swagger UI.

Now that all of that is set up, we can actually write some XML comments and attributes that will enhance our Swagger UI. As an example, put this on top of ModelsController’s Post method:

/// <summary>
/// Creates a new vehicle model for the given make.
/// </summary>
/// <param name="makeId">The ID of the vehicle make to add the model to.</param>
/// <param name="model">The data to create the new model with.</param>
/// <response code="201">When the request is valid.</response>
/// <response code="404">When the specified vehicle make does not exist.</response>
/// <response code="409">When there's already another model in the same make with the same name.</response>
[HttpPost]
[ProducesResponseType(StatusCodes.Status201Created)]
[ProducesResponseType(StatusCodes.Status404NotFound)]
[ProducesResponseType(StatusCodes.Status409Conflict)]
public async Task<ActionResult<ModelSpecification>> Post([FromRoute] int makeId, ModelSpecification model)
{
    // ...
}

Then, the Swagger UI now looks like this for this endpoint:

Configuring the app via settings files and environment variables

Another aspect that’s important to web applications is having them be configurable via things like configuration files or environment variables. The framework already has provision for this, we just need to use it. I’m talking about the appsettings files.

We have two of them created for us by default: appsettings.json which is applied in all environments, and appsettings.Development.json that is applied only under development environments. The environment is given by the ASPNETCORE_ENVIRONMENT environment variable, and it can be set to either Development, Staging, or Production by default. That means that if we had, for example, an appsettings.Staging.json file, the settings defined within would be loaded if the ASPNETCORE_ENVIRONMENT environment variable were set to Staging. You get the idea.

You can learn more about configuration and environments in the official documentation.

Anyway, let’s add a new setting on appsettings.json:

{
  // ...
+ "DefaultOffer": 77
}

We’ll use this setting to give default offers when we’re not able to calculate appropriate quotes for vehicles. This can happen if we don’t have rules, or if the ones we have don’t match any of the incoming vehicle features or if for some other reason the final sum ends up in zero or negative number. We can use this setting in our QuoteService like so:

// ...
+using Microsoft.Extensions.Configuration;

namespace VehicleQuotes.Services
{
    public class QuoteService
    {
        // ...
+       private readonly IConfiguration _configuration;

-       public QuoteService(VehicleQuotesContext context)
+       public QuoteService(VehicleQuotesContext context, IConfiguration configuration)
        {
            _context = context;
+           _configuration = configuration;
        }

        // ...

        public async Task<SubmittedQuoteRequest> CalculateQuote(QuoteRequest request)
        {
            // ...

+           if (response.OfferedQuote <= 0)
+           {
+               response.OfferedQuote = _configuration.GetValue<int>("DefaultOffer", 0);
+           }

            quoteToStore.OfferedQuote = response.OfferedQuote;

            // ...
        }

        // ...
    }
}

Here, we’ve added a new parameter to the constructor to specify that VehicleQuotesContext has a dependency on IConfiguration. This prompts the framework to provide an instance of that when instantiating the class. We can use that instance to access the settings that we defined in the appsettings.json file via its GetValue method, like I demonstrated above.

The value of the settings in appsettings.json can be overridden by environment variables as well. On Linux, for example, we can run the app and set an environment value with a line like this:

$ DefaultOffer=123 dotnet run

This will make the application use 123 instead of 77 when it comes to the DefaultOffer setting. This flexibility is great from a DevOps perspective. And we had to do minimal work in order to get that going.

That’s all for now

And that’s it! In this article we’ve gone through many of the features offered in .NET 5, ASP.NET Core, and Entity Framework Core to support some of the most common use cases when it comes to developing Web API applications.

We’ve installed .NET 5 and created an ASP.NET Core Web API project with EF Core and a few bells and whistles, created controllers to support many different endpoints, played a little bit with routes and response codes, created and built upon a data model and updated a database via entities and migrations, implemented database constraints using unique indexes, implemented input validation using both built-in and custom validation attributes, implemented resource models as DTOs for defining the contract of some of our API endpoints, tapped into the built-in dependency injection capabilities, explored and improved the auto-generated Swagger UI, added seed data for our database, learned about configuration via settings files and environment variables.

.NET 5 is looking great.

Database Design: Using Composite Keys

2021-05-20T00:00:00+00:00

Photo by Chuttersnap

Whether to use single-column or composite keys is another long-debated subject of database design. I previously wrote to support using natural keys and here I want to make good arguments for using composite keys.

Single-column vs. Composite

Single-column keys are widely used nowadays. I wouldn’t be surprised if many developers today don’t even think database design with composite keys is possible, even though they were essential in the beginning. Relational databases make no assumption that the keys must be composed of a single column.

Let’s see the composite keys with the corporate database example again. First, we’d need departments and employees:

CREATE TABLE departments (
  department_id text NOT NULL,
  department_location text NOT NULL,

  PRIMARY KEY (department_id)
);

CREATE TABLE employees (
  username text NOT NULL,
  department_id text NOT NULL,

  PRIMARY KEY (username),
  FOREIGN KEY (department_id) REFERENCES departments
);

Then our database grows, and we need to split the departments into multiple teams. Here’s what they’ll look like:

| id | department  | team           | members |
| -- | ----------- | -------------- | ------- |
|  1 | sysadmin    | infrastructure |       5 |
|  2 | sysadmin    | internal_tools |       3 |
|  3 | development | internal_tools |       4 |
|  4 | development | web_site       |       8 |

As you noticed there are 2 teams named internal_tools, so we cannot use this as the primary key column. We can add a surrogate auto-increment column to use as the primary key, or make the department and team the primary key. Let’s go with the surrogate key option first to demonstrate the problem:

CREATE TABLE teams (
  team_id int NOT NULL GENERATED ALWAYS AS IDENTITY,
  department_id text NOT NULL,
  team_name text NOT NULL
  team_members int NOT NULL,

  PRIMARY KEY (team_id),
  UNIQUE (department_id, team_name),
  FOREIGN KEY (department_id) REFERENCES departments
);

As you noticed, we used the surrogate column as the primary key, and added an additional unique index to ensure the team name to be unique in with the department. Now, let’s relate the employees with the teams:

ALTER TABLE employees
  ADD COLUMN team_id int NOT NULL,
  ADD FOREIGN KEY (team_id) REFERENCES teams;

Now, we know both the department and the team of an employee, but the problem is that they can point to inconsistent rows. For example, I can INSERT myself as an employee:

| username | department_id | team_id |
| -------- | ------------- | ------- |
| hasegeli | sysadmin      |       3 |

team_id 3 is in development department, so now you’d never know if I am in the sysadmin or development department. This is a very common source of data integrity problems in the databases. Applications have no good option to handle this. They would typically crash or sometimes show the employee in one department and sometimes in the other.

You cannot easily add a constraint to the database to prevent this. The best option would be to remove the department_id when the team_id is added as we know the department of the team anyway, but this option is expensive and not always possible, for example when the team_id can be NULL.

Relations with Composite Keys

Now, let’s create the teams table with a composite key:

DROP TABLE teams CASCADE;

CREATE TABLE teams (
  department_id text NOT NULL,
  team_id text NOT NULL,
  team_members int NOT NULL,

  PRIMARY KEY (department_id, team_id),
  FOREIGN KEY (department_id) REFERENCES departments
);

ALTER TABLE employees
  ALTER COLUMN team_id TYPE text,
  ADD FOREIGN KEY (department_id, team_id) REFERENCES teams;

With this method, we ensure data integrity and don’t need to disturb the existing users of the employees table while adding the teams. They can still reliably use the department_id column. We can also set the team_id as NULL and still maintain the data integrity.

Ease of Change

As we have already seen, one of the benefits of using composite keys is to respond to database model changes easier and without compromising data integrity. This becomes relevant in many real world scenarios. For example, let’s add the employees to rooms which must belong to the same department:

CREATE TABLE rooms (
  department_id text NOT NULL,
  room_id text NOT NULL,
  room_location text NOT NULL,

  PRIMARY KEY (department_id, room_id),
  FOREIGN KEY (department_id) REFERENCES departments
);

ALTER TABLE employees
  ADD COLUMN room_id text,
  ADD FOREIGN KEY (department_id, room_id) REFERENCES rooms;

Nothing special needed to be done in here, and it is often easy to respond to more complicated change requests. Composite keys play well with database constraints. For example, let’s add an employee rank which needs to be unique in every department:

ALTER TABLE employees
  ADD COLUMN employee_rank int,
  ADD CHECK (employee_rank > 0),
  ADD UNIQUE (department_id, employee_rank);

This was so easy because we have the department_id in here. Now, let’s imagine a more complicated change request. We need to divide the departments into sections:

CREATE TABLE sections (
  department_id text NOT NULL,
  section_id text NOT NULL,
  section_location text NOT NULL,

  PRIMARY KEY (department_id, section_id),
  FOREIGN KEY (department_id) REFERENCES departments
);

ALTER TABLE teams
  ADD COLUMN section_id text NOT NULL,
  DROP CONSTRAINT teams_pkey CASCADE,
  ADD PRIMARY KEY (department_id, section_id, team_id),
  ADD FOREIGN KEY (epartment_id, section_id) REFERENCES sections;

ALTER TABLE employees
  ADD COLUMN section_id text NOT NULL,
  ADD FOREIGN KEY (department_id, section_id, team_id) REFERENCES teams;

As you see, we can make this happen with minimal impact to the users of the database. As a final practice, let’s reserve employee ranks between 1 and 10 for the main section of every department:

ALTER TABLE employees
  ADD CHECK (section_id = 'main' OR employee_rank > 10);

Querying

Another advantage of using composite keys is to have more possibility for joining of tables. For example using the tables we created we can join employees to sections without using the teams tables. This would not be possible if we had used single-column keys everywhere.

Join conditions get complicated with composite keys. The USING clause helps. To demonstrate, let’s join all of the tables we created so far:

SELECT *
  FROM employees
    JOIN deparments USING (department_id)
    JOIN sections USING (department_id, section_id)
    JOIN rooms USING (department_id, room_id)
    JOIN teams USING (department_id, section_id, team_id);

The USING clause only works if you name the columns the same on all tables. I recommend doing so.

Another advantage of USING clause is to eliminate duplicate columns on the result set. If you run this query, you would not see the department_id column repeated 5 times.

Performance Considerations

One disadvantage of using composite keys is to store more data on tables as references. You would also need more space for the indexes as the reference columns often need to be indexed. However storage is the cheapest of resources, and the performance advantages easily outweigh the extra storage.

The main performance advantage of using composite keys is eliminating the need for many joins as mentioned before. However, when you do need to join many tables, the query planner would have many different paths. It’s the query planners’ strong suit to find the best join order. Composite keys allow them to come up with better plans in many scenarios. To demonstrate this, let’s get our join-all-tables query and add some WHERE conditions:

SELECT *
  FROM employees
    JOIN deparments USING (department_id)
    JOIN sections USING (department_id, section_id)
    JOIN rooms USING (department_id, room_id);
    JOIN teams USING (department_id, section_id, team_id)
  WHERE username LIKE 'a%' AND
    department_location LIKE 'b%' AND
    section_location LIKE 'c%' AND
    room_location LIKE 'd%';
    team_members > 3;

Now we added 5 conditions using 5 columns on 5 different tables. The query planner can estimate which conditions are more selective and plan to join the tables from the smaller one to bigger one.

Enumerated Types in Rails and PostgreSQL

2021-04-29T00:00:00+00:00

Photo by Jared Tarbell, used under CC BY 2.0, cropped from original.

Enumerated types are a useful programming tool when dealing with variables that have a predefined, limited set of potential values. An example of an enumerated type from Wikipedia is “the four suits in a deck of playing cards may be four enumerators named Club, Diamond, Heart, and Spade, belonging to an enumerated type named suit”.

I use enumerated types in my Rails applications most often for model attributes like “status” or “category”. Rails’ implementation of enumerated types in ActiveRecord::Enum provides a way to define sets of enumerated types and automatically makes some convenient methods available on models for working with enumerated attributes. The simple syntax does belie some potential pitfalls when it comes to longer-term maintenance of applications, however, and as I’ll describe later in this post, I would caution against using this basic 1-line syntax in most cases:

enum status: [:active, :archived]

The Rails implementation of enumerated types maps values to integers in database rows by default. This can be surprising the first time it is encountered, as a Rails developer looking to store status values like “active” or “archived” would typically create a string-based column. Instead, Rails looks for an numeric type column and stores the index of the selected enumerated value (0 for active, 1 for archived, etc.).

This exposes one of the first potential drawbacks of this minimalist enumerated type implementation: the stored integer values can be difficult to interpret outside the context of the Rails application. Although querying records in a Rails console will map the integer values back to their enumerated equivalents, other database clients are simply going to return the mapped integer values instead, leaving it up to the developer to look up what those 0 or 1 values are supposed to represent.

A larger problem that arises from defining an enum as an array of values is that the values are tied to the order of elements in an array. This means any change to the order or length of the array can have unwanted consequences on the mapped values.

# don't do this; the index of active is changed from 0 to 1, archived from 1 to 2
enum status: [:abandoned, :active, :archived]

At a minimum, I would recommend using this hash-based syntax for defining enumerated types with explicit integer mapping:

enum status: {
  active: 0,
  archived: 1
}

This provides the benefit of documenting which integers are mapped to which enumerated values, and also provides more flexibility for future adjustments. For example, a new status value can now be added to the enumerated type without disrupting any of the existing records:

enum status: {
  abandoned: 2,
  active: 0,
  archived: 1
}

For Rails applications with PostgreSQL databases, it’s possible to go one step further and get most of the best of both worlds: the efficiency of using predefined enumerated types while still maintaining the ability to store meaningful string values at the database level. This is made possible by combining Rails enums with PostgreSQL Enumerated Types.

This technique requires using a migration to first define a new enumerated type in the database, and then creating a column in the model’s table to use that PostgreSQL type:

class AddEnumeratedStatusToDevices < ActiveRecord::Migration[5.2]
  def up
    execute <<-SQL
      CREATE TYPE device_status AS ENUM ('abandoned', 'active', 'archived');
    SQL

    add_column :devices, :status, :device_status
    add_index :devices, :status
  end

  def down
    remove_index :devices, :status
    remove_column :devices, :status

    execute <<-SQL
      DROP TYPE device_status;
    SQL
  end
end

The corresponding model code to use this new enumerated type looks similar to before, but now the values are mapped to strings:

class Device < ApplicationRecord
  enum status: {
    abandoned: "abandoned",
    active: "active",
    archived: "archived"
  }
end

This combination of Rails and PostgreSQL enumerated types has become my preferred approach in most situations. One limitation to be aware of with this approach is that PostgreSQL enumerated types can be extended with ALTER TYPE, but existing values cannot be removed. There is a small bit of additional development overhead introduced with the need to manage the enumerated type at both the Rails and the PostgreSQL level, but I like having the option of querying records by the string values of attributes, and the use of a PostgreSQL enumerated type provides for more efficient database storage than simply using a string type column.

Database Design: Using Natural Keys

2021-03-15T00:00:00+00:00

Whether to use natural or surrogate keys is a long-debated subject of database design. I am a fan of using natural keys. I think there are even more compelling reasons to use natural keys in databases as the systems grow more complex and interdependent.

Natural or Surrogate

Let’s start by what we mean by natural. It’s not trivial to define this. In today’s world of APIs, someone’s surrogate key is another’s natural key. Wikipedia defines natural keys as “a type of unique key in a database formed of attributes that exist and are used in the external world outside the database”. This makes it clear that the keys we get from APIs are our natural keys. But how about the ones generated by us to be used in the external world?

When applications expose the keys on the URLs and APIs, others start relying on them. This is where our choices become important. When all those different applications generate their own keys instead of using the keys they got from other places, life becomes difficult for no reason.

Let’s elaborate with an example corporate database where employees are identified by their usernames and the departments with their domains. So our data would look like this:

| department        | username  | job                     |
| ----------------- | --------- | ----------------------- |
| sysadm.corp-x.com | hasegeli  | Database Administrator  |
| sysadm.corp-x.com | john      | System Administrator    |
| dep1.corp-x.com   | jane      | Developer               |

When we design this using surrogate keys, it’d look like this:

CREATE TABLE departments (
  id int NOT NULL GENERATED ALWAYS AS IDENTITY,
  domain text NOT NULL,

  PRIMARY KEY (id),
  UNIQUE (domain)
);

CREATE TABLE employees (
  id int NOT NULL GENERATED ALWAYS AS IDENTITY,
  username text NOT NULL,

  PRIMARY KEY (id),
  UNIQUE (username)
);

CREATE TABLE department_employees (
  id int NOT NULL GENERATED ALWAYS AS IDENTITY,
  department_id int NOT NULL,
  employee_id int NOT NULL,
  job text NOT NULL,

  PRIMARY KEY (id),
  UNIQUE (department_id, employee_id),
  FOREIGN KEY (department_id) REFERENCES departments (id),
  FOREIGN KEY (employee_id) REFERENCES employees (id)
);

Then, we would have to make a query joining them all together:

SELECT d.domain, e.username, de.job
FROM department_employees de
JOIN departments d ON de.department_id = d.id
JOIN employees e ON de.employee_id = e.id;

Instead, we could just have it as we need it on a single table:

CREATE TABLE department_employees (
  domain text NOT NULL,
  username text NOT NULL,
  job text NOT NULL,

  PRIMARY KEY (domain, username)
);

And, we could just query it:

SELECT domain, username, job
FROM department_employees;

Good Examples of Natural Keys

A primary key must be always available (NOT NULL) and unique. It should also be compact and easy to represent. Here are some good examples of natural keys:

Keys acquired from various APIs
Usernames
Domains
Email addresses
ISO country codes
IATA airport codes
Dates

I think easy representation matters for unchanged use among many systems. For that reason, I don’t think names with spaces, symbols, or non-ASCII Unicode letters are good keys.

UUIDs and hashes/digests like MD5 and SHA are also popular as primary keys. I don’t think they are a good choice as surrogate keys. Auto-incrementing IDs are easier to use, generate, store, predict — although, when they are defined in the external world and we have to store them, there’s nothing wrong with using them as the primary keys.

Another consideration when selecting keys is their stability. Things that can change are not good keys. This is not only because it is more work to change the keys everywhere, but because different parts of the system cannot rely on them.

The context matters to evaluate a key’s stability. For example, a phone number is not a good key for the users table, because users may change their phone number. However, it is perfect for the phone numbers table.

Also, I don’t think we should discard some useful keys just because they can change in an edge case. It is a good trade-off to allow the rare change rather than to complicate the system with a surrogate key.

Enterprise Architecture

I don’t know what you understand from the word “enterprise” but I have heard people advocating streamlined, unpractical solutions in the name of “enterprise” for a long time. I have seen people using layers of layers in complex frameworks and ORMs with caches and buffers, managers and supervisors, factories and workers, locks and semaphores, queues and consumers, all in the name of “enterprise architecture”.

Many of those frameworks have a rule like requiring all database tables to have a single-column auto-incremented primary key named “id”. Such a rule is easy to enforce and follow but it’s limiting. It would not be practical, maybe not even possible to use natural keys as the primary keys within such framework.

Even though I am writing this post to support using natural keys, strong conventions built around surrogate keys do certainly help developing complex systems in the “enterprise” environments. They help to keep systems consistent when the developers are more motivated to write code than to read what’s there, while working until they get a better offer from another company.

Performance Consideration

Performance is often an argument in favour of surrogate keys. Surrogate keys can be small auto-incrementing integers which would keep the indexes small, next branches easy to predict, so well performing.

Surrogate keys would take a bit of extra space since otherwise we don’t need to have them at all, but the space savings of using them as foreign keys would easily outweigh this. However, using surrogate keys may require the queries to have more tables joined together.

Some relational databases maintain the tables indexed by the primary key. Such systems are more affected by the size of the primary key. On PostgreSQL all indexes are secondary including the primary key, therefore the primary key is not any more important than the other indexes on the table.

Composite Keys

Natural keys are often unique together with other keys. This forces the database designer to use multiple columns as keys. Such design has many advantages, which I may cover in future blog posts.

Job opening: PostgreSQL database administrator & developer

2020-11-13T00:00:00+00:00

Photo by manhhai, cropped & filtered, CC BY 2.0

We are seeking a full-time PostgreSQL database administrator and developer to support and consult with our clients. For this role we prefer those living in the Western Hemisphere.

End Point is an Internet technology consulting company based in New York City, with 50 employees serving many clients ranging from small family businesses to large corporations. The company turned 25 years old this year!

Even before the pandemic most of us worked remotely from home offices. We collaborate using SSH, Git, project tracking tools, Zulip chat, video conferencing, and of course email and phones.

What you bring:

5+ years of hands-on professional experience and technical expertise with:

Database design and systems architecture
Advanced SQL and query optimization
Programming for web applications, stored procedures, and scripting, in Ruby, Java, Python, PHP, Perl, JavaScript, C#, Go, PL/pgSQL, bash, etc.
Performance tuning, scalability, connection pooling
Cloud database hosting services, especially at AWS, Google Cloud, and Azure
Security and encryption
Replication, PITR, backup, monitoring, and relevant tools
HA, failover, and disaster recovery
Database upgrades and migrations
Programming language interfaces to the database
Linux system administration and observation
DevOps and “configuration as code”
Troubleshooting problems and handling emergencies
Other databases such as MySQL, MongoDB, Oracle, SQL Server, Redis, Solr, Elasticsearch, etc.
Git version control
Bonus for familiarity with running PostgreSQL on Windows, and with the Bucardo replication system

These work traits are just as important:

Strong verbal and written communication skills
An eye for detail
Tenacity in solving problems and focusing on customer needs
A feeling of ownership of your projects
Work both independently and as part of a team
Ability to pass a criminal background check or get a security clearance when clients require
A good remote work environment

What work here offers:

Collaborate with knowledgeable, friendly, helpful, and diligent co-workers around the world
Flexible, sane work hours
Paid holidays and vacation
Annual bonus opportunity
For U.S. employees: health insurance subsidy and 401(k) retirement savings plan
Freedom from being tied to an office location
Use your desktop OS of choice: Linux, macOS, Windows
Use open source tools and contribute back as opportunity arises

Get in touch with us:

~~Please email us an introduction to jobs@endpointdev.com to apply.~~ (This job has been filled.) Include your location, a resume/CV, your Git repository or LinkedIn URLs, and whatever else may help us get to know you.

We look forward to hearing from you! Direct work seekers only, please—this role is not for agencies or subcontractors.

We are an equal opportunity employer and value diversity at our company. We do not discriminate on the basis of sex/gender, race, religion, color, national origin, sexual orientation, age, marital status, veteran status, or disability status.

Detecting gaps in time-series data in PostgreSQL

2020-10-26T00:00:00+00:00

Photo by Phoebe Baker

A client has a number of data feeds that are supposed to update at regular intervals. Like most things in the universe (thanks, entropy!) this does not always end up working out the way we want. We recently discovered that some of the data had not loaded as expected, and were brought in to assess the extent of the issue.

There were 2 main feeds that had issues, with different types of time data. The first had date-based data with batches and the second had full-range timestamps. We will examine each type individually, since they have similar — but not the same — characteristics.

Date-based, with batches

In the first data feed there was a table which tracked which files had been loaded by file date and batch number (there was a morning/evening batch designated by a batch field with either A or B). Since the files that had not loaded successfully did not have entries in this table we could find that one or both of the day’s batch files would be missing, but since the table tracked those which did get loaded, we needed to turn this list into something useful.

Any time I find myself considering sequences of data, I remember my friend generate_series(). While generate_series() is often called with numeric data, it can also generate date ranges, which can come in very handy:

postgres=# SELECT * FROM generate_series('2020-01-01'::date, '2020-10-01'::date, interval '1 day') LIMIT 10;
    generate_series
------------------------
 2020-01-01 00:00:00-06
 2020-01-02 00:00:00-06
 2020-01-03 00:00:00-06
 2020-01-04 00:00:00-06
 2020-01-05 00:00:00-06
 2020-01-06 00:00:00-06
 2020-01-07 00:00:00-06
 2020-01-08 00:00:00-06
 2020-01-09 00:00:00-06
 2020-01-10 00:00:00-06
(10 rows)

Let’s create our sample table and populate with data to simulate the situation we ran into:

CREATE TABLE loader_manifest (filename text, batch char, status char, processed_at timestamptz);

INSERT INTO loader_manifest (filename, batch, status, processed_at)
SELECT
    to_char(generate_series('2020-01-01'::date, '2020-10-22'::date, '1 day'), 'YYYYMMDD'),
    batch,
    'L',
    now()
FROM (VALUES ('A'::char), ('B')) batches(batch)
ORDER BY 1,2
;

-- create some gaps to later detect

DELETE FROM loader_manifest WHERE filename = '20200218' AND batch = 'B';
DELETE FROM loader_manifest WHERE filename = '20200426' AND batch = 'A';
DELETE FROM loader_manifest WHERE filename = '20201019';

The data that we were trying to match against looked similar to the following:

postgres=# SELECT * FROM loader_manifest LIMIT 5;
 filename | batch | status |         processed_at
----------+-------+--------+-------------------------------
 20200101 | A     | L      | 2020-10-18 11:25:05.878229-05
 20200101 | B     | L      | 2020-10-18 11:25:05.878229-05
 20200102 | A     | L      | 2020-10-18 11:25:05.878229-05
 20200102 | B     | L      | 2020-10-18 11:25:05.878229-05
 20200103 | A     | L      | 2020-10-18 11:25:05.878229-05
(5 rows)

Since either the batch or entire days could have been missing we basically needed to generate the list of expected combinations and use a LEFT JOIN to find which values in this table were missing. While this example has only 2 batch options, this solution would work for additional numbers of expected batches.

Because generate_series() is a set-returning function we can use it as a source in our FROM clause, in combination with the explicit values we want for our batch field. We also will need to extract the date pieces out in a way that we can match the expected filename format. We use something similar to the following:

postgres=# SELECT to_char(filename, 'YYYYMMDD') AS filename, batch FROM generate_series('2020-01-01'::date, '2020-10-01'::date, interval '1 day') AS filename, (VALUES ('A'), ('B')) AS batches(batch) LIMIT 10;
 filename  | batch
----------+-------
 20200101  | A
 20200101  | B
 20200102  | A
 20200102  | B
 20200103  | A
 20200103  | B
 20200104  | A
 20200104  | B
 20200105  | A
 20200105  | B
(10 rows)

So as we can see, this will generate the list of all expected filename/batch combinations that we can then check the original table against for missing values:

SELECT
    candidates.*
FROM (
    SELECT
        to_char(filename,'YYYYMMDD') AS filename,
        batch
    FROM
        generate_series('2020-01-01'::date, '2020-10-01'::date, interval '1 day') AS filename,
        (values ('A'), ('B')) AS batches(batch)
) candidates
NATURAL LEFT JOIN
    loader_manifest
WHERE loader_manifest.filename IS NULL

Here candidates is the generated list of all expected file/batch combinations.

And the results from our sample dataset:

 filename | batch
----------+-------
 20200218 | B
 20200426 | A
 20201019 | A
 20201019 | B
(4 rows)

As expected, this has identified the rows that we know are missing from the table. Success!

Non-date-based

The second type of query had to identify similar gaps in loaded data, but used a different approach. This data feed had a processing_time_utc field which would vary, but we expected this to have fairly regular updates. This data feed could be processed somewhat erratically, but if the data was complete we would expect to see records with processing_time_utc approximately every 10 minutes. If we found that there were gaps much larger than every 10 minutes we could suspect missing data and would need to locate/reprocess the underlying data files.

This approach also uses generate_series(), but since we do not have a single date that we can check existence of, the JOIN approach does not work directly. This means that in order to find the gaps we care about, the need to check that at least some data exists within each time period in question.

Here’s the query:

SELECT
    day
FROM generate_series('2020-07-01'::date, '2020-08-01'::date, interval '10 minutes') AS day
LEFT JOIN LATERAL
    (SELECT day FROM the_table WHERE processing_time_utc >= day AND processing_time_utc < day + interval '10 minutes' LIMIT 1) gap ON day = gap.day
WHERE gap.day IS NULL;

The key here is that we use generate_series() with an interval of the window size we care about, and then use that same interval when checking for any row’s existence that fits that criteria. With the LIMIT 1 on the gap subquery, we stop at the first record we find in that range, and thus we can exclude in the antijoin to find our results.

I should note that this approach works regardless of the size of the logical date partitions, and can be used to identify all gaps of at least the interval size we are looking at; the same query can check for gaps every 10 minutes as for hourly, daily, weekly, etc. It is purely a matter of selecting the proper interval size in the query in question.

Also note that the processing_time_utc field is indexed, or this would be painfully slow. We use a LATERAL query to pull in one specific row that matches the time interval we care about and ignore any others.

Be aware that this technique finds only the cases where no data is present for the time interval, not less than expected. You could certainly adapt this approach to that situation by using an appropriate query.

Summary

These approaches definitely come in handy when trying to isolate missing data based on time; which tack you take will depend on the expected data present, as well as the granularity in which you identify your gaps.

Using CTEs to do a binary search of large tables with non-indexed correlated data

2020-10-02T00:00:00+00:00

Photo by Colin Knowles, used under CC BY-SA 2.0.

Query optimization can take different forms depending on the data represented and the required needs. In a recent case, we had a large table that we had to query for some non-indexed criteria. This table was on an appliance that we were unable to modify, so we had to find a way to query efficiently without indexes that would have made it easier.

The straightforward approach for this query was something along these lines:

SELECT accounts.*
FROM accounts
JOIN logs
    ON accounts.id = logs.account_id
WHERE
    logs.created_at BETWEEN $1 - 'interval 1 minute' AND $1 + 'interval 1 minute' AND
    logs.field1 = $2 AND
    logs.field2 = $3
FETCH FIRST ROW ONLY

Unfortunately, none of the fields involved in this query were indexed, nor could they be, due to our access level on this database system. This lack of indexes means that our query against those fields would end up doing a sequential scan of the whole table which made things unacceptably slow. This specific table held time-series data with ~ 100k records per 1 minute period over a period of several weeks, which meant we were dealing with a lot of data.

While we could not create any additional indexes to help us with this query, we could use some specific properties to help us:

There was a primary key field, id, which was unique and monotonic, i.e., always increasing in value.
This table was append only; no updates or deletes, so once data existed in the table it was always the same.
The field we actually care about (created_at) also ends up being monotonic: subsequent records would always have the same or later values.
Since records were created sequentially and the id field was always increasing, id values and created_at fields would be together be generally monotonic; this means there is an indexed field which we can use as a surrogate stand-in for the target field that we want to treat as an index. While due to the nature of logging ingest it is possible the created_at and id values are not strictly monotonic (for instance if there are multiple logging records being created by separate ingest processes, of which ids get assigned in chunks), for our purposes this was close enough, since we were looking around in a time window which was wider than we were actually expecting the message to appear.
Since we are looking for fields matching a specific window of time, we can substitute the non-indexed clause created_at BETWEEN <timestamp_min> AND <timestamp_max> with an expression matching the indexed statement id BETWEEN <id of first id gt timestamp_min> AND <id of first id gt timestamp_max> to get the same effective approach.
In order to find the specific id fields which match the created_at time ranges we are interested in, we would need to find the first id value which matched the criteria created_at > 'timestamp'::timestamp, as all subsequent id values would match that condition as well. This would effectively require a binary search of the table to check which records match the criteria, and return the smallest id value for which this criteria held.

So now that we have identified how we can use an indexed surrogate key to substitute for the non-indexed expression, we need to figure out how to calculate the ranges in question.

Based on some recent discoveries about optimizing simple-looking but poorly-performing queries using more complicated queries¹, I had instincts that this could be solved with a WITH RECURSIVE Common Table Expression. After toying around for a while and not coming up with the exact solution, I ended up visiting the #postgresql channel on FreeNode IRC Network. There, I presented the problem and got some interested responses, as this is the exact kind of question that database ~~nerds~~ experts love². The solution that user xocolatl (Vik Fearing) came up with for a basic binary search is the basis for the rest of my solution:

CREATE TABLE test_table (id integer PRIMARY KEY, ts timestamptz);

INSERT INTO test_table
SELECT g, date 'today' + interval '1s' * g
FROM generate_series(1, 1000000) AS g;

WITH RECURSIVE
search (min, max, middle, level) AS (
    SELECT min(id), max(id), (min(id)+max(id))/2, 0
    FROM test_table
        UNION ALL
    SELECT v.min, v.max, (v.min + v.max)/2, s.level+1
    FROM search AS s
    CROSS JOIN LATERAL (
        SELECT *
        FROM test_table AS e
        WHERE e.id >= s.middle
        ORDER BY e.id
        FETCH FIRST ROW ONLY
    ) AS e
    CROSS JOIN LATERAL (VALUES (
        CASE WHEN e.ts < now() THEN e.id ELSE s.min END,
        CASE WHEN e.ts < now() THEN s.max ELSE e.id END
    )) AS v (min, max)
    WHERE (v.min + v.max)/2 NOT IN (v.min, v.max)
)
SELECT *
FROM search AS s
JOIN test_table AS e ON e.id = s.middle
ORDER BY s.level DESC
FETCH FIRST ROW ONLY;

As expected, the solution involved a WITH RECURSIVE CTE.

The basic explanation here is that the search expression first starts with the min, max, and middle (mean) values of id for the table (the initialization expression), then iteratively adds additional rows to the results depending on if the table row with id >= middle matches our specific test criteria, then continues until one of the boundaries of the region is hit. (Since we are using integer division in our terminal expression (v.min + v.max)/2 NOT IN (v.min, v.max) we are guaranteed to hit one of the boundary conditions eventually in our search.)

A few other things worthy of note:

This approach uses the check e.ts < now() as the condition we are testing, which means in this specific example, the answer to this “closest id” query would actually change depend on when you run this query relative to when the initial test data was populated. However, we can replace that condition with whatever condition we want to use to test for our surrogate non-indexed data.
This approach will work whether or not there are gaps in the sequence. In order to properly handle gaps, we are selecting the first row with id >= middle ... FETCH FIRST ROW ONLY rather than just selecting id = middle, which you could do in a gapless sequence.
In addition to not caring about gaps, we are also not trying to make sure this is using a balanced binary search; it would be not worth the computing effort to find the middlest existing row in an index, as we’d need to know the number of rows in the segment we’re searching. Since in PostgreSQL this would entail a COUNT(*) from a subselect, this would be quite slow and not worth the trouble.

Considering my specific use case was trying to limit the records considered based on two created_at values I needed to calculate a search_min and search_max to find the start/end ids for each side in the interval.

Given this, I just modified the CTE to calculate those and add the additional conditions we were wanting to consider. I also had to turn the result query from a join against the found id value to a range; the final query is as follows:

WITH RECURSIVE
search_min (min, max, middle, level) AS (
    SELECT min(id), max(id), (min(id)+max(id))/2, 0
    FROM logs
        UNION ALL
    SELECT v.min, v.max, (v.min + v.max)/2, s.level+1
    FROM search_min AS s
    CROSS JOIN LATERAL (
        SELECT *
        FROM logs AS e
        WHERE e.id >= s.middle
        ORDER BY e.id
        FETCH FIRST ROW ONLY
    ) AS e
    CROSS JOIN LATERAL (VALUES (
        CASE WHEN extract(epoch FROM e.created_at)::integer < $1 THEN e.id ELSE s.min END,
        CASE WHEN extract(epoch FROM e.created_at)::integer < $1 THEN s.max ELSE e.id END
    )) AS v (min, max)
    WHERE (v.min + v.max)/2 NOT IN (v.min, v.max)
),
search_max (min, max, middle, level) AS (
    SELECT min(id), max(id), (min(id)+max(id))/2, 0
    FROM logs
        UNION ALL
    SELECT v.min, v.max, (v.min + v.max)/2, s.level+1
    FROM search_max AS s
    CROSS JOIN LATERAL (
        SELECT *
        FROM logs AS e
        WHERE e.id >= s.middle
        ORDER BY e.id
        FETCH FIRST ROW ONLY
    ) AS e
    CROSS JOIN LATERAL (VALUES (
        CASE WHEN extract(epoch FROM e.created_at)::integer < $2 THEN e.id ELSE s.min END,
        CASE WHEN extract(epoch FROM e.created_at)::integer < $2 THEN s.max ELSE e.id END
    )) AS v (min, max)
    WHERE (v.min + v.max)/2 NOT IN (v.min, v.max)
)
SELECT accounts.*
FROM accounts
JOIN logs
    ON logs.account_id = accounts.id
WHERE
    logs.field1 = $3 AND
    logs.field2 = $4 AND
    logs.id >= (SELECT middle FROM search_min ORDER BY level DESC FETCH FIRST ROW ONLY) AND
    logs.id <= (SELECT middle FROM search_max ORDER BY level DESC FETCH FIRST ROW ONLY)
ORDER BY logs.id
FETCH FIRST ROW ONLY

The final results were drastically improved. The initial query went from timing out in the webservice in question to returning results in a fraction of a second. Clearly this technique, while not as useful as directly indexing data we care about, can come in handy in some circumstances.

Another tool for the toolbag!

1 (back): Improving max() performance in PostgreSQL: GROUP BY vs. CTE

2 (back): See nerdsniping

Testing user-defined functions with pgTAP

Functions:

has_function()

functions_are()

ok() and is()

Testing more complex user-defined stored functions:

How To Set Up pgTAP for Writing PostgreSQL Database Unit Tests

Installing & setting up pgTAP

Writing simple database tests with pgTAP

More pgTAP tests

Test to check if a trigger is triggered

Test to check if a constraint is enforced

Organizing and naming your test files based on Test Anything Protocol (TAP)

Running your tests using pg_prove

pgTAP in EpiTrax

Conclusion

Find Text in Any Column of a PostgreSQL Table

Data Migration Tips

Customer Management

Migration Scripting

Data Migration History

Teamwork

Building and Hosting a Web App with .NET 6, Postgres and Linux

Preparing the database

Creating the projects

Setting up the database connection

Updating the config file

Displaying the test data

Installing and configuring Nginx

Final application adjustments

Time to publish!

Running the app on the server

Using systemd to run the app as a service

Viewing logs

Wrapping up

Upgrading PostgreSQL 14 to 15 on Fedora, RHEL, CentOS, Rocky, Alma Linux with PGDG RPM packages

PostgreSQL 15 changes

Prerequisites for upgrading

Stop PostgreSQL 14

Update your system

Install PostgreSQL 15

Create the new PostgreSQL 15 database cluster

Migrate your data with pg_upgrade

Start PostgreSQL 15

Try it out

Remove PostgreSQL 14

Reference

Running PostgreSQL on Docker

Introduction

Prerequisites

Get to know the official Postgres Image

Run the Postgres image as a basic Postgres container

Working with the Postgres container

Run Postgres image as a “real world” Postgres container

Cleaning up the running container

Managing Postgres container with Docker Compose

Conclusion

How to deploy a Django App with Aurora Serverless and AWS Copilot

The Prerequisites

The Django Project

The Deployment with AWS Copilot

How to query Aurora Serverless

The End

Formatting SQL code with pgFormatter within Vim

On Shapefiles and PostGIS

QGIS

GDAL

PostGIS

Examples

Import Shapefile data into PostGIS

Export PostGIS data as KML

Using pgTAP to automate database testing

Databases need tests too

What to test?

Iteratively improving

Looking back and ahead

Database Design: Using Documents

Example

Normalized Tables

Table Sizes

`has_function()`

`functions_are()`

`ok()` and `is()`