Our Blog

Database Design: Using Documents

By Emre Hasegeli
March 9, 2022

Using documents in relational databases is increasingly popular. This technique can be practical and efficient when used in fitting circumstances.

Example

Let’s start with an example. Imagine we are scraping web sites for external URLs and store them in a table. We’ll have the web sites table to store the scrape timestamp and another table to store all of the references.

CREATE TABLE web_sites (
    web_site_domain text NOT NULL,
    last_scraped_at timestamptz,

    PRIMARY KEY (web_site_domain)
);

CREATE TABLE refs (
    web_site_domain text NOT NULL,
    ref_location text NOT NULL,
    link_url text NOT NULL,

    PRIMARY KEY (web_site_domain, ref_location, link_url),
    FOREIGN KEY (web_site_domain) REFERENCES web_sites
);

We do not need to bother adding an id to the web_sites table, because we assume there won’t be too many of them. The domain is small and more practical to use as an identifier. If you are curious about advantages of using natural keys, see my previous article.

Normalized Tables

There may be many thousands of unique URLs for a single web site and other web sites may refer to the same URLs. To try to minimize the storage, we can keep the …

Using a YubiKey as authentication for an encrypted disk

By Zed Jensen
March 7, 2022

Image by Silas Köhler on Unsplash

Recently I built a small desktop computer to run applications that were a bit much for my laptop to handle, intending to bring it with me when I work outside my apartment. However, there was an immediate issue with this plan. Because this computer was intended for use with sensitive information/​source code, I needed to encrypt the disk, which meant that I’d need to enter a passphrase before I could boot it up.

I didn’t really want to haul a keyboard and monitor around with me, so I came up with an alternative solution: using a YubiKey as my method of authentication. This allowed me to avoid the need to type a password without giving up security. In this post I’ll show you how you can do the same.

Preparation

First off, you need a YubiKey, if you don’t have one already. I ended up getting the YubiKey 5C NFC.

While I waited for my YubiKey to arrive, I installed Ubuntu 20.04 with full-disk encryption (using the default option of LUKS, or Linux Unified Key Setup) on the computer. I set a passphrase like normal—the process I describe in this post allows access with either this passphrase or the YubiKey.

Next, there were two packages …

Optimizing media delivery with Cloudinary

By Juan Pablo Ventoso
March 1, 2022

I remember how we needed to deal with different image formats and sizes years ago: From using the WordPress-style approach of automatically saving different resolutions on the server when uploading a picture, to using a PHP script to resize or crop images on the fly and return the result as a response to the frontend. Of course, many of those approaches were expensive, and not fully optimized for different browsers or device sizes.

With those experiences in mind, it was a nice surprise for me to discover Cloudinary when working on a new project a couple of months ago. It’s basically a cloud service that saves and delivers media content with a lot of transformations and management options for us to use. There is a free version with a usage limit: Up to 25K transformations or 25 GB of storage/​bandwidth, which should be enough for most non-enterprise websites. The cheapest paid service is $99 per month.

Here’s a list of the image features we used on that project. I know they offer many other things that can be used as well, but I think this is a good start for anyone who hasn’t used this service yet:

Resizing and cropping

When you make a request for an image, you …

String Processing in SQL, the Hard Way

By Josh Tolley
February 28, 2022

Photo by Kristy Lee

When I was a kid, my family visited a prehistoric cliff-dweller settlement, reconstructed and preserved as a museum exhibit. The homes were built in caves in high sandstone cliffs, and though I followed a well-marked path to visit them, the original inhabitants climbed in and out via holes cut into the rock face. Apparently to deter unwanted visitors, the builders carefully spaced the foot holds so that climbers needed to start their ascent on the correct foot. Someone who didn’t know the secret of the path, or who carelessly forgot, might get halfway up the cliff and discover he or she couldn’t stretch to the next foot hold and had to turn back.

The other day I found myself confronted with several comma-separated lists of strings, and a problem: I needed the longest possible substring from each, starting from the beginning, ending in a comma, and shorter than a certain length, and I needed to do it in the database. “Easy peasy,” you may say to yourself, if you have a little SQL programming experience and a penchant for cheesy clichés, and you’d be right except for one thing: I let my mind wander a little too far down into the …

Migrating a Node.js app database from MongoDB to PostgreSQL

By Phineas Jensen
February 25, 2022

Recently I worked on a web app designed for tracking, editing, and reporting on archaeological survey data. When I joined the project, it used React on the front-end, Node.js on the back-end, GraphQL as the API interface, and MongoDB as the database. For the most part, this architecture and choice of technology worked well, and I was quickly able to make meaningful contributions to the app both in the user interface and in the backend. However, after a while, things started to get difficult and we began to consider why we were using MongoDB, and what might be better.

MongoDB vs. PostgreSQL

First, why was MongoDB difficult to use in our case? The biggest difficulty was using their query language for certain aspects of our database. Part of the database involved representing artifacts found at a site using a specific type hierarchy defined by the state government. Representing that hierarchy within Mongo was difficult, and querying it even more so. It was hard to find how to query a recursive structure using the MongoDB query language.

As I worked on this, scouring the documentation and Q&A websites to find ways to query this data efficiently, I began to realize that it was a …

Data Reporting with EpiTrax

By Shannon Sandall
February 16, 2022

Early COVID-19 form-building in Kansas

In March of 2020, many states struggled with disease surveillance systems that were not ready for the COVID-19 pandemic. States were forced to wait lengthy periods for their vendors or systems administrators to stand up customized forms for COVID-19 investigations.

State of Kansas informatics staff were able to configure their form within a matter of hours. When the first COVID-19 records came rolling in, the state was ready to go. The Kansas Department of Health and Environment (KDHE) was using EpiTrax, a National Electronic Disease Surveillance System (NEDSS) compliant application created by the state of Utah.

EpiTrax contains a powerful form-building utility that was easily customizable, allowing jurisdictional users to create forms for investigation and reporting purposes. Users could configure questions and value sets based on configurable lists, cutting down on free-text entries and ensuring data quality. Forms could be attached automatically or manually and assigned to specified agencies and conditions.

Screenshot of the form builder

December 2021 updates

Flash forward to December 2021. State of Utah developers rolled out an update …

Image compression: WebP presets, HEIC, AVIF, JPEG XL

By Jon Jensen
February 15, 2022

How time flies. Eight years ago I wrote the blog post WebP images experiment on End Point website to describe and demonstrate how the WebP image format can store an equivalent-quality image in a much smaller file size than the older JPEG, PNG, and GIF formats.

My WebP examples there were 17–23% of the JPEGs they came from, or about 5–6× smaller. While experimenting with higher levels of compression, I found that WebP tends to leave less-noticeable artifacts than JPEG does.

The main drawback at the time was that, among major browsers, only Chrome and Opera supported WebP, and back then, Chrome was far less popular than it is now.

Can I use it?

Since Apple’s iOS 14 and macOS 11 (Big Sur) became available in late 2020, the WebP image format now works in all the currently supported major operating systems and browsers: Linux, Windows, and macOS, running Chromium, Chrome, Brave, Edge, Opera, Firefox, and Safari. You can see the specifics at the ever-useful site “Can I use”.

It only took about 10 years! 😁

So for you who are hosting websites, all your site visitors can see WebP images and animations except those vanishing few using Internet Explorer (now long past its …

Fun with Rational Numbers

By Darius Clynes
February 11, 2022

So, you say you adored learning long division in elementary school? Well here we are revisiting that wonderful activity we have missed so much: fabulous magical moments like dividing 1.0 by 7. The especially tedious exercise of dragging over one more of the endless 0’s and seeing how many 7’s would fit into that woefully large remainder. Soon the delight mounts as you desperately hope the next digit will be the last and you discover that beautiful 1 as a remainder.

Undoubtedly you have wondered, like I have, what would happen if you divided 1.0 by 19? Would it repeat after only 8 digits? How many digits would it really take before it repeats?

That’s exactly how I stumbled into the wonderful world of rational numbers. There I was, wondering and wandering along the gloomy Belgian beach — the tide way out as far as you could see — in the winter wind, and I couldn’t stop thinking how long it would take for that mantissa to repeat. I had actually already tried 1.0 over 19. Finally, on that miserable day, I got the courage to divide 1 by 23.

Fortunately for me, I wasn’t alone, and I was able to enlist, coax, oblige, implore the help of baffled little Susan, my daughter who was 7 …