Our Blog

Working around SPF problems delivering to Gmail

By Jon Jensen
March 30, 2022

Photo by Garrett Skinner

Email deliverability

Legitimate email delivery keeps getting harder. Spammers and phishers never stop flooding everyone’s inboxes with unwanted and harmful email, so automated defenses against junk mail are necessary. But they are not perfect, and good email sometimes gets flagged as spam.

When sending important “transactional” email such as for account confirmations, password resets, and ecommerce receipts, it is often worth using a paid email delivery service to increase deliverability. Those typically cost a flat amount per month for up to a certain quota of outgoing email, with overage charges for messages beyond that.

Many of our clients use one of those services and generally they have all worked well and differ mostly in pricing and feature set. Popular choices include SendGrid, Mandrill, Postmark, Mailgun, and Amazon SES.

We continue to have many cases where we want to be able to send potentially large amounts of automated email to ourselves, our clients, or our systems. This is usually for testing, notifications, or internal delivery to special mailboxes separate from our main mailboxes.

These other uses for sending email keep us …

Code Reviews

By Kevin Campusano
March 21, 2022

Last week, a few End Point team members and I came together to prepare a presentation on code reviews for the whole company. We went through the basics of “what”, “why”, and “how”.

We also, and perhaps most interestingly, made several recommendations that we’ve discovered after years of doing code reviews in a variety of teams and project sizes. A series of “lessons learned” so to speak.

I thought it’d be useful to capture that discussion in written form. Let’s start with the basics.

What is a code review?

Wikipedia’s article on code reviews says that a code review is…

A software quality assurance activity in which one or several people check a program mainly by viewing and reading parts of its source code, and they do so after implementation or as an interruption of implementation.

That is a precise but frankly wordy way to say “having somebody look at the code you’ve written”. This definition, however, touches on a few aspects that give us good insight into what code reviews are and what their purpose is.

First up, it tells us that code reviews are a software quality assurance …

Using pgTAP to automate database testing

By Josh Tolley
March 16, 2022

Photo from PxHere

Recently I started learning to tune pianos. There are many techniques and variations, but the traditional method, and the one apparently most accepted by ardent piano tuning purists, involves tuning one note to a reference, tuning several other notes in relation to the first, and testing the results by listening closely to different combinations of notes.

The tuner adjusts each new note in relation to several previously tuned notes. Physics being what it is, no piano can play all its tones perfectly, and one of the tricks of it all is adjusting each note to minimize audible imperfections. The tuner achieves this with an exacting series of musical intervals tested against each other.

Databases need tests too

One of our customers needed to add security policies to their PostgreSQL database, to limit data visibility for certain new users. This can quickly become complicated and ticklish, ensuring that the rules work properly for the affected users while leaving other users unmolested.

This struck me as an excellent opportunity to create some unit tests, not that there’s any short supply of good opportunities to add unit tests! This is not just because it helps …

Automating reading the screen and interacting with GUI programs on X Window System

By Constante “Tino” Gonzalez
March 10, 2022

A while back, Google Earth made some changes to the layer select menu in the sidebar, which broke a program that toggles the 3D imagery on VisionPort systems. These run the X Window System (also known as X11, or just X) on Ubuntu Linux.

In looking for a workaround, I found that shell scripts can fully interact with GUI apps using the xdotool, xwd, and convert commands. This script would send a series of keystrokes to open the sidebar, navigate the layers menu, and toggle the box for the 3D buildings layer.

Changing the series of keystrokes to match the new number of layers should have fixed the issue, but there was more to this script. The next part of the script would take a screenshot, crop the checkbox, and compare it to saved files of other cropped boxes. Fixing this part of the script required correcting the positions of the captures and replacing the reference files with ones that pictured the updated Google Earth checkbox states.

Here I will explain how the script works and how we changed it so that it no longer needs these reference files and ultimately runs faster.

Overview of how the script works

xwd takes a screenshot of a window on the screen. convert transforms the …

Database Design: Using Documents

By Emre Hasegeli
March 9, 2022

Using documents in relational databases is increasingly popular. This technique can be practical and efficient when used in fitting circumstances.

Example

Let’s start with an example. Imagine we are scraping web sites for external URLs and store them in a table. We’ll have the web sites table to store the scrape timestamp and another table to store all of the references.

CREATE TABLE web_sites (
    web_site_domain text NOT NULL,
    last_scraped_at timestamptz,

    PRIMARY KEY (web_site_domain)
);

CREATE TABLE refs (
    web_site_domain text NOT NULL,
    ref_location text NOT NULL,
    link_url text NOT NULL,

    PRIMARY KEY (web_site_domain, ref_location, link_url),
    FOREIGN KEY (web_site_domain) REFERENCES web_sites
);

We do not need to bother adding an id to the web_sites table, because we assume there won’t be too many of them. The domain is small and more practical to use as an identifier. If you are curious about advantages of using natural keys, see my previous article.

Normalized Tables

There may be many thousands of unique URLs for a single web site and other web sites may refer to the same URLs. To try to minimize the storage, we can keep the …

Using a YubiKey as authentication for an encrypted disk

By Zed Jensen
March 7, 2022

Image by Silas Köhler on Unsplash

Recently I built a small desktop computer to run applications that were a bit much for my laptop to handle, intending to bring it with me when I work outside my apartment. However, there was an immediate issue with this plan. Because this computer was intended for use with sensitive information/​source code, I needed to encrypt the disk, which meant that I’d need to enter a passphrase before I could boot it up.

I didn’t really want to haul a keyboard and monitor around with me, so I came up with an alternative solution: using a YubiKey as my method of authentication. This allowed me to avoid the need to type a password without giving up security. In this post I’ll show you how you can do the same.

Preparation

First off, you need a YubiKey, if you don’t have one already. I ended up getting the YubiKey 5C NFC.

While I waited for my YubiKey to arrive, I installed Ubuntu 20.04 with full-disk encryption (using the default option of LUKS, or Linux Unified Key Setup) on the computer. I set a passphrase like normal—the process I describe in this post allows access with either this passphrase or the YubiKey.

Next, there were two packages …

Optimizing media delivery with Cloudinary

By Juan Pablo Ventoso
March 1, 2022

I remember how we needed to deal with different image formats and sizes years ago: From using the WordPress-style approach of automatically saving different resolutions on the server when uploading a picture, to using a PHP script to resize or crop images on the fly and return the result as a response to the frontend. Of course, many of those approaches were expensive, and not fully optimized for different browsers or device sizes.

With those experiences in mind, it was a nice surprise for me to discover Cloudinary when working on a new project a couple of months ago. It’s basically a cloud service that saves and delivers media content with a lot of transformations and management options for us to use. There is a free version with a usage limit: Up to 25K transformations or 25 GB of storage/​bandwidth, which should be enough for most non-enterprise websites. The cheapest paid service is $99 per month.

Here’s a list of the image features we used on that project. I know they offer many other things that can be used as well, but I think this is a good start for anyone who hasn’t used this service yet:

Resizing and cropping

When you make a request for an image, you …

String Processing in SQL, the Hard Way

By Josh Tolley
February 28, 2022

Photo by Kristy Lee

When I was a kid, my family visited a prehistoric cliff-dweller settlement, reconstructed and preserved as a museum exhibit. The homes were built in caves in high sandstone cliffs, and though I followed a well-marked path to visit them, the original inhabitants climbed in and out via holes cut into the rock face. Apparently to deter unwanted visitors, the builders carefully spaced the foot holds so that climbers needed to start their ascent on the correct foot. Someone who didn’t know the secret of the path, or who carelessly forgot, might get halfway up the cliff and discover he or she couldn’t stretch to the next foot hold and had to turn back.

The other day I found myself confronted with several comma-separated lists of strings, and a problem: I needed the longest possible substring from each, starting from the beginning, ending in a comma, and shorter than a certain length, and I needed to do it in the database. “Easy peasy,” you may say to yourself, if you have a little SQL programming experience and a penchant for cheesy clichés, and you’d be right except for one thing: I let my mind wander a little too far down into the …