Introducing EP Audit

2026-05-11T00:00:00+00:00

Photo by Bimal Gharti Magar, 2026.

In the complex landscape of cloud security, staying compliant and prepared for audits is important. Enter EP Audit, a compliance-focused audit process, AI-powered but led by senior engineers. It’s designed for Azure, AWS, and Google Cloud setups. EP Audit emerged from necessity: When a financial services client required an Azure security audit, we at End Point built a solution that not only met their immediate needs, but also evolved into an internal tool we continue to refine. Today, it serves as a key part of our multi-cloud security audit framework and is tested against our own cloud accounts before being used in client engagements.

The Problem

Cloud environments are dynamic: New projects, personnel changes, and vendor updates can significantly alter configurations over time. As environments evolve, the security posture originally reviewed and approved by your engineers may no longer fully match what is running in production. Unused resources, overly permissive access, and orphaned infrastructure can also accumulate over time, increasing both risk and monthly costs. This often goes unnoticed until an audit, customer review, or security incident forces a closer look.

Modern compliance frameworks—SOC 2, ISO 27001, HIPAA, PCI DSS—require regular, documented security reviews. Yet many organizations still struggle to provide consistent documentation with timestamps, severity tracking, and proof of remediation. EP Audit helps address this gap by generating a structured audit trail throughout the review process.

What EP Audit Looks Like

An EP Audit engagement is structured as a scheduled review cycle with two phases: a Base scan to establish a security snapshot, and a Final scan performed after remediation work to verify fixes. Both phases use a command-line runner with a read-only audit identity, ensuring no changes are made during the audit itself. Each Base scan produces nine deliverables designed to provide both high-level summaries and detailed technical findings:

Executive Summary: Severity-ranked audit findings.
Client Report: Full narrative with risk scorecard and roadmap.
Structured Findings Report: Engineer-friendly findings by category.
Pre-filled Hardening Checklist: CIS-aligned checklist with auto-populated FAIL statuses.
Fillable Hardening Checklist: A remediation tool for your team.
Network Diagram: Auto-generated deployment topology.
Findings Workbook: Excel file with color-coded severity tabs.
Remediation Plan: Phased plan with affected resources.
Partner Remediation Identity Setup: Guide for temporary write scope during remediation.

These deliverables are standardized across Azure, AWS, and GCP, helping simplify multi-cloud audit reviews and recurring engagements.

Sample summary of findings by severity and category on EP Audit.

Every finding in EP Audit is mapped to controls from recognized security frameworks. For Azure, we reference the CIS Microsoft Azure Foundations Benchmark and the Microsoft Cloud Security Benchmark. AWS findings align with the CIS AWS Foundations Benchmark, AWS Foundational Security Best Practices, and related guidance. GCP findings follow the CIS GCP Foundation Benchmark and the Google Cloud Architecture Framework Security Pillar. This helps keep audit findings grounded in established industry standards rather than proprietary scoring systems or arbitrary recommendations.

A Service, Not Just a Tool

There are already plenty of cloud configuration scanners available. What matters is how the findings are reviewed, prioritized, and applied in real environments.

Senior Engineering Review: Our engineers provide context and actionable recommendations for each finding, bridging the gap between automated results and practical solutions.
End Point-Specific Enhancements: We include checks beyond published frameworks, such as billing data analysis for orphaned resources and third-party security tool integration.
Internal Validation: Every framework change is tested on End Point’s own cloud accounts before client deployment.

A single audit provides a snapshot in time, but recurring audits make it easier to identify trends, validate remediation progress, and detect configuration drift over time. EP Audit includes year-over-year comparison reporting to help highlight both improvements and regressions. In practice, this helps support compliance reviews, procurement questionnaires, and ongoing security review processes while providing better visibility into how environments evolve over time.

Phased Remediation

EP Audit generates remediation scripts and recommendations organized by severity:

P1 (0–7 days): Critical issues like exposed management ports.
P2 (30 days): High-priority issues such as encryption gaps.
P3 (90 days): Medium-priority defense-in-depth weaknesses.
P4 (Maintenance): Low-priority configuration hygiene.

Scripts default to dry-run mode, allowing proposed changes to be reviewed before execution. Our team can assist with remediation directly, or clients can implement the recommendations independently using the provided guidance.

Over multiple engagements, organizations begin building a clearer picture of how their cloud environments evolve over time. Recurring audits make it easier to track remediation progress, identify recurring issues, and detect configuration drift before it becomes a larger problem. The resulting documentation can also help support compliance reviews, customer security questionnaires, cyber insurance requirements, internal security tracking, and reduce monthly costs.

Working with End Point

If your organization operates workloads in Azure, AWS, or Google Cloud and needs a structured, standards-aligned cloud security review process, feel free to contact us. If you are working with us already, you can talk with your representative to find out more.

Rocky Linux 9 at Hetzner Robot Made Quick and Easy

2023-07-01T00:00:00+00:00

Image by Jeffry Johar

Introduction

In my last blog post, I shared my experience of installing Rocky Linux 8 on my Hetzner robot server and subsequently upgrading it to Rocky Linux 9.

Rocky Linux project manager Brian Clemens said that method is not recommended and suggested using a boot kickstart for an automated installation. Thanks, Brian!

Despite my attempts to utilize kickstart, I encountered difficulties in booting my NVMe disk. During this process, I discovered another workaround for installing Rocky 9 using the installimage script. This method is also experimental, just like the previous one.

The Steps

Access the rescue mode (refer to my previous blog post if you need guidance).
Copy the existing Rocky Linux 9 image to Rocky Linux 8 and place it in the root directory with the following command:
```
cp /root/images/Rocky-91-amd64-base.tar.gz /root/Rocky-87-amd64-base.tar.gz
```

The filename should adhere to the required naming convention for the installimage script to function correctly.

Launch the installimage and select the Custom Image option.
Configure the disk settings as required, and at the end of the script, select the image accordingly.
```
IMAGE /root/Rocky-87-amd64-base.tar.gz
```
To save the configuration file, simply press F10, allowing the installation process to resume uninterrupted. You may encounter a warning indicating the absence of an image signature, which is perfectly normal. Once the installation is finished, proceed to reboot the system.
After the system restarts, you will have access to Rocky Linux 9. The installation will provide you with some fairly recent point-release of Rocky Linux, such as 9.1, but as always you need to update to the latest packages with dnf update.

Conclusion

That’s all, folks! This second method is much easier compared to the first one, as it swiftly takes you to Rocky 9 without any complications.

Rocky Linux 9 via Hetzner Robot for the impatient

2023-06-12T00:00:00+00:00

Image: https://www.pexels.com/photo/go-kart-17122631/

Update: A Better Way to Install Rocky Linux 9 at Hetzner Cloud

Hey everyone, I wanted to share an update regarding the installation process of Rocky Linux 9 at Hetzner Cloud.

After input from Brian Clemens of the Rocky Linux project and some further exploration and testing, I have a more efficient and straightforward method to get Rocky Linux up and running on the Hetzner platform. You can check out the new method in my second blog post here.

About Rocky Linux

Rocky Linux is a free and open-source community-driven operating system designed to be a drop-in replacement for Red Hat Enterprise Linux (RHEL).

Rocky fills the gap left by the end of CentOS, which was a popular Linux distribution based on the same source code as RHEL but offered as a free alternative with community support. CentOS Stream is their new offering, but it is just different enough to not be entirely compatible with RHEL.

Another alternative is AlmaLinux, and everything mentioned here applies to Alma with some adaptation.

Rocky 9 on Hetzner

Hetzner is a popular hosting company offering bare metal and virtual servers at very affordable prices. It is based in Germany and also has datacenters in Finland and the U.S.

At the time of this writing (June 2023), Rocky Linux 9 is not supported by Hetzner Robot to install on its dedicated physical servers. Despite Hetzner’s the installimage software indicating support, attempting to install it will result in the following error, at least on some of its servers:

“ERROR: We do not yet support rockylinux 91 on EFI systems”

If you are looking for a workaround, consider installing Rocky Linux 8 and upgrading it to version 9.

While it’s important to note that this installation approach hasn’t undergone comprehensive vendor testing, we have used it and I’ll provide you with the following step-by-step instructions to accomplish it.

Provisioning the Server

To get started, follow Hetzner’s guide to order and provision a server from Hetzner Robot.

Once the server has been provisioned, you may proceed to the next step.

Enabling Rescue Mode

Access the Hetzner Robot interface and navigate to the “Rescue” tab.
Choose Linux as the rescue media and select the SSH key option to access the server. Alternatively, use the generated root password.
Click the “Activate rescue system” button.

Rebooting the Server

Switch to the “Reset” tab.
Select the “CTRL+ALT+DEL” option.
Click “Send” to reboot the server.

Accessing the Rescue System

After a few minutes, SSH into the server using root@[server IPv4 or server IPv6 address] to access the rescue system.

Installing Rocky Linux 8:

Once in the rescue system, proceed with the installation of Rocky Linux 8. For this installation, we’re going to configure the disk storage to use RAID 6 and LVM, with filesystems as follows:

/boot/efi ESP: 256MB
/boot ext4: 1GB
LVM vg0 (rest of the disk)
/dev/vg0/root ext4: 50GB
/dev/vg0/home ext4: 100GB
/dev/vg0/swap swap: 1GB
Rest of vg0: unallocated, for future use

Run the command: imageinstall and you should get the following menu.

Choose “Rocky Linux” → “Rocky Linux 8.2”. After confirming the selection, you will be brought to the configuration file.
In the configuration file, put these values that set up the planned filesystems and LVMs as specified.

PART /boot ext4 1G
PART /boot/efi esp 256M
PART lvm vg0 all
LV vg0 root / ext4 50G
LV vg0 home /home ext4 100G
LV vg0 swap swap swap 1G

Press F10 to exit the configuration file editor, and the OS installation will resume.

Completing the Installation

Once the installation is done, reboot the rescue system. After a few minutes, SSH into the server again using root@[server IPv4 or server IPv6 address]. The SSH key and root password will be the same as the rescue system.
Execute the following to verify the OS is Rocky Linux 8:

cat /etc/os-release

Upgrading to Rocky Linux 9:

Execute the following commands to upgrade the OS to Rocky Linux 9.2:

export REPO_URL="https://download.rockylinux.org/pub/rocky/9/BaseOS/x86_64/os/Packages/r"
export RELEASE_PKG="rocky-release-9.2-1.5.el9.noarch.rpm"
export REPOS_PKG="rocky-repos-9.2-1.5.el9.noarch.rpm"
export GPG_KEYS_PKG="rocky-gpg-keys-9.2-1.5.el9.noarch.rpm"
dnf install $REPO_URL/$RELEASE_PKG $REPO_URL/$REPOS_PKG $REPO_URL/$GPG_KEYS_PKG
rm -rf /usr/share/redhat-logos
dnf --releasever=9 --allowerasing --setopt=deltarpm=false distro-sync -y

Execute the following to change the hostname to whatever you want it to be:

hostnamectl set-hostname mynewhostname

Reboot the system to complete the OS upgrade.

Accessing the Rocky Linux 9 System:

After a few minutes, SSH into the server again using root@[server IPv4 or server IPv6 address].
Remove dnf modules from Rocky Linux 8.

dnf module disable python36 virt

Execute the following to update the RPM database.

rpm --rebuilddb

Execute the following to verify the OS is Rocky Linux 9.

cat /etc/os-release

Conclusion

That’s all, folks.

You can get past the EFI issue you encountered by installing Rocky Linux 8 on Hetzner Robot and upgrading it in-place to Rocky Linux 9.

With these instructions, you can transition to the new and shiny Rocky Linux 9 without waiting for specific support by Hetzner Robot. Enjoy the enhanced features and capabilities of this updated version.

Bypassing a CDN to browse a website directly on your origin host

2023-01-20T00:00:00+00:00

Using a content distribution network (CDN) has many advantages over serving a website directly, and for any reasonably large website, you should use one. Those advantages include:

Caching at each of the CDN’s PoPs (points of presence).
Often thousands of PoPs around the world, so traffic will be quick for everyone regardless of how far away they are from your origin server.
Blocking of some Bad Guys automatically at the edge, including DDoS (distributed denial of service attacks) mitigation help.
Origin IP address insulation. Hiding the origin IP address is useful to protect against DDoSes, since CDNs are generally well-defended against DDoS and your origin server probably is not as much.

You should generally be cautious about revealing your websites’ origin IP address. We serve other sites from our origin directly, so we don’t worry too much about sharing it here.

Straight to the source

Sometimes, though, you need to bypass your CDN and test your website directly on its origin server. For example, if you need to test that your website would still work if the CDN goes down, or to sidestep CDN caching or content modification when troubleshooting a problem.

It can be surprisingly confusing to bypass the CDN by yourself and not for the rest of your visitors, especially with newer features like Secure DNS in Chrome and Firefox muddying up traditional DNS resolution behaviors.

We’ll use a browser extension called IPvFoo (available for Chrome and Firefox) to see which IP address our hostname resolves to. Install IPvFoo from the Chrome Web Store or Firefox’s addons.mozilla.org and then click on its indicator next to the search bar.

Before we change anything, www.endpointdev.com resolves to 104.21.30.147:

After a quick whois search, we see that this IP address belongs to Cloudflare, our CDN:

$ whois 104.21.30.147

✀
OrgName:        Cloudflare, Inc.
✀

We can also check the certificate issuer by clicking on the padlock icon in our browser’s search bar, then clicking on “Connection is secure” or similar to see all the details about the certificate issuer.

Before we change anything, we see the TLS/SSL certificate presented is issued by Cloudflare, as expected.

Make the hosts file point directly to our server

Common desktop and server operating systems typically come with a “hosts file” that can be used to provide hostname to IP address translation that happens before your designated DNS servers are consulted.

This is useful to test a hostname without ever adding it to DNS, or to override a DNS entry with your own IP address.

We want to change our hosts file so that www.endpointdev.com points directly to our site’s origin server. How to do this depends on your operating system:

Linux/macOS: Open /etc/hosts as root (using sudo or su) in your favorite editor such as vim or nano.
Windows: Open C:\Windows\System32\drivers\etc\hosts as administrator in a decent plain text editor, not a word processor — I used Notepad++.

Once you have your hosts file open, add a line to point your domain name directly to your origin server. For www.endpointdev.com:

23.239.26.161 www.endpointdev.com

Disable DNS-over-TLS & DNS-over-HTTPS

You may also need to disable any “secure DNS” in your browser, such as DNS-over-TLS or DNS-over-HTTPS.

Chrome

In Chrome, this is called Secure DNS, and you can disable it by going to Settings > Privacy and Security > Security > Use secure DNS, and disabling it.

Firefox

In Firefox, go to Network Settings, then disable DNS over HTTPS.

Then close any open tabs in your browser and quit it completely. Then start it again.

Browse the website directly

Now go to www.endpointdev.com.

IPvFoo is now showing the origin IP address we gave it, so we know that we’re being served data directly from that server.

We can also run whois on our origin server, for good measure:

$ whois 23.239.26.161

✀
OrgName:        Linode
✀

Now we’re getting the Linode IP address, not Cloudflare’s. This is expected, since our origin site is hosted on a server at Linode.

Check certificate issuer

You can also check that the certificate issuer changed. Keep in mind that sometimes your certificate issuer might be the same when serving directly as when serving through a CDN, if you got a certificate signed by the CDN, or if you gave your CDN your TLS key & certificate to use.

But if there is a different certificate being served, we will likely see it was signed by a different certificate authority (CA):

After sidestepping Cloudflare our certificate is issued by Sectigo, so this is more proof our changes are working.

Clean up

When you’re all done, open /etc/hosts again and either delete the line you added or comment it out by putting a # at the beginning of the line.

# 23.239.26.161 www.endpointdev.com

Save.

Then close all browser windows, quit, and restart, and when you go to www.endpointdev.com, Cloudflare should be in the middle again.

Mobile

Note that there is no easy way to do something like this on iOS with an iPhone or iPad, as far as I can tell. And on Android, you either need a rooted phone, or a fancy VPN that lets you change your hostname mapping. I tried Hosts Go and it worked, despite some annoying ads which you don’t have to deal with in an /etc/hosts file 😁.

If you need to test on mobile, or on multiple devices, you could also use a DNS server on your local network, and configure any device to point to your local DNS. Pi-hole is a great option for this; see our blog post from a couple years ago for some of its other features.

Working around SPF problems delivering to Gmail

2022-03-30T00:00:00+00:00

Photo by Garrett Skinner

Email deliverability

Legitimate email delivery keeps getting harder. Spammers and phishers never stop flooding everyone’s inboxes with unwanted and harmful email, so automated defenses against junk mail are necessary. But they are not perfect, and good email sometimes gets flagged as spam.

When sending important “transactional” email such as for account confirmations, password resets, and ecommerce receipts, it is often worth using a paid email delivery service to increase deliverability. Those typically cost a flat amount per month for up to a certain quota of outgoing email, with overage charges for messages beyond that.

Many of our clients use one of those services and generally they have all worked well and differ mostly in pricing and feature set. Popular choices include SendGrid, Mandrill, Postmark, Mailgun, and Amazon SES.

We continue to have many cases where we want to be able to send potentially large amounts of automated email to ourselves, our clients, or our systems. This is usually for testing, notifications, or internal delivery to special mailboxes separate from our main mailboxes.

These other uses for sending email keep us involved in the fight for good email deliverability from our own servers, which we have worked at over many years, long predating these paid email delivery services.

Sender Policy Framework

One of the longest-running tools to fight spam is SPF, the Sender Policy Framework.

SPF is an open standard that provides a way for a receiving mail server to verify that the sending server is authorized to send email for the message’s “envelope” sender domain. The envelope sender address or “return-path” is not normally seen by email recipients, but is used behind the scenes by servers. It may or may not be the same as the sender seen in the “From” header.

The SPF policy for each domain is set in a special DNS TXT record for that domain.

The important thing is that each sender’s email belongs to a domain with a valid SPF record showing that the sending servers are allowed to send for that domain, and that all other servers should not be allowed to send email for that domain.

For example, our endpointdev.com domain currently has this TXT record to define its SPF policy:

v=spf1 a:maildrop.endpointdev.com include:_spf.google.com include:servers.mcsv.net -all

Let’s look at each of those space-separated elements:

v=spf1 designates this TXT record as an SPF policy, version 1 (the only one so far).
a:maildrop.endpointdev.com means to allow the A (IPv4) and/or AAAA (IPv6) IP address(es) of hostname maildrop.endpointdev.com as a valid source.
include:_spf.google.com means to look up another DNS TXT record at _spf.google.com (for Gmail, our main email provider here) and add its SPF policy to ours.
include:servers.mcsv.net is the same thing, but for servers.mcsv.net (for Mailchimp, to allow it to deliver email newsletters for our domain).
-all means to disallow any other senders.

With such a policy, receiving mail servers can immediately reject any incoming email claiming to be sent by us for our domain if it didn’t come from one of our designated servers.

This obviously doesn’t stop all spam, but it stops a whole class of forged senders, which is very helpful.

The key point to note is that SPF applies to the sending email server at the moment it connects to the receiving email server. It doesn’t deal with anything else.

One other point to note is that SPF policies are limited to a fairly small total number of DNS lookups via include elements, so we can’t endlessly add new valid sending servers to our list.

Email server trails

Based on the above SPF policy, if we want to send email from address notifier@endpointdev.com, it will have to be sent through maildrop.endpointdev.com, Gmail, or Mailchimp. Messages coming from any other sending server should be rejected by the receiving server. They don’t have to behave that way, but it is in their interest to do so if they don’t like spam.

We have an internal server we’ll call dashboard.endpointdev.com, which sends email notifications from address notifier@endpointdev.com.

Since we don’t want to bloat our SPF policy, we’ll have our server dashboard.endpointdev.com route its outgoing email through our mail forwarding service called maildrop, which lives on two or more servers behind the DNS name maildrop.endpointdev.com.

This is a good idea for several reasons:

It keeps all our outgoing email flowing through a few places so we can easily monitor them for any problems.
We don’t need to have SMTP daemons running on all our servers just to send outbound email.
We don’t need to worry about the quotas or pricing of commercial emailing services when sending less-important or internal-only email.

Since SPF is designed for a receiving email server to check that the server connecting to it to send email is authorized to do so for that email address’s domain, it shouldn’t matter what server the email originated on.

Gmail misuses header information in SPF checks

We recently discovered that Gmail has been misusing email header information in its SPF checks.

When one of our outgoing emails originated from server dashboard.endpointdev.com and was then forwarded to maildrop.endpointdev.com which then delivered it to Gmail, Gmail looked at the earliest sender server it could find in the Received headers of the email message, found dashboard.endpointdev.com, and flagged it as an SPF failure because our SPF policy didn’t include dashboard.endpointdev.com [206.191.128.233].

This can be seen in this excerpt of relevant email headers. (Some specific details here were changed to protect the innocent.) Note that email headers appear in reverse chronological order, so the most recent events are at the top:

Received: from maildrop14.epinfra.net (maildrop14.epinfra.net. [69.25.178.35])
        by mx.google.com with ESMTPS id l20si5561179oos.78.2022.01.25.10.52.05
        for <notifications@endpointdev.com>
        (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
        Tue, 25 Jan 2022 10:52:05 -0800 (PST)
Received-SPF: fail (google.com: domain of notifier@endpointdev.com does not designate 206.191.128.233 as permitted
    sender) client-ip=206.191.128.233;
Authentication-Results: mx.google.com;
       dkim=pass header.i=@endpointdev.com header.s=maildrop header.b=hR445V77;
       spf=fail (google.com: domain of notifier@endpointdev.com does not designate 206.191.128.233 as permitted
    sender) smtp.mailfrom=notifier@endpointdev.com
Received: from dashboard.endpointdev.com (dashboard.endpointdev.com [206.191.128.233])
    by maildrop14.epinfra.net (Postfix) with ESMTP id A2AA03E8A7
    for <notifications@endpointdev.com>; Tue, 25 Jan 2022 18:52:05 +0000 (UTC)
To: <notifications@endpointdev.com>

That is wrong! The SPF check should have been done against maildrop14.epinfra.net [69.25.178.35] because that is the IP address that actually connected to Gmail to send the email. That server is one of our infrastructure hostnames allowed to send email as part of the maildrop.endpointdev.com DNS record, so checking it would have led Gmail to give a passing SPF result.

Why did Gmail do this? I don’t know, and at the time didn’t find any public discussion that would explain it. I suspect it has something to do with Gmail’s internal systems being comprised of many, many servers, and the SPF check being done long after the email was passed on from the initial receiving point through various other servers. Then Gmail parses the headers to find out who the sender was, and gets confused.

We can avoid this problem by not having maildrop mention our original sending server dashboard.endpointdev.com at all.

Why should it mention it in the first place? It’s helpful for tracing problems when debugging, but really is TMI (too much information) for normal email sending, and exposes internal infrastructure details that would be better omitted anyway.

Since dashboard.endpointdev.com is running the very flexible and configurable Postfix email server, we can direct it to remove any Received headers that mention our internal hostnames.

By default Postfix in /etc/postfix/main.cf has the header_checks directive set to look at a table to match regular expressions and take specified actions.

So we added a regular expression to match and designated the action IGNORE, to the file /etc/postfix/header_checks:

/^Received:\ (from|by)\ .*(epinfra\.net|endpointdev\.com|localhost|localdomain)/  IGNORE

Then we update the map database file so that it takes immediate effect for new email flowing through Postfix:

postmap /etc/postfix/header_checks

When we sent another notification email from dashboard.endpointdev.com and received it in Gmail we saw the email’s headers look like this:

Received: from maildrop14.epinfra.net (maildrop14.epinfra.net. [69.25.178.35])
        by mx.google.com with ESMTPS id g72si1894187vke.271.2022.01.25.11.03.37
        for <notifications@endpointdev.com>
        (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
        Tue, 25 Jan 2022 11:03:37 -0800 (PST)
Received-SPF: pass (google.com: domain endpointdev.com configured 69.25.178.35 as internal address)
Authentication-Results: mx.google.com;
       dkim=pass header.i=@endpointdev.com header.s=maildrop header.b=qkccUkkU;
       spf=pass (google.com: domain endpointdev.com configured 69.25.178.35 as internal address)
    smtp.mailfrom=notifier@endpointdev.com
To: <notifications@endpointdev.com>

There is no more mention of dashboard or its IP address, so Gmail runs its SPF check against the proper IP address 69.25.178.35 which belongs to server maildrop14.epinfra.net which is part of the maildrop.endpointdev.com DNS name. Gmail now validates that IP address is allowed to send for the endpointdev.com domain and gives a “pass” result for its SPF check.

Perhaps this will help your legitimate email delivery too!

Reference

SRV DNS records in Terraform and Cloudflare

2018-06-26T00:00:00+00:00

(Photo by David Goehring, CC BY 2.0, cropped)

At End Point we are using Terraform for a few clients to manage their web hosting infrastructure as code (IaC). Terraform is particularly helpful when working with multiple cloud or infrastructure providers and stitching together their services.

For example, for one web application that involves failover from the primary production infrastructure to a secondary location at a different provider, we are using Cloudflare as a CDN to provide caching, DDoS mitigation, and traffic routing in front of virtual servers at DigitalOcean and Amazon Web Services (AWS).

We decided we wanted to store all of their infrastructure configuration in Terraform, not just what is required for the web application, so we can recreate their entire infrastructure from their Git repository.

This all went fine until we got to their email DNS records. Our client is using Microsoft Office 365 for their email, which requires some SRV records. Terraform’s Cloudflare provider works fine with the universal MX records, but when we first wanted to do this, the Terraform provider for Cloudflare did not support SRV records at all.

Luckily for us, Terraform recently (6 April 2018) gained support for DNS SRV records as mentioned in the release notes and described in more detail in the pull request that added the feature.

Great! So now we can get on with this.

I began by naively assuming that the SRV record data should be given in space-separated form like many DNS interfaces use, including BIND and Cloudflare’s web interface itself. I tried setting it like this:

resource "cloudflare_record" "_sipfederationtls_tcp" {
  domain = "${var.domain}"
  name   = "_sip._tcp.${var.subdomain}"
  type   = "SRV"
  value  = "100 1 443 sipdir.online.lync.com."
}

But that resulted in an error. So when in doubt, consult the documentation, right? I did that:

Those make it clear that a data element is required, but give no indication what format is needed.

This is not currently documented for Cloudflare’s API, nor for Terraform’s Cloudflare provider. User EpiqSty helpfully recorded asking Cloudflare’s support, who recommended reverse-engineering the format:

In the meantime, what I would suggest is you can create a SRV record via our UI and then you can use the API to do a GET request on the DNS records, that should show you all the fields that are required.

Ok, then!

It turns out that we must provide the component parts of the SRV record disassembled in key/value pairs as the Cloudflare API expects, which looks like this in Terraform config:

resource "cloudflare_record" "_sip_tls" {
  domain = "${var.domain}"
  name   = "_sip._tls.${var.subdomain}"
  type   = "SRV"
  data   = {
    service  = "_sip"
    proto    = "_tls"
    name     = "${var.subdomain}."
    priority = 100
    weight   = 1
    port     = 443
    target   = "sipdir.online.lync.com."
  }
}

One unexpected aspect of this is that it requires duplicating the parts that make up the name although the name doesn’t seem to be used by Cloudflare, which assembles the name from the data components service, proto, and name. The name is apparently just there for Terraform, unlike other DNS records where the name is used as the DNS record value.

The Terraform Cloudflare provider also gained support for the much more obscure DNS LOC record type at the same time, and it works similarly.

Thanks to Ben Vickers who contributed the new feature to Terraform!

systemd: a primer from the trenches

2018-06-18T00:00:00+00:00

Gears image by Guy Sie, CC BY-SA 2.0, cropped & scaled

systemctl: Let’s get back to basics

‘‘Help me systemd, you are my only hope.’’

Sometimes going back to day zero brings clarity to what seems like hopeless or frustrating situation for users from the Unix SysV init world. Caveat: I previously worked at Red Hat for many years before joining the excellent team at End Point and I have been using systemd for as long. I quite honestly have forgotten most of the SysV init days. Although at End Point we work daily on Debian, Ubuntu, CentOS, and BSD variants.

Here is a short and sweet primer to get your fingers wet, before we dive into some of the heavier subjects with systemd.

Did you know that systemd has many utilities you can run?

systemctl
timedatectl
journalctl
loginctl
systemd-notify
systemd-analyze - analyze system
systemd-cgls - show cgroup tree
systemd-cgtop
systemd-nspawn

And systemd consists of several daemons:

systemd
journald
networkd
logind
timedated
udevd
system-boot
tmpfiles
session

That’s a long way from the old SysV init days. But in all essence it’s not that different. The one thing that stands out to me is we have more information with less typing then previously. That can only be a good thing, right?

Well, let’s see! There are many many web pages out there that list systemd or systemctl switches/flags. However in everyday use I want to speed up the work I do, I want information at my fingertips, and I find flags and switches which mean something sure do make it easier.

Pro Tip 1: Tab completion

Before you begin playing with the commands, you should install bash-completion. Some distros don’t auto-complete with systemd until you install that, and without tab auto-completion you miss out on a lot of systemctl.

As an example when you tab for completion you will see many of the systemctl options:

# systemctl
add-requires           enable                 is-system-running      preset                 show
add-wants              exit                   kexec                  preset-all             show-environment
cancel                 force-reload           kill                   reboot                 start
cat                    get-default            link                   reenable               status
condreload             halt                   list-dependencies      reload                 stop
condrestart            help                   list-jobs              reload-or-restart      suspend
condstop               hibernate              list-machines          rescue                 switch-root
daemon-reexec          hybrid-sleep           list-sockets           reset-failed           try-reload-or-restart
daemon-reload          import-environment     list-timers            restart                try-restart
default                is-active              list-unit-files        revert                 unmask
disable                is-enabled             list-units             set-default            unset-environment
edit                   is-failed              mask                   set-environment
emergency              isolate                poweroff               set-property

systemctl vs. old school commands

Here we will list out new systemctl commands and the corresponding old SysV command, followed by systemctl flags and explanation.

Go ahead and run each command to get a feel for what it displays. Remember each command usually has switches/flags you can use.

Let’s start at the top and work down:

systemctl

Formerly: service

Used in conjunction with ABRT it can show you some great debug info and runtime metadata, categorized by their respective groupings of loaded, active, running and a description of the unit.

systemctl status

Formerly: service --status-all or initctl list.

Check all system services’ status. Normally during a server update I will run this and output it to a file. When the server reboots I can run it again and diff this file to ensure all things started.

The output is great. It shows me the PID path and potentially the arguments which were run for the process or service. Saves me psing the process.

Note that each distro deals differently with service --status-all output.

systemctl status serviceName -l

Formerly: service serviceName status

Good flags: is-active, -a, -l

As it suggests, show me the status and information related to the service unit file. Other good info included is whether the service is enabled or chkconfig is on, uptime, PID and cgroup info, and any other information associated with the service.

Tips:

The -l flag will usually output enough information to diagnose a service start or reload problem without having to go into the logs.
You can view more than one service by separating them with spaces, e.g.: systemctl status httpd mysql postfix.

systemctl enable|disable NameofService

Formerly: chkconfig ServiceName on|off

You might find on some distros that chkconfig is still present. It doesn’t do what you think it does with systemd systems.

systemctl start|stop|restart httpd

Formerly: service httpd start|stop|restart

Good unit commands: reload-or-restart

As it suggests, start, stop, or restart services/processes/units.

The reload-or-restart command tells the services to reload if it is able and if not then restart, similar to the old service serviceName force-reload. Some services don’t allow a reload. Nagios is one example where a reload-or-restart works because it doesn’t allow reloads.

systemctl reload httpd

Formerly: service httpd reload

Perform a graceful reload of a configuration you may have just changed. Example: I’ve just made some changes to httpd conf and need to gracefully reload them without restarting the web service.

systemctl daemon reload

Formerly: chkconfig serviceName --add

Graceful reloads configuration files on a running service/process. See below for an explanation of “daemon reload”. Basically, if you have added in a new service and made many config changes, use daemon-reload.

systemctl list-unit-files

Good flags: --type=service

Formerly: ls /etc/rc.d/init.d/ /etc/rc.d/rc.local

Prints unit files from /usr/lib/systemd/system/ and /etc/systemd/system/. Slightly different to list-units; rarely used but has any interesting output. You may want to use this in monitoring scripts you write.

systemctl list-units

Good flags: -a (--all), -t serviceName

Formerly:

chkconfig --list
ntsysv
ls /etc/rc.d/init.d/ /etc/rc.d/rc.local
sysv-rc-conf
initctl list

I prefer to use list-units over list-unit-files. It shows more information and is shorter to type. Or you could install the sysvinit-utils package, which by default is not installed on systemd distros.

systemctl list-sockets

Formerly:

lsof
ss -s
cat /proc/net/*

Sockets of all types can be viewed from one command. Although time to time I will still use lsof or ss depending on the socket type I want to look at.

systemctl list-timers

Formerly: crontab -e

systemd offers a way to schedule tasks like crontab. I’m going to go out on a limb here and say they both do the same thing, except systemd timers may be more readable by human eyes, are logged to the journal, easier to debug and enable or disable.

My caveat is that I still use cron jobs in my daily work, because I’m familiar with them and emailing is still an issue from a timer.

Tip:

set a systemd timer to a calendar day, month, year to trigger.

systemctl list-jobs

Requires further explanation in another blog post.

systemctl –failed

Show me failed services. systemctl status will highlight at the top if units have failed, especially useful after a reboot.

systemctl get-default

Formerly:

runlevel
chkconfig --list

To list targets run systemctl list-units -t target.

Gets the run level default for the system. Not often used, but good to know when you start having boot issues or need to change to a different run level to fix things.

Tips:

There are many tables comparing the SysV runlevels to systemd. Get your Google on and search.
systemctl set-default graphical.target will set a graphical user shell. For all those that like a good desktop.

systemctl set-default multi-user.target

Formerly: telinit runlevel

The systemd multi-user.target is equivalent to runlevels 2, 3, and 4.

systemctl shutdown or reboot

Formerly: shutdown -r|-h now

Reboot/shutdown and poweroff the system. Personally I still use shutdown.

systemctl cat serviceName

Formerly: cat /etc/init.d/$init_file

Shows me the system service (unit) file contents and options. We will go more into these commands later as we work through building and maintaining our own unit files.

Tips:

systemctl cat shows all unit file information and snippets involved with the unit file.
If you use Vim and Arch Linux, you can enable vim-systemd to help with syntax highlighting.

systemctl list-dependencies serviceName

What’s really depending on a given service. The --all flag will show everything from --before, --after, --reverse.

systemctl show serviceName

Flags: -p shows a single property of a service.

Shows more than using systemctl cat servicename. Don’t forget tab completion is your friend. Running the -p flag and using tab will help you.

systemctl mask serviceName

Formerly: update-rc.d serviceName disable

Never want someone starting a service ever? mask is your friend and a little sneaky. See if your system admins pick this one up. Good April Fool’s day trick on them. Use unmask to return it to its normal state.

systemctl edit serviceName

Formerly: edit /etc/init.d/scriptName

Good options: --full

Yes, that’s right! Edit the service file without having to go find it on disk. That has saved me a bit of time.

Tips:

Careful with this. The plain edit creates an override file in /etc/systemd/system to complement the original unit file.
If you need to edit the original unit file use the --full flag, which allows you to edit the unit file without creating a snippet.
If you make a mistake in your unit file: systemctl revert serviceName

systemctl -o

Formerly: Edit Apache config to set log level to warn or debug, /etc/init.d/httpd reload, view logs

Good options: --output=verbose

Particularly good if you have a service acting up. Outputs a short standard message or a very verbose message using different flags.

Tip:

journalctl -u serviceName can help you here, but I often find it easier to include --output=verbose

systemctl isolate

This deserves its own small blog post. Isolate can be used to rescue systems automagically following kernel reboot failures, but requires some special work.

systemd-delta

Check your unit files to see if someone has been changing things on you. Especially useful if you are writing your own unit files.

Tip:

Used in conjunction with systemctl cat or edit, delta can help you see what was what.

Gotchas

Sometimes you need to use the suffix such as ‘config_file@openvpn.service’. systemd will always think services are services unless you use the suffix like .target or .socket, so make sure you tell the system so.
If you want multiple services running use a prefix, such as; ssh1@sshd.service ssh2@sshd.service with different configs. Handy for multiple openvpn servers.
Mount points will always be determined as mount points.
If you have made a lot of configuration changes and want to gracefully load these without restarting everything try systemctl daemon reload. This is not the same as the above reload action. Daemon reload is for systemd and not the unit files it controls. This is a safe command to run since it keeps sockets open while it does its thing.
Reboots on newer systems only really need to be done when new kernels are presented. Systemd is gracious and good at processing new packages and enabling these changes during a yum update.
Autocomplete on CentOS is not present until you install bash-completion. systemctl takes on a life of its own when you install this utility. Tabbing out will list all systemctl options. Very handy!
Systemd does not use the /etc/inittab file even if you have it present.
Converting your init scripts to systemd is easier than you think. Create a systemd unit file and add in 10 basic lines to call the bin or init script. You have basically created a systemd managed init script. Don’t forget to go back and one day convert it completely.
Targets vs. runlevels: A target in systemd is a runlevel in sysV, names replace numbers; runlevel 3 = multi-user.target, runlevel 1 = rescue.target

Who’s got the goods on speed?

Is systemd faster than SysV init? Parallel processing says it is? You be the judge!

One great test is to build a new machine which doesn’t have systemd installed. Reboot the machine and check your boot time. On an older SysV system you may have use “tuned”, “systemtap”, “numastat” etc. to gather performance information.

Then install systemd. Better still, upgrade from a old version of Linux to a new version of linux from ‘init’ to ‘systemd’ and then run ‘systemd-analyze’.

systemd-analyze will show you boot times. Notice that systemd starts fewer services at boot because it only starts what is necessary to get the server booting.

Not a bad way to collect a baseline on a newly-built server. Add it to your Ansible facts for the server so you have a historical view and collection of boot times. In fact add it to your monitoring system and be proactive in your monitoring of server boot times while performing your maintenance cycles.

How do you see what is taking its sweet time during boot or what is borking your beautiful server following an update/upgrade? Wonder no more!

# systemd-analyze blame
# systemd-analyze time

For example on my system, I can see that my top 10 culprits for a potentially slow boot are:

# systemd-analyze blame
          7.346s dracut-initqueue.service
          6.787s systemd-cryptsetup@luks.service
          5.378s NetworkManager-wait-online.service
          2.308s abrtd.service
          1.444s docker.service
          1.395s plymouth-quit-wait.service
          1.358s lvm2-pvscan@8:1.service
          1.145s lvm2-pvscan@253:0.service
          1.072s fwupd.service
           609ms docker-storage-setup.service

I might disable the Docker service by systemctl disable docker and start it when I need it.

What else can systemd-analyze show me?

# systemd-analyze critical-chain
The time after the unit is active or started is printed after the "@" character.
The time the unit takes to start is printed after the "+" character.

graphical.target @5.209s
└─multi-user.target @5.209s
  └─abrt-journal-core.service @5.209s
    └─abrtd.service @2.897s +2.308s
      └─livesys.service @2.881s +13ms
        └─basic.target @2.804s
          └─sockets.target @2.804s
            └─dbus.socket @2.804s
              └─sysinit.target @2.797s
                └─systemd-update-utmp.service @2.782s +14ms
                  └─auditd.service @2.634s +145ms
                    └─systemd-tmpfiles-setup.service @2.590s +41ms
                      └─fedora-import-state.service @2.566s +22ms
                        └─local-fs.target @2.562s
                          └─run-user-42.mount @4.614s
                            └─local-fs-pre.target @964ms
                              └─lvm2-monitor.service @321ms +442ms
                                └─dm-event.socket @320ms
                                  └─-.slice

Let’s look at how long our network targets take to start:

# systemd-analyze critical-chain network.target
The time after the unit is active or started is printed after the "@" character.
The time the unit takes to start is printed after the "+" character.

network.target @2.618s
└─network.service @2.403s +214ms
  └─NetworkManager-wait-online.service @1.458s +941ms
    └─NetworkManager.service @1.417s +40ms
      └─network-pre.target @1.415s
        └─firewalld.service @976ms +438ms
          └─polkit.service @891ms +83ms
            └─basic.target @885ms
              └─paths.target @885ms
                └─brandbot.path @885ms
                  └─sysinit.target @881ms
                    └─systemd-update-utmp.service @872ms +7ms
                      └─auditd.service @702ms +168ms
                        └─systemd-tmpfiles-setup.service @676ms +24ms
                          └─rhel-import-state.service @653ms +22ms
                            └─local-fs.target @652ms
                              └─boot.mount @523ms +128ms
                                └─local-fs-pre.target @522ms
                                  └─lvm2-monitor.service @369ms +152ms
                                    └─lvm2-lvmetad.service @397ms
                                      └─lvm2-lvmetad.socket @368ms
                                        └─-.slice

Let’s go one step further and output the entire system hierarchy. With this one you get a nice image:

# systemd-analyze dot | dot -Tpng -o system-stuff.png

Pro Tip 2: Remote commands

I have a server which needs a service restarted or checked constantly. Running systemctl remotely will show me or allow me to do this:

# systemctl status sshd -H root@server.domain.tld

Or:

# systemctl -H root@server.domain.tld status httpd

I might make an alias for it. Obviously this is a pretty useless example, if you’re having to manually do this for a service/process you should fix the problem on the server. However for edge cases it can be quite handy. Use your imagination: We could use this for monitoring which takes a local ssh user found on all machines and pass this for some returned output to a monitoring server.

Pro Tip 3: What relies on my service

Figure out what targets/runlevel a target runs at:

# systemctl show httpd -p wants multi-user.target

Pro Tip 4: Monitoring

Check processes, service association, and busiest processes. You can still grep/awk the output if you wish.

Instead of using top or something like this:

ps xawf -eo pid,user,cgroup,args

use the following:

# systemd-cgls
# systemd-cgtop

Summing up

I know this is only touching the surface of systemctl or systemd as whole but from a day-to-day context this should help you play in the systemd world. I think the biggest part that people struggle with is workable, usable examples.

Stay tuned for future posts on unit files, unit targets, systemctl isolate and slices, journalctl, timedatectl, and loginctl.

Instant TLS Upgrades Through Proxy Magic!

2018-06-14T00:00:00+00:00

TLS shutdowns are real

The payment gateways have been warning for years about the impending and required TLS updates. Authorize.net and PayPal—to name a few—have stopped accepting transaction requests from servers using TLS 1.0. Despite the many warnings about this (and many delays in the final enforcement date), some projects are affected by this and payments are coming to a stop, customers cannot checkout, and e-commerce is at a standstill.

Ideally, getting to security compliance would include a larger migration to update your underlying operating system and your application. But a migration and software update can be an expensive project and in some cases, the business can’t wait weeks while this is done.

End Point has worked with several clients recently to try to remedy the situation by using a reverse proxy to fix this and we’ve had good success on getting payments flowing again.

What is a proxy?

A proxy is a mid-point, essentially a digital middleman, moving your data from one place to another. In two recent client instances, we ended up using nginx (the stack’s webserver) as the reverse proxy, basically running a separate server for just shuttling requests to/from the payment gateway. Since we want to be able to run the gateway in both live and test modes, we use two separate server definitions in our nginx include, one for each.

Since the proxy is talking to the gateway in TLS 1.2 the payment gateway is happy. Since the application can talk http to the proxy running on the same machine, your application is happy. Since payments are now flowing, the business is happy.

Why use a proxy?

While we always impress on clients the importance of staying up-to-date with their entire stack (operating system, language, application frameworks), this is not always practical for some sites, whether for cost reasons or some technical limitations which keep them on a specific library or framework version. In our case, these clients had been migrated to CentOS 7 already (which supports TLS 1.2), but the versions of Ruby and Ruby on Rails they were on were too old to be able to use the TLS 1.2 libraries built-in.

This can then be a fast way to get payments flowing again, ideally the first step on updating the site to work with a modern stack. This can also minimize development costs compared to doing a full migration or complete stack upgrade.

Steps

Run on an operating system version that supports TLS 1.2 natively. In practice for us, this is CentOS 7, but modern Debian, Ubuntu, and other Linux distributions would work.
Locate the existing payment gateway URLs for live and test modes. For Authorize.net, this differs depending on if you are connecting to their legacy systems or their modern systems.
Set up and configure the proxy. We used nginx for this; a later section covers the sample configs with explanations. We created a config file that we can just drop in an /etc/nginx/sites/ directory to be able to quickly set up and deploy the reverse proxies.
Adjust the configuration of the software to point to alternate payment gateway URLs. If you are using Rails’ ActiveMerchant, this can be accomplished via an application-level configuration override. We provide an example of the overriding of the live/test mode URLs for a Rails application later.

nginx sample config

# apitest.authorize.net

server {
    listen 127.0.0.1:1337 default_server;

    resolver 8.8.8.8;

    access_log /var/log/nginx/sites/authnet-proxies/access_log;
    error_log /var/log/nginx/sites/authnet-proxies/error_log warn;

    location / {
        proxy_pass_header Authorization;
        proxy_pass https://apitest.authorize.net$request_uri;
        proxy_set_header Host apitest.authorize.net;
        proxy_redirect off;
    }
}

# api.authorize.net

server {
    listen 127.0.0.1:1338 default_server;

    resolver 8.8.8.8;

    access_log /var/log/nginx/sites/authnet-proxies/access_log;
    error_log /var/log/nginx/sites/authnet-proxies/error_log warn;

    location / {
        proxy_pass_header Authorization;
        proxy_pass https://api.authorize.net$request_uri;
        proxy_set_header Host api.authorize.net;
        proxy_redirect off;
    }
}

In this file, we are defining a separate server block for each service (test mode and production), choosing an arbitrarily defined TCP port. Because nginx is speaking TLS 1.2 to the upstream gateway and we are speaking plain HTTP only to localhost and not storing any information regarding the request, we are able to fulfill the PCI DSS requirements.

A few notes about this:

We needed the resolver line for this to work given our configuration/setup because we are referring to the upstream servers by domain name instead of IP addresses; we just used the easily-memorable Google public DNS resolver servers for this purpose.
We need the Authorization header passed through the proxy, as this is what contains the site credentials when making a request from ActiveMerchant.
We need the Host header defined explicitly in the server block, as we want to override the localhost value that will be sent by the application due to application configuration. Authorize.net was (understandably) very picky about this being provided and correct.
The port numbers here are arbitrary and just need to match what we configure the application to connect to; one for the test server and one for production.
This setup will work for multiple development environments running on the same server as long as they are configured to point to the correct test/live server. There is nothing inherently insecure about doing this, as no authorization is shared, it just bounces the connection.

Rails sample application config

A configuration file for the Authorize.net CIM gateway, for clients using ActiveMerchant as the base for payment gateway integration:

require File.expand_path('../boot', __FILE__)
require 'rails/all'

module MyApp
  class Application < Rails::Application
    ActiveMerchant::Billing::AuthorizeNetCimGateway.test_url = 'http://localhost:1337/xml/v1/request.api'
    ActiveMerchant::Billing::AuthorizeNetCimGateway.live_url = 'http://localhost:1338/xml/v1/request.api'
  end
end

And another one for the legacy Authorize.net gateway:

require File.expand_path('../boot', __FILE__)
require 'rails/all'

module MyApp
  class Application < Rails::Application
    ActiveMerchant::Billing::AuthorizeNetGateway.test_url = 'http://localhost:1337/gateway/transaction.dll'
    ActiveMerchant::Billing::AuthorizeNetGateway.live_url = 'http://localhost:1338/gateway/transaction.dll'
  end
end

Ensure that you override the URLs depending on the gateway setup that your merchant account is set up for. Also make sure that your localhost URLs are using the ports corresponding to the server blocks set up in nginx to make sure these are proxying to the correct server.

While this example is using Authorize.net, you could use a similar setup for any payment gateway which allows you to set the URLs.

Summary

Hopefully this gives you some ideas about how you can use reverse proxies to upgrade the outgoing connection strength. If you need assistance getting something like this set up, feel free to contact us for help.

(Co-authored by Elizabeth Garrett Christensen.)

Logstash: Removing fields with empty values

2017-11-22T00:00:00+00:00

The Elastic stack is a nice toolkit for collecting, transporting, transforming, aggregating, searching, and reporting on log data from many sources. It was formerly known as the ELK stack, after its main components Elasticsearch, Logstash, and Kibana, but with the addition of Beats and other tools, the company now calls it simply the Elastic stack.

We are using it in a common configuration, on a central log server that receives logs via rsyslog over TLS, which are then stored in local files and processed further by Logstash.

When conservation is recommended

When forwarding logs on to SaaS log services such as Logentries, SumoLogic, etc., we have a limited amount of data transfer and storage allotted to us. So we need to either economize on what we send them, pay for a more expensive plan, or retain a shorter period of history.

For some very busy logs (nginx logs in JSON format) we decided to delete fields with empty values from the log event during the filter phase in Logstash. This removes a lot of data from the log message we send to the log service over the wire, and reduces the size of each log event stored in their system.

I expected this to be simple, but that expectation sometimes proves to be false. :)

Trying the prune filter

The most obvious way would be to use the Logstash prune filter, which is designed for just such a use case. However, the prune filter doesn’t handle nested keys, as explained in the documentation:

NOTE: This filter currently only support operations on top-level fields, i.e. whitelisting and blacklisting of subfields based on name or value does not work.

That is too bad.

Pruning with custom Ruby code

Several people have posted alternative solutions to this in the past. A representative recipe to have Logstash delete empty fields looked like this:

# This doesn’t work in Logstash 5 and newer ...
filter {
  ruby {
    code => "event.to_hash.delete_if {|field, value| value == '' }"
  }
}

And sadly, it doesn’t work.

Logstash 5 event API changes

It used to work with older versions of Logstash, but no longer. Logstash was originally written in Ruby, specifically JRuby for running on the JVM. But for Logstash 5 it was rewritten in Java, and though JRuby extensions are still possible, the Ruby event API has changed so that the log data is no longer provided as a mutable hash that the above code expects. (See also the Logstash event API documentation.)

Custom Ruby code to prune in Logstash 5+

So I came up with Ruby code that works using the new Logstash event API. It is more complicated , but it is still pretty straightforward:

filter {
  # remove fields with empty values
  ruby {
    code => "
      def walk_hash(parent, path, hash)
        path << parent if parent
        hash.each do |key, value|
          walk_hash(key, path, value) if value.is_a?(Hash)
          @paths << (path + [key]).map {|p| '[' + p + ']' }.join('')
        end
        path.pop
      end

      @paths = []
      walk_hash(nil, [], event.to_hash)

      @paths.each do |path|
        value = event.get(path)
        event.remove(path) if value.nil? || (value.respond_to?(:empty?) && value.empty?)
      end
    "
  }
}

We first recursively walk through the whole data structure that the API converts to a Ruby hash for us. We get all nested field names and store in an array their Logstash-style paths like "[nginx][access][upstream_addr]" that the API expects. Then we walk through the paths and use the API to check for empty values, and remove them. This way we also avoid changing the hash while still walking through it.

With that configuration and code in a file in /etc/logstash/conf.d/ (this is on CentOS 7 using the logstash RPM from Elastic) all the fields with empty values are removed.

Some other log event trimming

In addition we added a few other filters to remove or limit the size of fields that we are happy to have on our own central log server for archival or forensic purposes, but that we don’t need to send to our paid log service for the kinds of reporting we are doing there:

mutate {
  remove_field => [
    "@version", "beat", "host", "input_type", "offset", "source", "type",
    "[geoip][location]",
    "[nginx][access][pipe]",
    "[nginx][access][remote_port]",
    "[nginx][access][ssl_session_id]",
    "[nginx][access][ssl_session_reused]",
    "[nginx][access][upstream_bytes_received]",
    "[nginx][access][upstream_connect_time]",
    "[nginx][access][upstream_response_length]",
    "[nginx][access][upstream_status]"
  ]
}

if "beats_input_codec_plain_applied" in [tags] {
  mutate {
    remove_tag => ["beats_input_codec_plain_applied"]
  }
}

truncate {
  length_bytes => 1024
}

For example, sometimes the client sends an absurdly long HTTP Referer request header, or the URI requested is very long—we see plenty longer than 5000 characters. We are happy to truncate those to save space.

We also do not need to waste space in our paid log service with the repetitive tag beats_input_codec_plain_applied or the same Filebeat version in every single log event.

Finis

This is working for us on Logstash 5.6.3, but should work on Logstash 5.0 and newer.

From Zero to HTTPS in an afternoon

2017-11-20T00:00:00+00:00

I’ve been hosting my own humble personal web site since 2012. I had never bothered setting up HTTPS for my domain, but after hearing about the Let’s Encrypt project, I was completely out of excuses.

For the unfamiliar, Let’s Encrypt offers free and fully automatic HTTPS certificates. The web cares about HTTPS now more than ever. Deeply interactive interfaces like geolocation and user media (camera, microphone) are too sensitive to trust an insecure transport. By leveraging the security features present in modern browsers, users can expect a reasonable safety from attacks that would exploit the weaknesses of HTTP.

To take the security mission even further, I decided to completely containerize my server and expose only a couple of ports. Using a Docker composition made it very easy to deploy up-to-date nginx and keep it isolated from the rest of my host shard.

The first mission was to set up certificates with certbot, the EFF’s free certificate tool. certbot has a plugin that writes nginx configuration for you, but in this case I didn’t want nginx installed on my host at all. Instead of following the nginx-specific instructions for my platform, I opted for the webroot plugin to just give me a certificate and let me figure out how to set it up. A certbot invocation and a few seconds later I have certificates for my site in /etc/letsencrypt/live/www.mvollrath.net.

Next I went shopping for nginx Docker images. The official nginx image has everything I want: the latest and greatest mainline nginx based on stable Debian. I considered the Alpine variant, but felt like Debian was a better choice for me; familiarity outweighs a few tens of MB of image size.

The nginx image ships with a default configuration serving a single root directory over HTTP. Since HTTPS was the point of this experiment, I set out to correct this. I started by creating a project directory on the host to house all the configuration needed to build out my server. Then I started up a container with the vanilla configuration and copied config files /etc/nginx/nginx.conf and /etc/nginx/conf.d/default.conf out of the container to the project directory. With those config files now in my possession, I created a simple Dockerfile to inject them into a new image based on the library nginx image.

FROM nginx:latest

COPY nginx.conf /etc/nginx/nginx.conf
COPY default.conf /etc/nginx/conf.d/default.conf

With that out of the way, I started hacking config to create my ideal HTTPS server. First I set up a redirect to force all traffic to the HTTPS site.

server {
    listen 80;
    server_name mvollrath.net www.mvollrath.net;
    return 301 https://$server_name$request_uri;
}

Now I would need a good way to get my certificates into the container. Docker compose has a handy secrets directive to make this really painless.

version: '3.1'  # must be at least 3.1 for secrets feature
secrets:
  ssl_privkey:
    file: "/etc/letsencrypt/live/www.mvollrath.net/privkey.pem"
  ssl_fullchain:
    file: "/etc/letsencrypt/live/www.mvollrath.net/fullchain.pem"
services:
  nginx:
    container_name: "nginx"
    build: "."
    ports:
    - "80:80"
    - "443:443"
    volumes:
    - "/home/matt/htdocs:/usr/share/nginx/html:ro"
    restart: "on-failure"
    secrets:
    - "ssl_privkey"
    - "ssl_fullchain"

This mounts the provided secrets in /run/secrets to be scooped up by the site config.

server {
    listen 443 ssl http2 default_server;

    server_name mvollrath.net www.mvollrath.net;
    ssl_certificate /run/secrets/ssl_fullchain;
    ssl_certificate_key /run/secrets/ssl_privkey;

    [...]
}

Now I can update my server by running docker-compose build --pull and then docker-compose up -d. This may cause a momentary outage while the containers are being swapped, but for a personal site this is nothing to sweat over. I dropped these commands in a cron script since I like updates but would rather not have to think about updates.

With my new HTTPS site now exposed to the world, I found some free HTTPS validation tools to check my work and optimize the configuration a few notches beyond the “pretty good” nginx defaults. If you’ve deployed an HTTPS site for work or pleasure, check out the collection of web security tools rounded up by Phin in September.

I was really happy with the Let’s Encrypt tools and user experience. If you’re still hosting HTTP with no HTTPS option, consider using the free tools to get your free certificate and help your users protect their privacy. If you’re interested in using HTTPS wherever available, consider using the HTTPS Everywhere extension offered by the EFF.

Web Security Services Roundup

2017-09-19T00:00:00+00:00

Security is often a very difficult thing to get right, especially when it’s not easy to find reliable or up-to-date information or the process of testing can be confusing and complicated. We have a lot of history and experience working on the security of websites and servers, and we’ve found many tools and websites to be very helpful. Here is a collection of those.

Server-side security

There are a number of tools available that can scan your website to check for common vulnerabilities and the quality of SSL/TLS configuration, as well as give great tips on how to improve security for your website.

Qualys SSL Labs Server Test takes a simple domain name, performs a series of tests from a variety of clients, and returns a simple letter grade (from A+ down to F) indicating the quality of your SSL/TLS configuration, as well as a detailed summary for a host of configuration options. It covers certificates key and algorithms; TLS and SSL configurations; cipher suites; handshakes on a wide variety of platforms including Android, iOS, Chrome, Firefox, Internet Explorer and Edge, Safari, and others; common protocols and vulnerabilities; and other details.
HTTP Security Report does a similar scan, but provides a much more simplified summary of a website, with a numeric score from 0 to 100. It gives a simple, easy to understand list of results, with a green check mark or a red X to indicate whether something is configured for security or not. It also provides short paragraphs explaining settings and recommended configurations.
HT-Bridge SSL/TLS Server Test is very similar to Qualys SSL Labs Server Test, but provides some valuable extra information, such as PCI-DSS, HIPAA, and NIST guidelines compliance, as well as industry best practices and basic analysis of third-party content.
securityheaders.io is another letter-grade scan, but focuses on server headers only. It provides simple explanations for each recommended server header and links to guides on how to configure them correctly.
Observatory by Mozilla scans and gives information on HTTP, TLS, and SSH configuration, as well as simple summaries from other websites, including Qualys, HT-Bridge, and securityheaders.io as covered above.
SSL-Tools is focused on SSL and TLS configuration and certificates, with tools to scan websites and mail servers, check for common vulnerabilities, and decode certificates.
Microsoft Site Scan performs a series of simple tests, focused more on general website guidelines and best practices, including tests for outdated libraries and plugins which can be a security issue.
testssl.sh, the final website scanning tool I’ll cover, is a more advanced bash script that covers many of the same things these other websites do, but provides lots of options for fine-tuning test methods, returned information, and testing abnormal configurations. It’s also open source and doesn’t rely on any third parties.

These websites provide valuable information on SSL/TLS which can be used to create a secure, fast, and functional server configuration:

Security/Server Side TLS on the Mozilla wiki is a fantastic page which provides great summaries, recommendations, and reference information on many TLS topics, including handshakes, OCSP Stapling, HSTS, HPKP, certificate ciphers, and common attacks.
Mozilla SSL Configuration Generator is a simple tool that generates boilerplate server configuration files for common servers, including Apache and Nginx, and specific server and OpenSSL versions. It also allows you to target “modern”, “intermediate”, or “old” clients and servers, which will give the best configuration possible for each level.
Is TLS Fast Yet? is a great, simple, and to-the-point informational website which explains why TLS is so important and how to improve its performance so it has the smallest impact possible on your website’s speed.

Client-side security

These websites provide information and diagnostic tools to ensure that you are using a secure browser.

badssl.com gives a list of links to subdomains with various SSL configurations, including badly configured SSL, so you can have a good idea of what a well-configured website looks like versus one with errors in configuration, weak ciphers or key exchange protocols, or insecure HTTP forms.
IPv6 Test checks your network and browser for IPv6 support, showing you your ISP, reverse DNS pointers, both your IPv4 and IPv6 addresses, and giving an idea of when your computer or network may have problems with dual-stack IPv4 + IPv6 remote hosts or DNS.
How’s My SSL? and Qualys Labs SSL Client Test both check your browser for support of SSL/TLS versions, protocols, ciphers, and features, as well as susceptibility to common vulnerabilities.

General Tools

NeverSSL is a simple website that promises to never use SSL. Many public wifi networks require you to go through a payment or login page, which can be blocked when trying to access a well-secured website such as Google, Facebook, Twitter, or Amazon, which can cause trouble connecting to that website. NeverSSL provides an easy and simple way to access that login website.
crt.sh is a search engine for public TLS certificate information. It provides a history of certificates for a given domain name, with information including issuer and issue date, as well as an advanced search.
Digital Attack Map is an interactive map showing DDoS attacks across the world.
The Internet-Wide Scan Data Repository is a public archive of scans across the internet, intended for research and provided by the University of Michigan Censys Team.
take-a-screenshot.org is a simple website that shows how to take a screenshot on a variety of operating systems and desktop environments. It’s a fantastic tool to help less technically-minded people share their screens or issues they’re having.

Linode IPv6 issues with NetworkManager on CentOS 7

2017-04-04T00:00:00+00:00

In End Point, we use different hosting providers based on the specific task needs. One provider we use extensively with good results is Linode.

During a routine CentOS 7 system update, we noticed a very strange behavior where our IPv6 assigned server address was wrong after restarting the server.

IPv6 on Linode and SLAAC

Linode is offering IPv6 on all their VPS, and IPv6 dynamic addresses are assigned to servers using SLAAC.

In the provided CentOS 7 server image, this is managed by NetworkManager by default. After some troubleshooting, we noticed that during the update the NetworkManager package was upgraded from 1.0.6 to 1.4.0.

This was a major update, and it turned out that the problem was a change in the configuration defaults between the two version.

Privacy stable addressing

Since 1.2, NetworkManager added the Stable Privacy Addressing feature. This allows for some form of tracking prevention, with the IPv6 address to be stable on a network but changing when entering another network, and still remain unique.

This new interesting feature has apparently become the default after the update, with the ipv6.addr-gen-mode property set to “stable-privacy”. Setting it to “eui64” maintains the old default behavior.

Privacy Extension

Another feature apparently also caused some problems on our VPS: the Privacy Extension. This is a simple mechanism that somewhat randomizes the network hardware’s (MAC) address, to add another layer of privacy. Alas, this is used in address generation, and that randomization seemed to be part of the problem we were seeing.

This too has become the default, with the ipv6.ip6-privacy property set to 1. Setting it to 0 turns off the feature.

To sum it up

In the end, after the update, we could restore the old behavior and resolve our issues by running, in a root shell:

nmcli connection modify "Wired connection 1" ipv6.ip6-privacy 0
nmcli connection modify "Wired connection 1" ipv6.addr-gen-mode eui64

After a reboot, the IPv6 address finally matched the one actually assigned by Linode, and everything was working ok again.

If you want to know more on Privacy Extensions and Privacy Stable Addressing, this great blog post by Lubomir Rintel helped us a lot understanding what was going on.

TriSano Case Study

2017-01-24T00:00:00+00:00

Overview

End Point has been working with state and local health agencies since 2008. We host disease outbreak surveillance and management systems and have expertise providing clients with the sophisticated case management tools they need to deliver in-house analysis, visualization, and reporting—combined with the flexibility to comply with changing state and federal requirements. End Point provides the hosting infrastructure, database, reporting systems, and customizations that agencies need in order to service to their populations.

Our work with health agencies is a great example of End Point’s ability to use our experience in open source technology, Ruby on Rails, manage and back up large secure datasets, and integrate reporting systems to build and support a full-stack application. We will discuss one such client in this case study.

Why End Point?

End Point is a good fit for this project because of our expertise in several areas including reporting and our hosting capabilities. End Point has had a long history of consultant experts in PostgreSQL and Ruby on Rails, which are the core software behind this application.

Also, End Point specializes in customizing open-source software, which can save not-for-profit and state agencies valuable budget dollars they can invest in other social programs.

Due to the secure nature of the medical data in these database, we and our clients must adhere to all HIPAA and CDC policies regarding hosting of data handling, server hosting, and staff authorization and access auditing.

Team

Steve Yoman

Steve serves as the project manager for both communication and internal development for End Point’s relationship with the client. Steve brings many years in project management to the table for this job and does a great job keeping track of every last detail, quote, and contract item.

Selvakumar Arumugam

Selva is one of those rare engineers who is gifted with both development and DevOps expertise. He is the main developer on daily tasks related to the disease tracking system. He also does a great job navigating a complex hosting environment and has helped the client make strides towards their future goals.

Josh Tolley

Josh is one of End Point’s most knowledgeable database and reporting experts. Josh’s knowledge of PostgreSQL is extremely helpful to make sure that the data is secure and stable. He built and maintains a standalone reporting application based on Pentaho.

Application

The disease tracking system consists of several applications including a web application, reporting application, two messaging areas, and SOAP services that relay data between internal and external systems.

TriSano: The disease tracking web app is an open source Ruby on Rails application based on the TriSano product, originally built at the Collaborative Software Initiative. This is a role-based web application where large amounts of epidemiological data can be entered manually or by data transfer.

Pentaho: Pentaho is a PostgreSQL reporting application that allows you to run a separate reporting service or embed reports into your website. Pentaho has a community version and an enterprise version, which is what is used on this particular project. This reporting application provides OLAP services, dashboarding, and generates ad hoc and static reports. Josh Tolley customized Pentaho so that the client can download or create custom reports depending on their needs.

Two Messaging Area applications: The TriSano system also serves as the central repository for messaging feeds used to collect data from local health care providers, laboratories throughout the state, and the CDC.

SOAP services run between the TriSano web app, the Pentaho reporting application, and the client’s data systems translate messages into the correct formats and relay the information to each application.

Into the Future

Based on the success over 9+ years working on this project, the client continues to work with their End Point team to manage their few non open-source software licenses, create long term security strategies, and plan and implement all of their needs related to the continuous improvement and changes in epidemiology tracking. We partner with the client so they can focus their efforts on reading results and planning for the future health of their citizens. This ongoing partnership is something End Point is very proud to be a part of and we hope to continue our work in this field well into the future.

A Beginner’s Guide to PCI DSS Compliance and TLS Versions

2016-03-29T00:00:00+00:00

I recently did some research for one of End Point’s ecommerce clients on their PCI compliance and wanted to share some basic information for those of you who are new to this topic.

TLS

TLS (Transport Layer Security) is a standard for secure communications between applications. TLS is the current version of what used to be called SSL, the secure sockets layer. In the case of a financial transaction, this is the communication between the website selling a product and the end user. TLS works by encrypting data between two endpoints to ensure any sensitive data (such as financial details and private customer information) is exchanged securely. As security measures increase, new versions of TLS are released. To date, TLS 1.2 is the most up-to-date, with TLS 1.1 being considered safe, and TLS 1.0 being phased out. For details about OS versions supporting the latest TLS standards, please see Jon Jensen’s write-up here.

Compliance with PCI DSS

As all online retailers know, becoming and staying compliant with PCI DSS (Payment Card Industry Data Security Standard) is a big job. PCI is THE ecommerce security standard and in order to accept payment with Visa, MasterCard, American Express, and Discover, you must comply with their security standards.

As the Internet security landscape changes, PCI DSS standards are updated and reflect new risks and adjustments in security protections. As of today, PCI is requiring vendors to upgrade TLS 1.1 or above by June of 2016, with an optional extension until June 2018.

Compliance Assessors

Here’s where things get tricky. PCI does not actually do their own compliance, instead each merchant must have a neutral third party help them fulfill their PCI requirements. These are called ‘assessors’ and there are a large number of companies that offer this service along with help for other security-related tasks.

In preparation for the new requirements, many of the assessor companies are including the new TLS standards in their current compliance protocols.

What does that mean? Well, it means that even though PCI might not be requiring you to have TLS 1.1 until June of 2018, your compliance assessor might be require you to do it right now.

Bite the Bullet

So, now, given that you know you this change is coming AND you need it to get your compliance done, you might as well get your site updated. So where’s the catch?

Unsupported browsers

The big catch is that some browsers do not support TLS 1.1 or 1.2. In those cases, some of your users will not be able to complete a payment transaction and will instead hit an error screen and cannot continue. They are:

Internet Explorer on Windows XP
Internet Explorer older than version 11 on any version of Windows
the stock Android browser on versions of Android before 5.0
Safari 6 or older on Mac OS X 10.8 (Mountain Lion) or older
Safari on iOS 4 or older
very, very old versions of Firefox or Chrome that have been set not to auto-update

Okay, so how many people still use those old browsers? We’ll take a look at some of the breakdowns here:

http://www.w3schools.com/browsers/browsers_explorer.asp

You might be thinking, ‘That doesn’t seem like very many people’. And that’s true. However, every site has a different customer base and browsers use varies widely by demographics. So where can you go to find out what kinds of browser’s your customers use?

Google Analytics, your old friend

If you have Google Analytics setup, you can go through the Audience/Technology/Browser&OS screens to find out what kind of impact this might have.

Plan for the Worst

Now armed with your information, you will probably want to go ahead and get your website on the newest TLS version. The change is coming anyways but help your staff and web team plan for the worst by making sure everyone knows about the browser limitations and can help your customers through the process.

Server Compatibility Notes

For many ecommerce sites, enabling TLS 1.1 and 1.2 is easy, just changing a configuration setting and restarting the web server. But on older operating systems, such as the still supported and very popular Red Hat Enterprise Linux 5 and CentOS Linux 5, TLS 1.0 is the newest supported version. Various workarounds might be possible, but the only real solution is to migrate to a newer version of the operating system. There can be cost and time factors to consider, so it’s best to plan ahead. Ask us or your in-house developers whether a migration will be necessary!

Need Help?

As End Point’s client liaison, I’m happy to chat with anyone who needs answers or advice about PCI DSS and your ecommerce site.

E-commerce website encryption changes

2015-07-24T00:00:00+00:00

The big picture

Computer security is a moving target, and during the past few years it’s been moving faster than ever.

In the e-commerce world, the PCI Security Standards Council sets the rules for what merchants and vendors must do to have what they consider to be a sufficiently secure environment to handle cardholder data such as credit card numbers, expiration dates, and card security codes.

PCI DSS 3.1, released on 15 April 2015 puts us all on notice that TLS 1.0 is considered unfit to use for e-commerce website encryption (HTTPS), and will be disallowed soon. The new rules specify that new software implementations must not use TLS versions prior to 1.1. Existing implementations must require TLS 1.1 or 1.2 no later than 30 June 2016.

They provide some guidance on Migrating from SSL and early TLS and explain what is expected in more detail.

Long ago we were required to disable SSL 2, and last year we were expected to disable SSL 3, the predecessor to TLS 1.0. That turned out to not be particularly hard or cause too many problems, because almost all systems that supported SSL 3 also supported TLS 1.0.

This time we are not so lucky. Many clients (such as browsers) and servers did not support TLS beyond version 1.0 until fairly recently. That means much more work is involved in meeting these new requirements than just changing some settings and restarting servers.

Almost every client (browser) and server that supports TLS 1.1 also supports TLS 1.2, and almost everything that doesn’t support TLS 1.2 doesn’t support TLS 1.1 either. So to keep things simpler here I’ll just talk about TLS 1.0 vs. TLS 1.2 below, and TLS 1.1 can be assumed to apply as well where TLS 1.2 is mentioned.

At End Point we deploy, support, and host e-commerce sites for many customers, so I’ll talk about the server side of this first. Note that servers can act as both server and client in TLS connections, since servers often make outgoing HTTPS connections as well as accepting incoming requests. Let’s review the situation with each of the major Linux server operating systems.

Debian

Debian 8 is the current version, and supports TLS 1.2. It is scheduled to be supported until April 2020.

Debian 7 supports TLS 1.2, and has planned support until May 2018.

Debian’s support lifetime has historically depended on how quickly future releases come, but recently the project began to offer long-term support (LTS) for Debian 6, which was supposed to be at end of life, so it will be supported until February 2016. It also only supports TLS 1.0.

Ubuntu

Ubuntu’s long-term support (LTS) server versions are supported for 5 years. Currently supported versions 12.04 and 14.04 both handle TLS 1.2.

Some sites are still using Ubuntu 10.04, which supports only TLS 1.0, but its support ended in April 2015, so it should not be used any longer in any case.

Red Hat Enterprise Linux (RHEL) and CentOS

Red Hat and CentOS are “enterprise” operating systems with a very long support lifetime of 10 years. Because that is so long, the oldest supported versions may become practically unusable due to changes in the world such as the deprecation of TLS 1.0.

RHEL/CentOS 7 is the current version, supported until June 2024. It supports TLS 1.2.

RHEL/CentOS 6 is supported until November 2020. It is mostly ok for TLS 1.2. One exception is that the bundled version of curl doesn’t support TLS > 1.0 for some reason, so if you have applications making curl client calls to other systems, they may break without workarounds.

RHEL/CentOS 5 is the oldest version still supported, until March 2017, and it is very widely used, but it does not supprt TLS > 1.0.

Old server remediation

If you’re on an older server that doesn’t support TLS 1.2, the best thing to do is upgrade or migrate to a newer operating system, as soon as possible.

The common versions of OpenSSL that don’t support TLS 1.2 also do not support Server Name Indication used for hosting multiple HTTPS sites on the same IP address. That is now becoming more commonly used since the thing holding back acceptance was old versions of Windows XP that didn’t support it, and they are now are mostly dead. You can control whether you need SNI on the server side, avoiding it by continuing to get a separate IP address for each HTTPS site you host. But when you are a client of someone else’s service that requires SNI, you’ll wish you had it.

So migrate. That’s easier said than done, of course. Moving to a new OS version involves a lot of new system library versions, language versions, web server version, etc. Some things aren’t compatible. It takes work and time. That’s life, so accept it and move ahead. Advocate it, schedule it, do it. But in the meantime, if you must cope with old servers, there are some not entirely terrible options.

You could use plain HTTP on a local private network to talk to a newer server running stunnel or an nginx proxy to do the TLS layer, or use a VPN if you have no private network.

You can use CDN in front of your site, which will certainly support TLS 1.2, and covers the path between the end user and the CDN, at least.

You can build your own versions of OpenSSL, any libraries that link to OpenSSL such as curl or wget, Apache or nginx, etc. This is tempting but is a terrible option, because you are almost certain to not update this hand-crafted stack often enough in the future to protect against new vulnerabilities in it. Sidestepping the operating system’s native package management for core infrastructure software like this is usually a mistake.

You could avoid that problem by using someone else’s backported parallel-install packages of all that, if you can find some, and if you think they’re trustworthy, and if they’re going to maintain them so you can get later updates. I’m not familiar with anyone doing this, but it may be out there and could be hired for the right ongoing price.

But the best bet is to start planning your upgrade or migration as soon as possible.

Browser support for TLS 1.2

Of course the other half of the connection is the client, primarily end-users’ web browsers. On Wikipedia is a very detailed table showing various browsers’ support of various features, broken down by version, here: TLS support history of web browsers. My summary follows:

Google Chrome and Mozilla Firefox have been automatically updating themselves for a long time, so unless you or your system administrator have disabled the auto-update, they will work with TLS 1.2.

Internet Explorer 8, 9, and 10 support TLS 1.2, but it is disabled by default. Not until IE 11 can TLS 1.2 be counted on to work.

Apple Safari 7, 8, and 9 for Mac OS X support TLS 1.2.

The built-in browser on Android < 4.4 doesn’t support TLS > 1.0, and Android 4.4 has TLS > 1.0 disabled by default, which is the same thing for most users. So anyone with Android < 5.0 will not be able to connect to your site unless they’re using a separate and newer mobile browser such as Chrome and Firefox.

Browser support for TLS 1.0

I have not heard of any timelines being announced for browsers to disable TLS 1.0 yet. I suspect that’s because there are still so many servers that only support TLS 1.0. But before too long we may start to see servers catch up and then I expect browsers will eventually disable TLS 1.0.

Other clients

There are too many non-browser clients to list here. We’ve already mentioned curl; there is also wget, and the web client libraries in Perl, Python, Ruby, and Java. PHP uses libcurl. NodeJS and Go are likely not from the operating system and newer than it, so may be more current. At any rate, some of those clients will be old and won’t support TLS 1.2, so when other sites stop allowing TLS 1.0 connections, whatever you were talking to them for will stop working.

PCI DSS will require client applications to stop using TLS 1.0 also, which may mean that applications need to be configured to require TLS 1.2 for outgoing HTTPS connections.

Summary

Your systems need to stop supporting TLS 1.0 by June 2016 at the latest. Start planning the migration now! We are available to help our current and new clients test, assess needs, and plan upgrades and migrations.

Reference

“SSL and early TLS are no longer considered to be strong cryptography and cannot be used as a security control after June 30, 2016. Prior to this date, existing implementations that use SSL and/or early TLS must have a formal Risk Mitigation and Migration Plan in place. Effective immediately, new implementations must not use SSL or early TLS.” ―Sections 2.2.3, 2.3, 4.1 of PCI DSS 3.1
How’s My SSL? — tells you how secure your TLS client is
Qualys SSL Labs client test — a different client test
Qualys SSL Labs server test — very useful, and shows client compatibility with a server’s configuration
linuxlifecycle.com — Support Life Cycles for Enterprise Linux Distributions

Cron Wrapper: Keep your cron jobs environment sane

2015-02-06T00:00:00+00:00

It is becoming more common for developers to not use the operating system packages for programming languages. Perl, Python, Ruby, and PHP are all making releases of new versions faster than the operating systems can keep up (at least without causing compatibility problems). There are now plenty of tools to help with this problem. For Perl we have Perlbrew and plenv. For Ruby there is rbenv and RVM. For Python there is Virtualenv. For PHP there is PHP version. These tools are all great for many different reasons but they all have issues when being used with cron jobs. The cron environment is very minimal on purpose. It has a very restrictive path, very few environment variables and other issues. As far as I know, all of these tools would prefer using the env command to get the right version of the language you are using. This works great while you are logged in but tends to fail bad as a cron job. The cron wrapper script is a super simple script that you put before whatever you want to run in your crontab which will ensure you have the right environment variables set.

#!/bin/bash -l

exec "$@"

The crontab entry would look something like this:

34 12 * * * bin/cron-wrapper bin/blog-update.pl

The -l on the executing of bash makes it act like it is logging in. Therefore it picks up anything in the ~/.bash_profile and has that available to the env command. This means the cron job runs in the same environment that is setup when you run it from the command line, helping to stop those annoying times where it works fine from the command line but breaks in cron. Jon Jensen went into much greater detail on the benefits of using the -l here. Hope this helps!

CentOS 7 on Hetzner server with more than 2 TB disk

2015-01-22T00:00:00+00:00

We use a variety of hosting providers for ourselves and our clients, including Hetzner. They provide good servers for a great price, have decent support, and we’ve been happy with them for our needs.

Recently I was given the task of building out a new development server for one of our clients, and we wanted it to be set up identically to another one of their servers but with CentOS 7. I placed the order for the hardware with Hetzner and then began the procedure for installing the OS.

Hetzner provides a scripted install process that you can kick off after booting the machine into rescue mode. I followed this process and selected CentOS 7 and proceeded through the whole process without a problem. After rebooting the server and logging in to verify everything, I noticed that the disk space was capped at 2 TB, even though the machine had two 3 TB drives in it (in hardware RAID 1). I looked at the partitions and found the partition table was “msdos”. Ah ha!

At this point painful memories of running into this problem before hit me. I reviewed our notes of what we had done last time, and felt like it was worth a shot even though this time I’m dealing with CentOS 7. I went through the steps up to patching anaconda and then found that anaconda for CentOS 7 is newer and the files are different. I couldn’t find any files that care about the partition table type, so I didn’t patch anything.

I then tried to run the CentOS 7 install as-is. This only got me so far because I then ran into trouble with NetworkManager timing out and not starting.

A screenshot of the CentOS 7 installer failing (anaconda) similar to what I was seeing.

Baffled, I looked into what may have been causing the trouble and discovered that the network was not set up at all and it looked as if no network interfaces existed. WHAT?? At this point I dug through dmesg and found that the network interfaces did indeed exist but udevd had renamed them. Ugh!

Many new Linux distributions are naming network interfaces based on their physical connection to the system: those embedded on the motherboard get named em1, em2, etc. Apparently I missed the memo on this one, as I was still expecting eth0, eth1, etc. And from all indications, so was NetworkManager because it could not find the network interfaces!

Rather than spend more time going down this route, I decided to change gears and look to see if there was any way to patch the Hetzner install scripts to use a GPT partition table with my install instead of msdos. I found and read through the source code for their scripts and soon stumbled on something that just might solve my problem. In the file /root/.oldroot/nfs/install/functions.sh I found mention of a config variable FORCE_GPT. If this is set to “1” then it will try to use a GPT partition table unless it thinks the OS won’t like it, and it thinks that CentOS won’t like it (no matter the version). But if you set FORCE_GPT to “2” it will use a GPT partition table no matter what. This config setting just needs to be added to the file you edit where you list out your partitions and LVM volumes.

FORCE_GPT 2

PART /boot ext3 512M
PART lvm   vg0  all

LV  vg0  swap swap   swap  32G
LV  vg0  root  /     ext4 100G
LV  vg0  home  /home ext4 400G

I then ran the installer script and added the secret config option and… Bingo! It worked perfectly! No need to manually patch anything or install manually. And now we have a CentOS 7 server with full 3 TB of disk space usable.

(parted) print
Model: DELL PERC H710 (scsi)
Disk /dev/sda: 3000GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: pmbr_boot

Number  Start   End     Size    File system  Name  Flags
 3      1049kB  2097kB  1049kB                     bios_grub
 1      2097kB  539MB   537MB   ext3
 2      539MB   3000GB  2999GB                     lvm

Adventures in Downgrading my Linode Plan

2014-09-09T00:00:00+00:00

I recently went through the process of downgrading and downsizing my Linode plan and I wanted to share a few of the [small] hoops that I had to jump through to get there, with the help of the Linode Support team.

Background

I’ve had a small personal WordPress site running for more than a few years now. I also use this server for personal Ruby on Rails development. When I began work on that site, I tried out a few shared hosting providers such as Bluehost and GoDaddy because of the low cost (Bluehost was ~$6/mo) at the time. However, I quickly encountered common limitations of shared server hosting:

Shared hosting providers typically make it very difficult to run Ruby on Rails, especially edge versions of Ruby on Rails. It’s possible this has improved over the last few years, but when you are a developer and want to experiment (not locally), shared hosting providers are not going to give you the freedom to do so.
Shared hosting providers do not give you control of specific performance settings (e.g. use of mod_gzip, expires headers), so I was suffering from lack of control for my little WordPress site as well as my Rails sites. While this is another limitation that may have improved over the last few years, ultimately you are limited by non-root access as a shared server user.

Enter Linode

I looked to virtual server providers such as Linode, Slicehost, and Rackspace after experiencing these common limitations. At the time of my transition, Linode and Slicehost were comparatively priced, but because End Point had successful experiences with Linode for several clients up to that point, I decided to make the jump to Linode. I can’t remember what my initial Linode size was (I think 512MB), but I chose the smallest available option at $20/mo, plus $5/mo for backups. I am not a sysadmin expert like my fellow coworkers (Richard, Jon, the list goes on …), but I managed to get PHP and Ruby on Rails running on Apache with MySQL, and several of the best practices for speeding up your web site in place.

Fast forward about 2 years, and I’ve been very happy with Linode. I can only remember one specific instance where my server has gone down, and the support team has always been very responsive. They also release occasionally free upgrades at the $20/mo price point, and the current offering at that price point is the Linode 2GB (see more here). But lately, I’ve been hearing that Digital Ocean has been gaining momentum with a few cheaper options, and I considered making the jump. But I missed the recent announcement that Linode introduced a new $10/mo. plan back in June (hooray!), so I’m happy to stay with Linode at this lower price point that is suitable for my small, but optimized WordPress site and small Ruby on Rails experiments.

How to Downsize

In a perfect world, it would seem that to quickly downsize your Linode instance, you would first click on the “Resize” tab upon logging in to the Linode dashboard, click on the lower plan that you want, and then click “Resize this Linode now!”, as shown in the screenshot below:

The Linode resize options.

Things went a little differently for me. First, I received this message when I tried to resize:

“Pending free upgrades must be performed before you are able to resize. Please visit the dashboard to upgrade.”

So I headed to my dashboard and clicked on the free upgrade link on the bottom right in the dashboard. I then encountered this message:

“Linodes with configuration profiles referencing a 32 bit kernel are currently not eligible for this upgrade. For more information please see our switching kernels guide, or redeploy this Linode using a 64 bit distro.”

My main Linode Configuration Profile was 64 bit, but my Restore Configuration Profile was running the 32 bit kernel. So, I first had to update that by clicking on the “Edit” link, selecting the right kernel, and saving those changes. That took a few minutes to take effect.

My two configuration profiles needed to be on 64 bit kernel to allow for the Linode upgrade.

Then, I was ready for the free upgrade, which took another few minutes after the server booted down, migrated, and booted back up. Next, I headed back to the “Resize” tab on the dashboard and tried to proceed on the downgrade. I immediately received an error message notifying me that my disk images exceeded the resized option I wanted to switch to (24GB). Upon examining my dashboard, my disk images showed ~28GB allocated to the various disk images:

My disk image space exceeded the 24GB allotted for the Linode 1024 plan.

I was directed by the Linode support team to edit the disk image to get under that 24GB allowed amount. They also explained that I must verify my current app(s) didn’t exceed what I was going to downsize to, using “df -h” while logged into my server. I had already verified previously where disk space was going on my server and cleaned out some cached files and old log files, so I knew the space used was well under 24GB. The only additional step here was that I had to shut down my server first from the dashboard before reducing the disk image space. So I went through all that, and the disk image adjustment took another few minutes. After the disk image size was adjusted, I booted up the server again and verified it was still running.

Editing my Disk Image

Finally, after all that, I went to the “Resize” tab again and selected the Linode 1024 plan and proceeded. The new plan was implemented within a few minutes, automagically booting down my server and restarting it after completion. My billing information was also updated almost immediately, showing that I will now pay $12.50/mo for the Linode 1024 plan with backups.

Conclusion

In list form, here are the steps I went through to reach my final destination:

Updated kernels to 64 bit for all configuration profiles.
Applied pending, free upgrades.
Manually shut down server.
Applied change to reduce disk image space.
Rebooted server (not necessary, but I verified at this point things were still running).
Resized to Linode 1024 plan.

While this process wasn’t as trivial as I had hoped, the support folks were super responsive, often responding within a minute or two when I had questions. I’m happy to stay with Linode at this offering and it allows them to remain competitive with both virtual private hosting providers and as an appealing alternative to shared hosting providers. The Linode 1024 plan is also a great starting point for a proof-of-concept or staging server that may be scaled up later as applications move to production and increase in traffic. Linode has plans ranging from the $10/mo plan I have (1GB of RAM, 24GB SSD storage, etc.) all the way up to a 96GB RAM, 1920 GB SSD storage plan at $960/mo.

The Beauty of IPMI

2014-08-01T00:00:00+00:00

For our Liquid Galaxy installations, we use a master computer known as a “head node” and a set of slave computers known as “display nodes.” The slave computers all PXE-boot from the head node, which directs them to boot from a specific ISO disk image.

In general, this system works great. We connect to the head node and from there can communicate with the display nodes. We can boot them, change their ISO, and do all sorts of other maintenance tasks.

There are two main settings that we change in the BIOS to make things run smoothly. First is that we set the machine to power on when AC power is restored. Second, we set the machine’s boot priority to use the network.

Occasionally, though, the CMOS battery has an issue, and the BIOS settings get lost. How do we get in and boot the machine up? This is where ipmitool has really become quite handy.

Today we had a problem with one display node at one of our sites. It seems that all of the machines in the Liquid Galaxy were rebooted, or otherwise powered off and then back on. One of them just didn’t come up, and it was causing me much grief. We have used ipmitool in the past to be able to help us administer the machines.

IPMI stands for Intelligent Platform Media Interface, and it gives the administrator some non-operating system level access to the machine. Most vendors have some sort of management interface (HP’s iLO, Dell’s DRAC), including our Asus motherboards. The open source ipmitool is the tool we use on our Linux systems to be able to interface with the IPMI module on the motherboard.

I connected to the head node and ran the following command and got the following output:

admin@headnode:~ ipmitool -H 10.42.41.33 -I lanplus -P 'xxxxxx' chassis status
System Power         : off
Power Overload       : false
Power Interlock      : inactive
Main Power Fault     : false
Power Control Fault  : false
Power Restore Policy : always-off
Last Power Event     : ac-failed
Chassis Intrusion    : inactive
Front-Panel Lockout  : inactive
Drive Fault          : false
Cooling/Fan Fault    : true

While Asus’s Linux support is pretty lacking, and most of the options we find here don’t work with with the open source ipmitool, we did find “System Power : off” in the output, which is a pretty good indicator of our problem. This tells me that the BIOS settings have been lost for some reason, as we had previously set the system to power on when AC power was restored. I ran the following to tell it to boot into the BIOS, then powered on the machine:

admin@headnode:~ ipmitool -H 10.42.41.33 -I lanplus -P 'xxxxxx' chassis bootdev bios
admin@headnode:~ ipmitool -H 10.42.41.33 -I lanplus -P 'xxxxxx' chassis power on

At this point, the machine is ready for me to be able to access the BIOS through a terminal window. I opened a new terminal window and typed the following:

admin@headnode:~ ipmitool -H ipmi-lg2-3 -U admin -I lanplus sol activate
Password:

After typing in the password, I get the ever-helpful dialog below:

[SOL Session operational.  Use ~? for help]

I didn’t bother with the ~? because I knew that the BIOS would eventually just show up in my terminal. There are, however, other commands that pressing ~? would show.

See, look at this terminal version of the BIOS that we all know and love!

Now that the BIOS was up, it’s as if I was really right in front of the computer typing on a keyboard attached to it. I was able to get in and change the settings for the APM, so that the system will power on upon restoration of AC power. I also verified that the machine is set to boot from the network port before saving changes and exiting. The next thing I knew, the system was booting up PXE, which then pointed it to the proper ISO, and then it was all the way up and running.

And this, my friends, is why systems should have IPMI. I state the obvious here when I say that life as a system administrator is so much easier when one can get into the BIOS on a remote system.

Managing Multiple Hosts and SSH Identities with OpenSSH

2013-12-12T00:00:00+00:00

When I started working at End Point I was faced with the prospect of having multiple SSH identities for the first time. I had historically used an RSA SSH key with the default length of 2048 bits, but for my work at End Point I needed to generate a new key that was 4096 bits long.

Although I could have used ssh-copy-id to copy my new SSH public key to all of my old servers, I liked the idea of maintaining separate “personal” and “work” identities and decided to look for a way to automatically use the right key based on the server I was trying to connect to.

For the first few days I was specifying my new identity on the command line using:

ssh -i .ssh/endpoint_rsa patrick@server.example.com

That worked, but I often forgot to specify my new SSH identity when connecting to a server, only realizing my mistake when I was prompted for a password instead of being authenticated automatically.

Host Definitions

I had previously learned the value of creating an ssh_config file when I replaced a series of command-line aliases with equivalent entries in the SSH config file.

Instead of creating aliases in my shell:

alias server1='ssh -p 2222 -L 3389:192.168.1.99:3389 patrick@server1.example.com'

I learned that I could add an equivalent entry to my ~/.ssh/config file:

Host server1
  HostName server1.example.com
  Port 2222
  User patrick
  LocalForward 3389 192.168.1.99:3389

Then, to connect to that server, all I needed to do was run ssh server1 and all of the configuration details would be pulled in from the SSH config file. Replacing my series of shell aliases with Host definitions had the added benefit of automatically carrying over to other tools like git and mosh which read the same configuration.

Switching Identities Automatically

There’s an easy solution to managing multiple SSH identities if you only use one identity per server; use ssh-add to store all of your keys in the SSH authentication agent. For example, I used ssh-add ~/.ssh/endpoint_rsa to add my new key, and ssh-add -l to verify that it was showing up in the list of known keys. After adding all of your keys to the agent, it will automatically try them in order for SSH connections until it finds one that authenticates successfully.

Manually Defining Identities

If you need more control over which identity an SSH session is using, the IdentityFile option in ssh_config lets you specify which key will be used to authenticate. Here’s an example:

Host server2
  HostName server2.example.com
  User patrick
  IdentityFile ~/.ssh/endpoint_rsa

This usage is particularly helpful when you have a server that accepts more than one of your identities and you need to control which one should be used.

Comparing installed RPMs on two servers

2013-09-27T00:00:00+00:00

Sometimes I’m called on to deal with a problem that shows up only on one of two or more servers that are supposed to be configured identically, or nearly identically. One of the first things I do is run rpm -qa | sort on each machine and diff the output to see which RPM packages may be missing on one or the other server. I’ve never bothered to package this functionality up into a script because it’s so simple.

To exclude minor version differences, you need to specify a custom rpm --queryformat that leaves the version number off.

To understand what you’re seeing when it appears that some package is different but seems the same, you’re often looking at multiple architectures of packages (e.g. i386 and x86_64) which RPM doesn’t show in its default query format.

Finally, to turn the diff output into a list of RPMs to install via yum, I usually do some combination of grep and sed to pick out the RPMs I need.

After all that the process isn’t entirely simple anymore, and I recently decided it was easier to script it than explain it all to someone else. I first looked around to see what scripts others have come up with, since this is certainly not a new need. I found the blog post “Compare the RPM Packages Installed on Two Different Servers” which gives a very simple example of the manual labor version I’ve long done.

In that blog post’s comments, people link to various scripts others have done. I checked them out and found that they are all way overcomplicated for my needs, and the simple approach I want just needed to be scripted after all. So here is my script:

#!/bin/sh

test -n "$TMPDIR" || TMPDIR=/tmp
tmpfilebase="$TMPDIR/rpmdb-compare.$$"

if test $# -ne 2
then
    echo "Usage: $0 [user@]server1 [user@]server2" >&2
    echo "Special name localhost means this server, without ssh" >&2
    exit 1
fi

for i in "$@"
do
    tmpfile="$tmpfilebase.$i"
    if test $i = localhost
    then
        rpm -qa --qf '%{NAME}.%{ARCH}\n' | sort > $tmpfile
    else
        ssh $i "rpm -qa --qf '%{NAME}.%{ARCH}\n'" | sort > $tmpfile
    fi
done

echo "RPMs in $1 only:"
comm -23 "$tmpfilebase.$1" "$tmpfilebase.$2"

echo
echo "RPMs in $2 only:"
comm -13 "$tmpfilebase.$1" "$tmpfilebase.$2"

rm -f "$tmpfilebase.$1" "$tmpfilebase.$2"

I noticed one of those commenters mentioned using comm instead of diff/grep/sed, and so I used that too. Now the script is easier for me too, and helps avoid copying and leaving temporary files sitting around.

To run it, just do:

./rpmdb-compare host1 host2 > mylist

With the output redirected to file mylist, you can edit it to result in a list of RPMs that need to be installed on one server, then do this on that server:

< mylist yum -y install

It’s a good idea to test it first without the -y option, which will cause yum to abort the installation and gives you a chance to see if any unexpected dependencies will be dragged in.

Also, don’t blindly install every package you don’t know the purpose of. Watch out for RPMs that may not belong everywhere due to hardware differences such as Ethernet firmware, RAID controller, IPMI, etc.

Apache accidental DNS hostname lookups

2013-09-18T00:00:00+00:00

Logging website visitor traffic is an interesting thing: Which details should be logged? How long and in what form should you keep log data afterward? That includes questions of log rotation frequency, file naming, and compression. And how do you analyze the data later, if at all?

Allow me to tell a little story that illustrates a few limited areas around these questions.

Reverse DNS PTR records

System administrators may want to make more sense of visitor IP addresses they see in the logs, and one way to do that is with a reverse DNS lookup on the IP address. The network administrators for the netblock that the IP address is part of have the ability to set up a PTR (pointer) record, or not. You can find out what it is, if anything.

For example, let’s look at DNS for End Point’s main website at www.endpoint.com using the standard Unix tool “host”:

% host www.endpoint.com
www.endpoint.com has address 208.43.132.31
www.endpoint.com has IPv6 address 2607:f0d0:2001:103::31
% host 208.43.132.31
31.132.43.208.in-addr.arpa domain name pointer 208.43.132.31-static.reverse.softlayer.com.
% host 2607:f0d0:2001:103::31
1.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.3.0.1.0.1.0.0.2.0.d.0.f.7.0.6.2.ip6.arpa domain name pointer 2607.f0d0.2001.0103.0000.0000.0000.0031-static.v6reverse.softlayer.com.

The www.endpoint.com name points to both an IPv4 and an IPv6 address, so there are two answers. When each of those IP addresses is looked up, each shows a PTR record pointing to a subdomain of softlayer.com, which gives a clue about where our site is hosted.

(As an aside: Why don’t we use a prettier or more specific PTR record? We could set it to almost whatever we want. Well, there are dozens of websites hosted on those IP addresses, so which one should be in the PTR record? There’s no obvious choice, and it doesn’t matter for normal network functioning, so we just left it the way it was.)

So, is a PTR record like these useful to know about visitors to your website? Sometimes. Let’s take a look at a random sample of visitors to a different website we manage. How much can you tell about each of the visitors based on their reverse DNS PTR records? Is it a bot, someone at home or the office, in which country, and who is their Internet provider? How common is it for a visitor’s IP address to have no PTR record? And keep in mind that most of the visitors have no idea what their IP address or its PTR record is.

% host 93.137.189.55
55.189.137.93.in-addr.arpa domain name pointer 93-137-189-55.adsl.net.t-com.hr.
% host 66.249.73.121
121.73.249.66.in-addr.arpa domain name pointer crawl-66-249-73-121.googlebot.com.
% host 88.134.68.31
31.68.134.88.in-addr.arpa domain name pointer 88-134-68-31-dynip.superkabel.de.
% host 67.49.156.20
20.156.49.67.in-addr.arpa domain name pointer cpe-67-49-156-20.hawaii.res.rr.com.
% host 123.211.36.234
234.36.211.123.in-addr.arpa domain name pointer CPE-123-211-36-234.lnse3.cha.bigpond.net.au.
% host 91.75.70.162 
Host 162.70.75.91.in-addr.arpa. not found: 3(NXDOMAIN)
% host 209.82.97.10
10.97.82.209.in-addr.arpa domain name pointer mail02.westjet.com.
% host 76.70.117.223
223.117.70.76.in-addr.arpa domain name pointer bas1-toronto26-1279686111.dsl.bell.ca.
% host 101.160.207.115 
Host 115.207.160.101.in-addr.arpa. not found: 3(NXDOMAIN)
% host 184.198.177.214
214.177.198.184.in-addr.arpa domain name pointer 184-198-177-214.pools.spcsdns.net.
% host 84.199.97.130
130.97.199.84.in-addr.arpa domain name pointer 84-199-97-130.iFiber.telenet-ops.be.
% host 182.19.87.24 
Host 24.87.19.182.in-addr.arpa. not found: 3(NXDOMAIN)
% host 62.34.219.216
216.219.34.62.in-addr.arpa domain name pointer i01v-62-34-219-216.d4.club-internet.fr.
216.219.34.62.in-addr.arpa domain name pointer lns-c10k01-v-62-34-219-216.dsl.sta.abo.bbox.fr.
% host 187.86.213.190
Host 190.213.86.187.in-addr.arpa. not found: 3(NXDOMAIN)
% host 15.211.201.84
84.201.211.15.in-addr.arpa domain name pointer zccy01cs104.houston.hp.com.
% host 161.69.46.150
150.46.69.161.in-addr.arpa domain name pointer miv-scan015.scanalert.com.
% host 77.182.146.99
99.146.182.77.in-addr.arpa domain name pointer essn-4db69263.pool.mediaWays.net.
% host 107.0.160.152 
152.160.0.107.in-addr.arpa domain name pointer 107-0-160-152-ip-static.hfc.comcastbusiness.net.

Did you notice that one IP address returned two different PTR records? That is allowed, though uncommon, as I mentioned in my blog post Multiple reverse DNS pointers per IP address a few years back. Many reverse DNS control panels provided by commodity hosting providers won’t allow you to assign multiple PTR records, but if you get your reverse DNS delegated to a real nameserver you control, you can do it.

Finding the IP address owner: whois

The reverse DNS PTR can be set misleadingly, such that a forward lookup on the name does not point back to the same IP address. In the end the way to really know who controls that IP address (or at least a network provider who supplies the ultimately responsible person) is with a “whois” lookup. We can check that the 208.43.132.31 IP address really is hosted at SoftLayer, and for which customer, like this:

% whois 208.43.132.31
[Querying whois.arin.net]
[Redirected to rwhois.softlayer.com:4321]
[Querying rwhois.softlayer.com]
[rwhois.softlayer.com]
%rwhois V-1.5:003fff:00 rwhois.softlayer.com (by Network Solutions, Inc. V-1.5.9.5)
network:Class-Name:network
network:ID:NETBLK-SOFTLAYER.208.43.128.0/19
network:Auth-Area:208.43.128.0/19
network:Network-Name:SOFTLAYER-208.43.128.0
network:IP-Network:208.43.132.0/27
network:IP-Network-Block:208.43.132.0-208.43.132.31
network:Organization;I:End Point Corporation
network:Street-Address:920 Broadway, Suite 701
network:City:New York
network:State:NY
network:Postal-Code:10010
network:Country-Code:US
network:Tech-Contact;I:sysadmins@softlayer.com
network:Abuse-Contact;I:abuse@endpoint.com
network:Admin-Contact;I:IPADM258-ARIN
network:Created:2007-06-18 12:15:54
network:Updated:2010-11-21 18:59:43
network:Updated-By:ipadmin@softlayer.com

%referral rwhois://root.rwhois.net:4321/auth-area=.
%ok

So you can see that’s really End Point’s IP address, at SoftLayer.

Use your local whois tool or search for a web-based one and look up a few of the IP addresses that didn’t have reverse DNS PTR records in our log cross-section above. The results are interesting.

Reverse lookups in Apache httpd

Now let’s say that as a system administrator you would like to see the PTR records for visitor IP addresses on your Apache httpd website. It may be tempting to use the HostnameLookups configuration directive to do real-time lookups and put them in the log alongside the IP address. It’s easy but not wise to put the PTR record instead of the IP address, because it may not point back to the IP address, and even if it does, it can change over time, and will not provide a complete picture of the connection later on.

However, if you read the HostnameLookups documentation, you’ll see the authors recommend it not be enabled on busy production servers because of the extra network traffic and delay for visitors, especially for any netblocks with slow DNS servers (and there are many out there). This is important! It really should almost never be enabled for any public site.

Most web server administrators learn this early on and wouldn’t dream of enabling HostnameLookups.

However, I recently came across a situation where we inadvertently were doing the equivalent without explicitly enabling HostnameLookups. How? By limiting access based on the remote hostname! Read the documentation on the Allow directive, under the section “A (partial) domain-name”:

This configuration will cause Apache to perform a double reverse DNS lookup on the client IP address, regardless of the setting of the HostnameLookups directive. It will do a reverse DNS lookup on the IP address to find the associated hostname, and then do a forward lookup on the hostname to assure that it matches the original IP address. Only if the forward and reverse DNS are consistent and the hostname matches will access be allowed.

This makes perfect sense, but it is a pretty big likely unexpected side effect to using something like:

Allow from .example.com

In our case it was an even less obvious case that didn’t make us think of hostnames at all:

Allow from localhost

Here localhost was written, perhaps to save some effort or maybe increase clarity vs. writing out 127.0.0.1 (IPv4) and ::1 (IPv6). Mentally it’s so easy to view “localhost” is a direct alias for 127.0.0.1 and ::1 that we can forget that the name “localhost” is just a convention, and requires a lookup like any other name. Those familiar with the MySQL database may know that it actually assigns special confusing meaning to the word “localhost” to make a UNIX socket connection instead of a TCP connection to 127.0.0.1 or whatever “localhost” is defined as on the system!

You may also be thinking that looking up 127.0.0.1 is fast because that is usually mapped to “localhost” in /etc/hosts. True, but every other visitor who is not in /etc/hosts gets the slow DNS PTR lookup instead! And depending on the operating system, you may see “ip6-localhost” or “ip6-loopback” (Debian 7, Ubuntu 12.04), “localhost6” (RHEL 5/6, Fedora 19) in /etc/hosts, or something else. So it’s important to spell out the addresses:

Allow from 127.0.0.1
Allow from ::1

Doing so immediately stops the implicit HostnameLookups behavior and speeds up the website. In this case it wasn’t a problem, since it was for a private, internal website that couldn’t be visited at all by anyone not first allowed through a firewall, so traffic levels were relatively low. That access control is part of why localhost needed to be allowed in the first place. But it would have been very bad on a public production system due to the slowdown in serving traffic.

The right way

If you really want to see PTR records for every visitor IP address, you can use Apache’s logresolve log post-processing program. Or you can let an analytics package do it for you.

So, lesson learned: It’s not just HostnameLookups you need to keep turned off. Also watch out for the Apache Allow directive and don’t use it with anything other than numeric IP addresses!

GNU Screen logtstamp string

2013-07-24T00:00:00+00:00

A short note on GNU Screen configuration:

You can add configuration to ~/.screenrc or another configuration file named by -c $filename upon invocation, and among the many options are some to enable logging what happens in the screen windows. This is useful when using screen as a reattachable daemonizer.

Consider this configuration:

logfile path/to/screen-output.%Y%m%d.log
logfile flush 1
logtstamp on
logtstamp after 5
log on

That works nicely. With logfile we specify the name of the log file, using some % escapes as per “STRING ESCAPES” in the manpage to put the date in the logfile name.

With logfile flush 1 we request that every 1 second the output be flushed to the log, making it easier to follow with tail -f.

The logtstamp on option writes a timestamp to the log after a default 2 minutes of inactivity. We shorten that to 5 seconds with logtstamp after 5.

Finally, log on turns on the logging.

Now, what if we want to customize the timestamp? The default looks like this:

-- 0:process-name -- time-stamp -- Jul/24/13  9:09:56 --

The manpage says that can be customized with logtstampt string ..., where the default is:

-- %n:%t -- time-stamp -- %M/%d/%y %c:%s --\n

The manpage earlier says that arguments may be separated by single or double quotes. Doing so with the default shown doesn’t work, because a literal \n shows up in the logfile.

The solution I worked out by trial and error is that you must double-quote the string and use octal escape value \012. Single-quoting that will output a literal backslash 0 1 2, and \n simply is not a recognized escape. Thus our final configuration directive is:

logtstamp string "-- time-stamp -- %Y-%m-%d %0c:%s --\012"

which results in output like:

-- time-stamp -- 2013-07-24 09:59:35 --

If you use more than one screen window with this configuration, all windows’ output will go into the same logfile. Use the %n escape string to include a window number in the logfile name if you’d like them kept separate.

Converting root filesystem from ext3 to ext4 on CentOS and RHEL 5.9

2013-06-12T00:00:00+00:00

Here’s a quick explanation of the procedure to convert the root / filesystem on RHEL and CentOS 5.9 from ext3 to ext4, because ext3 wasn’t available during install time.

Note that this is not a configuration Red Hat supports, but it works fine. (I believe you cannot convert the /boot filesystem to ext4 on standard RHEL/CentOS 5 because its GRUB can’t handle it, but you can convert all the other filesystems.)

Ideally do this only on a fairly freshly-installed system you don’t mind losing. Back everything up first unless this is a system you don’t mind destroying! This is a risky operation and (ahem) things can go wrong.

You’ll need direct console or KVM access to the server. You can do without that if you can remount -o ro / but that usually won’t work with sshd or other daemons that keep files open on the / filesystem.

You will of course need to adapt the current kernel version and root filesystem block device path in the examples below.

Now, to live dangerously:

yum -y install e4fsprogs
Edit /etc/fstab so that the / filesystem is mounted as ext4 (which works with the existing ext3 filesystem as well). If you’re using a battery-backed RAID controller you may want to add the nobarrier mount option. See man mount to read about what that choice entails.
mkinitrd –with=ext4 –with=ext3 -f /boot/initrd-2.6.18-348.el5.img 2.6.18-348.el5
shutdown -r now
Boot into single-user mode: use GRUB to edit the linux arguments adding “single” to the end
fsck.ext3 -pf /dev/vg0/lv_root
tune4fs -O extents,uninit_bg,dir_index /dev/vg0/lv_root
fsck.ext4 -yfD /dev/vg0/lv_root
shutdown -r now
Allow it to boot normally into multi-user mode.

There are lots of articles out there about converting from ext3 to ext4, but none I found that covered RHEL/CentOS 5 specifically and contained all the needed steps and actually worked. This is based on info from here for Debian and here for CentOS.

Converting RHEL 5.9 and 6.4 to CentOS

2013-04-04T00:00:00+00:00

CentOS is, by design, an almost identical rebuild of Red Hat Enterprise Linux (RHEL). Any given version of each OS should behave the same as the other and packages and yum repositories built for one should work for the other unchanged. Any exception I would call a bug.

Because Red Hat is the source or origin of packages that ultimately end up in CentOS, there is an inherent delay between when Red Hat releases new packages and when they appear in CentOS. CentOS is financed by optional donations of work, hosting, and money, while Red Hat Enterprise Linux is financed by requiring customers to purchase entitlements to use the software and get various levels of support from Red Hat.

Thanks to this close similarity and the tradeoff between rapidity of updates vs. cost and entitlement tracking, we find reasons to use both RHEL and CentOS, depending on the situation.

Sometimes we want to convert RHEL to CentOS or vice versa, on a running machine, without the expense and destabilizing effect of having to reinstall the operating system. In the past I’ve written on this blog about converting from CentOS 6 to RHEL 6, and earlier about converting from RHEL 5 to CentOS 5.

I recently needed to migrate several servers from RHEL to CentOS, and found an update of the procedure was in order because some URLs and package versions had changed. Here are current instructions on how to migrate from RHEL 5.9 to CentOS 5.9, and RHEL 6.4 to CentOS 6.4.

These commands should of course be run as root, and observed carefully by a human eye to look for any errors or warnings and adapt accordingly.

RHEL 5.9 to CentOS 5.9 conversion, 64-bit (x86_64)

cd
mkdir centos
cd centos
wget http://mirror.centos.org/centos/5.9/os/x86_64/RPM-GPG-KEY-CentOS-5
wget http://mirror.centos.org/centos/5.9/os/x86_64/CentOS/centos-release-5-9.el5.centos.1.x86_64.rpm
wget http://mirror.centos.org/centos/5.9/os/x86_64/CentOS/centos-release-notes-5.9-0.x86_64.rpm
wget http://mirror.centos.org/centos/5.9/os/x86_64/CentOS/yum-3.2.22-40.el5.centos.noarch.rpm
wget http://mirror.centos.org/centos/5.9/os/x86_64/CentOS/yum-updatesd-0.9-5.el5.noarch.rpm
wget http://mirror.centos.org/centos/5.9/os/x86_64/CentOS/yum-fastestmirror-1.1.16-21.el5.centos.noarch.rpm
wget http://mirror.centos.org/centos/5.9/os/x86_64/CentOS/gamin-python-0.1.7-10.el5.x86_64.rpm
yum erase yum-rhn-plugin rhn-client-tools rhn-virtualization-common rhn-setup rhn-check rhnsd yum-updatesd
yum clean all
rpm --import RPM-GPG-KEY-CentOS-5
rpm -e --nodeps redhat-release
yum localinstall *.rpm
yum upgrade
shutdown -r now

RHEL 5.9 to CentOS 5.9 conversion, 32-bit (i386)

cd
mkdir centos
cd centos
wget http://mirror.centos.org/centos/5.9/os/i386/RPM-GPG-KEY-CentOS-5
wget http://mirror.centos.org/centos/5.9/os/i386/CentOS/centos-release-5-9.el5.centos.1.i386.rpm
wget http://mirror.centos.org/centos/5.9/os/i386/CentOS/centos-release-notes-5.9-0.i386.rpm
wget http://mirror.centos.org/centos/5.9/os/i386/CentOS/yum-3.2.22-40.el5.centos.noarch.rpm
wget http://mirror.centos.org/centos/5.9/os/i386/CentOS/yum-updatesd-0.9-5.el5.noarch.rpm
wget http://mirror.centos.org/centos/5.9/os/i386/CentOS/yum-fastestmirror-1.1.16-21.el5.centos.noarch.rpm
wget http://mirror.centos.org/centos/5.9/os/i386/CentOS/gamin-python-0.1.7-10.el5.i386.rpm
yum erase yum-rhn-plugin rhn-client-tools rhn-virtualization-common rhn-setup rhn-check rhnsd yum-updatesd
yum clean all
rpm --import RPM-GPG-KEY-CentOS-5
rpm -e --nodeps redhat-release
yum localinstall *.rpm
yum upgrade
shutdown -r now

RHEL 6.4 to CentOS 6.4 conversion, 64-bit (x86_64)

cd
mkdir centos
cd centos
wget http://mirror.centos.org/centos/6.4/os/x86_64/RPM-GPG-KEY-CentOS-6
wget http://mirror.centos.org/centos/6.4/os/x86_64/Packages/centos-release-6-4.el6.centos.10.x86_64.rpm
wget http://mirror.centos.org/centos/6.4/os/x86_64/Packages/yum-3.2.29-40.el6.centos.noarch.rpm
wget http://mirror.centos.org/centos/6.4/os/x86_64/Packages/yum-utils-1.1.30-14.el6.noarch.rpm
wget http://mirror.centos.org/centos/6.4/os/x86_64/Packages/yum-plugin-fastestmirror-1.1.30-14.el6.noarch.rpm
yum erase yum-rhn-plugin rhn-client-tools rhn-virtualization-common rhn-setup rhn-check rhnsd yum-updatesd subscription-manager
yum clean all
rpm --import RPM-GPG-KEY-CentOS-6
rpm -e --nodeps redhat-release-server
yum localinstall *.rpm
yum upgrade
shutdown -r now

We don’t use 32-bit (i386) RHEL or CentOS 6, so you’re on your own with that, but it should be very straightforward to adapt the x86_64 instructions.

If during the yum localinstall you get an error like this that references a URL containing %24releasever:

[Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404 Not Found"
Error: Cannot retrieve repository metadata (repomd.xml) for repository

Then you need to temporarily disable that add-on yum repository until after the conversion is complete by editing /etc/yum.repos.d/name.repo to change enabled=1 to enabled=0. The problem here is caused by the repo configuration using the releasever yum variable which is undefined mid-conversion because we forcibly removed the redhat-release* package that defines it. We can’t expect the OS to know what kind it is in the middle of its identity crisis and change!

If all goes well, nothing will look any different at all, except you’ll now see:

# cat /etc/redhat-release
CentOS release 5.9 (Final)

or:

# cat /etc/redhat-release
CentOS release 6.4 (Final)

Introducing EP Audit

The Problem

What EP Audit Looks Like

A Service, Not Just a Tool

Phased Remediation

Working with End Point

Rocky Linux 9 at Hetzner Robot Made Quick and Easy

Introduction

The Steps

Conclusion

Rocky Linux 9 via Hetzner Robot for the impatient

Update: A Better Way to Install Rocky Linux 9 at Hetzner Cloud

About Rocky Linux

Rocky 9 on Hetzner

Provisioning the Server

Enabling Rescue Mode

Rebooting the Server

Accessing the Rescue System

Installing Rocky Linux 8:

Completing the Installation

Upgrading to Rocky Linux 9:

Accessing the Rocky Linux 9 System:

Conclusion

Bypassing a CDN to browse a website directly on your origin host

Straight to the source

Make the hosts file point directly to our server

Disable DNS-over-TLS & DNS-over-HTTPS

Chrome

Firefox

Browse the website directly

Check certificate issuer

Clean up

Mobile

Working around SPF problems delivering to Gmail

Email deliverability

Sender Policy Framework

Email server trails

Gmail misuses header information in SPF checks

Don’t share TMI

Reference

SRV DNS records in Terraform and Cloudflare

systemd: a primer from the trenches

systemctl: Let’s get back to basics

Pro Tip 1: Tab completion

systemctl vs. old school commands

systemctl

systemctl status

systemctl status serviceName -l

systemctl enable|disable NameofService

systemctl start|stop|restart httpd

systemctl reload httpd

systemctl daemon reload

systemctl list-unit-files

systemctl list-units

systemctl list-sockets

systemctl list-timers

systemctl list-jobs

systemctl –failed

systemctl get-default

systemctl set-default multi-user.target

systemctl shutdown or reboot

systemctl cat serviceName

systemctl list-dependencies serviceName

systemctl show serviceName

systemctl mask serviceName

systemctl edit serviceName

systemctl -o

systemctl isolate

systemd-delta

Gotchas

Who’s got the goods on speed?

Pro Tip 2: Remote commands

Pro Tip 3: What relies on my service

Pro Tip 4: Monitoring

Summing up

Instant TLS Upgrades Through Proxy Magic!

TLS shutdowns are real

What is a proxy?

Why use a proxy?

Steps