SSH and Old Servers

2023-07-21T00:00:00+00:00

We recently updated one of our backup servers from Oracle Linux 8 to Oracle Linux 9, a free rebuild of Red Hat Enterprise Linux 9, also known as RHEL 9. As with any newer OS, there are bound to be unexpected little differences which crop up and which need to be handled.

In the case of our backup server, we found that it could no longer SSH to older servers, which includes a couple running CentOS 5 (yes, there really are a few of those out in the wild), some CentOS 6, and one Debian 8. We saw similar connection problems when we first went from CentOS 7 to Oracle Linux 8, and we addressed the issues by creating a Host entry for the servers in our ~/.ssh/config file.

Here is an example of of the entry that we used previously:

Host old-server-1 old-server-2 old-server-3
  User root
  Hostname %h.example.com
  KexAlgorithms +diffie-hellman-group-exchange-sha1

This worked fine for a couple of years until we upgraded from Oracle Linux 8 to 9. At this point we started to see a different problem when trying to connect.

Unable to negotiate with <nnn.nnn.nnn.nnn> port 22: no matching host key type found. Their
offer: ssh-rsa,ssh-dss

Troubleshooting

One of the good troubleshooting methods for diagnosing SSH connectivity problems is to enable some verbosity when connecting. This can be done via the use of the -v option, which can be stacked up to three times for increasing levels of detail. In this case, I used ssh -vvv $hostname, which is the highest level, and I was able to see another error message in the copious output:

ssh_dispatch_run_fatal: Connection to <nnn.nnn.nnn.nnn> port 22: error in libcrypto

It was at this time that I had to consult with The Fountain of Knowledge (the Internet) to come up with a solution. There were some suggestions to add HostKeyAlgorithms or PubkeyAcceptedKeyTypes entries for those hosts in the local SSH config, but that didn’t work in this case.

What I did find, though, is that RHEL 9-based OSes now forbid via crypto policy the ability to connect using SHA-1, but SHA-1 is the default crypto policy in use by CentOS 5 and CentOS 6! To get around this, I needed to add SHA-1 support to our existing policy. You can compare the crypto policies for RHEL 8 systems and RHEL 9 systems to see which is right for your use case.

update-crypto-policies --set DEFAULT:SHA1

Here I also found a bit of info on re-enabling SHA-1.

A problem for every solution

One more problem that we ran into with one of the CentOS 5 servers is that we were getting a different SSH failure even after having updated the crypto policy. This new problem was about the length of the host key on the old remote server.

Bad server host key: Invalid key length

That is because RHEL 9 by default doesn’t like short RSA host keys. Thankfully, there is an option that can be enabled at the SSH client level to help us overcome the limitation. It is solved in the usual manner of adding an option to the ~/.ssh/config file. The option is known as RequiredRSASize, and it specifies the minimum length of RSA key that is required to be considered valid.

Host old-server-1 old-server-2 old-server-3
  User root
  Hostname %h.example.com
  KexAlgorithms +diffie-hellman-group-exchange-sha1
  RequiredRSASize 1024

Perfect … enough

Obviously we don’t want to let old systems just sit out there and keep doing their thing. They increase the attack surface that hackers can target, and it’s important to stay on top of updating and upgrading. In an ideal world, we’d prefer that we simply replace and modernize the old, thus securing everything, and deal with the pain of upgrading all at one time. We like to just get the headache over with.

In the real world, though, some old systems just have to stay in place for a little longer (or “much longer” in this case) than we’d prefer. It’s helpful to figure out acceptable ways to work around the limitations when these issues arise, while wherever possible making the exceptions only for the specific old hosts we want to tolerate.

Mount a remote filesystem over SSH using SSHFS

2023-05-25T00:00:00+00:00

While creating and debugging software, it is important to reduce the amount of friction between each iteration of making a change to the software and then testing that change. Over time, even small amounts of friction can lead to fatigue and decreased performance of a developer. Because of this, we should take every opportunity to make our workflow as smooth and comfortable as possible.

A common source of friction when developing software running on remote systems is the separation between your personal computer and the server. Your personal computer likely has an IDE configured just the way you like. The server, on the other hand, is likely configured to be easily available to everyone on your team.

You could copy files back and forth between systems using SFTP or some other file transfer protocol. This works for quick one-off changes, but for development requiring multiple iterations you likely want a more streamlined workflow.

If only there was a way to use the software installed on your personal system to edit files on a remote system, without copying the files back and forth…

There is! SSHFS is a tool for mounting and interacting with remote directories over SSH.

Notice

The SSHFS repository has been archived for around a year at the time of writing; the latest release was on May 26, 2022. The repository now has a note about being orphaned and inviting other developers to take over the project. While there are a few forks (most notably deadbeefsociety’s and stephenxxiu’s) with some traction, none is a definitive replacement yet.

One alternative suggested by the ArchWiki is using rclone’s mount feature.

For macOS users, it’s important to note that macFUSE is no longer open-source as of 2017.

Installation

This guide contains installation instructions for macOS and Linux systems only.

If you are using Windows, check out sshfs-win.

macOS instructions

Download the macFUSE and SSHFS packages from the osxfuse website and install them as an administrator.

Linux instructions

SSHFS is available from the package repositories of most distributions. This is usually the preferred way to install software on Linux.

On Debian and Ubuntu systems, the following command will install the latest version of SSHFS available:

sudo apt update && sudo apt install sshfs

Usage

After installation you may interact with SSHFS via the sshfs command.

Before we mount a remote directory, let’s create a local directory that will serve as the local mount point:

mkdir /tmp/remote

Let’s say we want to mount the remote directory /var/www/html of the remote system at example.com while connected as the remote user wildfly. We want to mount this remote directory at the local directory /tmp/remote. To accomplish this, we would execute the following command:

sshfs -o compression=no,cache=yes,cache_timeout=20,auto_cache,idmap=user wildfly@example.com:/var/www/html /tmp/remote

The following mount options are given using the -o argument:

compression=no - Disable compression. The performance impact of enabling compression is not worth the minimal bandwidth savings.
cache=yes - Enable caching. Files that are repeatedly accessed within a certain period of time will only be retrieved once.
cache_timeout=20 - Set cache timeout, in seconds.
auto_cache - Enable caching based on file modification times.
idmap=user - Translate between UID of connecting user and remote user. This causes remote files to appear to be owned by you, even though they are actually owned by the remote user on the remote system.

To unmount on linux, run fusermount -u /tmp/remote. On macOS, run umount /tmp/remote.

SSH host key verification: a few useful tips

2023-04-13T00:00:00+00:00

The SSH connections between a client and a remote server begin with a host key verification as an initial handshake. If the default key algorithm is not supported between the client and server, the SSH connection attempt is closed with no matching host key type response.

$ sftp username@domain
Unable to negotiate with xx.xx.xx.xx port 22: no matching host key type found. Their offer: ssh-rsa
Connection closed.
Connection closed

In this article, we will explore the host key verification process and discuss ways to handle the non-matching host key issue to establish the connection.

Host keys

By default, OpenSSH automatically generates a public-private key pair on the server and stores it in /etc/ssh. These keys, known as host keys, are created using several encryption algorithms including RSA, DSA, ECDSA, and ed25519.

The pair of private and public keys is available on the host server at the path /etc/ssh.

$ ls /etc/ssh | grep key
ssh_host_dsa_key
ssh_host_dsa_key.pub
ssh_host_ecdsa_key
ssh_host_ecdsa_key.pub
ssh_host_ed25519_key
ssh_host_ed25519_key.pub
ssh_host_rsa_key
ssh_host_rsa_key.pub

How does OpenSSH decide which host key to use?

During SSH connection establishment, the OpenSSH server presents a default host key signature algorithm to the SSH client. To check the list of key signature algorithms and their order, you can perform a query with ssh -Q sig.

$ ssh -Q sig
ssh-ed25519
sk-ssh-ed25519@openssh.com
ssh-rsa
rsa-sha2-256
rsa-sha2-512
ssh-dss
ecdsa-sha2-nistp256
ecdsa-sha2-nistp384
ecdsa-sha2-nistp521
sk-ecdsa-sha2-nistp256@openssh.com
webauthn-sk-ecdsa-sha2-nistp256@openssh.com

HostKeyAlgorithms

You can also get a list of key types and signature algorithms with ssh -Q HostKeyAlgorithms:

$ ssh -Q HostKeyAlgorithms
ssh-ed25519
ssh-ed25519-cert-v01@openssh.com
sk-ssh-ed25519@openssh.com
sk-ssh-ed25519-cert-v01@openssh.com
ssh-rsa
rsa-sha2-256
rsa-sha2-512
ssh-dss
ecdsa-sha2-nistp256
ecdsa-sha2-nistp384
ecdsa-sha2-nistp521
sk-ecdsa-sha2-nistp256@openssh.com
webauthn-sk-ecdsa-sha2-nistp256@openssh.com
ssh-rsa-cert-v01@openssh.com
rsa-sha2-256-cert-v01@openssh.com
rsa-sha2-512-cert-v01@openssh.com
ssh-dss-cert-v01@openssh.com
ecdsa-sha2-nistp256-cert-v01@openssh.com
ecdsa-sha2-nistp384-cert-v01@openssh.com
ecdsa-sha2-nistp521-cert-v01@openssh.com
sk-ecdsa-sha2-nistp256-cert-v01@openssh.com

The default list of host keys can be modified on the OpenSSH server by configuring the /etc/ssh/sshd_config file, as described below.

Symbol	Description	Example
`+<algorithm_name>`	Algorithm will be appended to default set of list	`+rsa-sha`
`-<algorithm_name>`	Algorithm will be removed	`-rsa-sha`
`^<algorithm_name>`	Algorithm will be default value on top of list	`^rsa-sha`

You can see more in-depth info about SSH configuration with man 5 ssh_config.

Identify host key type

SSH client and server logs provide valuable information about the host key algorithm types that are offered during SSH connections. In case of a mismatch between the client and server, the logs will record the offered key information from the other end. This information will be useful for troubleshooting and identifying the root cause of the issue.

SSH client output:

$ sftp -o HostKeyAlgorithms=ssh-rsa  username@domain
Unable to negotiate with <server_id> port 22: no matching host key type found.
Their offer: rsa-sha2-512,rsa-sha2-256

SSH Server log (/var/log/auth.log):

2022-06-28T22:26:46.692753-07:00 localhost sshd[445086]:
Unable to negotiate with <client_id> port 55872: no matching host key type found.
Their offer: ssh-rsa [preauth]

Third-party application logs

Some third-party applications return the MD5 hash code in the event of a failed SSH connection attempt. This code can be useful in identifying the exact matching key algorithm that needs to be included in the server.

Limiting the SSH algorithms for the SFTP connection to
the following: none, zlib, zlib@openssh.com, aes256-chc, aes192-che,
aes128-cbe, diffie-hellman-group14-sha256, ecdh-sha2-nistp256,
ecdh-sha2-nistp384, ecdh-sha2-nistp521, ssh-dss, ssh-rsa, hmac-sha2-
256, hmac-sha2-512, hmac-shal.

Configuring server validation for the SSH connection using
public keys: 3072: 8b 7e de 33 d8 f4 f5 82 d6 86 68 41 17 ba 72 4c.

The MD5 value of a key found in a third-party application log can be used to identify the matching key on the server with the help of the ssh-keygen utility.

$ ssh-keygen -lf ssh_host_rsa_key.pub
3072 SHA256:kkcdlhNcj8DndHvyLBXGfhI/oZkrTRIAbwuqqd58uY0

$ ssh-keygen -E md5 -lf ssh_host_rsa_key.pub
3072 MD5:8b:7e:de:33:d8:f4:f5:82:d6:86:68:41:17:ba:72:4c root@ndpbh-test-epitrax-app01 (RSA)

ssh-rsa disabled by default

Since the release of OpenSSH version 8.8, the ssh-rsa key algorithm has been disabled. As a result, accessing the latest version of the OpenSSH server from an older client version may lead to non-matching host key failures.

$ ssh -V
OpenSSH_8.9p1 Ubuntu-3, OpenSSL 3.0.2 15 Mar 2022

Notice: http://www.openssh.com/txt/release-8.7

Imminent deprecation notice
===========================
OpenSSH will disable the ssh-rsa signature scheme by default in the
next release.

Legacy Support: https://www.openssh.com/txt/release-8.8

Potentially-incompatible changes
================================

This release disables RSA signatures using the SHA-1 hash algorithm
by default. This change has been made as the SHA-1 hash algorithm is
cryptographically broken, and it is possible to create chosen-prefix
hash collisions for <USD$50K [1]

For most users, this change should be invisible and there is
no need to replace ssh-rsa keys. OpenSSH has supported RFC8332
RSA/SHA-256/512 signatures since release 7.2 and existing ssh-rsa keys
will automatically use the stronger algorithm where possible.

Incompatibility is more likely when connecting to older SSH
implementations that have not been upgraded or have not closely tracked
improvements in the SSH protocol. For these cases, it may be necessary
to selectively re-enable RSA/SHA1 to allow connection and/or user
authentication via the HostkeyAlgorithms and PubkeyAcceptedAlgorithms
options. For example, the following stanza in ~/.ssh/config will enable
RSA/SHA1 for host and user authentication for a single destination host:

    Host old-host
        HostkeyAlgorithms +ssh-rsa
	PubkeyAcceptedAlgorithms +ssh-rsa

We recommend enabling RSA/SHA1 only as a stopgap measure until legacy
implementations can be upgraded or reconfigured with another key type
(such as ECDSA or Ed25519).

[1] "SHA-1 is a Shambles: First Chosen-Prefix Collision on SHA-1 and
    Application to the PGP Web of Trust" Leurent, G and Peyrin, T
    (2020) https://eprint.iacr.org/2020/014.pdf

Solution

The SSH client and server offer options to specify the host key algorithm to use during SSH connections. Therefore, you can modify the client or server configuration to establish a connection based on the restrictions of one of the systems involved.

Establishing an SSH connection using the most secure host key algorithm is highly recommended for optimal security. However, in situations where there are limitations on upgrading or changing the algorithm, users must make decisions based on the security policies of their application environment.

In such cases, a suggested solution is to carefully evaluate the security requirements of the application environment and choose the host key algorithm that best aligns with those requirements. This may involve a trade-off between security and compatibility, so it is important to weigh the potential risks and benefits of each option.

SSH Client

When the privilege to configure the SSH server is restricted, the SSH client can be configured to use a specific algorithm. To do this, you can use the -o HostKeyAlgorithms=<host_key> option with the SSH client to specify a particular host key algorithm type.

$ sftp username@server
Unable to negotiate with <server> port 22: no matching host key type found. Their offer: ssh-rsa
Connection closed.
Connection closed

In the above case, the SSH server uses ssh-rsa as the host key algorithm to establish connections. To use ssh-rsa as the key algorithm for the SSH client, you can configure it as follows.

$ sftp -o HostKeyAlgorithms=ssh-rsa username@server
username@server~:$

SSH Server

In some cases, it may not be possible to configure the SSH client to use a specific key when a third-party proprietary application is used to establish the connection. In such scenarios, the SSH server can be configured to include the required key.

In our scenario, the third-party application expects the ssh-rsa key type, but the latest version of OpenSSH server does not provide ssh-rsa as it has been deprecated.

To support old client attempts with ssh-rsa, the key should be added to the bottom of the list with a “+” symbol. This will allow the server to present the deprecated ssh-rsa key to old clients, while still offering more secure key algorithms to newer clients. The new lines in /etc/ssh/sshd_config should look like the following:

HostKeyAlgorithms +ssh-rsa
PubkeyAcceptedAlgorithms +ssh-rsa

Before restarting the SSH service, it is advisable to test the configuration for syntax errors to prevent potential issues, like the following which occurs if you add an extra space to Host KeyAlgorithms +ssh-rsa:

# sshd -t
/etc/ssh/sshd_config: line 32: Bad configuration option: Host
/etc/ssh/sshd_config: terminating, 1 bad configuration options

Once you’ve verified the configuration and sshd -t gives no output, go ahead and restart the service to apply the changes.

# sshd -t

# systemctl reload sshd.service

With this configuration, SSH clients can connect to the server using ssh-rsa. To verify the host key type used in an SSH connection, you can use the verbose (-v) option with the ssh command. This will display additional information about the SSH connection, including the host key algorithm being used. Here we increase the verbosity by adding the maximum of 3 -v options.

$ ssh -vvv -oHostKeyAlgorithms=ssh-rsa username@domain
…
debug2: host key algorithms: ssh-rsa
debug1: Will attempt key: .ssh/id_rsa RSA
SHA256:CmARXi+/8UwrvzwS7RkJNqD/rhroYTnfB285OnovVFs agent
…

Conclusion

SSH is an incredible tool, but errors like the ones described here can make it a bit harder to work with if you’re not aware of what’s going on behind the scenes. Let us know in the comments if you’ve had any similar experiences!

ssh-askpass on macOS for SSH agent confirmation

2022-11-23T00:00:00+00:00

Photo by Kristoffer Trolle, CC BY 2.0

At End Point Dev we mostly use SSH keys for authentication when connecting to remote servers and Git services. The majority of the time, the servers we are trying to visit are barred from direct access and require a middle “jump server” instead.

Enabling SSH agent forwarding makes it easier to reuse SSH private keys. It keeps the private keys on our local machine and uses them to authenticate with each server in the chain without entering a password.

However, this approach comes with an inherent risk of the agent being hijacked if one of the servers is compromised. This means a bad guy could use the SSH keys to compromise downstream servers.

In this post, we’ll cover a simple way to protect against SSH agent hijacking. We will see in detail on macOS how to configure a system-wide agent using ssh-askpass to pop up a graphical window to ask for confirmation before using the agent.

How it works

It is strongly recommended to use the -c option on the ssh-add command when adding your SSH keys to the agent in order to protect yourself against SSH agent hijacking.

With this, every time a request is made to utilize the private key stored in the SSH agent, ssh-askpass will display a prompt on your local computer asking you to approve the usage of the key. By doing this, it becomes more difficult for a remote attacker to use the private key without authorization.

If you don’t want to use the ssh-askpass agent confirmation, I recommend using the OpenSSH feature ProxyJump rather than agent forwarding to get an equivalent level of security.

Installing ssh-askpass on macOS

So let’s set this up. The recommended way is with Homebrew:

Install ssh-askpass with Homebrew

$ brew tap theseal/ssh-askpass
$ brew install ssh-askpass

You might see some warnings. Go ahead and proceed with them.

Now we need to start the Homebrew services. Note that this is a brew service, not a regular macOS daemon service.

$ brew services start theseal/ssh-askpass/ssh-askpass
=> Successfully started `ssh-askpass` (label: homebrew.mxcl.ssh-askpass)

Behind the scenes, it just sets the SSH_ASKPASS and SUDO_ASKPASS environment variables and stops ssh-agent, so that the SSH agent can pick up these environment variables when it restarts.

To list the services and make sure it’s started:

$ brew services list | grep ssh-askpass
ssh-askpass started ~/Library/LaunchAgents/homebrew.mxcl.ssh-askpass.plist

Install ssh-askpass with MacPorts

If you prefer MacPorts, do:

$ sudo port install ssh-askpass

Install ssh-askpass from source code

And of course you can install from source if you wish:

$ cp ssh-askpass /usr/local/bin/
$ cp ssh-askpass.plist ~/Library/LaunchAgents/
$ launchctl load -w ~/Library/LaunchAgents/ssh-askpass.plist

Configure the SSH agent with the `ssh-add -c` option

Let’s first verify that the agent is running, then add the private key with the confirmation option:

$ ssh-add -l
The agent has no identities.
$ ssh-add -c .ssh/id_rsa
Enter passphrase for .ssh/id_rsa (will confirm after each use):
Identity added: .ssh/id_rsa (.ssh/id_rsa)
The user must confirm each use of the key

The Identity will get added if you provide the correct passphrase for the key. This can be confirmed by listing the keys again with ssh-add -l.

ssh-askpass agent confirmation

Now let’s log into a remote server.

You will be prompted to confirm the private key’s usage with the pop-up window:

Set up keyboard shortcuts

Since it’s too easy to hit the spacebar and accept a connection, ssh-askpass defaults to the cancel option. We can use keyboard shortcuts to press “OK” by following the below steps.

Go to “System Preferences” and then “Keyboard”.

macOS 10.15 Catalina, 11 Big Sur, 12 Monterey

Go to “Shortcuts” tab
check the option “Use keyboard navigation to move focus between controls”.

macOS 13 Ventura

Turn on the “Keyboard navigation” option.

Now you can press tab ⇥ and then the spacebar to select “OK”.

Enjoy!

SSH Key Auth using KeeAgent with Git Bash and Windows CLI OpenSSH

2022-08-08T00:00:00+00:00

In a previous blog post we showed how to configure KeePass and KeeAgent on Windows to provide SSH key agent forwarding with confirmation while using PuTTY and other PuTTY agent compatible programs. In this post we’ll expand on that by showing how to use the same key agent to provide SSH key auth when using Git Bash and the Windows command line OpenSSH.

Git Bash support

Open KeePass, click on Tools → Options, select the KeeAgent tab.

Create C:\Temp if it does not exist.

Check the two boxes in the Cygwin/MSYS Integration section.

Directly after each box, fill in the path: C:\Temp\cygwin-ssh.socket for the Cygwin compatible socket file, and C:\Temp\msys-ssh.socket for the msysGit compatible socket file.

Click OK.

Open Git Bash.

Create the file ~/.bash_profile with the contents:

test -f ~/.profile && . ~/.profile
test -f ~/.bashrc && . ~/.bashrc

Create the file ~/.bashrc with the contents:

export SSH_AUTH_SOCK="C:\Temp\cygwin-ssh.socket"

Close and reopen Git Bash.

You should now be able to SSH with Git Bash using your loaded SSH key and a dialog box should appear to approve the use of the key.

Windows command line OpenSSH support

Open KeePass, click on Tools → Options, select the KeeAgent tab.

Scroll down and click on the box next to “Enable agent for Windows OpenSSH (experimental).”

Click OK.

Open a Windows Command Prompt.

You should now be able to SSH with Windows CLI using your loaded SSH key and a dialog box should appear to approve the use of the key.

Windows SSH key agent forwarding confirmation

2022-07-26T00:00:00+00:00

At End Point we use SSH keys extensively, primarily for authentication with servers for remote shell access as well as with Git services including GitHub, GitLab, and Bitbucket. Most of the time the servers we are attempting to reach are blocked from direct access and require that we go through an intermediate “jump server”.

Because of this need to jump from server to server we utilize SSH key forwarding that allows us to use the private key stored on our local system to authenticate with each of the servers in the chain. When we reach our destination server we can use the same private key to authenticate with the Git hosting service and perform git commands without having to enter a password.

One of the best practices when using SSH key forwarding is to use an option called key confirmation. When key confirmation is turned on, each time a request is made to use the private key that is loaded in the SSH agent a prompt will appear on your local machine to approve the use of the key. This reduces the ability for an attacker to use your private key without approval.

For the longest time SSH key confirmation was not available on Windows. One of the most popular SSH clients on Windows is PuTTY and its agent (pageant) does not support the option. Many other SSH key compatible Windows applications use PuTTY’s agent for SSH key caching and as a result these applications also lack the ability for key confirmation.

KeePass and KeeAgent

To use key confirmation on Windows we need to utilize two programs, KeePass and KeeAgent. KeePass is an open source password manager and KeeAgent is a plugin for KeePass that provides a PuTTY-compatible SSH key agent. It also appears that KeeAgent has support for integration with the Windows shell, Cygwin/MSYS, and Git Bash.

The instructions below will assume that you already have a SSH key in PuTTY that you’d like to use with key confirmation and that you have previously used PuTTY with key forwarding.

You should start by installing KeePass. Then install KeeAgent.

Once both are installed create a new KeePass database, or use your existing database if you are already a KeePass user.

Add a new entry to the database and name it SSH key. Enter your SSH key password into the Password and Repeat fields.

Then click on the KeeAgent tab and check ‘Allow KeeAgent to use this entry’.

In the Private Key section select External File and point it at your PuTTY private key. If you have entered the correct password on the first tab you should see your key comment and fingerprint listed. Then press OK.

Verify that confirmation is enabled by clicking on Tools -> Options and selecting the KeeAgent tab.

Press OK.

Then go to File -> Save. Close KeePass and re-open it. You’ll be asked to enter your KeePass password and then you can verify that the agent is loaded with your key by clicking Tools -> KeeAgent.

Now when we use PuTTY or another PuTTY agent-compatible program we’ll be presented with a confirmation dialog. Clicking Yes will allow the key to be used.

Notice that the default selected option is No. This is different than the standard openssh-askpass on Linux, which defaults to Yes. If you’re typing along in a fury and the confirmation window pops up and you hit Enter or space, it will decline the use of your SSH key, rather than accepting it.

If you have enabled SSH key forwarding in the PuTTY options for the connection you’ll be using you can then SSH to other servers using the same key and each time you do so the confirmation will be presented to you.

If you close KeePass the key agent will be closed and unavailable for future connections. Re-opening KeePass will allow the key to be used again.

If you use Windows and SSH agent forwarding but have never tried agent confirmation to protect against malicious use of your secret key, give KeePass and KeeAgent a try!

Using SSH tunnels to get around network limitations

2022-01-26T00:00:00+00:00

SSH is an extremely useful way to use computers that you aren’t in front of physically.

It can also be used to overcome some unique networking challenges, particularly those where one computer needs to connect to another in an unorthodox way. Let me show you a couple of uses of SSH tunnels that have come in handy for me personally.

Serving content without a public IP address

In the past, I wrote about maintaining a Minecraft server to play on with my friends. In that case I was dealing with being physically separated from the server hardware I intended to use, but once I got that machine back, I realized that I still had a problem: My ISP and their networking gear didn’t support port forwarding, meaning that I couldn’t connect to my server from outside my home network. But even if I could have, the public IP address I was assigned changed regularly.

One solution I found was to use a reverse SSH tunnel to forward traffic from a publicly-visible virtual server to my local server.

To set this up, you just need your local machine and a server with a publicly visible IP address. I used a virtual machine from UpCloud, which costs $5 per month, but you could use any other server, as long as it has its own IP address. Setup is fairly simple.

First, on the local machine, we create a new SSH public & private key pair just for this connection. This serves as a way to authenticate without a password, but without using a personal SSH key that has access to many more places.

ssh-keygen -t ed25519 -f ~/.ssh/ed25519 -q -N ""

Next, we create a new OS user proxy on the server. Creating a user specifically for this purpose lets us make sure that our password-free SSH key doesn’t give access to any sensitive data on the remote server. On Linux:

useradd -m proxy

Then, we add the public key generated earlier to authorized_keys in the new proxy user’s ~/.ssh folder.

Finally, on the local machine, we run this command:

ssh -fN -i ~/.ssh/id_ed25519 -R 0.0.0.0:25565:localhost:25565 proxy@myserver

Here’s what each part of that does:

-f tells ssh to run in the background.
-N doesn’t run a command on the remote host, which is perfect since we’re just forwarding traffic.
-i specifies the SSH key we’re using.
-R is a bit more complicated. 0.0.0.0:25565 specifies which traffic to intercept on the remote host and localhost:25565 where to forward it to. Note that 0.0.0.0 means to listen on all IP addresses, such as 127.0.0.1, your internal network NAT address, and any others. 25565 is a TCP port number (UDP isn’t supported by SSH out of the box) and will vary based on the application you’re using.
proxy@myserver is the user and hostname of the remote server.

And that’s it! The reverse tunnel will now forward traffic from our publicly visible server to the specified port to our machine.

Another good use for this setup is when you have an IPv4 address at the edge of a network but only IPv6 internally. This is becoming more common as IPv4 addresses become more scarce.

Also note that for more permanent situations, a VPN like WireGuard or OpenVPN might be a better choice than an SSH tunnel; however, for lower-volume traffic SSH works just fine, and it is often quicker to set up.

Splitting a multi-container Docker app between multiple machines

There are many other uses for SSH tunnels. One of our clients has an application with a lot of different moving parts that all need to communicate with each other for the entire application to function. For instance, there’s a database container, a container for part of the backend that needs CUDA support, and several others, in addition to a React frontend. Applications like this can of course slow your computer down quite a bit. What if you only need to make modifications to the frontend or another light part of the application? I asked around and found that a couple of coworkers had already found a workaround.

The solution is simple but effective: If you have another computer available — in my case, a desktop computer that I don’t usually use for work — you can run the performance-intensive containers on that machine. This allows you to work on the lighter parts of the application without experiencing slowdown (or fan noise, or other things like that) on your laptop.

Because the different parts of the application were already using Docker, it wasn’t hard to run the different pieces on separate machines, but they needed to know to talk to each other across the network. SSH tunnels let us do that without modifying our Docker configuration!

For the rest of this example I’ll refer to the two computers in this scenario as “the laptop” and “the desktop”, running the frontend and backend portions respectively, but keep in mind that you could do this with other setups as well.

1. Collecting some info

Before we can set everything up, we need to know which ports our application is communicating on. Examining the Dockerfiles for the backend containers in this application, I found that it was using ports 9933, 9934, and 10004. Normally, our frontend application would be communicating with these containers by talking to http://localhost:9933 and so on, but once we know which ports it’s going to use, we can set up SSH to forward the traffic to our desktop machine instead.

The other important piece of info we’ll need is the (private home network) IP address of our desktop machine. In this case, mine is 192.168.0.55, so we’ll use that.

2. Setting up the SSH config

The most important step to making the different machines talk to each other is going to be in our SSH config, ~/.ssh/config.

We’ll start by coming up with an alias hostname for our desktop, like backend, and creating a section for it in our config file. We then add a LocalForward line for each port we want to forward like so:

Host backend
	HostName 192.168.0.55
	User zed
	ForwardAgent yes
	LocalForward 9933 127.0.0.1:9933
	LocalForward 9934 127.0.0.1:9934
	LocalForward 10004 127.0.0.1:10004

3. Connecting the computers

For the different parts of the application to talk to each other, there needs to be an active SSH connection. I usually open an SSH connection like normal (make sure to use the hostname we set in step 2!), start a tmux session, and then run the backend portions of the app in the tmux session:

[zed@laptop ~]$ ssh desktop-machine
[zed@desktop ~]$ get-stuff-started

However, if you’d prefer, you can also open a reverse SSH connection that will run in the background until killed:

[zed@laptop ~]$ ssh -fN desktop-machine
[zed@laptop ~]$

4. Test it!

Now that we’ve got an SSH session running to forward traffic to the desktop machine, you can try spinning up the React app and see if it connects properly.

Conclusion

There are many other uses for SSH tunnels. Feel free to let us know in the comments how you’ve used them!

Setting up SSH in Visual Studio Code

2022-01-06T00:00:00+00:00

Visual Studio Code is a powerful code editor that can create a customized IDE for your development. VS Code’s default configuration is great for working locally but lacks the functionality to give the same experience for remote SSH development. Enter the extension Remote SSH.

Installation

Installing the Remote SSH extension is really easy! First you access the Extension Marketplace with Ctrl+Shift+X or by clicking View > Extensions in the menu, then you just search for and select Remote - SSH.

Setting up your SSH config file

To configure your connection, you’ll need to add a few lines to your SSH config. Click the green Open a Remote Window icon on the bottom left corner:

Select Open SSH Configuration File... and select the config file you want to use. I use the Linux default, /home/$USER/.ssh/config. Add the Host, HostName, and User as required and save:

Host MySite
  HostName site.endpointdev.com
  User couragyn

Connecting

Click the green Open a Remote Window icon on the bottom left corner, select Connect to Host..., and pick your desired host, in this case MySite. If your public SSH key isn’t on the remote server, you will be prompted to enter a password. If your key is on the server, it will state it has your fingerprint and prompt you to continue.

You’re now connected and can use VS Code’s features like Terminal and Debug Console just like you would locally.

Opening the working directory

Wouldn’t it be nice to have VS Code automatically open to the correct folder once your SSH connection is established? Unfortunately there isn’t a way to set a folder location in the settings yet; you’d need to click Open Folder and navigate to the project root every time you connect.

There is, however, a workaround to make this a bit less tedious:

Click Open Folder
Navigate to the project root
Click File > Save Workplace As...
Save your .code-workspace file somewhere it won’t be picked up by Git

Now open your workspace again with a new connection. If the workspace was recently used, you can use File > Open Recent > $Workspace.code-workspace; otherwise go to File > Open Workspace... and select your .code-workspace file. This should get you set up right in the correct directory after you’ve connected.

SSH with multiple hops

Sometimes you will need to SSH into one location before tunneling into another. To connect to a remote host through an intermediate jump host, you will need to add ForwardAgent and ProxyJump to the config file, like this:

Host MySite
  HostName site.endpointdev.com
  User couragyn
  ForwardAgent yes

Host SiteThatNeedsToGoThroughMySite
  HostName completely.different.com
  User couragyn
  ProxyJump MySite

Happy remote development!

Lock down your security with GPG on a YubiKey

2021-09-10T00:00:00+00:00

Photo by Mauro Sbicego on Unsplash

Gnu Privacy Guard (GnuPG or GPG) is a tool we use a lot at End Point. Its ubiquity and quite decent security is a perfect fit for us — and there’s a way to make it even safer.

GPG uses the OpenPGP standard to encrypt files. Normally, one creates a PGP key on their computer and just keeps the keyfile safe. A password is generally used, but as with any private key, it’s only as safe as the computer it’s on.

Got a YubiKey and not sure what to do with it? Want to get a little more secure with your encryption?

In case you haven’t heard of them, YubiKeys are hardware USB keys that can be used as a multi-factor authentication (MFA) token, or to fill in one-time password (OTP) fields (like those generated by Google Authenticator) on sites that don’t support the YubiKey directly as an MFA token.

Using a smart card like a YubiKey can increase GPG’s security, especially if the key is generated on an air-gapped machine. This way the keyfile is stored in the hardware security token, and is never exposed to the internet.

In addition, you can even store an SSH key on the card, which will enable you to log in to remote Linux machines while keeping your private key secured.

While there isn’t full password locking on hardware tokens, YubiKey and almost all OpenPGP keys have two PINs — a user PIN and an administrative PIN to reset the user PIN. If you enter either or both three times incorrectly, the card will lock and you’ll need to reload from backup (or, in some cases, throw the card away) which is why it’s critical to have a backup.

There are several options for smart cards beyond the YubiKey. You can use any OpenPGP compatible card and reader, or an all-in-one solution that’s compatible with OpenPGP.

The following instructions do require a basic understanding of the command line and of how to create a live CD/USB stick, but if you need to use GPG, you’re probably already at least somewhat familiar with these requirements.

An air-gapped machine isn’t required for these instructions. You could do this on any machine you trust, but using a machine with a fresh OS that hasn’t been connected to the internet affords the highest level of security. Simply booting from a live CD/USB is fairly easy. Choosing an operating sytem that comes with the smart card daemon (scdaemon) will help. If you don’t, make sure you download the scdaemon package for your operating system to use with the live CD/USB.

Prepare

Before you begin, you’ll need:

A smart card solution as described above.
A backup smart card, or external media on which to store an encrypted copy of the key.
A machine on which to generate the key.
A live OS. For this demonstration I used Tails, The Amnesic Incognito Live System. It’s specifically designed to not store logs or keep data from one reboot to another.
A way to access these instructions, like a second computer, phone, printout, or a very good memory.

Boot up the live machine. Note that if you’re using Tails, there are two settings you’ll need to choose on the welcome screen. Click the plus button and do the following:

Set an administrative password. Tails doesn’t set a root password by default, and thus disallows root access for better security. You can set one yourself.
Disable networking in the ‘Network Connection’ section.

Access and configure the card

Once booted, run an admin terminal, or load a terminal and run sudo -i. It’ll prompt you for the password you just set.

Ensure you can access the card and that the smart card daemon is installed by running gpg --edit-card. It should display information about your smart card.

If you’re using an air-gapped machine and your live OS is missing requisite packages, don’t access the internet with the machine in order to install it, since that would break the air gap. Instead, copy the installation files across using sneakernet: Add the files to a USB drive, perhaps the one you’ll be using to back up your PGP key). The packages for a Debian-based machine are: scdaemon, libccid, pcscd, rng-tools, and gnupg2. Debian has instructions on how to get packages to an air-gapped machine.

Set a PIN for your card, if you haven’t already. Start with:

$ gpg --edit-card

The default PIN for a YubiKey should be 123456, and the default admin PIN should be 12345678. Check the documentation that came with your key, though!

Enable admin features first:

gpg/card> admin

This should return Admin commands are allowed.

Set the passwords, both for the regular PIN and the admin PIN:

gpg/card> passwd

1 - change PIN <- Default 123456
2 - unblock PIN <- To reset the pin with the AdminPin / Reset Code
3 - change Admin PIN <- Default 12345678
4 - set the Reset Code <- See below
Q - quit

Do not mix up your PIN and admin PIN! You can lock up your card, which will require a factory reset.

The reset code is set if you are setting up the card for someone else to use, and wish to give them a way to reset the PIN without having full access to the rest of the admin functions.

Generate the keys

Run the following to generate the key:

$ gpg --expert --full-gen-key

Key type: 1 (RSA & RSA). (You can also use “ECC & ECC” if you’re brave. These types of keys may not work with older systems and implementations of GPG, however, so your mileage may vary. If you do want to use ECC & ECC, use Curve 25519 in the next step.)
Key size: this should be the maximum supported by your key. YubiKey 4 or 5 can support up to 4096. Use this for both main and subkey.
Expiry: This is your choice. I’d set it to a year or two.
Real name, email, and comment: I recommend leaving the comment blank, since most of the time the email address will be enough information.
Next, GPG will ask you to move your mouse around — don’t sprain anything while generating entropy!

Now your key is generated. If you only have one spare storage device which you want to use for backups, copy the revocation certificate and the SSH public key to storage, sneak this on to your main computer, and only then copy the keys over to the backup drive. Don’t attach the backup drive to anything but an air-gapped machine once it holds your key!

You might as well generate an SSH key now. Even if you don’t use it, there’s no harm in having it.

$ gpg --expert --edit-key <your email/key id>

Run gpg/card> addkey to add a key.
Type 8, RSA (set your own capabilities). If this option doesn’t show up, ensure you used --expert above.
Enable authentication and disable signing and encrypting — type s, e, a, then q to save.
4096 bits is the best number to use, at least for RSA on modern YubiKeys. If you have an older key you may be limited to 3072 or 2048.
You can choose to have the key expire, but ensure you have a backup method of logging in.
Confirm your choices, then quit, confirming you want to save.

For more in-depth instructions, visit https://opensource.com/article/19/4/gpg-subkeys-ssh

Export the public keys:

$ gpg --export --armor <key ID> > /path/to/thumbdrive/<email>_pub.asc

$ gpg --export-ssh <key ID> > /path/to/thumbdrive/YubiKey_id_rsa.pub

Move the keys to your card

Once you have the key added to your keyring, you’ll need to transfer that key to your card:

$ gpg --edit-key <key id>

To export the public SSH key you’ll need to put on remote servers, you can run the command: gpg --export-ssh-key 0x123456789ABCDE

gpg> keytocard — confirm you want to move the primary key and store this in position 1 of the card.
- To select the encryption key, type key 1.
- Run keytocard to store the encryption key in the encryption slot.
If you have an encryption key:
- To select the authentication key, run key 2.
- To deselect the key first key, run key 1. You should only have one key with * marking it.
- Store in the authentication slot: keytocard.
Repeat this for as many subkeys as you have.
Once you’re done, quit and confirm the saved changes.

Time to test your new key!

And that’s it. Publish your new public GPG key, use your new SSH key, secure in the knowledge that your private key is protected from malicious attacks by an additional hardware layer.

What do I do if it all went wrong or I locked up the card?

You can start again. The following command will restore the GPG-compatible portion of your YubiKey to factory settings. You will lose any keys stored on the card. I don’t believe it’ll cause any OTP/2FA set up with the card to be lost, but I make no guarantees.

ONLY RUN THE FOLLOWING IF YOUR PIN IS LOCKED AND THE SMART CARD IS UNUSABLE:

$ gpg --edit-card
gpg/card> factory-reset

Follow the confirmation steps on screen.

Consolidating Multiple SFTP Accounts Into One Master Account

2020-03-16T00:00:00+00:00

Photo by Dan Meyers

Recently, a client implemented a data-intensive workflow to generate various reports and insights from a list of facilities as part of an EpiTrax installation. Because a significant portion of these files contain sensitive healthcare data, they needed to strictly comply with HIPAA. Optimally, facilities should be able to transfer files securely and exclusively to our server. One of the best methods of achieving this is to create individual SSH File Transfer Protocol (SFTP) accounts for each source.

SFTP account

Private SFTP accounts were established for each facility and the data was received at a designated path. At these individual points of contact, a third-party application picks up the data and processes further into the pipeline. The following demonstrates how SFTP accounts are developed and configured:

Create a user group for SFTP accounts:

$ addgroup sftpusers

Configure the following settings in sshd_config (this enables an SFTP account and sets the default location as the home path):

$ vi /etc/ssh/sshd_config
...
# override default of no subsystems
Subsystem       sftp    internal-sftp...

Match Group sftpusers
    ChrootDirectory /home/%u
    AllowTCPForwarding no
    X11Forwarding no
    ForceCommand internal-sftp

Restart SSH server to apply changes:

$ systemctl restart ssh

Create an SFTP user account for a facility and place in a folder on the home path to receive data:

# set new user name
sftpuser=the-new-username
useradd $sftpuser
usermod -g sftpusers -s /usr/sbin/nologin $sftpuser
mkdir -p /home/$sftpuser/INPUT_PATH/
chown -R root:root /home/$sftpuser

Mount multiple accounts to one account

The goal here is to point the data from many facilities to one location, but using a single account and path for multiple sites’ data could result in a breach of security and/or privacy. Mounting the receiving path of a facility’s data onto a single master account and then to a “mount point” with a unique facility name takes care of this issue. The process next consolidates files from individual paths on a master account in one place where the application picks up messages for further processing.

The SFTP accounts and the master account should be attached to the same group. This will permit individual accounts to write on the master account-mounted path. In turn, the master account can read files from the same location. This location now has administrative rights for both the SFTP user the group. Group permission of the mounted folder is set to sftpgroup and user permission is set to the facility account.

# create master user account
$ adduser master
$ passwd master

# Add master user to sftpgroup group
usermod -a -G sftpgroup master
getent group sftpgroup

# MOUNT_PATH and sub folders
mkdir -p /home/master/MOUNT_PATH/{Input,Pickup,Backup,Archive}
chown -R master:master /home/master/MOUNT_PATH

We wrote a script to automate the process of creating an SFTP account, mounting it at the master account path, and adding fstab entries to save the mount in the case of a reboot. The script not only saves time, but also avoids human error when creating accounts for all facilities.

#!/bin/bash

# ./create_sftp_account.sh sftpfacilityone FACILITY_ONE

sftpuser=$1
facility_name=$2

# Create SFTP account and add to sftpgroup
useradd $sftpuser
usermod -g sftpusers -s /usr/sbin/nologin $sftpuser
usermod -a -G sftpgroup $sftpuser
mkdir -p /home/$sftpuser/INPUT_PATH
chown root:root /home/$sftpuser

# Create path specific to facility in master account Input folder
mkdir -p /home/master/MOUNT_PATH/Input/$facility_name
chown -R $sftpuser:sftpgroup /home/master/MOUNT_PATH/Input/$facility_name

# Mount sftp account into master account and set permissions
mount --bind /home/master/MOUNT_PATH/Input/$facility_name /home/$sftpuser/INPUT_PATH
chown -R $sftpuser:sftpgroup /home/$sftpuser/INPUT_PATH
chmod g=rwxs /home/$sftpuser/INPUT_PATH

# Add fstab entry to persist and mount on reboot
echo "/home/master/MOUNT_PATH/Input/$facility_name /home/$sftpuser/INPUT_PATH none bind 0 0" >> /etc/fstab
echo "Created user $sftpuser at $facility_name mount point successfully"

Files at one location

Now, data files from facilities are available at individual folders under MOUNT_PATH/Input on the master account. This enables third-party applications to pick up files in a straightforward way to proceed with further processing. It also helps our client access the files for review from the master account easily without navigating into each separate account.

Summary

Mounting multiple SFTP accounts onto one master account turns out to be an efficient and beneficial method of consolidating data. Both safe and secure, running separate SFTP accounts establishes an exclusive private link between facilities and servers. The master account has the unique ability to access files belonging to each facility in order to process the data further.

Tip: In order to avoid broken mounts, check the status by using the command mount -fav. Problems with mount configurations can cause broken mounts after rebooting the server.

The mystery of the disappearing SSH key

2017-04-13T00:00:00+00:00

Photo by Jay Huang

SSH (Secure Shell) is one of the programs I use every single day at work, primarily to connect to our client’s servers. Usually it is a rock-solid program that simply works as expected, but recently I discovered it behaving quite strangely - a server I had visited many times before was now refusing my attempts to login. The underlying problem turned out to be a misguided decision by the developers of OpenSSH to deprecate DSA keys. How I discovered this problem is described below (as well as two solutions).

The use of the ssh program is not simply limited to logging in and connecting to remote servers. It also supports many powerful features, one of the most important being the ability to chain multiple connections with the ProxyCommand option. By using this, you can “login” to servers that you cannot reach directly, by linking together two or more servers behind the scenes.

As as example, let’s consider a client named “Acme Anvils” that strictly controls access to its production servers. They make all SSH traffic come in through a single server, named dmz.acme-anvils.com, and only on port 2222. They also only allow certain public IPs to connect to this server, via whitelisting. On our side, End Point has a server, named portal.endpoint.com, that I can use as a jumping off point, which has a fixed IP that we can give to our clients to whitelist. Rather than logging in to “portal”, getting a prompt, and then logging in to “dmz”, I can simply add an entry in my ~/.ssh/config file to automatically create a tunnel between the servers - at which point I can reach the client’s server by typing “ssh acmedmz”:

##
## Client: ACME ANVILS
##

## Acme Anvil's DMZ server (dmz.acme-anvils.com)
Host acmedmz
User endpoint
HostName 555.123.45.67
Port 2222
ProxyCommand ssh -q greg@portal.endpoint.com nc -w 180s %h %p

Notice that the “Host” name may be set to anything you want. The connection to the client’s server uses a non-standard port, and the username changes from “greg” to “endpoint”, but all of that is hidden away from me as now the login is simply:

[greg@localhost]$ ssh acmedmz
[endpoint@dmz]$

It’s unusual that I’ll actually need to do any work on the dmz server, of course, so the tunnel gets extended another hop to the db1.acme-anvils.com server:

##
## Client: ACME ANVILS
##

## Acme Anvil's DMZ server (dmz.acme-anvils.com)
Host acmedmz
User endpoint
HostName 555.123.45.67
Port 2222
ProxyCommand ssh -q greg@portal.endpoint.com nc -w 180s %h %p

## Acme Anvil's main database (db1.acme-anvils.com)
Host acmedb1
User postgres
HostName db1
ProxyCommand ssh -q acmedmz nc -w 180s %h %p

Notice how the second ProxyCommand references the “Host” of the section above it. Neat stuff. When I type “ssh acemdb1”, I’m actually connecting to the portal.endpoint.com server, then immediately running the netcat (nc) command in the background, then going through netcat to dmz.acme-anvils.com and running a second netcat command on that server, and finally going through both netcats to login to the db1.acme-anvils.com server. It sounds a little complicated, but quickly becomes part of your standard tool set once you wrap your head around it. After you update your .ssh/config file, you soon forget about all the tunneling and feel as though you are connecting directly to all your servers. That is, until something breaks, as it did recently for me.

The actual client this happened with was not “Acme Anvils”, of course, and it was a connection that went through four servers and three ProxyCommands, but for demonstration purposes let’s pretend it happened on a simple connection to the dmz.acme-anvils.com server. I had not connected to the server in question for a long time, but I needed to make some adjustments to a tail_n_mail configuration file. The first login attempt failed completely:

[greg@localhost]$ ssh acmedmz
endpoint@dmz.acme-anvils.com's password:

Although the connection to portal.endpoint.com worked fine, the connection to the client server failed. This is not an unusual problem: it usually signifies that either ssh-agent is not running, or that I forgot to feed it the correct key via the ssh-add program. However, I quickly discovered that ssh-agent was working and contained all my usual keys. Moreover, I was able to connect to other sites with no problem! On a hunch, I tried breaking down the connections into manual steps. First, I tried logging in to the “portal” server. It logged me in with no problem. Then I tried to login from there to dmz.acme-anvils.com—which also logged me in with no problem! But trying to get there via ProxyCommand still failed. What was going on?

When in doubt, crank up the debugging. For the ssh program, using the -v option turns on some minimal debugging. Running the original command from my computer with this option enabled quickly revealed the problem:

[greg@localhost]$ ssh -v acmedmz
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017
debug1: Reading configuration data /home/greg/.ssh/config
debug1: /home/greg/.ssh/config line 1227: Applying options for acmedmz
debug1: Reading configuration data /etc/ssh/ssh_config
...
debug1: Executing proxy command: exec ssh -q greg@portal.endpoint.com nc -w 180s 555.123.45.67 2222
...
debug1: Authenticating to dmz.acme-anvils.com:2222 as 'endpoint'
...
debug1: Host 'dmz.acme-anvils.com' is known and matches the ECDSA host key.
...
debug1: Skipping ssh-dss key /home/greg/.ssh/greg2048dsa.key - not in PubkeyAcceptedKeyTypes
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/greg/.ssh/greg4096rsa.key
debug1: Next authentication method: password
endpoint@dmz.acme-anvils.com's password:

The problem is that my DSA key (the “ssh-dss key”) was rejected by my ssh program. As we will see below, DSA keys are rejected by default in recent versions of the OpenSSH program. But why was I still able to login when not hopping through the middle server? The solution lays in the fact that when I use the ProxyCommand, my ssh program is negotiating with the final server, and is refusing to use my DSA key. However, when I ssh to the portal.endpoint.com server, and then on to the next one, the second server has no problem using my (forwarded) DSA key! Using the -v option on the connection from portal.endpoint.com to dmz.acme-anvils.com reveals another clue:

[greg@portal]$ ssh -v endpoint@dmz.acme-anvils.com:2222
...
debug1: Connecting to dmz [1234:5678:90ab:cd::e] port 2222.
...
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/greg/.ssh/endpoint2.ssh
debug1: Authentications that can continue: publickey,password
debug1: Offering DSA public key: /home/greg/.ssh/endpoint.ssh
debug1: Server accepts key: pkalg ssh-dss blen 819
debug1: Authentication succeeded (publickey).
Authenticated to dmz ([1234:5678:90ab:cd::e]:2222).
...
debug1: Entering interactive session.
[endpoint@dmz]$

If you look closely at the above, you will see that we first offered an RSA key, which was rejected, and then we successfully offered a DSA key. This means that the endpoint@dm account has a DSA, but not a RSA, public key inside of its ~/.ssh/authorized_keys file. Since I was able to connect to portal.endpoint.com, its ~/.ssh/authorized_keys file must have my RSA key.

For the failing connection, ssh was able to use my RSA key to connect to portal.endpoint.com, run the netcat command, and then continue on to the dmz.acme-anvils.com server. However, this connection failed as the only key my local ssh program would provide was the RSA one, which the dmz server did not have.

For the working connection, ssh was able to connect to portal.endpoint.com as before, and then into an interactive prompt. However, when I then connected via ssh to dmz.acme-anvils.com, it was the ssh program on portal, not my local computer, which negotiated with the dmz server. It had no problem using a DSA key, so I was able to login. Note that both keys were happily forwarded to portal.endpoint.com, even though my ssh program refused to use them!

The quick solution to the problem, of course, was to upload my RSA key to the dmz.acme-anvils.com server. Once this was done, my local ssh program was more than happy to login by sending the RSA key along the tunnel.

Another solution to this problem is to instruct your SSH programs to recognize DSA keys again. To do this, add this line to your local SSH config file ($HOME/.ssh/config), or to the global SSH config file (/etc/ssh/config):

PubkeyAcceptedKeyTypes +ssh-dss

As mentioned earlier, this whole mess was caused by the OpenSSH program deciding to deprecate DSA keys. Their rationale for targeting all DSA keys seems a little weak at best: certainly I don’t feel that my 2048-bit DSA key is in any way a weak link. But the writing is on the wall now for DSA, so you may as well replace your DSA keys with RSA ones (and an ed25519 key as well, in anticipation of when ssh-agent is able to support them!). More information about the decision to force out DSA keys can be found in this great analysis of the OpenSSH source code.

SSH Key Access Recovery on EC2 Instances

2016-10-02T00:00:00+00:00

Can’t access your EC2 instance? Throw it away and spin up a new one! Everyone subscribes to the cattle server pattern, yes?

Not quite, of course. Until you reach a certain scale, that’s not as easy to maintain as a smaller grouping of pet servers. While you can certainly run with that pattern on Amazon, EC2 instances aren’t quite as friendly about overcoming access issues as other providers are. That’s to say, even if you have access to the AWS account there’s no interface for forcing a root password change or the like.

But sometimes you need that, as an End Point client did recently. It was a legacy platform, and the party that set up the environment wasn’t available. However an issue popped up that needed solved, so we needed a way to get in. The process involves some EBS surgery and does involve a little bit of downtime, but is fairly straightforward. The client’s system was effectively down already, so taking it all the way offline had little impact.

Also do make sure this is the actual problem, and not that the connection is blocked by security group configuration or some such.

For safe keeping, it’s recommended to create a snapshot of the original root volume. If something goes wrong, you can then roll back to that point.

Create temporary working instance in the same availability zone as the instance to fix. A micro instance is fine. Don’t forget to set a key that you do have access to.

Find the instance you need to fix and note the root volume ID. Double check that it has an elastic IP assignment: when stopped the instance IP address will change if not associated with a static elastic IP. Similarly, ephemeral storage will be cleared (but hopefully you’re not relying on having anything permanently there anyway.)

Stop, do not terminate, the instance to be fixed.

Find the root volume for the instance to be fixed using the ID found earlier (or just click it within the instance details pane) and detach it from the instance. Attach it to your working instance.

Connect in to your working instance, and mount that volume as /mnt (or anywhere, really. Just using /mnt as an example here.)

Copy any needed ssh keys into the .ssh/authorized_keys under /mnt/home/ubuntu/ or /mnt/home/ec2-user/ depending on the base distro used for the image, or even just to /mnt/root/. And/or make any other fixes needed.

When all is good, umount the volume and detach it from the working instance. Attach it back to the original instance as /dev/sda1 (even though it doesn’t say that’s an option.)

Boot original instance. If all goes well you should now be able to connect in using the ssh key you added. Ensure that everything comes up on boot. Terminate the temporary working instance (and do make sure you don’t terminate the wrong one.)

That’s not the only approach, of course. If you have a partially functional system, for example, you may be better off immediately creating a volume from the snapshot created in step 0, mounting that in another instance for the modifications, and then performing the stop, root volume swap, and start in quick succession. That’ll minimize any actual downtime, at the potential expense of losing any data changes happening in between the snapshot and the reboot.

Either way just remember there are options, and all is not lost!

One-time password SSH solutions

2015-02-02T00:00:00+00:00

[how encryption was done in the 18th century]

In a previous article I explained how I used a one‑time password system to enable SSH on my Chromebook. While this is still working great for me, there are a few problems that can pop up.

Problem 1: Tripping the alarm

Normally a single password is asked for when logging in using one-time passwords via the otpw program. However, otpw will issue a three‑password prompt when it thinks someone may be trying to perform a race attack (in other words, it think two people are trying to log in at once). The prompt will change from the normal one to this:

Password 206/002/011:

This alarm can be tripped from SSH timing out, or from getting pulled away from your computer mid‑login. In theory, it could be someone trying to break in to my laptop, but that is very unlikely. Needless to say, trying to squint at a paper and keyboard in the dark is hard enough with one password, much less three! It’s usually much easier (in most cases) for me to walk to my laptop and remove the ~/.otpw.lock file, which will clear the “intruder alarm” and cause otpw to prompt for a single password again. Another solution is to set a timeout on clearing the lock, for example via a cronjob that removes the lock file if it was created more than 10 minutes ago. Finally, it may be helpful to have a way to remotely clear the lock. I’ve not written it yet, but one good approach would be to use port knocking to remove the lock file.

Problem 2: Lost or compromised passwords

The physical sheet of paper containing your one‑time passwords is definitely a single point of failure—but an easy one to remedy. Maybe you lost the paper, maybe your arch‑nemesis stole it, or maybe it got destroyed in a freak hunting accident. No worries at all, just generate a new one! Run the otpw‑bin command again:

$ otpw-gen -e 30 | lpr

When you do so, the old ~/.otpw file is completely overwritten, and the old sheet of paper is now completely worthless. Solutions don’t get any easier than that. If your sheet is stolen and you need to quickly disable one‑time passwords, you can also just manually remove the .otpw file, which effectively turns off one‑time passwords for that account.

Problem 3: Unusable passwords

The password to use for login is randomly determined, so one time you may be asked to look up password 312 and the next 031. Sometimes, however, you cannot use one of the passwords on your printout. Perhaps you spilled something on it. Perhaps (as has happened to me!) the paper was folded in such a way that the crease made it difficult to read the characters. Perhaps your keyboard has a really hard time typing a certain letter. :) Whatever the reason, there are two solutions. First, generate a new sheet by rerunning otpw‑bin as described in Problem 2 above. Second, you can mark the current password as “complete” and have otpw move on to the next number.

Forcing otpw to advance to the next number is slightly tricky. Basically, you need to modify the .otpw file and change the current password hash to a line of hyphens. Here is what the top of a .otpw file looks like after 3 successful logins:

OTPW1
392 3 12 5
---------------
---------------
---------------
077jA:EAgMCJ2uM
097yG3IDv%gyUB7
1077T7EQq%K7E/F
101xeS3I+zMw8GZ
109xCEBXYFb3%3v
121AzwjOJYyBqD%
068WewLA3EIsLmx
065Jdq=2WDwHZ9D
089npYNavK9MIVA

The first line states the format of the file, while the second line indicates the number of passwords generated, the digits per password number, the digits in the hash, and the digits in the actual passwords. All the other lines are passwords—either an unused one consisting of the number and the hash, or a line of hyphens. The goal is to replace the current password with hyphens. Here’s a quick recipe to do so:

perl -ni -e 'print unless /^\d\d\d\S/ and ! $x++' ~/.otpw

We do an in‑place edit (-i) of the .otpw file, looping through a line at a time (-n), and run a command against each line (-e …). The first time we find a line that starts with three numbers and then contains non‑whitespace, we skip it. Everything else is printed and thus goes back into the file.

Those are some ways to handle three of the common problems that can occur when using one‑time passwords. Have other problems, or better solutions to the above? Let me know in the comments.

SSH one-time passwords (otpw) on chromebook

2015-01-21T00:00:00+00:00

Henri Coandă Bucureşt Airport
by Cristian Bortes

A little while ago, I bought a Chromebook as an alternative to my sturdy-but-heavy laptop. So far, it has been great—quick boot up, no fan, long battery life, and light as a feather. Perfect for bringing from room to room, and for getting some work done in a darkened bedroom at night. The one large drawback was a lack of SSH, a tool I use very often. I’ll describe how I used one-time passwords to overcome this problem, and made my Chromebook a much more productive tool.

The options for using SSH on Chrome OS are not that good. I downloaded and tried a handful of apps, but each had some significant problems. One flaw shared across all of them was a lack of something like ssh-agent, which will cache your SSH passphrase so that you don’t have to type it every time you open a new SSH session. An option was to use a password-less key, or a very short passphrase, but I did not want to make everything less secure. The storage of the SSH private key was an issue as well—the Chromebook has very limited storage options, and relies on putting most things “in the cloud”.

What was needed was a way to use SSH in a very insecure environment, while providing as much security as possible. Eureka! A one-time password system is exactly what I needed. Specifically, the wonderful otpw program. Chromebooks have a simple shell (accessed via ctrl-alt-t) that has SSH support. So the solution was to use one-time passwords and not store anything at all on the Chromebook.

Rather than trying to get otpw setup on all the servers I might need to reach, I simply set it up on my main laptop, carefully allowed incoming SSH connections, and now I can ssh from my Chromebook to my laptop. From there, to the world. Best of all, when I ssh in, I can use the already running ssh-agent on the laptop! All it takes is memorizing a single passphrase and securing a sheet of paper (which is far easier to secure than an entire Chromebook :)

Here are some details on how I set things up. On the Chromebook, nothing is needed except to open up a crosh tab with ctrl-alt-t, and run ssh. On the laptop side, the first step is to install the otpw program, and then configure PAM so that it uses it:

$ sudo aptitude install otpw-bin
$ sudo cat >> /etc/pam.d/ssh
  auth     required  pam_otpw.so
  session  optional  pam_otpw.so

That is the bare minimum, but I also wanted to make sure that only ‘local’ machines could SSH in. While there are a number of ways to do this, such as iptables or /etc/hosts.allow, I decided the best approach was to configure sshd itself. The “Match” directive instructs that the lines after it only take effect on a positive match. Thus:

$ sudo cat >> /etc/ssh/sshd_config
AllowUsers nobodyatall
Match Address 192.168.1.0/24,127.0.0.0
AllowUsers greg
$ service ssh restart

The next step is to create the one-time password list. This is done with the otwp-gen program; here is the command I use:

$ otpw-gen -e 30 | lpr
Generating random seed ...

If your paper password list is stolen, the thief should not gain access to your account with this information alone. Therefore, you need to memorize and enter below a prefix password. You will have to enter that each time directly before entering the one-time password (on the same line).

When you log in, a 3-digit password number will be displayed. It identifies the one-time password on your list that you have to append to the prefix password. If another login to your account is in progress at the same time, several password numbers may be shown and all corresponding passwords have to be appended after the prefix password. Best generate a new password list when you have used up half of the old one.

Enter new prefix password: 
Reenter prefix password: 

Creating '~/.otpw'.
Generating new one-time passwords ...

The otpw-gen command creates a file named .otpw in your home directory, which contains the hash of all the one-time passwords to use. In the example above, the -e controls the entropy of the generated passwords—in other words, how long they are. otpw-gen will not accept an entropy lower than 30, which will generate passwords that are five characters long. The default entropy, 48, generates passwords that are eight characters long, which I found a little too long to remember when trying to read from the printout in a dark room. :). Rather than show the list of passwords on the screen, or save them to a local file, the output goes directly to the printer. otpw-gen does a great job of formatting the page, and it ends up looking like this:

Here are some close-ups of what the passwords look like at various entropies:

Sample output with a low entropy of 30:
OTPW list generated 2015-07-12 13:23 on gregsbox

000 GGS%F  056 bTqut  112 f8iJs  168 lQVjk  224 gNG2x  280 -x8ke  336 egm5n
001 urHLf  057 a/Wwh  113 -PEpV  169 9ABpK  225 -K2db  281 babfX  337 feeED
002 vqrX:  058 rZszx  114 r3m8a  170 -UzX3  226 g74RI  282 gusBJ  338 ;Tr4m
003 fa%6G  059 -i4FZ  115 nPEaJ  171 o64FR  227 uBu:h  283 uBo/U  339 ;pYY8
004 -LYZY  060 vWDnw  116 f5Sb+  172 hopr+  228 rWXvb  284 rksPQ  340 ;v6GN

Sample output with the default entropy of 48:
OTPW list generated 2015-15-05 15:53 on gregsbox

000 tcsx qqlb  056 ougp yuzo  112 lxwt oitl  168 giap vqsj  224 vtvk rjc/
001 mfui ukph  057 wbpw aktt  113 kert wozj  169 ihed psyx  225 ducx pze=
002 wwsj hdcr  058 jmwa mguo  114 idtk zrzw  170 ecow fepm  226 ikru hty+
003 aoeb klnz  059 pvie fbfc  115 fmlb sptb  171 ftrd jotb  227 mqns ivq:
004 yclw hyml  060 slvj ezfi  116 djsy ycse  172 butg guzm  228 pfyv ytq%
005 eilj cufp  061 zlma yxxl  117 skyf ieht  173 vbtd rmsy  229 pzyn zlc/

Sample output with a high entropy of 79:
OTPW list generated 2015-07-05 18:74 on gregsbox

000 jeo SqM bQ9Y ato  056 AyT jsc YbU0 rXB  112 Og/ I3O 39nY W/Z
001 AFk W+5 J+2m e1J  057 MXy O9j FjA8 8q;  113 a6A 8R9 /Ofr E4s
002 02+ XPB 8B2S +qT  058 Cl4 6g2 /9Bk KO=  114 HEK vd3 T2TT Rr.
003 Exb jqE iK49 rfX  059 Qhz eU+ J2VG kwQ  115 aJ7 tg1 dJsr vf.
004 Bg1 b;5 p0qI f/m  060 VKz dpa G7;e 7jR  116 kaL OSw dC8e kx.

The final step is to SSH from the Chromebook to the laptop! Hit ctrl-alt-t, and you will get a new tab with a crosh prompt. From there, attempt to ssh to the laptop, and you will see the usual otpw prompt:

$ ssh greg@192.168.1.10
Password 140:

So you type in the passphrase you entered above when running the otpw-gen command, then pull out your sheet of paper and look up the matching password next to number 140. Voila! I am now connected securely to my more powerful computer, and can SSH from there to anywhere I am used to going to from my laptop. I can even run mutt as if I were at the laptop! A nice workaround for the limitations of the Chromebook.

Scripting ssh master connections

2014-03-17T00:00:00+00:00

At End Point, security is a top priority. We just phased out the last of the 1024-bit keys for all of our employees—those of us in ops roles that have keys lots of places had done so a long while back. Similarly, since we’ll tend to have several sessions open for a long while, a number of us will use ssh-agent’s -c (confirm) option. That forces a prompt for confirmation of each request the agent gets. It can get a little annoying (especially since it takes the focus over to one monitor, even if I’m working on the other) but it combats SSH socket hijacking when we have the agent forwarded to remote servers.

Working on server migrations is where it gets really annoying. I like to write little repeatable scripts that I can tweak and re-run as needed. They’re usually simple little things, starting with a bunch of rsync’s or pipe-over-ssh’s for pg_dump or any other data we need to move across. With any more than a couple of those ssh connections in there, repeatedly hitting the confirm button gets irritating fast. And if a large transfer takes a while, I’ll go off to do something else, later getting an unexpected confirmation box when I’m not thinking about the running script. Unexpected SSH auth confirmations, of course, get denied. So the script has to be re-run, and the vicious cycle repeats anew.

ssh has a neat ability to multiplex over a single connection. I have it set to do that locally in auto mode, and that made me wonder if it could be used in these scripts so I only have to authorize the connection once. Well, of course it can, and it turns out to be nothing special. But here’s what I got to work:

ssh
    -o ControlPath=~/.ssh/user@server.domain.foo  <em># Set the socket location</em>
    -M  <em># Defines master mode for the client</em>
    -N  <em># Don't bother to do anything remotely yet</em>
    -f  <em># Drop into the background so we can continue on</em>
    user@server.domain.foo  <em># And the typical connection username/host</em>

At the end of the script, remember to shut down the control socket by passing it an exit command, otherwise it’ll leave a connection hanging around out there:

ssh -o ControlPath=~/.ssh/user@server.domain.foo -O exit user@server.domain.foo

In between, it’s just a matter of using the ControlPath option. It can get a little repetitive, so a variable can be helpful. The latest iteration I have wraps the ssh command together with its -o ControlPath option, which can be executed directly or passed to rsync, as below.

As an example, here’s a somewhat stripped-down version of the script we used to move bucardo.org to a new host, minus some error handling and such:

#!/bin/bash

MSSH="ssh -o ControlPath=~/.ssh/root@bucardo.org" 
$MSSH -MNf bucardo.org

# Tell rsync to use ssh pointed at the master socket
rsync -e "$MSSH" -aHAX --del bucardo.org:/var/lib/git/ /var/lib/git/

# Pipe pg_dump (or whatever you need) over ssh
$MSSH bucardo.org 'su - postgres -c "pg_dump -c wikidb"' | su - postgres -c "psql wikidb"

# sed -i commands or anything else that's needed to fix up the local configuration

$MSSH -O exit bucardo.org

Ansiblizing SSH Keys

2014-03-03T00:00:00+00:00

It is occasionally the case that several users share a particular account on a few boxes, such as in a scenario where a test server and a production server share a deployment account, and several developers work on them. In these situations the preference is to authenticate the users with their ssh keys through authorized_keys on the account they are sharing, which leads to the problem of keeping the keys synchronized when they are updated and changed. We add the additional parameter that perhaps any given box will have a few users of the account that aren’t shared by the others, but otherwise allow a core of developers to access them. Now extend this scenario across hundreds of machines, and the maintenance becomes difficult or impossible when updating any of the core accounts. Obviously this is a job for a remote management framework like Ansible.

Our Example Scenario

We have developers Alice, Bob and Carla which need access to every box. We have additional developers Dwayne and Edward that only need access to one box each. We have a collection of servers: dev1, dev2, staging and prod. All of the servers have an account called web_deployment.

The authorized_keys for web_deployment on each box contains:

dev1
- alice
- bob
- carla
dev2
- alice
- bob
- carla
- dwayne
staging
- alice
- bob
- carla
- edward
prod
- alice
- bob
- carla

Enter Ansible

Ansible is setup for every box already. The basic strategy for managing the keys is to copy a default authorized_keys file from the ansible host containing Alice, Bob and Carla (since they are present on all of the destination machines) and assemble the keys with a collection of keys local to the host (Dwayne’s key on dev2, and Edward’s key on staging). To perform the assembly action we also want to provide a script so that the keys can be manually manipulated (local keys changed) without touching the ansible box. The script is thus:

#!/usr/bin/env bash
set -u -o errexit -o pipefail

target_ssh_dir="/home/web_deployment/.ssh"
base_authorized_key_file="authorized_keys"

local_authorized_keys="${target_ssh_dir}/${base_authorized_key_file}.local"
hosting_authorized_keys="${target_ssh_dir}/${base_authorized_key_file}.hosting"
target_authorized_keys="${target_ssh_dir}/${base_authorized_key_file}"

tmp_authorized_keys="${target_ssh_dir}/${base_authorized_key_file}.tmp"

authorized_keys_backup_dir="${target_ssh_dir}/history"

# BEGIN multiline configuration_management_disclaimer string variable
configuration_management_disclaimer="\n\
# ******************************************************************************\n\
# This file is automatically managed by End Point Configuration management
# system. In order to change it please apply your changes
# to $local_authorized_keys and run $0\n\
# so to assemble a new $target_authorized_keys\n\
# ******************************************************************************\n\
"
# END multiline configuration_management_disclaimer string variable

# BEGIN assembling tmp file
echo -e "$configuration_management_disclaimer" > $tmp_authorized_keys

echo -e "# BEGIN STANDARD HOSTING KEYS\n" >> $tmp_authorized_keys
cat $hosting_authorized_keys >> $tmp_authorized_keys
echo -e "# END STANDARD HOSTING KEYS\n" >> $tmp_authorized_keys

if [[ -r $local_authorized_keys ]]
then
  echo -e "# BEGIN LOCAL KEYS\n" >> $tmp_authorized_keys
  cat $local_authorized_keys >> $tmp_authorized_keys
  echo -e "# END LOCAL KEYS\n" >> $tmp_authorized_keys
fi

echo -e "$configuration_management_disclaimer" >> $tmp_authorized_keys
# END assembling tmp file

# BEGIN check (and do) backup of old file
if ! cmp $tmp_authorized_keys $target_authorized_keys &> /dev/null
then
  mkdir -p $authorized_keys_backup_dir

  backup_old_auth_keys="${authorized_keys_backup_dir}/${base_authorized_key_file}_$(date '+%Y%m%dT%H%M%z')"
  cat $target_authorized_keys > $backup_old_auth_keys
fi
# END check (and do) backup of old file

cat $tmp_authorized_keys > $target_authorized_keys

rm $tmp_authorized_keys

if [ -d $authorized_keys_backup_dir ]
then
  if [ -n "$(find $authorized_keys_backup_dir -maxdepth 0 -type d)" ]
  then
    chmod -R u=rwX,go= $authorized_keys_backup_dir
  fi
fi

if [ -f $local_authorized_keys ]
then
  if [ -n "$(find $local_authorized_keys -maxdepth 0 -type f)" ]
  then
    chmod u=rw $local_authorized_keys
  fi
fi

if [ -f $hosting_authorized_keys ]
then
  if [ -n "$(find $hosting_authorized_keys -maxdepth 0 -type f)" ]
  then
    chmod u=rw $hosting_authorized_keys
  fi
fi

We then use an Ansible task to distribute the files to the destination hosts:

# tasks/authorized_keys_deploy.yml
---
  - name: Create /home/web_deployment subdirectories
    file: path=/home/web_deployment/{{ item }}
          state=directory
          owner=web_deployment
          group=web_deployment
          mode=0700
    with_items:
      - .ssh
      - bin

  - name: Copy /home/web_deployment/.ssh/authorized_keys.universal
    template: src=all/home/web_deployment/.ssh/authorized_keys.universal.j2
          dest=/home/web_deployment/.ssh/authorized_keys.hosting
          owner=web_deployment
          group=web_deployment
          mode=0600
    notify:
      - Use shellscript to locally assemble authorized_keys

  - name: Copy /home/web_deployment/bin/assemble_authorized_keys.sh
    copy: src=files/all/home/web_deployment/bin/assemble_authorized_keys.sh
          dest=/home/web_deployment/bin/assemble_authorized_keys.sh
          owner=web_deployment
          group=web_deployment
          mode=0700
    notify:
      - Use shellscript to locally assemble authorized_keys

This task is invoked by an Ansible playbook:

# authorized_keys_deploy.yml
---
- name: authorized_keys file deployment/management
  hosts: authorized_keys_servers
  user: root

  handlers:
  - include: handlers/authorized_keys_deploy.yml

  tasks:
  - include: tasks/authorized_keys_deploy.yml

And finally the handler which invokes the assembly script:

# handlers/authorized_keys_deploy.yml
---
  - name: Use shellscript to locally assemble authorized_keys
    command: "/home/web_deployment/bin/assemble_authorized_keys.sh"

A note about this setup: The authorized_keys.universal has the extension .j2, and is invoked as a Jinja2 template. This allows server-specific conditionals amongst other things. It is useful for example when per-key shell features are used (for example restricting one particular key to invoking rsync for backups), and if the OS selection is mixed thus requiring the paths to differ between hosts.

Conclusion

We hope that this example is helpful. There are some clear directions for improvement and ways to make this suit other scenarios, such as having the universal keys list also merged with an additional keys list select by host or other information such as OS or abstract values associated with the host. One could also envision a system in which some arbitrary collection of key files are merged at the destination server selected through the aforementioned means or others.

Shout out to Lele Calo for his putting together the Ansible setup for this procedure.

Use Ansible/Jinja2 templates to change file content based on target OS

2013-12-19T00:00:00+00:00

In the End Point hosting team we really love automating repetitive tasks, especially when it involves remembering many little details which can over time be forgotten, like differences of coreutils location between some versions of Ubuntu (Debian), CentOS (Red Hat) and OpenBSD variants.

In our environment we bind the backup SSH user authorized_keys entry to a custom command in order to have it secured by being, among other aspects, tied to a specific rsync call.

So in our case the content of our CentOS authorized_keys would be something like:

command="/bin/nice -15 /usr/bin/rsync --server --daemon .",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa AAAB3[...]Q== endpoint-backup

Sadly that’s only true for CentOS systems so that if you want to automate the distribution of authorized_keys (as we’ll show in another post) to different Linux distributions (like Ubuntu) you may need to tweak it to comply to the new standard “/usr/bin” location, which will be eventually adopted by all new Linux versions overtime.. RHEL 7.x onward included.

To do the OS version detection we decided to use an Ansible/Jinja2 template by placing the following line in the Ansible task:

- name: Deploy /root/.ssh/authorized_keys
  template: src=all/root/.ssh/authorized_keys.j2
            dest=/root/.ssh/authorized_keys
            owner=root
            group=root
            mode=0600

And inside the actual file place a slightly modified version of the line above:

command="{% if ansible_os_family != "RedHat" %}/usr{% endif %}/bin/nice -15 /usr/bin/rsync --server --daemon .",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa AAAB3[...]Q== endpoint-backup"

So that if the target OS is not part of the “RedHat” family it will add the “/usr” in front of the “/bin/nice” absolute path.

Easy peasy, ain’t it?

Now go out there and exploit this feature to all your needs.

Managing Multiple Hosts and SSH Identities with OpenSSH

2013-12-12T00:00:00+00:00

When I started working at End Point I was faced with the prospect of having multiple SSH identities for the first time. I had historically used an RSA SSH key with the default length of 2048 bits, but for my work at End Point I needed to generate a new key that was 4096 bits long.

Although I could have used ssh-copy-id to copy my new SSH public key to all of my old servers, I liked the idea of maintaining separate “personal” and “work” identities and decided to look for a way to automatically use the right key based on the server I was trying to connect to.

For the first few days I was specifying my new identity on the command line using:

ssh -i .ssh/endpoint_rsa patrick@server.example.com

That worked, but I often forgot to specify my new SSH identity when connecting to a server, only realizing my mistake when I was prompted for a password instead of being authenticated automatically.

Host Definitions

I had previously learned the value of creating an ssh_config file when I replaced a series of command-line aliases with equivalent entries in the SSH config file.

Instead of creating aliases in my shell:

alias server1='ssh -p 2222 -L 3389:192.168.1.99:3389 patrick@server1.example.com'

I learned that I could add an equivalent entry to my ~/.ssh/config file:

Host server1
  HostName server1.example.com
  Port 2222
  User patrick
  LocalForward 3389 192.168.1.99:3389

Then, to connect to that server, all I needed to do was run ssh server1 and all of the configuration details would be pulled in from the SSH config file. Replacing my series of shell aliases with Host definitions had the added benefit of automatically carrying over to other tools like git and mosh which read the same configuration.

Switching Identities Automatically

There’s an easy solution to managing multiple SSH identities if you only use one identity per server; use ssh-add to store all of your keys in the SSH authentication agent. For example, I used ssh-add ~/.ssh/endpoint_rsa to add my new key, and ssh-add -l to verify that it was showing up in the list of known keys. After adding all of your keys to the agent, it will automatically try them in order for SSH connections until it finds one that authenticates successfully.

Manually Defining Identities

If you need more control over which identity an SSH session is using, the IdentityFile option in ssh_config lets you specify which key will be used to authenticate. Here’s an example:

Host server2
  HostName server2.example.com
  User patrick
  IdentityFile ~/.ssh/endpoint_rsa

This usage is particularly helpful when you have a server that accepts more than one of your identities and you need to control which one should be used.

SSH ProxyCommand with netcat and socat

2013-04-24T00:00:00+00:00

Picture by Tambako the Jaguar

Most of my day to day is work is conducted via a terminal, using Secure Shell (SSH) to connect to various servers. I make extensive use of the local SSH configuration file, ~/.ssh/config file, both to reduce typing by aliasing connections, and to allow me to seamlessly connect to servers, even when a direct connection is not possible, by use of the ProxyCommand option.

There are many servers I work on that cannot be directly reached, and this is where the ProxyCommand option really comes to the rescue. It allows you to chain two or more servers, such that your SSH connections bounce through the servers to get to the one that you need. For example, some of our clients only allow SSH connections from specific IPs. Rather than worry about which engineers need to connect, and what IPs they may have at the moment, engineers can access servers through certain trusted shell servers. Then our engineers can SSH to one of those servers, and from there on to the client’s servers. As one does not want to actually SSH twice every time a connection is needed, the ProxyCommand option allows a quick tunnel to be created. Here’s an example entry for a .ssh/config file:

Host proxy
User greg
HostName proxy.example.com

Host acme
User gmullane
HostName pgdev.acme.com
ProxyCommand ssh -q proxy nc -w 180 %h %p

So now when we run the command ssh acme, ssh actually first logs into proxy.example.com as the user “greg”, runs the nc (netcat) command (after plugging in the host and port parameters for us), and then logs in to gmullane@pgdev.acme.com from proxy.example.com. We don’t see any of this happening: one simply types “ssh acme” and gets a prompt on the pgdev.acme.com server.

Often times more than one “jump” is needed, but it is easy to chain servers together, such that you can log into a third server by running two ProxyCommands. Recently, this situation arose but with a further wrinkle. There was a server, we’ll call it calamity.acme.com, which was not directly reachable via SSH from the outside world, as it was a tightly locked down production box. However, it was reachable by other boxes within the company’s intranet, including pgdev.acme.com. Thus to login as gmullane on the calamity server, the .ssh/config file would normally look like this:

Host proxy
User greg
HostName proxy.example.com

Host acme
User gmullane
HostName pgdev.acme.com
ProxyCommand ssh -q proxy nc -w 180 %h %p

Host acme_calamity
User gmullane
## This is calamity.acme.com, but pgdev.acme.com cannot resolve that, so we use the IP
HostName 192.168.7.113
ProxyCommand ssh -q acme nc -w 180 %h %p

Thus, we’d expect to run ssh acme_calamity and get a prompt on the calamity box. However, this was not the case. Although I was able to ssh from proxy to acme, and then from acme to calamity, things were failing because acme did not have the nc (netcat) program installed. Further investigation showed that it was not even available via packaging, which surprised me a bit as netcat is a pretty old, standard, and stable program. However, a quick check showed that the socat program was available, so I installed that instead. The socat program is similar to netcat, but much more advanced. I did not need any advanced functionality, however, just a simple bidirectional pipe which the SSH connections could flow over. The new entry in the config file thus became:

Host acme_calamity
User gmullane
HostName 192.168.13.123
ProxyCommand ssh -q acme socat STDIN TCP:%h:%p

After that, everything worked as expected. It’s perfectly fine to mix socat and netcat as we’ve done here; at the end of the day, they are simple dumb pipes (although socat allows them to be not so simple or dumb if desired!). The arguments to socat are simply the two sides of the pipe. One is stdin (sometimes written as stdio or a single dash), and the other is a TCP connection to a specific host and port (which SSH will plug in). You may also see it written as TCP4, which simply forces IPv4 only, where TCP encompasses IPv6 as well.

The options to netcat are very similar, but shorter as it already defaults to using stdin for the one side, and because it defaults to a TCP connection, so we can leave that out as well. The “-w 180” simply establishes a three minute timeout so the connection will close itself on a problem rather than hanging out until manually killed.

Even if both netcat and socat were not available, there are other solutions. In addition to other programs, it is easy enough to write a quick Perl script to create your own bidirectional pipe!

OpenSSH Tips and Tricks with Matt Vollrath

2012-06-14T00:00:00+00:00

Matt Vollrath’s presentation focused on unique solutions Liquid Galaxy administration requires.

Specifically, Liquid Galaxy requires secure access to many public sites, which we don’t have physical access to. OpenSSH helps handle these remote challenges securely and quickly.

Multiplexing for Speed

The LG master node can send commands to all the slave nodes at the same time. This can be helpful to examine all display nodes’ current states without manual work. The multiplexed connection uses options -f (background), -M (control socket), -N (no command), and -T (prevents pseudo-terminal from being allocated), and any further connections to the host do not need to authenticate, for speed.:

ssh -fMNT hostname &

Include the ampersand to be able to track the PID of the background ssh connection. This connection will be maintained until it fails or is killed.

Additionally -O allows us to examine the state of the control connection as well as exit. As of OpenSSH 5.9, you can also use -O with the “stop” command to not accept any more multiplexers.

When combined with a simple bash script to inspect the host connection first, the script can create that multiplexed connection first. This allows for very fast work. Without control connection it takes 700ms to run a command; with sockets, 85ms.

For Liquid Galaxy, everything is done over SSH so this speed is critical for a responsive experience while maintaining security. This responsive experience is multiplied for each node in the setup, which in LG’s case, may be up to 8 nodes.

Reverse SSH for Security

In order to allow access behind deployments’ firewalls, the LG head node connects to a proxy server and then forwards its own SSH port to that proxy server, allowing secure remote connectivity from any network which will allow output SSH.

SSH config wildcards and multiple Postgres servers per client

2011-01-07T00:00:00+00:00

The SSH config file has some nice features that help me to keep my sanity among a wide variety of servers spread across many different clients. Nearly all of my Postgres work is done by using SSH to connect to remote client sites, so the ability to connect to the various servers easily and intuitively is important. I’ll go over an example of how a ssh config file might progress as you deal with an ever‑expanding client.

Some quick background: the ssh config file is a per‑user configuration file for the SSH program. It typically exists as ~/.ssh/config. It has two main purposes: setting global configuration items (such as ForwardX11 no), and setting things on a host‑by‑host basis. We’ll be focusing on the latter.

Inside the ssh config file, you can create Host sections which specify options that apply only to one or more matching hosts. The sections are applied if the host name you type in as the argument to the ssh command matches what is after the word “Host”. As we’ll see, this also allows for wildcards, which can be very useful.

I’m going to walk through a hypothetical client, Acme Corporation, and show how the ssh config can grow as the client does, until the final example mirrors an actual section of my ssh config section file.

So, you’ve just got a new Postgres client called Acme Corporation, and they are using Amazon Web Services (AWS) to host their server. We’re coming in as the postgres user, and have our public ssh keys already in place inside ~postgres/.ssh/authorized_keys on their server. The hostname is ec2‑456‑55‑123‑45.compute‑1.amazonaws.com. So, generally, we would connect by running:

$ ssh postgres@ec2‑456‑55‑123‑45.compute‑1.amazonaws.com

That’s a lot to type each time! We could create a bash alias to handle this, but it’s better to use the ssh config file instead. We’ll add this to the end of our ssh config:

##
## Client: Acme Corporation
##

Host  acmecorp
User postgres
Hostname  ec2-456-55-123-45.compute-1.amazonaws.com

Now we can simply use ‘acmecorp’ in place of that ugly string:

$ ssh acmecorp

Notice that we don’t need to specify the user anymore: ssh config plugs that in for us. We can still override it if we need to connect as someone else:

$ ssh greg@acmecorp

The next week, Acme Corporation decides that rather than allow anyone to SSH to their servers, they will use iptables or something similar to restrict access to select known hosts. Because different people with different IPs at End Point may need to access Acme, and because we don’t want to have Acme have to open a new hole each time we connect from a different place, we will connect from a shared company box. In this case, the box is vp.endpoint.com. Acme arranges to allow SSH from that box to their servers, and each End Point employee has a login on the vp.endpoint.com box. What we need to do now is create a SSH tunnel. Inside of the ssh config file, we add a new line to the entry for ‘acmecorp’:

Host  acmecorp
User  postgres
Hostname  ec2-456-55-123-45.compute-1.amazonaws.com
ProxyCommand  ssh -q greg@vp.endpoint.com nc -w 180 %h %p

Now, when we run this:

$ ssh acmecorp

…everything looks the same to us, but what we are really doing is connecting to vp.endpoint.com, running the nc (netcat) command, and then connecting to the amazonaws.com box over the new netcat connection. (The arguments to netcat specify that the connection should be closed if there is the connection goes away for 180 seconds, and the host and port should be echoed along). As far as amazonaws.com is concerned, we are connecting from vp.endpoint.com. As far as we are concerned, we are going directly to amazonaws.com. A nice side effect, and a big reason why we don’t simply use bash aliases, is that the scp program will use these aliases as well. So we can now do something like this:

$ scp check_postgres.pl acmecorp:

This will copy the check_postgres.pl program from our computer to the Acme one, going through the tunnel at vp.endpoint.com.

Business has been good for Acme lately and they finally have conceded to your strong suggestion to set up a warm standby server (using Postgres’ Point In Time Recovery system). This new server is located at ec2‑456‑55‑123‑99.compute‑1.amazonaws.com, and the internal host name they give it is maindb‑replica (the original box is known as maindb‑db). This new server requires another host entry to ssh config. Rather than copy over the same ProxyCommand, we’ll refactor the information out into a separate host entry. What we end up with is this:

Host  acmetunnel
User  greg
Hostname  vp.endpoint.com

Host  acmedb
User  postgres
Hostname  ec2-456-55-123-45.compute-1.amazonaws.com
ProxyCommand  ssh -q acmetunnel nc -w 180 %h %p

Host  acmereplica
User  postgres
Hostname  ec2-456-55-123-99.compute-1.amazonaws.com
ProxyCommand  ssh -q acmetunnel nc -w 180 %h %p

We also changed the name from acmecorp to just “acme” as that’s enough to uniquely identify among our clients, and who wants to type more than they have to?

Next, the company adds a QA box they want End Point to help setup. This box, however, is not reachable from outside their network; it can be reached only from other hosts in their network. Luckily, we already have access to some of those. What we’ll do is extend our tunnel by one more host, so that the path we travel from us to the Acme QA box is:

Local box → vp.endpoint.com → acreplica → acqa

Here’s the section of the ssh config after we’ve added in the QA box:

Host  acmetunnel
User  greg
Hostname  vp.endpoint.com

Host  acmedb
User  postgres
Hostname  ec2-456-55-123-45.compute-1.amazonaws.com
ProxyCommand  ssh -q acmetunnel nc -w 180 %h %p

Host  acmereplica
User  postgres
Hostname  ec2-456-55-123-99.compute-1.amazonaws.com
ProxyCommand  ssh -q acmetunnel nc -w 180 %h %p

Host  acmeqa
User  postgres
Hostname  qa
ProxyCommand  ssh -q acreplica nc -w 180 %h %p

Note that we don’t need the full hostname at this point for the “acmeqa” Hostname, as we can simply say ‘qa’ and the acreplica box knows how to get there.

There is still some unwanted repetition in the file, so let’s take advantage of the fact that the “Host” item inside the ssh config file will take wildcards as well. It’s not really apparent until you use wildcards, but a ssh host can match more than one “Host” section in the ssh config file, and thus you can achieve a form of inheritance. (However, once something has been set, it cannot be changed, so you always want to set the more specific items first). Here’s what the file looks like after adding a wildcard section:

Host  acme*
User  postgres
ProxyCommand  ssh -q greg@vp.endpoint.com nc -w 180 %h %p

Host  acmedb
Hostname  ec2-456-55-123-45.compute-1.amazonaws.com

Host  acmereplica
Hostname  ec2-456-55-123-99.compute-1.amazonaws.com

Host  acmeqa
User  root
Hostname  qa
ProxyCommand  ssh -q acreplica nc -w 180 %h %p

Notice that the file is now simplified quite a bit. If we run this command:

$ ssh acmereplica

…then the Host acme* section sets up both the User and the ProxyCommand. It then also matches on the Host acmereplica section and applies the Hostname there.

Note that we have removed the “acmetunnel” section. Now that all the ProxyCommands are in a single place, we can simply go back to the original ProxyCommand and specify the exact user and host.

All of the above presumes we want to login as the postgres user, but there are also times when we need to login as a different user (e.g. ‘root’). We can again use wildcards, this time to match the end of the host, to specify which user we want. Anything ending in the letter “r” means we log in as user root, and anything ending in the letter “p” means we log in as user postgres. Our final ssh config section for Acme is now:

##
## Client: Acme Corporation
##

Host  acme*
ProxyCommand  ssh -q greg@vp.endpoint.com nc -w 180 %h %p
Host  acme*r
User  root
Host  acme*p
User  postgres

Host  acmedb*
Hostname  ec2-456-55-123-45.compute-1.amazonaws.com

Host  acmereplica*
Hostname  ec2-456-55-123-99.compute-1.amazonaws.com

Host  acmeqa*
Hostname  qa
ProxyCommand  ssh -q acreplica nc -w 180 %h %p

From this point on, if Acme decides to add a new server, adding it into our ssh config is as simple as adding two lines:

Host  acmedev*
Hostname  ec2-456-55-999-45.compute-1.amazonaws.com

This automatically sets up two hosts for us, “acmedevr” and “acmedevp”. What if we leave out the ending “r” or “p” and just ssh to “acmedev”? Then we’ll connect as the default user, or $ENV{USER} (in my case, “greg”).

Have fun configuring your ssh config file, don’t be afraid to leave lots of comments inside of it, and of course keep it in version control!

Long Lasting SSH Multiplexing Made Simplish

2010-09-01T00:00:00+00:00

My Digression

To start off digressing just a little, I am primarily a developer, lately on longer projects of relatively significant size which means that I stay logged in to the same system for weeks (so long as my X session, local connection, and remote server allow it). I’m also a big believer in lots of screen real estate and using it all when you have it, so I have two monitors running high resolutions (1920x1200), and I use anywhere from 4-6 virtual pages/desktops/screens/(insert term of preference here). On 2-3 of those spaces I generally keep 8 “little black screens” (as my wife likes to call them) or terminals open in 2 rows of 4 windows. In each of those terminals I stay logged in to the same system as long as those aforementioned outside sources will allow me. A little absurd I know…

If you are still with me you may be thinking “why don’t you just use screen?”, I do, in each of those terminals for the inevitable times when network hiccups occur, etc. Next you’ll ask, “why don’t you just use separate tabs/windows/partitions/(whatever they are called) within screen?”, I do, generally I have 2-3 (or more) in each screen instance set to multiple locations on the remote file system. In general I’ve just found being able to look across a row of terminals at various different libs, a database client, tailed logs, etc. is much faster than toggling between windows (heaven forbid they’d be minimized), screen pages, virtual desktops, or even shell job control (though I use a good bit of that too). Inevitably I also seem to pop up an additional window (or two) every now and again and ssh into the same remote destination for the occasional one off command (particularly since I’ve never quite gotten the SSH key thing figured out with long running screens, potential future blog post?). So the bottom line, for any given sizable project I probably ssh to the same location a minimum of 8 times each time I set up my desktop and over the course of a particular X session maybe 20 (or more) times? (End digression)

The Point

As fast as SSHing is, waiting for a prompt when doing it that many times (particularly over the course of a year, or career) really starts to make the clock tick by (or at least it seems that way). Enter “multiplexing” which is essentially a wonderful feature in newer SSH that allows you to start one instance with a control channel that handles many of the slow parts (such as authentication) and so long as it is still running when you connect to the same remote location the new connection uses the existing control channel and is lightning fast getting you to the prompt. Simple enough, to turn on multiplexing you can add the following to your ~/.ssh/config:

# for multiplexing
ControlMaster auto
ControlPath ~/.ssh/multi/master-%r@%h:%p

The above indicates that a master should be used for each connection, and the location of where to store the master’s control socket file. The %r, %h, and %p are expanded to the login, host, and port respectively which is usually enough to make it unique. This should be enough to start using multiplexing, but…and you knew there had to be one…when the master’s control connection is lost all of the slaves to that connection lose their’s as well. With the occasional hung terminal window, or accidental closing of it (if you can remember which is master to begin with), etc. you quickly find that when you normally would not lose connection in a separate terminal window you all of a sudden have lost all of your connections (8+ in my case) which is really painful. Here is where the fun comes in, I use the “-n” and “-N” flags to SSH in a terminal window when I first load up an X session and background the process:

> ssh -Nn user@remote &

The above redirects stdin from /dev/null (a necessary evil when backgrounding SSH procs), prevents the execution of a remote command (meaning we don’t want a shell), and puts the new process in the local shell’s background. Unfortunately, and the part that took me the longest to figure out, is that SSH really likes to have a TTY around (we aren’t using the daemon after all) so simply killing the original terminal window will cause the SSH process to die and zap there went your control connection and all the little children. To get around this little snafu I follow the backgrounding of my SSH process with a bash specific built in disassociating it from the TTY:

> disown

Now I am free to close the original terminal window, the SSH process lives on in the background (as if it were a daemon) and keeps the control connection open so that whenever I use SSH to that remote location (or ‘scp’, etc.) I get an instant response.

Other notes

I could probably set this up to occur when I start X initially automatically, but with a flaky connection I end up needing to do it a few times per X session anyways and you would have to watch out for the sequence of events making sure it occurred after any ssh-agent set up was required.

I tried using ‘nohup’ and can’t remember if I ran into an actual show stopper problem, or if I could just never quite get it to do what I wanted before I stumbled on bash’s disown.

SSH and Old Servers

Troubleshooting

A problem for every solution

Perfect … enough

Mount a remote filesystem over SSH using SSHFS

Notice

Installation

macOS instructions

Linux instructions

Usage

SSH host key verification: a few useful tips

Host keys

How does OpenSSH decide which host key to use?

HostKeyAlgorithms

Identify host key type

Third-party application logs

ssh-rsa disabled by default

Solution

SSH Client

SSH Server

Conclusion

ssh-askpass on macOS for SSH agent confirmation

How it works

Installing ssh-askpass on macOS

Install ssh-askpass with Homebrew

Install ssh-askpass with MacPorts

Install ssh-askpass from source code

Configure the SSH agent with the ssh-add -c option

ssh-askpass agent confirmation

Set up keyboard shortcuts

macOS 10.15 Catalina, 11 Big Sur, 12 Monterey

macOS 13 Ventura

SSH Key Auth using KeeAgent with Git Bash and Windows CLI OpenSSH

Git Bash support

Windows command line OpenSSH support

Windows SSH key agent forwarding confirmation

KeePass and KeeAgent

Using SSH tunnels to get around network limitations

Serving content without a public IP address

Splitting a multi-container Docker app between multiple machines

1. Collecting some info

2. Setting up the SSH config

3. Connecting the computers

4. Test it!

Conclusion

Setting up SSH in Visual Studio Code

Installation

Setting up your SSH config file

Connecting

Opening the working directory

SSH with multiple hops

Lock down your security with GPG on a YubiKey

Got a YubiKey and not sure what to do with it? Want to get a little more secure with your encryption?

Prepare

Access and configure the card

Generate the keys

Move the keys to your card

What do I do if it all went wrong or I locked up the card?

Consolidating Multiple SFTP Accounts Into One Master Account

SFTP account

Mount multiple accounts to one account

Files at one location

Summary

The mystery of the disappearing SSH key

SSH Key Access Recovery on EC2 Instances

One-time password SSH solutions

Problem 1: Tripping the alarm

Problem 2: Lost or compromised passwords

Problem 3: Unusable passwords

SSH one-time passwords (otpw) on chromebook

Scripting ssh master connections

Ansiblizing SSH Keys

Our Example Scenario

Enter Ansible

Conclusion

Use Ansible/Jinja2 templates to change file content based on target OS

Managing Multiple Hosts and SSH Identities with OpenSSH

Host Definitions

Switching Identities Automatically

Manually Defining Identities

Configure the SSH agent with the `ssh-add -c` option