Migrating from Legacy Networking to systemd-networkd on Ubuntu 24.04

2025-11-25T00:00:00+00:00

During a recent Ubuntu server upgrade, I migrated a production system from the legacy ifupdown networking stack to the modern systemd-networkd service. This migration was part of preparing several remote production servers for a smooth upgrade to Ubuntu 24.04, which relies heavily on systemd-managed components.

In this post, I’ll walk through the full migration process, how we automated it safely across multiple remote servers, and the lessons learned from implementing rollback and watchdog mechanisms for zero-downtime transitions.

Why migrate to systemd-networkd?

Ubuntu 24.04 uses systemd-networkd as the default backend for network management. The traditional ifupdown scripts (/etc/network/interfaces) are no longer the recommended way to configure networking. Some of the modern features enabled by systemd-networkd include:

Predictable network interface naming (e.g., ens3, eth0, etc.)
Built-in support for bridges, VLANs, and bonds
Faster boot times with asynchronous network handling
Unified configuration under /etc/systemd/network

Migrating now also avoids future compatibility issues and takes advantage of systemd’s integrated logging and management.

Step 1: Check the current setup

Before the migration, I confirmed the system was still using ifupdown:

ls /etc/network/interfaces
cat /etc/network/interfaces

Output showed legacy configuration similar to:

auto eth0
iface eth0 inet static
    address 10.42.41.1
    netmask 255.255.0.0

Step 2: Install and enable systemd-networkd

Since systemd-networkd wasn’t already active, I installed and enabled it:

sudo apt install systemd-networkd systemd-resolved -y
sudo systemctl enable systemd-networkd
sudo systemctl enable systemd-resolved

Then, I stopped ifupdown to prevent conflicts.

Step 3: Create a network configuration file

Each interface now has its own .network file under /etc/systemd/network/.

DHCP-based interface:

# /etc/systemd/network/10-eth0.network
[Match]
Name=eth0

[Network]
DHCP=yes

Static interface:

# /etc/systemd/network/20-eth1.network
[Match]
Name=eth1

[Network]
Address=10.42.41.1/16
Gateway=10.42.0.1
DNS=8.8.8.8
DNS=1.1.1.1

Step 4: Configure systemd-resolved

This step ensures your system’s DNS resolver works correctly with systemd-networkd.

systemd-resolved listens locally on 127.0.0.53 and handles DNS resolution, caching, and per-interface settings.

Link your system’s /etc/resolv.conf to use it:

sudo ln -sf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf

Then to verify:

resolvectl status

If we skip this, the system might lose DNS resolution after reboot. Applications like apt, curl, and ping may fail to resolve hostnames even though interfaces are up.

Restart and verify

Restart both services:

sudo systemctl restart systemd-networkd
sudo systemctl restart systemd-resolved

Check active links:

networkctl list

Detailed view for one interface:

networkctl status eth0

Sample output:

● 2: eth0
       Type: ether
      State: routable (configured)
     Address: 10.42.41.1/16
     Gateway: 10.42.0.1
     DNS: 8.8.8.8

Test:

ping -c 3 8.8.8.8
ping -c 3 google.com

Step 6: Reboot and confirm

Reboot once to ensure everything persists. After the reboot,

systemctl status systemd-networkd
systemctl status systemd-resolved

and confirm interface and DNS are still functional.

Handling remote migration and downtime risks

Performing this migration remotely on production systems without physical console access required special care. I prepared several layers of protection to prevent accidental lockouts.

Console or IPMI verification: Ensured each node had out-of-band access in case SSH connectivity was lost.

Timed rollback safety script: A background process automatically reverted to legacy networking if the new configuration didn’t come up within a few minutes.

#!/bin/bash
sleep 300
if ! ping -c1 8.8.8.8 &>/dev/null; then
    systemctl disable systemd-networkd
    systemctl enable networking
    reboot
fi

Automating the migration and rollback process

To streamline upgrades across multiple servers, I created automation scripts that handle migration, rollback, and ongoing monitoring.

migrate_to_systemd_networkd.sh

This script performs a complete, logged migration with built-in validation and rollback. It checks for valid .network files, disables legacy services, enables systemd-networkd, tests connectivity, and if all retries fail, restores the old configuration automatically.

Key features

Validates configuration syntax using systemd-analyze verify
Masks conflicting services (ifup@eth*, NetworkManager)
Performs up to three connectivity tests
Restarts SSH and dependent services
Schedules a controlled reboot or user-confirmed one
Rolls back cleanly if connectivity fails

upgrade_watchdog.sh

During do-release-upgrade, networking may restart multiple times. This watchdog script runs in the background to ensure systemd-networkd remains active and services like SSH come back automatically.

Executes every 10 minutes
Verifies /etc/systemd/network/*.network exists and services are running as expected if it fails, this script restarts the service
Restarts systemd-networkd, ssh, and dependent services

Together, both scripts allowed us to perform fully remote OS upgrades with zero manual downtime and consistent recovery paths.

Conclusion

Migrating to systemd-networkd modernizes Ubuntu servers and simplifies long-term maintenance. With the automation scripts and safety mechanisms in place, we successfully migrated multiple remote servers with no downtime and automatic rollback protection.

This upgrade not only prepares infrastructure for Ubuntu 24.04 but also integrates cleanly with our automation and monitoring systems, ensuring reliable networking for years ahead.

Testing Ansible Automation with Molecule

2025-03-28T00:00:00+00:00

Photo by Patrick Lindenberg

1. Why Test with Molecule?

Molecule is a test framework for Ansible roles. It supports testing with multiple instances, operating systems and distributions, virtualization providers, test frameworks, and testing scenarios.

Molecule is useful because it increases confidence in your automation and ensures roles are reliable. It catches issues early in the development cycle, reducing production problems. It also ensures consistency across different environments and platforms.

2. Install Prerequisites

Make sure you have Python >= 3.6 and pip installed. Then install Ansible and Molecule using pip:

pip install ansible
pip install molecule

Molecule uses the delegated driver by default. Other drivers can be installed separately from PyPI, most of them being included in the molecule-plugins package. We are going to use the Docker driver, so let’s install that by running:

pip install "molecule[docker]"

3. Ansible Monorepo Structure

We use the Ansible monorepo structure. This means our playbooks, variables, scripts, roles, plugins, inventory scripts, and configuration all reside and are version controlled together in the same repository.

Here is the high level layout:

~/ansible-endpoint
├── ansible.cfg
├── ansible_hosts
├── molecule/
├── playbooks/
└── roles/
    ├── apache_server/
    ├── ...
    ├── nginx_server/
    └── ...

We’ll focus on roles/nginx_server, showing how to add and configure the Molecule scenarios for testing and also how to verify it.

4. Creating Your First Molecule Scenario

Inside the roles/nginx_server directory, let’s initialize the Molecule scenario:

cd roles/nginx_server
molecule init scenario --driver-name docker --scenario-name default

This command will create the following files:

roles/
└── nginx_server/
    ├── tasks/
    │   └── main.yml
    ├── molecule/
    │   └── default/
    │       ├── molecule.yml
    │       ├── converge.yml
    │       ├── verify.yml
    │       └── destroy.yml
    └── ...

Here are some details about the files created by Molecule:

molecule.yml — The Molecule configuration file.
```
---
driver:
  name: docker
platforms:
  - name: "molecule-rocky9-${CI_JOB_ID}"
    image: geerlingguy/docker-rockylinux9-ansible:latest
    privileged: true
    pre_build_image: true
    volumes:
      - /sys/fs/cgroup:/sys/fs/cgroup:rw
    cgroupns_mode: host
    command: /usr/sbin/init
provisioner:
  name: ansible
```
Key blocks from molecule.yml:
- provisioner is the tool that will be used to provision the scenario. We are using Ansible to run the scenario itself.
- driver — As mentioned above, we are using Docker as the driver.
- platforms — Here we define the target platform for the scenario. We are using the Rocky 9 Docker image as the target platform. Thanks to Jeff Geerling who created the Rocky 9 Docker image for Ansible testing with systemd in it.

converge.yml — The playbook to converge the scenario. This tells Molecule to apply the nginx_server role to our test container (the “instance” defined in molecule.yml).

---
- name: Converge
  hosts: all
  gather_facts: false
  tasks:
    - name: Run the nginx_server role
      hosts: all
      roles:
        - nginx_server

verify.yml — The playbook to verify whether the converge was successful. This tells Molecule to verify that our role has been correctly installed on the Docker instance. Here we are going to check if NGINX is installed, running and also responding to requests.

---
- name: Verify
  hosts: all
  tasks:
    - name: Check if NGINX is installed
      ansible.builtin.package:
        name: nginx
        state: present
      check_mode: true
      register: install
      failed_when: (install is changed) or (install is failed)

    - name: Check if NGINX service is running
      ansible.builtin.service:
        name: nginx
        state: started
        enabled: true
      check_mode: true
      register: service
      failed_when: (service is changed) or (service is failed)

    - name: Verify NGINX is up and running
      ansible.builtin.uri:
        url: http://localhost
        status_code: 200

destroy.yml — Destroys the instance defined in the molecule.yml file. There won’t be any file, but the default Docker driver will automatically destroy the instance after the test is finished. If you want to override this behavior, you can add a destroy.yml file to the scenario directory.

5. Running Molecule tests

A Molecule run consists of the various lifecycle events:
- create
- …
- converge
- …
- verify
- …
- destroy

These are some of the important events in the Molecule lifecycle run. To run the full lifecycle Molecule run, we can use the following command:

cd roles/nginx_server
molecule test

This command executes the following steps in sequence:

Create: Spins up a test Rocky 9 container as mentioned in the molecule.yml file.
Converge: Applies the nginx_server role inside the container.
Verify: Verifies whether the NGINX server is installed, running, and responding to requests.
Destroy: Cleans up the container.

We can also run each step individually:

molecule create
molecule converge
molecule verify
molecule destroy

To avoid rebuilding the containers, we can run commands from converge step.

6. Integrating Molecule with GitLab CI/CD

We want to run the Molecule tests whenever a merge request which changes the particular role is created.

To automatically run the Molecule tests, we can use the GitLab CI/CD.

Here is an example of a .gitlab-ci.yml file that runs our Molecule tests in the Docker based environment for our nginx_server role.

stages:
  - test

before_script:
  - pip install molecule ansible docker molecule-docker

molecule_test:
  stage: test
  variables:
    ROLE_PATH: "roles/nginx_server"
  script:
    - cd $ROLE_PATH
    - molecule test
  rules:
    - changes:
        - $ROLE_PATH/**

With this setup, we’ve set up a Molecule scenario for a specific Ansible role, then run tests both manually and automatically via GitLab CI. This allows tests to trigger whenever a merge request modifies that role, ensuring a quick feedback loop and a higher degree of reliability in the automation.

To learn more about Molecule, here are some useful links:

Automating DNS Management with GitLab CI/CD

2024-09-06T00:00:00+00:00

Photo by Simon Kadula

At End Point, managing DNS records across multiple domains has historically been a manual task. This blog post details our journey from manual processes to an automated workflow using GitLab CI/CD.

Our Initial Approach

With multiple domains and frequent updates necessary to manage the servers, manual handling of DNS changes became a bottleneck. Initially, our process looked like this:

Make changes to the OpenTofu configuration files
Create a merge request (MR) in GitLab
They would run tofu plan manually and paste the plan output into the MR for review
A coworker would review the MR and approve the changes
Once merged, the engineer would manually run tofu apply to implement the changes

While this process worked, automating it could enhance our productivity and minimize errors, integrating our DNS management directly into our CI/CD pipeline.

The Solution: Automating with GitLab CI/CD

Change Submission: Engineers make changes to the OpenTofu files and submit a merge request
Plan Creation: A GitLab CI/CD job automatically generates an OpenTofu plan when changes are proposed
Review Process: A coworker reviews the automatically generated plan in the MR
Applying Changes: Once approved and merged, another CI/CD job automatically runs tofu apply to implement the changes

Implementation Details:

1. `.gitlab-ci.yaml`

If you are doing this on a self-hosted GitLab instance you would need to import the repo to your hosted GitLab workspace and use it there.

We divided our GitLab CI/CD pipelines into three main stages:

Validation: Where we check if everything looks right
Planning: Where we create a plan to show the changes
Deployment: Where the changes get applied once everything is approved

It’s important to note that our CI/CD pipeline runs on a dedicated worker. This means only this worker has access to the secret keys needed for DNS management. By isolating these credentials to a single, controlled environment, we significantly reduce the risk of unauthorized access or accidental exposure of sensitive data.

include:
  - component: gitlab.com/components/opentofu/job-templates@main
    inputs:
      version: 0.24.0
      opentofu_version: 1.6.2
      auto_apply: false

before_script:
  - source /home/gitlab-runner/.tofu.env

stages: [ validate, build, deploy]

fmt-example.com:
  extends: [ .opentofu:fmt ]
  variables:
    TF_ROOT: "example.com"
    TF_STATE_NAME: "example_com"
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
      changes:
        - example.com/*
    - if: '$CI_COMMIT_BRANCH == "main"'
      changes:
        - example.com/*

validate-example.com:
  extends: [ .opentofu:validate ]
  variables:
    TF_ROOT: "example.com"
    TF_STATE_NAME: "example_com"
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
      changes:
        - example.com/*
    - if: '$CI_COMMIT_BRANCH == "main"'
      changes:
        - example.com/*

plan-example.com:
  extends: [ .opentofu:plan ]
  variables:
    TF_ROOT: "example.com"
    TF_STATE_NAME: "example_com"
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
      changes:
        - example.com/*
    - if: '$CI_COMMIT_BRANCH == "main"'
      changes:
        - example.com/*
  after_script:
    - source /home/gitlab-runner/.tofu.env
    - ./gitlab-comment-tofu-log.sh $TF_ROOT/plan_output.txt

apply-example.com:
  extends: [ .opentofu:apply ]
  variables:
    TF_ROOT: "example.com"
    TF_STATE_NAME: "example_com"
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'
      changes:
        - example.com/*
  after_script:
    - source /home/gitlab-runner/.tofu.env
    - ./gitlab-comment-tofu-log.sh $TF_ROOT/apply_output.txt

2. Posting OpenTofu plan/apply logs as merge request comments

To facilitate code reviews, we created a script (gitlab-comment-tofu-log.sh) that automatically formats and posts the OpenTofu plan and apply logs as comments on merge requests. This ensures that reviewers can easily understand the changes before approving them.

Here is the script:

#!/bin/bash

set -e

echo "GitLab MR Comment Script Starts"

if [ $# -eq 0 ]; then
    echo "Error: No output file path provided"
    echo "Usage: $0 <path_to_output_file>"
    exit 1
fi

output_file="$1"
echo "Using output file: $output_file"

echo "Checking environment variables"
if [ -z "$GITLAB_API_TOKEN" ]; then
    echo "Error: GITLAB_API_TOKEN is not set"
    exit 1
fi

if [ -z "$CI_PROJECT_ID" ]; then
    echo "Error: CI_PROJECT_ID is not set"
    exit 1
fi

if [ -n "$CI_MERGE_REQUEST_IID" ]; then
    echo "Using CI_MERGE_REQUEST_IID: $CI_MERGE_REQUEST_IID"
else
    echo "CI_MERGE_REQUEST_IID not set, fetching from API"
    CURRENT_COMMIT_SHA=$(git rev-parse HEAD)
    echo "Current commit SHA: $CURRENT_COMMIT_SHA"

    MR_INFO=$(curl --silent --header "PRIVATE-TOKEN: $GITLAB_API_TOKEN" \
        "https://<link_to_self_hosted_gitlab>/api/v4/projects/$CI_PROJECT_ID/repository/commits/$CURRENT_COMMIT_SHA/merge_requests")

    if [ "$(echo $MR_INFO | jq '. | length')" -eq 0 ]; then
        echo "Error: No merge request found for this commit" >&2
        exit 1
    fi

    CI_MERGE_REQUEST_IID=$(echo $MR_INFO | jq '.[0].iid')
    echo "Found Merge Request IID: $CI_MERGE_REQUEST_IID"
fi

echo "Reading output file: $output_file"
if [ ! -f "$output_file" ]; then
    echo "Error: File $output_file not found"
    exit 1
fi

file_name=$(basename "$output_file")
if [[ "$file_name" == plan_* ]]; then
    heading="<br>Plan log for $TF_ROOT"
    output=$(awk '/OpenTofu used the selected providers/,0' "$output_file")
elif [[ "$file_name" == apply_* ]]; then
    heading="<br>Apply log for $TF_ROOT"
    output=$(awk '/OpenTofu has been successfully initialized/,0' "$output_file")
else
    heading="<br>Tofu Output for $TF_ROOT"
fi

echo "Filtered output read. Length: ${#output} characters"

echo "Cleaning and formatting output"
cleaned_output=$(echo -e "$output" | sed -e 's/\x1b\[[0-9;]*m//g' \
    -e 's/\\u001b\[[0-9;]*m//g' \
    -e 's/\\n/\n/g' \
    -e 's/\\"//g' \
    -e 's/^"//g' \
    -e 's/"$//g')
echo "Cleaned output. Length: ${#cleaned_output} characters"

# Format the output with heading and code block
formatted_output="$(printf '## %s\n\n```\n%s\n```' "$heading" "$cleaned_output")"
echo "Formatted output. Length: ${#formatted_output} characters"

echo "Posting comment to GitLab MR"
response=$(curl --silent --show-error --fail \
     --request POST \
     --header "PRIVATE-TOKEN: $GITLAB_API_TOKEN" \
     --header "Content-Type: application/json" \
     --data "$(jq -n --arg body "$formatted_output" '{"body": $body}')" \
     "https://code.self_hosted_gitlab.com/api/v4/projects/$CI_PROJECT_ID/merge_requests/$CI_MERGE_REQUEST_IID/notes")


if [ $? -eq 0 ]; then
    echo "Comment successfully posted to GitLab MR"
else
    echo "Error: Failed to post comment to GitLab MR. Response: $response"
    exit 1
fi

echo "Script completed successfully"

Here is the tofu plan output comment:

Here is the tofu apply output comment:

3. Migrating to GitLab’s Remote HTTP Backend

Transitioning your local Tofu state to a remote backend hosted by GitLab can streamline state management and enhance security. Follow these straightforward steps to achieve this.

First, set up the necessary environment variables with your specific details:

export TF_STATE_NAME="example_net"
export TF_HTTP_ADDRESS="https://<link_to_GitLab_instance>>/api/v4/projects/460/terraform/state/$TF_STATE_NAME"
export TF_HTTP_USERNAME="<create_a_user>"
export TF_HTTP_PASSWORD="<create_a_password>"

Initialize the remote backend with Tofu, specifying your new backend configuration:

tofu init -migrate-state \
  -backend-config="address=$TF_HTTP_ADDRESS" \
  -backend-config="username=$TF_HTTP_USERNAME" \
  -backend-config="password=$TF_HTTP_PASSWORD"

Verify the migration by running:
```
tofu plan
```

After these steps, your Tofu environment will be fully integrated with GitLab’s remote HTTP backend, with state management and enhanced collaboration and security.

Automation of our DNS management with GitLab CI/CD and OpenTofu has transformed our operations, making them more efficient, error-resistant, and secure. We encourage other teams to explore similar automation strategies to enhance their infrastructure management processes.

Kubernetes From The Ground Up With AWS EC2

2022-10-06T00:00:00+00:00

Photo by Darry Lin

One way to learn Kubernetes infrastructure is to build it from scratch. This way of learning was introduced by the founding father of Kubernetes himself: Mr. Kelsey Hightower. The lesson is known as “Kubernetes The Hard Way”.

For this blog entry I would like to take a less demanding approach than Kubernetes The Hard Way, while still being educational. I would like to highlight only the major steps in creating a Kubernetes cluster and what is covered in CKA (Certified Kubernetes Administrator) exams. Thus we are going to use the kubeadm tools to build the Kubernetes cluster.

The steps of creating a Kubernetes cluster are hidden to you if you are using a Kubernetes as a service such as AWS EKS, GCP GKE or the enterprise suites of Kubernetes such as Red Hat Openshift or VMware Tanzu. All of these products let you use Kubernetes without the need to worry about creating it.

Prerequisites

For this tutorial we will need the following from AWS:

An active AWS account
EC2 instances with Amazon Linux 2 as the OS
AWS Keys for SSH to access control node and managed nodes
Security group which allows SSH and HTTP
A decent editor such as Vim or Notepad++ to create the inventory and the playbook

EC2 Instances provisioning

Provisioning of the the control plane, a.k.a. the master node:

Go to AWS Console → EC2 → Launch Instances.
Set the Name tag to Master.
Select the Amazon Linux 2 AMI.
Select a key pair. If there are no available key pairs, please create one according to Amazon’s instructions.
Allow SSH and 6443 TCP ports.
Set Number of Instances to 1.
Click Launch Instance.

Provisioning of the worker nodes, a.k.a. the minions:

Go to AWS Console → EC2 → Launch Instances.
Set the Name tag to Node.
Select the Amazon Linux 2 AMI.
Select a key pair. If there are no available key pairs, please create one according to Amazon’s instructions.
Allow SSH TCP port.
Set Number of Instances to 2.
Click Launch Instance.

Installing the container runtime

All Kubernetes nodes require some sort of container runtime engine. For these nodes we are going to use Docker. Log in to all EC2 instances and execute the following:

Install Docker.

sudo yum update -y
sudo amazon-linux-extras install docker -y
sudo usermod -a -G docker ec2-user
sudo service docker start
sudo systemctl enable docker.service
sudo su - ec2-user

Verify the Docker installation.

docker ps

We should get an empty Docker status:

CONTAINER ID   IMAGE          COMMAND                  CREATED       STATUS      PORTS                    NAMES

Install TC (Traffic Controller). This is required by the kubeadm tool.
```
sudo yum install tc -y
```

Kubernetes control plane setup

Add the Kubernetes repository. Log in to the node and paste the following:

cat <<'EOF' | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-$basearch
enabled=1
gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
EOF

Install the Kubernetes binaries for Control Plane (kubelet, kubeadm, kubectl) and enable it.

sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
sudo systemctl enable --now kubelet

Initiate the Control Plane. The --ignore-preflight-errors switch is required because we are using a system which has fewer than 2 CPUs and less than 2 GB of RAM. The --pod-network-cidr value is the default value for flannel (a networking add-on).
```
sudo kubeadm init --ignore-preflight-errors=NumCPU,Mem --pod-network-cidr=10.244.0.0/16
```
There are 3 important points from the output of this command. They are the successful note on the cluster initalization, the kubeconfig setup and the worker node joining string. The following is a sample output:
Create the configuration file for kubectl a.k.a. kubeconfig to connect to the Kubernetes cluster. The scripts are from previous output:
```
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
```

Install the pod network add-on. We are going to use flannel.

kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml

Kubernetes worker nodes setup

Execute the following in all worker nodes:

Add the Kubernetes repository. Log in to the node and paste the following

cat <<'EOF' | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-$basearch
enabled=1
gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm
EOF

Install the Kubernetes binaries for worker nodes (kubelet, kubeadm) and enable kubelet.

sudo yum install -y kubelet kubeadm --disableexcludes=kubernetes
sudo systemctl enable --now kubelet

Execute the join command with sudo. This command is from step #3 in the Kubernetes Control Plane Setup section.

Hello, Kubernetes :)

We have successfully created a Kubernetes cluster. Let’s check on the cluster and try to deploy some sample applications.

Get the latest status of the nodes. You might need to wait a minute or more for all nodes to become Ready.

kubectl get nodes

Sample output:

Deploy a sample Nginx web server

kubectl create deployment mynginx --image=nginx

Scale the Deployment to have 6 replicas and check on where the pods run. The pods should be assigned randomly to the available worker nodes.

kubectl scale --replicas=6 deployment/mynginx
kubectl get pods -o wide

Sample output:

Conclusion

That’s all, folks. I hope this blog entry has shed some insights on what it takes to create a Kubernetes cluster. Have a nice day :)

CI/CD with Azure DevOps

2022-08-28T00:00:00+00:00

Photo by Dylan Wooters, 2022.

A development process that includes manual builds, tests, and deployments can work for a small-scale project, but as your codebase and team grow, it can quickly become time-consuming and unwieldy. This can be particularly true if you’re a .NET developer. We all know the struggle of merging in the latest feature and clicking build in Visual Studio, only to have it fail, citing cryptic errors. “Maybe a clean will fix it?”

If this sounds like your current situation, it’s likely time to consider building a Continuous Integration and Continuous Deployment pipeline, commonly known as “CI/CD”. A good CI/CD pipeline will help you automate the painful areas of building, testing, and deploying your code, as well as help to enforce best practices like pull requests and build verification.

There are many great options to choose from when selecting a CI/CD tool. There are self-hosted options like Jenkins and TeamCity. There are also providers like GitHub and Azure DevOps, which offer CI/CD alongside cloud-hosted source control. All of these options have pros and cons, but if you’re looking for a large feature set, flexibility, and in particular good support for .NET solutions, you should consider Azure DevOps.

In this post, I’ll show you how to set up an Azure CI/CD pipeline for a self-hosted .NET MVC web solution targeting two different environments.

Creating Environments

The first step is to sign up for a free account at Azure DevOps. Once you have your account created, you can then create your Environments. The Environments are the places where you want your app to be deployed.

In the case of a recent project here at End Point Dev, we had three different environments—UAT, Stage, and Production—all running as IIS websites on self-hosted Windows VMs. The UAT environment was on the UAT server, and the Stage and Production environments were on the Production server. So you’ll want to plan out your environments accordingly.

Once you do so, head to Azure DevOps and then click on Environments under the Pipelines section, then click Create Environment.

Then enter a name from the environment and select Virtual Machines and click Next.

In the following window, select Generic Provider, and then select your OS, which in our case is Windows. You will see a registration script with a copy icon next to it. Click the icon to copy the (rather lengthy) PowerShell registration script to the clipboard, and then paste it into a text file.

Next, connect to the target environment. Open a PowerShell window as Administrator, copy and paste the registration script, and then press Enter. The PowerShell script will then do its magic and register the environment with Azure. It may take a minute to run, but afterwards you should see a success message.

Now, if you head back to Azure DevOps and click on the Environment, you should see the server name of the machine that you ran the PowerShell script on. The server name is referred to as a Resource in Azure.

You will then want to complete the above steps for each of your additional target environments, for example, Staging and Production. In our case, Staging and Production are hosted on the same web server, so we only need to create one additional environment (Production) in Azure.

Creating the Pipelines

Now it’s time to create the actual Pipeline. The Pipeline will be responsible for building and deploying your app to the Environments that you created in the previous step. To start, Click on Pipelines in Azure DevOps, and then click the Create Pipeline button.

You’ll then be asked where your source code lives. In our case, the source code exists in a repo in Azure DevOps. The nice thing about Azure is that you can also target source code that exists on another provider, for example, GitHub, Bitbucket, or even a self-hosted Git repo (Other Git). Click on your hosting provider and follow the instructions.

Once you’re connected to your source provider, then you can configure the Pipeline. The “Pipeline” is actually just a YAML file within your target repo. You can read all about the Azure YAML syntax here. We’ll choose a Starter Pipeline, which opens up a text editor and enables you to create your YAML file.

Delete the sample text that appears in the editor, and then enter the following YAML. This is a pre-baked YAML pipeline that restores and builds a .NET web project, and then publishes the build assets to your three different target environments. In the next section, we will take a deep dive into how the YAML works so that you can edit it to fit your needs.

trigger:
- uat
- staging
- main

pool:
  vmImage: 'windows-latest'

variables:
  projectPath: 'ci_tutorial/ci_tutorial_web.csproj'
  packageName: 'ci_tutorial_web.zip'
  solution: '**/*.sln'
  buildPlatform: 'AnyCPU'
  artifactName: 'AzureDrop'
  ${{ if eq(variables['Build.SourceBranchName'], 'uat') }}:
    websiteName: 'ci_tutorial_uat'
    buildConfiguration: 'UAT'
    environmentName: 'UAT'
    targetVM: 'UATWEBAPP1'
  ${{ if eq(variables['Build.SourceBranchName'], 'staging') }}:
    websiteName: 'ci_tutorial_staging'
    buildConfiguration: 'Staging'
    environmentName: 'Production'
    targetVM: 'WEBAPP1'
  ${{ if eq(variables['Build.SourceBranchName'], 'main') }}:
    websiteName: 'ci_tutorial_prod'
    buildConfiguration: 'Release'
    environmentName: 'Production'
    targetVM: 'WEBAPP1'

stages:
- stage: build
  jobs:
  - job: RestoreAndBuild
    steps:
    - task: NuGetToolInstaller@1
    - task: NuGetCommand@2
      inputs:
        restoreSolution: '$(solution)'
    - task: VSBuild@1
      inputs:
        solution: '$(projectPath)'
        msbuildArgs: '/p:DeployOnBuild=true /p:WebPublishMethod=Package /p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true /p:PackageLocation="$(build.artifactStagingDirectory)"'
        platform: '$(buildPlatform)'
        configuration: '$(buildConfiguration)'
    #- task: VSTest@2
    #  inputs:
    #    platform: '$(buildPlatform)'
    #    configuration: '$(buildConfiguration)'
    - task: PublishBuildArtifacts@1
      inputs:
        PathtoPublish: '$(Build.ArtifactStagingDirectory)'
        ArtifactName: '$(artifactName)'
        publishLocation: 'Container'

- stage: Deploy
  displayName: Deploy to IIS
  dependsOn: Build
  jobs:
  - deployment: DeploytoIIS
    displayName: Deploy the web application to dev environment
    environment:
      name: ${{ variables.environmentName }}
      resourceName: ${{ variables.targetVM }}
      resourceType: VirtualMachine
    strategy:
      runOnce:
        deploy:
          steps:
          - task: DownloadBuildArtifacts@0
            inputs:
              buildType: 'current'
              downloadType: 'specific'
              downloadPath: '$(System.ArtifactsDirectory)'
          - task: IISWebAppManagementOnMachineGroup@0
            displayName: 'Create App Pool and Website'
            inputs:
              WebsiteName: '$(websiteName)'
              WebsitePhysicalPath: '%SystemDrive%\inetpub\wwwroot\$(websiteName)'
              CreateOrUpdateAppPoolForWebsite: true
              AppPoolNameForWebsite: '$(websiteName)'
          # For testing, to confirm target website
          - task: PowerShell@2
            displayName: Display target websitename
            inputs:
              targetType: 'inline'
              script: 'Write-Host "Target Website Name: $(websiteName)"'
          - task: IISWebAppDeploymentOnMachineGroup@0
            displayName: 'Deploy IIS Website'
            inputs:
              WebSiteName: '$(websiteName)'
              Package: '$(System.ArtifactsDirectory)\$(artifactName)\$(packageName)'

Understanding the YAML

The first section of the YAML, trigger, determines which branches will trigger the pipeline when they receive a push. In our case, we have three branches: uat, staging, and main.

trigger:
- uat
- staging
- main

The variables section defines variables that are used later in the pipeline YAML. An important part of this section is the use of if statements to assign values to certain variables (i.e., environmentName) based on the source branch for the build. This mechanism allows the pipeline to deploy to several different environments using a single YAML file.

Note that the if statements also switch the build configuration. If you have config transforms, this will be crucial for ensuring that the correct config files are deployed to each environment.

${{ if eq(variables['Build.SourceBranchName'], 'uat') }}:
  websiteName: 'ci_tutorial_uat'
  buildConfiguration: 'UAT'
  environmentName: 'UAT'
  targetVM: 'UATWEBAPP1'
${{ if eq(variables['Build.SourceBranchName'], 'staging') }}:
  websiteName: 'ci_tutorial_staging'
  buildConfiguration: 'Staging'
  environmentName: 'Production'
  targetVM: 'WEBAPP1'
${{ if eq(variables['Build.SourceBranchName'], 'main') }}:
  websiteName: 'ci_tutorial_prod'
  buildConfiguration: 'Release'
  environmentName: 'Production'
  targetVM: 'WEBAPP1'

Next, we get into the actual actions, or stages, of the pipeline. This pipeline has two stages: build and deploy. The build stage runs a NuGet restore and then a VS build, and then publishes the build output/artifacts to an Azure cloud storage location. The last bit is something that took me a while to wrap my head around: The build does not occur on the target environment via the runner—it actually occurs up in “Azure land”, on a Windows cloud VM.

The VM on which the build occurs can be defined using the pool: vmImage section. In our case, we are running the build on windows-latest, which currently translates to the brand new Windows Server 2022.

The deploy stage then downloads the build artifacts from Azure and deploys them to IIS on the target machine, using the runner that you installed as part of the environment setup above. Note that the variables defined in the if statements come into play here—this is how the pipeline knows which environment/VM to target.

environment:
  name: ${{ variables.environmentName }}
  resourceName: ${{ variables.targetVM }}
  resourceType: VirtualMachine

The IISWebAppManagementOnMachineGroup task will automatically create an app pool and website in IIS for the application. This feature can be toggled using the CreateOrUpdateAppPoolForWebsite setting. If a website already exists based on the WebsiteName, Azure will simply deploy to that existing app pool and website.

Adding Branch Protection and Build Validation

In order to ensure successful deployments to Production, you will likely want to add in branch protection for your main or master branch, and also consider build validation.

Branch protection allows you to enforce reviews for pull requests into certain branches. Build validation is a feature that pre-compiles your .NET code to ensure that it builds prior to merging a pull request. Both features help to improve code quality and prevent broken deployments.

To add branch protection, go to Repo → Branches → your target branch → right side menu, then select Branch Policies.

To enforce reviews, turn on “Require a minimum number of reviewers”, and adjust the number of reviewers and settings as required.

For build validation, scroll down to the Build Validation section, and click the plus button to add a new policy. Select the target build pipeline, adjust the policy settings as necessary (see below), and then give it a display name. Click save, and the build validation policy will be applied to the branch.

An important note here is that the build validation essentially runs the CI pipeline. In our case, that means it will run the build and the deployment. This is not ideal; we only want it to run the build (NuGet restore and VS build). So, it’s advisable to create a separate pipeline with only the build stage. This can be done easily by copying our example YAML and removing the “Deploy” stage, then saving it as a new pipeline. Then, choose that new pipeline as the target build pipeline in the validation policy.

Now when you open up a PR, it can only be merged once a review has been applied, and the build validation has run successfully.

Monitoring and Troubleshooting the Pipeline

You can see the overall status of your pipeline by clicking on the Pipelines section in the left-side navigation. Green means good!

To dive into more details, click on the pipeline name. This will show all the recent pipeline runs. Then click on a particular run. This will bring up the run details, including a flowchart for each stage.

To dive further, click on a specific stage. You’ll then see all of the tasks in the stage, with actual output details on the right, similar to what you would see in the output window in Visual Studio.

If you encounter an error in your pipeline, the stage details page is a great place to troubleshoot. Azure will show you the task that generated the error by displaying a red error icon next to the task. You can then click on the task and trace the output on the right to determine the exact error message.

Wrapping Up

Hopefully, this post gives you a good overview of CI/CD options for a self-hosted .NET application. Azure is almost infinitely configurable, so there are many other options to explore that are not covered in this post, whether it be within the pipeline YAML, or through other settings like branch policies/protection. If you have any questions, please feel free to leave a comment. Happy automating!

DevOps & Kubernetes engineer job opening

2021-11-04T00:00:00+00:00

We are looking for a full-time, salaried DevOps / Kubernetes engineer to work on cloud hosting projects with our clients and our internal hosting team.

End Point Dev is an Internet technology consulting company based in New York City, with 50 employees serving many clients ranging from small family businesses to large corporations. We are going strong after 26 years in business!

Even before the pandemic most of us worked remotely from home offices. We collaborate using SSH, Git, project tracking tools, Zulip chat, video conferencing, and of course email and phones.

What you will be doing:

Automate, set up, support, and maintain complex containerized applications in the cloud.
Audit and improve security, backups, reliability, monitoring.
Work together with End Point Dev co-workers and our clients’ in-house staff.
Use your desktop operating system of choice: Linux, macOS, or Windows.
Work with open source software and contribute back as opportunity arises.

You’ll need professional experience with:

Production Kubernetes administration on Amazon EKS (2+ years)
Linux and common distributions including Ubuntu, RHEL/CentOS
Public clouds such as AWS, GCP, Azure, Linode, DigitalOcean
Containerization with Docker and/or Podman
IaC tools such as Terraform, CloudFormation, Ansible, Chef, Puppet, Salt
CI/CD pipelines, release management
Git version control, GitHub or GitLab
Scripting and programming with bash, Python, Ruby, Go, etc.
OS fundamentals, networking, and firewalls
HTTP, REST APIs

You have these important work traits:

Strong verbal and written communication skills
An eye for detail
Tenacity in solving problems and focusing on customer needs
A feeling of ownership of your projects
Work both independently and as part of a team

What work here offers:

Collaboration with knowledgeable, friendly, helpful, and diligent co-workers around the world
Freedom from being tied to an office location
Flexible, sane work hours
Paid holidays and vacation
Annual bonus opportunity
(For U.S. employees:) Health insurance subsidy and 401(k) retirement savings plan

Get in touch with us:

~~Please email us an introduction to jobs@endpointdev.com to apply.~~ (This job has been filled.)

Include your location, a resume/CV, your Git repository or LinkedIn URLs, and whatever else may help us get to know you.

We look forward to hearing from you! Direct work seekers only, please—this role is not for agencies or subcontractors.

We are an equal opportunity employer and value diversity at our company. We do not discriminate on the basis of sex/gender, race, religion, color, national origin, sexual orientation, age, marital status, veteran status, or disability status.

Installing Ubuntu 18.04 to a different partition from an existing Ubuntu installation

2020-04-06T00:00:00+00:00

Photo by Patryk Grądys on Unsplash

Our Liquid Galaxy systems are running on Ubuntu 14.04 LTS (Trusty). We decided to upgrade them to Ubuntu 18.04 LTS (Bionic) since Ubuntu 14.04 LTS reached its end of life on April 30, 2019.

Upgrading from Ubuntu 14.04 LTS

The recommended way to upgrade from Ubuntu 14.04 LTS is to first upgrade to 16.04 LTS, then to 18.04 LTS, which will continue to receive support until April 2023:

14.04 LTS → 16.04 LTS → 18.04 LTS

Ubuntu has LTS → LTS upgrades, allowing you to skip intermediate non-LTS releases, but we can’t skip intermediate LTS releases; we have to go via 16.04, unless we want to do a fresh install of 18.04 LTS.

For a little more longevity, we decided to do a fresh install of Ubuntu 18.04 LTS. Not only is this release supported into 2023 but it will offer a direct upgrade route to Ubuntu 20.04 LTS when it’s released in April 2020.

Installing Clean Ubuntu 18.04 LTS from Ubuntu 14.04 LTS

Install debootstrap

The debootstrap utility installs a very minimal Debian system. Debootstrap will install a Debian-based OS into a sub-directory. You don’t need an installation CD for this. However, you need to have access to the corresponding Linux distribution repository (e.g. Debian or Ubuntu).

apt-get update
apt-get -y install debootstrap

Creating a new root partition

Create a logical volume with size 12G and format the filesystem to ext4:

lvcreate -L12G -n ROOT_VG/ROOT_VOLUME
mkfs.ext4 /dev/ROOT_VG/ROOT_VOLUME

Mounting the new root partition

Mount the partition at /mnt/root18. This will be the root (/) of your new system.

mkdir -p /mnt/root18
mount /dev/ROOT_VG/ROOT_VOLUME /mnt/root18

Bootstrapping the new root partition

Debootstrap can download the necessary files directly from the repository. You can substitute any Ubuntu archive mirror for ports.ubuntu.com/ubuntu-ports in the command example below. Mirrors are listed here.

Replace $ARCH below with your architecture: amd64, arm64, armhf, i386, powerpc, ppc64el, or s390x.

debootstrap --arch $ARCH $DISTRO $ROOT_MOUNTPOINT
debootstrap --arch amd64 bionic /mnt/root18

Installing fstab

This just changes the root (/) partition path in the new installation while keeping the /boot partition intact. For example, /dev/mapper/headVG-root / → /dev/mapper/headVG-root18 /. Since device names are not guaranteed to be the same after rebooting or when a new device is connected, we use UUIDs (Universally Unique Identifiers) to refer to partitions in fstab. We don’t need to use UUIDs for logical volumes since their device names won’t change.

OLD_ROOT_PATH="$(awk '$2 == "/" { print $1 }' /etc/fstab)"
sed "s:^${OLD_ROOT_PATH}\s:/dev/mapper/headVG-root18 :" /etc/fstab > /mnt/root18/etc/fstab

Mounting things in the new root partition

Bind /dev to the new location, then mount /sys, /proc, and /dev/pts from your host system to the target system.

mount --bind /dev /mnt/root18/dev
mount -t sysfs none /mnt/root18/sys
mount -t proc none /mnt/root18/proc
mount -t devpts none /mnt/root18/dev/pts

Configuring apt

Debootstrap will have created a very basic /mnt/root18/etc/apt/sources.list that will allow installing additional packages. However, I suggest that you add some additional sources, such as the following, for source packages and security updates:

echo "deb http://us.archive.ubuntu.com/ubuntu bionic main universe
deb-src http://us.archive.ubuntu.com/ubuntu bionic main universe
deb http://security.ubuntu.com/ubuntu bionic-security main universe
deb-src http://security.ubuntu.com/ubuntu bionic-security main universe" > /mnt/root18/etc/apt/sources.list

Make sure to run apt update with chroot after you have made changes to the target system sources list.

Now we’ve got a real Ubuntu system, if a rather small one, on disk. chroot into it to set up the base configurations.

LANG=C.UTF-8 chroot /mnt/root18 /bin/bash

Installing required packages and running chef-client

As we are maintaining most of the Liquid Galaxy configuration and packages with Chef, we need to install chef-client, configure it on the new target system, and run chef-client to complete the setup.

Copy the chef configuration and persistent net udev rules into place:

cp -a /etc/chef /mnt/root18/etc/
cp /etc/udev/rules.d/70-persistent-net.rules /mnt/root18/etc/udev/rules.d/

Install and run chef-client and let it set up our user login:

cat <<EOF | chroot /mnt/root18
apt-get update && apt-get install -y curl wget
curl -L https://omnitruck.chef.io/install.sh | bash -s -- -v 12.5.1
chef-client -E production_trusty -o 'recipe[users]'
EOF

Next, chroot and install the required packages:

cat <<EOF | chroot /mnt/root18
mount /boot
apt-get update && apt-get install -y --no-install-recommends linux-image-generic lvm2 openssh-server ifupdown net-tools
locale-gen en_US.UTF-8
EOF

Set Ubuntu 14.04 to boot default

Create file 42_custom_template_trusty:

#!/bin/sh
# Entry for trusty system
menuentry 'TRUSTY' --class liquid --class gnu-linux --class gnu --class os {
        # Skipped lines
        linux   /trusty/vmlinuz-$trusty_kernel_version root=/dev/mapper/headVG-root ro nomodeset biosdevname=0 modprobe.blacklist=gma500_gfx quiet
        initrd  /trusty/initrd.img-$trusty_kernel_version
}

Create file 42_custom_template_bionic:

#!/bin/sh
# Entry for bionic system
menuentry 'BIONIC' --class liquid --class gnu-linux --class gnu --class os {
        # Skipped lines
        linux   /vmlinuz-$bionic_kernel_version-generic root=/dev/mapper/headVG-root18 ro nomodeset net.ifnames=0 biosdevname=0 modprobe.blacklist=gma500_gfx quiet
        initrd  /initrd.img-$bionic_kernel_version-generic
}

On the Current system (Trusty):

Back up the current Trusty kernel files into /boot/trusty and create a custom menu entry configuration for Ubuntu 14.04 on 42_custom_trusty. Update /etc/default/grub to set Ubuntu 14.04 as the default menu entry and run update-grub to apply it to the current system. This will be used as a fail-safe method to run Trusty again if there is a problem with the new installation.

mkdir -vp /boot/trusty
cp -v /boot/*-generic /boot/trusty/

envsubst '${trusty_kernel_version}' < 42_custom_template_trusty > /etc/grub.d/42_custom_template_trusty
chmod +x /etc/grub.d/42_custom_template_trusty

sed -i 's/GRUB_DEFAULT=.*/GRUB_DEFAULT="TRUSTY"/' /etc/default/grub
update-grub

On the target system (Bionic):

Create the custom menu entry for Ubuntu 14.04 and Ubuntu 18.04 on the target system.

mkdir -p /mnt/root18/etc/grub.d
envsubst '${trusty_kernel_version}' < 42_custom_template_trusty > /mnt/root18/etc/grub.d/42_custom_template_trusty
envsubst '${bionic_kernel_version}' < 42_custom_template_bionic > /mnt/root18/etc/grub.d/42_custom_template_bionic
chmod +x /mnt/root18/etc/grub.d/42_custom_template_{trusty,bionic}

chroot into the target system and update /etc/default/grub to set Ubuntu 14.04 as the default menu entry and run update-grub. This will also update the GRUB configuration to boot Ubuntu 14.04 as default and update the 0th menu entry to Ubuntu 18.04 (Bionic).

cat <<EOF | chroot /mnt/root18
sed -i 's/GRUB_DEFAULT=.*/GRUB_DEFAULT="TRUSTY"/' /etc/default/grub
update-grub
EOF

Boot into Bionic

To boot into Ubuntu 18.04 (Bionic), reboot the system after grub-reboot bionic and test if the bionic system is working as expected.

$ grub-reboot bionic
$ reboot

Reboot and test our new 0th GRUB entry:

$ grub-reboot 0
$ reboot

A normal reboot returns to Ubuntu 14.04 (Trusty) since the default menu entry is still set to Ubuntu 14.04 (Trusty).

Set Ubuntu 18.04 to boot default

To set our new Ubuntu 18.04 installation as the default menu entry, change GRUB_DEFAULT to 0 in /etc/default/grub and run update-grub to apply it. The next reboot will boot into Ubuntu 18.04.

sed -i 's/GRUB_DEFAULT=.*/GRUB_DEFAULT=0/' /etc/default/grub
update-grub

Congratulations! You now have a freshly installed Ubuntu 18.04 system.

Job opening: Linux system administration and DevOps remote engineer

2019-04-18T00:00:00+00:00

Photo by Kevin Horvat on Unsplash

We are looking for a full-time, salaried engineer to work during business hours in UTC-10 to UTC-6 (somewhere between Hawaii Time and Mountain Time) in the fun space where operations and development overlap!

End Point is a 23-year-old Internet technology consulting company based in New York City, with about 50 employees, most working remotely from home offices. We collaborate using SSH, GitLab, GitHub, chat, video conferencing, and good old email and phones.

We serve many development and hosting clients ranging from small family businesses to large corporations.

What you will be doing:

Remotely set up and maintain Linux servers (mostly RHEL/CentOS, Debian, and Ubuntu), with custom web applications
Audit and improve security, backups, reliability, monitoring
Support developer use of major language ecosystems
Automate provisioning with Terraform, Ansible, Chef, Puppet, etc.
Troubleshoot problems with performance, automation, security
Use open source tools and contribute back as opportunity arises
Use your desktop OS of choice: Linux, macOS, Windows

What you bring:

Professional experience with Linux system administration and web application support:

Cloud providers such as DigitalOcean, Linode, AWS, Azure, Google Cloud, Heroku, etc.
Networking
TLS and PKI
DNS
Web servers and HTTP
Databases such as PostgreSQL, MySQL, Solr, Elasticsearch, CouchDB, MongoDB, etc.
Libraries in Ruby gems, PHP PEAR/PECL, Python PyPI, Node.js npm, Perl CPAN, Java/JVM JARs, etc.
Security consciousness, and ideally familiarity with PCI DSS, HIPAA, etc.

And just as important:

Strong verbal and written communication skills
A good remote work environment
An eye for detail
Tenacity in solving problems
Ownership of projects to get things done well
Work both independently and as part of a team
Focus on customer needs
Be part of emergency on-call rotation including weekends
Willingness to shift work time after hours for occasional scheduled work

What work here offers:

Flexible full-time work hours
Paid holidays and vacation
For U.S. employees: health insurance subsidy and 401(k) retirement savings plan
Annual bonus opportunity
Freedom from being tied to an office location
Collaborate with knowledgeable, friendly, and diligent co-workers around the world

Getting in touch with us:

~~Please email us an introduction to jobs@endpointdev.com to apply.~~ (This job has been filled.) Include your location, a resume/CV, your GitHub or LinkedIn URLs, or whatever else helps us get to know you.

We look forward to hearing from you! Full-time employment seekers only, please—this role is not for agencies or subcontractors.

SRV DNS records in Terraform and Cloudflare

2018-06-26T00:00:00+00:00

(Photo by David Goehring, CC BY 2.0, cropped)

At End Point we are using Terraform for a few clients to manage their web hosting infrastructure as code (IaC). Terraform is particularly helpful when working with multiple cloud or infrastructure providers and stitching together their services.

For example, for one web application that involves failover from the primary production infrastructure to a secondary location at a different provider, we are using Cloudflare as a CDN to provide caching, DDoS mitigation, and traffic routing in front of virtual servers at DigitalOcean and Amazon Web Services (AWS).

We decided we wanted to store all of their infrastructure configuration in Terraform, not just what is required for the web application, so we can recreate their entire infrastructure from their Git repository.

This all went fine until we got to their email DNS records. Our client is using Microsoft Office 365 for their email, which requires some SRV records. Terraform’s Cloudflare provider works fine with the universal MX records, but when we first wanted to do this, the Terraform provider for Cloudflare did not support SRV records at all.

Luckily for us, Terraform recently (6 April 2018) gained support for DNS SRV records as mentioned in the release notes and described in more detail in the pull request that added the feature.

Great! So now we can get on with this.

I began by naively assuming that the SRV record data should be given in space-separated form like many DNS interfaces use, including BIND and Cloudflare’s web interface itself. I tried setting it like this:

resource "cloudflare_record" "_sipfederationtls_tcp" {
  domain = "${var.domain}"
  name   = "_sip._tcp.${var.subdomain}"
  type   = "SRV"
  value  = "100 1 443 sipdir.online.lync.com."
}

But that resulted in an error. So when in doubt, consult the documentation, right? I did that:

Those make it clear that a data element is required, but give no indication what format is needed.

This is not currently documented for Cloudflare’s API, nor for Terraform’s Cloudflare provider. User EpiqSty helpfully recorded asking Cloudflare’s support, who recommended reverse-engineering the format:

In the meantime, what I would suggest is you can create a SRV record via our UI and then you can use the API to do a GET request on the DNS records, that should show you all the fields that are required.

Ok, then!

It turns out that we must provide the component parts of the SRV record disassembled in key/value pairs as the Cloudflare API expects, which looks like this in Terraform config:

resource "cloudflare_record" "_sip_tls" {
  domain = "${var.domain}"
  name   = "_sip._tls.${var.subdomain}"
  type   = "SRV"
  data   = {
    service  = "_sip"
    proto    = "_tls"
    name     = "${var.subdomain}."
    priority = 100
    weight   = 1
    port     = 443
    target   = "sipdir.online.lync.com."
  }
}

One unexpected aspect of this is that it requires duplicating the parts that make up the name although the name doesn’t seem to be used by Cloudflare, which assembles the name from the data components service, proto, and name. The name is apparently just there for Terraform, unlike other DNS records where the name is used as the DNS record value.

The Terraform Cloudflare provider also gained support for the much more obscure DNS LOC record type at the same time, and it works similarly.

Thanks to Ben Vickers who contributed the new feature to Terraform!

Disaster Recovery — Miami to Dallas in one hit

2017-10-03T00:00:00+00:00

Hurricane Irma came a knocking but didn’t blow Miami away.

Hurricanes are never fun, and neither is knowing that your business servers may be in the direct path of a major outage due to a natural disaster.

Recently we helped a client prepare a disaster recovery environment, 4 days out pending Hurricane Irma approaching Miami—where it happens their entire server ecosystem sat. In recent months we had been discussing a long-term disaster recovery plan for this client’s infrastructure, but the final details hadn’t been worked out yet, and no work begun on it, when the news of the impending storm started to arrive.

Although the Miami datacenter they are using is highly regarded and well-rated, the client had the foresight to think about an emergency project to replicate their entire server ecosystem out of Miami to somewhere a little safer.

Fire suits on, battle helmets ready, GO! Two team members jumped in and did an initial review of the ecosystem: six Linux servers, one Microsoft SQL Server database with about 50 GB of data, and several little minions. All of this hosted on a KVM virtualization platform. OK, easy, right? Export the VM disks and stand up in a new datacentre on a similar KVM-based host.

But no downtime could be scheduled at that time, and cloning the database would take too much time to make a replica. If we can’t clone the VM disks without downtime we need another plan. Enter our save-the-day idea: use the database clone from the development machine.

So first things first: Build the servers in the new environment using the cloned copy, rsync the entire disk across, hoping that a restart would return the machine into a working state, take the development database and use the Microsoft tools at our disposal to replicate one current snapshot, then replicate row level details, in preparation of reducing the amount of lost data, should the datacenter lose connection.

Over the course of the next 16 hours, with 2 DevOps engineers, we replicated 6 servers, migrated 100+ GB of data twice and had a complete environment ready for a manual cutover.

Luckily the hurricane didn’t cause any damage to the datacenter, which never lost power or connectivity, so the sites never went down.

Even though it wasn’t used in this case, this emergency work did have a silver lining: Until this client’s longer-term disaster recovery environment is built, these system replicas can serve as cold standby, which is much better than having to rebuild from backups if their primary infrastructure goes out.

That basic phase is out of the way, so the client can breathe easier. Along the way, we produced some updated documentation, and gained deeper knowledge of the client’s software ecosystem. That’s a win!

The key takeaways here: Don’t panic, and do a little planning prior. Leaving things to the last minute rarely works well. And preferable to all of the above: Plan and build your long-term disaster recovery environment now, well before any possible disaster is on the horizon!

To all the people affected by Hurricane Irma we wish a speedy rebuild.

Separate and Not Equal: Development Environments to Support People, Process, and Automation

2017-01-18T00:00:00+00:00

I believe in the separation of software environments for the long term peace and prosperity of developers, product owners, and users. Ideally, there’s a dedicated environment purpose-built to support each community: the developers, the testers, the acceptors, and the users.

I learned this process from taking part in software development projects over the course of many years and working with a variety of different teams, including my own dev team, our clients’ in-house IT teams, and often other consulting firms. While each group had its own approach, one consistent factor for success was the separation of environments to support key roles.

Dev

Development environments will vary by project and technology stack. A common decision is developing locally or creating a remote VM, and both approaches have trade-offs. Developing remotely allows you to setup an environment that mirrors QA and Production, and it also allows you to use remote desktop to connect to your dev environment from any machine: PC, Mac, or Linux. However, developing remotely comes with latency, and often it doesn’t feel as smooth as working locally. More importantly, if I find myself traveling or otherwise without a stable internet connection, developing remotely can be challenging if not impossible.

The setup of the dev environment should be quick and painless, and the project should have clear documentation on how to get started developing. We often keep this in a Wiki in the project’s Git repository. Pairing this documentation with a pre-built dev environment is the ultimate way to get new developers up-and-running. This can be done by packaging up a VM, which can then be run locally or remotely using virtualization software like VirtualBox or a cloud platform like AWS. Vagrant is another option worth checking out, as it touts a one-command setup of a complete dev environment.

QA

The QA environment serves two purposes: as a place where various team members’ code can be built and run together, and a place where the specific task of QA can be performed. When you move to QA testing, it’s a good idea to create a separate branch in source control.This allows developers to push to a common QA branch, which can then be deployed to the QA environment, ideally using a CI tool like TeamCity.

Once the QA testers have signed off, you can move into the UAT/Staging environment. It’s crucial that this environment match production as closely as possible, to rule out any “environmental bugs” that may creep in after deployment to Production. This means ensuring that all the framework versions match (.NET, Node, etc.), system resources are similar, and that the same security is in place (firewalls, VPNs). The data should also replicate what will be found in production, if not outright synced from production using a tool like SQL Delta.

Stage

In staging, the software is put before a specially designated group of users for a dress rehearsal.

If I’m developing a desktop or mobile application (not a web app), I ensure that testing on a variety of different OS versions is incorporated into the UAT process. For example, I use Apple’s TestFlight to test across a variety of devices using different iOS versions.

Prod

Production is the main stage, and code should obviously only be pushed there after proper testing. Even if I have customers, clients, and/or bosses yelling at me, I try and resist the urge to push a fix directly to Production! Although the fix may have worked perfectly in my dev environment, it might not work in production, and then I can easily find myself in even deeper trouble than before. If there is not time to coordinate UAT, then at the very least I push to QA, where the build can be verified in a separate environment, and another set of eyes can perform some testing. It goes without saying that all of this can be made less frightful with unit testing, a topic that we will explore later.

It’s also important to ensure that production has adequate resources before going live. If I’m building a new version or replacement of existing software, I review the existing reporting/analytics to know things like concurrent user count and storage requirements per user. This may seem obvious, but in the rush to get software out to market, basic things can be overlooked. Also, I ensure that I have some baseline security in place on production. This means encrypting sensitive data in config files, not storing passwords (just salted hashes!), and if I’m using a cloud provider like AWS or Azure, I add two-factor authentication to the admin account.

Perl Dancer Conference 2016 Day 1

2016-11-30T00:00:00+00:00

Perl Dancer Conference Day 1

The Perl Dancer Conference is a great event, now in its third year. The event took place in the same location as last year in Vienna, Austria at the Hotel Schani Wien. For those of you who have never visited Vienna, it is a perfect place to bring the family. From visiting the beautiful parks to taking a scenic ride on the Danube River, the beautiful and historic city is known for its rich art and musical culture, and has much to offer.

I was very excited to not only attend but also give a talk this year. My talk titled “Dancing in the Clouds” also coincided with the release of 2 new Perl modules Etcd3 and Dancer2::Plugin::Etcd. This article will be the first of a 3 part series, with the final article a focus on my talk and usage examples with the new modules.

Sawyer X (Sawyer X) — A bus tour through Dancer core

The Captain of Dancer core, SawyerX, took us on a bus tour through the core functionality of Dancer2. Using practical examples of code blocks from core, he explained how different areas of the code base worked. I personally enjoyed his explanation of how hooks are implemented and created. Learning from the 1st iteration of Dancer, the second version shows maturity and stability.

Stefan Hornburg (Racke) — No Act on ACT

If you have ever taken the time to review a Perl conference’s website or even purchase tickets to attend you have no doubt been in contact with ACT. “Act (A Conference Toolkit) is a multilingual, template-driven, multi-conference website that can manage the users, talks, schedule, and payment for your conference.” While this package has been around for many years, it is somewhat dreaded because of its lack of features.

Stefan outlines his work with Interchange6::Schema and the perl.dance website painting a picture of the replacement for ACT. Utilizing Dancer2, DBIx::Class, Moo and other modern Perl tools the infrastructure outlined is very compelling. The package has a user admin, e-commerce, and even a module to print out the passes. Although he notes that this is not a plug and play replacement for ACT yet, with a bit of work and support, it could be the future of Perl conference management.

Andrew Beverly — Implementing i18n in a Dancer application using Plugin::LogReport

Andrew extended his talk last year about internationalization with the Dancer2::Plugin::LogReport module. Using great examples, he not only outlined the usage of the module, but also got into details of how the process works on a technical level. Explaining the different ways that internationalization can be done, he begins to outline how he achieved his task of deploying i18n in a Dancer app.

Theo van Hoesel — Quickstep

Theo was a great addition to the conference this year. He was able to make the event on very short notice after Dancer core Jason Crome was not able to attend due to injury. Theo outlined the Act::Voyager project briefly and the general requirements of adding user friendly features to the ACT toolkit. He also spent a good amount of time explaining the concept of web caching and how many of the existing modules failed in the task of complying with RFC7234. He then explained how all of this brought him to create HTTP::Caching and how it has “The RFC 7234 compliant brains to do caching right”. Along with this contribution part of the HTTP::Bundle, his Dancer focused Dancer2::Plugin::HTTP::Bundle was explained.

Job van Achterberg (jkva) — Dancing with Disabilities

Job’s talk was a very interesting look into how taking a considerate approach to development and small changes to your code can improve a disabled web user’s experience. Using the tools in macOS Job showed how simple things such as naming a list are reflected in a disabled users ability to get information. What I found very interesting in this presentation was how awkward the tools were to use even for an experienced pro like Job. It really made me think a lot about the challenges the disabled face in something many of us take for granted.

Jason Lewis — The Lazy Programmer’s Guide to building HTML tables in Dancer2

Jason has become a regular on the #dancer freenode IRC channel. This year he decided to travel all the way from Australia to give his presentation on his experiences replacing Crystal Reports with Dancer2 Template::Toolkit and DataTables. Although a great deal of the presentation was focused on the features of the jQuery plugin DataTables, he gave nice examples of code he used to populate reports and the hurdles he faced replacing Crystal Reports functionality. The reports looked beautiful and were very easy to convert to other data types such as PDF and CSV.

Stefan Seifert (nine) — Perl 5 and Perl 6 — a great team

Stefan is a great presence at the conference, and his fun and witty personality carried over to his presentation. After opening with a really funny skit as a reporter reading the the news, he continued to outline the current state of Perl6 and how important it is for all of us as a community to embrace the fact that we are all in this together. He reminded us of the perils of Python 3’s launch and the lack of support even today. He then began to focus on the capabilities of using Perl 5 with Perl 6 together with Inline::Perl5 and Inline::Perl6 modules. To be honest before his talk I had given Perl 6 very little time. Stefan’s talk opened my eyes to the possibilities of utilizing the two versions together and the advantages that ecosystem has.

Please stop back for links to day 2 of the conference and a breakdown of my talk outlining etcd integration with Perl and Dancer2.

8 Simple Steps to Saner Software Development

2016-11-10T00:00:00+00:00

While these might seem obvious for seasoned developers, many projects, especially legacy software, still fail to follow most of the steps below.

In future posts, I’ll expand on these steps and give more details. For now, here’s an overview of some basic guidelines for making the process of software development smoother (& saner) for both the client and the dev team.

Separate environments:

Ideally there should be Dev, QA, UAT/Staging, and Production environments. And, UAT/Staging should be as close to Production as possible. It’s amazing how many software projects are still done on someone’s local machine and pushed directly to Production.

Use a bug tracking tool:

This is pretty obvious. Bugs need to be logged and labeled so that they can be tracked through stages of development (i.e., To-Do, In Progress, Completed). Using a bug tracking tool like Jira or GitHub also helps the non-technical stakeholders comment on and clarify requirements/issues.

Have a source control strategy:

It doesn’t have to be fancy. For example, if you’re using Git, it should be more than having multiple developers working out of the master branch. Ideally link branches to features or bugs defined in a bug tracking tool, and keep in mind the separation of environments, especially if the application is already in Production.

Do pull/merge requests:

Even a short five minute review will improve the overall quality of code, reduce bugs, and communicate changes between developers responsible for different parts of the application. Web-based source control systems like GitLab, GitHub, and Bitbucket make this simple.

Write unit tests:

A lot of developers either feel pressure to move quickly or get lazy and don’t write tests. Regression testing becomes a nightmare without unit tests. Plus it makes you feel better to see all those tests passing.

Use a continuous integration tool:

TeamCity, Jenkins, or even basic GitHooks are some options. Automated deployments save time and, if integrated with unit tests, reduce the bugs that make it to Production.

Setup notifications:

If the CI tool alerts team members of deployment status, hundreds of “is the deployment done yet?” emails and chat messages can be saved. Slack integration is a good option here.

Consolidate project credentials:

Have a secure and centralized location for managing passwords. We like keeping a KeePass checked into the relevant project’s repo.

Building Containers with Habitat

2016-10-17T00:00:00+00:00

Many Containers, Many Build Systems

When working with modern container systems like Docker, Kubernetes, and Mesosphere, each provide methods for building your applications into their containers. However each build process is specific to that container system, and using similar applications across tiers of container environments would require maintaining each container’s build environment. When approaching this problem for multiple container environments, Chef Software created a tool to unify these build systems and create container-agnostic builds which could be exported into any of the containers. This tool is called Habitat which also provide some pre-built images to get applications started quickly.

I recently attended a Habitat Hack event locally in Portland (Oregon) which helped me get more familiar with the system and its capabilities. We worked together in teams to take a deeper dive into various aspects of how Habitat works, you can read about our adventures over on the Chef blog.

To examine how the various parts of the build environment work, I picked an example Node.js application from the Habitat Documentation to build and customize.

Node.js Application into a Docker Container

For the most basic Habitat build, you must define a plan.sh file which will contain all the build process logic as well as all configuration values to define the application. Within my Node.js example, this file contains this content:

pkg_origin=daehlie
pkg_name=mytutorialapp
pkg_version=0.3.0
pkg_maintainer="Kirk Harr <kharr@endpoint.com>"
pkg_license=()
pkg_source=https://s3-us-west-2.amazonaws.com/${pkg_name}/${pkg_name}-${pkg_version}.tar.gz
pkg_shasum=e4e988d9216775a4efa4f4304595d7ff31bdc0276d5b7198ad6166e13630aaa9
pkg_filename=${pkg_name}-${pkg_version}.tar.gz
pkg_deps=(core/node)
pkg_expose=(8080)

do_build() {
  npm install
}

do_install() {
  cp package.json ${pkg_prefix}
  cp server.js ${pkg_prefix}

  mkdir -p ${pkg_prefix}/node_modules/
  cp -vr node_modules/* ${pkg_prefix}/node_modules/
}

Within this is defined all the application details like the name of the author, the version of the application being packaged, as well as the package name. Each package can be defined with a license for the code in use as well as any code dependencies, like the Node.js application server (core/node), as well as the repository URL for locating these files. There are also two executable statements which build the package dependencies, and perform final installation setup during the eventual package installation.

Additionally to define this application we must provide the logic for how to start the Node application server, and provide configuration on what ports to listen on as well as the message to be displayed once it has started. To do so we must create a stub Node.js config.json which provides the port and message values:

{
    "message": "Hello, World",
    "port": "8080"
}

We also need two hooks which will be executed at package install time and at runtime for the application respectively. These are named, init and run in our case, with init setting up the symbolic links to the various Node.js components from the core/node package which will be included in the build, and run provides the entry point for the applications flow effectively starting the npm application server. Just like with a Dockerfile, any additional logic needed during the process would be included in these two hooks, depending on if the logic was specific to install time or run time.

Injected Configuration Values

In this example, both the message to be displayed to the user, as well as the port that the Node.js application server will listen on are hard-coded into our build, and all the images that resulted from it would be identical. In order to allow for some customizing of the resulting image, you can replace the hard-coded values in the Node.js config.json into variables which can be replaced during the build process:

{
    "message": "{{cfg.message}}",
    "port": "{{cfg.port}}"
}

To complete the replacement, we would provide a “Tom’s Obvious, Minimal Language” (.toml) file with has a key-value pair for each of these configuration variables we want to set. This .toml file will be interpreted during each build to populate these values, creating an opportunity to customize our builds by injecting specific values into the variables defined in the application configuration. Here is an example of the syntax from this example:

# Message of the Day
message = "Hello, World of Habitat"

# The port number that is listening for requests.
port = 8080

Conclusions

Habitat seeks to fill in the gaps between the various container formats for Docker, Kubernetes and others, by allowing common build infrastructure and dependency libraries to be unified in distribution. By utilizing the same build infrastructure, it becomes more feasible to have a hybrid environment with various container formats in use, without creating duplicate build infrastructure which basically performs the same task slightly differently right at the end to package the application into the proper container format. Habitat helps to decouple the actual build process and all that plumbing, from the process of exporting the build image into the proper format for whatever container is in use. In that way as new container formats are developed, all that is required to accommodate them is expanding the export function for that new format, without any changes to the overall build process or customization of your code.

DevOpsDays India — 2015

2015-09-30T00:00:00+00:00

DevOpsIndia 2015 was held at The Royal Orchid in Bengaluru on Sep 12-13, 2015. After saying hello to a few familiar faces who I often see at the conferences, I collected some goodies and entered into the hall. Everything was set up for the talks. Niranjan Paranjape, one of the organizers, was giving the introduction and overview of the conference.

Justin Arbuckle from Chef gave a wonderful keynote talk about the “Hedgehog Concept” and spoke more about the importance of consistency, scale and velocity in software development.

In addition, he quoted “A small team with generalists who have a specialization, deliver far more than a large team of single skilled people.”

A talk on “DevOps of Big Data infrastructure at scale” was given by Rajat Venkatesh from Qubole. He explained the architecture of Qubole Data Service (QDS), which helps to autoscale the Hadoop cluster. In short, scale up happens based on the data from Hadoop Job Tracker about the number of jobs running and time to complete the jobs. Scale down will be done by decommissioning the node, and the server will be chosen by which is reaching the boundary of an hour. This is because most of the cloud service providers charge for an hour regardless of whether the usage is 1 minute or 59 minutes.

Vishal Uderani, a DevOps guy from WebEngage, presented “Automating AWS infrastructure and code deployment using Ansible.” He shared the issues facing the environments like task failure due to ssh timeout on executing a giant task using Ansible and solved by monitoring the task after triggering the task to get out of the system immediately. Integrating Rundeck with Ansible is an alternative for enterprise Ansible Tower. He also stated the following reasons for using Ansible:

Good learning curve
No agents will be running on the client side, which avoids having to monitor the agents at client nodes
Great deployment tool

Vipul Sharma from CodeIgnition stated the importance of Resilience Testing. The application should be tested periodically to be tough and strong enough to handle any kind of load. He said Simian Army can be used to create the problems in environments and then resolving them to make them flawless. Simian Army can used to improve application using security monkey, chaos monkey, janitor monkey, etc… Also “Friday Failure” is a good method to identify the problem and improve the application.

Avi Cavale from Shippable gave an awesome talk on “Modern DevOps with Docker”. He talk started with “What is the first question that arises during an outage ? … What changed ?”After fixing these issues, the next question will be “Who made the change?” Both questions are bad for the business. Change is the root cause of all outage but business requires changes. In his own words, DevOps is a culture of embracing change. Along with that he explained the zero downtime ways to deploy the changes using a container.

He said DevOps is a culture, make it FACT(F-Frictionless, A-Agile, C-Continuous and T-Transparency).

Rahul Mahale from SecureDB gave a demo on Terraform, a tool for build and orchestration in Cloud. It features “Infrastructure as Code” and also provides an option to generate the diagrams and graphs of the present infrastructure.

Shobhit and Sankalp from CodeIgnition shared their experience on solving network based issues. Instead of whitelisting the user’s location every time manually to provide the access to systems, they created a VPN to enable access only to users, not locations. They have resolved two more additional kind of issues by adding Router to bind two networks using FIP. Another issue is that they need to whitelist to access third party services from containers, but it was hard to whitelist all the containers. Therefore, they created and whitelisted VMs and containers accessed the third party services through VMs.

Ankur Trivedi from Xebia Labs spoke about the “Open Containers Initiative” project. He explained the evolution of the containers (Docker—2013 & rkt—2014). The various distributions of containers are compared based on the Packing, Identity, Distribution and Runtime capabilities. Open Containers is supported by the community and various companies who are doing extensive work on containers in order to standardize them.

Vamsee Kanala, a DevOps consultant, presented a talk on “Docker Networking—A Primer”. He spoke about Bridge networking, Host Networking, Mapped container networking and None (Self Managed) with dockers. The communications between the containers can happen through:

Port mapping
Link
Docker composing (programmatically)

In addition, he explained the tools which feature the clustering of containers, and listed the tools that have their own way of clustering and advantages:

Kubernetes
Mesos
Docker Swarm

Aditya Patawari from BrowserStack gave a demo on “Using Kubernetes to Build Fault Tolerant Container Clusters”. Kubernetes has a feature called “Replication Controllers,” which helps to maintain number of pods running at any time. “Kubernetes Services” defines a policy to enable access among the pods which provisions the pods as micro services.

Arjun Shenoy from LinkedIn introduced a tool called “SIMOORG.” The tool was developed at LinkedIn and does the failure induction in a cluster for testing the stability of the code. It is a components-based Open Source framework and few components are replaceable with external ones.

Dharmesh Kakadia, a researcher from Microsoft, gave a wonderful talk on “Mesos is the Linux”. He started with a wonderful explanation on Micro services (relating with linux commands, each command is a micro service) which is simplest, independently updatable, runnable and deployable. Mesos is a “Data Center Kernel” which takes care of scalability, fault tolerance, load balance, etc… in Data Center.

At the end, I got a chance to do some hands-on things on Docker and played with some of its features. It was a wonderful conference to learn more about configuration management and the containers world.

The ‘name’ attribute is required in cookbook metadata: Solving a Vagrant/Chef Provisioning Issue

2015-04-21T00:00:00+00:00

When Vagrant/Chef Provisioning Goes South

I recently ran into the following error when provisioning a new vagrant machine via the vagrant up command:

[2015-04-21T17:10:35+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
[2015-04-21T17:10:35+00:00] ERROR: Cookbook loaded at path(s) [/tmp/vagrant-chef/path/to/my-cookbook] has invalid metadata: The 'name' attribute is required in cookbook metadata
[2015-04-21T17:10:35+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

After some googling and digging I learned version 12 of chef-client introduced a breaking change. From version 12 on, every cookbook requires a name attribute in their metadata.rb file. A quick grep through the metadata.rb files in the project revealed several did not include name attributes. You would be correct at this point to suggest I could have added name attributes to the cookbook metadata files and been done with this. However, in this case I was a consumer of the cookbooks and was not involved in maintaining them so an alternate solution was preferable.

Selecting a Specific Version of Chef in Vagrant

My idea for a solution was to install the most recent chef-client release prior to version 12. I was not sure how to do this initially but along the way I learned that by default, Vagrant will install the most recent release of chef-client. The Vagrant documentation for Chef provisioners described what I needed to do. The Chef version could be specified in config.vm.provision block in the Vagrantfile:

config.vm.provision :chef_solo do |chef|
      chef.version = "11.18"
      chef.cookbooks_path = "cookbooks"
      chef.data_bags_path = "data_bags"

      # List of recipes to run
      chef.add_recipe "vagrant_main::my_project"
  end

With this configuration change, chef-client 11.18 completed the provisioning step successfully.

Testing your chef repo pull requests with chef-zero, Vagrant and Jenkins

2015-02-18T00:00:00+00:00

All Liquid Galaxy setups deployed by End Point are managed by Chef. Typical deployment consists of approx 3 to 9 Linux boxes from which only 1 is managed and the rest is an ISO booted from this machine via network with copy-on-write root filesystem. Because of this, typical deployment involves more steps than just updating your code and restarting application. Deployment + rollback may be even 10 times longer compared with typical web application. Due to this fact, we need to test our infrastructure extensively.

What are we to do in order to make sure that our infrastructure is tested well before it hits production?

Scary? It’s not.

Workflow broken down by pieces

lg_chef.git repo—where we keep cookbooks, environments and node definitions
GitHub pull request—artifact of infrastructure source code tested by Jenkins
Vagrant—virtual environment in which chef is run in order to test the artifact. There’s always 1 master node and few Vagrant boxes that boot an ISO from master via tftp protocol
chef-zero—Chef flavor used to converge and test the infrastructure on the basis of GitHub pull request
chef-server/chef-client—Chef flavor used to converge and test production and pre-production environment
Jenkins — Continuous Integration environment that runs the converge process and part of the tests
Tests—two frameworks used—BATS (for the integration tests on the top) and minitest (for after-converge tests)
lg-live-build—our fork of Debian live build used to build the ISO that is booted by Vagrant slaves

Workflow broken down by the order of actions

User submits GitHub pull request to lg_chef.git repo
GitHub pull request gets picked up by Jenkins
Jenkins creates 1 master Vagrant node and several slave nodes to act as slaves
chef-zero converges master Vagrant box and runs minitest
BATS tests run on the freshly converged Vagrant masterbox. Few steps are performed here: ISO is built, it’s distributed to the slaves, slaves boot the ISO and final integration tests are run to see whether slaves have all the goodness.
If points 1 to 5 are **green,**developer merges the changes, uploads the updated cookbooks, node definitions, roles and environments and runs the final tests.

What didn’t work for us and why

kitchen-vagrant —because it didn’t play well with Jenkins (or JVM itself) and didn’t know how to use advanced Vagrant features for specifying multiple networking options, interfaces and drivers. However it supports using your own Vagrantfile.erb
We’ve had some doubts about keeping all the cookbooks, environments and node definitions in one repo because chef-server/chef-client tests can only test your stuff if it’s uploaded to the Chef server, but chef-zero came in handy

The code

As previously mentioned, we needed our own vagrant template file.

Vagrant.configure("2") do |config|
  <% if  @data[:chef_type] == "chef_zero" %>
  config.chef_zero.enabled = true
  config.chef_zero.chef_server_url = "<%= @data[:chef_server_url] %>"
  config.chef_zero.roles = "../../roles/"
  config.chef_zero.cookbooks = "../../cookbooks/"
  config.chef_zero.environments = "../../environments/"
  config.chef_zero.data_bags = "../integration/default/data_bags/"
  <% else %>
  config.chef_zero.enabled = false
  <% end %>
  config.omnibus.chef_version = "<%= @data[:chef_version] %>"
  config.vm.define "<%= @data[:headnode][:slug] %>" do |h|
  h.vm.box = "<%= @data[:headnode][:box] %>"
    h.vm.box_url = "<%= @data[:headnode][:box_url] %>"
    h.vm.hostname = "<%= @data[:headnode][:hostname] %>"
    h.vm.network(:private_network, {:ip => '10.42.41.1'})
    h.vm.synced_folder ".", "/vagrant", disabled: true
    h.vm.provider :virtualbox do |p|
      <% @data[:headnode][:customizations].each  do |key, value| %>
        p.customize ["modifyvm", :id, "<%= key %>", "<%= value %>"]
      <% end %>
    end
    h.vm.provision :chef_client do |chef|
      <% if @data[:chef_type] == "chef_zero" %>
      chef.environment = "<%= @data[:headnode][:provision][:environment] %>"
      chef.run_list = <%= @data[:run_list] %>
      chef.json = <%= @data[:node_definition] %>
      chef.chef_server_url = "<%= @data[:chef_server_url] %>"
      <% else %>
      chef.chef_server_url = "<%= @data[:headnode][:provision][:chef_server_url] %>"
      <% end  %>
      chef.validation_key_path = "<%= @data[:headnode][:provision][:validation_key_path] %>"
      chef.encrypted_data_bag_secret_key_path = "<%= @data[:headnode][:provision][:encrypted_data_bag_secret_key_path] %>"
      chef.verbose_logging = <%= @data[:headnode][:provision][:verbose_logging] %>
      chef.log_level = "<%= @data[:headnode][:provision][:log_level] %>"
  <% end  %>
  config.omnibus.chef_version = "<%= @data[:chef_version] %>"
  config.vm.define "<%= @data[:headnode][:slug] %>" do |h|
  h.vm.box = "<%= @data[:headnode][:box] %>"
    h.vm.box_url = "<%= @data[:headnode][:box_url] %>"
    h.vm.hostname = "<%= @data[:headnode][:hostname] %>"
    h.vm.network(:private_network, {:ip => '10.42.41.1'})
    h.vm.synced_folder ".", "/vagrant", disabled: true
    h.vm.provider :virtualbox do |p|
      <% @data[:headnode][:customizations].each  do |key, value| %>
        p.customize ["modifyvm", :id, "<%= key %>", "<%= value %>"]
      <% end %>
    end
    h.vm.provision :chef_client do |chef|
      <% if @data[:chef_type] == "chef_zero" %>
      chef.environment = "<%= @data[:headnode][:provision][:environment] %>"
      chef.run_list = <%= @data[:run_list] %>
      chef.json = <%= @data[:node_definition] %>
      chef.chef_server_url = "<%= @data[:chef_server_url] %>"
      <% else %>
      chef.chef_server_url = "<%= @data[:headnode][:provision][:chef_server_url] %>"
      <% end  %>
      chef.validation_key_path = "<%= @data[:headnode][:provision][:validation_key_path] %>"
      chef.encrypted_data_bag_secret_key_path = "<%= @data[:headnode][:provision][:encrypted_data_bag_secret_key_path] %>"
      chef.verbose_logging = <%= @data[:headnode][:provision][:verbose_logging] %>
      chef.log_level = "<%= @data[:headnode][:provision][:log_level] %>"
      chef.node_name = "<%= @data[:headnode][:provision][:node_name] %>"
    end
  end

  #display nodes
  <% @data[:display_nodes][:nodes].each  do |dn| %>
  config.vm.define "<%= dn[:slug] %>" do |dn_config|
    dn_config.vm.box_url = "<%= @data[:display_nodes][:global][:box_url] %>"
    dn_config.vm.hostname = "<%= dn[:hostname] %>"
    dn_config.vm.box = "<%= @data[:display_nodes][:global][:box] %>"
    dn_config.vm.synced_folder ".", "/vagrant", disabled: true
    dn_config.vm.boot_timeout = 1
    dn_config.vm.provider :virtualbox do |p|
    <% @data[:display_nodes][:global][:customizations].each  do |key, value| %>
      p.customize ["modifyvm", :id, "<%= key %>", "<%= value %>"]
    <% end %>
      p.customize ["modifyvm", :id, "--macaddress1", "<%= dn[:mac] %>"]
      p.customize ["createhd", "--filename", "../files/<%= dn[:slug] %>.vmdk", "--size", 80*1024]
      p.customize ["storageattach", :id, "--storagectl", "IDE Controller", "--port", 0, "--device", 0, "--type", "hdd", "--medium", "none"]
      p.customize ["storageattach", :id, "--storagectl", "IDE Controller", "--port", 0, "--device", 0, "--type", "hdd", "--medium", "../files/<%= dn[:slug] %>.vmdk"]
      p.customize ["storagectl", :id, "--name", "SATA Controller", "--add", "sata",  "--controller", "IntelAHCI", "--hostiocache", "on"]
      p.customize ["storageattach", :id, "--storagectl", "SATA Controller", "--port", 1, "--device", 0, "--type", "hdd", "--medium", "../files/ipxe_<%= dn[:slug] %>.vmdk"]
    end
  end
  <% end %>
end

It renders a Vagrant File out of following data:

{
  "description" : "This file is used to generate Vagrantfile and run_test.sh and also run tests. It should contain _all_ data needed to render the templates and run teh tests.",
  "chef_version" : "11.12.4",
  "chef_type" : "chef_zero",
  "vagrant_template_file" : "vagrantfile.erb",
  "run_tests_template_file" : "run_tests.sh.erb",
  "chef_server_url" : "http://192.168.1.2:4000",
  "headnode" :
    {
    "slug" : "projectX-pull-requests",
    "box" : "opscode-ubuntu-14.04",
    "box_url" : "https://opscode-vm-bento.s3.amazonaws.com/vagrant/virtualbox/opscode_ubuntu-14.04_chef-provisionerless.box",
    "hostname" : "lg-head",
    "bats_tests_dir" : "projectX-pr",
    "customizations" : {
      "--memory" : "2048",
      "--cpus": "2",
      "--nic1" : "nat",
      "--nic2": "intnet",
      "--nic3": "none",
      "--nic4" : "none",
      "--nictype1": "Am79C970A",
      "--nictype2": "Am79C970A",
      "--intnet2": "projectX-pull-requests"
    },
    "provision" : {
      "chef_server_url" : "https://chefserver.ourdomain.com:40443",
      "validation_key_path" : "~/.chef/validation.pem",
      "encrypted_data_bag_secret_key_path" : "~/.chef/encrypted_data_bag_secret",
      "node_name" : "lg-head-projectXtest.liquid.glx",
      "environment" : "pull_requests",
      "verbose_logging" : true,
      "log_level" : "info"
    }
  },
  "display_nodes" : {
    "global" : {
      "box" : "opscode-ubuntu-14.04",
      "box_url" : "https://opscode-vm-bento.s3.amazonaws.com/vagrant/virtualbox/opscode_ubuntu-14.04_chef-provisionerless.box",
      "customizations" : {
        "--memory" : "2048",
        "--cpus" : "1",
        "--boot1" : "floppy",
        "--boot2" : "net",
        "--boot3" : "none",
        "--boot4" : "none"
    },
    "provision" : {
      "chef_server_url" : "https://chefserver.ourdomain.com:40443",
      "validation_key_path" : "~/.chef/validation.pem",
      "encrypted_data_bag_secret_key_path" : "~/.chef/encrypted_data_bag_secret",
      "node_name" : "lg-head-projectXtest.liquid.glx",
      "environment" : "pull_requests",
      "verbose_logging" : true,
      "log_level" : "info"
    }
  },
  "display_nodes" : {
    "global" : {
      "box" : "opscode-ubuntu-14.04",
      "box_url" : "https://opscode-vm-bento.s3.amazonaws.com/vagrant/virtualbox/opscode_ubuntu-14.04_chef-provisionerless.box",
      "customizations" : {
        "--memory" : "2048",
        "--cpus" : "1",
        "--boot1" : "floppy",
        "--boot2" : "net",
        "--boot3" : "none",
        "--boot4" : "none",
        "--intnet1" : "projectX-pull-requests",
        "--nicpromisc1": "allow-all",
        "--nic1" : "intnet",
        "--nic2": "none",
        "--nic3": "none",
        "--nic4": "none",
        "--nictype1": "Am79C970A",
        "--ioapic": "on"
      }
    },
    "nodes" : [
      {
      "slug" : "projectX-pull-requests-kiosk",
      "hostname" : "kiosk",
      "mac" : "5ca1ab1e0001"
    },
    {
      "slug" : "projectX-pull-requests-display",
      "hostname" : "display",
      "mac" : "5ca1ab1e0002"
    }
    ]
  }
}

As a result we get an on-the-fly Vagrantfile that’s used during the testing:

Vagrant.configure("2") do |config|

  config.chef_zero.enabled = true
  config.chef_zero.chef_server_url = "http://192.168.1.2:4000"
  config.chef_zero.roles = "../../roles/"
  config.chef_zero.cookbooks = "../../cookbooks/"
  config.chef_zero.environments = "../../environments/"
  config.chef_zero.data_bags = "../integration/default/data_bags/"

  config.omnibus.chef_version = "11.12.4"
  config.vm.define "projectX-pull-requests" do |h|
  h.vm.box = "opscode-ubuntu-14.04"
    h.vm.box_url = "https://opscode-vm-bento.s3.amazonaws.com/vagrant/virtualbox/opscode_ubuntu-14.04_chef-provisionerless.box"
    h.vm.hostname = "lg-head"
    h.vm.network(:private_network, {:ip => '10.42.41.1'})
    h.vm.synced_folder ".", "/vagrant", disabled: true
    h.vm.provider :virtualbox do |p|
        p.customize ["modifyvm", :id, "--memory", "2048"]
        p.customize ["modifyvm", :id, "--cpus", "2"]
        p.customize ["modifyvm", :id, "--nic1", "nat"]
        p.customize ["modifyvm", :id, "--nic2", "intnet"]
        p.customize ["modifyvm", :id, "--nic3", "none"]
        p.customize ["modifyvm", :id, "--nic4", "none"]
        p.customize ["modifyvm", :id, "--nictype1", "Am79C970A"]
        p.customize ["modifyvm", :id, "--nictype2", "Am79C970A"]
        p.customize ["modifyvm", :id, "--intnet2", "projectX-pull-requests"]
    end
    h.vm.provision :chef_client do |chef|

      chef.environment = "pull_requests"
      chef.run_list = ["role[lg-head-nocms]", "recipe[lg_live_build]", "recipe[lg_tftproot]", "recipe[lg_projectX]", "recipe[lg_test]", "recipe[test_mode::bats]", "recipe[hostsfile::projectX]"]
      chef.json = {:sysctl=>{:params=>{:vm=>{:swappiness=>20}, :net=>{:ipv4=>{:ip_forward=>1}}}}, :test_mode=>true, :call_ep=>{:keyname=>"ProjectX CI Production", :fwdport=>33299}, :tftproot=>{:managed=>true}, :tags=>[], :lg_cms=>{:remote=>"github"}, :monitor=>true, :lg_grub=>{:cmdline=>"nomodeset biosdevname=0"}, :projectX=>{:repo_branch=>"development", :display_host=>"42-a", :kiosk_host=>"42-b", :sensors_host=>"42-b", :maps_url=>"https://www.google.com/maps/@8.135687,-75.0973243,17856994a,40.4y,1.23h/data=!3m1!1e3?esrch=Tactile::TactileAcme,Tactile::ImmersiveModeEnabled"}, :liquid_galaxy=>{:touchscreen_link=>"/dev/input/lg_active_touch", :screenshotd=>{:screen_rows=>"1", :screen_columns=>"1"}, :screenshot_service=>true, :display_nodes=>[{:hostname=>"42-c"}, {:allowed_pages=>["Google Maps", "Pacman Doodle", "jellyfish", "Doodle Selection", "ProjectX Video Player", "Composer kiosk"], :hostname=>"42-b", :mac=>"5c:a1:ab:1e:00:02", :features=>"mandatory_windows, plain_gray, starry_skies", :bad_windows_names=>"Google Earth - Login Status", :mandatory_windows_names=>"awesome", :screens=>[{:display=>":0", :crtc=>"default", :grid_order=>"0"}], :screen_rotation=>"normal", :audio_device=>"{type hw; card DGX; device 0}", :onboard_enable=>true, :keyboard_enable=>true, :mouse_enable=>true, :cursor_enable=>true, :background_extension=>"jpg", :background_mode=>"zoom-fill", :projectX=>{:extensions=>{:kiosk=>"ProjectX Kiosk", :google_properties_menu=>"Google Properties Menu", :onboard=>"Onboard", :no_right_click=>"Right Click Killer", :render_statistics=>"Render Statistics"}, :browser_slug=>"lgS0", :urls=>"https://www.google.com/maps", :ros_nodes=>[{:name=>"rfreceiver_reset", :pkg=>"rfreceiver", :type=>"kill_browser.py"}, {:name=>"proximity", :pkg=>"maxbotix", :type=>"sender.py"}, {:name=>"spacenav", :pkg=>"spacenav_node", :type=>"spacenav_node"}, {:name=>"leap", :pkg=>"leap_motion", :type=>"sender.py"}, {:name=>"projectX_nav", :pkg=>"projectX_nav", :type=>"projectX_nav"}, {:name=>"onboard", :pkg=>"onboard", :type=>"listener.py"}, {:name=>"rosbridge", :pkg=>"rosbridge_server", :type=>"rosbridge_websocket", :params=>[{:name=>"certfile", :value=>"/home/lg/etc/ros.crt"}, {:name=>"keyfile", :value=>"/home/lg/etc/ros.key"}]}]}, :browser_infinite_url=>"http://lg-head/projectX-loader.html"}, {:hostname=>"42-a", :mac=>"5c:a1:ab:1e:00:01", :features=>"mandatory_windows, plain_gray, starry_skies, erroneous_text", :bad_windows_names=>"Google Earth - Login Status", :mandatory_windows_names=>"awesome", :screens=>[{:display=>":0", :crtc=>"default", :grid_order=>"1"}], :keyboard_enable=>true, :mouse_enable=>true, :cursor_enable=>true, :background_extension=>"jpg", :background_mode=>"zoom-fill", :nvidia_mosaic=>true, :manual_layout=>{:default=>"1024x768+0+0"}, :projectX=>{:extensions=>{:display=>"ProjectX Large Display", :pacman=>"pacman", :render_statistics=>"Render Statistics"}, :browser_slug=>"lgS0", :urls=>"https://www.google.com/maps", :ros_nodes=>[{:name=>"geodata", :pkg=>"geodata", :type=>"geodata_server.py"}]}, :browser_infinite_url=>"http://lg-head/projectX-loader.html", :default_browser_bin=>"google-chrome", :allowed_pages=>["Google Maps", "Pacman Doodle", "jellyfish", "ProjectX Video Player", "Composer wall"]}], :has_cec=>false, :google_office=>false, :viewsync_master=>"42-b", :has_touchscreen=>false, :has_spacenav=>true, :support_name=>"projectX-ci", :podium_interface=>"http://lg-head", :podium_display=>"42-b:default"}}
      chef.chef_server_url = "http://192.168.1.2:4000"

      chef.validation_key_path = "~/.chef/validation.pem"
      chef.encrypted_data_bag_secret_key_path = "~/.chef/encrypted_data_bag_secret"
      chef.verbose_logging = true
      chef.log_level = "info"
      chef.node_name = "lg-head-projectXtest.liquid.glx"
    end
  end

  #display nodes

  config.vm.define "projectX-pull-requests-kiosk" do |dn_config|
    dn_config.vm.box_url = "https://opscode-vm-bento.s3.amazonaws.com/vagrant/virtualbox/opscode_ubuntu-14.04_chef-provisionerless.box"
    dn_config.vm.hostname = "kiosk"
    dn_config.vm.box = "opscode-ubuntu-14.04"
    dn_config.vm.synced_folder ".", "/vagrant", disabled: true
    dn_config.vm.boot_timeout = 1
    dn_config.vm.provider :virtualbox do |p|
      p.customize ["modifyvm", :id, "--memory", "2048"]
      p.customize ["modifyvm", :id, "--cpus", "1"]
      p.customize ["modifyvm", :id, "--boot1", "floppy"]
      p.customize ["modifyvm", :id, "--boot2", "net"]
      p.customize ["modifyvm", :id, "--boot3", "none"]
      p.customize ["modifyvm", :id, "--boot4", "none"]
      p.customize ["modifyvm", :id, "--intnet1", "projectX-pull-requests"]
      p.customize ["modifyvm", :id, "--nicpromisc1", "allow-all"]
      p.customize ["modifyvm", :id, "--nic1", "intnet"]
      p.customize ["modifyvm", :id, "--nic2", "none"]
      p.customize ["modifyvm", :id, "--nic3", "none"]
      p.customize ["modifyvm", :id, "--nic4", "none"]
      p.customize ["modifyvm", :id, "--nictype1", "Am79C970A"]
      p.customize ["modifyvm", :id, "--ioapic", "on"]
      p.customize ["modifyvm", :id, "--macaddress1", "5ca1ab1e0001"]
      p.customize ["createhd", "--filename", "../files/projectX-pull-requests-kiosk.vmdk", "--size", 80*1024]
      p.customize ["storageattach", :id, "--storagectl", "IDE Controller", "--port", 0, "--device", 0, "--type", "hdd", "--medium", "none"]
      p.customize ["storageattach", :id, "--storagectl", "IDE Controller", "--port", 0, "--device", 0, "--type", "hdd", "--medium", "../files/projectX-pull-requests-kiosk.vmdk"]
      p.customize ["storagectl", :id, "--name", "SATA Controller", "--add", "sata",  "--controller", "IntelAHCI", "--hostiocache", "on"]
      p.customize ["storageattach", :id, "--storagectl", "SATA Controller", "--port", 1, "--device", 0, "--type", "hdd", "--medium", "../files/ipxe_projectX-pull-requests-kiosk.vmdk"]
    end
  end

  config.vm.define "projectX-pull-requests-display" do |dn_config|
    dn_config.vm.box_url = "https://opscode-vm-bento.s3.amazonaws.com/vagrant/virtualbox/opscode_ubuntu-14.04_chef-provisionerless.box"
    dn_config.vm.hostname = "display"
    dn_config.vm.box = "opscode-ubuntu-14.04"
    dn_config.vm.synced_folder ".", "/vagrant", disabled: true
    dn_config.vm.boot_timeout = 1
    dn_config.vm.provider :virtualbox do |p|
      p.customize ["modifyvm", :id, "--memory", "2048"]
      p.customize ["modifyvm", :id, "--cpus", "1"]
      p.customize ["modifyvm", :id, "--boot1", "floppy"]
      p.customize ["modifyvm", :id, "--boot2", "net"]
      p.customize ["modifyvm", :id, "--boot3", "none"]
      p.customize ["modifyvm", :id, "--boot4", "none"]
      p.customize ["modifyvm", :id, "--intnet1", "projectX-pull-requests"]
      p.customize ["modifyvm", :id, "--nicpromisc1", "allow-all"]
      p.customize ["modifyvm", :id, "--nic1", "intnet"]
      p.customize ["modifyvm", :id, "--nic2", "none"]
      p.customize ["modifyvm", :id, "--nic3", "none"]
      p.customize ["modifyvm", :id, "--nic4", "none"]
      p.customize ["modifyvm", :id, "--nictype1", "Am79C970A"]
      p.customize ["modifyvm", :id, "--ioapic", "on"]
      p.customize ["modifyvm", :id, "--macaddress1", "5ca1ab1e0002"]
      p.customize ["createhd", "--filename", "../files/projectX-pull-requests-display.vmdk", "--size", 80*1024]
      p.customize ["storageattach", :id, "--storagectl", "IDE Controller", "--port", 0, "--device", 0, "--type", "hdd", "--medium", "none"]
      p.customize ["storageattach", :id, "--storagectl", "IDE Controller", "--port", 0, "--device", 0, "--type", "hdd", "--medium", "../files/projectX-pull-requests-display.vmdk"]
      p.customize ["storagectl", :id, "--name", "SATA Controller", "--add", "sata",  "--controller", "IntelAHCI", "--hostiocache", "on"]
      p.customize ["storageattach", :id, "--storagectl", "SATA Controller", "--port", 1, "--device", 0, "--type", "hdd", "--medium", "../files/ipxe_projectX-pull-requests-display.vmdk"]
    end
  end

end

Finally we have the environment stored in one Vagrantfile. The missing part is to how to run tests on it.

The testing script does the following:

#/bin/bash
set -e
# FUNCTIONS

function halt_vm () {
  vms=`vboxmanage list vms | grep "vagrant_$1_" | awk {'print $1'} | sed s/'"'//g`
  echo "Stopping VM $vms"
  stop_result=$(for vm in $vms ; do vboxmanage controlvm $vm poweroff; echo $?; done)
  echo "Output of stopping VM $1 : $stop_result"
}

function boot_vm () {
  vms=`vboxmanage list vms | grep "vagrant_$1_" | awk {'print $1'} | sed s/'"'//g`
  echo "Booting VM $vms"
  start_result=$(for vm in $vms ; do vboxmanage startvm $vm --type headless; echo $?; done)
  echo "Output of booting VM $1 : $start_result"
  echo "Sleeping additional 15 secs after peacefull boot"
  sleep 15
}

function add_keys () {
  for i in `find /var/lib/jenkins/.ssh/id_rsa* | grep -v '.pub'` ; do ssh-add $i ; done
}

#vars

knifeclient_name=lg-head-projectXtest.liquid.glx
headnode_name=projectX-pull-requests

# TEST SCENARIO

cd test/vagrant

# teardown of previous sessions
vagrant destroy projectX-pull-requests-kiosk -f
vagrant destroy projectX-pull-requests-display -f
vagrant destroy $headnode_name -f

echo "Not managing knife client because => chef_zero "
echo "All ssh keys presented below"
ssh-add -l

# headnode
vagrant up ${headnode_name}

# displaynodes

result=$(vboxmanage convertfromraw ../files/ipxe.usb ../files/ipxe_projectX-pull-requests-kiosk.vmdk --format=VMDK ; vagrant up projectX-pull-requests-kiosk; echo $?)
echo "projectX-pull-requests-kiosk : $result"

result=$(vboxmanage convertfromraw ../files/ipxe.usb ../files/ipxe_projectX-pull-requests-display.vmdk --format=VMDK ; vagrant up projectX-pull-requests-display; echo $?)
echo "projectX-pull-requests-display : $result"

# test phase
OPTIONS=`vagrant ssh-config  ${headnode_name} | grep -v ${headnode_name} | awk -v ORS=' ' '{print "-o " $1 "=" $2}'`
scp ${OPTIONS} ../integration/projectX-pr/bats/*.bats vagrant@${headnode_name}:/tmp/bats_tests

ssh ${OPTIONS} ${headnode_name} '/usr/local/bin/bats /tmp/bats_tests/pre_build_checks.bats'

halt_vm projectX-pull-requests-kiosk
halt_vm projectX-pull-requests-display

echo "Building teh ISO (it may take a long time)"
ssh ${OPTIONS} ${headnode_name} '/usr/local/bin/bats /tmp/bats_tests/build_iso.bats'

ssh ${OPTIONS} ${headnode_name} '/usr/local/bin/bats /tmp/bats_tests/set_grub_to_make_partitions.bats'

echo "Booting nodes"

boot_vm projectX-pull-requests-kiosk
boot_vm projectX-pull-requests-display

echo "Sleeping 30 secs for the DNS to boot and setting the grub to boot the ISO"
sleep 30

ssh ${OPTIONS} ${headnode_name} '/usr/local/bin/bats /tmp/bats_tests/set_grub_to_boot_the_iso.bats'
echo "Sleeping for 4 mins for the displaynodes to boot fresh ISO"
sleep 240

echo "Running the tests inside the headnode:"

ssh ${OPTIONS} ${headnode_name} '/usr/local/bin/bats /tmp/bats_tests/post_checks.bats'

So finally we get the following pipeline:

Clone Chef pull request from GitHub
Create Vagrantfile on the basis of Vagrantfile template
Create run_tests.sh script for running the tests
Destroy all previously created Vagrant boxes
Create one Chef Vagrant box
Create ISO Vagrant boxes with ipxe bootloader
Converge the Vagrant box with Chef
Copy BATS tests onto the headnode
Run initial BATS tests that build an ISO
Boot display nodes with the newly created ISO
Run final integration tests on the stack

Elapsed time—between 40 and 50 minutes.

Cron Wrapper: Keep your cron jobs environment sane

2015-02-06T00:00:00+00:00

It is becoming more common for developers to not use the operating system packages for programming languages. Perl, Python, Ruby, and PHP are all making releases of new versions faster than the operating systems can keep up (at least without causing compatibility problems). There are now plenty of tools to help with this problem. For Perl we have Perlbrew and plenv. For Ruby there is rbenv and RVM. For Python there is Virtualenv. For PHP there is PHP version. These tools are all great for many different reasons but they all have issues when being used with cron jobs. The cron environment is very minimal on purpose. It has a very restrictive path, very few environment variables and other issues. As far as I know, all of these tools would prefer using the env command to get the right version of the language you are using. This works great while you are logged in but tends to fail bad as a cron job. The cron wrapper script is a super simple script that you put before whatever you want to run in your crontab which will ensure you have the right environment variables set.

#!/bin/bash -l

exec "$@"

The crontab entry would look something like this:

34 12 * * * bin/cron-wrapper bin/blog-update.pl

The -l on the executing of bash makes it act like it is logging in. Therefore it picks up anything in the ~/.bash_profile and has that available to the env command. This means the cron job runs in the same environment that is setup when you run it from the command line, helping to stop those annoying times where it works fine from the command line but breaks in cron. Jon Jensen went into much greater detail on the benefits of using the -l here. Hope this helps!

CentOS 7 on Hetzner server with more than 2 TB disk

2015-01-22T00:00:00+00:00

We use a variety of hosting providers for ourselves and our clients, including Hetzner. They provide good servers for a great price, have decent support, and we’ve been happy with them for our needs.

Recently I was given the task of building out a new development server for one of our clients, and we wanted it to be set up identically to another one of their servers but with CentOS 7. I placed the order for the hardware with Hetzner and then began the procedure for installing the OS.

Hetzner provides a scripted install process that you can kick off after booting the machine into rescue mode. I followed this process and selected CentOS 7 and proceeded through the whole process without a problem. After rebooting the server and logging in to verify everything, I noticed that the disk space was capped at 2 TB, even though the machine had two 3 TB drives in it (in hardware RAID 1). I looked at the partitions and found the partition table was “msdos”. Ah ha!

At this point painful memories of running into this problem before hit me. I reviewed our notes of what we had done last time, and felt like it was worth a shot even though this time I’m dealing with CentOS 7. I went through the steps up to patching anaconda and then found that anaconda for CentOS 7 is newer and the files are different. I couldn’t find any files that care about the partition table type, so I didn’t patch anything.

I then tried to run the CentOS 7 install as-is. This only got me so far because I then ran into trouble with NetworkManager timing out and not starting.

A screenshot of the CentOS 7 installer failing (anaconda) similar to what I was seeing.

Baffled, I looked into what may have been causing the trouble and discovered that the network was not set up at all and it looked as if no network interfaces existed. WHAT?? At this point I dug through dmesg and found that the network interfaces did indeed exist but udevd had renamed them. Ugh!

Many new Linux distributions are naming network interfaces based on their physical connection to the system: those embedded on the motherboard get named em1, em2, etc. Apparently I missed the memo on this one, as I was still expecting eth0, eth1, etc. And from all indications, so was NetworkManager because it could not find the network interfaces!

Rather than spend more time going down this route, I decided to change gears and look to see if there was any way to patch the Hetzner install scripts to use a GPT partition table with my install instead of msdos. I found and read through the source code for their scripts and soon stumbled on something that just might solve my problem. In the file /root/.oldroot/nfs/install/functions.sh I found mention of a config variable FORCE_GPT. If this is set to “1” then it will try to use a GPT partition table unless it thinks the OS won’t like it, and it thinks that CentOS won’t like it (no matter the version). But if you set FORCE_GPT to “2” it will use a GPT partition table no matter what. This config setting just needs to be added to the file you edit where you list out your partitions and LVM volumes.

FORCE_GPT 2

PART /boot ext3 512M
PART lvm   vg0  all

LV  vg0  swap swap   swap  32G
LV  vg0  root  /     ext4 100G
LV  vg0  home  /home ext4 400G

I then ran the installer script and added the secret config option and… Bingo! It worked perfectly! No need to manually patch anything or install manually. And now we have a CentOS 7 server with full 3 TB of disk space usable.

(parted) print
Model: DELL PERC H710 (scsi)
Disk /dev/sda: 3000GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: pmbr_boot

Number  Start   End     Size    File system  Name  Flags
 3      1049kB  2097kB  1049kB                     bios_grub
 1      2097kB  539MB   537MB   ext3
 2      539MB   3000GB  2999GB                     lvm

Highlights of OpenWest conference 2014

2014-05-16T00:00:00+00:00

Last week I was able to not only attend the OpenWest conference, held in Orem, UT at the UVU campus, but I was also able to be a presenter. This year’s conference had their largest attendance to date and also their most diverse list of tracks and presentations. Many of the presentations are posted on YouTube.

My own presentation was on Git Workflows That Work (previous blog post, talk slides). It was part of the “Tools” track and went pretty well. Rather than recap my own presentation, I’d like to briefly touch on a few others I enjoyed.

Building a Scalable Codebase using Domain Driven Design

The folks at Insidesales.com presented on Domain Driven Design. This was something I had heard of but wasn’t very familiar with. The main principles of DDD are: understand the business purpose (not just the app requirements), rigorous organization (of code, services, design, etc.), distinct layers, and functional cohesion. The biggest point I got was that all business logic should be located in a “domain” separated from the application and separated from the data. The application is then just a thin wrapper around API calls to the business logic within the domain. DDD is different from MVC though because each layer is distinct and could have its own design. Thus MVC could be a smaller piece within a given layer.

As I listened to this presentation I remembered a project I worked on at a previous job building a Feeds Admin to create and manage product feeds to third parties (search engines, shopping portals, etc). I had built that using DDD without realizing it at the time. It came naturally as I wanted to cleanly organize and separate the logic for each feed (with different data pulled from the database, different feed formats, different transport methods, etc.). So I definitely have seen the benefits and principles of DDD in practice.

Mentoring Devs into DevOps

This presentation by Justin Carmony discussed how his team has been going through a transition of empowering developers to also do some Ops work. They had roughly 30 developers, 2 operations admins, and over 300 servers in their infrastructure. They also had several different tools and ways of doing things between the different teams of developers; for example, some used Capistrano and some used Ansible, and some used Jenkins and others Travis CI, some used Vagrant and some didn’t, etc.). They considered hiring for a new DevOps position to own and mentor everyone in standardizing tools and processes, but instead promoted one of their star developers to that role. This was a tough decision because it meant that they would not be able to use his skills a developer. But Justin said it was definitely worth it because within just 3 months this developer had helped everyone to standardized on tools and release procedures, making the entire team more productive and efficient.

They also switched to using Salt for configuration management for all servers at this same time. And to empower the developers and expose them to the world of Ops, they switched to using Vagrant managed by Salt for all development environments. Developers were able to make some Salt changes if needed to their own environment, and then submit a git pull request to Ops for peer review of the Salt configuration and Ops would then merge it and deploy it to production if they accepted the changes.

Justin mentioned that it is still an on-going transition for them. Ultimately the main points he presented are:

There is a long continuum or gradient of skill level and access level between Dev and Ops and it is better to think of DevOps as a continuum as well and not a strict role with strict access levels. Some Devs can have access to dashboards and graphs of systems while others can manage build tools and configuration. “DevOps” can be spread out.
Team culture matters. It can be very difficult to improve processes and get people to embrace change for the better, so positive influences by everyone involved can make a big difference. Business owners need to be supportive as well.

Retrospectus

Daniel Evans gave an interesting presentation on retrospective meetings. For those not familiar with these meetings, they are for discussing “what went well” and “what could be improved” over the last project/sprint/time period/etc. I’ve participated in these types of meetings for many years and have seen the benefits that can come from them. Daniel also talked about what these meetings are not: they are not meant for deep dives into problems, blame, or griping. They should be focused on the good that has been done and on solutions to fix problems going forward. And even then, if the solutions are not quickly apparent then you should not spend too much time trying to find them. Those conversations should happen outside of this meeting.

His presentation was quite short and didn’t really cover much that I didn’t know already. However, at the end he then turned the rest of the time into a retrospective meeting reviewing his presentation, with everyone participating. This turned out to be pretty fun and was a good exercise, and covered a lot more then a regular Q & A session would have.

Puppet, Salt, and DevOps (a review of the MountainWest DevOps conference)

2014-03-27T00:00:00+00:00

Last week I attended the MountainWest DevOps conference held in Salt Lake City, Utah. This was a one day conference with a good set of presenters and lightning talks. There were several interesting topics presented, but I’ll only review a few I wanted to highlight.

I Serve No Master!

Aaron Gibson of Adaptive Computing discussed a very common problem with Puppet (and other configuration management systems): they work well in the scenario they were designed for but what about when the situation isn’t typical? Aaron had a situation where developers and QA engineers could instantiate systems themselves via OpenStack, however the process for installing their company’s software stack on those VMs was inconsistent, mostly manual, and took many hours. One of the pain points he shared, which I related to, was dealing with registering a puppet node with a puppet master—the sometimes painful back and forth of certificate issuing and signing.

His solution was to remove the puppet master completely from the process. Instead he created a bash wrapper script to execute a workflow around what was needed, still using puppet manifests on each system but run locally. This wrapper tool, called “Builder”, relies on property files to customize the config and allow the script to manage different needs. This new script allowed them to keep using puppet to manage these self-serve OpenStack servers gaining the benefits of consistency and removing manual setup steps, providing the ability to automate installs with Jenkins or other tools. But it freed them from having to use a puppet master for nodes that were disposable. It also helped to reduce software install time from 12 hours down to a 11 minutes.

His Builder tool is still an internal only tool for his company, but he discussed some next steps he would like to add, including better reporting and auditing of executions. I pinged him after the conference on twitter and mentioned that Rundeck might be a good fit to fill that gap. I used Rundeck for 2 years at my last job, integrating nicely with other automation tools and providing reporting and auditing as well as access control of arbitrary jobs.

Automating cloud factories and the internet assembly line with SaltStack

Tom Hatch of Salt Stack spoke about Salt as an automation and remote execution platform. I’ve done quite a bit of work with Salt recently with a client and so I was pretty familiar with Salt. But one thing that he mentioned I didn’t know was that Salt was originally designed as a Cloud management tool, not necessarily a configuration management tool. However in the course of time configuration management became a higher priority for the Salt dev team to focus on. Tom mentioned that recently they have been working on Cloud management tools again—providing integration with Rackspace, AWS, xen, and more. I’ll have to dig more into these tools and give them a try.

How I Learned to Stop Worrying and Love DevOps

Bridget Kromhout of 8thBridge (since aquired by Fluid) spoke on the culture of DevOps and her journey from a corporate, strictly siloed environment to a small start-up that embraced DevOps. One of the first things she brought up that was different was the focus and approach to goals of each organization. In an organization where Ops teams are strictly separate from Developers, they often butt heads and have a limited vision of priorities. Each focuses on the goals of their own team or department, and have little understanding of the goals of the other departments. This leads to an adversarial relationship and culture of not caring much about different teams or departments.

In contrast, the organization that embraces DevOps as a culture will see to it that Ops and Devs work together on whatever solution best reaches the goals of the whole organization. In doing so, barriers will have to be questioned. Any “special snowflake” servers/applications/etc. that only one person knows and can touch can’t exist in this culture. Instead, any unique customizations need to be minimized through automation, documentation (sharing knowledge), and reporting/monitoring. This doesn’t mean root access for all—but it means reducing barriers as much as possible. Good habits from the Ops world to keep include: monitoring, robustness, security, scaling, and alerting.

The main pillars of DevOps are: culture, automation, measurement, and sharing. Culture is important and can be supportive or rejecting of the other pillars. Without a culture in the organization that supports DevOps, it will fizzle back into siloed “us vs. them” enmity.

Thanks to all the presenters and those that put on the conference. It was a great experience and I am glad I attended.

Provisioning a Development Environment with Packer, Part 2

2014-03-14T00:00:00+00:00

In my previous post on provisioning a development environment with Packer I walked through getting a server setup with an operating system installed. This post will be focused setting up Ansible so that I can setup my development environment just the way I like it. Packer supports many different methods for provisioning. After playing with some of them, I decided that Ansible was a good mix of simplicity and functionality.

A Packer provisioner is simply a configuration template that is added to the json configuration file. The “provisioners” section of the configuration file takes an array of json objects which means that you aren’t stuck with just one kind of provisioner. For example, you could run some shell scripts using the shell provisioner, then upload some files using the File Uploads provisioner, followed by your devops tool of choice (puppet, salt, chef, or ansible). You can even roll-your-own provisioner if desired. Here’s an example provisioner setup for the shell provisioner:

{
  "variables": {...},
  "builders" : [...],
  "provisioners" [
    {
      "type": "shell",
      "inline": [ "echo foo" ]
    }
  ]
}

Sudo and User Considerations

Packer will login to the server over ssh and run your provisioners. The big headache that always comes out of this is some provisioners require sudo, or being logged in as root, to run their commands. Packer, however, will login as the user that was created during the build stage (see the “builders” section in the code snippet in the previous post). There are a couple of ways to handle this. First, you can do everything in packer as root. I don’t love this approach because I like to simulate the way that I setup a machine by hand and I never login as root if I can help it. The second method is to grant your user sudo access. This gets a little tricky so I’ll just show the a code snippet and then explain it below.

{
  "type": "shell",
  "execute_command": "echo '{{user `ssh_pass`}}' | {{ .Vars }} sudo -E -S sh '{{ .Path }}'",
  "inline": [
    "echo '%sudo    ALL=(ALL)  NOPASSWD:ALL' >> /etc/sudoers"
  ]
}

Utilizing the shell provisioner, the execute_command option is used to specify to the provisioner that whenever a command is run, use this command. The commands provided to the inline array are compiled into a single shell script which is injected as the .Path variable. To quote the Packer documentation:

The -S flag tells sudo to read the password from stdin, which in this case is being piped in with the value of [the user variable ssh_pass.] The -E flag tells sudo to preserve the environment, allowing our environmental variables to work within the script.

By setting the execute_command to this, your script(s) can run with root privileges without worrying about password prompts.

Taking advantage of this trick, a command can now be placed in the inline section that will add all members of the sudo group to the sudoers file granting them permission to use sudo without a password. Now I know this isn’t secure but for my purpose, which is to create a custom development enviornment on a Virtual Machine running only on my machine, this will be just fine. I also use this example to illustrate how to run commands as sudo.

The Ansible Provisioner

Once the shell provisioner is working, Ansible can be installed on the new machine and then executed using Packer’s Ansible provisioner. The easiest way to do this that I found was to have the shell provisioner install Ansible on the virtual machine as well as upload an ssh public key so that the Ansible user could log in. My “provisioners” section looks like this:

"provisioners": [
  {
    "type": "shell",
    "inline": [
      "mkdir .ssh",
      "echo '{{user `public_key`}}' >> .ssh/authorized_keys"
    ]
  },
  {
    "type": "shell",
    "execute_command": "echo '{{user `ssh_pass`}}' | {{ .Vars }} sudo -E -S sh '{{ .Path }}'",
    "inline": [
      "add-apt-repository ppa:rquillo/ansible",
      "apt-get update",
      "apt-get install -y ansible",
      "echo '%sudo    ALL=(ALL)  NOPASSWD:ALL' >> /etc/sudoers"
    ]
  },
  {
    "type": "ansible-local",
    "playbook_file": "site.yml"
  }
]

This configuration uses a variable in which I placed an ssh public key (I could also have used the File Uploads provisioner for this), installs Ansible from an updated PPA, and grants the user sudo priveliges via the sudo group as explained above. This also shows how you can execute more than one statement at a time using the “shell” provisioner.

Ansible Provisioning Tip

This tip could probably apply to any of the devops tools you’d like to use. If you are creating your Ansible yml files for the first time, you will likely run into the issue where you spend a lot of time waiting for the machine to build and provision only to discover your Ansible script is wrong. Troubleshooting becomes a problem because if anything fails during the provision, Packer will stop running and delete the Virtual Machine leaving you with no option other than fixing your mistake and then waiting for the entire process to run again.

One way I found around this is to take the last section of the provisioners array out, build your machine, and then move the base machine into a new directory once it’s been successfully built. From there you can start the machine manually and then run the ansible-playbook command from your local machine while you develop your playbook. Once you have a working playbook, add the ansible-local section back to the provisioners array and rebuild your machine with Packer. That should speed up your development and troubleshooting cycles.

A Hiccup with Ansible and Packer

Ansible allows you to create template files that can be used for configuration files and the like. According to the documentation, you can specify the location of the templates and other files using the playbook_paths option in the provisioner. I could not get this to work and after a lot of troubleshooting and looking at the code for the provisioner I am convinced there is a bug copying the playbook_paths directories to the remote machine. I’ve posted on the Packer discussion group about this but haven’t had any response on it yet. Once I get to the bottom the issue I’ll post an update here.

Conclusion

Packer has turned out to be a fabulous resource for me in quickly ramping up development environments. Ansible has also been a breath for fresh air for provisioning those machines. I’ve previously used chef and other devops tools which only led to a great deal of frustration. I’m happy to have some new tools in my belt that took very little time to learn and to get working.

Provisioning a Development Environment with Packer, Part 1

2014-03-12T00:00:00+00:00

I recently needed to reconstruct an old development environment for a project I worked on over a year ago. The codebase had aged a little and I needed old versions of just about everything from the OS and database to Ruby and Rails. My preferred method for creating a development environment is to setup a small virtual machine (VM) that mimics the production environment as closely as possible.

Introducing Packer

I have been hearing a lot of buzz lately about Packer and wanted to give it a shot for setting up my environment. Packer is a small command line tool written in the increasingly popular Go programming language. It serves three primary purposes:

Building a machine based on a set of configuration parameters
Running a provisioner to setup the machine with a desired set of software and settings
Performing any post processing instructions on that machine

Packer is really simple to install and I would refer you to their great documentation to get it set up. Once set up, you will have the packer command at your disposal. To build a new machine, all you need to is call:

packer build my_machine.json

The file my_machine.json can be the name of any json file and contains all the information Packer needs to setup your machine. The configuration json has three major sections: variables, builders, and provisioners. Variables are simply key value pairs that you can reference later in the builders and provisioners sections.

The Builder Configuration

Builders takes an array of JSON objects that specify different ways to build your machines. You can think of them as instructions on how to get your machine setup and running. For example, to get a machine up and running you need to create a machine, install an operating system (OS) and create a user so that you can login to the machine. There are many different types of builders, but for the example here, I’ll just use the vmware-iso machine type. Here’s a working JSON configuration file:

{
  "variables": {
    "ssh_name": "mikefarmer",
    "ssh_pass": "mikefarmer",
    "hostname": "packer-test"
  },

  "builders": [
    {
      "type": "vmware-iso",
      "iso_url": "os/ubuntu-12.04.4-server-amd64.iso",
      "iso_checksum": "e83adb9af4ec0a039e6a5c6e145a34de",
      "iso_checksum_type": "md5",
      "ssh_username": "{{user `ssh_name`}}",
      "ssh_password": "{{user `ssh_pass`}}",
      "ssh_wait_timeout": "20m",
      "http_directory" : "preseeds",
      "http_port_min" : 9001,
      "http_port_max" : 9001,
      "shutdown_command": "echo {{user `ssh_pass`}} | sudo -S shutdown -P now",
      "boot_command": [
        "<esc><esc><enter><wait>",
        "/install/vmlinuz noapic ",
        "preseed/url=http://{{ .HTTPIP }}:{{ .HTTPPort }}/precise_preseed.cfg ",
        "debian-installer=en_US auto locale=en_US kbd-chooser/method=us ",
        "hostname={{user `hostname`}} ",
        "fb=false debconf/frontend=noninteractive ",
        "keyboard-configuration/modelcode=SKIP keyboard-configuration/layout=USA ",
        "keyboard-configuration/variant=USA console-setup/ask_detect=false ",
        "initrd=/install/initrd.gz -- <enter>"
      ]
    }
  ]
}

The documentation for these settings is really good but I want to point out a few things that weren’t immediately clear. Some of these pertain mostly to the vmware-iso builder type, but I believe they are worth pointing out because some of them apply to other builder types as well.

First, the iso_url setting can be either an absolute path, a relative path, or a fully qualified url. The relative path is relative to the directory where you run the packer command. So here, when I run packer, I need to make sure that I do so from a directory that has an os subdirectory with the ubuntu iso located therein.

Next, once the ISO is downloaded, Packer will automatically start up your VMWare client and boot the virtual machine. Immediately after that, Packer will start up a VNC client and server along with a mini web server to provide information for your machine. The http_port_min and http_port_max specify which ports to use for the VNC clients. Setting them to the same will allocate just that port for it to use. The http_directory setting provides the name of a local directory to use for the mini web server as the document root. This is important for providing your VM with a preseed file, more about the preseed file will be discussed below.

Since we are using Ubuntu as our main machine, we will need to use sudo to send the shutdown command. The shutdown_command setting is used to gracefully shut down the machine at the conclusion of the run and provisioning of the machine.

Installing your OS

The boot_command is a series of keystrokes that you can send to the machine via VNC. If you have set up a Linux machine from scratch you know that you have to enter in a bunch of information to the machine about how to set it up for the first time such as time zone, keyboard layout, how to partition the hard drive, host name, etc. All these keystrokes needed to set up your machine can be used here.

But if you think about it, that’s a ton of keystrokes and this command could get quite long. A better way to approach this is to use a preseed file. A preseed.cfg file contains the same information you enter when you setup a machine for the first time. This isn’t something provided by Packer, but it is provided by the operating system to automatically provision machines. For Ubuntu, a preseed file is used like so:

When you boot from the startup media (in this case an ISO image), you can choose the location of the preseed file via a URL.
The preseed file is uploaded into memory and the configuration is read.
The installation process begins using information from the preseed file to enter the values where the user would normally enter them.

So how do we get the preseed file up to the machine? Remember that little web server that Packer sets up? Well, the IP address and port is made available to the virtual machine when it boots from the ISO. The following line tells the OS where to find the web server and the configuration file:

"preseed/url=http://{{ .HTTPIP }}:{{ .HTTPPort }}/precise_preseed.cfg"

Strings in Packer can be interpolated using a simple template format similar to mustache. The double curly braces tell Packer to insert a variable here instead of text. The HTTPIP and HTTPPort variables are made available by Packer to the template.

One more important note about the preseed file: You need to make sure that the settings for the username and password are the same as listed in your variables section so that you can login to the machine once it is built. Where do you get a preseed file? I found one on a blog titled Packer in 10 Minutes by @kappataumu. I only had to modify a few settings that were specific to my setup.

Remember that http_directory mentioned above? Well, that directory needs to include your preseed file. I’ve named mine precise_preseed.cfg for Ubuntu 12.04 Precise Pangolin.

Next up is provisioning but that is such a big topic by itself that I’ll move that into a separate blog post. The config file above will work as-is and once run it should setup a basic Ubuntu server for you. Go ahead and give it a try and let me know in the comments how it worked out for you.

Super Powers

I said that Packer has 3 primary purposes earlier. Well, I lied.

Packer’s superpower is that it can perform those 3 purposes over any number of machines, whether virtual, hosted, or otherwise in parallel. Supported machines are currently:

Amazon EC2 (AMI)
DigitalOcean
Docker
Google Compute Engine
OpenStack
QEMU
VirtualBox
VMware

Consider for a moment that you can now automatically setup and provision multiple machines with the same environment using a single command. Now you are seeing the power of Packer.

Setting a server role in Salt (comparing Puppet and Salt)

2013-12-23T00:00:00+00:00

There are many ways to solve a given problem, and this is no truer than with configuration management. Salt (https://saltstack.com) is a fairly new tool in the configuration management arena joining the ranks of Puppet, Chef, and others. It has quickly gained and continues to grow in popularity, boasting its scalable architecture and speed. And so with multiple tools and multiple ways to use each tool, it can get a little tricky to know how best to solve your problem.

Recently I’ve been working with a client to convert their configuration management from Puppet to Salt. This involved reviewing their Puppet configs and designs and more-or-less mapping them to the equivalent for Salt. Most features do convert pretty easily. However, we did run into something that didn’t at first/assigning a role to a server.

We wanted to preserve the “feeling” of the configs where possible. In Puppet they had developed and used a convention for using some custom variables in their configs to assign an “environment” and a “role” for each server. These variables were assigned in the node and role manifests. But in Salt we struggled to find a similar way to do that, but here is what we learned.

In Puppet, once a server’s “role” and “environment” variables were set, then they could be used in other manifest files to select the proper source for a given config file like so:

    file    {
        "/etc/rsyslog.conf":
            source  =>
            [
                "puppet:///rsyslog/rsyslog.conf.$hostname",
                "puppet:///rsyslog/rsyslog.conf.$system_environment-$system_role",
                "puppet:///rsyslog/rsyslog.conf.$system_role",
                "puppet:///rsyslog/rsyslog.conf.$system_environment",
                "puppet:///rsyslog/rsyslog.conf"
            ],
            ensure  => present,
            owner   => "root",
            group   => "root",
            mode    => "644"
    }

Puppet will search the list of source files in order and use the first one that exists. For example, if $hostname = ‘myniftyhostname’ and $system_environment = ‘qa’ and $system_role = ‘sessiondb’, then it will use rsyslog.conf.myniftyhostname if it exists on the Puppet master, or if not then use rsyslog.conf.qa-sessiondb if it exists, or if not then rsyslog.conf.sessiondb if it exists, or if not then rsyslog.conf.qa if it exists, or if not then rsyslog.conf.

In Salt, environment is built into the top.sls file, where you match your servers to their respective state file(s), and can be used within state files as {{ env }}. Salt also allows for multiple sources for a managed file to be listed in order and it will use the first one that exists in the same way as Puppet. We were nearly there; however, setting the server role variable was not as straight forward in Salt.

We first looked at using Jinja variables (which is the default templating system for Salt), but soon found that setting a Jinja variable in one state file does not carry over to another state file. Jinja variables remain only in the scope of the file they were created in, at least in Salt.

The next thing we looked at was using Pillar, which is a way to set custom variables from the Salt master to given hosts (or minions). Pillar uses a structure very similar to Salt’s top.sls structure- matching a host with its state files. But since the hostnames for this client vary considerably and don’t lend themselves to pattern matching easily, this would be cumbersome to manage both the state top.sls file and the Pillar top.sls file and keep them in sync. It would require basically duplicating the list of hosts in two files, which could get out of sync over time.

We asked the salt community on #salt on Freenode.net how they might solve this problem, and the recommended answer was to set a custom grain. Grains are a set of properties for a given host, collected from the host itself- such as, hostname, cpu architecture, cpu model, kernel version, total ram, etc. There are multiple ways to set custom grains, but after some digging we found how to set them from within a state file. This meant that we could do something like this in a “role” state file:

# sessiondb role
# {{ salt['grains.setval']('server_role','sessiondb') }}

include:
  - common
  - postgres

And then within the common/init.sls and postgres/init.sls state files we could use that server_role custom grain in selecting the right source file, like this:

/etc/rsyslog.conf:
  file.managed:
    - source:
      - salt://rsyslog/files/rsyslog.conf.{{ grains['host'] }}
      - salt://rsyslog/files/rsyslog.conf.{{ env }}-{{ grains['server_role'] }}
      - salt://rsyslog/files/rsyslog.conf.{{ grains['server_role'] }}
      - salt://rsyslog/files/rsyslog.conf.{{ env }}
      - salt://rsyslog/files/rsyslog.conf
    - mode: 644
    - user: root
    - group: root

This got us to our desired config structure. But like I said earlier, there are probably many ways to handle this type of problem. This may not even be the best way to handle server roles and environments in Salt, if we were more willing to change the “feeling” of the configs. But given the requirements and feedback form our client, this worked fine.

Use Ansible/Jinja2 templates to change file content based on target OS

2013-12-19T00:00:00+00:00

In the End Point hosting team we really love automating repetitive tasks, especially when it involves remembering many little details which can over time be forgotten, like differences of coreutils location between some versions of Ubuntu (Debian), CentOS (Red Hat) and OpenBSD variants.

In our environment we bind the backup SSH user authorized_keys entry to a custom command in order to have it secured by being, among other aspects, tied to a specific rsync call.

So in our case the content of our CentOS authorized_keys would be something like:

command="/bin/nice -15 /usr/bin/rsync --server --daemon .",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa AAAB3[...]Q== endpoint-backup

Sadly that’s only true for CentOS systems so that if you want to automate the distribution of authorized_keys (as we’ll show in another post) to different Linux distributions (like Ubuntu) you may need to tweak it to comply to the new standard “/usr/bin” location, which will be eventually adopted by all new Linux versions overtime.. RHEL 7.x onward included.

To do the OS version detection we decided to use an Ansible/Jinja2 template by placing the following line in the Ansible task:

- name: Deploy /root/.ssh/authorized_keys
  template: src=all/root/.ssh/authorized_keys.j2
            dest=/root/.ssh/authorized_keys
            owner=root
            group=root
            mode=0600

And inside the actual file place a slightly modified version of the line above:

command="{% if ansible_os_family != "RedHat" %}/usr{% endif %}/bin/nice -15 /usr/bin/rsync --server --daemon .",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa AAAB3[...]Q== endpoint-backup"

So that if the target OS is not part of the “RedHat” family it will add the “/usr” in front of the “/bin/nice” absolute path.

Easy peasy, ain’t it?

Now go out there and exploit this feature to all your needs.

Migrating from Legacy Networking to systemd-networkd on Ubuntu 24.04

Why migrate to systemd-networkd?

Step 1: Check the current setup

Step 2: Install and enable systemd-networkd

Step 3: Create a network configuration file

Step 4: Configure systemd-resolved

Restart and verify

Step 6: Reboot and confirm

Handling remote migration and downtime risks

Automating the migration and rollback process

migrate_to_systemd_networkd.sh

upgrade_watchdog.sh

Conclusion

Testing Ansible Automation with Molecule

1. Why Test with Molecule?

2. Install Prerequisites

3. Ansible Monorepo Structure

4. Creating Your First Molecule Scenario

5. Running Molecule tests

6. Integrating Molecule with GitLab CI/CD

Automating DNS Management with GitLab CI/CD

Our Initial Approach

The Solution: Automating with GitLab CI/CD

Implementation Details:

1. .gitlab-ci.yaml

2. Posting OpenTofu plan/apply logs as merge request comments

3. Migrating to GitLab’s Remote HTTP Backend

Kubernetes From The Ground Up With AWS EC2

Prerequisites

EC2 Instances provisioning

Installing the container runtime

Kubernetes control plane setup

Kubernetes worker nodes setup

Hello, Kubernetes :)

Conclusion

CI/CD with Azure DevOps

Creating Environments

Creating the Pipelines

Understanding the YAML

Adding Branch Protection and Build Validation

Monitoring and Troubleshooting the Pipeline

Wrapping Up

DevOps & Kubernetes engineer job opening

What you will be doing:

You’ll need professional experience with:

You have these important work traits:

What work here offers:

Get in touch with us:

Installing Ubuntu 18.04 to a different partition from an existing Ubuntu installation

Upgrading from Ubuntu 14.04 LTS

Installing Clean Ubuntu 18.04 LTS from Ubuntu 14.04 LTS

Install debootstrap

Creating a new root partition

Mounting the new root partition

Bootstrapping the new root partition

Installing fstab

Mounting things in the new root partition

Configuring apt

Installing required packages and running chef-client

Set Ubuntu 14.04 to boot default

On the Current system (Trusty):

On the target system (Bionic):

Boot into Bionic

Set Ubuntu 18.04 to boot default

Job opening: Linux system administration and DevOps remote engineer

What you will be doing:

What you bring:

What work here offers:

Getting in touch with us:

SRV DNS records in Terraform and Cloudflare

Disaster Recovery — Miami to Dallas in one hit

Separate and Not Equal: Development Environments to Support People, Process, and Automation

Dev

QA

Stage

Prod

Perl Dancer Conference 2016 Day 1

Perl Dancer Conference Day 1

Sawyer X (Sawyer X) — A bus tour through Dancer core

Stefan Hornburg (Racke) — No Act on ACT

1. `.gitlab-ci.yaml`