Automating DNS Management with GitLab CI/CD

2024-09-06T00:00:00+00:00

At End Point, managing DNS records across multiple domains has historically been a manual task. This blog post details our journey from manual processes to an automated workflow using GitLab CI/CD.

Our Initial Approach

With multiple domains and frequent updates necessary to manage the servers, manual handling of DNS changes became a bottleneck. Initially, our process looked like this:

Make changes to the OpenTofu configuration files
Create a merge request (MR) in GitLab
They would run tofu plan manually and paste the plan output into the MR for review
A coworker would review the MR and approve the changes
Once merged, the engineer would manually run tofu apply to implement the changes

While this process worked, automating it could enhance our productivity and minimize errors, integrating our DNS management directly into our CI/CD pipeline.

The Solution: Automating with GitLab CI/CD

Change Submission: Engineers make changes to the OpenTofu files and submit a merge request
Plan Creation: A GitLab CI/CD job automatically generates an OpenTofu plan when changes are proposed
Review Process: A coworker reviews the automatically generated plan in the MR
Applying Changes: Once approved and merged, another CI/CD job automatically runs tofu apply to implement the changes

Implementation Details:

1. `.gitlab-ci.yaml`

If you are doing this on a self-hosted GitLab instance you would need to import the repo to your hosted GitLab workspace and use it there.

We divided our GitLab CI/CD pipelines into three main stages:

Validation: Where we check if everything looks right
Planning: Where we create a plan to show the changes
Deployment: Where the changes get applied once everything is approved

It’s important to note that our CI/CD pipeline runs on a dedicated worker. This means only this worker has access to the secret keys needed for DNS management. By isolating these credentials to a single, controlled environment, we significantly reduce the risk of unauthorized access or accidental exposure of sensitive data.

include:
  - component: gitlab.com/components/opentofu/job-templates@main
    inputs:
      version: 0.24.0
      opentofu_version: 1.6.2
      auto_apply: false

before_script:
  - source /home/gitlab-runner/.tofu.env

stages: [ validate, build, deploy]

fmt-example.com:
  extends: [ .opentofu:fmt ]
  variables:
    TF_ROOT: "example.com"
    TF_STATE_NAME: "example_com"
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
      changes:
        - example.com/*
    - if: '$CI_COMMIT_BRANCH == "main"'
      changes:
        - example.com/*

validate-example.com:
  extends: [ .opentofu:validate ]
  variables:
    TF_ROOT: "example.com"
    TF_STATE_NAME: "example_com"
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
      changes:
        - example.com/*
    - if: '$CI_COMMIT_BRANCH == "main"'
      changes:
        - example.com/*

plan-example.com:
  extends: [ .opentofu:plan ]
  variables:
    TF_ROOT: "example.com"
    TF_STATE_NAME: "example_com"
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
      changes:
        - example.com/*
    - if: '$CI_COMMIT_BRANCH == "main"'
      changes:
        - example.com/*
  after_script:
    - source /home/gitlab-runner/.tofu.env
    - ./gitlab-comment-tofu-log.sh $TF_ROOT/plan_output.txt

apply-example.com:
  extends: [ .opentofu:apply ]
  variables:
    TF_ROOT: "example.com"
    TF_STATE_NAME: "example_com"
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'
      changes:
        - example.com/*
  after_script:
    - source /home/gitlab-runner/.tofu.env
    - ./gitlab-comment-tofu-log.sh $TF_ROOT/apply_output.txt

2. Posting OpenTofu plan/apply logs as merge request comments

To facilitate code reviews, we created a script (gitlab-comment-tofu-log.sh) that automatically formats and posts the OpenTofu plan and apply logs as comments on merge requests. This ensures that reviewers can easily understand the changes before approving them.

Here is the script:

#!/bin/bash

set -e

echo "GitLab MR Comment Script Starts"

if [ $# -eq 0 ]; then
    echo "Error: No output file path provided"
    echo "Usage: $0 <path_to_output_file>"
    exit 1
fi

output_file="$1"
echo "Using output file: $output_file"

echo "Checking environment variables"
if [ -z "$GITLAB_API_TOKEN" ]; then
    echo "Error: GITLAB_API_TOKEN is not set"
    exit 1
fi

if [ -z "$CI_PROJECT_ID" ]; then
    echo "Error: CI_PROJECT_ID is not set"
    exit 1
fi

if [ -n "$CI_MERGE_REQUEST_IID" ]; then
    echo "Using CI_MERGE_REQUEST_IID: $CI_MERGE_REQUEST_IID"
else
    echo "CI_MERGE_REQUEST_IID not set, fetching from API"
    CURRENT_COMMIT_SHA=$(git rev-parse HEAD)
    echo "Current commit SHA: $CURRENT_COMMIT_SHA"

    MR_INFO=$(curl --silent --header "PRIVATE-TOKEN: $GITLAB_API_TOKEN" \
        "https://<link_to_self_hosted_gitlab>/api/v4/projects/$CI_PROJECT_ID/repository/commits/$CURRENT_COMMIT_SHA/merge_requests")

    if [ "$(echo $MR_INFO | jq '. | length')" -eq 0 ]; then
        echo "Error: No merge request found for this commit" >&2
        exit 1
    fi

    CI_MERGE_REQUEST_IID=$(echo $MR_INFO | jq '.[0].iid')
    echo "Found Merge Request IID: $CI_MERGE_REQUEST_IID"
fi

echo "Reading output file: $output_file"
if [ ! -f "$output_file" ]; then
    echo "Error: File $output_file not found"
    exit 1
fi

file_name=$(basename "$output_file")
if [[ "$file_name" == plan_* ]]; then
    heading="<br>Plan log for $TF_ROOT"
    output=$(awk '/OpenTofu used the selected providers/,0' "$output_file")
elif [[ "$file_name" == apply_* ]]; then
    heading="<br>Apply log for $TF_ROOT"
    output=$(awk '/OpenTofu has been successfully initialized/,0' "$output_file")
else
    heading="<br>Tofu Output for $TF_ROOT"
fi

echo "Filtered output read. Length: ${#output} characters"

echo "Cleaning and formatting output"
cleaned_output=$(echo -e "$output" | sed -e 's/\x1b\[[0-9;]*m//g' \
    -e 's/\\u001b\[[0-9;]*m//g' \
    -e 's/\\n/\n/g' \
    -e 's/\\"//g' \
    -e 's/^"//g' \
    -e 's/"$//g')
echo "Cleaned output. Length: ${#cleaned_output} characters"

# Format the output with heading and code block
formatted_output="$(printf '## %s\n\n```\n%s\n```' "$heading" "$cleaned_output")"
echo "Formatted output. Length: ${#formatted_output} characters"

echo "Posting comment to GitLab MR"
response=$(curl --silent --show-error --fail \
     --request POST \
     --header "PRIVATE-TOKEN: $GITLAB_API_TOKEN" \
     --header "Content-Type: application/json" \
     --data "$(jq -n --arg body "$formatted_output" '{"body": $body}')" \
     "https://code.self_hosted_gitlab.com/api/v4/projects/$CI_PROJECT_ID/merge_requests/$CI_MERGE_REQUEST_IID/notes")


if [ $? -eq 0 ]; then
    echo "Comment successfully posted to GitLab MR"
else
    echo "Error: Failed to post comment to GitLab MR. Response: $response"
    exit 1
fi

echo "Script completed successfully"

Here is the tofu plan output comment:

Here is the tofu apply output comment:

3. Migrating to GitLab’s Remote HTTP Backend

Transitioning your local Tofu state to a remote backend hosted by GitLab can streamline state management and enhance security. Follow these straightforward steps to achieve this.

First, set up the necessary environment variables with your specific details:

export TF_STATE_NAME="example_net"
export TF_HTTP_ADDRESS="https://<link_to_GitLab_instance>>/api/v4/projects/460/terraform/state/$TF_STATE_NAME"
export TF_HTTP_USERNAME="<create_a_user>"
export TF_HTTP_PASSWORD="<create_a_password>"

Initialize the remote backend with Tofu, specifying your new backend configuration:

tofu init -migrate-state \
  -backend-config="address=$TF_HTTP_ADDRESS" \
  -backend-config="username=$TF_HTTP_USERNAME" \
  -backend-config="password=$TF_HTTP_PASSWORD"

Verify the migration by running:
```
tofu plan
```

After these steps, your Tofu environment will be fully integrated with GitLab’s remote HTTP backend, with state management and enhanced collaboration and security.

Automation of our DNS management with GitLab CI/CD and OpenTofu has transformed our operations, making them more efficient, error-resistant, and secure. We encourage other teams to explore similar automation strategies to enhance their infrastructure management processes.

Introduction to Terraform with AWS

2022-11-09T00:00:00+00:00

Photo by Jeffry Johar

Terraform is a tool from HashiCorp to enable infrastructure as code (IaC). With it users can define and manage IT infrastructure in source code form.

Terraform is a declarative tool. It will ensure the desired state as defined by the user.

Terraform comes with multiple plugins or providers which enable it to manage a wide variety of cloud providers and technologies such as but not limited to AWS, GCP, Azure, Kubernetes, Docker and others.

This blog will go over how to use Terraform with AWS.

Prerequisites

For this tutorial we will need the following:

An active AWS account.
An internet connection to download required files.
A decent editor such as Vim or Notepad++ to edit the configuration files.

Install AWS CLI

We need to set up the AWS CLI (command-line interface) for authentication and authorization to AWS.

Execute the following command to install the AWS CLI on macOS:

$ curl -O https://awscli.amazonaws.com/AWSCLIV2.pkg
$ sudo installer -pkg AWSCLIV2.pkg -target /

For other OSes see Amazon’s docs.

Execute the following command and enter the AWS Account and Access Keys:

$ aws configure

Install Terraform

We need to install Terraform. It is just a command line tool. Execute the following to install Terraform on macOS:

$ brew tap hashicorp/tap
$ brew install hashicorp/tap/terraform

For other OSes see Terraform’s installation docs.

Create the Terraform configuration file

Before we can create any Terraform configuration file for a project, we need to create a directory where Terraform will pick up any configuration in the current directory and will store the state of the created infrastructure in a file.

The name of the directory can be anything. For this tutorial we are going to name it terraform-aws. Create the directory and cd to it:

$ mkdir terraform-aws
$ cd terraform-aws

Create the following file and name it main.tf. This is the main configuration file for our Terraform project. This configuration will provision an EC2 instance, install Amazon Linux 2 as the OS and install Nginx as the web server. The comments start with a hash #. They describe each section’s function. For simplicity, the configuration is using the default VPC that comes with the selected AWS region.

# Set AWS as the cloud provider
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.16"
    }
  }

  required_version = ">= 1.2.0"
}

# Set AWS region
provider "aws" {
  region = "ap-southeast-1"
}

# Set the default VPC as the VPC
resource "aws_default_vpc" "main" {
  tags = {
    Name = "Default VPC"
  }
}

# Set AWS security group to allow SSH and HTTP
resource "aws_security_group" "ssh_http" {
  name        = "ssh_http"
  description = "Allow SSH and HTTP"
  vpc_id      = aws_default_vpc.main.id

  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"] # make this your IP address or range
  }
  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}


# AWS EC2 configuration
# The user_data contains the script to install Nginx
resource "aws_instance" "app_server" {
  ami           = "ami-0b89f7b3f054b957e"
  instance_type = "t2.micro"
  key_name = "kaptenjeffry"
  vpc_security_group_ids = [aws_security_group.ssh_http.id]
  user_data = <<EOF
   #!/bin/bash
   sudo yum update
   sudo amazon-linux-extras install nginx1 -y
   sudo systemctl start nginx
  EOF

  tags = {
    Name = "Nginx by Terraform"
  }
}

# EC2 Public IP
output "app_server_public_ip" {
  description = "Public IP address of app_server"
  value       = aws_instance.app_server.public_ip
}

Initialize the project

Initialize the project by downloading the required plugin. For this example, it will download the AWS plugin. Initialize the project by executing the following command:

$ terraform init

Validate the configuration file

Check the syntax of the configuration file:

$ terraform validate

Apply the configuration

This will make Terraform create and provision the resources specified in the configuration file. It will ask to review the configuration; answer yes to proceed. Take note of the public IP of the provisioned EC2.

$ terraform apply

Sample output:

Access the provisioned EC2 and Nginx

Use the key_name that is configured in main.tf and the generated public IP address to SSH to the EC2 Instance.

$ ssh -i kaptenjeffry.pem ec2-user@46.137.236.88

Use the generated public IP address in a web browser to access the Nginx service. Please make sure to use http protocol since the Nginx is running on port 80.

Conclusion

That’s all, folks. This is the bare minimum Terraform configuration to quickly deploy an EC2 instance at AWS.

For more cool stuffs you can visit the Terraform main documentation for AWS.

Have a nice day :)

Deploying (Minecraft) Servers Automatically with Terraform

2020-07-16T00:00:00+00:00

Last year I bought an old Dell Optiplex on eBay to use as a dedicated Minecraft server for my friends and me. It worked well for a while, but when my university switched to online classes and I moved home, I left it at my college apartment and was unable to fix it (or retrieve our world save) when it failed for some reason. I still wanted to play Minecraft with friends, though, so I had to figure out a solution in the meantime.

I’d previously used a basic DigitalOcean droplet as a Minecraft server, but that had suffered with lag issues, especially with more than two or three people logged in. Their $5 tier of virtual machine provides 1GB of RAM and 1 CPU core, so it shouldn’t be too much of a surprise that it struggled with a Minecraft server. However, more performant virtual machines cost a lot more, and I wanted to keep my solution as cheap as possible.

I mentioned this to a co-worker and he pointed out that most companies don’t actually charge for virtual machines on a monthly basis; in reality, it’s an hourly rate based on when your virtual machine instance actually exists. So, he suggested I create a virtual machine and start my Minecraft server every time I wanted to play, then shut it down and delete it when I was finished, thus saving the cost of running it when it wasn’t being used.

Of course, you could do this manually in your provider’s dev console, but who wants to manually download dependencies, copy your world over, and set up a new server every time you want to play Minecraft? Not me! Instead, I used Terraform, an open-source tool that lets you describe your desired infrastructure and then sets it up for you.

In this post, I’ll show how I got my server setup streamlined into one Terraform configuration file that creates a virtual machine, runs a setup script on it, copies my Minecraft world to it with rsync, starts the Minecraft server, and adds a DNS entry for your new server.

Picking a provider

As I mentioned earlier, I’ve used DigitalOcean in the past, but at the recommendation of my co-worker, I decided to try UpCloud, a similar service based in Helsinki, Finland. They have datacenters in the US, Europe, and Singapore (the one I used is in Chicago—a full list of locations can be found here). They offer several affordable tiers of virtual machines, including these:

Memory	CPU	Storage	Transfer	Price
1 GB	1	25 GB	1 TB	$5/mo
2 GB	1	50 GB	2 TB	$10/mo
4 GB	2	80 GB	4 TB	$20/mo
8 GB	4	160 GB	5 TB	$40/mo

There are more tiers, but for a Minecraft server I didn’t need anything that powerful. I decided on the $20/month tier—or, in my case, $0.03/hour.

Installing Terraform

The process for setting up Terraform depends on what provider you’re using. If you’d like to use DigitalOcean or another provider, just make sure they’re on Terraform’s list of supported providers. Alternatively (as is the case for UpCloud), some providers aren’t officially supported but do have plugins that work well.

In my case, I followed UpCloud’s instructions for installation here. The rest of that tutorial does a great job showing you how to use Terraform with UpCloud via their plugin, but for instructions more suited to other use cases I’d recommend looking at Terraform’s wealth of learning resources, which can be found here.

You’ll also need to install the UpCloud plugin. This is detailed in the earlier tutorial at UpCloud’s site, so I’ll forgo putting it here.

Creating a Terraform configuration file

Once you’ve got Terraform and UpCloud’s plugin installed, create a directory for your project (mkdir minecraft) and run terraform init in it. Once you’ve done that, using Terraform is as simple as creating a configuration file describing your server. How exactly that looks depends on your provider, but my initial file, which I named server.tf, looked something like this:

provider "upcloud" {
  # Your UpCloud credentials are read from the environment variables
  # export UPCLOUD_USERNAME="Username for UpCloud API user"
  # export UPCLOUD_PASSWORD="Password for UpCloud API user"
}

resource "upcloud_server" "minecraft1" {
  # System hostname
  hostname = "minecraft1"

  # Availability zone
  zone = "us-chi1"

  # Number of CPUs and memory in GB
  plan = "2xCPU-4GB"

  storage_devices {
    # OS root disk size
    size = 25

    # Template UUID for Ubuntu 18.04
    storage = "01000000-0000-4000-8000-000030080200"

    tier   = "maxiops"
    action = "clone"
  }

  # Include at least one public SSH key
  login {
    user = "root"
    keys = [
      "PUBLIC_SSH_KEY_HERE"
    ]
    create_password = false
  }

  # Configuring connection details
  connection {
    host        = self.ipv4_address
    type        = "ssh"
    user        = "root"
    private_key = file("./id_rsa_upcloud")
  }
}

There are a few important sections here:

The hostname (which will be important later)
The zone (Full list here)
The plan (more information on the plan codes here)
The login and connection sections

The last two in particular need some attention; I created an SSH key specifically for managing my Minecraft server and included the public key in the login section and a link to the private key in the connection section.

After providing credentials via environment variables as per UpCloud’s tutorial, you can run terraform plan to see what your current configuration looks like:

zed@zeds-pc:~/minecraft $ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.


------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # upcloud_server.minecraft1 will be created
  + resource "upcloud_server" "minecraft1" {
      + cpu                  = (known after apply)
      + hostname             = "minecraft1"
      + id                   = (known after apply)
      + ipv4                 = true
      + ipv4_address         = (known after apply)
      + ipv4_address_private = (known after apply)
      + ipv6                 = true
      + ipv6_address         = (known after apply)
      + mem                  = (known after apply)
      + plan                 = "2xCPU-4GB"
      + private_networking   = true
      + title                = (known after apply)
      + zone                 = "us-chi1"

      + login {
          + create_password   = false
          + keys              = [
              + "PUBLIC_SSH_KEY_HERE",
            ]
          + password_delivery = "none"
          + user              = "root"
        }

      + storage_devices {
          + action  = "clone"
          + address = (known after apply)
          + id      = (known after apply)
          + size    = 25
          + storage = "01000000-0000-4000-8000-000030080200"
          + tier    = "maxiops"
          + title   = (known after apply)
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.

------------------------------------------------------------------------

Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.

This command shows you what’ll happen when you run terraform apply, which also shows you the changes that are going to happen, after which it prompts you to apply them. We’ll hold off for now, since we need more than a blank Ubuntu server; we need to get our Minecraft world and libraries onto this server.

Adding startup scripts

To get my new virtual machine running a Minecraft server, I needed to perform some initial setup. Specifically, I needed to copy my server files from my local machine, then download the Java runtime and start the server on the newly created VM. To accomplish this, I created a setup script and uploaded it to a private server; you could also use a public GitHub repo or similar and just download it from there. I also had to configure my server.tf to get this script and copy over my Minecraft world after creation.

The sections I added to my upcloud resource block in my server.tf file look like this:

  provisioner "remote-exec" {
    inline = [ 
      "wget http://zedjensen.com/init_minecraft.sh && chmod +x init_minecraft.sh && ./init_minecraft.sh -f && rm init_minecraft.sh"
    ]   
  }

  provisioner "local-exec" {
    command = "rsync --delete -r -e 'ssh -F ./ssh_config -i ./id_rsa_upcloud' server root@${self.ipv4_address}:~/minecraft/"
  }

  provisioner "remote-exec" {
    inline = [ 
      "bash ~/minecraft/start_server.sh"
    ]   
  }

The provisioners remote-exec and local-exec tell Terraform to run these commands on the new virtual machine and my local machine respectively. As you can see, I have the remote machine fetching an init script I wrote and running it, then deleting it. The local exec then copies over my Minecraft server, using the SSH key I created specifically for this, and using an alternate SSH config that uses /dev/null as the known_hosts file, avoiding security warnings on later runs when the machine at minecraft.zedjensen.com isn’t the same machine as it was before. ${self.ipv4_address} is replaced by Terraform with the IP address of the new server. Finally, I run start_server.sh from the files I copied over with rsync.

For reference, the relevant parts of my init script and ssh_config look something like this:

#!/bin/bash

echo "StrictHostKeyChecking accept-new" >> /etc/ssh/ssh_config
service sshd restart

i=0
tput sc
while ! apt install openjdk-8-jdk-headless tmux zip unzip -y &> /dev/null; do
    case $i in
        0 ) j="-" ;;
        1 ) j="\\" ;;
        2 ) j="|" ;;
        3 ) j="/" ;;
    esac
    tput rc echo =en "\r[$j] Waiting to install dependencies..."
    sleep 1
done

mkdir -p minecraft/server

StrictHostKeyChecking no
UserKnownHostsFile /dev/null

Adding a DNS record with Cloudflare

After the last steps, you should be able to get a server up and running, but you’d have to use the IP address to connect to it directly. I have the domain zedjensen.com, so I decided to use minecraft.zedjensen.com. Luckily, Terraform also supports Cloudflare, so I set up Cloudflare as my DNS provider for zedjensen.com and added a new section to my server.tf:

provider "cloudflare" {
  version = "~> 2.0"
  email = "zed@whatever.com"
  # Your CloudFlare API token is also read from environment variables
  # api_token = "MY_API_TOKEN"
}

resource "cloudflare_record" "minecraft1" {
  name = "minecraft.zedjensen.com"
  zone_id = "ZONE_ID_HERE"
  value = upcloud_server.minecraft1.ipv4_address
  type = "A"

  depends_on = [upcloud_server.minecraft1]
}

As you can probably guess, this resource block tells Terraform to add a new A record to my DNS configuration for minecraft.zedjensen.com with my new server’s IP address. The last part, depends_on, tells Terraform to wait until it’s done with the server setup, otherwise it’ll try to do them at the same time, before it knows the IP address for the new VM. Don’t forget to put your API token in your environment variables too (this one would be CLOUDFLARE_API_KEY).

Adding scripts for destruction

Lastly, I needed to make sure that when I was shutting my server down, my Minecraft world was safely copied off first. Terraform supports hooks for shutting down as well! I added a final section to the server block of my server.tf:

server {

  // ...

  provisioner "local-exec" {
    when = destroy
    command = "./copy_back.sh ${self.ipv4_address}"
  }
}

This is similar to the provisioners I added earlier, but it runs only when I’m destroying my server. In addition, if the command fails, Terraform will abort the destruction of the server, giving me a chance to see what went wrong and make a backup manually if needed. The copy_back.sh command I’m using here is just a helper script that copies the world back to my local machine; I also put the entire directory in a zipfile and copy that back for redundancy.

Trying it out

After completing all the above steps, here’s my completed server.tf:

provider "upcloud" {
  # Your UpCloud credentials are read from the environment variables
  # export UPCLOUD_USERNAME="Username for UpCloud API user"
  # export UPCLOUD_PASSWORD="Password for UpCloud API user"
}

provider "cloudflare" {
  version = "~> 2.0"
  email = "zed@whatever.com"
  api_token = "MY_API_TOKEN"
}

resource "upcloud_server" "minecraft1" {
  # System hostname
  hostname = "minecraft1"

  # Availability zone
  zone = "us-chi1"

  # Number of CPUs and memory in GB
  plan = "2xCPU-4GB"

  storage_devices {
    # OS root disk size
    size = 25

    # Template UUID for Ubuntu 18.04
    storage = "01000000-0000-4000-8000-000030080200"

    tier   = "maxiops"
    action = "clone"
  }

  # Include at least one public SSH key
  login {
    user = "root"
    keys = [
      "MY PUBLIC KEY HERE"
    ]
    create_password = false
  }

  # Configuring connection details
  connection {
    host        = self.ipv4_address
    type        = "ssh"
    user        = "root"
    private_key = file("./id_rsa_upcloud")
  }

  provisioner "remote-exec" {
    inline = [
      "wget http://zedjensen.com/init_minecraft.sh && chmod +x init_minecraft.sh && ./init_minecraft.sh -f && rm init_minecraft.sh"
    ]
  }

  provisioner "local-exec" {
    command = "rsync --delete -r -e 'ssh -F ./ssh_config -i ./id_rsa_upcloud' server root@${self.ipv4_address}:~/minecraft/"
  }

  provisioner "remote-exec" {
    inline = [
      "bash ~/minecraft/start_server.sh"
    ]
  }

  provisioner "local-exec" {
    when = destroy
    command = "./get_backup ${self.ipv4_address}"
  }

  provisioner "local-exec" {
    when = destroy
    command = "./run_rsync ${self.ipv4_address}"
  }
}

resource "cloudflare_record" "minecraft1" {
  name = "minecraft.zedjensen.com"
  zone_id = "ZONE_ID_HERE"
  value = upcloud_server.minecraft1.ipv4_address
  type = "A"

  depends_on = [upcloud_server.minecraft1]
}

Now that we’ve defined our server configuration, created scripts to set the remote server up, and added provisioners for startup and takedown, we should be able to run our script and see the magic happen. We’ll use the -out option this time so that we only have to review the configuration once.

zed@zeds-pc:~/minecraft $ terraform plan -out plan

Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.


------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # cloudflare_record.minecraft1 will be created
  + resource "cloudflare_record" "minecraft1" {
      + created_on  = (known after apply)
      + hostname    = (known after apply)
      + id          = (known after apply)
      + metadata    = (known after apply)
      + modified_on = (known after apply)
      + name        = "minecraft.zedjensen.com"
      + proxiable   = (known after apply)
      + proxied     = false
      + ttl         = (known after apply)
      + type        = "A"
      + value       = (known after apply)
      + zone_id     = "ZONE_ID_HERE"
    }

  # upcloud_server.minecraft1 will be created
  + resource "upcloud_server" "minecraft1" {
      + cpu                  = (known after apply)
      + hostname             = "minecraft1"
      + id                   = (known after apply)
      + ipv4                 = true
      + ipv4_address         = (known after apply)
      + ipv4_address_private = (known after apply)
      + ipv6                 = true
      + ipv6_address         = (known after apply)
      + mem                  = (known after apply)
      + plan                 = "2xCPU-4GB"
      + private_networking   = true
      + title                = (known after apply)
      + zone                 = "us-chi1"

      + login {
          + create_password   = false
          + keys              = [
              + "SSH_KEY_HERE"
            ]
          + password_delivery = "none"
          + user              = "root"
        }

      + storage_devices {
          + action  = "clone"
          + address = (known after apply)
          + id      = (known after apply)
          + size    = 25
          + storage = "01000000-0000-4000-8000-000030080200"
          + tier    = "maxiops"
          + title   = (known after apply)
        }
    }

Plan: 2 to add, 0 to change, 0 to destroy.

------------------------------------------------------------------------

This plan was saved to: plan

To perform exactly these actions, run the following command to apply:
    terraform apply "plan"

After reviewing the list of changes to be made, we can apply our changes by following the instructions and running terraform apply plan:

zed@zeds-pc:~/minecraft $ terraform apply plan

upcloud_server.minecraft1: Creating...
upcloud_server.minecraft1: Still creating... [10s elapsed]
upcloud_server.minecraft1: Still creating... [20s elapsed]
upcloud_server.minecraft1: Still creating... [30s elapsed]
upcloud_server.minecraft1: Still creating... [40s elapsed]
upcloud_server.minecraft1: Still creating... [50s elapsed]
upcloud_server.minecraft1: Provisioning with 'remote-exec'...
upcloud_server.minecraft1 (remote-exec): Connecting to remote host via SSH...
upcloud_server.minecraft1 (remote-exec):   Host: 111.222.111.222
upcloud_server.minecraft1 (remote-exec):   User: root
upcloud_server.minecraft1 (remote-exec):   Password: false
upcloud_server.minecraft1 (remote-exec):   Private key: true
upcloud_server.minecraft1 (remote-exec):   Certificate: false
upcloud_server.minecraft1 (remote-exec):   SSH Agent: true
upcloud_server.minecraft1 (remote-exec):   Checking Host Key: false
upcloud_server.minecraft1 (remote-exec): Connected!
upcloud_server.minecraft1 (remote-exec): --2020-07-17 20:36:02--  http://zedjensen.com/init_minecraft.sh
upcloud_server.minecraft1 (remote-exec): Resolving zedjensen.com (zedjensen.com)... 111.222.111.222
upcloud_server.minecraft1 (remote-exec): Connecting to zedjensen.com (zedjensen.com)|111.222.111.222|:80... connected.
upcloud_server.minecraft1 (remote-exec): HTTP request sent, awaiting response... 200 OK
upcloud_server.minecraft1 (remote-exec): Length: 1369 (1.3K) [application/octet-stream]
upcloud_server.minecraft1 (remote-exec): Saving to: 'init_minecraft.sh'

upcloud_server.minecraft1 (remote-exec):       init_   0%       0  --.-KB/s
upcloud_server.minecraft1 (remote-exec): init_minecr 100%   1.34K  --.-KB/s    in 0s

upcloud_server.minecraft1 (remote-exec): 2020-07-17 20:36:03 (205 MB/s) - 'init_minecraft.sh' saved [1369/1369]

upcloud_server.minecraft1 (remote-exec):
upcloud_server.minecraft1: Still creating... [1m0s elapsed]
upcloud_server.minecraft1: Still creating... [1m10s elapsed]
upcloud_server.minecraft1: Still creating... [1m20s elapsed]
upcloud_server.minecraft1: Still creating... [1m30s elapsed]
upcloud_server.minecraft1: Still creating... [1m40s elapsed]
upcloud_server.minecraft1 (remote-exec):
upcloud_server.minecraft1: Still creating... [1m50s elapsed]
upcloud_server.minecraft1: Still creating... [2m0s elapsed]
upcloud_server.minecraft1 (remote-exec):
upcloud_server.minecraft1: Still creating... [2m10s elapsed]
upcloud_server.minecraft1: Provisioning with 'local-exec'...
upcloud_server.minecraft1 (local-exec): Executing: ["/bin/sh" "-c" "rsync --delete -r -e 'ssh -F ./ssh_config -i ./id_rsa_upcloud' server root@111.222.111.222:~/minecraft/"]
upcloud_server.minecraft1 (local-exec): Warning: Permanently added '111.222.111.222' (ECDSA) to the list of known hosts.
upcloud_server.minecraft1: Still creating... [2m20s elapsed]

...

upcloud_server.minecraft1: Still creating... [9m40s elapsed]
upcloud_server.minecraft1: Provisioning with 'remote-exec'...
upcloud_server.minecraft1 (remote-exec): Connecting to remote host via SSH...
upcloud_server.minecraft1 (remote-exec):   Host: 111.222.111.222
upcloud_server.minecraft1 (remote-exec):   User: root
upcloud_server.minecraft1 (remote-exec):   Password: false
upcloud_server.minecraft1 (remote-exec):   Private key: true
upcloud_server.minecraft1 (remote-exec):   Certificate: false
upcloud_server.minecraft1 (remote-exec):   SSH Agent: true
upcloud_server.minecraft1 (remote-exec):   Checking Host Key: false
upcloud_server.minecraft1 (remote-exec): Connected!
upcloud_server.minecraft1: Still creating... [9m50s elapsed]
upcloud_server.minecraft1: Creation complete after 9m51s [id=stuff]
cloudflare_record.minecraft1: Creating...
cloudflare_record.minecraft1: Creation complete after 3s [id=things]

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.

State path: terraform.tfstate

Awesome! It takes a while to transfer files over, but once it’s done, my Minecraft server is up and running! To destroy, it’s just as easy:

zed@zeds-pc:~/minecraft$ terraform destroy
upcloud_server.minecraft1: Refreshing state... [id=things]
cloudflare_record.minecraft1: Refreshing state... [id=stuff]

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  # cloudflare_record.minecraft1 will be destroyed
  - resource "cloudflare_record" "minecraft1" {
      - created_on  = "2020-07-15T22:25:04.460564Z" -> null
      - data        = {} -> null
      - hostname    = "minecraft.zedjensen.com" -> null
      - id          = "stuff" -> null
      - metadata    = {
          - "auto_added"             = "false"
          - "managed_by_apps"        = "false"
          - "managed_by_argo_tunnel" = "false"
          - "source"                 = "primary"
        } -> null
      - modified_on = "2020-07-15T22:25:04.460564Z" -> null
      - name        = "minecraft.zedjensen.com" -> null
      - priority    = 0 -> null
      - proxiable   = true -> null
      - proxied     = false -> null
      - ttl         = 1 -> null
      - type        = "A" -> null
      - value       = "111.222.111.222" -> null
      - zone_id     = "ZONE_ID_HERE" -> null
    }

  # upcloud_server.minecraft1 will be destroyed
  - resource "upcloud_server" "minecraft1" {
      - cpu                  = 2 -> null
      - hostname             = "minecraft1" -> null
      - id                   = "things" -> null
      - ipv4                 = true -> null
      - ipv4_address         = "111.222.111.222" -> null
      - ipv4_address_private = "IPv4 addr" -> null
      - ipv6                 = true -> null
      - ipv6_address         = "IPv6 addr" -> null
      - mem                  = 4096 -> null
      - plan                 = "2xCPU-4GB" -> null
      - private_networking   = true -> null
      - title                = "minecraft1 (managed by terraform)" -> null
      - zone                 = "us-chi1" -> null

      - login {
          - create_password   = false -> null
          - keys              = [
              - "SSH_KEY_HERE"
            ] -> null
          - password_delivery = "none" -> null
          - user              = "root" -> null
        }

      - storage_devices {
          - action      = "clone" -> null
          - address     = "virtio:0" -> null
          - backup_rule = {} -> null
          - id          = "stuff" -> null
          - size        = 25 -> null
          - storage     = "01000000-0000-4000-8000-000030080200" -> null
          - tier        = "maxiops" -> null
          - title       = "terraform-minecraft1-disk-0" -> null
        }
    }

Plan: 0 to add, 0 to change, 2 to destroy.

Do you really want to destroy all resources?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes

cloudflare_record.minecraft1: Destroying... [id=stuff]
cloudflare_record.minecraft1: Destruction complete after 0s
upcloud_server.minecraft1: Destroying... [id=things]
upcloud_server.minecraft1: Provisioning with 'local-exec'...
upcloud_server.minecraft1 (local-exec): Executing: ["/bin/sh" "-c" "./get_backup 209.50.58.100"]
upcloud_server.minecraft1 (local-exec): Warning: Permanently added '209.50.58.100' (ECDSA) to the list of known hosts.
upcloud_server.minecraft1 (local-exec): Warning: Permanently added '209.50.58.100' (ECDSA) to the list of known hosts.
upcloud_server.minecraft1: Still destroying... [id=things, 20s elapsed]
upcloud_server.minecraft1: Provisioning with 'local-exec'...
upcloud_server.minecraft1 (local-exec): Executing: ["/bin/sh" "-c" "./run_rsync 209.50.58.100"]
upcloud_server.minecraft1 (local-exec): Warning: Permanently added '209.50.58.100' (ECDSA) to the list of known hosts.
upcloud_server.minecraft1 (local-exec): Warning: Permanently added '209.50.58.100' (ECDSA) to the list of known hosts.
upcloud_server.minecraft1: Still destroying... [id=things, 30s elapsed]
upcloud_server.minecraft1: Still destroying... [id=things, 40s elapsed]
upcloud_server.minecraft1: Still destroying... [id=things, 50s elapsed]
upcloud_server.minecraft1: Still destroying... [id=things, 1m1s elapsed]
upcloud_server.minecraft1: Destruction complete after 1m7s

Destroy complete! Resources: 2 destroyed.

Terraform proved to be a great solution for my Minecraft dilemma, but this could come in handy for lots of other purposes too. Let us know what you use Terraform for!

Camping in the Clouds with Terraform and Ansible

2019-02-05T00:00:00+00:00

Photo by Andrew E. Larsen · CC BY-ND 2.0

Right, so, show of hands: How many of you work on some bit of web code by doing a git clone to your own laptop, developing the feature or bug fix, running through manual testing of the app until you’re happy with it, and off it goes back up to the repo when done? I’m curious, and I have a few questions for you:

Have you ever had a bit of code that worked locally, but didn’t in production because of some difference in systems, dependencies, or something else in the stack?
How do you show off your work to a client or management for approval? Can you demo several alternate changes to the same site at the same time?
How do you bring in coworkers to “look over your shoulder“ and help with something, especially ones that are far away?
How do you get a new coworker up to speed if they’re doing development themselves?
If you’re working on multiple things, do you create multiple clones?
How’re your backups?

Are you fidgeting nervously thinking about that? Sorry. ☹ But also, check out this little thing: DevCamps. It’s been an End Point staple for quite a while now, so if you’ve read our blog before you might have heard about it.

Long story short: In addition to any local development you do, this system will spin up your own little environments (“camps”) on a remote development server. Each camp includes a checkout of the code, separate httpd/nginx and app processes, and a dedicated database with a clone of the data. What’s that all mean?

Well, figuring that dev server is configured the same, you’ll be working right in a stack identical to production.
Each camp gets its own port, so you can link your beautiful results to someone and they’ll see exactly what the site will look like when deployed.
Bring coworkers in to a shared tmux session for code review or pair programming.
And if they just can’t resist working on the same project they can create their own camp, without having to install dependencies and get the stack operational locally.
Create multiple camps so you can work on new projects when in-flight projects get put on the back burner.
Those server backups your ops team promises are working will automatically preserve your work.
And unicorns become real.

Recently a client brought their app into “The Cloud.“ Kind of. No Kubernetes there, so I can’t use that coveted “k8s” blog post tag yet. It’s still a Java app sitting on a single big VM, but it’s still technically running in the cloud. So we used that opportunity to redo how this “campsite” works and made it cloudy, too. Each camp is a dedicated set of VMs that we spin up (and, for cost savings, shut down) as needed.

We do that using Terraform. We’re already using that for this client’s production systems. Terraform has a concept of workspaces, which basically clones your single set of systems and lets you apply that multiple times over. You can name the workspace pretty much anything, but we’ve stuck with the traditional camp method of just assigning numbers. Thus, each camp is a numbered workspace, with an identical VM for the app and an identical VM for the database. That gets handed over to Ansible for the same provisioning process.

The configuration split is actually a little more complex than that. We have a ‘prod’ and a ‘dev’ module, which is mostly just different instance sizes. Then, combined with appropriate DNS naming parameters, ‘camp’, ‘staging’, and ‘prod’ configs. Just pull from those as needed. I’ll probably take you into that in more depth in a later post.

But, instead of one big server which is a fixed cost no matter how many camps we’re running, each cloud camp adds to the cost only as long as it’s running. So, like I said above, shutting down the camps is a key point of this. We run them on smaller instances generally, but if you run a number of camps at once that cost adds up fast. Shutting down the camps when they’re not needed saves money. There’s a break-even line somewhere in there, and I’ll leave calculating that as an exercise for the reader. But there are other benefits, too:

On traditional camps running on a single system, they all share the same IP address and each gets its own port number. But running on multiple systems with multiple addresses you don’t have to worry about clients with odd egress firewalls that block traffic to nonstandard ports.
Filesystem paths shouldn’t matter inside apps, but sometimes they do, and running camps in their own VMs means the apps can be deployed exactly where they are in production.
Same with database ports: You get to talk to the system Postgres on its default port, and potentially even on a separate host.
We tend to run our monolithic camp servers with SELinux disabled, but with cloud camps you can run with SELinux enforcing in staging environments just like in production.

Cloud camps take you the rest of the way there. You can feel like you’re working and testing right on production without really feeling all that dirty about it. And, when you’re nearly done, you can control the resources assigned to the VMs, for example to switch the sizing configuration so that smaller VMs you use for development briefly become production-sized for load testing. Briefly.

I do love making things open source, but in this case I’m not sure there’s really anything to publish; it’s just the same Terraform configuration with just a little bit of tuning to make workspaces work better. But you’ve heard enough from me today, so I’ll write more about all those details later.

There, I hope I’ve planted some ideas in your mind about how you can change that development methodology for the better. On the whole we work better collaboratively, and camp-like systems for development can really assist that. It can feel weird and cumbersome at first, but the clients we have that have adopted it are generally happy campers.

It’s still early on, but our cloud-based camp experiment is showing lots of promise, too. Keep it in mind when you’re starting a development project.

SRV DNS records in Terraform and Cloudflare

2018-06-26T00:00:00+00:00

(Photo by David Goehring, CC BY 2.0, cropped)

At End Point we are using Terraform for a few clients to manage their web hosting infrastructure as code (IaC). Terraform is particularly helpful when working with multiple cloud or infrastructure providers and stitching together their services.

For example, for one web application that involves failover from the primary production infrastructure to a secondary location at a different provider, we are using Cloudflare as a CDN to provide caching, DDoS mitigation, and traffic routing in front of virtual servers at DigitalOcean and Amazon Web Services (AWS).

We decided we wanted to store all of their infrastructure configuration in Terraform, not just what is required for the web application, so we can recreate their entire infrastructure from their Git repository.

This all went fine until we got to their email DNS records. Our client is using Microsoft Office 365 for their email, which requires some SRV records. Terraform’s Cloudflare provider works fine with the universal MX records, but when we first wanted to do this, the Terraform provider for Cloudflare did not support SRV records at all.

Luckily for us, Terraform recently (6 April 2018) gained support for DNS SRV records as mentioned in the release notes and described in more detail in the pull request that added the feature.

Great! So now we can get on with this.

I began by naively assuming that the SRV record data should be given in space-separated form like many DNS interfaces use, including BIND and Cloudflare’s web interface itself. I tried setting it like this:

resource "cloudflare_record" "_sipfederationtls_tcp" {
  domain = "${var.domain}"
  name   = "_sip._tcp.${var.subdomain}"
  type   = "SRV"
  value  = "100 1 443 sipdir.online.lync.com."
}

But that resulted in an error. So when in doubt, consult the documentation, right? I did that:

Those make it clear that a data element is required, but give no indication what format is needed.

This is not currently documented for Cloudflare’s API, nor for Terraform’s Cloudflare provider. User EpiqSty helpfully recorded asking Cloudflare’s support, who recommended reverse-engineering the format:

In the meantime, what I would suggest is you can create a SRV record via our UI and then you can use the API to do a GET request on the DNS records, that should show you all the fields that are required.

Ok, then!

It turns out that we must provide the component parts of the SRV record disassembled in key/value pairs as the Cloudflare API expects, which looks like this in Terraform config:

resource "cloudflare_record" "_sip_tls" {
  domain = "${var.domain}"
  name   = "_sip._tls.${var.subdomain}"
  type   = "SRV"
  data   = {
    service  = "_sip"
    proto    = "_tls"
    name     = "${var.subdomain}."
    priority = 100
    weight   = 1
    port     = 443
    target   = "sipdir.online.lync.com."
  }
}

One unexpected aspect of this is that it requires duplicating the parts that make up the name although the name doesn’t seem to be used by Cloudflare, which assembles the name from the data components service, proto, and name. The name is apparently just there for Terraform, unlike other DNS records where the name is used as the DNS record value.

The Terraform Cloudflare provider also gained support for the much more obscure DNS LOC record type at the same time, and it works similarly.

Thanks to Ben Vickers who contributed the new feature to Terraform!