Introduction to Terraform with AWS

2022-11-09T00:00:00+00:00

Photo by Jeffry Johar

Terraform is a tool from HashiCorp to enable infrastructure as code (IaC). With it users can define and manage IT infrastructure in source code form.

Terraform is a declarative tool. It will ensure the desired state as defined by the user.

Terraform comes with multiple plugins or providers which enable it to manage a wide variety of cloud providers and technologies such as but not limited to AWS, GCP, Azure, Kubernetes, Docker and others.

This blog will go over how to use Terraform with AWS.

Prerequisites

For this tutorial we will need the following:

An active AWS account.
An internet connection to download required files.
A decent editor such as Vim or Notepad++ to edit the configuration files.

Install AWS CLI

We need to set up the AWS CLI (command-line interface) for authentication and authorization to AWS.

Execute the following command to install the AWS CLI on macOS:

$ curl -O https://awscli.amazonaws.com/AWSCLIV2.pkg
$ sudo installer -pkg AWSCLIV2.pkg -target /

For other OSes see Amazon’s docs.

Execute the following command and enter the AWS Account and Access Keys:

$ aws configure

Install Terraform

We need to install Terraform. It is just a command line tool. Execute the following to install Terraform on macOS:

$ brew tap hashicorp/tap
$ brew install hashicorp/tap/terraform

For other OSes see Terraform’s installation docs.

Create the Terraform configuration file

Before we can create any Terraform configuration file for a project, we need to create a directory where Terraform will pick up any configuration in the current directory and will store the state of the created infrastructure in a file.

The name of the directory can be anything. For this tutorial we are going to name it terraform-aws. Create the directory and cd to it:

$ mkdir terraform-aws
$ cd terraform-aws

Create the following file and name it main.tf. This is the main configuration file for our Terraform project. This configuration will provision an EC2 instance, install Amazon Linux 2 as the OS and install Nginx as the web server. The comments start with a hash #. They describe each section’s function. For simplicity, the configuration is using the default VPC that comes with the selected AWS region.

# Set AWS as the cloud provider
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.16"
    }
  }

  required_version = ">= 1.2.0"
}

# Set AWS region
provider "aws" {
  region = "ap-southeast-1"
}

# Set the default VPC as the VPC
resource "aws_default_vpc" "main" {
  tags = {
    Name = "Default VPC"
  }
}

# Set AWS security group to allow SSH and HTTP
resource "aws_security_group" "ssh_http" {
  name        = "ssh_http"
  description = "Allow SSH and HTTP"
  vpc_id      = aws_default_vpc.main.id

  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"] # make this your IP address or range
  }
  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}


# AWS EC2 configuration
# The user_data contains the script to install Nginx
resource "aws_instance" "app_server" {
  ami           = "ami-0b89f7b3f054b957e"
  instance_type = "t2.micro"
  key_name = "kaptenjeffry"
  vpc_security_group_ids = [aws_security_group.ssh_http.id]
  user_data = <<EOF
   #!/bin/bash
   sudo yum update
   sudo amazon-linux-extras install nginx1 -y
   sudo systemctl start nginx
  EOF

  tags = {
    Name = "Nginx by Terraform"
  }
}

# EC2 Public IP
output "app_server_public_ip" {
  description = "Public IP address of app_server"
  value       = aws_instance.app_server.public_ip
}

Initialize the project

Initialize the project by downloading the required plugin. For this example, it will download the AWS plugin. Initialize the project by executing the following command:

$ terraform init

Validate the configuration file

Check the syntax of the configuration file:

$ terraform validate

Apply the configuration

This will make Terraform create and provision the resources specified in the configuration file. It will ask to review the configuration; answer yes to proceed. Take note of the public IP of the provisioned EC2.

$ terraform apply

Sample output:

Access the provisioned EC2 and Nginx

Use the key_name that is configured in main.tf and the generated public IP address to SSH to the EC2 Instance.

$ ssh -i kaptenjeffry.pem ec2-user@46.137.236.88

Use the generated public IP address in a web browser to access the Nginx service. Please make sure to use http protocol since the Nginx is running on port 80.

Conclusion

That’s all, folks. This is the bare minimum Terraform configuration to quickly deploy an EC2 instance at AWS.

For more cool stuffs you can visit the Terraform main documentation for AWS.

Have a nice day :)

Kubernetes From The Ground Up With AWS EC2

2022-10-06T00:00:00+00:00

Photo by Darry Lin

One way to learn Kubernetes infrastructure is to build it from scratch. This way of learning was introduced by the founding father of Kubernetes himself: Mr. Kelsey Hightower. The lesson is known as “Kubernetes The Hard Way”.

For this blog entry I would like to take a less demanding approach than Kubernetes The Hard Way, while still being educational. I would like to highlight only the major steps in creating a Kubernetes cluster and what is covered in CKA (Certified Kubernetes Administrator) exams. Thus we are going to use the kubeadm tools to build the Kubernetes cluster.

The steps of creating a Kubernetes cluster are hidden to you if you are using a Kubernetes as a service such as AWS EKS, GCP GKE or the enterprise suites of Kubernetes such as Red Hat Openshift or VMware Tanzu. All of these products let you use Kubernetes without the need to worry about creating it.

Prerequisites

For this tutorial we will need the following from AWS:

An active AWS account
EC2 instances with Amazon Linux 2 as the OS
AWS Keys for SSH to access control node and managed nodes
Security group which allows SSH and HTTP
A decent editor such as Vim or Notepad++ to create the inventory and the playbook

EC2 Instances provisioning

Provisioning of the the control plane, a.k.a. the master node:

Go to AWS Console → EC2 → Launch Instances.
Set the Name tag to Master.
Select the Amazon Linux 2 AMI.
Select a key pair. If there are no available key pairs, please create one according to Amazon’s instructions.
Allow SSH and 6443 TCP ports.
Set Number of Instances to 1.
Click Launch Instance.

Provisioning of the worker nodes, a.k.a. the minions:

Go to AWS Console → EC2 → Launch Instances.
Set the Name tag to Node.
Select the Amazon Linux 2 AMI.
Select a key pair. If there are no available key pairs, please create one according to Amazon’s instructions.
Allow SSH TCP port.
Set Number of Instances to 2.
Click Launch Instance.

Installing the container runtime

All Kubernetes nodes require some sort of container runtime engine. For these nodes we are going to use Docker. Log in to all EC2 instances and execute the following:

Install Docker.

sudo yum update -y
sudo amazon-linux-extras install docker -y
sudo usermod -a -G docker ec2-user
sudo service docker start
sudo systemctl enable docker.service
sudo su - ec2-user

Verify the Docker installation.

docker ps

We should get an empty Docker status:

CONTAINER ID   IMAGE          COMMAND                  CREATED       STATUS      PORTS                    NAMES

Install TC (Traffic Controller). This is required by the kubeadm tool.
```
sudo yum install tc -y
```

Kubernetes control plane setup

Add the Kubernetes repository. Log in to the node and paste the following:

cat <<'EOF' | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-$basearch
enabled=1
gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
EOF

Install the Kubernetes binaries for Control Plane (kubelet, kubeadm, kubectl) and enable it.

sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
sudo systemctl enable --now kubelet

Initiate the Control Plane. The --ignore-preflight-errors switch is required because we are using a system which has fewer than 2 CPUs and less than 2 GB of RAM. The --pod-network-cidr value is the default value for flannel (a networking add-on).
```
sudo kubeadm init --ignore-preflight-errors=NumCPU,Mem --pod-network-cidr=10.244.0.0/16
```
There are 3 important points from the output of this command. They are the successful note on the cluster initalization, the kubeconfig setup and the worker node joining string. The following is a sample output:
Create the configuration file for kubectl a.k.a. kubeconfig to connect to the Kubernetes cluster. The scripts are from previous output:
```
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
```

Install the pod network add-on. We are going to use flannel.

kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml

Kubernetes worker nodes setup

Execute the following in all worker nodes:

Add the Kubernetes repository. Log in to the node and paste the following

cat <<'EOF' | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-$basearch
enabled=1
gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm
EOF

Install the Kubernetes binaries for worker nodes (kubelet, kubeadm) and enable kubelet.

sudo yum install -y kubelet kubeadm --disableexcludes=kubernetes
sudo systemctl enable --now kubelet

Execute the join command with sudo. This command is from step #3 in the Kubernetes Control Plane Setup section.

Hello, Kubernetes :)

We have successfully created a Kubernetes cluster. Let’s check on the cluster and try to deploy some sample applications.

Get the latest status of the nodes. You might need to wait a minute or more for all nodes to become Ready.

kubectl get nodes

Sample output:

Deploy a sample Nginx web server

kubectl create deployment mynginx --image=nginx

Scale the Deployment to have 6 replicas and check on where the pods run. The pods should be assigned randomly to the available worker nodes.

kubectl scale --replicas=6 deployment/mynginx
kubectl get pods -o wide

Sample output:

Conclusion

That’s all, folks. I hope this blog entry has shed some insights on what it takes to create a Kubernetes cluster. Have a nice day :)

Ansible tutorial with AWS EC2

2022-08-11T00:00:00+00:00

Photo by David Buchi

Ansible is a tool to manage multiple remote systems from a single command center. In Ansible, the single command center is known as the control node and the remote systems to be managed are known as managed nodes. The following describes the 2 nodes:

Control node:
- The command center where Ansible is installed.
- Supported systems are Unix and Unix-like (Linux, BSD, macOS).
- Python and sshd are required.
- Remote systems to be managed are listed in a YAML or INI file called an inventory.
- Tasks to be executed are defined in a YAML file called a playbook.
Managed node:
- The remote systems to be managed.
- Supported systems are Unix/Unix-like, Windows, and Appliances (eg: Cisco, NetApp).
- Python and sshd are required for Unix/Unix-like.
- PowerShell and WinRM are required for Windows.

In this tutorial we will use Ansible to manage multiple EC2 instances. For simplicity, we are going to provision EC2 instances in the AWS web console. Then we will configure one EC2 as the control node that will be managing multiple EC2 instances as managed nodes.

Prerequisites

For this tutorial we will need the following from AWS:

An active AWS account.
EC2 instances with Amazon Linux 2 as the OS.
AWS Keys for SSH to access control node and managed nodes.
Security group which allows SSH and HTTP.
A decent editor such as Vim or Notepad++ to create the inventory and the playbook.

EC2 Instances provisioning

The following are the steps to provision EC2 instances with the AWS web console.

Go to AWS Console → EC2 → Launch Instances.
Select the Amazon Linux 2 AMI.
Select a key pair. If there are no available key pairs, please create one according to Amazon’s instructions.
Allow SSH and HTTP.
Set Number of Instances to 4.
Click Launch Instance.

Ansible nodes and SSH keys

In this section we will gather the IP addresses of EC2 instances and set up the SSH keys.

Go to AWS Console → EC2 → Launch Instances.
Get the Public IPv4 addresses.
We will choose the first EC2 to be the Ansible control node and the rest to be the managed nodes:
- control node: 13.215.159.65
- managed nodes: 18.138.255.51, 13.229.198.36, 18.139.0.15

ssh -i kaptenjeffry.pem ec2-user@13.215.159.65

Open another terminal and copy the key pair to the control node

scp -i kaptenjeffry.pem kaptenjeffry.pem ec2-user@13.215.159.65:~/.ssh

Go back to the control node terminal. Try to log in from the control node to one of the managed nodes by using the key pair. This is to ensure the key pair is usable to access the managed nodes.

ssh -i .ssh/kaptenjeffry.pem ec2-user@18.138.255.51

ssh-keyscan -t ecdsa-sha2-nistp256 13.229.198.36 18.139.0.15 >> .ssh/known_hosts

Ansible Installation and Configuration

In this section we will install Ansible in the control node and create the inventory file.

In the control node, execute the following commands to install Ansible:
```
sudo yum update
sudo amazon-linux-extras install ansible2
ansible --version
```
Where:
- yum update updates all installed packages using the yum package manager,
- amazon-linux-extras install installs Ansible, and
- ansible --version checks the installed version of Ansible.
Create a file named myinventory.ini. Insert the IP addresses that we identified earlier to be the managed nodes in the following format:

[mynginx]
red ansible_host=18.138.255.51
green ansible_host=13.229.198.36
blue ansible_host=18.139.0.15

Where:

[mynginx] is the group name of the managed nodes,
red, green, and blue are the aliases of each managed node, and
ansible_host=x.x.x.x sets the IP Address each managed node.

myinventory.ini is a basic inventory file in a INI format. An inventory file could be either in INI or YAML format. For more information on inventory see the Ansible docs.

Ansible modules and Ansible ad hoc commands

Ansible modules are scripts to do a specific task at managed nodes. For example, there are modules to check availability, copy files, install applications, and lots more. To get the full list of modules, you can check the official Ansible modules page.

A quick way to use Ansible modules is with an ad hoc command. Ad hoc commands use the ansible command-line interface to execute modules at the managed nodes. The usage is as follows:

ansible <pattern> -m <module> -a "<module options>" -i <inventory>

Where:

<pattern> is the IP address, hostname, alias or group name,
-m module is name of the module to be used,
-a "<module options>" sets options for the module, and
-i <inventory> is the inventory of the managed nodes.

Ad hoc command examples

The following are some example of Ansible ad hoc commands:

ping checks SSH connectivity and Python interpreter at the managed node. To use the ping module against the mynginx group of servers (all 3 hosts: red, green, and blue), run:

ansible mynginx -m ping -i myinventory.ini

copy copies files to a managed node. To copy a text file (/home/ec2-user/hello.txt in our test case) from the Control node to /tmp/ at all managed nodes in the mynginx group, run:

ansible mynginx -m copy \
-a 'src=/home/ec2-user/hello.txt dest=/tmp/hello.txt' \
-i myinventory.ini

shell executes a shell script at a managed node. To use module shell to execute uptime at all managed nodes in the mynginx group, run:

ansible mynginx -m shell -a 'uptime' -i myinventory.ini

Ansible playbooks

Ansible playbooks are configuration files in a YAML format that tell Ansible what to do. A playbook executes its assigned tasks sequentially from top to bottom. Tasks in playbooks are grouped by a block of instructions called a play. The following diagram shows the high level structure of a playbook:

Now we are going to use a playbook to install Nginx at our three managed nodes as depicted in the following diagram:

Create the following YAML file and name it nginx-playbook.yaml. This is a playbook with one play that will install and configure Nginx service at the managed node.

---
- name: Installing and Managing Nginx Server 
  hosts: mynginx   
  become: True
  vars:
    nginx_version: 1
    nginx_html: /usr/share/nginx/html
    user_home: /home/ec2-user
    index_html: index.html
  tasks:
    - name: Install the latest version of nginx
      command: amazon-linux-extras install nginx{{ nginx_version }}=latest -y

    - name: Start nginx service
      service:
        name: nginx
        state: started

    - name: Enable nginx service
      service:
         name: nginx
         enabled: yes
    - name: Copy index.html to managed nodes
      copy:
        src:  "{{ user_home }}/{{ index_html }}"
        dest: "{{ nginx_html }}"

Where:

name (top most) is the name of this play,
hosts specifies the managed nodes for this play,
become says whether to use superuser privilege (sudo for Linux),
vars defines variables for this play,
tasks is the start of the task section,
name (under task section) specifies the name of each task, and
name (in a service section) specifies the name of a module.

Let’s try to execute this playbook. Firstly we need to create the source index.html to be copied to managed nodes.

echo 'Hello World!' > index.html

Execute ansible-playbook against our playbook. Just like the ad hoc command, we need to specify the inventory with the -i switch.

ansible-playbook nginx-playbook.yaml -i myinventory.ini

Now we can curl our managed nodes to check on the Nginx service and the custom index.html.

curl 18.138.255.51
curl 13.229.198.36
curl 18.139.0.15

Conclusion

That’s all, folks. We have successfully managed EC2 instances with Ansible. This tutorial covered the fundamentals of Ansible to start managing remote servers.

Ansible rises above its competitors due to its simplicity of its installation, configuration, and usage. To get further information about Ansible you may visit its official documentation.

How to deploy a Django App with Aurora Serverless and AWS Copilot

2022-06-26T00:00:00+00:00

Photo by Виктор Куликов

AWS Copilot has the capability to provision an external database for its containerized work load. The database options are DynamoDB (NoSQL), Aurora Serverless (SQL), and S3 Buckets. For this blog we are going to provision and use Aurora Serverless with a containerized Django app. Aurora Serverless comes with 2 options for its engine: MySQL or PostgreSQL.

Watch Amazon’s 2-minute introduction video to get the basic idea of Aurora Serverless.

We are going to work with the same Django application from my last article on AWS Copilot.

In my last article, the Django application was deployed with SQLite as the database. The application’s data is stored in SQLite which resides internally inside the container. The problem with this setup is the data is not persistent. Whenever we redeploy the application, the container will get a new filesystem. Thus all old data will be removed automatically.

Now we are moving away the application’s data externally so that the life of the data does not depend on the container. We are going to put the data on the Aurora Serverless with PostgreSQL as the engine.

Django with SQLite as the internal database

Django with Aurora Serverless as the external database

The Prerequisites

Docker, AWS CLI, and AWS Copilot CLI are required. Please refer to my last article for how to install them.

The Django Project

Create a Django project by using a Python Docker Image. You can clone my Git project to get the Dockerfile, docker-compose.yaml and requirements.txt that I’m using:

$ git clone https://github.com/aburayyanjeffry/django-copilot.git django-aurora

Go to the django-aurora directory and execute docker-compose to create a Django project named “mydjango”.

$ cd django-aurora
$ docker-compose run web django-admin startproject mydjango

The Deployment with AWS Copilot

Execute the following command to create a AWS Copilot application with the name of “mydjango”, a load balancer container with the service name “django-web” which is made from the Dockerfile in the current directory.

$ copilot init \
-a mydjango \
-t "Load Balanced Web Service" -n django-web \
-d ./Dockerfile

Answer N to the following question. We want to defer the deployment until we have set up the database.

All right, you're all set for local development.

 Would you like to deploy a test environment? [? for help] (y/N) N

We need to create an environment for our application. Execute the following to create an environment named test for the “mydjango” application with the default configuration.

$ copilot env init \
--name test \
--profile default \
--app mydjango \
--default-config

Now we are going to generate a config for our Aurora Serverless database. Basically this is the CloudFormation template that will be used for Aurora Serverless.

Execute the following to generate the configuration for an Aurora cluster named “mydjango-db” that we will use for the “django-web” application. The Aurora cluster will be using the PostgreSQL engine and the database name will be “mydb”.

$ copilot storage init \
-n mydjango-db \
-t Aurora -w \
django-web \
--engine PostgreSQL \
--initial-db mydb

Take note of the injected environment variable name. This is where the database info and credentials are stored, and we will use this variable in later steps.

✔ Wrote CloudFormation template at copilot/django-web/addons/mydjango-db.yml

Recommended follow-up actions:
 - Update django-web's code to leverage the injected environment variable MYDJANGODB_SECRET.

Edit mydjango/settings.py to include the following. We will pass the injected environment variable we got previously to the function for getting the DBINFO variables.

from pathlib import Path
import os
import json
...
ALLOWED_HOSTS = ['*']
...
DBINFO = json.loads(os.environ.get('MYDJANGODB_SECRET', '{}'))
DATABASES = {
   'default': {
       'ENGINE': 'django.db.backends.postgresql',
       'HOST': DBINFO['host'],
       'PORT': DBINFO['port'],
       'NAME': DBINFO['dbname'],
       'USER': DBINFO['username'],
       'PASSWORD': DBINFO['password'],
   }
}

Deploy the application:

$ copilot deploy --name django-web

Open the terminal of the service:

$ copilot svc exec

Execute the following commands to migrate the initial database and to create a superuser account:

$ python manage.py migrate
$ python manage.py createsuperuser

Execute the following command to check on the environment variable. Take note of the MYDJANGODB_SECRET variable. It is the variable that holds the database information.

$ env | grep MYDJANGODB_SECRET

How to query Aurora Serverless

We can use the Query Editor at AWS Console for RDS to query Aurora Serverless.

Click the DB base on the DB identifier from the injected environment variable and click Modify.

Click the check box for Data API.

Select Apply Immediately.

Click Query Editor and fill in the Database information from the injected environment variable.

Now you may use the Query Editor to query the database. Execute the following query to list all tables in the database:

The End

That’s all, folks. We have deployed a containerized Django application and an Aurora Serverless with AWS Copilot. For further info on AWS Copilot visit its website.

How to deploy a containerized Django app with AWS Copilot

2022-06-21T00:00:00+00:00

Generally there are 2 major options at AWS when it comes to deployment of containerized applications. You can either go for EKS or ECS.

EKS (Elastic Kubernetes Service) is the managed Kubernetes service by AWS. ECS (Elastic Container Service), on the other hand, is AWS’s own way to manage your containerized application. You can learn more about EKS and ECS on the AWS website.

For this post we will use ECS.

The chosen one and the sidekick

With ECS chosen, now you have to find a preferably easy way to deploy your containerized application on it.

There are quite a number of resources from AWS that are needed for your application to live on ECS, such as VPC (Virtual Private Cloud), Security Group (firewall), EC2 (virtual machine), Load Balancer, and others. Creating these resources manually is cumbersome so AWS has came out with a tool that can automate the creation of all of them. The tool is known as AWS Copilot and we are going to learn how to use it.

Install Docker

Docker or Docker Desktop is required for building the Docker image later. Please refer to my previous article on how to install Docker Desktop on macOS, or follow Docker’s instructions for Linux and Windows.

Set up AWS CLI

We need to set up the Docker AWS CLI (command-line interface) for authentication and authorization to AWS.

Execute the following command to install the AWS CLI on macOS:

$ curl -O "https://awscli.amazonaws.com/AWSCLIV2.pkg"
$ sudo installer -pkg AWSCLIV2.pkg -target /

For other OSes see Amazon’s docs.

Execute the following command and enter the AWS Account and Access Keys.

$ aws configure

Install AWS Copilot CLI

Now it’s time for the main character: AWS Copilot.

Install AWS Copilot with Homebrew for macOS:

$ brew install aws/tap/copilot-cli

See AWS Copilot Installation for other platforms.

The Django project

Create a Django project by using a Python Docker Image. You can clone my Git project to get the Dockerfile, docker-compose.yaml and requirements.txt that I’m using.

$ git clone https://github.com/aburayyanjeffry/django-copilot.git

Go to the django-pilot directory and execute docker-compose to create a Django project named “mydjango”.

$ cd django-copilot
$ docker-compose run web django-admin startproject mydjango .

Edit mydjango/settings.py to allow all hostnames for its URL. This is required because by default AWS will generate a random URL for the application. Find the following variable and set the value as follows:

ALLOWED_HOSTS = ['*']

The Deployment with AWS Copilot

Create an AWS Copilot “Application”. This is a grouping of services such as web app or database, environments (development, QA, production), and CI/CD pipelines. Execute the following command to create an Application with the name of “mydjango”.

$ copilot init -a mydjango

Select the Workload type. Since this Django is an internet-facing app we will choose the “Load Balanced Web Service”.

Which workload type best represents your architecture?  [Use arrows to move, type to filter, ? for more help]
    Request-Driven Web Service  (App Runner)
  > Load Balanced Web Service   (Internet to ECS on Fargate)
    Backend Service             (ECS on Fargate)
    Worker Service              (Events to SQS to ECS on Fargate)
    Scheduled Job               (Scheduled event to State Machine to Fargate)

Give the Workload a name. We are going to name it “mydjango-web”.

Workload type: Load Balanced Web Service

  What do you want to name this service? [? for help] mydjango-web

Select the Dockerfile in the current directory.

Which Dockerfile would you like to use for mydjango-web?  [Use arrows to move, type to filter, ? for more help]
  > ./Dockerfile
    Enter custom path for your Dockerfile
    Use an existing image instead

Accept to create a test environment.

All right, you're all set for local development.

  Would you like to deploy a test environment? [? for help] (y/N) y

Wait and see. At the end of the deployment you will get the URL of your application. Open it in a browser.

Now let’s migrate some data, create a superuser, and try to log in. The Django app comes with a SQLite database. Execute the following command to get a terminal for the Django app:

$ copilot svc exec

Once in the terminal, execute the following to migrate the initial data and to create the superuser:

$ python manage.py migrate
$ python manage.py createsuperuser

Now you may access the admin page and login by using the created credentials.

You should see the Django admin:

A mini cheat sheet

AWS Copilot commands	Remarks
`copilot app ls`	To list available Applications
`copilot app show -n appname`	To get the details of an Application
`copilot app delete -n appname`	To delete an Application
`copilot svc ls`	To list available Services
`copilot svc show -n svcname`	To get the details of a Service
`copilot svc delete -n svcname`	To delete a Service

The End

That’s all, folks.

AWS Copilot is a tool to automate the deployment of AWS infrastructure for our containerized application needs. It takes away most of the worries about infrastructure and enables us to focus sooner on the application development.

For further info on AWS Copilot visit its website.

Generate PDF with Chrome, Puppeteer, and Serverless Stack

2022-01-02T00:00:00+00:00

Function as a Service (FaaS) solutions are becoming more and more mainstream today. The FaaS model allows developers to not have to worry about managing infrastructure and instead focus on writing the application logic. In the FaaS model, developers write individual functions that run specific tasks that are deployed together on a FaaS platform, including but not limited to Amazon Web Services (AWS), Azure, and Google Cloud Platform (GCP).

These functions don’t always run like a traditional application server does, waiting for a request. Instead, these FaaS platforms only spin up instances of these functions whenever there is traffic and will shut them back down once there are no more requests after a given period of time. This helps make FaaS a really cheap platform while traffic is low. It is a good approach for on-demand tasks that are part of the application but not necessarily the most common path in a customer’s everyday journey within the application.

The tools

In this tutorial, we’re going to be looking at implementing PDF download with Chrome, Puppeteer, and Serverless Stack, but first, let’s have a brief introduction of the tools that we are going to be using.

1. Serverless Stack

Serverless Stack is a framework for building full-stack serverless apps. The bigger player in this space is the Serverless framework. Serverless Stack is giving the Serverless framework (yes, I know, it can be very confusing at times) a challenge to the throne. The latter has been around for many years now and has been the authority for building serverless apps. There are pros and cons to both of them.

Serverless Stack’s biggest advantage is the live Lambda debugging, a faster development process and overall better development experience. With the Serverless framework, developers have to either constantly deploy to the cloud to test any change, or spend some time setting up serverless-offline for simulated Lambda and API gateway environments on their local machines.

But with the Serverless framework, developers are not tied to using a specific platform, as it can be used to deploy functions on AWS, Azure, and Google Cloud Platform. Serverless Stack, on the other hand, only allows deploying to AWS since its internals are built on top of AWS CDK.

2. Chrome and Puppeteer

Chrome will be used in this tutorial as a headless browser, controlled using Puppeteer. Puppeteer is a Node.js package that provides an API to make Chrome or Chromium navigate to a specific URL, take screenshots, click buttons, type in input fields, as well as take PDF screenshots. Note that the PDF screenshot feature is specific to Chrome and not Chromium, so make sure to double check that you are using the right browser.

Implementing PDF screenshot and download with Serverless Stack, Chrome, and Puppeteer

Now, let’s get to the meat of this tutorial. We are going to implement PDF generation with Chrome and Puppeteer, deployed on AWS Lambda through Serverless Stack. Here is what we will do:

Use Puppeteer in Chrome to navigate to a receipt page that we implemented.
Instruct Chrome to take a PDF screenshot of the page.
Stream the PDF file back to the user.
Deploy this implementation on AWS Lambda using Serverless Stack and demonstrate the development experience throughout.

PDF generation is a great use case for FaaS. It represents something that doesn’t happen very often and is triggered on demand by the user. Instead of letting our monolith application handle that intermittent traffic, we can delegate the task to FaaS in the cloud which will do this job without worrying us about scaling the infrastructure to cater to this particular use case. Then our application can just focus on handling the regular web traffic for general use.

Prerequisite: Sample receipt pages

First, let’s create some demo pages for PDF screenshots. I’m not going to go in-depth on these since it’s not the topic of the day. I have deployed these simple HTML pages that are styled like receipts on this Surge instance here. You can find the code on GitHub here.

Getting started

Let’s generate a new Serverless Stack project.

$ npx create-serverless-stack@latest pdf-generator
$ cd pdf-generator
$ npm install

Congratulations, we have just created a new project with Serverless Stack! Here we get a basic template of a working Serverless Stack app. You can go here for more information on how to get started with Serverless Stack.

First, let’s look at the files we get from bootstrapping the project and customize them wherever we see fit.

1. sst.json

{
  "name": "pdf-generator",
  "region": "us-east-1",
  "main": "stacks/index.js"
}

This file is the entry point to the Serverless Stack app, and here we can define the region of choice, the name of the app, and the main file that Serverless Stack will use.

2. stacks/index.js

 // stacks/index.js
 app.setDefaultFunctionProps({
   runtime: "nodejs12.x"
 });

-new MyStack(app, "my-stack");
+new MyStack(app, "pdf-generator");

stacks/index.js is the main file declared in sst.json. When building the project, Serverless Stack will use this file as an entry point to our application. This file is pretty simple:

It configures the Lambda functions to use the Node.js 12.x runtime.
It registers a stack called my-stack. This is the name of the CloudFormation stack on AWS. We definitely want to give it a better name than my-stack here, so let’s rename that to pdf-generator.

3. stacks/MyStack.js

import * as sst from "@serverless-stack/resources";

export default class MyStack extends sst.Stack {
  constructor(scope, id, props) {
    super(scope, id, props);

    // Create a HTTP API
    const api = new sst.Api(this, "Api", {
      routes: {
        "GET /": "src/lambda.handler",
      },
    });

    // Show the endpoint in the output
    this.addOutputs({
      "ApiEndpoint": api.url,
    });
  }
}

MyStack.js is where we declare the resources we need inside a given CloudFormation stack. This can be anything from a cluster of Lambda functions, API Gateway endpoints, DynamoDB tables, S3 buckets, etc. The full list of resources that we can create with Serverless Stack is listed here in the documentation. For our project we just need one API endpoint with API gateway, so what we are provided with here is already sufficient.

Deploying the project

Now, we can try to deploy this project to AWS. Ensure that the AWS Access Key ID and AWS Secret Access Key are set in the development environment, and then run npx sst start.

# export AWS credentials
export AWS_ACCESS_KEY_ID=<access key id>
export AWS_SECRET_ACCESS_KEY=<secret access key>

# or if the credentials are set in ~/.aws/credentials
export AWS_PROFILE=<AWS profile name>

# Start the app in development mode
npx sst start

After a few minutes, we will have the Cloudformation stack deployed on our AWS account. Keep in mind that the first deployment will take a bit longer than subsequent deployments. When the deployment is done, we should be able to see that it is watching for file changes from our local machine.

threadlightly-pdf-generator-pdf-generator | CREATE_COMPLETE | AWS::ApiGatewayV2::Route | ApiRouteGET8AC7D3F8
threadlightly-pdf-generator-pdf-generator | CREATE_COMPLETE | AWS::Lambda::Permission | ApiRouteGETthreadlightlypdfgeneratorpdfgeneratorApiRouteGET7230D6CDPermissionE4542537
threadlightly-pdf-generator-pdf-generator | CREATE_COMPLETE | AWS::CloudFormation::Stack | threadlightly-pdf-generator-pdf-generator

 ✅  threadlightly-pdf-generator-pdf-generator


Stack threadlightly-pdf-generator-pdf-generator
  Status: deployed
  Outputs:
    ApiEndpoint: https://<yourapigatewayurl>.execute-api.us-east-1.amazonaws.com


==========================
 Starting Live Lambda Dev
==========================

Transpiling Lambda code...
Debug session started. Listening for requests...

Let’s head to the API URL given in the output and we should get a response back like this:

Hello, World! Your request was received at 30/Nov/2021:06:21:13 +0000.

Developing with Serverless Stack

Let’s try updating one of the files provided to see how we can test the changes we have made and will make.

 // stacks/MyStack.js
 const api = new sst.Api(this, "Api", {
   routes: {
-    "GET /": "src/lambda.handler",
+    "GET /downloads/receipt": "src/lambda.handler",
   },
 });

 // stacks/index.js
   runtime: "nodejs12.x"
 });

-new MyStack(app, "my-stack");
+new MyStack(app, "pdf-generator");

Here we did two things:

We updated the API endpoint to GET /downloads/receipt instead of just GET /.
We made the Lambda function return the URL given in the query string.

Once we save these changes we should see Serverless Stack automatically reloading our code and since we changed the API endpoint path, Serverless Stack will need to make changes to our infrastructure (i.e. the API Gateway resources), and whenever it detects that it has to change the infrastructure of our application, it will immediately prompt us to redeploy.

Rebuilding code...
Rebuilding infrastructure...
Done building code
Press ENTER to redeploy infrastructure

Once we hit ENTER, Serverless Stack will automatically update our infrastructure for us: delete our old endpoint, create a new endpoint on GET /downloads/receipt, and hook up our src/lambda.js file to handle that endpoint.

When it’s ready, try hitting the API gateway URL again, but this time append /downloads/receipt?url=https://google.com to it.

// https://<yourapigatewayurl>.execute-api.us-east-1.amazonaws.com/downloads/receipt?url=https://google.com
Hello! You've requested to print the receipt at page https://google.com

Great, now we have learned how to make changes to our Serverless Stack application during development. Let’s go ahead and add Puppeteer and Chrome to our Lambda function.

 // stacks/MyStack.js
 import * as sst from "@serverless-stack/resources";
+import { LayerVersion } from "@aws-cdk/aws-lambda";
+
+const layerArn = "arn:aws:lambda:us-east-1:764866452798:layer:chrome-aws-lambda:25";

 // Create a HTTP API
 const api = new sst.Api(this, "Api", {
   routes: {
-     "GET /downloads/receipt": "src/lambda.handler",
+     "GET /downloads/receipt": {
+       function: {
+         handler: "src/lambda.handler",
+         // Increase the timeout for generating screenshots
+         timeout: 15,
+         // Load Chrome in a Layer
+         layers: [LayerVersion.fromLayerVersionArn(this, "Layer", layerArn)],
+         // Exclude bundling it in the Lambda function
+         bundle: { externalModules: ["chrome-aws-lambda"] },
+       }
+     },
   },
 });

Here, we are configuring our AWS Lambda function to use a Lambda layer that includes Chrome in our Lambda functions. You can think of these Lambda layers as being similar to NPM packages that you pull for extending your projects except that this is a 3rd party package for extending AWS Lambda functions instead.

With this layer, your AWS Lambda function will boot with the Chrome binary already included. This layer is maintained here. Make sure you’re using the ARN from the right region. In this tutorial, we are using us-east-1 per Serverless Stack’s default, so we are going to pick the ARN for us-east-1.

Now, let’s update our Lambda handler to:

Use Puppeteer to instruct Chrome to navigate to the URL passed in the parameter.
Take a screenshot of the web page and save a PDF out of that. Puppeteer provides a handy page.pdf API to do just that.
Ultimately, stream that PDF file back to the user.

Here’s the code for that, with some comments to help you out:

+import chrome from "chrome-aws-lambda";
+
+const puppeteer = chrome.puppeteer;
+
 export async function handler(event) {
-  return {
-    statusCode: 200,
-    headers: { "Content-Type": "text/plain" },
-    body: `Hello! You've requested to print the receipt at page ${event.queryStringParameters.url}`,
-  };
+  let browser
+  let response
+  try {
+    const { url } = event.queryStringParameters;
+
+    browser = await puppeteer.launch({
+      args: chrome.args,
+      executablePath: await chrome.executablePath,
+    });
+
+    const page = await browser.newPage();
+    // Use A5 size at 150dpi
+    const width = 874
+    const height = 1240
+
+    await page.setViewport({ width, height });
+
+    // Navigate to the url
+    await page.goto(url, { waitUntil: 'networkidle2' });
+
+    // Take the screenshot
+    await page.pdf({path: 'receipt.pdf', width: width + "px", height: height + "px", printBackground: true});
+
+    response = {
+      statusCode: 200,
+      body: JSON.stringify({message: 'Screenshot taken'})
+    };
+  } catch(err) {
+    response = {
+      statusCode: 500,
+      body: JSON.stringify({message: `An error occured. ${err.message}`})
+    }
+  } finally {
+    await browser && browser.close()
+  }
+  return response
 }

Let’s also not forget to install Puppeteer and Chrome.

$ npm install puppeteer puppeteer-core chrome-aws-lambda

Now, give Serverless Stack some time to reload our new changes. Once we have the API endpoint ready to go, let’s head to the browser and hit our API endpoint, passing the URL to the sample receipt page that we did earlier as a query param.

// Visit https://<yourapigatewayurl>.execute-api.us-east-1.amazonaws.com/downloads/receipt?url=https://unwieldy-key.surge.sh/index.html
// Response:
{"message":"Screenshot taken"}

Great! We are getting back a 200 status code and a message saying “Screenshot taken”. That means our code works and there were no exceptions. We’re halfway there.

Now, if we check the project folder, we should now see a new file called receipt.pdf there. This is the receipt.pdf captured by Puppeteer. We can open this file and verify that this is the demo receipt page that I deployed earlier that we passed to the endpoint.

We can also hit the API endpoint with the second sample receipt URL (i.e. url=https://unwieldy-key.surge.sh/receipt-2.html) and we should see the receipt.pdf file gets replaced with the receipt from the new URL.

Note that in a real-world application, the URL you would want to screenshot might need some form of authentication, so you are going to have to figure that out for your application. In this case the sample receipt page is accessible publicly, so we don’t run into this problem, but chances are your users’ receipt page will not be accessible publicly like this.

One way you could solve this is to create a special user account that has permission to visit these authenticated pages for this purpose. Then, prior to visiting the receipt page, you’d program Puppeteer to first log in as the special user and then head to the receipt page URL to be screenshot.

However, we are only halfway there. We can see the receipt.pdf file now only because the receipt.pdf file was made available to us by Serverless Stack, which automatically downloads that file to our local machine in development mode. In a real AWS Lambda execution, the file will remain in the filesystem of that Lambda execution context instead, and the user will not see it.

Hence we have to read and stream that file back to the user instead of returning a JSON message saying “Screenshot taken”. Getting a 200 status code is a great start, but that still does not give our users the actual PDF that they wanted, so let’s update our Lambda handler to do just that:

 // src/lambda.js
+import * as fs from "fs";

 // Take the screenshot
 await page.pdf({path: 'receipt.pdf', width: width + "px", height: height + "px", printBackground: true});
+const buffer = fs.readFileSync('receipt.pdf')

 response = {
   statusCode: 200,
-  body: JSON.stringify({message: 'Screenshot taken'})
+  headers: {'Content-type' : 'application/pdf'},
+  body: buffer.toString('base64'),
+  isBase64Encoded : true,
 };

The changes this time are pretty simple:

Import the fs module from Node.js, which will give us access to readFileSync API that can read files on the disk.
Read the generated receipt.pdf file as a Buffer.
Return the file’s Buffer as a base64 string in the response body.
Set the content-type to be application/pdf. Setting the content-type appropriately will tell the browser what to do with the file. Some browsers will display the PDF in an in-browser PDF reader, while others will download the file.

That’s it! Now let’s test the change. Hit the same URL again.

Bingo, the endpoint now returns the PDF to the user! Now we can integrate this endpoint with our main application to delegate these on-demand PDF generation to managed FaaS infrastructure.

Conclusion

FaaS platforms are a really great way to offload intermittent and on-demand traffic from our main application. This ensures our application runs smoother and can serve our general customers better, while at the same time handling these on-demand tasks really well.

Caveats

Building a full-blown application on the stack is also possible, but in my personal experience, it comes with its own complexities and tradeoffs that may or may not be worth the engineering time.

For example, databases are an issue with going full-blown development on FaaS. Because FaaS can scale indefinitely, the number of connections to the database is a limitation.

You can switch to FaaS-friendly databases such as DynamoDB or FaunaDB, but these are NoSQL databases and require a different mindset in structuring the application data than the simpler relational databases and that can cost tremendous engineering time. Relational databases are easier to work with for most types of applications.

Nonetheless, more and more tools are being developed to tackle these complexities, such as PlanetScale and Serverless Aurora v2 (still in preview). All in all, it is definitely a trend worth following closely.

Facial Recognition Using Amazon DeepLens: Counting Liquid Galaxy Interactions

2019-05-01T00:00:00+00:00

I have been exploring the possible uses of a machine-learning-enabled camera for the Liquid Galaxy. The Amazon Web Services (AWS) DeepLens is a camera that can receive and transmit data over wifi, and that has computing hardware built in. Since its hardware enables it to use machine learning models, it can perform computer vision tasks in the field.

The Amazon DeepLens camera

This camera is the first of its kind—likely the first of many, given the ongoing rapid adoption of Internet of Things (IoT) devices and computer vision. It came to End Point’s attention as hardware that could potentially interface with and extend End Point’s immersive visualization platform, the Liquid Galaxy. We’ve thought of several ways computer vision could potentially work to enhance the platform, for example:

Monitoring users’ reactions
Counting unique visitors to the LG
Counting the number of people using an LG at a given time

The first idea would depend on parsing facial expressions. Perhaps a certain moment in a user experience causes people to look confused, or particularly delighted—valuable insights. The second idea would generate data that could help us assess the platform’s impact, using a metric crucial to any potential clients whose goals involve engaging audiences. The third idea would create a simpler metric: the average number of people engaging with the system over a period of time. Nevertheless, this idea has a key advantage over the second: it doesn’t require distinguishing between people, which makes it a much more tractable project. This post focuses on the third idea.

To set up the camera, the user has to plug it into a power outlet and connect it to wifi. The camera will still work even with a slow network connection, though when the connection is slower the delay between the camera seeing something and reporting it is longer. However, this delay was hardly noticable on my home network which has slow-to-moderate speeds of about 17 Mbps down and 33 Mbps up.

Computer Vision and the Amazon DeepLens

A deep learning model is a neural network with multiple layers of processing units. It is called “deep” because it has multiple layers. The inputs and outputs of each processing unit are numbers. These units are roughly analogous to neurons: they receive input from units in the previous layer, and output it to units in the next layer after transforming it based on a function. These “activation functions” can change in a variety of ways. The last layer’s outputs translate into the results. These models work because these functions get tuned based on how well the model works. For example, to make a model that labels each human face in a picture and draws a box around it, we would start with a corpus of pictures with boxes drawn around faces, as well as the versions of the pictures without the boxes drawn. We would test the model on the non-labeled images by checking—for each picture—whether the output generated by the model is correct. If not, the computer chooses different unit functions, tries again, and compares the results. Repeating this process thousands of times yields models which work remarkably well for a wide range of tasks, including computer vision.

In deep learning for computer vision, training on large sets of labeled images enables models to generalize about visual characteristics. The training process takes a lot of computing resources, but once models are trained, they can produce results quickly and with relative ease. This is why the DeepLens is able to perform computer vision with its limited computing resources.

Since the DeepLens is an Amazon product, it comes as no surprise that the user interface and backend for DeepLens consist of AWS services. One of the most important is SageMaker, which is used to train, manage, optimize, and deploy machine learning models such as neural networks. It includes hosted Jupyter notebooks (Jupyter is a development environment for data science), as well as the computing resources required for model training and storage. With SageMaker, users can train computer vision models for deployment to DeepLens, or import and adjust pretrained models from various sources.

Remote management of the DeepLens depends on AWS Lambda, a “serverless” cloud service that provides an environment to run backend code and integrate with other cloud services. It runs the show, allowing users to manage everything from the camera’s behavior to what happens to gathered data. Another service, AWS Greengrass, connects the instructions from AWS Lambda to the DeepLens, managing tasks like authentication, updates, and reactions to local events.

Amazon’s IoT service saves information about each DeepLens, and allows users to manage their devices, for example by choosing which model is active on the device, or viewing a live stream from the camera. It also keeps track of what’s going on with the hardware, even when it’s off. When a model is running on the DeepLens, you can view a live stream of its inferences about what it’s seeing (the labeled images). Amazon has released various pretrained models designed to work on the DeepLens. Using a model for detecting faces, we can get a live stream that looks like this:

Me looking at the DeepLens in my kitchen

Facial recognition inferences on multiple people. (Witness my smile of satisfaction at finally finding enthusiastic subjects of facial recognition.)

Each face that the camera detects gets a box around it, along with the model’s level of certainty that it is a face. The above pictures were the results of an attempt to simulate the conditions where this could be used.

The Model

The model I used was trained on data from ImageNet, a public database with hundreds or thousands of images associated with nouns. (For example they have 1537 pictures of folding chairs.) ImageNet is commonly used to train and test computer vision models.

However, the training for this model didn’t stop there: Amazon used transfer learning from another large image dataset, MS-COCO, to fine-tune the model for face detection. Transfer learning works essentially by retraining the last layer of an already-trained model. In this way it harnesses the “insights” of the existing model (e.g. about shapes, colors, and positions) by repurposing this information to make predictions about something else. In this case, whether something is a face.

Since this model was pretrained and optimized by Amazon for the DeepLens, it provides a low effort route to implementing a computer vision model on the DeepLens. I didn’t have to do any of the processing on my own hardware. The DeepLens hardware took care of all the predictions, though the biggest resource savings were from not having to train the model myself (which can take days, or longer).

When the facial recognition model is deployed and the DeepLens is on, an AWS Lambda function written in Python repeatedly prompts the camera to get frames from the camera:

frame = awscam.getLastFrame()

…to resize the frames before inference (the model accepts frames of particular size):

frame_resize = cv2.resize(frame, (input_height, input_width))

…to pass the frames to the model:

parsed_inference_results = model.parseResult(model_type, model.doInference(frame_resize))

…and to use the results to draw boxes around the faces:

cv2.rectangle(frame, (xmin, ymin), (xmax, ymax), (255, 165, 20), 10)

As you can see from how often “cv2” appears in the code above, this implementation relies heavily on code from OpenCV, an open source computer vision framework. Finally, the results are sent to the cloud:

client.publish(topic=iot_topic, payload=json.dumps(cloud_output))

In the last code snippet above, iot_topic refers to an Amazon “MQTT topic” (Message Queuing Telemetry Transport), for IoT devices. MQTT is the standard connectivity framework for DeepLens and many other IoT devices. One of its advantages for this context is that it can handle situations with intermittent connectivity, by smoothly queueing messages for when the network connection is stable. The essence of MQTT is to enable publishing and subscribing to different topics. The system of topics enables results from a DeepLens to trigger other processes. For example, the DeepLens could publish a message when it sees a face, and this could prompt another cloud service to do something else, such as save what time and how long the face appeared.

I wanted to test how data from this model would compare to a human’s perception. The first step was to understand what data the camera offers. It produces data about each frame analyzed: a timestamp (in 13-digit Unix time), and the predicted probability that something it identifies is a face. To gather this data, I used the AWS IoT service to manually subscribe to a secure MQTT topic where the DeepLens published its predictions. Each frame processed produces data like this:

{
  "format": "json",
  "payload": {
    "face": 0.5654296875
  },
  "qos": 0,
  "timestamp": 1554853281975,
  "topic": "$aws/things/deeplens_bnU5sr2sSD2ecW5YkfJZtw/infer"
}

The data generated by a single frame (with one face) when processed by the DeepLens.

For my purposes, I was only interested in the timestamps and payloads (which contain the number of faces identified, and their probabilities). I decided to test the facial recognition model under several different conditions:

No faces present
One face present
Multiple faces present

For condition 1 I just aimed it at an empty room for 20 minutes, and for condition 2 I sat in front of the camera for 20 minutes. For condition 3, I aimed the camera at a public space for 20 minutes, and while it was running I kept an ongoing count of the number of people looking in the general direction of the camera (I put the camera in front of a wall with a TV on it so people would be more likely to look towards it). Then I averaged my count over the duration of the sample, which resulted in an average engagement number of 2.5 people, meaning that on average, 2.5 people were looking at the camera. In an attempt to minimize bias, I made my human-eye assessment before looking at any of the data.

I’ll spoil one aspect of the results right away: there were no false positives under any condition. Even the lower probability guesses corresponded to actual faces, though this result might not hold true in a room with lots of face-like art, that’s not too common of a scenario. This simplified things, since it meant there was no need to set a lower bound on the probabilities which we should count—any face detected by the camera is a face. This also highlights one of my remaining questions about the model: is there useful information to be gained from the probabilities?

Another important note: I noticed early in the experiment that it almost never detects a face farther than 15 feet away. For the use case of a Liquid Galaxy, the 15-foot range is too short to capture all types of engagement (some people look at it across the room), but from my experience with the system I think that users within this range could be accurately described as focused users—something worth measuring, but certainly not everything worth measuring. After noticing this, I retested condition 2 with my face about 5 feet from the DeepLens, after initially trying it from across a room.

How did the DeepLens counts compare to my counts?

The model matched my performance in conditions 1 and 2, which makes a strong statement about its reliability in relatively static and close-up conditions such as looking at an empty room, or looking at someone stare at their laptop across a small table. In contrast, it did not count as many faces as I did in condition 3—so I’m happy to report I can still outperform A.I. on something.

Anyway, this suggests that the model is somewhat conservative, at least compared to my count (likely partly due to my eyes having a range larger than 15 feet). Therefore, when considering usage statistics gathered by a similar method, it might make most sense to think of the results as a lower bound, e.g. “the average number of people focused on the system was more than 2.1”.

It would be useful to experiment with the multiple faces condition again, to see how robust these findings are. It would also be helpful to keep track of factors like how much people move, the lighting, and the orientation of the camera, to see if they might impact the results. It would also be useful to automate the data collection and analysis.

This investigation has showed me that the DeepLens has a lot of potential as a tool for measuring engagement. Perhaps a future post will examine how it can be used to count users.

Thanks for reading! You are welcome to learn more about End Point Liquid Galaxy and AWS DeepLens.

Amazon AWS upgrades to Postgres with Bucardo

2017-06-06T00:00:00+00:00

(Bird-chasing photo by Doug Waldron)

Many of our clients at End Point are using the incredible Amazon Relational Database Service (RDS), which allows for quick setup and use of a database system. Despite minimizing many database administration tasks, some issues still exist, one of which is upgrading. Getting to a new version of Postgres is simple enough with RDS, but we’ve had clients use Bucardo to do the upgrade, rather than Amazon’s built-in upgrade process. Some of you may be exclaiming “A trigger-based replication system just to upgrade?!” While using it may seem unintuitive, there are some very good reasons to use Bucardo for your RDS upgrade:

Minimize application downtime

Many businesses are very sensitive to any database downtime, and upgrading your database to a new version always incurs that cost. Although RDS uses the ultra-fast pg_upgrade –links method, the whole upgrade process can take quite a while—or at least too long for the business to accept. Bucardo can reduce the application downtime from around seven minutes to ten seconds or less.

Upgrade more than one version at once

As of this writing (June 2017), RDS only allows upgrading of one major Postgres version at a time. Since pg_upgrade can easily handle upgrading older versions, this limitation will probably be fixed someday. Still, it means even more application downtime—to the tune of seven minutes for each major version. If you are going from 9.3 to 9.6 (via 9.4 and 9.5), that’s at least 21 minutes of application downtime, with many unnecessary steps along the way. The total time for Bucardo to jump from 9.3 to 9.6 (or any major version to another one) is still under ten seconds.

Application testing with live data

The Bucardo upgrade process involves setting up a second RDS instance running the newer version, copying the data from the current RDS server, and then letting Bucardo replicate the changes as they come in. With this system, you can have two “live” databases you can point your applications to. With RDS, you must create a snapshot of your current RDS, upgrade that, and then point your application to the new (and frozen-in-time) database. Although this is still useful for testing your application against the newer version of the database, it is not as useful as having an automatically-updated version of the database.

Control and easy rollback

With Bucardo, the initial setup costs, and the overhead of using triggers on your production database, is balanced a bit by ensuring you have complete control over the upgrade process. The migration can happen when you want, at a pace you want, and can even happen in stages as you point some of the applications in your stack to the new version, while keeping some pointed at the old. And rolling back is as simple as pointing apps back at the older version. You could even set up Bucardo as “master-master”, such that both new and old versions can write data at the same time (although this step is rarely necessary).

Database bloat removal

Although the pg_upgrade program that Amazon RDS uses for upgrading is extraordinarily fast and efficient, the data files are seldom, if ever, changed at all, and table and index bloat is never removed. On the other hand, an upgrade system using Bucardo creates the tables from scratch on the new database, and thus completely removes all historical bloat. (Indeed, one time a client thought something had gone wrong, as the new version’s total database size had shrunk radically - but it was simply removal of all table bloat!).

Statistics remain in place

The pg_upgrade program currently has a glaring flaw—no copying of the information in the pg_statistic table. Which means that although an Amazon RDS upgrade completes in about seven minutes, the performance will range somewhere from slightly slow to completely unusable, until all those statistics are regenerated on the new version via the ANALYZE command. How long this can take depends on a number of factors, but in general, the larger your database, the longer it will take—a database-wide analyze can take hours on very large databases. As mentioned above, upgrading via Bucardo relies on COPYing the data to a fresh copy of the table. Although the statistics also need to be created when using Bucardo, the time cost for this does NOT apply to the upgrade time, as it can be done any time earlier, making the effective cost of generating statistics zero.

Upgrading RDS the Amazon way

Having said all that, the native upgrade system for RDS is very simple and fast. If the drawbacks above do not apply to you—or can be suffered with minimal business pain - then this way should always be the upgrade approach to use. Here is a quick walk through of how an Amazon RDS upgrade is done.

For this example, we will create a new Amazon RDS instance. The creation is amazingly simple: just log into aws.amazon.com, choose RDS, choose PostgreSQL (always the best choice!), and then fill in a few details, such as preferred version, server size, etc. The “DB Engine Version” was set as “PostgreSQL 9.3.16-R1”, the “DB Instance Class” as db.t2.small – 1 vCPU, 2 GiB RAM, and “Multi-AZ Deployment” as no. All other choices are the default. To finish up this section of the setup, “DB Instance Identifier” was set to gregtest, the “Master Username” to greg, and the “Master Password” to b5fc93f818a3a8065c3b25b5e45fec19

Clicking on “Next Step” brings up more options, but the only one that needs to change is to specify the “Database Name” as gtest. Finally, the “Launch DB Instance” button. The new database is on the way! Select “View your DB Instance” and then keep reloading until the “Status” changes to Active.

Once the instance is running, you will be shown a connection string that looks like this: gregtest.zqsvirfhzvg.us-east-1.rds.amazonaws.com:5432. That standard port is not a problem, but who wants to ever type that hostname out, or even have to look at it? The pg_service.conf file comes to the rescue with this new entry inside the ~/.pg_service.conf file:

[gtest]
host=gregtest.zqsvirfhzvg.us-east-1.rds.amazonaws.com
port=5432
dbname=gtest
user=greg
password=b5fc93f818a3a8065c3b25b5e45fec19
connect_timeout=10

Now we run a quick test to make sure psql is able to connect, and that the database is an Amazon RDS database:

$ psql service=gtest -Atc "show rds.superuser_variables"
session_replication_role

We want to use the pgbench program to add a little content to the database, just to give the upgrade process something to do. Unfortunately, we cannot simply feed the “service=gtest” line to the pgbench program, but a little environment variable craftiness gets the job done:

$ unset PGSERVICEFILE PGSERVICE PGHOST PGPORT PGUSER PGDATABASE
$ export PGSERVICEFILE=/home/greg/.pg_service.conf PGSERVICE=gtest
$ pgbench -i -s 4
NOTICE:  table "pgbench_history" does not exist, skipping
NOTICE:  table "pgbench_tellers" does not exist, skipping
NOTICE:  table "pgbench_accounts" does not exist, skipping
NOTICE:  table "pgbench_branches" does not exist, skipping
creating tables...
100000 of 400000 tuples (25%) done (elapsed 0.66 s, remaining 0.72 s)
200000 of 400000 tuples (50%) done (elapsed 1.69 s, remaining 0.78 s)
300000 of 400000 tuples (75%) done (elapsed 4.83 s, remaining 0.68 s)
400000 of 400000 tuples (100%) done (elapsed 7.84 s, remaining 0.00 s)
vacuum...
set primary keys...
done.

At 68MB in size, this is still not a big database—so let’s create a large table, then create a bunch of databases, to make pg_upgrade work a little harder:

## Make the whole database 1707 MB:
$ psql service=gtest -c "CREATE TABLE extra AS SELECT * FROM pgbench_accounts"
SELECT 400000
$ for i in {1..5}; do psql service=gtest -qc "INSERT INTO extra SELECT * FROM extra"; done

## Make the whole cluster about 17 GB:
$ for i in {1..9}; do psql service=gtest -qc "CREATE DATABASE gtest$i TEMPLATE gtest" ; done
$ psql service=gtest -c "SELECT pg_size_pretty(sum(pg_database_size(oid))) FROM pg_database WHERE datname ~ 'gtest'"
17 GB

To start the upgrade, we log into the AWS console, and choose “Instance Actions”, then “Modify”. Our only choices for instances are “9.4.9” and “9.4.11”, plus some older revisions in the 9.3 branch. Why anything other than the latest revision in the next major branch (i.e. 9.4.11) is shown, I have no idea! Choose 9.4.11, scroll down to the bottom, choose “Apply Immediately”, then “Continue”, then “Modify DB Instance”. The upgrade has begun!

How long will it take? All one can do is keep refreshing to see when your new database is ready. As mentioned above, 7 minutes and 30 seconds is the total time. The logs show how things break down:

11:52:43 DB instance shutdown
11:55:06 Backing up DB instance
11:56:12 DB instance shutdown
11:58:42 The parameter max_wal_senders was set to a value incompatible with replication. It has been adjusted from 5 to 10.
11:59:56 DB instance restarted
12:00:18 Updated to use DBParameterGroup default.postgres9.4

How much of that time is spent on upgrading though? Surprisingly little. We can do a quick local test to see how long the same database takes to upgrade from 9.3 to 9.4 using pg_upgrade –links: 20 seconds! Ideally Amazon will improve upon the total downtime at some point.

Upgrading RDS with Bucardo

As an asynchronous, trigger-based replication system, Bucardo is perfect for situations like this where you need to temporarily sync up two concurrent versions of Postgres. The basic process is to create a new Amazon RDS instance of your new Postgres version (e.g. 9.6), install the Bucardo program on a cheap EC2 box, and then have Bucardo replicate from the old Postgres version (e.g. 9.3) to the new one. Once both instances are in sync, just point your application to the new version and shut the old one down. One way to perform the upgrade is detailed below.

Some of the steps are simplified, but the overall process is intact. First, find a temporary box for Bucardo to run on. It doesn’t have to be powerful, or have much disk space, but as network connectivity is important, using an EC2 box is recommended. Install Postgres (9.6 or better, because of pg_dump) and Bucardo (latest or HEAD recommended), then put your old and new RDS databases into your pg_service.conf file as “rds93” and “rds96” to keep things simple.

The next step is to make a copy of the database on the new Postgres 9.6 RDS database. We want the bare minimum schema here: no data, no triggers, no indexes, etc. Luckily, this is simple using pg_dump:

$ pg_dump service=rds93 --section=pre-data | psql -q service=rds96

From this point forward, no DDL should be run on the old server. We take a snapshot of the post-data items right away and save it to a file for later:

$ pg_dump service=rds93 --section=post-data -f rds.postdata.pg

Time to get Bucardo ready. Recall that Bucardo can only replicate tables that have a primary key or unique index. But if those tables are small enough, you can simply copy them over at the final point of migration later.

$ bucardo install
$ bucardo add db A dbservice=rds93
$ bucardo add db B dbservice=rds96
## Create a sync and name it 'migrate_rds':
$ bucardo add sync migrate_rds tables=all dbs=A,B

That’s it! The current database will now have triggers that are recording any changes made, so we may safely do a bulk copy to the new database. This step might take a very long time, but that’s not a problem.

$ pg_dump service=rds93 --section=data | psql -q service=rds96

Before we create the indexes on the new server, we start the Bucardo sync to copy over any rows that were changed while the pg_dump was going on. After that, the indexes, primary keys, and other items can be created:

$ bucardo start
$ tail -f log.bucardo ## Wait until the sync finishes once
$ bucardo stop
$ psql service=rds96 -q -f rds.postdata.pg

For the final migration, we simply stop anything from writing to the 9.3 database, have Bucardo perform a final sync of any changed rows, and then point your application to the 9.6 database. The whole process can happen very quickly: well under a minute for most cases.

Upgrading major Postgres versions is never a trivial task, but both Bucardo and pg_upgrade allow it to be orders of magnitude faster and easier than the old method of using the pg_dump utility. Upgrading your Amazon AWS Postgres instance is fast and easy using the AWS pg_upgrade method, but it has limitations, so having Bucardo help out can be a very useful option.

Elastic Beanstalk Whenever

2015-01-27T00:00:00+00:00

I recently got the opportunity to pick up development on a Ruby on Rails application that was originally set up to run on AWS using their Elastic Beanstalk deployment tools. One of our first tasks was to move some notification hooks out of the normal workflow into scripts and schedule those batch scripts using cron.

Historically, I’ve had extremely good luck with Whenever. In my previous endeavors I’ve utilized Capistrano which Whenever merged with seamlessly. With how simple it was to integrate Whenever with Capistrano, I anticipated a similar experience dealing with Elastic Beanstalk. While the integration was not as seamless as Capistrano, I did manage to make it work.

My first stumbling block was finding documentation on how to do after or post hooks. I managed to find this forum post and this blog post which helped me out a lot. The important detail is that there is a “post” directory to go along with “pre” and “enact”, but it’s not present by default, so it can be easy to miss.

I used Marcin’s delayed_job config as a base. The first thing I had to address was an apparent change in Elastic Beanstalk’s configuration structure. Marcin’s config has

  . /opt/elasticbeanstalk/support/envvars

but that file doesn’t exist on the system I was working on. With a small amount of digging, I found:

  . /opt/elasticbeanstalk/containerfiles/envvars

in one of the other ebextensions. Inspecting that file showed a definition for and exportation of $EB_CONFIG_APP_CURRENT suggesting this is a similar file just stored in a different location now.

Another change that appears to have occurred since Marcin developed his config is that directories will be created automatically if they don’t already exist when adding a file in the files section of the config. That allows us to remove the entire commands section to simplify things.

That left me with a config that looked like:

files:
  "/opt/elasticbeanstalk/hooks/appdeploy/post/99_update_cron.sh"
  mode: "000755"
  owner: root
  group: root
  content: |
    #! /usr/bin/env bash
    . /opt/elasticbeanstalk/containerfiles/envvars
    su -c "cd $EB_CONFIG_APP_CURRENT; bundle exec whenever --update-cron" - $EB_CONFIG_APP_USER

This command completed successfully but on staging the cron jobs failed to run. The reason for that was an environment mismatch. The runner entries inside the cron commands weren’t receiving a RAILS_ENV or other type of environment directive so they were defaulting to production and failing when no database was found.

After some greping I was able to find a definition for RACK_ENV in:

/opt/elasticbeanstalk/containerfiles/envvars.d/sysenv

Making use of it, I came up with this final version:

files:
  "/opt/elasticbeanstalk/hooks/appdeploy/post/99_update_cron.sh"
  mode: "000755"
  owner: root
  group: root
  content: |
    #! /usr/bin/env bash
    . /opt/elasticbeanstalk/containerfiles/envvars
    . /opt/elasticbeanstalk/containerfiles/envvars.d/sysenv
    su -c "cd $EB_CONFIG_APP_CURRENT; bundle exec whenever --update-cron --set='environment=$RACK_ENV'" - $EB_CONFIG_APP_USER

PostgreSQL EC2/EBS/RAID 0 snapshot backup

2010-02-23T00:00:00+00:00

One of our clients uses Amazon Web Services to host their production application and database servers on EC2 with EBS (Elastic Block Store) storage volumes. Their main database is PostgreSQL.

A big benefit of Amazon’s cloud services is that you can easily add and remove virtual server instances, storage space, etc. and pay as you go. One known problem with Amazon’s EBS storage is that it is much more I/O limited than, say, a nice SAN.

To partially mitigate the I/O limitations, they’re using 4 EBS volumes to back a Linux software RAID 0 block device. On top of that is the xfs filesystem. This gives roughly 4x the I/O throughput and has been effective so far.

They ship WAL files to a secondary server that serves as warm standby in case the primary server fails. That’s working fine.

They also do nightly backups using pg_dumpall on the master so that there’s a separate portable (SQL) backup not dependent on the server architecture. The problem that led to this article is that extra I/O caused by pg_dumpall pushes the system beyond its I/O limits. It adds both reads (from the PostgreSQL database) and writes (to the SQL output file).

There are several solutions we are considering so that we can keep both binary backups of the database and SQL backups, since both types are valuable. In this article I’m not discussing all the options or trying to decide which is best in this case. Instead, I want to consider just one of the tried and true methods of backing up the binary database files on another host to offload the I/O:

Create an atomic snapshot of the block devices
Spin up another virtual server
Mount the backup volume
Start Postgres and allow it to recover from the apparent “crash” the server had (since there wasn’t a clean shutdown of the database before the snapshot
Do whatever pg_dump or other backups are desired
Make throwaway copies of the snapshot for QA or other testing

The benefit of such snapshots is that you get an exact backup of the database, with whatever table bloat, indexes, statistics, etc. exactly as they are in production. That’s a big difference from a freshly created database and import from pg_dump.

The difference here is that we’re using 4 EBS volumes with RAID 0 striped across them, and there isn’t currently a way to do an atomic snapshot of all 4 volumes at the same time. So it’s no longer “atomic” and who knows what state the filesystem metadata and the file data itself would be in?

Well, why not try it anyway? Filesystem metadata doesn’t change that often, especially in the controlled environment of a Postgres data volume. Snapshotting within a relatively short timeframe would be pretty close to atomic, and probably look to the software (operating system and database) like some kind of strange crash since some EBS volumes would have slightly newer writes than others. But aren’t all crashes a little unpredictable? Why shouldn’t the software be able to deal with that? Especially if we have Postgres make a checkpoint right before we snapshot.

I wanted to know if it was crazy or not, so I tried it on a new set of services in a separate AWS account. Here are the notes and some details of what I did:

1. Created one EC2 image:

Amazon EC2 Debian 5.0 lenny AMI built by Eric Hammond
Debian AMI ID ami-4ffe1926 (x86_64)
Instance Type: High-CPU Extra Large (c1.xlarge) — 7 GB RAM, 8 CPU cores

2. Created 4 x 10 GB EBS volumes

3. Attached volumes to the image

4. Created software RAID 0 device:

mdadm -C /dev/md0 -n 4 -l 0 -z max /dev/sdf /dev/sdg /dev/sdh /dev/sdi

5. Created XFS filesystem on top of RAID 0 device:

mkfs -t xfs -L /pgdata /dev/md0

6. Set up in /etc/fstab and mounted:

mkdir /pgdata
# edit /etc/fstab, with noatime
mount /pgdata

7. Installed PostgreSQL 8.3

8. Configured postgresql.conf to be similar to primary production database server

9. Created empty new database cluster with data directory in /pgdata

10. Started Postgres and imported a play database (from public domain census name data and Project Gutenberg texts), resulting in about 820 MB in data directory

11. Ran some bulk inserts to grow database to around 5 GB

12. Rebooted EC2 instance to confirm everything came back up correctly on its own

13. Set up two concurrent data-insertion processes:

50 million row insert based on another local table (INSERT INTO … SELECT …), in a single transaction (hits disk hard, but nothing should be visible in the snapshot because the transaction won’t have committed before the snapshot is taken)
Repeated single inserts in autocommit mode (Python script writing INSERT statements using random data from /usr/share/dict/words piped into psql), to verify that new inserts made it into the snapshot, and no partial row garbage leaked through

14. Started those “beater” jobs, which mostly consumed 2-3 CPU cores

15. Manually inserted a known test row and created a known view that should appear in the snapshot

16. Started Postgres’s backup mode that allows for copying binary data files in a non-atomic manner, which also does a CHECKPOINT and thus also a filesystem sync:

SELECT pg_start_backup('raid_backup');

17. Manually inserted a 2nd known test row & 2nd known test view that I don’t want to appear in the snapshot after recovery

18. Ran snapshot script which calls ec2-create-snapshot on each of the 4 EBS volumes—during first run, run serially quite slowly taking about 1 minute total; during second run, run in parallel such that the snapshot point was within 1 second for all 4 volumes

19. Tell Postgres the backup’s over:

SELECT pg_stop_backup();

20. Ran script to create new EBS volumes derived from the 4 snapshots (which aren’t directly usable and always go into S3), using ec2-create-volume --snapshot

21. Run script to attach new EBS volumes to devices on the new EC2 instance using ec2-attach-volume

22. Then, on the new EC2 instance for doing backups:

mdadm --assemble --scan
mount /pgdata
Start Postgres
Count rows on the 2 volatile tables; confirm that the table with the in-process transaction doesn’t show any new rows, and that the table getting individual rows committed to reads correctly
VACUUM VERBOSE — and confirm no errors or inconsistencies detected
pg_dumpall # confirmed no errors and data looks sound

It worked! No errors or problems, and pretty straightforward to do.

Actually before doing all the above I first did a simpler trial run with no active database writes happening, and didn’t make any attempt for the 4 EBS snapshots to happen simultaneously. They were actually spread out over almost a minute, and it worked fine. With the confidence that the whole thing wasn’t a fool’s errand, I then put together the scripts to do lots of writes during the snapshot and made the snapshots run in parallel so they’d be close to atomic.

There are lots of caveats to note here:

This is an experiment in progress, not a how-to for the general public.
The data set that was snapshotted was fairly small.
Two successful runs, even with no failures, is not a very big sample set. :)
I didn’t use Postgres’s point-in-time recovery (PITR) here at all—I just started up the database and let Postgres recover from an apparent crash. Shipping over the few WAL logs from the master collected during the pg_backup run after the snapshot copying is complete would allow a theoretically fully reliable recovery to be made, not just a practically non-failing recovery as I did above.

So there’s more work to be done to prove this technique viable in production for a mission-critical database, but it’s a promising start worth further investigation. It shows that there is a way to back up a database across multiple EBS volumes without adding noticeably to its I/O load by utilizing the Amazon EBS data store’s snapshotting and letting a separate EC2 server offload the I/O of backups or anything else we want to do with the data.

Walden University Presidential Youth Debate goes live

2008-10-22T00:00:00+00:00

This afternoon was the launch of Walden University’s Presidential Youth Debate website, which features 14 questions and video responses from Presidential candidates Barack Obama and John McCain. The video responses are about 44 minutes long overall.

The site has a fairly simple feature set but is technologically interesting for us. It was developed by Dan Collis-Puro and Ethan Rowe using Radiant, PostgreSQL, CentOS Linux, Ruby on Rails, Phusion Passenger, Apache, Varnish, and Amazon S3.

Nice work, guys!

Introduction to Terraform with AWS

Prerequisites

Install AWS CLI

Install Terraform

Create the Terraform configuration file

Initialize the project

Validate the configuration file

Apply the configuration

Access the provisioned EC2 and Nginx

Conclusion

Kubernetes From The Ground Up With AWS EC2

Prerequisites

EC2 Instances provisioning

Installing the container runtime

Kubernetes control plane setup

Kubernetes worker nodes setup

Hello, Kubernetes :)

Conclusion

Ansible tutorial with AWS EC2

Prerequisites

EC2 Instances provisioning

Ansible nodes and SSH keys

Ansible Installation and Configuration

Ansible modules and Ansible ad hoc commands

Ad hoc command examples

Ansible playbooks

Conclusion

How to deploy a Django App with Aurora Serverless and AWS Copilot

The Prerequisites

The Django Project

The Deployment with AWS Copilot

How to query Aurora Serverless

The End

How to deploy a containerized Django app with AWS Copilot

The chosen one and the sidekick

Install Docker

Set up AWS CLI

Install AWS Copilot CLI

The Django project

The Deployment with AWS Copilot

A mini cheat sheet

The End

Generate PDF with Chrome, Puppeteer, and Serverless Stack

The tools

1. Serverless Stack

2. Chrome and Puppeteer

Implementing PDF screenshot and download with Serverless Stack, Chrome, and Puppeteer

Prerequisite: Sample receipt pages

Getting started

1. sst.json

2. stacks/index.js

3. stacks/MyStack.js

Deploying the project

Developing with Serverless Stack

Conclusion

Caveats

Further readings

Facial Recognition Using Amazon DeepLens: Counting Liquid Galaxy Interactions

The Amazon DeepLens camera

Computer Vision and the Amazon DeepLens

The Model

How did the DeepLens counts compare to my counts?

Amazon AWS upgrades to Postgres with Bucardo

Minimize application downtime

Upgrade more than one version at once

Application testing with live data

Control and easy rollback

Database bloat removal

Statistics remain in place

Upgrading RDS the Amazon way

Upgrading RDS with Bucardo

Elastic Beanstalk Whenever

PostgreSQL EC2/EBS/RAID 0 snapshot backup

Walden University Presidential Youth Debate goes live