Testing Ansible Automation with Molecule

2025-03-28T00:00:00+00:00

1. Why Test with Molecule?

Molecule is a test framework for Ansible roles. It supports testing with multiple instances, operating systems and distributions, virtualization providers, test frameworks, and testing scenarios.

Molecule is useful because it increases confidence in your automation and ensures roles are reliable. It catches issues early in the development cycle, reducing production problems. It also ensures consistency across different environments and platforms.

2. Install Prerequisites

Make sure you have Python >= 3.6 and pip installed. Then install Ansible and Molecule using pip:

pip install ansible
pip install molecule

Molecule uses the delegated driver by default. Other drivers can be installed separately from PyPI, most of them being included in the molecule-plugins package. We are going to use the Docker driver, so let’s install that by running:

pip install "molecule[docker]"

3. Ansible Monorepo Structure

We use the Ansible monorepo structure. This means our playbooks, variables, scripts, roles, plugins, inventory scripts, and configuration all reside and are version controlled together in the same repository.

Here is the high level layout:

~/ansible-endpoint
├── ansible.cfg
├── ansible_hosts
├── molecule/
├── playbooks/
└── roles/
    ├── apache_server/
    ├── ...
    ├── nginx_server/
    └── ...

We’ll focus on roles/nginx_server, showing how to add and configure the Molecule scenarios for testing and also how to verify it.

4. Creating Your First Molecule Scenario

Inside the roles/nginx_server directory, let’s initialize the Molecule scenario:

cd roles/nginx_server
molecule init scenario --driver-name docker --scenario-name default

This command will create the following files:

roles/
└── nginx_server/
    ├── tasks/
    │   └── main.yml
    ├── molecule/
    │   └── default/
    │       ├── molecule.yml
    │       ├── converge.yml
    │       ├── verify.yml
    │       └── destroy.yml
    └── ...

Here are some details about the files created by Molecule:

molecule.yml — The Molecule configuration file.
```
---
driver:
  name: docker
platforms:
  - name: "molecule-rocky9-${CI_JOB_ID}"
    image: geerlingguy/docker-rockylinux9-ansible:latest
    privileged: true
    pre_build_image: true
    volumes:
      - /sys/fs/cgroup:/sys/fs/cgroup:rw
    cgroupns_mode: host
    command: /usr/sbin/init
provisioner:
  name: ansible
```
Key blocks from molecule.yml:
- provisioner is the tool that will be used to provision the scenario. We are using Ansible to run the scenario itself.
- driver — As mentioned above, we are using Docker as the driver.
- platforms — Here we define the target platform for the scenario. We are using the Rocky 9 Docker image as the target platform. Thanks to Jeff Geerling who created the Rocky 9 Docker image for Ansible testing with systemd in it.

converge.yml — The playbook to converge the scenario. This tells Molecule to apply the nginx_server role to our test container (the “instance” defined in molecule.yml).

---
- name: Converge
  hosts: all
  gather_facts: false
  tasks:
    - name: Run the nginx_server role
      hosts: all
      roles:
        - nginx_server

verify.yml — The playbook to verify whether the converge was successful. This tells Molecule to verify that our role has been correctly installed on the Docker instance. Here we are going to check if NGINX is installed, running and also responding to requests.

---
- name: Verify
  hosts: all
  tasks:
    - name: Check if NGINX is installed
      ansible.builtin.package:
        name: nginx
        state: present
      check_mode: true
      register: install
      failed_when: (install is changed) or (install is failed)

    - name: Check if NGINX service is running
      ansible.builtin.service:
        name: nginx
        state: started
        enabled: true
      check_mode: true
      register: service
      failed_when: (service is changed) or (service is failed)

    - name: Verify NGINX is up and running
      ansible.builtin.uri:
        url: http://localhost
        status_code: 200

destroy.yml — Destroys the instance defined in the molecule.yml file. There won’t be any file, but the default Docker driver will automatically destroy the instance after the test is finished. If you want to override this behavior, you can add a destroy.yml file to the scenario directory.

5. Running Molecule tests

A Molecule run consists of the various lifecycle events:
- create
- …
- converge
- …
- verify
- …
- destroy

These are some of the important events in the Molecule lifecycle run. To run the full lifecycle Molecule run, we can use the following command:

cd roles/nginx_server
molecule test

This command executes the following steps in sequence:

Create: Spins up a test Rocky 9 container as mentioned in the molecule.yml file.
Converge: Applies the nginx_server role inside the container.
Verify: Verifies whether the NGINX server is installed, running, and responding to requests.
Destroy: Cleans up the container.

We can also run each step individually:

molecule create
molecule converge
molecule verify
molecule destroy

To avoid rebuilding the containers, we can run commands from converge step.

6. Integrating Molecule with GitLab CI/CD

We want to run the Molecule tests whenever a merge request which changes the particular role is created.

To automatically run the Molecule tests, we can use the GitLab CI/CD.

Here is an example of a .gitlab-ci.yml file that runs our Molecule tests in the Docker based environment for our nginx_server role.

stages:
  - test

before_script:
  - pip install molecule ansible docker molecule-docker

molecule_test:
  stage: test
  variables:
    ROLE_PATH: "roles/nginx_server"
  script:
    - cd $ROLE_PATH
    - molecule test
  rules:
    - changes:
        - $ROLE_PATH/**

With this setup, we’ve set up a Molecule scenario for a specific Ansible role, then run tests both manually and automatically via GitLab CI. This allows tests to trigger whenever a merge request modifies that role, ensuring a quick feedback loop and a higher degree of reliability in the automation.

To learn more about Molecule, here are some useful links:

Middleware: Is that still a thing?

2022-05-26T00:00:00+00:00

The simple answer to the question in the title is simply, yes! Despite the term being many decades old and well past its hype peak, middleware is still very much a thing and has become a key part of the technical landscape that is critical in day-to-day functioning of systems.

So there are still some questions to be answered: What is middleware? What does it do? And maybe most importantly: Why do we care?

What is it?

In its simplest meaning, middleware is an application that sits between other applications and shuffles data between them. There is normally one system requesting the information and the middleware figures out where to get that requested data and makes the request to another system.

An easy example of this is buying something at a retail store using your credit or debit card. When you swipe your card, the business makes a request to a service (some middleware) to ask if there’s enough room on the card for that purchase. Then that system makes a request to the appropriate bank or card holding company to ask the same question. The bank or card holding company replies with either a yes or no and that answer is then relayed back to the terminal where you swiped your card.

You might be thinking, “Why do we need the middleware? Just have the terminal ask the right bank!” Well, that’s a benefit of middleware: The terminal software doesn’t need to have all the same logic coded into it to know whether or how to talk to Visa or Mastercard as well as Citibank or Capital One or Bank of America to get the answer. It just talks to the middleware and lets it figure all of that out. This way you can have many different types of terminals or credit card taking applications available but only one middleware application needs to have the smarts and authorization to know where and how to get the answer.

The term middleware is most often used to describe business systems internal to a company. It is a subset of the broader concept “API” (application programming interface), which includes services provided for external users, either the general public or specific authorized customers or business partners. If a service is used directly by humans of the general public, it is typically called SaaS (software as a service).

What does it do?

As I explained in the example above, the simplistic definition of middleware is an application that shuffles data between two systems. In reality, middleware is sometimes way more complicated than that!

In that example of a credit card charge terminal, using middleware also allows for the retailer to set centralized company-wide rules for fraud screening, add or remove supported payment methods, or change which payment gateway is used, without having to update each terminal individually.

Middleware also can do data manipulation and mapping between two systems. Let’s consider a more complicated case like booking a cross-country flight.

You’ve probably used sites like Google Flights or Expedia to search for ticket costs. You go to their site, put in your starting airport(s) and your destination airport(s), select the dates when you’d like to travel, and maybe check the box saying you’ve got some wiggle room on when you leave and come back. After that you click the search button and within a few seconds, you’ve got tons of flight options to sort through! The search results can be sorted by price, number of stops, and airlines, to name a few.

So taking a step back to marvel at what just happened, how do you think they got that much data from that many sources so quickly? Well, the answer is middleware.

At some point, a middleware system made a request to each of the airlines to ask for their available flights with their dates, costs, amenities, etc. and stored that information on their own system to be used for your search. There are many different ways that data could be captured. We won’t delve into all that excitement, but what we do know is some system (middleware) had to fetch this data from each of the airlines and store it to be used for your search. It has to organize and store that data in a way that makes sense to the search engine to use so you can get back your search results in a speedy fashion.

There’s a good chance that the data from American Airlines will differ in its structure and data points compared to Delta. It’s the middleware’s job to take each of those data feeds and organize and store them somewhere for the search engine to use. In this part of the example, the middleware is taking a request to fetch the Delta flight data, turning that request into a request to another system for the data then doing whatever data manipulation and storage of that data to satisfy the search engine.

Why do we care?

The main reason that we care is we’ve become used to having a single location (website, application etc.) that allows us access to data from multiple places.

In our credit card example, there are many different types of credit cards we can use to check out at our local gas station. Having them require us to only use a Shell gas card would be bad for their business! We’ve grown accustomed to being able to use our Visa, Mastercard, American Express, or Shell gas card at that same gas station. This is all made possible due to middleware behind the scenes doing the hard work.

In many cases, the function of gathering data from many sources in one place, keeping it current, and allowing it to be searched, sorted, etc. is not just one means to an end, but it actually is the whole business! It is why services like Google Flights and Expedia exist.

Middleware can serve many other purposes even when only used internally by a single company:

authentication — making sure who- or whatever made a request is supposed to be able to
authorization — making sure the kind of request being made is one this user is allowed to
auditing — logging or sampling traffic in middleware is often more comprehensive and convenient than at each service individually
rate-limiting — ensuring the user is not using too many resources too fast, or that one service isn’t overused and interfering with access to other services
caching — storing answers to frequent requests to reuse for a limited time, reducing the number of live requests to busy internal applications and databases
billing — tracking usage so it can be paid for or at least each user’s usage is fairly recorded
fault-tolerance — retrying a request if an internal system is temporarily unavailable, or rerouting requests to a different system, to shield users from outages
load balancing — spreading requests across multiple services to allow greater total throughput
translation — modern APIs typically use JSON to structure their data, and middleware can translate between a nice JSON interface and older systems using a variety of more complex and archaic data formats
migrations — transparently moving from an old system to a newer one, possibly in phases or temporarily for testing, without users having to change anything or even be aware

In closing, while middleware isn’t commonly mentioned or even thought about, it is a very important part of what makes the technological world function.

Creating Telegram bots with Google Apps Script

2022-01-17T00:00:00+00:00

In a previous post on this blog, Afif wrote about how to use Google Apps Script with Google Forms. Coincidentally, last year I learned a bit about how to use Google Apps Script with Telegram Bot as a personal ledger tool, as outlined in this post by Mars Escobin.

The Telegram Bot I created from Mars’s code looks like this:

In this post I will share a bit on how to adapt Mars’s code and use Telegram Bot to get the input from the user and let Google Apps Script call Google’s cloud-based services (translation and finance) to later return the outputs to the user.

The initial process of creating a Telegram bot is outlined on Telegram’s website. After receiving Telegram’s API key we can use it inside our Google Apps Script editor.

Translation Bot

One way to use the Telegram Bot and Google Cloud Services which came to mind was to create a translation bot. Although there are undoubtedly tons of mobile apps out there which do the same thing, I wanted to learn about it by using Telegram. So I found a class that I could use in Apps Script to realize that.

Google Apps Script can be used to manipulate the Google Translate capability by calling the Language.App class. The translation capability is invoked with LanguageApp.translate(text, sourceLanguage, targetLanguage).

In addition to fetching the input from the users, I also wanted to store the searched word inside a Google Sheets spreadsheet.

So by using this:

sheet.appendRow([formattedDate, item[0], item[1], item[2], myTranslationOutput]);

we can append the searched word inside the Google spreadsheet, and later append the input and output which we got from the user.

In my case, I removed the inline keyBoard function from Mars’s code since I didn’t plan to use it in my bot. Instead, I will use the inputs which were sent through the item variable and let LanguageApp translate it. The translation will virtually take a few seconds, so if we do not put an interval within our code, sendMessage will reply with an empty output for the translation.

In order to handle this, I used the Utilities.sleep() function prior to return sendMessage() so that I will be able to grab the answer before returning the output to the requestor.

The following are my changes for creating the translation bot:

var token = "<insert your Telegram API token here>";
var telegramUrl = "https://api.telegram.org/bot" + token;
var webAppUrl = "<insert your webAppURL which is generated from Google Apps Script UI here>";

function sendMessage(id, text) {
  var data = {
    method: "post",
    payload: {
      method: "sendMessage",
      chat_id: String(id),
      text: text,
      parse_mode: "HTML",

    }
  };
  UrlFetchApp.fetch('https://api.telegram.org/bot' + token + '/', data);
}

function doPost(e) {
  var contents = JSON.parse(e.postData.contents);
  var ssId = "<insert your webAppURL which is generated from Google Apps Script UI here>";
  var sheet = SpreadsheetApp.openById(ssId).getSheetByName("<sheet name here>");

  if (contents.message) {
    var id = contents.message.from.id;
    var text = contents.message.text;

    if (text.indexOf(",") !== -1) {
      var dateNow = new Date;
      var formattedDate = dateNow.getDate() + "/" + (dateNow.getMonth() + 1);
      var item = text.split(",");
      var myTranslationOutput = LanguageApp.translate(item[0], item[1], item[2]);

      sheet.appendRow([formattedDate, item[0], item[1], item[2], myTranslationOutput]);

      Utilities.sleep(200);
      return sendMessage(id, "The translation of " + item[0] + " is " + myTranslationOutput);
    } else {
      return sendMessage(id, "The word that you key in will be kept for our analysis purpose\nPlease use this format : word, source language code, target language code \nRefer cloud.google.com/translate/docs/languages");
    }
  }
}

Currency Converter Bot

There are at least two possible ways to create a currency converter bot with Telegram and Google Apps Script. I could either invoke a curl-like method by calling an external API (which is not related to Google) or just manipulating whatever Google Finance offers. I did some searching but I could not find any built-in class in order to do the conversion within the code. However, I remember that Google Sheets could actually call Google Finance within its cell. So I decided to let Sheets do the conversion and then I will fetch the result and return it to the requester.

This is shown in the following snippet:

sheet.getRange('a2').setValue(item[0]);
sheet.getRange('b2').setValue(item[1]);
sheet.getRange('c2').setValue(item[2]);
sheet.getRange('d2').setValue('=GOOGLEFINANCE("currency:"&b2&c2)*a2');

And later we fetched the value from d2 cell with

var value = SpreadsheetApp.getActiveSheet().getRange('d2').getValue();

As the default value is taking many decimal points, I made it fixed to two decimal points.

value = value.toFixed(2);

I then returned the value which later converted the currency code to uppercase with .toUpperCase().

return sendMessage(id, item[1].toUpperCase() + " " + item[0] + " = " + item[2].toUpperCase() + " " + value);

You can see my changes in the following scripts:

var token = "<insert your Telegram API token here>";
var telegramUrl = "https://api.telegram.org/bot" + token;
var webAppUrl = "<insert your webAppURL which is generated from Google Apps Script UI here>";

function setWebhook() {
  var url = telegramUrl + "/setWebhook?url=" + webAppUrl;
  var response = UrlFetchApp.fetch(url);
  Logger.log(response.getContentText());
}


function sendMessage(id, text) {
  var data = {
    method: "post",
    payload: {
      method: "sendMessage",
      chat_id: String(id),
      text: text,
      parse_mode: "HTML",

    }
  };
  UrlFetchApp.fetch('https://api.telegram.org/bot' + token + '/', data);
}

function doPost(e) {
  var contents = JSON.parse(e.postData.contents);
  var ssId = "<insert the spreadsheet ID here, you can get it from the browser URL bar>";
  var sheet = SpreadsheetApp.openById(ssId).getSheetByName("<sheet name here>");

  if (contents.message) {
    var id = contents.message.from.id;
    var text = contents.message.text;

    if (text.indexOf(",") !== -1) {
      var dateNow = new Date;
      var formattedDate = dateNow.getDate() + "/" + (dateNow.getMonth() + 1);
      var item = text.split(",");

      sheet.getRange('a2').setValue(item[0]);
      sheet.getRange('b2').setValue(item[1]);
      sheet.getRange('c2').setValue(item[2]);
      sheet.getRange('d2').setValue('=GOOGLEFINANCE("currency:"&b2&c2)*a2');

      SpreadsheetApp.getActiveSheet().getRange('c2').setValue('=GOOGLEFINANCE("currency:"&item[1]&item[2])*item[0]');

      var value = SpreadsheetApp.getActiveSheet().getRange('d2').getValue();
      Utilities.sleep(10);

      sheet.appendRow([formattedDate, item[0], item[1], item[2], value]);

      value = value.toFixed(2);

      return sendMessage(id, item[1].toUpperCase() + " " + item[0] + " = " + item[2].toUpperCase() + " " + value)
    } else {
      return sendMessage(id, "The word that you key in will be kept for our analysis purpose\nPlease use this format : amount, source currency code, target currency code")
    }
  }
}

Special Notes

Throughout the process, I found several ways for us to refer to a spreadsheet that we want inside the code. For example, we can use:

var ssId = "<spreadsheet's ID>";
var sheet = SpreadsheetApp.openById(ssId).getSheetByName("<the sheet's name>");

This can select a specific sheet, if your spreadsheet file contains multiple different sheets (tabs).

Or we could use SpreadsheetApp.getActiveSpreadsheet() but this method depends on the active sheet inside the spreadsheet’s UI, as described here.

Nevertheless, both of the methods above are part of the SpreadsheetApp Class.

There are many more things that could be done by the Google Apps Script. It is really helpful for automating anything that we routinely do across many files. In my example that I gave above, the function is only being used on two different spreadsheets — as a placeholder so that I could get the result to be returned to my Telegram bot.

Database integration testing with .NET

2022-01-12T00:00:00+00:00

Ruby on Rails is great. We use it at End Point for many projects with great success. One of Rails’ cool features is how easy it is to write database integration tests. Out of the box, Rails projects come with all the configuration necessary to set up a database that’s exclusive for the automated test suite. This database includes all the tables and other objects that exist within the regular database that the app uses during its normal execution. So, it is very easy to write automated tests that cover the application components that interact with the database.

ASP.NET Core is also great! However, it doesn’t have this feature out of the box. Let’s see if we can’t do it ourselves.

The sample project

As a sample project we will use a REST API that I wrote for another article. Check it out if you want to learn more about the ins and outs of developing REST APIs with .NET. You can find the source code on GitHub.

The API is very straightforward. It provides a few endpoints for CRUDing some database tables. It also provides an endpoint which, when given some vehicle information, will calculate a monetary value for that vehicle. That’s a feature that would be interesting for us to cover with some tests.

The logic for that feature is backed by a specific class and it depends heavily on database interactions. As such, that class is a great candidate for writing a few automated integration tests against. The class in question is QuotesService which is defined in Services/QuoteService.cs. The class provides features for fetching records from the database (the GetAllQuotes method) as well as creating new records based on data from the incoming request and a set of rules stored in the database itself (the CalculateQuote method).

In order to add automated tests, the first step is to organize our project so that it supports them. Let’s do that next.

Organizing the source code to allow for automated testing

In general, the source code of most real world .NET applications is organized as one or more “projects” under one “solution”. A solution is a collection of related projects, and a project is something that produces a deployment artifact. An artifact is a library (i.e. a *.dll file) or something that can be executed like a console or web app.

Our sample app is a stand-alone “webapi” project, meaning that it’s not within a solution. For automated tests, however, we need to create a new project for tests, parallel to our main one. Now that we have two projects instead of one, we need to reorganize the sample app’s source code to comply with the “projects in a solution” structure I mentioned earlier.

Let’s start by moving all the files in the root directory into a new VehicleQuotes directory. That’s one project. Then, we create a new automated tests project by running the following, still from the root directory:

dotnet new xunit -o VehicleQuotes.Tests

That creates a new automated tests project named VehicleQuotes.Tests (under a new aptly-named VehicleQuotes.Tests directory) which uses the xUnit.net test framework. There are other options when it comes to test frameworks in .NET, such as MSTest and NUnit. We’re going to use xUnit.net, but the others should work just as well for our purposes.

Now, we need to create a new solution to contain those two projects. Solutions come in the form of *.sln files and we can create ours like so:

dotnet new sln -o vehicle-quotes

That should’ve created a new vehicle-quotes.sln file for us. We should now have a file structure like this:

.
├── vehicle-quotes.sln
├── VehicleQuotes
│   ├── VehicleQuotes.csproj
│   └── ...
└── VehicleQuotes.Tests
    ├── VehicleQuotes.Tests.csproj
    └── ...

Like I said, the *.sln file indicates that this is a solution. The *.csproj files identify the individual projects that make up the solution.

Now, we need to tell dotnet that those two projects belong in the same solution. These commands do that:

dotnet sln add ./VehicleQuotes/VehicleQuotes.csproj
dotnet sln add ./VehicleQuotes.Tests/VehicleQuotes.Tests.csproj

Finally, we update the VehicleQuotes.Tests project so that it references the VehicleQuotes project. That way, the test suite will have access to all the classes defined in the REST API. Here’s the command for that:

dotnet add ./VehicleQuotes.Tests/VehicleQuotes.Tests.csproj reference ./VehicleQuotes/VehicleQuotes.csproj

With all that setup out of the way, we can now start writing some tests.

You can learn more about project organization in the official online documentation.

Creating a DbContext instance to talk to the database

The VehicleQuotes.Tests automated tests project got created with a default test file named UnitTest1.cs. You can delete it or ignore it, since we will not use it.

In general, it’s a good idea for the test project to mimic the directory structure of the project that it will be testing. Also, we already decided that we would focus our test efforts on the QuoteService class from the VehicleQuotes project. That class is defined in VehicleQuotes/Services/QuoteService.cs, so let’s create a similarly located file within the test project which will contain the test cases for that class. Here: VehicleQuotes.Tests/Services/QuoteServiceTests.cs. These would be the contents:

// VehicleQuotes.Tests/Services/QuoteServiceTests.cs

using System;
using Xunit;

namespace VehicleQuotes.Tests.Services
{
    public class QuoteServiceTests
    {
        [Fact]
        public void GetAllQuotesReturnsEmptyWhenThereIsNoDataStored()
        {
            // Given

            // When

            // Then
        }
    }
}

This is the basic structure for tests using xUnit.net. Any method annotated with a [Fact] attribute will be picked up and run by the test framework. In this case, I’ve created one such method called GetAllQuotesReturnsEmptyWhenThereIsNoDataStored which should give away its intention. This test case will validate that QuoteService’s GetAllQuotes method returns an empty set when called with no data in the database.

Before we can write this test case, though, the suite needs access to the test database. Our app uses Entity Framework Core for database interaction, which means that the database is accessed via a DbContext class. Looking at the source code of our sample app, we can see that the DbContext being used is VehicleQuotesContext, defined in VehicleQuotes/Data/VehicleQuotesContext.cs. Let’s add a utility method to the QuoteServiceTests class which can be used to create new instances of VehicleQuotesContext:

// VehicleQuotes.Tests/Services/QuoteServiceTests.cs

// ...
using Microsoft.EntityFrameworkCore;
using VehicleQuotes.Services;

namespace VehicleQuotes.Tests.Services
{
    public class QuoteServiceTests
    {
        private VehicleQuotesContext CreateDbContext()
        {
            var options = new DbContextOptionsBuilder<VehicleQuotesContext>()
                .UseNpgsql("Host=db;Database=vehicle_quotes_test;Username=vehicle_quotes;Password=password")
                .UseSnakeCaseNamingConvention()
                .Options;

            var context = new VehicleQuotesContext(options);

            context.Database.EnsureCreated();

            return context;
        }

        // ...
    }
}

As you can see, we need to go through three steps to create the VehicleQuotesContext instance and get a database that’s ready for testing:

First, we create a DbContextOptionsBuilder and use that to obtain the options object that the VehicleQuotesContext needs as a constructor parameter. We needed to include the Microsoft.EntityFrameworkCore namespace in order to have access to the DbContextOptionsBuilder. For this, I just copied and slightly modified this statement from the ConfigureServices method in the REST API’s VehicleQuotes/Startup.cs file:

// VehicleQuotes/Startup.cs

public void ConfigureServices(IServiceCollection services)
{
    // ...

    services.AddDbContext<VehicleQuotesContext>(options =>
        options
            .UseNpgsql(Configuration.GetConnectionString("VehicleQuotesContext"))
            .UseSnakeCaseNamingConvention()
            .UseLoggerFactory(LoggerFactory.Create(builder => builder.AddConsole()))
            .EnableSensitiveDataLogging()
    );

    // ...
}

This is a method that runs when the application is starting up to set up all the services that the app uses to work. Here, it’s setting up the DbContext to enable database interaction. For the test suite, I took this statement as a starting point and removed the logging configurations and specified a hardcoded connection string that specifically points to a new vehicle_quotes_test database that will be used for testing.

If you’re following along, then you need a PostgreSQL instance that you can use to run the tests. In my case, I have one running that is reachable via the connection string I specified: Host=db;Database=vehicle_quotes_test;Username=vehicle_quotes;Password=password.

If you have Docker, a quick way to get a Postgres database up and running is with this command:

docker run -d \
    --name vehicle-quotes-db \
    -p 5432:5432 \
    --network host \
    -e POSTGRES_DB=vehicle_quotes \
    -e POSTGRES_USER=vehicle_quotes \
    -e POSTGRES_PASSWORD=password \
    postgres

That’ll spin up a new Postgres instance that’s reachable via localhost.

Secondly, now that we have the options parameter ready, we can quite simply instantiate a new VehicleQuotesContext:

var context = new VehicleQuotesContext(options);

Finally, we call the EnsureCreated method so that the database that we specified in the connection string is actually created.

context.Database.EnsureCreated();

This is the database that our test suite will use.

Defining the test database connection string in the appsettings.json file

One quick improvement that we can do to the code we’ve written so far is move the connection string for the test database into a separate configuration file, instead of having it hardcoded. Let’s do that next.

We need to create a new appsettings.json file under the VehicleQuotes.Tests directory. Then we have to add the connection string like so:

{
  "ConnectionStrings": {
    "VehicleQuotesContext": "Host=db;Database=vehicle_quotes_test;Username=vehicle_quotes;Password=password"
  }
}

This is the standard way of configuring connection strings in .NET. Now, to actually fetch this value from within our test suite code, we make the following changes:

// ...
+using Microsoft.Extensions.Hosting;
+using Microsoft.Extensions.Configuration;
+using Microsoft.Extensions.DependencyInjection;

namespace VehicleQuotes.Tests.Services
{
    public class QuoteServiceTests
    {
        private VehicleQuotesContext CreateDbContext()
        {
+           var host = Host.CreateDefaultBuilder().Build();
+           var config = host.Services.GetRequiredService<IConfiguration>();

            var options = new DbContextOptionsBuilder<VehicleQuotesContext>()
-               .UseNpgsql("Host=db;Database=vehicle_quotes_test;Username=vehicle_quotes;Password=password")
+               .UseNpgsql(config.GetConnectionString("VehicleQuotesContext"))
                .UseSnakeCaseNamingConvention()
                .Options;

            var context = new VehicleQuotesContext(options);

            context.Database.EnsureCreated();

            return context;
        }

        // ...
    }
}

First we add a few using statements. We need Microsoft.Extensions.Hosting so that we can have access to the Host class through which we obtain access to the application’s execution context. This allows us to access the built-in configuration service. We also need Microsoft.Extensions.Configuration to have access to the IConfiguration interface which is how we reference the configuration service which allows us access to the appsettings.json config file. And we also need the Microsoft.Extensions.DependencyInjection namespace which allows us to tap into the built-in dependency injection mechanism, through which we can access the default configuration service I mentioned before. Specifically, that namespace is where the GetRequiredService extension method lives.

All this translates into the few code changes that you see in the previous diff: first getting the app’s host, then getting the configuration service, then using that to fetch our connection string.

You can refer to the official documentation to learn more about configuration in .NET.

Writing a simple test case that fetches data

Now that we have a way to access the database from within the test suite, we can finally write an actual test case. Here’s the GetAllQuotesReturnsEmptyWhenThereIsNoDataStored one that I alluded to earlier:

// ...

namespace VehicleQuotes.Tests.Services
{
    public class QuoteServiceTests
    {
        // ...

        [Fact]
        public async void GetAllQuotesReturnsEmptyWhenThereIsNoDataStored()
        {
            // Given
            var dbContext = CreateDbContext();
            var service = new QuoteService(dbContext, null);

            // When
            var result = await service.GetAllQuotes();

            // Then
            Assert.Empty(result);
        }
    }
}

This one is a very simple test. We obtain a new VehicleQuotesContext instance that we can use to pass as a parameter when instantiating the component that we want to test: the QuoteService. We then call the GetAllQuotes method and assert that it returned an empty set. The test database was just created, so there should be no data in it, hence the empty resource set.

To run this test, we do dotnet test. I personally like a more verbose output so I like to use this variant of the command: dotnet test --logger "console;verbosity=detailed". Here’s what the output looks like.

$ dotnet test --logger "console;verbosity=detailed"
  Determining projects to restore...
  All projects are up-to-date for restore.
  VehicleQuotes -> /app/VehicleQuotes/bin/Debug/net5.0/VehicleQuotes.dll
  VehicleQuotes.Tests -> /app/VehicleQuotes.Tests/bin/Debug/net5.0/VehicleQuotes.Tests.dll
Test run for /app/VehicleQuotes.Tests/bin/Debug/net5.0/VehicleQuotes.Tests.dll (.NETCoreApp,Version=v5.0)
Microsoft (R) Test Execution Command Line Tool Version 16.11.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
/app/VehicleQuotes.Tests/bin/Debug/net5.0/VehicleQuotes.Tests.dll
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.3+1b45f5407b (64-bit .NET 5.0.12)
[xUnit.net 00:00:01.03]   Discovering: VehicleQuotes.Tests
[xUnit.net 00:00:01.06]   Discovered:  VehicleQuotes.Tests
[xUnit.net 00:00:01.06]   Starting:    VehicleQuotes.Tests
[xUnit.net 00:00:03.25]   Finished:    VehicleQuotes.Tests
  Passed VehicleQuotes.Tests.Services.QuoteServiceTests.GetAllQuotesReturnsEmptyWhenThereIsNoDataStored [209 ms]

Test Run Successful.
Total tests: 1
     Passed: 1
 Total time: 3.7762 Seconds

Resetting the state of the database after each test

Now we need to write a test that actually writes data into the database. However, every test case needs to start with the database in its original state. In other words, the changes that one test case does to the test database should not be seen, affect, or be expected by any subsequent test. That will make it so our test cases are isolated and repeatable. That’s not possible with our current implementation, though.

You can read more about the FIRST principles of testing here.

Luckily, that’s a problem that’s easily solved with Entity Framework Core. All we need to do is call a method that ensures that the database is deleted just before it ensures that it is created. Here’s what it looks like:

 private VehicleQuotesContext CreateDbContext()
 {
     var host = Host.CreateDefaultBuilder().Build();
     var config = host.Services.GetRequiredService<IConfiguration>();

     var options = new DbContextOptionsBuilder<VehicleQuotesContext>()
         .UseNpgsql(config.GetConnectionString("VehicleQuotesContext"))
         .UseSnakeCaseNamingConvention()
         .Options;

     var context = new VehicleQuotesContext(options);

+    context.Database.EnsureDeleted();
     context.Database.EnsureCreated();

     return context;
 }

And that’s all. Now every test case that calls CreateDbContext in order to obtain a DbContext instance will effectively trigger a database reset. Feel free to dotnet test again to validate that the test suite is still working.

Now, depending on the size of the database, this can be quite expensive. For integration tests, performance is not as big of a concern as for unit tests. This is because integration tests should be fewer in number and less frequently run.

We can make it better though. Instead of deleting and recreating the database before each test case, we’ll take a page out of Ruby on Rails’ book and run each test case within a database transaction which gets rolled back after the test is done. For now though, let’s write another test case: this time, one where we insert new records into the database.

If you want to hear a more in-depth discussion about automated testing in general, I go into further detail on the topic in this article: An introduction to automated testing for web applications with Symfony.

Writing another simple test case that stores data

Now let’s write another test that exercises QuoteService’s GetAllQuotes method. This time though, let’s add a new record to the database before calling it so that the method’s result is not empty. Here’s what the test looks like:

// ...
using VehicleQuotes.Models;
using System.Linq;

namespace VehicleQuotes.Tests.Services
{
    public class QuoteServiceTests
    {
        // ...

        [Fact]
        public async void GetAllQuotesReturnsTheStoredData()
        {
            // Given
            var dbContext = CreateDbContext();

            var quote = new Quote
            {
                OfferedQuote = 100,
                Message = "test_quote_message",

                Year = "2000",
                Make = "Toyota",
                Model = "Corolla",
                BodyTypeID = dbContext.BodyTypes.Single(bt => bt.Name == "Sedan").ID,
                SizeID = dbContext.Sizes.Single(s => s.Name == "Compact").ID,

                ItMoves = true,
                HasAllWheels = true,
                HasAlloyWheels = true,
                HasAllTires = true,
                HasKey = true,
                HasTitle = true,
                RequiresPickup = true,
                HasEngine = true,
                HasTransmission = true,
                HasCompleteInterior = true,

                CreatedAt = DateTime.Now
            };

            dbContext.Quotes.Add(quote);

            dbContext.SaveChanges();

            var service = new QuoteService(dbContext, null);

            // When
            var result = await service.GetAllQuotes();

            // Then
            Assert.NotEmpty(result);
            Assert.Single(result);
            Assert.Equal(quote.ID, result.First().ID);
            Assert.Equal(quote.OfferedQuote, result.First().OfferedQuote);
            Assert.Equal(quote.Message, result.First().Message);
        }
    }
}

First we include the VehicleQuotes.Models namespace so that we can use the Quotes model class. In our REST API, this is the class that represents the data from the quotes table. This is the main table that GetAllQuotes queries. We also include the System.Linq namespace, which allows us to use various collection extension methods (like Single and First) which we leverage throughout the test case to query lookup tables and assert on the test results.

Other than that, the test case itself is pretty self-explanatory. We start by obtaining an instance of VehicleQuotesContext via the CreateDbContext method. Remember that this also resets the whole database so that the test case can run over a clean slate. Then, we create a new Quote object and use our VehicleQuotesContext to insert it as a record into the database. We do this so that the later call to QuoteService’s GetAllQuotes method actually finds some data to return this time. Finally, the test case validates that the result contains a record and that its data is identical to what we set manually.

Neat! At this point we have what I think is the bare minimum infrastructure when it comes to serviceable and effective database integration tests, namely, access to a test database. We can take it one step further, though, and make things more reusable and a little bit better performing.

Refactoring into a fixture for reusability

We can use the test fixture functionality offered by xUnit.net in order to make the database interactivity aspect of our test suite into a reusable component. That way, if we had other test classes focused on other components that interact with the database, we could just plug that code in. We can define a fixture by creating a new file called, for example, VehicleQuotes.Tests/Fixtures/DatabaseFixture.cs with these contents:

using System;
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;

namespace VehicleQuotes.Tests.Fixtures
{
    public class DatabaseFixture : IDisposable
    {
        public VehicleQuotesContext DbContext { get; private set; }

        public DatabaseFixture()
        {
            DbContext = CreateDbContext();
        }

        public void Dispose()
        {
            DbContext.Dispose();
        }

        private VehicleQuotesContext CreateDbContext()
        {
            var host = Host.CreateDefaultBuilder().Build();
            var config = host.Services.GetRequiredService<IConfiguration>();

            var options = new DbContextOptionsBuilder<VehicleQuotesContext>()
                .UseNpgsql(config.GetConnectionString("VehicleQuotesContext"))
                .UseSnakeCaseNamingConvention()
                .Options;

            var context = new VehicleQuotesContext(options);

            context.Database.EnsureDeleted();
            context.Database.EnsureCreated();

            return context;
        }
    }
}

All this class does is define the CreateDbContext method that we’re already familiar with but puts it in a nice reusable package. Upon instantiation, as seen in the constructor, it stores a reference to the VehicleQuotesContext in its DbContext property.

With that, our QuoteServiceTests test class can use it if we make the following changes to it:

 using System;
 using Xunit;
-using Microsoft.EntityFrameworkCore;
 using VehicleQuotes.Services;
-using Microsoft.Extensions.Hosting;
-using Microsoft.Extensions.Configuration;
-using Microsoft.Extensions.DependencyInjection;
 using VehicleQuotes.Models;
 using System.Linq;
+using VehicleQuotes.Tests.Fixtures;

 namespace VehicleQuotes.Tests.Services
 {
-    public class QuoteServiceTests
+    public class QuoteServiceTests : IClassFixture<DatabaseFixture>
     {
+        private VehicleQuotesContext dbContext;

+        public QuoteServiceTests(DatabaseFixture fixture)
+        {
+            dbContext = fixture.DbContext;
+        }

-        private VehicleQuotesContext CreateDbContext()
-        {
-            var host = Host.CreateDefaultBuilder().Build();
-            var config = host.Services.GetRequiredService<IConfiguration>();

-            var options = new DbContextOptionsBuilder<VehicleQuotesContext>()
-                .UseNpgsql(config.GetConnectionString("VehicleQuotesContext"))
-                .UseSnakeCaseNamingConvention()
-                .Options;

-            var context = new VehicleQuotesContext(options);

-            context.Database.EnsureDeleted();
-            context.Database.EnsureCreated();

-            return context;
-        }

         [Fact]
         public async void GetAllQuotesReturnsEmptyWhenThereIsNoDataStored()
         {
             // Given
-            var dbContext = CreateDbContext();

             // ...
         }

         [Fact]
         public async void GetAllQuotesReturnsTheStoredData()
         {
             // Given
-            var dbContext = CreateDbContext();

             // ...
         }
     }
 }

Here we’ve updated the QuoteServiceTests class definition so that it inherits from IClassFixture<DatabaseFixture>. This is how we tell xUnit.net that our tests use the new fixture that we created. Next, we define a constructor that receives a DatabaseFixture object as a parameter. That’s how xUnit.net allows our test class to access the capabilities provided by the fixture. In this case, we take the fixture’s DbContext instance, and store it for later use in all of the test cases that need database interaction. We also removed the CreateDbContext method because now that’s defined within the fixture. We also removed a few using statements that became unnecessary.

One important aspect to note about this fixture is that it is initialized once per whole test suite run, not once per test case. Specifically, the code within the DatabaseFixture’s constructor gets executed once, before all of the test cases. Similarly, the code in DatabaseFixture’s Dispose method get executed once at the end, after all test cases have been run.

This means that our test database deletion and recreation step now happens only once for the entire test suite. This is not good with our current implementation because that means that individual test cases no longer run with a fresh, empty database. This can be good for performance though, as long as we update our test cases to run within database transactions. Let’s do just that.

Using transactions to reset the state of the database

Here’s how we update out test class so that each test case runs within a transaction:

 // ...

 namespace VehicleQuotes.Tests.Services
 {
-    public class QuoteServiceTests : IClassFixture<DatabaseFixture>
+    public class QuoteServiceTests : IClassFixture<DatabaseFixture>, IDisposable
     {
         private VehicleQuotesContext dbContext;

         public QuoteServiceTests(DatabaseFixture fixture)
         {
             dbContext = fixture.DbContext;

+            dbContext.Database.BeginTransaction();
         }

+        public void Dispose()
+        {
+            dbContext.Database.RollbackTransaction();
+        }

         # ...
     }
 }

The first thing to note here is that we added a call to BeginTransaction in the test class constructor. xUnit.net creates a new instance of the test class for each test case. This means that this constructor is run before each and every test case. We use that opportunity to begin a database transaction.

The other interesting point is that we’ve updated the class to implement the IDisposable interface’s Dispose method. xUnit.net will run this code after each test case, so we rollback the transaction.

Put those two together and we’ve updated our test suite so that every test case runs within the context of its own database transaction. Try it out with dotnet test and see what happens.

To learn more about database transactions with Entity Framework Core, you can look at the official docs.

You can learn more about xUnit.net’s test class fixtures in the samples repository.

Alright, that’s all for now. It is great to see that implementing automated database integration tests is actually fairly straightforward using .NET, xUnit.net, and Entity Framework. Even if it isn’t quite as easy as it is in Rails, it is perfectly doable.

Forwarding Google Forms responses to an external API

2021-11-16T00:00:00+00:00

Google Forms is a great form service that many people use for surveys, research, questionnaires, etc. It has an intuitive and flexible interface for building forms and is fairly easy to use for everyone. Once you get a response, you can view the results in the admin section of the form or in a Google Sheets document in which Google will automatically insert all your responses.

However, you may need to do something else with the responses. For example, what if you want to have the response printed in your Slack channel or Discord server? Or what if you want to use the raw data to make more complex visualizations than Google Sheets is capable of?

Google Apps Script to the rescue!

Google Apps Script is a development platform for building add-ons for Google products, such as Google Sheets, Google Docs, and Google Forms. You write your JavaScript in the code editor that Google provides for you, so there is nothing to install on your local machine to start developing. This code then gets executed on Google’s servers. These Google Apps Script projects can then be published as a Google Workspace add-on that others can use or shared within your organization.

Even though Google Apps Script is basically just JavaScript, there are a few key differences from a Node.js project. For example, in a Node.js environment you could use the https module to send an HTTP request to another external service (or you could grab a package like axios or node-fetch that can do the same thing). In Google Apps Script, however, you cannot use the https module because it is not a Node.js project per se and does not provide you with the ability to install external NPM packages. Instead, there is a limited set of standard libraries that comes built-in within a Google Apps Script project.

1. Start a new Google Apps Script project

Head to Google Forms and create a new form.
Click the triple-dots icon at the top-right corner of the form and choose “Script Editor”.
You should now see the Google Apps Script editor. Great! Let’s change the project name to something more descriptive (e.g. “Forward to API”).

2. Add triggers to the script for initialization

Google Apps Script allows you to install triggers to the current form. The ones we are interested in right now are onOpen, which runs when a user that has edit access opens the form, and onInstall, which runs when a user installs the add-on in the Google Forms form. There are other events that you might be interested in listed here.

We will also need to provide a way for users to set the outgoing URL for each response. The easiest would be to hardcode it in the code, but we can make our add-on a little bit more configurable; let’s also create a sidebar that contains the configuration interface for the users to change the settings for our add-on, in this case the destination URL for the responses.

Now, in the Google Apps Script code editor, paste the following code:

/**
 * Adds a custom menu to the active form to show the add-on sidebar.
 */
function onOpen(e) {
  FormApp.getUi()
    .createAddonMenu()
    .addItem('Configure', 'showSidebar')
    .addToUi();
}

/**
 * Runs when the add-on is installed.
 */
function onInstall(e) {
  onOpen(e);
}

/**
 * Opens a sidebar in the form containing the add-on's user interface for
 * configuring the notifications this add-on will produce.
 */
function showSidebar() {
  var sidebarPage = HtmlService.createHtmlOutputFromFile('sidebar')
    .setTitle('Your add-on configuration');
  FormApp.getUi().showSidebar(sidebarPage);
}

// Save settings

// Load settings

First, we would like to have a menu item on the form to access the add-on. This is done through the createAddonMenu().addItem function within the onOpen trigger. We also have the onInstall trigger that does the same thing as what onOpen is doing. With these, both installing the add-on for the first time and opening the form creates an add-on menu item called “Configure”.

createAddonMenu().addItem accepts two arguments, a label (Configure) and a function name to be executed when the item is selected (showSidebar). In the code above, it will run the showSidebar function. showSidebar initializes the add-on’s view by loading and running the HTML file we specified in createHtmlOutputFromFile. Since we are passing sidebar to the function, it will automatically assume the filename sidebar.html and load the file from our project.

In the previous step, we specified our menu item to show the sidebar. Now, let’s go ahead and create a new file called sidebar.html.

On the Files panel, click the + icon and choose HTML in the dropdown menu.
Name the file sidebar.html when prompted. This is important as we have specified the name sidebar to the createHtmlOutputFromFile function within showSidebar.
Open the HTML file and paste the following HTML content:

<!DOCTYPE html>
<html>
  <head>
    <base target="_top">
    <style>
      body {
        font-family: sans-serif;
        font-size: 14px;
      }

      input {
        border: 1px solid #3f3f3f;
        padding: 0.25rem;
        width: 100%;
      }

      #error {
        margin-top: 0.5rem;
      }

      .input-field {
        margin-bottom: 1rem;
      }
    </style>
  </head>
  <body>
    <form>
      <div class="input-field">
        <label for="url">URL to send responses to:</label>
        <input id="url" type="text" name="url" placeholder="e.g. https://some-api.com/accept/responses"/>
      </div>
      <div class="block" id="button-bar">
        <button class="action" id="save-settings">Save</button>
        <p id="response"></p>
      </div>
    </form>
    
    <script src="//ajax.googleapis.com/ajax/libs/jquery/3.6.0/jquery.min.js">
    </script>
    <script>
    // Our JavaScript code
    </script>
  </body>
</html>

Go back to your Google Forms form and refresh. You should now get an add-on menu on your form, with your add-on listed in the dropdown.
Choose your add-on and click “Configure”. This is the add-on menu item that we declared in step 1.
You should now be prompted to authorize the script. Follow the instructions to allow the script permission to run on your Google Forms forms.
Once you’re done, go to the add-on menu, choose your add-on, and choose “Configure” again. You should now see a sidebar on the right side of the form with a text field to fill in the destination URL and a submit button to save the settings.

4. Saving your configuration

Now we need to be able to save the URL we want to send the responses to and create a form trigger out of that. Let’s go ahead and add that:

Add the following script in the <script> block under the // Our JavaScript code comment.

// sidebar.html
$(function() {
  // Load settings from the server
  loadSettingsAndPopulateForm();

  // Listen to 'submit' event
  $('form').submit(saveSettingsToServer);
});

function loadSettingsAndPopulateForm() {
  google.script.run
      .withSuccessHandler(
        function(settings) {
          $('#url').val(settings.url)
        })
      .withFailureHandler(
        function(msg) {
          $('#response').text('Failed to fetch settings. ERROR: ' + msg);
        })
      .fetchSettings();
}

function saveSettingsToServer(event) {
  event.preventDefault();
  var button = $(this).find('button');
  button.attr('disabled', 'disabled');
  var settings = {
    'url': $('#url').val(),
  };

  // Save the settings on the server
  google.script.run
      .withSuccessHandler(
        function(msg, button) {
          button.removeAttr('disabled');
          $('#response').text('Saved settings successfully');
        })
      .withFailureHandler(
        function(msg, button) {
          button.removeAttr('disabled');
          $('#response').text('Failed to save settings. ERROR: ' + msg);
        })
      .withUserObject(button)
      .saveSettings(settings);
}

// Code.gs

/**
 * Used by the client-side via `google.script.run` to save setings from the form.
 */
function saveSettings(settings) {
  PropertiesService.getDocumentProperties().setProperties(settings);
  // adjustFormSubmitTrigger();
}

/**
 * Used by the client-side via `google.script.run` to load saved setings.
 */
function fetchSettings() {
  return PropertiesService.getDocumentProperties().getProperties();
}

Now, let’s go through the code:

The $(function() {}) block is run at document load. Two things happen here:

The loadSettingsAndPopulateForm function runs the fetchSettings function from the backend. Then it populates the text input field #url with the saved settings.
It installs an event listener on the configuration form submit to save the settings to the server. This listener, the saveSettingsToServer function, gathers all the values from the input fields, and runs the saveSettings function on the backend, with the input field values as the arguments.

google.script.run

google.script.run is the glue between frontend (the *.html files) and the backend (the *.gs) files. Once you declare a function (e.g. doStuff) on backend, you can use the function through google.script.run.doStuff from the frontend. withSuccessHandler and withFailureHandler are where you supply the callbacks for successful calls and failed calls to the backend respectively.

To see more, read the documentation.

5. Installing the form submit event handler

Now that we’re able to pinpoint the address we want send the form data to, we can start instructing Google Forms to send the data our way when someone submits the form. In order to do that, we:

Code the form submit trigger to submit the data on the form response.
If there’s no existing form submit trigger and the URL is set, install the form submit trigger.
If there is an existing form submit trigger and the URL is unset, remove the trigger.

Let’s write the form submit trigger first. Paste this code at the end of Code.gs:

function sendResponse(e) {
  var data = {
    "form": {
      "id": e.source.getId(),
      "title": e.source.getTitle() ? e.source.getTitle() : "Untitled Form",
      "is_private": e.source.requiresLogin(),
      "is_published": e.source.isAcceptingResponses(),
    },
    "response": {
      "id": e.response.getId(),
      "email": e.response.getRespondentEmail(),
      "timestamp": e.response.getTimestamp(),
      "data": e.response.getItemResponses().map(function(y) {
        return {
          h: y.getItem().getTitle(),
          k: y.getResponse()
        }
      }, this).reduce(function(r, y) {
        r[y.h] = y.k;
        return r
      }, {}),
    }
  };

  var options = {
    method: "post",
    payload: JSON.stringify(data),
    contentType: "application/json; charset=utf-8",
  };

  var settings = PropertiesService.getDocumentProperties();
  UrlFetchApp.fetch(settings.getProperty('url'), options);
};

Let’s break it down. First, we compile the data that we want to send to our application. Form triggers accept a variable in their parameter, which we named e here. e is an Event Object, and it provides us access to the form’s information (via e.source) as well as the responses to the form (via e.response). We gather all this in an object (called data in this case) and submit it to our app via the UrlFetchApp.

UrlFetchApp is a standard library in Google Apps Script. It allows us to make HTTP requests to other applications. Here, we are doing a POST HTTP request to the URL that we set (obtained via PropertiesService.getDocumentProperties.getProperty('url')).

Adjusting the form trigger

Right now, sendResponse is not hooked to any function. Let’s hook it to the form trigger with this code:

function adjustFormSubmitTrigger() {
  var form = FormApp.getActiveForm();
  var triggers = ScriptApp.getUserTriggers(form);
  var settings = PropertiesService.getDocumentProperties();
  var url = settings.getProperty('url')
  var triggerNeeded = url && url.length > 0;
  
  // Create a new trigger if required; delete existing trigger
  //   if it is not needed.
  var existingTrigger = null;
  for (var i = 0; i < triggers.length; i++) {
    if (triggers[i].getEventType() == ScriptApp.EventType.ON_FORM_SUBMIT) {
      existingTrigger = triggers[i];
      break;
    }
  }
  if (triggerNeeded && !existingTrigger) {
    var trigger = ScriptApp.newTrigger('sendResponse')
      .forForm(form)
      .onFormSubmit()
      .create();
  } else if (!triggerNeeded && existingTrigger) {
    ScriptApp.deleteTrigger(existingTrigger);
  }
}

Don’t forget to also uncomment // adjustFormSubmitTrigger in the saveSettings function.

This function is pretty straightforward. It checks if there is a URL saved in the settings and installs the form submit trigger if needed. If there is no URL saved or if it’s an empty string, then any existing form submit trigger will be deleted. Changing the URL will not do anything to the existing form submit trigger, since sendResponse will always pull the latest URL from the settings.

6. Test the triggers

Now we can test the submission. Let’s configure our add-on with a valid URL to our app, add a couple of questions to our form, then hit Preview in the top navigation bar to test our form submit trigger.

Once we’ve filled the form, hit Submit. You should now see your form response gets sent to your destination URL as a POST request.

Here’s what I get from a simple Express app I developed to receive the response:

Example app listening at http://localhost:4000

{
  "form": {
    "id": "1zm_anJLsYnnmI-MlRqLcEhK8Eemd90rg_oJmsThiONw",
    "title": "HR Benefits Survey",
    "is_private": true,
    "is_published": true
  },
  "response": {
    "id": "2_ABaOnufmy5dDzYsUl3-G5vlkQaOPoW-Jf5Wskk9SZHhXvpgnTzx6LGVq9YGDqivvo6TXpko",
    "email": "",
    "timestamp": "2021-11-07T08:56:08.299Z",
    "data": {
      "What's the most important thing for you?": "4-day workweek",
      "Select games you'd like to have at the office": [
        "Foosball",
        "Dart"
      ]
    }
  }
}

Conclusion

That’s it! We can now send Google Forms responses to our app for better data processing and visualization.

Salesforce Integration with Node.js

2020-03-27T00:00:00+00:00

Photo by Dylan Wooters, 2020

Salesforce is huge. It is currently the dominant customer relationship management (CRM) provider, accounting for around 20% of market share. Businesses are using Salesforce not only as a traditional CRM solution, but also for novel purposes. Salesforce can serve as a backend database and admin portal for custom apps, or as a reporting tool that pulls data from various systems.

This growth leads to increasing demand for Salesforce integrations. The term “Salesforce integration” may conjure up images of expensive enterprise software or dense API documentation, but it doesn’t have to be that way. You can work with Salesforce easily using Node.js and the npm package JSforce. An example of a project that might benefit from this kind of Node.js integration is an e-commerce website where order data is loaded to and from Salesforce for order fulfillment, tracking, and reporting.

In this post we’ll cover how to connect to Salesforce using JSforce, the basics of reading and writing data, as well as some advanced topics like working with large amounts of data and streaming data with Socket.IO.

Setting Up

You’ll first want to install Node.js on your local machine, if you haven’t done so already.

Next, create your Node app. This will vary with your requirements. I often use Express to build a REST API for integration purposes. Other times, if I am routinely loading data into Salesforce, I will create Node scripts and schedule them using cron. For the purposes of this post, we will create a small Node script that can be run on the command line.

Create a new directory for your project, and within that directory, run npm init to generate your package.json file. Then install JSforce with npm install jsforce.

Finally, create a file named script.js, which we will run on the command line for testing. To test the script at any time, simply navigate to your app’s directory and run node script.js.

At the top of the script, require jsforce, as well as the Node IO libraries fs and path. Then define an asynchronous function that will serve as our script body. This is where all of your Salesforce code will go.

var jsforce = require('jsforce');
var fs = require('fs');
var path = require('path');

run();
async function run() {
    // salesforce code goes here...
}

Connecting to Salesforce

I usually store my Salesforce credentials and instance URL as a JSON object in a separate file, which I gitignore. This ensures that sensitive data does not appear in Git. Below is the content of my salesforce-creds.json file. You’ll want to add your Salesforce username and password and update the instance URL, if necessary.

{
    "username": [your username],
    "password": [your password],
    "url": "https://na111.salesforce.com"
}

To connect to Salesforce simply retrieve the credentials from the file and use them with the JSforce Connection class to login. Be sure to wrap all JSforce code in a try-catch block, to catch any errors coming back from Salesforce.

let creds = JSON.parse(fs.readFileSync(path.resolve(__dirname, './salesforce-creds.json')).toString());
let conn = new jsforce.Connection({
    loginUrl: creds.url
});
try {
    await conn.login(creds.username, creds.password);
    console.log('Connected to Salesforce!');
    // now you can use conn to read/write data...
    await conn.logout();
} catch (err) {
    console.error(err);
}

Reading, Writing, and Deleting Data

Once connected, the easiest way to query data from Salesforce is to use the JSforce query function, and pass in an SOQL statement. This offers the most flexibility, as you can run queries for child and parent objects. Using SOQL, we can query all accounts and their contacts (children) in a single statement. Note, however, that there are limitations on relationship queries. You can only go down one level, from parent to child, but you can go up multiple levels from child to parent.

Writing and deleting data is simple with JSforce using the sobject class and the corresponding create/update/delete function. In the example below, we will query for accounts and contacts using SOQL, and then isolate and update a specific contact using sobject().update.

let soql = `select id, name,
    (SELECT Id, FirstName, LastName, Email_Verified__c, Enrollment_Status__c from Contacts)
    FROM Account`;
let accounts = await conn.query(soql);
let cooper = accounts.records
    .filter(x => x.Name === 'Twin Peaks Sheriff Dept.')[0].Contacts.records
    .filter(y => y.FirstName === 'Dale' && y.LastName === 'Cooper')[0];
console.log(cooper);

// Console output:
// {
//     attributes: {
//         type: 'Contact',
//         url: '/services/data/v42.0/sobjects/Contact/0033h000001sDzDAAU'
//     },
//     Id: '0033h000001sDzDAAU',
//     FirstName: 'Dale',
//     LastName: 'Cooper',
//     Email_Verified__c: true,
//     Enrollment_Status__c: 'Pending'
// }

cooper.Enrollment_Status__c = 'Accepted';
let ret = await conn.sobject('Contact').update(cooper);
if (ret.success) {
    console.log('Contact updated in Salesforce.');
}

Working with Large Amounts of Data

You may need to read and write large amounts of data, for example if you are using Salesforce for reporting and loading data to and from other systems.

Event-driven Querying

The record limit for standard promise-style SOQL querying, as in our example above, is 2000 records. To query more than that, it is best to shift to the event-driven style of querying. This will ensure that all records are successfully retrieved from Salesforce. You can use the maxFetch property to set the upper limit of records returned. By default, maxFetch is set to 10,000.

let contacts = [];
let soql = 'SELECT Id, FirstName, LastName, Email_Verified__c, Enrollment_Status__c from Contact';
let query = await conn.query(soql)
    .on("record", (record) => {
        contacts.push(record);
    })
    .on("end", async () => {
        console.log(`Fetched Contacts. Total records fetched: ${contacts.length}`);
    })
    .on("error", (err) => {
        console.error(err);
    })
    .run({
        autoFetch: true,
        maxFetch: 5000
    });

Loading Data with the Bulk API

Loading a large amount of data into Salesforce is best accomplished through the Bulk API via JSforce. There are a couple good reasons for this approach.

The Bulk API has better performance over other methods when working with large collections of objects.

The standard JSforce sobject create/update/delete functions have a 200 object limit. For operations on large collections, you must divide the total by 200, resulting in many separate API calls. By contrast, the Bulk API only uses a single API call. Since Salesforce imposes API limits, this makes the Bulk API a better choice.

Running a bulk operation is simple using the bulk.load method, which takes three parameters: the Salesforce object type, the operation type, and an array of objects. The method returns an array of objects with success/errors fields, as well as the id of the newly created object, if successful.

If you’re working with thousands of objects, it’s good to set the pollTimeout property manually to one minute or more, to avoid Salesforce connection timeouts. Also note that the possible values for operation type are: ‘insert’, ‘update’, ‘upsert’, ‘delete’, or ‘hardDelete’.

// set poll timeout to one minute for larger datasets
sfConnection.bulk.pollTimeout = 240000;

// normally you will have thousands of Accounts, this is just an example
let accounts = [{
        Name: 'Saul Goodman, LLC'
    },
    {
        Name: 'Los Pollos Hermanos Inc'
    },
    {
        Name: 'Hamlin, Hamlin & McGill'
    }
];
let results = await conn.bulk.load('Account', 'insert', accounts);
console.log(results);

// Console output:
// [
//     { id: '0013h000006bdd2AAA', success: true, errors: [] },
//     { id: '0013h000006bdd3AAA', success: true, errors: [] },
//     { id: '0013h000006bdd4AAA', success: true, errors: [] }
// ]

if (accounts.length === results.filter(x => x.success).length) {
    console.log('All account successfully loaded.');
}

WebSocket Streaming with Socket.io

Say you are building a web application for reporting, and the app contains a dashboard with data on all of your contacts in Salesforce. You want the dashboard to be updated whenever the data in Salesforce changes, and you also want this to happen without refreshing the web page.

To accomplish this, you can stream real-time data from Salesforce using JSforce and the Socket.IO library, which makes working with WebSockets quite simple.

The first step in this process is creating a PushTopic in Salesforce. This is basically a trigger that emits a notification anytime an object is created, updated, etc. in Salesforce. I created a PushTopic for Contacts by running the following Apex code in the Salesforce developer console.

PushTopic pushTopic = new PushTopic();
pushTopic.Name = 'UserChange';
pushTopic.Query = 'SELECT Id, FirstName, LastName, Email_Verified__c, Enrollment_Status__c FROM Contact';
pushTopic.ApiVersion = 48.0;
pushTopic.NotifyForOperationCreate = true;
pushTopic.NotifyForOperationUpdate = true;
pushTopic.NotifyForOperationUndelete = true;
pushTopic.NotifyForOperationDelete = true;
pushTopic.NotifyForFields = 'Referenced';
insert pushTopic;

Then, back in your Node app, install Express and Socket.IO.

Next, you’ll want to create a very basic Express server that will listen for updates from the Salesforce PushTopic, and emit them to your reporting site. Start by installing Express and Socket.IO.

npm install express
npm install socket.io

Then delete the run function in your script.js file, which contained the code from the samples above, and replace it with the following:

async function run() {
    // listen with express
    server.listen(3000, function() {
        console.log('listening on *:3000');
    });

    // connect to Salesforce
    let creds = JSON.parse(fs.readFileSync(path.resolve(__dirname, './salesforce-creds.json')).toString());
    let conn = new jsforce.Connection({
        loginUrl: creds.url
    });
    try {
        await conn.login(creds.username, creds.password);
    } catch (err) {
        console.error(err);
    }

    // when the client connects, emit streaming updates from salesforce to client
    io.on("connection", (socket) => {
        console.log('A socket connection was made!');
        let eventHandler = (message) => {
            console.log('New streaming event received from Salesforce:', message);
            socket.emit('UserChange', message);
        };
        conn.streaming.topic('UserChange').subscribe(eventHandler);
    });
}

Here is a step-by-step description of what is occurring in the code sample above:

The Express server is set to listen for connections on port 3000.
We connect to Salesforce and login.
Socket.IO is set to listen for incoming connections from clients.
A function called eventHandler that emits Salesforce streaming messages to the client is defined.
When a connection is made, eventHandler is attached to the Salesforce streaming topic as a callback, using the live Salesforce connection.

If you follow the nice little tutorial from Socket.IO and create the sample chat webpage, you can actually test the Salesforce streaming updates. In the chat page, add this script, which will log messages coming back from Salesforce.

<script>
    var socket = io();
    socket.on('UserChange', function(msg) {
        console.log(msg);
    });
</script>

Then update a contact in Salesforce, changing the contact’s first name. If everything works correctly, you should see the client connect via Socket.IO in the Node logs, and also see a streaming message from Salesforce logged in the browser’s console window.

Summary

Node.js and JSforce provide a straightforward and elegant way to interact with Salesforce. Whether you have an existing Node API that needs to work with Salesforce, or you are building a new application that is powered by Salesforce data, consider the recipes above as stepping stones for completing your project.

Data Center Relocation Initiatives: Daunting But Achievable

2018-12-20T00:00:00+00:00

Photo by Seeweb · CC BY-SA 2.0, cropped

Introduction

A customer asked for our help dealing with logistical nightmares they encountered during a hardware update and data center relocation project. The customer had two active data centers, and wanted to relocate one of them to a modern Tier 4 facility to improve its performance and provide redundancy for critical systems. They also wanted to consolidate or decommission equipment to reduce recurring expenses and reduce their carbon footprint. The client decided to move to a Tier 4 data center because they provide across-the-board redundancy within the data center.

Moving forward with Tier IV

Enterprises typically transition to Tier 4 data centers because they offer the highest uptime guarantees, and have no single points of failure. These facilities are fully redundant in terms of electrical circuits, cooling, and networking. This architecture is best able to withstand even the most serious technical incidents without server availability being affected. Tier IV facilities have contracts with disaster management companies who will provide them, for example, with fuel in the event that a natural disaster damages the power grid.

(More information about the four data center levels is at the end of this post.)

Key Issues

Some of the customer’s key concerns were:

1. To safely move their virtual environment to the new data center.

2. To protect assets during the relocation.

3. To completely shut down the current data center and migrate seamlessly to the new data center.

4. To update external DNS records without causing any downtime for the web applications used by their customers.

Project Planning

To manage data center relocations is challenging because there are many moving parts and variables. Typically, planning starts 6 months ahead of the scheduled relocation date. We seek to understand the role of each system by consulting in-depth with all the teams involved. We analyze each part of the client’s tech infrastructure, break it down, and strategize about the best approach for handling the migration of each component. Additional time to plan can help especially when a large group of stakeholders is involved. Successful planning requires hard and sometimes extensive work, but makes an otherwise overwhelming data center migration smooth and worry free.

Here are some of our procedures for pre-migration preparation:

1. Process the paperwork for new hardware. Identify, review, and approve all applications. Hire movers. Schedule and coordinate with internal management and key stakeholders like IT staff, system admins, contractors, circuit providers, etc.

2. Collect and analyze resource usage associated with the critical and non-critical systems and compare resource needs to resource availability at the new facility.

3. Design and schedule the migration of critical systems to the new data center. Discuss the plan with key stakeholders.

4. Make sure replicas of critical systems are up to date.

5. Set up a new environment with new hardware in the new data center and establish network connections.

6. Prepare hardware and virtualization technology in the new data center.

7. Test, retest, and burn in the new hardware (2–4 weeks of testing).

8. Create a non-essential server within the virtualization environment and test it.

A Challenge: Eliminating Downtime

To minimize or eliminate downtime requires very careful preparation. With this particular client, we had the luxury of their already having a secondary data center, which could accommodate a majority of the critical systems during the migration.

We originally planned to leverage this by simply changing the external DNS records to point to this data center during the migration, but during planning determined that the external DNS record would not replicate worldwide quickly enough to meet the uptime needs of their global operation.

Solutions

After discussing various approaches, we ended up using a load balancer. A load balancer is a network device that routes incoming traffic destined for a single destination (web site, application, or service) and ‘shares’ the incoming connections across multiple destination devices or services. In this case, the data center we were to migrate to housed the client’s critical systems like web servers, database, etc. The secondary data center housed a replica of the critical systems using a replication technology, e.g. Actifio, Rubrik, SAN mirroring, VMWare SRM, or Zerto — we used Actifio in this scenario.

The customer had multiple systems with external DNS records and public IP addresses. We did not change the public DNS names for the critical systems. We did change the public IP addresses associated with the public DNS names, to point to the load balancer, which then pointed to the appropriate data center throughout the process. This setup enabled a gradual, flexible, and fully controlled transition to the new data center.

Summary

Our client wanted to reduce their servers’ power usage 40% by consolidating into a hyper-converged system. Other goals were to reduce floor space by 50%, and monthly recurring data center charges by 50%. We transitioned them to a new hardware stack (compute, storage, network, backup environment) in a Tier 4 data center, and saved them 50% percent on data center operation costs. Additionally, during the process we corrected issues associated with disaster recovery and business continuity, leaving the client’s operations more resilient, affordable, and more secure.

Appendix: Data Center Classifications

Tier IV (Best)

Tier IV data centers have redundancies for every process and data protection stream. No single outage or error can shut down the system. To keep Tier IV ranking, data centers must have 99.995% minimum uptime per year. They come equipped with 2N+1 infrastructure (two times the amount required for operation plus a backup), which is considered “fully redundant.”

Tier III (Average)

N+1 (the amount required for operation plus a backup) fault tolerance, which means a Tier III facility can undergo routine maintenance without a hiccup in operations. 99.982% minimum uptime per annum, used for maintenance and overwhelming emergencies.

Tier II (Minimal)

99.749% minimum uptime. Tier II facilities have considerably more downtime than Tier III facilities, primarily because Tier II facilities have less redundancy. They do however have partial cooling redundancy and multiple power redundancies.

Tier I (Basic, Cheapest)

99.671% minimum uptime, with 28.8 hours of downtime each year.

Meet the End Point Windows consulting group

2018-07-27T00:00:00+00:00

As the share of internet traffic passing through mobile devices hovers around 50%, it’s easy to forget that a lot of computing still happens on desktops and laptops—and perhaps even easier to forget that the majority of those desktops and laptops are running Microsoft Windows.

At End Point, we take pride in being a multi-platform organization and using open source technologies in real-world business environments. Our internal chat service, wiki, and many other libraries and tools we employ across our client base are all open source. However, for various reasons many companies choose Microsoft’s solutions including long-time standards Windows Server, Active Directory, and Exchange.

In April 2017, Dan Briones came to End Point to head up the company’s new Windows infrastructure services. In 2018 we added two new engineers to the mix, Chris Hopkins and Charles Chang, to form a full-service Windows consulting team, all local to our New York City office on Park Avenue.

Dan Briones

Dan is the team lead and has over three decades of hands-on experience in IT systems management, systems integrations, migrations, virtualization, networking, security, orchestration, security, compliance, and maintenance, specializing in the Microsoft Windows ecosystem. He also develops applications using the .NET framework and SQL Server database. Dan works with many of our local New York clients, focusing on their network and infrastructure.

Charles Chang

Charles has 21 years of experience in IT ranging from managing Windows infrastructure, to datacenter relocation and hardware refresh initiatives, office relocation, virtualization, disaster recovery and business continuity strategy. Charles has worked extensively with major vendors such as VMware, IBM, EMC, Cisco, and Dell in building enterprise environments in the datacenter. He also has experience in implementing data loss prevention technology, web and email filtering to protect and provide a healthy infrastructure.

Chris Hopkins

Chris has 10 years of experience in IT, network support, and development. He has extensive cross-platform development and hardware knowledge. Chris excels at mail migrations, Active Directory and Windows Server environments, site-to-site website migrations, and cloud-based phone systems.

Learn more about the services our Windows consulting group offers and talk to us today about how we can help you too!

Integration Experiences

2012-06-13T00:00:00+00:00

Szymon Guz gave a talk which covered his experiences working for a large client on integrating an acquired business’ website and fulfillment processes into their existing backend and fulfillment systems.

He gave us a detailed overview of their existing shipping fulfillment infrastructure, along with specific technical issues he encountered when integrating with the fulfillment models of the additional website.

He also detailed some of the non-technical/project management challenges inherent when working with a large number of people on a project.

Testing Ansible Automation with Molecule

1. Why Test with Molecule?

2. Install Prerequisites

3. Ansible Monorepo Structure

4. Creating Your First Molecule Scenario

5. Running Molecule tests

6. Integrating Molecule with GitLab CI/CD

Middleware: Is that still a thing?

What is it?

What does it do?

Why do we care?

Creating Telegram bots with Google Apps Script

Translation Bot

Currency Converter Bot

Special Notes

Database integration testing with .NET

The sample project

Organizing the source code to allow for automated testing

Creating a DbContext instance to talk to the database

Defining the test database connection string in the appsettings.json file

Writing a simple test case that fetches data

Resetting the state of the database after each test

Writing another simple test case that stores data

Refactoring into a fixture for reusability

Using transactions to reset the state of the database

Forwarding Google Forms responses to an external API

Google Apps Script to the rescue!

1. Start a new Google Apps Script project

2. Add triggers to the script for initialization

3. Adding the configuration sidebar

Test your sidebar

4. Saving your configuration

google.script.run

5. Installing the form submit event handler

Adjusting the form trigger

6. Test the triggers

Conclusion

Salesforce Integration with Node.js

Setting Up

Connecting to Salesforce

Reading, Writing, and Deleting Data

Working with Large Amounts of Data

Event-driven Querying

Loading Data with the Bulk API

WebSocket Streaming with Socket.io

Summary

Data Center Relocation Initiatives: Daunting But Achievable

Introduction

Moving forward with Tier IV

Key Issues

Project Planning

A Challenge: Eliminating Downtime

Solutions

Summary

Appendix: Data Center Classifications

Tier IV (Best)

Tier III (Average)

Tier II (Minimal)

Tier I (Basic, Cheapest)

Meet the End Point Windows consulting group

Dan Briones

Charles Chang

Chris Hopkins

Integration Experiences