How to Analyze Application Logs and Extract Actionable Insights

2025-02-28T00:00:00+00:00

Logs often accumulate unnoticed—until something breaks. Then, they suddenly become vital tools for diagnosing issues. By combining just a few command-line techniques, you can quickly spot recurring problems, identify suspicious activity, and strengthen your application’s defenses.

Below is a sample error log we’ll reference in the examples:

[2025-01-01 12:34:56] [client 192.168.1.1:12345] PHP Notice: Undefined variable $foo in /var/www/html/index.php on line 45 | referer: http://example.com/index.php
[2025-01-01 12:34:56] [client 192.168.1.1:12345] PHP Notice: Undefined variable $foo in /var/www/html/index.php on line 45 | referer: http://example.com/index.php
[2025-01-01 12:34:57] [client 192.168.1.1:12345] PHP Notice: Undefined variable $foo in /var/www/html/index.php on line 45 | referer: http://example.com/index.php
[2025-01-01 12:35:00] [client 10.0.0.2:6789] PHP Warning: Division by zero in /var/www/html/script.php on line 23 | referer: http://example.com/script.php
[2025-01-01 13:35:01] [client 10.0.0.2:6789] PHP Warning: Division by zero in /var/www/html/script.php on line 23 | referer: http://example.com/script.php
[2025-01-01 13:36:10] [client 192.168.1.1:12345] PHP Fatal error: Call to undefined function baz() in /var/www/html/lib.php on line 78 | referer: http://example.com/index.php
[2025-01-01 14:36:11] [client 172.16.0.5:54321] PHP Fatal error: Call to undefined function baz() in /var/www/html/lib.php on line 78 | referer: http://example.com/index.php

A Note on Commands

Throughout this blog, you’ll notice repeated use of commands like awk, grep, sed, uniq, and sort. These tools are indispensable for log analysis, allowing you to filter, transform, and summarize data efficiently. Here are some tips for learning these tools:

Use man <command> (e.g., man awk) to access the manual and dive deeper into a specific command
Experiment with small examples to understand how commands like awk process patterns or how grep extracts data
For a broader look at how these commands enhance productivity, check out my blog post: Practical Linux Command Line Tips for Productivity and Efficiency

These commands are not just for log analysis—they’re powerful for any data manipulation task you encounter. With practice, they can become essential to your workflow.

1. Debugging Application Errors

1.1 Summarize Frequent Issues

sed -E 's/^\[.*\] //' error.log | sort | uniq -c | sort -nr

Purpose: Remove timestamps/client’s IP, then sort and count duplicates
Outcome: A quick snapshot of which errors appear most often

3 PHP Notice: Undefined variable $foo ...
2 PHP Warning: Division by zero ...
2 PHP Fatal error: Call to undefined function baz() ...

1.2 Categorize Errors by Type

grep -oE 'PHP [^:]+' error.log | sort | uniq -c | sort -nr

Purpose: Extract error types (e.g., Notice, Warning, Fatal error)
Outcome: Know whether notices, warnings, or fatal errors dominate

3 Notice
2 Warning
2 Fatal error

1.3 Find Problematic Scripts

awk -F'in ' '{print $2}' error.log | awk '{print $1}' | sort | uniq -c | sort -nr

Purpose: Identify which files generate the most errors
Outcome: Pinpoint error hotspots for focused debugging

3 /var/www/html/index.php
2 /var/www/html/script.php
2 /var/www/html/lib.php

2. Monitoring System Behavior

2.1 Track Problematic IPs

awk -F'\[client ' '{print $2}' error.log | awk -F':' '{print $1}' | sort | uniq -c | sort -nr

Purpose: Count how many errors each IP triggers
Outcome: Identify suspicious or high-traffic IPs

4 192.168.1.1
2 10.0.0.2
1 172.16.0.5

2.2 Spot High-Error Time Periods

awk -F'[][]' '{split($2, time, ":"); print $1, time[1]":"time[2]":00"}' error.log | sort | uniq -c | sort -nr

Purpose: Group errors by minute, revealing spikes or trends
Outcome: Correlate error surges with deployments or traffic peaks

3 2025-01-01 12:34:00
1 2025-01-01 14:36:00
1 2025-01-01 13:36:00
...

2.3 Analyze IP–Referer Pairs

grep "referer:" error.log | awk -F'\\[client ' '{print $2}' | awk -F':| referer: ' '{print $1, $3}' | sort | uniq -c | sort -nr

Purpose: Match IP addresses with the URLs they refer from
Outcome: Detect repeat offenders or malicious traffic patterns

3 192.168.1.1 Undefined variable $foo ...
2 10.0.0.2 Division by zero ...
1 172.16.0.5 Call to undefined function baz() ...

3. Enhancing Security

3.1 Watch Sensitive URLs

grep -E "admin|login" script_error_frequency.txt | sort -nr

Purpose: Flag frequent hits on admin or login pages
Outcome: Gauge whether sensitive endpoints are under attack

2 /var/www/html/login.php
2 /var/www/html/admin.php

3.2 Pinpoint Critical Times

grep -E "admin|login" error.log | awk -F'[][]' '{split($2, time, ":"); print $1, time[1]":"time[2]":00"}' | sort | uniq -c | sort -nr

Purpose: Identify specific time windows for sensitive access
Outcome: Cross-reference suspicious activity with your security logs

2 2025-01-01 12:34:00
1 2025-01-01 13:35:00
1 2025-01-01 12:35:00

Conclusion

These commands demonstrate how quickly you can glean insights from logs. With a little creativity, you can expand them to track response times, detect performance bottlenecks, and safeguard critical endpoints. Whether you’re tackling PHP errors or any other type of log data, the same principles apply: filter, sort, count, and investigate.

Curious to learn more? Combine these strategies with automation tools, integrate them into CI/CD pipelines, or hook them up to visual dashboards. Your logs will become a gold mine of actionable information.

Introduction to Playwright using C#

2023-10-12T00:00:00+00:00

Automating web application tasks is a necessary skill for software developers and testers. You need it for performing repetitive tasks, conducting end-to-end testing of web applications, and scraping data from websites. In this blog post, we’ll explore how to use C# with Playwright for automating tasks.

What is Playwright?

Playwright is an open-source automation framework developed by Microsoft that allows you to automate web applications using various programming languages, including C#. Playwright was created specifically to accommodate the needs of end-to-end testing, but we will use it as a library for web automation. Playwright supports Chromium, WebKit, and Firefox. It runs on a variety of systems: Windows, Linux, and macOS, locally or on a continuous integration (CI) platform, headless or headed with native mobile emulation.

To begin web automation using C# and Playwright, follow these steps:

Prerequisites: You should have .NET Core or .NET 5+ installed on your machine.
Create a New C# Project: Let’s start by creating a new C# console application.
```
dotnet new console -n EPWebAutomation
cd EPWebAutomation
```
Install Playwright NuGet Package: Open new project in Visual Studio or any preferred C# IDE and install the Microsoft.Playwright NuGet package and build the project using the following commands:
```
dotnet add package Microsoft.Playwright
dotnet build
```

Install required browsers: Run the following command to install the required browsers for Playwright:

pwsh bin/Debug/netX/playwright.ps1 install

# If the pwsh command does not work (throws TypeNotFound), make sure to use an up-to-date version of PowerShell.
dotnet tool update --global PowerShell

Initialize Playwright Instance: Initialize Playwright in your C# code by creating a new browser instance and take a screenshot in Chromium:

using Microsoft.Playwright;

using var playwright = await Playwright.CreateAsync();
await using var browser = await playwright.Chromium.LaunchAsync();
var page = await browser.NewPageAsync();

Navigate and Interact: You can now navigate to a web page using GotoAsync and interact with it using Playwright’s APIs. For instance, to navigate to a website and take a screenshot, you can use the below code:
```
await page.GotoAsync("https://playwright.dev/dotnet");
await page.ScreenshotAsync(new PageScreenshotOptions { Path = "screenshot.png" });
```

Locators: Playwright allows you to locate elements on the page using ID, CSS selectors, or XPaths. It also provides inbuilt locators.

await page.GotoAsync("https://computer-database.gatling.io/computers");
// fill dell in search text field using id selector
await page.Locator("#searchbox").FillAsync("dell");
// click on Filter by name button using id selector
await page.Locator("#searchsubmit").ClickAsync();
// click on a button using inbuild GetByText selector
await page.GetByText("Add a new computer").ClickAsync();

Fill form, select dropdown value and click elements: Playwright allows you to perform actions like filling out forms, selecting dropdown value and clicking elements. Here’s an example:

await page.FillAsync("#name", "asus abcde");
await page.SelectOptionAsync("#company", new[] { new SelectOptionValue() { Label = "ASUS" } });
await page.Locator("input[type=submit]").ClickAsync();

Cleanup: Don’t forget to close the browser when you’re done:
```
await browser.CloseAsync();
```

Advantages of Using C# and Playwright

Some of the useful features of Playwright with C# include:

Cross-Browser and cross-platform support: As mentioned above, Playwright works on modern rendering engines including Chromium, WebKit, and Firefox. You also have the option of running it locally or on CI, and of running headless or headed.
C# Language Familiarity: The Playwright API can be used with C#. If you’re already familiar with C#, this reduces the learning curve for web automation.
Robust Documentation: Playwright’s documentation and community support make it easier to troubleshoot issues and find solutions.
Headless and Headful Browsing: Playwright allows you to choose between headless and headed mode. Debugging issues in headed mode can be much easier, since we can see what’s going on in every line of code.
Powerful APIs: With Playwright’s APIs, you can perform complex web automation tasks with ease.
Powerful Tooling: Playwright has tooling like Codegen, Playwright inspector, and Trace Viewer. They can help make using Playwright easier.
Emulation: You can test your app on any browser or emulate a real device such as mobile phone or tablet. It can emulate behavior such as userAgent, geolocation, viewport, and many more. You can read more about it in the emulation documentation.

The powerful combination of C# and Playwright makes it a preferred tool for automating web tasks. When there are repetitive tasks or browser testing involved, Playwright can help you with getting the tasks done.

Quartz Scheduler as a Service

2022-04-18T00:00:00+00:00

Photo by Mat Brown from Pexels

Quartz Job Scheduler

“Quartz is a richly featured, open source job scheduling library that can be integrated within virtually any Java application — from the smallest stand-alone application to the largest e-commerce system.” (Quartz Scheduler overview)

Besides its advanced features, most basic and frequently used feature is job scheduling and job execution. Some frameworks like Spring Scheduler have their integration practice using Quartz Scheduler which allows using its default scheduling method.

In this post I am going to tell you a different approach to show how we can use Quartz Scheduler to schedule our jobs. We actually still will be using the existing scheduling mechanism of Quartz but we’re going to show how we can manage the scheduled and unscheduled jobs online. This way you can manage all the available jobs or create new ones on the fly.

Quartz Scheduler as a Service

Previously I led development of an enterprise “Business Service Management” software to replace IBM’s TBSM product at a major telco company in Turkey. This was a challenging project and found a solid place in the customer environment after a successful release.

Scheduled key performance indicators (KPI) retrieval and background reporting jobs were a significant part of this project. KPIs were either internal business service availability and health metrics or measured metrics calculated and stored in external data sources. Reports are also another type of schedulable jobs as many organizations need the data to be reported at certain intervals.

In an enterprise web application with such needs you would need to allow your customer to create their own customized scheduled jobs (KPIs, reports, etc.) in an easily manageable way. For this I came up with a simple solution by blending the existing Quartz Scheduler scheduling mechanism with some spice.

So here is the model we used:

A database table for creating/updating scheduler job definitions
Observer Schduler Job for observing the scheduler job table to watch for any updates in the scheduled jobs: new job, updated job, or disabled job, etc.
Business Job: You might define several schedulable business job types. KPI is one of those and I am going to give an example of it.

Simplicity should be a design goal, however the details can have their complexities.

This design doesn’t replace or provide an alternative to how Quartz Scheduler schedules its jobs. That is subject to job persistence and is out of this article’s scope. I am assuming we are scheduling the jobs all in Quartz Scheduler’s RAM-store or Job-store.

Read and Manage Job Data

Ideally you should store and manage the jobs as services in a database and you can then connect to this job storage either via DB connection or API. For security reasons even if you think that your application or services are internal and totally authenticated and authorised you should still perform DB operations via APIs. But for capability perspective yes you can use many ways to read and manage a data storage.

For this simple project I am not going to use a database but instead a JSON file as job service definitions repository. But you can simply convert this method to a database or API method.

I am going to use a JSON file named kpi.json in my project and define a simple set of attributes for each KPI item. Any service or scheduled job can have more or fewer attributes according to the requirements of the business use case.

Spring Application

You can use any framework or even without using any framework you can create your application from scratch and build a JAR. Here in this project I chose to go with Spring framework. You can also simply initialize a Spring application here.

Design

As I suggested a model above as a scheduling service solution, here is a high-level design of the model.

The overall solution would have a data storage for holding scheduled job service definitions and a UI for managing their attributes like enabling/disabling or changing scheduling dates etc.

In this solution we have two different Quartz Scheduler job types: observer job and business job. Observer job is a single job triggered frequently, say, every 5 seconds or every 1 minute, and checks the existing job definitions in the job storage. If it sees any update on the job definitions or new jobs it behaves accordingly. Business jobs are the job definitions found in job storage and designed to perform certain business actions. The business jobs can be notification jobs, KPI measuring jobs, and any other scheduled business jobs which should have their own scheduling interval.

In this example project I specifically used KPI term as the business case just to make it more relevant.

Scheduler

KPIJobWatcher class is responsible to schedule the observer job. In Spring application startup this is going to be our starting point to the scheduling service management.

Spring Application Startup

@SpringBootApplication
public class QSchedulerApplication {

	public static void main(String[] args) {
		SpringApplication.run(QSchedulerApplication.class, args);
	}

	Scheduler kpiScheduler;

	@EventListener(ApplicationReadyEvent.class)
	public void onAppStartUp() {
		try {
			// initializing KPI Trigger
			SchedulerFactory sf = new StdSchedulerFactory();
			kpiScheduler = sf.getScheduler();

			// watcher runs an observer job which monitors and manages KPI jobs
			KPIJobWatcher watcher = new KPIJobWatcher(kpiScheduler);
			watcher.run();
		} catch (Exception e) {
			e.printStackTrace();
		}
	}
}

KPIJobWatcher schedules the Observer job:

public class KPIJobWatcher {
    private static final Logger logger = LoggerFactory.getLogger(KPIJobWatcher.class);

    Scheduler kpiScheduler;
    public KPIJobWatcher(Scheduler s) {
        kpiScheduler = s;
    }

    /**
     * run KPIJobWatcher
     * @throws Exception
     */
    public void run() throws Exception {

        try {
            // Setting the KPI Job factory of observerScheduler
            KPIJobFactory jf = new KPIJobFactory((StdScheduler)kpiScheduler);
            kpiScheduler.setJobFactory(jf);

            // Scheduling KPI Observer Job
            JobDetail observerJob = newJob(KPIObserverJob.class)
                    .withIdentity("observerJob", "observergroup")
                    .build();

            SimpleTrigger trigger = newTrigger()
                    .withIdentity(observerJob.getKey() + "_trigger", "observergroup")
                    .withSchedule(org.quartz.SimpleScheduleBuilder.simpleSchedule()
                            .withIntervalInSeconds(10)
                            .repeatForever())
                    .build();

            Date ft = kpiScheduler.scheduleJob(observerJob, trigger);
            logger.info(observerJob.getKey() + " has been scheduled to run at: " + ft);

            // Starting KPI Observer Scheduler
            kpiScheduler.start();

        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

Before moving on to observer job here I want to notice that you can use a custom JobFactory and attach it to the current scheduler object so that you can use custom jobs with custom constructors created within this custom JobFactory as part of the factory design pattern.

JobFactory

public class KPIJobFactory implements JobFactory {
    Scheduler kpiScheduler;
    public KPIJobFactory(Scheduler s) {
        kpiScheduler = s;
    }

    public KPIObserverJob newJob(TriggerFiredBundle bundle, Scheduler Scheduler) throws SchedulerException {

        JobDetail jobDetail = bundle.getJobDetail();
        Class<KPIObserverJob> jobClass = (Class<KPIObserverJob>) jobDetail.getJobClass();
        try {
            // this is how we construct our custom job with custom factory
            return jobClass.getConstructor(Scheduler.getClass()).newInstance(kpiScheduler);
        } catch (Exception e) {
            e.printStackTrace();
        }
        return null;
    }
}

Observer Job

As suggested in the model above the observer job is triggered frequently and manages the overall scheduling job service in the background. I created the KPIObserverJob class as the observer job in this project and as you can see in the previous section KPIJobFactory creates instances of this observer job.

KPIObserverJob

Observer Job has some specific methods like ScheduleJob and UnscheduleJob to manage scheduling jobs.

public class KPIObserverJob implements Job {

    private static final Logger logger = LoggerFactory.getLogger(KPIObserverJob.class);

    List<JobDetail> jobList;
    Scheduler kpiScheduler;

    public KPIObserverJob(StdScheduler s) {
        kpiScheduler = s;
    }

    List<String> scheduledJobList;
    HashMap<String, JobDetail> alreadyScheduledJobList;

    String cronFormat = "SECOND MINUTE HOUR DAY_OF_MON MONTH DAY_OF_WEEK";

    @Override
    public void execute(JobExecutionContext context) throws JobExecutionException {
        scheduledJobList = new ArrayList<String>();
        alreadyScheduledJobList = new HashMap<String, JobDetail>();
        //JobKey jobKey = context.getJobDetail().getKey();

        jobList = new ArrayList<JobDetail>();

        // Get all KPIs and create their jobs
        CreateJobs();

        // Get the list of currently scheduled KPI jobs
        try {
            for (String groupName : kpiScheduler.getJobGroupNames()) {
                for (JobKey jk : kpiScheduler.getJobKeys(GroupMatcher.jobGroupEquals(groupName))) {
                    String jobName = jk.getName();
                    String jobGroup = jk.getGroup();

                    scheduledJobList.add(jobName);

                    JobDetail jd = kpiScheduler.getJobDetail(jk);
                    alreadyScheduledJobList.put(jobName, jd);

                    logger.info("already scheduled jobName {}", jobName);
                }
            }
        } catch (SchedulerException e) {
            e.printStackTrace();
        }

        // Schedule or unschedule KPI jobs if not done yet
        for (JobDetail job : jobList) {
            try {
                if (!scheduledJobList.contains(job.getKey().getName()))
                {
                    if (job.getJobDataMap().getInt("isRunning") == 1) {
                        logger.info("scheduling job: kpiJobName_{}", job.getJobDataMap().getString("kpiName"));
                        ScheduleJob(job);
                    }
                } else {
                    // Check any changes in the KPI job definition
                    JobDetail sJD = alreadyScheduledJobList.get("kpiJobName_" + job.getJobDataMap().getString("kpiName"));
                    if (!job.getJobDataMap().getString("cron").equals(sJD.getJobDataMap().getString("cron"))) {
                        logger.info("rescheduling job: kpiJobName {} , new cron: {}",
                                job.getJobDataMap().getString("kpiName"), job.getJobDataMap().getString("cron"));
                        UnscheduleJob(job.getJobDataMap().getString("kpiName"));
                        ScheduleJob(job);
                    }

                    if (job.getJobDataMap().getInt("isRunning") == 0) {
                        logger.info("Unscheduling: kpiJobName {}" + job.getJobDataMap().getString("kpiName"));
                        UnscheduleJob(job.getJobDataMap().getString("kpiName"));
                    }
                }
            } catch (SchedulerException e) {
                e.printStackTrace();
            }
        }

        // Finally unschedule deleted jobs if they are not listed anymore
        for (String kpiName : scheduledJobList) {
            boolean unschedule = true;
            if (!kpiName.equals("observerJob")) {
                JobDetail toBeRemovedJob = null;
                for (JobDetail jdetail : jobList) {
                    if (jdetail.getKey().getName().equals(kpiName)) {
                        unschedule = false;
                    }
                }

                if (unschedule) {
                    logger.info("Unscheduling: " + "kpiJobId" + kpiName.split("_")[1]);
                    UnscheduleJob(kpiName.split("_")[1]);
                }
            }
        }
    }

    private static final Type KPI_JSON_TYPE = new TypeToken<List<KPI_JSON>>() {}.getType();

    /**
     * Create Quartz Scheduler jobs from the job records read from a data source
     */
    private void CreateJobs() {

        Gson gson = new Gson();
        try {
            // kpi.json as a service data storage where we get KPI job data to be scheduled
            JsonReader reader = new JsonReader(new FileReader("kpi.json"));
            List<KPI_JSON> kpiList = gson.fromJson(reader, KPI_JSON_TYPE);

            for (KPI_JSON kpiItem : kpiList) {
                logger.info("Found KPI in kpi.json: {} , enabled: {}", kpiItem.getName(), kpiItem.getIsRunning());

                JobDetail job = newJob(KPIJSONJob.class)
                        .withIdentity("kpiJobName_" + kpiItem.getName(), "kpigroup")
                        .usingJobData("kpiName", kpiItem.getName())
                        .usingJobData("cron", kpiItem.getCron())
                        .usingJobData("lastRan", kpiItem.getLastRan())
                        .usingJobData("kpiDescription", kpiItem.getKpiDescription())
                        .usingJobData("lastMeasuredValue", kpiItem.getLastMeasuredValue())
                        .usingJobData("filename", kpiItem.getFilename())
                        .usingJobData("type", kpiItem.getType())
                        .usingJobData("isRunning", kpiItem.getIsRunning())
                        .build();

                jobList.add(job);
            }
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

    /**
     * Schedule a job
     * @param job
     * @throws SchedulerException
     */
    private void ScheduleJob(JobDetail job) throws SchedulerException {
        String cron = job.getJobDataMap().getString("cron");
        CronTrigger trigger = newTrigger()
                .withIdentity(job.getKey().getName() + "_trigger", "kpigroup")
                .withSchedule(cronSchedule(cron))
                .startNow()
                .build();
        Date ft = kpiScheduler.scheduleJob(job, trigger);
    }

    /**
     * Unschedule a job
     * @param kpiName
     */
    private void UnscheduleJob(String kpiName) {
        TriggerKey tk = new TriggerKey("kpiJobName_" + kpiName + "_trigger", "kpigroup");
        try {
            kpiScheduler.unscheduleJob(tk);
            kpiScheduler.deleteJob(new JobKey("kpiJobName_" + kpiName, "kpigroup"));
        } catch (SchedulerException e) {
            e.printStackTrace();
        }
    }
}

Business Jobs

Business jobs, as suggested in the model, can be any schedulable jobs. Managing/updating the business jobs frequently is a key point here. As the enterprise demands grow and change continuously, KPIs are generated at intervals (daily, weekly, monthly, etc.) and for frequent notification needs this kind of scheduling job management can be an important part of a solution.

Here I created KPIJSONJob as my business job:

public class KPIJSONJob implements Job {
    private static final Logger logger = LoggerFactory.getLogger(KPIJSONJob.class);
    private KPI_JSON kpi;

    @Override
    public void execute(JobExecutionContext context) throws JobExecutionException {
        JobDataMap dataMap = context.getJobDetail().getJobDataMap();

        kpi.setName(dataMap.getString("kpiName"));
        kpi.setKpiDescription(dataMap.getString("kpiDescription"));
        kpi.setIsRunning(dataMap.getInt("isRunning"));
        kpi.setFilename(dataMap.getString("filename"));
        kpi.setCron(dataMap.getString("cron"));
        kpi.setLastRan(dataMap.getString("lastRan"));
        kpi.setType(dataMap.getString("type"));
        kpi.setLastMeasuredValue(dataMap.getString("lastMeasuredValue"));

        this.processKPI();
    }

    public class KPIMeasured {
        public String name;
        public String value;
    }

    private static final Type KPIMEASURED_TYPE = new TypeToken<List<KPIMeasured>>() {}.getType();
    protected void processKPI() {
        // processKPI is supposed to get the KPI measured value from an external datasource and updates kpi.json
        // ...
    }

}

Running this Solution

Let’s give it a try and see it in action. Say we have the kpi.json as our job storage, with the following KPI jobs defined:

[
  {
    "name": "critical_ticket_count",
    "type": "JSON_FILE",
    "cron": "0 0 1 ? * * *",
    "isRunning": 0,
    "lastRan": "2022-04-01 01:00:00",
    "kpiDescription": "Open critical ticket count",
    "lastMeasuredValue": "7",
    "filename": "kpi_measured.json"
  },
  {
    "name": "failed_customer_api_call",
    "type": "JSON_FILE",
    "cron": "0 0 2 ? * * *",
    "isRunning": 0,
    "lastRan": "2022-04-01 02:00:00",
    "kpiDescription": "Last 24-Hour failed API call count",
    "lastMeasuredValue": "23",
    "filename": "kpi_measured.json"
  }
]

When we run the Spring application it starts logging like below:

2022-04-11 13:48:12.354  INFO 13033 --- [eduler_Worker-1] com.example.qscheduler.KPIObserverJob    : Found KPI in kpi.json: critical_ticket_count , enabled: 0
2022-04-11 13:48:12.355  INFO 13033 --- [eduler_Worker-1] com.example.qscheduler.KPIObserverJob    : Found KPI in kpi.json: failed_customer_api_call , enabled: 0
2022-04-11 13:48:12.355  INFO 13033 --- [eduler_Worker-1] com.example.qscheduler.KPIObserverJob    : already scheduled jobName observerJob

Initially I set the isRunning attribute of those KPI jobs to 0 and my scheduler service is not scheduling them. My KPIObserverJob triggers every 10 seconds because I set it to trigger that way in KPIJobWatcher.

Now let’s see if I update critical_ticket_count KPI’s isRunning value to 1:

2022-04-11 13:52:32.347  INFO 13033 --- [eduler_Worker-7] com.example.qscheduler.KPIObserverJob    : Found KPI in kpi.json: critical_ticket_count , enabled: 1
2022-04-11 13:52:32.348  INFO 13033 --- [eduler_Worker-7] com.example.qscheduler.KPIObserverJob    : Found KPI in kpi.json: failed_customer_api_call , enabled: 0
2022-04-11 13:52:32.348  INFO 13033 --- [eduler_Worker-7] com.example.qscheduler.KPIObserverJob    : already scheduled jobName observerJob
2022-04-11 13:52:32.349  INFO 13033 --- [eduler_Worker-7] com.example.qscheduler.KPIObserverJob    : scheduling job: kpiJobName_critical_ticket_count

As you can see from the logs ObserverJob noticed the enabled job and scheduled it.

Let’s change the cron scheduling rule of critical_ticket_count job to 0 0 3 ? * * * and see the logs again:

2022-04-11 13:55:12.354  INFO 13033 --- [eduler_Worker-3] com.example.qscheduler.KPIObserverJob    : Found KPI in kpi.json: critical_ticket_count , enabled: 1
2022-04-11 13:55:12.355  INFO 13033 --- [eduler_Worker-3] com.example.qscheduler.KPIObserverJob    : Found KPI in kpi.json: failed_customer_api_call , enabled: 0
2022-04-11 13:55:12.356  INFO 13033 --- [eduler_Worker-3] com.example.qscheduler.KPIObserverJob    : already scheduled jobName observerJob
2022-04-11 13:55:12.356  INFO 13033 --- [eduler_Worker-3] com.example.qscheduler.KPIObserverJob    : already scheduled jobName kpiJobName_critical_ticket_count
2022-04-11 13:55:12.356  INFO 13033 --- [eduler_Worker-3] com.example.qscheduler.KPIObserverJob    : rescheduling job: kpiJobName critical_ticket_count , new cron: 0 0 3 ? * * *

The Observer job now rescheduled the job since we changed its cron rule. These are all how we make observer job manage the KPI business jobs. If you have more attributes and if you want your observer job to reschedule or perform different operations on business job definition updates you should enrich your ObserverJob.

Extend by Creating a Management UI

Managing the scheduler jobs using a UI is not in the scope of this post. But that is not much different than managing any data on a web application. I encourage you to do your own implementations if this solution sounds useful to you.

Conclusion

This solution helps you create your own scheduling job management solution on the fly and lets you create, update, or delete the Quartz Scheduler jobs dynamically.

The complete implementation can be found in the GitHub project.

Automating reading the screen and interacting with GUI programs on X Window System

2022-03-10T00:00:00+00:00

A while back, Google Earth made some changes to the layer select menu in the sidebar, which broke a program that toggles the 3D imagery on VisionPort systems. These run the X Window System (also known as X11, or just X) on Ubuntu Linux.

In looking for a workaround, I found that shell scripts can fully interact with GUI apps using the xdotool, xwd, and convert commands. This script would send a series of keystrokes to open the sidebar, navigate the layers menu, and toggle the box for the 3D buildings layer.

Changing the series of keystrokes to match the new number of layers should have fixed the issue, but there was more to this script. The next part of the script would take a screenshot, crop the checkbox, and compare it to saved files of other cropped boxes. Fixing this part of the script required correcting the positions of the captures and replacing the reference files with ones that pictured the updated Google Earth checkbox states.

Here I will explain how the script works and how we changed it so that it no longer needs these reference files and ultimately runs faster.

Overview of how the script works

xwd takes a screenshot of a window on the screen. convert transforms the pixel data in the image into lines of text with location and color that we can easily search and read with grep, sed, or similar. xdotool interacts with GUI windows. It can find, focus, and send keystrokes and mouse commands, among other things.

Preparing

To illustrate, let’s make a simple case of looking for a button and clicking on it from the terminal. We will skip making a script for this example as most of this can be accomplished on a single command line with xdotool. The commands here are for the Ubuntu operating system and may be a little different on other systems.

If you would like to try it as you read along, you will need an image editor to quickly look up pixel positions and colors. GIMP works great and is pictured in this example.

You will also need to install some packages. On a terminal:

sudo apt install xdotool x11-apps imagemagick

Working through an example

1. To know which window to interact with, xdotool needs to know the window’s name. So let’s open a browser and navigate to endpointdev.com and then note the page title in the browser tab, “Secure Business Solutions”:

2. On the terminal:

xdotool search "secure business solutions"

Defaulting to search window name, class, and classname
69206019

This reminds us that it can search for windows in many ways and returns a window ID, in this example 69206019. Search results also get stored on the “window stack” and can be referenced as %1, %2, and so on.

3. Use this ID in the next terminal command. If you have more than one ID, you may need to refine your search:

xwd -id 69206019 -out endpointdev.xwd

xwd creates a screenshot of the window that we want to see and saves it to the filename passed with the -out argument.

4. Then run a convert command:

convert endpointdev.xwd endpointdev.txt

By converting the image to text we can use any text tools like grep, cut, diff, or sed to find colors and coordinates on the images. We just need to know how to read it. The first few lines of the endpointdev.txt file look like this:

# ImageMagick pixel enumeration: 1156,638,65535,srgb
0,0: (11565,11565,11565)  #2D2D2D  srgb(45,45,45)
1,0: (10537,10537,10537)  #292929  grey16
2,0: (4369,4369,4369)  #111111  srgb(17,17,17)
3,0: (0,0,0)  #000000  black
4,0: (0,0,0)  #000000  black
5,0: (0,0,0)  #000000  black

Each line represents a pixel:

The first two numbers are the x and y positions,
in parentheses are the decimal color values per RGB channel,
after # is the HTML RGB color value which we will be using in our example,
and last the srgb code or name for the color.

We can use any of these color codes for matching. To decode these color codes visually we use an image editor.

5. Open the endpointdev.xwd file that we created in an image editor. Here we will use GIMP:

6. Select the zoom tool (see the mouse pointer in the screenshot above to know which that is). Use it to draw a box around the VisionPort button or something else that we want our script to “see” and zoom in on it:

7. Next, use the color picker tool (check the mouse pointer in the screenshot above for this tool) and click on the circle, or any other spot you want to pick for your script:

8. Notice by the pointer above how the color picker box changed colors. Now clicking on this box brings up a new window:

Here we can find the HTML notation of the color we selected: 00ffcc (see the pointer in the screenshot above). We’ll call this “VisionPort button green”.

In my experience even things that look like the same color can have a unique pixel caused by a different shade on an edge or something else. Our old script checked a 25×25 pixel image to guarantee what it had was a check mark, but finding one of these colors unique enough to use as a key ensures that the check mark is next to it, so we can check the state on a single pixel too.

If one pixel is not unique enough we can look for a pattern of pixels and still store them as variables on the script to avoid saving and comparing files.

9. With this HTML hex RGB color code we go back to the terminal and search our file for it. Use capital letters for the color or else grep -i to make it case-insensitive:

grep 00FFCC endpointdev.txt

This will return a list of pixels with this color, such as:

827,107: (0,65535,52428)  #00FFCC  srgb(0,255,204)
839,107: (0,65535,52428)  #00FFCC  srgb(0,255,204)
853,107: (0,65535,52428)  #00FFCC  srgb(0,255,204)
878,107: (0,65535,52428)  #00FFCC  srgb(0,255,204)
883,107: (0,65535,52428)  #00FFCC  srgb(0,255,204)

10. For our purpose we can use any of those, so let’s try the first result. In the terminal run:

xdotool windowfocus 69206019 mousemove 827 107 click 1

Your browser should then follow the link to visionport.com and load that website’s home page in your browser:

Did it work for you too? 🎉

That is all we need to build a script that interacts with GUI apps. If you would like to practice, take another screenshot and use xdotool to look for and click the “contact” button. 😁

How it works in a little more detail

The last xdotool command chain’s steps are:

windowfocus 69206019 selects the ID that we searched for before,
mousemove moves the mouse pointer to the specified absolute coordinates, in this case the first pixel that matched the VisionPort button green color in the endpointdev.txt file,
click 1 sends a mouse left click at the mouse’s current position.

Updating our script to use this instead of the series of keystrokes took some investigation and testing but made it faster in the end.

One of the issues to overcome was that the VisionPort custom layers capture xdotool’s window and mouse commands, so to target a specific window on the VisionPort, we have to offset the coordinates of our commands according to the position of the window on the screen.

Chaining xdotool actions

Here is another example of chaining several xdotool actions in a single command:

xdotool search "secure business solutions" windowfocus key Alt+F4

That should close the browser window on most systems. If any of these commands failed, try replacing windowfocus with windowactivate; some xdotool commands depend on system support. Check xdotool’s manual for more options.

Command chain steps:

search <name>: Finds windows with that name, and stores the results in the “window stack” memory.
windowfocus: Selects a window. Defaults to the first window from xdotool’s window stack; no need to supply the window ID if following a search.
key Alt+F4: Sends the Alt+F4 keypress to the selected window. The + here means at the same time. Use spaces to separate a list of keys. By default there is a delay between keystrokes, and that is configurable.

Use the standard Unix sleep <seconds> in between other xdotool commands that may need delays.

xdotool takes a list of commands naturally, which can even be stored and read directly from text files. They can also be executed directly by making the file executable (with chmod +x) and setting the shebang line (first line of the file) to:

#!/usr/bin/xdotool

Such executable scripts are commonly named using a .xdo extension.

Fine-tuning

While we can convert the whole screenshot to text and search on it like we did in the example above, the files are large, so cropping a smaller section before converting to text is recommended.

We can crop a section, save it to a file and compare it to another file, like our script did. We can also crop an image to 1 pixel wide across a menu bar if that’s what we’re aiming for, or a column of icons in our case, to find all the button positions without much overhead. Doing this instead of only checking the area hard-coded for a check box gave our script the ability to find the check box based on the icon next to it when changing positions due to other tabs expanding on the menu.

It is also possible to use the multiprocessing features of shell scripts: When working on multiple window screenshots or crops, we can have them all run at the same time using the & operator at the end of the commands, then use the wait command to let them all finish before performing the text searches. These are very fast on small cropped files.

Learning more

The xdotool documentation has information on other ways to identify and interact with windows and is a great place to go to learn more.

Generate PDF with Chrome, Puppeteer, and Serverless Stack

2022-01-02T00:00:00+00:00

Function as a Service (FaaS) solutions are becoming more and more mainstream today. The FaaS model allows developers to not have to worry about managing infrastructure and instead focus on writing the application logic. In the FaaS model, developers write individual functions that run specific tasks that are deployed together on a FaaS platform, including but not limited to Amazon Web Services (AWS), Azure, and Google Cloud Platform (GCP).

These functions don’t always run like a traditional application server does, waiting for a request. Instead, these FaaS platforms only spin up instances of these functions whenever there is traffic and will shut them back down once there are no more requests after a given period of time. This helps make FaaS a really cheap platform while traffic is low. It is a good approach for on-demand tasks that are part of the application but not necessarily the most common path in a customer’s everyday journey within the application.

The tools

In this tutorial, we’re going to be looking at implementing PDF download with Chrome, Puppeteer, and Serverless Stack, but first, let’s have a brief introduction of the tools that we are going to be using.

1. Serverless Stack

Serverless Stack is a framework for building full-stack serverless apps. The bigger player in this space is the Serverless framework. Serverless Stack is giving the Serverless framework (yes, I know, it can be very confusing at times) a challenge to the throne. The latter has been around for many years now and has been the authority for building serverless apps. There are pros and cons to both of them.

Serverless Stack’s biggest advantage is the live Lambda debugging, a faster development process and overall better development experience. With the Serverless framework, developers have to either constantly deploy to the cloud to test any change, or spend some time setting up serverless-offline for simulated Lambda and API gateway environments on their local machines.

But with the Serverless framework, developers are not tied to using a specific platform, as it can be used to deploy functions on AWS, Azure, and Google Cloud Platform. Serverless Stack, on the other hand, only allows deploying to AWS since its internals are built on top of AWS CDK.

2. Chrome and Puppeteer

Chrome will be used in this tutorial as a headless browser, controlled using Puppeteer. Puppeteer is a Node.js package that provides an API to make Chrome or Chromium navigate to a specific URL, take screenshots, click buttons, type in input fields, as well as take PDF screenshots. Note that the PDF screenshot feature is specific to Chrome and not Chromium, so make sure to double check that you are using the right browser.

Implementing PDF screenshot and download with Serverless Stack, Chrome, and Puppeteer

Now, let’s get to the meat of this tutorial. We are going to implement PDF generation with Chrome and Puppeteer, deployed on AWS Lambda through Serverless Stack. Here is what we will do:

Use Puppeteer in Chrome to navigate to a receipt page that we implemented.
Instruct Chrome to take a PDF screenshot of the page.
Stream the PDF file back to the user.
Deploy this implementation on AWS Lambda using Serverless Stack and demonstrate the development experience throughout.

PDF generation is a great use case for FaaS. It represents something that doesn’t happen very often and is triggered on demand by the user. Instead of letting our monolith application handle that intermittent traffic, we can delegate the task to FaaS in the cloud which will do this job without worrying us about scaling the infrastructure to cater to this particular use case. Then our application can just focus on handling the regular web traffic for general use.

Prerequisite: Sample receipt pages

First, let’s create some demo pages for PDF screenshots. I’m not going to go in-depth on these since it’s not the topic of the day. I have deployed these simple HTML pages that are styled like receipts on this Surge instance here. You can find the code on GitHub here.

Getting started

Let’s generate a new Serverless Stack project.

$ npx create-serverless-stack@latest pdf-generator
$ cd pdf-generator
$ npm install

Congratulations, we have just created a new project with Serverless Stack! Here we get a basic template of a working Serverless Stack app. You can go here for more information on how to get started with Serverless Stack.

First, let’s look at the files we get from bootstrapping the project and customize them wherever we see fit.

1. sst.json

{
  "name": "pdf-generator",
  "region": "us-east-1",
  "main": "stacks/index.js"
}

This file is the entry point to the Serverless Stack app, and here we can define the region of choice, the name of the app, and the main file that Serverless Stack will use.

2. stacks/index.js

 // stacks/index.js
 app.setDefaultFunctionProps({
   runtime: "nodejs12.x"
 });

-new MyStack(app, "my-stack");
+new MyStack(app, "pdf-generator");

stacks/index.js is the main file declared in sst.json. When building the project, Serverless Stack will use this file as an entry point to our application. This file is pretty simple:

It configures the Lambda functions to use the Node.js 12.x runtime.
It registers a stack called my-stack. This is the name of the CloudFormation stack on AWS. We definitely want to give it a better name than my-stack here, so let’s rename that to pdf-generator.

3. stacks/MyStack.js

import * as sst from "@serverless-stack/resources";

export default class MyStack extends sst.Stack {
  constructor(scope, id, props) {
    super(scope, id, props);

    // Create a HTTP API
    const api = new sst.Api(this, "Api", {
      routes: {
        "GET /": "src/lambda.handler",
      },
    });

    // Show the endpoint in the output
    this.addOutputs({
      "ApiEndpoint": api.url,
    });
  }
}

MyStack.js is where we declare the resources we need inside a given CloudFormation stack. This can be anything from a cluster of Lambda functions, API Gateway endpoints, DynamoDB tables, S3 buckets, etc. The full list of resources that we can create with Serverless Stack is listed here in the documentation. For our project we just need one API endpoint with API gateway, so what we are provided with here is already sufficient.

Deploying the project

Now, we can try to deploy this project to AWS. Ensure that the AWS Access Key ID and AWS Secret Access Key are set in the development environment, and then run npx sst start.

# export AWS credentials
export AWS_ACCESS_KEY_ID=<access key id>
export AWS_SECRET_ACCESS_KEY=<secret access key>

# or if the credentials are set in ~/.aws/credentials
export AWS_PROFILE=<AWS profile name>

# Start the app in development mode
npx sst start

After a few minutes, we will have the Cloudformation stack deployed on our AWS account. Keep in mind that the first deployment will take a bit longer than subsequent deployments. When the deployment is done, we should be able to see that it is watching for file changes from our local machine.

threadlightly-pdf-generator-pdf-generator | CREATE_COMPLETE | AWS::ApiGatewayV2::Route | ApiRouteGET8AC7D3F8
threadlightly-pdf-generator-pdf-generator | CREATE_COMPLETE | AWS::Lambda::Permission | ApiRouteGETthreadlightlypdfgeneratorpdfgeneratorApiRouteGET7230D6CDPermissionE4542537
threadlightly-pdf-generator-pdf-generator | CREATE_COMPLETE | AWS::CloudFormation::Stack | threadlightly-pdf-generator-pdf-generator

 ✅  threadlightly-pdf-generator-pdf-generator


Stack threadlightly-pdf-generator-pdf-generator
  Status: deployed
  Outputs:
    ApiEndpoint: https://<yourapigatewayurl>.execute-api.us-east-1.amazonaws.com


==========================
 Starting Live Lambda Dev
==========================

Transpiling Lambda code...
Debug session started. Listening for requests...

Let’s head to the API URL given in the output and we should get a response back like this:

Hello, World! Your request was received at 30/Nov/2021:06:21:13 +0000.

Developing with Serverless Stack

Let’s try updating one of the files provided to see how we can test the changes we have made and will make.

 // stacks/MyStack.js
 const api = new sst.Api(this, "Api", {
   routes: {
-    "GET /": "src/lambda.handler",
+    "GET /downloads/receipt": "src/lambda.handler",
   },
 });

 // stacks/index.js
   runtime: "nodejs12.x"
 });

-new MyStack(app, "my-stack");
+new MyStack(app, "pdf-generator");

Here we did two things:

We updated the API endpoint to GET /downloads/receipt instead of just GET /.
We made the Lambda function return the URL given in the query string.

Once we save these changes we should see Serverless Stack automatically reloading our code and since we changed the API endpoint path, Serverless Stack will need to make changes to our infrastructure (i.e. the API Gateway resources), and whenever it detects that it has to change the infrastructure of our application, it will immediately prompt us to redeploy.

Rebuilding code...
Rebuilding infrastructure...
Done building code
Press ENTER to redeploy infrastructure

Once we hit ENTER, Serverless Stack will automatically update our infrastructure for us: delete our old endpoint, create a new endpoint on GET /downloads/receipt, and hook up our src/lambda.js file to handle that endpoint.

When it’s ready, try hitting the API gateway URL again, but this time append /downloads/receipt?url=https://google.com to it.

// https://<yourapigatewayurl>.execute-api.us-east-1.amazonaws.com/downloads/receipt?url=https://google.com
Hello! You've requested to print the receipt at page https://google.com

Great, now we have learned how to make changes to our Serverless Stack application during development. Let’s go ahead and add Puppeteer and Chrome to our Lambda function.

 // stacks/MyStack.js
 import * as sst from "@serverless-stack/resources";
+import { LayerVersion } from "@aws-cdk/aws-lambda";
+
+const layerArn = "arn:aws:lambda:us-east-1:764866452798:layer:chrome-aws-lambda:25";

 // Create a HTTP API
 const api = new sst.Api(this, "Api", {
   routes: {
-     "GET /downloads/receipt": "src/lambda.handler",
+     "GET /downloads/receipt": {
+       function: {
+         handler: "src/lambda.handler",
+         // Increase the timeout for generating screenshots
+         timeout: 15,
+         // Load Chrome in a Layer
+         layers: [LayerVersion.fromLayerVersionArn(this, "Layer", layerArn)],
+         // Exclude bundling it in the Lambda function
+         bundle: { externalModules: ["chrome-aws-lambda"] },
+       }
+     },
   },
 });

Here, we are configuring our AWS Lambda function to use a Lambda layer that includes Chrome in our Lambda functions. You can think of these Lambda layers as being similar to NPM packages that you pull for extending your projects except that this is a 3rd party package for extending AWS Lambda functions instead.

With this layer, your AWS Lambda function will boot with the Chrome binary already included. This layer is maintained here. Make sure you’re using the ARN from the right region. In this tutorial, we are using us-east-1 per Serverless Stack’s default, so we are going to pick the ARN for us-east-1.

Now, let’s update our Lambda handler to:

Use Puppeteer to instruct Chrome to navigate to the URL passed in the parameter.
Take a screenshot of the web page and save a PDF out of that. Puppeteer provides a handy page.pdf API to do just that.
Ultimately, stream that PDF file back to the user.

Here’s the code for that, with some comments to help you out:

+import chrome from "chrome-aws-lambda";
+
+const puppeteer = chrome.puppeteer;
+
 export async function handler(event) {
-  return {
-    statusCode: 200,
-    headers: { "Content-Type": "text/plain" },
-    body: `Hello! You've requested to print the receipt at page ${event.queryStringParameters.url}`,
-  };
+  let browser
+  let response
+  try {
+    const { url } = event.queryStringParameters;
+
+    browser = await puppeteer.launch({
+      args: chrome.args,
+      executablePath: await chrome.executablePath,
+    });
+
+    const page = await browser.newPage();
+    // Use A5 size at 150dpi
+    const width = 874
+    const height = 1240
+
+    await page.setViewport({ width, height });
+
+    // Navigate to the url
+    await page.goto(url, { waitUntil: 'networkidle2' });
+
+    // Take the screenshot
+    await page.pdf({path: 'receipt.pdf', width: width + "px", height: height + "px", printBackground: true});
+
+    response = {
+      statusCode: 200,
+      body: JSON.stringify({message: 'Screenshot taken'})
+    };
+  } catch(err) {
+    response = {
+      statusCode: 500,
+      body: JSON.stringify({message: `An error occured. ${err.message}`})
+    }
+  } finally {
+    await browser && browser.close()
+  }
+  return response
 }

Let’s also not forget to install Puppeteer and Chrome.

$ npm install puppeteer puppeteer-core chrome-aws-lambda

Now, give Serverless Stack some time to reload our new changes. Once we have the API endpoint ready to go, let’s head to the browser and hit our API endpoint, passing the URL to the sample receipt page that we did earlier as a query param.

// Visit https://<yourapigatewayurl>.execute-api.us-east-1.amazonaws.com/downloads/receipt?url=https://unwieldy-key.surge.sh/index.html
// Response:
{"message":"Screenshot taken"}

Great! We are getting back a 200 status code and a message saying “Screenshot taken”. That means our code works and there were no exceptions. We’re halfway there.

Now, if we check the project folder, we should now see a new file called receipt.pdf there. This is the receipt.pdf captured by Puppeteer. We can open this file and verify that this is the demo receipt page that I deployed earlier that we passed to the endpoint.

We can also hit the API endpoint with the second sample receipt URL (i.e. url=https://unwieldy-key.surge.sh/receipt-2.html) and we should see the receipt.pdf file gets replaced with the receipt from the new URL.

Note that in a real-world application, the URL you would want to screenshot might need some form of authentication, so you are going to have to figure that out for your application. In this case the sample receipt page is accessible publicly, so we don’t run into this problem, but chances are your users’ receipt page will not be accessible publicly like this.

One way you could solve this is to create a special user account that has permission to visit these authenticated pages for this purpose. Then, prior to visiting the receipt page, you’d program Puppeteer to first log in as the special user and then head to the receipt page URL to be screenshot.

However, we are only halfway there. We can see the receipt.pdf file now only because the receipt.pdf file was made available to us by Serverless Stack, which automatically downloads that file to our local machine in development mode. In a real AWS Lambda execution, the file will remain in the filesystem of that Lambda execution context instead, and the user will not see it.

Hence we have to read and stream that file back to the user instead of returning a JSON message saying “Screenshot taken”. Getting a 200 status code is a great start, but that still does not give our users the actual PDF that they wanted, so let’s update our Lambda handler to do just that:

 // src/lambda.js
+import * as fs from "fs";

 // Take the screenshot
 await page.pdf({path: 'receipt.pdf', width: width + "px", height: height + "px", printBackground: true});
+const buffer = fs.readFileSync('receipt.pdf')

 response = {
   statusCode: 200,
-  body: JSON.stringify({message: 'Screenshot taken'})
+  headers: {'Content-type' : 'application/pdf'},
+  body: buffer.toString('base64'),
+  isBase64Encoded : true,
 };

The changes this time are pretty simple:

Import the fs module from Node.js, which will give us access to readFileSync API that can read files on the disk.
Read the generated receipt.pdf file as a Buffer.
Return the file’s Buffer as a base64 string in the response body.
Set the content-type to be application/pdf. Setting the content-type appropriately will tell the browser what to do with the file. Some browsers will display the PDF in an in-browser PDF reader, while others will download the file.

That’s it! Now let’s test the change. Hit the same URL again.

Bingo, the endpoint now returns the PDF to the user! Now we can integrate this endpoint with our main application to delegate these on-demand PDF generation to managed FaaS infrastructure.

Conclusion

FaaS platforms are a really great way to offload intermittent and on-demand traffic from our main application. This ensures our application runs smoother and can serve our general customers better, while at the same time handling these on-demand tasks really well.

Caveats

Building a full-blown application on the stack is also possible, but in my personal experience, it comes with its own complexities and tradeoffs that may or may not be worth the engineering time.

For example, databases are an issue with going full-blown development on FaaS. Because FaaS can scale indefinitely, the number of connections to the database is a limitation.

You can switch to FaaS-friendly databases such as DynamoDB or FaunaDB, but these are NoSQL databases and require a different mindset in structuring the application data than the simpler relational databases and that can cost tremendous engineering time. Relational databases are easier to work with for most types of applications.

Nonetheless, more and more tools are being developed to tackle these complexities, such as PlanetScale and Serverless Aurora v2 (still in preview). All in all, it is definitely a trend worth following closely.

Appium: Automated Mobile Testing

2021-06-30T00:00:00+00:00

First impressions are everything. You can have the best, most robust application in the world, but if it looks like it’s from 2004 most users won’t give it a second look. Automated testing can help ensure that the app the user sees is consistent and fully functional no matter the iteration.

Selenium, Cypress, and other automated testing suites have become more and more popular for webapps. This trend has not carried over to mobile native app testing. This may be a bit surprising, as a fully functional frontend can be the difference between a professional-feeling app and a hacky one.

There are many frameworks that can be used to test mobile applications (Appium, UI Automator, Robotium, XCUITest, SeeTest, and TestComplete to name a few), but today we’ll be focusing on Appium.

Appium is an open source framework that’s easy to use out of the box. It can be used to test many versions of many different mobile OSes. It’s a one-stop shop to ensure your users are on the same page, no matter the platform.

Installation and setup for Linux

General

We will be testing the To-Do List app from the Google Play Store. What we’re testing and for which platform isn’t important, since we’re here to learn about the Appium methodology that can be applied to any mobile app.

AVD Android emulator

We will be using an AVD (Android Virtual Device) to test our app. There are many out there but I recommend Android Studio’s.

Once it’s running, just click Configure > AVD Manager.

Either create a new AVD or use the default one and start it up.

Appium

Download Appium from GitHub.

Appium can be run through the command line but we’re going to use the desktop app here. This gives you the ability to record a session and capture elements for automated tests.

Once it’s running, all we have to do is click Start Server.

Ruby and RSpec

We will be using Ruby and RSpec to run our tests. At the top level of our project we need to add these lines to our Gemfile:

/Gemfile:

source 'https://www.rubygems.org'

gem 'appium_lib', '~> 11.2'
gem 'appium_lib_core', '~> 4.2'

Make sure Ruby is installed, then run:

$ gem install rspec
$ bundle install

We also need to configure RSpec to find the correct AVD and APK to test.

/spec/spec_settings.rb:

ANDROID_PACKAGE = 'io.appium.android.apis'
def android_caps {
  caps: {
    # These settings match the settings for your AVD. They can be found in the AVD Manager.
    platformName: 'Android',
    platformVersion: '11',
    deviceName: 'Pixel_3a_API_30_x86',
    #  This points to the apk you are testing
    app: './To-do_list.apk',
    automationName: 'UIAutomator2',
  },
  appium_lib: {
    wait: 60
  }
}
end

Finally we need to create our test. Since we don’t know the elements yet we can start with a blank template.

/spec/TaskCRUD:

require 'spec_settings'
require 'appium_lib'

describe 'Tests basic CRUD functions of Task application' do

  before(:all) do
    @driver = Appium::Driver.new(android_caps, true).start_driver
  end

  after(:all) do
    @driver&.quit
  end

  it '...' do

  end
end

Putting it all together

Let’s boot it up and make sure everything is running as expected.

Run an AVD from Android Studio
Start the Appium Server
Run rspec in the project root

Even though there is no test written it should still run and pass the template test. Running it the first time will save the APK into the AVD virtual memory. We can now access the app on the AVD emulator and record tests.

Recording a test with Inspector

Open the Appium window that has the server running and click Start Inspector Session:

Under Attach to Session... you will see a dropdown containing the RSpec test we just tried to run. If it’s not here try running rspec again. Select this and click the Attach to Session button.

We are now connected to the Android Emulator via the Inspector. We can click on elements to see their properties and interact with them.

If we click the top record button it will save our actions and allow us to import them into a test.

Running a test

Let’s take what we got from the Inspector and populate our template test.

/spec/TaskCRUD:

require 'spec_settings'
require 'appium_lib'

describe 'Tests basic CRUD functions of Task application' do

  before(:all) do
    @driver = Appium::Driver.new(android_caps, true).start_driver
  end

  after(:all) do
    @driver&.quit
  end

  it 'creates a task and verifies its name' do
   el1 = @driver.find_elements(:id, "todolist.scheduleplanner.dailyplanner.todo.reminders:id/zk")
    el1.click
    el2 = @driver.find_elements(:id, "todolist.scheduleplanner.dailyplanner.todo.reminders:id/xd")
    el2.click
    el3 = @driver.find_elements(:id, "todolist.scheduleplanner.dailyplanner.todo.reminders:id/wt")
    el3.send_keys "task_1"
    el4 = @driver.find_elements(:id, "todolist.scheduleplanner.dailyplanner.todo.reminders:id/wr")
    el4.click
    el5 = @driver.find_elements(:xpath, "/hierarchy/android.widget.FrameLayout/android.widget.LinearLayout/androidx.recyclerview.widget.RecyclerView/android.widget.LinearLayout[3]/android.widget.TextView")
    el5.click
    el6 = @driver.find_elements(:id, "todolist.scheduleplanner.dailyplanner.todo.reminders:id/wo")
    el6.click
    el7 = @driver.find_elements(:xpath, "/hierarchy/android.widget.FrameLayout/android.widget.LinearLayout/android.widget.FrameLayout/android.widget.LinearLayout/android.widget.FrameLayout/android.widget.RelativeLayout/androidx.recyclerview.widget.RecyclerView")
    el7.click
    el8 = @driver.find_elements(:id, "todolist.scheduleplanner.dailyplanner.todo.reminders:id/xl")
    expect(el8.text).to eql 'Test-text-1'
  end
end

If we run it with the rspec command we can watch the test in progress on the AVD and then we receive a success message!

What’s next?

Additional tests

Tests should be written to cover as many aspects of the app as possible. The main focus should be on parts of the app that don’t change very often. Tests written for something in progress or that is due for a facelift will need to be rewritten when the time comes.

Test with other devices

An app can look perfect on a Galaxy S20 but look awful on a Pixel 5. That’s why it’s important to try many different AVD versions for your app. Android allows you to create many phone types that run many different firmware versions.

CI/CD integration

Manual testing is great but nothing beats automation. These tests should ideally be integrated into a continuous integration and continuous delivery (CI/CD) platform. This way each build will be automatically vetted for bugs.

For more info, check out this guide for integrations with CircleCI.

Catching CSS Regressions and Visual Bugs in Continuous Integration

2021-06-24T00:00:00+00:00

Photo by Alexandra Nicolae

Many web projects nowadays are equipped to simulate real users and automatically test a real use case, but these end-to-end tests still have a weakness. Browser automation tools run tests by performing the interactions and assertions in the context of the DOM of the pages. That is very different from how humans use web applications, which is by looking at the user interface. Due to this, the tests’ definition of a “functional web app” differs from that of a real human.

For example, let’s take a simple music player with two buttons:

One in green with a play icon, and
One in red with a stop icon.

The HTML would look something like this:

<!-- music player -->
<button class="button is-red"><i class="icon icon-stop"/></button>
<button class="button is-green"><i class="icon icon-play"/></button>

In our end-to-end tests, one would typically instruct the test to check:

If the expected classnames exist.
If clicking the stop button stops the music.
If clicking the play button plays the music.

If all of the above conditions were met, then the app is considered “functional”. However, the tests have no way to verify if the right colors are actually reflected on the buttons, or if the SVG icons on the button are properly loaded and shown to the users. Unfortunately, for a real user, both of these are very crucial in telling the user how the app functions. If the stylings for the buttons are not implemented or the SVG icons for the buttons fail to load, users will not have a clear indication on how the app works, and the app will not be considered “functional” by real users, even though the functionalities of both buttons are wired correctly in the code.

Web developers also frequently deal with styling regressions. This can happen due to the “cascading” characteristic of CSS: Badly scoped CSS values may affect other elements in unrelated areas unintentionally. Many times, styling regressions slip past even the end-to-end tests with browser automation because, again, end-to-end tests only verify that the app’s functionalities are wired together, but do not check if things appear the right way visually.

Visual regression testing to the rescue

Visual regression testing adds another quality gate in the workflow to allow developers to verify that, after a code change, the user sees what is expected. For example, if the code change is supposed to modify the appearance of an element, developers can verify that it’s doing just that, or at the very least, the code change should not adversely affect other areas. Visual regression testing involves taking screenshots of tested scenarios with changes and comparing them against the baseline (usually screenshots from the stable branch).

This type of test complements the other testing categories in the test pyramid and ensures user experience is not adversely affected from any given code changes. For more information on the test pyramid and different categories of testing, visit this blog post by my colleague, Kevin.

There are many tools that help with visual regression testing. Here are some tools that you could look into:

Percy.io
- Generous free tier to get you started.
- 5000 free screenshots per month with unlimited team members. This should get you going just fine for smaller projects.
- Integrate with many mainstream end-to-end tests frameworks.
Applitools
- Uses AI-powered computer vision for visual diffing. Said to reduce a lot of false positives and be more efficient.
Testim.io Automation
- Another service that uses AI-powered computer vision.
BackstopJS
- A Node.js library you can set up yourself.

Each tool has its own strengths and weaknesses. For today’s demonstration, we are going to look at implementing visual regression testing with Percy and see how it works.

Integrating Percy

Percy provides comprehensive guides to get you started with many popular end-to-end test frameworks such as Selenium, Cypress, TestCafe, and Capybara. You can check the documentation to see how to integrate it with your project. Since we work with Rails a lot at End Point, we are going to demonstrate Percy within a Rails project alongside Rspec/Capybara-driven end-to-end tests.

In this article, we are going to use this simple Rails project I built for fun as the demo. It is just a simple wiki project using Vue and Rails. Let’s start.

1. Register an account at percy.io

Head to percy.io and click the “Start for Free” button on the top-right side.
Choose the preferred sign up method. In this case, we are just going to use email and password, so we will choose Sign Up with Browserstack.
Fill in the email and password and complete the registration. It will then send an email to ask us to verify our email address. Let’s check our email and complete the email verification.
Next, we will see the Get Started screen. We will choose Create a New Project and give it a relevant name.
We have successfully created a new percy.io project and should be on the project page. There are some links to setup guides that can help us integrate Percy into our project.
Let’s navigate to Project Settings. We will scroll down to the Branch Settings section and make sure the Default base branch points to the primary branch of our project’s repository. This is usually main, master or develop depending on your team’s workflow. You might want to make the same branch auto-approved as well. That way, screenshots from this branch will be treated as the baseline and will be used when comparing against the development branches. In our case, we are using the main branch so we will specify that in both fields.

Great, we have now successfully set up Percy. There are other configurations in the Project Settings page (e.g. browsers to enable, diff sensitivity, and whether or not to allow public viewers), but we do not have to care about them for the purpose of this demo. You can always come back and tweak this later according to your project’s requirements.

2. Integrate with the Rails project

Now, let’s look at how to integrate Percy with our demo Rails project. Percy provides a comprehensive guide to do that here.

1. First, let’s export the PERCY_TOKEN provided to us in the Builds page into our environment variable.

export PERCY_TOKEN=<value>

2. Then, we will add percy-capybara into our Gemfile and run bundle install.

gem 'percy-capybara'

3. Next, let’s install @percy/agent via npm/Yarn.

yarn add --dev @percy/agent

That’s it! Now let’s look at adding Percy to our feature specs.

3. Percy in Feature Specs

Percy has to be integrated into feature specs (i.e. end-to-end tests) because it requires the app to be running in order for it to be able to take screenshots. Hence, it will not work in unit tests where the application context is mocked and chunks of code are tested in isolation.

First, we will check out the primary working branch main and take a look at the existing feature specs:

# spec/integration/document_spec.rb
require "rails_helper"

RSpec.feature "Add documents", js: true do
  before :each do
    User.create(email: 'user@example.com', password: 'password')

    visit "/documents/new"

    within("#new_user") do
      fill_in 'Email', with: 'user@example.com'
      fill_in 'Password', with: 'password'
    end

    click_button 'Log in'
  end

  scenario "User visits page to create new document" do
    expect(page).to have_text('Create new document')
  end

  scenario "User sees validation errors" do
    click_button 'Create'

    expect(page).to have_text("Title can't be blank")
    expect(page).to have_text("Body can't be blank")
    expect(page).to have_text("Body is too short (minimum is 10 characters)")
  end
end

The feature specs are pretty simple. /documents/new is a route that only registered users can access, so upon visiting the path, devise is going to ask the visitor to authenticate before they can access the page.

The first test case verifies that the user can see the Create new document page upon successfully signing in.
The second case verifies that the user will see validation errors if they do not fill in the required details for the new document.

Adding Percy into the mix

To add Percy, simply require "percy" at the top of the spec file and add Percy.snapshot(page, { name: '<screenshot description>' }) on the lines in which we want Percy to capture.

# spec/integration/document_spec.rb
require "rails_helper"
# Include Percy in our feature specs
require "percy"

RSpec.feature "Add documents", js: true do
  before :each do
    User.create(email: 'user@example.com', password: 'password')

    visit "/documents/new"

    within("#new_user") do
      fill_in 'Email', with: 'user@example.com'
      fill_in 'Password', with: 'password'
    end

    click_button 'Log in'
  end

  scenario "User visits page to create new document" do
    expect(page).to have_text('Create new document')
    # Screenshot when we're redirected to the create new document page
    Percy.snapshot(page, { name: 'Create document page' })
  end

  scenario "User sees validation errors" do
    click_button 'Create'

    expect(page).to have_text("Title can't be blank")
    expect(page).to have_text("Body can't be blank")
    expect(page).to have_text("Body is too short (minimum is 10 characters)")
    # Screenshot when we see the errors
    Percy.snapshot(page, { name: 'Create document page validation error' })
  end
end

That’s it! Easy enough, right? Next, let’s run the tests so that the screenshots are taken and sent to percy.io. To do this, we cannot run the usual bundle exec rspec. Instead, we will have to run Percy and pass rspec to it.

$ yarn percy exec -- rspec

Note: This is the command that you should use if you have continuous integration set up.

And we’re done here! Now, if we go to percy.io, we can see that a new build has been created for us. After a few minutes, we should see the screenshots of the app.

4. Introduce visual changes and see how percy.io compares

The last step we ran was on the primary working branch. Since we set this branch as the baseline and that the screenshots from this branch would be auto-approved, we are not going to be asked to verify anything on this build.

Therefore, to demonstrate the visual diff-ing feature of Percy, let’s create another branch highlight-create-button, and let’s make the Create Document button appear bigger as well as change the text to just say Create.

<%# app/views/documents/_form.html.erb %>

<%# ... %>

<%# Old button %>
<%#= form.submit class: 'button is-primary' %>

<%# New button %>
<%= form.submit "Create", class: 'button is-primary is-large' %>

Awesome! Now, let’s rerun yarn percy exec -- rspec and analyse the visual diff on percy.io.

Once percy.io is done processing the new screenshots, it will compare the screenshots against the baseline in the main branch. We will see this in our new build:

As you can see, Percy highlights any changed areas in red. This makes it easier for us to spot the differences. In this case, it is expected for the button to change, so we can just go ahead and approve the build with the green button at the top.

5. What about style regressions?

Let’s do one more demonstration. This time, let’s simulate a CSS change gone wrong:

Create another git branch called css-regression.

Add a text-indent: -5rem to the error messages to push the text out of its container. This is what it would look like:

Let’s run bundle exec rspec. We will get this output:

> bundle exec rspec
Capybara starting Puma...
* Version 5.3.2 , codename: Sweetnighter
* Min threads: 0, max threads: 4
* Listening on http://127.0.0.1:65318
..

Finished in 2.62 seconds (files took 1.3 seconds to load)
2 examples, 0 failures

As you can see, the tests were not able to catch this styling regression. This is because the error messages specified in the assertion still exist on the page, so the use case is not considered broken in the “eyes” of the regular feature specs.

Now, run yarn percy exec -- rspec to send the screenshots to percy.io to be processed.

Great, percy.io was able to catch the regression here in the visual diff! A reviewer can just mark the build as “Requested changes” and leave a comment in there to have it fixed.

6. Responsive design and cross-browser testing

Percy can also help test your app’s UI on different browsers — namely Edge, Firefox, and Chrome. By default, all three browsers are enabled. You do not need to do anything else.

It can be used to test against different screen sizes as well. You can just select the screen sizes that are relevant to you and instruct Percy to take screenshots at those screen sizes automatically. This is particularly helpful as testing on several screen size variants can be very time-consuming. With Percy, you do not have to pay attention to the screenshot variants that do not diverge from the baseline.

In order to specify screen sizes, just pass a list of screen sizes that you are interested in, like so:

Percy.snapshot(page, { name: 'Create document page', widths: [480, 768, 1024, 1366] })

Rerun yarn percy exec -- rspec. We can now see the different variants (browsers and screen sizes) of each scenario on the top-right corner of the screen.

That’s it! We have successfully demonstrated how Percy can help us verify expected UI changes and catch visual regressions.

Caveats

As helpful and as crucial as it is, visual regression testing is not without its caveats. One of the most common issues with visual regression testing is, in some cases, it can produce a huge amount of false positives. Major-but-intentional layout shifts and changes to shared elements may result in having to review a huge amount of screenshots in the project.

For example, let’s say you increase the height of the site header (which is a common element in all pages) by 30px. When Percy processes the screenshots, it will invalidate most of the screenshots in the project, because elements would have been shifted down by 30px as a result of the taller header. For large projects, reviewing each screenshot one-by-one can be a hassle, especially with the tens of permutations for each screenshot to cater different screen sizes and browsers. As a result, one would be tempted to just click “Approve build”, thinking all changed areas would just be elements being pushed down, but that may not necessarily be true for elements with position: absolute;; they might get hidden behind the now taller header, and this may cause visual regressions to still slip into production, which kind of defeats visual testing’s original purpose.

Other than that, dealing with external resources can also result in flaky builds (e.g. if a page has an iframe or links to an external image). There are a lot of ways in which an external resource can change, e.g. image quality, content, updated resource, etc., and because these resources are outside of our control, they may become a source of false positives and a nuisance to the reviewers.

One way to deal with this is to not screenshot whole pages. Instead, screenshot only the individual components that are relevant to each scenario. This way, layout shifts in other parts of the pages will not affect the build in major ways, and it also helps cut out the changes from external resources on the page. Unfortunately, Percy does not have the capability to do this yet.

Conclusion

So, there you go! I hope this article has given you a good introduction to visual regression testing, why should you have it, and how to get started with it. Hope it will benefit you and your team in pushing great web applications!

Other resources

Lazy Isn’t Bad: Write Lazy Scripts

2021-06-09T00:00:00+00:00

Photo by cottonbro

A quote attributed to Bill Gates says, “I choose a lazy person to do a hard job. Because a lazy person will find an easy way to do it.” Larry Wall’s list of the virtues of a programmer begins with “laziness,” which of course is a particular kind of laziness: the desire to automate things that a human has previously done manually and painstakingly.

It’s a philosophy that is generally good to adopt. Are you doing repetitive tasks that you hate every day? Weekly? At the end of every month? That’s probably a clue that you could take some pain out of your workday. If there’s some task that comes up repeatedly that requires you to fiddle with data which can be automated, even if it’s relatively easy to do manually, why not do it?

There are many ways in which people try to improve productivity. One of the more famous, Kanban, involves removing work in progress and optimizing flow. While implementing full Kanban is probably more than you want to do, we can do it the lazy way.

The lazy way means you put time and effort into your infrastructure, creating code or scripts that will do the job for you. Many people don’t think they have the time to put some hours into making this work. It is indeed a time sink, but you only write the script once, compared to many many times of doing the job. The time saved in the long run can be immense. Additionally, the job can be done more accurately; less human interaction means less human error.

Look at a few factors: What task is the most boring and repetitive, what task takes the longest, and what task is most easily automatable? Pick one and write a script for it. Don’t make it too fancy; just make something to get the work done. If you find yourself with extra time left over after the next time you do the task, repeat the process and see if you can shave off more time.

While writing this post, I have a particular client and situation in mind, but we’ve seen it play out similarly many times over the years: we have a longstanding client who needs information gathered from a database. The manual process requires a lot of eye-scanning, copying, and pasting individual data items, fiddling with data formats, reformatting ID numbers from one set to another, and so on. While our example scenario is complex and has several different steps where it can fork in different directions, each step is easy to automate on its own.

Part of this task involves copying numbers from a spreadsheet, changing the number format (stripping out excess characters), deduplicating the numbers, applying a database call to those numbers, and writing the success or failure back to the document.

The most time-consuming part was searching through large log files, trying to identify the right string from just the numbers — this was also the easiest part to automate.

While writing a script that could access the spreadsheet and the database might take an hour, writing a script that does the data processing and spits out a database query took perhaps ten minutes, and has easily saved me that long after only a few uses. I also don’t have to worry about getting the call wrong; it’s preformatted for me.

Another part of the task was downloading several files with different extensions from different directories. This didn’t take too long to set up manually, perhaps 30 seconds. But with a script to grab the files for me, it takes five, and only one command instead of downloading five different files by hand.

This customer has had this pop up several times per month, sometimes per week, and would have saved thousands of dollars in fees in the long run — not to mention delays and fuss — if they’d had us write an application so they could self-serve this. Sometimes, you only realise this in hindsight, but that’s no reason to put off biting the bullet now. There are always more iterations of data handling coming in the future.

Of course these smaller scriptlets didn’t automate the whole process completely. But each step was easily done, and the time taken for the whole task is vastly reduced. If something feels too big to automate in one go, but you can see a bit that could be scripted alone, only script that part. It’ll make you feel good about that part every time you fire the script off. And maybe you can find a way to automate another part later. And so on.

Could I have made the task a simple push of the button, or completely hands off, by fully automating it? Probably. But it would take many hours to integrate all the components, and might only save time in the very long term, while the process might change in the mean time. Writing a half-measure helper script to partially automate the process isn’t fully solving the problem, but it’s lazy — in the good way!

Automating Windows Service Installation

2021-04-23T00:00:00+00:00

Photo by Science in HD on Unsplash

For me, setting up a service started as a clean one-liner that used InstallUtil.exe, but as time went on, I accumulated additional steps. Adding external files & folders, setting a custom Service Logon Account, and even an SSL cert had to be configured first before the service could be used. An entire checklist was needed just to make sure the service would start successfully. That’s when I realized a proper installation method was needed here. This article will go over how to make a dedicated .msi installer for a Windows Service that can do all these things and more.

Creating an installer can be tricky, because not all the available features are easy to find. In fact, the setup project itself is not included by default in Visual Studio; you need to install an extension in order to create one. But once the installer is created, we can use it to do things like:

Configure the installer to copy the build output of a project to the C:\Program Files (x86) folder, as well as add custom files & folders to the installation
Add custom CLI flags to the installer to specify the Service Logon Account at install time
Add an installer class to the service and use the installation lifecycle hooks to write custom code that gets run at any stage of the installation.

A Note On Compatibility

For .NET Core and .NET 5.0 projects, you won’t be able to add an installer class. To use either .NET Core or .NET 5.0 to make a service instead, you’d need to make a different kind of project called a Worker Service. A Worker Service differs from a traditional Windows Service in that it’s more like a console application that spawns off a worker process on a new thread. It can be configured to run as a Windows service, but doesn’t have to be. So instead of using an installer, for a Worker Service you’d publish the project to an output directory and then use the SC.exe utility to add it as a Windows service:

dotnet publish -o C:\<PUBLISH_PATH>
SC CREATE <WORKER_NAME> C:\<PUBLISH_PATH>

Creating a Windows Setup Project

In order to create a .msi installer in Visual Studio 2019, you’ll need to install the Microsoft Visual Studio Installer Projects extension. While it’s not provided with a default installation of Visual Studio 2019, it’s an official Microsoft extension. Once you’ve installed it, you’ll be able to create any of the following projects:

To create an installer, you can create a new Setup Project. The build output from this project will be your .msi installer. The setup project has a few different views, which you can use to configure what the installer needs to accomplish. These views can be accessed by right-clicking on the project in the Solution Explorer and expanding View from the context menu:

Configuring the Installation File System

To configure what files need to be installed, you can use the File System view, which provides a UI with some folders added to it:

Here, clicking on any folder on the left shows its contents over on the right. It also populates the Properties Window with the information about the folder:

In the above example, we can see that the Application Folder is being output to a folder inside C:\Program Files (x86). You can add any folders you want to the file system by right-clicking on the file system to open the Special Folders context menu:

Some default folders are shown here for convenience. But let’s say we wanted to make some files get added to the C:\ProgramData folder. To do this, select “Custom Folder” and give it a name. Then, in the Properties Window, set the value of DefaultLocation to the correct path:

From here, you can use the right half of the view to add additional folders within C:\ProgramData\DotNetDemoService based on your needs.

Another thing you’ll likely want to do is put the DLLs from your application into a folder within C:\Program Files (x86). You can easily do this by mapping the primary build output of your project to the Application Folder in the installer’s file system. To do this, right-click on the Application Folder, and add project output:

From there you’ll be prompted to select the project and output type. Select your project, and “Primary Output”:

This will copy over the DLLs for your project and all of its dependencies.

Creating an Installer class

You may be wondering if it’s possible to define custom code to be run during the installation process. It is! For any project targeting .NET Framework 4.8 and under, you can add a class that extends System.Configuration.Install.Installer, and has the [RunInstaller(true)] attribute applied to it. After doing so, you’ll then be able to hook in and override any of the installation lifecycle methods. Taking a look into the definition of the System.Configuration.Install.Installer class reveals the list of overridable lifecycle hook methods you can use to add custom logic to the installation:

public virtual void Commit(IDictionary savedState);
public virtual void Install(IDictionary stateSaver);
public virtual void Rollback(IDictionary savedState);
public virtual void Uninstall(IDictionary savedState);
protected virtual void OnAfterInstall(IDictionary savedState);
protected virtual void OnAfterRollback(IDictionary savedState);
protected virtual void OnAfterUninstall(IDictionary savedState);
protected virtual void OnBeforeInstall(IDictionary savedState);
protected virtual void OnBeforeRollback(IDictionary savedState);
protected virtual void OnBeforeUninstall(IDictionary savedState);
protected virtual void OnCommitted(IDictionary savedState);
protected virtual void OnCommitting(IDictionary savedState);

It also defines event handlers for each of these steps as well:

public event InstallEventHandler BeforeInstall;
public event InstallEventHandler Committing;
public event InstallEventHandler AfterUninstall;
public event InstallEventHandler AfterRollback;
public event InstallEventHandler AfterInstall;
public event InstallEventHandler Committed;
public event InstallEventHandler BeforeRollback;
public event InstallEventHandler BeforeUninstall;

To add an installer class to the Windows Service project, there’s a helper you can use by right clicking on the designer view of the service and selecting “Add Installer” from the context menu:

This will add a new file called ProjectInstaller.cs to your project, which has its own designer view. The designer view has a corresponding ProjectInstaller.Designer.cs file that amends the ProjectInstaller class with the code generated by the designer. You’ll notice that this designer view already defines two objects, serviceInstaller1 and serviceProcessInstaller1.

These are special installer classes that will handle all the default installation tasks for your service. serviceInstaller1 is of type ServiceInstaller and handles defining the service name and if it should auto start when the machine boots up. serviceProcessInstaller1 is of type ServiceProcessInstaller and handles setting up the Service Logon Account, which the service will run with once installed. Both of these are already set up and invoked by the designer generated code in ProjectInstaller.Designer.cs.

Since both of these special service installers extend System.Configuration.Install.Installer, you can add custom code to occur at any point of the installation on these as well. The designer view again provides a GUI helper to add this in. Double-clicking on serviceInstaller1 will automatically add a new method to ProjectInstaller:

private void serviceInstaller1_AfterInstall(object sender, InstallEventArgs e) { }

It will also put some code into ProjectInstaller.Designer.cs which adds this method to the AfterInstall event of serviceInstaller1.

Adding Installer CLI Options

It’s also possible to add custom properties that you can pass to the installer as command line arguments. These can be done by defining Custom Actions on the primary build output of your project. To do this, go to the Custom Actions view of the installer project, right click on “Install” and select “Add Custom Action” from the context menu:

This will open up a dialog that prompts you to select a file in the installer’s file system to define a custom action for. In this case, we want to define a custom action on the primary build output. This way, the custom CLI options we are about to define will be passed to the project’s installer class.

After you click “OK”, the primary build output will show up in the Custom Actions view. When you click on it, you’ll notice that the properties window has a property called CustomActionData. In short, you can use it to define custom CLI arguments like this:

CustomActionData has its own syntax, so let’s dive deeper into what this actually does. We’re mapping the value of USERNAME and PASSWORD from the installer’s Properties Collection to the InstallContext of the installer class of your project under the Username and Password keys, respectively. The square brackets denote that the value is to be taken from the Properties Collection, and the quotes allow the value of the property to contain spaces. The forward slash denotes that we are adding a new key to the context. Any command line arguments passed to the installer are added to the Properties Collection by default.

Using Custom CLI Options in the Installer Class

Now that we have defined our custom action with the CLI arguments, we can go over to the project’s Installer class and access them via the Context property. In this example, we’re using the custom properties to define the Logon account for the service, which needs to be set right before the installation happens. We can use the Install(IDictionary stateSaver) lifecycle hook method for this purpose:

public override void Install(IDictionary stateSaver)
{
    // If no username or password is specified, fall back to installing the service to run as the SYSTEM account
    if (
        string.IsNullOrEmpty(this.Context.Parameters["Username"])
        || string.IsNullOrEmpty(this.Context.Parameters["Password"])
    ) {
        this.serviceProcessInstaller1.Account = System.ServiceProcess.ServiceAccount.LocalSystem;
    }

    // Otherwise, configure the service to run under the specified account.
    else
    {
        this.serviceProcessInstaller1.Username = this.Context.Parameters["Username"];
        this.serviceProcessInstaller1.Password = this.Context.Parameters["Password"];
    }

    // Run the base class install after the service has been configured.
    base.Install(stateSaver);
}

Conditionally Installing Files

It’s also possible to make the installer conditionally install files based on a value from the Properties Collection. One example of how this can be useful would be swapping in the production or development configuration file based on the value of a command line argument. We don’t need to write any additional code to do this, we just have to add a value for the Condition property in the Properties Window for the file:

The above condition will make the file settings.production.config be installed only if the DEBUG command line argument is not defined or is set to “false”. Like the custom actions, this property is also sourced from the Properties Collection.

Conclusion

And that’s it! I found that having a dedicated .msi installer was handy for making the setup of my Windows Service completely hands-free. While some of the features you need might seem buried within context menus, the flexibility of having the installer handle the service setup is well worth the effort.

Have any questions? Feel free to leave a comment!

Rapid Test-Driven Development in Julia

2020-11-09T00:00:00+00:00

[//]: # ( Kamil wrote: image I’ve obtained from www.freepik.com where I have a paid account. The license type says: Premium license (Unlimited use without attribution). https://www.freepik.com/free-vector/isometric-automated-production-line-concept-with-industrial-conveyor-belt-robotic-mechanical-arms-isolated_10055534.htm )

The Julia programming language has been rising in the ranks among the science-oriented programming languages lately. It has proven to be revolutionary in many ways. I’ve been watching its development for years now. It’s one of the most innovative of all the modern programming languages.

Julia’s design seems to be driven by two goals: to appeal to the scientific community and to achieve the best performance possible. This is an attempt to solve the “two languages problem” where data analysis and model building is performed using a slower interpreted language (like R or Python) while performance-critical parts are written in a faster language like C or C++.

The type-system is what allows Julia to meet its goals. The mix of strong and dynamic typing enables Python-like productivity with C++ or Rust-like performance. Julia is not an interpreted language. It compiles its code to native binary just like C, C++, Go, or Rust. The compilation and execution, though, are what sets it 1000 feet apart from all those other languages.

Here’s a simplified, brief outline of the steps in Julia’s code execution model:

Julia process starts up.
Code is parsed.
For each code chunk:
- If it hasn’t yet been compiled, decide whether to interpret or JIT compile it and then execute:
  - If compile then infer the types and use LLVM to produce native code.
  - Execute the newly-created native code.
- If it has been compiled, execute it.
Repeat until the program ends or the user closes the REPL.

It’s quite apparent that the compilation and type inference happen at a very different time compared to other compiled languages. Using Rust, you compile your code just once. Execution isn’t taxed by consecutive recompilation.

The result is quite a big negative surprise to Julia’s newcomers. Each time you run your app, there’s a significant slowdown before you see anything. It’s called the “time to first plot” issue. This is because, for example, a data scientist may want to generate some plots during her “exploratory data analysis”. Doing it in languages that are slower on paper — like R — makes those plots appear way quicker than in Julia.

There are more “time to first X” issues in Julia

Julia’s execution model makes more aspects trickier than just seeing the first plot. If you’re a software engineer who’s used to following the test-driven development (TDD) approach, you’re in for a big surprise.

In languages like Ruby or Rust, it’s easy to have a tool watch for any file changes and respond by running the project’s testing suite. I often use the watchexec tool which works with virtually any language, interpreter, or compiler. I run watchexec -cw . "bundle exec rspec --fail-fast" when working on a Ruby project, or watchexec -cw . "cargo test" with Rust.

With Julia this approach is not an option though — the “time to first test” is dramatically long. The wastefulness of continuous re-compilation steals my precious time, making me extremely unproductive.

Making it work in Julia

The “time to first X” issue is only a problem if we’re closing the session in which our code has already been compiled. If we could move the file-watching and test-running steps all into the same session, the testing suite would run slowly only the first time. Julia’s standard library has built-in file watching functions that we could use to reproduce the watchexec in our code:

julia> using FileWatching

julia> watch_file
watch_file (generic function with 2 methods)

julia> watch_folder
watch_folder (generic function with 4 methods)

We can use those to get notified about the changes in our project’s files whenever they happen. Let’s imagine the following simple project’s structure:

$ tree
.
└── src
    ├── App.jl
    └── nested
        └── Other.jl

2 directories, 2 files

How do we watch for file changes in Julia? Let’s start up the REPL and see:

julia> using FileWatching

help?> watch_file
search: watch_file watch_folder unwatch_folder

  watch_file(path::AbstractString, timeout_s::Real=-1)


  Watch file or directory path for changes until a change occurs or timeout_s seconds have elapsed.

  The returned value is an object with boolean fields changed, renamed, and timedout, giving the result of watching the file.

  This behavior of this function varies slightly across platforms. See https://nodejs.org/api/fs.html#fs_caveats (https://nodejs.org/api/fs.html#fs_caveats) for more detailed information.

julia> watch_file("src")

The REPL didn’t return from the watch_file function. We can now change the “src/App.jl” file and see what happens:

julia> watch_file("src")
FileWatching.FileEvent(true, true, false)

julia>

Good! The function returned a FileEvent struct. We can ask Julia for its definition:

help?> FileWatching.FileEvent
  No documentation found.

  Summary
  ≡≡≡≡≡≡≡≡≡

  struct FileWatching.FileEvent <: Any


  Fields
  ≡≡≡≡≡≡≡≡

  renamed  :: Bool
  changed  :: Bool
  timedout :: Bool

We can see it tells us whether the file’s been renamed, changed, or if the timeout happened.

So far so good, can we get it to notify us when the nested file changes too?

julia> watch_file("src")

Now changing the “src/nested/Other.jl”:

julia> watch_file("src")

Nothing happened. We’ll need to be specific about the nested directory to make it work:

julia> watch_file("src/nested")
FileWatching.FileEvent(true, true, false)

With those experiments we can now conclude that:

We’ll need to watch on all possible nested directories at the same time.
Watching blocks the current thread so for each folder to watch we need a separate thread.

We’ll need a list of folders. My first idea was to use the Glob package:

julia> using Glob

julia> glob("**/*")
2-element Array{String,1}:
 "src/App.jl"
 "src/nested"

This seems legit but let’s nest another folder. Here’s how the project’s structure would look now:

$ tree .
.
└── src
    ├── App.jl
    └── nested
        ├── Other.jl
        └── nested2
            └── YetAnother.jl

3 directories, 3 files

Trying the Glob package again:

julia> glob("**/*")
2-element Array{String,1}:
 "src/App.jl"
 "src/nested"

julia> glob("**/**/*")
2-element Array{String,1}:
 "src/nested/Other.jl"
 "src/nested/nested2"

Turns out that Julia’s Glob package doesn’t support extensions that allow “recursive” globbing. We’ll need to roll our own code to return all the possible nested folders:

julia> function subdirs(base="src")
         ret = [base]

         for (root, dirs, _) in walkdir(base)
           fulldirs = map(d -> joinpath(root, d), dirs)

           ret = vcat(vcat(vcat(map(subdirs, fulldirs)...), fulldirs), ret)
         end

         return ret |> unique
       end
subdirs (generic function with 2 methods)

julia> subdirs("src")
3-element Array{Any,1}:
 "src/nested/nested2"
 "src/nested"
 "src"

Being able to list all the nested directories, we can now work on the file-watching function. Here’s the plan of attack:

Create a “channel” to receive the file changes from other threads watching each of those directories.
Spin up a new thread for working through the stream from the channel specifically.
Spin up threads for every nested directory found and watch for changes at the same time.
When the file change is detected, queue it into the channel.

function onchange(f, basedirs=["src"])
  channel = Channel()

  function handle()
    should_continue = true

    for file in channel
      try
        f(file)
      catch err
        should_continue = typeof(err) != InterruptException

        @warn("Error in the hanlder:\n$err")
      end
    end
  end

  function schedule(file)
    put!(channel, file)
  end

  Threads.@spawn handle()

  subs = vcat(map(basedir -> subdirs(basedir), basedirs)...)

  @threads for dir in subs
    should_continue = true

    while true
      (file, event) = watch_folder(dir, 1)

      if event.changed
        try
          schedule(file)
        catch err
          should_continue = typeof(err) != InterruptException

          @warn("Error in the scheduler:\n$err")
        end
      end
    end
  end

  for dir in subs
    unwatch_folder(dir)
  end
end

Before we can run this code though, we need to mention one of other of Julia’s quirks. The @threads macro is cool and all, but it’s not going to work unless you start Julia with some predefined number of threads first:

$ JULIA_NUM_THREADS=4 julia

Let’s give it a go now:

julia> onchange(f -> println(f))

While the REPL is still “inside” the onchange function, let’s change some of those files in the dummy project and see what happens:

julia> onchange(f -> println(f))
4913
App.jl
App.jl
4913
Other.jl
Other.jl

It works! The output is weird but we do get something here. For each file change, we get three messages here. After being puzzled for hours with how Julia implements this file watching I decided to just not mind it and add the throttling to make it work for my testing needs. The idea is that the throttling will only run the suite once per each of those triples.

Fortunately, the Flux package comes with the throttle function that we can reuse:

function throttle(f, timeout; leading=true, trailing=false)
  cooldown = true
  later = nothing
  result = nothing

  function throttled(args...; kwargs...)
    yield()

    if cooldown
      if leading
        result = f(args...; kwargs...)
      else
        later = () -> f(args...; kwargs...)
      end

      cooldown = false
      @async try
      while (sleep(timeout); later != nothing)
          later()
          later = nothing
        end
      finally
        cooldown = true
      end
    elseif trailing
      later = () -> (result = f(args...; kwargs...))
    end

    return result
  end
end

And the final version of our function:

function onchange(f, basedirs=["src", "test"], timeout=1)
  channel = Channel()

  function handle()
    should_continue = true

    for file in channel
      try
        f(file)
      catch err
        should_continue = typeof(err) != InterruptException

        @warn("Error in the hanlder:\n$err")
      end
    end
  end

  function schedule(file)
    put!(channel, file)
  end

  throttled_schedule = throttle(schedule, timeout)

  Threads.@spawn handle()

  subs = vcat(map(basedir -> subdirs(basedir), basedirs)...)

  @threads for dir in subs
    should_continue = true

    while true
      (file, event) = watch_folder(dir, 1)

      if event.changed
        try
          throttled_schedule(file)
        catch err
          should_continue = typeof(err) != InterruptException

          @warn("Error in the scheduler:\n$err")
        end
      end
    end
  end

  for dir in subs
    unwatch_folder(dir)
  end
end

Putting it all together

Armed with the helper onchange function we can now set up our nice auto-test runner. Let’s add the “test/runtests.jl” file:

using Test

function runtests()
  @testset "the project" begin
    include("test/test_one.jl")
    include("test/test_two.jl")
  end

  nothing
end

function watchtest()
  onchange(_ -> runtests())
end

Now, with the watchtest function running the whole testing suite re-runs whenever any of the project’s files changes.

Final words

I found it easy to have a love-hate relationship with Julia. I have all the respect for its creators. They’re doing an amazing job and are very bold with bringing in innovation. Once the code is compiled, it’s amazingly fast. The language’s ecosystem, along with amazing packages is one of its strongest points.

However, it’s awfully slow when you run your functions for the first time in the current session. Also, developers often have to rethink the workflows they’re so used to. This article touches on one of those issues.

The way the file watching is implemented in the standard library leaves a lot of room for improvement. As an example, I’m getting the “renamed” flag instead of “changed” in the FileWatching.FileEvent when I’m changing the file. That’s why in code I’m just checking for the absence of the timeout. It feels like a dirty hack but what can you do? The watcher was also not working consistently when the timeout was not given.

The immaturity of the standard library isn’t a show-stopper for many. Julia is developing rapidly and we can expect it to get better and better over time. Other issues will need engineers themselves to rethink their paradigms. I think it’s good though — challenges are what’s making us evolve after all and radical innovation doesn’t happen that often.

Deploying (Minecraft) Servers Automatically with Terraform

2020-07-16T00:00:00+00:00

Last year I bought an old Dell Optiplex on eBay to use as a dedicated Minecraft server for my friends and me. It worked well for a while, but when my university switched to online classes and I moved home, I left it at my college apartment and was unable to fix it (or retrieve our world save) when it failed for some reason. I still wanted to play Minecraft with friends, though, so I had to figure out a solution in the meantime.

I’d previously used a basic DigitalOcean droplet as a Minecraft server, but that had suffered with lag issues, especially with more than two or three people logged in. Their $5 tier of virtual machine provides 1GB of RAM and 1 CPU core, so it shouldn’t be too much of a surprise that it struggled with a Minecraft server. However, more performant virtual machines cost a lot more, and I wanted to keep my solution as cheap as possible.

I mentioned this to a co-worker and he pointed out that most companies don’t actually charge for virtual machines on a monthly basis; in reality, it’s an hourly rate based on when your virtual machine instance actually exists. So, he suggested I create a virtual machine and start my Minecraft server every time I wanted to play, then shut it down and delete it when I was finished, thus saving the cost of running it when it wasn’t being used.

Of course, you could do this manually in your provider’s dev console, but who wants to manually download dependencies, copy your world over, and set up a new server every time you want to play Minecraft? Not me! Instead, I used Terraform, an open-source tool that lets you describe your desired infrastructure and then sets it up for you.

In this post, I’ll show how I got my server setup streamlined into one Terraform configuration file that creates a virtual machine, runs a setup script on it, copies my Minecraft world to it with rsync, starts the Minecraft server, and adds a DNS entry for your new server.

Picking a provider

As I mentioned earlier, I’ve used DigitalOcean in the past, but at the recommendation of my co-worker, I decided to try UpCloud, a similar service based in Helsinki, Finland. They have datacenters in the US, Europe, and Singapore (the one I used is in Chicago—a full list of locations can be found here). They offer several affordable tiers of virtual machines, including these:

Memory	CPU	Storage	Transfer	Price
1 GB	1	25 GB	1 TB	$5/mo
2 GB	1	50 GB	2 TB	$10/mo
4 GB	2	80 GB	4 TB	$20/mo
8 GB	4	160 GB	5 TB	$40/mo

There are more tiers, but for a Minecraft server I didn’t need anything that powerful. I decided on the $20/month tier—or, in my case, $0.03/hour.

Installing Terraform

The process for setting up Terraform depends on what provider you’re using. If you’d like to use DigitalOcean or another provider, just make sure they’re on Terraform’s list of supported providers. Alternatively (as is the case for UpCloud), some providers aren’t officially supported but do have plugins that work well.

In my case, I followed UpCloud’s instructions for installation here. The rest of that tutorial does a great job showing you how to use Terraform with UpCloud via their plugin, but for instructions more suited to other use cases I’d recommend looking at Terraform’s wealth of learning resources, which can be found here.

You’ll also need to install the UpCloud plugin. This is detailed in the earlier tutorial at UpCloud’s site, so I’ll forgo putting it here.

Creating a Terraform configuration file

Once you’ve got Terraform and UpCloud’s plugin installed, create a directory for your project (mkdir minecraft) and run terraform init in it. Once you’ve done that, using Terraform is as simple as creating a configuration file describing your server. How exactly that looks depends on your provider, but my initial file, which I named server.tf, looked something like this:

provider "upcloud" {
  # Your UpCloud credentials are read from the environment variables
  # export UPCLOUD_USERNAME="Username for UpCloud API user"
  # export UPCLOUD_PASSWORD="Password for UpCloud API user"
}

resource "upcloud_server" "minecraft1" {
  # System hostname
  hostname = "minecraft1"

  # Availability zone
  zone = "us-chi1"

  # Number of CPUs and memory in GB
  plan = "2xCPU-4GB"

  storage_devices {
    # OS root disk size
    size = 25

    # Template UUID for Ubuntu 18.04
    storage = "01000000-0000-4000-8000-000030080200"

    tier   = "maxiops"
    action = "clone"
  }

  # Include at least one public SSH key
  login {
    user = "root"
    keys = [
      "PUBLIC_SSH_KEY_HERE"
    ]
    create_password = false
  }

  # Configuring connection details
  connection {
    host        = self.ipv4_address
    type        = "ssh"
    user        = "root"
    private_key = file("./id_rsa_upcloud")
  }
}

There are a few important sections here:

The hostname (which will be important later)
The zone (Full list here)
The plan (more information on the plan codes here)
The login and connection sections

The last two in particular need some attention; I created an SSH key specifically for managing my Minecraft server and included the public key in the login section and a link to the private key in the connection section.

After providing credentials via environment variables as per UpCloud’s tutorial, you can run terraform plan to see what your current configuration looks like:

zed@zeds-pc:~/minecraft $ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.


------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # upcloud_server.minecraft1 will be created
  + resource "upcloud_server" "minecraft1" {
      + cpu                  = (known after apply)
      + hostname             = "minecraft1"
      + id                   = (known after apply)
      + ipv4                 = true
      + ipv4_address         = (known after apply)
      + ipv4_address_private = (known after apply)
      + ipv6                 = true
      + ipv6_address         = (known after apply)
      + mem                  = (known after apply)
      + plan                 = "2xCPU-4GB"
      + private_networking   = true
      + title                = (known after apply)
      + zone                 = "us-chi1"

      + login {
          + create_password   = false
          + keys              = [
              + "PUBLIC_SSH_KEY_HERE",
            ]
          + password_delivery = "none"
          + user              = "root"
        }

      + storage_devices {
          + action  = "clone"
          + address = (known after apply)
          + id      = (known after apply)
          + size    = 25
          + storage = "01000000-0000-4000-8000-000030080200"
          + tier    = "maxiops"
          + title   = (known after apply)
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.

------------------------------------------------------------------------

Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.

This command shows you what’ll happen when you run terraform apply, which also shows you the changes that are going to happen, after which it prompts you to apply them. We’ll hold off for now, since we need more than a blank Ubuntu server; we need to get our Minecraft world and libraries onto this server.

Adding startup scripts

To get my new virtual machine running a Minecraft server, I needed to perform some initial setup. Specifically, I needed to copy my server files from my local machine, then download the Java runtime and start the server on the newly created VM. To accomplish this, I created a setup script and uploaded it to a private server; you could also use a public GitHub repo or similar and just download it from there. I also had to configure my server.tf to get this script and copy over my Minecraft world after creation.

The sections I added to my upcloud resource block in my server.tf file look like this:

  provisioner "remote-exec" {
    inline = [ 
      "wget http://zedjensen.com/init_minecraft.sh && chmod +x init_minecraft.sh && ./init_minecraft.sh -f && rm init_minecraft.sh"
    ]   
  }

  provisioner "local-exec" {
    command = "rsync --delete -r -e 'ssh -F ./ssh_config -i ./id_rsa_upcloud' server root@${self.ipv4_address}:~/minecraft/"
  }

  provisioner "remote-exec" {
    inline = [ 
      "bash ~/minecraft/start_server.sh"
    ]   
  }

The provisioners remote-exec and local-exec tell Terraform to run these commands on the new virtual machine and my local machine respectively. As you can see, I have the remote machine fetching an init script I wrote and running it, then deleting it. The local exec then copies over my Minecraft server, using the SSH key I created specifically for this, and using an alternate SSH config that uses /dev/null as the known_hosts file, avoiding security warnings on later runs when the machine at minecraft.zedjensen.com isn’t the same machine as it was before. ${self.ipv4_address} is replaced by Terraform with the IP address of the new server. Finally, I run start_server.sh from the files I copied over with rsync.

For reference, the relevant parts of my init script and ssh_config look something like this:

#!/bin/bash

echo "StrictHostKeyChecking accept-new" >> /etc/ssh/ssh_config
service sshd restart

i=0
tput sc
while ! apt install openjdk-8-jdk-headless tmux zip unzip -y &> /dev/null; do
    case $i in
        0 ) j="-" ;;
        1 ) j="\\" ;;
        2 ) j="|" ;;
        3 ) j="/" ;;
    esac
    tput rc echo =en "\r[$j] Waiting to install dependencies..."
    sleep 1
done

mkdir -p minecraft/server

StrictHostKeyChecking no
UserKnownHostsFile /dev/null

Adding a DNS record with Cloudflare

After the last steps, you should be able to get a server up and running, but you’d have to use the IP address to connect to it directly. I have the domain zedjensen.com, so I decided to use minecraft.zedjensen.com. Luckily, Terraform also supports Cloudflare, so I set up Cloudflare as my DNS provider for zedjensen.com and added a new section to my server.tf:

provider "cloudflare" {
  version = "~> 2.0"
  email = "zed@whatever.com"
  # Your CloudFlare API token is also read from environment variables
  # api_token = "MY_API_TOKEN"
}

resource "cloudflare_record" "minecraft1" {
  name = "minecraft.zedjensen.com"
  zone_id = "ZONE_ID_HERE"
  value = upcloud_server.minecraft1.ipv4_address
  type = "A"

  depends_on = [upcloud_server.minecraft1]
}

As you can probably guess, this resource block tells Terraform to add a new A record to my DNS configuration for minecraft.zedjensen.com with my new server’s IP address. The last part, depends_on, tells Terraform to wait until it’s done with the server setup, otherwise it’ll try to do them at the same time, before it knows the IP address for the new VM. Don’t forget to put your API token in your environment variables too (this one would be CLOUDFLARE_API_KEY).

Adding scripts for destruction

Lastly, I needed to make sure that when I was shutting my server down, my Minecraft world was safely copied off first. Terraform supports hooks for shutting down as well! I added a final section to the server block of my server.tf:

server {

  // ...

  provisioner "local-exec" {
    when = destroy
    command = "./copy_back.sh ${self.ipv4_address}"
  }
}

This is similar to the provisioners I added earlier, but it runs only when I’m destroying my server. In addition, if the command fails, Terraform will abort the destruction of the server, giving me a chance to see what went wrong and make a backup manually if needed. The copy_back.sh command I’m using here is just a helper script that copies the world back to my local machine; I also put the entire directory in a zipfile and copy that back for redundancy.

Trying it out

After completing all the above steps, here’s my completed server.tf:

provider "upcloud" {
  # Your UpCloud credentials are read from the environment variables
  # export UPCLOUD_USERNAME="Username for UpCloud API user"
  # export UPCLOUD_PASSWORD="Password for UpCloud API user"
}

provider "cloudflare" {
  version = "~> 2.0"
  email = "zed@whatever.com"
  api_token = "MY_API_TOKEN"
}

resource "upcloud_server" "minecraft1" {
  # System hostname
  hostname = "minecraft1"

  # Availability zone
  zone = "us-chi1"

  # Number of CPUs and memory in GB
  plan = "2xCPU-4GB"

  storage_devices {
    # OS root disk size
    size = 25

    # Template UUID for Ubuntu 18.04
    storage = "01000000-0000-4000-8000-000030080200"

    tier   = "maxiops"
    action = "clone"
  }

  # Include at least one public SSH key
  login {
    user = "root"
    keys = [
      "MY PUBLIC KEY HERE"
    ]
    create_password = false
  }

  # Configuring connection details
  connection {
    host        = self.ipv4_address
    type        = "ssh"
    user        = "root"
    private_key = file("./id_rsa_upcloud")
  }

  provisioner "remote-exec" {
    inline = [
      "wget http://zedjensen.com/init_minecraft.sh && chmod +x init_minecraft.sh && ./init_minecraft.sh -f && rm init_minecraft.sh"
    ]
  }

  provisioner "local-exec" {
    command = "rsync --delete -r -e 'ssh -F ./ssh_config -i ./id_rsa_upcloud' server root@${self.ipv4_address}:~/minecraft/"
  }

  provisioner "remote-exec" {
    inline = [
      "bash ~/minecraft/start_server.sh"
    ]
  }

  provisioner "local-exec" {
    when = destroy
    command = "./get_backup ${self.ipv4_address}"
  }

  provisioner "local-exec" {
    when = destroy
    command = "./run_rsync ${self.ipv4_address}"
  }
}

resource "cloudflare_record" "minecraft1" {
  name = "minecraft.zedjensen.com"
  zone_id = "ZONE_ID_HERE"
  value = upcloud_server.minecraft1.ipv4_address
  type = "A"

  depends_on = [upcloud_server.minecraft1]
}

Now that we’ve defined our server configuration, created scripts to set the remote server up, and added provisioners for startup and takedown, we should be able to run our script and see the magic happen. We’ll use the -out option this time so that we only have to review the configuration once.

zed@zeds-pc:~/minecraft $ terraform plan -out plan

Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.


------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # cloudflare_record.minecraft1 will be created
  + resource "cloudflare_record" "minecraft1" {
      + created_on  = (known after apply)
      + hostname    = (known after apply)
      + id          = (known after apply)
      + metadata    = (known after apply)
      + modified_on = (known after apply)
      + name        = "minecraft.zedjensen.com"
      + proxiable   = (known after apply)
      + proxied     = false
      + ttl         = (known after apply)
      + type        = "A"
      + value       = (known after apply)
      + zone_id     = "ZONE_ID_HERE"
    }

  # upcloud_server.minecraft1 will be created
  + resource "upcloud_server" "minecraft1" {
      + cpu                  = (known after apply)
      + hostname             = "minecraft1"
      + id                   = (known after apply)
      + ipv4                 = true
      + ipv4_address         = (known after apply)
      + ipv4_address_private = (known after apply)
      + ipv6                 = true
      + ipv6_address         = (known after apply)
      + mem                  = (known after apply)
      + plan                 = "2xCPU-4GB"
      + private_networking   = true
      + title                = (known after apply)
      + zone                 = "us-chi1"

      + login {
          + create_password   = false
          + keys              = [
              + "SSH_KEY_HERE"
            ]
          + password_delivery = "none"
          + user              = "root"
        }

      + storage_devices {
          + action  = "clone"
          + address = (known after apply)
          + id      = (known after apply)
          + size    = 25
          + storage = "01000000-0000-4000-8000-000030080200"
          + tier    = "maxiops"
          + title   = (known after apply)
        }
    }

Plan: 2 to add, 0 to change, 0 to destroy.

------------------------------------------------------------------------

This plan was saved to: plan

To perform exactly these actions, run the following command to apply:
    terraform apply "plan"

After reviewing the list of changes to be made, we can apply our changes by following the instructions and running terraform apply plan:

zed@zeds-pc:~/minecraft $ terraform apply plan

upcloud_server.minecraft1: Creating...
upcloud_server.minecraft1: Still creating... [10s elapsed]
upcloud_server.minecraft1: Still creating... [20s elapsed]
upcloud_server.minecraft1: Still creating... [30s elapsed]
upcloud_server.minecraft1: Still creating... [40s elapsed]
upcloud_server.minecraft1: Still creating... [50s elapsed]
upcloud_server.minecraft1: Provisioning with 'remote-exec'...
upcloud_server.minecraft1 (remote-exec): Connecting to remote host via SSH...
upcloud_server.minecraft1 (remote-exec):   Host: 111.222.111.222
upcloud_server.minecraft1 (remote-exec):   User: root
upcloud_server.minecraft1 (remote-exec):   Password: false
upcloud_server.minecraft1 (remote-exec):   Private key: true
upcloud_server.minecraft1 (remote-exec):   Certificate: false
upcloud_server.minecraft1 (remote-exec):   SSH Agent: true
upcloud_server.minecraft1 (remote-exec):   Checking Host Key: false
upcloud_server.minecraft1 (remote-exec): Connected!
upcloud_server.minecraft1 (remote-exec): --2020-07-17 20:36:02--  http://zedjensen.com/init_minecraft.sh
upcloud_server.minecraft1 (remote-exec): Resolving zedjensen.com (zedjensen.com)... 111.222.111.222
upcloud_server.minecraft1 (remote-exec): Connecting to zedjensen.com (zedjensen.com)|111.222.111.222|:80... connected.
upcloud_server.minecraft1 (remote-exec): HTTP request sent, awaiting response... 200 OK
upcloud_server.minecraft1 (remote-exec): Length: 1369 (1.3K) [application/octet-stream]
upcloud_server.minecraft1 (remote-exec): Saving to: 'init_minecraft.sh'

upcloud_server.minecraft1 (remote-exec):       init_   0%       0  --.-KB/s
upcloud_server.minecraft1 (remote-exec): init_minecr 100%   1.34K  --.-KB/s    in 0s

upcloud_server.minecraft1 (remote-exec): 2020-07-17 20:36:03 (205 MB/s) - 'init_minecraft.sh' saved [1369/1369]

upcloud_server.minecraft1 (remote-exec):
upcloud_server.minecraft1: Still creating... [1m0s elapsed]
upcloud_server.minecraft1: Still creating... [1m10s elapsed]
upcloud_server.minecraft1: Still creating... [1m20s elapsed]
upcloud_server.minecraft1: Still creating... [1m30s elapsed]
upcloud_server.minecraft1: Still creating... [1m40s elapsed]
upcloud_server.minecraft1 (remote-exec):
upcloud_server.minecraft1: Still creating... [1m50s elapsed]
upcloud_server.minecraft1: Still creating... [2m0s elapsed]
upcloud_server.minecraft1 (remote-exec):
upcloud_server.minecraft1: Still creating... [2m10s elapsed]
upcloud_server.minecraft1: Provisioning with 'local-exec'...
upcloud_server.minecraft1 (local-exec): Executing: ["/bin/sh" "-c" "rsync --delete -r -e 'ssh -F ./ssh_config -i ./id_rsa_upcloud' server root@111.222.111.222:~/minecraft/"]
upcloud_server.minecraft1 (local-exec): Warning: Permanently added '111.222.111.222' (ECDSA) to the list of known hosts.
upcloud_server.minecraft1: Still creating... [2m20s elapsed]

...

upcloud_server.minecraft1: Still creating... [9m40s elapsed]
upcloud_server.minecraft1: Provisioning with 'remote-exec'...
upcloud_server.minecraft1 (remote-exec): Connecting to remote host via SSH...
upcloud_server.minecraft1 (remote-exec):   Host: 111.222.111.222
upcloud_server.minecraft1 (remote-exec):   User: root
upcloud_server.minecraft1 (remote-exec):   Password: false
upcloud_server.minecraft1 (remote-exec):   Private key: true
upcloud_server.minecraft1 (remote-exec):   Certificate: false
upcloud_server.minecraft1 (remote-exec):   SSH Agent: true
upcloud_server.minecraft1 (remote-exec):   Checking Host Key: false
upcloud_server.minecraft1 (remote-exec): Connected!
upcloud_server.minecraft1: Still creating... [9m50s elapsed]
upcloud_server.minecraft1: Creation complete after 9m51s [id=stuff]
cloudflare_record.minecraft1: Creating...
cloudflare_record.minecraft1: Creation complete after 3s [id=things]

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.

State path: terraform.tfstate

Awesome! It takes a while to transfer files over, but once it’s done, my Minecraft server is up and running! To destroy, it’s just as easy:

zed@zeds-pc:~/minecraft$ terraform destroy
upcloud_server.minecraft1: Refreshing state... [id=things]
cloudflare_record.minecraft1: Refreshing state... [id=stuff]

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  # cloudflare_record.minecraft1 will be destroyed
  - resource "cloudflare_record" "minecraft1" {
      - created_on  = "2020-07-15T22:25:04.460564Z" -> null
      - data        = {} -> null
      - hostname    = "minecraft.zedjensen.com" -> null
      - id          = "stuff" -> null
      - metadata    = {
          - "auto_added"             = "false"
          - "managed_by_apps"        = "false"
          - "managed_by_argo_tunnel" = "false"
          - "source"                 = "primary"
        } -> null
      - modified_on = "2020-07-15T22:25:04.460564Z" -> null
      - name        = "minecraft.zedjensen.com" -> null
      - priority    = 0 -> null
      - proxiable   = true -> null
      - proxied     = false -> null
      - ttl         = 1 -> null
      - type        = "A" -> null
      - value       = "111.222.111.222" -> null
      - zone_id     = "ZONE_ID_HERE" -> null
    }

  # upcloud_server.minecraft1 will be destroyed
  - resource "upcloud_server" "minecraft1" {
      - cpu                  = 2 -> null
      - hostname             = "minecraft1" -> null
      - id                   = "things" -> null
      - ipv4                 = true -> null
      - ipv4_address         = "111.222.111.222" -> null
      - ipv4_address_private = "IPv4 addr" -> null
      - ipv6                 = true -> null
      - ipv6_address         = "IPv6 addr" -> null
      - mem                  = 4096 -> null
      - plan                 = "2xCPU-4GB" -> null
      - private_networking   = true -> null
      - title                = "minecraft1 (managed by terraform)" -> null
      - zone                 = "us-chi1" -> null

      - login {
          - create_password   = false -> null
          - keys              = [
              - "SSH_KEY_HERE"
            ] -> null
          - password_delivery = "none" -> null
          - user              = "root" -> null
        }

      - storage_devices {
          - action      = "clone" -> null
          - address     = "virtio:0" -> null
          - backup_rule = {} -> null
          - id          = "stuff" -> null
          - size        = 25 -> null
          - storage     = "01000000-0000-4000-8000-000030080200" -> null
          - tier        = "maxiops" -> null
          - title       = "terraform-minecraft1-disk-0" -> null
        }
    }

Plan: 0 to add, 0 to change, 2 to destroy.

Do you really want to destroy all resources?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes

cloudflare_record.minecraft1: Destroying... [id=stuff]
cloudflare_record.minecraft1: Destruction complete after 0s
upcloud_server.minecraft1: Destroying... [id=things]
upcloud_server.minecraft1: Provisioning with 'local-exec'...
upcloud_server.minecraft1 (local-exec): Executing: ["/bin/sh" "-c" "./get_backup 209.50.58.100"]
upcloud_server.minecraft1 (local-exec): Warning: Permanently added '209.50.58.100' (ECDSA) to the list of known hosts.
upcloud_server.minecraft1 (local-exec): Warning: Permanently added '209.50.58.100' (ECDSA) to the list of known hosts.
upcloud_server.minecraft1: Still destroying... [id=things, 20s elapsed]
upcloud_server.minecraft1: Provisioning with 'local-exec'...
upcloud_server.minecraft1 (local-exec): Executing: ["/bin/sh" "-c" "./run_rsync 209.50.58.100"]
upcloud_server.minecraft1 (local-exec): Warning: Permanently added '209.50.58.100' (ECDSA) to the list of known hosts.
upcloud_server.minecraft1 (local-exec): Warning: Permanently added '209.50.58.100' (ECDSA) to the list of known hosts.
upcloud_server.minecraft1: Still destroying... [id=things, 30s elapsed]
upcloud_server.minecraft1: Still destroying... [id=things, 40s elapsed]
upcloud_server.minecraft1: Still destroying... [id=things, 50s elapsed]
upcloud_server.minecraft1: Still destroying... [id=things, 1m1s elapsed]
upcloud_server.minecraft1: Destruction complete after 1m7s

Destroy complete! Resources: 2 destroyed.

Terraform proved to be a great solution for my Minecraft dilemma, but this could come in handy for lots of other purposes too. Let us know what you use Terraform for!

Randomly spacing cron jobs

2020-06-30T00:00:00+00:00

Cron is the default job scheduler for the Unix operating system family. It is old and well-used infrastructure — it was first released 45 years ago, in May 1975!

On Linux, macOS, and other Unix-like systems, you can see any cron jobs defined for your current user with:

crontab -l

If nothing is printed out, your user doesn’t have any cron jobs defined.

You can see the syntax for defining the recurring times that jobs should run with:

man 5 crontab

Important in that document is the explanation of the space-separated time and date fields:

field          allowed values
-----          --------------
minute         0-59
hour           0-23
day of month   1-31
month          1-12 (or names, see below)
day of week    0-7 (0 or 7 is Sunday, or use names)

A field may contain an asterisk (*), which always stands for "first-last".

For example, to make a job run every Monday at 3:33 am in the server’s defined time zone:

33 3 * * 1 /path/to/executable

Random interval scheduling

Sometimes it may be good to schedule a cron job to run at a somewhat random time: generally not truly random, but maybe at an arbitrary time within a specified time range rather than at a specific recurring interval.

This can be useful to keep simultaneous cron jobs for different users from causing predictable spikes in resource usage, or to run at a time other than the start of a new minute, since cron’s interval resolution doesn’t go smaller than one minute.

There isn’t any simple built-in way to randomize the scheduling in classic cron, but there are several ways to get it done:

cronie RANDOM_DELAY

The version of cron included with Red Hat Enterprise Linux (RHEL), CentOS, and Fedora Linux is cronie. It allows us to set the variable RANDOM_DELAY for this purpose. From its manual:

The RANDOM_DELAY variable allows delaying job startups by random amount of minutes with upper limit specified by the variable. The random scaling factor is determined during the cron daemon startup so it remains constant for the whole run time of the daemon.

To use that we set the variable in a crontab before the jobs it should apply to. For example, to delay every job defined after that variable by a random time up to 10 minutes:

RANDOM_DELAY=10

When it starts, cronie logs its random scaling factor so you can tell how long each RANDOM_DELAY will work out to be during the lifetime of this cron daemon. It looks like this:

# systemctl status crond
● crond.service - Command Scheduler
     Loaded: loaded (/usr/lib/systemd/system/crond.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2020-06-29 19:05:50 MDT; 2min 29s ago
   Main PID: 60630 (crond)
      Tasks: 1 (limit: 19011)
     Memory: 3.4M
        CPU: 218ms
     CGroup: /system.slice/crond.service
             └─60630 /usr/sbin/crond -n

Jun 29 19:05:50 myhost systemd[1]: Started Command Scheduler.
Jun 29 19:05:50 myhost crond[60630]: (CRON) STARTUP (1.5.5)
Jun 29 19:05:50 myhost crond[60630]: (CRON) INFO (RANDOM_DELAY will be scaled with factor 66% if used.)
Jun 29 19:05:50 myhost crond[60630]: (CRON) INFO (running with inotify support)
Jun 29 19:05:50 myhost crond[60630]: (CRON) INFO (@reboot jobs will be run at computer's startup.)

If we restart cronie with systemctl restart crond, we will (probably) see a different scaling factor:

Jun 29 22:42:02 myhost crond[75200]: (CRON) INFO (RANDOM_DELAY will be scaled with factor 29% if used.)

Note that this feature is not available on Debian & Ubuntu because they use the parent project vixie-cron, aka ISC Cron, that cronie descended from.

bash sleep RANDOM

If your cron uses bash as its default shell for executing jobs, or if you set it to bash with SHELL=/bin/bash before the cron job in question, you will have available the variable RANDOM which returns numbers between 0 and 32767.

To scale that number to the range of seconds you want to delay starting your cron job, do some modulo arithmetic. For example, to wait for a random time between 0 and 600 seconds (= 10 minutes) before running your job, do:

sleep $(( RANDOM % 600 )); /path/to/executable

That works when run directly from the shell, but watch out! Both cronie and vixie-cron terminate any line at the first unescaped % character, so you must write \% in the cron job. Why do they do that? It’s an old feature: The rest of the line after % is sent as stdin (standard input) to the command.

So that same example in a cron job would look like:

@reboot sleep $(( RANDOM \% 600 )); /path/to/executable

Wrapper script

If you want to avoid the pitfall of forgetting to escape % in cron jobs, you can write a wrapper script that waits a configurable random duration and then exits, and run that before running your program.

Perl sleep random

Another way is to use Perl instead of bash:

@reboot perl -e 'sleep(rand(3000))'; /path/to/executable

Or similarly in another scripting language of your choice.

at

If you want to have your job possibly wait quite a long while and not have the cron job sitting there sleeping the whole time, you can create a cron job that runs at the start of every period during which you want things to run randomly, and it can schedule an at job to run at later random time. For example:

@reboot echo /path/to/executable | at now + $(( RANDOM \% 60 )) minutes

Program delay feature

If your program has some inherent reason to delay before starting its actual work, you could add an option to your program to do the initial wait within a certain number of seconds.

But generally this seems better done by a separate program since it is unlikely that startup delay is a function closely related to your program and it will just bloat it. Why not keep it simple(r)?

OpenBSD cron

In OpenBSD 6.7 cron a new feature was introduced using the symbol ~. The OpenBSD crontab manual explains:

A random value (within the legal range) may be obtained by using the ‘~~’ character in a field. The interval of the random value may be specified explicitly, for example “0~~30” will result in a random value between 0 and 30 inclusive. If either (or both) of the numbers on either side of the ‘~’ are omitted, the appropriate limit (low or high) for the field will be used.

There is an important caveat not mentioned in the manual, explained on StackOverflow by user Kusalananda:

The random times are picked randomly, but they are fixed until the crontab is reloaded, i.e. until the cron daemon restarts or when the crontab is edited with crontab -e. This would therefore not provide you with a new random value for each run of the job, like using sleep with $RANDOM would do.

Jenkins scheduling

It is not standard Unix cron, but also consider the cron-inspired scheduling functionality in the Jenkins continuous integration system.

There a special symbol H is used to indicate that a “hashed” value is used, similarly to OpenBSD cron’s ~ when no range endpoints are specified. As the Wikipedia cron article describes it:

Thus instead of a fixed number such as 20 * * * * which means at 20 minutes after the hour every hour, H * * * * indicates that the task is performed every hour at an unspecified but invariant time for each task. This allows spreading out tasks over time, rather than having all of them start at the same time and compete for resources.

Though this seems effectively random, it is actually stable based on the job name, as the Jenkins Pipeline Syntax cron syntax documentation notes:

The H symbol can be thought of as a random value over a range, but it actually is a hash of the job name, not a random function, so that the value remains stable for any given project.

Limiting concurrent runs

Finally, a separate but important consideration: If you have a job that takes a long time to run, and if this isn’t just a job to run once with @reboot each time the server starts, you may want to make sure a new instance of the job will not start running before the previous one has finished.

You can do that by using a shared lock file that tells when another instance of the job is running, for example:

flock -n /tmp/describe-your.lock -c /path/to/executable

See man 1 flock for more details on what it can do.

⏲

Happy scheduling!

DevOpsDays India — 2015

2015-09-30T00:00:00+00:00

DevOpsIndia 2015 was held at The Royal Orchid in Bengaluru on Sep 12-13, 2015. After saying hello to a few familiar faces who I often see at the conferences, I collected some goodies and entered into the hall. Everything was set up for the talks. Niranjan Paranjape, one of the organizers, was giving the introduction and overview of the conference.

Justin Arbuckle from Chef gave a wonderful keynote talk about the “Hedgehog Concept” and spoke more about the importance of consistency, scale and velocity in software development.

In addition, he quoted “A small team with generalists who have a specialization, deliver far more than a large team of single skilled people.”

A talk on “DevOps of Big Data infrastructure at scale” was given by Rajat Venkatesh from Qubole. He explained the architecture of Qubole Data Service (QDS), which helps to autoscale the Hadoop cluster. In short, scale up happens based on the data from Hadoop Job Tracker about the number of jobs running and time to complete the jobs. Scale down will be done by decommissioning the node, and the server will be chosen by which is reaching the boundary of an hour. This is because most of the cloud service providers charge for an hour regardless of whether the usage is 1 minute or 59 minutes.

Vishal Uderani, a DevOps guy from WebEngage, presented “Automating AWS infrastructure and code deployment using Ansible.” He shared the issues facing the environments like task failure due to ssh timeout on executing a giant task using Ansible and solved by monitoring the task after triggering the task to get out of the system immediately. Integrating Rundeck with Ansible is an alternative for enterprise Ansible Tower. He also stated the following reasons for using Ansible:

Good learning curve
No agents will be running on the client side, which avoids having to monitor the agents at client nodes
Great deployment tool

Vipul Sharma from CodeIgnition stated the importance of Resilience Testing. The application should be tested periodically to be tough and strong enough to handle any kind of load. He said Simian Army can be used to create the problems in environments and then resolving them to make them flawless. Simian Army can used to improve application using security monkey, chaos monkey, janitor monkey, etc… Also “Friday Failure” is a good method to identify the problem and improve the application.

Avi Cavale from Shippable gave an awesome talk on “Modern DevOps with Docker”. He talk started with “What is the first question that arises during an outage ? … What changed ?”After fixing these issues, the next question will be “Who made the change?” Both questions are bad for the business. Change is the root cause of all outage but business requires changes. In his own words, DevOps is a culture of embracing change. Along with that he explained the zero downtime ways to deploy the changes using a container.

He said DevOps is a culture, make it FACT(F-Frictionless, A-Agile, C-Continuous and T-Transparency).

Rahul Mahale from SecureDB gave a demo on Terraform, a tool for build and orchestration in Cloud. It features “Infrastructure as Code” and also provides an option to generate the diagrams and graphs of the present infrastructure.

Shobhit and Sankalp from CodeIgnition shared their experience on solving network based issues. Instead of whitelisting the user’s location every time manually to provide the access to systems, they created a VPN to enable access only to users, not locations. They have resolved two more additional kind of issues by adding Router to bind two networks using FIP. Another issue is that they need to whitelist to access third party services from containers, but it was hard to whitelist all the containers. Therefore, they created and whitelisted VMs and containers accessed the third party services through VMs.

Ankur Trivedi from Xebia Labs spoke about the “Open Containers Initiative” project. He explained the evolution of the containers (Docker—2013 & rkt—2014). The various distributions of containers are compared based on the Packing, Identity, Distribution and Runtime capabilities. Open Containers is supported by the community and various companies who are doing extensive work on containers in order to standardize them.

Vamsee Kanala, a DevOps consultant, presented a talk on “Docker Networking—A Primer”. He spoke about Bridge networking, Host Networking, Mapped container networking and None (Self Managed) with dockers. The communications between the containers can happen through:

Port mapping
Link
Docker composing (programmatically)

In addition, he explained the tools which feature the clustering of containers, and listed the tools that have their own way of clustering and advantages:

Kubernetes
Mesos
Docker Swarm

Aditya Patawari from BrowserStack gave a demo on “Using Kubernetes to Build Fault Tolerant Container Clusters”. Kubernetes has a feature called “Replication Controllers,” which helps to maintain number of pods running at any time. “Kubernetes Services” defines a policy to enable access among the pods which provisions the pods as micro services.

Arjun Shenoy from LinkedIn introduced a tool called “SIMOORG.” The tool was developed at LinkedIn and does the failure induction in a cluster for testing the stability of the code. It is a components-based Open Source framework and few components are replaceable with external ones.

Dharmesh Kakadia, a researcher from Microsoft, gave a wonderful talk on “Mesos is the Linux”. He started with a wonderful explanation on Micro services (relating with linux commands, each command is a micro service) which is simplest, independently updatable, runnable and deployable. Mesos is a “Data Center Kernel” which takes care of scalability, fault tolerance, load balance, etc… in Data Center.

At the end, I got a chance to do some hands-on things on Docker and played with some of its features. It was a wonderful conference to learn more about configuration management and the containers world.

Install Tested Packages on Production Server

2015-09-07T00:00:00+00:00

One of our customers has us to do scheduled monthly OS updates following a specific rollout process. First week of the month, we will update the test server and wait for a week to confirm that everything looks as expected in the application; then next week we apply the very same updates to the production servers.

Since not long ago we used to use aptitude to perform system updates. While doing the update on the test server, we also executed aptitude on production servers to “freeze” the same packages and version to be updated on following week. That helped to ensure that only tested packages would have been updated on the production servers afterward.

Since using aptitude in that way wasn’t particularly efficient, we decided to use directly apt-get to stick with our standard server update process. We still wanted to keep our test-production synced updated process cause software updates released between the test and the production server update are untested in the customer specific environment. Thus we needed to find a method to filter out the unneeded packages for the production server update.

In order to do so we have developed a shell script that automates the process and maintain the package version in test and production in sync during OS updates. I’ll explain the processes involved used on both the test and the production servers.

Test Server Update Process:

Update the repository index
Get the list of installed packages and version before update
Complete the dist-upgrade
Get the list of installed packages and version after update
Compare the packages before and after the update and generate a diff output which has the information about the packages installed, updated and removed

Example:

root@selva_test:~# ./auto-tested-server-update.sh -t test

    ##########################################################
    ############ Auto Tested Server Update Script ############
    ##########################################################

Packages index list and server update will be completed here...

Files were generated in default directory. after_* and before_* contains the installed packages list before and after server update. diff_* contains the changes of the packages in the system.

root@selva_test:~# ls tmp/
after_update_selva_test.2015-09-02.txt  before_update_selva_test.2015-09-02.txt  diff_server_update_selva_test.2015-09-02.txt

# head -5 tmp/before_update_selva_test.2015-09-02.txt 
accountsservice=0.6.15-2ubuntu9.6
acl=2.2.51-5ubuntu1
acpi-support=0.140.1
acpid=1:2.0.10-1ubuntu3
activity-log-manager-common=0.9.4-0ubuntu3.2

# head -5 tmp/after_update_selva_test.2015-09-02.txt 
accountsservice=0.6.15-2ubuntu9.6
acl=2.2.51-5ubuntu1
acpi-support=0.140.1
acpid=1:2.0.10-1ubuntu3
activity-log-manager-common=0.9.4-0ubuntu3.2

# cat tmp/diff_server_update_selva_test.2015-09-02.txt 
             > git=1:1.7.9.5-1ubuntu0.1
             > git-man=1:1.7.9.5-1ubuntu0.1
metacity=1:2.34.1-1ubuntu11          <
mysql-client=5.43-0ubuntu0.12.04.1         | mysql-client=5.5.43-0ubuntu0.12.04.1
ubuntu-desktop=1.267.1           <
unity-2d=5.14.0-0ubuntu1          <
             > wget=1.13.4-2ubuntu1

Production Server Update Process:

Get the diff output file (which has information about packages installed, updated and removed packages) from the test server
Collect the packages which will be changed on server update
Process the packages to categorize under install, upgrade and remove
Filter the packages from test server installed, upgraded and removed packages
Perform the install/upgrade/remove actions based on the filtered tested packages

Example:

root@selva_prod:~# ./auto-tested-server-update.sh -t prod -S selva_test -U root

    ##########################################################
    ############ Auto Tested Server Update Script ############
    ##########################################################

*** Production Server Update Process ***

      remote_user: root
      remote_server: selva_prod
      remote_path: /root/tmp
    
1./root/tmp/diff_server_update_selva_test.2015-09-02.txt
Enter a number to choose file: 1
INFO: You have selected the file: diff_server_update_selva_test.2015-09-02.txt
INFO: scp root@selva_test:/root/tmp/diff_server_update_selva_test.2015-09-02.txt /root/tmp
diff_server_update_selva_test.2015-09-02.txt                       100%  763     0.8KB/s   00:00  
  
INFO: Fetching the list of packages need to be changed...!

*********************** Installation *************************
INFO: No packages to install
**************************************************************

************************* Upgrade ****************************
apt-get install mysql-client=5.5.43-0ubuntu0.12.04.1
Reading package lists... Done
Building dependency tree
etc...
**************************************************************

************************** Removal ***************************
apt-get remove zenity=3.4.0-0ubuntu4 zenity-common=3.4.0-0ubuntu4
Reading package lists... Done
Building dependency tree
etc...
**************************************************************

As showed in the sample output above, the script will then run through different phases in which it will install/upgrade/remove all the needed packages in the target production system.

In this way we ensure that all packages tested in the test environment will be replayed on the production server to minimize the chances of introducing bugs and maximize application stability during production system updates.

The Git repository below contains the script and a few additional information to use it.

https://github.com/selvait90/auto-tested-server-update

Puppet, Salt, and DevOps (a review of the MountainWest DevOps conference)

2014-03-27T00:00:00+00:00

Last week I attended the MountainWest DevOps conference held in Salt Lake City, Utah. This was a one day conference with a good set of presenters and lightning talks. There were several interesting topics presented, but I’ll only review a few I wanted to highlight.

I Serve No Master!

Aaron Gibson of Adaptive Computing discussed a very common problem with Puppet (and other configuration management systems): they work well in the scenario they were designed for but what about when the situation isn’t typical? Aaron had a situation where developers and QA engineers could instantiate systems themselves via OpenStack, however the process for installing their company’s software stack on those VMs was inconsistent, mostly manual, and took many hours. One of the pain points he shared, which I related to, was dealing with registering a puppet node with a puppet master—the sometimes painful back and forth of certificate issuing and signing.

His solution was to remove the puppet master completely from the process. Instead he created a bash wrapper script to execute a workflow around what was needed, still using puppet manifests on each system but run locally. This wrapper tool, called “Builder”, relies on property files to customize the config and allow the script to manage different needs. This new script allowed them to keep using puppet to manage these self-serve OpenStack servers gaining the benefits of consistency and removing manual setup steps, providing the ability to automate installs with Jenkins or other tools. But it freed them from having to use a puppet master for nodes that were disposable. It also helped to reduce software install time from 12 hours down to a 11 minutes.

His Builder tool is still an internal only tool for his company, but he discussed some next steps he would like to add, including better reporting and auditing of executions. I pinged him after the conference on twitter and mentioned that Rundeck might be a good fit to fill that gap. I used Rundeck for 2 years at my last job, integrating nicely with other automation tools and providing reporting and auditing as well as access control of arbitrary jobs.

Automating cloud factories and the internet assembly line with SaltStack

Tom Hatch of Salt Stack spoke about Salt as an automation and remote execution platform. I’ve done quite a bit of work with Salt recently with a client and so I was pretty familiar with Salt. But one thing that he mentioned I didn’t know was that Salt was originally designed as a Cloud management tool, not necessarily a configuration management tool. However in the course of time configuration management became a higher priority for the Salt dev team to focus on. Tom mentioned that recently they have been working on Cloud management tools again—providing integration with Rackspace, AWS, xen, and more. I’ll have to dig more into these tools and give them a try.

How I Learned to Stop Worrying and Love DevOps

Bridget Kromhout of 8thBridge (since aquired by Fluid) spoke on the culture of DevOps and her journey from a corporate, strictly siloed environment to a small start-up that embraced DevOps. One of the first things she brought up that was different was the focus and approach to goals of each organization. In an organization where Ops teams are strictly separate from Developers, they often butt heads and have a limited vision of priorities. Each focuses on the goals of their own team or department, and have little understanding of the goals of the other departments. This leads to an adversarial relationship and culture of not caring much about different teams or departments.

In contrast, the organization that embraces DevOps as a culture will see to it that Ops and Devs work together on whatever solution best reaches the goals of the whole organization. In doing so, barriers will have to be questioned. Any “special snowflake” servers/applications/etc. that only one person knows and can touch can’t exist in this culture. Instead, any unique customizations need to be minimized through automation, documentation (sharing knowledge), and reporting/monitoring. This doesn’t mean root access for all—but it means reducing barriers as much as possible. Good habits from the Ops world to keep include: monitoring, robustness, security, scaling, and alerting.

The main pillars of DevOps are: culture, automation, measurement, and sharing. Culture is important and can be supportive or rejecting of the other pillars. Without a culture in the organization that supports DevOps, it will fizzle back into siloed “us vs. them” enmity.

Thanks to all the presenters and those that put on the conference. It was a great experience and I am glad I attended.

Spot On Cost Effective Performance Testing

2014-01-06T00:00:00+00:00

AWS is, in my humble opinion, pricey. However, they provide a nice alternative to the on-demand EC2 instances most people are familiar with: Spot instances. In essence, spot instances allow you to bid on otherwise compute idle time. Recent changes to the web console seem to highlight spot instances a bit more than they used to, but I still don’t see them mentioned often.

The advantage is you get the same selection of instance sizes, and they perform the same as a normal on-demand instance for (usually) a fraction of the price. The downside is that they may disappear at a moment’s notice if there’s not enough spare capacity when someone else spins up a normal on-demand instance, or simply outbids you. It certainly happened to us on occasion, but not as frequently as I originally expected. They also take a couple minutes to evaluate the bid price when you put in a request, which can be a bit of a surprise if you’re used to the almost-instantaneous on-demand instance provision time.

We made extensive use of spot instances in some of the software cluster testing we recently did. For our purposes those caveats were no big deal. If we got outbid, the test could always be restarted in a different Availability Zone with a little more capacity, or we just waited until the demand went down.

At the height of our testing we were spinning up 300 m1.xlarge instances at once. Even when getting the best price for those spot instances, the cost of running a cluster that large adds up quickly! Automation was very important. Our test scripts took hold of the entire process, from spinning up the needed instances, kicking off the test procedure and keeping an eye on it, retrieving the results (and all the server metrics, too) once done, then destroying the instances at the end.

Here’s how we did it:

In terms of high level key components, first, that test driver script was home-grown and fairly specific to the task. Something like Chef could have been taught to spin up spot instances, but those types of configuration management tools are better at keeping systems up and running. We needed the ability to run a task, and immediately shut down the instances when done. That script was written in Python, and leans on the boto library to control the instances.

Second, a persistent ‘head node’ was kept up and running as a normal instance. This ran a Postgres instance and provided a centralized place for the worker nodes to report back to. Why Postgres? I needed a simple way to number and count the nodes in a way immune to race conditions, and sequences were what came to mind. It also gave us a place to collect the test results and system metrics, and compress down before transferring out from AWS.

Third, customized AMI’s were used. Why script the installation of ssh keys, Java, YCSB or Cassandra or whatever, system configuration like our hyper 10-second interval sysstat, application parameters, etc, onto each of those 300 stock instances? Do it once on a tiny micro instance, get it how you want it, and snapshot the thing into an AMI. Everything’s ready to go from the start.

There, those are the puzzle pieces. Now how does it all fit together?

When kicking off a test we give the script a name, a test type, a data node count, and maybe a couple other basic parameters if needed. The script performs the calculations for dataset size, number of client nodes needed to drive those data nodes, etc. Once it has all that figured out the script creates two tables in Postgres, one for data nodes and one for client nodes, and then fires off two batch requests for spot instances. We give them a launch group name to make sure they’re all started in the same AZ, our customized AMI, a bit of userdata, and a bid price:

max_price = '0.10'
instance_type = 'm1.xlarge'
#(snip)
ec2.request_spot_instances(
    max_price, ami, instance_type=instance_type, count=count,
    launch_group=test_name, availability_zone_group=test_name,
    security_groups=['epstatic'],
    user_data=user_data,
    block_device_map=bdmap
)

Okay, at this point I’ll admit the AMI’s weren’t quite that simple, as there’s still some configuration that needs to happen on instance start-up. Luckily AWS gives us a handy way to do that directly from the API. When making its request for a bunch of spot instances, our script sets a block of userdata in the call. When userdata is formulated as text that appears to be a script—starting with a shebang, like #!/bin/bash—that script is executed on first boot. (If you have cloud-init in your AMI’s, to be specific, but that’s a separate conversation.) We leaned on that to relay test name and identifier, test parameters, and anything else our driver script needed to communicate to the instances at start. That thus became the glue that tied the instances back to the script execution. It also let us run multiple tests in parallel.

You may have also noticed the call explicitly specifies the block device map. This overrides any default mapping that may (or may not) be built into the selected AMI. We typically spun up micro instances when making changes to the images, and as those don’t have any instance storage available we couldn’t preconfigure that in the AMI. Setting it manually looks something like:

from boto.ec2.blockdevicemapping import BlockDeviceMapping, BlockDeviceType
bdmap = BlockDeviceMapping()
sdb = BlockDeviceType()
sdb.ephemeral_name = 'ephemeral0'
bdmap['/dev/sdb'] = sdb
sdc = BlockDeviceType()
sdc.ephemeral_name = 'ephemeral1'
bdmap['/dev/sdc'] = sdc
sdd = BlockDeviceType()
sdd.ephemeral_name = 'ephemeral2'
bdmap['/dev/sdd'] = sdd
sde = BlockDeviceType()
sde.ephemeral_name = 'ephemeral3'
bdmap['/dev/sde'] = sde

Then, we wait. The process AWS goes through to evalutate, provision, and boot takes a number of minutes. The script actually goes through a couple of stages at this point. Initially we only watched the tables in the Postgres database, and once the right number of instances reported in, the test was allowed to continue. But we soon learned that not all EC2 instances start as they should. Now the script gets the expected instance IDs, and tells us which ones haven’t reported in. If a few minutes pass, and one or two still aren’t reporting in (more on that in a bit) we know exactly which instances are having problems, and can fire up replacements.

An example output from the test script log, if i-a2c4bfd1 doesn’t show up soon and we can’t connect to it ourselves, we can be confident it’s never going to check in:

2014-01-02 05:01:46 Requesting node allocation from AWS...
2014-01-02 05:02:50 Still waiting on start-up of 300 nodes...
2014-01-02 05:03:51 Still waiting on start-up of 5 nodes...
2014-01-02 05:04:52 Checking that all nodes have reported in...
2014-01-02 05:05:02 I see 294/300 data servers reporting...
2014-01-02 05:05:02 Missing Instances: i-e833499b,i-d63349a5,i-d43349a7,i-c63349b5,i-a2c4bfd1,i-d03349a3
2014-01-02 05:05:12 I see 294/300 data servers reporting...
2014-01-02 05:05:12 Missing Instances: i-e833499b,i-d63349a5,i-d43349a7,i-c63349b5,i-a2c4bfd1,i-d03349a3
2014-01-02 05:05:22 I see 296/300 data servers reporting...
2014-01-02 05:05:22 Missing Instances: i-e833499b,i-c63349b5,i-a2c4bfd1,i-d63349a5
2014-01-02 05:05:32 I see 298/300 data servers reporting...
2014-01-02 05:05:32 Missing Instances: i-a2c4bfd1,i-e833499b
2014-01-02 05:05:42 I see 298/300 data servers reporting...
2014-01-02 05:05:42 Missing Instances: i-a2c4bfd1,i-e833499b
2014-01-02 05:05:52 I see 299/300 data servers reporting...
2014-01-02 05:05:52 Missing Instances: i-a2c4bfd1
2014-01-02 05:06:02 I see 299/300 data servers reporting...
2014-01-02 05:06:02 Missing Instances: i-a2c4bfd1
2014-01-02 05:06:12 I see 299/300 data servers reporting...
2014-01-02 05:06:12 Missing Instances: i-a2c4bfd1
2014-01-02 05:06:22 I see 299/300 data servers reporting...
2014-01-02 05:06:22 Missing Instances: i-a2c4bfd1
2014-01-02 05:06:32 I see 299/300 data servers reporting...
2014-01-02 05:06:32 Missing Instances: i-a2c4bfd1

Meanwhile on the AWS side, as each instance starts up that userdata mini-script writes out its configuration to various files. The instance then kicks off a phone home script, which connects back to the Postgres instance on the head node, adds its own ID, IP address, and hostname, and receives back its node number. (Hurray INSERT … RETURNING!) It also discovers any local instance storage it has, and configures that automatically. The node is then configured for its application role, which may depend on what it’s discovered so far. For example, nodes 2-n the Cassandra cluster will look up the IP address for node 1, and use that for its gossip host, as well as use their node numbers for the ring position calculation. Voila, hands-free cluster creation for Cassandra, MongoDB, or whatever we need.

Back on the script side, once everything’s reported in and running as expected, a sanity check is run on the nodes. For example with Cassandra it checks that the ring reports the correct number of data nodes, or similarly for MongoDB that the correct number of shard servers are present. If something’s wrong, the human that kicked off the test (who hopefully hasn’t run off to dinner expecting that all is well at this point) is given the opportunity to correct the problem. Otherwise, we continue with the tests, and the client nodes are all instructed to begin their work at the same time, beginning with the initial data load phase.

Coordinated parallel execution isn’t easy. Spin off threads within the Python script and wait until each returns? Set up asynchronous connections to each, and poll to see when each is done? Nah, pipe the node IP address list using the subprocess module to:

xargs -P <em>(node count)</em> -n 1 -I {} ssh root@{} <em>(command)</em>

It almost feels like cheating. Each step is distributed to all the client nodes at once, and doesn’t return until all of the nodes complete.

Between each step, we perform a little sanity check, and push out a sysstat comment. Not strictly necessary, but if we’re looking through a server’s metrics it makes it easy to see which phase/test we’re looking at, rather than try to refer back to timestamps.

run_across_nodes(data_nodes+client_nodes, "/usr/lib/sysstat/sadc -C \\'Workload {0} finished.\\' -".format(workload))

When the tests are all done, it’s just a matter of collecting the test results (the output from the load generators) and the metrics. The files are simply scp’d down from all the nodes. The script then issues terminate() commands to AWS for each of the instances it’s used, and considers itself done.

Fun AWS facts we learned along the way:

Roughly 1% of the instances we spun up were duds. I didn’t record any hard numbers, but we routinely had instances that never made it through the boot process to report in, or weren’t at all accessible over the network. Occasionally it seemed like shortly after those were terminated, a subsequent run would be more likely to get a dud instance. Presumably I was just landing back on the same faulty hardware. I eventually learned to leave the dead ones running long enough to kick off the tests I wanted, then terminate them once everything else was running smoothly.

On rare occasion, instances were left running after the script completed. I never got around to figuring out if it was a script bug or if AWS didn’t act on a .terminate() command, but I soon learned to keep an eye on the running instance list to make sure everything was shut down when all the test runs were done for the day.

Setting a server role in Salt (comparing Puppet and Salt)

2013-12-23T00:00:00+00:00

There are many ways to solve a given problem, and this is no truer than with configuration management. Salt (https://saltstack.com) is a fairly new tool in the configuration management arena joining the ranks of Puppet, Chef, and others. It has quickly gained and continues to grow in popularity, boasting its scalable architecture and speed. And so with multiple tools and multiple ways to use each tool, it can get a little tricky to know how best to solve your problem.

Recently I’ve been working with a client to convert their configuration management from Puppet to Salt. This involved reviewing their Puppet configs and designs and more-or-less mapping them to the equivalent for Salt. Most features do convert pretty easily. However, we did run into something that didn’t at first/assigning a role to a server.

We wanted to preserve the “feeling” of the configs where possible. In Puppet they had developed and used a convention for using some custom variables in their configs to assign an “environment” and a “role” for each server. These variables were assigned in the node and role manifests. But in Salt we struggled to find a similar way to do that, but here is what we learned.

In Puppet, once a server’s “role” and “environment” variables were set, then they could be used in other manifest files to select the proper source for a given config file like so:

    file    {
        "/etc/rsyslog.conf":
            source  =>
            [
                "puppet:///rsyslog/rsyslog.conf.$hostname",
                "puppet:///rsyslog/rsyslog.conf.$system_environment-$system_role",
                "puppet:///rsyslog/rsyslog.conf.$system_role",
                "puppet:///rsyslog/rsyslog.conf.$system_environment",
                "puppet:///rsyslog/rsyslog.conf"
            ],
            ensure  => present,
            owner   => "root",
            group   => "root",
            mode    => "644"
    }

Puppet will search the list of source files in order and use the first one that exists. For example, if $hostname = ‘myniftyhostname’ and $system_environment = ‘qa’ and $system_role = ‘sessiondb’, then it will use rsyslog.conf.myniftyhostname if it exists on the Puppet master, or if not then use rsyslog.conf.qa-sessiondb if it exists, or if not then rsyslog.conf.sessiondb if it exists, or if not then rsyslog.conf.qa if it exists, or if not then rsyslog.conf.

In Salt, environment is built into the top.sls file, where you match your servers to their respective state file(s), and can be used within state files as {{ env }}. Salt also allows for multiple sources for a managed file to be listed in order and it will use the first one that exists in the same way as Puppet. We were nearly there; however, setting the server role variable was not as straight forward in Salt.

We first looked at using Jinja variables (which is the default templating system for Salt), but soon found that setting a Jinja variable in one state file does not carry over to another state file. Jinja variables remain only in the scope of the file they were created in, at least in Salt.

The next thing we looked at was using Pillar, which is a way to set custom variables from the Salt master to given hosts (or minions). Pillar uses a structure very similar to Salt’s top.sls structure- matching a host with its state files. But since the hostnames for this client vary considerably and don’t lend themselves to pattern matching easily, this would be cumbersome to manage both the state top.sls file and the Pillar top.sls file and keep them in sync. It would require basically duplicating the list of hosts in two files, which could get out of sync over time.

We asked the salt community on #salt on Freenode.net how they might solve this problem, and the recommended answer was to set a custom grain. Grains are a set of properties for a given host, collected from the host itself- such as, hostname, cpu architecture, cpu model, kernel version, total ram, etc. There are multiple ways to set custom grains, but after some digging we found how to set them from within a state file. This meant that we could do something like this in a “role” state file:

# sessiondb role
# {{ salt['grains.setval']('server_role','sessiondb') }}

include:
  - common
  - postgres

And then within the common/init.sls and postgres/init.sls state files we could use that server_role custom grain in selecting the right source file, like this:

/etc/rsyslog.conf:
  file.managed:
    - source:
      - salt://rsyslog/files/rsyslog.conf.{{ grains['host'] }}
      - salt://rsyslog/files/rsyslog.conf.{{ env }}-{{ grains['server_role'] }}
      - salt://rsyslog/files/rsyslog.conf.{{ grains['server_role'] }}
      - salt://rsyslog/files/rsyslog.conf.{{ env }}
      - salt://rsyslog/files/rsyslog.conf
    - mode: 644
    - user: root
    - group: root

This got us to our desired config structure. But like I said earlier, there are probably many ways to handle this type of problem. This may not even be the best way to handle server roles and environments in Salt, if we were more willing to change the “feeling” of the configs. But given the requirements and feedback form our client, this worked fine.

Controlling interactive programs with pexpect-u

2013-10-23T00:00:00+00:00

A client I am working with requires that various machines have Ubuntu 10.04.4 installed along with certain software dependencies prior to installation of their own software.

In order to have our client avoid the tedious task of spinning up a new machine for each new client of theirs, I decided to attempt to automate the process (minus the OS installation) in Python.

A couple of the software installations require the user to interact with a console application. For example, this is the Matlab Runtime Environment installer:

and the Passenger installer:

Here I used the Python package pexpect-u which allows you to spawn child applications and control them automatically.

To spawn the Matlab installer I run:

import pexpect
child = pexpect.spawn("sudo ./MCRInstaller.bin -console")

Now we tell pexpect what to expect:

child.expect("Press 1 for Next, 3 to Cancel or 5 to Redisplay \[1\]")

And we send a command with:

child.sendline("1")

The package can be found here and the source code includes many more examples, one of which might be of use for this very client: hive.py

This client has various installations of a Rails application and each sends requests to the same location on a server. Because this location changes periodically, each of the Rails application installations need a configuration variable updated. I think the following should do the trick:

python hive.py user1:pass2@host1 user2:pass2@host2 ... userN:passN@hostN
grep -rl 'old_location_path_string' /path/to/config/file | xargs sed -i 's/old_location_path_string/new_location_path_string/'

Let’s see it in action. I’ve set up two Ubuntu virtual machines and will be changing a string in a file that exists on both machines in the same location. First I login:

Let’s look at the contents of the files:

Now I’ll replace the string “Hello” with “Goodbye”:

Because the user of each machine has permissions over the test.conf file, I was able to modify it without sudo. If you’ll be needing to use sudo to make any changes, take a look at the :sync example here, however, note that if your sudo passwords differ on at least one of the remote machines this will not work.

Interchange Form Testing with WWW::Mechanize

2013-09-17T00:00:00+00:00

Recently, I encountered a testing challenge that involved making detailed comparisons between the old and new versions of over 200 separate form-containing HTML (Interchange) pages.

Because the original developers chose to construct 200+ slightly-different pages, rather than a table-driven Interchange flypage (curses be on them forever and ever, amen), an upgrade to change how the pages prepared the shopping cart meant making over 200 similar edits. (Emacs macros, yay!) Then I had to figure out how to verify that each of the 200 new versions did something at least close to what the 200 old versions did.

Fortunately, I had easy ways to identify which pages needed testing, construct URLs to the new and old pages, and even a way to “script” how to operate on the page-under-test. And I had WWW::Mechanize, which has saved my aft end more than once.

WWW::Mechanize is a pretty mature (originally 2008) “browser-like” system for fetching and acting on web pages. You can accept and store cookies, find and follow links, handle redirection, forms, you name it—but not Javascript. Sorry, but there are other tools in the box that can help you if you are working with more interactive pages.

In my case, lack of JS wasn’t an issue. I just needed a way to fetch a page, tweak a form element or two, and submit the page’s POST for server processing. Then if I could capture the server-side state of my session, I’d be golden.

 1 #!/usr/local/bin/perl
 2 use strict;
 3 use warnings;
 4 use WWW::Mechanize;
 5 use Test::More;
 6 
 7 our $BASE = 'http://www.example.com/';
 8 
 9 my %common = (
10     agent => 'compare-pages',
11     autocheck => 1,
12     cookie_jar => { },
13     quiet => 1,
14     redirect_ok => 1,
15     timeout => 15,
16 );
17 my $old = WWW::Mechanize->new(
18     %common,
19 );
20 my $new = WWW::Mechanize->new(
21     %common,
22 );
23 
24 for my $page (@ARGV ? @ARGV : <>) {
25     print $page;
26     chomp $page;
27     $new->get( $BASE . 'newstuff/' . $page . '?mv_pc=RESET');
28     my $new_form = $new->form_with_fields('last_product');
29     $new->submit();
30     $new->form_with_fields('mv_todo');
31     $new->submit();
32     $new->get( $BASE . 'show-the-dump' );
33     $new->content =~ m/#+\s+SESSION\s+#+\n(.+)\n#+\s+END SESSION\s+#+/s;
34     my $new_session = eval $1;
35     delete $new_session->{carts}{main}[0]{$_} for qw(some fields);
36 
37     $old->get( $BASE . $page . '?mv_pc=RESET' );
38     my $old_form = $old->form_with_fields('order_item', 'mv_order_deliverydate');
39     $old->select('mv_order_deliverydate', {n => 2});
40     $old->submit();
41     $old->get( $BASE . 'show-the-dump' );
42     $old->content =~ m/#+\s+SESSION\s+#+\n(.+)\n#+\s+END SESSION\s+#+/s;
43     my $old_session = eval $1;
44     delete $old_session->{carts}{main}[0]{$_} for qw(other fields);
45 
46     is_deeply($old_session->{carts}{main}, $new_session->{carts}{main}, "$page : carts match") or exit;
47 }
48 
49 done_testing;
50 exit;

2-5: Very few external modules are needed for this. WWW::Mechanize is quite complete (but it has a slew of prerequisites). Test::More is used just to make our comparisons easier.
7: This will be the URL base for our requests.
9-22: we set up two separate user agents so that they don’t share cookies, history, or any state information that would confuse our comparisons.
27, 37: retrieving the pages under test. Note that in my case, “newstuff/” distinguished the new version from the original.
28, 38: specifying which form on the retrieved page is to be considered the “current” one. Note that I’m not using the returned value here (although it came in handy during debugging). “form_with_fields” lets you pick a form based on one or more fields named within it. In the event that there’s more than one, you get the first (and Mechanize complains with a warning—but we’ve turned that off via the “quiet” option, above).
32, 41: In the interests of security, I’ve not shown the actual page we use to dump the session internals. However, for Interchange users it’s just a page with a “[dump]” tag. You might write something that produces plain text, or CSV, or JSON. In my case, the session dump contains Data::Dumper-style output that I can feed into Perl’s “eval” function.
35, 44: The two data structures resulting from the “old” and “new” pages aren’t exactly alike, so I remove the bits I don’t care about.
46: And Test::More to the rescue, saving me from having to re-invent the code that will compare a possibly-complex data structure down to the scalar members. I have it exit after a failure, since in my case one error usually meant a whole family of corrections that needed to be applied to several related pages.

And that’s all! My testing now consists of:

$ grep "some pattern that identifies my 200" *.html | perl compare_pages.pl

I also had to adjust my Interchange configuration so my script would be accepted as a “robot”:

RobotUA compare-pages

As a result of this testing, I identified at least a few pages where the “old” and “new” forms did not result in the same cart configuration, so I was able to fix that before it went live and caused untold confusion.

I hope this excursion into page-testing has proven interesting.

Deploying password files with Chef

2013-04-03T00:00:00+00:00

Today I worked on a Chef recipe that needed to deploy an rsync password file from an encrypted data bag. Obtaining the password from the data bag in the recipe is well documented, but I knew that great care should be taken when writing the file. There are a plethora of ways to write strings to files in Chef, but many have potential vulnerabilities when dealing with secrets. Caveats:

The details of execute resources may be gleaned from globally-visible areas of proc.
The contents of a template may be echoed to the chef client.log or stored in cache, stacktrace or backup areas.
Some chef resources which write to files can be made to dump the diff or contents to stdout when run with verbosity.

With tremendous help from Jay Feldblum in freenode#chef, we came up with a safe, optimized solution to deploy the password from a series of ruby blocks:

pw_path = Pathname("/path/to/pwd/file")
pw_path_uid = 0
pw_path_gid = 0
pw = Chef::EncryptedDataBagItem.load("bag", "item")['password']

ruby_block "#{pw_path}-touch" do
  block   { FileUtils.touch pw_path } # so that we can chown & chmod it before writing the pw to it
  not_if  { pw_path.file? }
end

ruby_block "#{pw_path}-chown" do
  block   { FileUtils.chown pw_path_uid, pw_path_gid, pw_path }
  not_if  { s = pw_path.stat ; s.uid == pw_path_uid && s.gid == pw_path_gid }
end

ruby_block "#{pw_path}-chmod" do
  block   { FileUtils.chmod 0600, pw_path }
  not_if  { s = pw_path.stat ; "%o" % s.mode == "100600" }
end

ruby_block "#{pw_path}-content" do
  block   { pw_path.open("w") {|f| f.write pw} }
  not_if  { pw_path.read == pw } # NOTE: a secure compare method might make this even better
end

Puppet custom facts and Ruby plugins to set a homedir

2012-03-21T00:00:00+00:00

Photo of Swedish Chef by A. M. Kuchling

Puppet is an indispensable tool for system admins, but it can be tricky at times to make it work the way you need it to. One such problem one of our clients had recently was that they needed to track a file inside a user’s home directory via Puppet (a common event). However, for various reasons the user’s home directory was not the same on all the servers! As Puppet uses hard-coded paths to track files, this required the use of a custom Puppet “fact” and a helper Ruby script plugin to solve.

Normally, you can use Puppet to track a file by simply adding a file resource section to a puppet manifest. For example, we might control such a file inside a manifest named “foobar” by writing the file puppet/modules/foobar/init.pp as so:

class foobar {

  user {
    "postgres":
      ensure     => present,
      managehome => true;
  }

  file {
    "/home/postgres/.psqlrc":
      ensure      => present,
      owner       => postgres,
      group       => postgres,
      mode        => 644,
      source      => [
                      "puppet:///foobar/$pg_environment/psqlrc",
                      "puppet:///foobar/psqlrc",
                      ],
      require     => User["postgres"];
  }
}

This is a bare-bones example (the actual username and file were different), but gets the idea across. While we want to ensure that the postgres user has the correct .psqlrc file, we also need to make sure that the postgres user itself exists. Hence the user section at the top of the script. This will ask puppet to create that user if it does not already exist. We also added the “managehome” parameter, to ensure that the new user also has a home directory. If this parameter is false (or missing), puppet runs a plain useradd command (or its equivalent); if the parameter is true, it adds the -m or –create-home argument, which is what we need, as we need to monitor a file in that directory.

As there is no point in trying to manage the .psqlrc file before the user is created, we make the User creation check a pre-requisite via the “require” parameter; basically, this helps puppet determine the order in which it runs things (using a directed acyclic graph, a feature that should be familiar to git fans).

The file resource in this manifest, named “/home/postgres/.psqlrc”, ensures that the file exists and matches the version stored in puppet. Most of the parameters are straightforward, but the source is not quite as intuitive. Here, rather than giving a simple string as the value for the parameter source, we give it an array of strings. Puppet will walk through the list until it finds the first one that exists, and use that for the actual file to use as /home/postgres/.psqlrc on each box using this manifest. This allows us to have different versions of the .psqlrc file for different arbitrary classes of boxes, but without having to write a separate manifest for each one. Instead, they all use the same manifest and simply change the $pg_environment variable, usually at the puppet “role” level.

The syntax puppet:///foobar/ is a way of telling puppet that the file is underneath the main puppet directory, inside the “foobar” directory, and in a subdirectory called “files”. The level above “files” is where one would create different subdirectories based on $pg_environment, so your module might look like this:

modules/
   └──foobar/
         ├─manifests/
         │     └─init.pp
         └──files/
              ├─psqlrc
              ├─production/
              │     └─psqlrc
              └──development/
                    └─psqlrc

In the above, we have three versions of the psqlrc file stored: one for boxes with a $pg_environment of “production”, one for a $pg_environment of “development”, and a default one for boxes that do not have $pg_environment set (or whose $pg_environment string does not have a matching subdirectory).

All well and good, but the problem comes when the user in question already exists on more than one server, and has a different home directory depending on the server! We can no longer say “/home/postgres/.psqlrc”, because on some boxes, what we really want is “/var/lib/pgsql/.psqlrc”. There are a couple of wrinkles that prevent us from simply saying something like “$HOME/.psqlrc”.

The first wrinkle is that Puppet runs as root, and what we need here is the $HOME of the postgres user, not root. The second wrinkle is that, even if we were to come up with a clever way to figure it out (say, by parsing /etc/passwd with an exec resource), we cannot add run-time code to our manifest and have it get stored into a variable. The reason is that the first thing Puppet does on a client is compiles all the manifests into a static catalog, that is then applied. Which introduces another wrinkle: even if we were to somehow know this information beforehand, what about the case where the user does not exist? We can ask puppet to create the user, but it is way, way too late in the game at that point to apply the location of the new home directory into our manifest.

We stated that the first thing puppet does is compile a catalog, but that’s not strictly true: it actually walks through the manifests and does a few other things as well, including executing any plugins. We can use this fact to create a custom fact for each client server—this new fact will contain the location of the home directory for the postgres user on that server.

There’s a few steps to get it all working. The first thing to know is that puppet provides a number of “facts” that get stored as simple key/value pairs, and these are available as variables you can use inside of your manifests. For example, you can put PostGIS on any of your hosts that contain the string “gis” somewhere in their hostname by saying:

  if $::hostname =~ /gis/ {
    package {
      'postgis':
        ensure => latest;
    }
  }

This list of facts can be expanded by the use of “custom facts”, which basically means we add our own variables that we can access in our manifests. In this particular case, we are going to create a variable named “$postgres_homedir”, which we can then utilize in our manifest.

A custom fact is created by a Ruby function: this function should be in its own file, located in the “lib/facter” directory of the relevant module. So in our case, we will create a small ruby file named “postgres_homedir.rb” and stick it here:

modules/
   └──foobar/
         ├─lib/
         │  └─facter/
         │       └─postgres_homedir.rb
         ├─manifests/
         │     └─init.pp
         └──files/
              ├─psqlrc
              ├─production/
              │     └─psqlrc
              └──development/
                    └─psqlrc

The function itself follows a fairly standard format: The only unique parts are the actual system calls and the name of the variable:

# postgres_homedir.rb

Facter.add("postgres_homedir") do
  setcode do
    system('useradd -m postgres 2>/dev/null')
    Facter::Util::Resolution.exec('/bin/grep "^postgres:" /etc/passwd | cut -d: -f6').chomp
  end
end

Since we’ve already shown how having Puppet create the user happens way too late in the game, and because we know that the foobar module always needs that user to exist, we’ve moved the user creation to a simple system call in this Ruby script. The -m makes sure that a home directory is created, and then the next line extracts the home directory and stores it in the global puppet variable $postgres_homedir. The ‘useradd’ line feels the least clean of all of this, and alternatives are welcome, but having the system do a ‘useradd’ and returning a (silenced) error each time the puppet client is run seems a fairly small price to pay for having this all work (and shorter than checking for existence, doing a conditional, etc).

Now that we have a way of knowing what the home directory of the postgres user will be before the manifest is compiled into a catalog, we can rewrite puppet/foobar/manifests/init.pp like so:

class foobar {

  file {

    postgres_psqlrc:
      path        => "${::postgres_homedir}/.psqlrc",
      ensure      => present,
      owner       => postgres,
      group       => postgres,
      mode        => 644,
      source      => [
                      "puppet:///postgres/$pg_environment/psqlrc",
                      "puppet:///postgres/psqlrc",
                      ];
  }
}

Voila! We no longer have to worry about the user existing, because we have already done that in the Ruby script. We also no longer have to worry about what the home directory is set to, for we have a handy top-level variable we can use. Note the use of the :: to indicate this is in the root namespace; Puppet variables have a scope and a name, such as $alpha::bravo.

Rather than leave the title of this resource as the “path” (most puppet resources have sane defaults like that), we have explicitly set the path, as having a variable in the resource title is ugly and can make referring to it elsewhere very tricky. We also changed the on-disk copy of .psqlrc to psqlrc: while normally the files are the same, there is no reason to keep it as a “hidden” file inside the puppet repo.

Let’s take a look at this module in action. We’ll manually run puppetd on one of our clients, using the handy –test argument, which expands to –onetime –verbose –ignorecache –no-daemonize –no-usecacheonfailure. Notice how our plugins are retrieved and loaded before the catalog is built, and that our postgres user now has the file in question:

$ puppetd --test
info: Retrieving plugin
notice: /File[/var/lib/puppet/lib/facter/postgres_homedir.rb]/ensure: 
  content changed '{md5}0642408678c90dced5c3e34dc40c3415'
    to '{md5}0642408678c90dced5c3e34dc40c3415'
info: Loading downloaded plugin /var/lib/puppet/lib/facter/postgres_homedir.rb
info: Caching catalog for somehost.example.com
info: Applying configuration version '1332065118'
notice: //foobar/File[postgres_psqlrc]/ensure: content changed 
  '{md5}08731a768885aa295d3f0856748f31d5'
    to '{md5}08731a768885aa295d3f0856748f31d5'
Changes:
            Total: 1
Resources:
          Applied: 1
      Out of sync: 1
        Scheduled: 195
            Total: 179
Time:
 Config retrieval: 1.63
             Exec: 0.00
             File: 6.28
       Filebucket: 0.00
            Group: 0.00
        Mailalias: 0.00
          Package: 0.13
         Schedule: 0.00
          Service: 0.51
             User: 0.01
            Total: 8.56
notice: Finished catalog run in 13.26 seconds

So we were able to solve out original problem via the use of custom facts, a Ruby plugin, and some minor changes to our manifest. While you don’t have to go through all of this effort often, it’s nice that Puppet is flexible enough to allow you do so!

Global Variables in Interchange Jobs

2011-11-28T00:00:00+00:00

Those familiar with writing global code in Interchange are certainly familiar with the number of duplicate references of certain global variables in different namespaces. For example, the Values reference is found in both the main namespace ($::Values) as well as in Vend::Interpolate ($Values usually from within usertags). One can also access the Values reference through the Session reference, which itself can be found in main ($::Session), Vend ($Vend::Session), and Vend::Interpolate ($Session usually from within usertags) namespaces with, e.g., $::Session->{values}. Most times, as long as context allows, any of those access points are interchangeable, and there’s a good mix you see from developers using all of them.

In recent work for a client, I had developed an actionmap that incorporated access to the session for some of its coding—certainly not an uncommon occurrence. When I work in global space, I tend to use the main namespace references since they are available in all contexts within Interchange (or so I thought). The actionmap was constructed, tested, and put into production, where it worked as expected.

After a short period of operation, the client came to us and noted that in their actual operating procedure, the actionmap must process many more data points than we had it operate on in testing, causing it to take much more time. Thus, for their usual workload, they found the process was timing out and Interchange housekeeping reaping the process.

After a brief discussion, we decided the expedient course of action was to convert the work from a browser-initiated actionmap into an Interchange job. The code was easily exposed as a usertag as well, so in very short order we had the same functionality available as a job, where the job was now triggered by the browser access previously running the actionmap.

The change resolved the immediate problem, so now all work was completing, but the client brought a new issue to our attention. The reporting from the job was not as it was supposed to be. None of the code had been modified in the changeover, and the code when run as an actionmap produced the proper reporting.

The problem tracked down eventually to that session access. When the code was run in the context of the job, the Session reference was not copied into the main (or, as it turns out, Vend::Interpolate) namespace. Without the assumed session values in place, it was causing the report to produce invalid output.

To demonstrate, I constructed a simple usertag to dump the reference addresses of the 5 mentioned global variables:

UserTag  ic-globals  Routine <<EOR
sub {
    return <<EOP;
.     \$Session: $Session
\$Vend::Session: $Vend::Session
    \$::Session: $::Session

       \$Values: $Values
     \$::Values: $::Values
EOP
}
EOR

I then created both a test page and an Interchange job that only called [ic-globals]. Running them both demonstrates the problem quite clearly.

From test page:

      $Session: HASH(0xb0e1898)
$Vend::Session: HASH(0xb0e1898)
    $::Session: HASH(0xb0e1898)

       $Values: HASH(0xb0e1dd8)
     $::Values: HASH(0xb0e1dd8)

Output from job:

      $Session:
$Vend::Session: HASH(0xb221fa0)
    $::Session:

       $Values: HASH(0x926ddd8)
     $::Values: HASH(0x926ddd8)

Interchange jobs provide yet a new context where you must consider your global variable usage. In particular, if you find code executed in the context of a job produces inconsistencies with the same code in other contexts, review your global variable usage and confirm those variables are what you assume they are.

NY Puppet Users Group meeting

2009-02-04T00:00:00+00:00

Several of us from End Point attended the first NY Puppet Users Group meeting last night, and we had a great time. Brian Gupta and Larry Ludwig did an excellent time coordinating, even though the weather and the venue didn’t cooperate. After a little shuffling, we found ourselves at The Perfect Pint, where we had a large table in a fairly quiet room where we could meet each other and talk about Puppet.

The group was split fairly evenly between those who have used Puppet and those who are considering it: everyone there understood the need for solid configuration management (CM), so much of the discussions (at least at my end of the table) centered around

comparisons of different CM systems
challenges in getting Puppet on legacy systems (e.g., that may not even have Ruby)
features people would like to see in Puppet
how to make the business case for Puppet

All in all it was a great time, and I really appreciated Brian and Larry’s leadership, as well as their flexibility. Not only did they move the venue at the last minute to better accommodate the group, but they were super hosts in coordinating the date and time: several of us were only in NY for a short time, but they managed to juggle things around so that as many as possible could attend. That flexibility says a lot for the Puppet community, and we’re looking forward to participating in future meetings.

How to Analyze Application Logs and Extract Actionable Insights

A Note on Commands

1. Debugging Application Errors

1.1 Summarize Frequent Issues

1.2 Categorize Errors by Type

1.3 Find Problematic Scripts

2. Monitoring System Behavior

2.1 Track Problematic IPs

2.2 Spot High-Error Time Periods

2.3 Analyze IP–Referer Pairs

3. Enhancing Security

3.1 Watch Sensitive URLs

3.2 Pinpoint Critical Times

Conclusion

Introduction to Playwright using C#

What is Playwright?

Advantages of Using C# and Playwright

Quartz Scheduler as a Service

Quartz Job Scheduler

Quartz Scheduler as a Service

Read and Manage Job Data

Spring Application

Design

Scheduler

Spring Application Startup

JobFactory

Observer Job

KPIObserverJob

Business Jobs

Running this Solution

Extend by Creating a Management UI

Conclusion

Automating reading the screen and interacting with GUI programs on X Window System

Overview of how the script works

Preparing

Working through an example

How it works in a little more detail

Chaining xdotool actions

Fine-tuning

Learning more

Generate PDF with Chrome, Puppeteer, and Serverless Stack

The tools

1. Serverless Stack

2. Chrome and Puppeteer

Implementing PDF screenshot and download with Serverless Stack, Chrome, and Puppeteer

Prerequisite: Sample receipt pages

Getting started

1. sst.json

2. stacks/index.js

3. stacks/MyStack.js

Deploying the project

Developing with Serverless Stack

Conclusion

Caveats

Further readings

Appium: Automated Mobile Testing

Installation and setup for Linux

General

AVD Android emulator

Appium

Ruby and RSpec

Putting it all together

Recording a test with Inspector

Running a test

What’s next?

Additional tests

Test with other devices

CI/​CD integration

Catching CSS Regressions and Visual Bugs in Continuous Integration

Visual regression testing to the rescue

Integrating Percy

1. Register an account at percy.io

2. Integrate with the Rails project

3. Percy in Feature Specs

Adding Percy into the mix

4. Introduce visual changes and see how percy.io compares

5. What about style regressions?

6. Responsive design and cross-browser testing

Caveats

Conclusion

CI/CD integration