Our Blog

Using tail_n_mail after hours

By Greg Sabino Mullane
October 23, 2017

(Photo of Turtle Island by Edwin Poon)

Someone recently asked me something about tail_n_mail, a program that watches over your log files, scans for certain patterns, and sends out an email if matches are found. It is frequently used to watch over Postgres logs so you can receive an automatic email alert when Bad Things start happening to your database. The questioner wanted to know if it was possible for tail_n_mail to change its behavior based on the time of day—​would it be able to do things differently outside of “business hours”? Although tail_n_mail cannot do so directly, a simple solution is to use alternate configuration files—​which get swapped by cron—​and the INHERIT keyword.

To demonstrate the solution, let’s spin up a Postgres 10 instance, route the logs to syslog, setup tail_n_mail, and then create separate configuration files for different times of the week. First, some setup:

$ initdb --version
initdb (PostgreSQL) 10.0
$ initdb --data-checksums data
$ cat >> data/postgresql.conf << EOT
log_line_prefix=''
log_destination='syslog'
EOT
$ echo 'local0.*  /var/log/postgres.log' | sudo tee -a /etc/rsyslog.conf > /dev/null
$ sudo …

Liquid Galaxy at ASTC 2017

By Ben Witten
October 18, 2017

End Point is pleased to be participating in ASTC 2017, alongside our partners BWC Visual Technology. ASTC, which stands for the Association of Science-Technology Centers, is holding their annual conference at The Tech Museum of Innovation, located in San Jose, CA. We were excited to hear that the conference takes place at The Tech Museum, as we have a Liquid Galaxy set up in the museum!

A 3-screen desktop Liquid Galaxy display will be set up by Liquid Galaxy Engineer Josh Ausborne at Booth 1103. This display will be showcasing content that includes Sketchfab and Unity 3D Models, Cesium content with interactive weather data, 360 panoramic video, Google Earth/Google Streetview content, and engaging presentations about National Parks and National Marine Sanctuaries.

We are very excited to be showcasing our technology with BWC Visual Technology. BWC is a distributor and licensed re-seller of state-of-the-art, interactive exhibit technology for museums and science centers. We have great respect for their team and technologies, and are excited to be showcasing Liquid Galaxy with them.

Liquid Galaxy is currently featured in many science and technology centers around the world. Please …

PKIX path validation failed — Debugging

By Selvakumar Arumugam
October 4, 2017

I recently ran into a case working on an application with a PKIX path validation error on a site that had a valid certificate. I was able to solve the issue using OpenSSL to debug.

Typically, the PKIX path validation error arises due to SSL certificate expiry, but I ran into the same error even when the system was configured with a valid certificate. There are two web applications in our scenario, AppX and AppY. AppX uses AppY’s authentication mechanism to allow the users to login with same user account. AppX sends a POST request using HttpClient with necessary arguments to SSL enabled AppY and allows the user to login based on the response.

HttpClient httpclient = new DefaultHttpClient();
// ...
HttpPost httppost = new HttpPost("https://app2domain.com/sessions");

try {
    resp = httpclient.execute(httppost);
}
catch (Exception e) {
    throw new Exception("Exception: ", e);
}

Error

The AppX was isolated to new server and it started throwing PKIX path validation failed error while sending requests to AppY.

Exception: javax.net.ssl.SSLHandshakeException:
sun.security.validator.ValidatorException: PKIX path validation failed: …

AngularJS is not Angular

By Piotr Hankiewicz
October 3, 2017

Why I’m doing this?

I often see that people use AngularJS and Angular names interchangeably. It’s an obvious mistake for someone familiar with the front-end programming.

If you don’t believe me (why would you?) just go to the Angular team repository or Wikipedia (Angular and AngularJS).

AngularJS (1.X)

AngularJS is a popular open-source JavaScript framework created by Miško Hevery released in 2010 that was subsequently taken over by Google. It’s not the same thing as Angular. The latest stable version of the framework is 1.6.6.

Angular (2+)

Angular is an open-source TypeScript-based framework created by Google from scratch. It’s been totally rewritten and not compatible with AngularJS. The latest stable version of the framework is 4.4.4. They started from version 2 to avoid even more confusion.

Summary

I feel I have made the world better. Go now and correct everyone (it’s such fun)!

Disaster Recovery — Miami to Dallas in one hit

By Ian Neilsen
October 3, 2017

Hurricane Irma came a knocking but didn’t blow Miami away.

Hurricanes are never fun, and neither is knowing that your business servers may be in the direct path of a major outage due to a natural disaster.

Recently we helped a client prepare a disaster recovery environment, 4 days out pending Hurricane Irma approaching Miami—​where it happens their entire server ecosystem sat. In recent months we had been discussing a long-term disaster recovery plan for this client’s infrastructure, but the final details hadn’t been worked out yet, and no work begun on it, when the news of the impending storm started to arrive.

Although the Miami datacenter they are using is highly regarded and well-rated, the client had the foresight to think about an emergency project to replicate their entire server ecosystem out of Miami to somewhere a little safer.

Fire suits on, battle helmets ready, GO! Two team members jumped in and did an initial review of the ecosystem: six Linux servers, one Microsoft SQL Server database with about 50 GB of data, and several little minions. All of this hosted on a KVM virtualization platform. OK, easy, right? Export the VM disks and stand up in a new datacentre on a …

Using PostgreSQL cidr and inet types, operators, and functions

By Phineas Jensen
September 28, 2017

A common problem in database design is how to properly store IP addresses: They are essentially positive 32 or 128 bit integers, but they’re more commonly written as blocks of 8 or 16 bit integers written in decimal and separated by periods (in IPv4) or written in hexadecimal and separated by colons (in IPv6). That may lead many people to store IP addresses as strings, which comes with a host of downsides.

For example, it’s difficult to do subnet comparisons and validation requires complex logic or regular expressions, especially when considering the variety of ways IPv6 addresses can be stored with upper- or lower-case hexadecimal letters, :0000: or :0: for a group of zeros, or :: as several groups of zeros (but only once in an address).

Storing them as integers or a fixed number of bytes forces parsing and a uniform representation, but is otherwise no better.

To solve these problems in PostgreSQL, try using the inet and cidr types, which are designed for storing host and network addresses, respectively, in either IPv4 or IPv6 or both.

In many cases, these types could end up simply being used as glorified integers: they display nicely as IP addresses should be, and support …

Working on production systems

By Greg Sabino Mullane
September 27, 2017

(Not as easy as it looks)
Photo by Flickr user 'edenpictures'

As consultants, the engineers at End Point are often called upon to do work on production systems—​in other words, one or more servers that are vital to our client’s business. These range from doing years of planning to perform a major upgrade for a long-standing client, down to jumping into a brand-new client for an emergency fix. Either way, the work can be challenging, rewarding, and a little bit nerve-wracking.

Regardless of how you end up there, following some basic good practices may reduce the chance of problems coming up, keep you calm during the chaos, and make the process easier for both you and the client.

Preparation

Unless this is a true emergency, doing major work on a production system (e.g. upgrading a database server) should involve a good amount or preparation. By the time the big night arrives (yes, it is always late at night!) you should have gone through this list:

• Do lots and lots of testing. Use systems as close as possible to production, and run through the process until it becomes second nature.
• Know the impact on the business, and the downtime window anticipated by the rest of the company. …

Liquid Galaxy Goes to Chile for IMPAC4

By Bryan Berry
September 26, 2017

Marine Protected Areas: Bringing the people and ocean together

Earlier this month, The Marine Conservation Institute and the Waitt Foundation brought a Liquid Galaxy to Chile to showcase interactive and panoramic ocean content at the Fourth International Marine Protected Areas Congress in La Serena.  This conference was a convergence of marine biologists, ocean agencies, and environmentalists from around the globe to discuss the state of and future development initiatives of Marine Protected Areas worldwide.

 The Marine Conservation Institute is working on a mapping project called MPAtlas that visually catalogs the development of Marine Protected Areas across the globe as well as the Global Ocean Refuge System which pushes for elevated standards for Marine Protected Areas and advocates for a 30% protected marine ecosystem by 2030.

mpatlas.org

We built new content to showcase the GLORES areas in Google Earth as well as data visualizations of the global system of Marine Protected Areas. In addition, we collected any past oceanographic related content we’ve developed for the Liquid Galaxy platform. This included underwater panoramic media from the Catlin Seaview Survey, …