Our Blog

Liquid Galaxy Featured on Reef Builders

By Ben Witten
July 21, 2016

The Liquid Galaxy was recently featured on the front page of Reef Builders, the original and longest running saltwater fish blog and a leading source of aquarium news. Reef Builders writer Nicole Helgason wrote the story, which can be found here.

The Liquid Galaxy is an amazing tool for aquariums (not to mention other kinds of museums, offices, educational institutions, and more) around the world. It is a particularly effective for aquariums due to the underwater imagery shown on the system, as well as End Point’s Content Management System that allows users the opportunity to tell underwater “stories” with supporting images, videos, and content. We have deployed to aquariums and science museums throughout the US, Europe, and East Asia.

The Liquid Galaxy lets visitors explore coral reefs and underwater environments the exact same way they navigate Street View (it’s the same app and data set) with a full set of screens to give a totally immersive experience. While viewing the dazzling immersive display, the user can make use of the Liquid Galaxy touchpad and 3D joystick to look around and navigate through the display.

A video demonstrating how the Liquid Galaxy is utilized in …

Poor Man’s Linux Remote Desktop Using VNC Server

By Ed Huott
July 21, 2016

Even the most hardcore Linux hackers want to run GUI applications sometimes. The problem is how do you do that with Linux when it’s running on a remote server with no access to the console?

Remote Desktop is a built-in feature of Windows Server that lets you access a virtual remote desktop session that’s not tied to a physical console. Many would be tempted to believe that most Linux distributions don’t come with this kind of functionality out of the box. Au contraire! This article explains one method for accessing this feature using only packages included in most stock Linux installations. The examples given here are based on Debian/Ubuntu.

The core ingredient of our recipe is the vnc4server package, which includes Xvnc, the X VNC (Virtual Network Computing) server. A description of what that is comes right from the Xvnc(1) manual page itself:

Xvnc is the X VNC (Virtual Network Computing) server. It is based on a standard X server, but it has a “virtual” screen rather than a physical one. X applications display themselves on it as if it were a normal X display, but they can only be accessed via a VNC viewer - see vncviewer(1).

So Xvnc is really two servers in one. To the …

Book review: “Two Scoops of Django: Best Practices for Django 1.8”

By Phineas Jensen
July 13, 2016

Two Scoops of Django: Best Practices for Django 1.8 is, shockingly, a book about best practices. It’s not a Django library reference, or a book about Django fundamentals, or tips and tricks. It’s a book designed to help web developers, novices and experts alike, avoid common pitfalls in every stage of the web development process, specifically the process of developing with Django.

The book can be used as a reference of best practices and a cover-to-cover guide to best practices. I’ve done both and found it to be enjoyable, accessible, and educational when read cover-to-cover and a valuable reference when setting up a new Django project or doing general Django development. It covers a huge range of material, answering questions like:

• Where should I store secret keys?
• Should I use virtualenv?
• How should I structure my Django project?
• When should I use ‘blank’ and ‘null’ in model fields?
• Should I use Class-Based Views or Function-Based Views?
• How should I structure my URLConfs?
• When should I use Forms?
• Where should templates be stored?
• Should I write custom template tags and filters?
• What package should I use to create a REST API?
• What core components should I replace?
• How can I …

Cesium on the Liquid Galaxy

By Dave Jenkins
July 13, 2016

Data visualization continues to evolve, with ever-more complex data sets available openly, and a corresponding increased pace in visualization tools. In mapping GIS data, the Cesium app is gaining quite a bit of traction. As we continue to branch out with new functionality and visualization apps for the Liquid Galaxy, we wanted to try the Cesium app as well.

Cesium is written all in JavaScript WebGL and offers some nice advantages over other engines: it’s open source, it’s flexible, and it’s quick. It can accept an array of points, shapefiles, 3D models, and even KML. The JavaScript then chews these up and delivers a nice consistent 3D environment that we can fly through with the SpaceNav controller, set scenes in a presentation to tell a story, or mix together with video or graphic popups for a fully immersive multimedia experience. Cesium is open source, and provides a great deal of flexibility and accessibility to build different kinds of data visualizations and interactions. There are a lot of new startups exploiting this platform and we welcome the opportunity to work with them.

As we’ve written previously, the main advantage of the Liquid Galaxy platform is the ability to …

Disabling Postgres constraints for pg_dump

By Greg Sabino Mullane
July 13, 2016

Photo by Jocelyn Kinghorn

Constraints in Postgres are very powerful and versatile: not only are foreign keys, primary keys, and column uniqueness done internally via constraints, but you may create your own quite easily (at both the column and table level). Most of the time constraints are simply set and forget, but there is one time constraints may become a problem: copying the database using the pg_dump program.

The issue is that constraints are usually added before the data is copied to the new table via the COPY command. This means the constraint fires for each added row, to make sure that the row passes the conditions of the constraint. If the data is not valid, however, the COPY will fail, and you will not be able to load the output of your pg_dump into a new database. Further, there may be a non-trivial performance hit doing all that validation. Preventing the constraint from firing may provide a significant speed boost, especially for very large tables with non-trivial constraints.

Let’s explore one way to work around the problem of pg_dump failing to work because some of the data is not valid according to the logic of the constraints. While it would be quicker to make some …

Liquid Galaxy on PBS

By Ben Witten
July 12, 2016

PBS recently aired a segment about the Liquid Galaxy! Just before we presented at New York Tech Meetup, we were interviewed about the Liquid Galaxy for SciTech Now, a PBS publication. The interview took place in NYU’s Skirball Center For The Performing Arts, which is where New York Tech Meetup takes place every month.

The Liquid Galaxy segment, which can be viewed above, features Ben Goldstein and me talking with complementary visuals playing at the same time.

Ben Goldstein opens the segment by talking about how the Liquid Galaxy is a panoramic system that engages your peripheral vision, and is immersive in that way.

I go on to add that the system consists of large paneled screens set up in an arch around the viewer. The Liquid Galaxy includes a touchscreen and 3D joystick that allows users can fly around the world. From there, with the use of End Point’s Content Management System, users can add images, video, kml, other overlay, to add interactivity and build custom presentations on the system. Thus far, the Liquid Galaxy has been particularly popular in real estate, museums, aquariums, research libraries, hospitality, and travel.

Ben concludes the segment by talking about how …

Scrape web content with PHP (no API? no problem)

By Piotr Hankiewicz
July 7, 2016

Introduction

There is a lot of data flowing everywhere. Not structured, not useful pieces of data moving here and there. Getting this data and structuring, processing can make it really expensive. There are companies making billions of dollars just (huh?) for scraping web content and showing in a nice form.

Another reason for doing such things can be for example, lack of an API from a source website. In this case, it’s the only way to get data that you need to process.

Today I will show you how to get web data using PHP and that it can be as easy as pie.

Just do it

There are multiple scraping scripts ready to use. I can recommend one of them: PHP Simple HTML DOM Parser. It’s extremely easy to start with and initial cost is almost nothing, it’s open sourced also.

First, download a library from an official site: https://sourceforge.net/project/showfiles.php?group_id=218559. You can use a composer version too, it’s here: https://github.com/sunra/php-simple-html-dom-parser.

Let’s say that you have downloaded this file already. It’s just a one PHP file called simple_html_dom.php. Create a new PHP file called scraper.php and include mentioned library like this:

<?php

require( …

WAL-based Estimates For When a Record Was Changed

By Josh Williams
July 1, 2016

I originally titled this: Inferring Record Timestamps by Analyzing PITR Streams for Transaction Commits and Cross-Referencing Tuple xmin Values. But that seemed a little long, though it does sum up the technique.

In other words, it’s a way to approximate an updated_at timestamp column for your tables when you didn’t have one in the first place.

PostgreSQL stores the timestamp of a transaction’s commit into the transaction log. If you have a hot standby server, you can see the value for the most-recently-applied transaction as the output of the pg_last_xact_replay_timestamp() function. That’s useful for estimating replication lag. But I hadn’t seen any other uses for it, at least until I came up with the hypothesis that all the available values could be extracted wholesale, and matched with the transaction ID’s stored along with every record.

If you’re on 9.5, there’s track_commit_timestamps in postgresql.conf, and combined with the pg_xact_commit_timestamp(xid) function has a similar result. But it can’t be turned on retroactively.

This can – sort of. So long as you have those transaction logs, at least. If you’re doing Point-In-Time Recovery you’re likely to at least have …