Getting realtime output using Python Subprocess
By Kannan Ponnusamy
January 28, 2015
The Problem
When I launch a long running unix process within a python script, it waits until the process is finished, and only then do I get the complete output of my program. This is annoying if I’m running a process that takes a while to finish. And I want to capture the output and display it in the nice manner with clear formatting.
Using the subprocess and shlex library
Python has a “batteries included” philosophy. I have used 2 standard libraries to solve this problem.
import subprocess
import shlex- subprocess—Works with additional processes.
- shlex—Lexical analysis of shell-style syntaxes.
subprocess.popen
To run a process and read all of its output, set the stdout value to PIPE and call communicate().
import subprocess
process = subprocess.Popen(['echo', '"Hello stdout"'], stdout=subprocess.PIPE)
stdout = process.communicate()[0]
print 'STDOUT:{}'.format(stdout)The above script will wait for the process to complete and then it will display the output. So now we are going to read the stdout line by line and display it in the console untill it completes the process.
output = process.stdout.readline()This will read a line from the stdout. …
shell python
Postgres session_replication role - Bucardo and Slony’s powerful ally
By Greg Sabino Mullane
January 28, 2015
One of the lesser known Postgres parameters is also one of the most powerful: session_replication_role. In a nutshell, it allows you to completely bypass all triggers and rules for a specified amount of time. This was invented to allow replication systems to bypass all foreign keys and user triggers, but also can be used to greatly speed up bulk loading and updating.
The problem with disabling triggers
Once upon a time, there were two replication systems, Slony and Bucardo, that both shared the same problem: triggers (and rules) on a “target” table could really mess things up. In general, when you are replicating table information, you only want to replicate the data itself, and avoid any side effects. In other words, you need to prevent any “post-processing” of the data, which is what rules and triggers may do. The disabling of those was done in a fairly standard, but very ugly method: updating the system catalogs for the tables in question to trick Postgres into thinking that there were no rules or triggers. Here’s what such SQL looks like in the Bucardo source code:
$SQL = q{
UPDATE pg_class
SET …bucardo postgres replication
Social Innovation Summit 2014
By Erik Samsoe
January 28, 2015
In November, Josh Ausborne and I set up a Liquid Galaxy at the Sofitel Hotel in Redwood City for the 2014 Social Innovation Summit. Prior to the event the End Point content team worked together with Chris Busselle and Sasha Buscho from Google.org to create presentations featuring 10 grantee organizations.
With the Liquid Galaxy we were able to feature “Street View” panoramas of people enjoying the High Line in New York City, penguins standing on the shoreline for Penguin Foundation, and seals swimming underwater for Conservation International. The touchscreen and space navigator control device enabled users to view 360 degrees of the landscape as if they had been teleported to each location.
image credit: Jensen Sutta
I was thrilled to see the Google.org team in front of the system sharing the larger narrative associated with each project. This highlights one of the many opportunities of the Liquid Galaxy—the opportunity to share, explore and collaborate in real time, in front of an immersive array of screens.
image credit: Chris Busselle
One of the prepared presentations highlighted a data collaboration with Polaris to fight human trafficking. With Google Earth, Chris …
conference visionport
Elastic Beanstalk Whenever
By Kent Krenrich
January 27, 2015
I recently got the opportunity to pick up development on a Ruby on Rails application that was originally set up to run on AWS using their Elastic Beanstalk deployment tools. One of our first tasks was to move some notification hooks out of the normal workflow into scripts and schedule those batch scripts using cron.
Historically, I’ve had extremely good luck with Whenever. In my previous endeavors I’ve utilized Capistrano which Whenever merged with seamlessly. With how simple it was to integrate Whenever with Capistrano, I anticipated a similar experience dealing with Elastic Beanstalk. While the integration was not as seamless as Capistrano, I did manage to make it work.
My first stumbling block was finding documentation on how to do after or post hooks. I managed to find this forum post and this blog post which helped me out a lot. The important detail is that there is a “post” directory to go along with “pre” and “enact”, but it’s not present by default, so it can be easy to miss.
I used Marcin’s delayed_job config as a base. The first thing I had to address was an apparent change in Elastic Beanstalk’s configuration structure. Marcin’s config has
. …rails sysadmin aws
CentOS 7 on Hetzner server with more than 2 TB disk
By Spencer Christensen
January 22, 2015
We use a variety of hosting providers for ourselves and our clients, including Hetzner. They provide good servers for a great price, have decent support, and we’ve been happy with them for our needs.
Recently I was given the task of building out a new development server for one of our clients, and we wanted it to be set up identically to another one of their servers but with CentOS 7. I placed the order for the hardware with Hetzner and then began the procedure for installing the OS.
Hetzner provides a scripted install process that you can kick off after booting the machine into rescue mode. I followed this process and selected CentOS 7 and proceeded through the whole process without a problem. After rebooting the server and logging in to verify everything, I noticed that the disk space was capped at 2 TB, even though the machine had two 3 TB drives in it (in hardware RAID 1). I looked at the partitions and found the partition table was “msdos”. Ah ha!
At this point painful memories of running into this problem before hit me. I reviewed our notes of what we had done last time, and felt like it was worth a shot even though this time I’m dealing with CentOS 7. I went through the …
redhat devops hosting
PGConf.US NYC: "Choosing a Logical Replication System"
By David Christensen
January 22, 2015
I’m excited to have my talk “Choosing a Logical Replication System” accepted to PGConf.US! I’ll be speaking on Friday, March 27th from 2:00 - 2:50, as part of the Strategy track.
In this talk I will cover a variety of existing Logical Replication systems for PostgreSQL and go over some of the differences between requirements, supported capabilities, and why you might choose one system over another. I’ll also cover some of the changes in PostgreSQL 9.4.
conference postgres
Cleaner redirection in Perl Dancer
By Jeff Boes
January 21, 2015
Recently I worked on a project using the Perl web application framework Dancer that had multiple paths to order a product:
/product => /cart => /checkout => /receiptThat’s the standard approach. Then there was a “phone order” approach:
/create_order => /checkout => /receiptA “phone order” is one taken down (usually by phone), where the user who is logged in is not the same as the user who “owns” the order. Thus, one user is ordering on behalf of another: the order must be recorded as part of the second user’s order history, the various shipping and billing information must come from that user’s stored information, and even the product pricing has to be calculated as though that customer were doing the ordering rather than the logged-in user.
As a consequence, the phone order page flow actually ended up as:
get /create_order => post /create_order => /checkoutThe submission of the /create_order page was processed in an environment that knew about this “proxy” ordering arrangement, thus could do some particularly special-case processing, and then the idea was to pass off to the /checkout page, which would finalize the order including payment information. …
dancer perl
SSH one-time passwords (otpw) on chromebook
By Greg Sabino Mullane
January 21, 2015
A little while ago, I bought a Chromebook as an alternative to my sturdy-but-heavy laptop. So far, it has been great—quick boot up, no fan, long battery life, and light as a feather. Perfect for bringing from room to room, and for getting some work done in a darkened bedroom at night. The one large drawback was a lack of SSH, a tool I use very often. I’ll describe how I used one-time passwords to overcome this problem, and made my Chromebook a much more productive tool.
The options for using SSH on Chrome OS are not that good. I downloaded and tried a handful of apps, but each had some significant problems. One flaw shared across all of them was a lack of something like ssh-agent, which will cache your SSH passphrase so that you don’t have to type it every time you open a new SSH session. An option was to use a password-less key, or a very short passphrase, but I did not want to make everything less secure. The storage of the SSH private key was an issue as well—the Chromebook has very limited storage options, and relies on putting most things “in the cloud”.
What was needed was a way to use SSH in a very insecure environment, …
chrome security ssh sysadmin