Our Blog

img.bi, a secret encrypted image sharing service tool

By Emanuele “Lele” Calò
July 30, 2015

After a fairly good experience with dnote installed on our own servers as an encrypted notes sharing service, my team decided that it would have been nice to have a similar service for images.

We found a nice project called img.bi that is based on NodeJS, Python, Redis and a lot of client-side JavaScript.

The system is divided into two components: the HTML/JS frontend and a Python FastCGI API.

Unfortunately the documentation is a still in its very early stage and it’s lacking a meaningful structure and a lot of needed information.

Here’s an overview of the steps we followed to setup img.bi on our own server behind nginx.

First of all we chose that we wanted to have as much as possible running and confined to a regular user, which is always a good idea with such young and potentially vulnerable tools. We chose to use the imgbi user.

Then since we wanted to keep as clean as possible the root user environment (and system status), we also decided to use pyenv. To be conservative we chose the latest Python 2.7 stable release, 2.7.10.

git clone https://github.com/yyuu/pyenv.git ~/.pyenv
echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bash_profile
echo 'export …

Nothing more permanent than a temporary fix

By Jon Jensen
July 28, 2015

A brief thought:

You may have heard the saying that nothing is more permanent than a temporary fix. Or that prototypes are things we just haven’t yet recognized will be permanent. Or some variation on the theme.

As an illustration of this, I recently came across the initial commit to the source code repository of our endpoint.com website when we ported it to Ruby on Rails back in April 2007. Our then co-worker PJ’s comment is a perfect example of how long-lasting some of our planned temporary work can be:

commit 2ee55da6ed953c049b3ef6f9f132ed3c1e0d4de9
Author: PJ Cabreras <pj@endpoint.com>
Date:   Wed Apr 18 13:07:46 2007 +0000

    Initial test setup of repository for mkcamp testing -- will probably throw away later
    
    git-svn-id: file:///home/camp/endpoint/svnrepo/trunk@1 7e1941c4-622e-0410-b359-a11864f70de7

It’s wise to avoid big architecture up front for experimental things we don’t know the needed shape and size of. But we should plan on iterating and being agile (in the real basic sense of the word), because we may never have the chance to start over from scratch. And starting over from scratch is often ill-advised in any case.

E-commerce website encryption changes

By Jon Jensen
July 24, 2015

The big picture

Computer security is a moving target, and during the past few years it’s been moving faster than ever.

In the e-commerce world, the PCI Security Standards Council sets the rules for what merchants and vendors must do to have what they consider to be a sufficiently secure environment to handle cardholder data such as credit card numbers, expiration dates, and card security codes.

PCI DSS 3.1, released on 15 April 2015 puts us all on notice that TLS 1.0 is considered unfit to use for e-commerce website encryption (HTTPS), and will be disallowed soon. The new rules specify that new software implementations must not use TLS versions prior to 1.1. Existing implementations must require TLS 1.1 or 1.2 no later than 30 June 2016.

They provide some guidance on Migrating from SSL and early TLS and explain what is expected in more detail.

Long ago we were required to disable SSL 2, and last year we were expected to disable SSL 3, the predecessor to TLS 1.0. That turned out to not be particularly hard or cause too many problems, because almost all systems that supported SSL 3 also supported TLS 1.0.

This time we are not so lucky. Many clients (such as browsers) and servers did …

Selectively firing Postgres triggers

By Greg Sabino Mullane
July 15, 2015

Lake Wallaga by Tony Heyward

Being able to disable Postgres triggers selectively can be an important skill when doing tasks like bulk updates, in which you only want a subset of the triggers on the table to be fired. Read below for the long explanation, but the TL;DR version of the best solution is to set a WHEN clause on the trigger you wish to skip, making it conditional on a variable such as session_replication_role, or application_name

CREATE TRIGGER mytrig AFTER INSERT ON foobar FOR EACH
  ROW WHEN (current_setting('session_replication_role') <> 'local') EXECUTE PROCEDURE myfunc();
BEGIN;
SET LOCAL session_replication_role = 'local';
UPDATE foobar SET baz = 123;
COMMIT;

I decided to spin up a free Heroku “Hobby Dev” database to illustrate the solutions. Generating a test table was done by using the Pagila project, as it has tables which contain triggers. Heroku gives you a randomly generated user and database name. To install the Pagila schema, I did:

$ export H="postgres://vacnvzatmsnpre:2iCDp-46ldaFxgdIx8HWFeXHM@ec2-34-567-89.compute-1.amazonaws.com:5432/d5q5io7c3alx9t"
$ cd pagila-0.10.1
$ psql $H -q -f pagila-schema.sql
$ psql $H -q …

How fast is pg_upgrade anyway?

By Greg Sabino Mullane
July 1, 2015

Shark photo by Rennett Stowe

Back in the old days, upgrading Postgres required doing a pg_dump and loading the resulting logical SQL into the new database. This could be a very slow, very painful process, requiring a lot of downtime. While there were other solutions (such as Bucardo) that allowed little (or even zero) downtime, setting them up was a large complex task. Enter the pg_upgrade program, which attempts to upgrade a cluster with minimal downtime. Just how fast is it? I grew tired of answering this question from clients with vague answers such as “it depends” and “really, really fast” and decided to generate some data for ballpark answers.

Spoiler: it’s either about 3.5 times as fast as pg_dump, or insanely fast at a flat 15 seconds or so. Before going further, let’s discuss the methodology used.

I used the venerable pgbench program to generate some sample tables and data, and then upgraded the resulting database, going from Postgres version 9.3 to 9.4. The pgbench program comes with Postgres, and simply requires an –initialize argument to create the test tables. There is also a –scale argument you can provide to increase the amount of initial data—​each …

Python string formatting and UTF-8 problems workaround

By Muhammad Najmi bin Ahmad Zabidi
July 1, 2015

Recently I worked on a program which required me to filter hundred of lines of blog titles. Throughout the assignment I stumbled upon a few interesting problems, some of which are outlined in the following paragraphs.

Non Roman characters issue

During the testing session I missed one title and investigating why it happened, I found that it was simply because the title contained non-Roman characters.

Here is the code’s snippet that I was previously using:

for e in results:                                                                                                                        
    simple_author=e['author'].split('(')[1][:-1].strip()                                                             
    if freqs.get(simple_author,0) < 1:                                                                                               
        print parse(e['published']).strftime("%Y-%m-%d") , "--",simple_author, "--", e['title']

And here is the fixed version

for e in results:                                                                                                                        
    simple_author=e[ …

Documenting web services with Perl POD and AJAX

By Jeff Boes
June 26, 2015

Perl POD is a handy, convenient, but low-tech approach to embedded documentation. Consider a web service in Dancer:

get time => sub {
  return scalar(localtime());
};

(Disclaimer: my actual use-case of this technique was even more legacy: I was documenting Interchange Actionmaps that returned images, JSON, etc.)

Your application might have several, or even dozens of these, with various parameters, returning data in JSON or TXT or CSV or who-knows-what. I chose to document these in Perl POD (Plain Old Documentation) format, e.g.,

=pod

=head1 time

Retrieves the current time

=over 3

=item Parameters

None.

=item Example

=begin html

<script src="/js/example-time.js" type="text/javascript"></script>

=end html

=back

=cut

This block gets inserted right in-line with the web service code, so it’s immediately obvious to anyone maintaining it (and thus has the best chance of being maintained if and when the code changes!). Now I can generate an HTML page directly from my Perl code:

$ pod2html MyPackage.pm

Your output looks something like this (excerpted for clarity):

time

Retrieves the current time Parameters

None.

Example

Where the magic comes in …

IPython Tips and Tricks

By Kannan Ponnusamy
June 18, 2015

Recently I have been working on Python automation scripts. Very often I use IPython to develop/debug the code.

IPython is an advanced interactive python shell. It is a powerful tool which has many more features. However, here I would like to share some of the cool tricks of IPython.

Getting help

Typing object_name? will print all sorts of details about any object, including docstrings, function definition lines (for call arguments) and constructor details for classes.

In [1]: import datetime
In [2]: datetime.datetime?
Docstring:
datetime(year, month, day[, hour[, minute[, second[, microsecond[,tzinfo]]]]])

The year, month and day arguments are required. tzinfo may be None, or an
instance of a tzinfo subclass. The remaining arguments may be ints or longs.
File:      /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/datetime.so
Type:      type

Magic commands

Edit

This will bring up an editor to type multiline code and execute the resulting code.

In [3]: %edit
IPython will make a temporary file named: /var/folders/xh/2m0ydjs51qxd_3y2k7x50hjc0000gn/T/ipython_edit_jnVJ51/ipython_edit_NdnenL.py

In [3]: %edit -p

This will bring up the editor with the …