Our Blog

We’re hiring Rails developers!

By Jon Jensen
August 22, 2013

This position has been filled. See our active job listings here.

We’re looking for a few more talented Ruby on Rails developers to consult with our clients and develop their web applications. Do you like to focus on solving business problems? Do you take responsibility for getting a job done well without intensive oversight? Then please read on!

End Point is an 18-year-old web consulting company based in New York City, with 38 full-time employees working mostly remotely from home offices in the United States, Canada, and Europe. Our team is made up of strong ecommerce, database, and system administration talent, working together using ssh, tmux and screen, IRC, phone, Google+ hangouts, and Skype.

We serve over 200 clients ranging from small family businesses to large corporations, using a variety of open source technologies including Ruby, Python, Perl, Git, PostgreSQL, MySQL, CouchDB, Redis, Elasticsearch, jQuery, and many more, on Linux.

What you will be doing:

• Help clients determine their web application needs
• Build, test, release, and maintain web applications for our clients
• Work with open source tools and contribute back as opportunity arises
• Use your desktop platform of choice: Linux, Mac OS X, or Windows

What you will need: …

The Un-unaccentable Character

By Josh Williams
August 18, 2013

I typed “Unicode” into an online translator, and it responded saying it had no idea what the language was but it roughly translates to “Surprise!”

Recently a client sent over a problem getting some of their Postgres data through an ASCII-only ETL process. They only needed to worry about some occasional accent marks, and not any of the more uncommon or odd Unicode characters, thankfully. ☺ Or so we thought. The unaccent extension was a great starting point, but the problem they sent over boiled down to this:

postgres=# SELECT unaccent('e é ѐ');
 unaccent 
----------
 e e ѐ
(1 row)

unaccent() worked, except for that odd ѐ, which then failed the ETL task. That’s exactly what unnaccent is supposed to handle. The character è even appears in the unaccent.rules file. So what gives?

Well, if you’re in the habit of piping blog posts through hexdump (and who isn’t?) then you probably already know the answer. But even if not, you may already suspect that we’re dealing with a different character that just looks the same. And you’d be right. Specifically, the è in the rules file is from the more common Latin set, and the ѐ that doesn’t work is from the Cyrillic set. Pretty …

Give me my /var/log/rpmpkgs back!

By Emanuele “Lele” Calò
August 12, 2013

When switching from RHEL5 to RHEL6 everyone had fears and hopes about things which would have been lost and gained.

One of the lost ones is /var/log/rpmpkgs which is a nice tool which helps system administrator staying sane when a server rebuild or migration is needed by giving them the list of packages installed up to the day before.

How this feature works is that basically a daily cronjob dumps the installed packages in the log file /var/log/rpmpkgs, along with various information, for the sake of system maintainers.

What happened is that while this tool was included in the RPM package ’til RHEL5 (and CentOS 5.x), when releasing RHEL6 (and CentOS 6.x) they decided to split it and create a specific package called rpm-cron.

So if you’re among the ones who misses this useful feature, please fire up your SSH connection and type

yum install rpm-cron

And rejoice of the useful tool being back where it should be.

Speeding Up Some FDW Queries

By Szymon Lipiński
August 8, 2013

There was a very interesting question about PostgreSQL optimization. It was about speeding up a query on foreign tables.

Foreign Data Wrappers

FDW is quite a nice idea, it allows to use different sources of data and access them like a normal database table.

You can find some more information about writing custom FDW handlers or use some already created. This way you can connect to another database, or even use CSV files as Postgres tables without loading them into database.

Introduction

Let’s take a couple of tables and a view created on the top of them:

CREATE TABLE t_10_20(i INTEGER);
CREATE TABLE t_15_20(i INTEGER);
CREATE TABLE t_10_16(i INTEGER);

CREATE VIEW all_tables AS
  SELECT i FROM t_10_20
  UNION ALL
  SELECT i FROM t_15_20
  UNION ALL
  SELECT i FROM t_10_16;

I assume, as it was in the original question, that there are some strict ranges of data in the tables. For the example I encoded the ranges in table names, so table t_10_20 contains values from the range [10,20] and table t_10_16 has values from [10,16];

The above view will be used for getting all data.

For filling them up, I used a function which I wrote long time ago, it returns a random number with uniform …

Buy One Get One Promotion with Spree

By Brian Buchalter
August 7, 2013

Implementing a “Buy One, Get One Free” promotion in Spree requires implementation of a custom promotion action and appropriate use of existing promotion rules. This article implements the promotion by automatically adding and removing immutable “get one” line items whose price is zero and whose quantity always mirrors its paid “buy one” counterpart. Although written and tested with Spree’s 1-3-stable branch, the core logic of this tutorial will work with any version of Spree.

Promotion Eligibility

Begin by creating a new promotion using a meaningful name. Set the “event name” field to be “Order contents changed” so the promotion’s actions are updated as the order is updated. Save this new promotion, so we can then configure Rules and Actions. In the Rules section, select the “Product(s)” rule and click Add. Now choose the products you’d like to be eligible for your promotion. If you’d like to include broader sets such as entire taxonomies (and have implemented the custom promotion rules to do so), feel free to use them. When we implement the promotion action, you’ll be able to make things work.

You should now have a product rule that selects some subset of products eligible for …

Little Spree Big Performance Problems

By Marina Lohova
August 5, 2013

Recently I worked on an online food store serving an area with very little infrastructure. As a result, the orders tended to be really big with lots of products.

The website worked in the following environment:

• Ruby 1.9.2
• Spree 0.60
• Heroku, Bamboo stack
• PostgreSQL 9.2.4

H12 timeout errors

The performance problems started when we migrated Bamboo to Cedar on Heroku and replaced Thin webserver with Unicorn. We started getting a lot of Heroku Request timeout errors - H12:

The problems happened mostly when logging in to admin dashboard or during the checkout for the certain orders. H12 errors occur when a HTTP request takes longer than 30 seconds to complete. For example, if a Rails app takes 35 seconds to render the page, the HTTP router returns a 503 after 30 seconds and abandons the incomplete Rails request for good. The Rails request will keep working and logging the normal errorless execution. After completion, the request will indefinitely hang in the application dyno.

We started debugging H12: we set Unicorn timeout to 20 seconds to prevent the runaway requests and installed the rack-timeout gem with the timeout of 10 seconds to raise an error on a slow request. It all came …

Disney Liquid Galaxy in Sao Paolo

By Dave Jenkins
August 1, 2013

End Point recently had the pleasure to work with Disney and Google to bring the Liquid Galaxy to Disney Expo in Sao Paolo, Brazil.

Disney saw the Liquid Galaxy at the Google office in Sao Paulo and recognized the “WOW!” factor that the display platform can provide, and the Disney Expo organizers saw a great fit for promoting the release of the upcoming animated film PLANES. Disney engaged End Point to develop a custom menu of “fly to” locations featured in the movie.

Attendees at the Expo experienced those locations in immersive high definition across 7 screens surrounding the viewer. End Point created a custom menu for the touch screen with one-touch buttons that “flew” the users to locations featured in PLANES:

• USA — Statue of Liberty
• ICELAND — Reykjavik Botanical Garden
• GERMANY — Deutsches Museum
• INDIA — Taj Mahal
• NEPAL — Himalayas
• CHINA — Great Wall
• MEXICO — Pyramids of Yucatan

The experience for the attendees at the expo was such that they could virtually fly to these locations just like the characters in the movie. Other options on the touch screen menu featured the Disney resort properties:

• USA — Orlando — Walt Disney World (Magic Kingdom)
• USA — Anaheim / California — …

Merging JSONs in PostgreSQL

By Szymon Lipiński
July 29, 2013

PostgreSQL’s JSON support is great, however there is one thing missing, and it will be missing in the next PostgreSQL release. It is not too easy to manipulate the values stored in such a JSON field. Fortunately there is an easy way to do anything you want, you can use some external programming language.

Merging JSONs

Sometimes you need to update one JSON with values from another. Or you just need to change one field in a JSON value. There is no easy way to do it in PostgreSQL, but with the help of external language, it seems trivial.

Let’s use simple JSONs:

WITH j AS (
  SELECT
    '{"a":42, "b":"test"}'::JSON a,
    '{"a":1764, "x":"test x"}'::JSON b
)
SELECT a, b
FROM j;

a           |            b
-------------------------------------------------------
 {"a":42, “b”:"test"} | {"a":1764, “x”:“test x”}<p></p>

Let’s assume I have value a stored in a table, and I want to update it with values from b. So I want to do something like:

UPDATE data SET j = something(j, b) WHERE id = 10;

The question is: how to merge those JSONs.

Merging

For merging I’ve written a simple plpython …