Giraffes and Liquid Galaxy
By Josh Tolley
August 25, 2011
We build lots and lots of Google Earth tours, typically for use with a Liquid Galaxy. These tours tell Google Earth to start at one point and fly to another, displaying various images along the way. They’re written in an XML-based language called Keyhole Markup Language, or KML. Various KML elements describe points on the globe, 3D objects, tracks through space, or other items, and a series of Google-specific KML extensions can define an automated tour in which Google Earth flies through these objects.
Now for a bit of dogma: writing large volumes of XML by hand is a drag. Writing KML is no exception. So when I started building KML tours, I was keen to build or find tools to make it easier. For my first KML project, making a tour through each End Point employee’s home city, I built a Perl script to convert a file of employee data to KML. This worked well enough, but I soon learned typing out all the KML was just half the pain. The other half was adjusting camera paths, camera angles, timing, and all sorts of other little things. Unfortunately all these adjustments had to be done by trial and error, and all by hand. Getting Google Earth’s camera to circle a particular point, for …
google-earth visionport kamelopard kml
PostgreSQL log analysis / PGSI
By Greg Sabino Mullane
August 19, 2011
Image by “exfordy” on Flickr
End Point recently started working with a new client (a startup in stealth mode, cannot name names, etc.) who is using PostgreSQL because of the great success some of the people starting the company have had with Postgres in previous companies. One of the things we recommend to our clients is a regular look at the database to see where the bottlenecks are. A good way to do this is by analyzing the logs. The two main tools for doing so are PGSI (Postgres System Impact) and pgfouine. We prefer PGSI for a few reasons: the output is better, it considers more factors, and it does not require you to munge your log_line_prefix setting quite as badly.
Both programs work basically the same: given a large number of log lines from Postgres, normalize the queries, see how long they took, and produce some pretty output.If you only want to look at the longest queries, it’s usually enough to set your log_min_duration_statement to something sane (such as 200), and then run a daily tail_n_mail job against it. This is what we are doing with this client, and it sends a daily report that looks like this:
Date: Mon Aug 29 11:22:33 2011 UTC
Host: acme-postgres-1
Minimum …analytics database monitoring performance postgres
jQuery and hidden elements
By Jeff Boes
August 16, 2011
Out of sight, not out of mind.
While extending some jQuery functionality on a page, I noted that a form element’s “change” handler wasn’t being invoked, which meant some of the page initialization code would be left out. So I helpfully added it:
$('input[name=foobar]').change(...).change();What I failed to contemplate was how this would impact another page, which just happened to have * some * (but not all) of the same form elements, and referenced the same JS code at page load. Specifically, my page’s sibling had:
<input name="foobar" type="hidden">And of course this broke in production.
Well, that’s interesting. A change handler on a hidden form input field isn’t usually all that useful, so I figured out I really needed:
$('input[name=foobar]').filter(':visible').change(...).change();It happens that the “.filter()” step is actually more efficient than doing it all in one selector (“input[name=foobar]:visible”), because of some obscurities within jQuery. That little discovery was of value to an End Point co-worker, who realized she could shave a little time off a page load elsewhere, so my minor page malfunction will redeem …
javascript jquery
Changing postgresql.conf from a script
By Greg Sabino Mullane
August 10, 2011
Image by “TheBusyBrain” on Flickr
The modify_postgres_conf script for Postgres allows you to change your postgresql.conf file from the command line, via a cron job, or any time when you want to automate the process.
Postgres runs as a background daemon. The configuration parameters it runs with are stored in a file named postgresql.conf. To change the behavior of Postgres, one must usually edit this file, and then tell Postgres that you have made the changes. Sometimes all that is needed is to ‘HUP’ or reload Postgres. Most changes fall into this category. Other changes require a full restart of Postgres, which entails disconnecting all current clients.
Thus, to make a change, one must edit the file, find the item to change (the file consists of “name = value” lines), change it, then send a signal to the main Postgres process so it picks up the change. Finally, you should …
database perl postgres tools
Remove specific CGI variables using Apache
By David Christensen
August 10, 2011
Sometime you need to remove a single CGI variable from a query string in your published URLs; for instance, one of our client’s site had gotten spidered with a session id in the generated links, so we wanted to ensure that those URLs would be appropriately updated when re-spidered. Apache’s mod_rewrite to the rescue!
The following snippet serves to rewrite any URL which has a query string parameter named id to one the exact same without that CGI variable. Since mod_rewrite uses PCRE, we can use this to our advantage by using \b word break anchors to ensure we’re only picking up a CGI variable named exactly the same, so (say) id=bar will be removed but tid=foo will pass on through the rewrite unaltered.
# already assume RewriteEngine on
RewriteCond %{QUERY_STRING} ^(.*)\bid=(\w*)\b(.*)
RewriteRule (.*) $1?%1%3 [R=301,L]Note in the above example that we know the range of values that was present for the id variable, so \w is sufficient to determine the full value to remove in this case. $N is the replacement from the Nth RewriteRule group and %N is the replacement from the Nth RewriteCond group. We use [R=301] to trigger an external 301 redirect, which search engines should see as …
ecommerce sysadmin tips tools
The rails_admin gem in Ecommerce
By Steph Skardal
August 9, 2011
Update: Since writing this article, this gem has had several updates. All rails_admin configuration now must be pulled out of ActiveRecord models and into the initializer. The look and feel has also been updated. Read about it more here. Additionally, End Point has released a Ruby on Rails Ecommerce Engine with a sleek admin interface driven by rails_admin. Read more about it here.
I recently installed the rails_admin gem into a new Rails project. This particular site is currently running on Interchange, but it is not an ecommerce business, so Interchange is overkill for the site. The client has recently decided to make the switch to Rails (and DevCamps) with our help, and they have a moderate budget to do so. For the first increment, we considered installing phpMyAdmin for them to work directly with the data until a nice admin interface was built, but instead I spent a bit of time installing the rails_admin gem which has been on my mind for the last 6 months. The result: I’m extremely impressed with what rails_admin has to offer.
To show some examples of rails_admin in action, I’ll use the data model from a Sinatra ecommerce setup I recently wrote about, because it has a basic …
ecommerce ruby rails sinatra
Rails Optimization: Digging Deeper
By Steph Skardal
August 5, 2011
I recently wrote about raw caching performance in Rails and advanced Rails performance techniques. In the latter article, I explained how to use a Rails low-level cache to store lists of things during the index or list request. This technique works well for list pages, but it doesn’t necessarily apply to requests to an individual thing, or what is commonly referred to as the “show” action in Rails applications.
In my application, the “show” action loaded at ~200 ms/request with low concurrency, with the use of Rails fragment caching. And with high concurrency, the requests shot up to around 2000 ms/request. This wasn’t cutting it! So, I pursued implementing full-page caching with a follow-up AJAX request, outlined by this diagram:
First, the fully-cached is loaded (quickly). Next, an AJAX request is made to retrieve access information. The access information returns a JSON object with information on whether or not there is a user, and if that user has edit access to that thing. If there is no user, the page stays as is. If there is a user, but he does not have edit permissions, the log out button is shown and the username is populated. If there is a user and he has edit …
javascript performance rails
DevCamps news
By Jon Jensen
August 4, 2011
DevCamps is a system for managing development, integration, staging, and production environments. It was developed by End Point for, and with the help of, some of our ecommerce clients. It grew over the space of several years, and really started to become its own standalone project in 2007.
Camps are a behind-the-scenes workhorse of our web application development at End Point, and don’t always get much attention because everyone’s too busy using camps to get work done! But this summer a few things are happening.
In early July we unveiled a redesign of the devcamps.org website that features a more whimsical look, a better explanation of what camps are all about, and endorsements by business and developer users. Marko Bijelic of Hipinspire did the design. Take a look:
In less than two weeks, on August 17, I’m going to be giving a talk on camps at YAPC::EU in Riga, Latvia. YAPC::EU is Europe’s annual Perl conference, and will be a nice place to talk about camps.
Many Perl developers are doing web applications, which is camps’ main focus, so that’s reason enough. But camps also started around the Interchange application server, which is written in Perl. And the camp …
camps conference perl open-source