Optimizing media delivery with Cloudinary

2022-03-01T00:00:00+00:00

I remember how we needed to deal with different image formats and sizes years ago: From using the WordPress-style approach of automatically saving different resolutions on the server when uploading a picture, to using a PHP script to resize or crop images on the fly and return the result as a response to the frontend. Of course, many of those approaches were expensive, and not fully optimized for different browsers or device sizes.

With those experiences in mind, it was a nice surprise for me to discover Cloudinary when working on a new project a couple of months ago. It’s basically a cloud service that saves and delivers media content with a lot of transformations and management options for us to use. There is a free version with a usage limit: Up to 25K transformations or 25 GB of storage/bandwidth, which should be enough for most non-enterprise websites. The cheapest paid service is $99 per month.

Here’s a list of the image features we used on that project. I know they offer many other things that can be used as well, but I think this is a good start for anyone who hasn’t used this service yet:

Resizing and cropping

When you make a request for an image, you can instruct the Cloudinary API to retrieve it with a given size, which will trigger a transformation on their end before delivering the content. You can also use a cropping method: fill, fill with padding, scale down, etc.

Gravity position

When we specify a gravity position to crop an image, the service will keep the area of the image we decide to use as the focal point. We can choose a corner (for example, top left), but also—and this is probably one of the most interesting capabilities on this service—we can specify “special positions”: By using machine learning, we can instruct Cloudinary to use face detection, or even focus on other objects, like an animal or a flower in the picture.

Automatic format

Another cool feature is the automatic format, which will use your request headers to find the most efficient picture format for your browser type and version. For example, if the browser supports it, Cloudinary will return the image in WebP format, which is generally more efficient than standard JPEG, as End Point CTO Jon Jensen demonstrates on his recent blog post.

Automatic format in action: Returning a WebP image in Chrome

Other features

There are many other options for us to choose when querying their API, like setting up a default placeholder when we don’t have an image, applying color transformations, removing red eyes, among other things. The Transformation reference page on their documentation section is a great resource.

NuxtJS integration

The project I mentioned above was a NuxtJS application with a Node.js backend. And since there’s a NuxtJS module for Cloudinary, it made sense to use it instead of building the queries to the API from scratch.

The component works great, except for one bug that we found that didn’t allow us to fully use their image component with server-side rendering enabled. Between that drawback and some issues trying to use the lazy loading setting, we ended up creating a Vue component ourselves that used a standard image tag instead. But we still used their component to generate most of the API calls and render the results.

Below is an example of using the Cloudinary Image component on a Vue template:

<template>
  <div>
    <cld-image
      :public-id="publicId"
      width="200"
      height="200"
      crop="fill"
      gravity="auto:subject"
      radius="max"
      fetchFormat="auto"
      quality="auto"
      alt="An image example with Cloudinary"
    />
  </div>
</template>

Alternatives

Of course, Cloudinary is not the only image processing and CDN company out there: There are other companies offering similar services, like Cloudflare images, Cloudimage, or imagekit.io.

Do you know any other good alternatives, or have you used any other Cloudinary feature that is not listed here? Feel free to add a comment below!

Image compression: WebP presets, HEIC, AVIF, JPEG XL

2022-02-15T00:00:00+00:00

How time flies. Eight years ago I wrote the blog post WebP images experiment on End Point website to describe and demonstrate how the WebP image format can store an equivalent-quality image in a much smaller file size than the older JPEG, PNG, and GIF formats.

My WebP examples there were 17–23% of the JPEGs they came from, or about 5–6× smaller. While experimenting with higher levels of compression, I found that WebP tends to leave less-noticeable artifacts than JPEG does.

The main drawback at the time was that, among major browsers, only Chrome and Opera supported WebP, and back then, Chrome was far less popular than it is now.

Can I use it?

Since Apple’s iOS 14 and macOS 11 (Big Sur) became available in late 2020, the WebP image format now works in all the currently supported major operating systems and browsers: Linux, Windows, and macOS, running Chromium, Chrome, Brave, Edge, Opera, Firefox, and Safari. You can see the specifics at the ever-useful site “Can I use”.

It only took about 10 years! 😁

So for you who are hosting websites, all your site visitors can see WebP images and animations except those vanishing few using Internet Explorer (now long past its end of support by Microsoft and dangerous to use), and people (or their organizations) who intentionally do not allow their older browsers and operating systems to be updated.

That means you may want to continue using JPEG, PNG, and/or GIF images in places crucial for rendering your main site features, for people to be able to understand the main things your site is trying to communicate.

But for any images that are less essential and primarily making it prettier, or where you don’t mind suggesting that users of old browsers upgrade to see them, WebP can now be your default image format.

How do I create WebP images?

Mobile phones and digital cameras typically save JPEG, HEIF, or raw (uncompressed) images. Some stock photography collections offer WebP downloads, but many still use JPEG.

So in many cases you won’t be starting with a WebP image, and you’ll convert some other image to WebP, likely after cropping, scaling, and other adjustments.

GIMP (GNU Image Manipulation Program) supports WebP images since about 2017, and Adobe Photoshop does not natively but can use the free WebPShop plugin.

The oldest way to convert images to WebP, and still very useful for batch processing or fine-tuning, is Google’s WebP converter “cwebp”.

cwebp settings

With the power “cwebp” offers comes some complexity, but it is mostly harmless.

Run this to see its many options:

cwebp -longhelp

Or read the same thing in its online documentation.

There among other things you will see the -z option which activates preset features for lossless encoding, with an integer level chosen from 0 to 9 where 0 is fastest but compresses less and 9 is slowest but compresses better. Use this to replace PNG files when you want no degradation of the image at all.

The documentation also shows the useful -preset and -hint options for lossy compression similar to what JPEG does, but better:

-preset <string> ....... preset setting, one of:
                          default, photo, picture,
                          drawing, icon, text

-hint <string> ......... specify image characteristics hint,
                         one of: photo, picture or graph

The meaning of a few of those terms, especially “photo” and “picture”, was not clear to me and not defined elsewhere in the documentation that I could see.

To find that out I had to make a quick trip into the source code, and there are comments explaining each option’s use case a bit:

The comments for the -preset option:

WEBP_PRESET_PICTURE,  // digital picture, like portrait, inner shot
WEBP_PRESET_PHOTO,    // outdoor photograph, with natural lighting
WEBP_PRESET_DRAWING,  // hand or line drawing, with high-contrast details
WEBP_PRESET_ICON,     // small-sized colorful images
WEBP_PRESET_TEXT      // text-like

And the comments for the -hint option:

WEBP_HINT_PICTURE,    // digital picture, like portrait, inner shot
WEBP_HINT_PHOTO,      // outdoor photograph, with natural lighting
WEBP_HINT_GRAPH,      // Discrete tone image (graph, map-tile etc).

So in short, “cwebp” considers a “picture” to be indoors and close-up, while “photo” is outdoors and more likely with more distant focus. That’s good to know.

Batch conversion

With that in mind, I can convert a pile of screenshots that have been collecting on my computer to refer to later. One kind of screenshots I sometimes take is of video meetings with mostly indoor views of people. I will use the “picture” preset for those.

A simple bash script works well to process many images in a row:

for infile in Screenshot*.png
do
    echo $infile
    base=$(basename "$infile" .png)
    cwebp -preset picture -v "$infile" -o "$base".webp
done

If you have many images to convert to WebP and want to do several at the same time to get done faster, you can use GNU parallel.

My screenshots when converted from PNG to WebP consistently take about 3% of the original space, 33–35× smaller! And the quality looks about the same. Amazing.

These screenshots are 2880×1800 pixels, mostly of Google Meet low-bandwidth video streams. The originals of these screenshots don’t look particularly good to begin with, with some blurriness. But exactly because of this, there is no reason for me to keep a larger high-quality original here. The much smaller WebP is fine.

Competitors to WebP

Other newer image formats have also been in the works for years, chasing some of the same goals. Should we skip WebP and use one of them instead?

HEIC

The HEIC (High-Efficiency Image Container) subset of the HEIF (High Efficiency Image File Format) standard uses High Efficiency Video Coding (HEVC, H.265) for storing images with a .heic suffix.

Compared to JPEG, HEIC offers the nice advantages of smaller file sizes for the same quality level (roughly half the size of an equivalent JPEG), and animation support (to replace GIF).

On the downside, HEIC is encumbered by patents that limit its use for major commercial projects, even on devices licensed for consumer use. It is also slower to encode/decode. And, concerning for archivists, HEIC shows severe visual damage to the entire image if part of the file is corrupted. In contrast, with corrupted JPEG files the visual damage is typically localized to particular smaller square regions, not the entire image.

HEIC has been used in Apple’s operating systems since the 2017 release of the iPhone 7 and iOS 11 and macOS 10.13 (High Sierra). Support was later added to Windows 10, Android 9, and Ubuntu 20.04.

As of this writing, no major browsers support HEIC natively, not even Apple’s own Safari.

So for now, HEIC is primarily used by Apple to store photos more efficiently on its mobile devices.

AVIF

The AVIF (AV1 Image File Format) competes with HEIC and, confusingly, uses the same HEIF container file format that HEIC does. That confusion is reduced in practice by its use of the separate file extension .avif.

AVIF is supported in current Chrome, Firefox, and Opera. Support was added to WebKit in 2021, but it still has not made its way into Safari. It also works in newer VLC, GIMP, Windows, Android, etc.

Netflix has a very detailed blog post comparing AVIF to JPEG and showing AVIF’s many advantages.

JPEG XL

A semi-compatible successor to JPEG has long been in the works, and JPEG XL seems likely to eventually fill that role.

Whereas the other new image formats mentioned above usually lose some quality when recompressing JPEG and other images that were already lossy-compressed, according to the Joint Photographic Experts Group (JPEG):

Existing JPEG files can be losslessly transcoded to JPEG XL, significantly reducing their size.

It has been reported that JPEG XL is expected to become available in its final standard form in 2022, and support is already available in preliminary form in some software (see the Wikipedia JPEG XL article).

That of course means that for now, no major browsers support JPEG XL natively.

Use WebP now

So all the other new options are not yet usable for general web images. WebP is the current obvious choice, whether you want a lossy replacement for JPEG photos, a lossless replacement for PNG images, or a replacement for GIF animations.

Our developers have set up automatic server-side app conversion of high-quality PNG originals to WebP or JPEG on the fly, with the image size dependent on the browser viewport size. And we have worked with Cloudinary, Cloudflare, and other CDNs to use their image conversion services. We are available to help with your projects too.

Reference

The Mozilla Development Network (MDN) has excellent documentation of web image format details, filename suffixes, and support in the major browsers in its Image file type and format guide.

Web Projects for a Rainy Day

2020-03-25T00:00:00+00:00

Image by Yellowstone NPS on Flickr

With the COVID-19 quarantine disrupting life for many of us, I thought I’d put together a list of things you can do with your website on a rainy day. These are things to keep your business moving even if you’re at home and some of your projects are stuck waiting on things to reopen. If you’re looking for some useful things to do to fill your days over the next few months, this post is for you!

Major Version Updates

Make a list of your entire stack, from OS to database to development frameworks. Note the current version and research the current supported versions. I find Wikipedia pages to be fairly reliable for this (e.g. en.wikipedia.org/wiki/CentOS). Ok, so what things need to be updated, or will need to be in the next year? Start on those now and use some downtime to get ahead of your updates.

Sample of a client’s stack review

Software	Purpose	Our version	Release date	End of support	Next update	Newest version	Notes
CentOS	OS for e-commerce server	7	July 2014	June 2024	Not imminent	8	https://wiki.centos.org/About/Product
Nginx	Web server	1.16.0	March 2020	Unclear	Not imminent	1.16.1	https://nginx.org/
PostgreSQL	Database server	9.5.20	January 2016	Feb 2020	Medium term, to version 11	12	https://www.postgresql.org/support/versioning/
Rails	App framework for store	5.1	February 2017	Current	Long Term, to version 6	6	https://rubygems.org/gems/rails/versions
Spree	Ecommerce and admin gem	3.3	April 2017	Current	Long Term, to version 4	4	https://rubygems.org/gems/spree/versions
Elasticsearch	Search platform for product import/search	5.6.x	September 2017	March 2019	Immediate, to version 6.8	7.4	https://www.elastic.co/support/eol
WordPress	Info site	5.2.3	September 2019	Unclear	5.2.4 shipped recently	5.2	https://codex.wordpress.org/Supported_Versions

Content Cleanup & SEO Review

Everyone’s website gets cluttered with outdated content. Take a look at your pages, review, and update what needs to be changed. Pay attention to search engine optimization (SEO) concerns as you go through it. Make sure your content has headers, accurate keywords, and good meta-descriptions. Research SEO best practices if you need a refresher.

Nowadays, reducing repeated content has huge benefits for SEO so we recommend any content review includes a review of duplication. If you have a small site, you can go through your content and SEO manually. Larger projects can utilize tools on the market such as Siteliner or WPOptimize.

While you’re taking a dive into content, don’t forget to review your Google Analytics and understand what content is being used and what isn’t. Google has added many new features to Analytics and Ads, so it’s a good idea to refresh yourself on the updated documentation and new features.

Reporting

A lot of clients with big ecommerce data sets or other applications that collect data benefit from a separate reporting or business analytics tool. A rainy day can be a good time to think about what reports you want on last year’s business, what data will help you plan for the future. End Point has worked with a few different reporting tools that easily add on to your database, like Pentaho or Jasper and those can be really useful.

Documentation

I wouldn’t be a good project manager if I didn’t throw this one in the list. Documentation is so, so important, yet really we can always do more. End Point uses a few different tools, including wikis running MediaWiki and Google Docs, for keeping track of project details. Now’s a good time to set up a nice documentation system or do a big review and make sure everything is updated and back in order. Maybe dream of a vacation you might be able to take when this is over and make sure everything’s ready for you to do that.

Disaster Recovery Tests

For anyone with business-critical infrastructure, you need to ensure you know how to get everything back up and running in the case of a major failure, either with on-premises or cloud hosting. Now’s a good time to clarify with your hosting vendor things like: What are your backups like? What is your disaster recovery plan? What is the timeline for recovering the application in the event of a major failure? If you can, take time to do a simulation and make sure all the pieces are there if they need to be. Simply said, we also need to test our backups in order to ensure that they actually work.

Redesign

If you’ve been meaning to refresh your website, a rainy day is prime time to do it. Designers and developers are looking for projects and you’ll have extra time on your hands to oversee the process, spend time reviewing and testing, and get things done just the way you want.

Automated Testing

Good developers want an automated test suite as part of their application. Not all applications were built with this from the beginning and many didn’t have the budget or time to get it done. With extra time on your hands, this can be a great time to start building your test suite or to improve the coverage of your existing one.

Unit tests in particular are a good place to start. Unit tests are great not only because they help validate software correctness and protect against regressions, but also because they require a well-factored and modular system. This means that, while writing your unit tests, you will often be forced to go back to your application’s code base and refactor it to make it testable, to make it better. Investing in creating a solid unit test suite is a great bang for your buck. You can also look at implementing continuous integration—having a pipeline to let multiple developers deploy code throughout the day and configure your automated tests into the workflow.

Versioning & Deployment Tools

When you’re cleaning house, take a look at your Git version control repository and make sure everything important is in there. We have a few projects that have a main project in Git, but sometimes the smaller projects and one-offs can go astray. This is a good time to get everything organized into one repository, or make sure external repositories are connected and integrated.

Automated DevOps deployment tools can also be nice to work on. Tools like Ansible and Chef can take a lot of time to set up and test, but they have some great time-saving and accuracy advantages down the line. Our in-house security experts also recommend tools like AIDE and OSSEC which automate monitoring file changes daily.

Security Audit and Monitoring

Taking a few days to review your personal security and that of your application is something you should do regularly and now’s a good time to plan for it. Charlie’s got a great security post that’s a good top-level review. For application security, End Point uses some tools for vulnerability scanning. We also have a checklist of basic security items that include password handling, PII data, and other common security holes. For certain projects/clients we must also take HIPAA or PCI DSS compliance into account. Also, don’t neglect to review your TLS status and ensure that web applications run on TLS 1.2 and are TLS 1.3 ready. This also may relate to the underlying operating systems—whether they are able to support the latest TLS version natively.

Optimization and Performance:

Most of the time new features have higher priority than improving the performance of an existing system. It could be the right time to review core functionalities and list out the areas that need improvement in serving a better experience to customers by optimization. The areas can be focused on optimizing code, database queries, image size, data compression over network, adding cache, CDN, and so on. We’ve been moving quite a few clients to the Cloudflare DNS and CDN service and we’ve been really happy with it. Optimization work will definitely influence the customer retention rate which helps to increase profitability long term.

Refactoring

Along the same lines as optimization, code refactoring can have long term gains in performance and ease of future development. Think of this like house cleaning. It is always easier to find any item in the house when things are arranged in an orderly manner. Similarly, the organized and clean code base will play a vital role in future code changes and development, helping to reduce the chances of unexpected bugs, save time making changes at one place and improving code readability. Disciplined refactoring delivers readable, reusable, non-redundant code. Refactoring can be applied to your databases and user interfaces as well.

Want to get started on some background projects for your website? Talk to us today.

Decreasing your website load time

2020-01-07T00:00:00+00:00

Photo by Johan Larsson, used under CC BY 2.0

We live in a competitive world, and the web is no different. Improving latency issues is crucial to any Search Engine Optimization (SEO) strategy, increasing the website’s ranking and organic traffic (visitors from search engines) as a result.

There are many factors that can lead to a faster response time, including optimization of your hosting plan, server proximity to your main traffic source, or utilization of a Content Distribution Network (CDN) if you are expecting visitors on an international level. Some of these solutions and many others can be implemented with only a couple hours of coding.

Inline styles and scripts for the topmost content

Nobody enjoys waiting for long load times. When opening a Google search link, being met with a blank page or a loading GIF for several seconds can seem agonizing. That’s why optimizing the initial rendering of your page is crucial.

The content that immediately appears to the user without the need to scroll down is referred to as “above-the-fold”. This is where your optimization efforts should be aimed. So here’s a plan to load and display as quickly as possible:

First, differentiate the critical styles and scripts you need to render the topmost content, and separate them from the rest of our stylesheet and external script references.
Then, minify the separated styles and scripts, and insert them directly on our page template, right before the closing </head> tag.
Finally, take the stylesheet and scripts link references from the <head> tag (where it’s usually located) and move them to the end of the above-the-fold content.

Now, the user won’t have to wait until all references are loaded before seeing content. Tip: Remember to use the async tag on scripts whenever possible.

example.html:

<head>
    <style>{above-the-fold minified inline styles goes here}</style>
    <script type="text/javascript">{above-the-fold critical scripts goes here}</script>
</head>
<body>
    <div class="above-the-fold-content"></div>
    <link rel="stylesheet" href="{below-the-fold minified stylesheet reference goes here}" />
    <script async src="{below-the-fold minified javascript reference goes here}" />
    <div class="below-the-fold-content"></div>
</body>

Deferred loading of ads

If you’re monetizing your website through Google AdSense or another ad agency that uses scripts to load ads, consider loading ads after the content is fully rendered. This may have a small impact on your revenue, but will improve the user’s experience while optimizing the load speed.

Although there are several ways to achieve this, a technique I have successfully used on many websites is removing all of the script references to Google AdSense until your page is fully loaded. A short delay can be added in order to allow some browsing time before showing ads.

Remove script references, the comment, and extra spaces from your original ad code, to convert it from something like this…

<script async src="https://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js"></script>
<!-- Your ad name -->
<ins class="adsbygoogle"
     style="display:inline-block;width:728px;height:90px"
     data-ad-client="ca-pub-XXXXXXXXXXXXXXXXX"
     data-ad-slot="XXXXXXXXX"></ins>
<script>
     (adsbygoogle = window.adsbygoogle || []).push({});
</script>

… to something like this:

<ins class="adsbygoogle" style="display:inline-block;width:728px;height:90px" data-ad-client="ca-pub-XXXXXXXXXXXXXXXXX" data-ad-slot="XXXXXXXXX"></ins>

A lot shorter, isn’t it? This will create an empty slot in which the ad will be displayed after the page is fully rendered. To accomplish that, a new script like the one below must be added (assuming jQuery is present on the website):

async-ads.js:

// Create a script reference
function addScript(src, async, callback) {
    var js = document.createElement("script");
    js.type = "text/javascript";
    if (async)
        js.async = true;
    if (callback)
        js.onload = callback;
    js.src = src;
    document.body.appendChild(js);
}

// Called when document is ready
$(document).ready(function() {

    // Wait for one second to ensure the user started browsing
    setTimeout(function() {
        (adsbygoogle = window.adsbygoogle || []);
        $("ins.adsbygoogle").each(function() {
            $("<script>(adsbygoogle = window.adsbygoogle || []).push({})</script>").insertAfter($(this));
        });
        addScript("https://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js", true);
    }, 1000);

});

This code will wait for one second once the document is ready, and then leave instructions for Google to push a new ad for each slot. Finally, the AdSense external script will be loaded so that Google will read the instructions and start filling all the slots with ads.

Tip: Enabling balancing from your AdSense dashboard may improve the average load speed as well as the user’s experience since ads will not be shown when the expected revenue is deprecable. And if you’re still on the fence about showing fewer ads, try out an experiment like I did. A balance of 50% worked well in my case, but the right balance will depend on your niche and website characteristics.

Lazy load for images

Because the user will most likely spend the majority of the visit reading above-the-fold content (and may even leave before scrolling at all), loading all images from content below-the-fold at first is impractical. Implementing a custom lazy-loading script (also referred to as deferred-loading or loading-on-scroll) for images can be an easy process. Even though changes to the backend would be likely, the concept of this approach is simple:

Replacing the src attributes from all images that will have lazy loading with a custom attribute such as data-src (this part will probably require backend changes) and set a custom class for them, like lazy.
Creating a script that will copy the data-src content into the src attribute as we scroll through the page.

lazy-load.js:

;(function($) {

    $.fn.lazy = function(threshold, callback) {
        var $w = $(window),
        th = threshold || 0,
        attrib = "data-src",
        images = this,
        loaded;
        this.one("lazy", function() {
            var source = this.getAttribute(attrib);
            source = source || this.getAttribute("data-src");
            if (source) {
                this.setAttribute("src", source);
                if (typeof callback === "function") callback.call(this);
            }
        });

        function lazy() {
            var inview = images.filter(function() {
                var $e = $(this);
                if ($e.is(":hidden")) return;
                var wt = $w.scrollTop(),
                wb = wt + $w.height(),
                et = $e.offset().top,
                eb = et + $e.height();
                return eb >= wt - th && et <= wb + th;
            });
            loaded = inview.trigger("lazy");
            images = images.not(loaded);
        }

        $w.scroll(lazy);
        $w.resize(lazy);
        lazy();
        return this;
    };

})(window.jQuery);

$(document).ready(function() {
    $('.lazy').each(function () {
        $(this).lazy(0, function() {
        $(this).load(function() {
            this.style.opacity = 1;
        });
    });
});

// Set the correct attribute when printing
var beforePrint = function() {
    $("img.lazy").each(function() {
        $(this).trigger("lazy");
        this.style.opacity = 1;
    });
};
if (window.matchMedia) {
    var mediaQueryList = window.matchMedia('print');
    mediaQueryList.addListener(function(mql) {
        if (mql.matches)
            beforePrint();
    });
}
window.onbeforeprint = beforePrint;

This script will search for all <img> tags with class lazy, and change the data-src attribute to the src attribute once the image becomes visible due to scrolling. It also includes some additional logic to set the src attribute before printing the page.

Server-side caching

Instead of performing all the backend rendering calculations every time, server-side caching allows you to output the same content to the clients over a period of time from a temporary copy of the response. This not only results in a decreased response time but also saves some resources on the server.

There are several ways to enable server-side caching, depending on factors such as the backend language and hosting platform (e.g. Windows/IIS vs. Linux/Apache), among other things. For this example, we will use ASP.NET (C#) since I’m mostly a Windows user.

The best and most efficient way to do this is by adding a declaration in the top of our ASP.NET page:

<%@ OutputCache Duration="10" VaryByParam="id;date" %>

This declaration is telling the compiler that we want to cache the output from the server for 10 minutes, and we will save different versions based on the id and date URL parameters. So pages like:

https://www.your-url.com/cached-page/?id=1&date=2020-01-01
https://www.your-url.com/cached-page/?id=2&date=2020-01-01
https://www.your-url.com/cached-page/?id=2&date=2020-02-01

will be saved and then served from different cache copies. If we only set the id parameter as a source for caching, pages with different dates will be served from the same cache source (this can be useful as the date parameter is only evaluated on frontend scripts and ignored in the backend).

There are other configurations in ASP.NET to set our output cache policy. The output can be set to be based on the browser, the request headers, or even custom strings. This page has more useful information on this subject.

GZip compression

GZip compression—when the client supports it—allows compressing the response before sending it over the network. In this way, more than 70% of the bandwidth can be saved when loading the website. Enabling GZip compression for dynamic and static content on a Windows Server with IIS is simple: Just go to the “Compression” section on the IIS Manager and check the options “Enable dynamic/static content compression”.

However, if you are running an ASP.NET MVC/WebForms website, this won’t be enough. For all backend responses to be compressed before sending them to the client, some custom code will also need to be added to the global.asax file in the website root:

global.asax:

<%@ Application Language="C#" %>

<script runat="server">

    void Application_PreRequestHandlerExecute(object sender, EventArgs e)
    {
        HttpApplication app = sender as HttpApplication;
        string acceptEncoding = app.Request.Headers["Accept-Encoding"];
        System.IO.Stream prevUncompressedStream = app.Response.Filter;

        if (app.Context.CurrentHandler == null)
            return;

        if (!(app.Context.CurrentHandler is System.Web.UI.Page ||
            app.Context.CurrentHandler.GetType().Name == "SyncSessionlessHandler") ||
            app.Request["HTTP_X_MICROSOFTAJAX"] != null)
            return;

        if (acceptEncoding == null || acceptEncoding.Length == 0)
            return;

        if (Request.ServerVariables["SCRIPT_NAME"].ToLower().Contains(".axd")) return;
        if (Request.ServerVariables["SCRIPT_NAME"].ToLower().Contains(".js")) return;
        if (Request.QueryString.ToString().Contains("_TSM_HiddenField_")) return;

        acceptEncoding = acceptEncoding.ToLower();

        if (acceptEncoding.Contains("deflate") || acceptEncoding == "*")
        {
            app.Response.Filter = new System.IO.Compression.DeflateStream(prevUncompressedStream,
                System.IO.Compression.CompressionMode.Compress);
            app.Response.AppendHeader("Content-Encoding", "deflate");
        }
        else if (acceptEncoding.Contains("gzip"))
        {
            app.Response.Filter = new System.IO.Compression.GZipStream(prevUncompressedStream,
                System.IO.Compression.CompressionMode.Compress);
            app.Response.AppendHeader("Content-Encoding", "gzip");
        }
    }

</script>

To make sure our code is working properly, an external tool like this will inform you if GZip is enabled or not.

Summary

While there are many ways of decreasing the load time of a website, most are common and expensive. However, with a few minor tweaks, we can offer a better user experience in addition to improve our position in the search engine results. Every bit of optimization counts towards the goal with SEO. Load time is a very important factor (to both the developer and the user), especially on mobile platforms where users expect to get what they want instantly.

The image below is a Google Analytics report from one of my websites where, over several months, I implemented most of these formulas. A month ago, I made the latest change of deferring ad loading, which had an observable impact on the average loading speed of the page:

Do you have any other page load optimization techniques? Leave a comment below!

Extensible Binary Encoding with CBOR

2019-03-18T00:00:00+00:00

CBOR is a relatively new IETF draft standard extensible binary data format. Compared to similar formats like MessagePack and BSON, CBOR was developed from the ground up with clear goals:

unambiguous encoding of most common data formats from Internet standards

code compactness for encoder and decoder

no schema description needed

reasonably compact serialization

applicability to constrained and unconstrained applications

good JSON conversion

extensibility

—RFC 7049 Appendix E, Copyright © 2013 IETF Trust, Bormann & Hoffman

In the context of data storage and messaging, most developers can relate to CBOR as a binary drop-in replacement for JSON. While CBOR doesn’t share the human readability of JSON, it can efficiently and unambiguously encode types of data that JSON struggles with. CBOR can also be extended with tags to optimize serialization beyond its standard primitives.

Encoding Binary Data

JSON is a ubiquitous data format for web and beyond, for many good reasons, but encoding blobs of binary data is an area where JSON falters. For example, if you are designing a JSON protocol to wrap the storage or transfer of arbitrary objects, your options are:

Require that all input data can be represented as JSON. When possible this is potentially a reasonable solution, but limits the types of data that can be encoded. Notable exceptions include most popular image encodings, excluding SVG.
Base64 encode any binary data values to a string. This can encode any binary data, but increases the size of the data by a minimum of 1/3, incurs encoding and decoding cost, and requires magic to indicate that the string is Base64 encoded.
Encode the bytes as an array of numbers or a hex string. These are probably not things you should do, but it seemed worth mentioning that these techniques increase the size of the data by anywhere from 2x to 5x and also require magic to indicate that the data is really binary.

With CBOR, binary blobs of any length are supported out of the box and are encoded 1:1. Encoding and decoding CBOR byte strings is extremely fast, even in higher-level languages. In the case of JavaScript, CBOR byte strings can be transcoded to and from the very fast and efficient Uint8Array.

For example, let’s try encoding a simple object with two fields: “name” and “data”. An abstract view of this object:

name: "Strawberry Pie"
data: <00 01 02 03 04 05 06 07 08 09>

A C struct of this raw data would be absolute minimum of 24 bytes.

If you Base64 the data into JSON, your output might look like this:

{"name":"Strawberry Pie","data":"AAECAwQFBgcICQ=="}

Length: 51 bytes.

This illustrates the “magic” problem with Base64 encoding binary data in JSON. Unless you have a JSON schema or special protocol fields, the decoder has no indication that the data in this object needs to be Base64 decoded, because it looks like a string. It is not self-describing.

The CBOR representation of the same input data, as a hex string:

a2646e616d656e5374726177626572727920506965696a7065675f646174614a00010203040506070809

Whoa, there. What do all these numbers and letters mean? I miss my JSON! The Node cbor package has a handy cbor2comment tool to annotate this hex string for us.

  a2                -- Map, 2 pairs
    64              -- String, length: 4
      6e616d65      -- {Key:0}, "name"
    6e              -- String, length: 14
      5374726177626572727920506965 -- {Val:0}, "Strawberry Pie"
    64              -- String, length: 4
      64617461 -- {Key:1}, "data"
    4a              -- Bytes, length: 10
      00010203040506070809 -- {Val:1}, 00010203040506070809

Length: 35 bytes.

Now let’s benchmark JSON against CBOR using real JPEG data. For this test I used a modified version of cbor-js, a library compatible with both Node and browsers, and encoded a 910,226 byte JPEG of strawberry rhubarb pie.

|                           | JSON      | CBOR    |
| :------------------------ | --------: | ------: |
| Median encoding time (ms) | 3.983     | 0.538   |
| Median decoding time (ms) | 3.151     | 0.006   |
| Encoded size (bytes)      | 1,213,676 | 910,262 |

As the numbers show, CBOR is both faster and more concise for this particular data. Also, CBOR pie tastes better.

Food always looks good in pictures!

Optimizing CBOR with Tags

In the case of encoding homogeneous numeric arrays, CBOR encoders can struggle with optimizing the packing of the data. For example, if you have an array of floating point numbers in a higher-level language like Python or JavaScript, the CBOR encoder implementation won’t necessarily determine how many bits are required to encode the numbers, defaulting to the largest available. Additionally, each value in the array will be individually described as a floating point number. This increases the cost and size of the data considerably.

CBOR has an answer to this problem. The Draft Typed Array Tags spec includes tags specifying typed arrays which happen to match JavaScript TypedArray flavors.

For example, let’s say you have a Float32Array with a few values:

const floats = new Float32Array([1.234567, 2.345678, 3.456789]);

JSON has a really funny way of encoding a Float32Array, lookit all those bytes:

{"0":1.2345670461654663,"1":2.3456780910491943,"2":3.456789016723633}

Length: 69 bytes.

If we were to help the encoder by sending it a regular Array it would still be pretty verbose, but the precision we weren’t using is truncated, so the overall length will vary wildly depending on the values in the Array:

[1.234567,2.345678,3.456789]

Length: 28 bytes.

The well-meaning Node cbor library can encode the Float32Array directly, but doesn’t try to optimize for size:

  83                -- Array, 3 items
    fb              -- Float, next 8 bytes
      3ff3c0c960000000 -- [0], 1.2345670461654663
    fb              -- Float, next 8 bytes
      4002c3f2e0000000 -- [1], 2.3456780910491943
    fb              -- Float, next 8 bytes
      400ba78100000000 -- [2], 3.456789016723633

Length: 28 bytes.

Look at all that wasted precision, 64 bits for each 32-bit float, this just won’t do! The CBOR spec allows you to “tag” data for special treatment. According to the list of registered CBOR tags, “IEEE 754 binary32, little endian, Typed Array” is tag 85. The consecutive bytes of the three numbers follow.

  d8                --  next 1 byte
    55              -- Tag #85
      4c            -- Bytes, length: 12
        4b069e3f971f1640083c5d40 -- 4b069e3f971f1640083c5d40

Length: 15 bytes.

The contained 12 byte string is equivalent to the values in the ArrayBuffer underneath our Float32Array.

The CBOR decoder will spit out the tag alongside its associated blob, so our optimization is self-described. Now we need to ensure that a byte string with this tag is correctly converted to the typed array, minding endianness. In a mature JavaScript implementation, this is both easy and very fast. Because we encoded the values of the ArrayBuffer underneath a Float32Array, we can construct a new Float32Array from the ArrayBuffer.

const floats = new Float32Array(bytes);  // assuming platform and bytes are same endianness!

This is a very scalable way to read and write arrays of numeric values into JavaScript.

When to Use CBOR

As King Crimson teaches us, “it doesn’t mean you should / just because you can.”

I’ve found CBOR useful as an alternative to JSON for large (>2kB) chunks of raw data or numeric arrays. In many other cases JSON is equivalent or superior, because most languages have native JSON encoding and decoding in their standard libraries. CBOR encoders are not always so optimized, but the gap is closing with wider adoption. Run your own tests and benchmarks on real data against real libraries before deciding to use CBOR.

Many readers will recognize that schemaful formats such as protocol buffers are an endgame for structured data. If the infrastructure demands make sense for your application, this is also a good way to go. If you’re working with an application that already uses JSON, the difference in development and maintenance costs of porting it to CBOR or protobuf-likes should be measured against the size and performance gains of each approach.

It’s tempting to let numbers alone decide what format to use, but don’t underestimate the value of JSON’s human readability for structured data. The value of human readability diminishes quickly when dealing with large numeric arrays or binary blobs, so once again CBOR is an appealing choice for these data.

Roundup of some useful websites

2018-12-21T00:00:00+00:00

The world is a big place, and the Internet has gotten pretty big too. There are always new projects being created, and I want to share some useful and interesting ones from my growing list:

Squoosh image compressor

Squoosh, hosted at squoosh.app, is an open source in-browser tool for experimenting with image compression, made by the Chrome development team.

With Squoosh you can load an image in your browser, convert it to different image file formats (JPEG, WebP, PNG, BMP) using various compression algorithms and settings, and compare the result side-by-side with either the original image or the image compressed using other options.

The screenshot above demonstrates Squoosh running in Firefox 64 on Linux. Click on it to see a larger, lossless PNG screenshot. The photo was taken by my son Phin in northern Virginia, and is a typical imperfect mobile phone photo. On the left is the original, and on the right I am showing how bad gradients in the sky can look when compressed too much—maybe a quality level of 12 (out of 100) was too low. It does make for a very compact file size, though. 😄

Squoosh’s interface has a convenient slider bar so you can compare any part of the two versions of the image side by side. You can zoom and pan the image as well.

It is neat to see JavaScript tools (in this case TypeScript specifically) doing work in the browser that has traditionally been done by native apps.

Nerd Fonts

If you want access to an amazing number of symbols in a font, check out nerdfonts.com. There you can mix and match symbols from many popular developer-oriented fonts such as Font Awesome, Powerline Symbols, Material Design, etc.

I probably should have chosen some fun symbols to demonstrate it here, but I could tell that was a rabbit hole I would not soon emerge from!

glot.io code pastebin

There are many public pastebins these days, but glot.io distinguishes itself by allowing you to run real code on their server in nearly 50 languages.

It offers both public and private pastes, has an API, and is open source.

Firefox Send

Firefox Send at send.firefox.com is a browser-based service for securely sharing files temporarily, for only one download during a maximum of 24 hours.

Handy for keeping unwanted bloat out of email, chat, or shared file storage for ephemeral files.

Similarly, transfer.sh is a terminal-based file upload and download tool.

As a command-line tool it easily integrates with other standard tools, so you can pipe output from other programs directly to it. If you have sensitive data to share you don’t need to trust the service—you can pipe your data through gpg or some other encryption tool before it leaves your computer.

transfer.sh is open source and can be self-hosted too.

It even has a Tor onion service so uploads and/or downloads can be as private as possible in hostile environments.

Doing what you don’t want to do

And finally, some timeless tips for making our human “software” work.

Often just one or two annoying little things can block us from making progress on larger projects that overall we really enjoy. How can you motivate yourself to push ahead when you have work that needs to be done, but you don’t want to do it?

Read the brief but helpful article 10 Ways to Do What You Don’t Want to Do by Leo Babauta to get some good ideas. A few of the points mentioned especially resonate with me:

Why do I need to do it?
What is stopping me?
Embrace that it won’t be fun and do it anyway.
Set constraints.

Then do at least a little bit of the work to get started. As our co-worker Mike Heins has said to me on a few occasions over the years, you’ll never finish until you start.

Learning from data basics II: simple Bayesian Networks

2016-04-12T00:00:00+00:00

In my last article I presented an approach that simplifies computations of very complex probability models. It makes these complex models viable by shrinking the amount of needed memory and improving the speed of computing probabilities. The approach we were exploring is called the Naive Bayes model.

The context was the e-commerce feature in which a user is presented with the promotion box. The box shows the product category the user is most likely to buy.

Though the results we got were quite good, I promised to present an approach that gives much better ones. While the Naive Bayes approach may not be acceptable in some scenarios due to the gap between approximated and real values, the approach presented in this article will make this distance much, much smaller.

Naive Bayes as a simple Bayesian Network

When exploring the Naive Bayes model, we said that there is a probabilistic assumption the model makes in order to simplify the computations. In the last article I wrote:

The Naive Bayes assumption says that the distribution factorizes the way we did it only if the features are conditionally independent given the category.

Expressing variable dependencies as a graph

Let’s imagine the visual representation of the relations between the random variables in the Naive Bayes model. Let’s make it into a directed acyclic graph. Let’s mark the dependence of one variable on another as a graph edge from the parent node pointing to it’s dependent node.

Because of the assumption the Naive Bayes model enforces, its structure as a graph looks like the following:

You can notice there are no lines between all the “evidence” nodes. The assumption says that knowing the category, we have all needed knowledge about every single evidence node. This makes category the parent of all the other nodes. Intuitively, we can say that knowing the class (in this example, the category) we know everything about all features. It’s easy to notice that this assumption doesn’t hold in this example.

In our fake data generator, we made it so that e.g. relationship status depends on age. We’ve also made the category depend on sex and age directly. This way we can’t say that knowing category we know everything about e. g. age. The random variables age and sex are not independent even if we know the value of category. It is clear that the above graph does not model the dependency relationships between these random variables.

Let’s draw a graph that represents our fake data model better:

The combination of a graph like the one above and the probability distribution that follows the independencies it describes are known as a Bayesian Network.

Using the graph representation in practice - the chain rule for Bayesian Networks

The fact that our distribution is part of the Bayesian Network, allows us to use the formula for simplifying the distribution itself. The formula is called the chain rule for Bayesian Networks and for our particular example looks like the following:

p(cat, sex, age, rel, loc) = p(sex) * p(age) * p(loc) * p(rel | age) * p(cat | sex, age)

You can notice that the equation is just a product of a number of factors. There’s one factor for each random variable. The factors for variables that in the graph don’t have any parents are expressed as p(var) while those that do are expressed as p(var | par) or p(var | par1, par2…).

Notice that the Naive Bayes model fits perfectly into this equation. If you were to take the first graph presented in this article—for the Naive Bayes, and use the above equation, you’d get exactly the formula we used in the last article.

Coding the updated probabilistic model

Before going further, I strongly advise you to make sure you read the previous article - about the Naive Bayes model - to fully understand the classes used in the code in this section.

Let’s take our chain rule equation and simplify it:

p(cat, sex, age, rel, loc) = p(sex) * p(age) * p(loc) * p(rel | age) * p(cat | sex, age)

Again a conditional distrubution can be expressed as:

p(a | b) = p(a, b) / p(b)

This gives us:

p(cat, sex, age, rel, loc) = p(sex) * p(age) * p(loc) * (p(rel, age)/ p(age)) * (p(cat, sex, age) / p(sex, age))

We can easily factor out the p(age) with:

p(cat, sex, age, rel, loc) = p(sex) * p(loc) * p(rel, age) * (p(cat, sex, age) / p(sex, age))

Let’s define needed random variables and factors:

category = RandomVariable.new :category, [ :veggies, :snacks, :meat, :drinks, :beauty, :magazines ]
age      = RandomVariable.new :age,      [ :teens, :young_adults, :adults, :elders ]
sex      = RandomVariable.new :sex,      [ :male, :female ]
relation = RandomVariable.new :relation, [ :single, :in_relationship ]
location = RandomVariable.new :location, [ :us, :canada, :europe, :asia ]

loc_dist     = Factor.new [ location ]
sex_dist     = Factor.new [ sex ]
rel_age_dist = Factor.new [ relation, age ]
cat_age_sex_dist = Factor.new [ category, age, sex ]
age_sex_dist = Factor.new [ age, sex ]

full_dist = Factor.new [ category, age, sex, relation, location ]

The learning part is as trivial as in the Naive Bayes case. The only difference is the set of distributions involved:

Model.generate(1000).each do |user|
  user.baskets.each do |basket|
    basket.line_items.each do |item|
      loc_dist.observe! location: user.location
      sex_dist.observe! sex: user.sex
      rel_age_dist.observe! relation: user.relationship, age: user.age
      cat_age_sex_dist.observe! category: item.category, age: user.age, sex: user.sex
      age_sex_dist.observe! age: user.age, sex: user.sex
      full_dist.observe! category: item.category, age: user.age, sex: user.sex,
        relation: user.relationship, location: user.location
    end
  end
end

The inference part is also very similar to the one from the previous article. Here too the only difference are the distributions involved:

infer = -> (age, sex, rel, loc) do
  all = category.values.map do |cat|
    pl  = loc_dist.value_for location: loc
    ps  = sex_dist.value_for sex: sex
    pra = rel_age_dist.value_for relation: rel, age: age
    pcas = cat_age_sex_dist.value_for category: cat, age: age, sex: sex
    pas = age_sex_dist.value_for age: age, sex: sex
    { category: cat, value: (pl * ps * pra * pcas) / pas }
  end

  all_full = category.values.map do |cat|
    val = full_dist.value_for category: cat, age: age, sex: sex,
            relation: rel, location: loc
    { category: cat, value: val }
  end

  win      = all.max      { |a, b| a[:value] <=> b[:value] }
  win_full = all_full.max { |a, b| a[:value] <=> b[:value] }

  puts "Best match for #{[ age, sex, rel, loc ]}:"
  puts "   #{win[:category]} => #{win[:value]}"
  puts "Full pointed at:"
  puts "   #{win_full[:category]} => #{win_full[:value]}\n\n"
end

The results

Now let’s run the inference procedure with the same set of examples as in the previous post to compare the results:

infer.call :teens, :male, :single, :us
infer.call :young_adults, :male, :single, :asia
infer.call :adults, :female, :in_relationship, :europe
infer.call :elders, :female, :in_relationship, :canada

Which yields:

Best match for [:teens, :male, :single, :us]:
   snacks => 0.020610837341908994
Full pointed at:
   snacks => 0.02103999999999992

Best match for [:young_adults, :male, :single, :asia]:
   meat => 0.001801062449999991
Full pointed at:
   meat => 0.0010700000000000121

Best match for [:adults, :female, :in_relationship, :europe]:
   beauty => 0.0007693377820183494
Full pointed at:
   beauty => 0.0008300000000000074

Best match for [:elders, :female, :in_relationship, :canada]:
   veggies => 0.0024346445741176875
Full pointed at:
   veggies => 0.0034199999999999886

Just as with using the Naive Bayes, we got correct values for all cases. When you look closer though, you can notice that the resulting probability values were much closer to the original, full distribution ones. The approach we took here makes the values differ only a couple times in 10000. That result could make a difference in the e-commerce shop from the example if it were visited by millions of customers each month.

Learning from data basics: the Naive Bayes model

2016-03-23T00:00:00+00:00

Have you ever wondered what is the machinery behind some of the algorithms for doing seemingly very intelligent tasks? How is it possible that the computer program can recognize faces in photos, turn an image into a text or even classify some emails as legitimate or as spam?

Today, I’d like to present one of the simplest models for performing classification tasks. The model enables extremely fast execution, making it very practical in many use cases. The example I’ll choose will enable us to extend the discussion about the most optimal approach to another blog post.

The problem

Imagine that you’re working on an e-commerce store for your client. One of the requirements is to present the currently logged in user with a “promotion box” somewhere on the page. The goal is to maximize our chances of having the user put the product from the box into the basket. There’s one promotional box and a couple of different categories of products to choose the actual product from.

Thinking about the solution—using probability theory

One of the obvious directions we may want to turn towards is to use probability theory. If we could collect the data about the user’s previous choices and his or her characteristics, we can use probability to select the product category best suited for the current user. We would then choose a product from this category that currently has an active promotion.

Quick theory refresher for programmers

As we’ll be exploring the probability approaches using Ruby code, I’d like to very quickly walk you through some of the basic concepts we will be using from now on.

Random variables

The simplest probability scenario many of us are already accustomed with is the coin toss results distribution. Here we’re throwing the coin, noting whether we get heads or tails. In this experiment, we call “got heads” and “got tails” probability events. We can also shift the terminology a bit by calling them: two values of the “toss result” random variable.

So in this case we’d have a random variable—let’s call it T (for “toss”) that can take values of: “heads” or “tails”. We then define the probability distribution P(T) as a function from the random variable value to a real number between 0 and 1 inclusively on both sides. In real world the probability values after e. g 10000 tosses might look like the following:

+-------+---------------------+
| toss  | value               |
+-------+---------------------+
| heads | 0.49929999999999947 |
| tails |   0.500699999999998 |
+-------+---------------------+

These values are nearing 0.5 more and more with the greater number of tosses.

Factors and probability distributions

We’ve shown a simple probability distribution. To ease the comprehension of the Ruby code we’ll be working with, let me introduce the notion of the factor. We called the “table” from the last example a probability distribution. The table represented a function from a random variable’s value to a real number between [0, 1]. The factor is a generalization of that notion. It’s a function from the same domain, but returning any real number. We’ll explore the usability of this notion in some of our next articles.

The probability distribution is a factor that adds two constraints:

its values are always in the range [0, 1] inclusively
the sum of all it’s values is exactly 1

Simple Ruby modeling of random variables and factors

We need to have some ways of computing probability distributions. Let’s define some simple tools we’ll be using in this blog series:

# Let's define a simple version of the random variable
# - one that will hold discrete values
class RandomVariable
  attr_accessor :values, :name

  def initialize(name, values)
    @name = name
    @values = values
  end
end

# The following class really represents here a probability
# distribution. We'll adjust it in the next posts to make
# it match the definition of a "factor". We're naming it this
# way right now as every probability distribution is a factor
# too.
class Factor
  attr_accessor :_table, :_count, :variables

  def initialize(variables)
    @_table = {}
    @_count = 0.0
    @variables = variables
    initialize_table
  end

  # We're choosing to represent the factor / distribution
  # here as a table with value combinations in one column
  # and probability values in another. Technically, we're using
  # Ruby's Hash. The following method builds the the initial hash
  # with all the possible keys and values assigned to 0:
  def initialize_table
    variables_values = @variables.map do |var|
      var.values.map do |val|
        { var.name.to_sym => val }
      end.flatten
    end # [ [ { name: value } ] ]   
    @_table = variables_values[1..(variables_values.count)].inject(variables_values.first) do |all_array, var_arrays|
      all_array = all_array.map do |ob|
        var_arrays.map do |var_val|
          ob.merge var_val
        end
      end.flatten
      all_array
    end.inject({}) { |m, item| m[item] = 0; m }
  end

  # The following method adjusts the factor by adding information
  # about observed combination of values. This in turn adjusts probability
  # values for all the entries:
  def observe!(observation)
    if !@_table.has_key? observation
      raise ArgumentError, "Doesn't fit the factor - #{@variables} for observation: #{observation}"
    end

    @_count += 1

    @_table.keys.each do |key|
      observed = key == observation
      @_table[key] = (@_table[key] * (@_count == 0 ? 0 : (@_count - 1)) + 
       (observed ? 1 : 0)) / 
         (@_count == 0 ? 1 : @_count)
    end

    self
  end

  # Helper method for getting all the possible combinations
  # of random variable assignments
  def entries
    @_table.each
  end

  # Helper method for testing purposes. Sums the values for the whole
  # distribution - it should return 1 (close to 1 due to how computers
  # handle floating point operations)
  def sum
    @_table.values.inject(:+)
  end

  # Returns a probability of a given combination happening
  # in the experiment
  def value_for(key)
    if @_table[key].nil?
      raise ArgumentError, "Doesn't fit the factor - #{@varables} for: #{key}"
    end
    @_table[key]
  end

  # Helper method for testing purposes. Returns a table object
  # ready to be printed to stdout. It shows the whole distribution
  # as a table with some columns being random variables values and
  # the last one being the probability value
  def table
    rows = @_table.keys.map do |key|
      key.values << @_table[key]
    end
    table = Terminal::Table.new rows: rows, headings: ( @variables.map(&:name) << "value" )
    table.align_column(@variables.count, :right)
    table
  end

  protected

  def entries=(_entries)
    _entries.each do |entry|
      @_table[entry.keys.first] = entry.values.first
    end
  end

  def count
    @_count
  end

  def count=(_count)
    @_count = _count
  end
end

Notice that we’re using here the terminal-table gem as a helper for printing out the factors in an easy to grasp fashion. You’ll need the following requires:

require 'rubygems'
require 'terminal-table'

The scenario setup

Let’s imagine that we have the following categories to choose from:

category = RandomVariable.new :category, [ :veggies, :snacks, :meat, :drinks, :beauty, :magazines ]

And the following user features on each request:

age      = RandomVariable.new :age,      [ :teens, :young_adults, :adults, :elders ]
sex      = RandomVariable.new :sex,      [ :male, :female ]
relation = RandomVariable.new :relation, [ :single, :in_relationship ]
location = RandomVariable.new :location, [ :us, :canada, :europe, :asia ]

Let’s define the data model that resembles logically the one we could have in our real e-commerce application:

class LineItem
  attr_accessor :category

  def initialize(category)
    self.category = category
  end
end

class Basket
  attr_accessor :line_items

  def initialize(line_items)
    self.line_items = line_items
  end
end

class User
  attr_accessor :age, :sex, :relationship, :location, :baskets

  def initialize(age, sex, relationship, location, baskets)
    self.age = age
    self.sex = sex
    self.relationship = relationship
    self.location = location
    self.baskets = baskets
  end
end

We want to utilize a user’s baskets in order to infer the most probable value for a category, given a set of user’s features. In our example, we can imagine that we’re offering authentication via Facebook. We can grab info about a user’s sex, location, age and whether she/he is in relationship or not. We want to find a category that’s being chosen the most by users with a given set of features.

As we don’t have any real data to play with, we’ll need a generator to create fake data of certain characteristics. Let’s first define a helper class with a method, that will allow us to choose a value out of a given list of options along with their weights:

class Generator
  def self.pick(options)
    items = options.inject([]) do |memo, keyval|
      key, val = keyval
      memo << Array.new(val, key)
      memo
    end.flatten
    items.sample
  end
end

With all the above we can define a random data generation model:

class Model

  # Let's generate `num` users (1000 by default)
  def self.generate(num = 1000)
    num.times.to_a.map do |user_index|
      gen_user
    end
  end

  # Returns a user with randomly selected traits and baskets
  def self.gen_user
    age = gen_age
    sex = gen_sex
    rel = gen_rel(age)
    loc = gen_loc
    baskets = gen_baskets(age, sex)

    User.new age, sex, rel, loc, baskets
  end

  # Randomly select a sex with 40% chance for getting a male
  def self.gen_sex
    Generator.pick male: 4, female: 6
  end

  # Randomly select an age with 50% chance for getting a teen
  # (among other options and weights)
  def self.gen_age
    Generator.pick teens: 5, young_adults: 2, adults: 2, elders: 1
  end

  # Randomly select a relationship status.
  # Depend the chance of getting a given option on the user's age
  def self.gen_rel(age)
    case age
      when :teens        then Generator.pick single: 7, in_relationship: 3
      when :young_adults then Generator.pick single: 4, in_relationship: 6
      else                    Generator.pick single: 8, in_relationship: 2
    end
  end

  # Randomly select a location with 40% chance for getting a united states
  # (among other options and weights)
  def self.gen_loc
    Generator.pick us: 4, canada: 3, europe: 1, asia: 2
  end

  # Randomly select 20 basket line items.
  # Depend the chance of getting a given option on the user's age and sex
  def self.gen_items(age, sex)
    num = 20

    num.times.to_a.map do |i|
      if (age == :teens || age == :young_adults) && sex == :female
        Generator.pick veggies: 1, snacks: 3, meat: 1, drinks: 1, beauty: 9, magazines: 6
      elsif age == :teens  && sex == :male
        Generator.pick veggies: 1, snacks: 6, meat: 4, drinks: 1, beauty: 1, magazines: 4
      elsif (age == :young_adults || age == :adults) && sex == :male
        Generator.pick veggies: 1, snacks: 4, meat: 6, drinks: 6, beauty: 1, magazines: 1
      elsif (age == :young_adults || age == :adults) && sex == :female
        Generator.pick veggies: 4, snacks: 4, meat: 2, drinks: 1, beauty: 6, magazines: 3
      elsif age == :elders && sex == :male
        Generator.pick veggies: 6, snacks: 2, meat: 2, drinks: 2, beauty: 1, magazines: 1
      elsif age == :elders && sex == :female
        Generator.pick veggies: 8, snacks: 1, meat: 2, drinks: 1, beauty: 4, magazines: 1
      else
        Generator.pick veggies: 1, snacks: 1, meat: 1, drinks: 1, beauty: 1, magazines: 1
      end
    end.map do |cat|
      LineItem.new cat
    end
  end

  # Randomly select 5 baskets depending the traits of the basket on user
  # age and sex
  def self.gen_baskets(age, sex)
    num = 5

    num.times.to_a.map do |i|
      Basket.new gen_items(age, sex)
    end
  end
end

Where is the complexity?

The approach described above doesn’t seem that exciting or complex. Usually reading about probability theory applied in the field of machine learning requires going through quite a dense set of mathematical notions. The field is also being actively worked on by researchers. This implies a huge complexity—certainly not the simple definition of probability that we got used to in high school.

The problem becomes a bit more complex if you consider efficiency of computing the probabilities. In our example, the joined probability distribution—to fully describe the scenario—needs to specify probability values for 383 cases:

p(:veggies, :teens, :male, :single, :us) # one of 384 combinations

Given that the probability distributions have to sum up to 1, the last case can be fully inferred from the sum of all the others. This means that we need 6 * 4 * 2 * 2 * 4 - 1 = 383 parameters in the model: 6 categories, 4 age classes, 2 sexes, 2 relationship kinds and 4 locations. Imagine adding one additional, 4 valued feature (a season). This would grow our number of parameters to 1535. And this is a very simple training example. We could have a model with close to 100 different features. The number of parameters would clearly be unmanageable even on the biggest servers we could put them in. This approach would also make it very painful to add additional features to the model.

Very simple but powerful optimization: The Naive Bayes model

In this section I’m going to present you with an equation we’ll be working with when optimizing our example. I’m not going to explain the mathematics behind it as you can easily read about them on e. g. Wikipedia.

The approach is called the Naive Bayes model. It is being used e .g. in spam filters. It also has been used in the past in medical diagnosis field.

It allows us to present the full probability distribution as a product of factors:

p(cat, age, sex, rel, loc) == p(cat) * p(age | cat) * p(sex | cat) * p(rel | cat) * p(loc | cat)

Where e. g. p(age | cat) represents the probability of a user being a certain age given that this user selects cat products most frequently. This is called the “posterior probability”. The above equation states that we can simplify the distribution to be a product of some number of much more easily manageable factors.

The category from our example is often called a class and the rest of random variables in the distribution are often called features.

In our example, the number of parameters we’ll need to manage when presenting the distribution in this form drops to:

(6 - 1) + (6 * 4 - 1) + (6 * 2 - 1) + (6 * 2 - 1) + (6 * 4 - 1) == 73

That’s just around 19% of the original amount! Also, adding another variable (season) would only add 23 new parameters (compared to 1152 in the full distribution case).

The Naive Bayes model limits the number of parameters we have to manage but it comes with very strong assumptions about the variables involved: in our example, that the user features are conditionally independent given the resulting category. Later on I’ll show why this isn’t true in this case even though the results will still be quite okay.

Implementing the Naive Bayes model

As we now have all the tools we need, let’s get back to the probability theory to figure out how best to model the Naive Bayes in terms of the Ruby blocks we now have.

The approach says that under the assumptions we discussed we can approximate the original distribution to be the product of factors:

p(cat, age, sex, rel, loc) = p(cat) * p(age | cat) * p(sex | cat) * p(rel | cat) * p(loc | cat)

Given the definition of the conditional probability we have that:

p(a | b) = p(a, b) / p(b)

Thus, we can express the approximation as:

p(cat, age, sex, rel, loc) = p(cat) * ( p(age, cat) / p(cat) ) * ( p(sex, cat) / p(cat) ) * ( p(rel, cat) / p(cat) ) * ( p(loc, cat) / p(cat) )

And then simplify it even further as:

p(cat, age, sex, rel, loc) = p(age, cat) * ( p(sex, cat) / p(cat) ) * ( p(rel, cat) / p(cat) ) * ( p(loc, cat) / p(cat) )

Let’s define all the factors we will need:

cat_dist     = Factor.new [ category ]
age_cat_dist = Factor.new [ age, category ]
sex_cat_dist = Factor.new [ sex, category ]
rel_cat_dist = Factor.new [ relation, category ]
loc_cat_dist = Factor.new [ location, category ]

Also, we want a full distribution to compare the results:

full_dist = Factor.new [ category, age, sex, relation, location ]

Let’s generate 1000 random users and looping through them and their baskets - adjust probability distributions for combinations of product categories and user traits:

Model.generate(1000).each do |user|
  user.baskets.each do |basket|
    basket.line_items.each do |item|
      cat_dist.observe! category: item.category
      age_cat_dist.observe! age: user.age, category: item.category
      sex_cat_dist.observe! sex: user.sex, category: item.category
      rel_cat_dist.observe! relation: user.relationship, category: item.category
      loc_cat_dist.observe! location: user.location, category: item.category
      full_dist.observe! category: item.category, age: user.age, sex: user.sex,
        relation: user.relationship, location: user.location
    end
  end
end

We can now print the distributions as tables to have an insight about the data:

[ cat_dist, age_cat_dist, sex_cat_dist, rel_cat_dist, 
  loc_cat_dist, full_dist ].each do |dist|
    puts dist.table
    # Let's print out the sum of all entries to ensure the
    # algorithm works well:
    puts dist.sum
    puts "\n\n"
end

Which yields the following to the console (the full distribution is truncated due to its size):

+-----------+---------------------+
| category  | value               |
+-----------+---------------------+
| veggies   |             0.10866 |
| snacks    | 0.19830999999999863 |
| meat      |             0.14769 |
| drinks    | 0.10115999999999989 |
| beauty    |             0.24632 |
| magazines | 0.19785999999999926 |
+-----------+---------------------+
0.9999999999999978

+--------------+-----------+----------------------+
| age          | category  | value                |
+--------------+-----------+----------------------+
| teens        | veggies   |  0.02608000000000002 |
| teens        | snacks    |  0.11347999999999969 |
| teens        | meat      |  0.06282999999999944 |
| teens        | drinks    |   0.0263200000000002 |
| teens        | beauty    |   0.1390699999999995 |
| teens        | magazines |  0.13322000000000103 |
| young_adults | veggies   | 0.010250000000000023 |
| young_adults | snacks    |  0.03676000000000003 |
| young_adults | meat      |  0.03678000000000005 |
| young_adults | drinks    |  0.03670000000000045 |
| young_adults | beauty    |  0.05172999999999976 |
| young_adults | magazines | 0.035779999999999916 |
| adults       | veggies   | 0.026749999999999927 |
| adults       | snacks    |  0.03827999999999962 |
| adults       | meat      | 0.034600000000000505 |
| adults       | drinks    | 0.028190000000000038 |
| adults       | beauty    |  0.03892000000000036 |
| adults       | magazines |  0.02225999999999998 |
| elders       | veggies   |  0.04558000000000066 |
| elders       | snacks    | 0.009790000000000047 |
| elders       | meat      | 0.013480000000000027 |
| elders       | drinks    | 0.009949999999999931 |
| elders       | beauty    | 0.016600000000000226 |
| elders       | magazines | 0.006600000000000025 |
+--------------+-----------+----------------------+
1.0000000000000013

+--------+-----------+----------------------+
| sex    | category  | value                |
+--------+-----------+----------------------+
| male   | veggies   |  0.03954000000000044 |
| male   | snacks    |   0.1132499999999996 |
| male   | meat      |  0.10851000000000031 |
| male   | drinks    |                0.073 |
| male   | beauty    | 0.023679999999999857 |
| male   | magazines |  0.05901999999999993 |
| female | veggies   |  0.06911999999999997 |
| female | snacks    |  0.08506000000000069 |
| female | meat      |  0.03918000000000006 |
| female | drinks    |  0.02816000000000005 |
| female | beauty    |  0.22264000000000062 |
| female | magazines |  0.13884000000000046 |
+--------+-----------+----------------------+
1.000000000000002

+-----------------+-----------+----------------------+
| relation        | category  | value                |
+-----------------+-----------+----------------------+
| single          | veggies   |  0.07722000000000082 |
| single          | snacks    |  0.13090999999999794 |
| single          | meat      |  0.09317000000000061 |
| single          | drinks    | 0.059979999999999915 |
| single          | beauty    |  0.16317999999999971 |
| single          | magazines |  0.13054000000000135 |
| in_relationship | veggies   | 0.031440000000000336 |
| in_relationship | snacks    |  0.06740000000000032 |
| in_relationship | meat      | 0.054520000000000006 |
| in_relationship | drinks    |  0.04118000000000009 |
| in_relationship | beauty    |  0.08314000000000002 |
| in_relationship | magazines |  0.06732000000000182 |
+-----------------+-----------+----------------------+
1.000000000000003

+----------+-----------+----------------------+
| location | category  | value                |
+----------+-----------+----------------------+
| us       | veggies   |  0.04209000000000062 |
| us       | snacks    |  0.07534000000000109 |
| us       | meat      | 0.055059999999999984 |
| us       | drinks    |  0.03704000000000108 |
| us       | beauty    |  0.09879000000000099 |
| us       | magazines |  0.07867999999999964 |
| canada   | veggies   | 0.027930000000000062 |
| canada   | snacks    |  0.05745999999999996 |
| canada   | meat      |  0.04288000000000003 |
| canada   | drinks    |  0.03078999999999948 |
| canada   | beauty    |  0.06397999999999997 |
| canada   | magazines | 0.053959999999999675 |
| europe   | veggies   | 0.013110000000000132 |
| europe   | snacks    |   0.0223200000000001 |
| europe   | meat      |  0.01730000000000005 |
| europe   | drinks    | 0.011859999999999964 |
| europe   | beauty    | 0.025490000000000183 |
| europe   | magazines | 0.020920000000000164 |
| asia     | veggies   |  0.02552999999999989 |
| asia     | snacks    |  0.04319000000000044 |
| asia     | meat      |  0.03244999999999966 |
| asia     | drinks    |  0.02147000000000005 |
| asia     | beauty    |  0.05805999999999953 |
| asia     | magazines |   0.0442999999999999 |
+----------+-----------+----------------------+
1.0000000000000029

+-----------+--------------+--------+-----------------+----------+------------------------+
| category  | age          | sex    | relation        | location | value                  |
+-----------+--------------+--------+-----------------+----------+------------------------+
| veggies   | teens        | male   | single          | us       |  0.0035299999999999936 |
| veggies   | teens        | male   | single          | canada   |  0.0024500000000000073 |
| veggies   | teens        | male   | single          | europe   |  0.0006999999999999944 |
| veggies   | teens        | male   | single          | asia     |  0.0016699999999999899 |
| veggies   | teens        | male   | in_relationship | us       |   0.001340000000000006 |
| veggies   | teens        | male   | in_relationship | canada   |  0.0010099999999999775 |
| veggies   | teens        | male   | in_relationship | europe   |  0.0006499999999999989 |
| veggies   | teens        | male   | in_relationship | asia     |   0.000819999999999994 |

(... many rows ...)

| magazines | elders       | male   | in_relationship | asia     | 0.00012000000000000163 |
| magazines | elders       | female | single          | us       |  0.0007399999999999966 |
| magazines | elders       | female | single          | canada   |  0.0007000000000000037 |
| magazines | elders       | female | single          | europe   |  0.0003199999999999965 |
| magazines | elders       | female | single          | asia     |  0.0005899999999999999 |
| magazines | elders       | female | in_relationship | us       |  0.0004899999999999885 |
| magazines | elders       | female | in_relationship | canada   | 0.00027000000000000114 |
| magazines | elders       | female | in_relationship | europe   | 0.00012000000000000014 |
| magazines | elders       | female | in_relationship | asia     | 0.00012000000000000014 |
+-----------+--------------+--------+-----------------+----------+------------------------+
1.0000000000000004

Let’s define a Proc for inferring categories based on user traits as evidence:

infer = -> (age, sex, rel, loc) do

  # Let's map through the possible categories and the probability
  # values the distibutions assign to them:
  all = category.values.map do |cat|
    pc  = cat_dist.value_for category: cat
    pac = age_cat_dist.value_for age: age, category: cat
    psc = sex_cat_dist.value_for sex: sex, category: cat
    prc = rel_cat_dist.value_for relation: rel, category: cat
    plc = loc_cat_dist.value_for location: loc, category: cat

    { category: cat, value: (pac * psc/pc * prc/pc * plc/pc) }
  end

  # Let's do the same with the full distribution to be able to compare
  # the results:
  all_full = category.values.map do |cat|
    val = full_dist.value_for category: cat, age: age, sex: sex,
            relation: rel, location: loc

    { category: cat, value: val }
  end

  # Here's we're getting the most probable categories based on the
  # Naive Bayes distribution approximation model and based on the full
  # distribution:
  win      = all.max      { |a, b| a[:value] <=> b[:value] }
  win_full = all_full.max { |a, b| a[:value] <=> b[:value] }

  puts "Best match for #{[ age, sex, rel, loc ]}:"
  puts "   #{win[:category]} => #{win[:value]}"
  puts "Full pointed at:"
  puts "   #{win_full[:category]} => #{win_full[:value]}\n\n"
end

The results

We’re ready now to use the model and see how well the Naive Bayes model performs in this particular scenario:

infer.call :teens, :male, :single, :us
infer.call :young_adults, :male, :single, :asia
infer.call :adults, :female, :in_relationship, :europe
infer.call :elders, :female, :in_relationship, :canada

This gave the following results on the console:

Best match for [:teens, :male, :single, :us]:
   snacks => 0.016252573282200262
Full pointed at:
   snacks => 0.01898999999999971

Best match for [:young_adults, :male, :single, :asia]:
   meat => 0.0037455794492659757
Full pointed at:
   meat => 0.0017000000000000016

Best match for [:adults, :female, :in_relationship, :europe]:
   beauty => 0.0012287311061725868
Full pointed at:
   beauty => 0.0003000000000000026

Best match for [:elders, :female, :in_relationship, :canada]:
   veggies => 0.002156365730474441
Full pointed at:
   veggies => 0.0013500000000000022

That’s quite impressive! Even though we’re using a simplified model to approximate the original distribution, the algorithm managed to infer the correct values in all cases. You can notice also that the results differ only by a couple of cases in 1000.

The approximation like that would certainly be very useful in a more complex e-commerce scenario, in the case where the number of evidence variables would be big enough to be unmanageable using the full distribution. There are use cases though, where a couple of errors in 1000 cases would be too many—the traditional example is medical diagnosis. There are also cases where the number of errors would be much greater just because the Naive Bayes assumption of conditional independence of variables is not always a fair an assumption. Is there a way to improve?

The Naive Bayes assumption says that the distribution factorizes the way we did it only if the features are conditionally independent given the category. The notion of conditional independence (apart from the formal mathematical definition) suggests that if some variables a and b are conditionally independent given c, then if we know the value of c then no additional information about b can alter our knowledge about a. In our example, knowing the category, let say :beauty doesn’t mean that e. g sex is independent from age. In real world examples, it’s often very hard to find a use case for Naive Bayes that would follow the assumption in all the cases.

There are alternative approaches that allow us to apply the assumptions that more rigidly follow the chosen data set. We will explore these in the next articles, building on top of what we saw here.

Temporary tables in SQL query optimization

2015-02-25T00:00:00+00:00

SQL queries can get complex in a big hurry. If you are querying multiple tables, and in particular if your query involves operations like UNION and INTERSECT, then you can find yourself in a big, messy pile of SQL. Worse, that big pile will often run slowly, sometimes to the point where a web application times out!

I won’t inflict a real-world example on you; that would be cruel. So let’s look at a “toy” problem, keeping in mind that this won’t illustrate any time-savings, just the technique involved.

Here’s the original SQL:

SELECT p.* FROM products p
JOIN (SELECT * FROM inventory WHERE /* complex clause here */) i USING (sku)
UNION ALL
SELECT p.* FROM clearance_products p
JOIN (SELECT * FROM inventory WHERE /* complex clause here */) i USING (sku)

Bonus hint: using “UNION ALL“ instead of just “UNION” will allow the query processor to skip an unnecessary step here. “UNION ALL” says you know the rows on either side of the clause are unique. “UNION” means the results will be post-processed to remove duplicates. This might save you more than a smidgen of time, depending on how large the two sub-queries get.

Now, many times the query optimizer will just do the right thing here. But sometimes (cough, cough-MySQL), your database isn’t quite up to the task. So you have to shoulder the burden and help out. That’s where we can apply a temporary table.

Temporary tables are created for the length of the database session; that’s different than a transaction. For a web application, that’s usually (not always) the length of the request (i.e., from the time your web application opens a database connection, until it explicitly closes it, or until it returns control to the web server, usually by passing it a completed page). For a script, it’s a similar duration, e.g. until the script exits.

CREATE TEMPORARY TABLE cross_inventory AS
SELECT * FROM inventory WHERE /* complex clause here */;

CREATE INDEX cross_inv_sku ON cross_inventory(sku);

There’s no significant difference for our purposes between a “permanent” and a “temporary” table. However, you do have to keep in mind that these tables are created without indexes, so if your goal is to improve the speed of queries involving the data here, adding an index after creating the table is usually desirable.

With all this in place, now we can:

SELECT p.* FROM products p
JOIN cross_inventory i USING (sku)
UNION
SELECT p.* FROM clearance_products p
JOIN cross_inventory i USING (sku)

Sometimes your temporary table will be built up not by a straightforward “CREATE … AS SELECT …”, but by your application:

CREATE TEMPORARY TABLE tmp_inventory AS SELECT * FROM inventory WHERE false;
CREATE INDEX tmp_inv_sku ON tmp_inventory(sku);

And then within the application:

# Pseudocode
while (more_data) {
  row = build_inv_record(more_data);
  sql_do('INSERT INTO tmp_inventory VALUES (?,?,...)', row);
}

Here, we are creating an empty “inventory” table template as a temporary table (“SELECT * FROM inventory WHERE false”), then adding rows to it from the application, and finally running our query. Note that in a practical application of this, it’s not likely to be a lot faster, because the individual INSERT statements will take time. But this approach may have some utility where the existing “inventory” table doesn’t have the data we want to JOIN against, or has the data, but not in a way we can easily filter.

I’ve used temporary tables (in a MySQL/Interchange/Perl environment) to speed up a query by a factor of two or more. It’s usually in those cases where you have a complex JOIN that appears in two or more parts of the query (again, usually a UNION). I’ve even had big-win situations where the same temporary table was used in two different queries during the same session.

A similar approach is the Common Table Expression (CTE) found in PostgreSQL starting with version 8.4. This allows you to identify the rows you would be pouring into your temporary table as a named result-set, then reference it in your query. Our “toy” example would become:

WITH cross_inventory AS
(SELECT * FROM inventory WHERE /* complex clause here */)
SELECT p.* FROM products p
JOIN cross_inventory i USING (sku)
UNION
SELECT p.* FROM clearance_products p
JOIN cross_inventory i USING (sku)

I’ve not had an opportunity to use CTEs yet, and of course they aren’t available in MySQL, so the temporary-table technique will still have a lot of value for me in the foreseeable future.

Interchange Loop Optimization

2015-02-09T00:00:00+00:00

It’s important to understand both how loops work in Interchange and the (very) fundamental differences between interpolating Interchange tag language (ITL) and the special loop tags (typically referred to as [PREFIX-*] in the literature). Absent this sometimes arcane knowledge, it is very easy to get stuck with inefficient loops even with relatively small loop sets. I’ll discuss both the function of loops and interpolation differences between the tag types while working through a [query] example. While all loop tags–[item-list], [loop], [search-list], and [query]–process similarly, it is to [query] where most complex loops will gravitate over time (to optimize the initiation phase of entering the loop) and where we have the most flexibility for coming up with alternative strategies to mitigate sluggish loop-processing.

Loop Processing

All loop tags are container tags in Interchange, meaning they have an open and close tag, and in between is the body. Only inside this body is it valid to define [PREFIX-] tags (notable exception of [PREFIX-quote] for the sql arg of [query]). This is because the [PREFIX-] tags are not true ITL. They are tightly coupled with the structure of the underlying rows of data and they are processed by distinct, optimized regular expressions serially. Outside the context of the row data from a result set, they are meaningless.

Moreover, the body of a loop tag is slurped into a scalar variable (as all bodies of container tags are handled via the ITL parser) and for each row in the record set of the loop, the contents are acted upon according to the [PREFIX-] tags defined within the body. The first important distinction to recognize here is, the per-row action on this scalar is limited to only the [PREFIX-] tags. The action occurring at loop time ignores any embedded ITL.

At the end of each row’s processing, the copy of the body tied to that one row is then concatenated to the results of all previous rows thus processed. For a loop with N rows (assuming no suppression by [if-PREFIX-*] conditionals) that means every instance of ITL originally placed into the loop body is now present N times in the fully assembled body string. Simple example:

[loop list='1 2 3']
[tmp junk][loop-code][/tmp]
[if scratch junk == 2]
I declare [loop-code] to be special!
[else]
Meh. [loop-code] is ordinary.
[/else]
[/if]
[/loop]

Once this result set with N=3 is processed, but before Interchange returns the results, the assembled return looks like the following string:

[tmp junk]1[/tmp]
[if scratch junk == 2]
I declare 1 to be special!
[else]
Meh. 1 is oridinary
[/else]
[/if]

[tmp junk]2[/tmp]
[if scratch junk == 2]
I declare 2 to be special!
[else]
Meh. 2 is oridinary
[/else]
[/if]

[tmp junk]3[/tmp]
[if scratch junk == 2]
I declare 3 to be special!
[else]
Meh. 3 is oridinary
[/else]
[/if]

Some important observations:

It doesn’t take much ITL to turn a loop body into a monster interpolation process. One must consider the complexity of the ITL in the body by a factor of the number of rows (total, or the “ml” matchlimit value).
ITL does nothing to short-circuit action of the [PREFIX-*] tags. Having [PREFIX-param], [PREFIX-calc], etc. inside an ITL [if] means all those loop tags parse regardless of the truth of the if condition.

ITL vs. Loop Tags

ITL maps to routines, both core and user-defined, determined at compile time. They are processed in order of discovery within the string handed to the ::interpolate_html() routine and have varied and complex attributes that must be resolved for each individual tag. Further, for many (if not most) tags, the return value is itself passed through a new call to ::interpolate_html(), acting on all embedded tags, in an action referred to as reparse. There is, relatively speaking, a good deal of overhead in processing through ::interpolate_html(), particularly with reparse potentially spawning off a great many more ::interpolate_html() calls.

Loop tags, by contrast, map to a pre-compiled set of regular expressions. In contrast to crawling the string and acting upon the tags in the order of discovery, each regex in turn is applied globally to the string. The size of the string is limited to the exact size of the single loop body, and there is no analogue to ITL’s reparse. Further, given this processing pattern, a careful observer might have noted that the order of operations can impact the structure. Specifically, tags processed earlier cannot depend on tags processed later. E.g., [PREFIX-param] processes ahead of [PREFIX-pos], and so:

[if-PREFIX-pos 2 eq [PREFIX-param bar]]

will work, but:

[if-PREFIX-param bar eq [PREFIX-pos 2]]

will not. While the above is a somewhat contrived exampled, the impacts of loop tag processing can be more easily seen in an example using [PREFIX-next]:

[PREFIX-next][PREFIX-param baz][/PREFIX-next]
Code I only want to run when baz is false, like this [PREFIX-exec foo][/PREFIX-exec] call

Because [PREFIX-next] is the absolute last loop tag to run, every other loop tag in the block is run before the next condition is checked. All [PREFIX-next] does is suppress the resulting body from the return, unlike Perl’s next, which short-circuits the remaining code in the loop block.

An Optimization Example

As long as you’re familiar with the idiosyncracies of [PREFIX-*] tags, you should make every effort to use them instead of ITL because they are substantially lighter weight and faster to process. A classic case that can yield remarkable performance gains is to directly swap an embedded [perl] or [calc] block with an equivalent [PREFIX-calc] block.

Let’s take a typical query with little consideration given to whether we use loop tags or ITL, not unlike many I’ve seen where the resource has just become unusably slow. This code originally was developed processing 50 records per page view, but the team using it has requested over time to increase that count.

[query
    list=1
    ml=500 [comment]Ouch! That's a big N[/comment]
    sql="
        SELECT *
        FROM transactions
        WHERE status = 'pending'
        ORDER BY order_date DESC
    "
]
Order [sql-param order_number]
[if global DEVELOPMENT]
    Show [sql-exec stats_crunch][sql-param stats][/sql-exec], only of interest to developers
[/if]
Date: [convert-date format="%b %d, %Y at %T"][sql-param order_date][/convert-date]
[if cgi show_inventory]
Inv:
[if cgi show_inventory eq all]
* Shipped: [either][sql-param is_shipped][or]pending[/either]
* Count: [inventory type=shipped sku=[sql-param sku]]
[/if]
* On Hand: [inventory type=onhand sku=[sql-param sku]]
* Sold: [inventory type=sold sku=[sql-param sku]]
* Shipping: [inventory type=shipping sku=[sql-param sku]]
[/if]
Order details:
    <a href="[area
                href=order_view
                form="
                    order=[sql-param order_number]
                    show_status=[either][cgi show_status][or]active[/either]
                    show_inventory=[cgi show_inventory]
                "
            ]">View [sql-param order_number]</a>
[/query]

Considering this block out of context, it doesn’t seem all that unreasonable. However, let’s look at some of the pieces individually and see what can be done.

We use [if] in 3 different circumstances in the block. However, those values they test are static. They don’t change on any iteration. (We are excluding the potential of any other ITL present in the block from changing their values behind the scenes.)
[convert-date] may be convenient, but it is only one of a number of ways to address date formatting. Our database itself almost certainly has date-formatting routines, but one of the benefits of [convert-date] is you could have a mixed format underlying the data and it can make sense out of the date to some degree. So perhaps that’s why the developer has used [convert-date] here.
Good chance that stats_crunch() is pretty complicated and that’s why the developer wrote a catalog or global subroutine to handle it. Since we only want to see it in the development environment, it’d be nice if it only ran when it was needed. Right now, because of ITL happening on reparse, stats_crunch() fires for every row even if we have no intention of using its output.
We need that link to view our order, but on reparse it means ::interpolate_html() has to parse 500 [area] tags along with [either] and [cgi] x 500. All of these tags are lightweight, but parsing numbers are really going to catch up to us here.

Our goal here is to replace any ITL we can with an equivalent use of a loop tag or, absent the ability to remove ITL logically, to wrap that ITL into a subroutine that can itself be called in loop context with [PREFIX-exec]. The first thing I want to address are those [if] and [either] tags, the lowest hanging fruit:

[query
    list=1
    ml=500
    sql="
        <b>SELECT *,
            '[if global DEVELOPMENT]1[/if]' AS is_development,
            [sql-quote][cgi show_inventory][/sql-quote] AS show_inventory,
            COALESCE(is_shipped,'pending') AS show_inventory_shipped</b>
        FROM transactions
        WHERE status = 'pending'
        ORDER BY order_date DESC
    "
]
Order [sql-param order_number]
<b>[if-sql-param is_development]</b>
    Show [sql-exec stats_crunch][sql-param stats][/sql-exec], only of interest to developers
<b>[/if-sql-param]</b>
Date: [convert-date format="%b %d, %Y at %T"][sql-param order_date][/convert-date]
<b>[if-sql-param show_inventory]</b>
Inv:
<b>[if-sql-param show_inventory eq all]</b>
* Shipped: <b>[sql-param show_inventory_shipped]</b>
* Count: [inventory type=shipped sku=[sql-param sku]]
<b>[/if-sql-param]</b>
* On Hand: [inventory type=onhand sku=[sql-param sku]]
* Sold: [inventory type=sold sku=[sql-param sku]]
* Shipping: [inventory type=shipping sku=[sql-param sku]]
<b>[/if-sql-param]</b>
Order details:
    <a href="[area
                href=order_view
                form="
                    order=[sql-param order_number]
                    show_status=[either][cgi show_status][or]active[/either]
                    show_inventory=[cgi show_inventory]
                "
            ]">View [sql-param order_number]</a>
[/query]

By moving those evaluations into the SELECT list of the query, we’ve reduced the number of interpolations to arrive at those static values to 1 or, in the case of the [either] tag, 0 as we’ve offloaded the calculation entirely to the database. If is_shipped could be something perly false but not null, we would have to adjust our field accordingly, but in either case could still be easily managed as a database calculation. Moreover, by swapping in [if-sql-param is_development] for [if global DEVELOPMENT], we have kept stats_crunch() from running at all when in the production environment.

Next, we’ll consider [convert-date]:

Date: [convert-date format="%b %d, %Y at %T"][sql-param order_date][/convert-date]

My first attempt would be to address this similarly to the [if] and [either] conditions, and try to render the formatted date from a database function as an aliased field. However, let’s assume the underlying structure of the data varies and that’s not easily accomplished, and we still want [convert-date]. Luckily, Interchange supports that same tag as a filter, and [PREFIX-filter] is a loop tag:

Date: <b>[sql-filter convert_date."%b %d, %Y at %T"]</b>[sql-param order_date]<b>[/sql-filter]</b>

[PREFIX-filter] is very handy to keep in mind as many transformation tags have a filter wrapper for them. E.g., [currency] -> [PREFIX-filter currency]. And if the one you’re looking at doesn’t, you can build your own, easily.

Now to look at that [inventory] tag. The most direct approach assumes that the code inside [inventory] can be run in Safe, which often it can even if [inventory] is global. However, if [inventory] does run-time un-Safe things (such as creating an object) then it may not be possible. In such a case, we would want to create a global sub, like our hypothetical stats_crunch(), and invoke it via [PREFIX-exec]. However, let us assume we can safely (as it were) invoke it via the $Tag object to demonstrate another potent loop option: [PREFIX-sub].

[if-sql-param show_inventory]
<b>[sql-sub show_inventory]
    my $arg = shift;
    return $Tag->inventory({ type => $arg, sku => $Row->{sku} });
[/sql-sub]</b>
Inv:
[if-sql-param show_inventory eq all]
* Shipped: [sql-param show_inventory_shipped]
* Count: <b>[sql-exec show_inventory]shipped[/sql-exec]</b>
[/if-sql-param]
* On Hand: <b>[sql-exec show_inventory]on_hand[/sql-exec]</b>
* Sold: <b>[sql-exec show_inventory]sold[/sql-exec]</b>
* Shipping: <b>[sql-exec show_inventory]shipping[/sql-exec]</b>
[/if-sql-param]

Let’s go over what this gives us:

[PREFIX-sub] creates an in-line catalog sub that is compiled at the start of processing, before looping actually begins. As such, the [PREFIX-sub] definitions can occur anywhere within the loop body and are then removed from the body to be parsed.
The body of the [PREFIX-exec] is passed to the sub as the first argument. We use that here for our static values to the “type” arg. If we also wanted to access [sql-param sku] from the call, we would have to include that in the body and set up a parser to extract it out of the one (and only) arg we can pass in. Instead, we can reference the $Row hash within the sub body just as we can do when using a [PREFIX-calc], with one minor adjustment to our [query] tag–we have to indicate to [query] we are operating on a row-hash basis instead of the default row-array basis. We do that by adding the hashref arg to the list:

[query
    list=1
    ml=500
    <b>hashref=1</b>

We still have access to the full functionality of [inventory] but we’ve removed the impact of having to parse that tag 2000 times (in the worst-case scenario) if left as ITL in the query body. If we run into Safe issues, that same sub body can either be created as a pre-compiled global sub or, if available, we can set our catalog AllowGlobal in which case catalog subs will no longer run under Safe.

Finally, all we have left to address is [area] and its args which themselves have ITL. I will leverage [PREFIX-sub] again as an easy way to manage the issue:

<b>[sql-sub area_order_view]
    my $show_status = $CGI->{show_status} || 'active';
    return $Tag->area({
        href => 'order_view',
        form => "order=$Row->{order_number}\n"
              . "show_status=$show_status\n"
              . "show_inventory=$CGI->{show_inventory}",
    });
[/sql-sub]</b>
Order details:
    <a href="<b>[sql-exec area_order_view][/sql-exec]</b>">View [sql-param order_number]</a>

By packaging all of [area]’s requirements into the sub body, I can address all of the ITL at once.

So now, let’s put together the entire [query] rewrite to see the final product:

[query
    list=1
    ml=500
    hashref=1
    sql="
        SELECT *,
            '[if global DEVELOPMENT]1[/if]' AS is_development,
            [sql-quote][cgi show_inventory][/sql-quote] AS show_inventory,
            COALESCE(is_shipped,'pending') AS show_inventory_shipped
        FROM transactions
        WHERE status = 'pending'
        ORDER BY order_date DESC
    "
]
Order [sql-param order_number]
[if-sql-param is_development]
    Show [sql-exec stats_crunch][sql-param stats][/sql-exec], only of interest to developers
[/if-sql-param]
Date: [sql-filter convert_date."%b %d, %Y at %T"][sql-param order_date][/sql-filter]
[if-sql-param show_inventory]
[sql-sub show_inventory]
    my $arg = shift;
    return $Tag->inventory({ type => $arg, sku => $Row->{sku} });
[/sql-sub]
Inv:
[if-sql-param show_inventory eq all]
* Shipped: [sql-param show_inventory_shipped]
* Count: [sql-exec show_inventory]shipped[/sql-exec]
[/if-sql-param]
* On Hand: [sql-exec show_inventory]on_hand[/sql-exec]
* Sold: [sql-exec show_inventory]sold[/sql-exec]
* Shipping: [sql-exec show_inventory]shipping[/sql-exec]
[/if-sql-param]
[sql-sub area_order_view]
    my $show_status = $CGI->{show_status} || 'active';
    return $Tag->area({
        href => 'order_view',
        form => "order=$Row->{order_number}\n"
              . "show_status=$show_status\n"
              . "show_inventory=$CGI->{show_inventory}",
    });
[/sql-sub]
Order details:
    <a href="[sql-exec area_order_view][/sql-exec]">View [sql-param order_number]</a>
[/query]

Voila! Our new query body is functionally identical to the original body, though admittedly a little more complicated to set up. However, the trade-off in efficiency is likely to be substantial.

I recently worked on a refactor for a client that was overall very similar to the above example, with a desired N value of 250. The code prior to refactoring took ~70s to complete. Once we had completed the refactor using the same tools as I’ve identified here, we brought down processing time to just under 3s, losing no functionality.

Time taken optimizing Interchange loops will almost always pay dividends.

WebP images experiment on End Point website

2014-01-28T00:00:00+00:00

WebP is an image format for RGB images on the web that supports both lossless (like PNG) and lossy (like JPEG) compression. It was released by Google in September 2010 with open source reference software available under the BSD license, accompanied by a royalty-free public patent license, making it clear that they want it to be widely adopted by any and all without any encumbrances.

Its main attraction is smaller file size at similar quality level. It also supports an alpha channel (transparency) and animation for both lossless and lossy images. Thus it is the first image format that offers the transparency of PNG in lossy images at much smaller file size, and animation only available in the archaic limited-color GIF format.

Comparing quality & size

While considering WebP for an experiment on our own website, we were very impressed by its file size to quality ratio. In our tests it was even better than generally claimed. Here are a few side-by-side examples from our site. You’ll only see the WebP version if your browser supports it:

13,420 bytes JPEG	2776 bytes WebP
14,734 bytes JPEG	3386 bytes WebP

The original PNG images were converted by ImageMagick to JPEG, and by cwebp -q 80 to WebP. I think we probably should increase the WebP quality a bit to keep a little of the facial detail that flattens out, but it’s amazing how good these images look for file sizes that are only 17% and 23% of the JPEG equivalent.

One of our website’s background patterns has transparency, making the PNG format a necessity, but it also has a gradient, which PNG compression is particularly inefficient with. WebP is a major improvement there, at 13% the size of the PNG. The image is large so I won’t show it here, but you can follow the links if you’d like to see it:

337,186 bytes	container-pattern.png
43,270 bytes	container-pattern.webp

Browser support

So, what is the downside? WebP is currently natively supported only in Chrome and Opera among the major browsers, though amazingly, support for other browsers can be added via WebPJS, a JavaScript WebP renderer.

Update! As of 2021, all current major browsers support WebP image rendering.

Why don’t the other browsers add support given the liberal license? Especially Firefox you’d expect to support it. In fact a patch has been pending for years, and a debate about adding support still smolders. Why?

WebP does not yet support progressive rendering, Exif tagging, non-RGB color spaces such as CMYK, and is limited to 16,384 pixels per side. Some Firefox developers feel that it would do the Internet community a disservice to support an image format still under development and cause uncertain levels of support in various clients, so they will not accept WebP in its current state.

Many batch image-processing tools now support WebP, and there is a free Photoshop plug-in for it. Some websites are quietly using it just because of the cost savings due to reduced bandwidth.

For our first experiment serving WebP images from the End Point website, I decided to serve WebP images only to browsers that claim to be able to support it. They advertise that support in this HTTP request header:

Accept: image/webp,*/*;q=0.8

That says explicitly that the browser can render image/webp, so we just need to configure the server to send WebP images. One way to do that is in the application server, by having it send URLs pointing to WebP files.

Let’s plan to have both common format (JPEG or PNG) and WebP files side by side, and then try a way that is transparent to the application and can be enabled or disabled very easily.

Web server rewrites

It’s possible to set up the web server to transparently serve WebP instead of JPEG or PNG if a matching file exists. Based on some examples other people posted, we used this nginx configuration:

set $webp "";
set $img "";
if ($http_accept ~* "image/webp") { set $webp "can"; }
if ($request_filename ~* "(.*)\.(jpe?g|png)$") { set $img $1.webp; }
if (-f $img) { set $webp "$webp-have"; }
if ($webp = "can-have") {
    add_header Vary Accept;
    rewrite "(.*)\.\w+$" $1.webp break;
    break;
}

It’s also good to add to /etc/nginx/mime.types:

image/webp .webp

so that .webp files are served with the correct MIME type instead of the default application/octet-stream, or worse, text/plain with perhaps a bogus character set encoding.

Then we just make sure identically-named .webp files match .png or .jpg files, such as those for our examples above:

-rw-rw-r-- 337186 Nov  6 14:10 container-pattern.png
-rw-rw-r--  43270 Jan 28 08:14 container-pattern.webp
-rw-rw-r--  14734 Nov  6 14:10 josh_williams.jpg
-rw-rw-r--   3386 Jan 28 08:14 josh_williams.webp
-rw-rw-r--  13420 Nov  6 14:10 marina_lohova.jpg
-rw-rw-r--   2776 Jan 28 08:14 marina_lohova.webp

A request for a given $file.png will work as normal in browsers that don’t advertise WebP support, while those that do will instead receive the $file.webp image.

The image is still being requested with a name ending in .jpg or .png, but that’s just a name as far as both browser and server are concerned, and the image type is determined by the MIME type in the HTTP response headers (and/or by looking at the file’s magic numbers). So the browser will have a file called $something.jpg in the DOM and in its cache, but it will actually be a WebP file. That’s ok, but could be confusing to users who save the file for whatever reason and find it isn’t actually the JPEG they were expecting.

301/302 redirect option

One remedy for that is to serve the WebP file via a 301 or 302 redirect instead of transparently in the response, so that the browser knows it’s dealing with a different file named $something.webp. To do that we changed the nginx configuration like this:

rewrite "(.*)\.\w+$" $1.webp permanent;

That adds a little bit of overhead, around 100-200 bytes unless large cookies are sent in the request headers, and another network round-trip or two, though it’s still a win with the reduced file sizes we saw. However, I found that it isn’t even necessary right now due to an interesting behavior in Chrome that may even be intentional to cope with this very situation. (Or it may be a happy accident.)

Chrome image download behavior

Versions of Chrome I tested only send the Accept: image/webp [etc.] request header when fetching images from an HTML page, not when you manually request a single file or asking the browser to save the image from the page by right-clicking or similar. In those cases the Accept header is not sent, so the server doesn’t know the browser supports WebP, so you get the JPEG or PNG you asked for. That was actually a little confusing to hunt down by sniffing the HTTP traffic on the wire, but it may be a nice thing for users as long as WebP is still less-known.

Batch conversion

It’s fun to experiment, but we needed to actually get all the images converted for our website. Surprisingly, even converting from JPEG isn’t too bad, though you need a higher quality setting and the file size will be larger. Still, for best image quality at the smallest file size, we wanted to start with original PNG images, not recompress JPEGs.

To make that easy, we wrote two shell scripts for Linux, bash, and cwebp. We found a few exceptional images that were larger in WebP than in PNG or JPEG, so the script deletes any WebP file that is not smaller, and our nginx configuration will in that case not find a .webp file and will serve the original PNG or JPEG.

Full-page download sizes compared

Here are performance tests run by WebPageTest.org using Chrome 32 on Windows 7 on a simulated cable Internet connection. The total download size difference is most impressive, and on a slower mobile network or with higher latency (greater distance from the server) would affect the download time more.

Page URL	With WebP	Without WebP
Bytes	Time	Bytes	Time
https://www.endpointdev.com/	374 KB	2.9s	850 KB	3.4s
https://www.endpointdev.com/team/	613 KB	3.6s	1308 KB	4.1s

Conclusion

This article is not even close to a comprehensive shootout between WebP and other image types. There are other sites that consider the image format technical details more closely and have well-chosen sample images.

My purpose here was to convert a real website in bulk to WebP without hand-tuning individual images or spending too much time on the project overall, and to see if the overall infrastructure is easy enough to set up, and the download size and speed improved enough to make it worth the trouble, and get real-world experience with it to see if we can recommend it for our clients, and in which situations.

So far it seems worth it, and we plan to continue using WebP on our website. With empty browser caches, visit www.endpointdev.com using Chrome and then one of the browsers that doesn’t support WebP, and see if you notice a speed difference on first load, or any visual difference.

I hope to see WebP further developed and more widely supported.

The Real Cost of Data Roundtrip

2012-10-03T00:00:00+00:00

Sometimes you need to perform some heavy database operations. I don’t know why very often programmers are afraid of using databases for that. They usually have some fancy ORM which performs all the operations, and the only way to change the data is to make some SELECT * from a table, create a bunch of unneeded objects, change one field, convert those changed objects into queries and send that to the database.

Have you ever thought about the cost of the roundtrip of data? The cost of getting all the data from database just to send changed data into the database? Why do that if there would be much faster way of achieving the same results?

Imagine that you have quite a heavy operation. Let’s make something which normally databases cannot do, some more complicated operation. Many programmers just don’t know that there is any other way than writing this in the application code. Let’s change all the HTML entities into real characters.

The HTML entities are a way of writing many different characters in HTML. This way you can write for instance the Euro currency sign “€” in HTML even if you don’t have it on your keyboard. You just have to write € or € instead. I don’t have to, as when I use UTF-8 encoding and write this character directly, it should be showed normally. What’s more I have this character on my keyboard.

I will convert the text stored in database changing all the htmlentities into real unicode characters. I will do it using three different methods.

The first will be a simple query run inside PostgreSQL
The second will be an external program which downloads the text column from database, changes it externally and loads into database.
The third method will be almost the same as the second, however it will download whole rows.

Generate Data

So, for this test I need to have some data. Let’s write a simple data generator.

First, a simple function for returning a random number within the given range.

CREATE FUNCTION random(INTEGER, INTEGER) RETURNS INTEGER AS $$
  SELECT floor ( $1 + ($2 - $1 + 1 ) * random())::INTEGER;
$$ LANGUAGE SQL;

Now the function for generating random texts of random length filled with the HTML entities.

CREATE FUNCTION generate_random_string() RETURNS TEXT AS $$
DECLARE
  items TEXT[] =
    ARRAY[
      'AAAA','BBBB','CCCC','DDDD','EEEE','FFFF','GGGG',
      'HHHH','IIII','JJJJ','KKKK','LLLL','MMMM','NNNN',
      'OOOO','PPPP','QQQQ','RRRR','SSSS','TTTT','UUUU',
      'VVVV','WWWW','XXXX','YYYY','ZZZZ',
      '&amp;', '&#34;', '&#39;', '&#38;','&#60;','&#62;',
      '&#162;','&#163;','&#164;','&#165;','&#166;','&#167;',
      '&#168;','&#169;','&#170;','&#171;','&#172;','&#173;',
      '&#174;','&#175;','&#176;','&#177;','&#178;','&#179;',
      '&#180;','&#181;','&#182;','&#183;','&#184;','&#185;',
      '&#186;','&#187;','&#188;','&#189;','&#190;'
    ];
  length INTEGER := random(500, 1500);
  result TEXT := '';
  items_length INTEGER := array_length(items, 1);
BEGIN
  FOR x IN 1..length LOOP
    result := result || items[ random(1, items_length) ];
  END LOOP;

  RETURN result;
END;
$$ LANGUAGE PLPGSQL;

The table for storing the data is created with the following query:

CREATE TABLE data (
    id SERIAL PRIMARY KEY,
    padding TEXT,
    t TEXT
);

Then I filled this table using a query generating 50k rows with random data:

INSERT INTO data(payload, t)
SELECT
    generate_random_string(),
    generate_random_string()
FROM
    generate_series(1, 50*1000);

Let’s check the table size:

SELECT pg_size_pretty(pg_relation_size('data'));
 pg_size_pretty
 ----------------
  207 MB
  (1 row)

As the table is filled with random data, I need to have two tables with exactly the same data.

CREATE TABLE query (id SERIAL PRIMARY KEY, payload TEXT, t TEXT);
CREATE TABLE script (id SERIAL PRIMARY KEY, payload TEXT, t TEXT);
CREATE TABLE script_full (id SERIAL PRIMARY KEY, payload TEXT, t TEXT);

INSERT INTO query SELECT * FROM data;
INSERT INTO script SELECT * FROM data;
INSERT INTO script_full SELECT * FROM data;

The Tests

SQL

Many programmers think that such operations are not normally available inside a database. However PostgreSQL has quite a nice feature, it can execute functions written in many different languages. For the purpose of this test I will use the language pl/perlu which allows me to use external libraries. I will also use HTML::Entities package for the conversion.

The function I wrote is quite simple:

CREATE FUNCTION decode_html_entities(text) RETURNS TEXT AS $$
    use HTML::Entities;
    return decode_entities($_[0]);
$$ LANGUAGE plperlu;

The update of the data can be done using the following query:

UPDATE query SET t = decode_html_entities(t);

Application

In order to have those tests comparable, I will write a simple perl script using exactly the same package for converting html entities.

#!/usr/bin/env perl

use DBI;
use HTML::Entities;
use Encode;

my $dbh = DBI->connect(
    "DBI:Pg:dbname=test;host=localhost",
    "szymon",
    "",
    {'RaiseError' => 1, 'pg_utf8_strings' => 1});

$dbh->do('BEGIN');

my $upd = $dbh->prepare("UPDATE script SET t = ? WHERE id = ?");

my $sth = $dbh->prepare("SELECT id, t FROM script");
$sth->execute();

while(my $row = $sth->fetchrow_hashref()) {
    my $t = decode_entities( $row->{'t'} );
    $t = encode("UTF-8", $t);
    $upd->execute( $t, $row->{'id'} );
}

$dbh->do('COMMIT');
$dbh->disconnect();

The Worst Application

There is another terrible idea implemented by programmers too often. Why select only the column you want to change? Let’s select all the rows and send them back to database.

This script will look like this (the important changes are in lines 17 and 23)

#!/usr/bin/env perl

use DBI;
use HTML::Entities;
use Encode;

my $dbh = DBI->connect(
    "DBI:Pg:dbname=test;host=localhost",
    "szymon",
    "",
    {'RaiseError' => 1, 'pg_utf8_strings' => 1});

$dbh->do('BEGIN');

my $upd = $dbh->prepare("UPDATE script_all SET t = ? WHERE id = ?");

my $sth = $dbh->prepare("SELECT id, payload, t FROM script_all");
$sth->execute();

while(my $row = $sth->fetchrow_hashref()) {
    my $t = decode_entities( $row->{'t'} );
    $t = encode("UTF-8", $t);
    $upd->execute( $t, $row->{'payload'}, $row->{'id'} );
}

$dbh->do('COMMIT');
$dbh->disconnect();

Results

The query using pl/perlu function executed in 26 seconds.

The script changing data externally execuded in 2 minutes 10 seconds (5 times slower)

The worst script getting and resending whole rows finished in 4 minutes 35 seconds (10 times slower).

I used quite a small number of rows. There were just 50k rows (about 200MB). On production servers the numbers are much bigger.

Just imagine that the code you developed for changing data could run 10 times faster if you’d do this in the database.

Speeding Up Integration Tests with PostgreSQL - Follow Up

2012-06-27T00:00:00+00:00

Last week I wrote a blog article about speeding up integration tests using PostgreSQL. I proposed there changing a couple of PostgreSQL cluster settings. The main drawback of this method is that those settings need to be changed for the whole cluster. When you have some important data in other databases, you can have a trouble.

In one of the comments Greg proposed using the unlogged table. This feature appeared in PostgreSQL 9.1. The whole difference is that you should use CREATE UNLOGGED TABLE instead of CREATE TABLE for creating all your tables.

For the unlogged table, the data is not written to the write-ahead log. All inserts, updates and deletes are much faster, however the table will be truncated at the server crash or unclean shutdown. Such table is not replicated to standby servers, which is obvious as there are replicated write-ahead logs. What is more important, the indexes created on unlogged tables are unlogged as well.

All the things I describe here are for integrations tests. When database crashes, then all the tests should be restarted and should prepare the database before running, so I really don’t care what happens with the data when something crashes.

The bad thing about unlogged tables is that you cannot change normal table to unlogged. There is nothing like:

ALTER TABLE SET UNLOGGED;

The easiest way which I found for changing the table into unlogged was to create a database dump and add UNLOGGED to all the table creation commands. To have it a little bit faster, I used this command:

pg_dump pbench | sed 's/^CREATE TABLE/CREATE UNLOGGED TABLE/' > pbench.dump.sql

This time I will just delete all the tables in the database and load this dump before running the tests instead of using pg_bench for generating the data:

psql -c "drop database pbench"
psql -c "create database pbench"
psql pbench < pbench.dump.sql

Time for tests. The previous tests results are in the previous blog article. I’m using standard PostgreSQL settings (the secure ones) and the same scale value for pg_bench.

The tests were made using exactly the same command as last time:

./pgbench -h localhost -c 1 -j 1 -T 300 pbench

Below are results combined with the results from previous article.

<table id="testres">
    <thead>
      <tr>
        <th class="noborder"> </th>
        <th colspan="3"> number of clients and threads </th>
      </tr>
      <tr>
        <th class="noborder"> </th>
        <th>1
        </th><th>2
        </th><th>3
      </th></tr>
    </thead>
    <tbody>
      <tr>
        <td class="name">normal settings</td>
        <td>78 tps</td>
        <td>80 tps</td>
        <td>99 tps</td>
      </tr>
      <tr>
        <td class="name">dangerous settings</td>
        <td>414 tps</td>
        <td>905 tps</td>
        <td>1215 tps</td>
      </tr>
      <tr>
        <td class="name">unlogged table</td>
        <td>420 tps</td>
        <td>720 tps</td>
        <td>1126 tps</td>
      </tr>
    </tbody>
</table>

As you can see, the efficiency with unlogged tables is almost as good as with the unsafe settings. The great thing is that it doesn’t influence other databases in the cluster, so you can use safe/default settings for other databases, and only use unlogged tables for the integration tests, which should be much faster now.

This solution works only with the PostgreSQL 9.1 and newer. If you have older PostgreSQL you have to use the previous method with unsafe settings, or better: just upgrade the database.

Website Performance Boot Camp at UTOSC 2012

2012-05-21T00:00:00+00:00

I’ll keep brief my last post about this year’s Utah Open Source Conference.

I was asked to give on both day one and day two a talk called “Website Performance Boot Camp” which carried this brief description:

What’s the difference between a snappy website and a sloth that you turn away from in frustration? A lot of little things, usually. It’s rarely worth doing 100% of the optimization you could do, but getting 75% of the way isn’t hard if you know where to look.

We’ll look at HTTP caching, compression, proxying, CDNs, CSS sprites, minification, and more, how to troubleshoot, and what’s best to leave alone when you have limited time or tolerance for risk.

Here is the video recording of the first time I presented the talk. (The technician noted its audio was “a little hot”.)

Use this Website Performance Boot Camp direct YouTube video link if the embedded video doesn’t work for you.

The slides for this Website Performance Boot Camp presentation are available.

Thanks again to the conference organizers and the other speakers and sponsors, and the nice venue Utah Valley University, for making it a great conference!

Interchange Search Caching with “Permanent More”

2012-01-02T00:00:00+00:00

Most sites that use Interchange take advantage of Interchange’s “more lists”. These are built-in tools that support an Interchange “search” (either the search/scan action, or result of direct SQL via [query]) to make it very easy to paginate results. Under the hood, the more list is a drill-in to a cached “search object”, so each page brings back a slice from the cache of the original search. There are extensive ways to modify the look and behavior of more lists and, with a bit of effort, they can be configured to meet design requirements.

Where more lists tend to fall short, however, is with respect to SEO. There are two primary SEO deficiencies that get business stakeholders’ attention:

There is little control over the construction of the URLs for more lists. They leverage the scan actionmap and contain a hash key for the search object and numeric data to identify the slice and page location. They possess no intrinsic value in identifying the content they reference.
The search cache by default is ephemeral and session-specific. This means all those results beyond page 1 the search engine has cataloged will result in dead links for search users who try to land directly on the more-listed pages.

It is the latter issue that I wish to address because there is—and has been for some time now—a simple mechanism called “permanent more” to remedy the default behavior.

You can leverage “permanent more” by adding the boolean mv_more_permanent, or the shorthand pm, to your search conditions. E.g.:

Link:

    <a href="[area search="
        co=1
        sf=category
        se=Foo
        op=rm
        more=1
        ml=5
        <b>pm=1</b>
    "]">All Foos</a>

Loop:

    [loop search="
        co=1
        sf=category
        se=Foo
        op=rm
        more=1
        ml=5
        <b>pm=1</b>
    "]
    ...loop body with [more-list]...
    [/loop]

Query:

    [query
        list=1
        more=1
        ml=10
        <b>pm=1</b>
        sql="SELECT * FROM products WHERE category LIKE '%Foo%'"
    ]
    ...same as loop but with 10 matches/page...
    [/query]

If the initial search is defined with the “permanent more” setting, it will produce the following adjustments:

The hash key used to store and identify the search cache is deterministic based on the search conditions. Many searches for Interchange are category driven. Thus, all end users who wish to browse a category end up clicking identical links, which create duplicate search caches, belonging uniquely to them. With permanent more, they all share the same cache, with the same identifier. As long as the search conditions don’t change, neither does the cache identifier. Even as the cache is refreshed with new executions of the search, the object remains in the same location. Thus, the results a search engine produced this morning reference links still valid now, tomorrow, or next week, provided they reference the same search conditions.
The cached search object has no session affinity. Any link referencing the cache with the correct hash key has access to the content.

Taken together, “permanent more” removes (for the most part, addressed later) dead links from more lists cataloged by search engines. There are, however, other benefits to “permanent more” beyond those intended as described above:

As stated in passing, standard Interchange search caching produces duplicate search objects for common search conditions. For a busy site, these caches can have an impact on storage. Typically, maintenance is implemented to clean up cache files for all such files whose age exceeds by some amount the session duration (standard is 48 hours). With permanent more, duplicate caches are eliminated. A cache location is reused by all users with the same search requirements, keeping data-storage requirements for caches to the minimum necessary. As searches change, ophaned caches can still easily be cleaned up as they will immediately start to age with no more access to them necessary for storage.
For the same reason that “permanent more” resolves search-engine links, it also resolves content management for individual sites using a reverse proxy for caching. Because most (and certainly the easiest) caching keys are based off of URL, the deterministic nature of the hash keys for “permanent more” allows assurance that the cached content in the proxy accurately reflects the search content over time, and that all users will hit the cached resource and not generate new, unique links with varying hash keys.

One shortcoming of “permanent more” to be aware of is the impact of changing data underneath the search. Even if search conditions do not change, the count and order of matching record sets may. So, e.g., enough products may be removed from a given category to cause the last page of a more list to become empty, which would cause any specific link into that page to become dead. More minor, but still a possibility, is the introduction or removal of products so that a particularly searched-for term has been “bumped” to another page within the search cache since the last time the search engine crawled the more lists. For searches backed by particularly volatile data, “permanent more” may not be sufficient to address search-engine or caching demands.

Finally, “permanent more” should be avoided for any search features that may cache data sensitive to an individual user. This is unlikely to happen as, under most circumstances, the configuration of the search itself will change based on the unique characteristics of the user executing the search (e.g., a username included in a query to review order history). However, it is still possible that context-sensitive information could be stored in the search object and, if so, all other users with access to the more lists would have access to that information.

A Performance Case Study

2011-02-09T00:00:00+00:00

I’m a sucker for a good performance case study. So, when I came across a general request for performance improvement suggestions at Inspiredology, I couldn’t help but experiment a bit.

The site runs on WordPress and is heavy on the graphics as it’s a site geared towards web designers. I inquired to the site administrators about grabbing a static copy of their home page and using it for a case study on our blog. My tools of choice for optimization were webpagetest.org and YSlow.

Here are the results of a 4-step optimization in visual form:

Inspiredology’s complete homepage.

The graph on the left shows the page load time in seconds for a first time view. Throughout optimization, page load time goes from 13.412 seconds to 9.212 seconds. Each step had a measurable impact. The graph on the right shows the page load time in seconds for a repeated view, and this goes from 7.329 seconds to 2.563 seconds throughout optimization. The first optimization step (CSS spriting and file combination) yielded a large performance improvement. I’m not sure why there’s a slight performance decrease between step 3 and step 4.

And here’s a summary of the changes involved in each step:

Step 1
- Addition of CSS Sprites: I wrote about CSS Sprites a while back and A List Apart has an older but still relevant article on CSS Sprites here. Repeating elements like navigation components, icons, and buttons are suitable for CSS sprites. Article or page-specific images are not typically suitable for CSS sprites. For Inspiredology’s site, I created two sprited images—one with a large amount of navigation components, and one with some of their large background images. You can find a great tool for building CSS rules from a sprited image here.
- Combination of JS and CSS files, where applicable. Any JavaScript or CSS files that are included throughout the site are suitable for combination. Files that can’t be combined include suckerfish JavaScript like Google Analytics or marketing service scripts.
- Moved JavaScript requests to the bottom of the HTML. This is recommended because JavaScript requests block parallel downloading. Moving them to the bottom allows page elements to be downloaded and rendered first, followed by JavaScript loading.
Step 2
- Image compression with jpegtran, pngcrush, convert. I use pngcrush often. I read about jpegtran in Yahoo’s Best Practices for Speeding Up Your Web Site. I wrote a bit about image compression a while ago and briefly experimented with image compression using imagemagick on Inspiredology’s images.
Step 3
- Addition of expires headers and disabling ETags: These are standard optimization suggestions. Jon Jensen wrote about using these a bit here and here.
Step 4
- Serving gzipped content with mod_deflate: Also a fairly standard optimization suggestion. Although, I should note I had some issues gzipping a couple of the files and since the site was in a temporary location, I didn’t spend much time troubleshoo
- A bit more cleanup of rogue html and CSS files. In particular, there was one HTML file requested that didn’t have any content in it and another that had JavaScript that I appended to the combined JavaScript file (combined

A side-by-side comparison of webpagetest.org’s original versus step 4 results highlights the reduction of requests in the waterfall and the large reduction in requests on the repeat view:

What Next?

At this point, webpagetest.org suggests the following changes:

Gzipping the remaining components has a potential of reducing total bytes of the first request by ~10%.
Additional image compression has the potential of reducing total bytes of the first request by about ~6%. This metric is based on their image compression check: “JPEG—Within 10% of a photoshop quality 50 will pass, up to 50% larger will warn and anything larger than that will fail.” Quite a few of Inspiredology’s jpgs did not pass this test and could be optimized further.
Use a CDN. This is a common optimization suggestion, but the cost of a CDN isn’t always justified for smaller sites.

I would suggest:

Revisiting CSS spriting to further optimize. I only spent a short time spriting and didn’t work out all the kinks. There were a few requests that I didn’t sprite because they were repeating elements, but repeating elements can be sprited together. Another 5 requests might be eliminated with additional CSS spriting.
Server-optimization: Inspiredology runs on WordPress. We’ve used the wp-cache plugin for a couple of our clients running WordPress, which I believe helps. But note that the case study presented here is a static page with static assets, so there is obviously a huge gain to be had by optimizing serving images, CSS, and JavaScript.
Database optimization: Again, there’s no database in play in this static page experiment. But there’s always room for improvement on database optimization. Josh Tolley recently made performance improvements for one of our clients running on Rails with postgreSQL using pgsi, our open source postgreSQL performance reporting tool, and had outrageously impressive benchmarked improvements.
I just read an article about CSS selectors. The combined.css file I created for this case study has 2000 lines. Although there might be only a small win with optimization here, surely optimization and cleanup of that file can be beneficial.
I recently wrote about several jQuery tips, including performance optimization techniques. This isn’t going to improve the serving of static assets, but it would be another customer-facing enhancement that can improve the usability of the site.

I highly recommend reading Yahoo’s Best Practices on Speeding Up Your Web Site. They have a great summary of performance recommendations, covering the topics described in this article and lots more.

PostgreSQL 9.0 High Performance Review

2010-11-30T00:00:00+00:00

I recently had the privilege of reading and reviewing the book PostgreSQL 9.0 High Performance by Greg Smith. While the title of the book suggests that it may be relevant only to PostgreSQL 9.0, there is in fact a wealth of information to be found which is relevant for all community supported versions of Postgres.

Acheiving the highest performance with PostgreSQL is definitely something which touches all layers of the stack, from your specific disk hardware, OS and filesystem to the database configuration, connection/data access patterns, and queries in use. This book gathers up a lot of the information and advice that I’ve seen bandied about on the IRC channel and the PostgreSQL mailing lists and presents it in one place.

While seemingly related, I believe some of the main points of the book could be summed up as:

Measure, don’t guess. From the early chapters which cover the lowest-level considerations, such as disk hardware/configuration to the later chapters which cover such topics as query optimization, replication and partitioning, considerable emphasis is placed on determining the metrics by which to measure performance before/after specific changes. This is the only way to determine the impact the changes you make have.
Tailor to your specific needs/workflows. While there are many good rules of thumb out there when it comes to configuration/tuning, this book emphasizes the process of determining/refining those more general numbers to tailoring configuration/setup to your specific database’s needs.
Review the information the database system itself gives you. Information provided by the pg_stat_* views can be useful in identifying bottlenecks in queries, unused/underused indexes.

This book also introduced me to a few goodies which I had not encountered previously. One of the more interesting ones is the pg_buffercache contrib module. This suite of functions allows you to peek at the internals of the shared_buffers cache to get a feel for which relations are heavily accessed on a block-by-block basis. The examples in the book show this being used to more accurately size shared_buffers based on the actual number of accesses to specific portions of different relations.

I found the book to be well-written (always a plus when reading technical books) and felt it covered quite a bit of depth given its ambitious scope. Overall, it was an informative and enjoyable read.

Speeding up the Spree demo site

2010-11-02T00:00:00+00:00

There’s a lot that can be done to speed up Spree, and Rails apps in general. Here I’m not going to deal with most of that. Instead I want to show how easy it is to speed up page delivery using standard HTTP server tuning techniques, demonstrated on demo.spreecommerce.com.

First, let’s get a baseline performance measure from the excellent webpagetest.org service using their remote Internet Explorer 7 tests:

First page load time: 2.1 seconds
Repeat page load time: 1.5 seconds

The repeat load is faster because the browser has images, JavaScript, and CSS cached, but it still has to check back with the server to make sure they haven’t changed. Full details are in this initial report.

The demo.spreecommerce.com site is run on a Xen VPS with 512 MB RAM, CentOS 5 i386, Apache 2.2, and Passenger 2.2. There were several things to tune in the Apache httpd.conf configuration:

mod_deflate was already enabled. Good. That’s a big help.
Enable HTTP keepalive: KeepAlive On and KeepAliveTimeout 3
Limit Apache children to keep RAM available for Rails: StartServers 5, MinSpareServers 2, MaxSpareServers 5
Limit Passenger pool size to 2 child processes (down from the default 6), to queue extra requests instead of using slow swap memory: PassengerMaxPoolSize 2
Enable browser & intermediate proxy caching of static files: ExpiresActive On and ExpiresByType image/jpeg “access plus 2 hours” etc. (see below for full example)
Disable ETags which aren’t necessary once Expires is enabled: FileETag None and Header unset ETag
Disable unused Apache modules: free up memory by commenting out LoadModule proxy, proxy_http, info, logio, usertrack, speling, userdir, negotiation, vhost_alias, dav_fs, autoindex, most authn_* and authz_* modules
Disable SSLv2 (for security and PCI compliance, not performance): SSLProtocol all -SSLv2 and SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP

After making these changes, without tuning Rails, Spree, or the database at all, a new webpagetest.org run reports:

First page load time: 1.2 seconds
Repeat page load time: 0.4 seconds

That’s an easy improvement, a reduction of 0.9 seconds for the initial load and 1.1 seconds for a repeat load! Complete details are in this follow-on report.

The biggest wins came from enabling HTTP keepalive, which allows serving multiple files from a single HTTP connection, and enabling static file caching which eliminates the majority of requests once the images, JavaScript, and CSS are cached in the browser.

Note that many of the resource-limiting changes I made above to Apache and Passenger would be too restrictive if more RAM or CPU were available, as is typical on a dedicated server with 2 GB RAM or more. But when running on a memory-constrained VPS, it’s important to put such limits in place or you’ll practically undo any other tuning efforts you make.

I wrote about these topics a year ago in a blog post about Interchange ecommerce performance optimization. I’ve since expanded the list of MIME types I typically enable static asset caching for in Apache. Here’s a sample configuration snippet to put in the container in httpd.conf:

    ExpiresActive On
    ExpiresByType image/gif   "access plus 2 hours"
    ExpiresByType image/jpeg  "access plus 2 hours"
    ExpiresByType image/png   "access plus 2 hours"
    ExpiresByType image/tiff  "access plus 2 hours"
    ExpiresByType text/css    "access plus 2 hours"
    ExpiresByType image/bmp   "access plus 2 hours"
    ExpiresByType video/x-flv "access plus 2 hours"
    ExpiresByType video/mpeg  "access plus 2 hours"
    ExpiresByType video/quicktime "access plus 2 hours"
    ExpiresByType video/x-ms-asf  "access plus 2 hours"
    ExpiresByType video/x-ms-wm   "access plus 2 hours"
    ExpiresByType video/x-ms-wmv  "access plus 2 hours"
    ExpiresByType video/x-ms-wmx  "access plus 2 hours"
    ExpiresByType video/x-ms-wvx  "access plus 2 hours"
    ExpiresByType video/x-msvideo "access plus 2 hours"
    ExpiresByType application/postscript        "access plus 2 hours"
    ExpiresByType application/msword            "access plus 2 hours"
    ExpiresByType application/x-javascript      "access plus 2 hours"
    ExpiresByType application/x-shockwave-flash "access plus 2 hours"
    ExpiresByType image/vnd.microsoft.icon      "access plus 2 hours"
    ExpiresByType application/vnd.ms-powerpoint "access plus 2 hours"
    ExpiresByType text/x-component              "access plus 2 hours"

Of course you’ll still need to tune your Spree application and database, but why not tune the web server to get the best performance you can there?

CSS Sprites and a “Live” Demo

2010-09-06T00:00:00+00:00

I’ve recently recommended CSS sprites to several clients, but the majority don’t understand what CSS sprites are or what their impact is. In this article I’ll present some examples of using CSS sprites and their impact.

First, an intro: CSS sprites is a technique that uses a combination of CSS rules and a single background image that is an aggregate of many smaller images to display the image elements on a webpage. The CSS rules set the boundaries and offset that define the part of the image to show. I like to refer to the technique as analogous to the “Ouija board”; the CSS acts as the little [rectangular] magnifying glass to show only a portion of the image.

It’s important to choose which images should be in a sprite based on how much each image is repeated throughout a site’s design and how often it might be replaced. For example, design border images and icons will likely be included in a sprite since they may be repeated throughout a site’s appearance, but a photo on the homepage that’s replaced daily is not a good candidate to be included in a sprite. I also typically exclude a site’s logo from a sprite since it may be used by externally linking sites. End Point uses CSS sprites, but only for a few elements:

End Point’s CSS sprite image only contains borders and Twitter and LinkedIn images.

“Live” Demo

I wanted to implement CSS sprites to demonstrate their performance impact. I chose to examine the homepages of my favorite ski resorts in Utah, Snowbird and Alta. First, I used WebPagetest.org to get a benchmark for each homepage:

Without sprites, Alta’s homepage loaded in 3.440 seconds with a repeat request load time of 1.643 seconds. 22 requests are made on the homepage.

Without sprites, Snowbird’s homepage loaded in 7.070 seconds with a repeat request load time of 3.146 seconds. 57 requests are made on the homepage.

After I benchmarked the two pages, I downloaded each homepage and its files and examined the images to build a CSS sprite image. I created the sprite images shown below. I chose to exclude the logo from each sprite, in addition to other time-sensitive images. Each sprited image contains navigation elements, icons, and a few homepage specific images.

My resulting sprited image for Alta.

My resulting sprited image for Snowbird.

I updated the HTML to remove individual image requests. Below are some examples:

Before:

<a href="https://www.alta.com/contact">
<img src="./alta_files/banner_contact.gif" name="ContactUs" width="100" height="20" border="0">
</a>

After:

<a href="/" class="sprite" id="banner_contact">
</a>

Before:

<a href="https://web.archive.org/web/20101128010936/http://shop.alta.com/CS/Browse.aspx?Catalog=AltaRetail&Category=Retail+Items" target="_self">
<img src="./alta_files/altaskishoplogo.jpg" alt="March Madness Logo" width="180" height="123" border="0">
</a>

After:

<a href="https://web.archive.org/web/20101128010936/http://shop.alta.com/CS/Browse.aspx?Catalog=AltaRetail&Category=Retail+Items" target="_self" id="skishoplogo" class="sprite">
</a>

Before:

<div class="icon">
<img src="./snowbird_files/icon_less_rain.gif" border="0" alt="Sunny" title="Sunny">
</div>

After:

<div class="icon sprite less_rain">
</div>

Before:

<div>
<input type="image" src="./snowbird_files/btn_check_rates.gif" border="0" alt="Check Rates">
</div>

After:

<div class="sprite" id="check_rates">
</div>

There are a few CSS tips to be aware of during CSS sprite implementation, such as:

Padding on sprited elements will affect the sprite position.
Links must have the “display:block” rule (combined with floating rules) to enforce a height and width.
Parts of the sprite that are repeating may not have any other elements along the repeating axis. For example, in End Point’s sprite, the top and bottom repeating border are in the sprite. No other images may be included in the sprite to the left or right of the borders.

The CSS for the new sprites is shown below. For all sprited elements, a height and width is set in addition to a background position. The height, width, and background position are the rules that define the region of the sprited image to show.

Alta CSS sprite rules

.sprite { background-image: url(sprites.gif); display: block; float: left; }
a#banner_logo { width: 100px; height: 100px; }
div#banner_top { width: 600px; height: 100px; background-position: -100px 0px; }
a#banner_home { width: 100px; height: 20px; background-position: -700px 0px; }
a#banner_contact { width: 100px; height: 20px; background-position: -700px -20px; }
a#banner_sitemap { width: 100px; height: 20px; background-position: -700px -40px; }
a#banner_press { width: 100px; height: 20px; background-position: -700px -60px; }
a#banner_weather { width: 100px; height: 20px; background-position: -700px -80px; }
a#skishoplogo { width: 180px; height: 123px; background-position: -298px 123px; }
a#skihistory { width: 164px; height: 61px; background-position: -479px 123px; margin-right: 40px; }
a#altaenviron { width: 298px; height: 61px; background-position: 0px 123px; }

Snowbird CSS sprite rules

.sprite { background: url(sprites.gif); }
a.sprite { display: block; float: left; margin-right: 5px; border: 2px solid #FFF; }
div#headerWeather div.icon { width: 36px; height: 26px; }
div.less_rain { background-position: -118px -142px; }
div.sunny { background-position: -177px -116px; }
div.partly_cloudy { background-position: -138px -116px; }
a#facebook { width: 66px; height: 25px; background-position: 0px -142px; }
a#twitter { width: 55px; height: 25px; background-position: -66px -142px; }
a#youtube { width: 48px; height: 25px; background-position: 0px -116px; }
a#flickr { width: 94px; height: 25px; background-position: -48px -116px; }
a#picofday { width: 218px; height: 22px; border: none; margin: 2px 0px 0px 2px; }
div#check_rates { width: 89px; height: 21px; background-position: 0px -261px; margin: 10px 0px; float: right; }
div#br_corner_light { width: 17px; height: 14px; background-position: -89px -268px; }
div#bl_corner_light { width: 17px; height: 14px; background-position: -105px -268px; }

After spriting the images shown above, I returned to WebPagetest.org to examine the impact of the sprites. In most cases, I maintained the image format to limit the performance change to spriting only. I also did not change HTML or CSS even if I noticed other performance improvement opportunities. Here are the new results:

With sprites, Alta’s homepage loaded in 2.768 seconds with a repeat request load time of 1.093 seconds.

With sprites, Snowbird’s homepage loaded in 6.289 seconds with a repeat request load time of 2.513 seconds.

A summary of differences shows:

Both pages decreased the number of requests by 10.
Alta decreased homepage load by 0.672 seconds, or 20% of the original page load. Snowbird decreased homepage load by 0.781 seconds, or 11% of the original page load.
On repeat views, Alta’s homepage load would be decreased by 0.665 seconds, or 40% of the original repeated page load. On repeat view, Snowbird’s homepage load would be decreased by 0.633 seconds, or 20% of the original repeat page load.

There is no reason to avoid sprites if a design has repeating elements or icons. An increase in performance will reduce load time for the customer. If a CDN is in place, CSS sprites can result in a decreased bandwidth cost. The improved performance also can indirectly positively influence search engine performance since search engines may use performance as an influencing factor in search. This improved performance can also make for a better mobile browsing experience. In our examples, the use of CSS sprites decreased first request page load time by 10-20%, but this amount may vary depending on the frequency of images used in a site’s design.

Efficiency of find -exec vs. find | xargs

2010-07-28T00:00:00+00:00

This is a quick tip for anyone writing a cron job to purge large numbers of old files.

Without xargs, this is a pretty common way to do such a purge, in this case of all files older than 31 days:

find /path/to/junk/files -type f -mtime +31 -exec rm -f {} \;

But that executes rm once for every single file to be removed, which adds a ton of overhead just to fork and exec rm so many times. Even on modern operating systems that are so efficient with fork, it can easily increase the I/O and load and runtime by 10 times or more than just running a single rm command with a lot of file arguments.

Instead do this:

find /path/to/junk/files -type f -mtime +31 -print0 | xargs -0 -r rm -f

That will run xargs once for each very long list of files to be removed, so the overhead of fork & exec is incurred very rarely, and the job can spend most of its effort actually unlinking files. (The xargs -r option says not to run the command if there is no input to xargs.)

How long can the argument list to xargs be? It depends on the system, but xargs –show-limits will tell us. Here’s output from a RHEL 5 x86_64 system (using findutils 4.2.27):

% xargs --show-limits
Your environment variables take up 2293 bytes
POSIX lower and upper limits on argument length: 2048, 129024
Maximum length of command we could actually use: 126731
Size of command buffer we are actually using: 126731

The numbers are similar on Debian Etch and Lenny.

And here’s output from an Ubuntu 10.04 x86_64 system (using findutils 4.4.2):

% xargs --show-limits
Your environment variables take up 1370 bytes
POSIX upper limit on argument length (this system): 2093734
POSIX smallest allowable upper limit on argument length (all systems): 4096
Maximum length of command we could actually use: 2092364
Size of command buffer we are actually using: 131072

Roughly 2 megabytes of arguments is a lot. But even the POSIX minimum of 4 kB is a lot better than processing one file at a time.

It doesn’t usually make much of a difference, but we can tune even more. Make sure the maximum number of files is processed at one time by first changing to the base directory so that the relative pathnames are shorter:

cd /path/to/junk/files && find . -type f -mtime +31 -print0 | xargs -0 -r rm -f

That way each file argument is shorter, e.g. ./junkfile compared to /path/to/junk/files/junkfile.

The above assumes you’re using GNU findutils, which includes find -print0 and xargs -0 for processing ASCII NUL-delimited filenames for safety when filenames include embedded spaces, newlines, etc.

Why is my load average so high?

2010-07-15T00:00:00+00:00

One of the most common ways people notice there’s a problem with their server is when Nagios, or some other monitoring tool, starts complaining about a high load average. Unfortunately this complaint carries with it very little information about what might be causing the problem. But there are ways around that. On Linux, where I spend most of my time, the load average represents the average number of process in either the “run” or “uninterruptible sleep” states. This code snippet will display all such processes, including their process ID and parent process ID, current state, and the process command line:

#!/bin/sh

ps -eo pid,ppid,state,cmd |\
    awk '$3 ~ /[RD]/ { print $0 }'

Most of the time, this script has simply confirmed what I already anticipated, such as, “PostgreSQL is trying to service 20 times as many simultaneous queries as normal.” On occasion, however, it’s very useful, such as when it points out that a backup job is running far longer than normal, or when it finds lots of “[pdflush]” operations in process, indicating that the system was working overtime to write dirty pages to disk. I hope it can be similarly useful to others.

On Linux, noatime includes nodiratime

2010-02-09T00:00:00+00:00

Note to performance-tweaking Linux sysadmins, pointed out to me by Selena Deckelmann: On Linux, the filesystem attribute noatime includes nodiratime, so there’s no need to say both “noatime,nodiratime” as I once did. (See this article on atime for details if you’re not familiar with it.)

Apparently the nodiratime attribute was added later as a subset of noatime applying only to directories to still offer a bit of performance boost in situations where noatime on files would cause trouble (as with mutt and a few other applications that care about atimes).

See also the related newer relatime attribute in the mount(8) manpage.

JPEG compression: quality or quantity?

2009-12-24T00:00:00+00:00

There are many aspects of JPEG files that are interesting to web site developers, such as:

The optimal trade off between quality and file size for any encoder and uncompressed source image.
Reducing size of an existing JPEG image when the uncompressed source is unavailable, but still finding the same optimal trade-off.
Comparison of different encoders and/or settings for quality at a given file size.

Two essential factors are file size and image quality. Bytes are objectively measurable, but image quality is much more nebulous. What to one person is a perfectly acceptable image is to another a grotesque abomination of artifacts. So the quality factor is subjective. For example, Steph sent me some images to compare compression artifacts. Here is the first one with three different settings in ImageMagick: 95, 50, and 8:

Compare the subtle (or otherwise) differences in the following images (mouseover shows the filesize and compression setting):

Mouseover each image for the file size and ImageMagick compression setting. Additional comparisons are below. Each image can be opened in a separate browser tab for easy A/B comparison. I think many would find the setting of 8 to have too many artifacts, even though it’s 10 times smaller than image compressed at a setting of 95. Some would find the setting of 50 to be an acceptable tradeoff between quality and size, since it sends 3.4 times fewer bytes.

Here is the code I wrote to make the comparison (shell script is great for this stuff):

#!/bin/bash
HTML_OUTFILE=comparison.html
echo '<html>' > $HTML_OUTFILE

write_img_html () {
    size=`du -h --apparent-size $1 | cut -f 1`
    if [ -n "$2" ]; then
       qual="setting: $2"
    fi
    cat <<EOF >>$HTML_OUTFILE
<a href="$1"><img src="$1" title="size: $size $qual"></a>
EOF
}

for name in image1 image2; do
    orig=$name-original.jpg
    resized=$name-300.png

    echo Resizing $orig to 300 on longest side: $resized...
    convert $orig -resize 300x300 $resized
    write_img_html $resized "lossless"

    for quality in 100 95 85 50 20 8 1; do
        echo Creating JPEG quality $quality...
        jpeg=$name-300-q-$quality.jpg
        convert $resized -strip -quality $quality $jpeg
        write_img_html $jpeg $quality
    done
done

Another factor that often comes into play is how artifacts in the image (e.g. aliasing, ringing, noise) combine with JPEG compression artifacts to exacerbate quality problems. So one way to get smaller file sizes is to reduce the other types of artifacts in the image, thereby allowing higher JPEG compression.

The most common source of artifacts is image resizing. If you are resizing the images, I strongly recommend using a program that has a high quality filter. Irfanview and ImageMagick are two good choices.

The ideal situation is this:

Uncompressed source image
Full-resolution if you will be handling the resize
Absent artifacts such as aliasing
Resize performed with good software like ImageMagick
JPEG compression chosen based on subjective quality assessment.

Choosing the trade-off between quality and file size is difficult in part because it varies by image content. Images with lots of small color details (e.g. bright fabric threads; AKA high spatial frequency chroma) stand less compression than images that only have medium sized details that do not have important and minute color information.

One of the settings that is important for small web images is removal of the color space profile (e.g. sRGB). The only time it is needed is when there is a good reason for using non-sRGB JPEG, such as when you are certain that your users will have color managed browsers. Removing it can shave off 5KB or so; software will assume images without profiles have an sRGB profile. It can be removed with the -strip parameter of ImageMagick.

As for choosing the specific compression settings, keep in mind that there are over 30 different types of options/techniques that can be used in compressing the image. Most image programs simplify that to a sliding scale from 0 to 100, 1 to 12, or something else. Keep in mind that even when programs use the same scale (e.g. 0 to 100), they probably have different ideas of what the numbers mean. 95 in one program may be very different than 95 in another.

If bandwidth is not an issue, then I use a setting of 95 on ImageMagick, because in normal images I can’t tell the difference between 95 and 100. But when file size in an important concern, I consider 85 to be the optimal setting. In this image, the difference should be clear, but I generally find that cutting filesize in half is worth it. Below 85, the artifacts are too onerous for my taste.

You don’t often hear about web site visitors’ dissatisfaction with compression artifacts, so you might be tempted to just reduce file sizes even beyond the point where it gets noticable. But I think there is a subliminal effect from the reduced image quality. Visitors may not stop visiting the site immediately, but my gut feeling is it leaves them with a certain impression in their mind or taste in their mouth. I would guess that user testing might result in comments such as “the X web site is not the same high-grade quality as the Y web site”, even if they don’t put it into words as specific as “the compression artifacts make X look uglier than Y”. Even if that pet theory is true, it still has to be balanced against the benefit of faster page loading times.

Ideally, the tradeoff between quality and page loading time would be a choice left to the user. Those who prefer fewer artifacts could set their browser to download larger, less-compressed image files than the default, while users with low bandwidth could set it for more compressed images to get a faster page load at the expense of quality. I could imagine an Apache module and corresponding Firefox add-on some day.

Regarding the situation where you want to reduce the file size of existing JPEGs, my advice is to first try (hard) to get the original source files. You can do better (for any given quality/size tradeoff) from those than you can by just manipulating the existing files. If that’s not possible, then the suboptimal workflows like jpegtran, jpegoptim, and doing a full decompress/recompress are the only alternative.

As far as comparing different encoders, I haven’t really looked into that except to compare ImageMagick and Photoshop, where I (subjectively) determined they both had about similar quality for file size (and vice-versa).

Here are all the comparison images. The file size and ImageMagick quality setting are in the rollover. I suggest opening images in browser tabs for easy A/B comparison.

Optimizing media delivery with Cloudinary

Resizing and cropping

Gravity position

Automatic format

Other features

NuxtJS integration

Alternatives

Image compression: WebP presets, HEIC, AVIF, JPEG XL

Can I use it?

How do I create WebP images?

cwebp settings

Batch conversion

Competitors to WebP

HEIC

AVIF

JPEG XL

Use WebP now

Reference

Web Projects for a Rainy Day

Major Version Updates

Sample of a client’s stack review

Content Cleanup & SEO Review

Reporting

Documentation

Disaster Recovery Tests

Redesign

Automated Testing

Versioning & Deployment Tools

Security Audit and Monitoring

Optimization and Performance:

Refactoring

Decreasing your website load time

Inline styles and scripts for the topmost content

Deferred loading of ads

Lazy load for images

Server-side caching

GZip compression

Summary

Extensible Binary Encoding with CBOR

Encoding Binary Data

Optimizing CBOR with Tags

When to Use CBOR

Roundup of some useful websites

Squoosh image compressor

Nerd Fonts

glot.io code pastebin

Firefox Send

transfer.sh command-​line file sharing

Doing what you don’t want to do

Learning from data basics II: simple Bayesian Networks

Naive Bayes as a simple Bayesian Network

Expressing variable dependencies as a graph

Using the graph representation in practice - the chain rule for Bayesian Networks

Coding the updated probabilistic model

The results

Learning from data basics: the Naive Bayes model

The problem

Thinking about the solution—​using probability theory

Quick theory refresher for programmers

Random variables

Factors and probability distributions

Simple Ruby modeling of random variables and factors

The scenario setup

Where is the complexity?

Very simple but powerful optimization: The Naive Bayes model

Implementing the Naive Bayes model

The results

Temporary tables in SQL query optimization

Interchange Loop Optimization

Loop Processing

ITL vs. Loop Tags

An Optimization Example

WebP images experiment on End Point website

Comparing quality & size

Browser support

Web server rewrites

301/302 redirect option

Chrome image download behavior

Batch conversion

Full-page download sizes compared

transfer.sh command-line file sharing

Thinking about the solution—using probability theory