https://www.endpointdev.com/blog/tags/compression/2022-03-01T00:00:00+00:00End Point DevOptimizing media delivery with Cloudinaryhttps://www.endpointdev.com/blog/2022/03/optimizing-image-delivery-with-cloudinary/2022-03-01T00:00:00+00:00Juan Pablo Ventoso
<p><img src="/blog/2022/03/optimizing-image-delivery-with-cloudinary/la-cumbrecita-202201.webp" alt="Beautiful cloudy mountain scene with river flowing by lush banks with people swimming, relaxing, and walking towards multistory buildings"></p>
<!-- Photo by Juan Pablo Ventoso -->
<p>I remember how we needed to deal with different image formats and sizes years ago: From using the WordPress-style approach of automatically saving different resolutions on the server when uploading a picture, to using a PHP script to resize or crop images on the fly and return the result as a response to the frontend. Of course, many of those approaches were expensive, and not fully optimized for different browsers or device sizes.</p>
<p>With those experiences in mind, it was a nice surprise for me to discover <a href="https://cloudinary.com/">Cloudinary</a> when working on a new project a couple of months ago. It’s basically a cloud service that saves and delivers media content with a lot of transformations and management options for us to use. <a href="https://cloudinary.com/pricing">There is a free version</a> with a usage limit: Up to 25K transformations or 25 GB of storage/bandwidth, which should be enough for most non-enterprise websites. The cheapest paid service is $99 per month.</p>
<p>Here’s a list of the image features we used on that project. I know they offer many other things that can be used as well, but I think this is a good start for anyone who hasn’t used this service yet:</p>
<h3 id="resizing-and-cropping">Resizing and cropping</h3>
<p>When you make a request for an image, you can instruct the Cloudinary API to <a href="https://cloudinary.com/documentation/resizing_and_cropping">retrieve it with a given size</a>, which will trigger a transformation on their end before delivering the content. You can also use a cropping method: fill, fill with padding, scale down, etc.</p>
<h3 id="gravity-position">Gravity position</h3>
<p>When we specify a <a href="https://cloudinary.com/documentation/resizing_and_cropping#control_gravity">gravity position</a> to crop an image, the service will keep the area of the image we decide to use as the focal point. We can choose a corner (for example, top left), but also—and this is probably one of the most interesting capabilities on this service—we can specify <a href="https://cloudinary.com/documentation/transformation_reference#g_special_position">“special positions”</a>: By using machine learning, we can instruct Cloudinary to use face detection, or even focus on other objects, like an animal or a flower in the picture.</p>
<h3 id="automatic-format">Automatic format</h3>
<p>Another cool feature is the <a href="https://cloudinary.com/documentation/transformation_reference#f_auto">automatic format</a>, which will use your request headers to find the most efficient picture format for your browser type and version. For example, if the browser supports it, Cloudinary will return the image in WebP format, which is generally more efficient than standard JPEG, as End Point CTO Jon Jensen demonstrates on his recent <a href="https://www.endpointdev.com/blog/2022/02/webp-heif-avif-jpegxl/">blog post</a>.</p>
<p><img src="/blog/2022/03/optimizing-image-delivery-with-cloudinary/image-response.jpg" alt="Screenshot of Chrome browser dev tools showing network response for a WebP image"><br>
Automatic format in action: Returning a WebP image in Chrome</p>
<h3 id="other-features">Other features</h3>
<p>There are many other options for us to choose when querying their API, like setting up a default placeholder when we don’t have an image, applying color transformations, removing red eyes, among other things. The <a href="https://cloudinary.com/documentation/transformation_reference">Transformation reference page</a> on their documentation section is a great resource.</p>
<h3 id="nuxtjs-integration">NuxtJS integration</h3>
<p>The project I mentioned above was a <a href="https://nuxtjs.org/">NuxtJS</a> application with a <a href="https://nodejs.org/">Node.js</a> backend. And since there’s a <a href="https://cloudinary.nuxtjs.org/">NuxtJS module for Cloudinary</a>, it made sense to use it instead of building the queries to the API from scratch.</p>
<p>The component works great, except for one bug that we found that didn’t allow us to fully use their image component with server-side rendering enabled. Between that drawback and some issues trying to use the lazy loading setting, we ended up creating a Vue component ourselves that used a standard image tag instead. But we still used their component to generate most of the API calls and render the results.</p>
<p>Below is an example of using the Cloudinary Image component on a Vue template:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-html" data-lang="html"><<span style="color:#b06;font-weight:bold">template</span>>
<<span style="color:#b06;font-weight:bold">div</span>>
<<span style="color:#b06;font-weight:bold">cld-image</span>
<span style="color:#369">:public-id</span>=<span style="color:#d20;background-color:#fff0f0">"publicId"</span>
<span style="color:#369">width</span>=<span style="color:#d20;background-color:#fff0f0">"200"</span>
<span style="color:#369">height</span>=<span style="color:#d20;background-color:#fff0f0">"200"</span>
<span style="color:#369">crop</span>=<span style="color:#d20;background-color:#fff0f0">"fill"</span>
<span style="color:#369">gravity</span>=<span style="color:#d20;background-color:#fff0f0">"auto:subject"</span>
<span style="color:#369">radius</span>=<span style="color:#d20;background-color:#fff0f0">"max"</span>
<span style="color:#369">fetchFormat</span>=<span style="color:#d20;background-color:#fff0f0">"auto"</span>
<span style="color:#369">quality</span>=<span style="color:#d20;background-color:#fff0f0">"auto"</span>
<span style="color:#369">alt</span>=<span style="color:#d20;background-color:#fff0f0">"An image example with Cloudinary"</span>
/>
</<span style="color:#b06;font-weight:bold">div</span>>
</<span style="color:#b06;font-weight:bold">template</span>>
</code></pre></div><h3 id="alternatives">Alternatives</h3>
<p>Of course, Cloudinary is not the only image processing and CDN company out there: There are other companies offering similar services, like <a href="https://www.cloudflare.com/products/cloudflare-images/">Cloudflare images</a>, <a href="https://www.cloudimage.io/">Cloudimage</a>, or <a href="https://imagekit.io/">imagekit.io</a>.</p>
<p>Do you know any other good alternatives, or have you used any other Cloudinary feature that is not listed here? Feel free to add a comment below!</p>
Image compression: WebP presets, HEIC, AVIF, JPEG XLhttps://www.endpointdev.com/blog/2022/02/webp-heif-avif-jpegxl/2022-02-15T00:00:00+00:00Jon Jensen
<p><img src="/blog/2022/02/webp-heif-avif-jpegxl/20211223-224644-sm.webp" alt="Rubble of a demolished house in front of a guilty-looking Caterpillar excavator"></p>
<!-- Photo by Jon Jensen -->
<p>How time flies. Eight years ago I wrote the blog post <a href="/blog/2014/01/webp-images-experiment-on-end-point/">WebP images experiment on End Point website</a> to describe and demonstrate how the WebP image format can store an equivalent-quality image in a much smaller file size than the older JPEG, PNG, and GIF formats.</p>
<p>My WebP examples there were 17–23% of the JPEGs they came from, or about 5–6× smaller. While experimenting with higher levels of compression, I found that WebP tends to leave less-noticeable artifacts than JPEG does.</p>
<p>The main drawback at the time was that, among major browsers, only Chrome and Opera supported WebP, and back then, Chrome was far less popular than it is now.</p>
<h3 id="can-i-use-it">Can I use it?</h3>
<p>Since Apple’s iOS 14 and macOS 11 (Big Sur) became available in late 2020, the WebP image format now works in all the currently supported major operating systems and browsers: Linux, Windows, and macOS, running Chromium, Chrome, Brave, Edge, Opera, Firefox, and Safari. You can see the specifics at the ever-useful site <a href="https://www.caniuse.com/webp">“Can I use”</a>.</p>
<p>It only took about 10 years! 😁</p>
<p>So for you who are hosting websites, all your site visitors can see WebP images and animations except those vanishing few using Internet Explorer (now long past its end of support by Microsoft and dangerous to use), and people (or their organizations) who intentionally do not allow their older browsers and operating systems to be updated.</p>
<p>That means you may want to continue using JPEG, PNG, and/or GIF images in places crucial for rendering your main site features, for people to be able to understand the main things your site is trying to communicate.</p>
<p>But for any images that are less essential and primarily making it prettier, or where you don’t mind suggesting that users of old browsers upgrade to see them, WebP can now be your default image format.</p>
<h3 id="how-do-i-create-webp-images">How do I create WebP images?</h3>
<p>Mobile phones and digital cameras typically save JPEG, HEIF, or raw (uncompressed) images. Some stock photography collections offer WebP downloads, but many still use JPEG.</p>
<p>So in many cases you won’t be starting with a WebP image, and you’ll convert some other image to WebP, likely after cropping, scaling, and other adjustments.</p>
<p>GIMP (GNU Image Manipulation Program) supports WebP images since about 2017, and Adobe Photoshop does not natively but can use the <a href="https://developers.google.com/speed/webp/docs/webpshop">free WebPShop plugin</a>.</p>
<p>The oldest way to convert images to WebP, and still very useful for batch processing or fine-tuning, is <a href="https://developers.google.com/speed/webp">Google’s WebP converter “cwebp”</a>.</p>
<h3 id="cwebp-settings">cwebp settings</h3>
<p>With the power “cwebp” offers comes some complexity, but it is mostly harmless.</p>
<p>Run this to see its many options:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sh" data-lang="sh">cwebp -longhelp
</code></pre></div><p>Or read the same thing in its <a href="https://developers.google.com/speed/webp/docs/cwebp">online documentation</a>.</p>
<p>There among other things you will see the <code>-z</code> option which activates preset features for lossless encoding, with an integer level chosen from 0 to 9 where 0 is fastest but compresses less and 9 is slowest but compresses better. Use this to replace PNG files when you want no degradation of the image at all.</p>
<p>The documentation also shows the useful <code>-preset</code> and <code>-hint</code> options for lossy compression similar to what JPEG does, but better:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">-preset <string> ....... preset setting, one of:
default, photo, picture,
drawing, icon, text
-hint <string> ......... specify image characteristics hint,
one of: photo, picture or graph
</code></pre></div><p>The meaning of a few of those terms, especially “photo” and “picture”, was not clear to me and not defined elsewhere in the documentation that I could see.</p>
<p>To find that out I had to make a quick trip into the source code, and there are comments explaining each option’s use case a bit:</p>
<p>The comments for the <a href="https://chromium.googlesource.com/webm/libwebp/+/refs/heads/main/src/webp/encode.h#159"><code>-preset</code> option</a>:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">WEBP_PRESET_PICTURE, // digital picture, like portrait, inner shot
WEBP_PRESET_PHOTO, // outdoor photograph, with natural lighting
WEBP_PRESET_DRAWING, // hand or line drawing, with high-contrast details
WEBP_PRESET_ICON, // small-sized colorful images
WEBP_PRESET_TEXT // text-like
</code></pre></div><p>And the comments for the <a href="https://chromium.googlesource.com/webm/libwebp/+/refs/heads/main/src/webp/encode.h#88"><code>-hint</code> option</a>:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">WEBP_HINT_PICTURE, // digital picture, like portrait, inner shot
WEBP_HINT_PHOTO, // outdoor photograph, with natural lighting
WEBP_HINT_GRAPH, // Discrete tone image (graph, map-tile etc).
</code></pre></div><p>So in short, “cwebp” considers a “picture” to be indoors and close-up, while “photo” is outdoors and more likely with more distant focus. That’s good to know.</p>
<h3 id="batch-conversion">Batch conversion</h3>
<p>With that in mind, I can convert a pile of screenshots that have been collecting on my computer to refer to later. One kind of screenshots I sometimes take is of video meetings with mostly indoor views of people. I will use the “picture” preset for those.</p>
<p>A simple <code>bash</code> script works well to process many images in a row:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash"><span style="color:#080;font-weight:bold">for</span> infile in Screenshot*.png
<span style="color:#080;font-weight:bold">do</span>
<span style="color:#038">echo</span> <span style="color:#369">$infile</span>
<span style="color:#369">base</span>=<span style="color:#080;font-weight:bold">$(</span>basename <span style="color:#d20;background-color:#fff0f0">"</span><span style="color:#369">$infile</span><span style="color:#d20;background-color:#fff0f0">"</span> .png<span style="color:#080;font-weight:bold">)</span>
cwebp -preset picture -v <span style="color:#d20;background-color:#fff0f0">"</span><span style="color:#369">$infile</span><span style="color:#d20;background-color:#fff0f0">"</span> -o <span style="color:#d20;background-color:#fff0f0">"</span><span style="color:#369">$base</span><span style="color:#d20;background-color:#fff0f0">"</span>.webp
<span style="color:#080;font-weight:bold">done</span>
</code></pre></div><p>If you have many images to convert to WebP and want to do several at the same time to get done faster, you can use <a href="https://www.gnu.org/software/parallel/">GNU parallel</a>.</p>
<p><strong>My screenshots when converted from PNG to WebP consistently take about 3% of the original space, 33–35× smaller! And the quality looks about the same. Amazing.</strong></p>
<p>These screenshots are 2880×1800 pixels, mostly of Google Meet low-bandwidth video streams. The originals of these screenshots don’t look particularly good to begin with, with some blurriness. But exactly because of this, there is no reason for me to keep a larger high-quality original here. The much smaller WebP is fine.</p>
<h3 id="competitors-to-webp">Competitors to WebP</h3>
<p>Other newer image formats have also been in the works for years, chasing some of the same goals. Should we skip WebP and use one of them instead?</p>
<h4 id="heic">HEIC</h4>
<p>The <a href="https://en.wikipedia.org/wiki/High_Efficiency_Image_File_Format#HEIC:_HEVC_in_HEIF">HEIC (High-Efficiency Image Container)</a> subset of the HEIF (High Efficiency Image File Format) standard uses High Efficiency Video Coding (HEVC, H.265) for storing images with a <code>.heic</code> suffix.</p>
<p>Compared to JPEG, HEIC offers the nice advantages of smaller file sizes for the same quality level (roughly half the size of an equivalent JPEG), and animation support (to replace GIF).</p>
<p>On the downside, HEIC is encumbered by patents that limit its use for major commercial projects, even on devices licensed for consumer use. It is also slower to encode/decode. And, concerning for archivists, HEIC shows severe visual damage to the entire image if part of the file is corrupted. In contrast, with corrupted JPEG files the visual damage is typically localized to particular smaller square regions, not the entire image.</p>
<p>HEIC has been used in Apple’s operating systems since the 2017 release of the iPhone 7 and iOS 11 and macOS 10.13 (High Sierra). Support was later added to Windows 10, Android 9, and Ubuntu 20.04.</p>
<p>As of this writing, no major browsers <a href="https://caniuse.com/heif">support HEIC natively</a>, not even Apple’s own Safari.</p>
<p>So for now, HEIC is primarily used by Apple to store photos more efficiently on its mobile devices.</p>
<h4 id="avif">AVIF</h4>
<p>The <a href="https://en.wikipedia.org/wiki/AVIF">AVIF (AV1 Image File Format)</a> competes with HEIC and, confusingly, uses the same HEIF container file format that HEIC does. That confusion is reduced in practice by its use of the separate file extension <code>.avif</code>.</p>
<p><a href="https://www.caniuse.com/avif">AVIF is supported</a> in current Chrome, Firefox, and Opera. Support was added to WebKit in 2021, but it still has not made its way into Safari. It also works in newer VLC, GIMP, Windows, Android, etc.</p>
<p>Netflix has a <a href="https://netflixtechblog.com/avif-for-next-generation-image-coding-b1d75675fe4">very detailed blog post</a> comparing AVIF to JPEG and showing AVIF’s many advantages.</p>
<h4 id="jpeg-xl">JPEG XL</h4>
<p>A semi-compatible successor to JPEG has long been in the works, and JPEG XL seems likely to eventually fill that role.</p>
<p>Whereas the other new image formats mentioned above usually lose some quality when recompressing JPEG and other images that were already lossy-compressed, according to the <a href="https://jpeg.org/jpegxl/">Joint Photographic Experts Group (JPEG)</a>:</p>
<blockquote>
<p>Existing JPEG files can be losslessly transcoded to JPEG XL, significantly reducing their size.</p>
</blockquote>
<p>It has been reported that JPEG XL is expected to become available in its final standard form in 2022, and support is already available in preliminary form in some software (see the <a href="https://en.wikipedia.org/wiki/JPEG_XL">Wikipedia JPEG XL article</a>).</p>
<p>That of course means that for now, no major browsers <a href="https://caniuse.com/jpegxl">support JPEG XL natively</a>.</p>
<h3 id="use-webp-now">Use WebP now</h3>
<p>So all the other new options are not yet usable for general web images. WebP is the current obvious choice, whether you want a lossy replacement for JPEG photos, a lossless replacement for PNG images, or a replacement for GIF animations.</p>
<p>Our developers have set up automatic server-side app conversion of high-quality PNG originals to WebP or JPEG on the fly, with the image size dependent on the browser viewport size. And we have worked with Cloudinary, Cloudflare, and other CDNs to use their image conversion services. We are available to help with your projects too.</p>
<h3 id="reference">Reference</h3>
<p>The Mozilla Development Network (MDN) has excellent documentation of web image format details, filename suffixes, and support in the major browsers in its <a href="https://developer.mozilla.org/en-US/docs/Web/Media/Formats/Image_types">Image file type and format guide</a>.</p>
Decreasing your website load timehttps://www.endpointdev.com/blog/2020/01/decreasing-website-load-time/2020-01-07T00:00:00+00:00Juan Pablo Ventoso
<p><img src="/blog/2020/01/decreasing-website-load-time/mobile-desktop-browsing.jpg" alt="Decreasing our website load time" /> <a href="https://www.flickr.com/photos/johanl/6798184016/">Photo</a> by <a href="https://www.flickr.com/photos/johanl/">Johan Larsson</a>, used under <a href="https://creativecommons.org/licenses/by/2.0/">CC BY 2.0</a></p>
<p>We live in a competitive world, and the web is no different. Improving latency issues is crucial to any Search Engine Optimization (SEO) strategy, increasing the website’s ranking and organic traffic (visitors from search engines) as a result.</p>
<p>There are many factors that can lead to a faster response time, including optimization of your hosting plan, server proximity to your main traffic source, or utilization of a Content Distribution Network (CDN) if you are expecting visitors on an international level. Some of these solutions and many others can be implemented with only a couple hours of coding.</p>
<h3 id="inline-styles-and-scripts-for-the-topmost-content">Inline styles and scripts for the topmost content</h3>
<p>Nobody enjoys waiting for long load times. When opening a Google search link, being met with a blank page or a loading GIF for several seconds can seem agonizing. That’s why optimizing the initial rendering of your page is crucial.</p>
<p>The content that immediately appears to the user without the need to scroll down is referred to as “above-the-fold”. This is where your optimization efforts should be aimed. So here’s a plan to load and display as quickly as possible:</p>
<ul>
<li>
<p>First, differentiate the critical styles and scripts you need to render the topmost content, and separate them from the rest of our stylesheet and external script references.</p>
</li>
<li>
<p>Then, <a href="https://www.imperva.com/learn/performance/minification/">minify</a> the separated <a href="https://csscompressor.com/">styles</a> and <a href="https://jscompress.com/">scripts</a>, and insert them directly on our page template, right before the closing <code></head></code> tag.</p>
</li>
<li>
<p>Finally, take the stylesheet and scripts link references from the <code><head></code> tag (where it’s usually located) and move them to the end of the above-the-fold content.</p>
</li>
</ul>
<p>Now, the user won’t have to wait until all references are loaded before seeing content. <b>Tip</b>: Remember to use the <a href="https://developer.mozilla.org/en-US/docs/Web/HTML/Element/script#attr-async">async</a> tag on scripts whenever possible.</p>
<p><strong>example.html:</strong></p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-html" data-lang="html"><<span style="color:#b06;font-weight:bold">head</span>>
<<span style="color:#b06;font-weight:bold">style</span>>{<span style="color:#a61717;background-color:#e3d2d2">above-the-fold</span> <span style="color:#a61717;background-color:#e3d2d2">minified</span> <span style="color:#a61717;background-color:#e3d2d2">inline</span> <span style="color:#a61717;background-color:#e3d2d2">styles</span> <span style="color:#a61717;background-color:#e3d2d2">goes</span> <span style="color:#a61717;background-color:#e3d2d2">here</span>}</<span style="color:#b06;font-weight:bold">style</span>>
<<span style="color:#b06;font-weight:bold">script</span> <span style="color:#369">type</span>=<span style="color:#d20;background-color:#fff0f0">"text/javascript"</span>>{above-the-fold critical scripts goes here}</<span style="color:#b06;font-weight:bold">script</span>>
</<span style="color:#b06;font-weight:bold">head</span>>
<<span style="color:#b06;font-weight:bold">body</span>>
<<span style="color:#b06;font-weight:bold">div</span> <span style="color:#369">class</span>=<span style="color:#d20;background-color:#fff0f0">"above-the-fold-content"</span>></<span style="color:#b06;font-weight:bold">div</span>>
<<span style="color:#b06;font-weight:bold">link</span> <span style="color:#369">rel</span>=<span style="color:#d20;background-color:#fff0f0">"stylesheet"</span> <span style="color:#369">href</span>=<span style="color:#d20;background-color:#fff0f0">"{below-the-fold minified stylesheet reference goes here}"</span> />
<<span style="color:#b06;font-weight:bold">script</span> <span style="color:#369">async</span> <span style="color:#369">src</span>=<span style="color:#d20;background-color:#fff0f0">"{below-the-fold minified javascript reference goes here}"</span> />
<<span style="color:#b06;font-weight:bold">div</span> <span style="color:#369">class</span>=<span style="color:#d20;background-color:#fff0f0">"below-the-fold-content"</span>></<span style="color:#b06;font-weight:bold">div</span>>
</<span style="color:#b06;font-weight:bold">body</span>>
</code></pre></div><h3 id="deferred-loading-of-ads">Deferred loading of ads</h3>
<p>If you’re monetizing your website through Google AdSense or another ad agency that uses scripts to load ads, consider loading ads after the content is fully rendered. This may have a small impact on your revenue, but will improve the user’s experience while optimizing the load speed.</p>
<p>Although there are several ways to achieve this, a technique I have successfully used on many websites is removing all of the script references to Google AdSense until your page is fully loaded. A short delay can be added in order to allow some browsing time before showing ads.</p>
<p>Remove script references, the comment, and extra spaces from your original ad code, to convert it from something like this…</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-html" data-lang="html"><<span style="color:#b06;font-weight:bold">script</span> <span style="color:#369">async</span> <span style="color:#369">src</span>=<span style="color:#d20;background-color:#fff0f0">"https://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js"</span>></<span style="color:#b06;font-weight:bold">script</span>>
<span style="color:#888"><!-- Your ad name --></span>
<<span style="color:#b06;font-weight:bold">ins</span> <span style="color:#369">class</span>=<span style="color:#d20;background-color:#fff0f0">"adsbygoogle"</span>
<span style="color:#369">style</span>=<span style="color:#d20;background-color:#fff0f0">"display:inline-block;width:728px;height:90px"</span>
<span style="color:#369">data-ad-client</span>=<span style="color:#d20;background-color:#fff0f0">"ca-pub-XXXXXXXXXXXXXXXXX"</span>
<span style="color:#369">data-ad-slot</span>=<span style="color:#d20;background-color:#fff0f0">"XXXXXXXXX"</span>></<span style="color:#b06;font-weight:bold">ins</span>>
<<span style="color:#b06;font-weight:bold">script</span>>
(adsbygoogle = <span style="color:#038">window</span>.adsbygoogle || []).push({});
</<span style="color:#b06;font-weight:bold">script</span>>
</code></pre></div><p>… to something like this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-html" data-lang="html"><<span style="color:#b06;font-weight:bold">ins</span> <span style="color:#369">class</span>=<span style="color:#d20;background-color:#fff0f0">"adsbygoogle"</span> <span style="color:#369">style</span>=<span style="color:#d20;background-color:#fff0f0">"display:inline-block;width:728px;height:90px"</span> <span style="color:#369">data-ad-client</span>=<span style="color:#d20;background-color:#fff0f0">"ca-pub-XXXXXXXXXXXXXXXXX"</span> <span style="color:#369">data-ad-slot</span>=<span style="color:#d20;background-color:#fff0f0">"XXXXXXXXX"</span>></<span style="color:#b06;font-weight:bold">ins</span>>
</code></pre></div><p>A lot shorter, isn’t it? This will create an empty slot in which the ad will be displayed after the page is fully rendered. To accomplish that, a new script like the one below must be added (assuming jQuery is present on the website):</p>
<p><strong>async-ads.js:</strong></p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-javascript" data-lang="javascript"><span style="color:#888">// Create a script reference
</span><span style="color:#888"></span><span style="color:#080;font-weight:bold">function</span> addScript(src, <span style="color:#080;font-weight:bold">async</span>, callback) {
<span style="color:#080;font-weight:bold">var</span> js = <span style="color:#038">document</span>.createElement(<span style="color:#d20;background-color:#fff0f0">"script"</span>);
js.type = <span style="color:#d20;background-color:#fff0f0">"text/javascript"</span>;
<span style="color:#080;font-weight:bold">if</span> (<span style="color:#080;font-weight:bold">async</span>)
js.<span style="color:#080;font-weight:bold">async</span> = <span style="color:#080;font-weight:bold">true</span>;
<span style="color:#080;font-weight:bold">if</span> (callback)
js.onload = callback;
js.src = src;
<span style="color:#038">document</span>.body.appendChild(js);
}
<span style="color:#888">// Called when document is ready
</span><span style="color:#888"></span>$(<span style="color:#038">document</span>).ready(<span style="color:#080;font-weight:bold">function</span>() {
<span style="color:#888">// Wait for one second to ensure the user started browsing
</span><span style="color:#888"></span> setTimeout(<span style="color:#080;font-weight:bold">function</span>() {
(adsbygoogle = <span style="color:#038">window</span>.adsbygoogle || []);
$(<span style="color:#d20;background-color:#fff0f0">"ins.adsbygoogle"</span>).each(<span style="color:#080;font-weight:bold">function</span>() {
$(<span style="color:#d20;background-color:#fff0f0">"<script>(adsbygoogle = window.adsbygoogle || []).push({})</script>"</span>).insertAfter($(<span style="color:#080;font-weight:bold">this</span>));
});
addScript(<span style="color:#d20;background-color:#fff0f0">"https://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js"</span>, <span style="color:#080;font-weight:bold">true</span>);
}, <span style="color:#00d;font-weight:bold">1000</span>);
});
</code></pre></div><p>This code will wait for one second once the document is ready, and then leave instructions for Google to push a new ad for each slot. Finally, the AdSense external script will be loaded so that Google will read the instructions and start filling all the slots with ads.</p>
<p><b>Tip</b>: Enabling balancing from your AdSense dashboard may improve the average load speed as well as the user’s experience since ads will not be shown when the expected revenue is deprecable. And if you’re still on the fence about showing fewer ads, <a href="https://fatstacksblog.com/adsense-ad-balance-experiment/">try out an experiment</a> like I did. A balance of 50% worked well in my case, but the right balance will depend on your niche and website characteristics.</p>
<h3 id="lazy-load-for-images">Lazy load for images</h3>
<p>Because the user will most likely spend the majority of the visit reading above-the-fold content (and may even leave before scrolling at all), loading all images from content below-the-fold at first is impractical. Implementing a custom lazy-loading script (also referred to as deferred-loading or loading-on-scroll) for images can be an easy process. Even though changes to the backend would be likely, the concept of this approach is simple:</p>
<ul>
<li>
<p>Replacing the <code>src</code> attributes from all images that will have lazy loading with a custom attribute such as <code>data-src</code> (this part will probably require backend changes) and set a custom class for them, like <code>lazy</code>.</p>
</li>
<li>
<p>Creating a script that will copy the <code>data-src</code> content into the <code>src</code> attribute as we scroll through the page.</p>
</li>
</ul>
<p><strong>lazy-load.js:</strong></p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-javascript" data-lang="javascript">;(<span style="color:#080;font-weight:bold">function</span>($) {
$.fn.lazy = <span style="color:#080;font-weight:bold">function</span>(threshold, callback) {
<span style="color:#080;font-weight:bold">var</span> $w = $(<span style="color:#038">window</span>),
th = threshold || <span style="color:#00d;font-weight:bold">0</span>,
attrib = <span style="color:#d20;background-color:#fff0f0">"data-src"</span>,
images = <span style="color:#080;font-weight:bold">this</span>,
loaded;
<span style="color:#080;font-weight:bold">this</span>.one(<span style="color:#d20;background-color:#fff0f0">"lazy"</span>, <span style="color:#080;font-weight:bold">function</span>() {
<span style="color:#080;font-weight:bold">var</span> source = <span style="color:#080;font-weight:bold">this</span>.getAttribute(attrib);
source = source || <span style="color:#080;font-weight:bold">this</span>.getAttribute(<span style="color:#d20;background-color:#fff0f0">"data-src"</span>);
<span style="color:#080;font-weight:bold">if</span> (source) {
<span style="color:#080;font-weight:bold">this</span>.setAttribute(<span style="color:#d20;background-color:#fff0f0">"src"</span>, source);
<span style="color:#080;font-weight:bold">if</span> (<span style="color:#080;font-weight:bold">typeof</span> callback === <span style="color:#d20;background-color:#fff0f0">"function"</span>) callback.call(<span style="color:#080;font-weight:bold">this</span>);
}
});
<span style="color:#080;font-weight:bold">function</span> lazy() {
<span style="color:#080;font-weight:bold">var</span> inview = images.filter(<span style="color:#080;font-weight:bold">function</span>() {
<span style="color:#080;font-weight:bold">var</span> $e = $(<span style="color:#080;font-weight:bold">this</span>);
<span style="color:#080;font-weight:bold">if</span> ($e.is(<span style="color:#d20;background-color:#fff0f0">":hidden"</span>)) <span style="color:#080;font-weight:bold">return</span>;
<span style="color:#080;font-weight:bold">var</span> wt = $w.scrollTop(),
wb = wt + $w.height(),
et = $e.offset().top,
eb = et + $e.height();
<span style="color:#080;font-weight:bold">return</span> eb >= wt - th && et <= wb + th;
});
loaded = inview.trigger(<span style="color:#d20;background-color:#fff0f0">"lazy"</span>);
images = images.not(loaded);
}
$w.scroll(lazy);
$w.resize(lazy);
lazy();
<span style="color:#080;font-weight:bold">return</span> <span style="color:#080;font-weight:bold">this</span>;
};
})(<span style="color:#038">window</span>.jQuery);
$(<span style="color:#038">document</span>).ready(<span style="color:#080;font-weight:bold">function</span>() {
$(<span style="color:#d20;background-color:#fff0f0">'.lazy'</span>).each(<span style="color:#080;font-weight:bold">function</span> () {
$(<span style="color:#080;font-weight:bold">this</span>).lazy(<span style="color:#00d;font-weight:bold">0</span>, <span style="color:#080;font-weight:bold">function</span>() {
$(<span style="color:#080;font-weight:bold">this</span>).load(<span style="color:#080;font-weight:bold">function</span>() {
<span style="color:#080;font-weight:bold">this</span>.style.opacity = <span style="color:#00d;font-weight:bold">1</span>;
});
});
});
<span style="color:#888">// Set the correct attribute when printing
</span><span style="color:#888"></span><span style="color:#080;font-weight:bold">var</span> beforePrint = <span style="color:#080;font-weight:bold">function</span>() {
$(<span style="color:#d20;background-color:#fff0f0">"img.lazy"</span>).each(<span style="color:#080;font-weight:bold">function</span>() {
$(<span style="color:#080;font-weight:bold">this</span>).trigger(<span style="color:#d20;background-color:#fff0f0">"lazy"</span>);
<span style="color:#080;font-weight:bold">this</span>.style.opacity = <span style="color:#00d;font-weight:bold">1</span>;
});
};
<span style="color:#080;font-weight:bold">if</span> (<span style="color:#038">window</span>.matchMedia) {
<span style="color:#080;font-weight:bold">var</span> mediaQueryList = <span style="color:#038">window</span>.matchMedia(<span style="color:#d20;background-color:#fff0f0">'print'</span>);
mediaQueryList.addListener(<span style="color:#080;font-weight:bold">function</span>(mql) {
<span style="color:#080;font-weight:bold">if</span> (mql.matches)
beforePrint();
});
}
<span style="color:#038">window</span>.onbeforeprint = beforePrint;
</code></pre></div><p>This script will search for all <code><img></code> tags with class <code>lazy</code>, and change the <code>data-src</code> attribute to the <code>src</code> attribute once the image becomes visible due to scrolling. It also includes some additional logic to set the <code>src</code> attribute before printing the page.</p>
<h3 id="server-side-caching">Server-side caching</h3>
<p>Instead of performing all the backend rendering calculations every time, server-side caching allows you to output the same content to the clients over a period of time from a temporary copy of the response. This not only results in a decreased response time but also saves some resources on the server.</p>
<p>There are several ways to enable server-side caching, depending on factors such as the backend language and hosting platform (e.g. Windows/IIS vs. Linux/Apache), among other things. For this example, we will use ASP.NET (C#) since I’m mostly a Windows user.</p>
<p>The best and most efficient way to do this is by adding a declaration in the top of our ASP.NET page:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-csharp" data-lang="csharp"><%<span style="color:#a61717;background-color:#e3d2d2">@</span> OutputCache Duration=<span style="color:#d20;background-color:#fff0f0">"10"</span> VaryByParam=<span style="color:#d20;background-color:#fff0f0">"id;date"</span> %>
</code></pre></div><p>This declaration is telling the compiler that we want to cache the output from the server for 10 minutes, and we will save different versions based on the <code>id</code> and <code>date</code> URL parameters. So pages like:</p>
<ul>
<li>https://www.your-url.com/cached-page/?id=1&date=2020-01-01</li>
<li>https://www.your-url.com/cached-page/?id=2&date=2020-01-01</li>
<li>https://www.your-url.com/cached-page/?id=2&date=2020-02-01</li>
</ul>
<p>will be saved and then served from different cache copies. If we only set the <code>id</code> parameter as a source for caching, pages with different dates will be served from the same cache source (this can be useful as the <code>date</code> parameter is only evaluated on frontend scripts and ignored in the backend).</p>
<p>There are other configurations in ASP.NET to set our output cache policy. The output can be set to be based on the browser, the request headers, or even custom strings. <a href="https://www.c-sharpcorner.com/UploadFile/chinnasrihari/Asp-Net-mvc-framework-server-side-html-caching-techniques/">This page</a> has more useful information on this subject.</p>
<h3 id="gzip-compression">GZip compression</h3>
<p>GZip compression—when the client supports it—allows compressing the response before sending it over the network. In this way, more than 70% of the bandwidth can be saved when loading the website. Enabling GZip compression for dynamic and static content on a Windows Server with IIS is simple: Just go to the “Compression” section on the IIS Manager and check the options “Enable dynamic/static content compression”.</p>
<p><img src="/blog/2020/01/decreasing-website-load-time/enabling-compression-iis.jpg" alt="Enabling compression in IIS"></p>
<p>However, if you are running an ASP.NET MVC/WebForms website, this won’t be enough. For all backend responses to be compressed before sending them to the client, some custom code will also need to be added to the <code>global.asax</code> file in the website root:</p>
<p><strong>global.asax:</strong></p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-csharp" data-lang="csharp"><%<span style="color:#a61717;background-color:#e3d2d2">@</span> Application Language=<span style="color:#d20;background-color:#fff0f0">"C#"</span> %>
<script runat=<span style="color:#d20;background-color:#fff0f0">"server"</span>>
<span style="color:#080;font-weight:bold">void</span> Application_PreRequestHandlerExecute(<span style="color:#888;font-weight:bold">object</span> sender, EventArgs e)
{
HttpApplication app = sender <span style="color:#080;font-weight:bold">as</span> HttpApplication;
<span style="color:#888;font-weight:bold">string</span> acceptEncoding = app.Request.Headers[<span style="color:#d20;background-color:#fff0f0">"Accept-Encoding"</span>];
System.IO.Stream prevUncompressedStream = app.Response.Filter;
<span style="color:#080;font-weight:bold">if</span> (app.Context.CurrentHandler == <span style="color:#080;font-weight:bold">null</span>)
<span style="color:#080;font-weight:bold">return</span>;
<span style="color:#080;font-weight:bold">if</span> (!(app.Context.CurrentHandler <span style="color:#080;font-weight:bold">is</span> System.Web.UI.Page ||
app.Context.CurrentHandler.GetType().Name == <span style="color:#d20;background-color:#fff0f0">"SyncSessionlessHandler"</span>) ||
app.Request[<span style="color:#d20;background-color:#fff0f0">"HTTP_X_MICROSOFTAJAX"</span>] != <span style="color:#080;font-weight:bold">null</span>)
<span style="color:#080;font-weight:bold">return</span>;
<span style="color:#080;font-weight:bold">if</span> (acceptEncoding == <span style="color:#080;font-weight:bold">null</span> || acceptEncoding.Length == <span style="color:#00d;font-weight:bold">0</span>)
<span style="color:#080;font-weight:bold">return</span>;
<span style="color:#080;font-weight:bold">if</span> (Request.ServerVariables[<span style="color:#d20;background-color:#fff0f0">"SCRIPT_NAME"</span>].ToLower().Contains(<span style="color:#d20;background-color:#fff0f0">".axd"</span>)) <span style="color:#080;font-weight:bold">return</span>;
<span style="color:#080;font-weight:bold">if</span> (Request.ServerVariables[<span style="color:#d20;background-color:#fff0f0">"SCRIPT_NAME"</span>].ToLower().Contains(<span style="color:#d20;background-color:#fff0f0">".js"</span>)) <span style="color:#080;font-weight:bold">return</span>;
<span style="color:#080;font-weight:bold">if</span> (Request.QueryString.ToString().Contains(<span style="color:#d20;background-color:#fff0f0">"_TSM_HiddenField_"</span>)) <span style="color:#080;font-weight:bold">return</span>;
acceptEncoding = acceptEncoding.ToLower();
<span style="color:#080;font-weight:bold">if</span> (acceptEncoding.Contains(<span style="color:#d20;background-color:#fff0f0">"deflate"</span>) || acceptEncoding == <span style="color:#d20;background-color:#fff0f0">"*"</span>)
{
app.Response.Filter = <span style="color:#080;font-weight:bold">new</span> System.IO.Compression.DeflateStream(prevUncompressedStream,
System.IO.Compression.CompressionMode.Compress);
app.Response.AppendHeader(<span style="color:#d20;background-color:#fff0f0">"Content-Encoding"</span>, <span style="color:#d20;background-color:#fff0f0">"deflate"</span>);
}
<span style="color:#080;font-weight:bold">else</span> <span style="color:#080;font-weight:bold">if</span> (acceptEncoding.Contains(<span style="color:#d20;background-color:#fff0f0">"gzip"</span>))
{
app.Response.Filter = <span style="color:#080;font-weight:bold">new</span> System.IO.Compression.GZipStream(prevUncompressedStream,
System.IO.Compression.CompressionMode.Compress);
app.Response.AppendHeader(<span style="color:#d20;background-color:#fff0f0">"Content-Encoding"</span>, <span style="color:#d20;background-color:#fff0f0">"gzip"</span>);
}
}
</script>
</code></pre></div><p>To make sure our code is working properly, an external tool like <a href="https://www.giftofspeed.com/gzip-test/">this</a> will inform you if GZip is enabled or not.</p>
<p><img src="/blog/2020/01/decreasing-website-load-time/gzip-compression-enabled.jpg" alt="It works!"></p>
<h3 id="summary">Summary</h3>
<p>While there are many ways of decreasing the load time of a website, most are common and expensive. However, with a few minor tweaks, we can offer a better user experience in addition to improve our position in the search engine results. Every bit of optimization counts towards the goal with SEO. Load time is a very important factor (to both the developer and the user), especially on mobile platforms where users expect to get what they want instantly.</p>
<p>The image below is a Google Analytics report from one of my websites where, over several months, I implemented most of these formulas. A month ago, I made the latest change of deferring ad loading, which had an observable impact on the average loading speed of the page:</p>
<p><img src="/blog/2020/01/decreasing-website-load-time/analytics-average-page-load.jpg" alt="Report from Google Analytics"></p>
<p>Do you have any other page load optimization techniques? <b>Leave a comment below!</b></p>
Roundup of some useful websiteshttps://www.endpointdev.com/blog/2018/12/roundup-of-some-useful-websites/2018-12-21T00:00:00+00:00Jon Jensen
<p><a href="/blog/2018/12/roundup-of-some-useful-websites/squoosh-demo-20181220b.png"><img src="/blog/2018/12/roundup-of-some-useful-websites/squoosh-demo-20181220a.jpg" /></a></p>
<p>The world is a big place, and the Internet has gotten pretty big too. There are always new projects being created, and I want to share some useful and interesting ones from my growing list:</p>
<h3 id="squoosh-image-compressor">Squoosh image compressor</h3>
<p>Squoosh, hosted at <a href="https://squoosh.app/">squoosh.app</a>, is an open source in-browser tool for experimenting with image compression, made by the Chrome development team.</p>
<p>With Squoosh you can load an image in your browser, convert it to different image file formats (JPEG, WebP, PNG, BMP) using various compression algorithms and settings, and compare the result side-by-side with either the original image or the image compressed using other options.</p>
<p>The screenshot above demonstrates Squoosh running in Firefox 64 on Linux. Click on it to see a larger, lossless PNG screenshot. The photo was taken by my son Phin in northern Virginia, and is a typical imperfect mobile phone photo. On the left is the original, and on the right I am showing how bad gradients in the sky can look when compressed too much—maybe a quality level of 12 (out of 100) was too low. It does make for a very compact file size, though. 😄</p>
<p>Squoosh’s interface has a convenient slider bar so you can compare any part of the two versions of the image side by side. You can zoom and pan the image as well.</p>
<p>It is neat to see JavaScript tools (in this case TypeScript specifically) doing work in the browser that has traditionally been done by native apps.</p>
<h3 id="nerd-fonts">Nerd Fonts</h3>
<p>If you want access to an amazing number of symbols in a font, check out <a href="https://nerdfonts.com/">nerdfonts.com</a>. There you can mix and match symbols from many popular developer-oriented fonts such as Font Awesome, Powerline Symbols, Material Design, etc.</p>
<p>I probably should have chosen some fun symbols to demonstrate it here, but I could tell that was a rabbit hole I would not soon emerge from!</p>
<h3 id="glotio-code-pastebin">glot.io code pastebin</h3>
<p>There are many public pastebins these days, but <a href="https://glot.io/">glot.io</a> distinguishes itself by allowing you to run real code on their server in nearly 50 languages.</p>
<p>It offers both public and private pastes, has an API, and is open source.</p>
<h3 id="firefox-send">Firefox Send</h3>
<p>Firefox Send at <a href="https://send.firefox.com/">send.firefox.com</a> is a browser-based service for securely sharing files temporarily, for only one download during a maximum of 24 hours.</p>
<p>Handy for keeping unwanted bloat out of email, chat, or shared file storage for ephemeral files.</p>
<h3 id="transfersh-command-line-file-sharing">transfer.sh command-line file sharing</h3>
<p>Similarly, <a href="https://transfer.sh/">transfer.sh</a> is a terminal-based file upload and download tool.</p>
<p>As a command-line tool it easily integrates with other standard tools, so you can pipe output from other programs directly to it. If you have sensitive data to share you don’t need to trust the service—you can pipe your data through gpg or some other encryption tool before it leaves your computer.</p>
<p>transfer.sh is open source and can be self-hosted too.</p>
<p>It even has a Tor onion service so uploads and/or downloads can be as private as possible in hostile environments.</p>
<h3 id="doing-what-you-dont-want-to-do">Doing what you don’t want to do</h3>
<p>And finally, some timeless tips for making our human “software” work.</p>
<p>Often just one or two annoying little things can block us from making progress on larger projects that overall we really enjoy. How can you motivate yourself to push ahead when you have work that needs to be done, but you don’t want to do it?</p>
<p>Read the brief but helpful article <a href="https://zenhabits.net/unwanted/">10 Ways to Do What You Don’t Want to Do</a> by Leo Babauta to get some good ideas. A few of the points mentioned especially resonate with me:</p>
<ul>
<li>Why do I need to do it?</li>
<li>What is stopping me?</li>
<li>Embrace that it won’t be fun and do it anyway.</li>
<li>Set constraints.</li>
</ul>
<p>Then do at least a little bit of the work to get started. As our co-worker <a href="/team/mike-heins/">Mike Heins</a> has said to me on a few occasions over the years, you’ll never finish until you start.</p>
Postgres WAL files: best compression methodshttps://www.endpointdev.com/blog/2017/03/postgres-wal-files-best-compression/2017-03-28T00:00:00+00:00Greg Sabino Mullane
<div class="separator" style="clear: both; text-align: center; float:right"><a href="/blog/2017/03/postgres-wal-files-best-compression/image-0.jpeg" imageanchor="1" style="clear:float; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="/blog/2017/03/postgres-wal-files-best-compression/image-0.jpeg"/></a><br/><small><a href="https://flic.kr/p/9sa2bM">Turtle turtle</a> by WO1 Larry Olson from <a href="https://www.flickr.com/people/familymwr/?rb=1">US Army</a></small>
</div>
<p>The <a href="https://postgresql.org">PostgreSQL database system</a> uses the write-ahead logging method to ensure that
a log of changes is saved before being applied to the actual data. The log
files that are created are known as <a href="https://www.postgresql.org/docs/current/static/wal-intro.html">WAL (Write Ahead Log) files</a>, and by default
are 16 MB in size each. Although this is a small size, a busy system can generate hundreds
or thousands of these files per hour, at which point disk space becomes an issue.
Luckily, WAL files are extremely compressible. I examined different programs to
find one that offered the best compression (as indicated by a smaller size)
at the smallest cost (as indicated by wall clock time). All of the methods tested worked
better than the venerable gzip program, which is suggested in the Postgres
documentation for the <a href="https://www.postgresql.org/docs/9.6/static/continuous-archiving.html">archive_command option</a>. The best overall solution was using the <a href="http://manpages.ubuntu.com/manpages/trusty/man1/pxz.1.html">pxz program</a> inside the <a href="https://www.postgresql.org/docs/current/static/runtime-config-wal.html#GUC-ARCHIVE-COMMAND">archive_command setting</a>, followed closely by use of the <a href="http://manpages.ubuntu.com/manpages/trusty/man1/7z.1.html">7za program</a>. Use of the built-in <a href="https://www.postgresql.org/docs/current/static/runtime-config-wal.html#GUC-WAL-COMPRESSION">wal_compression</a> option was an excellent solution as well, although not as
space-saving as using external programs via archive_command.</p>
<hr>
<p>A database system is a complex beast, involving many trade-offs. An important issue is speed:
waiting for changes to get written to disk before letting the client proceed can be a
very expensive solution. Postgres gets around this with the use of the Write Ahead Log, which
generates WAL files indicating what changes were made. Creating these files is much faster than
performing the actual updates on the underlying files. Thus, Postgres is able to tell the client
that the work is “done” when the WAL file has been generated. Should the system crash before
the actual changes are made, the WAL files are used to replay the changes. As these
WAL files represent a continuous unbroken chain of all changes to the database, they can also
be used for Point in Time Recovery—in other words, the WAL files can be used to rewind the database
to any single point in time, capturing the state of the database at a specific moment.</p>
<p>Postgres WAL files are exactly 16 MB in size (although this size may be changed at compilation time,
it is extremely unheard of to do this). These files primarily sit around taking up disk space
and are only accessed when a problem occurs, so being able to compress them is a good
one-time exchange of CPU effort for a lower file size. In theory, the time to decompress
the files should also be considered, but testing revealed that all the programs
decompressed so quickly that it should not be a factor.</p>
<p>WAL files can be compressed in one of two ways. As of Postgres 9.5, the wal_compression
feature can be enabled, which instructs Postgres to compress parts of the WAL file
in-place when possible, leading to the ability to store much more information per 16 MB WAL file,
and thus reducing the total number generated. The second way is to compress with an external
program via the free-form archive_command parameter. Here is the canonical example
from the Postgres docs, showing use of the gzip program for archive_command:</p>
<pre tabindex="0"><code>archive_command = 'gzip < %p > /var/lib/pgsql/archive/%f'
</code></pre><p>It is widely known that gzip is no longer the best compression option for most tasks,
so I endeavored to determine which program was the best at WAL file compression—in terms
of final file size versus the overhead to create the file. I also wanted to examine how these
fared versus the new wal_compression feature.</p>
<hr>
<p>To compare the various compression methods, I examined all of the compression programs that
are commonly available on a Linux system, are known to be stable, and which perform at least
as good as the common utility gzip. The contenders were:</p>
<ul>
<li><strong>gzip</strong> — the canonical, default compression utility for many years</li>
<li><strong>pigz</strong> — parallel version of gzip</li>
<li><strong>bzip2</strong> — second only to gzip in popularity, it offers better compression</li>
<li><strong>lbzip2</strong> — parallel version of bzip</li>
<li><strong>xz</strong> — an excellent all-around compression alternative to gzip and bzip</li>
<li><strong>pxz</strong> — parallel version of xz</li>
<li><strong>7za</strong> — excellent compression, but suffers from complex arguments</li>
<li><strong>lrzip</strong> — compression program targeted at “large files”</li>
</ul>
<p>For the tests, 100 random WAL files were copied from a busy production Postgres system. Each of
those 100 files were compressed nine times by each of the programs above: from the “least compressed”
option (e.g. -1) to the “best compressed” option (e.g. -9).
The tests were performed on a 16-core system, with plenty of free RAM and nothing else running on the server.
Results were gathered by wrapping
each command with /usr/bin/time -verbose, which produces a nice breakdown of results.
To gather the data, the “Elapsed (wall clock) time” was used, along with size of
the compressed file. Here is some sample output of the time command:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-text" data-lang="text"> Command being timed: "bzip2 -4 ./0000000100008B91000000A5"
User time (seconds): 1.65
System time (seconds): 0.01
Percent of CPU this job got: 99%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:01.66
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 3612
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 938
Voluntary context switches: 1
Involuntary context switches: 13
Swaps: 0
File system inputs: 0
File system outputs: 6896
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
</code></pre></div><p>The wal_compression feature was tested by creating a new Postgres 9.6 cluster, then
running the <a href="https://www.postgresql.org/docs/current/static/pgbench.html">pgbench program</a> twice to generate WAL files—once with wal_compression enabled,
and once with it disabled. Then each of the resulting WAL files was compressed using each of the programs above.</p>
<hr>
<style><!--
table.gsmt { font-family: Monospace; padding: 0 0 3em 0; margin-left: auto; margin-right: auto; }
table.gsmt table td { padding: 0 0.5em 0 0.2em ; color: #222200; white-space: nowrap; font-size: smaller;}
table.gsmt table th { padding: 0em 0.5em 0em 0.5em; color: black; font-size: smaller; }
--></style>
<table border="0" class="gsmt" style="padding: 0 0 3em 0"><caption><b>Table 1.</b><br/>
Results of compressing 16 MB WAL files—average for 100 files</caption>
<tbody><tr>
<td><table border="1">
<tbody><tr><th>Command</th><th>Wall clock time (s)</th><th>File size (MB)</th></tr>
<tr><td>gzip -1</td><td>0.271</td><td>4.927</td></tr>
<tr><td>gzip -2</td><td>0.292</td><td>4.777</td></tr>
<tr><td>gzip -3</td><td>0.366</td><td>4.667</td></tr>
<tr><td>gzip -4</td><td>0.381</td><td>4.486</td></tr>
<tr><td>gzip -5</td><td>0.492</td><td>4.318</td></tr>
<tr><td>gzip -6</td><td>0.734</td><td>4.250</td></tr>
<tr><td>gzip -7</td><td>0.991</td><td>4.235</td></tr>
<tr><td>gzip -8</td><td>2.042</td><td>4.228</td></tr>
<tr><td>gzip -9</td><td>3.626</td><td>4.227</td></tr>
</tbody></table></td>
<td><table border="1">
<tbody><tr><th>Command</th><th>Wall clock time (s)</th><th>File size (MB)</th></tr>
<tr><td>bzip2 -1</td><td>1.540</td><td>3.817</td></tr>
<tr><td>bzip2 -2</td><td>1.531</td><td>3.688</td></tr>
<tr><td>bzip2 -3</td><td>1.570</td><td>3.638</td></tr>
<tr><td>bzip2 -4</td><td>1.625</td><td>3.592</td></tr>
<tr><td>bzip2 -5</td><td>1.667</td><td>3.587</td></tr>
<tr><td>bzip2 -6</td><td>1.707</td><td>3.566</td></tr>
<tr><td>bzip2 -7</td><td>1.731</td><td>3.559</td></tr>
<tr><td>bzip2 -8</td><td>1.752</td><td>3.557</td></tr>
<tr><td>bzip2 -9</td><td>1.784</td><td>3.541</td></tr>
</tbody></table></td>
<td><table border="1">
<tbody><tr><th>Command</th><th>Wall clock time (s)</th><th>File size (MB)</th></tr>
<tr><td>xz -1</td><td>0.962</td><td>3.174</td></tr>
<tr><td>xz -2</td><td>1.186</td><td>3.066</td></tr>
<tr><td>xz -3</td><td>5.911</td><td>2.711</td></tr>
<tr><td>xz -4</td><td>6.292</td><td>2.682</td></tr>
<tr><td>xz -5</td><td>6.694</td><td>2.666</td></tr>
<tr><td>xz -6</td><td>8.988</td><td>2.608</td></tr>
<tr><td>xz -7</td><td>9.194</td><td>2.592</td></tr>
<tr><td>xz -8</td><td>9.117</td><td>2.596</td></tr>
<tr><td>xz -9</td><td>9.164</td><td>2.597</td></tr>
</tbody></table></td>
</tr>
</tbody></table>
<table border="0" class="gsmt"><caption><b>Table 2.</b><br/>
Results of compressing 16 MB WAL file—average for 100 files</caption>
<tbody><tr>
<td><table border="1">
<tbody><tr><th>Command</th><th>Wall clock time (s)</th><th>File size (MB)</th>
</tr><tr><td>lrzip<br/> -l -L1</td><td>0.296</td><td>5.712</td></tr>
<tr><td>lrzip<br/> -l -L2</td><td>0.319</td><td>5.686</td></tr>
<tr><td>lrzip<br/> -l -L3</td><td>0.341</td><td>5.645</td></tr>
<tr><td>lrzip<br/> -l -L4</td><td>0.370</td><td>5.639</td></tr>
<tr><td>lrzip<br/> -l -L5</td><td>0.389</td><td>5.627</td></tr>
<tr><td>lrzip<br/> -l -L6</td><td>0.901</td><td>5.501</td></tr>
<tr><td>lrzip<br/> -l -L7</td><td>2.090</td><td>5.462</td></tr>
<tr><td>lrzip<br/> -l -L8</td><td>2.829</td><td>5.471</td></tr>
<tr><td>lrzip<br/> -l -L9</td><td>5.983</td><td>5.463</td></tr>
</tbody></table></td>
<td><table border="1">
<tbody><tr><th>Command</th><th>Wall clock time (s)</th><th>File size (MB)</th></tr>
<tr><td>lrzip<br/> -z -L1</td><td>3.582</td><td>3.353</td></tr>
<tr><td>lrzip<br/> -z -L2</td><td>3.577</td><td>3.342</td></tr>
<tr><td>lrzip<br/> -z -L3</td><td>3.601</td><td>3.326</td></tr>
<tr><td>lrzip<br/> -z -L4</td><td>11.971</td><td>2.799</td></tr>
<tr><td>lrzip<br/> -z -L5</td><td>11.890</td><td>2.741</td></tr>
<tr><td>lrzip<br/> -z -L6</td><td>11.971</td><td>2.751</td></tr>
<tr><td>lrzip<br/> -z -L7</td><td>12.861</td><td>2.750</td></tr>
<tr><td>lrzip<br/> -z -L8</td><td>30.080</td><td>2.483</td></tr>
<tr><td>lrzip<br/> -z -L9</td><td>33.171</td><td>2.482</td></tr>
</tbody></table></td>
<td><table border="1">
<tbody><tr><th>Command</th><th>Wall clock time (s)</th><th>File size (MB)</th></tr>
<tr><td>7za -bd -mx=1<br/> a test.7za</td><td>0.128</td><td>3.182</td></tr>
<tr><td>7za -bd -mx=2<br/> a test.7za</td><td>0.139</td><td>3.116</td></tr>
<tr><td>7za -bd -mx=3<br/> a test.7za</td><td>0.301</td><td>3.059</td></tr>
<tr><td>7za -bd -mx=4<br/> a test.7za</td><td>1.251</td><td>3.001</td></tr>
<tr><td>7za -bd -mx=5<br/> a test.7za</td><td>3.821</td><td>2.620</td></tr>
<tr><td>7za -bd -mx=6<br/> a test.7za</td><td>3.841</td><td>2.629</td></tr>
<tr><td>7za -bd -mx=7<br/> a test.7za</td><td>4.631</td><td>2.591</td></tr>
<tr><td>7za -bd -mx=8<br/> a test.7za</td><td>4.671</td><td>2.590</td></tr>
<tr><td>7za -bd -mx=9<br/> a test.7za</td><td>4.663</td><td>2.599</td></tr>
</tbody></table></td></tr>
</tbody></table>
<table border="0" class="gsmt"><caption><b>Table 3.</b><br/>
Results of compressing 16 MB WAL file—average for 100 files</caption>
<tbody><tr>
<td><table border="1">
<tbody><tr><th>Command</th><th>Wall clock time (s)</th><th>File size (MB)</th></tr>
<tr><td>pigz -1</td><td>0.051</td><td>4.904</td></tr>
<tr><td>pigz -2</td><td>0.051</td><td>4.755</td></tr>
<tr><td>pigz -3</td><td>0.051</td><td>4.645</td></tr>
<tr><td>pigz -4</td><td>0.051</td><td>4.472</td></tr>
<tr><td>pigz -5</td><td>0.051</td><td>4.304</td></tr>
<tr><td>pigz -6</td><td>0.060</td><td>4.255</td></tr>
<tr><td>pigz -7</td><td>0.081</td><td>4.225</td></tr>
<tr><td>pigz -8</td><td>0.140</td><td>4.212</td></tr>
<tr><td>pigz -9</td><td>0.251</td><td>4.214</td></tr>
</tbody></table></td>
<td><table border="1">
<tbody><tr><th>Command</th><th>Wall clock time (s)</th><th>File size (MB)</th></tr>
<tr><td>lbzip2 -1</td><td>0.135</td><td>3.801</td></tr>
<tr><td>lbzip2 -2</td><td>0.151</td><td>3.664</td></tr>
<tr><td>lbzip2 -3</td><td>0.151</td><td>3.615</td></tr>
<tr><td>lbzip2 -4</td><td>0.151</td><td>3.586</td></tr>
<tr><td>lbzip2 -5</td><td>0.151</td><td>3.562</td></tr>
<tr><td>lbzip2 -6</td><td>0.151</td><td>3.545</td></tr>
<tr><td>lbzip2 -7</td><td>0.150</td><td>3.538</td></tr>
<tr><td>lbzip2 -8</td><td>0.151</td><td>3.524</td></tr>
<tr><td>lbzip2 -9</td><td>0.150</td><td>3.528</td></tr>
</tbody></table></td>
<td><table border="1">
<tbody><tr><th>Command</th><th>Wall clock time (s)</th><th>File size (MB)</th></tr>
<tr><td>pxz -1</td><td>0.135</td><td>3.266</td></tr>
<tr><td>pxz -2</td><td>0.175</td><td>3.095</td></tr>
<tr><td>pxz -3</td><td>1.244</td><td>2.746</td></tr>
<tr><td>pxz -4</td><td>2.528</td><td>2.704</td></tr>
<tr><td>pxz -5</td><td>5.115</td><td>2.679</td></tr>
<tr><td>pxz -6</td><td>9.116</td><td>2.604</td></tr>
<tr><td>pxz -7</td><td>9.255</td><td>2.599</td></tr>
<tr><td>pxz -8</td><td>9.267</td><td>2.595</td></tr>
<tr><td>pxz -9</td><td>9.355</td><td>2.591</td></tr>
</tbody></table></td></tr>
</tbody></table>
<table border="0" class="gsmt"><caption><b>Table 4.</b><br/>Results of Postgres wal_compression option</caption>
<tbody><tr>
<td><table border="1">
<tbody><tr><th>Modifications</th><th>Total size of WAL files (MB)</th></tr>
<tr><td>No modifications</td><td>208.1</td></tr>
<tr><td>wal_compression enabled</td><td>81.0</td></tr>
<tr><td>xz -2</td><td>8.6</td></tr>
<tr><td>wal_compression enabled PLUS xz -2</td><td>9.4</td></tr>
</tbody></table>
</td></tr></tbody></table>
<hr>
<p>Table 1 shows some baseline compression values for the three popular programs
gzip, bzip2, and xz. Both gzip and bzip2 show little change in the file sizes as the
compression strength is raised. However, xz has a relatively large jump when going
from -2 to -3, although the time cost increases to an unacceptable 5.9 seconds. As a starting
point, something well under one second is desired.</p>
<p>Among those three, xz is the clear winner, shrinking the file to 3.07 MB with a compression
argument of -2, and taking 1.2 seconds to do so. Both gzip and bzip2 never even reach this
file size, even when using a -9 argument. For that matter, the best compression gzip can
ever achieve is 4.23 MB, which the other programs can beat without breakng a sweat.</p>
<p>Table 2 demonstrates two ways of invoking the lrzip program: the -l option (lzo compression -
described in the lrzip documentation as “ultra fast”) and the -z option (zpaq compression -
“extreme compression, extreme slow”). All of those superlatives are supported by the data. The -l
option runs extremely fast: even at -L5 the total clock time is still only .39 seconds. Unfortunately,
the file size hovers around an undesirable 5.5 MB, no matter what compression level is used. The -z
option produces the smallest file of all the programs here (2.48 MB) at a whopping cost of over 30
seconds! Even the smallest compression level (-L1) takes 3.5 seconds to produce a 3.4 MB file. Thus,
lrzip is clearly out of the competition.</p>
<div class="separator" style="clear: both; text-align: center; padding-bottom:1em;"><a href="/blog/2017/03/postgres-wal-files-best-compression/image-1.jpeg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" id="jA0EBAMCQV4Upc7MVzZgycARvnUd7ZPBWXA9iXj+1nagDqP0bIKB3vspSuKJKLudmKh2tPgjwLFfxFkN34sgCWCeyYnxTt/zDAR3GpmBoiPQbMmxJO2QBdjcFr6e3R5/tMNH5MsIQklNNOM/EBtZid0PshHrOEooRj4xhSO74FtVZiXNR/hx0tr1QdPHs8XS5qWaKCG4PG69JN/k74CevdILYAAdhENPNZV48aOJwZq5D2A3+65ZYcNWfBXpHrnecboKT607iQUBC7zUnzqPCl31RVmQ0EmRX7zElgin3B3jwho==9+3U" src="/blog/2017/03/postgres-wal-files-best-compression/image-1.jpeg"/></a><br/>Compression in action <small>(<a href="https://flic.kr/p/9sgf6f">photo</a> by <a href="https://www.flickr.com/people/familymwr/?rb=1">Eric Armstrong</a>)</small></div>
<p>The most interesting program is without a doubt 7za. Unlike the others, it is organized around
creating archives, and thus doesn’t do in-place compression as the others do. Nevertheless,
the results are quite good. At the lowest level, it takes a mere 0.13 seconds to produce a
3.18 MB file. As it takes xz 1.19 seconds to produce a nearly equivalent 3.10 MB file, 7za
is the clear winner … if we had only a single core available. :)</p>
<p>It is rare to find a modern server with a single processor, and a crop of compression programs have
appeared to support this new paradigm. First up is the amazing pigz, which is a parallel
version of gzip. As Table 3 shows, pigz is extraordinarily fast on our 16 core box, taking
a mere 0.05 seconds to run at compression level -1, and only 0.25 seconds to run at
compression level -9. Sadly, it still suffers from the fact that gzip is simply not good
at compressing WAL files, as the smallest size was 4.23 MB. This rules out pigz from
consideration.</p>
<p>The bzip2 program has been nipping at the heels of gzip for many years, so naturally it
has its own parallel version, known as lbzip2. As Table 3 shows, it is also amazingly fast.
Not as fast as pigz, but with a speed of under 0.2 seconds—even at the highest compression level!
There is very little variation among the compression levels used, so it is fair to simply state
that lbzip2 takes 0.15 seconds to shrink the WAL file to 3.5 MB. A decent entry.</p>
<p>Of course, the xz program has a parallel version as well, known as pxz. Table 3 shows that
the times still vary quite a bit, and reach into the 9 second range at higher compression levels,
but does very well at -2, taking a mere 0.17 seconds to produce a 3.09 MB file. This is
comparable to the previous winner, 7za, which took 0.14 seconds to create a 3.12 MB
file.</p>
<p>So the clear winners are 7za and pxz. I gave the edge to pxz, as (1) the file size was
slightly smaller at comparable time costs, and (2) the odd syntax for 7za for both compressing
and decompressing was annoying compared with the simplicity of “xz -2” and “xz -d”.</p>
<p>Now, what about the built-in compression offered by the wal_compression option?
As Table 4 shows, the compression for the WAL files I tested went from 208 MB to
81 MB. This is a significant gain, but only equates to compressing a single WAL
file to 6.23 MB, which is a poor showing when compared to the compression programs above.
It should be pointed out that the wal_compression option is sensitive to
your workload, so you might see reports of greater and lesser compressions.</p>
<p>Of interest is that the WAL files generated by turning on wal_compression are
capable of being further compressed by the archive_command option, and doing
a pretty good job of it as well—going from 81 MB of WAL files to 9.4 MB of
WAL files. However, using just xz in the archive_command without wal_compression
on still yielded a smaller overall size, and means less CPU because the data is only
compressed once.</p>
<p>It should be pointed out that wal_compression has other advantages, and that
comparing it to archive_command is not a fair comparison, but this article was
primarily about the best compression option for storing WAL files long-term.</p>
<p>Thus, the overall winner is “pxz -2”, followed closely by 7za and its bulky
arguments, with honorable mention given to wal_compression. Your particular
requirements might guide you to another conclusion, but hopefully nobody shall
simply default to using gzip anymore.</p>
<hr>
<p>Thanks to my colleague Lele for encouraging me to try pxz, after I was already happy with xz. Thanks to the authors of xz, for providing an amazing program that has an incredible number of knobs for tweaking. And a final thanks to the authors of the wal_compression feature, which is a pretty nifty trick!</p>
HTTP/2 is on the way!https://www.endpointdev.com/blog/2015/03/http2-is-on-way/2015-03-13T00:00:00+00:00Jon Jensen
<h3 id="https-and-spdy">HTTPS and SPDY</h3>
<p>Back in August 2014, we made our websites <a href="/">www.endpoint.com</a> and <a href="https://www.visionport.com/">liquidgalaxy.endpoint.com</a> HTTPS-only, which allowed us to turn on <a href="https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security">HTTP Strict Transport Security</a> and earn a grade of <a href="https://www.ssllabs.com/ssltest/analyze.html?d=endpointdev.com&latest">A+ from Qualys’ SSL Labs</a> server test.</p>
<p>Given the widely-publicized surveillance of Internet traffic and injection of advertisements and tracking beacons into plain HTTP traffic by some unscrupulous Internet providers, we felt it would be good to start using TLS encryption on even our non-confidential public websites.</p>
<p>This removed any problems switching between HTTP for most pages and HTTPS for the contact form and the POST of submitted data. Site delivery over HTTPS also <a href="https://webmasters.googleblog.com/2014/08/https-as-ranking-signal.html">serves as a ranking signal</a> for Google, though presumably still a minor one.</p>
<p>Doesn’t SSL/TLS slow down a website? Simply put, not really these days. See <a href="https://istlsfastyet.com/">Is TLS Fast Yet?</a> for lots of details. And:</p>
<p>Moving to HTTPS everywhere on our sites also allowed us to take advantage of nginx’s relatively new <a href="https://en.wikipedia.org/wiki/SPDY">SPDY</a> (pronounced “speedy”) capability. SPDY is an enhancement to HTTPS created by Google to increase web page delivery time by compressing headers and multiplexing many requests in a single TCP connection. It is only available on HTTPS, so it also incentivizes sites to stop using unencrypted HTTP in order to get more speed, with security as a bonus. Whereas people once avoided HTTPS because SSL/TLS was slower, SPDY turned that idea around. We began offering SPDY for our sites in October 2014.</p>
<p>On the browser side, SPDY was initially only supported by Chrome and Firefox. Later support was added to Opera, Safari 8, and partially in IE 11. So most browsers can use it now.</p>
<p>There is only partial server support: In the open source world, nginx fully supports SPDY now, but Apache’s mod_spdy is incomplete and development on it has stalled.</p>
<p>Is SPDY here to stay? After all it was an experimental Google protocol. Instead of getting on track to become an Internet standard protocol as is, it was used as the starting point for the next version of HTTP, HTTP/2. That sounded like good news, except that the current version HTTP/1.1 was standardized in 1999 and hadn’t really changed since then. Many of us wondered if HTTP/2 would get mired in the standardization process and take years to see the light of day.</p>
<p>However, the skeptics were wrong! HTTP/2 was completed over about 3 years, and its official RFC form is now being finalized. Having it be the next version of HTTP will go a long way toward getting more implementation and adoption, since it is no longer a single company’s project. On the other hand, basing HTTP/2 on SPDY meant that there was a widely-used proof of concept out there already, so discussions didn’t get lost in the purely theoretical. The creators of SPDY at Google were heavily involved in the HTTP/2 standardization process, so their lessons were not lost, and it appears that HTTP/2 will be even better.</p>
<h3 id="what-is-different-in-http2">What is different in HTTP/2?</h3>
<ul>
<li>Request and response multiplexing in a single TCP connection (no need for 6+ connections to the same host!)</li>
<li>Stream prioritization (prioritizing files that the client most needs first)</li>
<li>Server push (of files the server expects the client to need, before the client knows it), and client stream cancellation (in case the server or the client is wrong and wants to abort a stream)</li>
<li>Binary framing (no more hand-typing requests via telnet, sadly)</li>
<li>Header compression (greatly reducing the bloat of large cookies)</li>
<li>Backward-compatibility with HTTP/1.1 and autodiscovery of HTTP/2 support (transparent upgrading for users)</li>
<li>When TLS is used, require TLS 1.2 and minimum acceptable cipher strength (to help retire weak TLS setups)</li>
</ul>
<p>For front-end web developers, these back-end plumbing changes have some very nice consequences. As described in <a href="https://mattwilcox.net/web-development/http2-for-front-end-web-developers">HTTP2 for front-end web developers</a>, you will soon be able to stop using many of the annoying workarounds for HTTP/1.1’s weaknesses: no more sprites, combining CSS & JavaScript files, inlining images in CSS, sharding across many subdomains, etc.</p>
<p>This practically means that the web can largely go back to working the way it was designed, with different files for different things, independent caching of small files, serve assets from the same place.</p>
<h3 id="what-is-not-changing">What is <em>not</em> changing?</h3>
<p>Most of HTTP/1.1 basic semantics remain the same, with most of the changes being to the “wrapping” or transport of the data. All this stays the same:</p>
<ul>
<li>built on TCP</li>
<li>stateless</li>
<li>same request methods</li>
<li>same request headers (including cookies)</li>
<li>same response headers and body</li>
<li>may be unencrypted or layered on TLS (although so far, Chrome and Firefox have stated that they will only support HTTP/2 over TLS, and IE so far only supports HTTP/2 over TLS as well)</li>
<li>no changes in HTML, CSS, client-side scripting, same-origin security policy, etc.</li>
</ul>
<h3 id="the-real-point-speed">The real point: speed</h3>
<p>Speed and efficiency are the main advantages of HTTP/2. It will use less data transfer for both requests and responses. It will use fewer TCP connections, lightening the load on clients, servers, firewalls, and routers.</p>
<p>As clients adapt more to HTTP/2, it will probably provide a faster perceived experience as servers push the most important CSS, images, and JavaScript proactively to the client before it has even parsed the HTML.</p>
<p>See these <a href="https://blog.httpwatch.com/2015/01/16/a-simple-performance-comparison-of-https-spdy-and-http2/">simple benchmarks between HTTP/1.1, SPDY, and HTTP/2</a>.</p>
<h3 id="when-can-we-use-it">When can we use it?</h3>
<p>Refreshingly, Google has announced that they are happy to kill their own creation SPDY: <a href="https://blog.chromium.org/2016/02/transitioning-from-spdy-to-http2.html">they will drop support for SPDY from Chrome in early 2016</a> in favor of HTTP/2.</p>
<p><a href="http://bitsup.blogspot.com/2015/02/http2-is-live-in-firefox.html">Firefox uses HTTP/2 by default</a> where possible, and Chrome has an option to enable HTTP/2. IE 11 for Windows 10 beta supports HTTP/2. You can see if your browser supports HTTP/2 now by using the <a href="https://http2.golang.org/">Go language HTTP/2 demo server</a>.</p>
<p>On the server side, Google and Twitter already have been opportunistically serving HTTP/2 for a while. <a href="https://nginx.com/blog/how-nginx-plans-to-support-http2/">nginx plans to add support this year</a>, and an experimental <a href="https://icing.github.io/mod_h2/">Apache module mod_h2</a> is available now. The H2O open-source C-based web server <a href="http://blog.kazuhooku.com/2015/02/h2o-new-http-server-goes-version-100-as.html">supports HTTP/2 now</a>, as does Microsoft IIS for Windows beta 10.</p>
<p>So we probably have at least a year until the most popular open source web servers easily support HTTP/2, but by then most browsers will probably support it and it should be an easy transition, as SPDY was. As long as you’re ready to go HTTPS-only for your site, anyway. :)</p>
<p>I think HTTP/2 will be a good thing!</p>
<h3 id="give-me-more-details">Give me more details!</h3>
<p>I highly recommend that system administrators and developers read the excellent <a href="https://daniel.haxx.se/http2/">http2 explained</a> PDF book by Daniel Stenberg, Firefox developer at Mozilla, and author of curl. It explains everything simply and well.</p>
<p>Other reference materials:</p>
<ul>
<li><a href="https://www.ietf.org/blog/2015/02/http2-approved/">HTTP/2 Approved</a> on the IETF blog, by Mark Nottingham, chair the IETF HTTP Working Group</li>
<li><a href="https://http2.github.io/">HTTP/2 home page</a> with specifications and FAQs</li>
<li><a href="http://chimera.labs.oreilly.com/books/1230000000545/ch12.html">High Performance Browser Networking chapter 12 on HTTP/2</a> by Ilya Grigorik</li>
<li><a href="https://en.wikipedia.org/wiki/HTTP/2">HTTP/2 on Wikipedia</a></li>
<li><a href="https://www.mnot.net/blog/2014/01/30/http2_expectations">Nine Things to Expect from HTTP/2</a> by Mark Nottingham</li>
<li><a href="http://queue.acm.org/detail.cfm?id=2555617">Making the Web Faster with HTTP 2.0: HTTP continues to evolve</a> by Ilya Grigorik of Google</li>
<li><a href="https://daniel.haxx.se/blog/2015/03/06/tls-in-http2/">TLS in HTTP/2</a>: Daniel Stenberg on TLS in HTTP/2 being mandatory in effect if not in the specification, and discusses opportunistic encryption</li>
<li><a href="http://robbysimpson.com/2015/01/26/http2-and-the-internet-of-things/">HTTP/2 and the Internet of Things</a> by Robby Simpson of GE Digital Energy</li>
<li><a href="http://queue.acm.org/detail.cfm?id=2716278">HTTP/2.0—The IETF is Phoning It In: Bad protocol, bad politics</a> by Poul-Henning Kamp, author of Varnish</li>
</ul>
Supporting Apple Retina displays on the Webhttps://www.endpointdev.com/blog/2014/05/supporting-apple-retina-displays-on-web/2014-05-27T00:00:00+00:00Phineas Jensen
<p>Apple’s <a href="https://en.wikipedia.org/wiki/Retina_Display">Retina displays</a> (on Mac desktop & laptop computers, and on iPhones and iPads) have around twice the pixel density of traditional displays. Most recent Android phones and tablets have higher-resolution screens as well.</p>
<p>I was recently given the task of adding support for these higher-resolution displays to our <a href="/">End Point company website</a>. Our imagery had been created prior to Retina displays being commonly used, but even now many web developers still overlook supporting high-resolution screens because it hasn’t been part of the website workflow before, because they aren’t simple to cope with, and since most people don’t notice any lack of sharpness without comparing low & high-resolution images side by side.</p>
<p>Most images which are not designed for Retina displays look blurry on them, like this:</p>
<p><a href="/blog/2014/05/supporting-apple-retina-displays-on-web/image-0-big.png" imageanchor="1" style="display:inline"><img border="0" height="266" src="/blog/2014/05/supporting-apple-retina-displays-on-web/image-0.png" width="266"/></a>
<a href="/blog/2014/05/supporting-apple-retina-displays-on-web/image-1-big.png" imageanchor="1" style="display:inline"><img border="0" height="266" src="/blog/2014/05/supporting-apple-retina-displays-on-web/image-1.png" width="266"/></a></p>
<p>The higher-resolution image is on the left, and the lower-resolution image is on the right.</p>
<p>Now, to solve this problem, you need to serve a larger, higher quality image to Retina displays. There are several different ways to do this. I’ll cover a few ways to do it, and explain how I implemented it for our site.</p>
<h3 id="retinajs">Retina.js</h3>
<p>As I was researching ways to implement support for Retina displays, I found that a popular suggestion is the JavaScript library <a href="http://imulus.github.io/retinajs/">Retina.js</a>. Retina.js automatically detects Retina screens, and then for each image on the page, it checks the web server for a Retina image version under the same name with @2x before the suffix. For example, when fetching the image background.jpg on a Retina-capable system, it would afterward look for <a href="mailto:background@2x.jpg">background@2x.jpg</a> and serve that if it’s available.</p>
<p>Retina.js makes it relatively painless to deal with serving Retina images to the correct people, but it has a couple of large problems. First, it fetches and replaces the Retina image <em>after</em> the default image, serving both the normal and Retina images to Retina users, greatly increasing download size and time.</p>
<p>Second, Retina.js does not use the correct image if the browser window is moved from a Retina display to a non-Retina display or vice versa when using multiple monitors. For example, if an image is loaded on a standard 1080p monitor and then the browser is moved to a Retina display, it will show the incorrect, low-res image.</p>
<h3 id="using-css-for-background-images">Using CSS for background images</h3>
<p>Doesn’t the “modern web” have a way to handle this natively in HTML & CSS? For sites using CSS background images, CSS media queries will do the trick:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-css" data-lang="css">@<span style="color:#080;font-weight:bold">media</span> <span style="color:#b06;font-weight:bold">only</span> <span style="color:#b06;font-weight:bold">screen</span> <span style="color:#b06;font-weight:bold">and</span> (<span style="color:#b06;font-weight:bold">-webkit-min-device-pixel-ratio</span>: <span style="color:#b06;font-weight:bold">2</span>), (<span style="color:#b06;font-weight:bold">min-resolution</span>: <span style="color:#b06;font-weight:bold">192dpi</span>) {
.<span style="color:#b06;font-weight:bold">icon</span> {
<span style="color:#080;font-weight:bold">background-image</span>: <span style="color:#038">url</span>(<span style="color:#2b2;background-color:#f0fff0">icon@2x.png</span>);
<span style="color:#080;font-weight:bold">background-size</span>: <span style="color:#00d;font-weight:bold">20</span><span style="color:#888;font-weight:bold">px</span> <span style="color:#00d;font-weight:bold">20</span><span style="color:#888;font-weight:bold">px</span>;
}
}
</code></pre></div><p>But this method only works with CSS background images, so for our site and a lot of other sites, it will only be useful for a small number of images.</p>
<p>Take a look at this <a href="https://css-tricks.com/snippets/css/retina-display-media-query/">CSS-Tricks</a> page for some excellent examples of Retina (and other higher-res display) support.</p>
<h3 id="server-side-checks-for-retina-images">Server-side checks for Retina images</h3>
<p>A very efficient way to handle all types of images is to have the browser JavaScript set a cookie that tells the web server whether to serve Retina or standard images. That will keep data transfer to a minimum, with a minimum of trickery required in the browser. You’ll still need to create an extra Retina-resolution image for every standard image on the server. And you’ll need to have a dynamic web process run for every image served. The <a href="https://web.archive.org/web/20141109113432/http://retina-images.complexcompulsions.com/">Retina Images</a> open source PHP program shows how to do this.</p>
<h3 id="why-we-didnt-use-these-methods">Why we didn’t use these methods</h3>
<p>There is one reason common to all of these methods which made us decide against them: All of them require you to maintain multiple versions of each image. This ends up taking a lot of time and effort. It also means your content distribution network (CDN) or other HTTP caches will have twice as many image files to load and cache, increasing cache misses and data transfer. It also uses more disk space, which isn’t a big problem for the small number of images on our website, but on an ecommerce website with many thousands of images, it adds up quickly.</p>
<p>We would feel compelled to have the separate images if it were necessary if the Retina images were much larger and slow down the browsing experience for non-Retina users for no purpose. But instead we decided on the following solution that we saw others describe.</p>
<h3 id="serving-retina-images-to-everybody-how-we-did-it">Serving Retina images to everybody (how we did it)</h3>
<p>We read that you can serve Retina images to everyone, but we immediately thought that wouldn’t work out well. We were sure that the Retina images would be several times larger than the normal images, wasting a ton of bandwidth for anyone not using a Retina screen. We were very pleasantly surprised to find out that this wasn’t the case at all.</p>
<p>After testing on a few images, I found I could get Retina images within 2-3 KB of the normal images while keeping the visual fidelity, by dropping the JPEG compression rate. How? Because the images were being displayed at a smaller size than they were, the compression artifacts weren’t nearly as noticeable.</p>
<p>These are the total file sizes for each image on our <a href="/team/">team page</a>:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">Retina Normal Filename
10K 9.3K adam_spangenthal.jpg
13K 13K adam_vollrath.jpg
12K 11K benjamin_goldstein.jpg
7.6K 4.2K bianca_rodrigues.jpg
14K 13K brian_buchalter.jpg
13K 15K brian_gadoury.jpg
7.5K 8.0K brian_zenone.jpg
9.8K 6.6K bryan_berry.jpg
12K 11K carl_bailey.jpg
6.9K 15K dave_jenkins.jpg
13K 13K david_christensen.jpg
7.7K 21K emanuele_calo.jpg
16K 16K erika_hamby.jpg
13K 11K gerard_drazba.jpg
14K 14K greg_davidson.jpg
14K 12K greg_sabino_mullane.jpg
14K 15K jeff_boes.jpg
14K 12K jon_jensen.jpg
13K 12K josh_ausborne.jpg
13K 14K josh_tolley.jpg
13K 11K josh_williams.jpg
8.9K 9.5K kamil_ciemniewski.jpg
13K 21K kent_krenrich.jpg
15K 12K kiel_christofferson.jpg
9.9K 11K kirk_harr.jpg
7.7K 13K marco_manchego.jpg
12K 13K marina_lohova.jpg
14K 11K mark_johnson.jpg
7.3K 13K matt_galvin.jpg
15K 12K matt_vollrath.jpg
6.6K 14K miguel_alatorre.jpg
13K 14K mike_farmer.jpg
7.1K 19K neil_elliott.jpg
9.9K 9.0K patrick_lewis.jpg
13K 5.6K phin_jensen.jpg
12K 14K richard_templet.jpg
12K 9.9K rick_peltzman.jpg
14K 13K ron_phipps.jpg
9.7K 14K selvakumar_arumugam.jpg
9.3K 15K spencer_christensen.jpg
12K 12K steph_skardal.jpg
15K 18K steve_yoman.jpg
6.7K 15K szymon_guz.jpg
7.5K 6.8K tim_case.jpg
15K 21K tim_christofferson.jpg
9.3K 12K will_plaut.jpg
12K 14K wojciech_ziniewicz.jpg
12K 9.9K zed_jensen.jpg
TOTALS
Retina: 549.4K
Normal: 608.8K
</code></pre></div><p>This is where I found the biggest, and best, surprise. The cumulative size of the Retina image files was <em>less</em> than that of the original images. So now we have support for Retina displays, making our website look nice on modern screens, while actually using less data transfer. We don’t need JavaScript, cookies, or any extra server-side trickery to do this. And best of all, we don’t have to maintain a separate set of Retina images.</p>
<p>Once you’ve seen the difference in quality on a Retina screen or a new Android phone, you’ll wonder how you ever were able to tolerate the lower-resolution images. And at least for our selection of JPEG images, there’s not even a file size penalty to pay!</p>
<h3 id="reference-reading">Reference reading</h3>
<ul>
<li><a href="https://ivomynttinen.com/blog/a-guide-for-creating-a-better-retina-web/">A guide for creating a better retina web</a> by Ivo Mynttinen</li>
<li><a href="https://www.leemunroe.com/designing-for-high-resolution-retina-displays/">5 Things I Learned Designing For High-Resolution Retina Displays</a> by Lee Munroe</li>
<li><a href="https://developer.apple.com/library/safari/documentation/NetworkingInternet/Conceptual/SafariImageDeliveryBestPractices/Introduction/Introduction.html">About Proper Image Delivery on the Web</a> on the Safari Developer Library</li>
<li><a href="https://developer.apple.com/library/safari/documentation/NetworkingInternet/Conceptual/SafariImageDeliveryBestPractices/ServingImagestoRetinaDisplays/ServingImagestoRetinaDisplays.html">Serving Images Efficiently to Displays of Varying Pixel Density</a> on the Safari Developer Library</li>
</ul>
WebP images experiment on End Point websitehttps://www.endpointdev.com/blog/2014/01/webp-images-experiment-on-end-point/2014-01-28T00:00:00+00:00Jon Jensen
<p>WebP is an image format for RGB images on the web that supports both lossless (like PNG) and lossy (like JPEG) compression. It was released by Google in September 2010 with open source reference software available under the BSD license, accompanied by a royalty-free public patent license, making it clear that they want it to be widely adopted by any and all without any encumbrances.</p>
<p>Its main attraction is smaller file size at similar quality level. It also supports an alpha channel (transparency) and animation for both lossless and lossy images. Thus it is the first image format that offers the transparency of PNG in lossy images at much smaller file size, and animation only available in the archaic limited-color GIF format.</p>
<h3 id="comparing-quality--size">Comparing quality & size</h3>
<p>While considering WebP for an experiment on our own website, we were very impressed by its file size to quality ratio. In our tests it was even better than generally claimed. Here are a few side-by-side examples from our site. You’ll only see the WebP version if your browser supports it:</p>
<table cellspacing="10">
<tbody><tr>
<td><img alt="" height="133" src="/blog/2014/01/webp-images-experiment-on-end-point/image-0.jpeg" width="133"/><br/>12,956 bytes JPEG</td>
<td><img alt="" height="133" src="/blog/2014/01/webp-images-experiment-on-end-point/image-1.webp" width="133"/><br/>2186 bytes WebP</td>
</tr>
<tr>
<td><img alt="" height="133" src="/blog/2014/01/webp-images-experiment-on-end-point/image-2.jpeg" width="133"/><br/>11,149 bytes JPEG</td>
<td><img alt="" height="133" src="/blog/2014/01/webp-images-experiment-on-end-point/image-3.webp" width="133"/><br/>2530 bytes WebP</td>
</tr>
</tbody></table>
<p>The original PNG images were converted by ImageMagick to JPEG, and by <code>cwebp -q 80</code> to WebP. I think we probably should increase the WebP quality a bit to keep a little of the facial detail that flattens out, but it’s amazing how good these images look for file sizes that are only 17% and 23% of the JPEG equivalent.</p>
<p>One of our website’s background patterns has transparency, making the PNG format a necessity, but it also has a gradient, which PNG compression is particularly inefficient with. WebP is a major improvement there, at 13% the size of the PNG. The image is large so I won’t show it here, but you can follow the links if you’d like to see it:</p>
<table cellspacing="15">
<tbody><tr>
<td align="right">337,186 bytes</td><td><a href="https://jon.endpointdev.com/blog/container-pattern.png">container-pattern.png</a></td>
</tr>
<tr>
<td align="right">43,270 bytes</td><td><a href="https://jon.endpointdev.com/blog/container-pattern.webp">container-pattern.webp</a></td>
</tr>
</tbody></table>
<h3 id="browser-support">Browser support</h3>
<p>So, what is the downside? WebP is currently natively supported only in Chrome and Opera among the major browsers, though amazingly, support for other browsers can be added via WebPJS, a JavaScript WebP renderer.</p>
<p><strong>Update! As of 2021, all current major browsers support WebP image rendering.</strong></p>
<p>Why don’t the other browsers add support given the liberal license? Especially Firefox you’d expect to support it. In fact a patch has been pending for years, and a debate about adding support still smolders. Why?</p>
<p>WebP does not yet support progressive rendering, Exif tagging, non-RGB color spaces such as CMYK, and is limited to 16,384 pixels per side. Some Firefox developers feel that it would do the Internet community a disservice to support an image format still under development and cause uncertain levels of support in various clients, so they will not accept WebP in its current state.</p>
<p>Many batch image-processing tools now support WebP, and there is a free Photoshop plug-in for it. Some websites are quietly using it just because of the cost savings due to reduced bandwidth.</p>
<p>For our first experiment serving WebP images from the End Point website, I decided to serve WebP images only to browsers that claim to be able to support it. They advertise that support in this HTTP request header:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">Accept: image/webp,*/*;q=0.8
</code></pre></div><p>That says explicitly that the browser can render image/webp, so we just need to configure the server to send WebP images. One way to do that is in the application server, by having it send URLs pointing to WebP files.</p>
<p>Let’s plan to have both common format (JPEG or PNG) and WebP files side by side, and then try a way that is transparent to the application and can be enabled or disabled very easily.</p>
<h3 id="web-server-rewrites">Web server rewrites</h3>
<p>It’s possible to set up the web server to transparently serve WebP instead of JPEG or PNG if a matching file exists. Based on some examples other people posted, we used this nginx configuration:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">set $webp "";
set $img "";
if ($http_accept ~* "image/webp") { set $webp "can"; }
if ($request_filename ~* "(.*)\.(jpe?g|png)$") { set $img $1.webp; }
if (-f $img) { set $webp "$webp-have"; }
if ($webp = "can-have") {
add_header Vary Accept;
rewrite "(.*)\.\w+$" $1.webp break;
break;
}
</code></pre></div><p>It’s also good to add to /etc/nginx/mime.types:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">image/webp .webp
</code></pre></div><p>so that .webp files are served with the correct MIME type instead of the default application/octet-stream, or worse, text/plain with perhaps a bogus character set encoding.</p>
<p>Then we just make sure identically-named .webp files match .png or .jpg files, such as those for our examples above:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">-rw-rw-r-- 337186 Nov 6 14:10 container-pattern.png
-rw-rw-r-- 43270 Jan 28 08:14 container-pattern.webp
-rw-rw-r-- 14734 Nov 6 14:10 josh_williams.jpg
-rw-rw-r-- 3386 Jan 28 08:14 josh_williams.webp
-rw-rw-r-- 13420 Nov 6 14:10 marina_lohova.jpg
-rw-rw-r-- 2776 Jan 28 08:14 marina_lohova.webp
</code></pre></div><p>A request for a given $file.png will work as normal in browsers that don’t advertise WebP support, while those that do will instead receive the $file.webp image.</p>
<p>The image is still being requested with a name ending in .jpg or .png, but that’s just a name as far as both browser and server are concerned, and the image type is determined by the MIME type in the HTTP response headers (and/or by looking at the file’s magic numbers). So the browser will have a file called $something.jpg in the DOM and in its cache, but it will actually be a WebP file. That’s ok, but could be confusing to users who save the file for whatever reason and find it isn’t actually the JPEG they were expecting.</p>
<h3 id="301302-redirect-option">301/302 redirect option</h3>
<p>One remedy for that is to serve the WebP file via a 301 or 302 redirect instead of transparently in the response, so that the browser knows it’s dealing with a different file named $something.webp. To do that we changed the nginx configuration like this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">rewrite "(.*)\.\w+$" $1.webp permanent;
</code></pre></div><p>That adds a little bit of overhead, around 100-200 bytes unless large cookies are sent in the request headers, and another network round-trip or two, though it’s still a win with the reduced file sizes we saw. However, I found that it isn’t even necessary right now due to an interesting behavior in Chrome that may even be intentional to cope with this very situation. (Or it may be a happy accident.)</p>
<h3 id="chrome-image-download-behavior">Chrome image download behavior</h3>
<p>Versions of Chrome I tested only send the Accept: image/webp [etc.] request header when fetching images from an HTML page, not when you manually request a single file or asking the browser to save the image from the page by right-clicking or similar. In those cases the Accept header is not sent, so the server doesn’t know the browser supports WebP, so you get the JPEG or PNG you asked for. That was actually a little confusing to hunt down by sniffing the HTTP traffic on the wire, but it may be a nice thing for users as long as WebP is still less-known.</p>
<h3 id="batch-conversion">Batch conversion</h3>
<p>It’s fun to experiment, but we needed to actually get all the images converted for our website. Surprisingly, even converting from JPEG isn’t too bad, though you need a higher quality setting and the file size will be larger. Still, for best image quality at the smallest file size, we wanted to start with original PNG images, not recompress JPEGs.</p>
<p>To make that easy, we wrote <a href="https://gist.github.com/jonjensen/8677031">two shell scripts</a> for Linux, bash, and cwebp. We found a few exceptional images that were larger in WebP than in PNG or JPEG, so the script deletes any WebP file that is not smaller, and our nginx configuration will in that case not find a .webp file and will serve the original PNG or JPEG.</p>
<h3 id="full-page-download-sizes-compared">Full-page download sizes compared</h3>
<p>Here are performance tests run by WebPageTest.org using Chrome 32 on Windows 7 on a simulated cable Internet connection. The total download size difference is most impressive, and on a slower mobile network or with higher latency (greater distance from the server) would affect the download time more.</p>
<table id="pagespeeds">
<tbody><tr>
<th align="center" rowspan="2">Page URL</th>
<th align="center" colspan="2">With WebP</th>
<th align="center" colspan="2">Without WebP</th>
</tr>
<tr>
<th align="center">Bytes</th>
<th align="center">Time</th>
<th align="center">Bytes</th>
<th align="center">Time</th>
</tr>
<tr>
<td>https://www.endpointdev.com/</td>
<td align="right">374 KB</td>
<td align="right">2.9s</td>
<td align="right">850 KB</td>
<td align="right">3.4s</td>
</tr>
<tr>
<td>https://www.endpointdev.com/team/</td>
<td align="right">613 KB</td>
<td align="right">3.6s</td>
<td align="right">1308 KB</td>
<td align="right">4.1s</td>
</tr>
</tbody></table>
<h3 id="conclusion">Conclusion</h3>
<p>This article is not even close to a comprehensive shootout between WebP and other image types. There are other sites that consider the image format technical details more closely and have well-chosen sample images.</p>
<p>My purpose here was to convert a real website in bulk to WebP without hand-tuning individual images or spending too much time on the project overall, and to see if the overall infrastructure is easy enough to set up, and the download size and speed improved enough to make it worth the trouble, and get real-world experience with it to see if we can recommend it for our clients, and in which situations.</p>
<p>So far it seems worth it, and we plan to continue using WebP on our website. With empty browser caches, visit <a href="/">www.endpointdev.com</a> using Chrome and then one of the browsers that doesn’t support WebP, and see if you notice a speed difference on first load, or any visual difference.</p>
<p>I hope to see WebP further developed and more widely supported.</p>
<h3 id="further-reading">Further reading</h3>
<ul>
<li><a href="https://developers.google.com/speed/webp/">WebP project home</a></li>
<li><a href="http://en.wikipedia.org/wiki/WebP">Wikipedia on WebP</a></li>
<li>Firefox debate: <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=600919">bug #600919</a>, <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=856375">bug #856375</a></li>
<li><a href="http://webpjs.appspot.com/">WebPJS</a></li>
</ul>
A Performance Case Studyhttps://www.endpointdev.com/blog/2011/02/performance-case-study/2011-02-09T00:00:00+00:00Steph Skardal
<table cellpadding="5" cellspacing="0" width="100%">
<tbody><tr>
<td valign="top">
<p>I’m a sucker for a good performance case study. So, when I came across a general request for performance improvement suggestions at <a href="https://inspiredology.com/">Inspiredology</a>, I couldn’t help but experiment a bit.</p>
<p>The site runs on WordPress and is heavy on the graphics as it’s a site geared towards web designers. I inquired to the site administrators about grabbing a static copy of their home page and using it for a case study on our blog. My tools of choice for optimization were <a href="https://www.webpagetest.org/">webpagetest.org</a> and <a href="http://yslow.org/">YSlow</a>.</p>
<p>Here are the results of a 4-step optimization in visual form:</p>
<table cellpadding="0" celspacing="0" width="100%">
<tbody><tr><td>
<img src="https://chart.apis.google.com/chart?chxl=0:|Step+%234|Step+%233|Step+%232|Step+%231|Original&chxr=0,0,15|1,0,15&chxt=y,t&chbh=a&chs=300x225&cht=bhg&chco=A2C180&chds=0,15&chd=t:13.412,11.957,10.561,10.243,9.212&chtt=First+Request+Load+Time+(seconds)"/>
</td><td>
<img src="https://chart.apis.google.com/chart?chxl=0:|Step+%234|Step+%233|Step+%232|Step+%231|Original&chxr=0,0,15|1,0,15&chxt=y,t&chbh=a&chs=300x225&cht=bhg&chco=A2C180&chds=0,15&chd=t:7.329,3.717,3.278,2.434,2.563&chtt=Repeat+Request+Load+Time+(seconds)"/>
</td></tr></tbody></table>
</td>
<td align="center" valign="top">
<a href="/blog/2011/02/performance-case-study/image-2-big.png" onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}"><img alt="" border="0" id="BLOGGER_PHOTO_ID_5571698267043922738" src="/blog/2011/02/performance-case-study/image-2.png" style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 135px; height: 400px;"/></a>
<p>Inspiredology’s complete homepage.</p>
</td>
</tr>
</tbody></table>
<p>The graph on the left shows the page load time in seconds for a first time view. Throughout optimization, page load time goes from 13.412 seconds to 9.212 seconds. Each step had a measurable impact. The graph on the right shows the page load time in seconds for a repeated view, and this goes from 7.329 seconds to 2.563 seconds throughout optimization. The first optimization step (CSS spriting and file combination) yielded a large performance improvement. I’m not sure why there’s a slight performance decrease between step 3 and step 4.</p>
<p>And here’s a summary of the changes involved in each step:</p>
<ul>
<li>
<p>Step 1</p>
<ul>
<li>Addition of CSS Sprites: I <a href="/blog/2010/09/css-sprites/">wrote about CSS Sprites</a> a while back and A List Apart has an older but still relevant article on CSS Sprites <a href="https://web.archive.org/web/20110217044912/www.alistapart.com/articles/sprites">here</a>. Repeating elements like navigation components, icons, and buttons are suitable for CSS sprites. Article or page-specific images are not typically suitable for CSS sprites. For Inspiredology’s site, I created two sprited images—one with a large amount of navigation components, and one with some of their large background images. You can find a great tool for building CSS rules from a sprited image <a href="https://web.archive.org/web/20110217071401/http://www.spritebox.net/">here</a>.</li>
<li>Combination of JS and CSS files, where applicable. Any JavaScript or CSS files that are included throughout the site are suitable for combination. Files that can’t be combined include suckerfish JavaScript like Google Analytics or marketing service scripts.</li>
<li>Moved JavaScript requests to the bottom of the HTML. This is recommended because JavaScript requests block parallel downloading. Moving them to the bottom allows page elements to be downloaded and rendered first, followed by JavaScript loading.</li>
</ul>
</li>
<li>
<p>Step 2</p>
<ul>
<li>Image compression with jpegtran, pngcrush, convert. I use pngcrush often. I read about jpegtran in <a href="https://developer.yahoo.com/performance/rules.html">Yahoo’s Best Practices for Speeding Up Your Web Site</a>. I <a href="/blog/2009/12/jpeg-compression-quality-or-quantity/">wrote a bit about image compression</a> a while ago and briefly experimented with image compression using imagemagick on Inspiredology’s images.</li>
</ul>
</li>
<li>
<p>Step 3</p>
<ul>
<li>Addition of expires headers and disabling ETags: These are standard optimization suggestions. <a href="/team/jon-jensen/">Jon Jensen</a> wrote about using these a bit <a href="/blog/2010/11/speeding-up-spree-demo-site/">here</a> and <a href="/blog/2009/10/performance-optimization-of/">here</a>.</li>
</ul>
</li>
<li>
<p>Step 4</p>
<ul>
<li>Serving gzipped content with mod_deflate: Also a fairly standard optimization suggestion. Although, I should note I had some issues gzipping a couple of the files and since the site was in a temporary location, I didn’t spend much time troubleshoo</li>
<li>A bit more cleanup of rogue html and CSS files. In particular, there was one HTML file requested that didn’t have any content in it and another that had JavaScript that I appended to the combined JavaScript file (combined</li>
</ul>
</li>
</ul>
<p>A side-by-side comparison of webpagetest.org’s original versus step 4 results highlights the reduction of requests in the waterfall and the large reduction in requests on the repeat view:</p>
<table width="100%">
<tbody><tr>
<td valign="top">
<a href="/blog/2011/02/performance-case-study/image-3-big.png" onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}"><img alt="" border="0" id="BLOGGER_PHOTO_ID_5571698261206853250" src="/blog/2011/02/performance-case-study/image-3.png" style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 318px; height: 400px;"/></a>
</td>
<td valign="top">
<a href="/blog/2011/02/performance-case-study/image-4-big.png" onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}"><img alt="" border="0" id="BLOGGER_PHOTO_ID_5571698261191315554" src="/blog/2011/02/performance-case-study/image-4.png" style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 400px; height: 365px;"/></a>
</td>
</tr>
</tbody></table>
<h3 id="what-next">What Next?</h3>
<p>At this point, <a href="https://www.webpagetest.org/">webpagetest.org</a> suggests the following changes:</p>
<ul>
<li>Gzipping the remaining components has a potential of reducing total bytes of the first request by ~10%.</li>
<li>Additional image compression has the potential of reducing total bytes of the first request by about ~6%. This metric is based on their image compression check: “JPEG—Within 10% of a photoshop quality 50 will pass, up to 50% larger will warn and anything larger than that will fail.” Quite a few of Inspiredology’s jpgs did not pass this test and could be optimized further.</li>
<li>Use a CDN. This is a common optimization suggestion, but the cost of a CDN isn’t always justified for smaller sites.</li>
</ul>
<p>I would suggest:</p>
<ul>
<li>Revisiting CSS spriting to further optimize. I only spent a short time spriting and didn’t work out all the kinks. There were a few requests that I didn’t sprite because they were repeating elements, but repeating elements can be sprited together. Another 5 requests might be eliminated with additional CSS spriting.</li>
<li>Server-optimization: Inspiredology runs on WordPress. We’ve used the <a href="https://wordpress.org/plugins/wp-cache/">wp-cache</a> plugin for a couple of our clients running WordPress, which I believe helps. But note that the case study presented here is a static page with static assets, so there is obviously a huge gain to be had by optimizing serving images, CSS, and JavaScript.</li>
<li>Database optimization: Again, there’s no database in play in this static page experiment. But there’s always room for improvement on database optimization. <a href="/team/josh-tolley/">Josh Tolley</a> recently made performance improvements for one of our clients running on Rails with postgreSQL using <a href="https://bucardo.org/Pgsi/">pgsi</a>, our open source postgreSQL performance reporting tool, and had outrageously impressive benchmarked improvements.</li>
<li>I just read an article about CSS selectors. The combined.css file I created for this case study has 2000 lines. Although there might be only a small win with optimization here, surely optimization and cleanup of that file can be beneficial.</li>
<li>I recently wrote about several <a href="/blog/2011/01/jquery-tips-ecommerce/">jQuery tips</a>, including performance optimization techniques. This isn’t going to improve the serving of static assets, but it would be another customer-facing enhancement that can improve the usability of the site.</li>
</ul>
<p>I highly recommend reading <a href="https://developer.yahoo.com/performance/rules.html">Yahoo’s Best Practices on Speeding Up Your Web Site</a>. They have a great summary of performance recommendations, covering the topics described in this article and lots more.</p>
Postgres SQL Backup Gzip Shrinkage, aka Don’t Panic!!!https://www.endpointdev.com/blog/2010/01/postgres-sql-backup-gzip-shrinkage-aka/2010-01-09T00:00:00+00:00David Christensen
<p>I was investigating a recent Postgres server issue, where we had
discovered that one of the RAM modules on the server in question had
gone bad. Unsurprisingly, one of the things we looked at was the
possibility of having to do a restore from a SQL dump, as if there had
been any potential corruption to the data directory, a base backup
would potentially have been subject to the same possible errors that
we were trying to restore to avoid.</p>
<p>As it was already the middle of the night (anyone have a server
emergency during the normal business hours?), my investigations were
hampered by my lack of sleep.</p>
<p>If there had been some data directory corruption, the pg_dump process
would likely fail earlier than in the backup process, and we’d expect
the dumps to be truncated; ideally this wasn’t the case, as memory
testing had not shown the DIMM to be bad, but the sensor had alerted
us as well.</p>
<p>I logged into the backup server and looked at the backup dumps; from
the alerts that we’d gotten, the memory was flagged bad on January 3.
I listed the files, and noticed the following oddity:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain"> -rw-r--r-- 1 postgres postgres 2379274138 Jan 1 04:33 backup-Jan-01.sql.gz
-rw-r--r-- 1 postgres postgres 1957858685 Jan 2 09:33 backup-Jan-02.sql.gz
</code></pre></div><p>Well, this was disconcerting. The memory event had taken place on the
3rd, but there was a large drop in size of the dumps between January
1st and January 2nd (more than 400MB of <em>compressed</em> output, for those of
you playing along at home). This indicated that either the memory
event took place earlier than recorded, or something somewhat
catastrophic had happened to the database; perhaps some large deletion
or truncation of some key tables.</p>
<p>Racking my brains, I tried to come up with an explanation: we’d had a
recent maintenance window that took place between January 1 and
January 2; we’d scheduled a CLUSTER/REINDEX to reclaim some of the
bloat which was in the database itself. But this would only reduce
the size of the data directory; the amount of live data would have
stayed the same or with a modest increase.</p>
<p>Obviously we needed to compare the two files in order to determine
what had changed between the two days. I tried:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain"> diff <(zcat backup-Jan-01.sql.gz | head -2300) <(zcat backup-Jan-02.sql.gz | head -2300)
</code></pre></div><p>Based on my earlier testing, this was the offset in the SQL dumps
which defined the actual schema for the database excluding the data;
in particular I was interested to see if there had been (say) any
temporarily created tables which had been dropped during the
maintenance window. However, this showed only minor changes (updates
to default sequence values). It was time to do a full diff of the
data to try and see if some of the aforementioned temporary tables had
been truncated or if some catastrophic deletion had occurred or…you
get the idea. I tried:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain"> diff <(zcat backup-Jan-01.sql.gz) <(zcat backup-Jan-02.sql.gz)
</code></pre></div><p>However, this approach fell down when diff ran out of memory. We
decided to unzip the files and manually diff the two files in case it
had something to do with the parallel unzips, and here was a mystery;
after unzipping the dumps in question, we saw the following:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain"> -rw-r--r-- 1 root root 10200609877 Jan 8 02:19 backup-Jan-01.sql
-rw-r--r-- 1 root root 10202928838 Jan 8 02:24 backup-Jan-02.sql
</code></pre></div><p>The uncompressed versions of these files showed sizes consistent with
slow growth; the Jan 02 backup was slightly larger than the Jan 01
backup. This was really weird! Was there some threshold in gzip
where given a particular size file it switched to a different
compression algorithm? Had someone tweaked the backup script to gzip
with a different compression level? Had I just gone delusional from
lack of sleep? Since gzip can operate on streams, the first option
seemed unlikely, and something I would have heard about before. I
verified that the arguments to gzip in the backup job had not changed,
so that took that choice off the table. Which left the last option,
but I had the terminal scrollback history to back me up.</p>
<p>We finished the rest of our work that night, but the gzip oddity stuck
with me through the next day. I was relating the oddity of it all to
a co-worker, when insight struck: since we’d CLUSTERed the table, that
meant that similar data (in the form of the tables’ multi-part primary
keys) had been reorganized to be on the same database pages, so when
pg_dump read/wrote out the data in page order, gzip had that much more
similarity in the same neighborhood to work with, which resulted in
the dramatic decrease in the compressed gzip dumps.</p>
<p>So the good news was that CLUSTER will save you space in your SQL
dumps as well (if you’re compressing), the bad news was that it took
an emergency situation and an almost heart-attack for this engineer to
figure it all out. Hope I’ve saved you the trouble… :-)</p>
JPEG compression: quality or quantity?https://www.endpointdev.com/blog/2009/12/jpeg-compression-quality-or-quantity/2009-12-24T00:00:00+00:00Daniel Browning
<p>There are many aspects of JPEG files that are interesting to web site developers, such as:</p>
<ul>
<li>The optimal trade off between quality and file size for any encoder and uncompressed source image.</li>
<li>Reducing size of an existing JPEG image when the uncompressed source is unavailable, but still finding the same optimal trade-off.</li>
<li>Comparison of different encoders and/or settings for quality at a given file size.</li>
</ul>
<p>Two essential factors are file size and image quality. Bytes are objectively measurable, but image quality is much more nebulous. What to one person is a perfectly acceptable image is to another a grotesque abomination of artifacts. So the quality factor is subjective. For example, Steph sent me some images to compare compression artifacts. Here is the first one with three different settings in ImageMagick: 95, 50, and 8:</p>
<table cellpadding="0" cellspacing="2">
<tbody><tr><td valign="top">
<a href="/blog/2009/12/jpeg-compression-quality-or-quantity/image-0.jpeg"><img src="/blog/2009/12/jpeg-compression-quality-or-quantity/image-0.jpeg" title="size: 27K setting: 95"/></a></td>
<td valign="top"><a href="/blog/2009/12/jpeg-compression-quality-or-quantity/image-1.jpeg"><img src="/blog/2009/12/jpeg-compression-quality-or-quantity/image-1.jpeg" title="size: 8.0K setting: 50"/></a></td>
<td valign="top">
<a href="/blog/2009/12/jpeg-compression-quality-or-quantity/image-2.jpeg"><img src="/blog/2009/12/jpeg-compression-quality-or-quantity/image-2.jpeg" title="size: 2.9K setting: 8"/></a></td></tr></tbody></table>
<p>Compare the subtle (or otherwise) differences in the following images (mouseover shows the filesize and compression setting):</p>
<table>
<tbody><tr>
<td><a href="/blog/2009/12/jpeg-compression-quality-or-quantity/image-3.jpeg"><img src="/blog/2009/12/jpeg-compression-quality-or-quantity/image-3.jpeg" title="size: 30K setting: 95"/></a></td>
<td><a href="/blog/2009/12/jpeg-compression-quality-or-quantity/image-4.jpeg"><img src="/blog/2009/12/jpeg-compression-quality-or-quantity/image-4.jpeg" title="size: 17K setting: 85"/></a></td>
</tr><tr>
<td><a href="/blog/2009/12/jpeg-compression-quality-or-quantity/image-5.jpeg"><img src="/blog/2009/12/jpeg-compression-quality-or-quantity/image-5.jpeg" title="size: 7.7K setting: 50"/></a></td>
<td><a href="/blog/2009/12/jpeg-compression-quality-or-quantity/image-6.jpeg"><img src="/blog/2009/12/jpeg-compression-quality-or-quantity/image-6.jpeg" title="size: 4.1K setting: 20"/></a></td>
</tr></tbody></table>
<p>Mouseover each image for the file size and ImageMagick compression setting. Additional comparisons are below. Each image can be opened in a separate browser tab for easy A/B comparison. I think many would find the setting of 8 to have too many artifacts, even though it’s 10 times smaller than image compressed at a setting of 95. Some would find the setting of 50 to be an acceptable tradeoff between quality and size, since it sends 3.4 times fewer bytes.</p>
<p>Here is the code I wrote to make the comparison (shell script is great for this stuff):</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash"><span style="color:#c00;font-weight:bold">#!/bin/bash
</span><span style="color:#c00;font-weight:bold"></span><span style="color:#369">HTML_OUTFILE</span>=comparison.html
<span style="color:#038">echo</span> <span style="color:#d20;background-color:#fff0f0">'<html>'</span> > <span style="color:#369">$HTML_OUTFILE</span>
write_img_html () {
<span style="color:#369">size</span>=<span style="color:#d20;background-color:#fff0f0">`</span>du -h --apparent-size <span style="color:#369">$1</span> | cut -f 1<span style="color:#d20;background-color:#fff0f0">`</span>
<span style="color:#080;font-weight:bold">if</span> [ -n <span style="color:#d20;background-color:#fff0f0">"</span><span style="color:#369">$2</span><span style="color:#d20;background-color:#fff0f0">"</span> ]; <span style="color:#080;font-weight:bold">then</span>
<span style="color:#369">qual</span>=<span style="color:#d20;background-color:#fff0f0">"setting: </span><span style="color:#369">$2</span><span style="color:#d20;background-color:#fff0f0">"</span>
<span style="color:#080;font-weight:bold">fi</span>
cat <span style="color:#d20;background-color:#fff0f0"><<EOF >>$HTML_OUTFILE
</span><span style="color:#d20;background-color:#fff0f0"><a href="$1"><img src="$1" title="size: $size $qual"></a>
</span><span style="color:#d20;background-color:#fff0f0">EOF</span>
}
<span style="color:#080;font-weight:bold">for</span> name in image1 image2; <span style="color:#080;font-weight:bold">do</span>
<span style="color:#369">orig</span>=<span style="color:#369">$name</span>-original.jpg
<span style="color:#369">resized</span>=<span style="color:#369">$name</span>-300.png
<span style="color:#038">echo</span> Resizing <span style="color:#369">$orig</span> to <span style="color:#00d;font-weight:bold">300</span> on longest side: <span style="color:#369">$resized</span>...
convert <span style="color:#369">$orig</span> -resize 300x300 <span style="color:#369">$resized</span>
write_img_html <span style="color:#369">$resized</span> <span style="color:#d20;background-color:#fff0f0">"lossless"</span>
<span style="color:#080;font-weight:bold">for</span> quality in <span style="color:#00d;font-weight:bold">100</span> <span style="color:#00d;font-weight:bold">95</span> <span style="color:#00d;font-weight:bold">85</span> <span style="color:#00d;font-weight:bold">50</span> <span style="color:#00d;font-weight:bold">20</span> <span style="color:#00d;font-weight:bold">8</span> 1; <span style="color:#080;font-weight:bold">do</span>
<span style="color:#038">echo</span> Creating JPEG quality <span style="color:#369">$quality</span>...
<span style="color:#369">jpeg</span>=<span style="color:#369">$name</span>-300-q-<span style="color:#369">$quality</span>.jpg
convert <span style="color:#369">$resized</span> -strip -quality <span style="color:#369">$quality</span> <span style="color:#369">$jpeg</span>
write_img_html <span style="color:#369">$jpeg</span> <span style="color:#369">$quality</span>
<span style="color:#080;font-weight:bold">done</span>
<span style="color:#080;font-weight:bold">done</span>
</code></pre></div><p>Another factor that often comes into play is how artifacts in the image (e.g. aliasing, ringing, noise) combine with JPEG compression artifacts to exacerbate quality problems. So one way to get smaller file sizes is to reduce the other types of artifacts in the image, thereby allowing higher JPEG compression.</p>
<p>The most common source of artifacts is image resizing. If you are resizing the images, I strongly recommend using a program that has a high quality filter. Irfanview and ImageMagick are two good choices.</p>
<p>The ideal situation is this:</p>
<ul>
<li>Uncompressed source image</li>
<li>Full-resolution if you will be handling the resize</li>
<li>Absent artifacts such as aliasing</li>
<li>Resize performed with good software like ImageMagick</li>
<li>JPEG compression chosen based on subjective quality assessment.</li>
</ul>
<p>Choosing the trade-off between quality and file size is difficult in part because it varies by image content. Images with lots of small color details (e.g. bright fabric threads; AKA high spatial frequency chroma) stand less compression than images that only have medium sized details that do not have important and minute color information.</p>
<p>One of the settings that is important for small web images is removal of the color space profile (e.g. sRGB). The only time it is needed is when there is a good reason for using non-sRGB JPEG, such as when you are certain that your users will have color managed browsers. Removing it can shave off 5KB or so; software will assume images without profiles have an sRGB profile. It can be removed with the -strip parameter of ImageMagick.</p>
<p>As for choosing the specific compression settings, keep in mind that there are over 30 different types of options/techniques that can be used in compressing the image. Most image programs simplify that to a sliding scale from 0 to 100, 1 to 12, or something else. Keep in mind that even when programs use the same scale (e.g. 0 to 100), they probably have different ideas of what the numbers mean. 95 in one program may be very different than 95 in another.</p>
<p>If bandwidth is not an issue, then I use a setting of 95 on ImageMagick, because in normal images I can’t tell the difference between 95 and 100. But when file size in an important concern, I consider 85 to be the optimal setting. In this image, the difference should be clear, but I generally find that cutting filesize in half is worth it. Below 85, the artifacts are too onerous for my taste.</p>
<p>You don’t often hear about web site visitors’ dissatisfaction with compression artifacts, so you might be tempted to just reduce file sizes even beyond the point where it gets noticable. But I think there is a subliminal effect from the reduced image quality. Visitors may not stop visiting the site immediately, but my gut feeling is it leaves them with a certain impression in their mind or taste in their mouth. I would guess that user testing might result in comments such as “the X web site is not the same high-grade quality as the Y web site”, even if they don’t put it into words as specific as “the compression artifacts make X look uglier than Y”. Even if that pet theory is true, it still has to be balanced against the benefit of faster page loading times.</p>
<p>Ideally, the tradeoff between quality and page loading time would be a choice left to the user. Those who prefer fewer artifacts could set their browser to download larger, less-compressed image files than the default, while users with low bandwidth could set it for more compressed images to get a faster page load at the expense of quality. I could imagine an Apache module and corresponding Firefox add-on some day.</p>
<p>Regarding the situation where you want to reduce the file size of existing JPEGs, my advice is to first try (hard) to get the original source files. You can do better (for any given quality/size tradeoff) from those than you can by just manipulating the existing files. If that’s not possible, then the suboptimal workflows like jpegtran, jpegoptim, and doing a full decompress/recompress are the only alternative.</p>
<p>As far as comparing different encoders, I haven’t really looked into that except to compare ImageMagick and Photoshop, where I (subjectively) determined they both had about similar quality for file size (and vice-versa).</p>
<p>Here are all the comparison images. The file size and ImageMagick quality setting are in the rollover. I suggest opening images in browser tabs for easy A/B comparison.</p>
<table>
<tbody><tr>
<td><a href="/blog/2009/12/jpeg-compression-quality-or-quantity/image-7.png"><img src="/blog/2009/12/jpeg-compression-quality-or-quantity/image-7.png" title="size: 87K setting: lossless"/></a></td>
<td><a href="/blog/2009/12/jpeg-compression-quality-or-quantity/image-8.jpeg"><img src="/blog/2009/12/jpeg-compression-quality-or-quantity/image-8.jpeg" title="size: 74K setting: 100"/></a></td>
<td><a href="/blog/2009/12/jpeg-compression-quality-or-quantity/image-0.jpeg"><img src="/blog/2009/12/jpeg-compression-quality-or-quantity/image-0.jpeg" title="size: 27K setting: 95"/></a></td>
<td><a href="/blog/2009/12/jpeg-compression-quality-or-quantity/image-10.jpeg"><img src="/blog/2009/12/jpeg-compression-quality-or-quantity/image-10.jpeg" title="size: 15K setting: 85"/></a></td>
</tr><tr>
<td><a href="/blog/2009/12/jpeg-compression-quality-or-quantity/image-1.jpeg"><img src="/blog/2009/12/jpeg-compression-quality-or-quantity/image-1.jpeg" title="size: 8.0K setting: 50"/></a></td>
<td><a href="/blog/2009/12/jpeg-compression-quality-or-quantity/image-12.jpeg"><img src="/blog/2009/12/jpeg-compression-quality-or-quantity/image-12.jpeg" title="size: 4.8K setting: 20"/></a></td>
<td><a href="/blog/2009/12/jpeg-compression-quality-or-quantity/image-2.jpeg"><img src="/blog/2009/12/jpeg-compression-quality-or-quantity/image-2.jpeg" title="size: 2.9K setting: 8"/></a></td>
<td><a href="/blog/2009/12/jpeg-compression-quality-or-quantity/image-14.jpeg"><img src="/blog/2009/12/jpeg-compression-quality-or-quantity/image-14.jpeg" title="size: 1.7K setting: 1"/></a></td>
</tr></tbody></table>
<table><tbody><tr>
<td><a href="/blog/2009/12/jpeg-compression-quality-or-quantity/image-15.png"><img src="/blog/2009/12/jpeg-compression-quality-or-quantity/image-15.png" title="size: 87K setting: lossless"/></a></td>
<td><a href="/blog/2009/12/jpeg-compression-quality-or-quantity/image-16.jpeg"><img src="/blog/2009/12/jpeg-compression-quality-or-quantity/image-16.jpeg" title="size: 80K setting: 100"/></a></td>
</tr><tr>
<td><a href="/blog/2009/12/jpeg-compression-quality-or-quantity/image-3.jpeg"><img src="/blog/2009/12/jpeg-compression-quality-or-quantity/image-3.jpeg" title="size: 30K setting: 95"/></a></td>
<td><a href="/blog/2009/12/jpeg-compression-quality-or-quantity/image-4.jpeg"><img src="/blog/2009/12/jpeg-compression-quality-or-quantity/image-4.jpeg" title="size: 17K setting: 85"/></a></td>
</tr><tr>
<td><a href="/blog/2009/12/jpeg-compression-quality-or-quantity/image-5.jpeg"><img src="/blog/2009/12/jpeg-compression-quality-or-quantity/image-5.jpeg" title="size: 7.7K setting: 50"/></a></td>
<td><a href="/blog/2009/12/jpeg-compression-quality-or-quantity/image-6.jpeg"><img src="/blog/2009/12/jpeg-compression-quality-or-quantity/image-6.jpeg" title="size: 4.1K setting: 20"/></a></td>
</tr><tr>
<td><a href="/blog/2009/12/jpeg-compression-quality-or-quantity/image-21.jpeg"><img src="/blog/2009/12/jpeg-compression-quality-or-quantity/image-21.jpeg" title="size: 2.2K setting: 8"/></a></td>
<td><a href="/blog/2009/12/jpeg-compression-quality-or-quantity/image-22.jpeg"><img src="/blog/2009/12/jpeg-compression-quality-or-quantity/image-22.jpeg" title="size: 1.3K setting: 1"/></a></td>
</tr></tbody></table>
XZ compressionhttps://www.endpointdev.com/blog/2009/11/xz-compression/2009-11-23T00:00:00+00:00Jon Jensen
<p><a href="https://en.wikipedia.org/wiki/Xz">XZ</a> is a new free compression file format that is starting to be more widely used. The LZMA2 compression method it uses first became popular in the <a href="https://www.7-zip.org/">7-Zip</a> archive program, with an analogous Unix command-line version called <em>7z</em>.</p>
<p>We used XZ for the first time in the <a href="http://www.icdevgroup.org/i/dev">Interchange project</a> in the <a href="http://www.icdevgroup.org/i/dev/news?mv_arg=00039">Interchange 5.7.3</a> packages. Compared to gzip and bzip2, the file sizes were as follows:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">interchange-5.7.3.tar.gz 2.4M
interchange-5.7.3.tar.bz2 2.1M
interchange-5.7.3.tar.xz 1.7M
</code></pre></div><p>Getting that tighter compression comes at the cost of its runtime being about 4 times slower than bzip2, but a bonus is that it decompresses about 3 times faster than bzip2. The combination of significantly smaller file sizes and faster decompression made it a clear win for distributing software packages, leading to it being the format used for packages in <a href="https://docs.fedoraproject.org/release-notes/f12/en-US/html/">Fedora 12</a>.</p>
<p>It’s also easy to use on Ubuntu 9.10, via the standard <em><a href="https://tukaani.org/xz/">xz-utils</a></em> package. When you install that with apt-get, aptitude, etc., you’ll get a scary warning about it replacing <em>lzma</em>, a core package, but this is safe to do because xz-utils provides compatible replacement binaries /usr/bin/lzma and friends (lzcat, lzless, etc.). There is also built-in support in <a href="https://www.gnu.org/software/tar/">GNU tar</a> with the new –xz aka -J options.</p>
Performance optimization of icdevgroup.orghttps://www.endpointdev.com/blog/2009/10/performance-optimization-of/2009-10-23T00:00:00+00:00Jon Jensen
<p>Some years ago Davor Ocelić redesigned <a href="http://www.icdevgroup.org/">icdevgroup.org</a>, Interchange’s home on the web. Since then, most of the attention paid to it has been on content such as news, documentation, release information, and so on. We haven’t looked much at implementation or optimization details. Recently I decided to do just that.</p>
<p><strong>Interchange optimizations</strong></p>
<p>There is currently no separate logged-in user area of icdevgroup.org, so Interchange is primarily used here as a templating system and database interface. The automatic read/write of a server-side user session is thus unneeded overhead, as is periodic culling of the old sessions. So I turned off permanent sessions by making all visitors appear to be search bots. Adding to interchange.cfg:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">RobotUA *
</code></pre></div><p>That would not work for most Interchange sites, which need a server-side session for storing mv_click action code, scratch variables, logged-in state, shopping cart, etc. But for a read-only content site, it works well.</p>
<p>By default, Interchange writes user page requests to a special tracking log as part of its UserTrack facility. It also outputs an X-Track HTTP response header with some information about the visit which can be used by a (to my knowledge) long defunct analytics package. Since we don’t need either of those features, we can save a tiny bit of overhead. Adding to catalog.cfg:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">UserTrack No
</code></pre></div><p>Very few Interchange sites have any need for UserTrack anymore, so this is commonly a safe optimization to make.</p>
<p><strong>HTTP optimizations</strong></p>
<p>Today I ran the excellent webpagetest.org test, and this was the <a href="http://www.webpagetest.org/result/091023_2M8V/">icdevgroup.org test result</a>. Even though icdevgroup.org is a fairly simple site without much bloat, two obvious areas for improvement stood out.</p>
<p>First, gzip/deflate compression of textual content should be enabled. That cuts down on bandwidth used and page delivery time by a significant amount, and with modern CPUs adds no appreciable extra CPU load on either the client or the server.</p>
<p>We’re hosting icdevgroup.org on Debian GNU/Linux with Apache 2.2, which has a reasonable default configuration of mod_deflate that does this, so it’s easy to enable:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">a2enmod deflate
</code></pre></div><p>That sets up symbolic links in /etc/apache2/mods-enabled for deflate.load and deflate.conf to enable mod_deflate. (Use a2dismod to remove them if needed.)</p>
<p>I added two content types for CSS & JavaScript to the default in deflate.conf:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">AddOutputFilterByType DEFLATE text/html text/plain text/xml text/css application/x-javascript
</code></pre></div><p>That used to be riskier when very old browsers such as Netscape 3 and 4 claimed to support compressed CSS & JavaScript but actually didn’t. But those browsers are long gone.</p>
<p>The next easy optimization is to enable proxy and browser caching of static content: images, CSS, and JavaScript files. By doing this we eliminate all HTTP requests for these files; the browser won’t even check with the server to see if it has the current version of these files once it has loaded them into its cache, making subsequent use of those files blazingly fast.</p>
<p>There is, of course, a tradeoff to this. Once the browser has the file cached, you can’t make it fetch a newer version unless you change the filename. So we’ll set a cache lifetime of only one hour. That’s long enough to easily cover most users' browsing sessions at a site like this, but short enough that if we need to publish a new version of one of these files, it will still propagate fairly quickly.</p>
<p>So I added to the Apache configuration file for this virtual host:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">ExpiresActive On
ExpiresByType image/gif "access plus 1 hour"
ExpiresByType image/jpeg "access plus 1 hour"
ExpiresByType image/png "access plus 1 hour"
ExpiresByType text/css "access plus 1 hour"
ExpiresByType application/x-javascript "access plus 1 hour"
FileETag None
Header unset ETag
</code></pre></div><p>This adds the HTTP response header “Cache-Control: max-age=3600” for those static files. I also have Apache remove the ETag header which is not needed given this caching and the Last-modified header.</p>
<p>There are cases where the above configuration would be too broad, for example, if you have:</p>
<ul>
<li>images that differ with the same filename, such as CAPTCHAs</li>
<li>static files that vary based on logged-in state</li>
<li>dynamically-generated CSS or JavaScript files with the same name</li>
</ul>
<p>If the website is completely static, including the HTML, or identical for all users at the same time even though dynamically generated, we could also enable caching the HTML pages themselves. But in the case of icdevgroup.org, that would probably cause trouble with the Gitweb repository browser, live documentation searches, etc.</p>
<p>After those changes, we can see the <a href="http://www.webpagetest.org/result/091023_2M91/">results of a new webpagetest.org run</a> and see that we reduced the bytes transferred, and the delivery time. It’s especially dramatic to see how much faster subsequent page views of the Hall of Fame are, since it has many screenshot thumbnail images.</p>
<p>Optimizing a simple non-commerce site such as icdevgroup.org is easy and even fun. With caution and practicing on a non-production system, complex ecommerce sites can be optimized using the same techniques, with even more dramatic benefits.</p>
rsync and bzip2 or gzip compressed datahttps://www.endpointdev.com/blog/2009/10/rsync-and-bzip2-or-gzip-compressed-data/2009-10-06T00:00:00+00:00Jon Jensen
<p>A few days ago, I learned that gzip has a custom option <code>--rsyncable</code> on Debian (and thus also Ubuntu). This <a href="https://beeznest.wordpress.com/2005/02/03/rsyncable-gzip/">old write-up</a> covers it well, or you can just <code>man gzip</code> on a Debian-based system and see the <code>--rsyncable</code> option note.</p>
<p>I hadn’t heard of this before and think it’s pretty neat. It resets the compression algorithm on block boundaries so that rsync won’t view every block subsequent to a change as completely different.</p>
<p>Because bzip2 has such large block sizes, it forces rsync to resend even more data for each plaintext change than plain gzip does, as <a href="https://web.archive.org/web/20090825013526/http://blog.arithm.com/2008/09/06/rsync-and-bzip2compressed-data/">noted here</a>.</p>
<p>Enter <a href="http://compression.ca/pbzip2/">pbzip2</a>. Based on how it works, I suspect that pbzip2 will be friendlier to rsync, because each thread’s compressed chunk has to be independent of the others. (However, pbzip2 can only operate on real input files, not stdin streams, so you can’t use it with e.g. <code>tar cj</code> directly.)</p>
<p>In the case of <code>gzip --rsyncable</code> and <code>pbzip2</code>, you trade a little lower compression efficency (< 1% or so worse) for reduced network usage by rsync. This is probably a good tradeoff in many cases.</p>
<p>But even more interesting for me, a couple of days ago Avery Pennarun posted an article about his experimental code to use the same principles to <a href="https://apenwarr.ca/log/?m=200910#04">more efficiently store deltas of large binaries in Git repositories</a>. It’s painful to deal with large binaries in any version control system I’ve used, and most people simply say, “don’t do that”. It’s too bad, because when you have everything else related to a project in version control, why not some large images or audio files too? It’s much more convenient for storage, distribution, complete documentation, and backups.</p>
<p>Avery’s experiment gives a bit of hope that someday we’ll be able to store big file changes in Git much more efficiently. (Though it doesn’t affect the size of the initial large object commits, which will still be bloated.)</p>
SDCH: Shared Dictionary Compression over HTTPhttps://www.endpointdev.com/blog/2009/07/sdch-shared-dictionary-compression-over/2009-07-27T00:00:00+00:00Jon Jensen
<p>Here’s something new in HTTP land to play with: Shared Dictionary Compression over HTTP (SDCH, apparently pronounced “sandwich”) is a new HTTP 1.1 extension announced by Wei-Hsin Lee of Google last September. Lee explains that with it “a user agent obtains a site-specific dictionary that then allows pages on the site that have many common elements to be transmitted much more quickly.” SDCH is applied before gzip or deflate compression, and Lee notes 40% better compression than gzip alone in their tests. Access to the dictionaries stored in the client is scoped by site and path just as cookies are.</p>
<p>The first client support was in the Google Toolbar for Internet Explorer, but it is now going to be much more widely used because it is supported in the Google Chrome browser for Windows. (It’s still not in the latest Chrome developer build for Linux, or at any rate not enabled by default if the code is there.)</p>
<p>Only Google’s web servers support it to date, as far as I know. Someone intended to start a mod_sdch project for Apache, but there’s no code at all yet and no activity since September 2008.</p>
<p>It is interesting to consider the challenge this will have on HTTP proxies that filter content, since the entire content would not be available to the proxy to scan during a single HTTP conversation. Sneakily-split malicious payloads would then be reassembled by the browser or other client, not requiring JavaScript or other active reassembly methods. <a href="http://prxbx.com/forums/showthread.php?tid=1379&pid=12519">This forum thread</a> discusses this threat and gives an example of stripping the Accept-encoding: sdch request headers to prevent SDCH from being used at all. Though the threat is real, it’s hard to escape the obvious analogy with TCP filtering, which had to grow from stateless to more difficult stateful TCP packet inspection. New features mean not just new benefits but also new complexity, but that’s not reason to reflexively reject them.</p>
<p>SDCH references:</p>
<ul>
<li><a href="https://groups.google.com/forum/#!forum/SDCH">SDCH Google Group</a> which includes the specification PDF and ongoing discussion</li>
<li><a href="http://assets.en.oreilly.com/1/event/7/Shared%20Dictionary%20Compression%20Over%20HTTP%20Presentation.ppt">Wei-Hsin Lee’s presentation slides on SDCH</a></li>
<li><a href="https://lists.w3.org/Archives/Public/ietf-http-wg/2008JulSep/0441.html">IETF mailing list announcement of SDCH</a> and ensuing discussion thread</li>
<li><a href="http://www.webadminblog.com/index.php/2008/06/24/the-velocity-2008-conference-experience-part-vi/">Velocity 2008 conference notes</a> where the pronunciation of SDCH is given as “sandwich”</li>
<li>Vaporware <a href="https://code.google.com/archive/p/mod-sdch/">Apache mod_sdch project</a></li>
</ul>