Bypassing a CDN to browse a website directly on your origin host

2023-01-20T00:00:00+00:00

Using a content distribution network (CDN) has many advantages over serving a website directly, and for any reasonably large website, you should use one. Those advantages include:

Caching at each of the CDN’s PoPs (points of presence).
Often thousands of PoPs around the world, so traffic will be quick for everyone regardless of how far away they are from your origin server.
Blocking of some Bad Guys automatically at the edge, including DDoS (distributed denial of service attacks) mitigation help.
Origin IP address insulation. Hiding the origin IP address is useful to protect against DDoSes, since CDNs are generally well-defended against DDoS and your origin server probably is not as much.

You should generally be cautious about revealing your websites’ origin IP address. We serve other sites from our origin directly, so we don’t worry too much about sharing it here.

Straight to the source

Sometimes, though, you need to bypass your CDN and test your website directly on its origin server. For example, if you need to test that your website would still work if the CDN goes down, or to sidestep CDN caching or content modification when troubleshooting a problem.

It can be surprisingly confusing to bypass the CDN by yourself and not for the rest of your visitors, especially with newer features like Secure DNS in Chrome and Firefox muddying up traditional DNS resolution behaviors.

We’ll use a browser extension called IPvFoo (available for Chrome and Firefox) to see which IP address our hostname resolves to. Install IPvFoo from the Chrome Web Store or Firefox’s addons.mozilla.org and then click on its indicator next to the search bar.

Before we change anything, www.endpointdev.com resolves to 104.21.30.147:

After a quick whois search, we see that this IP address belongs to Cloudflare, our CDN:

$ whois 104.21.30.147

✀
OrgName:        Cloudflare, Inc.
✀

We can also check the certificate issuer by clicking on the padlock icon in our browser’s search bar, then clicking on “Connection is secure” or similar to see all the details about the certificate issuer.

Before we change anything, we see the TLS/SSL certificate presented is issued by Cloudflare, as expected.

Make the hosts file point directly to our server

Common desktop and server operating systems typically come with a “hosts file” that can be used to provide hostname to IP address translation that happens before your designated DNS servers are consulted.

This is useful to test a hostname without ever adding it to DNS, or to override a DNS entry with your own IP address.

We want to change our hosts file so that www.endpointdev.com points directly to our site’s origin server. How to do this depends on your operating system:

Linux/macOS: Open /etc/hosts as root (using sudo or su) in your favorite editor such as vim or nano.
Windows: Open C:\Windows\System32\drivers\etc\hosts as administrator in a decent plain text editor, not a word processor — I used Notepad++.

Once you have your hosts file open, add a line to point your domain name directly to your origin server. For www.endpointdev.com:

23.239.26.161 www.endpointdev.com

Disable DNS-over-TLS & DNS-over-HTTPS

You may also need to disable any “secure DNS” in your browser, such as DNS-over-TLS or DNS-over-HTTPS.

Chrome

In Chrome, this is called Secure DNS, and you can disable it by going to Settings > Privacy and Security > Security > Use secure DNS, and disabling it.

Firefox

In Firefox, go to Network Settings, then disable DNS over HTTPS.

Then close any open tabs in your browser and quit it completely. Then start it again.

Browse the website directly

Now go to www.endpointdev.com.

IPvFoo is now showing the origin IP address we gave it, so we know that we’re being served data directly from that server.

We can also run whois on our origin server, for good measure:

$ whois 23.239.26.161

✀
OrgName:        Linode
✀

Now we’re getting the Linode IP address, not Cloudflare’s. This is expected, since our origin site is hosted on a server at Linode.

Check certificate issuer

You can also check that the certificate issuer changed. Keep in mind that sometimes your certificate issuer might be the same when serving directly as when serving through a CDN, if you got a certificate signed by the CDN, or if you gave your CDN your TLS key & certificate to use.

But if there is a different certificate being served, we will likely see it was signed by a different certificate authority (CA):

After sidestepping Cloudflare our certificate is issued by Sectigo, so this is more proof our changes are working.

Clean up

When you’re all done, open /etc/hosts again and either delete the line you added or comment it out by putting a # at the beginning of the line.

# 23.239.26.161 www.endpointdev.com

Save.

Then close all browser windows, quit, and restart, and when you go to www.endpointdev.com, Cloudflare should be in the middle again.

Mobile

Note that there is no easy way to do something like this on iOS with an iPhone or iPad, as far as I can tell. And on Android, you either need a rooted phone, or a fancy VPN that lets you change your hostname mapping. I tried Hosts Go and it worked, despite some annoying ads which you don’t have to deal with in an /etc/hosts file 😁.

If you need to test on mobile, or on multiple devices, you could also use a DNS server on your local network, and configure any device to point to your local DNS. Pi-hole is a great option for this; see our blog post from a couple years ago for some of its other features.

Jamstack Conf Virtual 2020: Thoughts & Highlights

2020-06-16T00:00:00+00:00

Welcome to Jamstack Conf Virtual 2020

Last week I attended Jamstack Conf Virtual 2020. It had originally been slated to take place in London, UK but was later transformed into a virtual event in light of the COVID-19 pandemic. The conference began at 2pm London time (thankfully I double-checked this the night before!)—6am for those of us in the Pacific Time Zone.

Up early for #jamstackconf 😎☕️ pic.twitter.com/ydjvrHWCZH
— Greg Davidson (@syncopated) May 27, 2020

Before getting too much further I wanted to mention that if you are not familiar with the Jamstack, You can read more about it at jamstack.org.

To virtually participate in the conference we used an app called Hopin. I had not heard of it before but was impressed with how well it worked. There were over 3000 attendees from 130+ countries one of the times I checked. Phil Hawksworth was the Host/MC for the event and did a great job. There were virtual spaces for the stage, sessions, expo (vendors), and networking. If you opted to, the networking feature paired you with a random attendee for a video chat. I’m not sure what I expected going into it but I thought it was fun. I met a fellow developer from the Dominican Republic. The experience was very similar though more serendipitous than the hallway track or lunch line at an in-person conference.

Keynote

Matt Biilmann opened the conference with a keynote address about the challenges we face as a developer community trying to improve access to accurate, timely and locally relevant information to a global audience. Many billions of users with all kinds of devices and varying levels of connectivity. He moved on to share how Netlify is trying to enable developers to “build websites instead of infrastructure” and “ensure all the best practices become common practices” through features like git-based deployments, build plugins, and edge handlers (more on those later).

State of the Jamstack Survey results

Laurie Voss walked us through the results of the Jamstack Survey 2020. There were some interesting findings and surprises. Later on I read Laurie’s post (which Matt had mentioned in his talk) and found that very interesting as well.

Fireside chat with Harper Reed

Frances Berriman chatted with Harper Reed and asked him about his application of Jamstack principles from years ago when he led the technology team for Barack Obama’s election campaign. He described the need to “get far with very limited resources” and spoke about experimenting with serving HTML from Google App Engine and Amazon S3. Using pre-built HTML allowed them to scale very efficiently and he opted to use message queues rather than interacting with the database to keep things very quick for users.

Another benefit Harper noted was how quickly and easily new team members could be onboarded. Folks who knew HTML, CSS and JavaScript would be up to speed and productive in no time. He admitted it’s more complicated today 😀. Speed is another benefit—he just loves when he comes across a super-fast web site (often mostly static).

Lightning Launch: Netlify ⚡

David Calavera gave a demo of Netlify’s new Edge Handlers feature which lets developers add logic to their code at the edge (e.g. the CDN servers geographically closest to the user). He demonstrated how Edge Handlers make it possible to examine the headers of the request to tailor the response to that unique request. Check out the video of his talk to watch him live code an example. I believe Cloudflare Workers and Fastly’s Edge Compute are operating in a similar problem space. I plan to look into each of these offerings more thoroughly in future.

Lightning Launch: Prismic ⚡

Renaud Bressand from Prismic demoed Slicemachine—a new feature from Prismic which lets you combine nuxt/Vue components with content managed in Prismic. It looked like a very compelling way of enabling better collaboration between developers and content creators.

Lightning Launch: RedwoodJS ⚡

Tom Preston-Werner demoed his latest project RedwoodJS. I had heard Tom speak about this on a few podcasts recently and it was nice to see him demo it for us. It looks interesting and feels reminiscent of Rails. RedwoodJS simplifies wiring up your database to a GraphQL API (they use Prisma for this) and integrating it into a React application. Tom is also the creator of Jekyll—a Jamstack-style tool which has been around for many years. It was nice to see several speakers give him some recognition for his work on that project.

The COVID Tracking Project: 0 to 2M API requests in 3 months

Erin Kissane gave an inspiring talk about her work on The COVID Tracking Project. She described it as an “interim public data rescue project”. Erin and some friends, journalists and volunteers worked together to create the site. They started by scraping COVID data from each state’s web site and storing it in a Google Sheet. They used Gatsby, Contentful, and Sass modules and hosted the site on Netlify. Using the Jamstack approach allowed the site to remain performant and continue to be responsive under some huge traffic spikes. Over time, they iterated on the design and continue to improve the site daily. I highly recommend checking out the video of the talk when you get a chance.

Jamstack for emerging markets

Christian Nwamba described some of the challenges of building sites for users in Nigeria (low power devices, spotty connectivity, unreliable power). He shared that 55% of the most visited sites in Nigeria are global companies (Google, Facebook/Instagram, Netflix, Stack Overflow etc.). Christian reviewed a large, popular banking site in Nigeria and noted its many shortcomings.

To demonstrate how the bank might do better he built an app for transferring money between friends & family built in the Jamstack style and using serverless functions. The most interesting thing I picked up from this was his method of using serverless functions to protect the app secrets (API keys, etc.). The front end of the application did not need to concern itself with this—the serverless functions kept the secrets safe and acted as a proxy between the frontend and the backend APIs. Be sure to take a look at Christian’s code if you are interested.

Managing diabetes with Jamstack

Jamie Bradley taught us about diabetes and the Jamstack app he built to help himself and others manage it. He built HeySugar with Gatsby and Sanity, hosted it on Netlify. He’s making it easier for others to deploy their own instances as well.

Selling tickets without servers. Or frameworks. Or cookies.

Jan van Hellemond has volunteered for years at a very popular conference in Europe (Fronteers I think). In previous years tickets sold out in 6 minutes one year, and 30 seconds in another! Their ticket vendor was struggling to handle to the load and this caused them to oversell early bird tickets. Jan built a Jamstack site to sell their tickets and was very pleased with the performance. He used simple, single-purpose vanilla JavaScript components and simple, single-purpose API handlers (serverless functions).

Jan prerendered as much as possible and seeded a relational database with the tickets available for sale. As the tickets were sold, they were marked sold with a database update. Webhooks were used as customers stepped through the checkout flow. Jan joked about deploying to production on a Friday afternoon because of how safe and simple the deployment process was. There were no services to restart, etc. because “it’s just files”. It was cool to see a practical example and that Jan used the basic building blocks of the web (HTML, JavaScript, CSS, old school links) successfully without reaching for a large JavaScript framework.

The business side of the Jamstack

Ana Rossetto shared how her company has been having great success with the Jamstack. Previously, the agency had been building projects for clients with Drupal. She walked through a project she and the team built to encourage people to buy from small, locally-owned businesses. She was impressed with what they were able to create in a very short amount of time.

Build plugin authors’ session

After the main talks there were several sessions. In the Hopin app, I was able to peek into some of these and the presenters and attendees were chatting (both text chat and video). This was very much like the experience of peeking into conference rooms at a physical venue and choosing whether to stay and participate or move on. After some wandering I chose to attend a session with three Netlify build plugin authors.

Peter Müller built Subfont with a friend. He demonstrated how subsetting web fonts (i.e. only loading the characters & glyphs you really need) can dramatically improve frontend performance. He compared the WebPageTest results for Google Fonts with and without Subfont and Subfont was seconds faster! I have subset and locally hosted webfonts in several client projects here at End Point. It takes time and is a manual process. Peter’s plugin makes this excellent performance improvement relatively painless.

David Darnes demoed his build plugin to turn Ghost content into Markdown files. Very interesting and it seemed flexible enough to work with other tools as well.

Gleb Bahmutov presented the build plugin he created to run Cypress tests as part of your Netlify build process. It was amazing to see how simple it was (single devDependency and single line in the netlify.toml config file).

Videos from the conference

Netlify has already put all of the Jamstack Conf Virtual 2020 talks on YouTube so head over there and check those out if you’d like to. Thanks very much to the team at Netlify for organizing the conf, all of the speakers, vendors and attendees!

Cloudflare and Vue SSR; Client Hydration Failures

2019-06-11T00:00:00+00:00

Photo by freestocks, used under CC0 1.0

I recently worked on getting a client’s Vue application converted to use Server Side Rendering (SSR). SSR works by generating the app’s initial HTML structure using a server-side process and then using client-side JavaScript to initialize or “hydrate” the application state on the client-side.

This worked out well in development and testing, however when it came time to roll things out to production, we ended up with a non-functioning application. While the server-rendered content was displaying inline, the client hydration piece was failing, and I was seeing browser console errors that I had never encountered before:

The error in Chrome:

chunk-vendors.342a7610.js:21 Uncaught DOMException: Failed to execute 'appendChild' on 'Node': This node type does not support this method.
...

The error in Safari:

[Error] HierarchyRequestError: The operation would yield an incorrect node tree.
	get (chunk-vendors.342a7610.js:27:30686)
	er (chunk-vendors.342a7610.js:27:30565)
	Le (chunk-vendors.342a7610.js:27:27806)
	xe (product-listing.0773c46e.js:1:18553)
[Error] TypeError: undefined is not an object (evaluating 't.$scopedSlots') — chunk-vendors.342a7610.js:27:27998
	on (chunk-vendors.342a7610.js:27:12025)
	rn (chunk-vendors.342a7610.js:27:11937)
	nn (chunk-vendors.342a7610.js:27:11584)
	(anonymous function) (chunk-vendors.342a7610.js:27:12759)
	fn (chunk-vendors.342a7610.js:27:12135)
	promiseReactionJob

Since the client hydration was failing, this resulted in a non-functioning front-end application which we had to quickly roll back. I’d scratched my head over this quite a bit, as this had been well-tested in development and staging.

After testing direct access to the origin server via /etc/hosts modification, we determined that the server itself was returning content that was working fine, so this pointed to something about the caching layer.

The errors being generated were similar to errors I’d encountered before during development where the generated DOM in the SSR content was mismatched with the version of the client-side Vue application.

One of the big differences here was that production was being served via Cloudflare (CF), however we had not run into any issues with stale versions (i.e., older versions of code being returned) and the like in the past. Our resources were stamped hashes via webpack, so per-application version shouldn’t have been utilizing cached old versions of the generated resources in question.

Because we could not really debug this issue on production and this was still just a theory that CF was the issue, we ended up setting up a development environment behind Cloudflare.

Sure enough, the same issue started occurring in the development environment with Cloudflare fronting it; now, however, we had time to debug/diagnose without affecting production users.

After comparing the static JS for the app returned through CF against the raw source files, I noticed that they were slightly smaller. We had already minified the source as part of our production build, but CF was further minifying things, removing newlines and JS comments that had been left in the prod build. This made me realize that something on the CF side was minifying resources further.

After some research, it turns out the CF has an AutoMin setting to allow the automatic minification of served resources (JS, CSS, and HTML). This setting was contributing to the minification that I was seeing. However, this could not explain what I was seeing on the JS minification alone; these were verified non-functional changes that should not affect how the app ran.

I turned my attention to the HTML of the page itself. Comparing the CF-returned version with the version returned by the app, I quickly saw that minification was affecting this returned HTML as well, cleaning up spacing/formatting and HTML comments as well.

And herein lay the issue; Vue SSR sticks in HTML comments as placeholders for nodes which are non-visible at server render time. So for example, if you had a Vue component like so:

<template>
  <div>
    My component!
    <div v-if="display">
      <p v-for="l in list">
        {{ l }}
      </p>
    </div>
  </div>
</template>

<script>
export default {
  name: 'MyComponent',
  data () {
    return {
      list: [1, 2, 3]
    }
  }
  props: {
    display: {
      type: Boolean,
      default: false
    }
  }
}
</script>

This would render in Vue SSR as the following HTML (assuming the component was not shown by the calling app):

<div>My component!<!-----></div>

Here, the  placeholder is a significant HTML comment, standing in for a non-visible component in the virtual DOM (the false condition in the v-if directive). Rehydration requires this to be in place for proper handling, so when CF stripped out HTML comments from the returned page output this disrupted the generated server-side DOM. This meant that the server-side DOM no longer matched what Vue was expecting and the client-side hydration failed and the app failed to init properly.

I tested disabling AutoMin for the test site, which resulted in a functioning application, verifying this was the issue. We did not want to turn off AutoMin globally, so I looked around for ways to narrow this down to only the affected pages (all of which had a common URL prefix).

I came across CF’s Page Rules settings, which let you modify its handling of specific URLs. I added an exception for the pages in question, turning off only the AutoMin setting for these pages, and voilà, the app worked while leaving this setting on for the rest of the site.

TL;DR:

Cloudflare’s AutoMin setting can interfere with Vue SSR output by removing significant HTML comments. The fix is to disable AutoMin on the relevant pages using Page Rules.