Using a Containerized Nginx Proxy to Serve a Multi-Application .NET System

2024-09-24T00:00:00+00:00

We recently blogged about how we deployed a system made of multiple .NET applications using Docker containers. In order to make them accessible over the internet, we created a reverse proxy using Nginx.

In that case, we installed and configured the Nginx instance directly in the server, as opposed to the rest of the applications, which ran within containers. That approach did and still does work well for us.

In this article, we’re going to explore an alternative strategy. One where we push the containerization aspect further and deploy and run the Nginx instance itself in a Docker container.

Reintroducing the demo project

Like I said, our system has multiple runtime components, each one of them running in their own container. We have two ASP.NET Core web applications: an Admin Portal and a Web API. They live in this Git repository. And we also have a Postgres database, which the apps interact with.

We also have another repository where the deployment-related files are stored. Among others, there are the expected compose.yaml and Dockerfiles that describe the entire infrastructure.

Throughout this post we will update those deployment configuration files to add an Nginx reverse proxy.

Let’s see what that would look like.

Adding the proxy service in `compose.yaml`

First we need to add the new container in the compose.yaml’s services section. It doesn’t need to be too complicated:

# compose.yaml

services:

# ...

  proxy:
    # The proxy container will be based on this Dockerfile, which we'll define
    # soon.
    build:
      context: .
      dockerfile: Dockerfile.Proxy

    # Here we expose the Nginx proxy via port 8888. This can be anything. In
    # fact, if you want to do multiple parallel deployments on the same machine,
    # that is, with many Nginx instances running at the same time, you can
    # adjust this setting appropriately to prevent port conflicts. Making sure
    # that each instance has its own port.
    ports:
      - 8888:80

    # We want the proxy to start up after the admin-portal and web-api services
    # are up and running. The depends_on setting helps with that.
    depends_on:
      admin-portal:
        condition: service_started
      web-api:
        condition: service_started

#...

Writing the proxy `Dockerfile`

As you saw, the proxy configuration in compose.yaml leverages an external Dockerfile to build the container image that will run our Nginx proxy. This file is also very straightforward. It uses the official Nginx image from Docker Hub and it looks like this:

# Dockerfile.Proxy

# We can pick the version and flavor that we like. In our case here, this is an
# image based on the latest release of Nginx, and the latest release of Debian.
FROM nginx:1.27.1-bookworm

# Unsurprisingly, we have a custom configuration that we want the proxy to use.
# This is how we make sure it does. We copy it into the default location inside
# the image.
COPY proxy/nginx.conf /etc/nginx/nginx.conf

Configuring the proxy

Now we have to configure the Nginx proxy to route requests to both our web applications. Here’s an nginx.conf that does just that:

# proxy/nginx.conf

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;

    keepalive_timeout  65;

    # We have to comment out or remove this line to make sure the default
    # configuration that comes in the official image is not applied.
    # include /etc/nginx/conf.d/*.conf;

    # Our customizations start here:
    server {
        # With this listen directive, we configure our proxy to expect
        # connections coming from the default HTTP port: 80.
        listen 80;

        # This location directive makes sure all requests coming to URLs that
        # look like .../admin are routed to the Admin Portal web app.
        location ~ ^/admin(/?)(.*) {
            proxy_pass http://admin-portal:8080;
            proxy_http_version 1.1;
            proxy_set_header Host $host;
            proxy_set_header Connection keep-alive;
        }

        # This location directive makes sure all requests coming to URLs that
        # look like .../api are routed to the Web API.
        location ~ ^/api(/?)(.*) {
            proxy_pass http://web-api:8080;
            proxy_http_version 1.1;
            proxy_set_header Host $host;
            proxy_set_header Connection keep-alive;
        }
    }
}

What I’ve done here is take the default Nginx configuration file that comes right out of the box, and replace the include /etc/nginx/conf.d/*.conf; line with my own server block directive.

As explained in the official image’s Docker Hub page, a quick way of obtaining a copy of this default file is using this command: docker run --rm --entrypoint=cat nginx /etc/nginx/nginx.conf > /host/path/nginx.conf.

The most interesting parts are the proxy_pass directives that take care of redirecting traffic to the apps. Notice how they refer to the Admin Portal using http://admin-portal:8080 and to the Web API by http://web-api:8080. These are the internal naming of these components within the containers’ virtual network, which gets created automatically by Docker Compose.

Remember that this Nginx instance is not running in the host machine directly. Instead it’s running in a container. In the same virtual network as the other containers described in compose.yaml. That’s why it has to use the hostnames assigned by compose.yaml (i.e. admin-portal and web-api) and the port through which the ASP.NET Core apps running within accept requests (i.e. 8080).

This is the idea:

It also uses proxy_set_header directives to set Host and Connection headers. This is typical practice when it comes to Nginx reverse proxies. You can read more about that on Nginx’s website.

Configuring the PathBase in ASP.NET Core apps

There’s an additional step that we have to take to make all this work. We have configured our Nginx proxy to serve both applications under the same “server”, and rely on different URL paths (i.e. /admin vs /api) to determine which app will receive which request. This means that we have to perform further configuration in the apps so that routing is done properly. Thankfully, all it takes is a one-liner in each of the apps’ Program.cs files. After the usual var app = builder.Build(); line, we do the following:

For the Admin Portal, we add this:

// source/VehicleQuotes.AdminPortal/Program.cs

// ...

app.UsePathBase("/admin");

// ...

And for the Web API:

// source/VehicleQuotes.WebApi/Program.cs

// ...

app.UsePathBase("/api");

// ...

With that, the apps are ready to handle requests coming from the Nginx proxy, which will include either the /admin or /api sections.

Deploying

Finally, we can deploy. Go into the directory where all the deployment configuration files live. That’s the one with the compose.yaml file. In other words, the root of the deploy repo. Once in there, run:

docker compose up --build

After a while, Docker compose will have built and deployed everything for us. Now you can navigate to the apps in any browser at http://localhost:8888/admin and http://localhost:8888/api/swagger/index.html.

As you click around, you can see the Nginx logs with:

docker compose logs proxy -f

To bring it all back down, you can run:

docker compose down

Cool! At the end of the day, Nginx is just another program that runs as a process in an operating system. And as such, it can be run in a container. One of the nice aspects about this setup is the convenience of having an entire system described in a set of files, and being able to bring everything up with a single command.

Using Docker Compose to Deploy a Multi-Application .NET System

2024-07-13T00:00:00+00:00

This post was co-authored by Juan Pablo Ventoso

We recently developed a system that involved several runtime components. It was an ecommerce site that included a database, a web API, an admin control panel web app, and a frontend SPA.

There are many ways to deploy such a system. For us, we wanted the infrastructure to be easily replicable for multiple environments with slightly different configurations. We wanted to be able to have, for example, a production and a staging version that could be deployed easily, with minimal configuration changes. We also wanted the infrastructure to be captured in files and version controlled, to further help replicability and maintainability.

With all that in mind, Docker Compose seemed like an ideal option. We could author a series of configuration files, parameterize environment-specific changes and, with a single command, we could spin up a whole environment to run the various applications within the system.

In this blog post, I’ll explain how we did that using a demo .NET code base that has a similar set of components. Let’s get started.

Getting familiar with the demo project

In .NET terms, our demo code base is organized as a solution with multiple projects. Two of those projects are ASP.NET web applications: a Razor Pages web app (the admin portal), and an MVC Web API. The rest are class libraries that define the core domain logic, tests, and other utilities. For deployment purposes, the web application projects are the interesting ones, as they produce executables that actually need to run as processes in the server. So, including the database, our demo system has three runtime components:

The database.
The admin portal.
The web API.

In the world of Docker, that would translate into three separate containers. Considering Docker Compose, that means three separate services.

Throughout this post, we will, step by step, build a compose.yaml, a set of Dockerfiles, and other configuration files which can be used with Docker Compose to deploy our system.

You can find the system’s source code on GitHub. The final version of the deployment files we’ll build in this article are also on GitHub.

Including the code base repository as a Git submodule

Considering the separation of our components, the reasons for the organization of our deployment configuration files and our code base starts to become apparent. The deployment files will live in their own repository. They do need access to the system’s source code, though, in order to build and run the apps. And that source code lives in its own repo. So, the deployment repository will include a source subdirectory, which will be a Git submodule that points to the repository where the system’s source code is stored.

Here’s the file structure that we’re aiming for:

.
├── compose.yaml
├── the various dockerfiles...
├── source
│   ├── vehicle-quotes.sln
│   └── the varous projects...
└── any other files and directories...

Including a Git repository inside another Git repository as a submodule is easy. In our case, we already have the parent Git repo, which is the one where the deployment config files live. To add the source code repo to it, we run a command like this:

git submodule add git@github.com:megakevin/end-point-blog-dotnet-8-demo.git source

Pretty straightforward. The command is calling git submodule add and passing it the location of the repo on GitHub and the directory in which to clone it. That’s it. As a result of that command, Git will have created a new source directory with the contents of the end-point-blog-dotnet-8-demo repository, and a .gitmodules file with contents like these:

[submodule "source"]
  path = source
  url = git@github.com:megakevin/end-point-blog-dotnet-8-demo.git

As you can see, this file contains the information Git needs to know that this repo has a submodule, where it is, and where it comes from.

Deploying the database

Let’s begin actually building out our system components with the database. Our apps use a PostgreSQL database to store information. Luckily for us, getting a PostgreSQL database up and running with Docker Compose is easy. All it takes is a compose.yaml file like this:

# ./compose.yaml

services:
  # This is the name of the service: "db". Other containers will use that as a
  # hostname when they need to interact with it.
  db:
    # The container will be based in the "16.3-bookworm" release of the public
    # PostgreSQL image that's available in Docker Hub
    image: postgres:16.3-bookworm
    # Always restart the container automatically if it ever closes for whatever
    # reason.
    restart: always
    # Makes the database accessible at port 5432.
    ports:
      - 5432:5432
    # Stores the PostgreSQL data files in a docker volume so that they are not
    # lost when the container stops or restarts.
    volumes:
      - db-postgres-data:/var/lib/postgresql/data
    # Sets some basic configuration for the initial database in the PostgreSQL
    # instance: its name, and a pair of credentials to access it.
    environment:
      - POSTGRES_DB=vehicle_quotes
      - POSTGRES_USER=vehicle_quotes
      - POSTGRES_PASSWORD=password

# Defines the Docker volume that the db service uses to store its data files.
volumes:
  db-postgres-data:

This is a very standard definition of a Docker Compose service. We called it db, made it available via the standard PostgreSQL port (5432), and set some basic configurations for it. Check out the PostgreSQL image’s page in Docker Hub to see more complicated use cases.

To bring it to life, we can run docker compose up -d in the directory where we’ve located the compose.yaml file. Running that command will prompt Docker to download the PostgreSQL image, build a container with it, and run it. It’ll also create the db-postgres-data volume we configured and a “network” that all the services we put in the compose.yaml file will be part of. Thanks to the -d option, it will run in daemon mode; that is, as a background process. So as soon as it’s done, it gives us control of our terminal back:

$ docker compose up -d

...

[+] Running 3/3
 ✔ Network end-point-blog-dotnet-docker-deploy_default            Created  0.0s
 ✔ Volume "end-point-blog-dotnet-docker-deploy_db-postgres-data"  Created  0.0s
 ✔ Container end-point-blog-dotnet-docker-deploy-db-1             Created  0.1s

Now that we have a database up and running, we have a couple of options for connecting to it. If we have the psql command line client installed in our machine, we can connect to it directly:

$ psql -h localhost -d vehicle_quotes -U vehicle_quotes -W
Password:
psql (16.3 (Ubuntu 16.3-0ubuntu0.24.04.1))
Type "help" for help.

vehicle_quotes=#

If not, we could first connect to the container, and then open psql from there:

$ docker compose exec db bash
root@edc038da3aa4:/# psql -h localhost -d vehicle_quotes -U vehicle_quotes -W
Password:
psql (16.3 (Debian 16.3-1.pgdg120+1))
Type "help" for help.

vehicle_quotes=#

Notice how we pass db and bash as parameters to the docker compose exec command. db is the name of our service, and bash is the command that we wish to execute within it. In this case we just want to open a shell, so we use bash.

Nice. That’s all it takes to set up the database. Now on to the applications.

Deploying the admin portal web app

Like I mentioned before, our code base contains an admin portal web application. In order to deploy it, we need first to define an image, which we’ll do with a Dockerfile, and then add the configuration to run a container based on that image using Docker Compose.

The image is a self-contained package that includes all the software that the application needs to run. It’s like an executable program. One that can be run by Docker, instead of directly by the operating system. In order to build images, we use Dockerfiles. For this ASP.NET app, the Dockerfile will perform two main tasks: Specify how to build the app and how to run it. Here’s what a Dockerfile for the admin portal project could look like:

# ./Dockerfile.AdminPortal

# Pull the .NET SDK image from Microsoft's repository. We will use it to build
# our app. We also give it a name of "build" so that we can reference it later.
FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build

# Now we copy all *.csproj files from the code base into the image and run
# dotnet restore. This will download and install all the NuGet packages that the
# various projects require.
WORKDIR /source

COPY source/vehicle-quotes.sln .
COPY source/VehicleQuotes.AdminPortal/VehicleQuotes.AdminPortal.csproj ./VehicleQuotes.AdminPortal/
COPY source/VehicleQuotes.Core/VehicleQuotes.Core.csproj ./VehicleQuotes.Core/
COPY source/VehicleQuotes.CreateUser/VehicleQuotes.CreateUser.csproj ./VehicleQuotes.CreateUser/
COPY source/VehicleQuotes.IntegrationTests/VehicleQuotes.IntegrationTests.csproj ./VehicleQuotes.IntegrationTests/
COPY source/VehicleQuotes.RazorTemplates/VehicleQuotes.RazorTemplates.csproj ./VehicleQuotes.RazorTemplates/
COPY source/VehicleQuotes.UnitTests/VehicleQuotes.UnitTests.csproj ./VehicleQuotes.UnitTests/
COPY source/VehicleQuotes.WebApi/VehicleQuotes.WebApi.csproj ./VehicleQuotes.WebApi/
RUN dotnet restore

# Next we copy all of the source code for all of the projects.
COPY source/VehicleQuotes.AdminPortal/. ./VehicleQuotes.AdminPortal/
COPY source/VehicleQuotes.Core/. ./VehicleQuotes.Core/
COPY source/VehicleQuotes.CreateUser/. ./VehicleQuotes.CreateUser/
COPY source/VehicleQuotes.IntegrationTests/. ./VehicleQuotes.IntegrationTests/
COPY source/VehicleQuotes.RazorTemplates/. ./VehicleQuotes.RazorTemplates/
COPY source/VehicleQuotes.UnitTests/. ./VehicleQuotes.UnitTests/
COPY source/VehicleQuotes.WebApi/. ./VehicleQuotes.WebApi/

# Now that we have everything in place, we use the dotnet publish command to
# build the VehicleQuotes.AdminPortal project.
WORKDIR /source/VehicleQuotes.AdminPortal
RUN dotnet publish -c release -o /app --no-restore

# In this last step, notice how we pull a different image here, this is the
# image that Microsoft recommends to use for running ASP.NET apps.
#
# Since by this point we already built the app, and have the executable binary
# for it, we don't need the full SDK anymore. dotnet/aspnet is a lighter weight
# image designed exclusively for running apps. In other words, it has the .NET
# runtime redistributable, not the full SDK.
#
# So here, all we do is copy the build assets from the "build" image into this
# new runtime-only one and put them in a /app directory. Then we call the .NET
# CLI to run the web app's DDL.
FROM mcr.microsoft.com/dotnet/aspnet:8.0
WORKDIR /app
COPY --from=build /app ./
ENTRYPOINT ["dotnet", "VehicleQuotes.AdminPortal.dll"]

With a Dockerfile like this, we could run the app as a stand-alone container. However, we don’t want to run it like that. Instead, we want it to be a service that’s deployable via Docker Compose, as part of a bigger ecosystem. In order to do that, we add the following to our compose.yaml file.

# ./compose.yaml

services:
  # The service is called "admin-portal" because that's the application that it
  # contains.
  admin-portal:
    # This is slightly more complex than the "db" service. Instead of using the
    # "image" option to download an off-the-shelf PostgreSQL image from
    # dockerhub, we use "build" and point to the Dockerfile we created in the
    # previous step. We also specify the current directory (i.e. the ".") as the
    # context in which the build will be performed.
    build:
      context: .
      dockerfile: Dockerfile.AdminPortal
    # We also configure this service to restart automatically if for whatever
    # reason it stops.
    restart: always
    # "8080" is the default port that ASP.NET Core 8 web applications listen to.
    # So here, we set it up so that all requests coming to the host machine to
    # port 8001 get sent to this container's port 8080. That way, reaching the
    # ASP.NET app that's running within.
    ports:
      - 8001:8080
    # This part is particular to our app. The admin portal offers the capability
    # of uploading files. Here, we're defining a Docker volume to ensure that
    # the uploaded files persist across container restarts. In this case, we are
    # linking the host machine's ./uploads directory to the container's
    # /app/wwwroot/uploads directory, which is where the uploaded files are
    # saved.
    volumes:
      - ./uploads:/app/wwwroot/uploads
    # We can use this section so set any environment variables that our app may
    # need. In the case of our admin portal here, we set the running environment
    # to Development, specify a database connection string, and the location to
    # save the uploaded files.
    environment:
      - ASPNETCORE_ENVIRONMENT=Development
      - ConnectionStrings__VehicleQuotesContext=Host=db;Database=vehicle_quotes;Username=vehicle_quotes;Password=password
      - QuoteImagesPath=/app/wwwroot/uploads
    # Finally, "entrypoint" specifies the command that should be used when
    # running the container. In this case, this is essentially the same as the
    # last line of the Dockerfile we saw above.
    entrypoint: ["sh", "-c", "dotnet VehicleQuotes.AdminPortal.dll"]

# ...

With this, we’re ready to see the application running. Run docker compose up -d again to update the infrastructure and include the new service:

$ docker compose up -d

...

[+] Running 2/2
 ✔ Container end-point-blog-dotnet-docker-deploy-db-1            Running   0.0s
 ✔ Container end-point-blog-dotnet-docker-deploy-admin-portal-1  Started   0.3s

After a while of downloading and building, navigating to http://localhost:8001 should show this:

Adding a maintenance container

At this point, we have a problem though. If we try clicking on the “Quotes” link in the top navigation bar, we see this:

We need to run the database migrations. In order to do that, we need an environment that has all of our source code and the full .NET SDK with the Entity Framework Core command line tool. That is, an environment that can build the app and run the .NET CLI. While we could install that in the host machine and run our migrations that way; we could also encapsulate it in a container. That way we get the benefits of portability, etc.

Of course, that maintenance container needs an image. And images are defined in Dockerfiles. Here’s a Dockerfile that would serve our purpose:

# ./Dockerfile.Maintenance

# Same as before, we pull the official image that contains the full .NET SDK.
FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build

# Here we install the psql command line client. A useful tool to have in a
# container meant for system maintenance.
RUN apt-get update && export DEBIAN_FRONTEND=noninteractive \
    && apt-get -y install --no-install-recommends postgresql-client

# We also install the dotnet-ef tool which allows us access to commands for
# managing the database and migrations.
RUN dotnet tool install dotnet-ef --global
ENV PATH="$PATH:/root/.dotnet/tools"

# Finally, we copy all of our source code.
WORKDIR /source

COPY source/vehicle-quotes.sln .
COPY source/VehicleQuotes.AdminPortal/VehicleQuotes.AdminPortal.csproj ./VehicleQuotes.AdminPortal/
COPY source/VehicleQuotes.Core/VehicleQuotes.Core.csproj ./VehicleQuotes.Core/
COPY source/VehicleQuotes.CreateUser/VehicleQuotes.CreateUser.csproj ./VehicleQuotes.CreateUser/
COPY source/VehicleQuotes.IntegrationTests/VehicleQuotes.IntegrationTests.csproj ./VehicleQuotes.IntegrationTests/
COPY source/VehicleQuotes.RazorTemplates/VehicleQuotes.RazorTemplates.csproj ./VehicleQuotes.RazorTemplates/
COPY source/VehicleQuotes.UnitTests/VehicleQuotes.UnitTests.csproj ./VehicleQuotes.UnitTests/
COPY source/VehicleQuotes.WebApi/VehicleQuotes.WebApi.csproj ./VehicleQuotes.WebApi/
RUN dotnet restore

COPY source/VehicleQuotes.AdminPortal/. ./VehicleQuotes.AdminPortal/
COPY source/VehicleQuotes.Core/. ./VehicleQuotes.Core/
COPY source/VehicleQuotes.CreateUser/. ./VehicleQuotes.CreateUser/
COPY source/VehicleQuotes.IntegrationTests/. ./VehicleQuotes.IntegrationTests/
COPY source/VehicleQuotes.RazorTemplates/. ./VehicleQuotes.RazorTemplates/
COPY source/VehicleQuotes.UnitTests/. ./VehicleQuotes.UnitTests/
COPY source/VehicleQuotes.WebApi/. ./VehicleQuotes.WebApi/

# No ENTRYPOINT here because there's no specific command to run.

This Dockerfile is very similar to the first portion of the one for the admin portal. It essentially creates a .NET development environment. Notice how it doesn’t include an ENTRYPOINT command. This is expected, as this container’s purpose is to allow us to connect to it to run any number of maintenance tasks on demand. There isn’t really any process that it needs to run upon start.

Now, we want to deploy the maintenance container when we do docker compose up -d. To that end, we add the following to the services section in our compose.yaml file:

# ./compose.yaml

services:

  # ...

  maintenance:
    # Similar to the admin-portal, we specify the context and Dockerfile to
    # build the image.
    build:
      context: .
      dockerfile: Dockerfile.Maintenance
    # The maintenance tasks will need to know where to find the database. So we
    # set the connection string environment variable.
    environment:
      - ConnectionStrings__VehicleQuotesContext=Host=db;Database=vehicle_quotes;Username=vehicle_quotes;Password=password
    # "sleep infinity" here makes sure the container doesn't close immediately
    # after starting up, which is the expected behavior when a container has no
    # start up command or has a command that just runs and ends. That is, as
    # opposed to a long running service, like a web app.
    command: sleep infinity

# ...

With these additions, running docker compose up -d again will result in the new maintenance container being created:

$ docker compose up -d

...

[+] Running 3/3
 ✔ Container end-point-blog-dotnet-docker-deploy-admin-portal-1  Running   0.0s
 ✔ Container end-point-blog-dotnet-docker-deploy-db-1            Running   0.0s
 ✔ Container end-point-blog-dotnet-docker-deploy-maintenance-1   Started   0.4s

Now we can finally connect to the brand new maintenance container:

$ docker compose exec maintenance bash
root@14613bcf1756:/source#

Then inspect the status of the migrations:

root@14613bcf1756:/source# dotnet ef migrations list -s ./VehicleQuotes.AdminPortal -p ./VehicleQuotes.Core
Build started...
Build succeeded.

...

20210625212939_AddLookupTables (Pending)
20210625224443_AddUniqueIndexesToLookupTables (Pending)
20210625232816_AddVehicleModelTables (Pending)
20210625234824_AddUniqueIndexesForVehicleModelTables (Pending)
20210627204444_AddQuoteRulesAndOverridesTables (Pending)
20210627213029_AddQuotesTable (Pending)
20210627230039_AddSeedDataForSizesAndBodyTypes (Pending)
20220530192346_FixDatetimeColumn (Pending)
20220605003253_AddIdentityTables (Pending)
20220609233914_AddUserApiKeysTable (Pending)
20240504211307_VariousNullabilityChanges (Pending)
20240606222539_AddQuoteImages (Pending)
root@14613bcf1756:/source#

And run them:

root@14613bcf1756:/source# dotnet ef database update -s ./VehicleQuotes.AdminPortal -p ./VehicleQuotes.Core
Build started...
Build succeeded.

...

Done.
root@14613bcf1756:/source#

With that done, we can now go to the browser again and bring up the http://localhost:8001/Quotes page. Which now looks like this:

An empty—but working—page for listing database records!

Deploying the web API

Now let’s look at how to deploy another of our system’s runtime components: the web API. The process of setting this up is nearly identical to the admin portal’s. After all, both are ASP.NET Core web applications. Really, the only difference is that one serves HTML and the other serves JSON. To the runtime, they are the same type of thing. And sure enough, here’s the web API’s Dockerfile:

# ./Dockerfile.WebApi

FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build

WORKDIR /source

# copy sln and csproj files and restore
COPY source/vehicle-quotes.sln .
COPY source/VehicleQuotes.AdminPortal/VehicleQuotes.AdminPortal.csproj ./VehicleQuotes.AdminPortal/
COPY source/VehicleQuotes.Core/VehicleQuotes.Core.csproj ./VehicleQuotes.Core/
COPY source/VehicleQuotes.CreateUser/VehicleQuotes.CreateUser.csproj ./VehicleQuotes.CreateUser/
COPY source/VehicleQuotes.IntegrationTests/VehicleQuotes.IntegrationTests.csproj ./VehicleQuotes.IntegrationTests/
COPY source/VehicleQuotes.RazorTemplates/VehicleQuotes.RazorTemplates.csproj ./VehicleQuotes.RazorTemplates/
COPY source/VehicleQuotes.UnitTests/VehicleQuotes.UnitTests.csproj ./VehicleQuotes.UnitTests/
COPY source/VehicleQuotes.WebApi/VehicleQuotes.WebApi.csproj ./VehicleQuotes.WebApi/
RUN dotnet restore

# copy everything else
COPY source/VehicleQuotes.AdminPortal/. ./VehicleQuotes.AdminPortal/
COPY source/VehicleQuotes.Core/. ./VehicleQuotes.Core/
COPY source/VehicleQuotes.CreateUser/. ./VehicleQuotes.CreateUser/
COPY source/VehicleQuotes.IntegrationTests/. ./VehicleQuotes.IntegrationTests/
COPY source/VehicleQuotes.RazorTemplates/. ./VehicleQuotes.RazorTemplates/
COPY source/VehicleQuotes.UnitTests/. ./VehicleQuotes.UnitTests/
COPY source/VehicleQuotes.WebApi/. ./VehicleQuotes.WebApi/

# build app
WORKDIR /source/VehicleQuotes.WebApi
RUN dotnet publish -c release -o /app --no-restore

# final image
FROM mcr.microsoft.com/dotnet/aspnet:8.0
WORKDIR /app
COPY --from=build /app ./
ENTRYPOINT ["dotnet", "VehicleQuotes.WebApi.dll"]

Very similar to the admin portal’s Dockerfile. The only differences are the sections for building and running the app towards the end of the file, on account of the directories and resulting DDL name being different. Other than that, they are pretty much identical.

Same story on the Docker Compose side of things:

# ./compose.yaml

services:

  #...

  web-api:
    build:
      context: .
      dockerfile: Dockerfile.WebApi
    restart: always
    # We want to expose this service in a different port than the admin portal,
    # we chose 8002.
    ports:
      - 8002:8080
    # A different set of environment variables, but the structure is the same.
    environment:
      - ASPNETCORE_ENVIRONMENT=Development
      - ConnectionStrings__VehicleQuotesContext=Host=db;Database=vehicle_quotes;Username=vehicle_quotes;Password=password
      - MailSettings__Server=SMTP_SERVER_URL
      - MailSettings__Port=2525
      - MailSettings__SenderName=Vehicle Quotes
      - MailSettings__SenderEmail=info@vehiclequotes.com
      - MailSettings__UserName=SMTP_SERVER_USER_NAME
      - MailSettings__Password=SMTP_SERVER_PASSWORD
    # The entrypoint command is different because the DLL has a different file
    # name.
    entrypoint: ["sh", "-c", "dotnet VehicleQuotes.WebApi.dll"]

#...

It has the same overall setup as the admin portal. This time we’ve chosen a different port, configured a different set of environment variables, and run the web API’s DLL. Other than that, it’s the same. Now docker compose up -d can be run again, and after a while you should see:

$ docker compose up -d

...

[+] Running 4/4
 ✔ Container end-point-blog-dotnet-docker-deploy-web-api-1       Started   0.4s
 ✔ Container end-point-blog-dotnet-docker-deploy-maintenance-1   Running   0.0s
 ✔ Container end-point-blog-dotnet-docker-deploy-db-1            Running   0.0s
 ✔ Container end-point-blog-dotnet-docker-deploy-admin-portal-1  Running   0.0s

The web API has a Swagger UI that should now be accessible at http://localhost:8002/swagger:

Excellent. We now have a very basic implementation of our deployment strategy. We have managed to build and run all the apps we need. We also have a special container with a full development environment that we can use to perform a number of maintenance tasks. However, there are various issues that we still need to address. Let’s do that next.

Looking at the logs

Before that, though, let’s look at the logs being produced by the applications that have been deployed. You can see the logs for the entire system with:

docker compose logs -f

That command produces a single log stream with messages from all the services. To look at the logs of a particular service, all we need to do is specify the service name. For example docker compose logs db -f or docker compose logs web-api -f.

That’s all there is to it as far as logs go. However, if we look at the logs for our current deployment, they reveal the first issue that we need to attend to…

Persisting data protection keys

If we look at the logs of the admin portal or web API containers, we see messages like this:

web-api-1       | warn: Microsoft.AspNetCore.DataProtection.Repositories.FileSystemXmlRepository[60]
web-api-1       |       Storing keys in a directory '/root/.aspnet/DataProtection-Keys' that may not be persisted outside of the container. Protected data will be unavailable when container is destroyed. For more information go to https://aka.ms/aspnet/dataprotectionwarning

As it turns out, the data protection subsystem in ASP.NET Core (which is used for cookies and the like) depends on the app being able to store and persist files. As the message says, this is problematic for containers. Because upon container destruction, which is a usual and expected part of a container’s lifecycle, the internal container’s file system gets wiped out. That is, unless we use Docker volumes to store the files that should persist beyond a particular container’s life. So, in order to get rid of these warnings, that’s exactly what we’re going to have to do.

First we create a data-protection-keys directory in our host machine. And inside it, we create one directory for each of the ASP.NET Core apps. It ends up looking like this:

data-protection-keys/
├── admin-portal
└── web-api

Next we have to make sure the apps in the containers have access to these directories. In order to do that, we need to configure the compose.yaml so that it creates new Docker volumes that are linked to those locations:

# ./compose.yaml

services:
  admin-portal:
    #...
    volumes:
      - ./data-protection-keys/admin-portal:/data-protection-keys/admin-portal
    #...

  web-api:
    #...
    volumes:
      - ./data-protection-keys/web-api:/data-protection-keys/web-api
    #...
#...

This means that, for example, whenever an app in the container references /data-protection-keys/admin-portal; it will actually be accessing the ./data-protection-keys/admin-portal directory in the host machine. Same deal for the web-api one.

We also need to configure the applications themselves so that they store their data protection keys in the locations that we’ve created. For that, we add environment variables for the containers to set the paths:

# ./compose.yaml

services:
  admin-portal:
    #...
    environment:
      - AdminPortalDataProtectionKeysPath=/data-protection-keys/admin-portal
    #...

  web-api:
    #...
    environment:
      - WebApiDataProtectionKeysPath=/data-protection-keys/web-api
    #...
#...

And add the following code to each of the apps’ Program.cs files:

// ./source/VehicleQuotes.AdminPortal/Program.cs

builder.Services.AddDataProtection().PersistKeysToFileSystem(
    new DirectoryInfo(
        builder.Configuration["AdminPortalDataProtectionKeysPath"] ??
            throw new InvalidOperationException("Config setting 'AdminPortalDataProtectionKeysPath' not found.")
    )
);

// ./source/VehicleQuotes.WebApi/Program.cs

builder.Services.AddDataProtection().PersistKeysToFileSystem(
    new DirectoryInfo(
        builder.Configuration["WebApiDataProtectionKeysPath"] ??
            throw new InvalidOperationException("Config setting 'WebApiDataProtectionKeysPath' not found.")
    )
);

There isn’t much to comment about the code really. It’s just some .NET boilerplate to configure that specific detail of the data protection services. It also makes sure that the values are always present. Errors are raised if not.

Now that everything is wired up like that, we can hit docker compose up -d --build and the warning message should be gone if we look at the logs.

Notice how we used the --build option this time. That tells Docker that it needs to rebuild the images from scratch. Image rebuild means running through the Dockerfiles again. And that means running dotnet build again. In short, we have to do this to make sure that the code changes that we made are included in the new builds.

Storing sensitive information with secrets

Another issue that we need to address is how to handle sensitive information like passwords in our config files. So far we’ve been putting them in plain text in compose.yaml. The problem with this is that this file is meant to be pushed to version control and we don’t want passwords in there. The spread of sensitive information like that should be more controlled. Ideally, we’d store them in files that never leave the server where the system is deployed.

Docker Compose has a feature that works just like that. Through Docker Compose secrets, we can create text files outside of the compose.yaml and put the database password and connection string in them. These files will live only in the server, never uploaded to version control.

Let’s create a new secrets directory and create these two files within it:

./secrets/vehicle-quotes-db-connection-string.txt:

Host=db;Database=vehicle_quotes;Username=vehicle_quotes;Password=password

./secrets/vehicle-quotes-db-password.txt:
```
password
```

Ending up looking like this:

secrets/
├── vehicle-quotes-db-connection-string.txt
└── vehicle-quotes-db-password.txt

Now, at the bottom of compose.yaml, we add a new secrets section:

# ./compose.yaml

#...

secrets:
  vehicle-quotes-db-password:
    file: ./secrets/vehicle-quotes-db-password.txt
  vehicle-quotes-db-connection-string:
    file: ./secrets/vehicle-quotes-db-connection-string.txt

Then we include the secrets in the services that need them:

# ./compose.yaml

services:
  admin-portal:
    #...
    secrets:
      - vehicle-quotes-db-connection-string
    #...

  web-api:
    #...
    secrets:
      - vehicle-quotes-db-connection-string
    #...

  db:
    #...
    secrets:
      - vehicle-quotes-db-password
    #...

  maintenance:
    #...
    secrets:
      - vehicle-quotes-db-connection-string
    #...
#...

With this, Docker Compose will add files in the resulting containers under the /run/secrets/ directory with the contents of their referenced secrets. So, for example, in the case of the admin-portal container, a /run/secrets/vehicle-quotes-db-connection-string file will be created in the container’s internal file system, with the same contents as the ./secrets/vehicle-quotes-db-connection-string.txt file. Similar thing for the others.

Now that the secrets are materialized as files within the containers, let’s see how we put them to use.

For the admin-portal container, we injected the vehicle-quotes-db-connection-string secret into it. This file contains the database connection string. And we need to pass that to the running app via an environment variable. In order to do so, we can change the entrypoint command to this:

# ./compose.yaml

services:
  admin-portal:
  #...
  entrypoint: [
    "sh", "-c",
    "export ConnectionStrings__VehicleQuotesContext=$(cat /run/secrets/vehicle-quotes-db-connection-string) &&
    dotnet VehicleQuotes.AdminPortal.dll"
  ]

#...

We’ve modified the command to set the ConnectionStrings__VehicleQuotesContext to the contents of the /run/secrets/vehicle-quotes-db-connection-string file. The export command defines the environment variable for this particular command; and the $(cat ...) part returns the contents of the file.

We should also remove the ConnectionStrings__VehicleQuotesContext variable from the admin-portal service’s environment section.

For the web-api service, we do the same thing. The only difference is that its entrypoint command calls for VehicleQuotes.WebApi.dll instead of VehicleQuotes.AdminPortal.dll.

For the db service, the setup is a little different. A little simpler in fact. The official PostgreSQL image has a shortcut for specifying the database user password via Docker Compose secrets. All we need to do is define this new POSTGRES_PASSWORD_FILE environment variable and remove the POSTGRES_PASSWORD one:

# ./compose.yaml

services:
  #...

  db:
  #...
  environment:
    - POSTGRES_PASSWORD_FILE=/run/secrets/vehicle-quotes-db-password
    # - POSTGRES_PASSWORD=password     <-- remove this one
#...

Try docker compose up -d again and test the apps. Everything should still work well.

In the maintenance container, we also have to remove the ConnectionStrings__VehicleQuotesContext environment variable. That unfortunately means that the connection string will no longer be automatically available for us to run database related tasks. Just like other containers, it will be in a /run/secrets/vehicle-quotes-db-connection-string file. So, whenever we want to interact with the database, like when running migrations, we need to manually export the variable. Something like this:

$ docker compose exec maintenance bash
root@2732a06871c0:/source# export ConnectionStrings__VehicleQuotesContext=$(cat /run/secrets/vehicle-quotes-db-connection-string)
root@2732a06871c0:/source# dotnet ef migrations list -s ./VehicleQuotes.AdminPortal -p ./VehicleQuotes.Core
Build started...
Build succeeded.

...

Parameterizing `compose.yaml` to support multiple deployment environments

A common requirement when deploying applications is to be able to do so in multiple environments. There’s generally a “live” or “production” environment where the system runs and end users access it. There can also be others: staging, test, development, etc. Ideally, we’d use the same set of Docker and Compose files, with slight changes, in order to deploy variants of the system depending on the environment.

Some settings like ports, passwords, or SMTP credentials are the types of things that usually vary per environment. Luckily for us, Docker Compose supports .env files that can be used to parameterize certain aspects of the compose.yaml file.

We can extract the values that vary from our compose.yaml file, and put them in a separate .env file that looks like this:

ADMIN_PORTAL_PORT=8001
WEB_API_PORT=8002
DB_PORT=5432
POSTGRES_DB=vehicle_quotes
POSTGRES_USER=vehicle_quotes
MailSettings__Server=SMTP_SERVER_URL
MailSettings__Port=2525
MailSettings__SenderName=Vehicle Quotes
MailSettings__SenderEmail=info@vehiclequotes.com
MailSettings__UserName=SMTP_SERVER_USER_NAME
MailSettings__Password=SMTP_SERVER_PASSWORD

Here we have the ports that we want to expose the various services on, the database name and user name, and some email delivery settings. This file wont be pushed to version control, and each deployment will have its own version of it.

The compose.yaml file can reference the values defined in the .env file using the following syntax: ${VAR_NAME}. Here’s how we change our compose.yaml to take advantage of the settings defined in the .env file:

# ./compose.yaml

services:
  admin-portal:
    #...
    ports:
      - ${ADMIN_PORTAL_PORT}:8080
    #...

  web-api:
    #...
    ports:
      - ${WEB_API_PORT}:8080
    #...
    environment:
      #...
      - MailSettings__Server=${MAIL_SETTINGS_SERVER}
      - MailSettings__Port=${MAIL_SETTINGS_PORT}
      - MailSettings__SenderName=${MAIL_SETTINGS_SENDER_NAME}
      - MailSettings__SenderEmail=${MAIL_SETTINGS_SENDER_EMAIL}
      - MailSettings__UserName=${MAIL_SETTINGS_USER_NAME}
      - MailSettings__Password=${MAIL_SETTINGS_PASSWORD}
    #...

  db:
    #...
    ports:
      - ${DB_PORT}:5432
    #...
    environment:
      - POSTGRES_DB=${POSTGRES_DB}
      - POSTGRES_USER=${POSTGRES_USER}
      #...
    #...

#...

Once again, we can run docker compose up -d and everything should be fine.

Waiting for the database to be ready

Sometimes some services need to wait for others to come online before they can start up. A common scenario is to wait for the database to be ready before running apps that depend on it. We can do that in Docker Compose thanks to the depends_on and healthcheck settings. We can update our compose.yaml so that the admin portal and web API services only start after the database is up and running and ready to receive requests. Here’s how:

# ./compose.yaml

services:
  admin-portal:
    #...
    # This specifies that the admin-portal service depends on the db to be
    # healthy in order to start up. The db service's healthcheck setting is how
    # it determines whether it is healthy or not.
    depends_on:
      db:
        condition: service_healthy

  web-api:
    #...
    # Exact same setting as the one in admin-portal.
    depends_on:
      db:
        condition: service_healthy

  db:
    #...
    # Here we configure this service to offer a healthcheck that other services
    # can use to determine if it's ready to be depended upon. In this case, we
    # leverage PostgreSQL's pg_isready tool. It is called by the specified
    # timeout, at the specified interval, and with as many retries as specified
    # in the settings below.
    healthcheck:
      test: "pg_isready -U ${POSTGRES_USER}"
      interval: 10s
      timeout: 5s
      retries: 5

#...

The most interesting part is the healthcheck setting in the db service which leverages a PostgreSQL-specific tool to check whether the database is ready. Other software will have other methods to do checks like this, but PostgreSQL’s is thankfully pretty straightforward.

Serving the apps with NGINX

Another common pattern for serving web applications is to use NGINX as a reverse proxy that funnels HTTP traffic coming from the internet into the application. As you may have noticed, we haven’t talked about HTTPS so far. This aspect is something that can be elegantly handled by NGINX as well. In this last section we see how we can set up NGINX to expose our apps to the world.

First, we need to create some custom items in the default NGINX configuration file. Its default location will depend on the OS version and flavor you installed NGINX on, but for Rocky Linux 9, it usually lives in /etc/nginx/nginx.conf. First, we need to declare our upstream servers that will point to the ports where the web API and admin portals are listening:

upstream admin_portal {
    server localhost:8001;
}

upstream web_api {
    server localhost:8002;
}

Then, we will add a server that listens in the standard port 80 and proxies the /admin and /api URLs to the admin_portal and web_api upstream servers respectively:

server {
    listen 80;
    server_name vehiclequotes.com;

    location /admin {
        proxy_pass http://admin_portal;
        proxy_http_version 1.1;
        proxy_set_header Connection keep-alive;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    location /api {
        proxy_pass http://web_api;
        proxy_http_version 1.1;
        proxy_set_header Connection keep-alive;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Those are the basic settings needed to serve our applications in a single domain through different locations. This allows having a live frontend application that will eventually consume our API endpoints that lives in the same server and domain, avoiding possible Cross-Origin Resource Sharing (CORS) issues when establishing the workflow between the different web applications.

If we also have an SSL certificate that we want to use to securely serve our apps, we can use a free service such as Let’s Encrypt to create it. Once we’re in possession of the certificate files and placed them on the server, we need to perform a few extra tweaks to our nginx.conf file.

First, let’s make our server listen on port 443 (HTTPS), and point to our .cer and .key files in the updated entry:

server {
    listen 443 ssl;
    server_name vehiclequotes.com;
    ssl_certificate /etc/certs/live/vehiclequotes.com/fullchain.cer;
    ssl_certificate_key /etc/certs/live/vehiclequotes.com/vehiclequotes.com.key;

    location /admin {
        proxy_pass http://admin_portal;
        proxy_http_version 1.1;
        proxy_set_header Connection keep-alive;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    location /api {
        proxy_pass http://web_api;
        proxy_http_version 1.1;
        proxy_set_header Connection keep-alive;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Second, let’s add a new server for port 80 (non-HTTPS) that will redirect permanently to our new secure location:

server {
    listen 80;
    server_name vehiclequotes.com;
    return 301 https://$server_name$request_uri;
}

Finally, let’s restart the NGINX service to apply our changes. The command that restarts the service will vary depending on the OS we’re running on the server. For Rocky Linux 9, we can do that by running the command sudo systemctl restart nginx.

You can find the resulting nginx.conf file in the project’s repo in GitHub.

That’s all for now

And that’s it! In this article, we’ve seen how we can approach deploying a .NET system into production using Docker Compose.

We saw how to organize the code and configuration files using Git submodules. We addressed a few important edge cases and gotchas like properly configuring Data Protection keys, having a container for performing maintenance tasks, ensuring certain files persist across restarts, and bringing up services in a certain order via depends_on settings.

We even saw how to allow our web applications to be accessible to the outside world with NGINX through a set of reverse proxying rules, and as a bonus, to be securely served with SSL.

Secure Your Dockerized Nginx with Let's Encrypt SSL Certificates

2024-06-27T00:00:00+00:00

Photo by Animesh Srivastava from Pexels.

In this tutorial I will demonstrate how to secure Nginx on Docker using HTTPS, leveraging free certificates from Let’s Encrypt. Let’s Encrypt certificates provide trusted and secure encryption at no cost, although they require renewal every 90 days. Fortunately, this renewal process can be automated with various tools. We will use acme.sh, a versatile Bash script compatible with major platforms. The tutorial will guide you through obtaining Let’s Encrypt certificates on the host system and mounting them as a volume in the Nginx container. Please ensure the following prerequisites are met before proceeding:

Working Docker Engine
Working domain name
A host with ports 80 and 443 that is accessible from the internet

1. Domain validation

First, we need an Nginx instance on Docker that will expose port 80 and have a directory on the host mounted for its web root. This is required by acme.sh for its file-based domain validation. I’ve prepared a Docker Compose file (docker-compose.yml) and an Nginx configuration file (nginx.conf) for this purpose. Git clone the following repository and change into the directory

git clone https://github.com/aburayyanjeffry/nginx-docker-acme.git
cd nginx-docker-acme

In nginx.conf, please note that the lines exposing port 443 and adding SSL certificates are commented because we don’t have the certificate yet. Listening on port 444 and trying to enable SSL without a valid certificate will cause errors and prevent Nginx from starting up. Port 443 is also commented out in docker-compose.yml because Nginx is not exposing port 443 yet, as is the line copying the ssh folder. Let’s start Nginx using Docker Compose:

docker compose up -d

2. Get the acme.sh

The following command will install acme.sh in the current user’s home directory. Make sure to use your email in the command.

curl https://get.acme.sh | sh -s email=jeffry@email.com

Log out and log in again to enable the acme.sh alias for the user. If acme.sh is not working, it’s probably because you missed this step. If the alias is not enabled, the acme.sh script is not defined.

3. Set the CA

Set Let’s Encrypt as the default Certificate Authority.

acme.sh --set-default-ca --server letsencrypt

4. Issue the certificate

Now we’ll proceed with issuing the certificate, a step that involves domain validation. Upon successful validation, the certificate will be issued.

acme.sh --issue -d jeffry.temphost.net -w /home/jeffry/nginx-docker-acme/nginx/html --keylength 4096

Make sure to replace jeffry.temphost.net with your domain and /home/jeffry/nginx-docker-acme/nginx/html with your web root directory. Note that /home/jeffry is the directory where the code was downloaded, making it the working directory. Be sure to update it to reflect your own working directory.

The -d flag specifies the domain, while -w designates the web root directory. This directory will be mounted as Nginx’s web root in Docker, where acme.sh will write the validation file.

We need to know the container name in order to restart it. The container name is the string in the last column from the docker ps output. In this example the container name is nginx-docker-acme-web-1.

[jeffry@docker ~]$ docker ps
CONTAINER ID   IMAGE     COMMAND                  CREATED          STATUS          PORTS                               NAMES
754c055d5b5e   nginx     "/docker-entrypoint...."   16 minutes ago   Up 16 minutes   0.0.0.0:80->80/tcp, :::80->80/tcp   nginx-docker-acme-web-1

5. Install the certificate

Uncomment the port 443 and SSL lines in nginx.conf and docker-compose.yaml. This will enable port 443 for Nginx and will make Docker expose it to the host after a restart it through Docker Compose later.

docker-compose.yaml

services:
  web:
    image: nginx
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/conf.d/default.conf
      - ./nginx/html:/usr/share/nginx/html
      - ./nginx/ssl:/etc/nginx/ssl

nginx/nginx.conf

server {
    listen 80;
    listen 443 ssl;

    server_name localhost;

    ssl_certificate /etc/nginx/ssl/cert.pem;
    ssl_certificate_key /etc/nginx/ssl/key.pem;

    location / {
        root /usr/share/nginx/html;
        index index.html;
    }
}

Create the ssl directory for Nginx and install the certificates there. This directory will be mounted by Nginx in Docker. Use the container name obtained from the previous steps as the value for the --reloadcmd switch. This is crucial because it will be used to restart Nginx when certificates are updated.

mkdir /home/jeffry/nginx-docker-acme/nginx/ssl
acme.sh --install-cert -d jeffry.temphost.net --key-file /home/jeffry/nginx-docker-acme/nginx/ssl/key.pem --fullchain-file /home/jeffry/nginx-docker-acme/nginx/ssl/cert.pem --reloadcmd "docker exec nginx-docker-acme-web-1 nginx -s reload"

6. Restart the Nginx container

Refresh the Nginx container by stopping and starting it from Docker Compose. You should be able to access Nginx over HTTPS with the Let’s Encrypt certificates.

docker compose down
docker compose up -d

7. Certificate Renewal Mechanism

The certificate will be automatically renewed by the cron job which was installed by acme.sh. You can verify the cron job by running crontab -l .

[jeffry@docker ~]$ crontab -l
13 7 * * * "/home/jeffry/.acme.sh"/acme.sh --cron --home "/home/jeffry/.acme.sh" > /dev/null

This ensures that the renewal process runs regularly and without manual intervention.

Setting up Let’s Encrypt SSL certificates for Nginx in a Docker environment using acme.sh is an easy process that enhances the security of your web applications. By leveraging acme.sh, you automate the certificate issuance and renewal process, ensuring your sites remain secure without manual intervention. Following the steps outlined in this tutorial, you now have a robust setup where Nginx serves your applications over HTTPS, backed by trusted SSL certificates from Let’s Encrypt. Thank you for following this tutorial. Have a great day!

Testing to defend against nginx add_header surprises

2020-05-29T00:00:00+00:00

These days when hosting websites it is common to configure the web server to send several HTTP response headers with every single request for security purposes.

For example, using the nginx web server we may add these directives to our http configuration scope to apply to everything served, or to specific server configuration scopes to apply only to particular websites we serve:

add_header Strict-Transport-Security max-age=2592000 always;
add_header X-Content-Type-Options    nosniff         always;

(See HTTP Strict Transport Security and X-Content-Type-Options at MDN for details about these two particular headers.)

The surprise (problem)

Once upon a time I ran into a case where nginx usually added the expected HTTP response headers, but later appeared to be inconsistent and sometimes did not. This is distressing!

Troubleshooting leads to the (re-)discovery that add_header directives are not always additive throughout the configuration as one would expect, and as every other server I can think of typically does.

If you define your add_header directives in the http block and then use an add_header directive in a server block, those from the http block will disappear.

If you define some add_header directives in the server block and then add another add_header directive in a location block, those from the http and/or server blocks will disappear.

This is even the case in an if block.

In the nginx add_header documentation we find the reason for the behavior explained:

There could be several add_header directives. These directives are inherited from the previous level if and only if there are no add_header directives defined on the current level.

This nginx directive has always behaved this way. Various people have warned about it in blog posts and online discussions for many years. But the situation remains the same, a trap for the unwary.

I have tried to imagine the rationale behind this behavior. Response headers often are set in groups, so the programmer who created this feature may have decided that any new scope’s add_header directives should start with a clean slate, unaffected by those set elsewhere. Hmm. The need for exclusive grouping of response headers is rare in my experience, and adding headers to the existing stack of tentative response headers is far more commonly what I want.

So while this behavior may make sense somewhere, it has not ever done so for me or anyone I have talked to about it. For us it is simply misbehavior, silent and easy to overlook when making later seemingly unrelated configuration adjustments.

Dangers

It often has security implications when headers you thought were being added to every response are not. Consider more fine-tuned and consequential security-related headers such as Content-Security-Policy, Vary for cache object separation, CORS headers Access-Control-*, etc.

Headers such as these are especially important when they need to be added based on logic spread across various configuration blocks, and that is exactly when nginx add_headers doesn’t work as expected.

Another pitfall is omitting the always option to add_header. Without that, the header will only be added to success responses (2XX and 3XX, but see the docs for specifics). We usually want security-related headers to be added even to 4XX and 5XX error responses.

Workaround using include

My first instinct was to work around the problems caused by this behavior by putting the standard add_header list in a file that I include everywhere. In some cases that works.

But despite the nginx include documentation saying that directive is allowed in “Context: any”, include is not allowed in an if block and will result in the fatal startup error:

“include” directive is not allowed here

So the only recourse in those cases is to repeat all needed add_header directives in every if block that uses add_header. Gross.

Repeating configuration manually means almost surely having the add_header directives in different configuration areas drift over time. So if we have to repeat ourselves, at least let’s do it with automation, such as by using configuration templating and preprocessing.

That is what I have most recently done. And we can still use native nginx include directives everywhere those are allowed.

nginx Headers More module

Many people have run into exactly this problem, and some of them developed a separate nginx module ngx_headers_more to solve most of these problems.

By using its more_set_headers directive, you get the expected additive behavior with previously-declared headers, regardless of the block scope:

Directives inherited from an upper level scope (say, http block or server blocks) are executed before the directives in the location block.

Note that although more_set_headers is allowed in location if blocks, it is not allowed in the server if blocks …

Fortunately I have not needed to use this in an if block in the server scope, so that one remaining limitation doesn’t pose a problem for me.

It also has options to set a header only for responses of a certain HTTP content type or status code.

The more_clear_headers directive allows the * wildcard for clearing all headers with the same prefix at once, such as Access-Control-*.

Installing ngx_headers_more

Because “Headers More” is a separate module, not part of standard nginx, it is not usually available without some extra work.

You can build it from source and install it manually, but of course that isn’t good to do on a production machine since it won’t get updated on its own.

You can use the OpenResty server built around nginx, which “Headers More” is part of. But you may not want all of that if you’re not writing a Lua web application.

Many Linux distributions and 3rd-party package repositories have prebuilt packages for “Headers More” which you can use:

Alpine
- nginx-mod-http-headers-more
Debian & Ubuntu
- nginx-extras
- libnginx-mod-http-headers-more-filter
RHEL/CentOS
- GetPageSpeed & Webtatic repos nginx-module-headers-more
- Aeris repo nginx-more

Search the excellent pkgs.org to find what you need if it isn’t already available through your package manager.

Apache

Apache httpd is still alive and well — actually better than ever. So depending on your situation, you may want to use that instead.

Apache’s Header directive has intuitive (to me) default behavior for setting response headers across the whole configuration, and many ways to deal with a possibly already-existing header:

add another header, or set exclusively (replace), or set only if this header doesn’t already exist
append to or merge into an existing header (for headers that accept multiple values)
edit an existing header with a regular expression search-and-replace
unset a header if one was previously set

I don’t know a way to have Apache clear a group of headers with a wildcard, or all headers at once, so they need to be individually cleared by name if that’s what you want.

Доверяй, но проверяй (Trust, but verify)

nginx was written by Igor Sysoev. Despite my disagreement with this one feature’s behavior, overall I find that nginx is excellent. Because of its open source release, excellent performance, and wide use, it has provided much-needed competition to Apache and Microsoft IIS. Thank you, Igor and all other contributors!

In the relevant spirit, since Igor is Russian, I close with the Russian proverb Доверяй, но проверяй: Trust, but verify.

Let us code (and configure) defensively, yet also test to avoid being surprised by missing headers.

We can manually test various HTTP responses are as we expect using curl -v or other HTTP clients to exercise various requests.

Even better, we can add to our automated test suite to confirm these HTTP response headers appear everywhere we expect, for static files and API endpoints backed by different application servers, and for various success and error responses.

Here is a test adapted from one I put together for one of our clients. It uses JavaScript in Node.js, the Jest test framework, and the Axios HTTP client. It ensures the security headers example I showed at the beginning of this article keeps working, even as we make nginx configuration changes over time:

const axios = require('axios');

const http = axios.create({
  baseURL: 'https://your.dom.ain',
});

describe('Check security headers', () => {
  const verifs = [
    { header: 'strict-transport-security', expect: (x) => x.toMatch(/max-age=\d{3,}/) },
    { header: 'x-content-type-options',    expect: (x) => x.toEqual('nosniff')        },
  ];

  const locs = [
    { path: '/robots.txt',                status: 200 },  // static
    { path: '/feed/endpoint/of/interest', status: 200 },  // API backend in PHP
    { path: '/api/other/auth/endpoint',   status: 403 },  // API backend in Perl
    { path: '/never/gonna/give/you/up!',  status: 404 },
    { path: '/api/dies/for/testing',      status: 500 },
  ];

  // throw no exceptions for non-success HTTP response status
  const conf = { validateStatus: () => true };

  for (const l of locs) {
    test(`${l.status} ${l.path}`, async () => {
      const res = await http.get(l.path, conf);
      expect(res.status).toBe(l.status);
      for (const v of verifs) {
        v.expect(expect(res.headers[v.header]));
      }
    });
  }
});

Here I run just this one test rather than the whole suite:

% jest -w 6 ./__tests__/webserver/security-headers.test.js
Determining test suites to run...
testing on https://https://your.dom.ain

 PASS  webserver/security-headers.test.js
  Check security headers
    ✓ 200 /robots.txt (55ms)
    ✓ 200 /feed/endpoint/of/interest (408ms)
    ✓ 403 /api/other/auth/endpoint (18ms)
    ✓ 404 /never/gonna/give/you/up! (6ms)
    ✓ 500 /api/dies/for/testing (12ms)

Test Suites: 1 passed, 1 total
Tests:       5 passed, 5 total
Snapshots:   0 total
Time:        2.721s, estimated 3s
Ran all test suites matching /.\/__tests__\/webserver\/security-headers.test.js/i.

This can also be extended to ensure that certain headers do not exist, or do not contain details that you do not want exposed:

the Server header should not reveal the nginx (see server_tokens) or Apache (see ServerTokens) version numbers
the X-Powered-By header should be absent, not exposing the fact that you are using PHP, and the version number — see the expose_php directive for php.ini
or with the Java Wildfly server, both of those headers are sent by default! — see instructions on how to omit them by editing XML or using jboss-cli

Add to the verifs array in the code above:

    { header: 'server',                    expect: (x) => x.not.toMatch(/\d/)         },
    { header: 'x-powered-by',              expect: (x) => x.toBeUndefined()           },

Now if (when) I forget about the nginx add_headers behavior, make changes, and inadvertently break things? Instead of it being unnoticed, my test suite will alert me so I can fix it before it goes into production!

Instant TLS Upgrades Through Proxy Magic!

2018-06-14T00:00:00+00:00

TLS shutdowns are real

The payment gateways have been warning for years about the impending and required TLS updates. Authorize.net and PayPal—to name a few—have stopped accepting transaction requests from servers using TLS 1.0. Despite the many warnings about this (and many delays in the final enforcement date), some projects are affected by this and payments are coming to a stop, customers cannot checkout, and e-commerce is at a standstill.

Ideally, getting to security compliance would include a larger migration to update your underlying operating system and your application. But a migration and software update can be an expensive project and in some cases, the business can’t wait weeks while this is done.

End Point has worked with several clients recently to try to remedy the situation by using a reverse proxy to fix this and we’ve had good success on getting payments flowing again.

What is a proxy?

A proxy is a mid-point, essentially a digital middleman, moving your data from one place to another. In two recent client instances, we ended up using nginx (the stack’s webserver) as the reverse proxy, basically running a separate server for just shuttling requests to/from the payment gateway. Since we want to be able to run the gateway in both live and test modes, we use two separate server definitions in our nginx include, one for each.

Since the proxy is talking to the gateway in TLS 1.2 the payment gateway is happy. Since the application can talk http to the proxy running on the same machine, your application is happy. Since payments are now flowing, the business is happy.

Why use a proxy?

While we always impress on clients the importance of staying up-to-date with their entire stack (operating system, language, application frameworks), this is not always practical for some sites, whether for cost reasons or some technical limitations which keep them on a specific library or framework version. In our case, these clients had been migrated to CentOS 7 already (which supports TLS 1.2), but the versions of Ruby and Ruby on Rails they were on were too old to be able to use the TLS 1.2 libraries built-in.

This can then be a fast way to get payments flowing again, ideally the first step on updating the site to work with a modern stack. This can also minimize development costs compared to doing a full migration or complete stack upgrade.

Steps

Run on an operating system version that supports TLS 1.2 natively. In practice for us, this is CentOS 7, but modern Debian, Ubuntu, and other Linux distributions would work.
Locate the existing payment gateway URLs for live and test modes. For Authorize.net, this differs depending on if you are connecting to their legacy systems or their modern systems.
Set up and configure the proxy. We used nginx for this; a later section covers the sample configs with explanations. We created a config file that we can just drop in an /etc/nginx/sites/ directory to be able to quickly set up and deploy the reverse proxies.
Adjust the configuration of the software to point to alternate payment gateway URLs. If you are using Rails’ ActiveMerchant, this can be accomplished via an application-level configuration override. We provide an example of the overriding of the live/test mode URLs for a Rails application later.

nginx sample config

# apitest.authorize.net

server {
    listen 127.0.0.1:1337 default_server;

    resolver 8.8.8.8;

    access_log /var/log/nginx/sites/authnet-proxies/access_log;
    error_log /var/log/nginx/sites/authnet-proxies/error_log warn;

    location / {
        proxy_pass_header Authorization;
        proxy_pass https://apitest.authorize.net$request_uri;
        proxy_set_header Host apitest.authorize.net;
        proxy_redirect off;
    }
}

# api.authorize.net

server {
    listen 127.0.0.1:1338 default_server;

    resolver 8.8.8.8;

    access_log /var/log/nginx/sites/authnet-proxies/access_log;
    error_log /var/log/nginx/sites/authnet-proxies/error_log warn;

    location / {
        proxy_pass_header Authorization;
        proxy_pass https://api.authorize.net$request_uri;
        proxy_set_header Host api.authorize.net;
        proxy_redirect off;
    }
}

In this file, we are defining a separate server block for each service (test mode and production), choosing an arbitrarily defined TCP port. Because nginx is speaking TLS 1.2 to the upstream gateway and we are speaking plain HTTP only to localhost and not storing any information regarding the request, we are able to fulfill the PCI DSS requirements.

A few notes about this:

We needed the resolver line for this to work given our configuration/setup because we are referring to the upstream servers by domain name instead of IP addresses; we just used the easily-memorable Google public DNS resolver servers for this purpose.
We need the Authorization header passed through the proxy, as this is what contains the site credentials when making a request from ActiveMerchant.
We need the Host header defined explicitly in the server block, as we want to override the localhost value that will be sent by the application due to application configuration. Authorize.net was (understandably) very picky about this being provided and correct.
The port numbers here are arbitrary and just need to match what we configure the application to connect to; one for the test server and one for production.
This setup will work for multiple development environments running on the same server as long as they are configured to point to the correct test/live server. There is nothing inherently insecure about doing this, as no authorization is shared, it just bounces the connection.

Rails sample application config

A configuration file for the Authorize.net CIM gateway, for clients using ActiveMerchant as the base for payment gateway integration:

require File.expand_path('../boot', __FILE__)
require 'rails/all'

module MyApp
  class Application < Rails::Application
    ActiveMerchant::Billing::AuthorizeNetCimGateway.test_url = 'http://localhost:1337/xml/v1/request.api'
    ActiveMerchant::Billing::AuthorizeNetCimGateway.live_url = 'http://localhost:1338/xml/v1/request.api'
  end
end

And another one for the legacy Authorize.net gateway:

require File.expand_path('../boot', __FILE__)
require 'rails/all'

module MyApp
  class Application < Rails::Application
    ActiveMerchant::Billing::AuthorizeNetGateway.test_url = 'http://localhost:1337/gateway/transaction.dll'
    ActiveMerchant::Billing::AuthorizeNetGateway.live_url = 'http://localhost:1338/gateway/transaction.dll'
  end
end

Ensure that you override the URLs depending on the gateway setup that your merchant account is set up for. Also make sure that your localhost URLs are using the ports corresponding to the server blocks set up in nginx to make sure these are proxying to the correct server.

While this example is using Authorize.net, you could use a similar setup for any payment gateway which allows you to set the URLs.

Summary

Hopefully this gives you some ideas about how you can use reverse proxies to upgrade the outgoing connection strength. If you need assistance getting something like this set up, feel free to contact us for help.

(Co-authored by Elizabeth Garrett Christensen.)

From Zero to HTTPS in an afternoon

2017-11-20T00:00:00+00:00

I’ve been hosting my own humble personal web site since 2012. I had never bothered setting up HTTPS for my domain, but after hearing about the Let’s Encrypt project, I was completely out of excuses.

For the unfamiliar, Let’s Encrypt offers free and fully automatic HTTPS certificates. The web cares about HTTPS now more than ever. Deeply interactive interfaces like geolocation and user media (camera, microphone) are too sensitive to trust an insecure transport. By leveraging the security features present in modern browsers, users can expect a reasonable safety from attacks that would exploit the weaknesses of HTTP.

To take the security mission even further, I decided to completely containerize my server and expose only a couple of ports. Using a Docker composition made it very easy to deploy up-to-date nginx and keep it isolated from the rest of my host shard.

The first mission was to set up certificates with certbot, the EFF’s free certificate tool. certbot has a plugin that writes nginx configuration for you, but in this case I didn’t want nginx installed on my host at all. Instead of following the nginx-specific instructions for my platform, I opted for the webroot plugin to just give me a certificate and let me figure out how to set it up. A certbot invocation and a few seconds later I have certificates for my site in /etc/letsencrypt/live/www.mvollrath.net.

Next I went shopping for nginx Docker images. The official nginx image has everything I want: the latest and greatest mainline nginx based on stable Debian. I considered the Alpine variant, but felt like Debian was a better choice for me; familiarity outweighs a few tens of MB of image size.

The nginx image ships with a default configuration serving a single root directory over HTTP. Since HTTPS was the point of this experiment, I set out to correct this. I started by creating a project directory on the host to house all the configuration needed to build out my server. Then I started up a container with the vanilla configuration and copied config files /etc/nginx/nginx.conf and /etc/nginx/conf.d/default.conf out of the container to the project directory. With those config files now in my possession, I created a simple Dockerfile to inject them into a new image based on the library nginx image.

FROM nginx:latest

COPY nginx.conf /etc/nginx/nginx.conf
COPY default.conf /etc/nginx/conf.d/default.conf

With that out of the way, I started hacking config to create my ideal HTTPS server. First I set up a redirect to force all traffic to the HTTPS site.

server {
    listen 80;
    server_name mvollrath.net www.mvollrath.net;
    return 301 https://$server_name$request_uri;
}

Now I would need a good way to get my certificates into the container. Docker compose has a handy secrets directive to make this really painless.

version: '3.1'  # must be at least 3.1 for secrets feature
secrets:
  ssl_privkey:
    file: "/etc/letsencrypt/live/www.mvollrath.net/privkey.pem"
  ssl_fullchain:
    file: "/etc/letsencrypt/live/www.mvollrath.net/fullchain.pem"
services:
  nginx:
    container_name: "nginx"
    build: "."
    ports:
    - "80:80"
    - "443:443"
    volumes:
    - "/home/matt/htdocs:/usr/share/nginx/html:ro"
    restart: "on-failure"
    secrets:
    - "ssl_privkey"
    - "ssl_fullchain"

This mounts the provided secrets in /run/secrets to be scooped up by the site config.

server {
    listen 443 ssl http2 default_server;

    server_name mvollrath.net www.mvollrath.net;
    ssl_certificate /run/secrets/ssl_fullchain;
    ssl_certificate_key /run/secrets/ssl_privkey;

    [...]
}

Now I can update my server by running docker-compose build --pull and then docker-compose up -d. This may cause a momentary outage while the containers are being swapped, but for a personal site this is nothing to sweat over. I dropped these commands in a cron script since I like updates but would rather not have to think about updates.

With my new HTTPS site now exposed to the world, I found some free HTTPS validation tools to check my work and optimize the configuration a few notches beyond the “pretty good” nginx defaults. If you’ve deployed an HTTPS site for work or pleasure, check out the collection of web security tools rounded up by Phin in September.

I was really happy with the Let’s Encrypt tools and user experience. If you’re still hosting HTTP with no HTTPS option, consider using the free tools to get your free certificate and help your users protect their privacy. If you’re interested in using HTTPS wherever available, consider using the HTTPS Everywhere extension offered by the EFF.

A caching, resizing, reverse proxying image server with Nginx

2016-05-25T00:00:00+00:00

While working on a complex project, we had to set up a caching reverse proxying image server with the ability of automatically resize any cached image on the fly.

Looking around on the Internet, we discovered an amazing blog post describing how Nginx could do that with a neat Image Filter module capable of resizing, cropping and rotating images, creating an Nginx-only solution.

What we wanted

What we wanted to achieve in our test configuration was to have a URL like:

http://www.example.com/image/x/

…that would retrieve the image at:

https://upload.wikimedia.org/

…then resize it on the fly, cache it and serve it.

Our setup ended up being almost the same as the one in that blog post, with some slight differences.

Requirements installation

First, as the post points out, the Image Filter module is not installed by default on many Linux distributions. As we’re using Nginx’s official repositories, it was just a matter of installing the nginx_module_image_filter package and restarting the service.

Cache Storage configuration

Continuing following the post’s great instructions, we set up the cache in our main http section, tuning each parameter to fit ur specific needs. We wanted a 10MB storage space for keys and 100MB for actual images, that will be removed after not being accessed for 40 days. The main configuration entry was then:

proxy_cache_path /tmp/nginx_cache levels=1:2 keys_zone=nginx_cache:10M max_size=100M inactive=40d;

This went straight in the http section of nginx.conf.

Caching Proxy configuration

Next, we configured our front facing virtual host. In our case, we needed the reverse proxy to live within an already existing site, and that’s why we chose the /image/ path prefix.

server {
      ...
  
      location /image/ {
          proxy_pass http://127.0.0.1:20000;
          proxy_cache nginx_cache;
          proxy_cache_key "$proxy_host$uri$is_args$args";
      }
  
      location / {
          # other locations we may need for the site.
          root /var/www/whatever;
      }
  
  }

Every URL starting with /image/ would be server from the cache if present, otherwise it would be proxied to our Resizing Server, and cached for 30 days, as desired.

Resizing Server configuration

We then configured the resizing server, using a regexp to extract the width, height and URL of the image we desire.

The server will proxy the request to https://upload.wikimedia.org/ looking for the image, resize it and then serve it back to the Caching Proxy. We preferred to keep it simple and tidy, as we didn’t actually need any aws-related configuration as the blog post did.

server {
      ...
  
      location ~ ^/image/([0-9]+)x([0-9]+)/(.+) {
          image_filter_buffer 20M; # Will return 415 if image is bigger than this
          image_filter_jpeg_quality 75; # Desired JPG quality
          image_filter_interlace on; # For progressive JPG
  
          image_filter resize $1 $2;
  
          proxy_pass https://upload.wikimedia.org/$3;
      }
  
  }

Note that we may also use image_filter resize and crop options, should we need different results than just resizing.

Testing the final result

You should now be able to fire up your browser and access an URL like:

http://www.example.com/image/150x150/wikipedia/commons/0/01/Tiger.25.jpg

…and enjoy your caching, resizing, reverse proxying image server.

Optionally securing access to your image server

As this was not a public server, we didn’t use any security mechanism to validate the request.

The original blog post, though, reports a very simple and clever way to prevent abuse from unauthorized access, using the Secure Link module.

To access your server you will now need to add an auth parameter to the request, with a secure token that can be easily calculated as an MD5 hash.

This is the simple Bash command we used to test it:

echo -n '/image/150x150/wikipedia/commons/0/01/Tiger.25.jpg your_secret' | openssl md5 -binary | openssl base64 | tr +/ -_ | tr -d =

…and the resulting URL would be:

http://www.example.com/image/150x150/wikipedia/commons/0/01/Tiger.25.jpg?auth=TwcXg954Rhkjt1RK8IO4jA

Conclusions

Thanks to Charles Leifer for explaining his findings so well and giving us a smooth path with only minor tweaks to make our project work.

DAD Trouble

2014-06-15T00:00:00+00:00

I never thought I’d say it, but these days technology is simply moving too fast for DAD. It’s just the way it is. Of course it’s not DAD’s fault, it’s just the world doesn’t want to wait.

Before I get to that, I want to mention some trouble we’d recently started seeing with nginx failing to start on boot. It’s just been on our most recently obtained servers, both Debian-based (including Ubuntu) and RHEL-based installations. Some were Linode VM’s, others were bare metal hardware systems. After boot and once we got in to try and see what was happening, nginx would happily start manually. The only clue was one line that had been left in the error log:

2014/06/14 23:33:20 [emerg] 2221#0: bind() to [2607:f0d0:2001:103::8]:80 failed (99: Cannot assign requested address)

And it wasn’t just nginx; Apache httpd in one instance gave us similar trouble:

Starting httpd: (99)Cannot assign requested address: make_sock: could not bind to address [2600:3c00::f03c:91ff:fe73:687f]:80
no listening sockets available, shutting down

As an interim fix, since at the moment these systems only had one IPv6 each, we told nginx or httpd to listen on all addresses. But not liking to leave a mystery unsolved, once we were able to schedule a long enough maintenance window on a system to reboot it a few times and see what’s going on, we found that the interface was in a “tentative” state for a short interval.

That was the clue we needed. For some reason, the boot process was allowed to continue before DAD (Duplicate Address Detection) has a chance to decide that if the interface is allowed to use the provided IPv6 address. It’s probably been doing this all along, but the servers that were affected just didn’t boot fast enough to try binding before the interface was ready. Now, things are faster, and service start-up was winning the race.

For us, the addresses are either static or autoconfigured, and we’re confident that a duplicate address situation won’t be a problem. So we turned off dad_transmits by setting this in sysctl.conf:

net.ipv6.conf.all.dad_transmits = 0

Success! No more bind problems on boot preventing a service from starting.

Incidentally, I believe this has been solved in Debian by making the interface wait until it’s out of the “tentative” state, but it doesn’t look like it’s been backported to current stable. It should be in Ubuntu as of the current LTS (14.04) however.

Happy Father’s Day!

Proxy Nginx ports using a regular expression

2014-03-19T00:00:00+00:00

I’m working on a big Rails project for Phenoms Fantasy Sports that uses the ActiveMerchant gem to handle Dwolla payments. One of the developers, Patrick, ran into an issue where his code wasn’t receiving the expected postback from the Dwolla gateway. His code looked right, the Dwolla account UI showed the sandbox transactions, but we never saw any evidence of the postback hitting our development server.

Patrick’s theory was that Dwolla was stripping the port number off the postback URL he was sending with the request. We tested that theory by using the RequestBin.com service for the postback URL, and it showed Dwolla making the postback successfully. Next, we needed to verify that Dwolla could hit our development server on port 80.

I started Nginx on port 80 of our dev server and Patrick fired his Dwolla transaction test again. The expected POST requests hit the Nginx logfile. Suspicions confirmed. It looked like we would just have to work around the Dwolla weirdness by proxying port 80 to the port that Patrick’s development instance was running on. Then we’d need a way to make that work for the other developers’ instances on the dev. server, as well.

Proxying a single port to another port with Nginx is easy, but that second requirement is a little more complicated. We did get a little lucky with this bit, however. We are using DevCamps.org “camps” for this project. DevCamps’ naming convention for a given development instance (AKA a “camp”) number uses a two digit camp number as the hostname and as part of the port number. For example, camp 42 would run on 42.camp.example.com:9042. (I bet you can already see where I’m going with this.)

I tweaked the Nginx config for the port 80 instance to use a regex to capture the hostname (“42” in this case) from the server_name portion of the HTTP request. It then appends that to the full hostname/IP address and the first two digits of that camp’s port number. That made the proxy work for everyone’s camp. Finally, I updated the config to work with any URI under the /dwolla directory.

We now have the following tidbit in our Nginx config:

server {
    # [SNIP unrelated config stuff]

    server_name ~^(?<portname>\d\d)\.camp\.;

    location /dwolla {
        proxy_pass        http://169.29.89.157:90${portname}$uri;
        proxy_set_header  X-Real-IP  $remote_addr;
    }
}

As root, I ran service nginx reload to pick up the new config changes. Now Nginx automagically proxies connections to specific ports based on the server’s hostname.

WebP images experiment on End Point website

2014-01-28T00:00:00+00:00

WebP is an image format for RGB images on the web that supports both lossless (like PNG) and lossy (like JPEG) compression. It was released by Google in September 2010 with open source reference software available under the BSD license, accompanied by a royalty-free public patent license, making it clear that they want it to be widely adopted by any and all without any encumbrances.

Its main attraction is smaller file size at similar quality level. It also supports an alpha channel (transparency) and animation for both lossless and lossy images. Thus it is the first image format that offers the transparency of PNG in lossy images at much smaller file size, and animation only available in the archaic limited-color GIF format.

Comparing quality & size

While considering WebP for an experiment on our own website, we were very impressed by its file size to quality ratio. In our tests it was even better than generally claimed. Here are a few side-by-side examples from our site. You’ll only see the WebP version if your browser supports it:

13,420 bytes JPEG	2776 bytes WebP
14,734 bytes JPEG	3386 bytes WebP

The original PNG images were converted by ImageMagick to JPEG, and by cwebp -q 80 to WebP. I think we probably should increase the WebP quality a bit to keep a little of the facial detail that flattens out, but it’s amazing how good these images look for file sizes that are only 17% and 23% of the JPEG equivalent.

One of our website’s background patterns has transparency, making the PNG format a necessity, but it also has a gradient, which PNG compression is particularly inefficient with. WebP is a major improvement there, at 13% the size of the PNG. The image is large so I won’t show it here, but you can follow the links if you’d like to see it:

337,186 bytes	container-pattern.png
43,270 bytes	container-pattern.webp

Browser support

So, what is the downside? WebP is currently natively supported only in Chrome and Opera among the major browsers, though amazingly, support for other browsers can be added via WebPJS, a JavaScript WebP renderer.

Update! As of 2021, all current major browsers support WebP image rendering.

Why don’t the other browsers add support given the liberal license? Especially Firefox you’d expect to support it. In fact a patch has been pending for years, and a debate about adding support still smolders. Why?

WebP does not yet support progressive rendering, Exif tagging, non-RGB color spaces such as CMYK, and is limited to 16,384 pixels per side. Some Firefox developers feel that it would do the Internet community a disservice to support an image format still under development and cause uncertain levels of support in various clients, so they will not accept WebP in its current state.

Many batch image-processing tools now support WebP, and there is a free Photoshop plug-in for it. Some websites are quietly using it just because of the cost savings due to reduced bandwidth.

For our first experiment serving WebP images from the End Point website, I decided to serve WebP images only to browsers that claim to be able to support it. They advertise that support in this HTTP request header:

Accept: image/webp,*/*;q=0.8

That says explicitly that the browser can render image/webp, so we just need to configure the server to send WebP images. One way to do that is in the application server, by having it send URLs pointing to WebP files.

Let’s plan to have both common format (JPEG or PNG) and WebP files side by side, and then try a way that is transparent to the application and can be enabled or disabled very easily.

Web server rewrites

It’s possible to set up the web server to transparently serve WebP instead of JPEG or PNG if a matching file exists. Based on some examples other people posted, we used this nginx configuration:

set $webp "";
set $img "";
if ($http_accept ~* "image/webp") { set $webp "can"; }
if ($request_filename ~* "(.*)\.(jpe?g|png)$") { set $img $1.webp; }
if (-f $img) { set $webp "$webp-have"; }
if ($webp = "can-have") {
    add_header Vary Accept;
    rewrite "(.*)\.\w+$" $1.webp break;
    break;
}

It’s also good to add to /etc/nginx/mime.types:

image/webp .webp

so that .webp files are served with the correct MIME type instead of the default application/octet-stream, or worse, text/plain with perhaps a bogus character set encoding.

Then we just make sure identically-named .webp files match .png or .jpg files, such as those for our examples above:

-rw-rw-r-- 337186 Nov  6 14:10 container-pattern.png
-rw-rw-r--  43270 Jan 28 08:14 container-pattern.webp
-rw-rw-r--  14734 Nov  6 14:10 josh_williams.jpg
-rw-rw-r--   3386 Jan 28 08:14 josh_williams.webp
-rw-rw-r--  13420 Nov  6 14:10 marina_lohova.jpg
-rw-rw-r--   2776 Jan 28 08:14 marina_lohova.webp

A request for a given $file.png will work as normal in browsers that don’t advertise WebP support, while those that do will instead receive the $file.webp image.

The image is still being requested with a name ending in .jpg or .png, but that’s just a name as far as both browser and server are concerned, and the image type is determined by the MIME type in the HTTP response headers (and/or by looking at the file’s magic numbers). So the browser will have a file called $something.jpg in the DOM and in its cache, but it will actually be a WebP file. That’s ok, but could be confusing to users who save the file for whatever reason and find it isn’t actually the JPEG they were expecting.

301/302 redirect option

One remedy for that is to serve the WebP file via a 301 or 302 redirect instead of transparently in the response, so that the browser knows it’s dealing with a different file named $something.webp. To do that we changed the nginx configuration like this:

rewrite "(.*)\.\w+$" $1.webp permanent;

That adds a little bit of overhead, around 100-200 bytes unless large cookies are sent in the request headers, and another network round-trip or two, though it’s still a win with the reduced file sizes we saw. However, I found that it isn’t even necessary right now due to an interesting behavior in Chrome that may even be intentional to cope with this very situation. (Or it may be a happy accident.)

Chrome image download behavior

Versions of Chrome I tested only send the Accept: image/webp [etc.] request header when fetching images from an HTML page, not when you manually request a single file or asking the browser to save the image from the page by right-clicking or similar. In those cases the Accept header is not sent, so the server doesn’t know the browser supports WebP, so you get the JPEG or PNG you asked for. That was actually a little confusing to hunt down by sniffing the HTTP traffic on the wire, but it may be a nice thing for users as long as WebP is still less-known.

Batch conversion

It’s fun to experiment, but we needed to actually get all the images converted for our website. Surprisingly, even converting from JPEG isn’t too bad, though you need a higher quality setting and the file size will be larger. Still, for best image quality at the smallest file size, we wanted to start with original PNG images, not recompress JPEGs.

To make that easy, we wrote two shell scripts for Linux, bash, and cwebp. We found a few exceptional images that were larger in WebP than in PNG or JPEG, so the script deletes any WebP file that is not smaller, and our nginx configuration will in that case not find a .webp file and will serve the original PNG or JPEG.

Full-page download sizes compared

Here are performance tests run by WebPageTest.org using Chrome 32 on Windows 7 on a simulated cable Internet connection. The total download size difference is most impressive, and on a slower mobile network or with higher latency (greater distance from the server) would affect the download time more.

Page URL	With WebP	Without WebP
Bytes	Time	Bytes	Time
https://www.endpointdev.com/	374 KB	2.9s	850 KB	3.4s
https://www.endpointdev.com/team/	613 KB	3.6s	1308 KB	4.1s

Conclusion

This article is not even close to a comprehensive shootout between WebP and other image types. There are other sites that consider the image format technical details more closely and have well-chosen sample images.

My purpose here was to convert a real website in bulk to WebP without hand-tuning individual images or spending too much time on the project overall, and to see if the overall infrastructure is easy enough to set up, and the download size and speed improved enough to make it worth the trouble, and get real-world experience with it to see if we can recommend it for our clients, and in which situations.

So far it seems worth it, and we plan to continue using WebP on our website. With empty browser caches, visit www.endpointdev.com using Chrome and then one of the browsers that doesn’t support WebP, and see if you notice a speed difference on first load, or any visual difference.

I hope to see WebP further developed and more widely supported.

Full Page Caching in Interchange 5

2013-10-28T00:00:00+00:00

I recently attended the eCommerce Innovation Conference 2013 with fellow End Pointer Richard Templet and presented on the work End Point has done to develop full-page caching in Interchange 5. The work is in final review currently for inclusion into core Interchange and should provide a roadmap for cache management in Nitesi/Interchange 6.

Parameters of a Caching Solution

In order to identify the full scope of what one means when one says an application uses caching, there are (at least) 3 aspects to define:

Where in the application stack is the cache being applied.
How long until the cache is expired.
What level of user state must it support.

The issue of cache duration is not addressed here as any such formulation will be specific to business requirements and tolerance of stale data. I will state, however, that even very brief cache durations can have an enormous impact on performance, particularly for sites that have a relatively small number of resources that absorb the bulk of the traffic. Cache durations of several minutes to several hours can drop the traffic that makes it to Interchange to a small fraction of the overall requests.

Caching and the Application Stack

Let’s examine an ideal target architecture for a simple implementation of an Interchange store, or stores. Note that we could also introduce load balancing layers to substantially boost capacity through horizontal scaling at multiple points in this stack, and if we did so we’d need to add those to our list to identify the impact of selecting a target cache point.

Browser
Reverse proxy (e.g., Varnish, Pound, nginx)
Web server (e.g., Apache httpd, nginx)
Interchange, with session storage
- Database (e.g., MySQL)
- Other applications (e.g., email server)
- Web services (e.g., payment gateway)

The higher up the stack we can cache, the more scalable our solution becomes. Interchange comes with [timed-build] which can be used to great effect to cache database results in particular, but also potentially other applications that could produce bottlenecks in performance. Moreover, because Interchange assembles the document, this is the last point in the stack that we can (directly, if at all) partially cache a resource. However, having all requests reach Interchange is, itself, a scalability issue that would have to be resolved with either horizontal scaling or pushing our cache farther up the stack.

It’s also possible to produce a static build of assets in the web server’s doc space, keeping requests from reaching Interchange at all. And while responses from a web server will have considerably less overhead and better response times than Interchange, both building and maintaining a static repository of Interchange assets is going to take some effort and, ultimately, will require horizontal scaling to relieve overload.

Our target point for the cache described herein is at the reverse proxy. We want to control our cache using standard cache headers and an nginx configuration that uses the full URL for its cache keys. The reverse proxy is chosen because:

Very fast and efficient at delivering documents.
Low maintenance as we are able to have proxy engine keep its data storage fresh according to document headers.
Assets can now be massively scaled through 3rd-party CDNs requiring no infrastructure investment and maintenance that horizontal scaling solutions lower in the stack would.
Full URL cache keys and cache headers allow us extend our cache all the way to the browser, for those user agents that respect the cache headers.

Core Interchange Support for Full-Page Caching

To provide general support for full-page caching in Interchange, we found it necessary to introduce some core features into Interchange 5. These features are in final review internally and will be committed to core soon.

SuppressCachedCookies

New boolean catalog configuration parameter that, when true, instructs Interchange not to write any cookies in a cacheable response. Cookies are inherently specific to the individual client being served and nginx will refuse to cache a resource containing Set-Cookie headers.

$::Instance->{Volatile}

Value indicates to critical core code what the request’s cache potential is. Three values:

undef - unknown, could be cached, but hasn’t been explicitly identified
true - cannot be cached. Indicates the requested resource is user-dependent and may produce different results for the same URL for different users.
false (other than undef) - explicitly treat as a “can be cached” resource. This setting can be used to reverse override other cache overrides.

The ternary nature of Volatile allows a developer to explicitly control the caching behavior of any given resource if circumstances require an adjustment to the default behavior.

[if-not-volatile]

New container tag whose body will only interpolate if the value of $::Instance->{Volatile} at the time of interpolation is false. Tag is particularly useful for placing settings for cache headers on shared resources (includes files, components, etc.) where the final document may or may not be cacheable.

OutputCookieHook

Catalog configuration parameter that takes the name of a catalog or global sub to execute just prior to Interchange writing its Set-Cookie headers. Setting was inspired by the need to maintain portions of the session on the client via cookies to allow some more stubborn session-dependent resources to be cacheable.

Obstacles to Full-Page Caching in Interchange

Standard coding practices in a typical Interchange catalog interfere with full-page caching in a number of ways, primarily:

Liberal coupling of resources with user sessions
Searches common to all site users (e.g., category lists) generate saved search objects that force session coupling
Heavy reliance on non-RESTful URLs, primarily those generated by the process actionmap

In most circumstances, these practices can be altered to produce fully cacheable resources, and particularly if the most heavily used components of the site are addressed first, such as the home page and category lists.

Catalog Changes to Mitigate Caching Obstacles

Precisely what changes are required depends on the specific coding practices used for the resources in question. However, there are a number of typical usage patterns that will to some degree affect almost all Interchange catalogs.

Use RESTful URLs

Avoid unnecessary dependence on the process actionmap, which is often used liberally precisely because it gives lots of hooks into cool and useful features. Avoid any other use of common URLs that produce varying resources based on session dependence.

Take advantage of writing custom actionmaps, which allow the developer extreme flexibility in URL construction. Because actionmaps make it easy and straightforward to produce unique URLs, they are ideal both for creating cacheable resources and fine-tuning SEO.

Permanent More for Category Lists

By default, search objects which Interchange uses for more lists, are restricted to access from the generating user’s session. This is a safeguard as often search results include personal data for access only to the requestor. However, for features such as category lists, this creates a difficult burden for the developer who wishes to cache the popular resources and whose results are identical across all users.

We can overcome this difficulty by making the search definitions for category lists, or other canned searches, include the permanent more indicator. Permanent more causes all identical searches to share a common search object accessible by the same URLs, and freeing the usual coupling with the session of the search originator.

Address Common Session Variables

There are certain session variables that are often found in page code and can cause a number of difficulties when trying to make a resource cacheable. Start by tracking through the use of each of the following and come up with a strategy to remove the dependencies so that the interpolated code is free of them:

[value]
[data session ___]
[set]/[seti]
Any [if] conditionals that use those respective bases (e.g., [if value], [if session], and [if scratch])

Cacheable Redirects

Any code that issues a redirect must do so consistently with respect to its URL. Any redirect will cache the http code and, if issued conditionally, will force all users accessing the cached resource to also redirect. This practice is seen often in Interchange catalogs, particularly when monitoring pages that are restricted to logged-in users. In summary, it’s OK to cache redirects, but just make sure that a given URL is either always, or never, redirected.

Profile and “Click” Code

It is common practice to define profile and click code in scratch variables. This is particularly true for click code defined with the [button] tag, which while convenient causes the click action to be defined under the hood in scratch space. In order for these event-driven features to work, the resource must compile that code and seed it in the session in anticipation of the user’s next actions. If those resources are cached, those important features are never added to the session as the result of a page load and, so, none of the actions will work.

All use of [button] or [set] to produce click or profile code should be moved into the profile files (typically found in etc/profile.*). There they are added to the Interchange global configuration at compile time and are thus available to all users without regard to the state of their sessions. This is good practice generally since it is often easy (particularly with [button]) to have multiple actions map to the same scratch key. When that happens, a user going through the browser back button can get invalid results on actions taken because the click or profile definitions have changed with respect to the anticipated such actions on the current page.

Set SuppressCachedCookies to Yes

As described earlier, this will tell the Interchange core not to write any cookies if the resource is to be cached.

Define Cache-Control Headers

At any point in the development of the response body, Interchange can be issued a pragma that tells it to treat the response as cacheable. This will interact with the core features described above to ensure that there is no impact on the user session as a result of this request, and put the correct headers in the output stream (as well as keep the cookie headers out).

Invocation looks something like [tag pragma cache_control]max-age=NNN[/tag], where NNN is the number of seconds the cache should persist.

Impact on Session Management

Any resource considered cacheable should a priori have neither impact nor dependence on a session. This must be true if we consider that, once cached, a user will interact with the page—and expect correct behavior—without ever touching Interchange. This introduces some new conditions associated with the session:

The initial user request against a cacheable resource will not generate a session. Why is this so? Apart from the already-noted discrepancy of accessing the resource for a cached vs. live hit, generating a session would necessitate producing the session cookie. Returning that cookie would invalidate the resource as cacheable. Further, one of the significant advantages of a reverse-proxy caching strategy is to provide protection in a DoS attack, and the user agents in such an attack are very unlikely to maintain a session. Thus, if we were failing to produce a cache on initial hits to allow setting a session, all those DoS hits would reach Interchange, and on top of that be churning out session entries on the server.
Session writing is suppressed on any response with a cacheable resource. Interchange must treat the response without any permanence because all accesses of the resource from the cache will never reach Interchange. If the request that produced the cache also wrote that user’s session, it would produce a deviation in behavior between the cached v. live requests.

Overrides on a Cacheable Response

Any action resulting in a POST is considered to imply the necessity of the user initiating the request reaching the session (or the database, or some other permanent storage controlled by the Interchange app). Thus POSTs by default force the Volatile setting to true. However, note this can be overridden by the developer if necessary (e.g., if a DoS hits the home page with a POST rather than the expected GET).

Similarly, any requests passing through either the “process” or “order” actionmap are assumed to require access to the session. “process” will most often be issued as a POST as well, although using “order” with a GET is common.

User State on Cacheable Resources

A big mistake a developer may make when considering full-page caching is to assume an all-or-nothing approach. Trying to compartmentalize an entire catalog into fully cacheable resources would be a daunting task, requiring essentially the construction of a fully client-side application and session management. This is neither realistic nor desirable.

A catalog can gain considerable benefit simply from evaluating those resources which do not require session entanglement at all and starting with them. Without considering users that are logged in or have items in their cart, under most circumstances the home and category list pages should be free from entanglement. With a bit of URL management, the resources can skip the cache when a user is logged in or has items in the cart.

“Read Only” User State

If caching is desirable on resources that cannot be decoupled from session influence, we can expose the necessary parts of the session to the client in the form of cookies and can refactor our document to contain client-side code to manage the session use. Typical examples of this would be personalization for logged in users, or the display of a small cart on all pages. The session data stored in the cookie is controlled exclusively by Interchange and is read-only on the client. Each time the session is accessed and updated, the cookie is re-written to the client.

Management of such a process is relatively easy with modern JavaScript frameworks such as jQuery. As a typical example, one might need to replace the following session-dependent code

[if session logged_in]
  Hi, [value fname]!
[/if]

with client-side management:

<span id="fname_display"></span>

and in the ready() event elsewhere with our session cookie data stored in valuesData:

if (valuesData.fname) {
  $('#fname_display').replaceWith('Hi, ' + valuesData.fname + '!');
}

The OutputCookieHook was developed as a convenient mechanism for constructing the proposed valuesData cookie above, allowing for a subroutine to build the cookie just prior to the core code that constructs the document headers but after any standard actions that would alter the session and would need to be captured in the cookie data.

“Read Write” User State

If state needs are more complex on a particularly popular resource, it may be necessary to allow our state cookie to also be updated from the client. With the tools described here, the developer can either amend the existing cookie, or construct a new one, that captures data input by the client through subsequent requests to cached resources. Once the user issues the next non-cached request, an Autoload subroutine could be constructed to identify that changes have occurred and then sync those changes back to the user’s session. While implementing read-write user state may be challenging, it is possible and has been done at End Point for clients where that need exists.

Hopefully this provides a good idea of how to get started when approaching full-page caching, which is possible in most web app frameworks, and soon will be much easier in Interchange 5 with the new core tools introduced here.

Custom 500 error page while using nginx proxying

2013-06-21T00:00:00+00:00

I was working with our customer Paper Source to setup a 500 error page that looked like the rest of the site when I ran into something interesting. I went through the nginx configuration and added this line to allow for a custom 500 error page just like I had done for the custom 404 error page.

error_page   500  =  /cgi-bin/paper/500.html;

What I noticed when I forced the site to create an Internal Server Error was that I was still getting the ugly normal Apache version of the 500 error page. It seemed like nginx was ignoring the error_page directive. I did some searching and found out that you have to use the proxy_intercept_errors directive.

proxy_intercept_errors on;

This directive allows nginx to recognize the 500 error code being returned from Apache and run its own directives to display the right page.

Paper Source: The Road to nginx Full Page Caching in Interchange

2013-01-03T00:00:00+00:00

Background & Motivation

During the recent holiday season, it became apparent that some efforts were needed to improve performance for Paper Source to minimize down-time and server sluggishness. Paper Source runs on Interchange and sells paper and stationery products, craft products, personalized invitations, and some great gifts! They also have over 40 physical stores which in addition to selling products, offer on-site workshops.

Over the holiday season, the website experienced a couple of instances where server load spiked causing extreme sluggishness for customers. Various parts of the site leverage Interchange’s timed-build tag, which creates static caches of parts of a page (equivalent to Rails’ and Django’s fragment caching). However, in all cases, Interchange is still being hit for the page request and often the pages perform repeated logic and database hits that opens an opportunity for optimization.

The Plan

The long-term plan for Paper Source is to move towards full page nginx caching, which will yield speedily served pages that do not require Interchange to be touched. However, there are several code and configuration hurdles that we have to get over first, described below.

Step 1: Identify Commonly Visited Pages

First, it’s important to recognize which pages are visited the most frequently and to tackle optimization on those pages first, essentially profiling the site to determine where we will gain the most from performance optimization. In the case of Paper Source, popular pages include:

Thumbnail page, or the template where multiple products are shown in list format
Product detail page, or the template that serves the basic product page
Swatching page, or the template that serves a special product page with special product options
Personalization detail page, or a template that serves the special product page for personalizeable products (e.g. wedding invitatations, birth announcements, etc.)

Step 2: Remove Dynamic User Elements on pages of interest

The next step in the process is to remove dynamic elements on the page, by having cookies or AJAX render these dynamic elements. Below are a couple of examples of these dynamic elements on two primary page templates.

The thumbnail page contains two dynamic elements: the mini-cart template, which shows how many items are in the user’s cart, and the log in information, which shows “my account” and “log out” links if the user is logged in, and shows a “log in” link if the user is not logged in.

In addition to the mini-cart and logged in elements, the product page contains additional dynamic elements which signify if a user has added an item to their cart, and presentation of the user’s previously viewed items.

In the examples above, the following changes were applied to replace these dynamic elements:

Mini-cart Component: This section utilizes cookies with the jQuery.cookie plugin. A cookie stored in the browser identifies the number of cart items and the cart subtotal. After the DOM loads, the mini-cart is rendered and displayed if the user has a non-empty cart. These cookies are manipulated whenever the cart contents are modified.
Login Component: This section also reads a browser-stored cookie. If the cookie indicates the user is logged in, the navigation elements are updated to reflect that.
Added to cart Component: On the product detail page, this code was invasively modified to allow items to be added to cart via AJAX. The AJAX call results in an update of the cart cookies specified above. The feedback of the AJAX call is presented to the user to indicate that the item has been successfully added.
Previously Viewed Component: Finally, the product page also contains previously viewed items. This is generated via a cookie that stores recent skus visited by the user, and each sku has an associated cookie that includes information such as the image source, link, description, and price. Because a maximum of three previously viewed items is shown, cookies here of older previously viewed items are deleted to minimize cookie build-up.

Step 3: Implement fully timed-build caching pages

During this incremental process to reach the end goal of full nginx caching, the next step is to implement fully timed-build pages, or use Interchange’s caching mechanism to fully cache these pages and reduce repetitive database hits and backend logic. In this step, the entire page is wrapped in a timed-build tag, which results in writing and serving a static cached file for that page. While this step is not a necessity, it does allow for us to deploy and test our changes in preparation for nginx caching. In adddition to giving us an opportunity to work out kinks, this step also gives us an added bump in performance because several of these page templates have no caching at all.

Step 4: Reproduce redirect logic outside of Interchange

Next up, we plan to move logic that handles page redirects outside of Interchange to nginx. At the moment, Interchange is responsible for handling 301 redirects on old product and navigation pages. This will need to be moved to nginx redirects to minimize the hits on Interchange here.

Step 5: Implement nginx architecture on camps

Another non-trivial step in this process will be to implement nginx architecture on DevCamps (or camps). DevCamps is an open source tool developed by End Point for developing on multiple instances of copies of the production server. Camps are heavily used for Paper Source because several End Point and internal Paper Source employees simultaneously work on different projects on their development instances or camps. Nginx caching will need to be set up to also work with the camp system in place.

Step 6: Turn nginx caching on!

Finally, we can turn on nginx caching for specific pages of interest. Nginx will then serve these fully cached pages and will avoid Interchange entirely. Cookies and AJAX will still be used to render the dynamic elements on the fully cached pages. While we’d ideally like to cache every page on the site except for the cart, checkout and my account pages, it makes more sense to find the bottlenecks and tackle them incrementally.

Where are we now?

At the moment, I’ve made progress on steps 1-3 for several subsets of pages, including the thumbnail and product detail pages. I plan to continue these steps for additional bottleneck pages. I have worked out out a couple of minor kinks throughout the recent progress, but things have been progressing well. Richard plans to make progress on the nginx related tasks in preparation for reaching the end goal.

nginx and lighttpd deployments growing

2008-08-30T00:00:00+00:00

Apache httpd is great. But it’s good to see Netcraft report that nginx and lighttpd continue to grow in popularity as well. Having active competition in the free software web server space is really beneficial to everyone, and these very lightweight and fast servers fill an important niche for dedicated static file serving, homegrown CDNs, etc. Thanks to all the developers involved!