.NET 10 is Coming: What's New, and Why It Matters

2025-09-29T00:00:00+00:00

Microsoft has everything almost ready for the upcoming release of .NET 10 in November this year, and it promises to be a major milestone for developers. Currently in beta, the previews released so far have shown improvements in performance, security, compatibility, and more. Let’s take a look and what’s coming, and why it matters.

Performance and runtime improvements

One of the core aspects of .NET 10 will be the improved efficiency in different areas. To mention a few of them:

Optimized compression streams such as GZipStream and async implementations for ZipArchive methods
Devirtualization improvements for interface methods, especially when arrays are involved, resulting in faster code execution
Faster stack allocation for objects through the expanded use of scape analysis

All of these features (and others) contribute to lower latency and more efficient resource usage. There is a comprehensive blog post by Stephen Toub, .NET team developer, where he runs several benchmarks proving how these improvements result in faster responses and smaller allocations.

Security and increased cryptographic support

.NET 10 also introduces enhancements to encryption and authentication, increasing application security with support for new encryption policies and ways of authentication.

Enhanced certificate thumbprint support adds the ability to search certificates by thumbprint using more secure hash algorithms (not just SHA-1)
WebAuthn support enables passwordless logins with secure authentication methods like biometrics or physical security keys
More cryptographic APIs: Better support for post-quantum cryptography APIs

API & web enhancements

For Web APIs, services, and Blazor apps, .NET 10 has several useful features aiming to increase productivity, security, and cloud capabilities.

Minimal APIs now support built-in validation through Data Annotations
OpenAPI 3.1 support: ASP.NET Core in .NET 10 will generate OpenAPI 3.1 documents with the latest JSON Schema specifications
Several improvements to Blazor assets delivery and environment configuration. One key change is that Blazor scripts will be served as a static web asset

AI, cloud, and DevOps

AI developers will also benefit from the new .NET, as well as applications residing in the cloud and serverless environments.

The ML.NET Framework will keep evolving with expanded capabilities
Cloud Native Development: .NET 10 is designed to be fundamental to cloud native deployments, thanks to smaller base images and the automatic trimming of unused bits
The HybridCache library unifies in-memory and distributed caching, with tagging for smarter invalidation

C# 14

With .NET 10, Microsoft is also rolling out a new version of the C# language, including several improvements that will result in cleaner syntax and better performance. Let’s mention some key features:

User-defined compound assignment operators: We will be able to define our own operators for types like +=, -=, etc.
Using nameof(List<>) without specifying type arguments if we only need the name, avoiding boilerplate in generic programming and logging
Partial constructors and partial events, expanding features like partial methods and properties from the previous language versions

You can find more details about all these upcoming changes on the official .NET website. Based on what we’ve seen above, .NET 10 promises to be one of the most significant releases to the .NET ecosystem, consolidating many incremental improvements that will result in measurable gains in performance, security, and productivity.

I encourage you to explore the .NET 10 Release Candidate 1 to try things yourself and get an early look into what’s about to come out in the .NET world!

Announcing End Point Ecommerce, Our New Open Source Project

2025-08-21T00:00:00+00:00

Today we’re pleased to announce a new open source project: End Point Ecommerce.

End Point Ecommerce is a minimalist ecommerce system that’s quick to set up and easy to understand. It is meant for developers to own, adapt, customize, and extend.

You can learn more about it on our landing page. You can also fork it today on GitHub. There are running demos here:

Are you interested in using End Point Ecommerce for your ecommerce site? Feel free to contact us and we can work together to help you deploy, customize, and maintain it.

Key features

Built with ASP.NET, PostgreSQL, and Authorize.net
Multiple deployment strategies, including Docker Compose
Code base designed to follow Clean Architecture and Domain Driven Design concepts, without being dogmatic about it — simplicity trumps all
Good test coverage with xUnit
Includes a backend admin portal for managing the store, a REST API for building user-facing store frontends, and a simple web frontend
Limited functionality — implements the bare minimum of ecommerce use cases
Product catalog, shopping cart, order submission, email delivery

How we got here

We recently had the opportunity to help one of our clients migrate their ecommerce site. They were using an established ecommerce engine and were having various problems with it. Their requirements were simple, and they didn’t need all the bells and whistles of big ecommerce solutions. They didn’t want to deal with the overhead, bloat, and difficulty of finding maintainers.

During initial investigation and planning, we didn’t find any options that were small and easy to set up, understand, and customize, so we decided to build them a new site from scratch.

We realize that there may be others out there in the same boat that we were. You want to set up a custom ecommerce site but you don’t want to deal with the complexity of bigger off-the-shelf solutions. You want something which you can thoroughly understand quickly, and modify to your heart’s content, as if it were your own code base that you developed from scratch.

Who is this for?

This project is meant for .NET developers who want a solid and simple foundation to build and customize ecommerce sites. To that end, the feature set is very basic, the code base is very straightforward, and the documentation is very developer-oriented. The idea being that developers can quickly get up to speed with how everything works, develop ownership over the code base, and implement their bespoke use cases.

This project is not meant for non-technical folks. There is no installation wizard, no templating engine, no plugin framework, no no-code customization capabilities.

If you want to set up a simple online store, or develop a highly customized one from scratch, think of this as a way of skipping the first few steps of building the foundational architecture and basic functionality.

Please check it out! And do whatever you want with it!

Implementing Azure Blob Storage in .NET 9

2025-06-02T00:00:00+00:00

Businesses keep moving toward scalable and cloud-based architectures. With this in mind, a client that was dealing with random errors in a .NET app when saving files locally on the web server decided to get rid of that process and replace it with an Azure Blob Storage implementation.

Why use Azure Blob Storage? It’s an efficient cloud object storage solution from Microsoft, designed to store unstructured data, optimized for storing and serving documents, media, logs, or binary data, especially in applications that expose this data through an API. The key features are high performance, redundancy, reliability, and scalability. There’s an SDK that we can use for easy integration and development, be it in .NET or other languages.

Let’s take a look at what that change involves. For this example, we will set up the integration in a .NET 9 application:

Install the NuGet package required to connect to Azure Blob Storage. We can do it with the dotnet CLI, or through the NuGet package manager.

dotnet add package Azure.Storage.Blobs

Then, we need to configure our connection in our appsettings.json file. We will use the connection string that Azure provides us when we create the new storage account.

{
  "AzureBlobStorage": {
    "ConnectionString": "{your-connection-string}",
    "ContainerName": "my-container"
  }
}

In Azure Blob Storage, containers are like folders that group blobs (or files) together within a storage account. In this setting, the ContainerName value tells our application which container inside that account will be used for storing and retrieving blobs.

Create a BlobService class to read and write to the container.

using Azure.Storage.Blobs;
using Microsoft.Extensions.Configuration;

public class BlobService
{
    private readonly BlobContainerClient _containerClient;

    /// <summary>
    /// Initializes a new instance of the <see cref="BlobService"/> class.
    /// </summary>
    /// <param name="configuration">Application configuration containing Azure Blob Storage settings.</param>
    public BlobService(IConfiguration configuration)
    {
        var connectionString = configuration["AzureBlobStorage:ConnectionString"];
        var containerName = configuration["AzureBlobStorage:ContainerName"];

        _containerClient = new BlobContainerClient(connectionString, containerName);
    }

    /// <summary>
    /// Uploads a file stream to Azure Blob Storage with the specified file name.
    /// </summary>
    /// <param name="fileName">The name of the file to be stored in blob storage.</param>
    /// <param name="fileStream">The stream representing the file content.</param>
    /// <returns>A task that represents the asynchronous upload operation.</returns>
    public async Task UploadFileAsync(string fileName, Stream fileStream)
    {
        var blobClient = _containerClient.GetBlobClient(fileName);
        await blobClient.UploadAsync(fileStream, overwrite: true);
    }

    /// <summary>
    /// Downloads a file from Azure Blob Storage.
    /// </summary>
    /// <param name="fileName">The name of the file to download from blob storage.</param>
    /// <returns>A task that returns a stream containing the blob's content.</returns>
    public async Task<Stream> DownloadFileAsync(string fileName)
    {
        var blobClient = _containerClient.GetBlobClient(fileName);
        var response = await blobClient.DownloadAsync();
        return response.Value.Content;
    }
}

Quite simple, isn’t it? Now, let’s see a way to easily expose file upload/download endpoints via ASP.NET Core minimal APIs. Introduced in .NET 6 and evolving further in .NET 9, minimal APIs provide a lightweight way to build HTTP APIs with a few lines of code. Instead of requiring full controller classes and attributes, minimal APIs let you define routes directly in your main Program.cs file, using simple lambda expressions. This makes them ideal for microservices, lightweight APIs, and internal tools where fast development and low overhead are essential.

For this example, we will create two endpoints that will use our BlobService class to communicate with Azure Blob Storage to save and retrieve files. In our Program.cs file, let’s add:

/// <summary>
/// Endpoint to upload a file to Azure Blob Storage.
/// </summary>
/// <param name="file">The uploaded file sent from the client (via multipart/form-data).</param>
/// <param name="blobService">Injected service used to handle blob storage operations.</param>
/// <returns>An HTTP 200 OK response when the upload succeeds.</returns>
app.MapPost("/upload", async (IFormFile file, BlobService blobService) =>
{
    using var stream = file.OpenReadStream();
    await blobService.UploadFileAsync(file.FileName, stream);
    return Results.Ok("File uploaded successfully");
});

/// <summary>
/// Endpoint to download a file from Azure Blob Storage.
/// </summary>
/// <param name="fileName">The name of the file to download (provided in the URL path).</param>
/// <param name="blobService">Injected service used to access blob storage.</param>
/// <returns>The file stream as a binary response with a download filename.</returns>
app.MapGet("/download/{fileName}", async (string fileName, BlobService blobService) =>
{
    var stream = await blobService.DownloadFileAsync(fileName);
    return Results.File(stream, "application/octet-stream", fileName);
});

Of course, we would want to configure authentication, add validations, and other security measures on these endpoints to prevent spammers from filling out our storage capacity 🙂. But the basics are there.

Also, there are recommended practices for any Azure integrations to consider, as we increase the robustness of our application:

Monitor and log all storage operations using Azure Application Insights. Integrate Application Insights into our .NET app to get detailed logging of the different HTTP calls to the Azure services.
Use Managed Identities: Instead of storing secrets in appsettings.json, we can assign a Managed Identity to our application service, and grant it the Storage Blob Data Contributor role at the storage account. That will allow the application to gain access to the container automatically, without storing credentials in our settings files.
Containers should be private by default. Avoid using “Blob” or “Container” public access levels, unless necessary. When public access is enabled, any person with a link to the container will be able to open or download the blob.

We will cover more details about using Managed Identities and Shared Access Signatures (SAS) for other Azure services in an upcoming blog post. Stay tuned!

Reducing page load times: Cloudflare and caching with ASP.NET

2025-01-14T00:00:00+00:00

When designing the architecture for a new website, it’s important to keep caching in mind. Caching allows you to store a copy of the pages or resources used by your web application in your local browser. Content distribution networks such as Cloudflare also leverage cache directives to distribute content to users more efficiently.

If we are going to use response caching on our web application, we should not include anything that depends on the identity context for the user in the page content in the response. Instead, we should render the same content to every user that hits the same URL. If we include any identity-specific content, the CDN will reuse it for other requests, which is a security and privacy risk.

A simple example is an ecommerce site where the customer logs in to make a purchase: If the homepage shows the user name and icon at the top bar, that content should be rendered from a non-cached request. Otherwise, CDN caching will show the same user and icon to other visitors. And worse: depending on the validations applied to the website, the profile page could even display personal information to other users!

When dealing with these situations, I often rely on asynchronous requests to a set of non-cached API endpoints to retrieve the identity information, and process it on the frontend to render the user-dependent content.

Adding response caching

In .NET Core, we have a simple way of adding response caching to an ASP.NET application. Let’s see how easy it is to enable caching on a Razor Pages app. First, in our Program.cs file, we need to add a couple lines:

+ builder.Services.AddResponseCaching();

var app = builder.Build();

+ app.UseResponseCaching();

Second, we need to include the [ResponseCache] attribute on our action methods to specify the caching options for that request. In the example below, we are adding a response cache directive of 30 seconds that will vary depending on the User-Agent header sent with the request:

+ [ResponseCache(VaryByHeader = "User-Agent", Duration = 30)]
public async Task<IActionResult> OnGet()
{
    return Page();
}

To test the new logic in place, we can navigate to the page in Chrome and review the response headers in DevTools. We should set the cache-control header to public, with the max-age attribute set to the duration in seconds we specified.

Using Cloudflare to distribute cached content

Once we have our cache responses configured, we can leverage Cloudflare to store our cached responses in the CDN, and improve our page load response time for the user while decreasing the number of requests to our server. Win-win!

By default, dynamic content is not cached in Cloudflare. This is a default configuration to avoid security implications: Cloudflare will disable caching on certain content type and file extensions (such as HTML, or in our case, ASP.NET pages). That means that once we deploy our web application and test it with the CDN active, we will get a response header similar to this:

The cf-cache-status header is a custom header added to the response by Cloudflare that indicates whether the content was cached by the CDN. In this case, we got the value DYNAMIC, meaning that Cloudflare does not consider the content eligible to cache.

To enforce caching, we need to add some additional rules: On Cloudflare’s dashboard, let’s select our website and, on the left nav, go to Rules → Page Rules. We will open a section where we can add custom rules that will apply to specific URL patterns. Here, I usually create two rules:

A page rule that will cache everything on the website. This rule will tell Cloudflare to cache dynamic and static content on the CDN. This is reasonable since we are aware of the implications of caching our pages, and already made the changes neccesary to avoid caching identity-related data.
A second page rule that will skip caching completely on the location where we will be delivering identity-specific content. For example, any API endpoints that will return the user details to be displayed at the top bar, or the user profile page, should be bypassed by the CDN. In our example, those API endpoints are inside the /api/user/* location.

Note: It’s important to set this rule to the first position in the list, so it’s not overridden by previous rules that were deployed.

Testing

After saving and deploying both rules, it’s time to test the results. When we navigate to our website’s homepage with Chrome, we will see the updated response headers:

The “HIT” value means the request hit a cached version of the resource in the CDN.

And finally, we should test the response headers on our identity-related API endpoints.

Yes! We’re getting a BYPASS value for the cf-cache-status header, meaning Cloudflare is not caching any content from that URL.

All right! This is as far as it goes for this post. Now, we have our ASP.NET pages cached, and Cloudflare is distributing the cached version through the CDN, reducing the number of requests to our servers and decreasing the response times for the user. Of course, this is just the tip of the iceberg; we can also add caching for assets and external resources, specify different strategies and durations for each resource type, and more. All that is material for upcoming blog posts.

Using a Containerized Nginx Proxy to Serve a Multi-Application .NET System

2024-09-24T00:00:00+00:00

We recently blogged about how we deployed a system made of multiple .NET applications using Docker containers. In order to make them accessible over the internet, we created a reverse proxy using Nginx.

In that case, we installed and configured the Nginx instance directly in the server, as opposed to the rest of the applications, which ran within containers. That approach did and still does work well for us.

In this article, we’re going to explore an alternative strategy. One where we push the containerization aspect further and deploy and run the Nginx instance itself in a Docker container.

Reintroducing the demo project

Like I said, our system has multiple runtime components, each one of them running in their own container. We have two ASP.NET Core web applications: an Admin Portal and a Web API. They live in this Git repository. And we also have a Postgres database, which the apps interact with.

We also have another repository where the deployment-related files are stored. Among others, there are the expected compose.yaml and Dockerfiles that describe the entire infrastructure.

Throughout this post we will update those deployment configuration files to add an Nginx reverse proxy.

Let’s see what that would look like.

Adding the proxy service in `compose.yaml`

First we need to add the new container in the compose.yaml’s services section. It doesn’t need to be too complicated:

# compose.yaml

services:

# ...

  proxy:
    # The proxy container will be based on this Dockerfile, which we'll define
    # soon.
    build:
      context: .
      dockerfile: Dockerfile.Proxy

    # Here we expose the Nginx proxy via port 8888. This can be anything. In
    # fact, if you want to do multiple parallel deployments on the same machine,
    # that is, with many Nginx instances running at the same time, you can
    # adjust this setting appropriately to prevent port conflicts. Making sure
    # that each instance has its own port.
    ports:
      - 8888:80

    # We want the proxy to start up after the admin-portal and web-api services
    # are up and running. The depends_on setting helps with that.
    depends_on:
      admin-portal:
        condition: service_started
      web-api:
        condition: service_started

#...

Writing the proxy `Dockerfile`

As you saw, the proxy configuration in compose.yaml leverages an external Dockerfile to build the container image that will run our Nginx proxy. This file is also very straightforward. It uses the official Nginx image from Docker Hub and it looks like this:

# Dockerfile.Proxy

# We can pick the version and flavor that we like. In our case here, this is an
# image based on the latest release of Nginx, and the latest release of Debian.
FROM nginx:1.27.1-bookworm

# Unsurprisingly, we have a custom configuration that we want the proxy to use.
# This is how we make sure it does. We copy it into the default location inside
# the image.
COPY proxy/nginx.conf /etc/nginx/nginx.conf

Configuring the proxy

Now we have to configure the Nginx proxy to route requests to both our web applications. Here’s an nginx.conf that does just that:

# proxy/nginx.conf

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;

    keepalive_timeout  65;

    # We have to comment out or remove this line to make sure the default
    # configuration that comes in the official image is not applied.
    # include /etc/nginx/conf.d/*.conf;

    # Our customizations start here:
    server {
        # With this listen directive, we configure our proxy to expect
        # connections coming from the default HTTP port: 80.
        listen 80;

        # This location directive makes sure all requests coming to URLs that
        # look like .../admin are routed to the Admin Portal web app.
        location ~ ^/admin(/?)(.*) {
            proxy_pass http://admin-portal:8080;
            proxy_http_version 1.1;
            proxy_set_header Host $host;
            proxy_set_header Connection keep-alive;
        }

        # This location directive makes sure all requests coming to URLs that
        # look like .../api are routed to the Web API.
        location ~ ^/api(/?)(.*) {
            proxy_pass http://web-api:8080;
            proxy_http_version 1.1;
            proxy_set_header Host $host;
            proxy_set_header Connection keep-alive;
        }
    }
}

What I’ve done here is take the default Nginx configuration file that comes right out of the box, and replace the include /etc/nginx/conf.d/*.conf; line with my own server block directive.

As explained in the official image’s Docker Hub page, a quick way of obtaining a copy of this default file is using this command: docker run --rm --entrypoint=cat nginx /etc/nginx/nginx.conf > /host/path/nginx.conf.

The most interesting parts are the proxy_pass directives that take care of redirecting traffic to the apps. Notice how they refer to the Admin Portal using http://admin-portal:8080 and to the Web API by http://web-api:8080. These are the internal naming of these components within the containers’ virtual network, which gets created automatically by Docker Compose.

Remember that this Nginx instance is not running in the host machine directly. Instead it’s running in a container. In the same virtual network as the other containers described in compose.yaml. That’s why it has to use the hostnames assigned by compose.yaml (i.e. admin-portal and web-api) and the port through which the ASP.NET Core apps running within accept requests (i.e. 8080).

This is the idea:

It also uses proxy_set_header directives to set Host and Connection headers. This is typical practice when it comes to Nginx reverse proxies. You can read more about that on Nginx’s website.

Configuring the PathBase in ASP.NET Core apps

There’s an additional step that we have to take to make all this work. We have configured our Nginx proxy to serve both applications under the same “server”, and rely on different URL paths (i.e. /admin vs /api) to determine which app will receive which request. This means that we have to perform further configuration in the apps so that routing is done properly. Thankfully, all it takes is a one-liner in each of the apps’ Program.cs files. After the usual var app = builder.Build(); line, we do the following:

For the Admin Portal, we add this:

// source/VehicleQuotes.AdminPortal/Program.cs

// ...

app.UsePathBase("/admin");

// ...

And for the Web API:

// source/VehicleQuotes.WebApi/Program.cs

// ...

app.UsePathBase("/api");

// ...

With that, the apps are ready to handle requests coming from the Nginx proxy, which will include either the /admin or /api sections.

Deploying

Finally, we can deploy. Go into the directory where all the deployment configuration files live. That’s the one with the compose.yaml file. In other words, the root of the deploy repo. Once in there, run:

docker compose up --build

After a while, Docker compose will have built and deployed everything for us. Now you can navigate to the apps in any browser at http://localhost:8888/admin and http://localhost:8888/api/swagger/index.html.

As you click around, you can see the Nginx logs with:

docker compose logs proxy -f

To bring it all back down, you can run:

docker compose down

Cool! At the end of the day, Nginx is just another program that runs as a process in an operating system. And as such, it can be run in a container. One of the nice aspects about this setup is the convenience of having an entire system described in a set of files, and being able to bring everything up with a single command.

Using Docker Compose to Deploy a Multi-Application .NET System

2024-07-13T00:00:00+00:00

This post was co-authored by Juan Pablo Ventoso

We recently developed a system that involved several runtime components. It was an ecommerce site that included a database, a web API, an admin control panel web app, and a frontend SPA.

There are many ways to deploy such a system. For us, we wanted the infrastructure to be easily replicable for multiple environments with slightly different configurations. We wanted to be able to have, for example, a production and a staging version that could be deployed easily, with minimal configuration changes. We also wanted the infrastructure to be captured in files and version controlled, to further help replicability and maintainability.

With all that in mind, Docker Compose seemed like an ideal option. We could author a series of configuration files, parameterize environment-specific changes and, with a single command, we could spin up a whole environment to run the various applications within the system.

In this blog post, I’ll explain how we did that using a demo .NET code base that has a similar set of components. Let’s get started.

Getting familiar with the demo project

In .NET terms, our demo code base is organized as a solution with multiple projects. Two of those projects are ASP.NET web applications: a Razor Pages web app (the admin portal), and an MVC Web API. The rest are class libraries that define the core domain logic, tests, and other utilities. For deployment purposes, the web application projects are the interesting ones, as they produce executables that actually need to run as processes in the server. So, including the database, our demo system has three runtime components:

The database.
The admin portal.
The web API.

In the world of Docker, that would translate into three separate containers. Considering Docker Compose, that means three separate services.

Throughout this post, we will, step by step, build a compose.yaml, a set of Dockerfiles, and other configuration files which can be used with Docker Compose to deploy our system.

You can find the system’s source code on GitHub. The final version of the deployment files we’ll build in this article are also on GitHub.

Including the code base repository as a Git submodule

Considering the separation of our components, the reasons for the organization of our deployment configuration files and our code base starts to become apparent. The deployment files will live in their own repository. They do need access to the system’s source code, though, in order to build and run the apps. And that source code lives in its own repo. So, the deployment repository will include a source subdirectory, which will be a Git submodule that points to the repository where the system’s source code is stored.

Here’s the file structure that we’re aiming for:

.
├── compose.yaml
├── the various dockerfiles...
├── source
│   ├── vehicle-quotes.sln
│   └── the varous projects...
└── any other files and directories...

Including a Git repository inside another Git repository as a submodule is easy. In our case, we already have the parent Git repo, which is the one where the deployment config files live. To add the source code repo to it, we run a command like this:

git submodule add git@github.com:megakevin/end-point-blog-dotnet-8-demo.git source

Pretty straightforward. The command is calling git submodule add and passing it the location of the repo on GitHub and the directory in which to clone it. That’s it. As a result of that command, Git will have created a new source directory with the contents of the end-point-blog-dotnet-8-demo repository, and a .gitmodules file with contents like these:

[submodule "source"]
  path = source
  url = git@github.com:megakevin/end-point-blog-dotnet-8-demo.git

As you can see, this file contains the information Git needs to know that this repo has a submodule, where it is, and where it comes from.

Deploying the database

Let’s begin actually building out our system components with the database. Our apps use a PostgreSQL database to store information. Luckily for us, getting a PostgreSQL database up and running with Docker Compose is easy. All it takes is a compose.yaml file like this:

# ./compose.yaml

services:
  # This is the name of the service: "db". Other containers will use that as a
  # hostname when they need to interact with it.
  db:
    # The container will be based in the "16.3-bookworm" release of the public
    # PostgreSQL image that's available in Docker Hub
    image: postgres:16.3-bookworm
    # Always restart the container automatically if it ever closes for whatever
    # reason.
    restart: always
    # Makes the database accessible at port 5432.
    ports:
      - 5432:5432
    # Stores the PostgreSQL data files in a docker volume so that they are not
    # lost when the container stops or restarts.
    volumes:
      - db-postgres-data:/var/lib/postgresql/data
    # Sets some basic configuration for the initial database in the PostgreSQL
    # instance: its name, and a pair of credentials to access it.
    environment:
      - POSTGRES_DB=vehicle_quotes
      - POSTGRES_USER=vehicle_quotes
      - POSTGRES_PASSWORD=password

# Defines the Docker volume that the db service uses to store its data files.
volumes:
  db-postgres-data:

This is a very standard definition of a Docker Compose service. We called it db, made it available via the standard PostgreSQL port (5432), and set some basic configurations for it. Check out the PostgreSQL image’s page in Docker Hub to see more complicated use cases.

To bring it to life, we can run docker compose up -d in the directory where we’ve located the compose.yaml file. Running that command will prompt Docker to download the PostgreSQL image, build a container with it, and run it. It’ll also create the db-postgres-data volume we configured and a “network” that all the services we put in the compose.yaml file will be part of. Thanks to the -d option, it will run in daemon mode; that is, as a background process. So as soon as it’s done, it gives us control of our terminal back:

$ docker compose up -d

...

[+] Running 3/3
 ✔ Network end-point-blog-dotnet-docker-deploy_default            Created  0.0s
 ✔ Volume "end-point-blog-dotnet-docker-deploy_db-postgres-data"  Created  0.0s
 ✔ Container end-point-blog-dotnet-docker-deploy-db-1             Created  0.1s

Now that we have a database up and running, we have a couple of options for connecting to it. If we have the psql command line client installed in our machine, we can connect to it directly:

$ psql -h localhost -d vehicle_quotes -U vehicle_quotes -W
Password:
psql (16.3 (Ubuntu 16.3-0ubuntu0.24.04.1))
Type "help" for help.

vehicle_quotes=#

If not, we could first connect to the container, and then open psql from there:

$ docker compose exec db bash
root@edc038da3aa4:/# psql -h localhost -d vehicle_quotes -U vehicle_quotes -W
Password:
psql (16.3 (Debian 16.3-1.pgdg120+1))
Type "help" for help.

vehicle_quotes=#

Notice how we pass db and bash as parameters to the docker compose exec command. db is the name of our service, and bash is the command that we wish to execute within it. In this case we just want to open a shell, so we use bash.

Nice. That’s all it takes to set up the database. Now on to the applications.

Deploying the admin portal web app

Like I mentioned before, our code base contains an admin portal web application. In order to deploy it, we need first to define an image, which we’ll do with a Dockerfile, and then add the configuration to run a container based on that image using Docker Compose.

The image is a self-contained package that includes all the software that the application needs to run. It’s like an executable program. One that can be run by Docker, instead of directly by the operating system. In order to build images, we use Dockerfiles. For this ASP.NET app, the Dockerfile will perform two main tasks: Specify how to build the app and how to run it. Here’s what a Dockerfile for the admin portal project could look like:

# ./Dockerfile.AdminPortal

# Pull the .NET SDK image from Microsoft's repository. We will use it to build
# our app. We also give it a name of "build" so that we can reference it later.
FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build

# Now we copy all *.csproj files from the code base into the image and run
# dotnet restore. This will download and install all the NuGet packages that the
# various projects require.
WORKDIR /source

COPY source/vehicle-quotes.sln .
COPY source/VehicleQuotes.AdminPortal/VehicleQuotes.AdminPortal.csproj ./VehicleQuotes.AdminPortal/
COPY source/VehicleQuotes.Core/VehicleQuotes.Core.csproj ./VehicleQuotes.Core/
COPY source/VehicleQuotes.CreateUser/VehicleQuotes.CreateUser.csproj ./VehicleQuotes.CreateUser/
COPY source/VehicleQuotes.IntegrationTests/VehicleQuotes.IntegrationTests.csproj ./VehicleQuotes.IntegrationTests/
COPY source/VehicleQuotes.RazorTemplates/VehicleQuotes.RazorTemplates.csproj ./VehicleQuotes.RazorTemplates/
COPY source/VehicleQuotes.UnitTests/VehicleQuotes.UnitTests.csproj ./VehicleQuotes.UnitTests/
COPY source/VehicleQuotes.WebApi/VehicleQuotes.WebApi.csproj ./VehicleQuotes.WebApi/
RUN dotnet restore

# Next we copy all of the source code for all of the projects.
COPY source/VehicleQuotes.AdminPortal/. ./VehicleQuotes.AdminPortal/
COPY source/VehicleQuotes.Core/. ./VehicleQuotes.Core/
COPY source/VehicleQuotes.CreateUser/. ./VehicleQuotes.CreateUser/
COPY source/VehicleQuotes.IntegrationTests/. ./VehicleQuotes.IntegrationTests/
COPY source/VehicleQuotes.RazorTemplates/. ./VehicleQuotes.RazorTemplates/
COPY source/VehicleQuotes.UnitTests/. ./VehicleQuotes.UnitTests/
COPY source/VehicleQuotes.WebApi/. ./VehicleQuotes.WebApi/

# Now that we have everything in place, we use the dotnet publish command to
# build the VehicleQuotes.AdminPortal project.
WORKDIR /source/VehicleQuotes.AdminPortal
RUN dotnet publish -c release -o /app --no-restore

# In this last step, notice how we pull a different image here, this is the
# image that Microsoft recommends to use for running ASP.NET apps.
#
# Since by this point we already built the app, and have the executable binary
# for it, we don't need the full SDK anymore. dotnet/aspnet is a lighter weight
# image designed exclusively for running apps. In other words, it has the .NET
# runtime redistributable, not the full SDK.
#
# So here, all we do is copy the build assets from the "build" image into this
# new runtime-only one and put them in a /app directory. Then we call the .NET
# CLI to run the web app's DDL.
FROM mcr.microsoft.com/dotnet/aspnet:8.0
WORKDIR /app
COPY --from=build /app ./
ENTRYPOINT ["dotnet", "VehicleQuotes.AdminPortal.dll"]

With a Dockerfile like this, we could run the app as a stand-alone container. However, we don’t want to run it like that. Instead, we want it to be a service that’s deployable via Docker Compose, as part of a bigger ecosystem. In order to do that, we add the following to our compose.yaml file.

# ./compose.yaml

services:
  # The service is called "admin-portal" because that's the application that it
  # contains.
  admin-portal:
    # This is slightly more complex than the "db" service. Instead of using the
    # "image" option to download an off-the-shelf PostgreSQL image from
    # dockerhub, we use "build" and point to the Dockerfile we created in the
    # previous step. We also specify the current directory (i.e. the ".") as the
    # context in which the build will be performed.
    build:
      context: .
      dockerfile: Dockerfile.AdminPortal
    # We also configure this service to restart automatically if for whatever
    # reason it stops.
    restart: always
    # "8080" is the default port that ASP.NET Core 8 web applications listen to.
    # So here, we set it up so that all requests coming to the host machine to
    # port 8001 get sent to this container's port 8080. That way, reaching the
    # ASP.NET app that's running within.
    ports:
      - 8001:8080
    # This part is particular to our app. The admin portal offers the capability
    # of uploading files. Here, we're defining a Docker volume to ensure that
    # the uploaded files persist across container restarts. In this case, we are
    # linking the host machine's ./uploads directory to the container's
    # /app/wwwroot/uploads directory, which is where the uploaded files are
    # saved.
    volumes:
      - ./uploads:/app/wwwroot/uploads
    # We can use this section so set any environment variables that our app may
    # need. In the case of our admin portal here, we set the running environment
    # to Development, specify a database connection string, and the location to
    # save the uploaded files.
    environment:
      - ASPNETCORE_ENVIRONMENT=Development
      - ConnectionStrings__VehicleQuotesContext=Host=db;Database=vehicle_quotes;Username=vehicle_quotes;Password=password
      - QuoteImagesPath=/app/wwwroot/uploads
    # Finally, "entrypoint" specifies the command that should be used when
    # running the container. In this case, this is essentially the same as the
    # last line of the Dockerfile we saw above.
    entrypoint: ["sh", "-c", "dotnet VehicleQuotes.AdminPortal.dll"]

# ...

With this, we’re ready to see the application running. Run docker compose up -d again to update the infrastructure and include the new service:

$ docker compose up -d

...

[+] Running 2/2
 ✔ Container end-point-blog-dotnet-docker-deploy-db-1            Running   0.0s
 ✔ Container end-point-blog-dotnet-docker-deploy-admin-portal-1  Started   0.3s

After a while of downloading and building, navigating to http://localhost:8001 should show this:

Adding a maintenance container

At this point, we have a problem though. If we try clicking on the “Quotes” link in the top navigation bar, we see this:

We need to run the database migrations. In order to do that, we need an environment that has all of our source code and the full .NET SDK with the Entity Framework Core command line tool. That is, an environment that can build the app and run the .NET CLI. While we could install that in the host machine and run our migrations that way; we could also encapsulate it in a container. That way we get the benefits of portability, etc.

Of course, that maintenance container needs an image. And images are defined in Dockerfiles. Here’s a Dockerfile that would serve our purpose:

# ./Dockerfile.Maintenance

# Same as before, we pull the official image that contains the full .NET SDK.
FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build

# Here we install the psql command line client. A useful tool to have in a
# container meant for system maintenance.
RUN apt-get update && export DEBIAN_FRONTEND=noninteractive \
    && apt-get -y install --no-install-recommends postgresql-client

# We also install the dotnet-ef tool which allows us access to commands for
# managing the database and migrations.
RUN dotnet tool install dotnet-ef --global
ENV PATH="$PATH:/root/.dotnet/tools"

# Finally, we copy all of our source code.
WORKDIR /source

COPY source/vehicle-quotes.sln .
COPY source/VehicleQuotes.AdminPortal/VehicleQuotes.AdminPortal.csproj ./VehicleQuotes.AdminPortal/
COPY source/VehicleQuotes.Core/VehicleQuotes.Core.csproj ./VehicleQuotes.Core/
COPY source/VehicleQuotes.CreateUser/VehicleQuotes.CreateUser.csproj ./VehicleQuotes.CreateUser/
COPY source/VehicleQuotes.IntegrationTests/VehicleQuotes.IntegrationTests.csproj ./VehicleQuotes.IntegrationTests/
COPY source/VehicleQuotes.RazorTemplates/VehicleQuotes.RazorTemplates.csproj ./VehicleQuotes.RazorTemplates/
COPY source/VehicleQuotes.UnitTests/VehicleQuotes.UnitTests.csproj ./VehicleQuotes.UnitTests/
COPY source/VehicleQuotes.WebApi/VehicleQuotes.WebApi.csproj ./VehicleQuotes.WebApi/
RUN dotnet restore

COPY source/VehicleQuotes.AdminPortal/. ./VehicleQuotes.AdminPortal/
COPY source/VehicleQuotes.Core/. ./VehicleQuotes.Core/
COPY source/VehicleQuotes.CreateUser/. ./VehicleQuotes.CreateUser/
COPY source/VehicleQuotes.IntegrationTests/. ./VehicleQuotes.IntegrationTests/
COPY source/VehicleQuotes.RazorTemplates/. ./VehicleQuotes.RazorTemplates/
COPY source/VehicleQuotes.UnitTests/. ./VehicleQuotes.UnitTests/
COPY source/VehicleQuotes.WebApi/. ./VehicleQuotes.WebApi/

# No ENTRYPOINT here because there's no specific command to run.

This Dockerfile is very similar to the first portion of the one for the admin portal. It essentially creates a .NET development environment. Notice how it doesn’t include an ENTRYPOINT command. This is expected, as this container’s purpose is to allow us to connect to it to run any number of maintenance tasks on demand. There isn’t really any process that it needs to run upon start.

Now, we want to deploy the maintenance container when we do docker compose up -d. To that end, we add the following to the services section in our compose.yaml file:

# ./compose.yaml

services:

  # ...

  maintenance:
    # Similar to the admin-portal, we specify the context and Dockerfile to
    # build the image.
    build:
      context: .
      dockerfile: Dockerfile.Maintenance
    # The maintenance tasks will need to know where to find the database. So we
    # set the connection string environment variable.
    environment:
      - ConnectionStrings__VehicleQuotesContext=Host=db;Database=vehicle_quotes;Username=vehicle_quotes;Password=password
    # "sleep infinity" here makes sure the container doesn't close immediately
    # after starting up, which is the expected behavior when a container has no
    # start up command or has a command that just runs and ends. That is, as
    # opposed to a long running service, like a web app.
    command: sleep infinity

# ...

With these additions, running docker compose up -d again will result in the new maintenance container being created:

$ docker compose up -d

...

[+] Running 3/3
 ✔ Container end-point-blog-dotnet-docker-deploy-admin-portal-1  Running   0.0s
 ✔ Container end-point-blog-dotnet-docker-deploy-db-1            Running   0.0s
 ✔ Container end-point-blog-dotnet-docker-deploy-maintenance-1   Started   0.4s

Now we can finally connect to the brand new maintenance container:

$ docker compose exec maintenance bash
root@14613bcf1756:/source#

Then inspect the status of the migrations:

root@14613bcf1756:/source# dotnet ef migrations list -s ./VehicleQuotes.AdminPortal -p ./VehicleQuotes.Core
Build started...
Build succeeded.

...

20210625212939_AddLookupTables (Pending)
20210625224443_AddUniqueIndexesToLookupTables (Pending)
20210625232816_AddVehicleModelTables (Pending)
20210625234824_AddUniqueIndexesForVehicleModelTables (Pending)
20210627204444_AddQuoteRulesAndOverridesTables (Pending)
20210627213029_AddQuotesTable (Pending)
20210627230039_AddSeedDataForSizesAndBodyTypes (Pending)
20220530192346_FixDatetimeColumn (Pending)
20220605003253_AddIdentityTables (Pending)
20220609233914_AddUserApiKeysTable (Pending)
20240504211307_VariousNullabilityChanges (Pending)
20240606222539_AddQuoteImages (Pending)
root@14613bcf1756:/source#

And run them:

root@14613bcf1756:/source# dotnet ef database update -s ./VehicleQuotes.AdminPortal -p ./VehicleQuotes.Core
Build started...
Build succeeded.

...

Done.
root@14613bcf1756:/source#

With that done, we can now go to the browser again and bring up the http://localhost:8001/Quotes page. Which now looks like this:

An empty—but working—page for listing database records!

Deploying the web API

Now let’s look at how to deploy another of our system’s runtime components: the web API. The process of setting this up is nearly identical to the admin portal’s. After all, both are ASP.NET Core web applications. Really, the only difference is that one serves HTML and the other serves JSON. To the runtime, they are the same type of thing. And sure enough, here’s the web API’s Dockerfile:

# ./Dockerfile.WebApi

FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build

WORKDIR /source

# copy sln and csproj files and restore
COPY source/vehicle-quotes.sln .
COPY source/VehicleQuotes.AdminPortal/VehicleQuotes.AdminPortal.csproj ./VehicleQuotes.AdminPortal/
COPY source/VehicleQuotes.Core/VehicleQuotes.Core.csproj ./VehicleQuotes.Core/
COPY source/VehicleQuotes.CreateUser/VehicleQuotes.CreateUser.csproj ./VehicleQuotes.CreateUser/
COPY source/VehicleQuotes.IntegrationTests/VehicleQuotes.IntegrationTests.csproj ./VehicleQuotes.IntegrationTests/
COPY source/VehicleQuotes.RazorTemplates/VehicleQuotes.RazorTemplates.csproj ./VehicleQuotes.RazorTemplates/
COPY source/VehicleQuotes.UnitTests/VehicleQuotes.UnitTests.csproj ./VehicleQuotes.UnitTests/
COPY source/VehicleQuotes.WebApi/VehicleQuotes.WebApi.csproj ./VehicleQuotes.WebApi/
RUN dotnet restore

# copy everything else
COPY source/VehicleQuotes.AdminPortal/. ./VehicleQuotes.AdminPortal/
COPY source/VehicleQuotes.Core/. ./VehicleQuotes.Core/
COPY source/VehicleQuotes.CreateUser/. ./VehicleQuotes.CreateUser/
COPY source/VehicleQuotes.IntegrationTests/. ./VehicleQuotes.IntegrationTests/
COPY source/VehicleQuotes.RazorTemplates/. ./VehicleQuotes.RazorTemplates/
COPY source/VehicleQuotes.UnitTests/. ./VehicleQuotes.UnitTests/
COPY source/VehicleQuotes.WebApi/. ./VehicleQuotes.WebApi/

# build app
WORKDIR /source/VehicleQuotes.WebApi
RUN dotnet publish -c release -o /app --no-restore

# final image
FROM mcr.microsoft.com/dotnet/aspnet:8.0
WORKDIR /app
COPY --from=build /app ./
ENTRYPOINT ["dotnet", "VehicleQuotes.WebApi.dll"]

Very similar to the admin portal’s Dockerfile. The only differences are the sections for building and running the app towards the end of the file, on account of the directories and resulting DDL name being different. Other than that, they are pretty much identical.

Same story on the Docker Compose side of things:

# ./compose.yaml

services:

  #...

  web-api:
    build:
      context: .
      dockerfile: Dockerfile.WebApi
    restart: always
    # We want to expose this service in a different port than the admin portal,
    # we chose 8002.
    ports:
      - 8002:8080
    # A different set of environment variables, but the structure is the same.
    environment:
      - ASPNETCORE_ENVIRONMENT=Development
      - ConnectionStrings__VehicleQuotesContext=Host=db;Database=vehicle_quotes;Username=vehicle_quotes;Password=password
      - MailSettings__Server=SMTP_SERVER_URL
      - MailSettings__Port=2525
      - MailSettings__SenderName=Vehicle Quotes
      - MailSettings__SenderEmail=info@vehiclequotes.com
      - MailSettings__UserName=SMTP_SERVER_USER_NAME
      - MailSettings__Password=SMTP_SERVER_PASSWORD
    # The entrypoint command is different because the DLL has a different file
    # name.
    entrypoint: ["sh", "-c", "dotnet VehicleQuotes.WebApi.dll"]

#...

It has the same overall setup as the admin portal. This time we’ve chosen a different port, configured a different set of environment variables, and run the web API’s DLL. Other than that, it’s the same. Now docker compose up -d can be run again, and after a while you should see:

$ docker compose up -d

...

[+] Running 4/4
 ✔ Container end-point-blog-dotnet-docker-deploy-web-api-1       Started   0.4s
 ✔ Container end-point-blog-dotnet-docker-deploy-maintenance-1   Running   0.0s
 ✔ Container end-point-blog-dotnet-docker-deploy-db-1            Running   0.0s
 ✔ Container end-point-blog-dotnet-docker-deploy-admin-portal-1  Running   0.0s

The web API has a Swagger UI that should now be accessible at http://localhost:8002/swagger:

Excellent. We now have a very basic implementation of our deployment strategy. We have managed to build and run all the apps we need. We also have a special container with a full development environment that we can use to perform a number of maintenance tasks. However, there are various issues that we still need to address. Let’s do that next.

Looking at the logs

Before that, though, let’s look at the logs being produced by the applications that have been deployed. You can see the logs for the entire system with:

docker compose logs -f

That command produces a single log stream with messages from all the services. To look at the logs of a particular service, all we need to do is specify the service name. For example docker compose logs db -f or docker compose logs web-api -f.

That’s all there is to it as far as logs go. However, if we look at the logs for our current deployment, they reveal the first issue that we need to attend to…

Persisting data protection keys

If we look at the logs of the admin portal or web API containers, we see messages like this:

web-api-1       | warn: Microsoft.AspNetCore.DataProtection.Repositories.FileSystemXmlRepository[60]
web-api-1       |       Storing keys in a directory '/root/.aspnet/DataProtection-Keys' that may not be persisted outside of the container. Protected data will be unavailable when container is destroyed. For more information go to https://aka.ms/aspnet/dataprotectionwarning

As it turns out, the data protection subsystem in ASP.NET Core (which is used for cookies and the like) depends on the app being able to store and persist files. As the message says, this is problematic for containers. Because upon container destruction, which is a usual and expected part of a container’s lifecycle, the internal container’s file system gets wiped out. That is, unless we use Docker volumes to store the files that should persist beyond a particular container’s life. So, in order to get rid of these warnings, that’s exactly what we’re going to have to do.

First we create a data-protection-keys directory in our host machine. And inside it, we create one directory for each of the ASP.NET Core apps. It ends up looking like this:

data-protection-keys/
├── admin-portal
└── web-api

Next we have to make sure the apps in the containers have access to these directories. In order to do that, we need to configure the compose.yaml so that it creates new Docker volumes that are linked to those locations:

# ./compose.yaml

services:
  admin-portal:
    #...
    volumes:
      - ./data-protection-keys/admin-portal:/data-protection-keys/admin-portal
    #...

  web-api:
    #...
    volumes:
      - ./data-protection-keys/web-api:/data-protection-keys/web-api
    #...
#...

This means that, for example, whenever an app in the container references /data-protection-keys/admin-portal; it will actually be accessing the ./data-protection-keys/admin-portal directory in the host machine. Same deal for the web-api one.

We also need to configure the applications themselves so that they store their data protection keys in the locations that we’ve created. For that, we add environment variables for the containers to set the paths:

# ./compose.yaml

services:
  admin-portal:
    #...
    environment:
      - AdminPortalDataProtectionKeysPath=/data-protection-keys/admin-portal
    #...

  web-api:
    #...
    environment:
      - WebApiDataProtectionKeysPath=/data-protection-keys/web-api
    #...
#...

And add the following code to each of the apps’ Program.cs files:

// ./source/VehicleQuotes.AdminPortal/Program.cs

builder.Services.AddDataProtection().PersistKeysToFileSystem(
    new DirectoryInfo(
        builder.Configuration["AdminPortalDataProtectionKeysPath"] ??
            throw new InvalidOperationException("Config setting 'AdminPortalDataProtectionKeysPath' not found.")
    )
);

// ./source/VehicleQuotes.WebApi/Program.cs

builder.Services.AddDataProtection().PersistKeysToFileSystem(
    new DirectoryInfo(
        builder.Configuration["WebApiDataProtectionKeysPath"] ??
            throw new InvalidOperationException("Config setting 'WebApiDataProtectionKeysPath' not found.")
    )
);

There isn’t much to comment about the code really. It’s just some .NET boilerplate to configure that specific detail of the data protection services. It also makes sure that the values are always present. Errors are raised if not.

Now that everything is wired up like that, we can hit docker compose up -d --build and the warning message should be gone if we look at the logs.

Notice how we used the --build option this time. That tells Docker that it needs to rebuild the images from scratch. Image rebuild means running through the Dockerfiles again. And that means running dotnet build again. In short, we have to do this to make sure that the code changes that we made are included in the new builds.

Storing sensitive information with secrets

Another issue that we need to address is how to handle sensitive information like passwords in our config files. So far we’ve been putting them in plain text in compose.yaml. The problem with this is that this file is meant to be pushed to version control and we don’t want passwords in there. The spread of sensitive information like that should be more controlled. Ideally, we’d store them in files that never leave the server where the system is deployed.

Docker Compose has a feature that works just like that. Through Docker Compose secrets, we can create text files outside of the compose.yaml and put the database password and connection string in them. These files will live only in the server, never uploaded to version control.

Let’s create a new secrets directory and create these two files within it:

./secrets/vehicle-quotes-db-connection-string.txt:

Host=db;Database=vehicle_quotes;Username=vehicle_quotes;Password=password

./secrets/vehicle-quotes-db-password.txt:
```
password
```

Ending up looking like this:

secrets/
├── vehicle-quotes-db-connection-string.txt
└── vehicle-quotes-db-password.txt

Now, at the bottom of compose.yaml, we add a new secrets section:

# ./compose.yaml

#...

secrets:
  vehicle-quotes-db-password:
    file: ./secrets/vehicle-quotes-db-password.txt
  vehicle-quotes-db-connection-string:
    file: ./secrets/vehicle-quotes-db-connection-string.txt

Then we include the secrets in the services that need them:

# ./compose.yaml

services:
  admin-portal:
    #...
    secrets:
      - vehicle-quotes-db-connection-string
    #...

  web-api:
    #...
    secrets:
      - vehicle-quotes-db-connection-string
    #...

  db:
    #...
    secrets:
      - vehicle-quotes-db-password
    #...

  maintenance:
    #...
    secrets:
      - vehicle-quotes-db-connection-string
    #...
#...

With this, Docker Compose will add files in the resulting containers under the /run/secrets/ directory with the contents of their referenced secrets. So, for example, in the case of the admin-portal container, a /run/secrets/vehicle-quotes-db-connection-string file will be created in the container’s internal file system, with the same contents as the ./secrets/vehicle-quotes-db-connection-string.txt file. Similar thing for the others.

Now that the secrets are materialized as files within the containers, let’s see how we put them to use.

For the admin-portal container, we injected the vehicle-quotes-db-connection-string secret into it. This file contains the database connection string. And we need to pass that to the running app via an environment variable. In order to do so, we can change the entrypoint command to this:

# ./compose.yaml

services:
  admin-portal:
  #...
  entrypoint: [
    "sh", "-c",
    "export ConnectionStrings__VehicleQuotesContext=$(cat /run/secrets/vehicle-quotes-db-connection-string) &&
    dotnet VehicleQuotes.AdminPortal.dll"
  ]

#...

We’ve modified the command to set the ConnectionStrings__VehicleQuotesContext to the contents of the /run/secrets/vehicle-quotes-db-connection-string file. The export command defines the environment variable for this particular command; and the $(cat ...) part returns the contents of the file.

We should also remove the ConnectionStrings__VehicleQuotesContext variable from the admin-portal service’s environment section.

For the web-api service, we do the same thing. The only difference is that its entrypoint command calls for VehicleQuotes.WebApi.dll instead of VehicleQuotes.AdminPortal.dll.

For the db service, the setup is a little different. A little simpler in fact. The official PostgreSQL image has a shortcut for specifying the database user password via Docker Compose secrets. All we need to do is define this new POSTGRES_PASSWORD_FILE environment variable and remove the POSTGRES_PASSWORD one:

# ./compose.yaml

services:
  #...

  db:
  #...
  environment:
    - POSTGRES_PASSWORD_FILE=/run/secrets/vehicle-quotes-db-password
    # - POSTGRES_PASSWORD=password     <-- remove this one
#...

Try docker compose up -d again and test the apps. Everything should still work well.

In the maintenance container, we also have to remove the ConnectionStrings__VehicleQuotesContext environment variable. That unfortunately means that the connection string will no longer be automatically available for us to run database related tasks. Just like other containers, it will be in a /run/secrets/vehicle-quotes-db-connection-string file. So, whenever we want to interact with the database, like when running migrations, we need to manually export the variable. Something like this:

$ docker compose exec maintenance bash
root@2732a06871c0:/source# export ConnectionStrings__VehicleQuotesContext=$(cat /run/secrets/vehicle-quotes-db-connection-string)
root@2732a06871c0:/source# dotnet ef migrations list -s ./VehicleQuotes.AdminPortal -p ./VehicleQuotes.Core
Build started...
Build succeeded.

...

Parameterizing `compose.yaml` to support multiple deployment environments

A common requirement when deploying applications is to be able to do so in multiple environments. There’s generally a “live” or “production” environment where the system runs and end users access it. There can also be others: staging, test, development, etc. Ideally, we’d use the same set of Docker and Compose files, with slight changes, in order to deploy variants of the system depending on the environment.

Some settings like ports, passwords, or SMTP credentials are the types of things that usually vary per environment. Luckily for us, Docker Compose supports .env files that can be used to parameterize certain aspects of the compose.yaml file.

We can extract the values that vary from our compose.yaml file, and put them in a separate .env file that looks like this:

ADMIN_PORTAL_PORT=8001
WEB_API_PORT=8002
DB_PORT=5432
POSTGRES_DB=vehicle_quotes
POSTGRES_USER=vehicle_quotes
MailSettings__Server=SMTP_SERVER_URL
MailSettings__Port=2525
MailSettings__SenderName=Vehicle Quotes
MailSettings__SenderEmail=info@vehiclequotes.com
MailSettings__UserName=SMTP_SERVER_USER_NAME
MailSettings__Password=SMTP_SERVER_PASSWORD

Here we have the ports that we want to expose the various services on, the database name and user name, and some email delivery settings. This file wont be pushed to version control, and each deployment will have its own version of it.

The compose.yaml file can reference the values defined in the .env file using the following syntax: ${VAR_NAME}. Here’s how we change our compose.yaml to take advantage of the settings defined in the .env file:

# ./compose.yaml

services:
  admin-portal:
    #...
    ports:
      - ${ADMIN_PORTAL_PORT}:8080
    #...

  web-api:
    #...
    ports:
      - ${WEB_API_PORT}:8080
    #...
    environment:
      #...
      - MailSettings__Server=${MAIL_SETTINGS_SERVER}
      - MailSettings__Port=${MAIL_SETTINGS_PORT}
      - MailSettings__SenderName=${MAIL_SETTINGS_SENDER_NAME}
      - MailSettings__SenderEmail=${MAIL_SETTINGS_SENDER_EMAIL}
      - MailSettings__UserName=${MAIL_SETTINGS_USER_NAME}
      - MailSettings__Password=${MAIL_SETTINGS_PASSWORD}
    #...

  db:
    #...
    ports:
      - ${DB_PORT}:5432
    #...
    environment:
      - POSTGRES_DB=${POSTGRES_DB}
      - POSTGRES_USER=${POSTGRES_USER}
      #...
    #...

#...

Once again, we can run docker compose up -d and everything should be fine.

Waiting for the database to be ready

Sometimes some services need to wait for others to come online before they can start up. A common scenario is to wait for the database to be ready before running apps that depend on it. We can do that in Docker Compose thanks to the depends_on and healthcheck settings. We can update our compose.yaml so that the admin portal and web API services only start after the database is up and running and ready to receive requests. Here’s how:

# ./compose.yaml

services:
  admin-portal:
    #...
    # This specifies that the admin-portal service depends on the db to be
    # healthy in order to start up. The db service's healthcheck setting is how
    # it determines whether it is healthy or not.
    depends_on:
      db:
        condition: service_healthy

  web-api:
    #...
    # Exact same setting as the one in admin-portal.
    depends_on:
      db:
        condition: service_healthy

  db:
    #...
    # Here we configure this service to offer a healthcheck that other services
    # can use to determine if it's ready to be depended upon. In this case, we
    # leverage PostgreSQL's pg_isready tool. It is called by the specified
    # timeout, at the specified interval, and with as many retries as specified
    # in the settings below.
    healthcheck:
      test: "pg_isready -U ${POSTGRES_USER}"
      interval: 10s
      timeout: 5s
      retries: 5

#...

The most interesting part is the healthcheck setting in the db service which leverages a PostgreSQL-specific tool to check whether the database is ready. Other software will have other methods to do checks like this, but PostgreSQL’s is thankfully pretty straightforward.

Serving the apps with NGINX

Another common pattern for serving web applications is to use NGINX as a reverse proxy that funnels HTTP traffic coming from the internet into the application. As you may have noticed, we haven’t talked about HTTPS so far. This aspect is something that can be elegantly handled by NGINX as well. In this last section we see how we can set up NGINX to expose our apps to the world.

First, we need to create some custom items in the default NGINX configuration file. Its default location will depend on the OS version and flavor you installed NGINX on, but for Rocky Linux 9, it usually lives in /etc/nginx/nginx.conf. First, we need to declare our upstream servers that will point to the ports where the web API and admin portals are listening:

upstream admin_portal {
    server localhost:8001;
}

upstream web_api {
    server localhost:8002;
}

Then, we will add a server that listens in the standard port 80 and proxies the /admin and /api URLs to the admin_portal and web_api upstream servers respectively:

server {
    listen 80;
    server_name vehiclequotes.com;

    location /admin {
        proxy_pass http://admin_portal;
        proxy_http_version 1.1;
        proxy_set_header Connection keep-alive;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    location /api {
        proxy_pass http://web_api;
        proxy_http_version 1.1;
        proxy_set_header Connection keep-alive;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Those are the basic settings needed to serve our applications in a single domain through different locations. This allows having a live frontend application that will eventually consume our API endpoints that lives in the same server and domain, avoiding possible Cross-Origin Resource Sharing (CORS) issues when establishing the workflow between the different web applications.

If we also have an SSL certificate that we want to use to securely serve our apps, we can use a free service such as Let’s Encrypt to create it. Once we’re in possession of the certificate files and placed them on the server, we need to perform a few extra tweaks to our nginx.conf file.

First, let’s make our server listen on port 443 (HTTPS), and point to our .cer and .key files in the updated entry:

server {
    listen 443 ssl;
    server_name vehiclequotes.com;
    ssl_certificate /etc/certs/live/vehiclequotes.com/fullchain.cer;
    ssl_certificate_key /etc/certs/live/vehiclequotes.com/vehiclequotes.com.key;

    location /admin {
        proxy_pass http://admin_portal;
        proxy_http_version 1.1;
        proxy_set_header Connection keep-alive;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    location /api {
        proxy_pass http://web_api;
        proxy_http_version 1.1;
        proxy_set_header Connection keep-alive;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Second, let’s add a new server for port 80 (non-HTTPS) that will redirect permanently to our new secure location:

server {
    listen 80;
    server_name vehiclequotes.com;
    return 301 https://$server_name$request_uri;
}

Finally, let’s restart the NGINX service to apply our changes. The command that restarts the service will vary depending on the OS we’re running on the server. For Rocky Linux 9, we can do that by running the command sudo systemctl restart nginx.

You can find the resulting nginx.conf file in the project’s repo in GitHub.

That’s all for now

And that’s it! In this article, we’ve seen how we can approach deploying a .NET system into production using Docker Compose.

We saw how to organize the code and configuration files using Git submodules. We addressed a few important edge cases and gotchas like properly configuring Data Protection keys, having a container for performing maintenance tasks, ensuring certain files persist across restarts, and bringing up services in a certain order via depends_on settings.

We even saw how to allow our web applications to be accessible to the outside world with NGINX through a set of reverse proxying rules, and as a bonus, to be securely served with SSL.

Uploading multiple files in a single request in an ASP.NET Core application

2024-07-04T00:00:00+00:00

We recently developed a web application for maintaining an ecommerce site’s product catalog. Unsurprisingly, one of the features involved the management of product images. Specifically, we wanted to create a page where all the images of a given product were displayed and new ones could be uploaded.

In addition to that, this wasn’t a single-page application. We weren’t using a JavaScript framework and instead were relying on regular server-rendered views with ASP.NET Razor Pages. Even so, we wanted to create a user experience with a good balance of usability and development complexity. So, we decided to create the capability of uploading several image files at once within a single request — that is, within a single HTML form submission.

In this article, I’m going to describe the solution that we came up with in order to make this happen.

To demonstrate the approach, I’ll use a sample ASP.NET Core solution that I’ve been building out throughout several blog posts. Its main feature is calculating the value of used cars and offering quotes for them. The system stores all generated quotes as database records. Given this context, we’ll add the functionality to upload image files for a given quote.

Let’s get started.

Each section is accompanied by a given commit that includes all the changes discussed within them. Feel free to check those as well to see how the finished product takes shape step by step.

Where to store the images

Commit: d65fd8.

The first decision that we have to make is where to store the image files. There are various options afforded to us by ASP.NET and our underlying PostgreSQL database, but here’s how we’re going to do it: we’ll store records in the database that represent the images associated with each quote. These records won’t actually contain image binary data, however, only references to them in the form of their file names. The files themselves will be stored in the project’s wwwroot directory. That way they can be easily served to browsers, as everything inside that directory is accessible to clients.

Starting off with a fresh ASP.NET Core Web App (with Razor Pages) project, here’s how that’s done.

First we have to make sure that the app has static file serving enabled. This is done with this function call in the Program.cs file:

// VehicleQuotes.AdminPortal/Program.cs

app.UseStaticFiles();

Next, we need to create the new database structure that will store the image records. In our app, we already have a Quote entity, backed by a corresponding quotes table in the database. The plan is to create a new QuoteImage entity and update Quote to “have many” of them.

Here’s what the new class looks like:

// VehicleQuotes.WebApi/Models/QuoteImage.cs

namespace VehicleQuotes.WebApi.Models;

public class QuoteImage
{
    public int ID { get; set; }
    public string FileName { get; set; } = default!;

    public int QuoteId { get; set; }
    public Quote Quote { get; set; } = default!;
}

We also have to add this navigation property to the Quote entity’s class:

// VehicleQuotes.WebApi/Models/Quote.cs

public class Quote
{
    // ...
    public ICollection<QuoteImage> QuoteImages { get; set; } = [];
}

And finally declare its corresponding DbSet in our VehicleQuotesContext class:

// VehicleQuotes.WebApi/Data/VehicleQuotesContext.cs

public class VehicleQuotesContext : IdentityUserContext<IdentityUser>
{
    // ...
    public DbSet<QuoteImage> QuoteImages { get; set; }
    // ...
}

Now all that’s left is to create and run the migration with these commands:

dotnet ef migrations add AddQuoteImages
dotnet ef database update

With that, our model and database are ready to store references to image files for the quotes on record.

vehicle_quotes=# \d quote_images;
                          Table "public.quote_images"
  Column   |  Type   | Collation | Nullable |             Default
-----------+---------+-----------+----------+----------------------------------
 id        | integer |           | not null | generated by default as identity
 file_name | text    |           | not null |
 quote_id  | integer |           | not null |
Indexes:
    "pk_quote_images" PRIMARY KEY, btree (id)
    "ix_quote_images_quote_id" btree (quote_id)
Foreign-key constraints:
    "fk_quote_images_quotes_quote_id" FOREIGN KEY (quote_id) REFERENCES quotes(id) ON DELETE CASCADE

Check out this commit to see the migration.

A Razor Page for uploading many files

Commit: feaa57.

Now we need to create a new Razor Page for uploading the image files. The new page will be reachable at this route: quotes/edit/{id}. {id} is the identifier of the quote record for which images will be uploaded. To that end, we’ll create the usual pair of files that make up a Razor Page: The PageModel at VehicleQuotes.AdminPortal/Pages/Quotes/Edit.cshtml.cs and the View (i.e. template) at VehicleQuotes.AdminPortal/Pages/Quotes/Edit.cshtml.

We’ll start with the PageModel. Let’s create the new file with these contents:

// VehicleQuotes.AdminPortal/Pages/Quotes/Edit.cshtml.cs

using Microsoft.AspNetCore.Mvc;
using Microsoft.AspNetCore.Mvc.RazorPages;
using Microsoft.EntityFrameworkCore;
using VehicleQuotes.WebApi;
using VehicleQuotes.WebApi.Models;

namespace VehicleQuotes.AdminPortal.Pages.Quotes;

public class EditModel : PageModel
{
    // This page will interact with the database so we need an instance of the
    // DbContext.
    private readonly VehicleQuotesContext _context;
    private readonly string _imagesPath;

    public EditModel(VehicleQuotesContext context)
    {
        _context = context;
        // Like we discussed, we will store the image files in the app's wwwroot
        // directory. Specifically, ~/wwwroot/uploads. It'd be better to
        // substitute this hard coded string with a config value from
        // appsettings.json.
        _imagesPath = "/path/to/wwwroot/uploads";
    }

    // The BindProperty attribute in this property makes it so that when a form
    // in this page is submitted, the framework takes the request payload and
    // uses it to populate this property. In our case, the form will submit a
    // set of files. We'll see how a little bit later.
    [BindProperty]
    public IEnumerable<IFormFile>? ImageFiles { get; set; }

    // This GET handler method does little else than render the corresponding
    // template when a browser requests the quotes/edit/{id} URL.
    // That is, the one defined in
    // VehicleQuotes.AdminPortal/Pages/Quotes/Edit.cshtml.
    public IActionResult OnGet(int id) => Page();

    // This is the method that handles POST requests on this page. It inspects
    // the incoming request payload and uses it to produce new quote image
    // records as well as save the uploaded files under ~/wwwroot/uploads.
    public async Task<IActionResult> OnPostSaveAsync(int id)
    {
        // First, try to find the quote record by the given id and...
        var quote = await FindQuote(id);

        // ...if it's not found, return a proper 404 response.
        if (quote == null)
        {
            return NotFound();
        }

        // If images were uploaded...
        if (ImageFiles is not null)
        {
            // ...iterate over each one of them...
            foreach (var imageFile in ImageFiles)
            {
                // ...store them in the file system...
                var imageFileName = await SaveImageFile(imageFile);
                // ...and add a new corresponding record to the quote's related
                // image records.
                quote.QuoteImages.Add(new() { FileName = imageFileName });
            }
        }

        // Finally, submit the database changes...
        await _context.SaveChangesAsync();

        // ...and effectively reload the page.
        return RedirectToPage("./Edit", new { Id = id });
    }

    // This method queries the database to find quote records by id.
    private async Task<Quote?> FindQuote(int id) =>
        await _context.Quotes
            .FirstOrDefaultAsync(m => m.ID == id);

    // This method has some mostly straightforward code to save the files from
    // the incoming HTTP request into the file system.
    private async Task<string> SaveImageFile(IFormFile fileToSave)
    {
        var extension = Path.GetExtension(fileToSave.FileName).ToLowerInvariant();
        // The only interesting thing that we have to keep in mind is that we
        // need to come up with a naming scheme that won't produce conflicts.
        // Here, we've taken the naive approach of just using
        // Path.GetRandomFileName to create randomized names.
        var fileName = $"{Path.GetRandomFileName()}{extension}";
        var filePath = Path.Combine(_imagesPath, fileName);

        using var stream = System.IO.File.Create(filePath);
        await fileToSave.CopyToAsync(stream);

        return fileName;
    }
}

Next we need to create the View. The job of this View is to define an HTML form that supports uploading many files. We have two options for this. The first option involves creating a form with a single file input element that uses the multiple attribute. That will allow the user to pick many files at the same time using their operating system’s file picker dialog box, looking something like this:

Option number two involves defining a form with multiple file input elements with each accepting one file. The number of <input> elements cannot be static however. We want to allow users to upload as many images as they want. So, we’d also need some JavaScript to dynamically add new <input> elements as the user picks more and more files.

Both options are valid, and deciding on one over the other will depend on the user experience you’re interested in creating and the amount of effort you can devote to it. For us, we went with option #2: A dynamic set of individual file input elements.

Before making it fully dynamic, though, let’s go through the exercise of building it with a static number of <input> elements as a first step. I think that’ll help clarify the process better.

So here’s the cshtml file with three <input type="file"> elements:

<!-- VehicleQuotes.AdminPortal/Pages/Quotes/Edit.cshtml -->

@page "{id}"
@model VehicleQuotes.AdminPortal.Pages.Quotes.EditModel

@{
    ViewData["Title"] = "Details";
}

<h1>Edit</h1>

<h4>Add Images</h4>
<hr />

<div class="row">
    <!--
    The form doesn't need too many details. We do have to make sure to set its
    enctype attribute to multipart/form-data to allow it to include files.
    -->
    <form method="post" enctype="multipart/form-data">
        <div id="quote-images-container" class="row">
            <div class="col-3 mb-3">
                <!--
                These are regular old HTML file input elements. They all have
                the same name, which matches that of the property in the
                PageModel that we want the framework to populate.
                That is, "ImageFiles".
                -->
                <input
                    type="file" accept="image/*"
                    id="ImageFiles_0" name="ImageFiles" class="form-control"
                >
            </div>

            <div class="col-3 mb-3">
                <input
                    type="file" accept="image/*"
                    id="ImageFiles_1" name="ImageFiles" class="form-control"
                >
            </div>

            <div class="col-3 mb-3">
                <input
                    type="file" accept="image/*"
                    id="ImageFiles_2" name="ImageFiles" class="form-control"
                >
            </div>
        </div>

        <div class="form-group mb-3">
            <!-- This button submits the form. -->
            <button
                type="submit"
                asp-page-handler="Save"
                class="btn btn-primary"
            >
                Save
            </button>
        </div>
    </form>
</div>

<div class="row">
    <a asp-page="./Index">Back to List</a>
</div>

This is how we design our form so that when submitted, ASP.NET knows that it has to populate ImageFiles (the collection of IFormFiles that we have defined in the PageModel) with the uploaded files. All we have to do is make sure that all the input elements share the same name attribute, and that said name is the same as the target property. ImageFiles in this case. The elements’ id attribute is actually superfluous.

If you’re following along and want to run this next part on your environment, you’ll need a quote record to be able to bring it up in the page we’ve been building and add images to it. You could do so by running the VehicleQuotes.WebApi project with dotnet run and POST to it using curl:
curl --location 'http://localhost:8001/api/Quotes' \
--header 'Content-Type: application/json' \
--data '{
 "year": "2020",
 "make": "Mazda",
 "model": "Mazda 3",
 "bodyType": "Sedan",
 "size": "Compact",
 "itMoves": true,
 "hasAllWheels": true,
 "hasAlloyWheels": true,
 "hasAllTires": true,
 "hasKey": true,
 "hasTitle": true,
 "requiresPickup": true,
 "hasEngine": true,
 "hasTransmission": true,
 "hasCompleteInterior": true
}'
Then connect to the database however you like to and run the following query to learn its id:
select * from quotes;
Use that id in the URL to the page.

At this point, the feature should work. If you run the app with dotnet run, and navigate to the page we’ve been building at http://localhost:{YOUR_PORT}/Quotes/Edit/{QUOTE_ID}, you should see something like this:

Pick three image files, hit the blue “Save” button and the files should be uploaded into the wwwroot/uploads diretory…

$ ls wwwroot/uploads
sqyjb1ui.qnv.png  x1c4s4fj.rqb.png  zjaym20f.wcx.png

…and three new records should be created on the quote_images table:

vehicle_quotes=# select * from quote_images;
 id |    file_name     | quote_id
----+------------------+----------
  1 | sqyjb1ui.qnv.png |        1
  2 | zjaym20f.wcx.png |        1
  3 | x1c4s4fj.rqb.png |        1
(3 rows)

Uploading any number of files

Commit: 5aaff6.

Now that we’ve seen the general approach and key details of working with the framework to upload multiple files, it’s more clear what we need to do to support any number of files. Like I said, what we need is “some JavaScript to dynamically add new <input> elements as the user picks more and more files”.

Let’s start by removing the three file input elements (that is, the whole <div id="quote-images-container"> element) and replacing them with this:

<!-- VehicleQuotes.AdminPortal/Pages/Quotes/Edit.cshtml -->

<!-- ... -->

<div id="quote-images-container" class="row">
    <div class="col-3 mb-3">
        <input type="file" accept="image/*" id="new-quote-image-file" class="form-control" />
    </div>
</div>

<template id="quote-image-template">
    <div class="col-3 mb-3">
        <input type="file" accept="image/*" name="" id="" class="form-control" />
    </div>
</template>

We’ve reduced the three file input elements to just one and, most importantly, we’ve added a template that we’ll use to add new ones. If you’re not familiar, you can read more about templates in MDN. But the basic idea is that this element will not actually be visible in the page as it is. Instead, we will write some JavaScript that uses it to create new file inputs as the user fills out the currently present ones.

We will create a new JavaScript file that will contain that logic. Let’s first add a reference to it in the View:

<!-- VehicleQuotes.AdminPortal/Pages/Quotes/Edit.cshtml -->

<!-- ... -->

@section scripts {
    <script src="~/js/quotes.new-images.js" asp-append-version="true"></script>
}

And here’s what that JavaScript file looks like:

// VehicleQuotes.AdminPortal/wwwroot/js/quotes.new-images.js

window.addEventListener('DOMContentLoaded', _event => {
    // First of all we attach the "addNewFileInput" function to the change event
    // of the default file input element.
    const newFileInput = document.querySelector("#new-quote-image-file");
    newFileInput.addEventListener("change", addNewFileInput);

    // We use this counter to set the ids of the file input elements as they get
    // added.
    let imageInputCount = 0;

    // This function, which handles the file input's change event, updates the
    // input element that triggered the event so that it can be submitted with
    // the form and also creates a new empty file input ready to accept a new
    // file. See the comments on the "convertToSubmittable" and
    // "createNewFileInput" functions to learn how.
    function addNewFileInput(event) {
        convertToSubmittable(event.target)
        createNewFileInput();

        imageInputCount++;
    }

    // This function takes the file input element that the user interacted with
    // and changes its name to "ImageFiles". That way, when the form is
    // submitted, ASP.NET knows that it needs include the file from this
    // element when populating the "ImageFiles" property.
    // We also detach the change event handler, because once a file
    // is selected on it, we don't want new inputs being created if the user
    // changes the file. It is only the new empty file input that will be
    // responsible of creating another new empty one once it is filled.
    function convertToSubmittable(fileInput) {
        fileInput.setAttribute("name", "ImageFiles");
        fileInput.setAttribute("id", `ImageFiles_${imageInputCount}`);

        fileInput.removeEventListener("change", addNewFileInput);
    }

    // Here we leverage the template to create and attach a new empty file input
    // element. Importantly, we set the addNewFileInput to handle its change
    // event. That way, when the user picks a file to put in the newly created
    // file input, the whole process runs again to get the element ready for
    // submission and create a new empty one.
    function createNewFileInput() {
        const template = document.querySelector("#quote-image-template");
        const clone = template.content.cloneNode(true);

        let fileInput = clone.querySelector("input");
        fileInput.setAttribute("id", "new-quote-image-file");

        appendToContainer(clone);

        fileInput.addEventListener("change", addNewFileInput);
    }

    function appendToContainer(element) {
        const container = document.querySelector("#quote-images-container");
        container.appendChild(element);
    }
});

So the overall strategy is something like this: At page load, we start with an empty file input. Once a file is selected for it, the input gets updated so that it is ready for submission. Then a new empty file input is created which will serve the same purpose as the initial one. This creates a “cycle” where a single new empty file input is always created as a result of users picking files, allowing them to keep adding files one by one.

So the page starts looking like this:

If we pick a file, the page turns into this:

Pick another, and now it looks like this:

And so on.

Now users can pick as many files as they want, hit the “Save” button, and everything just works.

And with that, the initial promise of this blog post is fulfilled. However, there are a few more improvements and features that we can add.

Validating the uploaded images

Commit: 06b547.

Something obvious that’s missing is validation. Right now, our app allows users to upload anything they want. We can limit that a bit with a couple of validation attributes.

For example, here’s one that only allows files with the most common image extensions:

// VehicleQuotes.AdminPortal/Validation/AllFilesHaveImageFileExtensionAttribute.cs

using System.ComponentModel.DataAnnotations;

namespace VehicleQuotes.AdminPortal.Validation;

[AttributeUsage(AttributeTargets.Property)]
public class AllFilesHaveImageFileExtensionAttribute : ValidationAttribute
{
    // These are the file extensions that we accept.
    private static readonly string[] imageExtensions =
        [".png", ".jpg", ".jpeg", ".gif", ".bmp"];

    // This method is called by the framework when processing the request to
    // validate the incoming payload.
    protected override ValidationResult? IsValid(
        object? value, ValidationContext context
    ){
        if (value == null) return ValidationResult.Success;

        ThrowIfTypeIsNotSupported(value);

        if (!CheckIfIsValid(value))
            return new ValidationResult(GetErrorMessage());

        return ValidationResult.Success;
    }

    // In this method we make sure that the attribute is only applied to
    // properties of type IEnumerable<IFormFile>. It only works with those.
    private void ThrowIfTypeIsNotSupported(object value)
    {
        if (value is not IEnumerable<IFormFile>)
        {
            throw new ArgumentException($"{GetType().Name} only works with properties of type IEnumerable<IFormFile>.");
        }
    }

    // Here we check that all the files in the collection meet the criteria
    // defined in "HasImageExtension".
    private static bool CheckIfIsValid(object value)
    {
        var files = (IEnumerable<IFormFile>)value;
        return files.All(HasImageExtension);
    }

    // This is the core of our validation logic. This method checks that each
    // given file has one of the extensions from the "imageExtensions" array.
    private static bool HasImageExtension(IFormFile file)
    {
        var extension = Path.GetExtension(file.FileName).ToLowerInvariant();

        if (string.IsNullOrEmpty(extension) || !imageExtensions.Contains(extension))
        {
            return false;
        }

        return true;
    }

    // Here we produce an error message that's easily understandable for users.
    // Letting them know which extensions our app supports.
    private static string GetErrorMessage()
    {
        var allowedExtensions = string.Join(", ", imageExtensions);
        return $"Only the following file extensions are allowed: {allowedExtensions}.";
    }
}

The attribute can be applied to the ImageFiles property in the PageModel like this:

// VehicleQuotes.AdminPortal/Pages/Quotes/Edit.cshtml.cs

[BindProperty]
[AllFilesHaveImageFileExtension] // <- Here's the new attribute.
public IEnumerable<IFormFile>? ImageFiles { get; set; }

And here’s another one that checks that none of the incoming files are empty:

// VehicleQuotes.AdminPortal/Validation/AllFilesAreNotEmptyAttribute.cs

using System.ComponentModel.DataAnnotations;

namespace VehicleQuotes.AdminPortal.Validation;

[AttributeUsage(AttributeTargets.Property)]
public class AllFilesAreNotEmptyAttribute : ValidationAttribute
{
    protected override ValidationResult? IsValid(object? value, ValidationContext context)
    {
        if (value == null) return ValidationResult.Success;

        ThrowIfTypeIsNotSupported(value);

        if (!CheckIfIsValid(value)) return new ValidationResult(GetErrorMessage());

        return ValidationResult.Success;
    }

    private void ThrowIfTypeIsNotSupported(object value)
    {
        if (value is not IEnumerable<IFormFile>)
        {
            throw new ArgumentException($"{GetType().Name} only works with properties of type IEnumerable<IFormFile>.");
        }
    }

    // This is the only part that's meaningfully different from the other
    // validation attribute we just wrote. Here, we simply check that the
    // uploaded file's length attribute is greater than zero. In other words:
    // we check that the file isn't empty.
    private static bool CheckIfIsValid(object value)
    {
        var files = (IEnumerable<IFormFile>)value;
        return files.All(file => file.Length > 0);
    }

    private static string GetErrorMessage() => "Some of the selected files appear to be empty.";
}

It can be applied similarly:

// VehicleQuotes.AdminPortal/Pages/Quotes/Edit.cshtml.cs

[BindProperty]
[AllFilesHaveImageFileExtension]
[AllFilesAreNotEmpty] // <- Here it is.
public IEnumerable<IFormFile>? ImageFiles { get; set; }

The last step is to actually trigger the input validation when the request is received. This can be done with this update to the OnPostSaveAsync handler:

public async Task<IActionResult> OnPostSaveAsync(int id)
{
    var quote = await FindQuote(id);

    if (quote == null)
    {
        return NotFound();
    }

+    if (!ModelState.IsValid)
+    {
+        return Page();
+    }

    if (ImageFiles is not null)
    {
        foreach (var imageFile in ImageFiles)
        {
            var imageFileName = await SaveImageFile(imageFile);
            quote.QuoteImages.Add(new() { FileName = imageFileName });
        }
    }

    await _context.SaveChangesAsync();

    return RedirectToPage("./Edit", new { Id = id });
}

Oh! And don’t forget the necessary using statement near the top of the file:

using VehicleQuotes.AdminPortal.Validation;

We probably also want to add a validation summary in the View so that an error message is shown whenever the validation fails. We put it inside the <form> element:

<form method="post" enctype="multipart/form-data">
+    <div asp-validation-summary="All" class="text-danger"></div>

    <!-- ... -->
</form>

Attempting to upload invalid files produces something like this:

There are more complex validations that can be done. Check out ASP.NET’s official docs to learn more.

Displaying the images

Commit: d15d35

So far we’ve added support for uploading images. We should have a way of displaying the ones that are already in the system as well. Luckily this is quite a straightforward problem to solve. To do that, we need to update our PageModel’s GET request handler so that it fetches the corresponding quote record from the database. Also, we need to create a new property to store this quote, so that it is accessible in the View.

All in all, the PageModel has to be updated to look like this:

// VehicleQuotes.AdminPortal/Pages/Quotes/Edit.cshtml.cs

// ...

public class EditModel : PageModel
{
    // ...

    // Here's the property we talked about. We need it so that the View can
    // access the quote record fetched in the handler.
    public Quote Quote { get; set; } = default!;

    // ...

    // public IActionResult OnGet(int id) => Page(); <- This gets removed.

    // This is our new GET request handler.
    public async Task<IActionResult> OnGetAsync(int id)
    {
        var quote = await FindQuote(id);

        // Return a 404 if the quote doesn't exist.
        if (quote == null)
        {
            return NotFound();
        }

        // Don't forget to store the quote in the property.
        Quote = quote;

        return Page();
    }

    // ...

    // We also need to update the query so that the image records are also
    // fetched, along with the particular quote we're interested in.
    private async Task<Quote?> FindQuote(int id) =>
        await _context.Quotes
            .Include(m => m.QuoteImages) // <- We need to add this.
            .FirstOrDefaultAsync(m => m.ID == id);
}

The View will need a way to calculate the images’ URLs. We could put that logic in the .cshtml template itself, but a simple property on QuoteImage would be a bit cleaner. Here’s what it would look like:

// VehicleQuotes.WebApi/Models/QuoteImage.cs

namespace VehicleQuotes.WebApi.Models;

public class QuoteImage
{
    // ...

    // Taking advantage of the fact that we know that the image files are stored
    // in the wwwroot/uploads directory, we can construct a URL that can be used
    // in the src attribute of <img> elements.
    public string Url => $"~/uploads/{FileName}";
}

Finally, in the View, we update the HTML template to iterate over the loaded quote’s QuoteImages and render them. Let’s add this code right above the “Add Images” header:

<!-- VehicleQuotes.AdminPortal/Pages/Quotes/Edit.cshtml -->

<h4>Existing Images</h4>
<hr />

<div class="row mb-4">
    @foreach (var image in Model.Quote.QuoteImages)
    {
        <div class="col-3">
            <img
                src="@image.Url"
                asp-append-version="true"
                alt="Quote image"
                class="mx-auto d-block w-75"
            />
        </div>
    }
</div>

The page should now look something like this:

Deleting the images

Commit: 2526f2.

The last piece of the puzzle that’s missing from our image file management screen is the ability to delete them. The Razor Pages way of doing this is with a new button that submits a form to a named page handler.

The form and button combo looks like this and can be added right below the <img> element that we added inside the foreach loop in the previous section:

<!-- VehicleQuotes.AdminPortal/Pages/Quotes/Edit.cshtml -->

<form method="post">
    <button
        type="submit"
        asp-page-handler="DeleteImage"
        asp-route-imageId="@image.ID"
        class="btn btn-outline-danger mx-auto d-block"
    >
        Delete
    </button>
</form>

The asp-page-handler attribute tells the framework which handler to call when this form gets submitted. We will write that handler shortly! The asp-route-imageId on the other hand, defines a parameter to pass to the handler.

Now for the handler, here’s the code that we need to add to our PageModel:

// VehicleQuotes.AdminPortal/Pages/Quotes/Edit.cshtml.cs

// This is the handler method. Its name matches the asp-page-handler attribute
// on the button that triggers the form submission. The id parameter comes from
// the URL route (i.e. quotes/edit/{id}). The imageId parameter comes from the
// asp-route-imageId attribute in the form's submit button.
public async Task<IActionResult> OnPostDeleteImageAsync(int id, int imageId)
{
    // Find the quote record by id...
    var quote = await FindQuote(id);

    // ...and return a 404 if no quote is found.
    if (quote == null)
    {
        return NotFound();
    }

    // Get the image referenced by imageId. That is, the one for which the
    // delete button was pressed.
    var imageToDelete = quote.QuoteImages.FirstOrDefault(i => i.ID == imageId);

    // Again, return a 404 if no such image record exists.
    if (imageToDelete == null)
    {
        return NotFound();
    }

    // If all goes well, we delete the image file...
    DeleteImageFile(imageToDelete.FileName);
    // ...and also delete the corresponding database record.
    _context.QuoteImages.Remove(imageToDelete);
    await _context.SaveChangesAsync();

    // And finally reload the page.
    return RedirectToPage("./Edit", new { Id = id });
}

// This method uses the typical .NET library features for deleting a file.
private void DeleteImageFile(string fileName)
{
    var filePath = Path.Combine(_imagesPath, fileName);
    System.IO.File.Delete(filePath);
}

With that, each image rendered in the page should now have a working “Delete” button that looks like this:

Alright! Now we have a fully functional image handling page where we can view, add and remove images associated with a particular entity in our system. The neat part is that users can select as many files as they want and upload them all at the same time within a single form submission. We did all this with Razor Pages, using framework features, and a little bit of JavaScript.

How to validate record uniqueness in ASP.NET

2024-05-28T00:00:00+00:00

In ASP.NET, the System.ComponentModel.DataAnnotations namespace includes many attributes that can be used to instruct the framework to perform basic validation tasks for us. There are built-in validators for ensuring that certain fields are present, that they meet character length limits, that they don’t exceed or fall short of certain amounts, that they match certain formats, and more.

One omission however, is checking for record uniqueness: making sure that no other record in the entity’s underlying persistent data storage has the same “name”, or the same “code”, or the same “any other field”.

In this article, we’re going to try to address this shortcoming by implementing this type of uniqueness validation ourselves. We will see two approaches: a simpler solution using a database index, and a more flexible one using a custom validation attribute. Let’s get started.

Throughout this article, I will be using a demo Web API application for code examples. If you’d like to see what the final implementation looks like, you can find all the code on GitHub.

The API is about calculating quotes for used vehicles based on their condition. It runs on .NET 8 and uses a PostgreSQL database, to which it connects using Entity Framework Core. As such it has various endpoints, entities and tables related to vehicle information like makes, models, etc.

Using a database unique index to enforce uniqueness

If your application uses a relational database to store its data, then the easiest solution to enforce record uniqueness is implementing a unique index. This way, we let the database itself enforce the rule.

With Entity Framework Core, we can apply an attribute to an entity class to specify that its underlying database table should define an index.

For example, imagine we have a Make entity class that looks like this:

public class Make
{
    public int ID { get; set; }
    public required string Name { get; set; }
}

If we need every Make record to have a unique Name, we can do so by adding this attribute to the class:

+ using Microsoft.EntityFrameworkCore;

// ...

+ [Index(nameof(Name), IsUnique = true)]
public class Make
{
    // ...
}

The Index attribute from the Microsoft.EntityFrameworkCore namespace, with its IsUnique property set to true takes care of that.

Now, if we were to create a migration with a command like this:

dotnet ef migrations add AddUniqueIndexToMakes

We’ll see something like this in the generated code:

// ...
migrationBuilder.CreateIndex(
    name: "ix_makes_name",
    table: "makes",
    column: "name",
    unique: true);
// ...

See how the framework takes notice of the Index attribute and produces some code to create an index in the database with the column and uniqueness constraint that we specified.

After applying the migration with dotnet ef database update, any attempt at inserting or updating records that violate this constraint will result in an error.

Suppose we have an endpoint for registering new makes that looks like this:

// ...

private readonly VehicleQuotesContext _context;

// ...

[HttpPost]
public async Task<ActionResult<Make>> PostMake(Make make)
{
    _context.Makes.Add(make);

    await _context.SaveChangesAsync();

    return Created();
}

If we POST to it, and give it a payload with a non-unique name…

$ curl -X 'POST' 'http://localhost:5000/api/Makes' \
  -H 'Content-Type: application/json' \
  -d '{
  "name": "Toyota"
}'

…here’s what we get back:

Microsoft.EntityFrameworkCore.DbUpdateException: An error occurred while saving the entity changes. See the inner exception for details.
 ---> Npgsql.PostgresException (0x80004005): 23505: duplicate key value violates unique constraint "ix_makes_name"

This is only a portion of the output, but sure enough, we see a Microsoft.EntityFrameworkCore.DbUpdateException being thrown with the message indicating that a unique constraint was not met. The request results in a 500 Internal Server Error response.

That gets the job done for sure: it prevents bad data from entering the system. But generally we don’t want to expose such verbose errors to our API clients. It’d be better, for example, to catch that exception and return a 400 series status code. A simple 400 Bad Request seems appropriate in this case. We can make the endpoint behave this way with these tweaks:

[HttpPost]
public async Task<ActionResult<Make>> PostMake(Make make)
{
    _context.Makes.Add(make);

+    try
+    {
        await _context.SaveChangesAsync();
+    }
+    catch (DbUpdateException)
+    {
+        return BadRequest();
+    }

    return Created();
}

Simple enough: we use a try-catch to handle the Microsoft.EntityFrameworkCore.DbUpdateException and return a response of our choosing. Trying the same request again results in a 400 Bad Request response with this body:

{
  "type": "https://tools.ietf.org/html/rfc9110#section-15.5.1",
  "title": "Bad Request",
  "status": 400,
  "traceId": "00-7ea5139de6f4ce471aa096dcf4a79ce5-dd308a30c4dd2b33-00"
}

Writing a custom validation attribute to enforce uniqueness

The database index solution will work in a lot of cases. However, there are some scenarios where it won’t. For example, the validation may need to be applied to input data that is not directly tied to a database table; or, maybe the app does not use a relational database at all. In cases where we can’t rely on unique indexes, a custom validation attribute is the best option.

The validation logic that the attribute would have to implement is simple. All it would have to do is query the underlying data store to see if a record with the given field value already exists, and if it does, trigger a validation error.

We’ll start by implementing a first iteration that works specifically for Make names. Then, we’ll extract the generic parts and create a reusable base attribute that can be used in many situations, with any class and field.

Before we can develop the attribute, consider the same Make class that we worked with before:

public class Make
{
    public int ID { get; set; }
    public required string Name { get; set; }
}

Also, we have a repository class that we use to access the underlying storage. Its public interface looks like this:

public interface IMakeRepository
{
    Make? FindByName(string name);
}

The actual implementation is not important to us right now (although you can find it here). It’s enough to know that we can use objects of this type to search for Make records by name.

Suppose also that we’ve included an instance of this type as a service via Dependency Injection. With something like this in the project’s Program.cs:

services.AddScoped<IMakeRepository, MakeRepository>();

With those pieces in place, a custom validation attribute that prevents Makes with the same name could look like this:

// Annotating the class with the AttributeUsage attribute is our way of letting
// the framework know that our UniqueMakeNameAttribute is meant to be applied to
// classes. It's produce a build error if we try to apply it to anything else.
[AttributeUsage(AttributeTargets.Class)]
// In order for our attribute to integrate correctly with the framework's model
// validation functionality, it needs to inherit from ValidationAttribute.
public class UniqueMakeNameAttribute : ValidationAttribute
{
    // Inheriting from ValidationAttribute requires implementing this method.
    // This is the method that gets called by the framework during model
    // validation.
    // Here, we're using it only to prepare the validation algorithm to run,
    // which is defined in the private Validate method.
    protected override ValidationResult? IsValid(
        // The framework uses this "value" parameter to pass in the full Make
        // object that's being validated.
        object? value,
        // The "context" parameter will contain various objects related to the
        // execution environment. It is used to obtain instances made available
        // by Dependency Injection.
        ValidationContext context
    ) {
        if (value == null) return ValidationResult.Success;

        // Here we use the validation context parameter to obtain an instance of
        // the repository
        var repo = context.GetService<IMakeRepository>();
        if (repo == null) return ValidationResult.Success;

        return Validate((Make)value, repo);
    }

    // This is where we actually run the custom validation logic.
    // Like I mentioned before, the logic is simple...
    private static ValidationResult? Validate(Make input, IMakeRepository repo)
    {
        // ...first it calls on the repository to try and find an existing
        // record wiht the same name...
        var existing = repo.FindByName(input.Name);

        // ...then, if another record with the same name exists, one which is
        // not its not the same being validated right now...
        if (existing != null && input.ID != existing.ID)
        {
            // ...return a ValidationResult which contains an understandable
            // error message.
            return new ValidationResult(
                $"The Make name '{input.Name}' is already in use."
            );
        }

        // If, on the other hand, no problem was found, return a successful
        // ValidationResult.
        return ValidationResult.Success;
    }
}

In order to put this logic to work, we use the attribute to annotate our Make class, like so:

+[UniqueMakeName]
public class Make
{
    public int ID { get; set; }
    public required string Name { get; set; }
}

Now, if we make a request to create a new Make and use an existing name, we get back a 400 Bad Request response with this payload:

{
  "type": "https://tools.ietf.org/html/rfc9110#section-15.5.1",
  "title": "One or more validation errors occurred.",
  "status": 400,
  "errors": {
    "": [
      "The Make name 'Toyota' is already in use."
    ]
  },
  "traceId": "00-30bc11fbb22071998c05660fdc454f5b-1ef36a85ecfeaf9b-00"
}

OK, that’s maybe the best response we’ve had so far. It includes not only a proper HTTP status code but also a human readable message that a frontend app can display to its users. Also, with this approach, we don’t have to use a try-catch in the endpoint’s action method like we did before. This is because our attribute plugs right into ASP.NET’s model validation functionality, which triggers automatically at the beginning of every API request.

A generic custom validation attribute to enforce uniqueness

The only disadvantage of our validation attribute is that it only works for one particular class and one particular property. I’m thinking we can make something a little bit more generic which we can reuse in many scenarios.

We could implement an abstract base class that defines the core validation logic as a template method. The overall algorithm will be implemented in the abstract base class. Then, more concrete derived classes will provide the specializations to handle particular objects and fields.

We can refactor our code to make that happen. We should identify the parts of the code that are generic and the ones that are specific. The generic ones won’t change, regardless of what’s being validated. So they will be part of the abstract base class. On the other hand, the specific ones will vary, so they are better defined in derived classes. The base class will provide hooks for the derived ones to implement and supply specializations.

Looking at our current code, here’s what I can identify:

[AttributeUsage(AttributeTargets.Class)]
public class UniqueMakeNameAttribute : ValidationAttribute
{
    protected override ValidationResult? IsValid(
        object? value,
        ValidationContext context
    ) {
        if (value == null) return ValidationResult.Success;

        // Depending on the entity being validated, the repository to use to
        // try to find a would-be duplicated record in the database will be
        // different. So, this line for obtaining a reference to the repository
        // is better implemented in a derived class.
        var repo = context.GetService<IMakeRepository>();
        if (repo == null) return ValidationResult.Success;

        // The type of "value" will be different depending on the class that
        // this attribute is applied to. That means that this casting to "Make"
        // belongs in a derived class.
        return Validate((Make)value, repo);
    }

    private static ValidationResult? Validate(Make input, IMakeRepository repo)
    {
        // Each repository will have its own method for trying to find records.
        // What's more, each entity will have its own field through which its
        // uniqueness is established. This is the responsibility of derived
        // classes.
        var existing = repo.FindByName(input.Name);

        // Every object being validated has its own method for determining if
        // they are equal to or different than what's found in the underlying
        // storage. That can't be part of the generic class.
        if (existing != null && input.ID != existing.ID)
        {
            // Depending on the object being validated, the error message will
            // need to be adjusted. That message belongs in derived classes.
            return new ValidationResult(
                $"The Make name '{input.Name}' is already in use."
            );
        }

        return ValidationResult.Success;
    }
}

With that figured out, here’s an abstract, and generic, base class that can be used as a basis for implementing many concrete custom validation attributes:

[AttributeUsage(AttributeTargets.Class)]
// This class is generic. TEntity is the type of the object that derived classes
// will validate. TRepo is the type of the repository that derived classes will
// use to query the database.
public abstract class BaseUniqueAttribute<TEntity, TRepo> : ValidationAttribute
{
    // These abstract methods are the hooks that derived classes will implement
    // in order to provide specializations for the core algorithm.
    // This one takes care of finding would-be repeated records.
    protected abstract TEntity? FindRecord(TEntity input, TRepo repository);
    // This one checks whether the existing record, if found, is the same being
    // validated.
    protected abstract bool IsSameRecord(TEntity input, TEntity existing);
    // This one produces the error message to return when validation fails.
    protected abstract string GetErrorMessage(TEntity input);

    protected override ValidationResult? IsValid(
        object? input,
        ValidationContext context
    ) {
        if (input == null) return ValidationResult.Success;

        // Here we use TRepo as a type parameter instead of IMakeRepository.
        // Derived classes will be responsible of defining which type the repo
        // will be.
        var repo = context.GetService<TRepo>();
        if (repo == null) return ValidationResult.Success;

        // Here we cast to TEntity instead of Make or any other concrete class.
        // The derived classes will specify the type.
        return Validate((TEntity)input, repo);
    }

    // The overall algorithm in this method remains pretty much identical.
    // Only some of the particular pieces have been replaced by abstract hooks
    // that are to be implemented in derived classes.
    private ValidationResult? Validate(TEntity input, TRepo repo)
    {
        var existing = FindRecord(input, repo);

        if (existing != null && !IsSameRecord(input, existing))
        {
            return new ValidationResult(GetErrorMessage(input));
        }

        return ValidationResult.Success;
    }
}

With that generic abstract class in place, we can develop new specific custom validation attributes very easily. For example, here’s a new version of our UniqueMakeNameAttribute:

public class UniqueMakeNameAttribute : BaseUniqueAttribute<Make, IMakeRepository>
{
    protected override Make? FindRecord(Make input, IMakeRepository repo) =>
        repo.FindByName(input.Name);

    protected override bool IsSameRecord(Make input, Make existing) =>
        input.ID == existing.ID;

    protected override string GetErrorMessage(Make input) =>
        $"The Make name '{input.Name}' is already in use.";
}

Very compact, huh? With this base class, new concrete validation attributes just need to inherit from BaseUniqueAttribute, provide it the proper type parameters (Make and IMakeRepository in this case), and implement the three abstract methods. All of which are simple one liners.

You can test the endpoint again now and should see the same behavior as before. That means the refactoring was successful!

Writing a unit test for the custom validation attribute

Speaking of tests, let’s write a unit test for this. Testing custom validation attributes involves calling their GetValidationResult method and checking that it returns as expected. This is a public method defined in the ValidationAttribute base class. This method is defined by the framework, but it eventually calls our own custom logic that we wrote in the IsValid override.

Since our validator depends on an IMakeRepository, we’re going to have to provide it as a mock object. The validator also needs a ValidationContext instance that it can use to fetch services from the Dependency Injection container. Our test will also have to provide that in some way.

Here’s what such a test could look like:

[Fact]
public void GetValidationResult_ReturnsFailure_WhenAnotherMakeExistsWithTheSameName()
{
    // Arrange

    var attribute = new UniqueMakeNameAttribute();

    var existingMake = new Make() { ID = 10, Name = "test_name" };
    var makeToValidate = new Make() { ID = 20, Name = "test_name" };

    // We have to build a mock of type IMakeRepository for the service provider
    // to return. The attribute uses this class to query the database.
    // This test wants to trigger a validation failure. As such, this mock repo
    // is configured to return an object. This simulates that there is already
    // another record in the database that has the same name as the one being
    // validated.
    var mockRepository = new Mock<IMakeRepository>();
    mockRepository
        .Setup(m => m.FindByName(It.IsAny<string>()))
        .Returns(existingMake);

    // We also have to build a IServiceProvider mock. This will be used to
    // create a ValidationContext instance that the attribute will use to obtain
    // services available via Dependency Injection. Specifically, the repo.
    // With this configuration, it will return the mock repository.
    var mockServiceProvider = new Mock<IServiceProvider>();
    mockServiceProvider
        .Setup(m => m.GetService(typeof(IMakeRepository)))
        .Returns(mockRepository.Object);

    // The validation attribute needs a ValidationContext instance in order to
    // work. So we build this one, giving it the mock we've constructed.
    var context = new ValidationContext(makeToValidate, mockServiceProvider.Object, null);

    // Act

    // Here we invoke the public method that's available for us to run the
    // validation logic. This test expects the validation to fail.
    var result = attribute.GetValidationResult(makeToValidate, context);

    // Assert

    // Two simple assertions here. One checking that the result is not a success
    // and another checking that the expected error message is returned.
    Assert.NotEqual(ValidationResult.Success, result);
    Assert.Equal("The Make name 'test_name' is already in use.", result?.ErrorMessage);
}

And that’s how you can test a custom validation attribute like the one we’ve built. This is very conventional as unit tests go. The only unusual things to keep in mind is the method we need to call in order to exercise the validation logic: GetValidationResult. And also how to construct and configure the ValidationContext object that the method needs as a parameter.

And that’s all for now. We’ve seen two ways of implementing uniqueness validation in ASP.NET. One approach was leveraging an underlying relational database to create unique indexes. A more complicated — but also more flexible — approach leverages framework features to create custom validation attributes. And we even saw how we can test them!

Writing integration tests for an ASP.NET Web API

2024-05-13T00:00:00+00:00

Integration tests exercise a system by instantiating major components and making them interact with each other. They are great for validating important use case scenarios in an end-to-end or close to end-to-end manner.

Full integration tests seldom use mocks or fake objects. Usually, the full stack is tested as if the entire system were running for real. For REST APIs, that generally means tests that involve issuing HTTP requests, validating HTTP responses, and asserting on changes made to a persistent data store, like a database.

In this article, we’re going to discuss how to write such tests for a Web API built using ASP.NET.

Introducing the project

I’ll use an existing ASP.NET Web API project to demonstrate how to write these tests. The API is part of a system that calculates the value of used cars and offers quotes for them. As such, the API has an endpoint for calculating a vehicle quote, given its information and condition: POST /api/Quotes. It also has an endpoint for administration purposes that returns all the quotes that have been stored in the system’s database: GET /api/Quotes. These are the two endpoints that we’ll want to test.

The source code is on GitHub, so feel free to browse. I’ve organized it so that the changes that we’ll make throughout this article are all contained in a single commit. You can see the diff here.

The code base is organized as a .NET solution, as evidenced by the vehicle-quotes.sln file at the root of the repository. The Web API project can be found inside the VehicleQuotes.WebApi directory. The endpoints that we want to test are defined in the controller at VehicleQuotes.WebApi/Controllers/QuotesController.cs.

Our plan is to develop integration tests that exercise the entire stack. That is, the API’s HTTP request handling as well as its database interactions. These are the steps that we’ll take in order to do that:

Create a new xUnit project where we will put our integration tests.
Define a test class fixture that will connect our tests to a test database.
Write some logic to run the tests within their own database transactions. This makes sure they don’t affect one another, and that they encounter the database in a clean state and also leave it that way.
Write some tests that interact with the API over HTTP.

Setting up the integration tests project

The first step is to create a new xUnit project and add it to our solution. This can be done with this pair of commands:

dotnet new xunit -o VehicleQuotes.IntegrationTests
dotnet sln add ./VehicleQuotes.IntegrationTests/VehicleQuotes.IntegrationTests.csproj

That will create a new xUnit project under the VehicleQuotes.IntegrationTests directory. It will have an empty test class file that can be deleted.

The project needs the Microsoft.AspNetCore.Mvc.Testing NuGet package. If we move into the VehicleQuotes.IntegrationTests directory, the package can be installed with this command:

dotnet add package Microsoft.AspNetCore.Mvc.Testing --version 8.0.4

This package will allow our tests to issue HTTP requests to the Web API. We’ll see how that’s done soon.

We also need to add a reference to the Web API project. Also from within the VehicleQuotes.IntegrationTests directory, we can do that with:

dotnet add reference ../VehicleQuotes.WebApi/VehicleQuotes.WebApi.csproj

That way our testing project will have access to the classes defined in the Web API project. Specifically, we’ll need the DB context and some entities. We’ll see why soon.

There’s one additional step that we need to do in the Web API project so that it is testable, and that’s explicitly defining its “Program” class. To do that, we add this line at the end of the VehicleQuotes.WebApi/Program.cs file:

// VehicleQuotes.WebApi/Program.cs
// ...
public partial class Program { }

I’ll admit: this is quite strange. But it is a requirement for integration testing. You’ll see how this comes into play when we start writing the tests. You can read more about it in the official docs.

With that, the project is set up. In the end, our VehicleQuotes.IntegrationTests/VehicleQuotes.IntegrationTests.csproj should look like this:

<!-- VehicleQuotes.IntegrationTests/VehicleQuotes.IntegrationTests.csproj -->
<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>

    <IsPackable>false</IsPackable>
    <IsTestProject>true</IsTestProject>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="coverlet.collector" Version="6.0.0" />
    <PackageReference Include="Microsoft.AspNetCore.Mvc.Testing" Version="8.0.4" />
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.8.0" />
    <PackageReference Include="xunit" Version="2.5.3" />
    <PackageReference Include="xunit.runner.visualstudio" Version="2.5.3" />
  </ItemGroup>

  <ItemGroup>
    <Using Include="Xunit" />
  </ItemGroup>

  <ItemGroup>
    <ProjectReference Include="..\VehicleQuotes.WebApi\VehicleQuotes.WebApi.csproj" />
  </ItemGroup>

</Project>

Writing a database fixture

Now, we need to make it possible for our API to interact with a test instance of our database when running within the context of our tests. This can be done with a properly configured test class fixture. Let’s see what that looks like.

First of all we need an appsettings file for our test project that contains the test database connection string. I created a VehicleQuotes.IntegrationTests/appsettings.Test.json file with these contents:

{
  "ConnectionStrings": {
    "VehicleQuotesContext": "Host=db;Database=vehicle_quotes_test;Username=vehicle_quotes;Password=password;Include Error Detail=True"
  },
  "Jwt": {
    "Key": "this is the secret key for the jwt, it must be kept secure",
    "Issuer": "vehiclequotes.endpointdev.com",
    "Audience": "vehiclequotes.endpointdev.com",
    "Subject": "JWT for vehiclequotes.endpointdev.com"
  },
  "DefaultOffer": 77
}

We also have to tell .NET that it needs to include this file when building the project to run the tests. We do so by adding the following to the VehicleQuotes.IntegrationTests/VehicleQuotes.IntegrationTests.csproj file:

<Project Sdk="Microsoft.NET.Sdk">
  <!-- ... -->
  <ItemGroup>
    <Content Include="appsettings.Test.json" CopyToOutputDirectory="PreserveNewest" />
  </ItemGroup>
  <!-- ... -->
</Project>

The important thing in this appsettings file is the ConnectionStrings.VehicleQuotesContext setting, which contains the test database connection string. Notice the value for Database in the connection string is appended with _test. This is how we make sure the tests run against a different database. The rest of the settings are unrelated to the test database, but need to be defined for the Web API to work. These will obviously be different for every project. All in all, this file is meant to be a test version of the Web API’s own appsettings.json file. You can find it here.

Our Web API supports authentication via Bearer Token. If you want to learn more about how I implemented that, here’s another blog post describing the process.

Next, we develop the test class fixture for enabling database access. We define a VehicleQuotes.IntegrationTests/Fixtures/DatabaseFixture.cs file that looks like this:

using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.Configuration;
using VehicleQuotes.WebApi;

namespace VehicleQuotes.IntegrationTests.Fixtures;

// For more info about this class, check:
// https://learn.microsoft.com/en-us/ef/core/testing/testing-with-the-database#creating-seeding-and-managing-a-test-database
public class DatabaseFixture
{
    private static readonly object _lock = new();
    private static bool _databaseInitialized;

    // Initializes the database specified in the connection string defined in
    // the appsettings.Test.json file.
    public DatabaseFixture()
    {
        // Tests can run in parallel. This lock is meant to make this code
        // thread safe.
        lock (_lock)
        {
            if (!_databaseInitialized)
            {
                using (var dbContext = CreateDbContext())
                {
                    // Delete the database and recreate it.
                    dbContext.Database.EnsureDeleted();
                    dbContext.Database.EnsureCreated();
                }

                _databaseInitialized = true;
            }
        }
    }

    // Creates a new VehicleQuotesContext instance configured with the
    // connection string defined in the appsettings.Test.json file.
    public VehicleQuotesContext CreateDbContext()
    {
        // Load up the appsettings.Test.json file
        var config = new ConfigurationBuilder()
            .AddJsonFile("appsettings.Test.json")
            .Build();

        // Create an instance of DbContextOptions using the connection string
        // defined in the appsettings.Test.json file.
        var options = new DbContextOptionsBuilder<VehicleQuotesContext>()
            .UseNpgsql(config.GetConnectionString("VehicleQuotesContext"))
            .UseSnakeCaseNamingConvention()
            .Options;

        var dbContext = new VehicleQuotesContext(options);

        return dbContext;
    }

    // Runs the given "test" within a database transaction created using the
    // given "dbContext". It rolls back the transaction when the "test" is done.
    public async Task WithTransaction(VehicleQuotesContext dbContext, Func<Task> test)
    {
        dbContext.Database.BeginTransaction();

        try
        {
            await test.Invoke();
        }
        catch
        {
            throw;
        }
        finally
        {
            dbContext.Database.RollbackTransaction();
        }
    }
}

I’ve made sure to include some comments on that class trying to explain what it does, so feel free to review. Much of it was inspired by .NET’s official docs.

This class serves the purpose of allowing the tests suite to connect to and interact with the test database. It does so by performing three tasks:

Resetting the database at the beginning of every test run. This happens in the constructor.
Allowing the creation of new VehicleQuotesContext instances which connect to the test database. Tests will use that to interact with the database.
Offering the capability for tests to be run within DB transactions. This makes sure they don’t affect one another, and that they encounter the database in a clean state and also leave it that way.

Writing some integration tests

Now we can finally start writing some tests. Let’s start with a simple one that makes a GET request to the “fetch all quotes” endpoint that I mentioned at the beginning: GET /api/Quotes. The one defined in the GetAll method in the VehicleQuotes.WebApi/Controllers/QuotesController.cs controller.

We create a new VehicleQuotes.IntegrationTests/Controllers/QuotesControllerTests.cs file and write our test in there. It looks like this:

using System.Net;
using Microsoft.AspNetCore.Mvc.Testing;
using Microsoft.AspNetCore.TestHost;
using Microsoft.Extensions.DependencyInjection;
using VehicleQuotes.IntegrationTests.Fixtures;
using VehicleQuotes.WebApi;

namespace GifBackend.IntegrationTests.WebApi.Controllers;

public class OldQuotesControllerTests : IClassFixture<WebApplicationFactory<Program>>, IClassFixture<DatabaseFixture>
{
    private readonly WebApplicationFactory<Program> _factory;
    private readonly VehicleQuotesContext _dbContext;

    public OldQuotesControllerTests(WebApplicationFactory<Program> factory, DatabaseFixture database)
    {
        _factory = factory;
        _dbContext = database.CreateDbContext();
    }

    protected HttpClient CreateHttpClient()
    {
        return _factory.WithWebHostBuilder(builder =>
        {
            builder.ConfigureTestServices(services =>
            {
                services.AddSingleton(_ => _dbContext);
            });
        })
        .CreateClient();
    }

    [Fact]
    public async Task GetQuotes_ReturnsOK()
    {
        // Arrange
        var client = CreateHttpClient();

        // Act
        var response = await client.GetAsync("/api/Quotes");

        // Assert
        Assert.Equal(HttpStatusCode.OK, response.StatusCode);
    }
}

First, turn your attention to the GetQuotes_ReturnsOK test case. Very simple as tests go, but there are a few interesting things taking place here.

The test case itself is indeed simple. All it does is create an HTTP client, use it to send a GET request to the endpoint that we want to test (using the client’s GetAsync method), and finally validate that the response is a 200 OK. How it does these things is the interesting part.

The HTTP client is created using the CreateHttpClient method. This method leverages _factory, a WebApplicationFactory<Program> instance that’s injected by the framework into our test class. Here, the generic type parameter Program is referring to the “Program” class from the Web API project. The one we defined in the Program.cs file. Notice also how our test class implements the IClassFixture<WebApplicationFactory<Program>> interface. That’s what signals to the framework that a WebApplicationFactory<Program> instance needs to be passed/injected via the constructor. This is the way that the Microsoft.AspNetCore.Mvc.Testing package allows us to express that “this test class contains tests for this web application”.

Full details in the official docs.

Notice also how when creating the HTTP client, a VehicleQuotesContext instance is set up as a singleton service. This is key. This DB context is obtained thanks to our DatabaseFixture. That means that it connects to the test database. We configured it to do so. By setting it up as a service like this, we make sure that the Web API application uses that instance whenever it interacts with the database. And using that instance means that it will use the test database. Since this is the same instance that we will use within our tests, both the tests suite and the application (when running within the context of the tests) will share the same database.

Long story short: This way of constructing the HTTP client and injecting our own DB context into the running application is the secret sauce that allows our tests to utilize the test instance of the database.

Obtaining an instance of the DatabaseFixture is the same as obtaining an instance of the WebApplicationFactory<Program>: all we have to do is make our test class implement the IClassFixture<DatabaseFixture> interface and define the constructor parameter so that the framework passes it in.

Writing some more integration tests

OK, now that we understand the basics, let’s write a few more test cases in order to demonstrate some other common scenarios.

A test that writes to and reads from the database

For example, here’s one that validates that the GET /api/Quotes endpoint actually returns the data that’s stored in the database.

[Fact]
public async Task GetQuotes_ReturnsTheQuotesFromTheDatabase()
{
    await _database.WithTransaction(_dbContext, async () => {
        // Arrange
        await CreateNewQuote("2024", "Toyota", "Corolla");
        await CreateNewQuote("2024", "Honda", "Civic");

        var client = CreateHttpClient();

        // Act
        var response = await client.GetAsync("/api/Quotes");

        // Assert
        Assert.Equal(HttpStatusCode.OK, response.StatusCode);

        var quotes = await response.Content.ReadFromJsonAsync<IEnumerable<SubmittedQuoteRequest>>();

        Assert.NotNull(quotes);
        Assert.Equal(2, quotes.Count());

        Assert.Equal("2024", quotes.First().Year);
        Assert.Equal("Toyota", quotes.First().Make);
        Assert.Equal("Corolla", quotes.First().Model);

        Assert.Equal("2024", quotes.Last().Year);
        Assert.Equal("Honda", quotes.Last().Make);
        Assert.Equal("Civic", quotes.Last().Model);
    });
}

private async Task<Quote> CreateNewQuote(string year, string make, string model)
{
    var bodyType = await _dbContext.BodyTypes.SingleAsync(bt => bt.Name == "Sedan");
    var size = await _dbContext.Sizes.SingleAsync(s => s.Name == "Compact");

    var quote = new Quote {
        Year = year,
        Make = make,
        Model = model,
        BodyTypeID = bodyType.ID,
        SizeID = size.ID,
        ItMoves = true,
        HasAllWheels = true,
        HasAlloyWheels = false,
        HasAllTires = true,
        HasKey = true,
        HasTitle = true,
        RequiresPickup = true,
        HasEngine = true,
        HasTransmission = true,
        HasCompleteInterior = false,
        OfferedQuote = 123,
        Message = "test_message",
        CreatedAt = DateTime.UtcNow
    };

    _dbContext.Quotes.Add(quote);

    _dbContext.SaveChanges();

    return quote;
}

This method introduces a few more interesting features:

It runs within a database transaction. Ensuring that any data changes are rolled back once the test is done.
It uses the singleton DB context to interact with the database. Inserting new records before executing the application under test.
It parses a JSON response body into a .NET object.

In detail, here’s what it does: It uses our DatabaseFixture’s WithTransaction method to run within a database transaction. The test’s strategy is simple: it first inserts new records into the database, leveraging the CreateNewQuote helper method. Then it sends a request to the Web API’s GET /api/Quotes endpoint. Finally, in the assertion section, it validates that the response came back with the correct HTTP status code. Then it parses the JSON response body into an object, and inspects that object to make sure that it has the correct data in it — that is, it contains the database records that were inserted at the beginning of the test case.

A test that makes a POST request

Using all these concepts, we can also write a test for the POST /api/Quotes endpoint. For example, here’s a test that validates that the endpoint stores new records in the database using the given payload:

[Fact]
public async Task PostQuote_CreatesANewQuoteRecord()
{
    await _database.WithTransaction(_dbContext, async () => {
        // Arrange
        var client = CreateHttpClient();

        Assert.Empty(_dbContext.Quotes);

        // Act
        var response = await client.PostAsJsonAsync(
            "/api/Quotes",
            new
            {
                Year = "1990",
                Make = "Toyota",
                Model = "Corolla",
                BodyType = "Sedan",
                Size = "Compact",
                ItMoves = true,
                HasAllWheels = true,
                HasAlloyWheels = false,
                HasAllTires = true,
                HasKey = true,
                HasTitle = true,
                RequiresPickup = false,
                HasEngine = true,
                HasTransmission = true,
                HasCompleteInterior = true
            }
        );

        // Assert
        Assert.Equal(HttpStatusCode.OK, response.StatusCode);

        Assert.Single(_dbContext.Quotes);

        var quote = _dbContext.Quotes.First();
        Assert.NotNull(quote);
        Assert.Equal("1990", quote.Year);
        Assert.Equal("Toyota", quote.Make);
        Assert.Equal("Corolla", quote.Model);
    });
}

The only new concept that this test introduces is the use of the HTTP client’s PostAsJsonAsync method to send POST requests. Notice how we can send any payload we want using an anonymous object. In the assertion phase, the test queries the database to check if the expected record was inserted.

All of this is made possible by the singleton VehicleQuotesContext instance. Both test code and application code are talking to the test database. And thanks to the transactions, each test cleans up after it’s done so that the next test can run with a clean slate.

A test that makes many requests

We can also write tests that span multiple requests. Here’s one for example that registers a new user account and logs in, so that it can then be allowed access to a secure endpoint:

using System.Net.Http.Headers;

// ...

[Fact]
public async Task GetQuotesSecure_ReturnsOK_WhenTheUserHasLoggedIn()
{
    await _database.WithTransaction(_dbContext, async () => {
        // Arrange
        var client = CreateHttpClient();

        await RegisterUser(client);
        var response = await Login(client);
        var authResponse = await response.Content.ReadFromJsonAsync<AuthenticationResponse>();

        Assert.NotNull(authResponse);

        // Act
        using var requestMessage = new HttpRequestMessage(HttpMethod.Get, "/api/Quotes/Secure");
        requestMessage.Headers.Authorization = new AuthenticationHeaderValue("Bearer", authResponse.Token);

        response = await client.SendAsync(requestMessage);

        // Assert
        Assert.Equal(HttpStatusCode.OK, response.StatusCode);
    });
}

private async Task<HttpResponseMessage> RegisterUser(HttpClient client)
{
    var response = await client.PostAsJsonAsync(
        "/api/Users",
        new
        {
            UserName = "test_user_name",
            Password = "test_password",
            Email = "test@email.com"
        }
    );

    Assert.Equal(HttpStatusCode.Created, response.StatusCode);

    return response;
}

private async Task<HttpResponseMessage> Login(HttpClient client)
{
    var response = await client.PostAsJsonAsync(
        "/api/Users/BearerToken",
        new
        {
            UserName = "test_user_name",
            Password = "test_password"
        }
    );

    Assert.Equal(HttpStatusCode.OK, response.StatusCode);

    return response;
}

This test is a little more complicated and introduces a few new concepts. All within its own database transaction, this test:

Uses the RegisterUser helper method to register a new user account. It calls the POST /api/Users endpoint for this.
Uses the Login helper method to log in. In our demo Web API project, this means obtaining a token that can be used for Bearer Token authentication. It calls the POST /api/Users/BearerToken endpoint for this.
Extracts the generated token from the response, and prepares a new request for the GET /api/Quotes/Secure endpoint using that token as an authentication header.
Sends the request and validates that it results in a successful HTTP status code.

Notice that we had to use the HTTP client’s more verbose SendAsync method instead of the more convenient GetAsync. This is because GetAsync doesn’t support sending headers, which we needed.

Disabling some application services

Before we’re done here, something useful about this approach is that it is possible to disable some services on the application under test. We’ve been writing full integration tests where all system components are exercised. It could be the case, however, that we would like to exclude some components from testing. For example, if the app under test sends emails, we might want to disable that. Or if it invokes another third party service, we might want the tests to not do that.

Because we’re able to inject services of our choosing to the running application (like we do with the DB context), it’s certainly possible for us to disable parts of the system. For example, imagine an application that sends emails using an IMailer derived class. One could inject a null object in its place. We could do this when creating the test HTTP client. Something like this:

private HttpClient CreateHttpClient()
{
    // Disable emails for integration tests
    var mockMailer = new Mock<IMailer>();
    mockMailer.Setup(m => m.SendMailAsync()).ReturnsAsync(true);

    return _factory.WithWebHostBuilder(builder =>
    {
        builder.ConfigureTestServices(services =>
        {
            services.AddSingleton(_ => _dbContext);
            services.AddTransient(_ => mockMailer.Object);
        });
    })
    .CreateClient();
}

Here, I created a mock object of the same type as the “Mailer” service that the application uses; then I added it to its Dependency Injection container. Now, every time the application calls for an IMailer instance, it will get the mock. A mock that does nothing.

And that’s it! I think a good amount of integration tests will end up utilizing and remixing various combinations of these basic concepts. I invite you to look at the demo app’s source code on GitHub, where I’ve added a few more tests. I also did some refactoring to make these features a little easier to reuse. Happy testing!

Using Razor templates to render HTML emails in .NET

2024-04-30T00:00:00+00:00

Update August 2025: .NET 8 has simplified the process of rendering HTML emails. You can read an updated post on this blog!

When it comes to sending emails, Ruby on Rails has an excellent solution in the form of Action Mailer.

The basic idea is that you can define email templates using ERB files. This is the same templating engine/language used for normal web application views. Then, application-level SMTP settings are configured for email delivery. Finally, a “Mailer” class can be developed that leverages the templates and the underlying email sending mechanism to send emails.

In Rails, all this comes right out the box. Setup is minimal, so this approach is a huge time saver for a task that’s very common in web applications.

In ASP.NET Core (or .NET in general), we don’t have such a convenient, built-in solution. However, it is possible to implement our own using the framework’s features.

In this article, I’m going to explain step by step what I did in a recent .NET project to develop functionality similar to what Action Mailer provides.

Throughout this article, I will be using a demo Web API application for code examples. If you’d like to see what the final implementation looks like, you can find all the code on GitHub. The API is about calculating quotes for used vehicles. Using the process described here, I added the feature to send emails when new quotes are generated.

In fact, I have all of these changes in a single commit. Here’s the diff.

The plan

So here’s the problem statement: I want to be able to send emails from my .NET app. The body of those emails need to be HTML, and they need to be built based on Razor templates — that is, I want to be able to define *.cshtml files for them. I also want to be able to define a “Mailer” class for each specific transaction or event that I want to send emails for. These “Mailer” classes are what the domain logic components will use directly to send the emails. They are the system’s gateway to email sending functionality.

To fulfill those requirements, we will need four elements:

A base component for sending emails.
A component for turning Razor templates (i.e. *.cshtml files) into email bodies.
The actual Razor templates.
A concrete component that domain logic can invoke to send transactional emails.

Step 1: Sending emails in .NET with the MailKit NuGet package

Creating a class that sends emails is easy using the MailKit NuGet package. I ended up using the approach discussed in this article by Dzenana Kajtaz for Mailtrap’s blog.

The first thing to do is to install the MailKit NuGet package. This command will do it:

dotnet add package MailKit --version 4.5.0

The next thing to do is add the necessary SMTP configuration settings into the project’s appsettings.json file. Here’s what one might look like when configured to use Mailtrap.

// VehicleQuotes.WebApi/appsettings.json
{
    // ...
    "MailSettings": {
        "Server": "sandbox.smtp.mailtrap.io",
        "Port": 587, // 25 or 465 or 587 or 2525
        "SenderName": "VehicleQuotes",
        "SenderEmail": "system@vehiclequotes.com",
        "UserName": "YOUR_SMTP_SERVER_USER_NAME",
        "Password": "YOUR_SMTP_SERVER_USER_PASSWORD"
    },
    // ...
}

Next, we define a class that matches the data structure of these settings. Here’s the one I ended up with:

// VehicleQuotes.WebApi/Configuration/MailSettings.cs
namespace VehicleQuotes.WebApi.Configuration;

public class MailSettings
{
    public required string Server { get; set; }
    public required int Port { get; set; }
    public required string SenderName { get; set; }
    public required string SenderEmail { get; set; }
    public required string UserName { get; set; }
    public required string Password { get; set; }
}

Now, to make the settings actually accessible to the system, we need to add them to the Dependency Injection container. I added this to the application’s bootstrapping logic in my Program.cs file, before the call to builder.Build():

builder.Services.Configure<Configuration.MailSettings>(config.GetSection("MailSettings"));

This essentially tells .NET to read the "MailSettings" section from appsettings.json and load the data in an object of type MailSettings. This object will be available to any class in the system thanks to Dependency Injection. We will use it later.

Now we need to define a class for sending emails. This one is low level. All it does is send emails, it doesn’t render Razor templates. Other components will take care of that. So for now, here’s the class responsible for sending emails:

// VehicleQuotes.WebApi/Services/Mailer.cs
using Microsoft.Extensions.Options;
using MailKit.Net.Smtp;
using MimeKit;
using VehicleQuotes.WebApi.Configuration;

namespace VehicleQuotes.WebApi.Services;

public class MailData
{
    public required string To { get; set; }
    public required string ToName { get; set; }
    public required string Subject { get; set; }
    public required string Body { get; set; }
}

public interface IMailer
{
    Task<bool> SendMailAsync(MailData mailData);
}

public class Mailer : IMailer
{
    private readonly MailSettings _mailSettings;

    public Mailer(IOptions<MailSettings> mailSettingsOptions)
    {
        _mailSettings = mailSettingsOptions.Value;
    }

    public async Task<bool> SendMailAsync(MailData mailData)
    {
        try
        {
            using var emailMessage = new MimeMessage();

            emailMessage.From.Add(new MailboxAddress(_mailSettings.SenderName, _mailSettings.SenderEmail));
            emailMessage.To.Add(new MailboxAddress(mailData.ToName, mailData.To));

            emailMessage.Subject = mailData.Subject;

            var emailBodyBuilder = new BodyBuilder
            {
                TextBody = mailData.Body,
                HtmlBody = mailData.Body
            };

            emailMessage.Body = emailBodyBuilder.ToMessageBody();

            using var mailClient = new SmtpClient();

            await mailClient.ConnectAsync(_mailSettings.Server, _mailSettings.Port, MailKit.Security.SecureSocketOptions.StartTls);
            await mailClient.AuthenticateAsync(_mailSettings.UserName, _mailSettings.Password);
            await mailClient.SendAsync(emailMessage);
            await mailClient.DisconnectAsync(true);

            return true;
        }
        catch
        {
            // TODO: log the email delivery failure
            return false;
        }
    }
}

This class is straightforward. It has a single method, SendMailAsync. The method receives the email subject, body and recipient within a MailData object. Then, it uses the conventional MailKit process to send the email: builds the message, sets sender and recipient, sets the body, connects to the server, authenticates, and sends the email.

For this class to be available at runtime, we need to add it to the Dependency Injection container. So, similar to how we did when loading the SMTP server configuration options, we add this line to the Program.cs file before the call to builder.Build().

builder.Services.AddTransient<Services.IMailer, Services.Mailer>();

Ok, with that, our system knows how to send emails. Let’s see about the next step now.

Step 2: Rendering Razor templates into strings

Now that we have our core mailer class, we see that it expects a string to use as a body for the emails it sends. So like I mentioned before, we need a component that can take Razor templates (i.e. *.cshtml files) and turn them into strings. Here’s how that’s done.

This component will live in a new Razor Class Library project. The *.cshtml templates will also live here. We can create the project and add it to the solution with commands like these:

dotnet new razorclasslib -o VehicleQuotes.RazorTemplates -s
dotnet sln add ./VehicleQuotes.RazorTemplates/VehicleQuotes.RazorTemplates.csproj

Now comes the star of the show, the class that renders Razor templates into strings:

using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Mvc;
using Microsoft.AspNetCore.Mvc.Abstractions;
using Microsoft.AspNetCore.Mvc.ModelBinding;
using Microsoft.AspNetCore.Mvc.Razor;
using Microsoft.AspNetCore.Mvc.Rendering;
using Microsoft.AspNetCore.Mvc.ViewEngines;
using Microsoft.AspNetCore.Mvc.ViewFeatures;
using Microsoft.AspNetCore.Routing;
using Microsoft.Extensions.DependencyInjection;

namespace VehicleQuotes.RazorTemplates.Services;

public interface IRazorViewRenderer
{
    Task<string> Render<TModel>(string viewName, TModel model);
}

public class RazorViewRenderer : IRazorViewRenderer
{
    private readonly IRazorViewEngine _viewEngine;
    private readonly ITempDataProvider _tempDataProvider;
    private readonly IServiceScopeFactory _serviceScopeFactory;

    public RazorViewRenderer(
        IRazorViewEngine viewEngine,
        ITempDataProvider tempDataProvider,
        IServiceScopeFactory serviceScopeFactory
    ) {
        _viewEngine = viewEngine;
        _tempDataProvider = tempDataProvider;
        _serviceScopeFactory = serviceScopeFactory;
    }

    public async Task<string> Render<TModel>(string viewName, TModel model)
    {
        using var scope = _serviceScopeFactory.CreateScope();

        var httpContext = new DefaultHttpContext() { RequestServices = scope.ServiceProvider };
        var actionContext = new ActionContext(httpContext, new RouteData(), new ActionDescriptor());

        var view = FindView(actionContext, viewName);

        var viewData = new ViewDataDictionary<TModel>(new EmptyModelMetadataProvider(), new ModelStateDictionary())
        {
            Model = model
        };

        var tempData = new TempDataDictionary(httpContext, _tempDataProvider);

        using var output = new StringWriter();

        var viewContext = new ViewContext(
            actionContext,
            view,
            viewData,
            tempData,
            output,
            new HtmlHelperOptions()
        );

        await view.RenderAsync(viewContext);

        return output.ToString();
    }

    private IView FindView(ActionContext actionContext, string viewName)
    {
        var getViewResult = _viewEngine.GetView(executingFilePath: null, viewPath: viewName, isMainPage: true);
        if (getViewResult.Success)
        {
            return getViewResult.View;
        }

        var findViewResult = _viewEngine.FindView(actionContext, viewName, isMainPage: true);
        if (findViewResult.Success)
        {
            return findViewResult.View;
        }

        var searchedLocations = getViewResult.SearchedLocations.Concat(findViewResult.SearchedLocations);
        var errorMessage = string.Join(
            Environment.NewLine,
            new[] { $"Unable to find view '{viewName}'. The following locations were searched:" }.Concat(searchedLocations)
        );

        throw new InvalidOperationException(errorMessage);
    }
}

This class is complicated. It leverages several obscure framework components that are not very commonly used. I was able to piece it together with help from here, here, and here.

That’s quite a bit of code, but most of it is ceremony in the service of executing two main steps:

Finding the *.cshtml file that corresponds to the given viewName.
Preparing an IView object that can be used to render the template. It does so using the given model object which contains the actual data to fill out the template placeholders.

Finally, in order to make this class available to the system, we add it to Dependency Injection. Here’s what that looks like:

builder.Services.AddMvcCore().AddRazorViewEngine(); // Necessary for non-GUI projects.
builder.Services.AddTransient<RazorTemplates.Services.IRazorViewRenderer, RazorTemplates.Services.RazorViewRenderer>();

The only interesting thing here is the services.AddMvcCore().AddRazorViewEngine(); line. I had to add that to my project because it is a Web API. That means that it doesn’t include all the services related to rendering views. Other project types that already include all the view-related services, like MVC or Razor Pages, may not need this line. Remember that our RazorViewRenderer class depends on all sorts of framework objects. This line makes sure that they are available.

Step 3: Defining the email templates

Now that our system knows how to render Razor templates into strings, let’s go ahead and actually implement some. The nice thing about this approach is that these templates are full Razor views. That means that features like layouts and partials are supported.

For the purposes of this demo, a simple layout with a header and a footer will suffice. Along with the body of the particular transactional email that we want to send.

Our layout could look like this:

<!-- VehicleQuotes.RazorTemplates/Views/Shared/EmailLayout.cshtml -->
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Email With Razor Templates</title>
</head>
<body>
    <p>Imagine this is a header</p>
    @RenderBody()
    <p>Imagine this is a footer</p>
</body>
</html>

Very simple as layouts go. It does little more than define the basic HTML document structure and call @RenderBody().

We can also add a _ViewStart.cshtml file that specifies this layout as the layout to use for all other templates under the Emails directory:

<!-- VehicleQuotes.RazorTemplates/Views/Emails/_ViewStart.cshtml -->
@{
    Layout = "EmailLayout";
}

Now, for the core contents of the email, we define a new template, along with a class that will serve as its view model.

My template ended up looking like this:

<!-- VehicleQuotes.RazorTemplates/Views/Emails/QuoteGenerated.cshtml -->
@using VehicleQuotes.RazorTemplates.ViewModels

@model QuoteGeneratedViewModel

<p>
    A quote has been generated for a @Model.Year @Model.Make @Model.Model.
</p>

<p>Quote ID: @Model.ID</p>
<p>Created At: @Model.CreatedAt.ToLongDateString()</p>
<p>Offered Amount: @Model.OfferedQuote</p>
<p>Message: @Model.Message</p>

And the view model:

// VehicleQuotes.RazorTemplates/ViewModels/QuoteGeneratedViewModel.cs
namespace VehicleQuotes.RazorTemplates.ViewModels;

public class QuoteGeneratedViewModel
{
    public required int ID { get; set; }
    public required DateTime CreatedAt { get; set; }
    public required int OfferedQuote { get; set; }
    public required string Message { get; set; }
    public required string Year { get; set; }
    public required string Make { get; set; }
    public required string Model { get; set; }
}

Very simple. The template specifies the type that it accepts as a view model using the @model directive. It then proceeds to render the email contents using regular old Razor syntax. The view model is just a simple POCO that defines the data that the template can work with.

Step 4: Putting it all together: the class that sends the “quote generated” email

Finally, we can define our specific Mailer class that, leveraging all the infrastructure we’ve put together, can send one specific type of transactional email. These classes are meant to be simple and boring. Here’s mine:

// VehicleQuotes.WebApi/Services/QuoteGeneratedMailer.cs
using VehicleQuotes.RazorTemplates.Services;
using VehicleQuotes.WebApi.ResourceModels;
using VehicleQuotes.RazorTemplates.ViewModels;

namespace VehicleQuotes.WebApi.Services;

public class QuoteGeneratedMailer
{
    private readonly IMailer _mailer;
    private readonly IRazorViewRenderer _razorViewRenderer;

    public QuoteGeneratedMailer(IMailer mailer, IRazorViewRenderer razorViewRenderer)
    {
        _mailer = mailer;
        _razorViewRenderer = razorViewRenderer;
    }

    public async Task SendAsync(QuoteGeneratedViewModel payload)
    {
        string body = await _razorViewRenderer.Render(
            "/Views/Emails/QuoteGenerated.cshtml", payload
        );

        await _mailer.SendMailAsync(new() {
            To = "test@email.com",
            ToName = "Mr. Recipient",
            Subject = $"VehicleQuotes - New Quote Generated - Quote #{payload.ID}",
            Body = body
        });
    }
}

Pretty neat, huh? This class receives instances of the base Mailer and the RazorViewRenderer via Dependency Injection and uses them to: 1. render the template and 2. send the email.

Like everything else, it also needs to be made available via Dependency Injection. All in all, I ended up with this nice bundle in my Program.cs file:

builder.Services.Configure<Configuration.MailSettings>(config.GetSection("MailSettings"));
builder.Services.AddScoped<Services.QuoteGeneratedMailer>();
builder.Services.AddMvcCore().AddRazorViewEngine();
builder.Services.AddTransient<RazorTemplates.Services.IRazorViewRenderer, RazorTemplates.Services.RazorViewRenderer>();
builder.Services.AddTransient<Services.IMailer, Services.Mailer>();

A good idea is to put these into an IServiceCollection extension method. That’s what I ended up doing, in fact.

Instances of a class like this can be used anywhere in the code. For example like this:

// VehicleQuotes.WebApi/Services/QuoteService.cs
// _mailer is a QuoteGeneratedMailer
// Imagine response is an object that contains all these fields.
await _mailer.SendAsync(
    new QuoteGeneratedViewModel
    {
        ID = response.ID,
        CreatedAt = response.CreatedAt,
        OfferedQuote = response.OfferedQuote,
        Message = response.Message,
        Year = response.Year,
        Make = response.Make,
        Model = response.Model
    }
);

Just build the view model object that it expects and off it goes.

And that’s it! That definitely took some elbow grease to get working as well as delving into pretty arcane framework features. In the end, however, we did manage to build something that offers a developer experience that’s very similar to Action Mailer. Once the core Mailer and the RazorViewRenderer are in place, all it takes to send a new transactional email is:

Defining a new template, with its view model.
Defining a new Mailer class that renders the template, uses it as the email’s body, and sends it.

Building a real-time application with SignalR, .NET, and three.js

2024-01-13T00:00:00+00:00

Image by Pixabay on Pexels

When data in a web application is changing in real time, users want to see those updates reflected in real time without refreshing the application. Adding real-time functionality to a .NET application is easy with the SignalR library.

Depending on the capabilities of the client and server, SignalR can use WebSockets, server-sent events, or long polling to establish a persistent connection between the client and the server. From the server side, SignalR pushes code to connected clients. SignalR is good for applications that require high-frequency updates from the server, such as real-time gaming.

C# code can be used to write SignalR hubs, which easily integrate with other ASP.NET features like dependency injection, authentication, authorization, and scalability.

What we’ll build

We will learn how to use SignalR for real-time communication to send the camera position of a cube, built using three.js, to all users on the page.

You can see the demo here.

Here is the description of our testing app:

A user can request control of a cube for 2 minutes
Control can be released manually, or it will be automatically released after 2 minutes
A user can control the camera position of cube, which will be seen by all users viewing the page
When one user is controlling the cube, other users requesting control will be added to a queue
Queued users will be granted control automatically when the controlling user’s time is over

First we will create a web application and create a SignalR communication between the client and server. After that, we can implement the above features.

Creating the web application

Create a new .NET application.

> dotnet new webapp -o EPSignalRControl

I edited the app using VS Code:

> code .\EPSignalRControl\

Installing and configuring SignalR

The SignalR server library is included in the ASP.NET Core shared framework. However, the JavaScript client library isn’t automatically included in the project. You can use Library Manager (LibMan) to get the client library from unpkg. unpkg is a fast global content delivery network for everything on npm.

> dotnet tool install -g Microsoft.Web.LibraryManager.Cli
> libman install @microsoft/signalr@latest -p unpkg -d wwwroot/js/signalr --files dist/browser/signalr.js

Creating a SignalR hub

We’ll create a CameraData model class for passing data between the SignalR server and client.

public class CameraData
{
    public double x { get; set; } = 0;
    public double y { get; set; } = 0;
    public double z { get; set; } = 5;
}

We’ll also create a ControlHub class which inherits from Hub.

using Microsoft.AspNetCore.SignalR;

namespace EPSignalRControl.Hubs;

public class ControlHub : Hub
{
    public async Task SendData(CameraData data)
    {
        // Broadcast the data to all clients
        await Clients.All.SendAsync("ReceiveData", data);
    }
}

The Hub class has Clients, Context, and Groups properties:

Clients can be used to invoke methods on the clients connected to this hub.
Context is the hub caller context for accessing information about the hub caller connection.
Groups is the group manager to manage connections in groups.

One thing to note is that hubs are transient, so we cannot store state in a property of the Hub class. Each hub method call is executed on a new Hub instance.

Clients.All calls a method on all connected clients. As a result, the control hub sends the data received from one client back to all connected clients. Similarly, we can use Clients.Caller to send back data to the client that invoked the hub method.

Then, we need to register the services using AddSignalR() and configure the endpoints using MapHub() required by SignalR in Program.cs

using EPSignalRControl.Hubs;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddRazorPages(); // Add services to the container
builder.Services.AddSignalR(); // Register SignalR service

var app = builder.Build();
if (!app.Environment.IsDevelopment())
{
    app.UseExceptionHandler("/Error");
    app.UseHsts();
}
app.UseHttpsRedirection();
app.UseStaticFiles();
app.UseRouting();
app.UseAuthorization();
app.MapRazorPages();

app.MapHub<ControlHub>("/controlHub"); // Map ControlHub to the '/controlHub' endpoint

app.Run();

Connecting to SignalR Hub using the SignalR JavaScript client library

We’ll reference the SignalR JavaScript library as well as create a site.js file and reference it in Index.cshtml.

<script src="~/js/signalr/dist/browser/signalr.js"></script>
<script type="module" src="~/js/site.js" asp-append-version="true"></script>

In the site.js file, we will:

Connect to the control hub using the endpoint /connecthub
Start the connection with our control hub
Send data to the server by invoking the SendData method of ControlHub
Add a listener for ReceiveData to receive the data sent from SignalR Hub

const controlHubConnection = new signalR.HubConnectionBuilder()
    .withUrl("/controlhub")
    .build();

controlHubConnection.on("ReceiveData", function (cameraData) {
    console.log(`position.x = ${cameraData.x}`);
    console.log(`position.y = ${cameraData.y}`);
    console.log(`position.z = ${cameraData.z}`);
});

controlHubConnection.start()
    .then(function () {
        console.log("Connected to ControlHub");
        console.log("Invoking SendData");
        controlHubConnection.invoke("SendData", {
            x: 100,
            y: 100,
            z: 100
    })
    .catch(function (err) {
        return console.error(err.toString());
    });
}).catch(function (err) {
    console.error("Error connecting to ControlHub: ", err);
});

At this point if we run the web application, open the browser, and look at the developer tools console, we will see:

Connected to ControlHub
Invoking SendData
position.x = 100
position.y = 100
position.z = 100

If we open the web application in a different tab, then in the first tab’s dev tools console we will see the position values are appended. This is because the new tab invoked SendData and the server received and sent back data to all connected clients.

Connected to ControlHub
Invoking SendData
position.x = 100
position.y = 100
position.z = 100
position.x = 100
position.y = 100
position.z = 100

This is the basic way of implementing SignalR in the project to achieve real-time functionality. Now, we’ll dive deep into creating more features to extend the app to more practical usage.

Adding features

Backend

Add a ControlRequest.cs model to hold the SignalR connection ID and the time of the request.

public class ControlRequest
{
    public string ConnectionId { get; set; } = string.Empty;
    public DateTime RequestTime { get; set; }
}

Add two variables in ControlHub.cs: currentControl to hold the details of the current active request, and controlQueue to hold the requests in queue.

private static ControlRequest? currentControl;
private static Queue<ControlRequest> controlQueue = new Queue<ControlRequest>();

Add a ControlTimer.cs class with properties for a ConcurrentDictionary mapping of the connection ID, the ControlHub clients, the ControlHub context, and the ControlTimer object.

using System.Collections.Concurrent;
using Microsoft.AspNetCore.SignalR;

public class ControlTimer : System.Timers.Timer
{
    public static ConcurrentDictionary<string, ControlTimer> ControlTimers = new();
    public HubCallerContext hubContext { get; set; }
    public IHubCallerClients hubCallerClients { get; set; }
    public ControlTimer(double interval) : base(interval) { }
}

Add two more variables in ControlHub.cs:

controlTimer to store control timer dictionary, control hub context, and control hub clients that can be accessed during the interval of the timer that runs for the active request.
CONTROL_TIME holds the time in seconds for which the active request has control.

private ControlTimer? controlTimer;
private const int CONTROL_TIME = 120; //seconds

Add a RequestControl method to ControlHub that is invoked by the client to request the control. The method adds the connection ID of the invoked request to the queue, and creates and adds the control timer for the currently invoked request. If the queue has only one request, it gives control to the current request, starts the timer and sends a message to the requesting client notifying that access is granted. If the queue has more than one request, the requesting client is notified that their request is queued.

public async Task RequestControl()
{
    var controlRequest = new ControlRequest
    {
        ConnectionId = Context.ConnectionId,
        RequestTime = DateTime.Now
    };

    // Add the control request to the queue
    controlQueue.Enqueue(controlRequest);

    // Add timer to dictionary for request
    controlTimer = new ControlTimer(500)
    {
        hubContext = Context,
        hubCallerClients = Clients
    };
    controlTimer = ControlTimer.ControlTimers.GetOrAdd(controlRequest.ConnectionId, controlTimer);

    // Grant control if the queue is empty or it's the first request
    if (controlQueue.Count == 1 || controlQueue.Peek() == controlRequest)
    {
        currentControl = controlRequest;

        SetupTimerForRelease(controlRequest);

        await Clients.Client(controlRequest.ConnectionId).SendAsync("ControlGranted");
    }
    else
    {
        await Clients.Client(controlRequest.ConnectionId).SendAsync("ControlQueued");
    }
}

Add a SetupTimerForRelease method to add calls to the ReleaseControlMiddleware method on a regular interval from the running timer.

public void SetupTimerForRelease(ControlRequest controlRequest)
{
    if (controlRequest != null && controlTimer == null)
    {
        controlTimer = ControlTimer.ControlTimers.GetOrAdd(controlRequest.ConnectionId, controlTimer);
    }
    controlTimer.Elapsed += new ElapsedEventHandler(ReleaseControlMiddleware);
    controlTimer.Enabled = true;
}

public void ReleaseControlMiddleware(object source, ElapsedEventArgs e)
{
    _ = AutoReleaseControl(source, e);
}

Add an AutoReleaseControl method, to be called by a running timer, which checks if the time for the current control has elapsed. The method sends the time remaining message to the user with control. If the time has elapsed, it calls ClearTimerAndControl to clear the timer and release control.

public async Task AutoReleaseControl(object source, ElapsedEventArgs e)
{
    var controlTimer = (ControlTimer)source;
    HubCallerContext hcallerContext = controlTimer.hubContext;
    IHubCallerClients hubClients = controlTimer.hubCallerClients;

    if (currentControl != null)
    {
        var elapsedSeconds = Math.Ceiling(DateTime.Now.Subtract(currentControl.RequestTime).TotalSeconds);
        await hubClients.Client(currentControl.ConnectionId).SendAsync("ControlRemaining", CONTROL_TIME - elapsedSeconds);
        if (elapsedSeconds >= CONTROL_TIME)
        {
            await ClearTimerAndControl(hubClients, hcallerContext);
        }
    }
}

Add ClearTimerAndControl method which:

Clears the timer for a given connection ID
Sends a message to the controlling client informing that their control time is released
Sends default camera position data to all the clients to reset the cube camera position
Dequeues the current control from the queue, which is the first item
Gives control to the next request in the queue
Sends a message to the client who is granted the control

private async Task ClearTimerAndControl(IHubCallerClients hubClients, HubCallerContext context)
{
    try
    {
        // Clear the timer when control is explicitly released
        ClearControlTimer(context.ConnectionId);

        await hubClients.Client(context.ConnectionId).SendAsync("ControlReleased");
        await hubClients.All.SendAsync("ReceiveData", new CameraData());
        // Release control
        currentControl = null;

        // Remove the first request from the queue
        if (controlQueue.Count > 0)
            controlQueue.Dequeue();

        // Grant control to the next in the queue
        if (controlQueue.Count > 0)
        {
            currentControl = controlQueue.Peek();
            currentControl.RequestTime = DateTime.Now;
            SetupTimerForRelease(currentControl);
            await hubClients.Client(currentControl.ConnectionId).SendAsync("ControlGranted", currentControl.RequestTime.ToString());
        }
    }
    catch (Exception ex)
    {
        await Clients.All.SendAsync(ex.ToString());
    }
}

Add a ClearControlTimer method which gets the control timer of the given connection ID. Removes the ReleaseControlMiddleware from the control timer and disables the control timer.

private void ClearControlTimer(string connectionId)
{
    controlTimer = ControlTimer.ControlTimers.GetOrAdd(connectionId, new ControlTimer(500));
    if (controlTimer != null)
    {
        controlTimer.Elapsed -= new ElapsedEventHandler(ReleaseControlMiddleware);
        controlTimer.Enabled = false;
        controlTimer = null;
    }
}

Add a ReleaseControl method to ControlHub, that is invoked manually by the controlling client to release the control. It checks if the requesting client has the current control access and calls the method to clear timer.

public async Task ReleaseControl()
{
    if (currentControl != null && currentControl.ConnectionId == Context.ConnectionId)
    {
        await ClearTimerAndControl(Clients, Context);
    }
}

Add a SendData method, which receives the data from the controlling user and then sends the received data to all the clients.

public async Task SendData(CameraData cameraData)
{
    controlTimer = ControlTimer.ControlTimers.GetOrAdd(Context.ConnectionId, new ControlTimer(500));
    if (controlTimer != null && controlTimer.hubContext != null && Context.ConnectionId == controlTimer.hubContext.ConnectionId)
    {
        // Broadcast the received sensor data to all clients
        await controlTimer.hubCallerClients.All.SendAsync("ReceiveData", cameraData);
    }
}

That’s all the code required for the backend. Now let’s add to the frontend.

Frontend

In Index.cshtml, we’ll include:

A receiver section to show the three.js cube and update the camera position using the received data from SignalR connection.
A section to show the message from backend.
A controller section with button to request the control using SignalR connection and show the three.js cube. The cube can be rotated with the mouse which sends the data to the backend using the SignalR connection. The backend then sends the data back to all the clients to update the cube camera position in receiver section.

@page
@model IndexModel
@{
    ViewData["Title"] = "Home page";
}

<div id="wrapper">
    <h3>Receiver</h3>
    <div id="receiver-wrapper"></div>

    <hr />
    <h3>Controller</h3>

    <button class="btn btn-primary" id="requestCtrlBtn" disabled></button>
    <div class="alert alert-info d-none" role="alert" id="message"></div>

    <div id="controller-wrapper"></div>
</div>

<script src="~/js/signalr/dist/browser/signalr.js"></script>
<script type="module" src="~/js/site.js" asp-append-version="true"></script>

Then we can add JavaScript code to implement the app functionality in the template. We’ll add:

A click handler to invoke the RequestControl method using the SignalR connection.
Listeners for the ControlGranted, ControlQueued, ControlReleased, and ControlRemaining events to update the UI based on the message included in the corresponding event.

let controlRequested = false;
let controlGranted = false;
let destroyController = false;

const requestCtrlBtn = document.getElementById('requestCtrlBtn')
requestCtrlBtn.disabled = true;
requestCtrlBtn.innerText = "Request Control";

const messageCtrl = document.getElementById('message')

requestCtrlBtn.addEventListener('click', () => {
    let action = 'RequestControl';
    if (controlGranted) {
        action = 'ReleaseControl';
    }
    controlHubConnection.invoke(action).catch(function (err) {
        return console.error(err.toString());
    });
})

controlHubConnection.on("ControlGranted", function () {
    controlGranted = true;
    controlRequested = false;
    requestCtrlBtn.disabled = false;
    requestCtrlBtn.classList.remove(...['btn-primary', 'btn-warning']);
    requestCtrlBtn.classList.add('btn-success');
    requestCtrlBtn.innerText = "Release Control";
    destroyController = createController();
});

controlHubConnection.on("ControlQueued", function () {
    controlRequested = true;
    controlGranted = false;
    requestCtrlBtn.disabled = true;
    requestCtrlBtn.classList.remove('btn-primary');
    requestCtrlBtn.classList.add('btn-warning');
    requestCtrlBtn.innerText = "Waiting in Queue";
});

controlHubConnection.on("ControlReleased", function () {
    controlRequested = false;
    controlGranted = false;
    requestCtrlBtn.disabled = false;
    requestCtrlBtn.innerText = "Request Control";
    requestCtrlBtn.classList.remove(...['btn-success', 'btn-warning']);
    requestCtrlBtn.classList.add('btn-primary');
    messageCtrl.innerText = '';
    messageCtrl.classList.add('d-none');

    if (destroyController != null) {
        destroyController();
        destroyController = null;
    }
});

controlHubConnection.on("ControlRemaining", function (seconds) {
    messageCtrl.classList.remove('d-none');
    messageCtrl.innerText = `Control Granted: ${seconds} seconds remaining`;
});

At the top of the file, import the three.js library along with the OrbitalControl add-on to control the cube camera position.

import * as THREE from 'three';
import { OrbitControls } from 'three/addons/controls/OrbitControls.js';

Add createRenderer, createScene, createCamera, and createCube helper methods that are required for creating cube and attaching it to the receiver as well as the controller section to show a cube.

function createRenderer(canvasDOMId) {
    // Load a Renderer
    let renderer = new THREE.WebGLRenderer({ alpha: false, antialias: true });
    renderer.setClearColor(0xC5C5C3);
    renderer.setPixelRatio(window.devicePixelRatio);
    renderer.setSize(250, 250);
    document.getElementById(canvasDOMId).appendChild(renderer.domElement);

    return renderer;
}

function createScene() {
    // Load 3D Scene
    let scene = new THREE.Scene();

    // Load Light
    var ambientLight = new THREE.AmbientLight(0xcccccc);
    scene.add(ambientLight);

    var directionalLight = new THREE.DirectionalLight(0xffffff);
    directionalLight.position.set(0, 0, 1).normalize();
    scene.add(directionalLight);

    return scene;
}

function createCamera(cameraZPosition = 10) {

    // Load Camera Perspective
    let camera = new THREE.PerspectiveCamera(50, 250 / 250, 1, 200);
    camera.position.z = 5;

    return camera;
}

function createCube() {
    const geometry = new THREE.BoxGeometry(2, 2, 2).toNonIndexed();;

    const positionAttribute = geometry.getAttribute('position');
    const colors = [];
    const color = new THREE.Color();
    for (let i = 0; i < positionAttribute.count; i += 3) {

        if (i >= 0 && i <= 11) {
            color.set(0xffff00); // x facing yellow
        }
        else if (i >= 12 && i <= 23) {
            color.set(0xff0000); // y facing red
        }
        else {
            color.set(0x0000ff); // z facing blue
        }

        // define the same color for each vertex of a triangle
        colors.push(color.r, color.g, color.b);
        colors.push(color.r, color.g, color.b);
        colors.push(color.r, color.g, color.b);

    }
    geometry.setAttribute('color', new THREE.Float32BufferAttribute(colors, 3));
    const material = new THREE.MeshBasicMaterial({ vertexColors: true });
    let cube = new THREE.Mesh(geometry, material);

    // wireframe
    var geo = new THREE.EdgesGeometry(cube.geometry); // or WireframeGeometry
    var mat = new THREE.LineBasicMaterial({ color: 0xffffff });
    var wireframe = new THREE.LineSegments(geo, mat);
    cube.add(wireframe);

    return cube;
}

Use the helper methods in the receiver section to create the cube and display it.

// Define variables
const receiverRenderer = createRenderer('receiver-wrapper');
const receiverScene = createScene();
const receiverCamera = createCamera();
const receiverCube = createCube();
receiverScene.add(receiverCube);

let receiverControls = new OrbitControls(receiverCamera, receiverRenderer.domElement);
receiverControls.enabled = false;
function animate() {
    requestAnimationFrame(animate);
    receiverControls.update();
    receiverRenderer.render(receiverScene, receiverCamera);
}
animate();

Listen to the ReceiveData event to update the receiver section cube with updated camera position data.

controlHubConnection.on("ReceiveData", function (cameraData) {
    receiverCamera.position.x = cameraData.x;
    receiverCamera.position.y = cameraData.y;
    receiverCamera.position.z = cameraData.z;
});

Add a createController method which is called when a user is granted control of the cube. It creates a cube and renders on the controller section and listens to the camera position change.

When the cube is rotated, it calls the server method SendData using the SignalR connection. It also returns a method that cancels the cube renderer and removes the listener from the controller cube. This returned method is called when the ControlReleased event is called from server.

function createController() {
    const controllerRenderer = createRenderer('controller-wrapper');
    const controllerScene = createScene();
    const controllerCamera = createCamera();
    const controllerCube = createCube();
    controllerScene.add(controllerCube);

    let controls = new OrbitControls(controllerCamera, controllerRenderer.domElement);
    function onPositionChange(o) {
        controlHubConnection.invoke("SendData", controllerCamera.position).catch(function (err) {
            return console.error(err.toString());
        });
    }
    controls.addEventListener('change', onPositionChange);

    let request;
    function controllerAnimate() {
        request = requestAnimationFrame(controllerAnimate);
        controls.update();
        controllerRenderer.render(controllerScene, controllerCamera);
    }

    controllerAnimate();

    return function () {
        document.getElementById('controller-wrapper').innerHTML = '';
        cancelAnimationFrame(request);
        controls.removeEventListener('change', onPositionChange);
    }
}

Wrapping Up

We learned the basics of SignalR and how to build a simple application to communicate between the backend and frontend using a SignalR connection. We also looked into a practical implementation on how can we use SignalR to control a graphical object built with three.js. We can extend the application to build apps like a multi-player game, or to send messages to the frontend from long-running backend background jobs.

You can get the code at GitHub.

Client Profile: J.G. Title Company

2023-12-27T00:00:00+00:00

J.G. Title is a prominent service company specializing in automotive dealership titling and registration processes across the country. They simplify operations for their dealers, businesses, and individuals by giving tax/fee quotes, document validation, and checklists for all jurisdictions.

Jordan Kivett, the owner of J.G. Title, envisioned an innovative web application to simplify dealership operations. He contacted us to request a quote for developing this solution and making his vision a reality. This project became a great opportunity for our .NET team to build a robust system with state-of-the-art technologies functioning seamlessly together.

The solution

J.G. Title submitted a detailed document containing a description of the requirements for the J.G. Title Suite app along with workflow diagrams explaining their current business processes. After analyzing all the documents and having some initial meetings, we decided to divide the project into different stages, and estimate each phase of work separately. We ended up with three phases:

Phase I: Create a product that allows users to enter deals into the system and retrieve the quote estimation, including taxes, fees, and charges for services across 50 states.
Phase II: Implement several UI improvements such as a detailed dealership dashboard, financial management, and integration with QuickBooks.
Phase III: Add an ambitious PDF document management section featuring document recognition, analysis, and manipulation, to perform automatic validations on the document’s contents and automatically generate PDF outputs from the deal’s details.

Tech stack

.NET 6 with C#
PostgreSQL 14
Rocky Linux 9 with Nginx

We had already worked with this combination of technologies for several clients, and we are confident in the combined power in a robust database such as Postgres along with .NET 6 running in a Linux environment. It has proven to be a stable, fast, and reliable setup for our web solutions.

We also used Trello for tracking the project’s progress and tasks, GitHub for version control, and Visual Studio 2022 or VS Code with .NET CLI as a coding environment. Some of us also used VS Code dev containers. pgAdmin is our usual tool to manage the local Postgres database, as well as Postman to test our API endpoints.

The team

Most of our .NET team is involved on this project. We also receive valuable help from our hosting team and our Postgres developers.

The client is also highly involved in all aspects of the development process, from requirements gathering to documenting, testing, and providing feedback. We have bi-weekly standup calls, and the entire team (End Point + J.G. Title) interacts actively through Trello, working together and updating each task as the work progresses.

Results

We began working on the project on October 25, 2022, and completed Phase I within the expected timeline, on June 7, 2023. The next phases were launched iteratively, with new deployments usually scheduled twice a week.

The application is now widely used by dealerships and the J.G. Title team. They have over 200 users registered in the system, with more than 8,000 deals quoted, and new deals being added at an increasing rate.

The J.G. Title Suite application, featured on the company’s website.

We are continuing work on several tasks related to Phase III of the project. As the application keeps growing and users send feedback, new phases of work will surely be added to the project in the future.

This partnership is a testament to our shared vision for efficiency and innovation, and we’re excited to continue reshaping the industry with J.G. Title Company!

I wrote the same app twice, with Hotwire and Blazor Server — here’s what I learned

2023-05-27T00:00:00+00:00

There’s been a very interesting movement that has emerged recently in the world of frontend web development: a rise of little-to-no-JavaScript frontend frameworks.

The promise here is that we would be able to develop web applications with rich interactive capabilities without the need to write a whole lot of JavaScript. As such, these new approaches present themselves as alternatives to the likes of Vue, React, Angular, etc.

Two recent technologies that try to fulfill this promise come from two of the most prolific web application development frameworks of today: Blazor, built on .NET, and Hotwire, built on Ruby on Rails.

Now, I love my JS frameworks as much as the next guy, but these new technologies are intriguing. So I decided to build the same application twice, with Hotwire and with Blazor. I learned a few things along the way that I would like to share in this blog post.

Note that there is a table of contents at the end of this post.

What this article is

I want to present some of my findings when working with these two technologies. I also want to discuss how they work and how they feel. How they are similar and how they are different. How they take different routes to arrive at their ultimately similar destinations. Maybe offer some pros and cons.

This post assumes sufficient familiarity with C#, ASP.NET, Ruby, Ruby on Rails and the current state of the art of web development. I won’t assume any familiarity with either Blazor or Hotwire, but this is not a tutorial for either, so I won’t explain in detail how to fully build apps with these technologies.

So who am I writing this for? Essentially, for anybody who is curious about these technologies and is interested in understanding the big picture of what they are about, how they compare to each other, and building their next project with one of them. So, this article is intended to serve more as an introduction to both, a starting point for a conversation to help you make a decision on what’s best for you and your team.

Spoiler alert: Both are great and you can’t go wrong with either. It all comes down to your team’s preferences and past experience.

One final thing worth noting is that I’m focusing this article on “Blazor Server” specifically. I’ll be using the word “Blazor” moving forward, for short. Blazor as a framework has three variants: Blazor Server, Blazor WebAssembly and Blazor Hybrid. You can learn more about them here.

Most of the examples that I’ll use in this post come from a couple of demo apps that I built in order to get my feet wet with these technologies. You can find both of them in GitHub. Here’s the Blazor one and here’s the Hotwire one. You can study both the source code and the commit history, which I tried my best to keep neatly organized. They are both functionally identical, and based on this excellent Hotwire tutorial.

An overview of Blazor

When it comes to how they are designed and the developer experience they offer, these two are very different. Let’s go over some of the key details of Blazor and then we’ll do the same with Hotwire. With that, the differences between them will become apparent.

The first thing we have to understand about Blazor is that it is a component framework, very much like Vue or React. So, with Blazor, applications are broken up into composable modules called “Razor/Blazor components” that are essentially independent pieces of GUI bundled with their corresponding logic. Each component has three parts to it:

HTML-like markup that describes the layout and UI elements to be rendered,
C# logic that defines the behavior of the component, like what actions to take when users interact with GUI elements, and
CSS for styling the GUI elements.

For example here’s a simple a Blazor component that allows displaying, editing, and deleting a particular type of record called “Quote”. Don’t worry about the details too much; we’ll go over some of them next. For now, I just want us to get a sense of what Blazor components look like:

@using Microsoft.EntityFrameworkCore

@inject IDbContextFactory<QuoteEditorBlazor.Data.QuoteEditorContext> dbContextFactory

@if (isEditing)
{
    <QuoteEditorBlazor.Shared.Quotes.Edit
        QuoteToEdit="QuoteToShow"
        OnCancel="HideEditForm"
    />
}
else
{
    <div class="quote">
        <a href="quotes/@QuoteToShow.ID">@QuoteToShow.Name</a>
        <div class="quote__actions">
            <a class="btn btn--light" @onclick="DeleteQuote">Delete</a>
            <a class="btn btn--light" @onclick="ShowEditForm">Edit</a>
        </div>
    </div>
}

@code {
    [Parameter]
    public QuoteEditorBlazor.Models.Quote QuoteToShow { get; set; }

    [Parameter]
    public EventCallback OnQuoteDeleted { get; set; }

    bool isEditing = false;

    void ShowEditForm()
    {
        isEditing = true;
    }

    void HideEditForm()
    {
        isEditing = false;
    }

    void DeleteQuote()
    {
        using var context = dbContextFactory.CreateDbContext();
        context.Quotes.Remove(QuoteToShow);
        context.SaveChanges();
        OnQuoteDeleted.InvokeAsync();
    }
}

You can see that we have some C# in the file (enclosed in a @code block) with some event handlers and parameters. We also have some markup written with a mixture of HTML and C#. This markup has conditionals, wires up click event handlers, renders data from a given record, renders another component, etc. That markup is really just Razor, a templating language that has been widely used in ASP.NET for a good while now. And we also have some top-level statements like @using and @inject for including classes and objects that the component can use.

As far as CSS goes, here’s how it works: Suppose these are the contents of a file called Example.razor. The CSS for it would have to be defined in an Example.razor.css file sitting right next to it. Syntax-wise, this would just be a plain old CSS file. One cool thing to mention about it, though, is that the CSS within it is visible only to the component. So there’s no risk of conflicting rules between components.

If you’re familiar with modern frontend web development and have used frameworks like Vue or React, this should look very familiar to you. In fact, I would venture to say this is one of Blazor’s most attractive points. If you come from that background, and know .NET, it’s not that big of a leap to get into Blazor. The development experience is very similar as its design shares many concepts with modern JS frameworks; they operate under a very similar mental model.

Of course, C# is not JavaScript and .NET is not a browser. So there are many differences when it comes to the nitty-gritty mechanics of things. So a thorough understanding of the .NET framework and its libraries is also important for mastering Blazor. Which, depending on your team’s background, may be a point in favor or against.

How Blazor works under the hood

As you can probably gather from the previous discussion, Blazor takes control of the entire GUI of the application and puts up a firm layer of abstraction over traditional native web technologies. All notion of JavaScript or code executing on a browser environment is greatly de-emphasized. In fact, most of Blazor executes in the server.

Essentially, all the code that you actually write executes on the server side. When a user begins using the app, a two-way, persistent SignalR connection is established between the browser and server. Whenever the client requests a page, the server renders it and sends it through the connection to the client browser, where there’s a component that interprets and displays it. Likewise, whenever the user interacts with some UI element, like clicking a button or a link, the action is sent to the server via this SignalR connection for it to be processed. If changes to the GUI are necessary, the server calculates them and sends them back to the user’s browser, where they will be interpreted and used to re-render the screen with the new state.

Microsoft’s official documentation has an excellent diagram that helps explain the process:

So, even though there is client side code running and browser DOM being manipulated, this is all happening under the hood. The developer doesn’t need to be concerned with that and can just focus on authoring C# code, for the most part.

This approach has a few implications worth noting. One is that this means higher load on the server compared to more traditional web applications. This is mainly because there needs to be a connection always open between clients and the server, by design. Classic HTTP is purely stateless, and connections are typically opened and closed multiple times throughout a user’s session, as they interact with the web app. Not so for Blazor, where this SignalR connection (most likely via WebSockets) is always alive. So, for scalability concerns, we need to keep in mind that the more concurrent users our app has, the more resources the server will consume, even if the users are somewhat idle.

An abstraction over the request/response cycle

A key element of Blazor is that it abstracts developers from the classic HTTP request/response model. With Blazor, one seldom has to consider that aspect, as all interactions between client and server are managed via the framework itself through the persistent connection we discussed in the last couple of paragraphs. To the developer, there’s no real separation between the two. This is a big departure from classic MVC-like frameworks where things like endpoints, actions, and controllers are front and center. Such concepts are simply not in play on Blazor apps. The idea is to make them feel more like desktop apps: fully integrated, monolithic, simple packages of GUI and functionality.

This could become a double-edged sword as, in general, it is always important to have a clear understanding of the underlying technologies that support the application stack you’re working with. That is, the web is still there, even if you can’t see it. But as long as you’re cognizant of that, this approach can have great advantages too.

For example, thanks to this approach, there’s no need to employ the classic SPA pattern of developing applications in two halves: 1. a backend Web API for domain logic written in some backend programming language, and 2. a frontend application written in JavaScript that implements the user experience and communicates with the backend over HTTP.

Blazor attempts to offer a simpler solution from a developer’s perspective, where everything lives in the same execution environment so there’s no need for inter-process communication over HTTP. The example Blazor component from before demonstrates such a case. Notice how this link has an event handler registered to its click event:

<a @onclick="DeleteQuote">Delete</a>

That DeleteQuote event handler, simply defined as the method below, directly leverages the DbContext to issue the delete command to the underlying database:

void DeleteQuote()
{
    using var context = dbContextFactory.CreateDbContext();
    context.Quotes.Remove(QuoteToShow);
    context.SaveChanges();
}

Pretty simple.

However, this flexibility also requires great discipline from the development team. It falls upon us not to clutter the GUI code with all manner of extraneous things like database calls and domain or application logic that have nothing to do with user interface concerns. The traditional two-halves pattern for SPAs has this separation between domain and GUI logic baked in by necessity. Blazor allows us to break free from it, but that does not mean that the separation is not useful or even necessary. For a small example like this, this works just fine, but for larger applications, a more modularized design should be considered as well. Maybe the introduction of abstractions like repositories or domain services? At the end of the day, the core software design principles still need to be applied.

How Blazor supports common frontend framework features

Something else to consider, which I touched on before, is that Blazor is built on top of .NET. That means that a solid understanding of .NET concepts is all but a necessity in order to be effective with Blazor. Most of the features that are now traditional and expected in frontend JavaScript frameworks exist in Blazor, and they are implemented using age-old .NET concepts. If your team has solid .NET experience, this is a blessing. If not, then Blazor requires a larger investment, one that could be overwhelming depending on your time constraints.

Here are a few examples of how Blazor implements classic frontend framework features:

Handling DOM events

We already saw how event handlers work: they are defined as regular C# methods. The way they are registered to respond to events is via attributes in the GUI elements like @onclick or @onchange. All traditional DOM events are available. Like this:

<a @onclick="DeleteQuote">Delete</a>

Including other Blazor components

We also saw how to render components from within other components. All it takes is adding the component to the template as if it was any other GUI element/HTML tag. The tag itself is the name of the component, which sometimes needs to be fully qualified. We saw an example before:

<QuoteEditorBlazor.Shared.Quotes.Edit
    QuoteToEdit="QuoteToShow"
    OnCancel="HideEditForm"
/>

The fully qualified component name is QuoteEditorBlazor.Shared.Quotes.Edit in this case, and that’s how we reference it. If we were to include the namespace with an @using statement (like @using QuoteEditorBlazor.Shared.Quotes near the top of the file) then we would be able to invoke the component just by its name of Edit.

Passing parameters to components

That previous snippet also demonstrates how to pass parameters to components. In this case, we have two: QuoteToEdit which is an object, and OnCancel which is a custom event. As you can see, parameters are passed as if they were HTML element attributes. In the case of QuoteToEdit, we’re passing it QuoteToShow, which is a C# Property defined in the @code section of the component:

public QuoteEditorBlazor.Models.Quote QuoteToShow { get; set; }

In order for the receiving component to be able to accept the parameter, it needs to define a property itself of the same type, annotated with the Parameter attribute. In this case, the QuoteEditorBlazor.Shared.Quotes.Edit component defines it like this:

[Parameter]
public QuoteEditorBlazor.Models.Quote QuoteToEdit { get; set; }

Notice how the name of the property is the same name used for the “HTML attribute” that was used in markup to pass the parameter to the component:

QuoteToEdit="QuoteToShow"

In this case, the expected parameter is of type QuoteEditorBlazor.Models.Quote.

Defining, handling and triggering custom component events

The other parameter, which is a custom event, accepts a method. It looks like this in the receiving component:

[Parameter]
public EventCallback OnCancel { get; set; }

Again, this is a Property annotated with the Parameter Attribute. The only difference is that the type of this one is EventCallback. That’s what allows it to accept a method. Then, inside the receiving component, the event can be triggered with code like this:

OnCancel.InvokeAsync();

This will execute whatever method the parent component has registered as a handler for this custom event. In this case, that would be our HideEditForm method.

Handling component lifecycle events

Other than DOM and custom events, much like in other frontend frameworks, Blazor components also offer ways of hooking up to their own internal lifecycle events. OnInitialized is one of the most important ones, which runs when the component is first starting up. To hook into it and run some code when it happens, all a Blazor component has to do is implement it as a method override within its code section. Something like this:

protected override void OnInitialized()
{
    // Do something here.
}

There’s more to learn about component lifecycle and the official documentation does a good job in explaining it.

Templates

Like we discussed before, Blazor also offers a rich templating syntax via Razor, which is also a hallmark of modern frontend frameworks. We have conditional rendering, support for loops, interpolation of data, model binding, etc.

Routing

Routing is also included in Blazor and the way that it works is with the @page directive that’s used on top of components like this:

@page "/quotes"

Not all components will use these, only those that correspond to whole pages. Most components will be used as portions of pages and as such, would not include these. A @page statement like the above will make the component that includes it accessible via an URL like http://mydomain.com/quotes. In other words, at the root of the site.

These routes also support parameters, which can be defined like this:

@page "/quotes/{quoteId:int}"

A component with this directive will be accessible through URLs like http://mydomain.com/quotes/123. The code in the component can access the route parameter quoteId by defining it as a Property with the matching type, annotated with the Parameter attribute. Like this:

[Parameter]
public int QuoteId { get; set; }.

State management

Global application state management à la Vuex or Redux is also available in Blazor. The cool thing about how this is implemented in Blazor is that there is no need for any additional library or special components. A global app store can be a simple C# object that’s configured to have a lifetime that spans that of the user’s session. Here’s an example of a class that’s used to store global flash messages:

namespace QuoteEditorBlazor.AppState;

public class FlashStore
{
    public List<string> Messages { get; private set; } = new List<string>();

    public event Action? MessagesChanged;

    public async void AddMessage(string message)
    {
        Messages.Add(message);
        MessagesChanged?.Invoke();

        await Task.Delay(TimeSpan.FromSeconds(5));

        Messages.Remove(message);
        MessagesChanged?.Invoke();
    }
}

Like I said: a plain and simple C# class. It offers a method for displaying a message for a few seconds. It does this by storing the given message into an internal variable and then, after a little while, it gets removed. It also offers a MessagesChanged event that other components can subscribe to which is triggered as messages are added and removed.

The lifetime of the instance is controlled via its dependency injection configuration. In our Program.cs, it would be configured like this:

builder.Services.AddScoped<FlashStore>();

Like I mentioned before, Blazor apps establish a persistent connection for the user throughout their session. This means that the instance of the app that is running on the server is also persistent. So, if we add FlashStore as a scoped service, one single instance of it will also persist throughout the session. That’s why we can rely on its internal variables to store global application state.

You can learn more about state management in Blazor here.

Then, you could have a component that renders those messages that looks like this:

@using QuoteEditorBlazor.AppState

@inject FlashStore flashStore

@implements IDisposable

<div class="flash">
    @foreach (var message in flashStore.Messages)
    {
        <div class="flash__message">
            @message
        </div>
    }
</div>

@code {
    protected override void OnInitialized()
    {
        flashStore.MessagesChanged += StateHasChanged;
    }

    public void Dispose()
    {
        flashStore.MessagesChanged -= StateHasChanged;
    }
}

Very simple too. This Blazor component uses a loop to render all the messages. There are a couple of interesting things about this one. First, the component gets access to the instance of the FlashStore class via the @inject directive near the top of the file. That’s how the flashStore variable is made available for the component to use both in C# code and in the template.

The second interesting element is how this component registers its StateHasChanged method to the MessagesChanged event defined in FlashStore. As you recall, FlashStore will trigger MessagesChanged every time messages are added or removed. By registering StateHasChanged to that event, we make sure that the component re-renders every time the messages list changes; which in turn ensures that the most current messages are always rendered. Kind of a neat trick.

And as you might expect, any piece of code throughout the app can submit new messages to FlashStore by getting hold of the global instance via dependency injection and then calling:

flashStore.AddMessage("Here's a flash message!");

Interoperability with JavaScript

For all the nice abstractions that Blazor presents us with, the reality is that sometimes we actually do have to break through them. In the case of JavaScript, this happens when, for example, we need access to some native browser feature, or when we need to integrate with some library for some widget or other type of capability. In Blazor, this is certainly possible.

There are many details to consider when it comes to JavaScript interoperability in Blazor. Check out the official documentation to learn what all is possible. For our purposes here though, it’s enough to know that calls from .NET to JS and vice versa are supported.

The most basic example of calling JS code from .NET code looks like the following. If we have a JS function like this:

window.showMessagefromServer = (message) => {
    alert(`The server says: ${message}`);
    return "The client says thanks!";
};

Such a method can be invoked from a Blazor component like this:

string interopResult = await js.InvokeAsync<string>("showMessagefromServer", "Have a nice day!");

The Blazor component that runs this code would have to inject an instance of IJSRuntime into the js variable with a statement like @inject IJSRuntime js. A key thing to note is that the JavaScript method needs to be attached to the window object in order for it to be accessible.

Like I said, the other way around also works: JavaScript code is able to call .NET logic defined in a server-side Blazor component. Here’s a simple example for us to get an idea of how it feels. We can have a method that looks like this on a Blazor component:

[JSInvokable]
public static Task<string> getMessageFromServer()
{
    return Task.FromResult("Have a nice day!");
}

This method is public and static. It is also possible to invoke non-static instance methods, but the setup is a little more complicated, and this is enough for our purposes here. Other than that, as you can see, the method needs to be annotated with the JSInvokable attribute and return a type of Task.

Here’s some JavaScript that calls this method:

window.returnArrayAsync = () => {
    DotNet.invokeMethodAsync('{YOUR APP ASSEMBLY NAME}', 'getMessageFromServer')
        .then(data => {
            alert(`I asked the server and it says: ${data}`);
        });
};

There are a few noteworthy things here. First we have the DotNet.invokeMethodAsync function which Blazor makes available to us in JavaScript land. That’s what we use to invoke .NET server side code. Next, the function needs to be given the assembly name of our app as well as the name of the method to invoke within the Blazor component. This is the one that was annotated with JSInvokable. Finally, the function itself is asynchronous and thus returns a promise.

An overview of Hotwire

Hotwire makes a promise similar to Blazor’s. However, the way it goes about it couldn’t be more different.

Hotwire is much simpler and more minimalistic than Blazor. Whereas in Blazor we let the framework take total control of the GUI, just like we do with other popular frontend frameworks, Hotwire feels more like a natural evolution of traditional pre-JavaScript-heavy web development. We still have controllers and views, we still render pages on the server, and we still work in tandem with HTTP’s request/response model.

What Hotwire gives us, if we were to boil it down to a single sentence, is a way to refresh only portions of our pages as a result of user interactions. That is, we’re not forced to reload the entire page, as is the case with non-JavaScript web applications. For example, in Hotwire we have the ability to submit a form or click a link and, as a result of that, only update a particular message, picture, or section.

ASP.NET veterans will find this awfully familiar. That’s because all the way back in version 3.5, ASP.NET AJAX offered a very similar feature: partial page updates with the UpdatePanel component. Indeed, Hotwire presents a very similar concept; only greatly improved and modernized.

When it comes to actual coding, Hotwire’s footprint is minimal. It augments traditional Ruby on Rails controllers and views to produce the desired effect of page refreshes that are partial and targeted. Let’s walk through an example and you’ll see how we can start with a regular looking Rails app and then, through minor adjustments, we end up with a more richly interactive experience.

Imagine we are beginning to develop support for CRUDing a particular type of record called “Quote”, and we have these files:

### app/controllers/quotes_controller.rb
class QuotesController < ApplicationController
  def index
  end

  def new
    @quote = Quote.new
  end
end

<!-- app/views/quotes/index.html.erb -->
<main class="container">
  <div class="header">
    <h1>Quotes</h1>
    <%= link_to "New quote", new_quote_path %>
  </div>
</main>

<!-- app/views/quotes/new.html.erb -->
<main class="container">
  <%= link_to "Back to quotes", quotes_path %>

  <div class="header">
    <h1>New quote</h1>
  </div>

  <%= render "form", quote: @quote %>
</main>

<!-- app/views/quotes/_form.html.erb -->
<%= simple_form_for quote do |f| %>
  <% if quote.errors.any? %>
    <div class="error-message">
      <%= quote.errors.full_messages.to_sentence.capitalize %>
    </div>
  <% end %>

  <%= f.input :name %>
  <%= f.submit %>
<% end %>

This sample is using the simple_form gem, which you can learn more about here.

If you’re familiar with Rails, then you understand what’s happening here. We have a simple index page with a heading and a link to another page. That other page contains a form to create new Quote records. It also contains a link to go back to the index page.

Partial page updates with Turbo Frames

As they are right now, these files would produce a traditional web application user experience. When links are clicked, the whole screen will be reloaded to show the page that the clicked link points to. But what if we wanted, for example, to have the new record creation form appear out of nowhere within the same index page, without a full page reload? Here’s what that would look like with Hotwire:

 <!-- app/views/quotes/index.html.erb -->
 <main class="container">
   <div class="header">
     <h1>Quotes</h1>
     <%= link_to "New quote",
                 new_quote_path,
+                data: { turbo_frame: dom_id(Quote.new) } %>
   </div>

+  <%= turbo_frame_tag Quote.new %>
 </main>

 <!-- app/views/quotes/new.html.erb -->
 <main class="container">
   <%= link_to sanitize("&larr; Back to quotes"), quotes_path %>

   <div class="header">
     <h1>New quote</h1>
   </div>

+  <%= turbo_frame_tag Quote.new do %>
     <%= render "form", quote: @quote %>
+  <% end %>
 </main>

Remember that these examples are taken from a fully working application. Feel free to read through the source code to have a more complete understanding of the context within which these files exist.

And that’s really all it takes. Let’s go over it. With these changes, whenever a user clicks on the “New quote” link, instead of the browser triggering the usual GET request to then reload the screen and show the creation page, Hotwire’s frontend component captures the click event and makes the request itself via XHR. The server then receives this request normally and routes it to the new action in the quotes controller. All that action does is render the new.html.erb template and send that back to the client as a response.

When the Hotwire frontend component receives this, it notices that some of the response is enclosed in a turbo_frame_tag whose ID is that of an empty new Quote object. Now, because the link that the user clicked also had the ID of an empty new Quote object as its data-turbo_frame attribute, Hotwire looks for a turbo_frame_tag with the same ID in the page that’s currently being shown an replaces its contents with the contents from the similarly named turbo_frame_tag from the incoming response.

With that, you’ve seen in action some of the key elements that make Hotwire work. First of all we have the turbo_frame_tag helper, which produces a so-called “Turbo Frame”. Turbo Frames are the main building block that we use for partial page updates. Turbo Frames essentially say: “This section of the page is allowed to be dynamically updated without a full page refresh.” In an app that uses Hotwire, whenever you’re in a page that includes a Turbo Frame, if a request is made (whether it be navigation or form submission), and if the resulting response includes within it another Turbo Frame with the same ID, then Hotwire will notice the match and update the current page’s Turbo Frame with the contents of the Turbo Frame from the response.

Looking back at the code, you can see how we achieved this. We added an empty Turbo Frame on the index page, right below the link to the creation page. We also wrapped the form from the creation page in a similarly named Turbo Frame. Finally, we added the data-turbo_frame attribute to the link in the index page to tell Hotwire that it should kick in for this link and target that specific Turbo Frame. If the link was inside the Turbo Frame, we would not have to do this. Since it is outside, Hotwire needs the little hint that says: “Treat this link as if it was inside this Turbo Frame.”

I feel like this is at the same time a little awkward to wrap your head around and deceptively simple. When compared to Blazor, which builds upon tried and true concepts (as far as the developer experience goes at least), Hotwire almost seems alien, with a much more unusual style. But all in all, one can’t deny just how clean and simple all of this looks, in that “Rails magic” kind of way. And once it clicks, you can begin to see a world of possibilities opening up. The Hotwire developers managed to identify and extract a general design pattern of web application interactions, one that can be leveraged to produce a lot of varying rich interactive user experiences.

One neat aspect worth noting is that, if the user were to disable JavaScript, the app would still fully work. It would gracefully degrade. This is a direct consequence of Hotwire’s paradigm of adding minimal features on top of the existing traditional Rails programming model.

Imperative rendering with Turbo Streams

Let’s consider another example now. This time we’ll see another key component of Hotwire in action which is called Turbo Streams.

Turbo Streams helps solve the problem that emerges when the declarative style of pure Turbo Frames is not enough to obtain the fine grained control and behavior that we need. In these cases, the server needs to issue specific commands to the frontend on how to update the GUI. Via Turbo Streams, we can achieve just that. Coding-wise, these look like regular Rails view templates, albeit with some funky syntax thanks to the Turbo Stream helpers.

In our example, we’ll add a list of quotes in the index page, right below the link and the form. We’ll also complete our quote creation implementation so that we’re actually able to submit the form. As a result of that, we want the GUI to be updated so that the form goes away and the newly created quote is shown in the list. These last two are the things that we’ll use Turbo Streams for. Of course, these updates to the page will be done without a full page refresh.

Let’s start by adding a list of quotes on the index page.

In the controller, we query the database for all the quote records and store them in a variable that the view can later access.

 ### app/controllers/quotes_controller.rb
 class QuotesController < ApplicationController
   def index
+    @quotes = Quote.all
   end

   def new
     @quote = Quote.new
   end
 end

In the index view template, we render the collection of records.

 <!-- app/views/quotes/index.html.erb -->
 <main class="container">
   <div class="header">
     <h1>Quotes</h1>
     <%= link_to "New quote",
                 new_quote_path,
                 data: { turbo_frame: dom_id(Quote.new) } %>
   </div>

   <%= turbo_frame_tag Quote.new %>

+  <%= render @quotes %>
 </main>

Now we need to define a “_quote” partial view so that render statement we added on the index view can work automagically.

<!-- app/views/quotes/_quote.html.erb -->
<div class="quote">
  <%= quote.name %>
</div>

With this, the render @quotes statement will loop through all the records in @quotes and render this partial view for each on of them. This template is very simple. All it does is render the name of the quote.

Now let’s add an action that can accept quote creation form submissions:

 ### app/controllers/quotes_controller.rb
 class QuotesController < ApplicationController
   def index
     @quotes = Quote.all
   end

   def new
     @quote = Quote.new
   end
+
+  def create
+    @quote = Quote.new(quote_params)
+
+    if @quote.save
+      redirect_to quotes_path, notice: "Quote was successfully created."
+    else
+      render :new
+    end
+  end
+
+  def quote_params
+    params.require(:quote).permit(:name)
+  end
 end

A typical Rails recipe for a record creation endpoint. It takes the parameters coming from the request and uses them to create a new record via the Quote Active Record model. If successful, it redirects to the index page; if not, it renders the creation page again (via the new action). The new.html.erb view template has some logic to render error messages when they are present so those are going to show up when that page is rendered as a result of unsuccessful calls to this create endpoint.

At this point, we’re able to view all the quotes on record and create new ones. Now here are the changes to Hotwire-ify this scenario.

First we wrap the list of quotes with a Turbo Frame named “quotes”:

 <!-- app/views/quotes/index.html.erb -->
 <main class="container">
   <div class="header">
     <h1>Quotes</h1>
     <%= link_to "New quote",
                 new_quote_path,
                 data: { turbo_frame: dom_id(Quote.new) } %>
   </div>

   <%= turbo_frame_tag Quote.new %>

+  <%= turbo_frame_tag "quotes" do %>
     <%= render @quotes %>
+  <% end %>
 </main>

Next, we employ Turbo Streams. Like I said, Turbo Streams materialize themselves in code as if they were view templates. So, a new file is added that looks like this:

<!-- app/views/quotes/create.turbo_stream.erb -->
<%= turbo_stream.prepend "quotes", partial: "quotes/quote", locals: { quote: @quote } %>
<%= turbo_stream.update Quote.new, "" %>

It sort of looks like a couple of imperative statements, does it not? The first line instructs Hotwire to prepend, in the "quotes" Turbo Frame, a new render of the app/views/quotes/_quote.html.erb partial view, while passing it the @quote object as a parameter. The second line updates the Quote.new Turbo Frame to be empty. When we remember that the "quotes" Turbo Frame is the one that contains the list of records and the Quote.new Turbo Frame is the one that contains the new quote creation form, this starts to make sense. This Turbo Streams view is making the newly created quote appear in the list; and at the same time, it is making the form disappear. From a user’s perspective, this all takes place after submitting the creation form. So the user experience makes complete sense. All that with no full page refresh.

And finally, the controller action needs to make use of this new view template like so:

 ### app/controllers/quotes_controller.rb

 def create
   @quote = Quote.new(quote_params)

   if @quote.save
-    redirect_to quotes_path, notice: "Quote was successfully created."
+    respond_to do |format|
+      format.html { redirect_to quotes_path, notice: "Quote was successfully created." }
+      format.turbo_stream
+    end
   else
     render :new, status: :unprocessable_entity
   end
 end

This is yet another familiar Rails pattern. This is how we specify different response formats, whether it be HTML, JSON, XML. Now, thanks to Hotwire, we can also specify Turbo Streams. This is one of the great aspects about Hotwire: it seamlessly integrates with Rails’ existing features and concepts.

Anyway, in this case, we invoke format.turbo_stream within the block passed to respond_to and that makes it so the app/views/quotes/create.turbo_stream.erb view template is included in this action’s response. Hotwire’s frontend component sees this coming as part of the response and acts accordingly, updating the GUI how it’s been specified.

Adding JavaScript with Stimulus

The final piece of the Hotwire puzzle is Stimulus, which allows us to integrate JavaScript functionality in a neat way. Stimulus is a very simple JavaScript framework, it does not take control of the entire UI. In fact, it does not render any HTML at all. Stimulus essentially offers a nice way of wiring up JS behavior to existing markup. Let’s look at a quick example of how it could hypothetically be used for showing a confirmation popup before deleting a record.

The JavaScript logic lives inside so-called “Stimulus controllers”. For our example, it could look something like this:

// app/javascript/controllers/confirmations_controller.js
import { Controller } from "@hotwired/stimulus"

export default class extends Controller {
  confirm() {
    if (confirm("Are you sure you want to delete this record?")) {
      // Carry on with the operation...
    }
  }
}

And now, the idea is to wire up this code so that it runs when the hypothetical delete button is clicked. Here’s what that button could like:

<button
  data-controller="confirmations"
  data-action="click->confirmations#confirm"
>
  Delete
</button>

Pretty self-explanatory. We specify the name of the controller to use via the data-controller attribute. We also specify, via data-action, what method to invoke within that controller as a result of which DOM event, click in this case.

And that’s basically all it takes to sprinkle some JavaScript on Hotwire apps. Stimulus does offer a few more useful features, which you can read more about in the official documentation. But for us for now, it’s enough to know that it exists, and what’s the main idea behind it.

So as you can hopefully see, Hotwire is much smaller in scope to Blazor. And yet, it allows us to do just the same: highly interactive applications with little to no JavaScript. This ought to make a lot of Rails developers happy.

A final comparison

I was initially thinking about ending this blog post with a flowchart of sorts to explain the process of deciding which of these two technologies you should use. But really, the decision is very simple: If your team is comfortable with .NET, use Blazor. If your team is comfortable with Ruby on Rails, use Hotwire. It’s obvious, so I won’t claim to have made a great discovery here.

The only thing to add is that if your team is familiar with modern frontend web development framework concepts, you’ll be even better served by Blazor and you’ll hit the ground running. If not, then even for seasoned .NET people, there will be a decent learning curve, but not steep enough to be deterred. Moreover, if your team has no modern frontend development experience at all, then Hotwire is a godsend; thanks to its “augment classic backend-heavy web app development” style.

With that said, let’s close out with a summary of main aspects of both technologies and how they compare to each other.

Overall, Hotwire is much simpler than Blazor. While Blazor is a full-fledged GUI component framework, Hotwire’s approach is more like an augmentation of classic non-JavaScript web development patterns. That said, Hotwire’s style is more unusual than Blazor’s, so if your team is already familiar with modern frontend web development, Blazor can be a great fit.

While both frameworks try to offer enough functionality to allow the development of rich interactive experiences without the need to write any JavaScript, the reality is that sometimes JavaScript does need to be written. Both technologies offer ways to make this happen. And while both are perfectly workable, Blazor’s solution is a bit more clunky than Hotwire’s.

When it comes to classic web technologies like HTTP’s request/response cycle and the separation between server and client, Blazor’s style offers a big deviation from them. It greatly de-emphasizes them and presents instead a completely different programming model, one more akin to desktop application development. The concepts of request, response, client, and server seem to vanish. Not so for Hotwire, which builds upon these classic technologies in a way where they still need to be considered and are in fact in the spotlight. While Blazor attempts to do away with these, Hotwire embraces them.

In Blazor, client events are sent to the server, the server renders the DOM updates and sends them to the client for updates. This happens via the persistent SignalR/WebSockets connection.

Hotwire, on the other hand, intercepts client events and sends classic HTTP requests (via AJAX/XHR) to the server. The server then executes the requests and sends back the responses to the client which carries out the necessary operations, generally speaking, updating sections of the page that’s already being displayed.

That means that at the end of the day, both frameworks do the rendering on the server side and send the rendered markup over the wire to the clients. But in Blazor, the client and server have a persistent connection, while Hotwire’s connections come and go as normal HTTP requests and responses.

A neat aspect of Hotwire’s programming model is that it allows an incremental approach to web development where you can start developing the app like you would a traditional, non-reactive, non-JS app, then augment it with a little code to give it SPA capabilities.

And that’s all for now! I for one am glad to see these types of technologies emerge. While there are many teams out there that are already effective and productive with the current landscape of frontend web development, these two are very interesting and seem capable in their own right.

Besides, having alternatives is never a bad thing. Depending mainly on your previous experience, these could be a great fit for projects new and old. It’s great to know that both .NET and Rails include these types of offerings and that they work pretty well.

Building and Hosting a Web App with .NET 6, Postgres and Linux

2022-11-03T00:00:00+00:00

For well over a decade, working with the .NET framework meant running Windows. With the release of .NET Core in 2016, developers were granted the freedom to choose their OS, including Linux; no longer were we bound to Windows. However, few took the plunge, at least in my experience. Why? Well, we are comfortable with what we know, and afraid of what we don’t.

The truth is that building a .NET application on Linux is not that hard, once you get over a few minor bumps in the road. And there are many advantages to this approach, including flexibility, simplicity, and lower costs.

To demonstrate this, we will create a simple .NET MVC web application that connects to Postgres. Then, we will host the app on Linux with Nginx. Shall we start?

Preparing the database

First, you’ll want to install Postgres locally. If you’re using a Mac, this step is very easy. Simply install Postgres.app and you’ll be ready to go.

If you’re using Windows, check out the Windows Installers page on the Postgres website to download the latest installer.

Creating the projects

To develop .NET 6 apps, you will need to install Visual Studio 2022. Check out the Visual Studio downloads page for options for both Windows and Mac.

Start by opening up Visual Studio and creating a new Web Application (MVC) project, and choosing .NET 6.0 as the target framework. I’ve named my project “DotNetSix.Demo”. Here are the steps as they look in Visual Studio on my Mac.

On the final screen, go ahead and give your solution a name, and then click Create. Visual Studio should create a new solution for you, with a single web project. It will automatically create the necessary MVC folders, including Controllers, Models, and Views.

Setting up the database connection

For this demo, we’ll create a simple app that tracks books that you’ve recently read. Let’s go ahead and add a few simple models for the database: Author and Book. You can create these files in the pre-existing Models folder.

Book Model:

using System.ComponentModel.DataAnnotations;
using System.Text.Json.Serialization;

namespace DotNetSix.Demo.Models
{
    public class Book
    {
        public int Id { get; set; }

        public string Title { get; set; }

        [Display(Name = "Publish Date")]
        [DisplayFormat(DataFormatString = "{0:d}")]
        public DateTime PublishDate { get; set; }

        public Author? Author { get; set; }
    }
}

Author Model:

using System.ComponentModel.DataAnnotations;
using System.Text.Json.Serialization;

namespace DotNetSix.Demo.Models
{
    public class Author
    {
        public int Id { get; set; }

        public string FirstName { get; set; }

        public string LastName { get; set; }

        public ICollection<Book>? Books { get; set; }
    }
}

Next, add a folder named “Data” in your project. This will hold a few classes necessary for connecting to Postgres and creating the database. Create a new class file in the folder called AppDBContext.cs. This class will use EF Core to setup a database connection.

AppDBContext.cs:

using Microsoft.EntityFrameworkCore;
using DotNetSix.Demo.Models;

namespace DotNetSix.Demo.Data
{
	public class AppDBContext : DbContext
	{
		public AppDBContext(DbContextOptions<AppDBContext> options) : base(options)
		{
			Database.EnsureCreated();
			DBInitializer.Initialize(this);
		}

		public DbSet<Book> Books { get; set; }
		public DbSet<Author> Authors { get; set; }
	}
}

Then create another class file in the folder called DBInitializer.cs. This class will initialize the Postgres database with test data.

DBInitializer.cs:

using DotNetSix.Demo.Models;

namespace DotNetSix.Demo.Data
{
    public static class DBInitializer
    {
        public static void Initialize(AppDBContext context)
        {
            // Look for any existing Book records.
            if (context.Books.Any())
            {
                return;   // DB has been seeded
            }

            var books = new Book[]
            {
                new Book
                {
                    Title="Blood Meridian",
                    PublishDate=DateTime.Parse("04-01-1985"),
                    Author=new Author{FirstName="Cormac",LastName="McCarthy"}
                },
                new Book
                {
                    Title="The Dog of the South",
                    PublishDate=DateTime.Parse("01-31-1979"),
                    Author=new Author{FirstName="Charles",LastName="Portis"}
                },
                new Book
                {
                    Title="Outline",
                    PublishDate=DateTime.Parse("05-15-2014"),
                    Author=new Author{FirstName="Rachel",LastName="Cusk"}
                },
            };

            context.Books.AddRange(books);
            context.SaveChanges();
        }
    }
}

At this point, you may notice some errors in your IDE. This is because we need to add two important Nuget packages: Microsoft.EntityFrameworkCore and Npgsql.EntityFrameworkCore.PostgreSQL. You can add these by right-clicking on the Dependencies folder and clicking “Manage Nuget Packages…”. Here’s how it looks in Visual Studio.

To round out the database connection, you’ll want to update your Program.cs file, adding the DB Context and the initializing the database. Also, you may encounter an error when you first run your application, citing incompatible dates between .NET and Postgres. To fix this, we will set the Npgsql.EnableLegacyTimestampBehavior to true.

Here is the complete Program.cs for your reference. Lines 7–8, 14, and 24–34 are what was added to the default Program.cs that is created as part of the web project.

Program.cs:

using DotNetSix.Demo.Data;
using DBContext = DotNetSix.Demo.Data.AppDBContext;
using Microsoft.EntityFrameworkCore;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddDbContext<DBContext>(options =>
    options.UseNpgsql(builder.Configuration.GetConnectionString("DemoDbContext")));

// Add services to the container.
builder.Services.AddControllersWithViews();

var app = builder.Build();
AppContext.SetSwitch("Npgsql.EnableLegacyTimestampBehavior", true);

// Configure the HTTP request pipeline.
if (!app.Environment.IsDevelopment())
{
    app.UseExceptionHandler("/Home/Error");
    // The default HSTS value is 30 days. You may want to change this for production scenarios, see https://aka.ms/aspnetcore-hsts.
    app.UseHsts();
}

using (var scope = app.Services.CreateScope())
{
    var services = scope.ServiceProvider;

    var context = services.GetRequiredService<DBContext>();
    // Note: if you're having trouble with EF, database schema, etc.,
    // uncomment the line below to re-create the database upon each run.
    //context.Database.EnsureDeleted();
    context.Database.EnsureCreated();
    DBInitializer.Initialize(context);
}

app.UseHttpsRedirection();
app.UseStaticFiles();

app.UseRouting();

app.UseAuthorization();

app.MapControllerRoute(
    name: "default",
    pattern: "{controller=Books}/{action=Index}/{id?}");

app.Run();

Note that the pattern in MapControllerRoute points to a new controller and action that we will create in the next section.

Updating the config file

The final task to connect to Postgres is to update the appsettings.json file with a connection string. Since I used Postgres.app to install Postgres, my connection string is simple. The username is the same as my Mac, and there is no password. Here is the full file:

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning"
    }
  },
  "AllowedHosts": "*",
  "ConnectionStrings": {
    "DemoDbContext": "Host=localhost;Database=demo;Username=dylan"
  }
}

Displaying the test data

Now that we have the database connection setup, let’s make some small changes to the controllers and views in order to see the data in Postgres.

Add a new file in the Controllers folder called BookController.cs. This file provides an Index controller action that queries the book data from Postgres using EFCore.

Controllers/BooksController.cs:

using Microsoft.EntityFrameworkCore;
using Microsoft.AspNetCore.Mvc;
using DotNetSix.Demo.Data;

namespace DotNetSix.Demo.Controllers
{
    public class BooksController : Controller
    {
        private readonly AppDBContext _context;
        private readonly IConfiguration _config;

        public BooksController(AppDBContext context, IConfiguration config)
        {
            _context = context;
            _config = config;
        }
        // GET: Papers
        public async Task<IActionResult> Index()
        {
            var appDBContext = _context.Books.Include(b => b.Author);
            return View(await appDBContext.ToListAsync());
        }
    }
}

Now create an accompanying view. Add a “Books” folder under Views, and then add a new file to the Books folder called Index.cshtml. This view will receive the data from the controller and display a simple table of recently read books.

Books/Index.cshtml:

@model IEnumerable<DotNetSix.Demo.Models.Book>

<head>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <link rel="stylesheet" href="./css/CreateStyleSheet.css" asp-append-version="true" />
    <link rel="stylesheet" href="~/css/site.css" asp-append-version="true" />
</head>

@{
    ViewData["Title"] = "Index";
}

<h1>Books Recently Read</h1>

<div class="table-wrapper">
    <div class="table-container">
        <table class="table">
            <thead>
                <tr>
                    <th>
                        @Html.DisplayNameFor(model => model.Title)
                    </th>
                    <th>
                        @Html.DisplayNameFor(model => model.Author)
                    </th>
                    <th>
                        @Html.DisplayNameFor(model => model.PublishDate)
                    </th>
                </tr>
            </thead>
            <tbody>
                @foreach (var item in Model)
                    {
                    <tr>
                        <td>
                            @Html.DisplayFor(modelItem => item.Title)
                        </td>
                        <td>
                            @{
                                var authorFullName = item.Author.FirstName + " " + item.Author.LastName;
                                @Html.DisplayFor(modelItem => authorFullName);
                            }
                        </td>
                        <td>
                            @Html.DisplayFor(modelItem => item.PublishDate)
                        </td>
                    </tr>
                }
            </tbody>
        </table>
    </div>
</div>

With the above files in place, you are now ready to run your app. Go ahead and run the project in Visual Studio. You should then see the Books index page in your browser. Hopefully it looks like this:

Installing and configuring Nginx

Just a heads up that the following sections borrow from the MSDN article on hosting ASP.NET Core on Linux. Be sure to check out the article if you’re in need of more info or additional help!

Now that we have the application running, we can prepare to deploy it to a Linux server. The first step in preparing your server to host the application is to install Nginx. Nginx will act as a reverse proxy to your .NET application running on localhost. To install Nginx, use the appropriate package manager for your Linux distro. For example, if you’re running Debian, use apt-get to install Nginx:

sudo apt-get update
sudo apt-get install nginx

You can verify the installation by running sudo nginx -v, which should display the version. Finally, start Nginx using the command sudo service nginx start.

Once Nginx is installed, you’ll need to configure it to host the .NET Core application. Go ahead and create a new Nginx configuration file in /etc/nginx/conf.d. In our case, we’ll name it dotnetsixdemo.conf. Within this configuration file, we’ll do a few things:

Redirect HTTP traffic to HTTPS.
Use HTTPS with an SSL cert installed by Let’s Encrypt certbot.
Configure Nginx to forward HTTP requests to the .NET application, which by default will run locally at http://127.0.0.1:5000.

Here is what our configuration file ends up looking like.

server {
    if ($host = dotnetsixdemo.org) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

    listen        80;
    listen       [::]:80;
    server_name  dotnetsixdemo.org;
    return 404; # managed by Certbot
}

server {
    server_name  dotnetsixdemo.org;

    location / {
        proxy_pass         http://127.0.0.1:5000;
        proxy_http_version 1.1;
        proxy_set_header   Upgrade $http_upgrade;
        proxy_set_header   Connection keep-alive;
        proxy_set_header   Host $host;
        proxy_cache_bypass $http_upgrade;
        proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto $scheme;
    }

    listen [::]:443 ssl; # managed by Certbot
    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/dotnetsixdemo.org/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/dotnetsixdemo.org/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}

Final application adjustments

Before we publish the app, we need to make a small change to the Program.cs file. This will allow redirect and security policies to work correctly given the Nginx reverse proxy setup.

using Microsoft.AspNetCore.HttpOverrides;

...

app.UseForwardedHeaders(new ForwardedHeadersOptions
{
    ForwardedHeaders = ForwardedHeaders.XForwardedFor | ForwardedHeaders.XForwardedProto
});

Additionally, you’ll want to update your appsettings.json file to reflect your target environment, in particular the connection string for Postgres.

Time to publish!

We are now ready to publish the app to the server. In the command prompt, navigate to the root directory of the project, and run the following command.

dotnet publish --configuration release

This will build the app and create a new output folder at /your-project-root/bin/Release/net6.0/publish.

At this point, all that’s left to do is to copy the contents of the publish folder to the server. You can do this using a variety of tools including SCP, SFTP, etc.

Running the app on the server

Once you’ve copied the app to the server, you can start the app by navigating to the directory where the app was copied, and then running the command dotnet [app_assembly].dll. For our demo app, the target DLL would be DotNetSix.Demo.dll.

This will run the app at http://127.0.0.1:5000, and it should now be accessible via the URL that you configured using Nginx. Based on the Nginx configuration provided above, that would be https://dotnetsixdemo.org. Go ahead and test your site in the browser to make sure it is accessible and the reverse proxy is working properly.

Using systemd to run the app as a service

As you might have noticed, there are a few problems with running the app directly using the dotnet command above. The app could easily stop running if the server encounters an issue, and the app is not monitorable. To fix this, let’s use systemd to run and monitor the app process.

Create a new service definition file by running sudo nano /etc/systemd/system/dot-net-six-demo.service and entering the following.

[Unit]
Description=Dotnet Six Demo App

[Service]
WorkingDirectory=/home/dotnetsix/publish
ExecStart=/usr/bin/dotnet /home/dotnetsix/publish/DotNetSix.Demo.dll
Restart=always
# Restart service after 10 seconds if the dotnet service crashes:
RestartSec=10
KillSignal=SIGINT
SyslogIdentifier=dotnetsixdemo
User=dotnetsix
Environment=ASPNETCORE_ENVIRONMENT=Production
Environment=DOTNET_PRINT_TELEMETRY_MESSAGE=false

[Install]
WantedBy=multi-user.target

Note that the paths in WorkingDirectory and ExecStart should match where you copied the application build files on the server, as part of the publish step above.

Also, the User option specifies the user that manages the service and runs the process. In our example, this is the dotnetsix user. You’ll want to create your own user, and importantly grant that user proper ownership of the application files.

To finalize the new service, save the service file and then start the service with sudo systemctl enable --now dot-net-six-demo.service. This will run the app now in the background via systemd and ensure it also starts up after the next reboot.

Viewing logs

Since we are now running the app via systemd, we can use the journalctl to view logs. To view the logs for the demo app, you would run the command sudo journalctl -fu dot-net-six-demo.service.

Wrapping up

That is all! We now have a .NET application running on our Linux server, hosted with Nginx via a reverse proxy, and connecting to a local PostgreSQL database. As we can see, there are several steps to take into account, but the process itself is not particularly complex.

See also my co-worker Kevin Campusano’s blog post for more tips on using .NET with PostgreSQL.

Have you had problems working with .NET and Linux? Do you have any alternative solutions to the ones proposed here? We await your comment.

CI/CD with Azure DevOps

2022-08-28T00:00:00+00:00

Photo by Dylan Wooters, 2022.

A development process that includes manual builds, tests, and deployments can work for a small-scale project, but as your codebase and team grow, it can quickly become time-consuming and unwieldy. This can be particularly true if you’re a .NET developer. We all know the struggle of merging in the latest feature and clicking build in Visual Studio, only to have it fail, citing cryptic errors. “Maybe a clean will fix it?”

If this sounds like your current situation, it’s likely time to consider building a Continuous Integration and Continuous Deployment pipeline, commonly known as “CI/CD”. A good CI/CD pipeline will help you automate the painful areas of building, testing, and deploying your code, as well as help to enforce best practices like pull requests and build verification.

There are many great options to choose from when selecting a CI/CD tool. There are self-hosted options like Jenkins and TeamCity. There are also providers like GitHub and Azure DevOps, which offer CI/CD alongside cloud-hosted source control. All of these options have pros and cons, but if you’re looking for a large feature set, flexibility, and in particular good support for .NET solutions, you should consider Azure DevOps.

In this post, I’ll show you how to set up an Azure CI/CD pipeline for a self-hosted .NET MVC web solution targeting two different environments.

Creating Environments

The first step is to sign up for a free account at Azure DevOps. Once you have your account created, you can then create your Environments. The Environments are the places where you want your app to be deployed.

In the case of a recent project here at End Point Dev, we had three different environments—UAT, Stage, and Production—all running as IIS websites on self-hosted Windows VMs. The UAT environment was on the UAT server, and the Stage and Production environments were on the Production server. So you’ll want to plan out your environments accordingly.

Once you do so, head to Azure DevOps and then click on Environments under the Pipelines section, then click Create Environment.

Then enter a name from the environment and select Virtual Machines and click Next.

In the following window, select Generic Provider, and then select your OS, which in our case is Windows. You will see a registration script with a copy icon next to it. Click the icon to copy the (rather lengthy) PowerShell registration script to the clipboard, and then paste it into a text file.

Next, connect to the target environment. Open a PowerShell window as Administrator, copy and paste the registration script, and then press Enter. The PowerShell script will then do its magic and register the environment with Azure. It may take a minute to run, but afterwards you should see a success message.

Now, if you head back to Azure DevOps and click on the Environment, you should see the server name of the machine that you ran the PowerShell script on. The server name is referred to as a Resource in Azure.

You will then want to complete the above steps for each of your additional target environments, for example, Staging and Production. In our case, Staging and Production are hosted on the same web server, so we only need to create one additional environment (Production) in Azure.

Creating the Pipelines

Now it’s time to create the actual Pipeline. The Pipeline will be responsible for building and deploying your app to the Environments that you created in the previous step. To start, Click on Pipelines in Azure DevOps, and then click the Create Pipeline button.

You’ll then be asked where your source code lives. In our case, the source code exists in a repo in Azure DevOps. The nice thing about Azure is that you can also target source code that exists on another provider, for example, GitHub, Bitbucket, or even a self-hosted Git repo (Other Git). Click on your hosting provider and follow the instructions.

Once you’re connected to your source provider, then you can configure the Pipeline. The “Pipeline” is actually just a YAML file within your target repo. You can read all about the Azure YAML syntax here. We’ll choose a Starter Pipeline, which opens up a text editor and enables you to create your YAML file.

Delete the sample text that appears in the editor, and then enter the following YAML. This is a pre-baked YAML pipeline that restores and builds a .NET web project, and then publishes the build assets to your three different target environments. In the next section, we will take a deep dive into how the YAML works so that you can edit it to fit your needs.

trigger:
- uat
- staging
- main

pool:
  vmImage: 'windows-latest'

variables:
  projectPath: 'ci_tutorial/ci_tutorial_web.csproj'
  packageName: 'ci_tutorial_web.zip'
  solution: '**/*.sln'
  buildPlatform: 'AnyCPU'
  artifactName: 'AzureDrop'
  ${{ if eq(variables['Build.SourceBranchName'], 'uat') }}:
    websiteName: 'ci_tutorial_uat'
    buildConfiguration: 'UAT'
    environmentName: 'UAT'
    targetVM: 'UATWEBAPP1'
  ${{ if eq(variables['Build.SourceBranchName'], 'staging') }}:
    websiteName: 'ci_tutorial_staging'
    buildConfiguration: 'Staging'
    environmentName: 'Production'
    targetVM: 'WEBAPP1'
  ${{ if eq(variables['Build.SourceBranchName'], 'main') }}:
    websiteName: 'ci_tutorial_prod'
    buildConfiguration: 'Release'
    environmentName: 'Production'
    targetVM: 'WEBAPP1'

stages:
- stage: build
  jobs:
  - job: RestoreAndBuild
    steps:
    - task: NuGetToolInstaller@1
    - task: NuGetCommand@2
      inputs:
        restoreSolution: '$(solution)'
    - task: VSBuild@1
      inputs:
        solution: '$(projectPath)'
        msbuildArgs: '/p:DeployOnBuild=true /p:WebPublishMethod=Package /p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true /p:PackageLocation="$(build.artifactStagingDirectory)"'
        platform: '$(buildPlatform)'
        configuration: '$(buildConfiguration)'
    #- task: VSTest@2
    #  inputs:
    #    platform: '$(buildPlatform)'
    #    configuration: '$(buildConfiguration)'
    - task: PublishBuildArtifacts@1
      inputs:
        PathtoPublish: '$(Build.ArtifactStagingDirectory)'
        ArtifactName: '$(artifactName)'
        publishLocation: 'Container'

- stage: Deploy
  displayName: Deploy to IIS
  dependsOn: Build
  jobs:
  - deployment: DeploytoIIS
    displayName: Deploy the web application to dev environment
    environment:
      name: ${{ variables.environmentName }}
      resourceName: ${{ variables.targetVM }}
      resourceType: VirtualMachine
    strategy:
      runOnce:
        deploy:
          steps:
          - task: DownloadBuildArtifacts@0
            inputs:
              buildType: 'current'
              downloadType: 'specific'
              downloadPath: '$(System.ArtifactsDirectory)'
          - task: IISWebAppManagementOnMachineGroup@0
            displayName: 'Create App Pool and Website'
            inputs:
              WebsiteName: '$(websiteName)'
              WebsitePhysicalPath: '%SystemDrive%\inetpub\wwwroot\$(websiteName)'
              CreateOrUpdateAppPoolForWebsite: true
              AppPoolNameForWebsite: '$(websiteName)'
          # For testing, to confirm target website
          - task: PowerShell@2
            displayName: Display target websitename
            inputs:
              targetType: 'inline'
              script: 'Write-Host "Target Website Name: $(websiteName)"'
          - task: IISWebAppDeploymentOnMachineGroup@0
            displayName: 'Deploy IIS Website'
            inputs:
              WebSiteName: '$(websiteName)'
              Package: '$(System.ArtifactsDirectory)\$(artifactName)\$(packageName)'

Understanding the YAML

The first section of the YAML, trigger, determines which branches will trigger the pipeline when they receive a push. In our case, we have three branches: uat, staging, and main.

trigger:
- uat
- staging
- main

The variables section defines variables that are used later in the pipeline YAML. An important part of this section is the use of if statements to assign values to certain variables (i.e., environmentName) based on the source branch for the build. This mechanism allows the pipeline to deploy to several different environments using a single YAML file.

Note that the if statements also switch the build configuration. If you have config transforms, this will be crucial for ensuring that the correct config files are deployed to each environment.

${{ if eq(variables['Build.SourceBranchName'], 'uat') }}:
  websiteName: 'ci_tutorial_uat'
  buildConfiguration: 'UAT'
  environmentName: 'UAT'
  targetVM: 'UATWEBAPP1'
${{ if eq(variables['Build.SourceBranchName'], 'staging') }}:
  websiteName: 'ci_tutorial_staging'
  buildConfiguration: 'Staging'
  environmentName: 'Production'
  targetVM: 'WEBAPP1'
${{ if eq(variables['Build.SourceBranchName'], 'main') }}:
  websiteName: 'ci_tutorial_prod'
  buildConfiguration: 'Release'
  environmentName: 'Production'
  targetVM: 'WEBAPP1'

Next, we get into the actual actions, or stages, of the pipeline. This pipeline has two stages: build and deploy. The build stage runs a NuGet restore and then a VS build, and then publishes the build output/artifacts to an Azure cloud storage location. The last bit is something that took me a while to wrap my head around: The build does not occur on the target environment via the runner—it actually occurs up in “Azure land”, on a Windows cloud VM.

The VM on which the build occurs can be defined using the pool: vmImage section. In our case, we are running the build on windows-latest, which currently translates to the brand new Windows Server 2022.

The deploy stage then downloads the build artifacts from Azure and deploys them to IIS on the target machine, using the runner that you installed as part of the environment setup above. Note that the variables defined in the if statements come into play here—this is how the pipeline knows which environment/VM to target.

environment:
  name: ${{ variables.environmentName }}
  resourceName: ${{ variables.targetVM }}
  resourceType: VirtualMachine

The IISWebAppManagementOnMachineGroup task will automatically create an app pool and website in IIS for the application. This feature can be toggled using the CreateOrUpdateAppPoolForWebsite setting. If a website already exists based on the WebsiteName, Azure will simply deploy to that existing app pool and website.

Adding Branch Protection and Build Validation

In order to ensure successful deployments to Production, you will likely want to add in branch protection for your main or master branch, and also consider build validation.

Branch protection allows you to enforce reviews for pull requests into certain branches. Build validation is a feature that pre-compiles your .NET code to ensure that it builds prior to merging a pull request. Both features help to improve code quality and prevent broken deployments.

To add branch protection, go to Repo → Branches → your target branch → right side menu, then select Branch Policies.

To enforce reviews, turn on “Require a minimum number of reviewers”, and adjust the number of reviewers and settings as required.

For build validation, scroll down to the Build Validation section, and click the plus button to add a new policy. Select the target build pipeline, adjust the policy settings as necessary (see below), and then give it a display name. Click save, and the build validation policy will be applied to the branch.

An important note here is that the build validation essentially runs the CI pipeline. In our case, that means it will run the build and the deployment. This is not ideal; we only want it to run the build (NuGet restore and VS build). So, it’s advisable to create a separate pipeline with only the build stage. This can be done easily by copying our example YAML and removing the “Deploy” stage, then saving it as a new pipeline. Then, choose that new pipeline as the target build pipeline in the validation policy.

Now when you open up a PR, it can only be merged once a review has been applied, and the build validation has run successfully.

Monitoring and Troubleshooting the Pipeline

You can see the overall status of your pipeline by clicking on the Pipelines section in the left-side navigation. Green means good!

To dive into more details, click on the pipeline name. This will show all the recent pipeline runs. Then click on a particular run. This will bring up the run details, including a flowchart for each stage.

To dive further, click on a specific stage. You’ll then see all of the tasks in the stage, with actual output details on the right, similar to what you would see in the output window in Visual Studio.

If you encounter an error in your pipeline, the stage details page is a great place to troubleshoot. Azure will show you the task that generated the error by displaying a red error icon next to the task. You can then click on the task and trace the output on the right to determine the exact error message.

Wrapping Up

Hopefully, this post gives you a good overview of CI/CD options for a self-hosted .NET application. Azure is almost infinitely configurable, so there are many other options to explore that are not covered in this post, whether it be within the pipeline YAML, or through other settings like branch policies/protection. If you have any questions, please feel free to leave a comment. Happy automating!

Implementing Backend Tasks in ASP.NET Core

2022-08-08T00:00:00+00:00

As we’ve already established, Ruby on Rails is great. The amount and quality of tools that Rails puts at our disposal when it comes to developing web applications is truly outstanding. One aspect of web application development that Rails makes particularly easy is that of creating backend tasks.

These tasks can be anything from database maintenance, file system cleanup, overnight heavy computations, bulk email dispatch, etc. In general, functionality that is typically initiated by a sysadmin in the backend, or scheduled in a cron job, which has no GUI, but rather, is invoked via command line.

By integrating with Rake, Rails allows us to very easily write such tasks as plain old Ruby scrips. These scripts have access to all the domain logic and data that the full-fledged Rails app has access to. The cherry on top is that the command-line interface to invoke such tasks is very straightforward. It looks something like this: bin/rails fulfillment:process_new_orders.

All this is included right out of the box for new Rails projects.

ASP.NET Core, which is also great, doesn’t support this out of the box like Rails does.

However, I think we should be able to implement our own without too much hassle, and have a similar sysadmin experience. Let’s see if we can do it.

There is a Table of contents at the end of this post.

What we want to accomplish

So, to put it in concrete terms, we want to create a backend task that has access to all the logic and data of an existing ASP.NET Core application. The task should be callable via command-line interface, so that it can be easily executed via the likes of cron or other scripts.

In order to meet these requirements, we will create a new .NET console app that:

References the existing ASP.NET Core project.
Loads all the classes from it and makes instances of them available via dependency injection.
Has a usable, Unix-like command-line interface that sysadmins would be familiar with.
Is invokable via the .NET CLI.

We will do all this within the context of an existing web application. One that I’ve been building upon thoughout a few articles.

It is a simple ASP.NET Web API backed by a Postgres database. It has a few endpoints for CRUDing automotive related data and for calculating values of vehicles based on various aspects of them.

You can find the code on GitHub. If you’d like to follow along, clone the repository and check out this commit: 9a078015ce. It represents the project as it was before applying all the changes from this article. The finished product can be found here.

You can follow the instructions in the project’s README file if you want to get the app up and running.

For our demo use case, we will try to develop a backend task that creates new user accounts for our existing application.

Let’s get to it.

Creating a new console app that references the existing web app as a library

The codebase is structured as a solution, as given away by the vehicle-quotes.sln file located at the root of the repository. Within this solution, there are two projects: VehicleQuotes which is the web app itself, and VehicleQuotes.Tests which contains the app’s test suite. For this article, we only care about the web app.

Like I said, the backend task that we will create is nothing fancy in itself. It’s a humble console app. So, we start by asking the dotnet CLI to create a new console app project for us.

From the repository’s root directory, we can do so with this command:

dotnet new console -o VehicleQuotes.CreateUser

That should’ve resulted in a new VehicleQuotes.CreateUser directory being created, and within it, (along with some other nuts and bolts) our new console app’s Program.cs (the code) and VehicleQuotes.CreateUser.csproj (the project definition) files. The name that we’ve chosen is straightforward: the name of the overall solution and the action that this console app is going to perform.

There’s more info regarding the dotnet new command in the official docs.

Now, since we’re using a solution file, let’s add our brand new console app project to it with:

dotnet sln add VehicleQuotes.CreateUser

OK, cool. That should’ve produced the following diff on vehicle-quotes.sln:

diff --git a/vehicle-quotes.sln b/vehicle-quotes.sln
index 537d864..5da277d 100644
--- a/vehicle-quotes.sln
+++ b/vehicle-quotes.sln
@@ -7,6 +7,8 @@ Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "VehicleQuotes", "VehicleQuo
 EndProject
 Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "VehicleQuotes.Tests", "VehicleQuotes.Tests\VehicleQuotes.Tests.csproj", "{5F6470E4-12AB-4E30-8879-3664ABAA959D}"
 EndProject
+Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "VehicleQuotes.CreateUser", "VehicleQuotes.CreateUser\VehicleQuotes.CreateUser.csproj", "{EDBB33E3-DCCE-4957-8A69-DC905D1BEAA4}"
+EndProject
 Global
        GlobalSection(SolutionConfigurationPlatforms) = preSolution
                Debug|Any CPU = Debug|Any CPU
@@ -44,5 +46,17 @@ Global
                {5F6470E4-12AB-4E30-8879-3664ABAA959D}.Release|x64.Build.0 = Release|Any CPU
                {5F6470E4-12AB-4E30-8879-3664ABAA959D}.Release|x86.ActiveCfg = Release|Any CPU
                {5F6470E4-12AB-4E30-8879-3664ABAA959D}.Release|x86.Build.0 = Release|Any CPU
+               {EDBB33E3-DCCE-4957-8A69-DC905D1BEAA4}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
+               {EDBB33E3-DCCE-4957-8A69-DC905D1BEAA4}.Debug|Any CPU.Build.0 = Debug|Any CPU
+               {EDBB33E3-DCCE-4957-8A69-DC905D1BEAA4}.Debug|x64.ActiveCfg = Debug|Any CPU
+               {EDBB33E3-DCCE-4957-8A69-DC905D1BEAA4}.Debug|x64.Build.0 = Debug|Any CPU
+               {EDBB33E3-DCCE-4957-8A69-DC905D1BEAA4}.Debug|x86.ActiveCfg = Debug|Any CPU
+               {EDBB33E3-DCCE-4957-8A69-DC905D1BEAA4}.Debug|x86.Build.0 = Debug|Any CPU
+               {EDBB33E3-DCCE-4957-8A69-DC905D1BEAA4}.Release|Any CPU.ActiveCfg = Release|Any CPU
+               {EDBB33E3-DCCE-4957-8A69-DC905D1BEAA4}.Release|Any CPU.Build.0 = Release|Any CPU
+               {EDBB33E3-DCCE-4957-8A69-DC905D1BEAA4}.Release|x64.ActiveCfg = Release|Any CPU
+               {EDBB33E3-DCCE-4957-8A69-DC905D1BEAA4}.Release|x64.Build.0 = Release|Any CPU
+               {EDBB33E3-DCCE-4957-8A69-DC905D1BEAA4}.Release|x86.ActiveCfg = Release|Any CPU
+               {EDBB33E3-DCCE-4957-8A69-DC905D1BEAA4}.Release|x86.Build.0 = Release|Any CPU
        EndGlobalSection
 EndGlobal

This allows the .NET tooling to know that we’ve got some intentional organization going on in our code base. That these projects each form part of a bigger whole.

It’s also nice to add a .gitignore file for our new VehicleQuotes.CreateUser project to keep things manageable. dotnet new can help with that if we were to navigate into the VehicleQuotes.CreateUser directory and run:
dotnet new gitignore

You can learn more about how to work with solutions via the .NET CLI in the official docs.

Now let’s modify our new project’s .csproj file so that it references the main web app project under VehicleQuotes. This will allow our console app to access all of the classes defined in the web app, as if it was a library or package.

If we move to the VehicleQuotes.CreateUser directory, we can do that with the following command:

dotnet add reference ../VehicleQuotes/VehicleQuotes.csproj

The command itself is pretty self-explanatory. It just expects to be given the .csproj file of the project that we want to add as a reference in order to do its magic.

Running that should’ve added the following snippet to VehicleQuotes.CreateUser/VehicleQuotes.CreateUser.csproj:

<ItemGroup>
  <ProjectReference Include="..\VehicleQuotes\VehicleQuotes.csproj" />
</ItemGroup>

This way, .NET allows the code defined in the VehicleQuotes project to be used within the VehicleQuotes.CreateUser project.

You can learn more about the add reference command in the official docs.

Setting up dependency injection in the console app

As a result of the previous steps, our new console app now has access to the classes defined within the web app. However, classes by themselves are no good if we can’t actually create instances of them that we can interact with. The premier method for making instances of classes available throughout a .NET application is via dependency injection. So, we need to set that up for our little console app.

Dependency injection is something that comes out of the box for ASP.NET Core web apps. Luckily for us, .NET makes it fairly easy to set it up in console apps as well by leveraging the same components.

For this app, we want to create user accounts. In the web app, user account management is done via ASP.NET Core Identity. Specifically, the UserManager class is used to create new user accounts. This console app will do the same.

Take a look at VehicleQuotes/Controllers/UsersController.cs to see how the user accounts are created. If you’d like to know more about integrating ASP.NET Core Identity into an existing web app, I wrote an article about it.

Before we do the dependency injection setup, let’s add a new class to our console app project that will encapsulate the logic of leveraging the UserManager for user account creation. This is the actual task that we want to perform. The new class will be defined in VehicleQuotes.CreateUser/UserCreator.cs and these will be its contents:

using Microsoft.AspNetCore.Identity;

namespace VehicleQuotes.CreateUser;

class UserCreator
{
    private readonly UserManager<IdentityUser> _userManager;

    public UserCreator(UserManager<IdentityUser> userManager) {
        _userManager = userManager;
    }

    public IdentityResult Run(string username, string email, string password)
    {
        var userCreateTask = _userManager.CreateAsync(
            new IdentityUser() { UserName = username, Email = email },
            password
        );

        var result = userCreateTask.Result;

        return result;
    }
}

This class is pretty lean. All it does is define a constructor that expects an instance of UserManager<IdentityUser>, which will be supplied via dependency injection; and a simple Run method that, when given a username, email, and password, asks the UserManager<IdentityUser> instance that it was given to create a user account.

Moving on to setting up dependency injection now, we will do it in VehicleQuotes.CreateUser/Program.cs. Replace the contents of that file with this:

using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using VehicleQuotes.CreateUser;

IHost host = Host.CreateDefaultBuilder(args)
    .UseContentRoot(System.AppContext.BaseDirectory)
    .ConfigureServices((context, services) =>
    {
        var startup = new VehicleQuotes.Startup(context.Configuration);
        startup.ConfigureServices(services);

        services.AddTransient<UserCreator>();
    })
    .Build();

var userCreator = host.Services.GetRequiredService<UserCreator>();
userCreator.Run(args[0], args[1], args[2]);

Let’s dissect this bit by bit.

First off, we’ve got a few using statements that we need in order to access some classes and extension methods that we need down below.

Next, we create and configure a new IHost instance. .NET Core introduced the concept of a “host” as an abstraction for programs; and packed in there a lot of functionality to help with things like configuration, logging and, most importantly for us, dependency injection. To put it simply, the simplest way of enabling dependency injection in a console app is to use a Host and all the goodies that come within.

There’s much more information about hosts in .NET’s official documentation.

Host.CreateDefaultBuilder(args) gives us an IHostBuilder instance that we can use to configure our host. In our case, we’ve chosen to call UseContentRoot(System.AppContext.BaseDirectory) on it, which makes it possible for the app to find assets (like appconfig.json files!) regardless of where its deployed and where its being called from.

This is important for us because, as you will see later, we will install this console app as a .NET Tool. .NET Tools are installed in directories picked by .NET and can be run from anywhere in the system. So we need to make sure that our app can find its assets wherever it has been installed.

After that, we call ConfigureServices where we do a nice trick in order to make sure our console app has all the same configuration as the web app as far as dependency injection goes.

You see, in ASP.NET Core, all the service classes that are to be made available to the application via dependency injection are configured within the web app’s Startup class’ ConfigureServices method. VehicleQuotes is no exception. So, in order for our console app to have access to all of the services (i.e. instances of classes) that the web app does, the console app needs to call that same code. And that’s exactly what’s happening in these two lines:

var startup = new VehicleQuotes.Startup(context.Configuration);
startup.ConfigureServices(services);

We create a new instance of the web app’s Startup class and call its ConfigureServices method. That’s the key element that allows the console app to have access to all the logic that the web app does. Including the services/classes provided by ASP.NET Core Identity like UserManager<IdentityUser>, which UserCreator needs in order to function.

Once that’s done, the rest is straightforward.

We also add our new UserCreator to the dependency injection engine via:

services.AddTransient<UserCreator>();

Curious about what Transient means? The official .NET documentation has the answer.

And that allows us to obtain an instance of it with:

var userCreator = host.Services.GetRequiredService<UserCreator>();

And then, it’s just a matter of calling its Run method like so, passing it the command-line arguments:

userCreator.Run(args[0], args[1], args[2]);

args is a special variable that contains an array with the arguments given by command line. That means that our console app can be called like this:

dotnet run test_username test_email@email.com mysecretpassword

Go ahead, you can try it out and see the app log what it’s doing. Once done, it will also have created a new record in the database.

$ psql -h localhost -U vehicle_quotes
psql (14.3 (Ubuntu 14.3-0ubuntu0.22.04.1), server 14.2 (Debian 14.2-1.pgdg110+1))
Type "help" for help.

vehicle_quotes=# select user_name, email from public."AspNetUsers";
   user_name   |           email            
---------------+----------------------------
 test_username | test_email@email.com
(1 rows)

Pretty neat, huh? At this point we have a console app that creates user accounts for our existing web app. It works, but it could be better. Let’s add a nice command-line interface experience now.

Improving the CLI with CommandLineParser

With help from CommandLineParser, we can develop a Unix-like command-line interface for our app. We can use it to add help text, examples, have strongly typed parameters and useful error messages when said parameters are not correctly provided. Let’s do that now.

First, we need to install the package in our console app project by running the following command from within the project’s directory (VehicleQuotes.CreateUser):

dotnet add package CommandLineParser

After that’s done, a new section will have been added to VehicleQuotes.CreateUser/VehicleQuotes.CreateUser.csproj that looks like this:

<ItemGroup>
  <PackageReference Include="CommandLineParser" Version="2.9.1" />
</ItemGroup>

Now our console app can use the classes provided by the package.

All specifications for CommandLineParser are done via a plain old C# class that we need to define. For this console app, which accepts three mandatory arguments, such a class could look like this:

using CommandLine;
using CommandLine.Text;

namespace VehicleQuotes.CreateUser;

class CliOptions
{
    [Value(0, Required = true, MetaName = "username", HelpText = "The username of the new user account to create.")]
    public string Username { get; set; }

    [Value(1, Required = true, MetaName = "email", HelpText = "The email of the new user account to create.")]
    public string Email { get; set; }

    [Value(2, Required = true, MetaName = "password", HelpText = "The password of the new user account to create.")]
    public string Password { get; set; }

    [Usage(ApplicationAlias = "create_user")]
    public static IEnumerable<Example> Examples
    {
        get
        {
            return new List<Example> {
                new (
                    "Create a new user account",
                    new CliOptions { Username = "name", Email = "email@domain.com", Password = "secret" }
                )
            };
        }
    }
}

I’ve decided to name it CliOptions but really, it could have been anything. Go ahead and create it in VehicleQuotes.CreateUser/CliOptions.cs. There are a few interesting elements to note here.

The key aspect is that we have a few properties: Username, Email, and Password. These represent our three command-line arguments. Thanks to the Value attributes that they have been annotated with, CommandLineParser will know that that’s their purpose. You can see how the attributes themselves also contain each argument’s specification like the order in which they should be supplied, as well as their name and help text.

This class also defines an Examples getter which is used by CommandLineParser to print out usage examples into the console when our app’s help option is invoked.

Other than that, the class itself is unremarkable. In summary, it’s a number of fields annotated with attributes so that CommandLineParser knows what to do with it.

In order to actually put it to work, we update our VehicleQuotes.CreateUser/Program.cs like so:

 using Microsoft.Extensions.DependencyInjection;
 using Microsoft.Extensions.Hosting;
+using CommandLine;
 using VehicleQuotes.CreateUser;
 
+void Run(CliOptions options)
+{
     IHost host = Host.CreateDefaultBuilder(args)
         .UseContentRoot(System.AppContext.BaseDirectory)
         .ConfigureServices((context, services) =>
         {
             var startup = new VehicleQuotes.Startup(context.Configuration);
             startup.ConfigureServices(services);
 
             services.AddTransient<UserCreator>();
         })
         .Build();
 
     var userCreator = host.Services.GetRequiredService<UserCreator>();
-    userCreator.Run(args[0], args[1], args[2]);
+    userCreator.Run(options.Username, options.Email, options.Password);
+}
+
+Parser.Default
+    .ParseArguments<CliOptions>(args)
+    .WithParsed(options => Run(options));

We’ve wrapped Program.cs’s original code into a method simply called Run.

Also, we’ve added this snippet at the bottom of the file:

Parser.Default
    .ParseArguments<CliOptions>(args)
    .WithParsed(options => Run(options));

That’s how we ask CommandLineParser to parse the incoming CLI arguments, as specified by CliOptions and, if it can be done successfully, then execute the rest of the program by calling the Run method.

Neatly, we also no longer have to use the args array directly in order to get the command-line arguments provided to the app, instead we use the options object that CommandLineParser creates for us once it has done its parsing. You can see it in this line:

userCreator.Run(options.Username, options.Email, options.Password);

options is an instance of our very own CliOptions class, so we can access the properties that we defined within it. These contain the arguments that were passed to the program.

If you were to try dotnet run right now, you’d see the following output:

VehicleQuotes.CreateUser 1.0.0
Copyright (C) 2022 VehicleQuotes.CreateUser

ERROR(S):
  A required value not bound to option name is missing.
USAGE:
Create a new user account:
  create_user name email@domain.com secret

  --help               Display this help screen.

  --version            Display version information.

  username (pos. 0)    Required. The username of the new user account to create.

  email (pos. 1)       Required. The email of the new user account to create.

  password (pos. 2)    Required. The password of the new user account to create.

As you can see, CommandLineParser detected that no arguments were given, and as such, it printed out an error message, along with the descriptions, help text and example that we defined. Basically the instructions on how to use our console app.

Deploying the console app as a .NET tool

OK, we now have a console app that does what it needs to do, with a decent interface. The final step is to make it even more accessible by deploying it as a .NET tool. If we do that, we’d be able to invoke it with a command like this:

dotnet create_user Kevin kevin@gmail.com secretpw

.NET makes this easy for us. There’s a caveat that we’ll discuss later, but for now, let’s go through the basic setup.

.NET tools are essentially just glorified NuGet packages. As such, we begin by adding some additional package-related configuration options to VehicleQuotes.CreateUser/VehicleQuotes.CreateUser.csproj. We add them as children elements to the <PropertyGroup>:

<PackAsTool>true</PackAsTool>
<PackageOutputPath>./nupkg</PackageOutputPath>
<ToolCommandName>create_user</ToolCommandName>
<VersionPrefix>1.0.0</VersionPrefix>

With that, we signal .NET that we want the console app to be packed as a tool, the path where it should put the package itself, and what its name will be. That is, how will it be invoked via the console (remember we want to be able to do dotnet create_user).

Finally, we specify a version number. When dealing with NuGet packages, versioning them is very important, as that drives caching and downloading logic in NuGet. More on that later when we talk about the aforementioned caveats.

Now, to build the package, we use:

dotnet pack

That will build the application and produce a VehicleQuotes.CreateUser/nupkg/VehicleQuotes.CreateUser.1.0.0.nupkg file.

We won’t make the tool available for the entire system. Instead, we will make it available from within our solution’s directory only. We can make that happen if we create a tool manifest file in the source code’s root directory. That’s done with this command, run from the root directory:

dotnet new tool-manifest

That should create a new file: .config/dotnet-tools.json.

Now, also from the root directory, we can finally install our tool:

dotnet tool install --add-source ./VehicleQuotes.CreateUser/nupkg VehicleQuotes.CreateUser

This is the regular command to install any tools in .NET. The interesting part is that we use the --add-source option to point it to the path where our freshly built package is located.

After that, .NET shows this output:

$ dotnet tool install --add-source ./VehicleQuotes.CreateUser/nupkg VehicleQuotes.CreateUser
You can invoke the tool from this directory using the following commands: 'dotnet tool run create_user' or 'dotnet create_user'.
Tool 'vehiclequotes.createuser' (version '1.0.0') was successfully installed. Entry is added to the manifest file /path/to/solution/.config/dotnet-tools.json.

It tells us all we need to know. Check out the .config/dotnet-tools.json to see how the tool has been added there. All this means that now we can run our console app as a .NET tool:

$ dotnet create_user --help
VehicleQuotes.CreateUser 1.0.0
Copyright (C) 2022 VehicleQuotes.CreateUser
USAGE:
Create a new user account:
  create_user name email@domain.com secret

  --help               Display this help screen.

  --version            Display version information.

  username (pos. 0)    Required. The username of the new user account to create.

  email (pos. 1)       Required. The email of the new user account to create.

  password (pos. 2)    Required. The password of the new user account to create.

Pretty sweet, huh? And yes, it has taken a lot more effort than what it would’ve taken in Ruby on Rails, but hey, the end result is pretty fabulous I think, and we learned a new thing. Besides, once you’ve done it once, the skeleton can be easily reused for all kinds of different backend tasks.

Now, before we wrap this up, there’s something we need to consider when actively developing these tools. That is, when making changes and re-installing constantly.

The main aspect to understand is that tools are just NuGet packages, and as such are beholden to the NuGet package infrastructure. Which includes caching. If you’re in the process of developing your tool and are quickly making and deploying changes, NuGet won’t update the cache unless you do one of two things:

Manually clear it with a command like dotnet nuget locals all --clear.
Bump up the version of the tool by updating the value of <VersionSuffix> in the project (.csproj) file.

This means that, unless you do one these, the changes that you make to the app between re-builds (with dotnet pack) and re-installs (with dotnet dotnet tool install) won’t ever make their way to the package that’s actually installed in your system. So be sure to keep that in mind.

Implementing Authentication in ASP.NET Core Web APIs

2022-06-17T00:00:00+00:00

Authentication is a complex space. There are many problem scenarios and many more solutions. When it comes to Web APIs written with ASP.NET Core, there are various fully featured options like Duende IdentityServer or Azure Active Directory. These promise to be “everything but the kitchen sink” solutions which are robust and allow you to deal with many complex requirements.

But what if our requirements dictate that we need something simpler? Do we have to roll out our own from scratch? Or does ASP.NET Core offer smaller, customizable, somewhat independent puzzle pieces that we can put together without having to write all the code ourselves and still have a good amount of control?

Spoiler alert: The answer to that last question is yes. And we’re going to talk about it in this very article.

There is a Table of contents at the end of this post.

Two approaches to authentication: JWT and API Keys

In this article, we’ll take an existing ASP.NET Core Web API and add authentication capabilities to it. Specifically, we’ll support two authentication schemes commonly used for Web APIs: JWT and API Keys. Also, we will use our own database for storage of user accounts and credentials.

The project that we will work with is a simple ASP.NET Web API backed by a Postgres database. It has a few endpoints for CRUDing automotive related data and for calculating values of vehicles based on various aspects of them. You can read all about the process of building it here. I also added a few database integration tests for it here.

You can find it on GitHub. If you’d like to follow along, clone the repository and checkout this commit: cd290c765fcd2c6693008d3dc76fa931098dcaa0. It represents the project as it was before applying all the changes from this article.

You can follow the instructions in the project’s README file in order to get the app up and running.

Managing user accounts with ASP.NET Core Identity

Let’s deal first with the requirement of storing the user accounts in our own database.

Luckily for us, ASP.NET Core provides us with Identity. This is an API that offers a comprehensive solution for authentication. It can connect with the aforementioned Duende IdentityServer for OpenID Connect and OAuth 2.0, supports authentication via third parties like Facebook or Google, and can fully integrate with user interfaces on web apps, complete with scaffolding/autogeneration capabilities too.

Most importantly for us, it supports management of user accounts stored in our own database. It includes data for third party logins, passwords, roles for authorization, email confirmation, access tokens, etc.

That’s a lot of of functionality baked in there. But we don’t need all that, so let’s see how we can take only the bits and pieces that we need in order to fulfill our specific requirements.

Specifically, we want:

Tables in our database for storage of user accounts and passwords.
A programmatic way to create, fetch and validate users using those tables.

It’s actually pretty simple.

Install the necessary NuGet packages

First, we need to install a couple of NuGet packages:

Microsoft.AspNetCore.Identity which contains the core library.
Microsoft.AspNetCore.Identity.EntityFrameworkCore which includes the classes that Identity needs in order to properly interact with Entity Framework, which is what we’re using in our Web API for interfacing with the database.

We can install them by running these two commands from the VehicleQuotes directory:

$ dotnet add package Microsoft.AspNetCore.Identity
$ dotnet add package Microsoft.AspNetCore.Identity.EntityFrameworkCore

That will add the following lines to the VehicleQuotes/VehicleQuotes.csproj project file:

 <Project Sdk="Microsoft.NET.Sdk.Web">
   ...
   <ItemGroup>
     ...
+    <PackageReference Include="Microsoft.AspNetCore.Identity" Version="2.2.0" />
+    <PackageReference Include="Microsoft.AspNetCore.Identity.EntityFrameworkCore" Version="6.0.5" />
     ...
   </ItemGroup>
 </Project>

Update the DbContext to include the Identity tables

Next step is to configure our DbContext class so that it includes the new tables that we need from the Identity library. So let’s go to VehicleQuotes/Data/VehicleQuotesContext.cs and update it to do so.

We need to include these new using statements at the top of the file so that we have access to the classes that we need from the NuGet packages we just installed:

using Microsoft.AspNetCore.Identity;
using Microsoft.AspNetCore.Identity.EntityFrameworkCore;

Next, instead of DbContext, the VehicleQuotesContext class should inherit from IdentityUserContext<IdentityUser>.

IdentityUserContext is a class provided by ASP.NET Core Identity that’s designed so that our DbContext can inherit from it and gain user management functionality. Namely, it includes a new DbSet (and consequently, a table) for holding user accounts (aptly named Users), among other things that we won’t need.

IdentityUserContext has a more feature rich counterpart called IdentityDbContext, which also includes DbSets to support roles based authorization. We don’t need all that so we use its simpler cousin. Feel free to explore the source code on GitHub to see all it offers.

The generic type parameter that we give it, IdentityUser, is a class that’s also provided by the Identity library. Its purpose is to serve as a default Entity Type for our user model and, as a consequence, our Users DbSet.

In summary, by having our DbContext class inherit from IdentityUserContext<IdentityUser>, we’re telling Identity that we want it to augment our DbContext (and database) to include the core user management tables and that we want our users table to have the same columns as IdentityUser.

If we wanted to include more columns in our users table, what we would have to do is create a new class, make it inherit from IdentityUser, define any additional fields that we want on it, and use that class as a type parameter to IdentityUserContext. For us for now, the default works just fine. You can learn more about customizing Identity in the official docs.

The change looks like this:

-public class VehicleQuotesContext : DbContext
+public class VehicleQuotesContext : IdentityUserContext<IdentityUser>

Finally, IdentityUserContext has some logic that it needs to run when it is being created. In order to allow it to run that logic, let’s add the following line to our VehicleQuotesContext’s OnModelCreating method:

 protected override void OnModelCreating(ModelBuilder modelBuilder)
 {
+    base.OnModelCreating(modelBuilder);
     // ...
 }

This calls IdentityUserContext’s own OnModelCreating implementation so that it can set itself up properly.

The complete file should be looking like this now:

using Microsoft.AspNetCore.Identity;
using Microsoft.AspNetCore.Identity.EntityFrameworkCore;
using Microsoft.EntityFrameworkCore;
using VehicleQuotes.Models;

namespace VehicleQuotes
{
    public class VehicleQuotesContext : IdentityUserContext<IdentityUser>
    {
        public VehicleQuotesContext (DbContextOptions<VehicleQuotesContext> options)
            : base(options)
        {
        }

        public DbSet<Make> Makes { get; set; }
        public DbSet<Size> Sizes { get; set; }
        public DbSet<BodyType> BodyTypes { get; set; }

        public DbSet<Model> Models { get; set; }
        public DbSet<ModelStyle> ModelStyles { get; set; }
        public DbSet<ModelStyleYear> ModelStyleYears { get; set; }

        public DbSet<QuoteRule> QuoteRules { get; set; }
        public DbSet<QuoteOverride> QuoteOverides { get; set; }

        public DbSet<Quote> Quotes { get; set; }

        protected override void OnModelCreating(ModelBuilder modelBuilder)
        {
            base.OnModelCreating(modelBuilder);

            modelBuilder.Entity<Size>().HasData(
                new Size { ID = 1, Name = "Subcompact" },
                new Size { ID = 2, Name = "Compact" },
                new Size { ID = 3, Name = "Mid Size" },
                new Size { ID = 5, Name = "Full Size" }
            );

            modelBuilder.Entity<BodyType>().HasData(
                new BodyType { ID = 1, Name = "Coupe" },
                new BodyType { ID = 2, Name = "Sedan" },
                new BodyType { ID = 3, Name = "Hatchback" },
                new BodyType { ID = 4, Name = "Wagon" },
                new BodyType { ID = 5, Name = "Convertible" },
                new BodyType { ID = 6, Name = "SUV" },
                new BodyType { ID = 7, Name = "Truck" },
                new BodyType { ID = 8, Name = "Mini Van" },
                new BodyType { ID = 9, Name = "Roadster" }
            );
        }
    }
}

Applying database changes

The Users DbSet that we added into our data model by inheriting from IdentityUserContext<IdentityUser> will become a table once we create and apply a database migration. That’s what we will do next.

That’s simple in ASP.NET Core. All we need to do is run a command like this:

dotnet ef migrations add AddIdentityTables

That should produce a new migration file named something like 20220605003253_AddIdentityTables.cs. If you explore it, you’ll see how it contains the definitions for a few new database tables. Including the one that we want: AspNetUsers. That’s the one that we will use to store our user account records.

Next, we apply the migration with:

dotnet ef database update

If you now connect to the database, you’ll see the new tables in there.

If you have the database running in a Docker container like we discussed in the beginning of the article, you should be able to connect to it with:

$ psql -h localhost -U vehicle_quotes

Then, to see the tables:

vehicle_quotes=# \dt
                    List of relations
 Schema |         Name          | Type  |     Owner      
--------+-----------------------+-------+----------------
 public | AspNetUserClaims      | table | vehicle_quotes
 public | AspNetUserLogins      | table | vehicle_quotes
 public | AspNetUserTokens      | table | vehicle_quotes
 public | AspNetUsers           | table | vehicle_quotes
 ...
(14 rows)

And that’s pretty much it when it comes to having a sensible storage for user accounts. That was pretty inexpensive, wasn’t it? All we had to do was install some NuGet packages, tweak our existing DbContext, and run some migrations.

The best thing is that we’re not done yet. We can also take advantage of ASP.NET Core Identity to manage users programmatically. Instead of interacting with these tables directly, we will use the service classes provided by Identity to create new users, fetch existing ones and validate their credentials.

Before we can do that though, we must first do some configuration in our app’s Startup class so that said services are properly set up to our liking and are made available to our application.

Configuring the Identity services

We need to add some code to VehicleQuotes/Startup.cs. With it, we configure the Identity system and add a few service classes to ASP.NET Core’s IoC container so that they are available to our app via Dependency Injection.

We need a new using statement:

using Microsoft.AspNetCore.Identity;

And the following code added to the ConfigureServices method:

services
    .AddIdentityCore<IdentityUser>(options => {
        options.SignIn.RequireConfirmedAccount = false;
        options.User.RequireUniqueEmail = true;
        options.Password.RequireDigit = false;
        options.Password.RequiredLength = 6;
        options.Password.RequireNonAlphanumeric = false;
        options.Password.RequireUppercase = false;
        options.Password.RequireLowercase = false;
    })
    .AddEntityFrameworkStores<VehicleQuotesContext>();

The call to AddIdentityCore makes several Identity utility classes available to the application. Among those, UserManager is the only one we will use. We will use it later to… well, manage users. You can also see how we’ve set a few options related to how the user accounts are handled. options.SignIn.RequireConfirmedAccount controls whether new accounts need to be confirmed via email before they are available. With options.User.RequireUniqueEmail, we tell Identity to enforce uniqueness of emails on user accounts. And finally the options.Password.* options configure the password strength requirements.

You can explore all the available options in the official docs.

Then, the call to AddEntityFrameworkStores tells the Identity system that it should use our VehicleQuotesContext for data storage.

Creating users

With that configuration out of the way, we can now write some code to create new user accounts. To keep things simple, we’ll add a new UsersController to our project that will expose a new endpoint that offers that functionality.

Let’s start with this in VehicleQuotes/Controllers/UsersController.cs:

using Microsoft.AspNetCore.Identity;
using Microsoft.AspNetCore.Mvc;

namespace VehicleQuotes.Controllers
{
    [Route("api/[controller]")]
    [ApiController]
    public class UsersController : ControllerBase
    {
        private readonly UserManager<IdentityUser> _userManager;

        public UsersController(
            UserManager<IdentityUser> userManager
        ) {
            _userManager = userManager;
        }
    }
}

As you can see, this controller defines a dependency on UserManager<IdentityUser>, an instance of which is injected via the constructor. This is one of the classes made available to us when we configured the Identity core services in our app’s Startup. We will use it to create new user records.

Before that though, we need to define a new class that encapsulates the payload for a request to our endpoint. When it comes to creating user accounts, all we need is a username, a password, and an email. As such, we add the following class in a new VehicleQuotes/ResourceModels/User.cs file:

using System.ComponentModel.DataAnnotations;

namespace VehicleQuotes.ResourceModels
{
    public class User
    {
        [Required]
        public string UserName { get; set; }
        [Required]
        public string Password { get; set; }
        [Required]
        public string Email { get; set; }
    }
}

Now, we shall use this type as the parameter for a new PostUser action method in the UserController which will expose the new user account creation endpoint in our API. The method looks like this:

// POST: api/Users
[HttpPost]
public async Task<ActionResult<User>> PostUser(User user)
{
    if (!ModelState.IsValid)
    {
        return BadRequest(ModelState);
    }

    var result = await _userManager.CreateAsync(
        new IdentityUser() { UserName = user.UserName, Email = user.Email },
        user.Password
    );

    if (!result.Succeeded)
    {
        return BadRequest(result.Errors);
    }

    user.Password = null;
    return Created("", user);
}

Be sure to also add the following using statements:

using System.Threading.Tasks;
using VehicleQuotes.ResourceModels;

System.Threading.Tasks allows us to reference the class Task<> which is the return type of the new PostUser method. VehicleQuotes.ResourceModels is where our new User class lives.

Thanks to the instance of UserManager<IdentityUser> that we’re holding onto, this method is very straightforward. The most interesting portion is this:

var result = await _userManager.CreateAsync(
    new IdentityUser() { UserName = user.UserName, Email = user.Email },
    user.Password
);

Here we’re newing up an IdentityUser instance using the given request parameters and passing it on to UserManager<IdentityUser>’s CreateAsync method, along with the password that was also given in the incoming request. This puts the Identity system to work for us and properly create a new user account.

Then, we can inspect its return value (which we capture in the result variable) to determine if the operation was successful. That way we can respond appropriately to our API’s caller.

Finally, with…

user.Password = null;
return Created("", user);

we return the data representing the newly created user account but we’re discreet and make sure not to include the password.

With that, we can test our API. Fire it up with dotnet run (or dotnet watch!) and send a POST request to our new endpoint at http://0.0.0.0:5000/api/Users with a payload like this:

{
  "userName": "newuser000",
  "email": "newuser000@endpointdev.com",
  "password": "password"
}

I just tried it in Postman and this is what it looked like:

Feel free to test it out further. Try repeated emails or usernames. Try passwords that don’t meet the criteria we defined when configuring Identity in the app’s Startup class. It all works as you’d expect.

You can inspect the database and see the newly created record too:

vehicle_quotes=# select id, user_name, email, password_hash from "AspNetUsers";
-[ RECORD 1 ]-+--------------------------------------
id            | aaad07b4-f109-4255-8caa-185fd7694c72
user_name     | newuser000
email         | newuser000@endpointdev.com
password_hash | AQAAAAEAACcQAAAAEJJTV7M2Ejqd3K3iC...

And there it is, that’s our record. With a hashed password and everything.

Now let’s see what would an endpoint to fetch users look like:

Fetching users

UserManager<IdentityUser> offers a FindByNameAsync method that we can use to retrieve users by name. We can add a new endpoint that leverages it like so:

// GET: api/Users/username
[HttpGet("{username}")]
public async Task<ActionResult<User>> GetUser(string username)
{
    IdentityUser user = await _userManager.FindByNameAsync(username);

    if (user == null)
    {
        return NotFound();
    }

    return new User
    {
        UserName = user.UserName,
        Email = user.Email
    };
}

This is even more straightforward than the previous one. We just call the FindByNameAsync method by giving it the username of the account we want, make sure that we actually found it, and then return the data.

For the response, we use the same User class that we created to represent the input for the creation endpoint. If we added more fields to the user profile, we could include them here. Alas, we only have username and email for now.

Restart the app and we can now make a request like this:

Pretty neat, huh? Try a username that does not exist and you should see the API respond with a 404.

One quick improvement that we can do before we move on: let’s change PostUser’s return statement to this:
return CreatedAtAction("GetUser", new { username = user.UserName }, user);
All that does is include a Location header on the POST Users endpoint’s response that contains the URL for the newly created user. That’s just being a good citizen.

Implementing JWT Bearer Token authentication

Now that we have the user management capabilities that we need, let’s implement some actual authentication. We will start by adding JWT-based authentication to our API.

The strategy that we will use is to create a new API endpoint that clients can POST credentials to and that will respond to them with a fresh, short-lived token. They will then be able to use that token for subsequent requests by including it via headers.

We will pick a random endpoint to secure, just to serve as an example. That is, an endpoint that requires authentication in order to be accessed. GET api/BodyTypes is a good candidate. It is defined in VehicleQuotes/Controllers/BodyTypesController.cs’s GetBodyTypes action method. Feel free to test it out.

Creating tokens

Let’s start by creating the new endpoint that clients will call to obtain the auth tokens. In order to do so, we need a few things:

A class that represents incoming request data.
A class that represents outgoing response data.
A class that can generate tokens for a given user.
A new action method in UsersController that does the work.

The request data structure

To deal with step one, let’s define a new class in VehicleQuotes/ResourceModels/AuthenticationRequest.cs that looks like this:

using System.ComponentModel.DataAnnotations;

namespace VehicleQuotes.ResourceModels
{
    public class AuthenticationRequest
    {
        [Required]
        public string UserName { get; set; }
        [Required]
        public string Password { get; set; }
    }
}

All users need to authenticate is provide a set of credentials: username and password. So we have this class that contains fields for both and that will be used as input for our endpoint.

The response data structure

Next, we need to define a class to represent that endpoint’s response. I’ve added the following class in VehicleQuotes/ResourceModels/AuthenticationResponse.cs:

using System;

namespace VehicleQuotes.ResourceModels
{
    public class AuthenticationResponse
    {
        public string Token { get; set; }
        public DateTime Expiration { get; set; }
    }
}

Just a simple data structure containing the token itself and a date letting clients know when they can expect it to expire.

The class that creates JWTs

Step 3 is creating a class that can produce the tokens. This is the most interesting part in terms of complexity. Before we can do that though, let’s add the following configurations to VehicleQuotes/appsettings.json:

  "Jwt": {
    "Key": "this is the secret key for the jwt, it must be kept secure",
    "Issuer": "vehiclequotes.endpointdev.com",
    "Audience": "vehiclequotes.endpointdev.com",
    "Subject": "JWT for vehiclequotes.endpointdev.com"
  },

These are values that we’ll need when creating the tokens. We’ll go over the purpose of each them as they come up as we continue writing our code.

Here we’ll gloss over some details on the inner workings of JWTs as a standard. You can learn more about them at jwt.io.

For now, let’s add the following class in VehicleQuotes/Services/JwtService.cs:

using System;
using System.IdentityModel.Tokens.Jwt;
using System.Security.Claims;
using System.Text;
using Microsoft.AspNetCore.Identity;
using Microsoft.Extensions.Configuration;
using Microsoft.IdentityModel.Tokens;
using VehicleQuotes.ResourceModels;

namespace VehicleQuotes.Services
{
    public class JwtService
    {
        private const int EXPIRATION_MINUTES = 1;

        private readonly IConfiguration _configuration;

        public JwtService(IConfiguration configuration)
        {
            _configuration = configuration;
        }

        public AuthenticationResponse CreateToken(IdentityUser user)
        {
            var expiration = DateTime.UtcNow.AddMinutes(EXPIRATION_MINUTES);

            var token = CreateJwtToken(
                CreateClaims(user),
                CreateSigningCredentials(),
                expiration
            );

            var tokenHandler = new JwtSecurityTokenHandler();

            return new AuthenticationResponse {
                Token = tokenHandler.WriteToken(token),
                Expiration = expiration
            };
        }

        private JwtSecurityToken CreateJwtToken(Claim[] claims, SigningCredentials credentials, DateTime expiration) =>
            new JwtSecurityToken(
                _configuration["Jwt:Issuer"],
                _configuration["Jwt:Audience"],
                claims,
                expires: expiration,
                signingCredentials: credentials
            );

        private Claim[] CreateClaims(IdentityUser user) =>
            new[] {
                new Claim(JwtRegisteredClaimNames.Sub, _configuration["Jwt:Subject"]),
                new Claim(JwtRegisteredClaimNames.Jti, Guid.NewGuid().ToString()),
                new Claim(JwtRegisteredClaimNames.Iat, DateTime.UtcNow.ToString()),
                new Claim(ClaimTypes.NameIdentifier, user.Id),
                new Claim(ClaimTypes.Name, user.UserName),
                new Claim(ClaimTypes.Email, user.Email)
            };

        private SigningCredentials CreateSigningCredentials() =>
            new SigningCredentials(
                new SymmetricSecurityKey(
                    Encoding.UTF8.GetBytes(_configuration["Jwt:Key"])
                ),
                SecurityAlgorithms.HmacSha256
            );
    }
}

The CreateToken method is the element that’s most worth discussing here. It receives an instance of IdentityUser as a parameter (which, remember, is the entity class that represents our user accounts), uses it to construct a JWT, and returns it within a AuthenticationResponse object (which is what we decided that our endpoint would return). To do so, we use various classes built into .NET.

The main class that represents the JWT is JwtSecurityToken. We new up one of those in the CreateJwtToken method with this call:

new JwtSecurityToken (
    _configuration["Jwt:Issuer"],
    _configuration["Jwt:Audience"],
    claims,
    expires: expiration,
    signingCredentials: credentials
);

“Issuer” and “Audience” are two important values for how JWTs work. They specify which entity is creating the token (i.e. the “Issuer”) and which entity is the token intended for (i.e. the “Audience”). We use the IConfiguration instance that we got as a dependency to fetch their values from the VehicleQuotes/appsettings.json file.

The “Issuer” and “Audience” parameters in JwtSecurityToken’s constructor correspond to JWT claims iss and aud, respectively. You can learn more about them and other claims in the RFC.

The next parameter that JwtSecurityToken’s constructor needs is the claims array. In JWT terms, a claim is essentially a statement about the entity for which the token is generated, some data that identifies it. For example, if we’re generating a token for a user, what you would expect to see in such a token’s claims are things like username, email, and any other non-secret profile info.

In our case, as you can see in the CreateClaims method, we add a number of claims:

new[] {
    new Claim(JwtRegisteredClaimNames.Sub, _configuration["Jwt:Subject"]),
    new Claim(JwtRegisteredClaimNames.Jti, Guid.NewGuid().ToString()),
    new Claim(JwtRegisteredClaimNames.Iat, DateTime.UtcNow.ToString()),
    new Claim(ClaimTypes.NameIdentifier, user.Id),
    new Claim(ClaimTypes.Name, user.UserName),
    new Claim(ClaimTypes.Email, user.Email)
};

Along with the user id, name and email, we also add sub, jti and iat claims. These are standardized claims of which you can learn more about in the RFC. The data that we put in here will make its way into the encoded token that our API caller eventually sees. We’ll see that later.

The next parameter to JwtSecurityToken is the expiration date of the token. Here we are setting it to just one minute in the future to comply with our original requirement that the token should be short-lived so that it only allows a handful of requests over a short period of time.

Finally there’s the signingCredentials parameter which tells the JwtSecurityToken how to cryptographically sign the token. As you can see in the code:

new SigningCredentials(
    new SymmetricSecurityKey(
        Encoding.UTF8.GetBytes(_configuration["Jwt:Key"])
    ),
    SecurityAlgorithms.HmacSha256
);

That’s a SigningCredentials instance that we create using the “key” that we have configured in VehicleQuotes/appsettings.json, along with the algorithm to use to produce it.

And that’s about it. There isn’t much else to this class. It is somewhat involved but that’s just how JWTs are created in .NET.

The action method that puts it all together

Now all we need to do is actually create an endpoint that exposes this functionality to clients. Since we have the core logic encapsulated in our JwtService class, the actual action method is simple. Here are the changes that we need to make in order to implement it:

First, on VehicleQuotes/Controllers/UsersController.cs, we add a new using statement so that we can reference our new JwtService class:

using VehicleQuotes.Services;

Next, we declare a new parameter of type JwtService in the controller’s constructor so that we signal to ASP.NET Core’s Dependency Injection subsystem that we want to use one of those. We also hold onto it via a new instance variable called _jwtService:

 // ...
 public class UsersController : ControllerBase
 {
     private readonly UserManager<IdentityUser> _userManager;
+    private readonly JwtService _jwtService;
 
     public UsersController(
         UserManager<IdentityUser> userManager,
+        JwtService jwtService
     ) {
         _userManager = userManager;
+        _jwtService = jwtService;
     }
     // ...
 }

We also need to tell ASP.NET Core that JwtService should be available for Dependency Injection. To do so, we can add this line in VehicleQuotes/Startup.cs’s ConfigureServices method:

services.AddScoped<Services.JwtService>();

Note that this way of defining dependencies is not recommended and only done this way here to keep things simple for an illustrative app that’s never going to run in production. Ideally what you want to do here is define the dependency as an abstraction (i.e. an interface) and a concrete implementation that fulfills it. For example:
services.AddScoped<ITokenCreationService, JwtService>();
This way, classes that depend on this service can reference the interface, not the concrete type. In doing that, they adhere to the Dependency Inversion principle and become more easily testable because they allow mocks to be provided as dependencies.

Finally, we write the actual action method:

// POST: api/Users/BearerToken
[HttpPost("BearerToken")]
public async Task<ActionResult<AuthenticationResponse>> CreateBearerToken(AuthenticationRequest request)
{
    if (!ModelState.IsValid)
    {
        return BadRequest("Bad credentials");
    }

    var user = await _userManager.FindByNameAsync(request.UserName);

    if (user == null)
    {
        return BadRequest("Bad credentials");
    }

    var isPasswordValid = await _userManager.CheckPasswordAsync(user, request.Password);

    if (!isPasswordValid)
    {
        return BadRequest("Bad credentials");
    }

    var token = _jwtService.CreateToken(user);

    return Ok(token);
}

Here too we’re leveraging the UserManager instance that ASP.NET Core Identity so graciously provided us with. In summary, we find the user account by name using the incoming request data, check if the given password is correct, then ask our JwtService to create a token for this user, and finally return it wrapped in a 200 response.

If you hit that endpoint with a POST and a set of existing credentials, you should see something like this:

If you take that big string that came back in the "token" field in the response JSON, and paste it in jwt.io’s token decoder, you should be able to see all the claims that we added to the token:

Notice that the keywords here are “Encoded” and “Decoded”. The claims that we put in our JWTs are not protected in any way. As such, we should never put secrets in there.

Securing an endpoint with JWT authentication

Now that we have a way for obtaining tokens, let’s see how we can actually use them to gain access to some resources. To demonstrate that, let’s secure an endpoint in a way that it denies unauthenticated requests.

Applying the Authorize attribute

First we need to signal ASP.NET Core that the endpoint requires auth. We do that by annotating the corresponding action method with the Authorize attribute. We’ve already decided that we were going to use VehicleQuotes/Controllers/BodyTypesController.cs’s GetBodyTypes method as a guinea pig. So, let’s go into that file and add the following using statement:

using Microsoft.AspNetCore.Authorization;

That will allow us access to the attribute, which we can apply to the action method like so:

 // GET: api/BodyTypes
+ [Authorize]
  [HttpGet]
  public async Task<ActionResult<IEnumerable<BodyType>>> GetBodyTypes()
  {
      return await _context.BodyTypes.ToListAsync();
  }

With this, we’ve told ASP.NET Core that we want it to require auth for this endpoint.

Enabling and configuring the JWT authentication

Now, we need to tell it how to actually perform the check. To do that, we need to install the Microsoft.AspNetCore.Authentication.JwtBearer NuGet package. We can do that from the VehicleQuotes directory with the following command:

$ dotnet add package Microsoft.AspNetCore.Authentication.JwtBearer

Once we have that installed, we get access to additional services which we can use to configure the “JwtBearer” authentication scheme. As usual, we do the configuration in VehicleQuotes/Startup.cs’s. Here’s what we need to do:

Add a few new using statements:

using Microsoft.AspNetCore.Authentication.JwtBearer;
using Microsoft.IdentityModel.Tokens;
using System.Text;

Add the following code at the end of the ConfigureServices method:

services
    .AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
    .AddJwtBearer(options =>
    {
        options.TokenValidationParameters = new TokenValidationParameters()
        {
            ValidateIssuer = true,
            ValidateAudience = true,
            ValidateLifetime = true,
            ValidateIssuerSigningKey = true,
            ValidAudience = Configuration["Jwt:Audience"],
            ValidIssuer = Configuration["Jwt:Issuer"],
            IssuerSigningKey = new SymmetricSecurityKey(
                Encoding.UTF8.GetBytes(Configuration["Jwt:Key"])
            )
        };
    });

The call to AddAuthentication includes all the internal core service classes that are needed to do authentication in our app.

The JwtBearerDefaults.AuthenticationScheme parameter that we give it is the name of the authentication scheme to use as the default. More on that later. For now, know that ASP.NET Core supports multiple authentication schemes to be used at the same time. In fact, our own plan here is to eventually support two auth schemes: JWTs and API Keys. We’re starting with JWT first and as such, that’s the one we specify as the default.

The call to AddJwtBearer configures the JWT authentication scheme. That is, it allows the app to perform authentication checks based on an incoming JWT token.

The most important part of the configuration is the values that we are passing to TokenValidationParameters. As you can see, we are able to specify which aspects of the incoming JWTs to validate. You can see all available options for TokenValidationParameters in the official documentation. Here we’ve chosen to validate obvious things like the issuer, audience, and signing key.

The last configuration step is to add this line right before app.UseAuthorization(); in the Configure method:

app.UseAuthentication();

That will enable the authentication middleware. That way the framework will perform authentication checks as part of the request processing pipeline. In other words, actually put all the configuration that we’ve done to good use.

Alright, now that we have all the configuration ready and have secured an endpoint, let’s try hitting it with a GET on api/BodyTypes and see what happens:

Ok! So far so good. We made an unauthenticated request into an endpoint that requires authentication and as a result we got a 401 back. That’s just what we wanted.

Now, let’s get a token by POSTing to api/Users/BearerToken:

We can copy that token and include as a header in the GET request to api/BodyTypes. The header key should be Authorization and the value should be Bearer <our token>. In Postman, we can setup a similar request if we choose the “Authorization” tab, select “Bearer Token” in the “Type” drop-down list and paste the token in the “Token” text box.

Do that, and you should now see a 200 response from the endpoint:

Neat! Now that we passed a valid token, it wants to talk to us again. Let a few minutes pass, enough for the token to expire and try the request again to see how it’s 401’ing again.

Implementing API Key authentication

Now let’s change gears from JWT and implement an alternate authentication strategy in our Web API: API Keys. Authentication with API Keys is fairly common in the web service world.

The core idea of API Keys is that the API provider (in this case, us) produces a secret string that is given to the clients for safekeeping. The clients then can use that key to access the API and make as many requests as they want. So it’s essentially just a glorified password.

The main functional differences when it comes to JWT as we have implemented it is that the API Keys won’t expire and that we will store them in our database. Then, when processing incoming requests, as part of the authentication step, our app will check if the given API Key exists in our internal records. If it does, then the request is allowed to go through, if it does not, then a 401 is sent back.

This is going to be interesting because ASP.NET Core does not include an authentication scheme out of the box for this kind of behavior. For JWT, we were able to use the Microsoft.AspNetCore.Authentication.JwtBearer package. For this we have to get our hands dirty and actually implement a custom authentication handler that will run the logic that we need.

Let’s get started.

Creating API Keys

Similarly to when we implemented JWTs, we’ll start by creating the new endpoint that clients will call to obtain their keys. In order to make it work, we’ll need:

A new model and database table to store the keys
A service class to create the keys.
A new action method in UsersController that does the work.

The API Key model and database table

For storing API Keys, we just need a simple table that is linked to users via a one-to-many relationship and includes a name and a value. Using Entity Framework, our model could look like this in VehicleQuotes/Models/UserApiKey.cs:

using System.ComponentModel.DataAnnotations;
using System.Text.Json.Serialization;
using Microsoft.AspNetCore.Identity;
using Microsoft.EntityFrameworkCore;

namespace VehicleQuotes.Models
{
    [Index(nameof(Value), IsUnique = true)]
    public class UserApiKey
    {
        [JsonIgnore]
        public int ID { get; set; }

        [Required]
        public string Value { get; set; }

        [JsonIgnore]
        [Required]
        public string UserID { get; set; }

        [JsonIgnore]
        public IdentityUser User { get; set; }
    }
}

Easy enough. We express the relationship between the “keys” and “users” tables via the User navigation property and the UserID field. We also define a unique index on the Value field and annotate some fields with Required because we don’t want them to allow null. Interestingly, we also annotated some of the fields with the JsonIgnore attribute so that when we include objects of this type in API responses (which, as you’ll see, we will), those fields are not included.

We also need to add a new DbSet on our DbContext class at VehicleQuotes/Data/VehicleQuotesContext.cs so that Entity Framework picks it up as part of the data model:

public DbSet<UserApiKey> UserApiKeys { get; set; }

With that, let’s now create and run a migration so that the new table can be added to the database. All it takes is running this to create the migration:

dotnet ef migrations add AddUserApiKeysTable

And this to apply it:

dotnet ef database update

Now the new table should appear in the database:

vehicle_quotes=# \dt
                    List of relations
 Schema |         Name          | Type  |     Owner      
--------+-----------------------+-------+----------------
...
 public | user_api_keys         | table | vehicle_quotes
 (15 rows)

The class that creates API Keys

The next step is very straightforward. All we need is some logic that, when given a user account (in the form of an instance of IdentityUser), can generate a new API Key, insert it in the database, and return it. The following unremarkable class, defined in VehicleQuotes/Services/ApiKeyService.cs, encapsulates said logic:

using System;
using Microsoft.AspNetCore.Identity;
using VehicleQuotes.Models;

namespace VehicleQuotes.Services
{
    public class ApiKeyService
    {
        private readonly VehicleQuotesContext _context;

        public ApiKeyService(VehicleQuotesContext context)
        {
            _context = context;
        }

        public UserApiKey CreateApiKey(IdentityUser user)
        {
            var newApiKey = new UserApiKey
            {
                User = user,
                Value = GenerateApiKeyValue()
            };

            _context.UserApiKeys.Add(newApiKey);

            _context.SaveChanges();

            return newApiKey;
        }

        private string GenerateApiKeyValue() =>
            $"{Guid.NewGuid().ToString()}-{Guid.NewGuid().ToString()}";
    }
}

As you can see, this class’s single public method, aptly named CreateApiKey, just takes the given user and creates a new UserApiKey that’s associated with it and saves it into the database. The key values themselves are just two GUIDs concatenated.

This is an admittedly simplistic and not-all-that-secure method for generating the keys themselves. For simplicity, we’ve gone with this. In a production app however, it’d be better to use a true cryptographically secure string. Here’s an article that explains how to do that with .NET.

Naturally, this class needs to be used elsewhere in the application, so let’s make it available for Dependency Injection by adding the following line to VehicleQuotes/Startup.cs’s ConfigureServices method.

services.AddScoped<Services.ApiKeyService>();

The action method that puts it all together

Now let’s finally write a new action method on VehicleQuotes/Controllers/UsersController.cs so that there’s an endpoint that clients can call to get API Keys. Before adding the action method though, as usual, we need to declare the following dependency as a constructor parameter and hold it in an instance variable:

 public class UsersController : ControllerBase
 {
     private readonly UserManager<IdentityUser> _userManager;
     private readonly JwtService _jwtService;
+    private readonly ApiKeyService _apiKeyService;
 
     public UsersController(
         UserManager<IdentityUser> userManager,
         JwtService jwtService,
+        ApiKeyService apiKeyService 
     ) {
         _userManager = userManager;
         _jwtService = jwtService;
+        _apiKeyService = apiKeyService;
     }}

The action method itself would look like this:

// POST: api/Users/ApiKey
[HttpPost("ApiKey")]
public async Task<ActionResult> CreateApiKey(AuthenticationRequest request)
{
    if (!ModelState.IsValid)
    {
        return BadRequest(ModelState);
    }

    var user = await _userManager.FindByNameAsync(request.UserName);

    if (user == null)
    {
        return BadRequest("Bad credentials");
    }

    var isPasswordValid = await _userManager.CheckPasswordAsync(user, request.Password);

    if (!isPasswordValid)
    {
        return BadRequest("Bad credentials");
    }

    var token = _apiKeyService.CreateApiKey(user);

    return Ok(token);
}

This method is very similar to the other one that generates JWTs. Painfully so. In fact the only difference is that this one calls the ApiKeyService’s CreateApiKey method instead of JwtService’s CreateToken, for obvious reasons.

These two are ripe for some refactoring to eliminate duplication but we won’t do that here. Take a look at the finished project on GitHub to see a neat refactoring option.

Anyway, you can now try a POST into api/Users/ApiKey, pass it a valid set of credentials, and you should see it respond with a brand new API Key:

The key can also be found in the database:

vehicle_quotes=# select id, SUBSTRING(value,1,13), user_id from user_api_keys;
 id |   substring   |               user_id                
----+---------------+--------------------------------------
  1 | efd2133b-cc4a | aaad07b4-f109-4255-8caa-185fd7694c72
(1 row)

Securing an endpoint with API Key authentication

Now that users have a way of getting API Keys, we can start securing endpoints with that authentication scheme. In order to acomplish that, we need to take three steps:

Implement a custom authentication handler that runs the authentication logic.
Configure the new authentication handler, plugging it into ASP.NET Core’s auth mechanisms.
Secure certain endpoints via the Authorization attribute, specifying the scheme that we want to use to do so.

We’ll do that next.

Implementing the API Key authentication handler

Like we’ve touched on before, the “authentication handler” is a class that can plug into ASP.NET Core’s authentication middleware and can run the authentication logic for a given authentication scheme. In practical terms, it’s a class that inherits from Microsoft.AspNetCore.Authentication.AuthenticationHandler and implements the HandleAuthenticateAsync method. Ours will live in VehicleQuotes/Authentication/ApiKey/ApiKeyAuthenticationHandler.cs and it looks like this:

using System.Security.Claims;
using System.Text.Encodings.Web;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Authentication;
using Microsoft.AspNetCore.Identity;
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;

namespace VehicleQuotes.Authentication.ApiKey
{
    class ApiKeyAuthenticationHandler : AuthenticationHandler<AuthenticationSchemeOptions>
    {
        private const string API_KEY_HEADER = "Api-Key";

        private readonly VehicleQuotesContext _context;

        public ApiKeyAuthenticationHandler(
            IOptionsMonitor<AuthenticationSchemeOptions> options,
            ILoggerFactory logger,
            UrlEncoder encoder,
            ISystemClock clock,
            VehicleQuotesContext context
        ) : base(options, logger, encoder, clock)
        {
            _context = context;
        }

        protected override async Task<AuthenticateResult> HandleAuthenticateAsync()
        {
            if (!Request.Headers.ContainsKey(API_KEY_HEADER))
            {
                return AuthenticateResult.Fail("Header Not Found.");
            }

            string apiKeyToValidate = Request.Headers[API_KEY_HEADER];

            var apiKey = await _context.UserApiKeys
                .Include(uak => uak.User)
                .SingleOrDefaultAsync(uak => uak.Value == apiKeyToValidate);

            if (apiKey == null)
            {
                return AuthenticateResult.Fail("Invalid key.");
            }

            return AuthenticateResult.Success(CreateTicket(apiKey.User));
        }

        private AuthenticationTicket CreateTicket(IdentityUser user)
        {
            var claims = new[] {
                new Claim(ClaimTypes.NameIdentifier, user.Id),
                new Claim(ClaimTypes.Name, user.UserName),
                new Claim(ClaimTypes.Email, user.Email)
            };

            var identity = new ClaimsIdentity(claims, Scheme.Name);
            var principal = new ClaimsPrincipal(identity);
            var ticket = new AuthenticationTicket(principal, Scheme.Name);

            return ticket;
        }
    }
}

The first thing to note is how we’ve defined the base class: AuthenticationHandler<AuthenticationSchemeOptions>. AuthenticationHandler is a generic, and it allows us to control the type of options object that it accepts. In this case, we’ve used AuthenticationSchemeOptions, which is just a default one already provided by the framework. This is because we don’t really need any custom options.

If we did, then we would define a new class that inherits from AuthenticationSchemeOptions, give it the new fields that we need, and use that as a generic time parameter instead. We would then be able to set values for these new fields during configuration in the app’s Startup class. Just like we’ve done with the “Jwt Bearer” auth scheme via the AddJwtBearer method.

The next point of interest in this class is its constructor. The key aspect of it is that it calls base and passes it a series of parameters that it itself gets. This is because AuthenticationHandler does have some constructor logic that needs to be run for things to work properly, so we make sure to do that.

Finally, there’s the HandleAuthenticateAsync method which does the actual authentication. This method inspects the incoming request looking for an “Api-Key” header. If it finds it, it takes its value and makes a query into the database to try and find a record in the user_api_keys table that matches the incoming value. If it finds that, it creates an AuthenticationTicket which contains the identity of the user that the key belongs to, and returns it wrapped in a AuthenticateResult.Success. That return value signals ASP.NET Core’s authentication middleware that the request is authentic. Otherwise, it returns AuthenticateResult.Fail, which prompts ASP.NET Core to halt the request and return a 401.

Something interesting to note is how the AuthenticationTicket, similarly to the JWT Bearer scheme’s JwtSecurityToken, also includes an array of “claims” that serve to identify the user that has been authenticated. The “claims” concept is central to how auth works in ASP.NET Core.

Enabling and configuring the API Key authentication

Now we need to tell the framework that we want it to use a new authentication scheme for our app, spearheaded by the custom authentication handler that we just wrote. There are two steps to make that happen.

First, we configure our new scheme in the app’s Startup class. For that, we need these two new using statements:

using Microsoft.AspNetCore.Authentication;
using VehicleQuotes.Authentication.ApiKey;

Then we add this right after the AddJwtBearer call in the ConfigureServices method:

 public void ConfigureServices(IServiceCollection services)
 {
     // ...
     services
         .AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
         .AddJwtBearer(options =>
         {
             options.TokenValidationParameters = new TokenValidationParameters()
             {
                 ValidateIssuer = true,
                 ValidateAudience = true,
                 ValidateLifetime = true,
                 ValidateIssuerSigningKey = true,
                 ValidAudience = Configuration["Jwt:Audience"],
                 ValidIssuer = Configuration["Jwt:Issuer"],
                 IssuerSigningKey = new SymmetricSecurityKey(
                     Encoding.UTF8.GetBytes(Configuration["Jwt:Key"])
                 )
             };
         })
+        .AddScheme<AuthenticationSchemeOptions, ApiKeyAuthenticationHandler>(
+            "ApiKey",
+            options => { }
+        );
 }

A simple configuration as far as auth schemes go. We specify both the types of the authentication handler and the options object that it accepts via the generic type parameters to the AddScheme method. "ApiKey" is just the name we’ve given our custom auth scheme. It can be anything as long as it’s not “Bearer”, which is the name of the Jwt Bearer scheme (which is the value returned by JwtBearerDefaults.AuthenticationScheme). We’ll refer back to it later. Finally, since we have no special options to give it, we specify a lambda that does nothing with its options.

Updating the Authorize attribute to use our two schemes

Now that the auth scheme is configured, we need to use it to secure an endpoint. We can use the same that we used for JWT: POST api/BodyTypes, defined in BodyTypesController’s GetBodyTypes action method. Remember though, that we wanted to have both auth schemes (JWT and API Key) work on the endpoint. For that, the Authorize attribute allows a comma separated string of scheme names as a parameter. So, we can get both our configured schemes working if we update the attribute like this:

[Authorize(AuthenticationSchemes = $"{JwtBearerDefaults.AuthenticationScheme},ApiKey")]
public async Task<ActionResult<IEnumerable<BodyType>>> GetBodyTypes()
{
    return await _context.BodyTypes.ToListAsync();
}

JwtBearerDefaults.AuthenticationScheme contains the name of the JWT Bearer auth scheme. Next to it, we just put “ApiKey”, which is the name we have given our new custom one.

It’d be nice to put that “ApiKey” string somewhere safe similar to how the “Jwt Bearer” scheme has it in a constant. Take a look at the finished project on GitHub to see one way to organize that and also how to make the Startup configuration a little less verbose by using extension methods.

And finally, let’s test our endpoint and marvel at the fruit of our work.

Make sure to create a new API Key by POSTing to api/Users/ApiKey, and copy the “value” from the response. We can use it as the value for the Api-Key header a GET request to api/BodyTypes. In Postman, we can do that by choosing the “Authorization” tab, selecting “API Key” in the “Type” drop-down list, writing “Api-Key” in the “Key” text box, and putting our key in the “Value” text box.

With that, we can make the request and see how the endpoint now allows authentication via API Keys:

Of course, requests authenticated via JWT will also work for this endpoint.

That’s all

And there you have it! We’ve updated an existing ASP.NET Core Web API application so that it supports authentication using two strategies: JWT and API Keys. We leveraged the Identity libraries to securely store and manage user accounts. We used ASP.NET Core’s built-in authentication capabilities to enable JWT generation and usage. For API Keys, the framework didn’t provide an implementation out of the box. However, it proved to be extensible enough so that we could implement our own.

Database integration testing with .NET

2022-01-12T00:00:00+00:00

Ruby on Rails is great. We use it at End Point for many projects with great success. One of Rails’ cool features is how easy it is to write database integration tests. Out of the box, Rails projects come with all the configuration necessary to set up a database that’s exclusive for the automated test suite. This database includes all the tables and other objects that exist within the regular database that the app uses during its normal execution. So, it is very easy to write automated tests that cover the application components that interact with the database.

ASP.NET Core is also great! However, it doesn’t have this feature out of the box. Let’s see if we can’t do it ourselves.

The sample project

As a sample project we will use a REST API that I wrote for another article. Check it out if you want to learn more about the ins and outs of developing REST APIs with .NET. You can find the source code on GitHub.

The API is very straightforward. It provides a few endpoints for CRUDing some database tables. It also provides an endpoint which, when given some vehicle information, will calculate a monetary value for that vehicle. That’s a feature that would be interesting for us to cover with some tests.

The logic for that feature is backed by a specific class and it depends heavily on database interactions. As such, that class is a great candidate for writing a few automated integration tests against. The class in question is QuotesService which is defined in Services/QuoteService.cs. The class provides features for fetching records from the database (the GetAllQuotes method) as well as creating new records based on data from the incoming request and a set of rules stored in the database itself (the CalculateQuote method).

In order to add automated tests, the first step is to organize our project so that it supports them. Let’s do that next.

Organizing the source code to allow for automated testing

In general, the source code of most real world .NET applications is organized as one or more “projects” under one “solution”. A solution is a collection of related projects, and a project is something that produces a deployment artifact. An artifact is a library (i.e. a *.dll file) or something that can be executed like a console or web app.

Our sample app is a stand-alone “webapi” project, meaning that it’s not within a solution. For automated tests, however, we need to create a new project for tests, parallel to our main one. Now that we have two projects instead of one, we need to reorganize the sample app’s source code to comply with the “projects in a solution” structure I mentioned earlier.

Let’s start by moving all the files in the root directory into a new VehicleQuotes directory. That’s one project. Then, we create a new automated tests project by running the following, still from the root directory:

dotnet new xunit -o VehicleQuotes.Tests

That creates a new automated tests project named VehicleQuotes.Tests (under a new aptly-named VehicleQuotes.Tests directory) which uses the xUnit.net test framework. There are other options when it comes to test frameworks in .NET, such as MSTest and NUnit. We’re going to use xUnit.net, but the others should work just as well for our purposes.

Now, we need to create a new solution to contain those two projects. Solutions come in the form of *.sln files and we can create ours like so:

dotnet new sln -o vehicle-quotes

That should’ve created a new vehicle-quotes.sln file for us. We should now have a file structure like this:

.
├── vehicle-quotes.sln
├── VehicleQuotes
│   ├── VehicleQuotes.csproj
│   └── ...
└── VehicleQuotes.Tests
    ├── VehicleQuotes.Tests.csproj
    └── ...

Like I said, the *.sln file indicates that this is a solution. The *.csproj files identify the individual projects that make up the solution.

Now, we need to tell dotnet that those two projects belong in the same solution. These commands do that:

dotnet sln add ./VehicleQuotes/VehicleQuotes.csproj
dotnet sln add ./VehicleQuotes.Tests/VehicleQuotes.Tests.csproj

Finally, we update the VehicleQuotes.Tests project so that it references the VehicleQuotes project. That way, the test suite will have access to all the classes defined in the REST API. Here’s the command for that:

dotnet add ./VehicleQuotes.Tests/VehicleQuotes.Tests.csproj reference ./VehicleQuotes/VehicleQuotes.csproj

With all that setup out of the way, we can now start writing some tests.

You can learn more about project organization in the official online documentation.

Creating a DbContext instance to talk to the database

The VehicleQuotes.Tests automated tests project got created with a default test file named UnitTest1.cs. You can delete it or ignore it, since we will not use it.

In general, it’s a good idea for the test project to mimic the directory structure of the project that it will be testing. Also, we already decided that we would focus our test efforts on the QuoteService class from the VehicleQuotes project. That class is defined in VehicleQuotes/Services/QuoteService.cs, so let’s create a similarly located file within the test project which will contain the test cases for that class. Here: VehicleQuotes.Tests/Services/QuoteServiceTests.cs. These would be the contents:

// VehicleQuotes.Tests/Services/QuoteServiceTests.cs

using System;
using Xunit;

namespace VehicleQuotes.Tests.Services
{
    public class QuoteServiceTests
    {
        [Fact]
        public void GetAllQuotesReturnsEmptyWhenThereIsNoDataStored()
        {
            // Given

            // When

            // Then
        }
    }
}

This is the basic structure for tests using xUnit.net. Any method annotated with a [Fact] attribute will be picked up and run by the test framework. In this case, I’ve created one such method called GetAllQuotesReturnsEmptyWhenThereIsNoDataStored which should give away its intention. This test case will validate that QuoteService’s GetAllQuotes method returns an empty set when called with no data in the database.

Before we can write this test case, though, the suite needs access to the test database. Our app uses Entity Framework Core for database interaction, which means that the database is accessed via a DbContext class. Looking at the source code of our sample app, we can see that the DbContext being used is VehicleQuotesContext, defined in VehicleQuotes/Data/VehicleQuotesContext.cs. Let’s add a utility method to the QuoteServiceTests class which can be used to create new instances of VehicleQuotesContext:

// VehicleQuotes.Tests/Services/QuoteServiceTests.cs

// ...
using Microsoft.EntityFrameworkCore;
using VehicleQuotes.Services;

namespace VehicleQuotes.Tests.Services
{
    public class QuoteServiceTests
    {
        private VehicleQuotesContext CreateDbContext()
        {
            var options = new DbContextOptionsBuilder<VehicleQuotesContext>()
                .UseNpgsql("Host=db;Database=vehicle_quotes_test;Username=vehicle_quotes;Password=password")
                .UseSnakeCaseNamingConvention()
                .Options;

            var context = new VehicleQuotesContext(options);

            context.Database.EnsureCreated();

            return context;
        }

        // ...
    }
}

As you can see, we need to go through three steps to create the VehicleQuotesContext instance and get a database that’s ready for testing:

First, we create a DbContextOptionsBuilder and use that to obtain the options object that the VehicleQuotesContext needs as a constructor parameter. We needed to include the Microsoft.EntityFrameworkCore namespace in order to have access to the DbContextOptionsBuilder. For this, I just copied and slightly modified this statement from the ConfigureServices method in the REST API’s VehicleQuotes/Startup.cs file:

// VehicleQuotes/Startup.cs

public void ConfigureServices(IServiceCollection services)
{
    // ...

    services.AddDbContext<VehicleQuotesContext>(options =>
        options
            .UseNpgsql(Configuration.GetConnectionString("VehicleQuotesContext"))
            .UseSnakeCaseNamingConvention()
            .UseLoggerFactory(LoggerFactory.Create(builder => builder.AddConsole()))
            .EnableSensitiveDataLogging()
    );

    // ...
}

This is a method that runs when the application is starting up to set up all the services that the app uses to work. Here, it’s setting up the DbContext to enable database interaction. For the test suite, I took this statement as a starting point and removed the logging configurations and specified a hardcoded connection string that specifically points to a new vehicle_quotes_test database that will be used for testing.

If you’re following along, then you need a PostgreSQL instance that you can use to run the tests. In my case, I have one running that is reachable via the connection string I specified: Host=db;Database=vehicle_quotes_test;Username=vehicle_quotes;Password=password.

If you have Docker, a quick way to get a Postgres database up and running is with this command:

docker run -d \
    --name vehicle-quotes-db \
    -p 5432:5432 \
    --network host \
    -e POSTGRES_DB=vehicle_quotes \
    -e POSTGRES_USER=vehicle_quotes \
    -e POSTGRES_PASSWORD=password \
    postgres

That’ll spin up a new Postgres instance that’s reachable via localhost.

Secondly, now that we have the options parameter ready, we can quite simply instantiate a new VehicleQuotesContext:

var context = new VehicleQuotesContext(options);

Finally, we call the EnsureCreated method so that the database that we specified in the connection string is actually created.

context.Database.EnsureCreated();

This is the database that our test suite will use.

Defining the test database connection string in the appsettings.json file

One quick improvement that we can do to the code we’ve written so far is move the connection string for the test database into a separate configuration file, instead of having it hardcoded. Let’s do that next.

We need to create a new appsettings.json file under the VehicleQuotes.Tests directory. Then we have to add the connection string like so:

{
  "ConnectionStrings": {
    "VehicleQuotesContext": "Host=db;Database=vehicle_quotes_test;Username=vehicle_quotes;Password=password"
  }
}

This is the standard way of configuring connection strings in .NET. Now, to actually fetch this value from within our test suite code, we make the following changes:

// ...
+using Microsoft.Extensions.Hosting;
+using Microsoft.Extensions.Configuration;
+using Microsoft.Extensions.DependencyInjection;

namespace VehicleQuotes.Tests.Services
{
    public class QuoteServiceTests
    {
        private VehicleQuotesContext CreateDbContext()
        {
+           var host = Host.CreateDefaultBuilder().Build();
+           var config = host.Services.GetRequiredService<IConfiguration>();

            var options = new DbContextOptionsBuilder<VehicleQuotesContext>()
-               .UseNpgsql("Host=db;Database=vehicle_quotes_test;Username=vehicle_quotes;Password=password")
+               .UseNpgsql(config.GetConnectionString("VehicleQuotesContext"))
                .UseSnakeCaseNamingConvention()
                .Options;

            var context = new VehicleQuotesContext(options);

            context.Database.EnsureCreated();

            return context;
        }

        // ...
    }
}

First we add a few using statements. We need Microsoft.Extensions.Hosting so that we can have access to the Host class through which we obtain access to the application’s execution context. This allows us to access the built-in configuration service. We also need Microsoft.Extensions.Configuration to have access to the IConfiguration interface which is how we reference the configuration service which allows us access to the appsettings.json config file. And we also need the Microsoft.Extensions.DependencyInjection namespace which allows us to tap into the built-in dependency injection mechanism, through which we can access the default configuration service I mentioned before. Specifically, that namespace is where the GetRequiredService extension method lives.

All this translates into the few code changes that you see in the previous diff: first getting the app’s host, then getting the configuration service, then using that to fetch our connection string.

You can refer to the official documentation to learn more about configuration in .NET.

Writing a simple test case that fetches data

Now that we have a way to access the database from within the test suite, we can finally write an actual test case. Here’s the GetAllQuotesReturnsEmptyWhenThereIsNoDataStored one that I alluded to earlier:

// ...

namespace VehicleQuotes.Tests.Services
{
    public class QuoteServiceTests
    {
        // ...

        [Fact]
        public async void GetAllQuotesReturnsEmptyWhenThereIsNoDataStored()
        {
            // Given
            var dbContext = CreateDbContext();
            var service = new QuoteService(dbContext, null);

            // When
            var result = await service.GetAllQuotes();

            // Then
            Assert.Empty(result);
        }
    }
}

This one is a very simple test. We obtain a new VehicleQuotesContext instance that we can use to pass as a parameter when instantiating the component that we want to test: the QuoteService. We then call the GetAllQuotes method and assert that it returned an empty set. The test database was just created, so there should be no data in it, hence the empty resource set.

To run this test, we do dotnet test. I personally like a more verbose output so I like to use this variant of the command: dotnet test --logger "console;verbosity=detailed". Here’s what the output looks like.

$ dotnet test --logger "console;verbosity=detailed"
  Determining projects to restore...
  All projects are up-to-date for restore.
  VehicleQuotes -> /app/VehicleQuotes/bin/Debug/net5.0/VehicleQuotes.dll
  VehicleQuotes.Tests -> /app/VehicleQuotes.Tests/bin/Debug/net5.0/VehicleQuotes.Tests.dll
Test run for /app/VehicleQuotes.Tests/bin/Debug/net5.0/VehicleQuotes.Tests.dll (.NETCoreApp,Version=v5.0)
Microsoft (R) Test Execution Command Line Tool Version 16.11.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
/app/VehicleQuotes.Tests/bin/Debug/net5.0/VehicleQuotes.Tests.dll
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.3+1b45f5407b (64-bit .NET 5.0.12)
[xUnit.net 00:00:01.03]   Discovering: VehicleQuotes.Tests
[xUnit.net 00:00:01.06]   Discovered:  VehicleQuotes.Tests
[xUnit.net 00:00:01.06]   Starting:    VehicleQuotes.Tests
[xUnit.net 00:00:03.25]   Finished:    VehicleQuotes.Tests
  Passed VehicleQuotes.Tests.Services.QuoteServiceTests.GetAllQuotesReturnsEmptyWhenThereIsNoDataStored [209 ms]

Test Run Successful.
Total tests: 1
     Passed: 1
 Total time: 3.7762 Seconds

Resetting the state of the database after each test

Now we need to write a test that actually writes data into the database. However, every test case needs to start with the database in its original state. In other words, the changes that one test case does to the test database should not be seen, affect, or be expected by any subsequent test. That will make it so our test cases are isolated and repeatable. That’s not possible with our current implementation, though.

You can read more about the FIRST principles of testing here.

Luckily, that’s a problem that’s easily solved with Entity Framework Core. All we need to do is call a method that ensures that the database is deleted just before it ensures that it is created. Here’s what it looks like:

 private VehicleQuotesContext CreateDbContext()
 {
     var host = Host.CreateDefaultBuilder().Build();
     var config = host.Services.GetRequiredService<IConfiguration>();

     var options = new DbContextOptionsBuilder<VehicleQuotesContext>()
         .UseNpgsql(config.GetConnectionString("VehicleQuotesContext"))
         .UseSnakeCaseNamingConvention()
         .Options;

     var context = new VehicleQuotesContext(options);

+    context.Database.EnsureDeleted();
     context.Database.EnsureCreated();

     return context;
 }

And that’s all. Now every test case that calls CreateDbContext in order to obtain a DbContext instance will effectively trigger a database reset. Feel free to dotnet test again to validate that the test suite is still working.

Now, depending on the size of the database, this can be quite expensive. For integration tests, performance is not as big of a concern as for unit tests. This is because integration tests should be fewer in number and less frequently run.

We can make it better though. Instead of deleting and recreating the database before each test case, we’ll take a page out of Ruby on Rails’ book and run each test case within a database transaction which gets rolled back after the test is done. For now though, let’s write another test case: this time, one where we insert new records into the database.

If you want to hear a more in-depth discussion about automated testing in general, I go into further detail on the topic in this article: An introduction to automated testing for web applications with Symfony.

Writing another simple test case that stores data

Now let’s write another test that exercises QuoteService’s GetAllQuotes method. This time though, let’s add a new record to the database before calling it so that the method’s result is not empty. Here’s what the test looks like:

// ...
using VehicleQuotes.Models;
using System.Linq;

namespace VehicleQuotes.Tests.Services
{
    public class QuoteServiceTests
    {
        // ...

        [Fact]
        public async void GetAllQuotesReturnsTheStoredData()
        {
            // Given
            var dbContext = CreateDbContext();

            var quote = new Quote
            {
                OfferedQuote = 100,
                Message = "test_quote_message",

                Year = "2000",
                Make = "Toyota",
                Model = "Corolla",
                BodyTypeID = dbContext.BodyTypes.Single(bt => bt.Name == "Sedan").ID,
                SizeID = dbContext.Sizes.Single(s => s.Name == "Compact").ID,

                ItMoves = true,
                HasAllWheels = true,
                HasAlloyWheels = true,
                HasAllTires = true,
                HasKey = true,
                HasTitle = true,
                RequiresPickup = true,
                HasEngine = true,
                HasTransmission = true,
                HasCompleteInterior = true,

                CreatedAt = DateTime.Now
            };

            dbContext.Quotes.Add(quote);

            dbContext.SaveChanges();

            var service = new QuoteService(dbContext, null);

            // When
            var result = await service.GetAllQuotes();

            // Then
            Assert.NotEmpty(result);
            Assert.Single(result);
            Assert.Equal(quote.ID, result.First().ID);
            Assert.Equal(quote.OfferedQuote, result.First().OfferedQuote);
            Assert.Equal(quote.Message, result.First().Message);
        }
    }
}

First we include the VehicleQuotes.Models namespace so that we can use the Quotes model class. In our REST API, this is the class that represents the data from the quotes table. This is the main table that GetAllQuotes queries. We also include the System.Linq namespace, which allows us to use various collection extension methods (like Single and First) which we leverage throughout the test case to query lookup tables and assert on the test results.

Other than that, the test case itself is pretty self-explanatory. We start by obtaining an instance of VehicleQuotesContext via the CreateDbContext method. Remember that this also resets the whole database so that the test case can run over a clean slate. Then, we create a new Quote object and use our VehicleQuotesContext to insert it as a record into the database. We do this so that the later call to QuoteService’s GetAllQuotes method actually finds some data to return this time. Finally, the test case validates that the result contains a record and that its data is identical to what we set manually.

Neat! At this point we have what I think is the bare minimum infrastructure when it comes to serviceable and effective database integration tests, namely, access to a test database. We can take it one step further, though, and make things more reusable and a little bit better performing.

Refactoring into a fixture for reusability

We can use the test fixture functionality offered by xUnit.net in order to make the database interactivity aspect of our test suite into a reusable component. That way, if we had other test classes focused on other components that interact with the database, we could just plug that code in. We can define a fixture by creating a new file called, for example, VehicleQuotes.Tests/Fixtures/DatabaseFixture.cs with these contents:

using System;
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;

namespace VehicleQuotes.Tests.Fixtures
{
    public class DatabaseFixture : IDisposable
    {
        public VehicleQuotesContext DbContext { get; private set; }

        public DatabaseFixture()
        {
            DbContext = CreateDbContext();
        }

        public void Dispose()
        {
            DbContext.Dispose();
        }

        private VehicleQuotesContext CreateDbContext()
        {
            var host = Host.CreateDefaultBuilder().Build();
            var config = host.Services.GetRequiredService<IConfiguration>();

            var options = new DbContextOptionsBuilder<VehicleQuotesContext>()
                .UseNpgsql(config.GetConnectionString("VehicleQuotesContext"))
                .UseSnakeCaseNamingConvention()
                .Options;

            var context = new VehicleQuotesContext(options);

            context.Database.EnsureDeleted();
            context.Database.EnsureCreated();

            return context;
        }
    }
}

All this class does is define the CreateDbContext method that we’re already familiar with but puts it in a nice reusable package. Upon instantiation, as seen in the constructor, it stores a reference to the VehicleQuotesContext in its DbContext property.

With that, our QuoteServiceTests test class can use it if we make the following changes to it:

 using System;
 using Xunit;
-using Microsoft.EntityFrameworkCore;
 using VehicleQuotes.Services;
-using Microsoft.Extensions.Hosting;
-using Microsoft.Extensions.Configuration;
-using Microsoft.Extensions.DependencyInjection;
 using VehicleQuotes.Models;
 using System.Linq;
+using VehicleQuotes.Tests.Fixtures;

 namespace VehicleQuotes.Tests.Services
 {
-    public class QuoteServiceTests
+    public class QuoteServiceTests : IClassFixture<DatabaseFixture>
     {
+        private VehicleQuotesContext dbContext;

+        public QuoteServiceTests(DatabaseFixture fixture)
+        {
+            dbContext = fixture.DbContext;
+        }

-        private VehicleQuotesContext CreateDbContext()
-        {
-            var host = Host.CreateDefaultBuilder().Build();
-            var config = host.Services.GetRequiredService<IConfiguration>();

-            var options = new DbContextOptionsBuilder<VehicleQuotesContext>()
-                .UseNpgsql(config.GetConnectionString("VehicleQuotesContext"))
-                .UseSnakeCaseNamingConvention()
-                .Options;

-            var context = new VehicleQuotesContext(options);

-            context.Database.EnsureDeleted();
-            context.Database.EnsureCreated();

-            return context;
-        }

         [Fact]
         public async void GetAllQuotesReturnsEmptyWhenThereIsNoDataStored()
         {
             // Given
-            var dbContext = CreateDbContext();

             // ...
         }

         [Fact]
         public async void GetAllQuotesReturnsTheStoredData()
         {
             // Given
-            var dbContext = CreateDbContext();

             // ...
         }
     }
 }

Here we’ve updated the QuoteServiceTests class definition so that it inherits from IClassFixture<DatabaseFixture>. This is how we tell xUnit.net that our tests use the new fixture that we created. Next, we define a constructor that receives a DatabaseFixture object as a parameter. That’s how xUnit.net allows our test class to access the capabilities provided by the fixture. In this case, we take the fixture’s DbContext instance, and store it for later use in all of the test cases that need database interaction. We also removed the CreateDbContext method because now that’s defined within the fixture. We also removed a few using statements that became unnecessary.

One important aspect to note about this fixture is that it is initialized once per whole test suite run, not once per test case. Specifically, the code within the DatabaseFixture’s constructor gets executed once, before all of the test cases. Similarly, the code in DatabaseFixture’s Dispose method get executed once at the end, after all test cases have been run.

This means that our test database deletion and recreation step now happens only once for the entire test suite. This is not good with our current implementation because that means that individual test cases no longer run with a fresh, empty database. This can be good for performance though, as long as we update our test cases to run within database transactions. Let’s do just that.

Using transactions to reset the state of the database

Here’s how we update out test class so that each test case runs within a transaction:

 // ...

 namespace VehicleQuotes.Tests.Services
 {
-    public class QuoteServiceTests : IClassFixture<DatabaseFixture>
+    public class QuoteServiceTests : IClassFixture<DatabaseFixture>, IDisposable
     {
         private VehicleQuotesContext dbContext;

         public QuoteServiceTests(DatabaseFixture fixture)
         {
             dbContext = fixture.DbContext;

+            dbContext.Database.BeginTransaction();
         }

+        public void Dispose()
+        {
+            dbContext.Database.RollbackTransaction();
+        }

         # ...
     }
 }

The first thing to note here is that we added a call to BeginTransaction in the test class constructor. xUnit.net creates a new instance of the test class for each test case. This means that this constructor is run before each and every test case. We use that opportunity to begin a database transaction.

The other interesting point is that we’ve updated the class to implement the IDisposable interface’s Dispose method. xUnit.net will run this code after each test case, so we rollback the transaction.

Put those two together and we’ve updated our test suite so that every test case runs within the context of its own database transaction. Try it out with dotnet test and see what happens.

To learn more about database transactions with Entity Framework Core, you can look at the official docs.

You can learn more about xUnit.net’s test class fixtures in the samples repository.

Alright, that’s all for now. It is great to see that implementing automated database integration tests is actually fairly straightforward using .NET, xUnit.net, and Entity Framework. Even if it isn’t quite as easy as it is in Rails, it is perfectly doable.

Kubernetes 101: Deploying a web application and database

2022-01-08T00:00:00+00:00

The DevOps world seems to have been taken over by Kubernetes during the past few years. And rightfully so, I believe, as it is a great piece of software that promises and delivers when it comes to managing deployments of complex systems.

Kubernetes is hard though. But it’s all good, I’m not a DevOps engineer. As a software developer, I shouldn’t care about any of that. Or should I? Well… Yes. I know that very well after being thrown head first into a project that heavily involves Kubernetes, without knowing the first thing about it.

Even if I wasn’t in the role of a DevOps engineer, as a software developer, I had to work with it in order to set up dev environments, troubleshoot system issues, and make sound design and architectural decisions.

After a healthy amount of struggle, I eventually gained some understanding on the subject. In this blog post I’ll share what I learned. My hope is to put out there the things I wish I knew when I first encountered and had to work with Kubernetes.

So, I’m going to introduce the basic concepts and building blocks of Kubernetes. Then, I’m going to walk you through the process of containerizing a sample application, developing all the Kubernetes configuration files necessary for deploying it into a Kubernetes cluster, and actually deploying it into a local development cluster. We will end up with an application and its associated database running completely on and being managed by Kubernetes.

In short: If you know nothing about Kubernetes, and are interested in learning, read on. This post is for you.

What is Kubernetes?

Simply put, Kubernetes (or K8s) is software for managing computer clusters. That is, groups of computers that are working together in order to process some workload or offer a service. Kubernetes does this by leveraging application containers. Kubernetes will help you out in automating the deployment, scaling, and management of containerized applications.

Once you’ve designed an application’s complete execution environment and associated components, using Kubernetes you can specify all that declaratively via configuration files. Then, you’ll be able to deploy that application with a single command. Once deployed, Kubernetes will give you tools to check on the health of your application, recover from issues, keep it running, scale it, etc.

There are a few basic concepts that we need to be familiar with in order to effectively work with Kubernetes. I think the official documentation does a great job in explaining this, but I’ll try to summarize.

Nodes, pods, and containers

First up are containers. If you’re interested in Kubernetes, chances are that you’ve already been exposed to some sort of container technology like Docker. If not, no worries. For our purposes here, we can think of a container as an isolated process with its own resources and file system in which an application can run.

A container has all the software dependencies that an application needs to run, including the application itself. From the application’s perspective, the container is its execution environment: the “machine” in which it’s running. In more practical terms, a container is a form of packaging, delivering, and executing an application. What’s the advantage? Instead of installing the application and its dependencies directly into the machine that’s going to run it, having it containerized allows for a container runtime (like Docker) to just run it as a self-contained unit. This makes it possible for the application to run anywhere that has the container runtime installed, with minimal configuration.

Something very closely related to containers is the concept of images. You can think of images as the blueprint for containers. An image is the spec, and the container is the instance that’s actually running.

When deploying applications into Kubernetes, this is how it runs them: via containers. In other words, for Kubernetes to be able to run an application, it needs to be delivered within a container.

Next is the concept of a node. This is very straightforward and not even specific to Kubernetes. A node is a computer within the cluster. That’s it. Like I said before, Kubernetes is built to manage computer clusters. A node is just one computer, either virtual or physical, within that cluster.

Then there are pods. Pods are the main executable units in Kubernetes. When we deploy an application or service into a Kubernetes cluster, it runs within a pod. Kubernetes works with containerized applications though, so it is the pods that take care of running said containers within them.

These three work very closely together within Kubernetes. To summarize: containers run within pods which in turn exist within nodes in the cluster.

There are other key components to talk about like deployments, services, replica sets, and persistent volumes. But I think that’s enough theory for now. We’ll learn more about all these as we get our hands dirty working though our example. So let’s get started with our demo and we’ll be discovering and discussing them organically as we go through it.

Installing and setting up Kubernetes

The first thing we need is a Kubernetes environment. There are many Kubernetes implementations out there. Google, Microsoft, and Amazon offer Kubernetes solutions on their respective cloud platforms, for example. There are also implementations that one can install and run on their own, like kind, minikube, and MicroK8s. We are going to use MicroK8s for our demo, for no particular reason other than “this is the one I know”.

When done installing, MicroK8s will have set up a whole Kubernetes cluster, with your machine as its one and only node.

Installing MicroK8s

So, if you’re in Ubuntu and have snapd, installing MicroK8s is easy. The official documentation explains it best. You install it with a command like this:

$ sudo snap install microk8s --classic --channel=1.21

MicroK8s will create a user group which is best to add your user account to so you can execute commands that would otherwise require admin privileges. You can do so with:

$ sudo usermod -a -G microk8s $USER
$ sudo chown -f -R $USER ~/.kube

With that, our very own Kubernetes cluster, courtesy of MicroK8s, should be ready to go. Check its status with:

$ microk8s status --wait-ready

You should see a “MicroK8s is running” message along with some specifications on your cluster. Including the available add-ons, which ones are enabled and which ones are disabled.

You can also shut down your cluster anytime with microk8s stop. Use microk8s start to bring it back up.

Introducing kubectl

MicroK8s also comes with kubectl. This is our gateway into Kubernetes, as this is the command line tool that we use to interact with it. By default, MicroK8s makes it so we can call it using microk8s kubectl .... That is, namespaced. This is useful if you have multiple Kubernetes implementations running at the same time, or another, separate kubectl. I don’t, so I like to create an alias for it, so that I can call it without having to use the microk8s prefix. You can do it like so:

$ sudo snap alias microk8s.kubectl kubectl

Now that all that’s done, we can start talking to our Kubernetes cluster. We can ask it for example to tell us which are the nodes in the cluster with this command:

$ kubectl get nodes

That will result in something like:

NAME     STATUS   ROLES    AGE   VERSION
pop-os   Ready    <none>   67d   v1.21.4-3+e5758f73ed2a04

The only node in the cluster is your own machine. In my case, my machine is called “pop-os” so that’s what shows up. You can get more information out of this command by using kubectl get nodes -o wide.

Installing add-ons

MicroK8s supports many add-ons that we can use to enhance our Kubernetes installation. We are going to need a few of them so let’s install them now. They are:

The dashboard, which gives us a nice web GUI which serves as a window into our cluster. In there we can see everything that’s running, read logs, run commands, etc.
dns, which sets up DNS for within the cluster. In general it’s a good idea to enable this one because other add-ons use it.
storage, which allows the cluster to access the host machine’s disk for storage. The application that we will deploy needs a persistent database so we need this plugin to make it happen.
registry, which sets up a container image registry that Kubernetes can access. Kubernetes runs containerized applications, containers are based on images. So, having this add-on allows us to define an image for our application and make it available to Kubernetes.

To install these, just run the following commands:

$ microk8s enable dashboard
$ microk8s enable dns
$ microk8s enable storage
$ microk8s enable registry

Those are all the add-ons that we’ll use.

Introducing the Dashboard

The dashboard is one we can play with right now. In order to access it, first run this:

$ microk8s dashboard-proxy

That will start up a proxy into the dashboard. The command will give you a URL and login token that you can use to access the dashboard. It results in an output like this:

Checking if Dashboard is running.
Dashboard will be available at https://127.0.0.1:10443
Use the following token to login:
<YOUR LOGIN TOKEN>

Now you can navigate to that URL in your browser and you’ll find a screen like this:

Make sure the “Token” option is selected and take the login token generated by the microk8s dashboard-proxy command from before and paste it in the field in the page. Click the “Sign In” button and you’ll be able to see the dashboard, allowing you access to many aspects of your cluster. It should look like this:

Feel free to play around with it a little bit. You don’t have to understand everything yet. As we work through our example, we’ll see how the dashboard and the other add-ons come into play.

There’s also a very useful command line tool called K9s, which helps in interacting with our cluster. We will not be discussing it further in this article but feel free to explore it if you need or want a command line alternative to the built-in dashboard.

Deploying applications into a Kubernetes cluster

With all that setup out of the way, we can start using our K8s cluster for what it was designed: running applications.

Deployments

Pods are very much the stars of the show when it comes to Kubernetes. However, most of the time we don’t create them directly. We usually do so through “deployments”. Deployments are a more abstract concept in Kubernetes. They basically control pods and make sure they behave as specified. You can think of them as wrappers for pods which make our lives easier than if we had to handle pods directly. Let’s go ahead and create a deployment so things will be clearer.

In Kubernetes, there are various ways of managing objects like deployments. For this post, I’m going to focus exclusively on the configuration-file-driven declarative approach as that’s the one better suited for real world scenarios.

You can learn more about the different ways of interacting with Kubernetes objects in the official documentation.

So, simply put, if we want to create a deployment, then we need to author a file that defines it. A simple deployment specification looks like this:

# nginx-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

This example is taken straight from the official documentation.

Don’t worry if most of that doesn’t make sense at this point. I’ll explain it in detail later. First, let’s actually do something with it.

Save that in a new file. You can call it nginx-deployment.yaml. Once that’s done, you can actually create the deployment (and its associated objects) in your K8s cluster with this command:

$ kubectl apply -f nginx-deployment.yaml

Which should result in the following message:

deployment.apps/nginx-deployment created

And that’s it for creating deployments! (Or any other type of object in Kubernetes for that matter.) We define the object in a file and then invoke kubectl’s apply command. Pretty simple.

If you want to delete the deployment, then this command will do it:
$ kubectl delete -f nginx-deployment.yaml
deployment.apps "nginx-deployment" deleted

Using kubectl to explore a deployment

Now, let’s inspect our cluster to see what this command has done for us.

First, we can ask it directly for the deployment with:

$ kubectl get deployments

Which outputs:

NAME               READY   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   3/3     3            3           2m54s

Now you can see that the deployment that we just created is right there with the name that we gave it.

As I said earlier, deployments are used to manage pods, and that’s just what the READY, UP-TO-DATE and AVAILABLE columns allude to with those values of 3. This deployment has three pods because, in our YAML file, we specified we wanted three replicas with the replicas: 3 line. Each “replica” is a pod. For our example, that means that we will have three instances of NGINX running side by side.

We can see the pods that have been created for us with this command:

$ kubectl get pods

Which gives us something like this:

NAME                                READY   STATUS    RESTARTS   AGE
nginx-deployment-66b6c48dd5-fs5rq   1/1     Running   0          55m
nginx-deployment-66b6c48dd5-xmnl2   1/1     Running   0          55m
nginx-deployment-66b6c48dd5-sfzxm   1/1     Running   0          55m

The exact names will vary, as the IDs are auto-generated. But as you can see, this command gives us some basic information about our pods. Remember that pods are the ones that actually run our workloads via containers. The READY field is particularly interesting in this sense because it tells us how many containers are running in the pods vs how many are supposed to run. So, 1/1 means that the pod has one container ready out of 1. In other words, the pod is fully ready.

Using the dashboard to explore a deployment

Like I said before, the dashboard offers us a window into our cluster. Let’s see how we can use it to see the information that we just saw via kubectl. Navigate into the dashboard via your browser and you should now see that some new things have appeared:

We now have new “CPU Usage” and “Memory Usage” sections that give us insight into the utilization of our machine’s resources.

There’s also “Workload status” that has some nice graphs giving us a glance at the status of our deployments, pods, and replica sets.

Don’t worry too much about replica sets right now, as we seldom interact with them directly. Suffice it to say, replica sets are objects that deployments rely on to make sure that the number of specified replica pods is maintained. As always, there’s more info in the official documentation.

Scroll down a little bit more and you’ll find the “Deployments” and “Pods” sections, which contain the information that we’ve already seen via kubectl before.

Feel free to click around and explore the capabilities of the dashboard.

Dissecting the deployment configuration file

Now that we have a basic understanding of deployments and pods and how to create them, let’s look more closely into the configuration file that defines it. This is what we had:

# nginx-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

This example is very simple, but it touches on the key aspects of deployment configuration. We will be building more complex deployments as we work through this article, but this is a great start. Let’s start at the top:

apiVersion: Under the hood, a Kubernetes cluster exposes its functionality via a REST API. We seldom interact with this API directly because we have kubectl that takes care of it for us. kubectl takes our commands, translates them into HTTP requests that the K8s REST API can understand, sends them, and gives us back the results. So, this apiVersion field specifies which version of the K8s REST API we are expecting to talk to.
kind: It represents the type of object that the configuration file defines. All objects in Kubernetes can be managed via YAML configuration files and kubectl apply. So, this field specifies which one we are managing at any given time.
metadata.name: Quite simply, the name of the object. It’s how we and Kubernetes refer to it.
metadata.labels: These help us further categorize cluster objects. These have no real effect in the system so they are useful for user help more than anything else.
spec: This contains the actual functional specification for the behavior of the deployment. More details below.
spec.replicas: The number of replica pods that the deployment should create. We already talked a bit about this before.
spec.selector.labels: This is one case when labels are actually important. Remember that when we create deployments, replica sets and pods are created with it. Within the K8s cluster, they each are their own individual objects though. This field is the mechanism that K8s uses to associate a given deployment with its replica set and pods. In practice, that means that whatever labels are in this field need to match the labels in spec.template.metadata.labels. More on that one below.
spec.template: Specifies the configuration of the pods that will be part of the deployment.
spec.template.metadata.labels: Very similar to metadata.labels. The only difference is that those labels are added to the deployment while these ones are added to the pods. The only notable thing is that these labels are key for the deployment to know which pods it should care about (as explained in above in spec.selector.labels).
spec.template.spec: This section specifies the actual functional configuration of the pods.
spec.template.spec.containers: This section specifies the configuration of the containers that will be running inside the pods. It’s an array so there can be many. In our example we have only one.
spec.template.spec.containers[0].name: The name of the container.
spec.template.spec.containers[0].image: The image that will be used to build the container.
spec.template.spec.containers[0].ports[0].containerPort: A port through which the container will accept traffic from the outside. In this case, 80.

You can find a detailed description of all the fields supported by deployment configuration files in the official API reference documentation. And much more!

Connecting to the containers in the pods

Kubernetes allows us to connect to the containers running inside a pod. This is pretty easy to do with kubectl. All we need to know is the name of the pod and the container that we want to connect to. If the pod is running only one container (like our NGINX one does) then we don’t need the container name. We can find out the names of our pods with:

$ kubectl get pods
NAME                                READY   STATUS    RESTARTS   AGE
nginx-deployment-66b6c48dd5-85nwq   1/1     Running   0          25s
nginx-deployment-66b6c48dd5-x5b4x   1/1     Running   0          25s
nginx-deployment-66b6c48dd5-wvkhc   1/1     Running   0          25s

Pick one of those and we can open a bash session in it with:

$ kubectl exec -it nginx-deployment-66b6c48dd5-85nwq -- bash

Which results in a prompt like this:

root@nginx-deployment-66b6c48dd5-85nwq:/#

We’re now connected to the container in one of our NGINX pods. There isn’t a lot to do with this right now, but feel free to explore it. It’s got its own processes and file system which are isolated from the other replica pods and your actual machine.

We can also connect to containers via the dashboard. Go back to the dashboard in your browser, log in again if the session expired, and scroll down to the “Pods” section. Each pod in the list has an action menu with an “Exec” command. See it here:

Click it and you’ll be taken to a screen with a console just like the one we obtained via kubectl exec:

The dashboard is quite useful, right?

Services

So far, we’ve learned quite a bit about deployments. How to specify and create them, how to explore them via command line and the dashboard, how to interact with the pods, etc. We haven’t seen a very important part yet, though: actually accessing the application that has been deployed. That’s where services come in. We use services to expose an application running in a set of pods to the world outside the cluster.

Here’s what a configuration file for a service that exposes access to our NGINX deployment could look like:

# nginx-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  type: NodePort
  selector:
    app: nginx
  ports:
    - name: "http"
      port: 80
      targetPort: 80
      nodePort: 30080

Same as with the deployment’s configuration file, this one also has a kind field that specifies what it is; and a name given to it via the metadata.name field. The spec section is where things get interesting.

spec.type specifies, well… The type of the service. Kubernetes supports many types of services. For now, we want a NodePort. This type of service makes sure to expose itself as a static port (given by spec.ports[0].nodePort) on every node in the cluster. In our setup, we only have one node, which is our own machine.
spec.ports defines which ports of the pods’ containers the service will expose.
spec.ports[0].name: The name of the port. To be used elsewhere to reference the specific port.
spec.ports[0].port: The port that will be exposed by the service.
spec.ports[0].targetPort: The port that the service will target in the container.
spec.ports[0].nodePort: The port that the service will expose in all the nodes of the cluster.

Same as with deployments, we can create such a service with the kubectl apply command. If you save the contents from the YAML above into a nginx-service.yaml file, you can run the following to create it:

$ kubectl apply -f nginx-service.yaml

And to inspect it and validate that it was in fact created:

$ kubectl get services
NAME            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes      ClusterIP   10.152.183.1    <none>        443/TCP        68d
nginx-service   NodePort    10.152.183.22   <none>        80:30080/TCP   27s

The dashboard also has a section for services. It looks like this:

Accessing an application via a service

We can access our service in a few different ways. We can use its “cluster IP” which we obtain from the output of the kubectl get services command. As given by the example above, that would be 10.152.183.22 in my case. Browsing to that IP gives us the familiar NGINX default welcome page:

Another way is by using the “NodePort”. Remember that the “NodePort” specifies the port in which the service will be available on every node of the cluster. With our current MicroK8s setup, our own machine is a node in the cluster, so we can also access the NGINX that’s running in our Kubernetes cluster using localhost:30080. 30080 is given by the spec.ports[0].nodePort field in the service configuration file from before. Try it out:

How cool is that? We have identical, replicated NGINX instances running in a Kubernetes cluster that’s installed locally in our machine.

Deploying our own custom application

Alright, by deploying NGINX, we’ve learned a lot about nodes, pods, deployments, services, and how they all work together to run and serve an application from a Kubernetes cluster. Now, let’s take all that knowledge and try to do the same for a completely custom application of our own.

What are we building

The application that we are going to deploy into our cluster is a simple one with only two components: a REST API written with .NET 5 and a Postgres database. You can find the source code in GitHub. It’s an API for supporting a hypothetical front end application for capturing used vehicle information and calculating their value in dollars.

If you’re interested in learning more about the process of actually writing that app, it’s all documented in another blog post: Building REST APIs with .NET 5, ASP.NET Core, and PostgreSQL.

If you’re following along, now would be a good time to download the source code of the web application that we’re going to be playing with. You can find it on GitHub. From now on, we’ll use that as the root directory of all the files we create and modify.

Also, be sure to delete or put aside the k8s directory. We’ll be building that throughout the rest of this post.

Deploying the database

Let’s begin with the Postgres database. Similar as before, we start by setting up a deployment with one pod and one container. We can do so with a deployment configuration YAML file like this:

# db-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vehicle-quotes-db
spec:
  selector:
    matchLabels:
      app: vehicle-quotes-db
  replicas: 1
  template:
    metadata:
      labels:
        app: vehicle-quotes-db
    spec:
      containers:
        - name: vehicle-quotes-db
          image: postgres:13
          ports:
            - containerPort: 5432
              name: "postgres"
          env:
            - name: POSTGRES_DB
              value: vehicle_quotes
            - name: POSTGRES_USER
              value: vehicle_quotes
            - name: POSTGRES_PASSWORD
              value: password
          resources:
            limits:
              memory: 4Gi
              cpu: "2"

This deployment configuration YAML file is similar to the one we used for NGINX before, but it introduces a few new elements:

spec.template.spec.containers[0].ports[0].name: We can give specific names to ports which we can reference later, elsewhere in the K8s configurations, which is what this field is for.
spec.template.spec.containers[0].env: This is a list of environment variables that will be defined in the container inside the pod. In this case, we’ve specified a few variables that are necessary to configure the Postgres instance that will be running. We’re using the official Postgres image from Dockerhub, and it calls for these variables. Their purpose is straightforward: they specify database name, username, and password.
spec.template.spec.containers[0].resources: This field defines the hardware resources that the container needs in order to function. We can specify upper limits with limits and lower ones with requests. You can learn more about resource management in the official documentation. In our case, we’ve kept it simple and used limits to prevent the container from using more than 4Gi of memory and 2 CPU cores.

Now, let’s save that YAML into a new file called db-deployment.yaml and run the following:

$ kubectl apply -f db-deployment.yaml

Which should output:

deployment.apps/vehicle-quotes-db created

After a few seconds, you should be able to see the new deployment and pod via kubectl:

$ kubectl get deployment -A
NAMESPACE            NAME                        READY   UP-TO-DATE   AVAILABLE   AGE
...
default              vehicle-quotes-db           1/1     1            1           9m20s

$ kubectl get pods -A
NAMESPACE            NAME                                         READY   STATUS    RESTARTS   AGE
...
default              vehicle-quotes-db-5fb576778-gx7j6            1/1     Running   0          9m22s

Remember you can also see them in the dashboard:

Connecting to the database

Let’s try connecting to the Postgres instance that we just deployed. Take note of the pod’s name and try:

$ kubectl exec -it <DB_POD_NAME> -- bash

You’ll get a bash session on the container that’s running the database. For me, given the pod’s auto-generated name, it looks like this:

root@vehicle-quotes-db-5fb576778-gx7j6:/#

From here, you can connect to the database using the psql command line client. Remember that we told the Postgres instance to create a vehicle_quotes user. We set it up via the container environment variables on our deployment configuration. As a result, we can do psql -U vehicle_quotes to connect to the database. Put together, it all looks like this:

$ kubectl exec -it vehicle-quotes-db-5fb576778-gx7j6 -- bash
root@vehicle-quotes-db-5fb576778-gx7j6:/# psql -U vehicle_quotes
psql (13.3 (Debian 13.3-1.pgdg100+1))
Type "help" for help.

vehicle_quotes=# \l
                                            List of databases
      Name      |     Owner      | Encoding |  Collate   |   Ctype    |         Access privileges
----------------+----------------+----------+------------+------------+-----------------------------------
 postgres       | vehicle_quotes | UTF8     | en_US.utf8 | en_US.utf8 |
 template0      | vehicle_quotes | UTF8     | en_US.utf8 | en_US.utf8 | =c/vehicle_quotes                +
                |                |          |            |            | vehicle_quotes=CTc/vehicle_quotes
 template1      | vehicle_quotes | UTF8     | en_US.utf8 | en_US.utf8 | =c/vehicle_quotes                +
                |                |          |            |            | vehicle_quotes=CTc/vehicle_quotes
 vehicle_quotes | vehicle_quotes | UTF8     | en_US.utf8 | en_US.utf8 |
(4 rows)

Pretty cool, don’t you think? We have a database running on our cluster now with minimal effort. There’s a slight problem though…

Persistent volumes and claims

The problem in our database is that any changes are lost if the pod or container were to shut down or reset for some reason. This is because all the database files live inside the container’s file system, so when the container is gone, the data is also gone.

In Kubernetes, pods are supposed to be treated as ephemeral entities. The idea is that pods should easily be brought down and replaced by new pods and users and clients shouldn’t even notice. This is all Kubernetes working as expected. That is to say, pods should be as stateless as possible to work well with this behavior.

However, a database is, by definition, not stateless. So what we need to do to solve this problem is have some available disk space from outside the cluster that can be used by our database to store its files. Something persistent that won’t go away if the pod or container goes away. That’s where persistent volumes and persistent volume claims come in.

We will use a persistent volume (PV) to define a directory in our host machine that we will allow our Postgres container to use to store data files. Then, a persistent volume claim (PVC) is used to define a “request” for some of that available disk space that a specific container can make. In short, a persistent volume says to K8s “here’s some storage that the cluster can use” and a persistent volume claim says “here’s a portion of that storage that’s available for containers to use”.

Configuration files for the PV and PVC

Start by tearing down our currently broken Postgres deployment:

$ kubectl delete -f db-deployment.yaml

Now let’s add two new YAML configuration files. One is for the persistent volume:

# db-persistent-volume.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: vehicle-quotes-postgres-data-persisent-volume
  labels:
    type: local
spec:
  claimRef:
    namespace: default
    name: vehicle-quotes-postgres-data-persisent-volume-claim
  storageClassName: manual
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/home/kevin/projects/vehicle-quotes-postgres-data"

In this config file, we already know about the kind and metadata fields. A few of the other elements are interesting though:

spec.claimRef: Contains identifying information about the claim that’s associated with the PV. Used to bind the PVC with a specific PVC. Notice how it matches the name defined in the PVC config file from below.
spec.capacity.storage: Is pretty straightforward in that it specifies the size of the persistent volume.
spec.accessModes: Defines how the PV can be accessed. In this case, we’re using ReadWriteOnce so that it can only be used by a single node in the cluster which is allowed to read from and write into the PV.
spec.hostPath.path: Specifies the directory in the host machine’s file system where the PV will be mounted. Simply put, the containers in the cluster will have access to the specific directory defined here. I’ve used /home/kevin/projects/vehicle-quotes-postgres-data because that makes sense on my own machine. If you’re following along, make sure to set it to something that makes sense in your environment.

hostPath is just one type of persistent volume which works well for development deployments. Managed Kubernetes implementations like the ones from Google or Amazon have their own types which are more appropriate for production.

We also need another config file for the persistent volume claim:

# db-persistent-volume-claim.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: vehicle-quotes-postgres-data-persisent-volume-claim
spec:
  volumeName: vehicle-quotes-postgres-data-persisent-volume
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi

Like I said, PVCs are essentially usage requests for PVs. So, the config file is simple in that it’s mostly specified to match the PV.

spec.volumeName: The name of the PV that this PVC is going to access. Notice how it matches the name that we defined in the PV’s config file.
spec.resources.requests: Defines how much space this PVC requests from the PV. In this case, we’re just requesting all the space that the PV has available to it, as given by its config file: 5Gi.

Configuring the deployment to use the PVC

After saving those files, all that’s left is to update the database deployment configuration to use the PVC. Here’s what the updated config file would look like:

 apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: vehicle-quotes-db
 spec:
   selector:
     matchLabels:
       app: vehicle-quotes-db
   replicas: 1
   template:
     metadata:
       labels:
         app: vehicle-quotes-db
     spec:
       containers:
         - name: vehicle-quotes-db
           image: postgres:13
           ports:
             - containerPort: 5432
               name: "postgres"
+          volumeMounts:
+            - mountPath: "/var/lib/postgresql/data"
+              name: vehicle-quotes-postgres-data-storage
           env:
             - name: POSTGRES_DB
               value: vehicle_quotes
             - name: POSTGRES_USER
               value: vehicle_quotes
             - name: POSTGRES_PASSWORD
               value: password
           resources:
             limits:
               memory: 4Gi
               cpu: "2"
+      volumes:
+        - name: vehicle-quotes-postgres-data-storage
+          persistentVolumeClaim:
+            claimName: vehicle-quotes-postgres-data-persisent-volume-claim

First, notice the volumes section at the bottom of the file. Here’s where we define the volume that will be available to the container, give it a name and specify which PVC it will use. The spec.template.volumes[0].persistentVolumeClaim.claimName needs to match the name of the PVC that we defined in db-persistent-volume-claim.yaml.

Then, up in the containers section, we define a volumeMounts element. We use that to specify which directory within the container will map to our PV. In this case, we’ve set the container’s /var/lib/postgresql/data directory to use the volume that we defined at the bottom of the file. That volume is backed by our persistent volume claim, which is in turn backed by our persistent volume. The significance of the /var/lib/postgresql/data directory is that this is where Postgres stores database files by default.

In summary: We created a persistent volume that defines some disk space in our machine that’s available to the cluster; then we defined a persistent volume claim that represents a request of some of that space that a container can have access to; after that we defined a volume within our pod configuration in our deployment to point to that persistent volume claim; and finally we defined a volume mount in our container that uses that volume to store the Postgres database files.

By setting it up this way, we’ve made it so that regardless of how many Postgres pods come and go, the database files will always be persisted, because the files now live outside of the container. They are stored in our host machine instead.

There’s another limitation that’s important to note. Just using the approach that we discussed, it’s not possible to deploy multiple replicas of Postgres which work in tandem and operate on the same data. Even though the data files can be defined outside of the cluster and persisted that way, there can only be one single Postgres instance running against it at any given time.

In production, the high availability problem is better solved leveraging the features provided by the database software itself. Postgres offers various options in that area. Or, if you are deploying to the cloud, the best strategy may be to use a relational database service managed by your cloud provider. Examples are Amazon’s RDS and Microsoft’s Azure SQL Database.

Applying changes

Now let’s see it in action. Run the following three commands to create the objects:

$ kubectl apply -f db-persistent-volume.yaml
$ kubectl apply -f db-persistent-volume-claim.yaml
$ kubectl apply -f db-deployment.yaml

After a while, they will show up in the dashboard. You already know how to look for deployments and pods. For persistent volumes, click the “Persistent Volumes” option under the “Cluster” section in the sidebar:

Persistent volume claims can be found in the “Persistent Volume Claims” option under the “Config and Storage” section in the sidebar:

Now, try connecting to the database (using kubectl exec -it <VEHICLE_QUOTES_DB_POD_NAME> -- bash and then psql -U vehicle_quotes) and creating some tables. Something simple like this would work:

CREATE TABLE test (test_field varchar);

Now, close psql and the bash in the pod and delete the objects:

$ kubectl delete -f db-deployment.yaml
$ kubectl delete -f db-persistent-volume-claim.yaml
$ kubectl delete -f db-persistent-volume.yaml

Create them again:

$ kubectl apply -f db-persistent-volume.yaml
$ kubectl apply -f db-persistent-volume-claim.yaml
$ kubectl apply -f db-deployment.yaml

Connect to the database again and you should see that the table is still there:

vehicle_quotes=# \c vehicle_quotes
You are now connected to database "vehicle_quotes" as user "vehicle_quotes".
vehicle_quotes=# \dt
           List of relations
 Schema | Name | Type  |     Owner
--------+------+-------+----------------
 public | test | table | vehicle_quotes
(1 row)

That’s just what we wanted: the database is persisting independently of what happens to the pods and containers.

Exposing the database as a service

Lastly, we need to expose the database as a service so that the rest of the cluster can access it without having to use explicit pod names. We don’t need this for our testing, but we do need it for later when we deploy our web app, so that it can reach the database. As you’ve seen, services are easy to create. Here’s the YAML config file:

# db-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: vehicle-quotes-db-service
spec:
  type: NodePort
  selector:
    app: vehicle-quotes-db
  ports:
    - name: "postgres"
      protocol: TCP
      port: 5432
      targetPort: 5432
      nodePort: 30432

Save that into a new db-service.yaml file and don’t forget to kubectl apply -f db-service.yaml.

Deploying the web application

Now that we’ve got the database sorted out, let’s turn our attention to the app itself. As you’ve seen, Kubernetes runs apps as containers. That means that we need images to build those containers. A custom web application is no exception. We need to build a custom image that contains our application so that it can be deployed into Kubernetes.

Building the web application image

The first step for building a container image is writing a Dockerfile. Since our application is a Web API built using .NET 5, I’m going to use a slightly modified version of the Dockerfile used by Visual Studio Code’s development container demo for .NET. These development containers are excellent for, well… development. You can see the original in the link above, but here’s mine:

# [Choice] .NET version: 5.0, 3.1, 2.1
ARG VARIANT="5.0"
FROM mcr.microsoft.com/vscode/devcontainers/dotnet:0-${VARIANT}

# [Option] Install Node.js
ARG INSTALL_NODE="false"

# [Option] Install Azure CLI
ARG INSTALL_AZURE_CLI="false"

# Install additional OS packages.
RUN apt-get update && export DEBIAN_FRONTEND=noninteractive \
    && apt-get -y install --no-install-recommends postgresql-client-common postgresql-client

# Run the remaining commands as the "vscode" user
USER vscode

# Install EF and code generator development tools
RUN dotnet tool install --global dotnet-ef
RUN dotnet tool install --global dotnet-aspnet-codegenerator
RUN echo 'export PATH="$PATH:/home/vscode/.dotnet/tools"' >> /home/vscode/.bashrc

WORKDIR /app

# Prevent the container from closing automatically
ENTRYPOINT ["tail", "-f", "/dev/null"]

There are many other development containers for other languages and frameworks. Take a look at the microsoft/vscode-dev-containers GitHub repo to learn more.

An interesting thing about this Dockerfile is that we install the psql command line client so that we can connect to our Postgres database from within the web application container. The rest is stuff specific to .NET and the particular image we’re basing this Dockerfile on, so don’t sweat it too much.

If you’ve downloaded the source code, this Dockerfile should already be there as Dockerfile.dev.

Making the image accessible to Kubernetes

Now that we have a Dockerfile, we can use it to build an image that Kubernetes is able to use to build a container to deploy. So that Kubernetes can see it, we need to build it in a specific way and push it into a registry that’s accessible to Kubernetes. Remember how we ran microk8s enable registry to install the registry add-on when we were setting up MicroK8s? That will pay off now, as that’s the registry to which we’ll push our image.

First, we build the image:

$ docker build . -f Dockerfile.dev -t localhost:32000/vehicle-quotes-dev:registry

That will take some time to download and set up everything. Once that’s done, we push the image to the registry:

$ docker push localhost:32000/vehicle-quotes-dev:registry

That will also take a little while.

Deploying the web application

The next step is to create a deployment for the web app. Like usual, we start with a deployment YAML configuration file. Let’s call it web-deployment.yaml:

# web-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vehicle-quotes-web
spec:
  selector:
    matchLabels:
      app: vehicle-quotes-web
  replicas: 1
  template:
    metadata:
      labels:
        app: vehicle-quotes-web
    spec:
      containers:
        - name: vehicle-quotes-web
          image: localhost:32000/vehicle-quotes-dev:registry
          ports:
            - containerPort: 5000
              name: "http"
            - containerPort: 5001
              name: "https"
          volumeMounts:
            - mountPath: "/app"
              name: vehicle-quotes-source-code-storage
          env:
            - name: POSTGRES_DB
              value: vehicle_quotes
            - name: POSTGRES_USER
              value: vehicle_quotes
            - name: POSTGRES_PASSWORD
              value: password
            - name: CUSTOMCONNSTR_VehicleQuotesContext
              value: Host=$(VEHICLE_QUOTES_DB_SERVICE_SERVICE_HOST);Database=$(POSTGRES_DB);Username=$(POSTGRES_USER);Password=$(POSTGRES_PASSWORD)
          resources:
            limits:
              memory: 2Gi
              cpu: "1"
      volumes:
        - name: vehicle-quotes-source-code-storage
          persistentVolumeClaim:
            claimName: vehicle-quotes-source-code-persisent-volume-claim

This deployment configuration should look very familiar to you by now as it is very similar to the ones we’ve already seen. There are a few notable elements though:

Notice how we specified localhost:32000/vehicle-quotes-dev:registry as the container image. This is the exact same name of the image that we built and pushed into the registry before.
In the environment variables section, the one named CUSTOMCONNSTR_VehicleQuotesContext is interesting for a couple of reasons:
- First, the value is a Postgres connection string being built off of other environment variables using the following format: $(ENV_VAR_NAME). That’s a neat feature of Kubernetes config files that allows us to reference variables to build other ones.
- Second, the VEHICLE_QUOTES_DB_SERVICE_SERVICE_HOST environment variable used within that connection string is not defined anywhere in our configuration files. That’s an automatic environment variable that Kubernetes injects on all containers when there are services available. In this case, it contains the hostname of the vehicle-quotes-db-service that we created a few sections ago. The automatic injection of this *_SERVICE_HOST variable always happens as long as the service is already created by the time that the pod gets created. We have already created the service so we should be fine using the variable here. As usual, there’s more info in the official documentation.

As you may have noticed, this deployment has a persistent volume. That’s to store the application’s source code. Or, more accurately, to make the source code, which lives in our machine, available to the container. This is a development setup after all, so we want to be able to edit the code from the comfort of our own file system, and have the container inside the cluster be aware of that.

Anyway, let’s create the associated persistent volume and persistent volume claim. Here’s the PV (save it as web-persistent-volume.yaml):

# web-persistent-volume.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: vehicle-quotes-source-code-persisent-volume
  labels:
    type: local
spec:
  claimRef:
    namespace: default
    name: vehicle-quotes-source-code-persisent-volume-claim
  storageClassName: manual
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/home/kevin/projects/vehicle-quotes"

And here’s the PVC (save it as web-persistent-volume-claim.yaml):

# web-persistent-volume-claim.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: vehicle-quotes-source-code-persisent-volume-claim
spec:
  volumeName: vehicle-quotes-source-code-persisent-volume
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

The only notable element here is the PV’s hostPath. I have it pointing to the path where I downloaded the app’s source code from GitHub. Make sure to do the same on your end.

Finally, tie it all up with a service that will expose the development build of our REST API. Here’s the config file:

# web-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: vehicle-quotes-web-service
spec:
  type: NodePort
  selector:
    app: vehicle-quotes-web
  ports:
    - name: "http"
      protocol: TCP
      port: 5000
      targetPort: 5000
      nodePort: 30000
    - name: "https"
      protocol: TCP
      port: 5001
      targetPort: 5001
      nodePort: 30001

Should be pretty self-explanatory at this point. In this case, we expose two ports, one for HTTP and another for HTTPS. Our .NET 5 Web API works with both so that’s why we specify them here. This configuration says that the service should expose port 30000 and send traffic that comes into that port from the outside world into port 5000 on the container. Likewise, outside traffic coming to port 30001 will be sent to port 5001 in the container.

Save that file as web-service.yaml and we’re ready to apply the changes:

$ kubectl apply -f web-persistent-volume.yaml
$ kubectl apply -f web-persistent-volume-claim.yaml
$ kubectl apply -f web-deployment.yaml
$ kubectl apply -f web-service.yaml

Feel free to explore the dashboard’s “Deployments”, “Pods”, “Services”, “Persistent Volumes”, and “Persistent Volume Claims” sections to see the fruits of our labor.

Starting the application

Let’s now do some final setup and start up our application. Start by connecting to the web application pod:

$ kubectl exec -it vehicle-quotes-web-86cbc65c7f-5cpg8 -- bash

Remember that the pod name will be different for you, so copy it from the dashboard or kubectl get pods -A.

You’ll get a prompt like this:

vscode ➜ /app (master ✗) $

Try ls to see all of the app’s source code files courtesy of the PV that we set up before:

vscode ➜ /app (master ✗) $ ls
Controllers     Dockerfile.prod  Models      README.md       Startup.cs            appsettings.Development.json  k8s      queries.sql
Data            K8S_README.md    Program.cs  ResourceModels  Validations           appsettings.json              k8s_wip
Dockerfile.dev  Migrations       Properties  Services        VehicleQuotes.csproj  database.dbml                 obj

Now it’s just a few .NET commands to get the app up and running. First, compile and download packages:

$ dotnet build

That will take a while. Once done, let’s build the database schema:

$ dotnet ef database update

And finally, run the development web server:

$ dotnet run

If you get the error message “System.InvalidOperationException: Unable to configure HTTPS endpoint.” while trying dotnet run, follow the error message’s instructions and run dotnet dev-certs https --trust. This will generate a development certificate so that the dev server can serve HTTPS.

As a result, you should see this:

vscode ➜ /app (master ✗) $ dotnet run
Building...
info: Microsoft.Hosting.Lifetime[0]
      Now listening on: https://0.0.0.0:5001
info: Microsoft.Hosting.Lifetime[0]
      Now listening on: http://0.0.0.0:5000
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Development
info: Microsoft.Hosting.Lifetime[0]
      Content root path: /app

It indicates that the application is up and running. Now, navigate to http://localhost:30000 in your browser of choice and you should see our REST API’s Swagger UI:

Notice that 30000 is the port we specified in the web-service.yaml’s nodePort for the http port. That’s the port that the service exposes to the world outside the cluster. Notice also how our .NET web app’s development server listens to traffic coming from ports 5000 and 5001 for HTTP and HTTPS respectively. That’s why we configured web-service.yaml like we did.

Outstanding! All our hard work has paid off and we have a full-fledged web application running in our Kubernetes cluster. This is quite a momentous occasion. We’ve built a custom image that can be used to create containers to run a .NET web application, pushed that image into our local registry so that K8s could use it, and deployed a functioning application. As a cherry on top, we made it so the source code is super easy to edit, as it lives within our own machine’s file system and the container in the cluster accesses it directly from there. Quite an accomplishment.

Now it’s time to go the extra mile and organize things a bit. Let’s talk about Kustomize next.

Putting it all together with Kustomize

Kustomize is a tool that helps us improve Kubernetes’ declarative object management with configuration files (which is what we’ve been doing throughout this post). Kustomize has useful features that help with better organizing configuration files, managing configuration variables, and support for deployment variants (for things like dev vs. test vs. prod environments). Let’s explore what Kustomize has to offer.

First, be sure to tear down all the objects that we have created so far as we will be replacing them later once we have a setup with Kustomize. This will work for that:

$ kubectl delete -f db-service.yaml
$ kubectl delete -f db-deployment.yaml
$ kubectl delete -f db-persistent-volume-claim.yaml
$ kubectl delete -f db-persistent-volume.yaml

$ kubectl delete -f web-service.yaml
$ kubectl delete -f web-deployment.yaml
$ kubectl delete -f web-persistent-volume-claim.yaml
$ kubectl delete -f web-persistent-volume.yaml

Next, let’s reorganize our db-* and web-* YAML files like this:

k8s
├── db
│   ├── db-deployment.yaml
│   ├── db-persistent-volume-claim.yaml
│   ├── db-persistent-volume.yaml
│   └── db-service.yaml
└── web
    ├── web-deployment.yaml
    ├── web-persistent-volume-claim.yaml
    ├── web-persistent-volume.yaml
    └── web-service.yaml

As you can see, we’ve put them all inside a new k8s directory, and further divided them into db and web sub-directories. web-* files went into the web directory and db-* files went into db. At this point, the prefixes on the files are a bit redundant so we can remove them. After all, we know what component they belong to because of the name of their respective sub-directories.

There’s already a k8s directory in the repo. Feel free to get rid of it as we will build it back up from scratch now.

So it should end up looking like this:

k8s
├── db
│   ├── deployment.yaml
│   ├── persistent-volume-claim.yaml
│   ├── persistent-volume.yaml
│   └── service.yaml
└── web
    ├── deployment.yaml
    ├── persistent-volume-claim.yaml
    ├── persistent-volume.yaml
    └── service.yaml

kubectl’s apply and delete commands support directories as well, not only individual files. That means that, at this point, to build up all of our objects you could simply do kubectl apply -f k8s/db and kubectl apply -f k8s/web. This is much better than what we’ve been doing until now where we had to specify every single file. Still, with Kustomize, we can do better than that…

The Kustomization file

We can bring everything together with a kustomization.yaml file. For our setup, here’s what it could look like:

# k8s/kustomization.yaml
kind: Kustomization

resources:
  - db/persistent-volume.yaml
  - db/persistent-volume-claim.yaml
  - db/service.yaml
  - db/deployment.yaml
  - web/persistent-volume.yaml
  - web/persistent-volume-claim.yaml
  - web/service.yaml
  - web/deployment.yaml

This first iteration of the Kustomization file is simple. It just lists all of our other config files in the resources section in their relative locations. Save that as k8s/kustomization.yaml and you can apply it with the following:

$ kubectl apply -k k8s

The -k option tells kubectl apply to look for a Kustomization within the given directory and use that to build the cluster objects. After running it, you should see familiar output:

service/vehicle-quotes-db-service created
service/vehicle-quotes-web-service created
persistentvolume/vehicle-quotes-postgres-data-persisent-volume created
persistentvolume/vehicle-quotes-source-code-persisent-volume created
persistentvolumeclaim/vehicle-quotes-postgres-data-persisent-volume-claim created
persistentvolumeclaim/vehicle-quotes-source-code-persisent-volume-claim created
deployment.apps/vehicle-quotes-db created
deployment.apps/vehicle-quotes-web created

Feel free to explore the dashboard or kubectl get commands to see the objects that got created. You can connect to pods, run the app, query the database, everything. Just like we did before. The only difference is that now everything is neatly organized and there’s a single file that serves as bootstrap for the whole setup. All thanks to Kustomize and the -k option.

kubectl delete -k k8s can be used to tear everything down.

Defining reusable configuration values with ConfigMaps

Another useful feature of Kustomize is ConfigMaps. These allow us to specify configuration variables in the Kustomization and use them throughout the rest of the resource config files. A good candidate to demonstrate their use are the environment variables that configure our Postgres database and the connection string in our web application.

We’re going to make changes to the config so be sure to tear everything down with kubectl delete -k k8s.

We can start by adding the following to the kustomization.yaml file:

 # k8s/kustomization.yaml
 kind: Kustomization

 resources:
   - db/persistent-volume.yaml
   - db/persistent-volume-claim.yaml
   - db/service.yaml
   - db/deployment.yaml
   - web/persistent-volume.yaml
   - web/persistent-volume-claim.yaml
   - web/service.yaml
   - web/deployment.yaml

+configMapGenerator:
+  - name: postgres-config
+    literals:
+      - POSTGRES_DB=vehicle_quotes
+      - POSTGRES_USER=vehicle_quotes
+      - POSTGRES_PASSWORD=password

The configMapGenerator section is where the magic happens. We’ve kept it simple and defined the variables as literals. configMapGenerator is much more flexible than that though, accepting external configuration files. The official documentation has more details.

Now, let’s see what we have to do to actually use those values in our configuration.

First up is the database deployment configuration file, k8s/db/deployment.yaml. Update its env section like so:

# k8s/db/deployment.yaml
# ...
env:
-  - name: POSTGRES_DB
-    value: vehicle_quotes
-  - name: POSTGRES_USER
-    value: vehicle_quotes
-  - name: POSTGRES_PASSWORD
-    value: password
+  - name: POSTGRES_DB
+    valueFrom:
+      configMapKeyRef:
+        name: postgres-config
+        key: POSTGRES_DB
+  - name: POSTGRES_USER
+    valueFrom:
+      configMapKeyRef:
+        name: postgres-config
+        key: POSTGRES_USER
+  - name: POSTGRES_PASSWORD
+    valueFrom:
+      configMapKeyRef:
+        name: postgres-config
+        key: POSTGRES_PASSWORD
# ...

Notice how we’ve replaced the simple key-value pairs with new, more complex objects. Their names are still the same, they have to be because that’s what the Postgres database container expects. But instead of a literal, hard-coded value, we have changed them to these valueFrom.configMapKeyRef objects. Their names match the name of the configMapGenerator we configured in the Kustomization. Their keys match the keys of the literal values that we specified in the configMapGenerator’s literals field. That’s how it all ties together.

Similarly, we can update the web application deployment configuration file, k8s/web/deployment.yaml. Its env section would look like this:

# k8s/web/deployment.yaml
# ...
env:
-  - name: POSTGRES_DB
-    value: vehicle_quotes
-  - name: POSTGRES_USER
-    value: vehicle_quotes
-  - name: POSTGRES_PASSWORD
-    value: password
+  - name: POSTGRES_DB
+    valueFrom:
+      configMapKeyRef:
+        name: postgres-config
+        key: POSTGRES_DB
+  - name: POSTGRES_USER
+    valueFrom:
+      configMapKeyRef:
+        name: postgres-config
+        key: POSTGRES_USER
+  - name: POSTGRES_PASSWORD
+    valueFrom:
+      configMapKeyRef:
+        name: postgres-config
+        key: POSTGRES_PASSWORD
  - name: CUSTOMCONNSTR_VehicleQuotesContext
    value: Host=$(VEHICLE_QUOTES_DB_SERVICE_SERVICE_HOST);Database=$(POSTGRES_DB);Username=$(POSTGRES_USER);Password=$(POSTGRES_PASSWORD)
# ...

This is the exact same change as with the database deployment. Out with the hard coded values and in with the new ConfigMap-driven ones.

Try kubectl apply -k k8s and you’ll see that things are still working well. Try to connect to the web application pod and build and run the app.

For data that’s important to secure like passwords, tokens, and keys, Kubernetes and Kustomize also offer Secrets and secretGenerator. Secrets are very similar to ConfigMaps in how they work, but are tailored specifically for handling secret data. You can learn more about them in the official documentation.

Creating variants for production and development environments

The crowning achievement of Kustomize is its ability to facilitate multiple deployment variants. Variants, as the name suggests, are variations of deployment configurations that are ideal for setting up various execution environments for an application. Think development, staging, production, etc., all based on a common set of reusable configurations to avoid superfluous repetition.

Kustomize does this by introducing the concepts of bases and overlays. A base is a set of configs that can be reused but not deployed on its own, and overlays are the actual configurations that use and extend the base and can be deployed.

To demonstrate this, let’s build two variants: one for development and another for production. Let’s consider the one we’ve already built to be the development variant and work towards properly specifying it as so, and then building a new production variant.

Note that the so-called “production” variant we’ll build is not actually meant to be production worthy. It’s just an example to illustrate the concepts and process of building bases and overlays. It does not meet the rigors of a proper production system.

Creating the base and overlays

The strategy I like to use is to just copy everything over from one variant to another, implement the differences, identify the common elements, and extract them into a base that both use.

Let’s begin by creating a new k8s/dev directory and move all of our YAML files into it. That will be our “development overlay”. Then, make a copy the k8s/dev directory and all of its contents and call it k8s/prod. That will be our “production overlay”. Let’s also create a k8s/base directory to store the common files. That will be our “base”. It should be like this:

k8s
├── base
├── dev
│   ├── kustomization.yaml
│   ├── db
│   │   ├── deployment.yaml
│   │   ├── persistent-volume-claim.yaml
│   │   ├── persistent-volume.yaml
│   │   └── service.yaml
│   └── web
│       ├── deployment.yaml
│       ├── persistent-volume-claim.yaml
│       ├── persistent-volume.yaml
│       └── service.yaml
└── prod
    ├── kustomization.yaml
    ├── db
    │   ├── deployment.yaml
    │   ├── persistent-volume-claim.yaml
    │   ├── persistent-volume.yaml
    │   └── service.yaml
    └── web
        ├── deployment.yaml
        ├── persistent-volume-claim.yaml
        ├── persistent-volume.yaml
        └── service.yaml

Now we have two variants, but they don’t do us any good because they aren’t any different. We’ll now go through each file one by one and identify which aspects need to be the same and which need to be different between our two variants:

db/deployment.yaml: I want the same database instance configuration for both our variants. So we copy the file into base/db/deployment.yaml and delete dev/db/deployment.yaml and prod/db/deployment.yaml.
db/persistent-volume-claim.yaml: This one is also the same for both variants. So we copy the file into base/db/persistent-volume-claim.yaml and delete dev/db/persistent-volume-claim.yaml and prod/db/persistent-volume-claim.yaml.
db/persistent-volume.yaml: This file defines the location in the host machine that will be available for the Postgres instance that’s running in the cluster to store its data files. I do want this path to be different between variants. So let’s leave them where they are and do the following changes to them: For dev/db/persistent-volume.yaml, change its spec.hostPath.path to "/path/to/vehicle-quotes-postgres-data-dev". For prod/db/persistent-volume.yaml, change its spec.hostPath.path to "/path/to/vehicle-quotes-postgres-data-prod". Of course, adjust the paths to something that makes sense in your environment.
db/service.yaml: There doesn’t need to be any difference in this file between the variants so we copy it into base/db/service.yaml and delete dev/db/service.yaml and prod/db/service.yaml.
web/deployment.yaml: There are going to be quite a few differences between the dev and prod deployments of the web application. So we leave them as they are. Later we’ll see the differences in detail.
web/persistent-volume-claim.yaml: This is also going to be different. Let’s leave it be now and we’ll come back to it later.
web/persistent-volume.yaml: Same as web/persistent-volume-claim.yaml. Leave it be for now.
wev/service.yaml: This one is going to be the same for both dev and prod so let’s do the usual and copy it into base/web/service.yaml and remove dev/web/service.yaml and prod/web/service.yaml

The decisions made when designing these overlays and the base may seem arbitrary. That’s because they totally are. The purpose of this article is to demonstrate Kustomize’s features, not produce a real-world, production-worthy setup.

Once all those changes are done, you should have the following file structure:

k8s
├── base
│   ├── db
│   │   ├── deployment.yaml
│   │   ├── persistent-volume-claim.yaml
│   │   └── service.yaml
│   └── web
│       └── service.yaml
├── dev
│   ├── kustomization.yaml
│   ├── db
│   │   └── persistent-volume.yaml
│   └── web
│       ├── deployment.yaml
│       ├── persistent-volume-claim.yaml
│       └── persistent-volume.yaml
└── prod
    ├── kustomization.yaml
    ├── db
    │   └── persistent-volume.yaml
    └── web
        ├── deployment.yaml
        ├── persistent-volume-claim.yaml
        └── persistent-volume.yaml

Much better, huh? We’ve gotten rid of quite a bit of repetition. But we’re not done just yet. The base also needs a Kustomization file. Let’s create it as k8s/base/kustomization.yaml and add these contents:

# k8s/base/kustomization.yaml
kind: Kustomization

resources:
  - db/persistent-volume-claim.yaml
  - db/service.yaml
  - db/deployment.yaml
  - web/service.yaml

configMapGenerator:
  - name: postgres-config
    literals:
      - POSTGRES_DB=vehicle_quotes
      - POSTGRES_USER=vehicle_quotes
      - POSTGRES_PASSWORD=password

As you can see, the file is very similar to the other one we created. We just list the resources that we moved into the base directory and define the database environment variables via the configMapGenerator. We need to define the configMapGenerator here because we’ve moved all the other files that use them into here.

Now that we have the base defined, we need to update the kustomization.yaml files of the overlays to use it. We also need to update them so that they only point to the resources that they need to.

Here’s how the changes to the “dev” overlay’s kustomization.yaml file look:

 # dev/kustomization.yaml
 kind: Kustomization

+bases:
+  - ../base

 resources:
   - db/persistent-volume.yaml
-  - db/persistent-volume-claim.yaml
-  - db/service.yaml
-  - db/deployment.yaml
   - web/persistent-volume.yaml
   - web/persistent-volume-claim.yaml
-  - web/service.yaml
   - web/deployment.yaml

-configMapGenerator:
-  - name: postgres-config
-    literals:
-      - POSTGRES_DB=vehicle_quotes
-      - POSTGRES_USER=vehicle_quotes
-      - POSTGRES_PASSWORD=password

As you can see we removed the configMapGenerator and the individual resources that were already defined in the base. Most importantly, we’ve added a bases element that indicates that our Kustomization over on the base directory is this overlay’s base.

The changes to the “prod” overlay’s kustomization.yaml file are identical. Go ahead and make them.

At this point, you can run kubectl apply -k k8s/dev or kubectl apply -k k8s/prod and things should work just like before.

Don’t forget to also do kubectl delete -k k8s/dev or kubectl delete -k k8s/prod when you’re done testing the previous commands, as we’ll continue doing changes to the configs. Keep in mind also that both variants can’t be deployed at the same time. So be sure delete one before applying the other.

Developing the production variant

I want our production variant to use a different image for the web application. That means a new Dockerfile. If you downloaded the source code from the GitHub repo, you should see the production Dockerfile in the root directory of the repo. It’s called Dockerfile.prod.

Here’s what it looks like:

# Dockerfile.prod
ARG VARIANT="5.0"
FROM mcr.microsoft.com/dotnet/sdk:${VARIANT}

RUN apt-get update && export DEBIAN_FRONTEND=noninteractive \
    && apt-get -y install --no-install-recommends postgresql-client-common postgresql-client

RUN dotnet tool install --global dotnet-ef
ENV PATH $PATH:/root/.dotnet/tools

RUN dotnet dev-certs https

WORKDIR /source

COPY . .

ENTRYPOINT ["tail", "-f", "/dev/null"]

The first takeaway from this production Dockerfile is that it is simpler, when compared to the development one. The image here is based on the official dotnet/sdk instead of the dev-ready one from vscode/devcontainers/dotnet. Also, this Dockerfile just copies all the source code into a /source directory within the image. This is because we want to “ship” the image with everything it needs to work without too much manual intervention. Also, we won’t be editing code live on the container, as opposed to how we set up the dev variant which allowed that. So we just copy the files over instead of leaving them out to provision them later via volumes. We’ll see how that pans out later.

Now that we have our production Dockerfile, we can build an image with it and push it to the registry so that Kubernetes can use it. So, save that file as Dockerfile.prod (or just use the one that’s already in the repo), and run the following commands:

Build the image with:

$ docker build . -f Dockerfile.prod -t localhost:32000/vehicle-quotes-prod:registry

And push it to the registry with:

$ docker push localhost:32000/vehicle-quotes-prod:registry

Now, we need to modify our prod variant’s deployment configuration so that it can work well with this new prod image. Here’s how the new k8s/prod/web/deployment.yaml should look:

# k8s/prod/web/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vehicle-quotes-web
spec:
  selector:
    matchLabels:
      app: vehicle-quotes-web
  replicas: 1
  template:
    metadata:
      labels:
        app: vehicle-quotes-web
    spec:
      initContainers:
        - name: await-db-ready
          image: postgres:13
          command: ["/bin/sh"]
          args: ["-c", "until pg_isready -h $(VEHICLE_QUOTES_DB_SERVICE_SERVICE_HOST) -p 5432; do echo waiting for database; sleep 2; done;"]
        - name: build
          image: localhost:32000/vehicle-quotes-dev:registry
          workingDir: "/source"
          command: ["/bin/sh"]
          args: ["-c", "dotnet restore -v n && dotnet ef database update && dotnet publish -c release -o /app --no-restore"]
          volumeMounts:
            - mountPath: "/app"
              name: vehicle-quotes-source-code-storage
          env:
            - name: POSTGRES_DB
              valueFrom:
                configMapKeyRef:
                  name: postgres-config
                  key: POSTGRES_DB
            - name: POSTGRES_USER
              valueFrom:
                configMapKeyRef:
                  name: postgres-config
                  key: POSTGRES_USER
            - name: POSTGRES_PASSWORD
              valueFrom:
                configMapKeyRef:
                  name: postgres-config
                  key: POSTGRES_PASSWORD
            - name: CUSTOMCONNSTR_VehicleQuotesContext
              value: Host=$(VEHICLE_QUOTES_DB_SERVICE_SERVICE_HOST);Database=$(POSTGRES_DB);Username=$(POSTGRES_USER);Password=$(POSTGRES_PASSWORD)
      containers:
        - name: vehicle-quotes-web
          image: localhost:32000/vehicle-quotes-dev:registry
          workingDir: "/app"
          command: ["/bin/sh"]
          args: ["-c", "dotnet VehicleQuotes.dll --urls=https://0.0.0.0:5001/"]
          ports:
            - containerPort: 5001
              name: "https"
          volumeMounts:
            - mountPath: "/app"
              name: vehicle-quotes-source-code-storage
          env:
            - name: POSTGRES_DB
              valueFrom:
                configMapKeyRef:
                  name: postgres-config
                  key: POSTGRES_DB
            - name: POSTGRES_USER
              valueFrom:
                configMapKeyRef:
                  name: postgres-config
                  key: POSTGRES_USER
            - name: POSTGRES_PASSWORD
              valueFrom:
                configMapKeyRef:
                  name: postgres-config
                  key: POSTGRES_PASSWORD
            - name: CUSTOMCONNSTR_VehicleQuotesContext
              value: Host=$(VEHICLE_QUOTES_DB_SERVICE_SERVICE_HOST);Database=$(POSTGRES_DB);Username=$(POSTGRES_USER);Password=$(POSTGRES_PASSWORD)
          resources:
            limits:
              memory: 2Gi
              cpu: "1"
      volumes:
        - name: vehicle-quotes-source-code-storage
          emptyDir: {}

This deployment config is similar to the one from the dev variant, but we’ve changed a few elements on it.

Init containers

The most notable change is that we added an initContainers section. Init Containers are one-and-done containers that run specific processes during pod initialization. They are good for doing any sort of initialization tasks that need to be run once, before a pod is ready to work. After they’ve done their task, they go away, and the pod is left with the containers specified in the containers section, like usual. In this case, we’ve added two init containers.

First is the await-db-ready one. This is a simple container based on the postgres:13 image that just sits there waiting for the database to become available. This is thanks to its command and args, which make up a simple shell script that leverages the pg_isready tool to continuously check if connections can be made to our database:

command: ["/bin/sh"]
args: ["-c", "until pg_isready -h $(VEHICLE_QUOTES_DB_SERVICE_SERVICE_HOST) -p 5432; do echo waiting for database; sleep 2; done;"]

This will cause pod initialization to stop until the database is ready.

Thanks to this blog post for the very useful recipe.

We need to wait for the database to be ready before continuing because of what the second init container does. Among other things, the build init container sets up the database. The database needs to be available for it to be able to to do that. The init container also downloads dependencies, builds the app, produces the deployable artifacts and copies them over to the directory from which the app will run: /app. You can see that all that is specified in the command and args elements, which define a few shell commands to do those tasks.

command: ["/bin/sh"]
args: ["-c", "dotnet restore -v n && dotnet ef database update && dotnet publish -c release -o /app --no-restore"]

Another interesting aspect of this deployment is the volume that we’ve defined. It’s at the bottom of the file, take a quick look:

volumes:
  - name: vehicle-quotes-source-code-storage
    emptyDir: {}

This one is different from the ones we’ve seen before which relied on persistent volumes and persistent volume claims. This one uses emptyDir. This means that this volume will provide storage that is persistent throughout the lifetime of the pod. That is, not tied to any specific container. In other words, even when the container goes away, the files in this volume will stay. This is a mechanism that’s useful when we want one container to produce some files that another container will use. In our case, the build init container produces the artifacts/binaries that the main vehicle-quotes-web container will use to actually run the web app.

The only other notable difference of this deployment is how its containers use the new prod image that we built before, instead of the dev one. That is, it uses localhost:32000/vehicle-quotes-prod:registry instead of localhost:32000/vehicle-quotes-dev:registry.

The rest of the deployment doesn’t have anything we haven’t already seen. Feel free to explore it.

As you saw, this prod variant doesn’t need to access the source code via a persistent volume. So, we don’t need PV and PVC definitions for it. So feel free to delete k8s/prod/web/persistent-volume.yaml and k8s/prod/web/persistent-volume-claim.yaml. Remember to also remove them from the resources section in k8s/prod/kustomization.yaml.

With those changes done, we can fire up our prod variants with:

$ kubectl apply -k k8s/prod

The web pod will take quite a while to properly start up because it’s downloading a lot of dependencies. Remember that you can use kubectl get pods -A to see their current status. Take note of their names and you would also be able to see container specific logs.

Use kubectl logs -f <vehicle-quotes-web-pod-name> await-db-ready to see the logs from the await-db-ready init container.
Use kubectl logs -f <vehicle-quotes-web-pod-name> build to see the logs from the build init container.

If this were an actual production setup, and we were worried about pod startup time, there’s one way we could make it faster. We could perform the “download dependencies” step when building the production image instead of when deploying the pods. So, we could have our Dockerfile.prod call dotnet restore -v n, instead of the build init container. That way building the image would take more time, but it would have all dependencies already baked in by the time Kubernetes tries to use it to build containers. Then the web pod would start up much faster.

This deployment automatically starts the web app, so after the pods are in the “Running” status (or green in the dashboard!), we can just navigate to the app via a web browser. We’ve configured the deployment to only work over HTTPS (as given by the ports section in the vehicle-quotes-web container), so this is the only URL that’s available to us: https://localhost:30001. We can navigate to it and see the familiar screen:

At this point, we finally have fully working, distinct variants. However, we can take our configuration a few steps further by leveraging some additional Kustomize features.

Using patches for small, precise changes

Right now, the persistent volume configurations for the databases of both variants are pretty much identical. The only difference is the hostPath. With patches, we can focus in on that property and vary it specifically.

To do it, we first copy either of the variant’s db/persistent-volume.yaml into k8s/base/db/persistent-volume.yaml. We also need to add it under resources on k8s/base/kustomization.yaml:

 # k8s/base/kustomization.yaml
 # ...
 resources:
+  - db/persistent-volume.yaml
   - db/persistent-volume-claim.yaml
 # ...

That will serve as the common ground for overlays to “patch over”. Now we can create the patches. First, the one for the one for the dev variant:

# k8s/dev/db/persistent-volume-host-path.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: vehicle-quotes-postgres-data-persisent-volume
spec:
  hostPath:
    path: "/home/kevin/projects/vehicle-quotes-postgres-data-dev"

And then the one for the prod variant:

# k8s/prod/db/persistent-volume-host-path.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: vehicle-quotes-postgres-data-persisent-volume
spec:
  hostPath:
    path: "/home/kevin/projects/vehicle-quotes-postgres-data-prod"

As you can see, these patches are sort of truncated persistent volume configs which only include the kind, metadata.name, and the value that actually changes: the hostPath.

Once those are saved, we need to include them in their respective kustomization.yaml. It’s the same modification to both k8s/dev/kustomization.yaml and k8s/prod/kustomization.yaml. Just remove the db/persistent-volume.yaml item from their resources sections and add the following to both of them:

patches:
  - db/persistent-volume-host-path.yaml

Right now, k8s/dev/kustomization.yaml should be:

# k8s/dev/kustomization.yaml
kind: Kustomization

bases:
  - ../base

resources:
  - web/persistent-volume.yaml
  - web/persistent-volume-claim.yaml
  - web/deployment.yaml

patches:
  - db/persistent-volume-host-path.yaml

And k8s/prod/kustomization.yaml should be:

# k8s/prod/kustomization.yaml
kind: Kustomization

bases:
  - ../base

resources:
  - web/deployment.yaml

patches:
  - db/persistent-volume-host-path.yaml

Overriding container images

Another improvement we can make is to use the images element in the kustomization.yaml files to control the web app images used by the deployments in the variants. This is easier for maintenance as it’s defined in one single, expected place. Also, the full name of the image doesn’t have to be used throughout the configs so it reduces repetition.

To put it in practice, add the following at the end of the k8s/dev/kustomization.yaml file:

images:
  - name: vehicle-quotes-web
    newName: localhost:32000/vehicle-quotes-dev
    newTag: registry

Similar thing with k8s/prod/kustomization.yaml, only use the prod image for this one:

images:
  - name: vehicle-quotes-web
    newName: localhost:32000/vehicle-quotes-prod
    newTag: registry

Now, we can replace any mention of localhost:32000/vehicle-quotes-dev in the dev variant, and any mention of localhost:32000/vehicle-quotes-prod in the prod variant with vehicle-quotes-web. Which is simpler.

In k8s/dev/web/deployment.yaml:

 # ...
     spec:
       containers:
         - name: vehicle-quotes-web
-          image: localhost:32000/vehicle-quotes-dev:registry
+          image: vehicle-quotes-web
           ports:
 # ...

And in k8s/prod/web/deployment.yaml:

 # ...

         - name: build
-          image: localhost:32000/vehicle-quotes-prod:registry
+          image: vehicle-quotes-web
           workingDir: "/source"
           command: ["/bin/sh"]
 # ...

       containers:
         - name: vehicle-quotes-web
-          image: localhost:32000/vehicle-quotes-prod:registry
+          image: vehicle-quotes-web
           workingDir: "/app"
 # ...

Once that’s all done, you should be able to kubectl apply -k k8s/dev or kubectl apply -k k8s/prod and everything should work fine. Be sure to kubectl delete before kubectl applying a different variant though, as both of them cannot coexist in the same cluster, due to many objects having the same name.

Closing thoughts

Wow! That was a good one. In this post I’ve captured all the knowledge that I wish I had when I first encountered Kubernetes. We went from knowing nothing to being able to put together a competent environment. We figured out how to install Kubernetes locally via MicroK8s, along with a few useful add-ons. We learned about the main concepts in Kubernetes like nodes, pods, images, containers, deployments, services, and persistent volumes. Most importantly, we learned how to define and create them using a declarative configuration file approach.

Then, we learned about Kustomize and how to use it to implement variants of our configurations. And we did all that by actually getting our hands dirty and, step by step, deploying a real web application and its backing database system. When all was said and done, a simple kubeclt apply -k <kustomization> was all it took to get the app up and running fully. Not bad, eh?

Useful commands

Start up MicroK8s: microk8s start
Shut down MicroK8s: microk8s stop
Check MicroK8s info: microk8s status
Start up the K8s dashboard: microk8s dashboard-proxy
Get available pods on all namespaces: kubectl get pods -A
Watch and follow the logs on a specific container in a pod: kubectl logs -f <POD_NAME> <CONTAINER_NAME>
Open a shell into the default container in a pod: kubectl exec -it <POD_NAME> -- bash
Create a K8s resource given a YAML file or directory: kubectl apply -f <FILE_OR_DIRECTORY_NAME>
Delete a K8s resource given a YAML file or directory: kubectl delete -f <FILE_OR_DIRECTORY_NAME>
Create K8s resources with Kustomize: kubectl apply -k <KUSTOMIZATION_DIR>
Delete K8s resources with Kustomize: kubectl delete -k <KUSTOMIZATION_DIR>
Build custom images for the K8s registry: docker build . -f <DOCKERFILE> -t localhost:32000/<IMAGE_NAME>:registry
Push custom images to the K8s registry: docker push localhost:32000/<IMAGE_NAME>:registry

.NET/C# developer job opening

2021-11-09T00:00:00+00:00

Photo by #WOCinTech Chat · CC BY 2.0, modified

We are seeking a full-time .NET/C# software developer based in the United States to work with us on our clients’ applications.

End Point Dev is an Internet technology consulting company based in New York City, with 50 employees serving many clients ranging from small family businesses to large corporations. The company is going strong after 26 years in business!

Even before the pandemic most of us worked remotely from home offices. We collaborate using SSH, Git, project tracking tools, Zulip chat, video conferencing, and of course email and phones.

What you will be doing:

Develop new web applications and support existing ones for our clients.
Work together with End Point Dev co-workers and our clients’ in-house staff.
Use your desktop operating system of choice: Windows, macOS, or Linux.
Enhance open source software and contribute back as opportunity arises.

You’ll need professional development experience with:

3+ years of development with .NET and C#
Databases such as SQL Server, PostgreSQL, Redis, Solr, Elasticsearch, etc.
Front-end web development with HTML, CSS, JavaScript and frameworks such as Vue, React, Angular
Security consciousness such as under PCI-DSS for ecommerce or HIPAA medical data
Git version control
Automated testing
HTTP, REST APIs

You have these important work traits:

Strong verbal and written communication skills
An eye for detail
Tenacity in solving problems and focusing on customer needs
A feeling of ownership of your projects
Work both independently and as part of a team
A good remote work environment

For some of our clients you will need to submit to and pass a criminal background check.

What work here offers:

Collaboration with knowledgeable, friendly, helpful, and diligent co-workers around the world
Work from your home office, or from our offices in New York City and the Tennessee Tri-Cities area
Flexible, sane work hours
Paid holidays and vacation
Health insurance subsidy and 401(k) retirement savings plan
Annual bonus opportunity

Get in touch with us:

~~Please email us an introduction to jobs@endpointdev.com to apply.~~ (This job has been filled.)

Include your location, a resume/CV, your Git repository or LinkedIn URLs, and whatever else may help us get to know you.

We look forward to hearing from you! Direct work seekers only, please—this role is not for agencies or subcontractors.

We are an equal opportunity employer and value diversity at our company. We do not discriminate on the basis of sex/gender, race, religion, color, national origin, sexual orientation, age, marital status, veteran status, or disability status.

.NET Conf 2021 is coming!

2021-10-27T00:00:00+00:00

It’s that time of the year again! It has been almost one year since .NET 5 was launched at .NET Conf 2020, unifying .NET Framework and .NET Core into a single open-source and cross-platform framework. With .NET 6 around the corner, it’s time to prepare for the new edition, .NET Conf 2021, starting on November 9th.

This edition will be the 11th online conference, and the Agenda will mainly focus on the .NET 6 launch and the new C# 10, along with some coding challenges and community sessions. The event is organized by both the .NET community and Microsoft. The main changes that will likely be discussed at the conference, based on what we’ve seen through the previews, might include:

Mobile development

We will see several changes to how web mobile app development works under .NET 6. With a smaller, optimized, and optional SDK for mobile based on Xamarin, now called Multi-platform App UI or MAUI, developing an app that targets multiple mobile platforms should be simpler than ever.

Hot reload (after some struggling)

The Hot Reload feature, which will allow us to make and apply changes to the code at execution time, will be finally available in .NET 6. There was some noise about it in the past days since Microsoft made a decision to lock it to Visual Studio 2022, which is a Windows-limited mostly-paid product. But after the open source community made it clear they were angry about it, Microsoft reversed the change and the feature will be available for all platforms.

Blazor 6

The new version of Blazor will also have improvements. One of the main features is the implementation of ahead-of-time compilation, that allows generating WebAssembly code during the publishing process. That way, the application performance is increased, since it can run natively instead of needing a .NET IL interpreter.

LTS (Long Term Support)

.NET 6 is a Long Term Support version, which means that it will have support for at least 3 years after its release. The current version, .NET 5, is a General Availability (GA) version, which means its support will likely end six months after the next release is available.

Several improvements were introduced to different aspects of the framework, including AOT compilation or the Minimal API Framework. There are also improvements to the build speed as we can see in the chart below. A complete list of breaking changes can be seen here.

Bonus: Although not part of the event itself, the new version of Visual Studio will also be available from November 8th, and it will be launched on an online event that Microsoft is preparing with several presentations from the development team.

As we did last year, the .NET team at End Point Dev will be listening to the talks and presentations from the conference, and trying out the new features that will be publicly available on November 9th. Exciting days ahead! We hope to see you there.

Deploying a .NET 5 app on IIS

2021-09-27T00:00:00+00:00

Photo by Christian Cable, CC BY 2.0

.NET 5 has been around for a few years now, after being released at .NET Conf 2020, containing the best of both worlds: .NET Core, including multi-platform support and several performance improvements, and .NET Framework, including Windows desktop development support with WPF and Windows Forms (UWP is also supported, but not officially yet).

A .NET Core-based project can be published into any platform (as long as we’re not depending on libraries targeted to .NET Framework), allowing us to save costs by hosting on Linux servers and increasing performance by having cheaper scalability options. But most developers are still using Windows with Internet Information Services (IIS) as the publishing target, likely due to the almost 20 years of history of .NET Framework, compared to the relatively short history of .NET Core, launched in 2016.

Our .NET project

We won’t review the steps needed to set up a new .NET 5 project, since this time we are only focusing on publishing what we already have developed. But to understand how our application will integrate with IIS and the framework, it’s important to note a fundamental change any .NET 5 project has in comparison with a .NET Framework one:

Since .NET 5 is .NET Core in its foundation, our project output will actually be a console application. If we create a new .NET Core project, no matter which version we are using, we will find a Program.cs file in the root with an application entry point in it, that will look somewhat similar to the one below:

public class Program
{
	public static void Main(string[] args)
	{
		BuildWebHost(args).Run();
	}

	public static IWebHost BuildWebHost(string[] args) =>
		WebHost.CreateDefaultBuilder(args)
			.UseStartup<Startup>()
			.Build();
}

The WebHost object will be the one processing requests to the app, as well as setting configuration like the content root, accessing environment variables, and logging.

This application needs to be executed by the dotnet process, which comes with any .NET 5 runtime.

Installing a .NET 5 runtime

The first step we need to do in our destination server, is to prepare the environment to run .NET 5 apps by installing the .NET 5 Hosting Bundle, a 65 MB setup file which has all the basic stuff needed to run .NET 5 on Windows. Since .NET can also run on Linux and macOS, installer executables are available for these operating systems as well.

Once the installation finishes, we will need to restart IIS by typing iisreset on an elevated command prompt.

Creating the application pool

It’s always recommended to create a new application pool for a new website that will be published. That allows us to run the website in a separate IIS process, which is safer and prevents other websites from crashing if an application throws an unhandled exception. To create a new application pool, right-click on the “Application pools” section on the IIS Manager sidebar and choose the option “Add application pool”.

Since .NET 5 is based on .NET Core, the application pool we create will not be loaded inside the .NET Framework runtime environment. All .NET 5 applications will run by calling the external dotnet process, which is the reason why we need to install a separate hosting bundle in the first place.

That means that, when we are creating our application pool, we will need to set the .NET CLR version to “No managed code” before saving changes, as shown below:

Creating the new website

With the bundle installed and a new application pool created, it’s time to add the new website where our application will be published to. Right-click on the “Sites” section on the IIS Manager sidebar and choose the “Add Website” option.

We can choose any name to identify the new website. The important thing is to point the website to the newly created application pool, and bind it to the correct IP address and domain/host name, as shown below:

Once we accept the changes, the new website will be automatically started, which means we should be able to reach the IP address/hostname we entered. We will get a default page or a 404 response, depending on how our IIS instance is configured, since we haven’t published our application yet.

Publishing our project

Finally, it’s time to publish our .NET application into the new website. If we have Visual Studio, we can have the IDE automatically upload our content to IIS and publish it by right-clicking on our project and choosing “Publish”. Or we can use the dotnet command with the publish parameter to do it ourselves.

I usually prefer to do a manual publish into a local folder, and then decide which content I need to copy into the destination. Sometimes we only update a portion of the backend logic, in which case copying the output DLLs is all we need to do.

If we choose to let Visual Studio handle the publishing process, we need to choose “Web Server (IIS) / Web deploy” as our publish destination and enter the information needed for Visual Studio to connect to the server and copy the files:

We need to make sure that the “site name” we enter here corresponds to the site name we entered when we created the new website on the server.

If you prefer to manually copy the files like me, on the destination screen choose “Folder” instead of “Web Server (IIS)”. That option will copy the project’s output into the specified folder, so it can be manually copied into our website’s root later.

And that’s it! Our project is now published into the website we created. We can hit our IP address/hostname again and we should now get our website’s default response.

To sum it up, the main difference between publishing a .NET Framework app and a .NET 5 app is that we need to install a runtime and do a couple of tweaks when setting up the new application pool. But in general, it’s a pretty straightforward process.

Monitoring Settings Changes in ASP.NET Core

2021-09-22T00:00:00+00:00

Photo by Linus Nylund on Unsplash

Did you know you can directly respond to config file changes in ASP.NET Core? By using the IOptionsMonitor<T> interface, it’s possible to run a lambda function every time a config file is updated.

For those who aren’t too familiar with it, ASP.NET Core configuration is handled using the Options Pattern. That is, you create a model class that has a field for each property in your config file, and then set up the application to map the configuration to that class, which can be injected into controllers and services. In your controllers, you can request the configuration class T as a dependency by using an options wrapper class like IOptions<T>.

While this works just fine in some cases, it doesn’t allow you to actually run code when the configuration is changed. However, if you request your configuration using IOptionsMonitor<T>, you get the ability to define a lambda function to run every time appsettings.json is changed.

One use case for this functionality would be if you wanted to maintain a list in the config file, and log every time that list was changed. In this article I’ll explain how to set up an ASP.NET Core 5.0 API to run custom code whenever changes are made to appsettings.json.

Setting up the API

In order to use the options pattern in your API, you’ll first need to add the options services to the container using the services.AddOptions() method. Then, you can register your custom configuration class (in this example, MyOptions) to be bound to a specific section in appsettings.json (in this example, "myOptions").

public void ConfigureServices(IServiceCollection services)
{
  // ...

  // Add options services to the container
  services.AddOptions();

  // Configure the app to map the "myOptions" section of
  // the config file to the MyOptions model class
  services.Configure<MyOptions>(
    Configuration.GetSection("myOptions")
  );
}

Monitoring Configuration Changes

Now that the API is set up correctly, in your controllers you can directly request the configuration using IOptionsMonitor<T>. You can also unpack the configuration instance itself by using the IOptionsMonitor<T>.CurrentValue property. This value will automatically get updated by IOptionsMonitor<T> when the configuration is changed, so you only need to retrieve it once in the constructor.

using System.Threading.Tasks;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Options;

[Route("api/[controller]")]
[ApiController]
public class MyController : ControllerBase
{
  private MyOptions MyOptions;

  public MyController(IOptionsMonitor<MyOptions> MyOptionsMonitor)
  {
    // Stores the actual configuration in the controller
    // to reference later. Changes to the config will be
    // reflected in MyOptions.
    this.MyOptions = MyOptionsMonitor.CurrentValue;

    // Registers a lambda function to be called every time
    // a configuration change is made
    MyOptionsMonitor.OnChange(async (opts) =>
    {
      // Write some code!
    });
  }
}

One small gotcha worth noting is that the OnChange function isn’t debounced. So if you’re using an IDE that automatically saves as you type, it’ll trigger the OnChange function rapidly as you edit appsettings.json.

Conclusion

And that’s it! It doesn’t take much to get your API to run custom code on config file changes.

Have any questions? Feel free to leave a comment!

Building REST APIs with .NET 5, ASP.NET Core, and PostgreSQL

2021-07-09T00:00:00+00:00

Photo by Sam Beasley

This is old news by now, but I’m still amazed by the fact that nowadays .NET is open source and can run on Linux. I truly believe that this new direction can help the technology realize its true potential, since it’s no longer shackled to Windows-based environments. I’ve personally been outside the .NET game for a good while, but with the milestone release that is .NET 5, I think now is a great time to dive back in.

So I thought of taking some time to do just that, really dive in, see what’s new, and get a sense of the general developer experience that the current incarnation of .NET offers. So in this blog post, I’m going to chronicle my experience developing a simple but complete REST API application. Along the way, I’ll touch on the most common problems that one runs into when developing such applications and how they are solved in the .NET world. So think of this piece as a sort of tutorial or overview of the most common framework features when it comes to developing REST APIs.

There’s a table of contents at the bottom.

First, let’s get familiar with what we’re building.

What we’re building

The demo application

You can find the finished product on my GitHub.

The application that we’ll be building throughout this article will address a request from a hypothetical car junker business. Our client wants to automate the process of calculating how much money to offer their customers for their vehicles, given certain information about them. And they want an app to do that. We are building the back-end component that will support that app. It is a REST API that allows users to provide vehicle information (year, make, model, condition, etc.) and will produce a quote of how much money our hypothetical client would be willing to pay for it.

Here’s a short list of features that we need to implement in order to fulfill that requirement:

Given a vehicle model and condition, calculate a price.
Store and manage rules that are used to calculate vehicle prices.
Store and manage pricing overrides on a vehicle model basis. Price overrides are used regardless of the current rules.
CRUD vehicle models so that overrides can be specified for them.

The data model

Here’s what our data model looks like:

The main table in our model is the quotes table. It stores all the requests for quotes received from our client’s customers. It captures all the relevant vehicle information in terms of model and condition. It also captures the offered quote; that is, the money value that our system calculates for their vehicle.

The quotes table includes all the fields that identify a vehicle: year, make, model, body style and size. It also includes a model_style_year_id field which is an optional foreign key to another table. This FK points to the model_style_years table which contains specific vehicle models that our system can store explicitly.

The idea of this is that, when a customer submits a request for a quote, if we have their vehicle registered in our database, then we can populate this foreign key and link the quote with the specific vehicle that it’s quoting. If we don’t have their vehicle registered, then we leave that field unpopulated. Either way, we can offer a quote. The only difference is the level or certainty of the quote.

The records in the model_style_years table represent specific vehicles. That whole hierarchy works like this: A vehicle make (e.g. Honda, Toyota, etc. in the makes table) has many models (e.g. Civic, Corolla, etc. in the models table), each model has many styles (the model_styles table). Styles are combinations of body types (the body_types table) and sizes (e.g. Mid-size Sedan, Compact Coupe, etc. in the sizes table). And finally, each model style has many years in which they were being produced (via the model_style_years table).

This model allows us very fine-grained differentiation between vehicles. For example, we can have a “2008 Honda Civic Hatchback which is a Compact car” and also a “1990 Honda Civic Hatchback which is a Sub-compact”. That is, same model, different year, size or body type.

We also have a quote_rules table which stores the rules that are applied when it comes to calculating a vehicle quote. The rules are pairs of key-values with an associated monetary value. So for example, rules like “a vehicle that has alloy wheels is worth $10 more” can be expressed in the table with a record where feature_type is “has_alloy_wheels”, feature_value is “true” and price_modifier is “10”.

Finally, we have a quote_overrides table which specifies a flat, static price for specific vehicles (via the link to the model_style_years table). The idea here is that if some customer requests a quote for a vehicle for which we have an override, no price calculation rules are applied and they are offered what is specified in the override record.

The development environment

Setting up the PostgreSQL database with Docker

For this project, our database of choice is PostgreSQL. Luckily for us, getting a PostgreSQL instance up and running is very easy thanks to Docker.

If you want to learn more about dockerizing a typical web application, take a look at this article that explains the process in detail.

Once you have Docker installed in your machine, getting a PostgreSQL instance is as simple as running the following command:

$ docker run -d \
    --name vehicle-quote-postgres \
    -p 5432:5432 \
    --network host \
    -e POSTGRES_DB=vehicle_quote \
    -e POSTGRES_USER=vehicle_quote \
    -e POSTGRES_PASSWORD=password \
    postgres

Here we’re asking Docker to run a new container based on the latest postgres image from DockerHub, name it vehicle-quote-postgres, specify the port to use the default PostgreSQL one, make it accessible to the local network (with the --network host option) and finally, specify a few environment variables that the postgres image uses when building our new instance to set up the default database name, user and password (with the three -e options).

After Docker is done working its magic, you should be able to access the database with something like this:

$ docker exec -it vehicle-quote-postgres psql -U vehicle_quote
psql (13.2 (Debian 13.2-1.pgdg100+1))
Type "help" for help.

vehicle_quote=#

This command is connecting to our new vehicle-quote-postgres container and then, from within the container, using the command line client psql in order to connect to the database.

If you have psql installed on your own machine, you can use it directly to connect to the PostgreSQL instance running inside the container:

$ psql -h localhost -U vehicle_quote

This is possible because we specified in our docker run command that the container would be accepting traffic over port 5432 (-p 5432:5432) and that it would be accesible within the same network as our actual machine (--network host).

Installing the .NET 5 SDK

Ok, with that out of the way, let’s install .NET 5.

.NET 5 truly is multi-platform, so whatever environment you prefer to work with, they’ve got you covered. You can go to the .NET 5 download page and pick your desired flavor of the SDK.

On Ubuntu 20.10, which is what I’m running, installation is painless. It’s your typical process with APT and this page from the official docs has all the details.

First step is to add the Microsoft package repository:

$ wget https://packages.microsoft.com/config/ubuntu/20.10/packages-microsoft-prod.deb -O packages-microsoft-prod.deb

$ sudo dpkg -i packages-microsoft-prod.deb

Then, install .NET 5 with APT like one would any other software package:

$ sudo apt-get update; \
  sudo apt-get install -y apt-transport-https && \
  sudo apt-get update && \
  sudo apt-get install -y dotnet-sdk-5.0

Run dotnet --version in your console and you should see something like this:

$ dotnet --version
5.0.301

Setting up the project

Creating our ASP.NET Core REST API project

Ok now that we have our requirements, database and SDK, let’s start setting up our project. We do so with the following command:

$ dotnet new webapi -o VehicleQuotes

This instructs the dotnet command line tool to create a new REST API web application project for us in a new VehicleQuotes directory.

As a result, dotnet will give you some messages, including this:

The template "ASP.NET Core Web API" was created successfully.

A new directory was created with our web application files. The newly created VehicleQuotes project looks like this:

.
├── appsettings.Development.json
├── appsettings.json
├── Controllers
│   └── WeatherForecastController.cs
├── obj
│   ├── project.assets.json
│   ├── project.nuget.cache
│   ├── VehicleQuotes.csproj.nuget.dgspec.json
│   ├── VehicleQuotes.csproj.nuget.g.props
│   └── VehicleQuotes.csproj.nuget.g.targets
├── Program.cs
├── Properties
│   └── launchSettings.json
├── Startup.cs
├── VehicleQuotes.csproj
└── WeatherForecast.cs

Important things to note here are the appsettings.json and appsettings.Development.json files which contain environment specific configuration values; the Controllers directory where we define our application controllers and action methods (i.e. our REST API endpoints); the Program.cs and Startup.cs files that contain our application’s entry point and bootstrapping logic; and finally VehicleQuotes.csproj which is the file that contains project-wide configuration that the framework cares about like references, compilation targets, and other options. Feel free to explore.

The dotnet new command has given us quite a bit. These files make up a fully working application that we can run and play around with. It even has a Swagger UI, as I’ll demonstrate shortly. It’s a great place to get started from.

You can also get a pretty comprehensive .gitignore file by running the dotnet new gitignore command.

From inside the VehicleQuotes directory, you can run the application with:

$ dotnet run

Which will start up a development server and give out the following output:

$ dotnet run
Building...
info: Microsoft.Hosting.Lifetime[0]
      Now listening on: https://localhost:5001
info: Microsoft.Hosting.Lifetime[0]
      Now listening on: http://localhost:5000
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Development
info: Microsoft.Hosting.Lifetime[0]
      Content root path: /home/kevin/projects/endpoint/blog/VehicleQuotes

Open up a browser window and go to https://localhost:5001/swagger to find a Swagger UI listing our API’s endpoints:

As you can see we’ve got a GET /WeatherForecast endpoint in our app. This is included by default in the webapi project template that we specified in our call to dotnet new. You can see it defined in the Controllers/WeatherForecastController.cs file.

Installing packages we’ll need

Now let’s install all the tools and libraries we will need for our application. First, we install the ASP.NET Code Generator tool which we’ll use later for scaffolding controllers:

$ dotnet tool install --global dotnet-aspnet-codegenerator

We also need to install the Entity Framework command line tools which help us with creating and applying database migrations:

$ dotnet tool install --global dotnet-ef

Now, we need to install a few libraries that we’ll use in our project. First are all the packages that allow us to use Entity Framework Core, provide scaffolding support and give us a detailed debugging page for database errors:

$ dotnet add package Microsoft.VisualStudio.Web.CodeGeneration.Design
$ dotnet add package Microsoft.EntityFrameworkCore.Design
$ dotnet add package Microsoft.EntityFrameworkCore.SqlServer
$ dotnet add package Microsoft.AspNetCore.Diagnostics.EntityFrameworkCore

We also need the EF Core driver for PostgreSQL which will allow us to interact with our database:

$ dotnet add package Npgsql.EntityFrameworkCore.PostgreSQL

Finally, we need another package that will allow us to use the snake case naming convention for our database tables, fields, etc. We need this because EF Core uses capitalized camel case by default, which is not very common in the PostgreSQL world, so this will allow us to play nice. This is the package:

$ dotnet add package EFCore.NamingConventions

Connecting to the database and performing initial app configuration

In order to connect to, query, and modify a database using EF Core, we need to create a DbContext. This is a class that serves as the entry point into the database. Create a new directory called Data in the project root and add this new VehicleQuotesContext.cs file to it:

using Microsoft.EntityFrameworkCore;

namespace VehicleQuotes
{
    public class VehicleQuotesContext : DbContext
    {
        public VehicleQuotesContext (DbContextOptions<VehicleQuotesContext> options)
            : base(options)
        {
        }
    }
}

As you can see this is just a simple class that inherits from EF Core’s DbContext class. That’s all we need for now. We will continue building on this class as we add new tables and configurations.

Now, we need to add this class into ASP.NET Core’s built-in IoC (inversion of control) container so that it’s available to controllers and other classes via Dependency Injection, and tell it how to find our database. Go to Startup.cs and add the following using statement near the top of the file:

using Microsoft.EntityFrameworkCore;

That will allow us to do the following change in the ConfigureServices method:

 public void ConfigureServices(IServiceCollection services)
 {

     // ...

+    services.AddDbContext<VehicleQuotesContext>(options =>
+        options
+            .UseNpgsql(Configuration.GetConnectionString("VehicleQuotesContext"))
+    );
 }

UseNpgsql is an extension method made available to us by the Npgsql.EntityFrameworkCore.PostgreSQL package that we installed in the previous step.

The services variable contains all the objects (known as “services”) that are available in the app for Dependency Injection. So here, we’re adding our newly created DbContext to it, specifying that it will connect to a PostgreSQL database (via the options.UseNpgsql call), and that it will use a connection string named VehicleQuotesContext from the app’s default configuration file. So let’s add the connection string then. To do so, change the appsettings.json like so:

 {
   "Logging": {
     "LogLevel": {
       "Default": "Information",
       "Microsoft": "Warning",
       "Microsoft.Hosting.Lifetime": "Information"
     }
   },
-  "AllowedHosts": "*"
+  "AllowedHosts": "*",
+  "ConnectionStrings": {
+      "VehicleQuotesContext": "Host=localhost;Database=vehicle_quote;Username=vehicle_quote;Password=password"
+  }
 }

This is your typical PostgreSQL connection string. The only gotcha is that it needs to be specified under the ConnectionStrings -> VehicleQuotesContext section so that our call to Configuration.GetConnectionString can find it.

Now let’s put the EFCore.NamingConventions package to good use and configure EF Core to use snake case when naming database objects. Add the following to the ConfigureServices method in Startup.cs:

public void ConfigureServices(IServiceCollection services)
{
    // ...

    services.AddDbContext<VehicleQuotesContext>(options =>
        options
            .UseNpgsql(Configuration.GetConnectionString("VehicleQuotesContext"))
+           .UseSnakeCaseNamingConvention()
    );
}

UseSnakeCaseNamingConvention is an extension method made available to us by the EFCore.NamingConventions package that we installed in the previous step.

Now let’s make logging a little bit more verbose with:

public void ConfigureServices(IServiceCollection services)
{
    // ...

    services.AddDbContext<VehicleQuotesContext>(options =>
        options
            .UseNpgsql(Configuration.GetConnectionString("VehicleQuotesContext"))
            .UseSnakeCaseNamingConvention()
+           .UseLoggerFactory(LoggerFactory.Create(builder => builder.AddConsole()))
+           .EnableSensitiveDataLogging()
    );
}

This will make sure full database queries appear in the log in the console, including parameter values. This could expose sensitive data so be careful when using EnableSensitiveDataLogging in production.

We can also add the following service configuration to have the app display detailed error pages when something related to the database or migrations goes wrong:

public void ConfigureServices(IServiceCollection services)
{
    // ...

+   services.AddDatabaseDeveloperPageExceptionFilter();
}

AddDatabaseDeveloperPageExceptionFilter is an extension method made available to us by the Microsoft.AspNetCore.Diagnostics.EntityFrameworkCore package that we installed in the previous step.

Finally, one last configuration I like to do is have the Swagger UI show up at the root URL, so that instead of using https://localhost:5001/swagger, we’re able to just use https://localhost:5001. We do so by by updating the Configure method this time, in the same Startup.cs file that we’ve been working on:

public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
    if (env.IsDevelopment())
    {
        app.UseDeveloperExceptionPage();
        app.UseSwagger();
-       app.UseSwaggerUI(c => c.SwaggerEndpoint("/swagger/v1/swagger.json", "VehicleQuotes v1"));
+       app.UseSwaggerUI(c => {
+           c.SwaggerEndpoint("/swagger/v1/swagger.json", "VehicleQuotes v1");
+           c.RoutePrefix = "";
+       });
    }

The magic is done by the c.RoutePrefix = ""; line which makes it so there’s no need to put any prefix in order to access the auto generated Swagger UI.

Try it out. Do dotnet run and navigate to https://localhost:5001 and you should see the Swagger UI there.

Building the application

Creating model entities, migrations and updating the database

Alright, with all that configuration out of the way, let’s implement some of our actual application logic now. Refer back to our data model. We’ll start by defining our three simplest tables: makes, sizes and body_types. With EF Core, we define tables via so-called POCO entities, which are simple C# classes with some properties. The classes become tables and the properties become the tables’ fields. Instances of these classes represent records in the database.

So, create a new Models directory in our project’s root and add these three files:

// Models/BodyType.cs
namespace VehicleQuotes.Models
{
    public class BodyType
    {
        public int ID { get; set; }
        public string Name { get; set; }
    }
}

// Models/Make.cs
namespace VehicleQuotes.Models
{
    public class Make
    {
        public int ID { get; set; }
        public string Name { get; set; }
    }
}

// Models/Size.cs
namespace VehicleQuotes.Models
{
    public class Size
    {
        public int ID { get; set; }
        public string Name { get; set; }
    }
}

Now, we add three corresponding DbSets to our DbContext in Data/VehicleQuoteContext.cs. Here’s the diff:

using Microsoft.EntityFrameworkCore;
+using VehicleQuotes.Models;

namespace VehicleQuotes
{
    public class VehicleQuotesContext : DbContext
    {
        public VehicleQuotesContext (DbContextOptions<VehicleQuotesContext> options)
            : base(options)
        {
        }

+       public DbSet<Make> Makes { get; set; }
+       public DbSet<Size> Sizes { get; set; }
+       public DbSet<BodyType> BodyTypes { get; set; }
    }
}

This is how we tell EF Core to build tables in our database for our entities. You’ll see later how we use those DbSets to access the data in those tables. For now, let’s create a migration script that we can later run to apply changes to our database. Run the following to have EF Core create it for us:

$ dotnet ef migrations add AddLookupTables

Now take a look at the newly created Migrations directory. It contains a few new files, but the one we care about right now is Migrations/{TIMESTAMP}_AddLookupTables.cs. In its Up method, it’s got some code that will modify the database structure when run. The EF Core tooling has inspected our project, identified the new entities, and automatically generated a migration script for us that creates tables for them. Notice also how the tables and fields use the snake case naming convention, just as we specified with the call to UseSnakeCaseNamingConvention in Startup.cs.

Now, to actually run the migration script and apply the changes to the database, we do:

$ dotnet ef database update

That command inspects our project to find any migrations that haven’t been run yet, and applies them. In this case, we only have one, so that’s what it runs. Look at the output in the console to see it working its magic step by step:

$ dotnet ef database update
Build started...
Build succeeded.
warn: Microsoft.EntityFrameworkCore.Model.Validation[10400]
      Sensitive data logging is enabled. Log entries and exception messages may include sensitive application data; this mode should only be enabled during development.

...

info: Microsoft.EntityFrameworkCore.Database.Command[20101]
      Executed DbCommand (4ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
      CREATE TABLE sizes (
          id integer GENERATED BY DEFAULT AS IDENTITY,
          name text NULL,
          CONSTRAINT pk_sizes PRIMARY KEY (id)
      );
info: Microsoft.EntityFrameworkCore.Database.Command[20101]
      Executed DbCommand (1ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
      INSERT INTO "__EFMigrationsHistory" (migration_id, product_version)
      VALUES ('20210625212939_AddLookupTables', '5.0.7');
Done.

Notice how it warns us about potential exposure of sensitive data because of that EnableSensitiveDataLogging option we opted into in Startup.cs. Also, EF Core related logs are extra verbose showing all database operations because of another configuration option that we applied there: the UseLoggerFactory(LoggerFactory.Create(builder => builder.AddConsole())) one.

You can connect to the database with the psql command line client and see that the changes took effect:

$ psql -h localhost -U vehicle_quote

...

vehicle_quote=# \c vehicle_quote 
psql (12.7 (Ubuntu 12.7-0ubuntu0.20.10.1), server 13.2 (Debian 13.2-1.pgdg100+1))
You are now connected to database "vehicle_quote" as user "vehicle_quote".
vehicle_quote=# \dt
                   List of relations
 Schema |         Name          | Type  |     Owner     
--------+-----------------------+-------+---------------
 public | __EFMigrationsHistory | table | vehicle_quote
 public | body_types            | table | vehicle_quote
 public | makes                 | table | vehicle_quote
 public | sizes                 | table | vehicle_quote
(4 rows)

vehicle_quote=# \d makes
                            Table "public.makes"
 Column |  Type   | Collation | Nullable |             Default              
--------+---------+-----------+----------+----------------------------------
 id     | integer |           | not null | generated by default as identity
 name   | text    |           |          | 
Indexes:
    "pk_makes" PRIMARY KEY, btree (id)

There are our tables in all their normalized, snake-cased glory. The __EFMigrationsHistory table is used internally by EF Core to keep track of which migrations have been applied.

Creating controllers for CRUDing our tables

Now that we have that, let’s add a few endpoints to support basic CRUD of those tables. We can use the dotnet-aspnet-codegenerator scaffolding tool that we installed earlier. For the three tables that we have, we would do:

$ dotnet aspnet-codegenerator controller \
    -name MakesController \
    -m Make \
    -dc VehicleQuotesContext \
    -async \
    -api \
    -outDir Controllers

$ dotnet aspnet-codegenerator controller \
    -name BodyTypesController \
    -m BodyType \
    -dc VehicleQuotesContext \
    -async \
    -api \
    -outDir Controllers

$ dotnet aspnet-codegenerator controller \
    -name SizesController \
    -m Size \
    -dc VehicleQuotesContext \
    -async \
    -api \
    -outDir Controllers

Those commands tell the scaffolding tool to create new controllers that:

Are named as given by the -name option.
Use the model class specified in the -m option.
Use our VehicleQuotesContext to talk to the database. As per the -dc option.
Define the methods using async/await syntax. Given by the -async option.
Are API controllers. Specified by the -api option.
Are created in the Controllers directory. Via the -outDir option.

Explore the new files that got created in the Controllers directory: MakesController.cs, BodyTypesController.cd and SizesController.cs. The controllers have been generated with the necessary Action Methods to fetch, create, update and delete their corresponding entities. Try dotnet run and navigate to https://localhost:5001 to see the new endpoints in the Swagger UI:

Try it out! You can interact with each of the endpoints from the Swagger UI and it all works as you’d expect.

Adding unique constraints via indexes

Ok, our app is coming along well. Right now though, there’s an issue with the tables that we’ve created. It’s possible to create vehicle makes with the same name. The same is true for body types and sizes. This doesn’t make much sense for these tables. So let’s fix that by adding a uniqueness constraint. We can do it by creating a unique database index using EF Core’s Index attribute. For example, we can modify our Models/Make.cs like so:

+using Microsoft.EntityFrameworkCore;

namespace VehicleQuotes.Models
{
+   [Index(nameof(Name), IsUnique = true)]
    public class Make
    {
        public int ID { get; set; }
        public string Name { get; set; }
    }
}

In fact, do the same for our other entities in Models/BodyType.cs and Models/Size.cs. Don’t forget the using Microsoft.EntityFrameworkCore statement.

With that, we can create a new migration:

$ dotnet ef migrations add AddUniqueIndexesToLookupTables

That will result in a new migration script in Migrations/{TIMESTAMP}_AddUniqueIndexesToLookupTables.cs. Its Up method looks like this:

protected override void Up(MigrationBuilder migrationBuilder)
{
    migrationBuilder.CreateIndex(
        name: "ix_sizes_name",
        table: "sizes",
        column: "name",
        unique: true);

    migrationBuilder.CreateIndex(
        name: "ix_makes_name",
        table: "makes",
        column: "name",
        unique: true);

    migrationBuilder.CreateIndex(
        name: "ix_body_types_name",
        table: "body_types",
        column: "name",
        unique: true);
}

As you can see, new unique indexes are being created on the tables and fields that we specified. Like before, apply the changes to the database structure with:

$ dotnet ef database update

Now if you try to create, for example, a vehicle make with a repeated name, you’ll get an error. Try doing so by POSTing to /api/Makes via the Swagger UI:

Responding with specific HTTP error codes (409 Conflict)

The fact that we can now enforce unique constraints is all well and good. But the error scenario is not very user friendly. Instead of returning a “500 Internal Server Error” status code with a wall of text, we should be responding with something more sensible. Maybe a “409 Conflict” would be more appropriate for this kind of error. We can easily update our controllers to handle that scenario. What we need to do is update the methods that handle the POST and PUT endpoints so that they catch the Microsoft.EntityFrameworkCore.DbUpdateException exception and return the proper response. Here’s how we would do it for the MakesController:

// ...
namespace VehicleQuotes.Controllers
{
    [Route("api/[controller]")]
    [ApiController]
    public class MakesController : ControllerBase
    {
        // ...

        [HttpPut("{id}")]
        public async Task<IActionResult> PutMake(int id, Make make)
        {
            // ...

            try
            {
                await _context.SaveChangesAsync();
            }
            // ...
+           catch (Microsoft.EntityFrameworkCore.DbUpdateException)
+           {
+               return Conflict();
+           }

            return NoContent();
        }

        [HttpPost]
        public async Task<ActionResult<Make>> PostMake(Make make)
        {
            _context.Makes.Add(make);
-           await _context.SaveChangesAsync();

+           try
+           {
+               await _context.SaveChangesAsync();
+           }
+           catch (Microsoft.EntityFrameworkCore.DbUpdateException)
+           {
+               return Conflict();
+           }

            return CreatedAtAction("GetMake", new { id = make.ID }, make);
        }

        // ...
    }
}

Go ahead and do the same for the other two controllers, and try again to POST a repeated make name via the Swagger UI. You should see this now instead:

Much better now, don’t you think?

Adding a more complex entity to the model

Now let’s work on an entity that’s a little bit more complex: the one we will use to represent vehicle models.

For this entity, we don’t want our API to be as low level as the one for the other three, where it was basically a thin wrapper over database tables. We want it to be a little bit more abstract and not expose the entire database structure verbatim.

Refer back to the data model. We’ll add models, model_styles and model_style_years. Let’s start by adding the following classes:

// Models/Model.cs
using System.Collections.Generic;

namespace VehicleQuotes.Models
{
    public class Model
    {
        public int ID { get; set; }
        public string Name { get; set; }
        public int MakeID { get; set; }

        public Make Make { get; set; }

        public ICollection<ModelStyle> ModelStyles { get; set; }
    }
}

// Models/ModelStyle.cs
using System.Collections.Generic;

namespace VehicleQuotes.Models
{
    public class ModelStyle
    {
        public int ID { get; set; }
        public int ModelID { get; set; }
        public int BodyTypeID { get; set; }
        public int SizeID { get; set; }

        public Model Model { get; set; }
        public BodyType BodyType { get; set; }
        public Size Size { get; set; }

        public ICollection<ModelStyleYear> ModelStyleYears { get; set; }
    }
}

// Models/ModelStyleYear.cs
namespace VehicleQuotes.Models
{
    public class ModelStyleYear
    {
        public int ID { get; set; }
        public string Year { get; set; }
        public int ModelStyleID { get; set; }

        public ModelStyle ModelStyle { get; set; }
    }
}

Notice how some of these entities now include properties whose types are other entities. Some of them are collections even. These are called Navigation Properties and are how we tell EF Core that our entities are related to one another. These will result in foreign keys being created in the database.

Take the Model entity for example. It has a property Make of type Make. It also has a MakeID property of type int. EF Core sees this and figures out that there’s a relation between the makes and models tables. Specifically, that models have a make. A many-to-one relation where the models table stores a foreign key to the makes table.

Similarly, the Model entity has a ModelStyles property of type ICollection<ModelStyleYear>. This tells EF Core that models have many model_styles. This one is a one-to-many relation from the perspective of the models table. The foreign key lives in the model_styles table and points back to models.

The official documentation is a great resource to learn more details about how relationships work in EF Core.

After that, same as before, we have to add the corresponding DbSets to our DbContext:

// Data/VehicleQuotesContext.cs
// ...

namespace VehicleQuotes
{
    public class VehicleQuotesContext : DbContext
    {
        // ...

+       public DbSet<Model> Models { get; set; }
+       public DbSet<ModelStyle> ModelStyles { get; set; }
+       public DbSet<ModelStyleYear> ModelStyleYears { get; set; }
    }
}

Don’t forget the migration script. First create it:

$ dotnet ef migrations add AddVehicleModelTables

And then apply it:

$ dotnet ef database update

Adding composite unique indexes

These vehicle model related tables also need some uniqueness enforcement. This time, however, the unique keys are composite. Meaning that they involve multiple fields. For vehicle models, for example, it makes no sense to have multiple records with the same make and name. But it does make sense to have multiple models with the same name, as long as they belong to different makes. We can solve for that with a composite index. Here’s how we create one of those in Model with EF Core:

using System.Collections.Generic;
+using Microsoft.EntityFrameworkCore;

namespace VehicleQuotes.Models
{
+   [Index(nameof(Name), nameof(MakeID), IsUnique = true)]
    public class Model
    {
        // ...
    }
}

Very similar to what we did with the Make, BodyType, and Size entities. The only difference is that this time we included multiple fields in the parameters for the Index attribute.

We should do the same for ModelStyle and ModelStyleYear:

using System.Collections.Generic;
+using Microsoft.EntityFrameworkCore;

namespace VehicleQuotes.Models
{
+   [Index(nameof(ModelID), nameof(BodyTypeID), nameof(SizeID), IsUnique = true)]
    public class ModelStyle
    {
        // ...
    }
}

+using Microsoft.EntityFrameworkCore;

namespace VehicleQuotes.Models
{
+   [Index(nameof(Year), nameof(ModelStyleID), IsUnique = true)]
    public class ModelStyleYear
    {
        // ...
    }
}

Don’t forget the migrations:

$ dotnet ef migrations add AddUniqueIndexesForVehicleModelTables

$ dotnet ef database update

Adding controllers with custom routes

Our data model dictates that vehicle models belong in a make. In other words, a vehicle model has no meaning by itself. It only has meaning within the context of a make. Ideally, we want our API routes to reflect this concept. In other words, instead of URLs for models to look like this: /api/Models/{id}; we’d rather them look like this: /api/Makes/{makeId}/Models/{modelId}. Let’s go ahead and scaffold a controller for this entity:

$ dotnet aspnet-codegenerator controller \
    -name ModelsController \
    -m Model \
    -dc VehicleQuotesContext \
    -async \
    -api \
    -outDir Controllers

Now let’s change the resulting Controllers/ModelsController.cs to use the URL structure that we want. To do so, we modify the Route attribute that’s applied to the ModelsController class to this:

[Route("api/Makes/{makeId}/[controller]/")]

Do a dotnet run and take a peek at the Swagger UI on https://localhost:5001 to see what the Models endpoint routes look like now:

The vehicle model routes are now nested within makes, just like we wanted.

Of course, this is just eye candy for now. We need to actually use this new makeId parameter for the logic in the endpoints. For example, one would expect a GET to /api/Makes/1/Models to return all the vehicle models that belong to the make with id 1. But right now, all vehicle models are returned regardless. All other endpoints behave similarly, there’s no limit to the operations on the vehicle models. The given makeId is not taken into consideration at all.

Let’s update the ModelsController’s GetModels method (which is the one that handles the GET /api/Makes/{makeId}/Models endpoint) to behave like one would expect. It should look like this:

[HttpGet]
public async Task<ActionResult<IEnumerable<Model>>> GetModels([FromRoute] int makeId)
{
    var make = await _context.Makes.FindAsync(makeId);

    if (make == null)
    {
        return NotFound();
    }

    return await _context.Models.Where(m => m.MakeID == makeId).ToListAsync();
}

See how we’ve included a new parameter to the method: [FromRoute] int makeId. This [FromRoute] attribute is how we tell ASP.NET Core that this endpoint will use that makeId parameter coming from the URL route. Then, we use our DbContext to try to find the make that corresponds to the given identifier. This is done in _context.Makes.FindAsync(makeId). Then, if we can’t find the given make, we return a 404 Not Found HTTP status code with the return NotFound(); line. Finally, we query the models table for all the records whose make_id matches the given parameter. That’s done in the last line of the method.

We have access to the DbContext because it has been injected as a dependency into the controller via its constructor by the framework.

The official documentation is a great resource to learn about all the possibilities when querying data with EF Core.

Let’s update the GetModel method, which handles the GET /api/Makes/{makeId}/Models/{id} endpoint, similarly.

[HttpGet("{id}")]
-public async Task<ActionResult<Model>> GetModel(int id)
+public async Task<ActionResult<Model>> GetModel([FromRoute] int makeId, int id)
{
-   var model = await _context.Models.FindAsync(id);
+   var model = await _context.Models.FirstOrDefaultAsync(m =>
+       m.MakeID == makeId && m.ID == id
+   );

    if (model == null)
    {
        return NotFound();
    }

    return model;
}

We’ve once again included the makeId as a parameter to the method and modified the EF Core query to use both the make ID and the vehicle model ID when looking for the record.

And that’s the gist of it. Other methods would need to be updated similarly. The next section will include these methods in their final form, so I won’t go through each one of them here.

Using resource models as DTOs for controllers

Now, I did say at the beginning that we wanted the vehicle model endpoint to be a bit more abstract. Right now it’s operating directly over the EF Core entities and our table. As a result, creating new vehicle models via the POST /api/Makes/{makeId}/Models endpoint is a pain. Take a look at the Swagger UI request schema for that endpoint:

This is way too much. Let’s make it a little bit more user friendly by making it more abstract.

To do that, we will introduce what I like to call a Resource Model (or DTO, Data Transfer Object, or View Model). This is a class whose only purpose is to streamline the API contract of the endpoint by defining a set of fields that clients will use to make requests and interpret responses. Something that’s simpler than our actual database structure, but still captures all the information that’s important for our application. We will update the ModelsController so that it’s able to receive objects of this new class as requests, operate on them, translate them to our EF Core entities and actual database records, and return them as a response. The hope is that, by hiding the details of our database structure, we make it easier for clients to interact with our API.

So let’s create a new ResourceModels directory in our project’s root and add these two classes:

// ResourceModels/ModelSpecification.cs
namespace VehicleQuotes.ResourceModels
{
    public class ModelSpecification
    {
        public int ID { get; set; }
        public string Name { get; set; }

        public ModelSpecificationStyle[] Styles { get; set; }
    }
}

// ResourceModels/ModelSpecificationStyle.cs
namespace VehicleQuotes.ResourceModels
{
    public class ModelSpecificationStyle
    {
        public string BodyType { get; set; }
        public string Size { get; set; }

        public string[] Years { get; set; }
    }
}

Thanks to these two, instead of that mess from above, clients POSTing to /api/Makes/{makeId}/Models will be able to use a request body like this:

{
  "name": "string",
  "styles": [
    {
      "bodyType": "string",
      "size": "string",
      "years": [
        "string"
      ]
    }
  ]
}

Which is much simpler. We have the vehicle model name and an array of styles. Each style has a body type and a size, which we can specify by their names because those are unique keys. We don’t need their integer IDs (i.e. primary keys) in order to find to them. Then, each style has an array of strings that contain the years in which those styles are available for that model. The make is part of the URL already, so we don’t need to also specify it in the request payload.

Let’s update our ModelsController to use these Resource Models instead of the Model EF Core entity. Be sure to include the namespace where the Resource Models are defined by adding the following using statement: using VehicleQuotes.ResourceModels;. Now, let’s update the GetModels method (which handles the GET /api/Makes/{makeId}/Models endpoint) so that it looks like this:

[HttpGet]
// Return a collection of `ModelSpecification`s and expect a `makeId` from the URL.
public async Task<ActionResult<IEnumerable<ModelSpecification>>> GetModels([FromRoute] int makeId)
{
    // Look for the make identified by `makeId`.
    var make = await _context.Makes.FindAsync(makeId);

    // If we can't find the make, then we return a 404.
    if (make == null)
    {
        return NotFound();
    }

    // Build a query to fetch the relevant records from the `models` table and
    // build `ModelSpecification` with the data.
    var modelsToReturn = _context.Models
        .Where(m => m.MakeID == makeId)
        .Select(m => new ModelSpecification {
            ID = m.ID,
            Name = m.Name,
            Styles = m.ModelStyles.Select(ms => new ModelSpecificationStyle {
                BodyType = ms.BodyType.Name,
                Size = ms.Size.Name,
                Years = ms.ModelStyleYears.Select(msy => msy.Year).ToArray()
            }).ToArray()
        });

    // Execute the query and respond with the results.
    return await modelsToReturn.ToListAsync();
}

The first thing that we changed was the return type. Instead of Task<ActionResult<IEnumerable<Model>>>, the method now returns Task<ActionResult<IEnumerable<ModelSpecification>>>. We’re going to use our new Resource Models as these endpoints’ contract, so we need to make sure we are returning those. Next, we considerably changed the LINQ expression that searches the database for the vehicle model records we want. The filtering logic (given by the Where) is the same. That is, we’re still searching for vehicle models within the given make ID. What we changed was the projection logic in the Select. Our Action Method now returns a collection of ModelSpecification objects, so we updated the Select to produce such objects, based on the records from the models table that match our search criteria. We build ModelSpecifications using the data coming from models records and their related model_styles and model_style_years. Finally, we asynchronously execute the query to fetch the data from the database and return it.

Next, let’s move on to the GetModel method, which handles the GET /api/Makes/{makeId}/Models/{id} endpoint. This is what it should look like:

[HttpGet("{id}")]
// Return a `ModelSpecification`s and expect `makeId` and `id` from the URL.
public async Task<ActionResult<ModelSpecification>> GetModel([FromRoute] int makeId, [FromRoute] int id)
{
    // Look for the model specified by the given identifiers and also load
    // all related data that we care about for this method.
    var model = await _context.Models
        .Include(m => m.ModelStyles).ThenInclude(ms => ms.BodyType)
        .Include(m => m.ModelStyles).ThenInclude(ms => ms.Size)
        .Include(m => m.ModelStyles).ThenInclude(ms => ms.ModelStyleYears)
        .FirstOrDefaultAsync(m => m.MakeID == makeId && m.ID == id);

    // If we couldn't find it, respond with a 404.
    if (model == null)
    {
        return NotFound();
    }

    // Use the fetched data to construct a `ModelSpecification` to use in the response.
    return new ModelSpecification {
        ID = model.ID,
        Name = model.Name,
        Styles = model.ModelStyles.Select(ms => new ModelSpecificationStyle {
            BodyType = ms.BodyType.Name,
            Size = ms.Size.Name,
            Years = ms.ModelStyleYears.Select(msy => msy.Year).ToArray()
        }).ToArray()
    };
}

Same as before, we changed the return type of the method to be ModelSpecification. Then, we modified the query so that it loads all the related data for the Model entity via its navigation properties. That’s what the Include and ThenInclude calls do. We need this data loaded because we use it in the method’s return statement to build the ModelSpecification that will be included in the response. The logic to build it is very similar to that of the previous method.

You can learn more about the various available approaches for loading data with EF Core in the official documentation.

Next is the PUT /api/Makes/{makeId}/Models/{id} endpoint, handled by the PutModel method:

[HttpPut("{id}")]
// Expect `makeId` and `id` from the URL and a `ModelSpecification` from the request payload.
public async Task<IActionResult> PutModel([FromRoute] int makeId, int id, ModelSpecification model)
{
    // If the id in the URL and the request payload are different, return a 400.
    if (id != model.ID)
    {
        return BadRequest();
    }

    // Obtain the `models` record that we want to update. Include any related
    // data that we want to update as well.
    var modelToUpdate = await _context.Models
        .Include(m => m.ModelStyles)
        .FirstOrDefaultAsync(m => m.MakeID == makeId && m.ID == id);

    // If we can't find the record, then return a 404.
    if (modelToUpdate == null)
    {
        return NotFound();
    }

    // Update the record with what came in the request payload.
    modelToUpdate.Name = model.Name;

    // Build EF Core entities based on the incoming Resource Model object.
    modelToUpdate.ModelStyles = model.Styles.Select(style => new ModelStyle {
        BodyType = _context.BodyTypes.Single(bodyType => bodyType.Name == style.BodyType),
        Size = _context.Sizes.Single(size => size.Name == style.Size),

        ModelStyleYears = style.Years.Select(year => new ModelStyleYear {
            Year = year
        }).ToList()
    }).ToList();

    try
    {
        // Try saving the changes. This will run the UPDATE statement in the database.
        await _context.SaveChangesAsync();
    }
    catch (Microsoft.EntityFrameworkCore.DbUpdateException)
    {
        // If there's an error updating, respond accordingly.
        return Conflict();
    }

    // Finally return a 204 if everything went well.
    return NoContent();
}

The purpose of this endpoint is to update existing resources. So, it receives a representation of said resource as a parameter that comes from the request body. Before, it expected an instance of the Model entity, but now, we’ve changed it to receive a ModelSpecification. The rest of the method is your usual structure of first obtaining the record to update by the given IDs, then changing its values according to what came in as a parameter, and finally, saving the changes.

You probably get the idea by now: since the API is using the Resource Model, we need to change input and output values for the methods and run some logic to translate between Resource Model objects and Data Model objects that EF Core can understand so that it can perform its database operations.

That said, here’s what the PostModel Action Method, handler of the POST /api/Makes/{makeId}/Models endpoint, should look like:

[HttpPost]
// Return a `ModelSpecification`s and expect `makeId` from the URL and a `ModelSpecification` from the request payload.
public async Task<ActionResult<ModelSpecification>> PostModel([FromRoute] int makeId, ModelSpecification model)
{
    // First, try to find the make specified by the incoming `makeId`.
    var make = await _context.Makes.FindAsync(makeId);

    // Respond with 404 if not found.
    if (make == null)
    {
        return NotFound();
    }

    // Build out a new `Model` entity, complete with all related data, based on
    // the `ModelSpecification` parameter.
    var modelToCreate = new Model {
        Make = make,
        Name = model.Name,

        ModelStyles = model.Styles.Select(style => new ModelStyle {
            // Notice how we search both body type and size by their name field.
            // We can do that because their names are unique.
            BodyType = _context.BodyTypes.Single(bodyType => bodyType.Name == style.BodyType),
            Size = _context.Sizes.Single(size => size.Name == style.Size),

            ModelStyleYears = style.Years.Select(year => new ModelStyleYear {
                Year = year
            }).ToArray()
        }).ToArray()
    };

    // Add it to the DbContext.
    _context.Add(modelToCreate);

    try
    {
        // Try running the INSERTs.
        await _context.SaveChangesAsync();
    }
    catch (Microsoft.EntityFrameworkCore.DbUpdateException)
    {
        // Return accordingly if an error happens.
        return Conflict();
    }

    // Get back the autogenerated ID of the record we just INSERTed.
    model.ID = modelToCreate.ID;

    // Finally, return a 201 including a location header containing the newly
    // created resource's URL and the resource itself in the response payload.
    return CreatedAtAction(
        nameof(GetModel),
        new { makeId = makeId, id = model.ID },
        model
    );
}

All that should be pretty self explanatory by now. Moving on to the DeleteModel method which handles the DELETE /api/Makes/{makeId}/Models/{id} endpoint:

[HttpDelete("{id}")]
// Expect `makeId` and `id` from the URL.
public async Task<IActionResult> DeleteModel([FromRoute] int makeId, int id)
{
    // Try to find the record identified by the ids from the URL.
    var model = await _context.Models.FirstOrDefaultAsync(m => m.MakeID == makeId && m.ID == id);

    // Respond with a 404 if we can't find it.
    if (model == null)
    {
        return NotFound();
    }

    // Mark the entity for removal and run the DELETE.
    _context.Models.Remove(model);
    await _context.SaveChangesAsync();

    // Respond with a 204.
    return NoContent();
}

And that’s all for that controller. Hopefully that demonstrated what it looks like to have endpoints that operate using objects other than the EF Core entities. Fire up the app with dotnet run and explore the Swagger UI and you’ll see the changes that we’ve made reflected in there. Try it out. Try CRUDing some vehicle models. And don’t forget to take a look at our POST endpoint specification which looks much more manageable now:

Which means that you can send in something like this, for example:

{
    "name": "Corolla",
    "styles": [
        {
            "bodyType": "Sedan",
            "size": "Compact",
            "years": [ "2000", "2001" ]
        }
    ]
}

This will work assuming you’ve created at least one make to add the vehicle model to, as well as a body type whose name is Sedan and a size whose name is Compact.

There’s also a ModelExists method in that controller which we don’t need anymore. You can delete it.

Validation using built-in Data Annotations

Depending on how “creative” you were in the previous section when trying to CRUD models, you may have run into an issue or two regarding the data that’s allowed into our database. We solve that by implementing input validation. In ASP.NET Core, the easiest way to implement validation is via Data Annotation attributes on the entities or other objects that controllers receive as request payloads. So let’s see about adding some validation to our app. Since our ModelsController uses the ModelSpecification and ModelSpecificationStyle Resource Models to talk to clients, let’s start there. Here’s the diff:

+using System.ComponentModel.DataAnnotations;

namespace VehicleQuotes.ResourceModels
{
    public class ModelSpecification
    {
        public int ID { get; set; }
+       [Required]
        public string Name { get; set; }

+       [Required]
        public ModelSpecificationStyle[] Styles { get; set; }
    }
}

+using System.ComponentModel.DataAnnotations;

namespace VehicleQuotes.ResourceModels
{
    public class ModelSpecificationStyle
    {
+       [Required]
        public string BodyType { get; set; }
+       [Required]
        public string Size { get; set; }

+       [Required]
+       [MinLength(1)]
        public string[] Years { get; set; }
    }
}

And just like that, we get a good amount of functionality. We use the Required and MinLength attributes from the System.ComponentModel.DataAnnotations namespace to specify that some fields are required, and that our Years array needs to contain at least one element. When the app receives a request to the PUT or POST endpoints — which are the ones that expect a ModelSpecification as the payload — validation kicks in. If it fails, the action method is never executed and a 400 status code is returned as a response. Try POSTing to /api/Makes/{makeId}/Models with a payload that violates some of these rules to see for yourself. I tried for example sending this:

{
  "name": null,
  "styles": [
    {
      "bodyType": "Sedan",
      "size": "Full size",
      "years": []
    }
  ]
}

And I got back a 400 response with this payload:

{
  "type": "https://tools.ietf.org/html/rfc7231#section-6.5.1",
  "title": "One or more validation errors occurred.",
  "status": 400,
  "traceId": "00-0fd4f00eeb9f2f458ccefc180fcfba1c-79a618f13218394b-00",
  "errors": {
    "Name": [
      "The Name field is required."
    ],
    "Styles[0].Years": [
      "The field Years must be a string or array type with a minimum length of '1'."
    ]
  }
}

Pretty neat, huh? With minimal effort, we have some basic validation rules in place and a pretty usable response for when errors occur.

To learn more about model validation, including all the various validation attributes included in the framework, check the official documentation: Model validation and System.ComponentModel.DataAnnotations Namespace.

Validation using custom attributes

Of course, the framework is never going to cover all possible validation scenarios with the built-in attributes. Case in point, it’d be great to validate that the Years array contains values that look like actual years. That is, four-character, digit-only strings. There are no validation attributes for that. So, we need to create our own. Let’s add this file into a new Validations directory:

// Validations/ContainsYearsAttribute.cs
using System;
using System.ComponentModel.DataAnnotations;
using System.Linq;
using System.Runtime.CompilerServices;

namespace VehicleQuotes.Validation
{
    // In the .NET Framework, attribute classes need to have their name suffixed with the word "Attribute".
    // Validation attributes need to inherit from `System.ComponentModel.DataAnnotations`'s `ValidationAttribute` class
    // and override the `IsValid` method.
    public class ContainsYearsAttribute : ValidationAttribute
    {
        private string propertyName;

        // This constructor is called by the framework when the attribute is applied to some member. In this specific
        // case, we define a `propertyName` parameter annotated with a `CallerMemberName` attribute. This makes it so
        // the framework sends in the name of the member to which our `ContainsYears` attribute is applied to.
        // We store the value to use it later when constructing our validation error message.
        // Check https://docs.microsoft.com/en-us/dotnet/api/system.runtime.compilerservices.callermembernameattribute?view=net-5.0
        // for more info on `CallerMemberName`.
        public ContainsYearsAttribute([CallerMemberName] string propertyName = null)
        {
            this.propertyName = propertyName;
        }

        // This method is called by the framework during validation. `value` is the actual value of the field that this
        // attribute will validate.
        protected override ValidationResult IsValid(object value, ValidationContext validationContext)
        {
            // By only applying the validation checks when the value is not null, we make it possible for this
            // attribute to work on optional fields. In other words, this attribute will skip validation if there is no
            // value to validate.
            if (value != null)
            {
                // Check if all the elements of the string array are valid years. Check the `IsValidYear` method below
                // to see what checks are applied for each of the array elements.
                var isValid = (value as string[]).All(IsValidYear);

                if (!isValid)
                {
                    // If not, return an error.
                    return new ValidationResult(GetErrorMessage());
                }
            }

            // Return a successful validation result if no errors were detected.
            return ValidationResult.Success;
        }

        // Determines if a given value is valid by making sure it's not null, nor empty, that its length is 4 and that
        // all its characters are digits.
        private bool IsValidYear(string value) =>
            !String.IsNullOrEmpty(value) && value.Length == 4 && value.All(Char.IsDigit);

        // Builds a user friendly error message which includes the name of the field that this validation attribute has
        // been applied to.
        private string GetErrorMessage() =>
            $"The {propertyName} field must be an array of strings containing four digits.";
    }
}

Check the comments in the code for more details into how that class works. Then, we apply our custom attribute to our ModelSpecificationStyle class in the same way that we applied the built in ones:

using System.ComponentModel.DataAnnotations;
+using VehicleQuotes.Validation;

namespace VehicleQuotes.ResourceModels
{
    public class ModelSpecificationStyle
    {
        // ...

        [Required]
        [MinLength(1)]
+       [ContainsYears]
        public string[] Years { get; set; }
    }
}

Now do a dotnet run and try to POST to /api/Makes/{makeId}/Models this payload:

{
  "name": "Rav4",
  "styles": [
    {
      "bodyType": "SUV",
      "size": "Mid size",
      "years": [ "not_a_year" ]
    }
  ]
}

That should make the API respond with this:

{
  "type": "https://tools.ietf.org/html/rfc7231#section-6.5.1",
  "title": "One or more validation errors occurred.",
  "status": 400,
  "traceId": "00-9980325f3e388f48a5975ef382d5b137-2d55da1bb9613e4f-00",
  "errors": {
    "Styles[0].Years": [
      "The Years field must be an array of strings containing four numbers."
    ]
  }
}

That’s our custom validation attribute doing its job.

There’s another aspect that we could validate using a custom validation attribute. What happens if we try to POST a payload with a body type or size that doesn’t exist? These queries from the PostModel method would throw an InvalidOperationException:

BodyType = _context.BodyTypes.Single(bodyType => bodyType.Name == style.BodyType)

and

Size = _context.Sizes.Single(size => size.Name == style.Size)

They do so because we used the Single method, which is designed like that. It tries to find a body type or size whose name is the given value, can’t find it, and thus, throws an exception.

If, for example, we wanted not-founds to return null, we could have used SingleOrDefault instead.

This unhandled exception results in a response that’s quite unbecoming:

So, to prevent that exception and control the error messaging, we need a couple of new validation attributes that go into the body_types and sizes tables and check if the given values exist. Here’s what one would look like:

// Validations/VehicleBodyTypeAttribute.cs
using System;
using System.ComponentModel.DataAnnotations;
using Microsoft.EntityFrameworkCore;
using System.Linq;

namespace VehicleQuotes.Validation
{
    public class VehicleBodyTypeAttribute : ValidationAttribute
    {
        protected override ValidationResult IsValid(object value, ValidationContext validationContext)
        {
            if (value == null) return ValidationResult.Success;

            var dbContext = validationContext.GetService(typeof(VehicleQuotesContext)) as VehicleQuotesContext;

            var bodyTypes = dbContext.BodyTypes.Select(bt => bt.Name).ToList();

            if (!bodyTypes.Contains(value))
            {
                var allowed = String.Join(", ", bodyTypes);
                return new ValidationResult(
                    $"Invalid vehicle body type {value}. Allowed values are {allowed}."
                );
            }

            return ValidationResult.Success;
        }
    }
}

You can find the other one here: Validations/VehicleSizeAttribute.cs.

These two are very similar to one another. The most interesting part is how we use the IsValid method’s second parameter (ValidationContext) to obtain an instance of VehicleQuotesContext that we can use to query the database. The rest should be pretty self-explanatory. These attributes are classes that inherit from System.ComponentModel.DataAnnotations’s ValidationAttribute and implement the IsValid method. The method then checks that the value under scrutiny exists in the corresponding table and if it does not, raises a validation error. The validation error includes a list of all allowed values. They can be applied to our ModelSpecificationStyle class like so:

// ...
namespace VehicleQuotes.ResourceModels
{
    public class ModelSpecificationStyle
    {
        [Required]
+       [VehicleBodyType]
        public string BodyType { get; set; }

        [Required]
+       [VehicleSize]
        public string Size { get; set; }

        //...
    }
}

Now, a request like this:

{
  "name": "Rav4",
  "styles": [
    {
      "bodyType": "not_a_body_type",
      "size": "Mid size",
      "years": [ "2000" ]
    }
  ]
}

Produces a response like this:

{
  "type": "https://tools.ietf.org/html/rfc7231#section-6.5.1",
  "title": "One or more validation errors occurred.",
  "status": 400,
  "traceId": "00-9ad59a7aff60944ab54c19a73be73cc7-eeabafe03df74e40-00",
  "errors": {
    "Styles[0].BodyType": [
      "Invalid vehicle body type not_a_body_type. Allowed values are Coupe, Sedan, Convertible, Hatchback, SUV, Truck."
    ]
  }
}

Implementing endpoints for quote rules and overrides

At this point we’ve explored many of the most common features available to us for developing Web APIs. So much so that implementing the next two pieces of functionality for our app doesn’t really introduce any new concepts. So, I wont discuss that here in great detail.

Feel free to browse the source code on GitHub if you want though. These are the relevant files:

The FeatureTypeAttribute class is interesting in that it provides another example of a validation attribute. This time is one that makes sure the value being validated is included in an array of strings that’s defined literally in the code.

Other than that, it’s all stuff we’ve already covered: models, migrations, scaffolding controllers, custom routes, resource models, etc.

If you are following along, be sure to add those files and run a dotnet ef database update to apply the migration.

Implementing the quote model

Let’s now start implementing the main capability of our app: calculating quotes for vehicles. Let’s start with the Quote entity. This is what the new Models/Quote.cs file containing the entity class will look like:

// Models/Quote.cs
using System;
using System.ComponentModel.DataAnnotations.Schema;

namespace VehicleQuotes.Models
{
    public class Quote
    {
        public int ID { get; set; }

        // Directly tie this quote record to a specific vehicle that we have
        // registered in our db, if we have it.
        public int? ModelStyleYearID { get; set; }

        // If we don't have the specific vehicle in our db, then store the
        // vehicle model details independently.
        public string Year { get; set; }
        public string Make { get; set; }
        public string Model { get; set; }
        public int BodyTypeID { get; set; }
        public int SizeID { get; set; }

        public bool ItMoves { get; set; }
        public bool HasAllWheels { get; set; }
        public bool HasAlloyWheels { get; set; }
        public bool HasAllTires { get; set; }
        public bool HasKey { get; set; }
        public bool HasTitle { get; set; }
        public bool RequiresPickup { get; set; }
        public bool HasEngine { get; set; }
        public bool HasTransmission { get; set; }
        public bool HasCompleteInterior { get; set; }

        public int OfferedQuote { get; set; }
        public string Message { get; set; }
        public DateTime CreatedAt { get; set; }

        public ModelStyleYear ModelStyleYear { get; set; }

        public BodyType BodyType { get; set; }
        public Size Size { get; set; }
    }
}

This should be pretty familiar by now. It’s a plain old class that defines a number of properties, one for each of the fields in the resulting table and a few navigation properties that serve to access related data.

The only aspect worth noting is that we’ve defined the ModelStyleYearID property as a nullable integer (with int?). This is because, like we discussed at the beginning, the foreign key from quotes to vehicle_style_years is actually optional. The reason being that we may receive a quote request for a vehicle that we don’t have registered in our database. We need to be able to support quoting those vehicles too, so if we don’t have the requested vehicle registered, then that foreign key will stay unpopulated and we’ll rely on the other fields (i.e. Year, Make, Model, BodyTypeID and SizeID) to identify the vehicle and calculate the quote for it.

Using Dependency Injection

So far we’ve been putting a lot of logic in our controllers. That’s generally not ideal, but fine as long as the logic is simple. The problem is that a design like that can quickly become a hindrance for maintainability and testing as our application grows more complex. For the logic that calculates a quote, we’d be better served by implementing it in its own class, outside of the controller that should only care about defining endpoints and handling HTTP concerns. Then, the controller can be given access to that class and delegate to it all the quote calculation logic. Thankfully, ASP.NET Core includes an IoC container by default, which allows us to use Dependency Injection to solve these kinds of problems. Let’s see what that looks like.

For working with quotes, we want to offer two endpoints:

A POST api/Quotes that captures the vehicle information, calculates the quote, keeps record of the request, and responds with the calculated value.
A GET api/Quotes that returns all the currently registered quotes in the system.

Using the Dependency Injection capabilities, a controller that implements those two could look like this:

using System.Collections.Generic;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Mvc;
using VehicleQuotes.ResourceModels;
using VehicleQuotes.Services;

namespace VehicleQuotes.Controllers
{
    [Route("api/[controller]")]
    [ApiController]
    public class QuotesController : ControllerBase
    {
        private readonly QuoteService _service;

        // When intiating the request processing logic, the framework recognizes
        // that this controller has a dependency on QuoteService and expects an
        // instance of it to be injected via the constructor. The framework then
        // does what it needs to do in order to provide that dependency.
        public QuotesController(QuoteService service)
        {
            _service = service;
        }

        // GET: api/Quotes
        [HttpGet]
        // This method returns a collection of a new resource model instead of just the `Quote` entity directly.
        public async Task<ActionResult<IEnumerable<SubmittedQuoteRequest>>> GetAll()
        {
            // Instead of directly implementing the logic in this method, we call on
            // the service class and let it take care of the rest.
            return await _service.GetAllQuotes();
        }

        // POST: api/Quotes
        [HttpPost]
        // This method receives as a paramater a `QuoteRequest` of just the `Quote` entity directly.
        // That way callers of this endpoint don't need to be exposed to the details of our data model implementation.
        public async Task<ActionResult<SubmittedQuoteRequest>> Post(QuoteRequest request)
        {
            // Instead of directly implementing the logic in this method, we call on
            // the service class and let it take care of the rest.
            return await _service.CalculateQuote(request);
        }
    }
}

As you can see, we’ve once again opted to abstract away clients from the implementation details of our data model and used Resource Models for the API contract instead of the Quote entity directly. We have one for input data that’s called QuoteRequest and another one for output: SubmittedQuoteRequest. Not very remarkable by themselves, but feel free to explore the source code in the GitHub repo.

This controller has a dependency on QuoteService, which it uses to perform all of the necessary logic. This class is not defined yet so let’s do that next:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using Microsoft.EntityFrameworkCore;
using VehicleQuotes.Models;
using VehicleQuotes.ResourceModels;

namespace VehicleQuotes.Services
{
    public class QuoteService
    {
        private readonly VehicleQuotesContext _context;

        // This constructor defines a dependency on VehicleQuotesContext, similar to most of our controllers.
        // Via the built in dependency injection features, the framework makes sure to provide this parameter when
        // creating new instances of this class.
        public QuoteService(VehicleQuotesContext context)
        {
            _context = context;
        }

        // This method takes all the records from the `quotes` table and constructs `SubmittedQuoteRequest`s with them.
        // Then returns that as a list.
        public async Task<List<SubmittedQuoteRequest>> GetAllQuotes()
        {
            var quotesToReturn = _context.Quotes.Select(q => new SubmittedQuoteRequest
            {
                ID = q.ID,
                CreatedAt = q.CreatedAt,
                OfferedQuote = q.OfferedQuote,
                Message = q.Message,

                Year = q.Year,
                Make = q.Make,
                Model = q.Model,
                BodyType = q.BodyType.Name,
                Size = q.Size.Name,

                ItMoves = q.ItMoves,
                HasAllWheels = q.HasAllWheels,
                HasAlloyWheels = q.HasAlloyWheels,
                HasAllTires = q.HasAllTires,
                HasKey = q.HasKey,
                HasTitle = q.HasTitle,
                RequiresPickup = q.RequiresPickup,
                HasEngine = q.HasEngine,
                HasTransmission = q.HasTransmission,
                HasCompleteInterior = q.HasCompleteInterior,
            });

            return await quotesToReturn.ToListAsync();
        }

        // This method takes an incoming `QuoteRequest` and calculates a quote based on the vehicle described by it.
        // To calculate this quote, it looks for any overrides before trying to use the currently existing rules defined
        // in the `quote_rules` table. It also stores a record on the `quotes` table with all the incoming data and the
        // quote calculation result. It returns back the quote value as well as a message explaining the conditions of
        // the quote.
        public async Task<SubmittedQuoteRequest> CalculateQuote(QuoteRequest request)
        {
            var response = this.CreateResponse(request);
            var quoteToStore = await this.CreateQuote(request);
            var requestedModelStyleYear = await this.FindModelStyleYear(request);
            QuoteOverride quoteOverride = null;

            if (requestedModelStyleYear != null)
            {
                quoteToStore.ModelStyleYear = requestedModelStyleYear;

                quoteOverride = await this.FindQuoteOverride(requestedModelStyleYear);

                if (quoteOverride != null)
                {
                    response.OfferedQuote = quoteOverride.Price;
                }
            }

            if (quoteOverride == null)
            {
                response.OfferedQuote = await this.CalculateOfferedQuote(request);
            }

            if (requestedModelStyleYear == null)
            {
                response.Message = "Offer subject to change upon vehicle inspection.";
            }

            quoteToStore.OfferedQuote = response.OfferedQuote;
            quoteToStore.Message = response.Message;

            _context.Quotes.Add(quoteToStore);
            await _context.SaveChangesAsync();

            response.ID = quoteToStore.ID;
            response.CreatedAt = quoteToStore.CreatedAt;

            return response;
        }

        // Creates a `SubmittedQuoteRequest`, initialized with default values, using the data from the incoming
        // `QuoteRequest`. `SubmittedQuoteRequest` is what gets returned in the response payload of the quote endpoints.
        private SubmittedQuoteRequest CreateResponse(QuoteRequest request)
        {
            return new SubmittedQuoteRequest
            {
                OfferedQuote = 0,
                Message = "This is our final offer.",

                Year = request.Year,
                Make = request.Make,
                Model = request.Model,
                BodyType = request.BodyType,
                Size = request.Size,

                ItMoves = request.ItMoves,
                HasAllWheels = request.HasAllWheels,
                HasAlloyWheels = request.HasAlloyWheels,
                HasAllTires = request.HasAllTires,
                HasKey = request.HasKey,
                HasTitle = request.HasTitle,
                RequiresPickup = request.RequiresPickup,
                HasEngine = request.HasEngine,
                HasTransmission = request.HasTransmission,
                HasCompleteInterior = request.HasCompleteInterior,
            };
        }

        // Creates a `Quote` based on the data from the incoming `QuoteRequest`. This is the object that gets eventually
        // stored in the database.
        private async Task<Quote> CreateQuote(QuoteRequest request)
        {
            return new Quote
            {
                Year = request.Year,
                Make = request.Make,
                Model = request.Model,
                BodyTypeID = (await _context.BodyTypes.SingleAsync(bt => bt.Name == request.BodyType)).ID,
                SizeID = (await _context.Sizes.SingleAsync(s => s.Name == request.Size)).ID,

                ItMoves = request.ItMoves,
                HasAllWheels = request.HasAllWheels,
                HasAlloyWheels = request.HasAlloyWheels,
                HasAllTires = request.HasAllTires,
                HasKey = request.HasKey,
                HasTitle = request.HasTitle,
                RequiresPickup = request.RequiresPickup,
                HasEngine = request.HasEngine,
                HasTransmission = request.HasTransmission,
                HasCompleteInterior = request.HasCompleteInterior,

                CreatedAt = DateTime.Now
            };
        }

        // Tries to find a registered vehicle that matches the one for which the quote is currently being requested.
        private async Task<ModelStyleYear> FindModelStyleYear(QuoteRequest request)
        {
            return await _context.ModelStyleYears.FirstOrDefaultAsync(msy =>
                msy.Year == request.Year &&
                msy.ModelStyle.Model.Make.Name == request.Make &&
                msy.ModelStyle.Model.Name == request.Model &&
                msy.ModelStyle.BodyType.Name == request.BodyType &&
                msy.ModelStyle.Size.Name == request.Size
            );
        }

        // Tries to find an override for the vehicle for which the quote is currently being requested.
        private async Task<QuoteOverride> FindQuoteOverride(ModelStyleYear modelStyleYear)
        {
            return await _context.QuoteOverides
                .FirstOrDefaultAsync(qo => qo.ModelStyleYear == modelStyleYear);
        }

        // Uses the rules stored in the `quote_rules` table to calculate how much money to offer for the vehicle
        // described in the incoming `QuoteRequest`.
        private async Task<int> CalculateOfferedQuote(QuoteRequest request)
        {
            var rules = await _context.QuoteRules.ToListAsync();

            // Given a vehicle feature type, find a rule that applies to that feature type and has the value that
            // matches the condition of the incoming vehicle being quoted.
            Func<string, QuoteRule> theMatchingRule = featureType =>
                rules.FirstOrDefault(r =>
                    r.FeatureType == featureType &&
                    r.FeatureValue == request[featureType]
                );

            // For each vehicle feature that we care about, sum up the the monetary values of all the rules that match
            // the given vehicle condition.
            return QuoteRule.FeatureTypes.All
                .Select(theMatchingRule)
                .Where(r => r != null)
                .Sum(r => r.PriceModifier);
        }
    }
}

Finally, we need to tell the framework that this class is available for Dependency Injection. Similarly to how we did with our VehicleQuotesContext, we do so in the Startup.cs file’s ConfigureServices method. Just add this line at the top:

services.AddScoped<Services.QuoteService>();

The core tenet of Inversion of Control is to depend on abstractions, not on implementations. So ideally, we would not have our controller directly call for a QuoteService instance. Instead, we would have it reference an abstraction, e.g. an interface like IQuoteService. The statement on Startup.cs would then look like this instead: services.AddScoped<Services.IQuoteService, Services.QuoteService>();.

This is important because it would allow us to unit test the component that depends on our service class (i.e. the controller in this case) by passing it a mock object — one that also implements IQuoteService but does not really implement all the functionality of the actual QuoteService class. Since the controller only knows about the interface (that is, it “depends on an abstraction”), the actual object that we give it as a dependency doesn’t matter to it, as long as it implements that interface. This ability to inject mocks as dependencies is invaluable during testing. Testing is beyond the scope of this article though, so I’ll stick with the simpler approach with a static dependency on a concrete class. Know that this is not a good practice when it comes to actual production systems.

And that’s all it takes. Once you add a few rules via POST /api/QuoteRules, you should be able to get some vehicles quoted with POST /api/Quotes. And also see what the system has stored via GET /api/Quotes.

And that’s all the functionality that we set out to build into our REST API! There are a few other neat things that I thought I’d include though.

Adding seed data for lookup tables

Our vehicle size and body type data isn’t meant to really change much. In fact, we could even preload that data when our application starts. EF Core provides a data seeding feature that we can access via configurations on the DbContext itself. For our case, we could add this method to our VehicleQuotesContext:

protected override void OnModelCreating(ModelBuilder modelBuilder)
{
    modelBuilder.Entity<Size>().HasData(
        new Size { ID = 1, Name = "Subcompact" },
        new Size { ID = 2, Name = "Compact" },
        new Size { ID = 3, Name = "Mid Size" },
        new Size { ID = 5, Name = "Full Size" }
    );

    modelBuilder.Entity<BodyType>().HasData(
        new BodyType { ID = 1, Name = "Coupe" },
        new BodyType { ID = 2, Name = "Sedan" },
        new BodyType { ID = 3, Name = "Hatchback" },
        new BodyType { ID = 4, Name = "Wagon" },
        new BodyType { ID = 5, Name = "Convertible" },
        new BodyType { ID = 6, Name = "SUV" },
        new BodyType { ID = 7, Name = "Truck" },
        new BodyType { ID = 8, Name = "Mini Van" },
        new BodyType { ID = 9, Name = "Roadster" }
    );
}

OnModelCreating is a hook that we can define to run some code at the time the model is being created for the first time. Here, we’re using it to seed some data. In order to apply that, a migration needs to be created and executed. If you’ve added some data to the database, be sure to wipe it before running the migration so that we don’t run into unique constraint violations. Here are the migrations:

$ dotnet ef migrations add AddSeedDataForSizesAndBodyTypes

$ dotnet ef database update

After that’s done, it no longer makes sense to allow creating, updating, deleting and fetching individual sizes and body types, so I would delete those endpoints from the respective controllers.

There are other options for data seeding in EF Core. Take a look: Data Seeding.

Improving the Swagger UI via XML comments

Our current auto-generated Swagger UI is pretty awesome. Especially considering that we got it for free. It’s a little lacking when it comes to more documentation about specific endpoint summaries or expected responses. The good news is that there’s a way to leverage C# XML Comments in order to improve the Swagger UI.

We can add support for that by configuring our project to produce, at build time, an XML file with the docs that we write. In order to do so, we need to update the VehicleQuotes.csproj like this:

<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <!-- ... -->
+   <GenerateDocumentationFile>true</GenerateDocumentationFile>
+   <NoWarn>$(NoWarn);1591</NoWarn>
  </PropertyGroup>

  <!-- ... -->
</Project>

GenerateDocumentationFile is the flag that tells the .NET 5 build tools to generate the documentation file. The NoWarn element prevents our build output from getting cluttered with a lot of warnings saying that some classes and methods are not properly documented. We don’t want that because we just want to write enough documentation for the Swagger UI. And that includes only the controllers.

You can run dotnet build and look for the new file in bin/Debug/net5.0/VehicleQuotes.xml.

Then, we need to update Startup.cs. First we need to add the following using statements:

using System.IO;
using System.Reflection;

And add the following code to the ConfigureServices method on Startup.cs:

public void ConfigureServices(IServiceCollection services)
{
    // ...

    services.AddSwaggerGen(c =>
    {
        c.SwaggerDoc("v1", new OpenApiInfo { Title = "VehicleQuotes", Version = "v1" });

+       c.IncludeXmlComments(
+           Path.Combine(
+               AppContext.BaseDirectory,
+               $"{Assembly.GetExecutingAssembly().GetName().Name}.xml"
+           )
        );
    });

    // ...
}

This makes it so the SwaggerGen service knows to look for the XML documentation file when building up the Open API specification file used for generating the Swagger UI.

Now that all of that is set up, we can actually write some XML comments and attributes that will enhance our Swagger UI. As an example, put this on top of ModelsController’s Post method:

/// <summary>
/// Creates a new vehicle model for the given make.
/// </summary>
/// <param name="makeId">The ID of the vehicle make to add the model to.</param>
/// <param name="model">The data to create the new model with.</param>
/// <response code="201">When the request is valid.</response>
/// <response code="404">When the specified vehicle make does not exist.</response>
/// <response code="409">When there's already another model in the same make with the same name.</response>
[HttpPost]
[ProducesResponseType(StatusCodes.Status201Created)]
[ProducesResponseType(StatusCodes.Status404NotFound)]
[ProducesResponseType(StatusCodes.Status409Conflict)]
public async Task<ActionResult<ModelSpecification>> Post([FromRoute] int makeId, ModelSpecification model)
{
    // ...
}

Then, the Swagger UI now looks like this for this endpoint:

Configuring the app via settings files and environment variables

Another aspect that’s important to web applications is having them be configurable via things like configuration files or environment variables. The framework already has provision for this, we just need to use it. I’m talking about the appsettings files.

We have two of them created for us by default: appsettings.json which is applied in all environments, and appsettings.Development.json that is applied only under development environments. The environment is given by the ASPNETCORE_ENVIRONMENT environment variable, and it can be set to either Development, Staging, or Production by default. That means that if we had, for example, an appsettings.Staging.json file, the settings defined within would be loaded if the ASPNETCORE_ENVIRONMENT environment variable were set to Staging. You get the idea.

You can learn more about configuration and environments in the official documentation.

Anyway, let’s add a new setting on appsettings.json:

{
  // ...
+ "DefaultOffer": 77
}

We’ll use this setting to give default offers when we’re not able to calculate appropriate quotes for vehicles. This can happen if we don’t have rules, or if the ones we have don’t match any of the incoming vehicle features or if for some other reason the final sum ends up in zero or negative number. We can use this setting in our QuoteService like so:

// ...
+using Microsoft.Extensions.Configuration;

namespace VehicleQuotes.Services
{
    public class QuoteService
    {
        // ...
+       private readonly IConfiguration _configuration;

-       public QuoteService(VehicleQuotesContext context)
+       public QuoteService(VehicleQuotesContext context, IConfiguration configuration)
        {
            _context = context;
+           _configuration = configuration;
        }

        // ...

        public async Task<SubmittedQuoteRequest> CalculateQuote(QuoteRequest request)
        {
            // ...

+           if (response.OfferedQuote <= 0)
+           {
+               response.OfferedQuote = _configuration.GetValue<int>("DefaultOffer", 0);
+           }

            quoteToStore.OfferedQuote = response.OfferedQuote;

            // ...
        }

        // ...
    }
}

Here, we’ve added a new parameter to the constructor to specify that VehicleQuotesContext has a dependency on IConfiguration. This prompts the framework to provide an instance of that when instantiating the class. We can use that instance to access the settings that we defined in the appsettings.json file via its GetValue method, like I demonstrated above.

The value of the settings in appsettings.json can be overridden by environment variables as well. On Linux, for example, we can run the app and set an environment value with a line like this:

$ DefaultOffer=123 dotnet run

This will make the application use 123 instead of 77 when it comes to the DefaultOffer setting. This flexibility is great from a DevOps perspective. And we had to do minimal work in order to get that going.

That’s all for now

And that’s it! In this article we’ve gone through many of the features offered in .NET 5, ASP.NET Core, and Entity Framework Core to support some of the most common use cases when it comes to developing Web API applications.

We’ve installed .NET 5 and created an ASP.NET Core Web API project with EF Core and a few bells and whistles, created controllers to support many different endpoints, played a little bit with routes and response codes, created and built upon a data model and updated a database via entities and migrations, implemented database constraints using unique indexes, implemented input validation using both built-in and custom validation attributes, implemented resource models as DTOs for defining the contract of some of our API endpoints, tapped into the built-in dependency injection capabilities, explored and improved the auto-generated Swagger UI, added seed data for our database, learned about configuration via settings files and environment variables.

.NET 5 is looking great.

Automating Windows Service Installation

2021-04-23T00:00:00+00:00

Photo by Science in HD on Unsplash

For me, setting up a service started as a clean one-liner that used InstallUtil.exe, but as time went on, I accumulated additional steps. Adding external files & folders, setting a custom Service Logon Account, and even an SSL cert had to be configured first before the service could be used. An entire checklist was needed just to make sure the service would start successfully. That’s when I realized a proper installation method was needed here. This article will go over how to make a dedicated .msi installer for a Windows Service that can do all these things and more.

Creating an installer can be tricky, because not all the available features are easy to find. In fact, the setup project itself is not included by default in Visual Studio; you need to install an extension in order to create one. But once the installer is created, we can use it to do things like:

Configure the installer to copy the build output of a project to the C:\Program Files (x86) folder, as well as add custom files & folders to the installation
Add custom CLI flags to the installer to specify the Service Logon Account at install time
Add an installer class to the service and use the installation lifecycle hooks to write custom code that gets run at any stage of the installation.

A Note On Compatibility

For .NET Core and .NET 5.0 projects, you won’t be able to add an installer class. To use either .NET Core or .NET 5.0 to make a service instead, you’d need to make a different kind of project called a Worker Service. A Worker Service differs from a traditional Windows Service in that it’s more like a console application that spawns off a worker process on a new thread. It can be configured to run as a Windows service, but doesn’t have to be. So instead of using an installer, for a Worker Service you’d publish the project to an output directory and then use the SC.exe utility to add it as a Windows service:

dotnet publish -o C:\<PUBLISH_PATH>
SC CREATE <WORKER_NAME> C:\<PUBLISH_PATH>

Creating a Windows Setup Project

In order to create a .msi installer in Visual Studio 2019, you’ll need to install the Microsoft Visual Studio Installer Projects extension. While it’s not provided with a default installation of Visual Studio 2019, it’s an official Microsoft extension. Once you’ve installed it, you’ll be able to create any of the following projects:

To create an installer, you can create a new Setup Project. The build output from this project will be your .msi installer. The setup project has a few different views, which you can use to configure what the installer needs to accomplish. These views can be accessed by right-clicking on the project in the Solution Explorer and expanding View from the context menu:

Configuring the Installation File System

To configure what files need to be installed, you can use the File System view, which provides a UI with some folders added to it:

Here, clicking on any folder on the left shows its contents over on the right. It also populates the Properties Window with the information about the folder:

In the above example, we can see that the Application Folder is being output to a folder inside C:\Program Files (x86). You can add any folders you want to the file system by right-clicking on the file system to open the Special Folders context menu:

Some default folders are shown here for convenience. But let’s say we wanted to make some files get added to the C:\ProgramData folder. To do this, select “Custom Folder” and give it a name. Then, in the Properties Window, set the value of DefaultLocation to the correct path:

From here, you can use the right half of the view to add additional folders within C:\ProgramData\DotNetDemoService based on your needs.

Another thing you’ll likely want to do is put the DLLs from your application into a folder within C:\Program Files (x86). You can easily do this by mapping the primary build output of your project to the Application Folder in the installer’s file system. To do this, right-click on the Application Folder, and add project output:

From there you’ll be prompted to select the project and output type. Select your project, and “Primary Output”:

This will copy over the DLLs for your project and all of its dependencies.

Creating an Installer class

You may be wondering if it’s possible to define custom code to be run during the installation process. It is! For any project targeting .NET Framework 4.8 and under, you can add a class that extends System.Configuration.Install.Installer, and has the [RunInstaller(true)] attribute applied to it. After doing so, you’ll then be able to hook in and override any of the installation lifecycle methods. Taking a look into the definition of the System.Configuration.Install.Installer class reveals the list of overridable lifecycle hook methods you can use to add custom logic to the installation:

public virtual void Commit(IDictionary savedState);
public virtual void Install(IDictionary stateSaver);
public virtual void Rollback(IDictionary savedState);
public virtual void Uninstall(IDictionary savedState);
protected virtual void OnAfterInstall(IDictionary savedState);
protected virtual void OnAfterRollback(IDictionary savedState);
protected virtual void OnAfterUninstall(IDictionary savedState);
protected virtual void OnBeforeInstall(IDictionary savedState);
protected virtual void OnBeforeRollback(IDictionary savedState);
protected virtual void OnBeforeUninstall(IDictionary savedState);
protected virtual void OnCommitted(IDictionary savedState);
protected virtual void OnCommitting(IDictionary savedState);

It also defines event handlers for each of these steps as well:

public event InstallEventHandler BeforeInstall;
public event InstallEventHandler Committing;
public event InstallEventHandler AfterUninstall;
public event InstallEventHandler AfterRollback;
public event InstallEventHandler AfterInstall;
public event InstallEventHandler Committed;
public event InstallEventHandler BeforeRollback;
public event InstallEventHandler BeforeUninstall;

To add an installer class to the Windows Service project, there’s a helper you can use by right clicking on the designer view of the service and selecting “Add Installer” from the context menu:

This will add a new file called ProjectInstaller.cs to your project, which has its own designer view. The designer view has a corresponding ProjectInstaller.Designer.cs file that amends the ProjectInstaller class with the code generated by the designer. You’ll notice that this designer view already defines two objects, serviceInstaller1 and serviceProcessInstaller1.

These are special installer classes that will handle all the default installation tasks for your service. serviceInstaller1 is of type ServiceInstaller and handles defining the service name and if it should auto start when the machine boots up. serviceProcessInstaller1 is of type ServiceProcessInstaller and handles setting up the Service Logon Account, which the service will run with once installed. Both of these are already set up and invoked by the designer generated code in ProjectInstaller.Designer.cs.

Since both of these special service installers extend System.Configuration.Install.Installer, you can add custom code to occur at any point of the installation on these as well. The designer view again provides a GUI helper to add this in. Double-clicking on serviceInstaller1 will automatically add a new method to ProjectInstaller:

private void serviceInstaller1_AfterInstall(object sender, InstallEventArgs e) { }

It will also put some code into ProjectInstaller.Designer.cs which adds this method to the AfterInstall event of serviceInstaller1.

Adding Installer CLI Options

It’s also possible to add custom properties that you can pass to the installer as command line arguments. These can be done by defining Custom Actions on the primary build output of your project. To do this, go to the Custom Actions view of the installer project, right click on “Install” and select “Add Custom Action” from the context menu:

This will open up a dialog that prompts you to select a file in the installer’s file system to define a custom action for. In this case, we want to define a custom action on the primary build output. This way, the custom CLI options we are about to define will be passed to the project’s installer class.

After you click “OK”, the primary build output will show up in the Custom Actions view. When you click on it, you’ll notice that the properties window has a property called CustomActionData. In short, you can use it to define custom CLI arguments like this:

CustomActionData has its own syntax, so let’s dive deeper into what this actually does. We’re mapping the value of USERNAME and PASSWORD from the installer’s Properties Collection to the InstallContext of the installer class of your project under the Username and Password keys, respectively. The square brackets denote that the value is to be taken from the Properties Collection, and the quotes allow the value of the property to contain spaces. The forward slash denotes that we are adding a new key to the context. Any command line arguments passed to the installer are added to the Properties Collection by default.

Using Custom CLI Options in the Installer Class

Now that we have defined our custom action with the CLI arguments, we can go over to the project’s Installer class and access them via the Context property. In this example, we’re using the custom properties to define the Logon account for the service, which needs to be set right before the installation happens. We can use the Install(IDictionary stateSaver) lifecycle hook method for this purpose:

public override void Install(IDictionary stateSaver)
{
    // If no username or password is specified, fall back to installing the service to run as the SYSTEM account
    if (
        string.IsNullOrEmpty(this.Context.Parameters["Username"])
        || string.IsNullOrEmpty(this.Context.Parameters["Password"])
    ) {
        this.serviceProcessInstaller1.Account = System.ServiceProcess.ServiceAccount.LocalSystem;
    }

    // Otherwise, configure the service to run under the specified account.
    else
    {
        this.serviceProcessInstaller1.Username = this.Context.Parameters["Username"];
        this.serviceProcessInstaller1.Password = this.Context.Parameters["Password"];
    }

    // Run the base class install after the service has been configured.
    base.Install(stateSaver);
}

Conditionally Installing Files

It’s also possible to make the installer conditionally install files based on a value from the Properties Collection. One example of how this can be useful would be swapping in the production or development configuration file based on the value of a command line argument. We don’t need to write any additional code to do this, we just have to add a value for the Condition property in the Properties Window for the file:

The above condition will make the file settings.production.config be installed only if the DEBUG command line argument is not defined or is set to “false”. Like the custom actions, this property is also sourced from the Properties Collection.

Conclusion

And that’s it! I found that having a dedicated .msi installer was handy for making the setup of my Windows Service completely hands-free. While some of the features you need might seem buried within context menus, the flexibility of having the installer handle the service setup is well worth the effort.

Have any questions? Feel free to leave a comment!

.NET 10 is Coming: What's New, and Why It Matters

Performance and runtime improvements

Security and increased cryptographic support

API & web enhancements

AI, cloud, and DevOps

C# 14

Announcing End Point Ecommerce, Our New Open Source Project

Key features

How we got here

Who is this for?

Implementing Azure Blob Storage in .NET 9

Reducing page load times: Cloudflare and caching with ASP.NET

Adding response caching

Using Cloudflare to distribute cached content

Testing

Using a Containerized Nginx Proxy to Serve a Multi-Application .NET System

Reintroducing the demo project

Adding the proxy service in compose.yaml

Writing the proxy Dockerfile

Configuring the proxy

Configuring the PathBase in ASP.NET Core apps

Deploying

Using Docker Compose to Deploy a Multi-Application .NET System

Getting familiar with the demo project

Including the code base repository as a Git submodule

Deploying the database

Deploying the admin portal web app

Adding a maintenance container

Deploying the web API

Looking at the logs

Persisting data protection keys

Storing sensitive information with secrets

Parameterizing compose.yaml to support multiple deployment environments

Waiting for the database to be ready

Serving the apps with NGINX

That’s all for now

Uploading multiple files in a single request in an ASP.NET Core application

Where to store the images

A Razor Page for uploading many files

Uploading any number of files

Validating the uploaded images

Displaying the images

Deleting the images

How to validate record uniqueness in ASP.NET

Using a database unique index to enforce uniqueness

Writing a custom validation attribute to enforce uniqueness

A generic custom validation attribute to enforce uniqueness

Writing a unit test for the custom validation attribute

Writing integration tests for an ASP.NET Web API

Introducing the project

Setting up the integration tests project

Writing a database fixture

Writing some integration tests

Writing some more integration tests

A test that writes to and reads from the database

A test that makes a POST request

A test that makes many requests

Disabling some application services

Using Razor templates to render HTML emails in .NET

The plan

Step 1: Sending emails in .NET with the MailKit NuGet package

Step 2: Rendering Razor templates into strings

Step 3: Defining the email templates

Step 4: Putting it all together: the class that sends the “quote generated” email

Building a real-time application with SignalR, .NET, and three.js

What we’ll build

Creating the web application

Installing and configuring SignalR

Creating a SignalR hub

Connecting to SignalR Hub using the SignalR JavaScript client library

Adding features

Backend

Frontend

Wrapping Up

Client Profile: J.G. Title Company

The solution

Tech stack

The team

Results

I wrote the same app twice, with Hotwire and Blazor Server — here’s what I learned

Adding the proxy service in `compose.yaml`

Writing the proxy `Dockerfile`

Parameterizing `compose.yaml` to support multiple deployment environments