CSTE Conference EpiTrax retrospective
By Steve Yoman
June 29, 2022
Last week we were in Louisville, Kentucky for the CSTE Conference. End Point staffed a conference booth to represent the EpiTrax public health surveillance system to a wonderful group of public health experts.
You can read some background about the conference and CSTE in our earlier blog post announcing our plans.
We really enjoyed meeting new friends in person after two years of canceled events due to the pandemic. We spoke with staff from health departments and disease surveillance teams from several state and local jurisdictions, as well as with experts from the CDC and other software and service vendors.
One of the highlights was going around to meet other people staffing booths at the conference. It charged us up to see and hear about all of the interesting and innovative things going on in the public health space at a time when there is so much that needs to be done. We were particularly struck by the efforts being made in onboarding and distributing ELRs and eCRs, areas where the Electronic Message Staging Area (EMSA, which we deploy and support) can complement and enrich those activities.
The open-source disease surveillance and reporting software EMSA and EpiTrax both enjoyed well-deserved attention as we demonstrated them to numerous groups seeking better solutions to serve people in their jurisdictions. People were very interested in the functionality EpiTrax has for case management, encounters, and NMI reporting.
Of course there is a lot more we could say about EpiTrax and EMSA. So, if you didn’t find us at the conference or if you are interested in what EpiTrax and EMSA can do, contact us here online. We are happy to give you a demo!
conference epitrax emsa
How to deploy a Django App with Aurora Serverless and AWS Copilot
By Jeffry Johar
June 26, 2022
Photo by Виктор Куликов
AWS Copilot has the capability to provision an external database for its containerized work load. The database options are DynamoDB (NoSQL), Aurora Serverless (SQL), and S3 Buckets. For this blog we are going to provision and use Aurora Serverless with a containerized Django app. Aurora Serverless comes with 2 options for its engine: MySQL or PostgreSQL.
Watch Amazon’s 2-minute introduction video to get the basic idea of Aurora Serverless.
We are going to work with the same Django application from my last article on AWS Copilot.
In my last article, the Django application was deployed with SQLite as the database. The application’s data is stored in SQLite which resides internally inside the container. The problem with this setup is the data is not persistent. Whenever we redeploy the application, the container will get a new filesystem. Thus all old data will be removed automatically.
Now we are moving away the application’s data externally so that the life of the data does not depend on the container. We are going to put the data on the Aurora Serverless with PostgreSQL as the engine.
Django with SQLite as the internal database
Django with Aurora Serverless as the external database
The Prerequisites
Docker, AWS CLI, and AWS Copilot CLI are required. Please refer to my last article for how to install them.
The Django Project
Create a Django project by using a Python Docker Image. You can clone my Git project to get the Dockerfile, docker-compose.yaml and requirements.txt that I’m using:
$ git clone https://github.com/aburayyanjeffry/django-copilot.git django-aurora
Go to the django-aurora directory and execute docker-compose to create a Django project named “mydjango”.
$ cd django-aurora
$ docker-compose run web django-admin startproject mydjango
The Deployment with AWS Copilot
Execute the following command to create a AWS Copilot application with the name of “mydjango”, a load balancer container with the service name “django-web” which is made from the Dockerfile in the current directory.
$ copilot init \
-a mydjango \
-t "Load Balanced Web Service" -n django-web \
-d ./Dockerfile
Answer N to the following question. We want to defer the deployment until we have set up the database.
All right, you're all set for local development.
Would you like to deploy a test environment? [? for help] (y/N) N
We need to create an environment for our application. Execute the following to create an environment named test for the “mydjango” application with the default configuration.
$ copilot env init \
--name test \
--profile default \
--app mydjango \
--default-config
Now we are going to generate a config for our Aurora Serverless database. Basically this is the CloudFormation template that will be used for Aurora Serverless.
Execute the following to generate the configuration for an Aurora cluster named “mydjango-db” that we will use for the “django-web” application. The Aurora cluster will be using the PostgreSQL engine and the database name will be “mydb”.
$ copilot storage init \
-n mydjango-db \
-t Aurora -w \
django-web \
--engine PostgreSQL \
--initial-db mydb
Take note of the injected environment variable name. This is where the database info and credentials are stored, and we will use this variable in later steps.
✔ Wrote CloudFormation template at copilot/django-web/addons/mydjango-db.yml
Recommended follow-up actions:
- Update django-web's code to leverage the injected environment variable MYDJANGODB_SECRET.
Edit mydjango/settings.py to include the following. We will pass the injected environment variable we got previously to the function for getting the DBINFO variables.
from pathlib import Path
import os
import json
...
ALLOWED_HOSTS = ['*']
...
DBINFO = json.loads(os.environ.get('MYDJANGODB_SECRET', '{}'))
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'HOST': DBINFO['host'],
'PORT': DBINFO['port'],
'NAME': DBINFO['dbname'],
'USER': DBINFO['username'],
'PASSWORD': DBINFO['password'],
}
}
Deploy the application:
$ copilot deploy --name django-web
Open the terminal of the service:
$ copilot svc exec
Execute the following commands to migrate the initial database and to create a superuser account:
$ python manage.py migrate
$ python manage.py createsuperuser
Execute the following command to check on the environment variable. Take note of the MYDJANGODB_SECRET variable. It is the variable that holds the database information.
$ env | grep MYDJANGODB_SECRET
How to query Aurora Serverless
We can use the Query Editor at AWS Console for RDS to query Aurora Serverless.
Click the DB base on the DB identifier from the injected environment variable and click Modify.
Click the check box for Data API.
Select Apply Immediately.
Click Query Editor and fill in the Database information from the injected environment variable.
Now you may use the Query Editor to query the database. Execute the following query to list all tables in the database:
The End
That’s all, folks. We have deployed a containerized Django application and an Aurora Serverless with AWS Copilot. For further info on AWS Copilot visit its website.
docker containers aws postgres
How to deploy a containerized Django app with AWS Copilot
By Jeffry Johar
June 21, 2022
Generally there are 2 major options at AWS when it comes to deployment of containerized applications. You can either go for EKS or ECS.
EKS (Elastic Kubernetes Service) is the managed Kubernetes service by AWS. ECS (Elastic Container Service), on the other hand, is AWS’s own way to manage your containerized application. You can learn more about EKS and ECS on the AWS website.
For this post we will use ECS.
The chosen one and the sidekick
With ECS chosen, now you have to find a preferably easy way to deploy your containerized application on it.
There are quite a number of resources from AWS that are needed for your application to live on ECS, such as VPC (Virtual Private Cloud), Security Group (firewall), EC2 (virtual machine), Load Balancer, and others. Creating these resources manually is cumbersome so AWS has came out with a tool that can automate the creation of all of them. The tool is known as AWS Copilot and we are going to learn how to use it.
Install Docker
Docker or Docker Desktop is required for building the Docker image later. Please refer to my previous article on how to install Docker Desktop on macOS, or follow Docker’s instructions for Linux and Windows.
Set up AWS CLI
We need to set up the Docker AWS CLI (command-line interface) for authentication and authorization to AWS.
Execute the following command to install the AWS CLI on macOS:
$ curl -O "https://awscli.amazonaws.com/AWSCLIV2.pkg"
$ sudo installer -pkg AWSCLIV2.pkg -target /
For other OSes see Amazon’s docs.
Execute the following command and enter the AWS Account and Access Keys.
$ aws configure
Install AWS Copilot CLI
Now it’s time for the main character: AWS Copilot.
Install AWS Copilot with Homebrew for macOS:
$ brew install aws/tap/copilot-cli
See AWS Copilot Installation for other platforms.
The Django project
Create a Django project by using a Python Docker Image. You can clone my Git project to get the Dockerfile, docker-compose.yaml and requirements.txt that I’m using.
$ git clone https://github.com/aburayyanjeffry/django-copilot.git
Go to the django-pilot directory and execute docker-compose to create a Django project named “mydjango”.
$ cd django-copilot
$ docker-compose run web django-admin startproject mydjango .
Edit mydjango/settings.py to allow all hostnames for its URL. This is required because by default AWS will generate a random URL for the application. Find the following variable and set the value as follows:
ALLOWED_HOSTS = ['*']
The Deployment with AWS Copilot
Create an AWS Copilot “Application”. This is a grouping of services such as web app or database, environments (development, QA, production), and CI/CD pipelines. Execute the following command to create an Application with the name of “mydjango”.
$ copilot init -a mydjango
Select the Workload type. Since this Django is an internet-facing app we will choose the “Load Balanced Web Service”.
Which workload type best represents your architecture? [Use arrows to move, type to filter, ? for more help]
Request-Driven Web Service (App Runner)
> Load Balanced Web Service (Internet to ECS on Fargate)
Backend Service (ECS on Fargate)
Worker Service (Events to SQS to ECS on Fargate)
Scheduled Job (Scheduled event to State Machine to Fargate)
Give the Workload a name. We are going to name it “mydjango-web”.
Workload type: Load Balanced Web Service
What do you want to name this service? [? for help] mydjango-web
Select the Dockerfile in the current directory.
Which Dockerfile would you like to use for mydjango-web? [Use arrows to move, type to filter, ? for more help]
> ./Dockerfile
Enter custom path for your Dockerfile
Use an existing image instead
Accept to create a test environment.
All right, you're all set for local development.
Would you like to deploy a test environment? [? for help] (y/N) y
Wait and see. At the end of the deployment you will get the URL of your application. Open it in a browser.
Now let’s migrate some data, create a superuser, and try to log in. The Django app comes with a SQLite database. Execute the following command to get a terminal for the Django app:
$ copilot svc exec
Once in the terminal, execute the following to migrate the initial data and to create the superuser:
$ python manage.py migrate
$ python manage.py createsuperuser
Now you may access the admin page and login by using the created credentials.
You should see the Django admin:
A mini cheat sheet
| AWS Copilot commands | Remarks |
|---|---|
copilot app ls |
To list available Applications |
copilot app show -n appname |
To get the details of an Application |
copilot app delete -n appname |
To delete an Application |
copilot svc ls |
To list available Services |
copilot svc show -n svcname |
To get the details of a Service |
copilot svc delete -n svcname |
To delete a Service |
The End
That’s all, folks.
AWS Copilot is a tool to automate the deployment of AWS infrastructure for our containerized application needs. It takes away most of the worries about infrastructure and enables us to focus sooner on the application development.
For further info on AWS Copilot visit its website.
docker containers cloud aws python django
Getting started with Docker and Kubernetes on macOS
By Jeffry Johar
June 20, 2022
What is the best way to master American English? One school of thought says that the best way to learn a language is to live in the country of the origin. For American English that would be the USA. Why is that so? Because we as the learners get to talk to native speakers daily. By doing this, we get to know how the natives use the language and its grammar in the real world.
The same goes for learning Docker and Kubernetes. The best way to learn Docker and Kubernetes is to get them in our MacBooks, laptops, and PCs. This way we can learn and try locally what works and what doesn’t work in our local host at any time, any day.
Lucky for us earthlings who enjoy GUIs, Docker now has Docker Desktop. As its name suggests, it is nicely built for the desktop. It comes with GUI and CLI to manage our Docker and Kubernetes needs. Please take note of the Docker Desktop license. It is free for personal use, education, and open source projects, and has a fee for enterprise usage. With that out of the way, let’s get things started.
Docker Desktop Installation
The official Docker Desktop for can be found on Docker’s website. It covers installation for macOS, Linux, and Windows. For this post we are going to install Docker Desktop for macOS using Brew. Execute the following command to proceed with the Docker Desktop installation:
brew install --cask docker
Then run it at Finder ➝ Application ➝ Docker.
Upon a successful installation, Docker Desktop will appear as the following:
Click the Docker icon at the top menu bar to ensure the Docker Desktop is running.
Run the first containerized application
For the first application we are going to run the latest version of nginx official image from hub.docker.com. Open the terminal and execute the following to run the nginx image as a background service at port 80:
docker run -d -p 80:80 nginx:latest
Run the following command to check on the application. This is the Docker equivalent to the standard ps Unix command to list processes.
docker ps
curl the application at localhost port 80:
curl http://localhost
Sample output:
Stop the application
Execute the following command to get the application information and take note of the container ID.
docker ps
Execute the following to stop the application by its container ID. In the following example the container ID is f7c19b95fcc2.
docker stop {container ID}
Run docker ps again to ensure that the stopped application is not being display as a running application.
Sample Output:
Enable Kubernetes
Click the Docker icon at the top menu bar and click Preferences:
Enable Kubernetes and click Apply and Restart:
Click the Docker icon at the top menu bar and ensure the Kubernetes is running:
Open the terminal and check on the Kubernetes nodes. The status should be Ready:
kubectl get nodes
Deploy the first application on Kubernetes
We are going to deploy the same official latest nginx images at Kubernetes. Execute the following command:
kubectl run mynginx --image=nginx:latest
Execute the following command to check on the application. Its status should be Running. On a slow machine this will take some time and we might need to do this multiple times.
kubectl get pod
Execute the following command to create the Kubernetes service resource for the application. A service in Kubernetes serves as an internal named load balancer.
kubectl expose pod mynginx --port 80
Execute the following command to redirect the localhost network to the Kubernetes network. By doing this we can curl or access the application from localhost:{port}. This is a foreground process, so it needs to be left open.
kubectl port-forward service/mynginx 8080:80
Open another terminal to curl localhost:8080 or open the address in a web browser.
curl http://localhost:8080
In a browser:
Clean up the Kubernetes resources
Ctrl-C at the port-forwarding process in the terminal and list all the running Kubernetes resources. We should see our application in a pod and its services:
kubectl get all
Now we need to delete these resources.
To delete the service:
kubectl delete service/mynginx
To delete the application:
kubectl delete pod/mynginx
Now list back all resources. The mynginx-related resources should not be displayed.
kubectl get all
To stop Docker Desktop
If we are done with Docker Desktop we can stop its services by going to the top menu bar and selecting the Quit Docker Desktop option. This will stop Docker and Kubernetes services.
That’s all, folks. We now have the tools to learn and explore Docker and Kubernetes in our own local host.
Now we may proceed with the official documentation and other tutorials to continue on the path to learn Docker and Kubernetes.
docker kubernetes containers
Implementing Authentication in ASP.NET Core Web APIs
By Kevin Campusano
June 17, 2022
Authentication is a complex space. There are many problem scenarios and many more solutions. When it comes to Web APIs written with ASP.NET Core, there are various fully featured options like Duende IdentityServer or Azure Active Directory. These promise to be “everything but the kitchen sink” solutions which are robust and allow you to deal with many complex requirements.
But what if our requirements dictate that we need something simpler? Do we have to roll out our own from scratch? Or does ASP.NET Core offer smaller, customizable, somewhat independent puzzle pieces that we can put together without having to write all the code ourselves and still have a good amount of control?
Spoiler alert: The answer to that last question is yes. And we’re going to talk about it in this very article.
There is a Table of contents at the end of this post.
Two approaches to authentication: JWT and API Keys
In this article, we’ll take an existing ASP.NET Core Web API and add authentication capabilities to it. Specifically, we’ll support two authentication schemes commonly used for Web APIs: JWT and API Keys. Also, we will use our own database for storage of user accounts and credentials.
The project that we will work with is a simple ASP.NET Web API backed by a Postgres database. It has a few endpoints for CRUDing automotive related data and for calculating values of vehicles based on various aspects of them. You can read all about the process of building it here. I also added a few database integration tests for it here.
You can find it on GitHub. If you’d like to follow along, clone the repository and checkout this commit: cd290c765fcd2c6693008d3dc76fa931098dcaa0. It represents the project as it was before applying all the changes from this article.
You can follow the instructions in the project’s README file in order to get the app up and running.
Managing user accounts with ASP.NET Core Identity
Let’s deal first with the requirement of storing the user accounts in our own database.
Luckily for us, ASP.NET Core provides us with Identity. This is an API that offers a comprehensive solution for authentication. It can connect with the aforementioned Duende IdentityServer for OpenID Connect and OAuth 2.0, supports authentication via third parties like Facebook or Google, and can fully integrate with user interfaces on web apps, complete with scaffolding/autogeneration capabilities too.
Most importantly for us, it supports management of user accounts stored in our own database. It includes data for third party logins, passwords, roles for authorization, email confirmation, access tokens, etc.
That’s a lot of of functionality baked in there. But we don’t need all that, so let’s see how we can take only the bits and pieces that we need in order to fulfill our specific requirements.
Specifically, we want:
- Tables in our database for storage of user accounts and passwords.
- A programmatic way to create, fetch and validate users using those tables.
It’s actually pretty simple.
Install the necessary NuGet packages
First, we need to install a couple of NuGet packages:
- Microsoft.AspNetCore.Identity which contains the core library.
- Microsoft.AspNetCore.Identity.EntityFrameworkCore which includes the classes that Identity needs in order to properly interact with Entity Framework, which is what we’re using in our Web API for interfacing with the database.
We can install them by running these two commands from the VehicleQuotes directory:
$ dotnet add package Microsoft.AspNetCore.Identity
$ dotnet add package Microsoft.AspNetCore.Identity.EntityFrameworkCore
That will add the following lines to the VehicleQuotes/VehicleQuotes.csproj project file:
<Project Sdk="Microsoft.NET.Sdk.Web">
...
<ItemGroup>
...
+ <PackageReference Include="Microsoft.AspNetCore.Identity" Version="2.2.0" />
+ <PackageReference Include="Microsoft.AspNetCore.Identity.EntityFrameworkCore" Version="6.0.5" />
...
</ItemGroup>
</Project>
Update the DbContext to include the Identity tables
Next step is to configure our DbContext class so that it includes the new tables that we need from the Identity library. So let’s go to VehicleQuotes/Data/VehicleQuotesContext.cs and update it to do so.
We need to include these new using statements at the top of the file so that we have access to the classes that we need from the NuGet packages we just installed:
using Microsoft.AspNetCore.Identity;
using Microsoft.AspNetCore.Identity.EntityFrameworkCore;
Next, instead of DbContext, the VehicleQuotesContext class should inherit from IdentityUserContext<IdentityUser>.
IdentityUserContext is a class provided by ASP.NET Core Identity that’s designed so that our DbContext can inherit from it and gain user management functionality. Namely, it includes a new DbSet (and consequently, a table) for holding user accounts (aptly named Users), among other things that we won’t need.
IdentityUserContexthas a more feature rich counterpart calledIdentityDbContext, which also includesDbSets to support roles based authorization. We don’t need all that so we use its simpler cousin. Feel free to explore the source code on GitHub to see all it offers.
The generic type parameter that we give it, IdentityUser, is a class that’s also provided by the Identity library. Its purpose is to serve as a default Entity Type for our user model and, as a consequence, our Users DbSet.
In summary, by having our DbContext class inherit from IdentityUserContext<IdentityUser>, we’re telling Identity that we want it to augment our DbContext (and database) to include the core user management tables and that we want our users table to have the same columns as IdentityUser.
If we wanted to include more columns in our users table, what we would have to do is create a new class, make it inherit from
IdentityUser, define any additional fields that we want on it, and use that class as a type parameter toIdentityUserContext. For us for now, the default works just fine. You can learn more about customizing Identity in the official docs.
The change looks like this:
-public class VehicleQuotesContext : DbContext
+public class VehicleQuotesContext : IdentityUserContext<IdentityUser>
Finally, IdentityUserContext has some logic that it needs to run when it is being created. In order to allow it to run that logic, let’s add the following line to our VehicleQuotesContext’s OnModelCreating method:
protected override void OnModelCreating(ModelBuilder modelBuilder)
{
+ base.OnModelCreating(modelBuilder);
// ...
}
This calls IdentityUserContext’s own OnModelCreating implementation so that it can set itself up properly.
The complete file should be looking like this now:
using Microsoft.AspNetCore.Identity;
using Microsoft.AspNetCore.Identity.EntityFrameworkCore;
using Microsoft.EntityFrameworkCore;
using VehicleQuotes.Models;
namespace VehicleQuotes
{
public class VehicleQuotesContext : IdentityUserContext<IdentityUser>
{
public VehicleQuotesContext (DbContextOptions<VehicleQuotesContext> options)
: base(options)
{
}
public DbSet<Make> Makes { get; set; }
public DbSet<Size> Sizes { get; set; }
public DbSet<BodyType> BodyTypes { get; set; }
public DbSet<Model> Models { get; set; }
public DbSet<ModelStyle> ModelStyles { get; set; }
public DbSet<ModelStyleYear> ModelStyleYears { get; set; }
public DbSet<QuoteRule> QuoteRules { get; set; }
public DbSet<QuoteOverride> QuoteOverides { get; set; }
public DbSet<Quote> Quotes { get; set; }
protected override void OnModelCreating(ModelBuilder modelBuilder)
{
base.OnModelCreating(modelBuilder);
modelBuilder.Entity<Size>().HasData(
new Size { ID = 1, Name = "Subcompact" },
new Size { ID = 2, Name = "Compact" },
new Size { ID = 3, Name = "Mid Size" },
new Size { ID = 5, Name = "Full Size" }
);
modelBuilder.Entity<BodyType>().HasData(
new BodyType { ID = 1, Name = "Coupe" },
new BodyType { ID = 2, Name = "Sedan" },
new BodyType { ID = 3, Name = "Hatchback" },
new BodyType { ID = 4, Name = "Wagon" },
new BodyType { ID = 5, Name = "Convertible" },
new BodyType { ID = 6, Name = "SUV" },
new BodyType { ID = 7, Name = "Truck" },
new BodyType { ID = 8, Name = "Mini Van" },
new BodyType { ID = 9, Name = "Roadster" }
);
}
}
}
Applying database changes
The Users DbSet that we added into our data model by inheriting from IdentityUserContext<IdentityUser> will become a table once we create and apply a database migration. That’s what we will do next.
That’s simple in ASP.NET Core. All we need to do is run a command like this:
dotnet ef migrations add AddIdentityTables
That should produce a new migration file named something like 20220605003253_AddIdentityTables.cs. If you explore it, you’ll see how it contains the definitions for a few new database tables. Including the one that we want: AspNetUsers. That’s the one that we will use to store our user account records.
Next, we apply the migration with:
dotnet ef database update
If you now connect to the database, you’ll see the new tables in there.
If you have the database running in a Docker container like we discussed in the beginning of the article, you should be able to connect to it with:
$ psql -h localhost -U vehicle_quotes
Then, to see the tables:
vehicle_quotes=# \dt
List of relations
Schema | Name | Type | Owner
--------+-----------------------+-------+----------------
public | AspNetUserClaims | table | vehicle_quotes
public | AspNetUserLogins | table | vehicle_quotes
public | AspNetUserTokens | table | vehicle_quotes
public | AspNetUsers | table | vehicle_quotes
...
(14 rows)
And that’s pretty much it when it comes to having a sensible storage for user accounts. That was pretty inexpensive, wasn’t it? All we had to do was install some NuGet packages, tweak our existing DbContext, and run some migrations.
The best thing is that we’re not done yet. We can also take advantage of ASP.NET Core Identity to manage users programmatically. Instead of interacting with these tables directly, we will use the service classes provided by Identity to create new users, fetch existing ones and validate their credentials.
Before we can do that though, we must first do some configuration in our app’s Startup class so that said services are properly set up to our liking and are made available to our application.
Configuring the Identity services
We need to add some code to VehicleQuotes/Startup.cs. With it, we configure the Identity system and add a few service classes to ASP.NET Core’s IoC container so that they are available to our app via Dependency Injection.
We need a new using statement:
using Microsoft.AspNetCore.Identity;
And the following code added to the ConfigureServices method:
services
.AddIdentityCore<IdentityUser>(options => {
options.SignIn.RequireConfirmedAccount = false;
options.User.RequireUniqueEmail = true;
options.Password.RequireDigit = false;
options.Password.RequiredLength = 6;
options.Password.RequireNonAlphanumeric = false;
options.Password.RequireUppercase = false;
options.Password.RequireLowercase = false;
})
.AddEntityFrameworkStores<VehicleQuotesContext>();
The call to AddIdentityCore makes several Identity utility classes available to the application. Among those, UserManager is the only one we will use. We will use it later to… well, manage users. You can also see how we’ve set a few options related to how the user accounts are handled. options.SignIn.RequireConfirmedAccount controls whether new accounts need to be confirmed via email before they are available. With options.User.RequireUniqueEmail, we tell Identity to enforce uniqueness of emails on user accounts. And finally the options.Password.* options configure the password strength requirements.
You can explore all the available options in the official docs.
Then, the call to AddEntityFrameworkStores tells the Identity system that it should use our VehicleQuotesContext for data storage.
Creating users
With that configuration out of the way, we can now write some code to create new user accounts. To keep things simple, we’ll add a new UsersController to our project that will expose a new endpoint that offers that functionality.
Let’s start with this in VehicleQuotes/Controllers/UsersController.cs:
using Microsoft.AspNetCore.Identity;
using Microsoft.AspNetCore.Mvc;
namespace VehicleQuotes.Controllers
{
[Route("api/[controller]")]
[ApiController]
public class UsersController : ControllerBase
{
private readonly UserManager<IdentityUser> _userManager;
public UsersController(
UserManager<IdentityUser> userManager
) {
_userManager = userManager;
}
}
}
As you can see, this controller defines a dependency on UserManager<IdentityUser>, an instance of which is injected via the constructor. This is one of the classes made available to us when we configured the Identity core services in our app’s Startup. We will use it to create new user records.
Before that though, we need to define a new class that encapsulates the payload for a request to our endpoint. When it comes to creating user accounts, all we need is a username, a password, and an email. As such, we add the following class in a new VehicleQuotes/ResourceModels/User.cs file:
using System.ComponentModel.DataAnnotations;
namespace VehicleQuotes.ResourceModels
{
public class User
{
[Required]
public string UserName { get; set; }
[Required]
public string Password { get; set; }
[Required]
public string Email { get; set; }
}
}
Now, we shall use this type as the parameter for a new PostUser action method in the UserController which will expose the new user account creation endpoint in our API. The method looks like this:
// POST: api/Users
[HttpPost]
public async Task<ActionResult<User>> PostUser(User user)
{
if (!ModelState.IsValid)
{
return BadRequest(ModelState);
}
var result = await _userManager.CreateAsync(
new IdentityUser() { UserName = user.UserName, Email = user.Email },
user.Password
);
if (!result.Succeeded)
{
return BadRequest(result.Errors);
}
user.Password = null;
return Created("", user);
}
Be sure to also add the following using statements:
using System.Threading.Tasks;
using VehicleQuotes.ResourceModels;
System.Threading.Tasks allows us to reference the class Task<> which is the return type of the new PostUser method. VehicleQuotes.ResourceModels is where our new User class lives.
Thanks to the instance of UserManager<IdentityUser> that we’re holding onto, this method is very straightforward. The most interesting portion is this:
var result = await _userManager.CreateAsync(
new IdentityUser() { UserName = user.UserName, Email = user.Email },
user.Password
);
Here we’re newing up an IdentityUser instance using the given request parameters and passing it on to UserManager<IdentityUser>’s CreateAsync method, along with the password that was also given in the incoming request. This puts the Identity system to work for us and properly create a new user account.
Then, we can inspect its return value (which we capture in the result variable) to determine if the operation was successful. That way we can respond appropriately to our API’s caller.
Finally, with…
user.Password = null;
return Created("", user);
we return the data representing the newly created user account but we’re discreet and make sure not to include the password.
With that, we can test our API. Fire it up with dotnet run (or dotnet watch!) and send a POST request to our new endpoint at http://0.0.0.0:5000/api/Users with a payload like this:
{
"userName": "newuser000",
"email": "newuser000@endpointdev.com",
"password": "password"
}
I just tried it in Postman and this is what it looked like:
Feel free to test it out further. Try repeated emails or usernames. Try passwords that don’t meet the criteria we defined when configuring Identity in the app’s Startup class. It all works as you’d expect.
You can inspect the database and see the newly created record too:
vehicle_quotes=# select id, user_name, email, password_hash from "AspNetUsers";
-[ RECORD 1 ]-+--------------------------------------
id | aaad07b4-f109-4255-8caa-185fd7694c72
user_name | newuser000
email | newuser000@endpointdev.com
password_hash | AQAAAAEAACcQAAAAEJJTV7M2Ejqd3K3iC...
And there it is, that’s our record. With a hashed password and everything.
Now let’s see what would an endpoint to fetch users look like:
Fetching users
UserManager<IdentityUser> offers a FindByNameAsync method that we can use to retrieve users by name. We can add a new endpoint that leverages it like so:
// GET: api/Users/username
[HttpGet("{username}")]
public async Task<ActionResult<User>> GetUser(string username)
{
IdentityUser user = await _userManager.FindByNameAsync(username);
if (user == null)
{
return NotFound();
}
return new User
{
UserName = user.UserName,
Email = user.Email
};
}
This is even more straightforward than the previous one. We just call the FindByNameAsync method by giving it the username of the account we want, make sure that we actually found it, and then return the data.
For the response, we use the same User class that we created to represent the input for the creation endpoint. If we added more fields to the user profile, we could include them here. Alas, we only have username and email for now.
Restart the app and we can now make a request like this:
Pretty neat, huh? Try a username that does not exist and you should see the API respond with a 404.
One quick improvement that we can do before we move on: let’s change
PostUser’s return statement to this:return CreatedAtAction("GetUser", new { username = user.UserName }, user);All that does is include a
Locationheader on the POST Users endpoint’s response that contains the URL for the newly created user. That’s just being a good citizen.
Implementing JWT Bearer Token authentication
Now that we have the user management capabilities that we need, let’s implement some actual authentication. We will start by adding JWT-based authentication to our API.
The strategy that we will use is to create a new API endpoint that clients can POST credentials to and that will respond to them with a fresh, short-lived token. They will then be able to use that token for subsequent requests by including it via headers.
We will pick a random endpoint to secure, just to serve as an example. That is, an endpoint that requires authentication in order to be accessed. GET api/BodyTypes is a good candidate. It is defined in VehicleQuotes/Controllers/BodyTypesController.cs’s GetBodyTypes action method. Feel free to test it out.
Creating tokens
Let’s start by creating the new endpoint that clients will call to obtain the auth tokens. In order to do so, we need a few things:
- A class that represents incoming request data.
- A class that represents outgoing response data.
- A class that can generate tokens for a given user.
- A new action method in
UsersControllerthat does the work.
The request data structure
To deal with step one, let’s define a new class in VehicleQuotes/ResourceModels/AuthenticationRequest.cs that looks like this:
using System.ComponentModel.DataAnnotations;
namespace VehicleQuotes.ResourceModels
{
public class AuthenticationRequest
{
[Required]
public string UserName { get; set; }
[Required]
public string Password { get; set; }
}
}
All users need to authenticate is provide a set of credentials: username and password. So we have this class that contains fields for both and that will be used as input for our endpoint.
The response data structure
Next, we need to define a class to represent that endpoint’s response. I’ve added the following class in VehicleQuotes/ResourceModels/AuthenticationResponse.cs:
using System;
namespace VehicleQuotes.ResourceModels
{
public class AuthenticationResponse
{
public string Token { get; set; }
public DateTime Expiration { get; set; }
}
}
Just a simple data structure containing the token itself and a date letting clients know when they can expect it to expire.
The class that creates JWTs
Step 3 is creating a class that can produce the tokens. This is the most interesting part in terms of complexity. Before we can do that though, let’s add the following configurations to VehicleQuotes/appsettings.json:
"Jwt": {
"Key": "this is the secret key for the jwt, it must be kept secure",
"Issuer": "vehiclequotes.endpointdev.com",
"Audience": "vehiclequotes.endpointdev.com",
"Subject": "JWT for vehiclequotes.endpointdev.com"
},
These are values that we’ll need when creating the tokens. We’ll go over the purpose of each them as they come up as we continue writing our code.
Here we’ll gloss over some details on the inner workings of JWTs as a standard. You can learn more about them at jwt.io.
For now, let’s add the following class in VehicleQuotes/Services/JwtService.cs:
using System;
using System.IdentityModel.Tokens.Jwt;
using System.Security.Claims;
using System.Text;
using Microsoft.AspNetCore.Identity;
using Microsoft.Extensions.Configuration;
using Microsoft.IdentityModel.Tokens;
using VehicleQuotes.ResourceModels;
namespace VehicleQuotes.Services
{
public class JwtService
{
private const int EXPIRATION_MINUTES = 1;
private readonly IConfiguration _configuration;
public JwtService(IConfiguration configuration)
{
_configuration = configuration;
}
public AuthenticationResponse CreateToken(IdentityUser user)
{
var expiration = DateTime.UtcNow.AddMinutes(EXPIRATION_MINUTES);
var token = CreateJwtToken(
CreateClaims(user),
CreateSigningCredentials(),
expiration
);
var tokenHandler = new JwtSecurityTokenHandler();
return new AuthenticationResponse {
Token = tokenHandler.WriteToken(token),
Expiration = expiration
};
}
private JwtSecurityToken CreateJwtToken(Claim[] claims, SigningCredentials credentials, DateTime expiration) =>
new JwtSecurityToken(
_configuration["Jwt:Issuer"],
_configuration["Jwt:Audience"],
claims,
expires: expiration,
signingCredentials: credentials
);
private Claim[] CreateClaims(IdentityUser user) =>
new[] {
new Claim(JwtRegisteredClaimNames.Sub, _configuration["Jwt:Subject"]),
new Claim(JwtRegisteredClaimNames.Jti, Guid.NewGuid().ToString()),
new Claim(JwtRegisteredClaimNames.Iat, DateTime.UtcNow.ToString()),
new Claim(ClaimTypes.NameIdentifier, user.Id),
new Claim(ClaimTypes.Name, user.UserName),
new Claim(ClaimTypes.Email, user.Email)
};
private SigningCredentials CreateSigningCredentials() =>
new SigningCredentials(
new SymmetricSecurityKey(
Encoding.UTF8.GetBytes(_configuration["Jwt:Key"])
),
SecurityAlgorithms.HmacSha256
);
}
}
The CreateToken method is the element that’s most worth discussing here. It receives an instance of IdentityUser as a parameter (which, remember, is the entity class that represents our user accounts), uses it to construct a JWT, and returns it within a AuthenticationResponse object (which is what we decided that our endpoint would return). To do so, we use various classes built into .NET.
The main class that represents the JWT is JwtSecurityToken. We new up one of those in the CreateJwtToken method with this call:
new JwtSecurityToken (
_configuration["Jwt:Issuer"],
_configuration["Jwt:Audience"],
claims,
expires: expiration,
signingCredentials: credentials
);
“Issuer” and “Audience” are two important values for how JWTs work. They specify which entity is creating the token (i.e. the “Issuer”) and which entity is the token intended for (i.e. the “Audience”). We use the IConfiguration instance that we got as a dependency to fetch their values from the VehicleQuotes/appsettings.json file.
The “Issuer” and “Audience” parameters in
JwtSecurityToken’s constructor correspond to JWT claimsissandaud, respectively. You can learn more about them and other claims in the RFC.
The next parameter that JwtSecurityToken’s constructor needs is the claims array. In JWT terms, a claim is essentially a statement about the entity for which the token is generated, some data that identifies it. For example, if we’re generating a token for a user, what you would expect to see in such a token’s claims are things like username, email, and any other non-secret profile info.
In our case, as you can see in the CreateClaims method, we add a number of claims:
new[] {
new Claim(JwtRegisteredClaimNames.Sub, _configuration["Jwt:Subject"]),
new Claim(JwtRegisteredClaimNames.Jti, Guid.NewGuid().ToString()),
new Claim(JwtRegisteredClaimNames.Iat, DateTime.UtcNow.ToString()),
new Claim(ClaimTypes.NameIdentifier, user.Id),
new Claim(ClaimTypes.Name, user.UserName),
new Claim(ClaimTypes.Email, user.Email)
};
Along with the user id, name and email, we also add sub, jti and iat claims. These are standardized claims of which you can learn more about in the RFC. The data that we put in here will make its way into the encoded token that our API caller eventually sees. We’ll see that later.
The next parameter to JwtSecurityToken is the expiration date of the token. Here we are setting it to just one minute in the future to comply with our original requirement that the token should be short-lived so that it only allows a handful of requests over a short period of time.
Finally there’s the signingCredentials parameter which tells the JwtSecurityToken how to cryptographically sign the token. As you can see in the code:
new SigningCredentials(
new SymmetricSecurityKey(
Encoding.UTF8.GetBytes(_configuration["Jwt:Key"])
),
SecurityAlgorithms.HmacSha256
);
That’s a SigningCredentials instance that we create using the “key” that we have configured in VehicleQuotes/appsettings.json, along with the algorithm to use to produce it.
And that’s about it. There isn’t much else to this class. It is somewhat involved but that’s just how JWTs are created in .NET.
The action method that puts it all together
Now all we need to do is actually create an endpoint that exposes this functionality to clients. Since we have the core logic encapsulated in our JwtService class, the actual action method is simple. Here are the changes that we need to make in order to implement it:
First, on VehicleQuotes/Controllers/UsersController.cs, we add a new using statement so that we can reference our new JwtService class:
using VehicleQuotes.Services;
Next, we declare a new parameter of type JwtService in the controller’s constructor so that we signal to ASP.NET Core’s Dependency Injection subsystem that we want to use one of those. We also hold onto it via a new instance variable called _jwtService:
// ...
public class UsersController : ControllerBase
{
private readonly UserManager<IdentityUser> _userManager;
+ private readonly JwtService _jwtService;
public UsersController(
UserManager<IdentityUser> userManager,
+ JwtService jwtService
) {
_userManager = userManager;
+ _jwtService = jwtService;
}
// ...
}
We also need to tell ASP.NET Core that JwtService should be available for Dependency Injection. To do so, we can add this line in VehicleQuotes/Startup.cs’s ConfigureServices method:
services.AddScoped<Services.JwtService>();
Note that this way of defining dependencies is not recommended and only done this way here to keep things simple for an illustrative app that’s never going to run in production. Ideally what you want to do here is define the dependency as an abstraction (i.e. an interface) and a concrete implementation that fulfills it. For example:
services.AddScoped<ITokenCreationService, JwtService>();This way, classes that depend on this service can reference the interface, not the concrete type. In doing that, they adhere to the Dependency Inversion principle and become more easily testable because they allow mocks to be provided as dependencies.
Finally, we write the actual action method:
// POST: api/Users/BearerToken
[HttpPost("BearerToken")]
public async Task<ActionResult<AuthenticationResponse>> CreateBearerToken(AuthenticationRequest request)
{
if (!ModelState.IsValid)
{
return BadRequest("Bad credentials");
}
var user = await _userManager.FindByNameAsync(request.UserName);
if (user == null)
{
return BadRequest("Bad credentials");
}
var isPasswordValid = await _userManager.CheckPasswordAsync(user, request.Password);
if (!isPasswordValid)
{
return BadRequest("Bad credentials");
}
var token = _jwtService.CreateToken(user);
return Ok(token);
}
Here too we’re leveraging the UserManager instance that ASP.NET Core Identity so graciously provided us with. In summary, we find the user account by name using the incoming request data, check if the given password is correct, then ask our JwtService to create a token for this user, and finally return it wrapped in a 200 response.
If you hit that endpoint with a POST and a set of existing credentials, you should see something like this:
If you take that big string that came back in the "token" field in the response JSON, and paste it in jwt.io’s token decoder, you should be able to see all the claims that we added to the token:
Notice that the keywords here are “Encoded” and “Decoded”. The claims that we put in our JWTs are not protected in any way. As such, we should never put secrets in there.
Securing an endpoint with JWT authentication
Now that we have a way for obtaining tokens, let’s see how we can actually use them to gain access to some resources. To demonstrate that, let’s secure an endpoint in a way that it denies unauthenticated requests.
Applying the Authorize attribute
First we need to signal ASP.NET Core that the endpoint requires auth. We do that by annotating the corresponding action method with the Authorize attribute. We’ve already decided that we were going to use VehicleQuotes/Controllers/BodyTypesController.cs’s GetBodyTypes method as a guinea pig. So, let’s go into that file and add the following using statement:
using Microsoft.AspNetCore.Authorization;
That will allow us access to the attribute, which we can apply to the action method like so:
// GET: api/BodyTypes
+ [Authorize]
[HttpGet]
public async Task<ActionResult<IEnumerable<BodyType>>> GetBodyTypes()
{
return await _context.BodyTypes.ToListAsync();
}
With this, we’ve told ASP.NET Core that we want it to require auth for this endpoint.
Enabling and configuring the JWT authentication
Now, we need to tell it how to actually perform the check. To do that, we need to install the Microsoft.AspNetCore.Authentication.JwtBearer NuGet package. We can do that from the VehicleQuotes directory with the following command:
$ dotnet add package Microsoft.AspNetCore.Authentication.JwtBearer
Once we have that installed, we get access to additional services which we can use to configure the “JwtBearer” authentication scheme. As usual, we do the configuration in VehicleQuotes/Startup.cs’s. Here’s what we need to do:
Add a few new using statements:
using Microsoft.AspNetCore.Authentication.JwtBearer;
using Microsoft.IdentityModel.Tokens;
using System.Text;
Add the following code at the end of the ConfigureServices method:
services
.AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
.AddJwtBearer(options =>
{
options.TokenValidationParameters = new TokenValidationParameters()
{
ValidateIssuer = true,
ValidateAudience = true,
ValidateLifetime = true,
ValidateIssuerSigningKey = true,
ValidAudience = Configuration["Jwt:Audience"],
ValidIssuer = Configuration["Jwt:Issuer"],
IssuerSigningKey = new SymmetricSecurityKey(
Encoding.UTF8.GetBytes(Configuration["Jwt:Key"])
)
};
});
The call to AddAuthentication includes all the internal core service classes that are needed to do authentication in our app.
The JwtBearerDefaults.AuthenticationScheme parameter that we give it is the name of the authentication scheme to use as the default. More on that later. For now, know that ASP.NET Core supports multiple authentication schemes to be used at the same time. In fact, our own plan here is to eventually support two auth schemes: JWTs and API Keys. We’re starting with JWT first and as such, that’s the one we specify as the default.
The call to AddJwtBearer configures the JWT authentication scheme. That is, it allows the app to perform authentication checks based on an incoming JWT token.
The most important part of the configuration is the values that we are passing to TokenValidationParameters. As you can see, we are able to specify which aspects of the incoming JWTs to validate. You can see all available options for TokenValidationParameters in the official documentation. Here we’ve chosen to validate obvious things like the issuer, audience, and signing key.
The last configuration step is to add this line right before app.UseAuthorization(); in the Configure method:
app.UseAuthentication();
That will enable the authentication middleware. That way the framework will perform authentication checks as part of the request processing pipeline. In other words, actually put all the configuration that we’ve done to good use.
Alright, now that we have all the configuration ready and have secured an endpoint, let’s try hitting it with a GET on api/BodyTypes and see what happens:
Ok! So far so good. We made an unauthenticated request into an endpoint that requires authentication and as a result we got a 401 back. That’s just what we wanted.
Now, let’s get a token by POSTing to api/Users/BearerToken:
We can copy that token and include as a header in the GET request to api/BodyTypes. The header key should be Authorization and the value should be Bearer <our token>. In Postman, we can setup a similar request if we choose the “Authorization” tab, select “Bearer Token” in the “Type” drop-down list and paste the token in the “Token” text box.
Do that, and you should now see a 200 response from the endpoint:
Neat! Now that we passed a valid token, it wants to talk to us again. Let a few minutes pass, enough for the token to expire and try the request again to see how it’s 401’ing again.
Implementing API Key authentication
Now let’s change gears from JWT and implement an alternate authentication strategy in our Web API: API Keys. Authentication with API Keys is fairly common in the web service world.
The core idea of API Keys is that the API provider (in this case, us) produces a secret string that is given to the clients for safekeeping. The clients then can use that key to access the API and make as many requests as they want. So it’s essentially just a glorified password.
The main functional differences when it comes to JWT as we have implemented it is that the API Keys won’t expire and that we will store them in our database. Then, when processing incoming requests, as part of the authentication step, our app will check if the given API Key exists in our internal records. If it does, then the request is allowed to go through, if it does not, then a 401 is sent back.
This is going to be interesting because ASP.NET Core does not include an authentication scheme out of the box for this kind of behavior. For JWT, we were able to use the Microsoft.AspNetCore.Authentication.JwtBearer package. For this we have to get our hands dirty and actually implement a custom authentication handler that will run the logic that we need.
Let’s get started.
Creating API Keys
Similarly to when we implemented JWTs, we’ll start by creating the new endpoint that clients will call to obtain their keys. In order to make it work, we’ll need:
- A new model and database table to store the keys
- A service class to create the keys.
- A new action method in
UsersControllerthat does the work.
The API Key model and database table
For storing API Keys, we just need a simple table that is linked to users via a one-to-many relationship and includes a name and a value. Using Entity Framework, our model could look like this in VehicleQuotes/Models/UserApiKey.cs:
using System.ComponentModel.DataAnnotations;
using System.Text.Json.Serialization;
using Microsoft.AspNetCore.Identity;
using Microsoft.EntityFrameworkCore;
namespace VehicleQuotes.Models
{
[Index(nameof(Value), IsUnique = true)]
public class UserApiKey
{
[JsonIgnore]
public int ID { get; set; }
[Required]
public string Value { get; set; }
[JsonIgnore]
[Required]
public string UserID { get; set; }
[JsonIgnore]
public IdentityUser User { get; set; }
}
}
Easy enough. We express the relationship between the “keys” and “users” tables via the User navigation property and the UserID field. We also define a unique index on the Value field and annotate some fields with Required because we don’t want them to allow null. Interestingly, we also annotated some of the fields with the JsonIgnore attribute so that when we include objects of this type in API responses (which, as you’ll see, we will), those fields are not included.
We also need to add a new DbSet on our DbContext class at VehicleQuotes/Data/VehicleQuotesContext.cs so that Entity Framework picks it up as part of the data model:
public DbSet<UserApiKey> UserApiKeys { get; set; }
With that, let’s now create and run a migration so that the new table can be added to the database. All it takes is running this to create the migration:
dotnet ef migrations add AddUserApiKeysTable
And this to apply it:
dotnet ef database update
Now the new table should appear in the database:
vehicle_quotes=# \dt
List of relations
Schema | Name | Type | Owner
--------+-----------------------+-------+----------------
...
public | user_api_keys | table | vehicle_quotes
(15 rows)
The class that creates API Keys
The next step is very straightforward. All we need is some logic that, when given a user account (in the form of an instance of IdentityUser), can generate a new API Key, insert it in the database, and return it. The following unremarkable class, defined in VehicleQuotes/Services/ApiKeyService.cs, encapsulates said logic:
using System;
using Microsoft.AspNetCore.Identity;
using VehicleQuotes.Models;
namespace VehicleQuotes.Services
{
public class ApiKeyService
{
private readonly VehicleQuotesContext _context;
public ApiKeyService(VehicleQuotesContext context)
{
_context = context;
}
public UserApiKey CreateApiKey(IdentityUser user)
{
var newApiKey = new UserApiKey
{
User = user,
Value = GenerateApiKeyValue()
};
_context.UserApiKeys.Add(newApiKey);
_context.SaveChanges();
return newApiKey;
}
private string GenerateApiKeyValue() =>
$"{Guid.NewGuid().ToString()}-{Guid.NewGuid().ToString()}";
}
}
As you can see, this class’s single public method, aptly named CreateApiKey, just takes the given user and creates a new UserApiKey that’s associated with it and saves it into the database. The key values themselves are just two GUIDs concatenated.
This is an admittedly simplistic and not-all-that-secure method for generating the keys themselves. For simplicity, we’ve gone with this. In a production app however, it’d be better to use a true cryptographically secure string. Here’s an article that explains how to do that with .NET.
Naturally, this class needs to be used elsewhere in the application, so let’s make it available for Dependency Injection by adding the following line to VehicleQuotes/Startup.cs’s ConfigureServices method.
services.AddScoped<Services.ApiKeyService>();
The action method that puts it all together
Now let’s finally write a new action method on VehicleQuotes/Controllers/UsersController.cs so that there’s an endpoint that clients can call to get API Keys. Before adding the action method though, as usual, we need to declare the following dependency as a constructor parameter and hold it in an instance variable:
public class UsersController : ControllerBase
{
private readonly UserManager<IdentityUser> _userManager;
private readonly JwtService _jwtService;
+ private readonly ApiKeyService _apiKeyService;
public UsersController(
UserManager<IdentityUser> userManager,
JwtService jwtService,
+ ApiKeyService apiKeyService
) {
_userManager = userManager;
_jwtService = jwtService;
+ _apiKeyService = apiKeyService;
}}
The action method itself would look like this:
// POST: api/Users/ApiKey
[HttpPost("ApiKey")]
public async Task<ActionResult> CreateApiKey(AuthenticationRequest request)
{
if (!ModelState.IsValid)
{
return BadRequest(ModelState);
}
var user = await _userManager.FindByNameAsync(request.UserName);
if (user == null)
{
return BadRequest("Bad credentials");
}
var isPasswordValid = await _userManager.CheckPasswordAsync(user, request.Password);
if (!isPasswordValid)
{
return BadRequest("Bad credentials");
}
var token = _apiKeyService.CreateApiKey(user);
return Ok(token);
}
This method is very similar to the other one that generates JWTs. Painfully so. In fact the only difference is that this one calls the ApiKeyService’s CreateApiKey method instead of JwtService’s CreateToken, for obvious reasons.
These two are ripe for some refactoring to eliminate duplication but we won’t do that here. Take a look at the finished project on GitHub to see a neat refactoring option.
Anyway, you can now try a POST into api/Users/ApiKey, pass it a valid set of credentials, and you should see it respond with a brand new API Key:
The key can also be found in the database:
vehicle_quotes=# select id, SUBSTRING(value,1,13), user_id from user_api_keys;
id | substring | user_id
----+---------------+--------------------------------------
1 | efd2133b-cc4a | aaad07b4-f109-4255-8caa-185fd7694c72
(1 row)
Securing an endpoint with API Key authentication
Now that users have a way of getting API Keys, we can start securing endpoints with that authentication scheme. In order to acomplish that, we need to take three steps:
- Implement a custom authentication handler that runs the authentication logic.
- Configure the new authentication handler, plugging it into ASP.NET Core’s auth mechanisms.
- Secure certain endpoints via the Authorization attribute, specifying the scheme that we want to use to do so.
We’ll do that next.
Implementing the API Key authentication handler
Like we’ve touched on before, the “authentication handler” is a class that can plug into ASP.NET Core’s authentication middleware and can run the authentication logic for a given authentication scheme. In practical terms, it’s a class that inherits from Microsoft.AspNetCore.Authentication.AuthenticationHandler and implements the HandleAuthenticateAsync method. Ours will live in VehicleQuotes/Authentication/ApiKey/ApiKeyAuthenticationHandler.cs and it looks like this:
using System.Security.Claims;
using System.Text.Encodings.Web;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Authentication;
using Microsoft.AspNetCore.Identity;
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
namespace VehicleQuotes.Authentication.ApiKey
{
class ApiKeyAuthenticationHandler : AuthenticationHandler<AuthenticationSchemeOptions>
{
private const string API_KEY_HEADER = "Api-Key";
private readonly VehicleQuotesContext _context;
public ApiKeyAuthenticationHandler(
IOptionsMonitor<AuthenticationSchemeOptions> options,
ILoggerFactory logger,
UrlEncoder encoder,
ISystemClock clock,
VehicleQuotesContext context
) : base(options, logger, encoder, clock)
{
_context = context;
}
protected override async Task<AuthenticateResult> HandleAuthenticateAsync()
{
if (!Request.Headers.ContainsKey(API_KEY_HEADER))
{
return AuthenticateResult.Fail("Header Not Found.");
}
string apiKeyToValidate = Request.Headers[API_KEY_HEADER];
var apiKey = await _context.UserApiKeys
.Include(uak => uak.User)
.SingleOrDefaultAsync(uak => uak.Value == apiKeyToValidate);
if (apiKey == null)
{
return AuthenticateResult.Fail("Invalid key.");
}
return AuthenticateResult.Success(CreateTicket(apiKey.User));
}
private AuthenticationTicket CreateTicket(IdentityUser user)
{
var claims = new[] {
new Claim(ClaimTypes.NameIdentifier, user.Id),
new Claim(ClaimTypes.Name, user.UserName),
new Claim(ClaimTypes.Email, user.Email)
};
var identity = new ClaimsIdentity(claims, Scheme.Name);
var principal = new ClaimsPrincipal(identity);
var ticket = new AuthenticationTicket(principal, Scheme.Name);
return ticket;
}
}
}
The first thing to note is how we’ve defined the base class: AuthenticationHandler<AuthenticationSchemeOptions>. AuthenticationHandler is a generic, and it allows us to control the type of options object that it accepts. In this case, we’ve used AuthenticationSchemeOptions, which is just a default one already provided by the framework. This is because we don’t really need any custom options.
If we did, then we would define a new class that inherits from
AuthenticationSchemeOptions, give it the new fields that we need, and use that as a generic time parameter instead. We would then be able to set values for these new fields during configuration in the app’sStartupclass. Just like we’ve done with the “Jwt Bearer” auth scheme via theAddJwtBearermethod.
The next point of interest in this class is its constructor. The key aspect of it is that it calls base and passes it a series of parameters that it itself gets. This is because AuthenticationHandler does have some constructor logic that needs to be run for things to work properly, so we make sure to do that.
Finally, there’s the HandleAuthenticateAsync method which does the actual authentication. This method inspects the incoming request looking for an “Api-Key” header. If it finds it, it takes its value and makes a query into the database to try and find a record in the user_api_keys table that matches the incoming value. If it finds that, it creates an AuthenticationTicket which contains the identity of the user that the key belongs to, and returns it wrapped in a AuthenticateResult.Success. That return value signals ASP.NET Core’s authentication middleware that the request is authentic. Otherwise, it returns AuthenticateResult.Fail, which prompts ASP.NET Core to halt the request and return a 401.
Something interesting to note is how the AuthenticationTicket, similarly to the JWT Bearer scheme’s JwtSecurityToken, also includes an array of “claims” that serve to identify the user that has been authenticated. The “claims” concept is central to how auth works in ASP.NET Core.
Enabling and configuring the API Key authentication
Now we need to tell the framework that we want it to use a new authentication scheme for our app, spearheaded by the custom authentication handler that we just wrote. There are two steps to make that happen.
First, we configure our new scheme in the app’s Startup class. For that, we need these two new using statements:
using Microsoft.AspNetCore.Authentication;
using VehicleQuotes.Authentication.ApiKey;
Then we add this right after the AddJwtBearer call in the ConfigureServices method:
public void ConfigureServices(IServiceCollection services)
{
// ...
services
.AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
.AddJwtBearer(options =>
{
options.TokenValidationParameters = new TokenValidationParameters()
{
ValidateIssuer = true,
ValidateAudience = true,
ValidateLifetime = true,
ValidateIssuerSigningKey = true,
ValidAudience = Configuration["Jwt:Audience"],
ValidIssuer = Configuration["Jwt:Issuer"],
IssuerSigningKey = new SymmetricSecurityKey(
Encoding.UTF8.GetBytes(Configuration["Jwt:Key"])
)
};
})
+ .AddScheme<AuthenticationSchemeOptions, ApiKeyAuthenticationHandler>(
+ "ApiKey",
+ options => { }
+ );
}
A simple configuration as far as auth schemes go. We specify both the types of the authentication handler and the options object that it accepts via the generic type parameters to the AddScheme method. "ApiKey" is just the name we’ve given our custom auth scheme. It can be anything as long as it’s not “Bearer”, which is the name of the Jwt Bearer scheme (which is the value returned by JwtBearerDefaults.AuthenticationScheme). We’ll refer back to it later. Finally, since we have no special options to give it, we specify a lambda that does nothing with its options.
Updating the Authorize attribute to use our two schemes
Now that the auth scheme is configured, we need to use it to secure an endpoint. We can use the same that we used for JWT: POST api/BodyTypes, defined in BodyTypesController’s GetBodyTypes action method. Remember though, that we wanted to have both auth schemes (JWT and API Key) work on the endpoint. For that, the Authorize attribute allows a comma separated string of scheme names as a parameter. So, we can get both our configured schemes working if we update the attribute like this:
[Authorize(AuthenticationSchemes = $"{JwtBearerDefaults.AuthenticationScheme},ApiKey")]
public async Task<ActionResult<IEnumerable<BodyType>>> GetBodyTypes()
{
return await _context.BodyTypes.ToListAsync();
}
JwtBearerDefaults.AuthenticationScheme contains the name of the JWT Bearer auth scheme. Next to it, we just put “ApiKey”, which is the name we have given our new custom one.
It’d be nice to put that “ApiKey” string somewhere safe similar to how the “Jwt Bearer” scheme has it in a constant. Take a look at the finished project on GitHub to see one way to organize that and also how to make the
Startupconfiguration a little less verbose by using extension methods.
And finally, let’s test our endpoint and marvel at the fruit of our work.
Make sure to create a new API Key by POSTing to api/Users/ApiKey, and copy the “value” from the response. We can use it as the value for the Api-Key header a GET request to api/BodyTypes. In Postman, we can do that by choosing the “Authorization” tab, selecting “API Key” in the “Type” drop-down list, writing “Api-Key” in the “Key” text box, and putting our key in the “Value” text box.
With that, we can make the request and see how the endpoint now allows authentication via API Keys:
Of course, requests authenticated via JWT will also work for this endpoint.
That’s all
And there you have it! We’ve updated an existing ASP.NET Core Web API application so that it supports authentication using two strategies: JWT and API Keys. We leveraged the Identity libraries to securely store and manage user accounts. We used ASP.NET Core’s built-in authentication capabilities to enable JWT generation and usage. For API Keys, the framework didn’t provide an implementation out of the box. However, it proved to be extensible enough so that we could implement our own.
Table of contents
authentication security dotnet
3 useful built-in objects and functions in JavaScript
By Phineas Jensen
June 15, 2022
I love learning (and learning about) programming languages. Right now I’m teaching myself Rust and trying to learn about 3D rendering. Every time I find a list of programming languages I have to look up every one that I haven’t heard of, and when I see posts about Zig or Haskell or any other cool project on Hacker News, I can’t help reading through the comments and seeing people discuss the unique features, quirks, and drawbacks of each language.
One thing I enjoy about learning about these languages (sometimes in an all-too-shallow way) is seeing all the different methods and syntaxes that exist for solving problems, and while it’s always tempting to think the grass is greener on the other side, it’s also important to do the same kind of exploration within the languages I’m using right now. Not only is it important, it actually makes using those languages a lot more enjoyable as I find new, more efficient ways to do things I’ve probably done dozens of times before.
With that spirit, here’s a little list of cool objects and functions in JavaScript that have given me that feeling of excitement and made the language more fun and satisfying to use. Like any JavaScript feature, support will vary from browser to browser, runtime to runtime, and version to version, but with tools like Webpack becoming ubiquitous, that’s becoming less of a problem.
The Set object
The Set object has a lot of use cases, but one I find myself using a lot is a set of selected items in a list like this:
Why is Set a good fit for this? Because a Set is a list of unique items (we shouldn’t have one item in the selected list twice), we can add and remove items from it easily, and we can check membership with a single, obvious method that returns a boolean. Let’s look at an example using an array first, which mimics code I’ve seen quite a bit.
const selected = [];
function select(item) {
if (selected.indexOf(item) == -1) {
selected.push(item);
}
}
function deselect(item) {
const index = selected.indexOf(item);
if (index != -1) {
selected.splice(index, 1);
}
}
function isSelected(item) {
return selected.indexOf(item) != -1;
}
This isn’t too complex, and it works well, but it’s pretty verbose. Using the indexOf function three times is a little messy, and the splice function is uncommon enough that it’s easy to forget exactly how it works. We can implement the same functions more simply using a Set:
const selected = new Set();
function select(item) {
selected.add(item);
}
function deselect(item) {
selected.delete(item);
}
function isSelected(item) {
return selected.has(item);
}
In fact, I’d say this functionality is so much simpler that we don’t even need to define new functions. Selecting an item by calling selected.add(...) reads just as well as or better than select(...), and the same goes for these other function names. Set handles it all for us.
Note that if you use a Set in React with useState, the component won’t re-render unless you create a new Set object when you update the state, like so:
function Example(props) {
const [selected, setSelected] = useState();
const addItem = (item) => {
selected.add(item);
setSelected(new Set(selected));
}
return items.map(item => (
<button onClick={() => addItem(item)}>
));
}
This is because React will bail out of the state update if the values are the same according to the Object.is function.
The URL API’s URLSearchParams object
Imagine you’re writing a function that needs to parse and use a URL’s search parameters. Pretty simple task, right? It wouldn’t be too hard to do by hand, but there’s probably a library out there somewhere, so you do a Google search for “javascript parse url parameters” and click on the first Stack Overflow answer in the results. You’re greeted with an answer that defines a function using .split() a few times and then a much longer and more complex function that decodes the parameters in a non-standard way. Is this really necessary?
With the built-in URLSearchParams object, no. Supported in every major browser except Internet Explorer (which just reached its end of support life), this nice object handles parsing and formatting of string parameters with proper encoding and decoding:
const params = new URLSearchParams('?foo=1');
params.append('bar', 'has weird & tricky characters'); // Add another parameter
console.log(params.toString()); // Prints 'foo=1&bar=has+weird+%26+tricky+characters'
It handles multiple parameters with the same key easily and supports iteration:
const params = new URLSearchParams('?foo=1&foo=2&foo=3');
console.log(params.getAll('foo')) // Prints ["1", "2", "3"]
for (const [k, v] of params) {
console.log(`${k}: ${v}`);
}
// Output:
// foo: 1
// foo: 2
// foo: 3
Much nicer, and requires no copy-pasting of code from Stack Overflow or installing dependencies!
The Array iterator functions
If you’ve used a lot of JavaScript or used languages like Perl, Python, Ruby, Rust, functional languages, or languages with iterators, you’ve probably seen functions like map or forEach. I’ve used them pretty extensively but still find that a lot of people don’t seem to know about them. For people coming from languages like C or C++, where they aren’t available, or for people who are fresh out of a university program where neat things like that are often not taught in favor of theory, that’s not much of a surprise.
Array.prototype.forEach is pretty straightforward. Compare the following equivalent code snippets:
const names = ["bob", "roberta", "alice", "reza"];
// This...
for (let i = 0; i < names.length; i++) {
console.log(names[i].substr(0,1).toUpperCase() + names[i].substr(1));
}
// ...is equivalent to this:
names.forEach(value => console.log(value.substr(0,1).toUpperCase() + value.substr(1)));
forEach takes as its argument a callback function, which it then calls once for each item in the array. The callback function can take more arguments than just the value as well; see its MDN page.
I personally find map more interesting and useful. Map is very similar to forEach, except that the values returned by the callback function are assembled to make a new array of the same length as the old one, essentially returning a transformed array.
const names = ["bob", "roberta", "alice", "reza"];
// This...
const uppercaseNames = [];
for (let i = 0; i < names.length; i++) {
const name = names[i];
uppercaseNames.push(name.substr(0,1).toUpperCase() + name.substr(1));
}
// ...is equivalent to this:
const uppercaseNames = names.map(name => name.substr(0,1).toUpperCase() + name.substr(1));
There are similar cool functions that return true or false if the callback function returns true for every or some item in the list:
const names = ["bob", "roberta", "alice", "reza", "spaces are good"];
// Both values will be false because of the "spaces are good" item
const allSpaceless = names.every(name => name.indexOf(" ") === -1);
// or...
const allSpaceless = !names.some(name => name.indexOf(" ") !== -1);
Or we can find individual or multiple items using a function, rather than just by value:
const names = ["bob", "roberta", "alice", "Reza", "spaces are good"];
const namesStartingWithR = names.filter(name => name[0].toLowerCase() === "r") // returns ["roberta", "Reza"]
const firstNameWithC = names.find(name => name.toLowerCase().includes("c")) // returns "alice"
Then there’s the complex but very cool reduce function, which starts with an initial value and replaces it with the result of the callback function on each iteration. map can be implemented using reduce:
const names = ["bob", "roberta", "alice", "reza"];
// This...
const uppercaseNames = names.map(name => name.substr(0,1).toUpperCase() + name.substr(1));
// ...is equivalent to this:
const uppercaseNames = names.reduce(
(previousValue, name) => {
previousValue.push(name.substr(0,1).toUpperCase() + name.substr(1));
return previousValue;
},
[] // Initial value
);
Obviously, it’s not terribly useful in this case, but can be extremely useful for things like summing items in a list or transforming a list into an object somehow.
Consider keeping the Array reference on hand to keep these functions close to your heart and start to learn some of their other features. As new versions of ECMAScript come out, new prototype functions will crop up too. Note that some of these functions will vary in performance compared to normal loop iterations. If you’re using them in heavy workload or low-latency scenarios, make sure to compare and benchmark performance as needed.
Conclusion
Okay, that was actually more than 3 things, but they fit into three categories, so I’m keeping the post’s name. JavaScript’s built-in objects and the generally available Web APIs have a lot more cool APIs and functions and features that are very much worth checking out. MDN (which I linked several times throughout this article) is a fantastic resource to learn about them, so make sure to get familiar with it as you work with JavaScript. It’ll pay off.
javascript tips
End Point booth at CSTE 2022 conference
By Shannon Sandall
June 9, 2022
Converging on Kentucky
We are excited to announce that End Point will be attending the CSTE (Council of State and Territorial Epidemiologists) conference in less than two weeks!
We will be running a booth during the 3-day conference in Louisville, Kentucky from Monday, June 20 through Wednesday, June 22, 2022. Our crew is scheduled to include Steve Yoman, Linda King, Ben Goldstein, and me.
Please come visit us in booth #35 to learn more about the EpiTrax Disease Surveillance System and EMSA (Electronic Message Staging Area). Now is the time for public health entities to turn to modern systems to perform disease surveillance, automate the ingestion of electronic laboratory records (ELRs) and electronic case reports (eCRs), report to the CDC through NMI reporting methods, track outbreaks, perform contact management, and do it all with a highly configurable system.
Our booth will feature a VisionPort Mini to showcase EpiTrax and EMSA! VisionPort is our product combining hardware and software to give organizations the ability to create dynamic, shared immersive multimedia experiences. We are delighted when we have opportunities like this to use VisionPort to inform conference attendees about our own offerings such as EpiTrax and EMSA.
We will be happy to demonstrate EpiTrax to you there. Here is a preview screenshot of the application’s disease outbreaks list view:
CSTE
What is CSTE? The Council of State and Territorial Epidemiologists is an organization that works to advance public health policy and epidemiologic capacity. They also provide information, education, and developmental support of practicing epidemiologists in a wide range of areas as well as expertise for program and surveillance efforts.
Here are some of the conference sessions we plan to attend:
- Monday, 2:00–3:15 PM: In-Person Concurrent Breakout Sessions: Surveillance / Informatics I: Not Your Grandma’s Old-Fashioned: Data Modernization
- Monday, 5:45–6:30 PM: Virtual Concurrent Roundtable Sessions / Networking Surveillance / Informatics II: Electronic Case Reporting and Public Health Agencies—Needs and Directions
- Tuesday, 8:45–10:00 AM: In-Person Concurrent Breakout Sessions: Surveillance / Informatics I: Taking off the Blinders: Innovative Methods for Detection and Surveillance
- Tuesday, 5:45–6:30 PM: In-Person Concurrent Roundtable Sessions / Networking: Surveillance / Informatics I: CSTE National ELR Workgroup Meet and Greet
- Wednesday, 7:30–8:15 AM: In-Person Concurrent Roundtable Sessions / Networking: Surveillance Informatics I: National Notifiable Disease Surveillance System (NNDSS): A Comprehensive Review
Come see what End Point and EpiTrax can do for you! We would love to meet with you in person, a privilege we don’t take for granted these days.
Or if you can’t make the conference, contact us here online.
conference epitrax emsa
Understanding Linear Regression
By Kürşat Kutlu Aydemir
June 1, 2022
Linear regression is a regression model which outputs a numeric value. It is used to predict an outcome based on a linear set of input.
The simplest hypothesis function of linear regression model is a univariate function as shown in the equation below:
$$ h_θ = θ_0 + θ_1x_1 $$
As you can guess this function represents a linear line in the coordinate system. The hypothesis function (h0) approximates the output given input.
θ0 is the intercept, also called bias term. θ1 is the gradient or slope.
A linear regression model can either represent a univariate or a multivariate problem. So we can generalize the equation of the hypothesis as summation:
$$ h_θ = \sum{θ_ix_i} $$
where x0 is always 1.
We can also represent the hypothesis equation with vector notation:
$$ h_θ = \begin{bmatrix} θ_0 & θ_1 & θ_2 \dots θ_n \end{bmatrix} x \begin{bmatrix} x_0 \\ x_1 \\ x_2 \\ \vdots \\ x_n \end{bmatrix} $$
Linear Regression Model
I am going to introduce a linear regression model using a gradient descent algorithm. Each iteration of a gradient descent algorithm calculates the following steps:
- Hypothesis h
- The loss
- Gradient descent update
The gradient descent update iteration stops when it reaches the convergence.
Although I am implementing a univariate linear regression model in this section, these steps apply to multivariate linear regression models as well.
Hypothesis
We start the initial hypothesis assumption with random parameters. Then we calculate the loss using L2 Loss function over the training dataset. In Python:
def hypothesis(X, theta):
return theta[0] + theta[1:] * X
In this function we took X input (univariate in this implementation) and theta parameter values. X represents the feature input of our dataset. Theta is the weights of the features. θ0 is called the bias term and θ1 is the gradient or slope.
L2 Loss Function
L2 Loss function — sometimes called Mean Squared Error (MSE) — is the total error of the current hypothesis over the given training dataset. During the training, by calculating the MSE, we can target minimizing the cumulative error.
$$ J(θ) = \frac{\sum{(h_θ(x_i) - y_i)^2}}{2m} $$
L2 loss function (MSE) simply calculates the error by summing the squares of each data point error by dividing the size of the dataset.
The more the linear function is aligned, the optimized center of the data points with an optimized slope would give us a minimized error which is our target in linear regression training.
Gradients of the Loss
Each time we iterate and calculate a new theta (θ), we get a new theta1 (slope) value. If we plot each slope value in the gradient descent batch update we will have a curve like this:
This curve has a minimum value which can’t get lower. Our goal is to find an optimal low value of theta1 that reaches a point where our curve doesn’t get lower anymore or the change can be ignored. That is where the convergence is achieved and the loss is minimized.
Let’s do a little bit more math. The gradient of the loss is the partial derivative of θ. We calculate partial differential of loss for θ0 and θ1 separately. For multivariate functions our θ1 is a generalized version for all available θi since the partial derivatives are calculated similarly. You can simply calculate the partial derivatives of loss function yourself too.
$$ \frac{∂}{∂θ_0}J(θ_0) = \frac{\sum{(h_0 - y_0)}}{m} $$
$$ \frac{∂}{∂θ_0}J(θ_i) = \frac{\sum{(h_i - y_i)x_i}}{m} $$
Since we know the hypothesis equation we can replace it in the derivatives as well:
def partial_derivatives(h, X, y):
return [np.mean((h.flatten() - y)), np.mean((h.flatten() - y) * X.flatten())]
Now we will calculate the gradients for given hypothesis given theta, X, and y:
def calc_gradients(theta, X, y):
gradient = [0, 0]
h = hypothesis(X, theta)
gradient = partial_derivatives(h, X, y)
return np.array(gradient)
Batch Gradient Descent
The gradient descent method I used in this implementation is called batch gradient descent which uses all the data available through the iterations, which slows down the overall convergence process. There are methods to improve the performance of gradient descent such as stochastic gradient descent.
Since we calculated the gradients for the given theta we will iterate as much as we can until the convergence.
$$ θ_1(new) = θ_1(current) - α * J'(θ_1(current)) $$
Here comes the convergence rate or so called learning rate (α) factor to decide how long we should jump through the iterations. If α is too small, convergence can be more accurate, but the performance will be too small. This also leads to overfitting. If α is too big, the performance will be better, but convergence couldn’t be calculated accurately or well enough.
There is no strict best value for α since it depends on the dataset for training the model. By evaluating the model you trained you can find the best alpha value for your dataset. You can refer to statistical measures like R2 score to determine the observed variance. But there usually won’t be single model parameter, hyperparameter, or statistical variable to refer to for regularization.
def gradient_update(X, y, theta, alpha, stop_threshold):
# initial loss
loss = L2_loss(hypothesis(X, theta), y)
old_loss = loss + stop_threshold
while( abs(old_loss - loss) > stop_threshold ):
# gradient descent update
gradients = calc_gradients(theta, X, y)
theta = theta - alpha * gradients
old_loss = loss
loss = L2_loss(hypothesis(X, theta), y)
print('Gradient Descent training stopped at loss %s, with coefficients: %s' % (loss, theta))
return theta
By performing batch gradient descent we actually train our algorithm and make it find the best theta values to fit the linear function. Now we can evaluate our algorithm and compare it with Sci-Kit Learn Linear Regression.
Evaluation
Since linear regression is a regression model, you should train and evaluate this model on regression datasets.
SK-Learn Diabetes dataset is a good regression dataset example. Below I loaded and prepared the dataset by splitting into training and test datasets.
from sklearn import datasets
from sklearn.model_selection import train_test_split
# Load the diabetes dataset
diabetes = datasets.load_diabetes()
diabetes_X = diabetes.data[:, np.newaxis, 2]
diabetes_y = diabetes.target
X_train, X_test, y_train, y_test = train_test_split(diabetes.data, diabetes_y, test_size=0.1)
Now we can evaluate our model:
from sklearn.metrics import mean_squared_error, r2_score
# initial random theta
theta = [100, 3]
stop_threshold = 0.1
# learning rate
alpha = 0.5
theta = gradient_update(X_train, y_train, theta, alpha, stop_threshold)
y_pred = hypothesis(X_test, theta)
print("Intercept (theta 0):", theta[0])
print("Coefficients (theta 1):", theta[1])
print("MSE:", mean_squared_error(y_test, y_pred))
print("R2 Score", r2_score(y_test, y_pred))
# Plot outputs using test data
plt.scatter(X_test, y_test, color='black')
plt.plot(X_test, y_pred, color='blue', linewidth=3)
plt.show()
When I run my linear regression model it finds the optimal theta values, finishes the training, and outputs as below. I noted sample evaluation scores below too.
Gradient Descent training stopped at loss 3753.11429796413, with coefficients: [151.6166715 850.81024746]
Intercept (theta 0): 151.61667150054697
Coefficients (theta 1): 850.8102474614635
MSE: 5320.89741757879
R2 Score 0.14348916154815183
Now let’s evaluate the SK-Learn linear regression model with the same training and test datasets we used. I’m going to use default parameters without optimizing.
# Sci-Kit Learn LinearRegression model evaluation
regr = linear_model.LinearRegression()
regr.fit(X_train, y_train)
y_pred = regr.predict(X_test)
print("Coef:", regr.coef_)
print("Intercept:", regr.intercept_)
print("MSE:", mean_squared_error(y_test, y_pred))
print("R2 Score", r2_score(y_test, y_pred))
# Plot outputs
plt.scatter(X_test, y_test, color='black')
plt.plot(X_test, y_pred, color='blue', linewidth=3)
plt.show()
The output and plot of the SK-Learn Linear Regression model is as below:
Coef: [993.14228074]
Intercept: 151.5751918329106
MSE: 5544.283378702411
R2 Score 0.10753047228113943
Notice the intercept of my linear regression model and SK-Learn’s linear regression model are very close with value of around ~151. MSE values are calculated very close too. Also both plotted their predictions very similarly as well.
Multivariate Linear Regression
We can add more features as we have more features in a dataset and prepare our hypothesis as below, similar to a univariate hypothesis.
$$ h_θ(x) = θ_0 + θ_1x_1 + … + θ_nx_n $$
A multivariate dataset can have multiple features and a single output like below.
| Feature1 | Feature2 | Feature3 | Feature4 | Target |
|---|---|---|---|---|
| 2 | 0 | 0 | 100 | 12 |
| 16 | 10 | 1000 | 121 | 18 |
| 5 | 5 | 450 | 302 | 14 |
Each feature is an independent variable (xi) of a dataset. Parameters (theta) are what we aim to find during the training just like the univariate model.
Linear Regression with Polynomial Functions
Sometimes a line function doesn’t fit the data well enough. Although if we are dealing with a polynomial function (having multiple features with exponential versions), it could fit the data better.
In this case the data itself is not linear but we are lucky that the parameter space is linear and we can still apply the linear regression over the non-linear dataset as well:
$$ h_θ(x) = θ_0 + θ_1x + θ_1x^2 … + θ_nx^n $$
$$ h_θ = \begin{bmatrix} 1 & x & x^2 \dots x^n \end{bmatrix} x \begin{bmatrix} θ_0 \\ θ_1 \\ θ_2 \\ \vdots \\ θ_n \end{bmatrix} $$
Here the data is non-linear but the parameters are linear and we can still apply the gradient descent algorithm.
Conclusion
In this post I implemented a linear regression model from scratch and evaluated by training it.
Linear regression is useful when your dataset variables can be related in a linear relation. In the real world, linear regression is very useful in forecasting.
machine-learning data-science python