• Home

  • Custom Ecommerce
  • Application Development
  • Database Consulting
  • Cloud Hosting
  • Systems Integration
  • Legacy Business Systems
  • Security & Compliance
  • GIS

  • Expertise

  • About Us
  • Our Team
  • Clients
  • Careers

  • Blog

  • EpiTrax

  • VisionPort

  • Contact
  • Our Blog

    Ongoing observations by End Point Dev people

    Nevada State EpiTrax Launch

    By Katrease Hale
    December 1, 2022

    Weathered red rocks jut out from the desert into the foreground, while a blue haze covers a mountain range in the background. Photo by Adrien Drj

    If COVID-19 has taught us anything, it is that the public health landscape can change quickly, and we need a disease surveillance system that is adaptable to support our ever-evolving climate.

    Having access to surveillance data for purposes of contact tracing, following trends, and monitoring evolving disease conditions allows health departments to be agile in response. This is a critical component in providing communities with a robust public health infrastructure.

    For all these reasons and many more, the State of Nevada embarked on a journey to migrate away from their surveillance system, NBS, to the open-source EpiTrax system created by the Utah Department of Health. Ultimately, the Nevada decisionmakers made this change because they needed to be on one state-wide system and wanted autonomy to customize the system.

    Nevada had been exploring this change for a while but due to unforeseen problems the window of time for implementation was incredibly narrow. In the first four months of End Point’s partnership with Nevada, the team was able to accomplish what was believed to be impossible in so short a time.

    The Four Month Sprint

    Some of the major milestones on the way to the new EpiTrax system included the following:

    Building Out Custom Forms

    EpiTrax is composed of core and form fields. Core fields are consistent across all conditions. Form fields allow jurisdictions to customize the data system to collect data specific to the needs of their area or the reporting requirements of their funders.

    In Nevada, we framed these forms to collect the data outlined in the Message Mapping Guide set forth by the CDC. Collecting form data is the first step in configuring the system to use the NMI module which allows for seamless reporting.

    Configuring the System

    EpiTrax is an intuitive system that is highly configurable. For example, EpiTrax will show specific fields by condition or condition type, allowing an investigator working on a COVID record to only see vaccines and treatments for COVID which reduces time and error in data entry.

    These parameters need to be defined and set up in the system for a large number of items. End Point was able to provide templates and examples from other states and having a framework for this documentation was vital.

    Testing the System

    Testing sounds easy: “You just log in and make sure the functionality is there.” However, to adequately test a new environment with multiple jurisdictions and a variety of user groups takes time, attention, and follow-through.

    Nevada worked with End Point to bring in users from across the state to test both the functionality of the system and disease-specific variables. Then the test failures were discussed daily and the programmers provided options and support for resolving problems. After modifications were made to the environment the second round of testing was implemented.

    Given the tight window of time to launch EpiTrax we were building (testing) the ship as we were sailing it and this was only possible given the commitment of everyone on the team.

    Training

    You cannot launch a statewide system without training the end users, answering questions, and providing a mechanism for ongoing support. End Point provided countless trainings ranging from general use to administrative functionality of the system, and also provided Nevada with a user manual template that they were able to build on and grow to fit their needs.

    Electronic Messaging Staging Area (EMSA)

    EMSA is an application that processes lab messages and delivers them into EpiTrax. The delivery of these messages is what creates the record for investigation in EpiTrax. Since this was a new process and system to Nevada, End Point worked with the Nevada team to configure the various vocabularies and rules.

    Going Live

    On September 12 EpiTrax officially launched in Nevada. End Point staff were there both in person and with a consistent virtual presence to work through user concerns, login issues, and workflow questions.

    Conclusion

    The customization and scope of EpiTrax are crucial components in allowing Nevada to optimize work processes, streamline the multiple data systems being used into one cohesive system, and greatly reduce redundancy.

    While switching systems can seem like a daunting and time-consuming task, so is continuing to operate in crisis mode with a system that stops your team from being dynamic. Any change for the better involving hundreds of people and a lot of data is going to take time, normally much longer than 4 months.

    The partnership between Nevada and End Point during this fast-paced launch has better positioned the state to use their data to support well-coordinated public health responses to emerging diseases or outbreaks.


    epitrax emsa clients

    A/B Testing

    Kürşat Kutlu Aydemir

    By Kürşat Kutlu Aydemir
    November 28, 2022

    A chemist in complete PPE holds two test tubes holding green liquid.
    Photo by Mikhail Nilov

    In statistics, A/B testing is “an experiment with two groups to establish which of two treatments, products, procedures, or the like is superior. Often one of the two treatments is the standard existing treatment, or no treatment” (Bruce 2020, 88).

    A/B testing is very useful when adapted to e-commerce and marketing for determining the better of two options for a webpage.

    Let’s consider a website where we want to analyze the page visits of page A and page B. Page A is the existing page (the control group), and page B is a new design of the web page (the treatment group).

    To prepare A/B testing we start with the following steps:

    1. Define hypotheses: null hypothesis (H0) and alternative hypothesis.
    2. Prepare control and treatment groups.

    Then we’ll apply the A/B test on the dataset.

    Purpose

    The new and existing versions of our web page can show different performance in terms of marketing, visitor attention, and “conversion” to a particular goal. By applying A/B tests we can understand which of the two web pages has better performance. We can also find out if any difference in performance is due to chance or due to a design change.

    Example

    For example, let’s say we want to compare the conversion rates for visitors of page A and page B. Here are the aggregated results of our collected data for both pages:

    Page A

    Conversions: 231
    Non-conversion visits: 11779

    Page B

    Conversions: 196
    Non-conversion visits: 9823

    The data to be sampled can vary. A very simple dataset could be a list of numeric results of observations: spent time on a control and a treatment web page, the weight of patients with and without treatments, etc. The permutation sampling of these results would be simply picking the values from the combined list of both groups.

    In our conversion example the nature of the groups is similar but the values are slightly different, as they store two states: conversion and not conversion. However, we can still use a simple statistic like the mean to make our permutation test calculation.

    Null hypothesis

    Let’s build our null hypothesis around our experiment. In this example the null hypothesis would be “conversion rate of A ≥ conversion rate of B”. So, the alternative hypothesis would be “conversion rate of B > conversion rate of A”.

    In a significance test we usually aim to answer the question of whether we will reject the null hypothesis or fail to reject the null hypothesis.

    Data preprocessing and clean-up

    Since the conversion/not conversion results are just yes/no (binary) results we might prefer to store them as values of 1 or 0.

    We will use Python with the pandas library to import conversion samples for Pages A and B as a pandas Series:

    import pandas as pd
    import random
    
    conversion = [1] * 427
    conversion.extend([0] * 21602)
    random.shuffle(conversion)
    conversion = pd.Series(conversion)
    

    Permutation test

    To determine if the difference between the control and treatment groups is due to chance or not we can use permutation testing.

    Permutation sampling is performed using the following steps, as listed in Practical Statistics (see the Reference section at the end):

    1. Combine the results from the different groups into a single data set.
    2. Shuffle the combined data and then randomly draw (without replacement) a resample of the same size as group A (which will contain some data from multiple groups).
    3. From the remaining data, randomly draw (without replacement) a resample of the same size as group B.
    4. Do the same for groups C, D, and so on. You have now collected one set of resamples that mirror the sizes of the original samples.
    5. Whatever statistic or estimate was calculated for the original samples (in our case, the percent difference between two groups), calculate it now for the resamples, and record. This constitutes one permutation iteration.
    6. Repeat the previous steps R times to yield a permutation distribution of the test statistic.

    For each step of the permutation test we need to calculate the percent difference between successful and unsuccessful conversions. Using our example aggregated data from above, we can do this like so:

    observed_percentage_diff = 100 * (196 / (196 + 9823) - 231 / (231 + 11779))
    

    Which results in 0.032885893156042734 for our observation.

    Below is a simple permutation test implementation from Practical Statistics.

    def perm_fun(x, nA, nB):
        n = nA + nB
        idx_B = set(random.sample(range(n), nB))
        idx_A = set(range(n)) - idx_B
        return x.loc[list(idx_B)].mean() - x.loc[list(idx_A)].mean()
    

    Now we need to repeat the sampling with a high R value, we’ll use 5000.

    R = 5000
    perm_diffs = [100 * perm_fun(conversion, 12010, 10019) for _ in range(R)]
    

    Results

    perm_diffs holds the percentage differences of the sampling data we generated. Now we have permutation test sampling differences and observed difference to be plotted.

    import matplotlib.pyplot as plt
    
    plt.hist(perm_diffs, bins=15)
    plt.axvline(x = observed_percentage_diff, color='black', lw=2)
    plt.show()
    

    The histogram from permutation testing results. The y-axis is labeled “frequency”. The x-axis spans from -0.6 to 0.8. There is a normal distribution peaking at x = -0.1, with y roughly equaling 1000. The lowest frequency values are at -0.6 and 0.6, with around 10 occurrences each.

    This plot shows us that the observed difference is well within the confidence level. Although we can easily identify the location observation, it is always good to check the p-value to be sure.

    In this example our p-value is found with:

    import numpy as np
    
    p_value = np.mean([diff > observed_percentage_diff for diff in perm_diffs])
    

    This results in 0.4204 for our data. Then we compare p_value with an alpha value, which is usually 0.05. If the p-value is less then alpha then we reject the null hypothesis. So in our example the p-value is extremely high, meaning we fail to reject the null hypothesis. Therefore, we can say that the difference between group A and group B is most likely due to chance rather than the treatment we applied.

    Conclusion

    The results of a significance test do not explicitly accept the null hypothesis, but rather help to understand if the treatments are affecting the results of the experiments or the results are most likely due to chance. A/B (or in some cases more groups, like C/D/E…) experiments are helpful to understand whether the treatment or chance is most likely the cause of the results, as well as finding which group performs the best.

    Reference

    Bruce, Peter, Andrew Bruce, and Peter Gedeck. Practical Statistics for Data Scientists, 2nd Edition. O’Reilly Media, Inc., 2020.


    testing data-science

    ssh-askpass on macOS for SSH agent confirmation

    Bharathi Ponnusamy

    By Bharathi Ponnusamy
    November 23, 2022

    A city street at night. A man sits on a bench, looking at his laptop as cyclists pass by.
    Photo by Kristoffer Trolle, CC BY 2.0

    At End Point Dev we mostly use SSH keys for authentication when connecting to remote servers and Git services. The majority of the time, the servers we are trying to visit are barred from direct access and require a middle “jump server” instead.

    Enabling SSH agent forwarding makes it easier to reuse SSH private keys. It keeps the private keys on our local machine and uses them to authenticate with each server in the chain without entering a password.

    However, this approach comes with an inherent risk of the agent being hijacked if one of the servers is compromised. This means a bad guy could use the SSH keys to compromise downstream servers.

    In this post, we’ll cover a simple way to protect against SSH agent hijacking. We will see in detail on macOS how to configure a system-wide agent using ssh-askpass to pop up a graphical window to ask for confirmation before using the agent.

    How it works

    Agent confirmation dialog reading “Allow us of key /Users/(blank)/.ssh/id_rsa? Key fingerprint (blank) (blank). The cancel button is highlighted.

    It is strongly recommended to use the -c option on the ssh-add command when adding your SSH keys to the agent in order to protect yourself against SSH agent hijacking.

    With this, every time a request is made to utilize the private key stored in the SSH agent, ssh-askpass will display a prompt on your local computer asking you to approve the usage of the key. By doing this, it becomes more difficult for a remote attacker to use the private key without authorization.

    If you don’t want to use the ssh-askpass agent confirmation, I recommend using the OpenSSH feature ProxyJump rather than agent forwarding to get an equivalent level of security.

    Installing ssh-askpass on macOS

    So let’s set this up. The recommended way is with Homebrew:

    Install ssh-askpass with Homebrew

    $ brew tap theseal/ssh-askpass
    $ brew install ssh-askpass
    

    You might see some warnings. Go ahead and proceed with them.

    Now we need to start the Homebrew services. Note that this is a brew service, not a regular macOS daemon service.

    $ brew services start theseal/ssh-askpass/ssh-askpass
    => Successfully started `ssh-askpass` (label: homebrew.mxcl.ssh-askpass)
    

    Behind the scenes, it just sets the SSH_ASKPASS and SUDO_ASKPASS environment variables and stops ssh-agent, so that the SSH agent can pick up these environment variables when it restarts.

    To list the services and make sure it’s started:

    $ brew services list | grep ssh-askpass
    ssh-askpass started ~/Library/LaunchAgents/homebrew.mxcl.ssh-askpass.plist
    

    Install ssh-askpass with MacPorts

    If you prefer MacPorts, do:

    $ sudo port install ssh-askpass
    

    Install ssh-askpass from source code

    And of course you can install from source if you wish:

    $ cp ssh-askpass /usr/local/bin/
    $ cp ssh-askpass.plist ~/Library/LaunchAgents/
    $ launchctl load -w ~/Library/LaunchAgents/ssh-askpass.plist
    

    Configure the SSH agent with the ssh-add -c option

    • Let’s first verify that the agent is running, then add the private key with the confirmation option:

      $ ssh-add -l
      The agent has no identities.
      $ ssh-add -c .ssh/id_rsa
      Enter passphrase for .ssh/id_rsa (will confirm after each use):
      Identity added: .ssh/id_rsa (.ssh/id_rsa)
      The user must confirm each use of the key
      
    • The Identity will get added if you provide the correct passphrase for the key. This can be confirmed by listing the keys again with ssh-add -l.

    ssh-askpass agent confirmation

    Now let’s log into a remote server.

    You will be prompted to confirm the private key’s usage with the pop-up window:

    Agent confirmation dialog. Identical to the previous dialog.

    Set up keyboard shortcuts

    Since it’s too easy to hit the spacebar and accept a connection, ssh-askpass defaults to the cancel option. We can use keyboard shortcuts to press “OK” by following the below steps.

    • Go to “System Preferences” and then “Keyboard”.

    macOS 10.15 Catalina, 11 Big Sur, 12 Monterey

    • Go to “Shortcuts” tab

    • check the option “Use keyboard navigation to move focus between controls”.

      macOS 10.15+ settings open to the Keyboard section. First highlighted is the “Shortcuts” tab, and second is a checkbox at the bottom of the window reading “Use keyboard navigation to move focus between controls.

    macOS 13 Ventura

    • Turn on the “Keyboard navigation” option.

      macOS 13.0 settings open to the keyboard tab, with a slider button reading “Keyboard navigation” highlighted.

    Now you can press tab ⇥ and then the spacebar to select “OK”.

    Enjoy!


    ssh mac security

    Introduction to Terraform with AWS

    Jeffry Johar

    By Jeffry Johar
    November 9, 2022

    Port Dickson, a Malaysian Beach. Rocks in the forground jut out into an inlet, across which is a line of red-roofed houses.
    Photo by Jeffry Johar

    Terraform is a tool from HashiCorp to enable infrastructure as code (IaC). With it users can define and manage IT infrastructure in source code form.

    Terraform is a declarative tool. It will ensure the desired state as defined by the user.

    Terraform comes with multiple plugins or providers which enable it to manage a wide variety of cloud providers and technologies such as but not limited to AWS, GCP, Azure, Kubernetes, Docker and others.

    This blog will go over how to use Terraform with AWS.

    Prerequisites

    For this tutorial we will need the following:

    • An active AWS account.
    • An internet connection to download required files.
    • A decent editor such as Vim or Notepad++ to edit the configuration files.

    Install AWS CLI

    We need to set up the AWS CLI (command-line interface) for authentication and authorization to AWS.

    Execute the following command to install the AWS CLI on macOS:

    $ curl -O https://awscli.amazonaws.com/AWSCLIV2.pkg
    $ sudo installer -pkg AWSCLIV2.pkg -target /
    

    For other OSes see Amazon’s docs.

    Execute the following command and enter the AWS Account and Access Keys:

    $ aws configure
    

    Install Terraform

    We need to install Terraform. It is just a command line tool. Execute the following to install Terraform on macOS:

    $ brew tap hashicorp/tap
    $ brew install hashicorp/tap/terraform
    

    For other OSes see Terraform’s installation docs.

    Create the Terraform configuration file

    Before we can create any Terraform configuration file for a project, we need to create a directory where Terraform will pick up any configuration in the current directory and will store the state of the created infrastructure in a file.

    The name of the directory can be anything. For this tutorial we are going to name it terraform-aws. Create the directory and cd to it:

    $ mkdir terraform-aws
    $ cd terraform-aws
    

    Create the following file and name it main.tf. This is the main configuration file for our Terraform project. This configuration will provision an EC2 instance, install Amazon Linux 2 as the OS and install Nginx as the web server. The comments start with a hash #. They describe each section’s function. For simplicity, the configuration is using the default VPC that comes with the selected AWS region.

    # Set AWS as the cloud provider
    terraform {
      required_providers {
        aws = {
          source  = "hashicorp/aws"
          version = "~> 4.16"
        }
      }
    
      required_version = ">= 1.2.0"
    }
    
    # Set AWS region
    provider "aws" {
      region = "ap-southeast-1"
    }
    
    # Set the default VPC as the VPC
    resource "aws_default_vpc" "main" {
      tags = {
        Name = "Default VPC"
      }
    }
    
    # Set AWS security group to allow SSH and HTTP
    resource "aws_security_group" "ssh_http" {
      name        = "ssh_http"
      description = "Allow SSH and HTTP"
      vpc_id      = aws_default_vpc.main.id
    
      ingress {
        from_port   = 22
        to_port     = 22
        protocol    = "tcp"
        cidr_blocks = ["0.0.0.0/0"] # make this your IP address or range
      }
      ingress {
        from_port   = 80
        to_port     = 80
        protocol    = "tcp"
        cidr_blocks = ["0.0.0.0/0"]
      }
      egress {
        from_port   = 0
        to_port     = 0
        protocol    = "-1"
        cidr_blocks = ["0.0.0.0/0"]
      }
    }
    
    
    # AWS EC2 configuration
    # The user_data contains the script to install Nginx
    resource "aws_instance" "app_server" {
      ami           = "ami-0b89f7b3f054b957e"
      instance_type = "t2.micro"
      key_name = "kaptenjeffry"
      vpc_security_group_ids = [aws_security_group.ssh_http.id]
      user_data = <<EOF
       #!/bin/bash
       sudo yum update
       sudo amazon-linux-extras install nginx1 -y
       sudo systemctl start nginx
      EOF
    
      tags = {
        Name = "Nginx by Terraform"
      }
    }
    
    # EC2 Public IP
    output "app_server_public_ip" {
      description = "Public IP address of app_server"
      value       = aws_instance.app_server.public_ip
    }
    

    Initialize the project

    Initialize the project by downloading the required plugin. For this example, it will download the AWS plugin. Initialize the project by executing the following command:

    $ terraform init
    

    Validate the configuration file

    Check the syntax of the configuration file:

    $ terraform validate
    

    Apply the configuration

    This will make Terraform create and provision the resources specified in the configuration file. It will ask to review the configuration; answer yes to proceed. Take note of the public IP of the provisioned EC2.

    $ terraform apply
    

    Sample output:

    Terraform Apply output. Highlighted is a line reading “Enter a value:”. “yes” has been entered as the answer. Also highlighted is a line under “Outputs:” reading “app_server_public_ip = “46.137.236.88”.

    Access the provisioned EC2 and Nginx

    Use the key_name that is configured in main.tf and the generated public IP address to SSH to the EC2 Instance.

    $ ssh -i kaptenjeffry.pem ec2-user@46.137.236.88
    

    Use the generated public IP address in a web browser to access the Nginx service. Please make sure to use http protocol since the Nginx is running on port 80.

    The default Nginx page in a web browser. The top of the page reads “Welcome to nginx on Amazon Linux!

    Conclusion

    That’s all, folks. This is the bare minimum Terraform configuration to quickly deploy an EC2 instance at AWS.

    For more cool stuffs you can visit the Terraform main documentation for AWS.

    Have a nice day :)


    terraform aws linux sysadmin

    Building and Hosting a Web App with .NET 6, Postgres and Linux

    Dylan Wooters

    By Dylan Wooters
    November 3, 2022

    Fishing boat in Dar es Salaam. A traditional fishing boat sits on the beach at low tide, with the fading light of sunset behind. In the background, other boats float on the Msasani Bay, and several high-rise buildings are visible to the right on the Masaki peninsula.

    For well over a decade, working with the .NET framework meant running Windows. With the release of .NET Core in 2016, developers were granted the freedom to choose their OS, including Linux; no longer were we bound to Windows. However, few took the plunge, at least in my experience. Why? Well, we are comfortable with what we know, and afraid of what we don’t.

    The truth is that building a .NET application on Linux is not that hard, once you get over a few minor bumps in the road. And there are many advantages to this approach, including flexibility, simplicity, and lower costs.

    To demonstrate this, we will create a simple .NET MVC web application that connects to Postgres. Then, we will host the app on Linux with Nginx. Shall we start?

    Preparing the database

    First, you’ll want to install Postgres locally. If you’re using a Mac, this step is very easy. Simply install Postgres.app and you’ll be ready to go.

    If you’re using Windows, check out the Windows Installers page on the Postgres website to download the latest installer.

    Creating the projects

    To develop .NET 6 apps, you will need to install Visual Studio 2022. Check out the Visual Studio downloads page for options for both Windows and Mac.

    Start by opening up Visual Studio and creating a new Web Application (MVC) project, and choosing .NET 6.0 as the target framework. I’ve named my project “DotNetSix.Demo”. Here are the steps as they look in Visual Studio on my Mac.

    Visual Studio. A window is displayed called New Project. It shows two templates, with the Web Application (Model-View-Controller) template selected. A button on the bottom right shows “Continue”.

    Visual Studio. A window is displayed called New Project. At the top is reads “Configure your new Web Application (Model-View-Controller)”. There is a Target Framework dropdown with “.NET 6.0” selected and second Authentication dropdown with “No Authentication” selected. Under Advanced, the “Configure for HTTPS” checkbox is checked, and the “Do not use top-level statements” checkbox is unchecked.

    On the final screen, go ahead and give your solution a name, and then click Create. Visual Studio should create a new solution for you, with a single web project. It will automatically create the necessary MVC folders, including Controllers, Models, and Views.

    Setting up the database connection

    For this demo, we’ll create a simple app that tracks books that you’ve recently read. Let’s go ahead and add a few simple models for the database: Author and Book. You can create these files in the pre-existing Models folder.

    Book Model:

    using System.ComponentModel.DataAnnotations;
    using System.Text.Json.Serialization;
    
    namespace DotNetSix.Demo.Models
    {
        public class Book
        {
            public int Id { get; set; }
    
            public string Title { get; set; }
    
            [Display(Name = "Publish Date")]
            [DisplayFormat(DataFormatString = "{0:d}")]
            public DateTime PublishDate { get; set; }
    
            public Author? Author { get; set; }
        }
    }
    

    Author Model:

    using System.ComponentModel.DataAnnotations;
    using System.Text.Json.Serialization;
    
    namespace DotNetSix.Demo.Models
    {
        public class Author
        {
            public int Id { get; set; }
    
            public string FirstName { get; set; }
    
            public string LastName { get; set; }
    
            public ICollection<Book>? Books { get; set; }
        }
    }
    
    

    Next, add a folder named “Data” in your project. This will hold a few classes necessary for connecting to Postgres and creating the database. Create a new class file in the folder called AppDBContext.cs. This class will use EF Core to setup a database connection.

    AppDBContext.cs:

    using Microsoft.EntityFrameworkCore;
    using DotNetSix.Demo.Models;
    
    namespace DotNetSix.Demo.Data
    {
    	public class AppDBContext : DbContext
    	{
    		public AppDBContext(DbContextOptions<AppDBContext> options) : base(options)
    		{
    			Database.EnsureCreated();
    			DBInitializer.Initialize(this);
    		}
    
    		public DbSet<Book> Books { get; set; }
    		public DbSet<Author> Authors { get; set; }
    	}
    }
    

    Then create another class file in the folder called DBInitializer.cs. This class will initialize the Postgres database with test data.

    DBInitializer.cs:

    using DotNetSix.Demo.Models;
    
    namespace DotNetSix.Demo.Data
    {
        public static class DBInitializer
        {
            public static void Initialize(AppDBContext context)
            {
                // Look for any existing Book records.
                if (context.Books.Any())
                {
                    return;   // DB has been seeded
                }
    
                var books = new Book[]
                {
                    new Book
                    {
                        Title="Blood Meridian",
                        PublishDate=DateTime.Parse("04-01-1985"),
                        Author=new Author{FirstName="Cormac",LastName="McCarthy"}
                    },
                    new Book
                    {
                        Title="The Dog of the South",
                        PublishDate=DateTime.Parse("01-31-1979"),
                        Author=new Author{FirstName="Charles",LastName="Portis"}
                    },
                    new Book
                    {
                        Title="Outline",
                        PublishDate=DateTime.Parse("05-15-2014"),
                        Author=new Author{FirstName="Rachel",LastName="Cusk"}
                    },
                };
    
                context.Books.AddRange(books);
                context.SaveChanges();
            }
        }
    }
    

    At this point, you may notice some errors in your IDE. This is because we need to add two important Nuget packages: Microsoft.EntityFrameworkCore and Npgsql.EntityFrameworkCore.PostgreSQL. You can add these by right-clicking on the Dependencies folder and clicking “Manage Nuget Packages…”. Here’s how it looks in Visual Studio.

    Visual Studio. The Nuget Packages window shows several packages in a column on the left, with the Microsoft.EntityFrameworkCore package selected. On the top right is a search bar to use to search for dependencies, and below it is an informational window on the dependency, as well as a “Add Package” button.

    To round out the database connection, you’ll want to update your Program.cs file, adding the DB Context and the initializing the database. Also, you may encounter an error when you first run your application, citing incompatible dates between .NET and Postgres. To fix this, we will set the Npgsql.EnableLegacyTimestampBehavior to true.

    Here is the complete Program.cs for your reference. Lines 7–8, 14, and 24–34 are what was added to the default Program.cs that is created as part of the web project.

    Program.cs:

    using DotNetSix.Demo.Data;
    using DBContext = DotNetSix.Demo.Data.AppDBContext;
    using Microsoft.EntityFrameworkCore;
    
    var builder = WebApplication.CreateBuilder(args);
    
    builder.Services.AddDbContext<DBContext>(options =>
        options.UseNpgsql(builder.Configuration.GetConnectionString("DemoDbContext")));
    
    // Add services to the container.
    builder.Services.AddControllersWithViews();
    
    var app = builder.Build();
    AppContext.SetSwitch("Npgsql.EnableLegacyTimestampBehavior", true);
    
    // Configure the HTTP request pipeline.
    if (!app.Environment.IsDevelopment())
    {
        app.UseExceptionHandler("/Home/Error");
        // The default HSTS value is 30 days. You may want to change this for production scenarios, see https://aka.ms/aspnetcore-hsts.
        app.UseHsts();
    }
    
    using (var scope = app.Services.CreateScope())
    {
        var services = scope.ServiceProvider;
    
        var context = services.GetRequiredService<DBContext>();
        // Note: if you're having trouble with EF, database schema, etc.,
        // uncomment the line below to re-create the database upon each run.
        //context.Database.EnsureDeleted();
        context.Database.EnsureCreated();
        DBInitializer.Initialize(context);
    }
    
    app.UseHttpsRedirection();
    app.UseStaticFiles();
    
    app.UseRouting();
    
    app.UseAuthorization();
    
    app.MapControllerRoute(
        name: "default",
        pattern: "{controller=Books}/{action=Index}/{id?}");
    
    app.Run();
    

    Note that the pattern in MapControllerRoute points to a new controller and action that we will create in the next section.

    Updating the config file

    The final task to connect to Postgres is to update the appsettings.json file with a connection string. Since I used Postgres.app to install Postgres, my connection string is simple. The username is the same as my Mac, and there is no password. Here is the full file:

    {
      "Logging": {
        "LogLevel": {
          "Default": "Information",
          "Microsoft.AspNetCore": "Warning"
        }
      },
      "AllowedHosts": "*",
      "ConnectionStrings": {
        "DemoDbContext": "Host=localhost;Database=demo;Username=dylan"
      }
    }
    

    Displaying the test data

    Now that we have the database connection setup, let’s make some small changes to the controllers and views in order to see the data in Postgres.

    Add a new file in the Controllers folder called BookController.cs. This file provides an Index controller action that queries the book data from Postgres using EFCore.

    Controllers/BooksController.cs:

    using Microsoft.EntityFrameworkCore;
    using Microsoft.AspNetCore.Mvc;
    using DotNetSix.Demo.Data;
    
    namespace DotNetSix.Demo.Controllers
    {
        public class BooksController : Controller
        {
            private readonly AppDBContext _context;
            private readonly IConfiguration _config;
    
            public BooksController(AppDBContext context, IConfiguration config)
            {
                _context = context;
                _config = config;
            }
            // GET: Papers
            public async Task<IActionResult> Index()
            {
                var appDBContext = _context.Books.Include(b => b.Author);
                return View(await appDBContext.ToListAsync());
            }
        }
    }
    

    Now create an accompanying view. Add a “Books” folder under Views, and then add a new file to the Books folder called Index.cshtml. This view will receive the data from the controller and display a simple table of recently read books.

    Books/Index.cshtml:

    @model IEnumerable<DotNetSix.Demo.Models.Book>
    
    <head>
        <meta charset="utf-8" />
        <meta name="viewport" content="width=device-width, initial-scale=1.0" />
        <link rel="stylesheet" href="./css/CreateStyleSheet.css" asp-append-version="true" />
        <link rel="stylesheet" href="~/css/site.css" asp-append-version="true" />
    </head>
    
    @{
        ViewData["Title"] = "Index";
    }
    
    <h1>Books Recently Read</h1>
    
    <div class="table-wrapper">
        <div class="table-container">
            <table class="table">
                <thead>
                    <tr>
                        <th>
                            @Html.DisplayNameFor(model => model.Title)
                        </th>
                        <th>
                            @Html.DisplayNameFor(model => model.Author)
                        </th>
                        <th>
                            @Html.DisplayNameFor(model => model.PublishDate)
                        </th>
                    </tr>
                </thead>
                <tbody>
                    @foreach (var item in Model)
                        {
                        <tr>
                            <td>
                                @Html.DisplayFor(modelItem => item.Title)
                            </td>
                            <td>
                                @{
                                    var authorFullName = item.Author.FirstName + " " + item.Author.LastName;
                                    @Html.DisplayFor(modelItem => authorFullName);
                                }
                            </td>
                            <td>
                                @Html.DisplayFor(modelItem => item.PublishDate)
                            </td>
                        </tr>
                    }
                </tbody>
            </table>
        </div>
    </div>
    
    

    With the above files in place, you are now ready to run your app. Go ahead and run the project in Visual Studio. You should then see the Books index page in your browser. Hopefully it looks like this:

    Demo page in the web browser. A new window in Brave is pointing to https://localhost:7281/ and displays a top-level navigation with our app name (DotNetsix.Demo) and Home and Privacy links. Below the navigation is a simple table displaying the book data that is saved in Postgres.

    Installing and configuring Nginx

    Just a heads up that the following sections borrow from the MSDN article on hosting ASP.NET Core on Linux. Be sure to check out the article if you’re in need of more info or additional help!

    Now that we have the application running, we can prepare to deploy it to a Linux server. The first step in preparing your server to host the application is to install Nginx. Nginx will act as a reverse proxy to your .NET application running on localhost. To install Nginx, use the appropriate package manager for your Linux distro. For example, if you’re running Debian, use apt-get to install Nginx:

    sudo apt-get update
    sudo apt-get install nginx
    

    You can verify the installation by running sudo nginx -v, which should display the version. Finally, start Nginx using the command sudo service nginx start.

    Once Nginx is installed, you’ll need to configure it to host the .NET Core application. Go ahead and create a new Nginx configuration file in /etc/nginx/conf.d. In our case, we’ll name it dotnetsixdemo.conf. Within this configuration file, we’ll do a few things:

    1. Redirect HTTP traffic to HTTPS.
    2. Use HTTPS with an SSL cert installed by Let’s Encrypt certbot.
    3. Configure Nginx to forward HTTP requests to the .NET application, which by default will run locally at http://127.0.0.1:5000.

    Here is what our configuration file ends up looking like.

    server {
        if ($host = dotnetsixdemo.org) {
            return 301 https://$host$request_uri;
        } # managed by Certbot
    
        listen        80;
        listen       [::]:80;
        server_name  dotnetsixdemo.org;
        return 404; # managed by Certbot
    }
    
    server {
        server_name  dotnetsixdemo.org;
    
        location / {
            proxy_pass         http://127.0.0.1:5000;
            proxy_http_version 1.1;
            proxy_set_header   Upgrade $http_upgrade;
            proxy_set_header   Connection keep-alive;
            proxy_set_header   Host $host;
            proxy_cache_bypass $http_upgrade;
            proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header   X-Forwarded-Proto $scheme;
        }
    
        listen [::]:443 ssl; # managed by Certbot
        listen 443 ssl; # managed by Certbot
        ssl_certificate /etc/letsencrypt/live/dotnetsixdemo.org/fullchain.pem; # managed by Certbot
        ssl_certificate_key /etc/letsencrypt/live/dotnetsixdemo.org/privkey.pem; # managed by Certbot
        include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
        ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
    }
    

    Final application adjustments

    Before we publish the app, we need to make a small change to the Program.cs file. This will allow redirect and security policies to work correctly given the Nginx reverse proxy setup.

    using Microsoft.AspNetCore.HttpOverrides;
    
    ...
    
    app.UseForwardedHeaders(new ForwardedHeadersOptions
    {
        ForwardedHeaders = ForwardedHeaders.XForwardedFor | ForwardedHeaders.XForwardedProto
    });
    

    Additionally, you’ll want to update your appsettings.json file to reflect your target environment, in particular the connection string for Postgres.

    Time to publish!

    We are now ready to publish the app to the server. In the command prompt, navigate to the root directory of the project, and run the following command.

    dotnet publish --configuration release
    

    This will build the app and create a new output folder at /your-project-root/bin/Release/net6.0/publish.

    At this point, all that’s left to do is to copy the contents of the publish folder to the server. You can do this using a variety of tools including SCP, SFTP, etc.

    Running the app on the server

    Once you’ve copied the app to the server, you can start the app by navigating to the directory where the app was copied, and then running the command dotnet [app_assembly].dll. For our demo app, the target DLL would be DotNetSix.Demo.dll.

    This will run the app at http://127.0.0.1:5000, and it should now be accessible via the URL that you configured using Nginx. Based on the Nginx configuration provided above, that would be https://dotnetsixdemo.org. Go ahead and test your site in the browser to make sure it is accessible and the reverse proxy is working properly.

    Using systemd to run the app as a service

    As you might have noticed, there are a few problems with running the app directly using the dotnet command above. The app could easily stop running if the server encounters an issue, and the app is not monitorable. To fix this, let’s use systemd to run and monitor the app process.

    Create a new service definition file by running sudo nano /etc/systemd/system/dot-net-six-demo.service and entering the following.

    [Unit]
    Description=Dotnet Six Demo App
    
    [Service]
    WorkingDirectory=/home/dotnetsix/publish
    ExecStart=/usr/bin/dotnet /home/dotnetsix/publish/DotNetSix.Demo.dll
    Restart=always
    # Restart service after 10 seconds if the dotnet service crashes:
    RestartSec=10
    KillSignal=SIGINT
    SyslogIdentifier=dotnetsixdemo
    User=dotnetsix
    Environment=ASPNETCORE_ENVIRONMENT=Production
    Environment=DOTNET_PRINT_TELEMETRY_MESSAGE=false
    
    [Install]
    WantedBy=multi-user.target
    

    Note that the paths in WorkingDirectory and ExecStart should match where you copied the application build files on the server, as part of the publish step above.

    Also, the User option specifies the user that manages the service and runs the process. In our example, this is the dotnetsix user. You’ll want to create your own user, and importantly grant that user proper ownership of the application files.

    To finalize the new service, save the service file and then start the service with sudo systemctl enable --now dot-net-six-demo.service. This will run the app now in the background via systemd and ensure it also starts up after the next reboot.

    Viewing logs

    Since we are now running the app via systemd, we can use the journalctl to view logs. To view the logs for the demo app, you would run the command sudo journalctl -fu dot-net-six-demo.service.

    Wrapping up

    That is all! We now have a .NET application running on our Linux server, hosted with Nginx via a reverse proxy, and connecting to a local PostgreSQL database. As we can see, there are several steps to take into account, but the process itself is not particularly complex.

    See also my co-worker Kevin Campusano’s blog post for more tips on using .NET with PostgreSQL.

    Have you had problems working with .NET and Linux? Do you have any alternative solutions to the ones proposed here? We await your comment.


    dotnet postgres linux

    Creating a Messaging App Using Spring for Apache Kafka, Part 5

    Kürşat Kutlu Aydemir

    By Kürşat Kutlu Aydemir
    October 31, 2022

    Close up photo of 5 pencils on a faux wood grain desk. The center pencil’s coating is a bright orange, while the other four are a dark green, almost black.

    I guess this is the longest break between posts in this series. But finally I had a chance to prepare a working example to finalize this series. So, up-to-date code is available in the GitHub repository.

    Activation and Login

    This was already implemented in the previous parts. However, we haven’t shown it in action yet. There are some code fixes and small changes within the workflow of activation and login steps, so you can refer to the GitHub repository for the latest code.

    Authentication and activation are managed through the AuthController class where these activation and login requests are handled. Let’s take a look at the REST endpoints handling these requests and explain the steps.

    Activation

    Our activation step uses a dummy mobile phone number. You can think of it similarly to activating a messaging application (like WhatsApp) using a phone number. I didn’t introduce any restrictions on the phone number for this application, so the phone number activation is just a pseudo-step and you can supply any number. In real life, the phone number activation would use SMS or other activation services to activate your chat application against the user’s phone number.

    The activation endpoint is /api/auth/getcode, and takes the mobile number as payload. A sample POST request is below:

    curl -H "Content-Type: application/json" -X POST localhost:8080/api/auth/getcode -d '{ "mobile": "01234" }'
    

    The response of this endpoint is of the class ActivationResponse which, in return, gives an activation code along with the provided mobile number. This activationCode will later be used to login and retrieve an access token for further loginless connections. Here is an example output of this response:

    {
      "mobile": "01234",
      "activationCode": "309652"
    }
    

    We provide a random activationCode and the client now needs to use this activation code to move one step further towards getting an access token for further authentication.

    Login

    The login step is supposed to be a one-time login after getting the activation code (phone number) which is used for login credentials. Our application returns an access token in return. The login endpoint is /api/auth/login. Request and response examples are below.

    curl -H "Content-Type: application/json" -X POST localhost:8080/api/auth/login -d '{ "mobile": "01234", "activationCode": "309652" }'
    

    POSTing to the login endpoint returns an access token. The access token is needed to make message-sending requests so that the client can be authenticated. Storing locally and reusing this access token is the responsibility of the client.

    {
      "accessToken": "5eab27f8-8748-4fdc-a4de-ac782ce17a74"
    }
    

    Chat Application

    The chat application is the final part of this series, and will show the messaging in action. The chat application is designed as a simple proof of concept and simulates a chatting session.

    To add a web UI to our Spring application I’ve added two static files, app.js and index.html, in the resources directory. index.html serves as the chat application’s UI and app.js is responsible for making AJAX calls and WebSocket connections from the client browser.

    Three text boxes. The first is labeled “Mobile number (pretend) activation”, and an “Activate” button follow. The next is labeled “login” with a “Login” button following. The next is labeled “Access Token, with no button following immediately. There is then a label reading “Chat Application:”, followed by two buttons reading “Start New Chat” and “End Chat”, in order from top to bottom.

    I added all three steps to the chat application UI to make the workflow clearer. As you see in the chat screen above, these three steps are activation, login, and starting a new chat. Let’s walk through these steps.

    Activation and login through the chat application

    Activation simply requires a pretend mobile phone number. When you click on Activate it sends a POST request to the /api/auth/getcode end point we mentioned above.

    function activate() {
      var mobileFormData = JSON.stringify({
        'mobile': $("#mobile").val(),
      });
    
      $.ajax({
        type: "POST",
        url: "http://localhost:8080/api/auth/getcode",
        data: mobileFormData,
        contentType: "application/json; charset=utf-8",
        success: function(result) {
          $("#activationCode").val(result.activationCode);
        },
      });
    }
    

    The resulting activationCode is automatically inputted into the login field. We already know what this activationCode is and where it is stored in the backend, so now our chat client can use this along with the mobile number to log in and get the access token. When you click on Login it POSTs to /api/auth/login and puts the access token into the Access Token input box this time.

    function login() {
      var loginFormData = JSON.stringify({
        'mobile': $("#mobile").val(),
        'activationCode': $("#activationCode").val(),
      });
    
      $.ajax({
        type: "POST",
        url: "http://localhost:8080/api/auth/login",
        data: loginFormData,
        contentType: "application/json; charset=utf-8",
        success: function(result) {
          $("#accessToken").val(result.accessToken);
        },
      });
    }
    

    Now finally our web client can connect to our WebSocket URI. Without a login step we could let our application accept WebSocket connections as well. However, we simulated an authentication step and we want to know who is connecting to the WebSocket handler.

    function connect() {
      ws = new WebSocket('ws://localhost:8080/messaging?accessToken=' + $("#accessToken").val());
        ws.addEventListener('error', (error) => {
        console.log("Error: ", error.message);
        setConnected(false);
        });
        ws.addEventListener('close', (event) => {
        console.log("Close: ", event.code + " - " + event.reason);
          setConnected(false);
        })
      ws.onmessage = function(event) {
        helloWorld(event.data);
        console.log("Connected to WebSocket " + event.data);
      }
      setConnected(true);
    }
    

    WebSocket Handshake

    In our WebSocketConfig class I added an WebSocket handshake interceptor WSHandshakeInterceptor to authenticate the connecting user by checking the accessToken. So, the accessToken is necessary at the first step while connecting to the WebSocket registry. If the client doesn’t provide an accessToken or provides an invalid one then the handshake will fail.

    @Override
    public boolean beforeHandshake(ServerHttpRequest serverHttpRequest, ServerHttpResponse serverHttpResponse, org.springframework.web.socket.WebSocketHandler webSocketHandler, Map<String, Object> map) throws Exception {
    
        LOG.info("hand shaking for ws session url  " + serverHttpRequest.getURI().getPath() + " ? " + serverHttpRequest.getURI().getQuery());
        String parameters[] = serverHttpRequest.getURI().getQuery().split("=");
    
        if (parameters.length == 2 && parameters[0].equals("accessToken")) {
            String accessToken = parameters[1];
    
            Long senderUserId = 0L;
            String senderId = cacheRepository.getUserIdByAccessToken(accessToken);
    
            if (senderId == null) {
                User sender = userRepository.findByToken(accessToken);
                if(sender != null) {
                    senderUserId = sender.getUserId();
                }
            } else {
                senderUserId = Long.valueOf(senderId);
            }
    
            LOG.info("Handshake found userId: " + senderUserId);
    
            if (senderUserId == 0L) {
                LOG.info("Handshake failed: user not found for given accessToken");
                map.put("CUSTOM-HEADER", "Handshake failed: user not found for given accessToken");
                return false;
            }
    
            map.put("CUSTOM-HEADER", "Handshake successful");
            LOG.info("Handshake successful");
            return true;
        }
    
        map.put("CUSTOM-HEADER", "Handshake failed: accessToken not found");
        LOG.info("Handshake failed: accessToken not found");
        return false;
    
    }
    

    If everything goes well and our web client is connected to WebSocket then the client can start sending messages to the other WebSocket sessions. We don’t manage offline messaging or other capabilities in this application. As you’ll see a WebSocket session will be able to send messages to only corresponding active WebSocket sessions. In real life examples you would manage offline messaging and deliver to the destination users whenever they become online.

    Chat in action

    When I open the chat application in two different browsers and login with different mobile numbers I am able to make them have a chat. I assume mobile number 01234 has two contacts 12345 and 23456 in their contact list.

    On this screen I chose the 12345 contact to start messaging. I entered my message and clicked on Send button. Since both clients connected to the WebSocket I was able to send the message to 12345, as you can see on the second client’s window.

    Two chat windows are open, with the same layout as the previous image. The left window’s layout is extended to include the following: two checkboxes reading “12345” and “23456”, with the first checked. Below is a text field containing a message reading “Hey dude, howdy?”. Below it is a button labeled “Send”. The right window is extended to include the following: a received message reading ‘{“msg”:“Hey dude, howdy?”,“senderId”:4,“topic”:“SEND_MESSAGE”}’, then an empty text area with a “Send” button.

    The messages are not tidied here; I made the WebSocket message list the message on the client where it connected to its WebSocket session.

    Here is the JavaScript code sending the message to WebSocket session when you click on the Send button:

    function sendData() {
      var data = JSON.stringify({
        'topic': 'SEND_MESSAGE',
        'message': {
          'accessToken': $("#accessToken").val(),
          'sendTo': $("#sendTo").val(),
          'msg': $("#messageArea").val(),
        },
      })
      ws.send(data);
    }
    

    When we send the message to WebSocket, our WebSocketHandler handles the received message through WebSocket session. However, it doesn’t directly send the message to the corresponding WebSocket session. We send our message to the Kafka topic SEND_MESSAGE using MessageSender.send() method at this point and Kafka manages the streaming messages.

    On the Kafka listener side of our application MessageReceiver.messagesSendToUser() is receiving the messages and it redirects the messages to the corresponding WebSocket session. So we took this workload from the WebSocket handler to the Kafka listener so it would be busy only with the messages sent by the clients. Another approach would be sending the messages directly to Kafka topics instead of WebSocket handler so we would give all the workload to Kafka services.

    @KafkaListener(topics = "SEND_MESSAGE", groupId = "foo")
    public void messagesSendToUser(@Payload String message, @Headers MessageHeaders headers) {
    
        JSONObject jsonObject = new JSONObject(message);
    
        LOG.info("Websocket message will be sent if corresponding destination websocket session is found");
        if (jsonObject.get("sendTo") != null
                && WebSocketPool.websockets.get(jsonObject.getLong("sendTo")) != null
                && WebSocketPool.websockets.get(jsonObject.getLong("sendTo")).size() > 0) {
    
            String accessToken = jsonObject.getString("accessToken");
            Long sendTo = jsonObject.getLong("sendTo");
            String msg = jsonObject.getString("msg");
    
            LOG.info("Websocket message is sent to " + sendTo);
    
            String topic = "SEND_MESSAGE";
    
            messageService.sendMessage(accessToken, sendTo, msg, topic);
    
        } else {
            LOG.info("Websocket session not found for given sendTo");
        }
    }
    

    Here is our logging for the message sent through the chat application.

    2022-10-07 16:36:41.371  INFO 37554 --- [nio-8080-exec-1] c.e.S.websocket.WebSocketHandler         : {"topic":"SEND_MESSAGE","message":{"accessToken":"5eab27f8-8748-4fdc-a4de-ac782ce17a74","sendTo":"5","msg":"Hey dude, howdy?"}}
    2022-10-07 16:36:41.411  INFO 37554 --- [ntainer#0-0-C-1] c.e.S.message.broker.MessageReceiver     : Websocket message will be sent if corresponding destination websocket session is found
    2022-10-07 16:36:41.411  INFO 37554 --- [ntainer#0-0-C-1] c.e.S.message.broker.MessageReceiver     : Websocket message is sent to 5
    2022-10-07 16:36:41.412  INFO 37554 --- [ntainer#0-0-C-1] c.e.S.message.broker.MessageReceiver     : Sending websocket message {"msg":"Hey dude, howdy?","senderId":4,"topic":"SEND_MESSAGE"}
    

    What is the point of using Kafka and WebSocket?

    Let’s go back to the question of why we used Kafka and WebSockets for a messaging app. We questioned this paradigm in part 1 of this series while discussing the design and architecture. It all depends on what you aim to do with a messaging application.

    There are times when WebSocket has the advantage over HTTP or XMPP, and we can compare these protocols on a case-by-case basis. However, those debates won’t be had in this post. WebSocket is simply a fast relaying and simple protocol which enables a web server app to communicate with a web browser, or more broadly, WebSocket clients, easily. So you can use WebSocket to create chat applications over the web and let the same user’s multiple sessions run concurrently on multiple clients.

    Kafka, on the other hand, acts as a message orchestrator, highly aware of what is coming in and going out and upwardly scalable. You are guaranteed to be able to process each message which arrives at Kafka without worrying about concurrency and delay issues which you often face with traditional threads or services. Kafka is not essentially a chat application backend but can handle the millions and even billions of messages easily. The main purpose of this example application is not creating a chat application, but as the title says, a messaging application.

    So, while chatting would designate a narrow scope, messaging covers a broad range of applications handling high loads of data streaming environments.

    Conclusion

    Kafka is used in several environments by lots of giants of industry and startups. A few industry examples would be financial services, telecom, manufacturing, and healthcare. You can make data streaming seamless and real-time using Kafka.

    Please share with us in the comments your experiences using Kafka and related technologies, even including ML and model training streams.


    spring kafka spring-kafka-series java

    Kubernetes Volume definition defaults to EmptyDir type with wrong capitalization of hostPath

    Ron Phipps

    By Ron Phipps
    October 26, 2022

    Cow with light red-brown fur and an inventory ear tag standing in a dry field with scattered desert grass and brush, in front of a fench Photo by Garrett Skinner

    Kubernetes Host Path volume mounts allow accessing a host system directory inside of a pod, which is helpful when doing development, for example to access the frequently-changing source code of an application being actively developed. This allows a developer to edit the code with their normal set of tools without having to jump through a bunch of hoops to get the code into a pod.

    We use this setup at End Point in development where the host system is running MicroK8s and there is a single pod for an application on a single node. In most other cases, host path volume mounts are not recommended. But here it means the developer can edit code on the host machine and the changes are immediately reflected within the pod without having to deploy a new image. If the application server running within the pod is also running in development mode with dynamic reloading, the changes can be viewed with a refresh of the browser accessing the application.

    While working on a test environment to run EpiTrax within Kubernetes, the need arose to set up a Host Path volume mount so that the source code on the host machine would be available within the pod. I used this simple Deployment definition:

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: epitrax
      namespace: app
      labels:
        app: epitrax
    spec:
      template:
        spec:
          securityContext:
            fsGroup: $USERID
          containers:
          - name: shell
            image: epitrax/epitrax
            command: ["sh", "-c", "tail -f /dev/null"]
            securityContext:
              runAsNonRoot: true
              runAsUser: $USERID
              runAsGroup: $USERID
            volumeMounts:
          - mountPath: /opt/jboss/epitrax
              name: epitrax-source-directory
          volumes:
          - name: epitrax-source-directory
            hostpath:
              type: Directory
              path: $PWD/projects/epitrax
    

    After applying this deployment and shelling into the pod I found that /opt/jboss/epitrax was an empty directory and not a host path volume. Describing the pod showed the following:

    Volumes:
      epitrax-source-directory:
        Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
        Medium:
        SizeLimit:  <unset>
    

    I tried changing many different things, viewed the various logs, and searched the Internet for reports of the same problem, but could not figure out what was wrong.

    Eventually I found a single GitHub issue on the Kubernetes project, which did not explain the trouble but did explain that the volume type always defaults to EmptyDir to match Docker’s behavior.

    That’s when I realized the problem: I had used hostpath (all lower case) instead of hostPath. Kubernetes could not find a valid volume type of hostpath so it defaulted to EmptyDir.

    Updating the volumes section to the following resolved the issue:

    # snip
          volumes:
          - name: epitrax-source-directory
            hostPath:
              type: Directory
              path: $PWD/projects/epitrax
    

    Be aware that Kubernetes will not warn or error out if there is an invalid volume type referenced in the volumes section—it will quietly default to EmptyDir!


    kubernetes docker

    Knocking on Kubernetes’s Door (Ingress)

    Jeffry Johar

    By Jeffry Johar
    October 20, 2022

    The door of Alhambra Palace, Spain. A still pool reflects grand doors, flanked on each side by arches and hedges.
    Photo by Alberto Capparelli

    According to the Merriam-Webster dictionary, the meaning of ingress is the act of entering or entrance. In the context of Kubernetes, Ingress is a resource that enables clients or users to access the services which reside in a Kubernetes cluster. Thus Ingress is the entrance to a Kubernetes cluster! Let’s get to know more about it and test it out.

    Prerequisites

    We are going to deploy Nginx Ingress at Kubernetes on Docker Desktop. Thus the following are the requirements:

    • Docker Desktop with Kubernetes enabled. If you are not sure how to do this, please refer to my previous blog on Docker Desktop and Kubernetes.
    • Internet access to download the required YAML and Docker Images.
    • git command to clone a Git repository.
    • A decent editor such as Vim or Notepad++ to view and edit the YAML.

    Ingress and friends

    To understand why we need Ingress, we need to know 2 other resources and their shortcomings in exposing Kubernetes services. Those 2 resources are NodePort and LoadBalancer. Then we will go over the details of Ingress.

    NodePort

    NodePort is a type of Kubernetes service which exposes the Kubernetes application at high-numbered ports. By default the range is from 30000–32767. Each of the worker nodes proxies the port. Thus, access to the service is by using the Kubernetes worker node IPs and the ports. In the following example the NodePort service is exposed at port 30000.

    A diagram of a Kubernetes Cluster. Within are 3 boxes, labeled as worker nodes, with the IP addresses: 192.168.1.1, 192.168.1.2, and 192.168.1.3. Each box contains several purple boxes labeled “Service Type: NodePort at port 30001”. They point to three blue boxes labeled “Pods”. Each worker node box points to a URL corresponding to its IP address: “http://192.168.1.1:30000” and so forth, with port 30000 for each.

    To have a single universal access and a secured SSL connection, we need some external load balancer in front of the Kubernetes cluster to do the SSL termination and to load balance the exposed IPs and ports from the worker nodes. This is illustrated in the following diagram:

    The same diagram as above, but this time all three IP address have arrows pointing bidirectionally to a single circle, labeled Load Balancer + SSL Termination. This circle points to a single URL, “https://someurl.com”.

    LoadBalancer

    LoadBalancer is another type of Kubernetes service which exposes Kubernetes services. Generally it is an OSI layer 4 load balancer which exposes a static IP address. The implementation of LoadBalancer depends on the Cloud or the Infrastructure provider, thus the capability of LoadBalancer varies.

    In the following example a LoadBalancer is exposed with the static public IP address 13.215.159.65 provided by a cloud provider. The IP could also be registered in DNS to allow resolution by a host name.

    A similar Kubernetes Cluster box, again with three boxes containing three blue Pods each, but this time every pod points to the same single box, encompassed only by the outer Kubernetes Cluster box. It is labeled “Service Type: LoadBalancer; Static IP: 13.215.159.65”. The outer box points to a URL: “http://13.215.159.65”, which in turn points to “http://someurl.com”.

    Ingress

    Ingress is a Kubernetes resource that serves as an OSI layer 7 load balancer. Unlike NodePort and LoadBalancer, Ingress is not a Kubernetes service. It is another Kubernetes resource that sits in front of a Kubernetes service. It enables routing, SSL termination, and virtual hosting. This is like a full-fledged load balancer inside the Kubernetes cluster!

    The following diagram shows that Ingress is able to route the someurl.com/web/ and someurl.com/app/ endpoints to the intended applications in the Kubernetes cluster, able to terminate SSL certificates, do virtual hosting and route the URL to the intended destination. Please take note that as of this writing, Ingress only supports the http and https protocols.

    An outer Kubernetes Cluster box contains three boxes again. Each box again has three Pods, but they are split into three colors (green, yellow, and red), distributed randomly through the three boxes. The three Pods of each color point to a matching box, in the larger Kubernetes Cluster box, reading “Service Type: ClusterIP; Name: X”, where X is Web, App, and Blog. These three boxes point to another box labeled “Ingress: SSL Termination; Routing; Virtual Hosting. This box points to a URL, “http://13.215.159.65”, which in turn points to a cloud icon with three URLS: “https://someurl.com/web/”, “https://someurl.com/app/”, “https://someurl.com”

    In order to get Ingress in a Kubernetes cluster we need to deploy 2 main things:

    • Ingress Controller is the engine of the Ingress. It is responsible for providing the Ingress capability to Kubernetes. The Ingress Controller is a separate module from Kubernetes core components. There are multiple Ingress Controllers available to use such as Nginx, Istio, NSX, and many more. See a complete list at the kubernetes.io page on Ingress controllers.
    • Ingress Resource is the configuration that manages the Ingress. It is made by applying the Ingress Resource YAML. This is a typical YAML file for Kubernetes resources which requires apiVersion, kind, metadata and spec. Go to kubernetes.io documentation on Ingress to learn more.

    How to deploy and use Ingress

    Now we are going to deploy the Nginx Ingress at Kubernetes in Docker Desktop. We will configure it to access an Nginx web server, a variant for Tomcat web application server and our old beloved Apache web server.

    Start your Docker Desktop with Kubernetes Enable. Right click at the Docker Desktop icon at the top right area of your screen (near the left in this cropped screenshot) to see that both Docker and Kubernetes are running:

    Docker Desktop running on MacOS, with macOS’s top bar Docker menu open. There are two green dots next to lines saying “Docker Desktop is running” and “Kubernetes is running”.

    Clone my repository to get the required deployments YAML files:

    git clone https://github.com/aburayyanjeffry/nginx-ingress.git
    

    Let’s go through the downloaded files.

    • 01-ingress-controller.yaml : This is the main deployment YAML for the Ingress controller. It will create a new namespace named “ingress-nginx”. Then it will create the required service account, role, rolebinding, clusterrole, clusterolebinding, service, configmap, and deployments in the “ingress-nginx” namespace. This YAML is from the official ingress-nginx documentation. To learn more about this deployment see the docs.
    • 02-nginx-webserver.yaml, 03-tomcat-webappserver.yaml, 04-httpd-webserver.yaml: These are the deployment YAML files for the sample applications. They are the typical Kubernetes configs which contain the services and deployments.
    • 05-ingress-resouce.yaml : This is the configuration of the Ingress. It is using the test domain *.localdev.me. This is a domain that is available in most modern operating systems. It can be used for testing without the need to edit the /etc/hosts file. Ingress is configured to route as the following diagram:

    An icon of several people points to three URLS: “http://demo.localdev.me”, “http://demo.localdev.me/tomcat/”, and “http://httpd.localdev.me”. These three point through an Nginx Ingress box to three logos: Nginx, Tomcat, and Apache, respectively. The Nginx Ingress box and the logos all lie within a larger Kubernetes Cluster box.

    Deploy the Ingress Controller. Execute the following to deploy the Ingress Controller:

    kubectl apply -f 01-ingress-controller.yaml
    

    Execute the following to check on the deployment. The pod must be running and the Deployment must be ready:

    kubectl get all -n ingress-nginx
    

    The results of the above command. Highlighted is a line giving the following values: name: “pod/ingress-nginx-controller-6bf7bc7f94-gfgdw”; ready: “1/1”; status: “Running”; restarts: “0”; age: “21s”. Two sections down, another line is highlighted, with the values: name: “deployment.apps/ingress-nginx-controller”; ready: “1/1”; up-to-date: “1”; available: “1”; age: “21s”

    Deploy the sample applications. Execute the following to deploy the sample applications:

    kubectl apply -f 02-nginx-webserver.yaml
    kubectl apply -f 03-tomcat-webappserver.yaml
    kubectl apply -f 04-httpd-webserver.yaml
    

    Execute the following to check on the deployments. All pods must be running and all deployments must be ready:

    kubectl get all
    

    The output of the above command. Highlighted are lines from a table with the following values for name: “pod/myhttpd-xxxxxx”, “pod/mynginx-xxxxxx”, and “pod/mytomcat-xxxxxx”. They share values for ready, status, restarts, and age: “1/1”, “Running”, “0”, and “13s”, respectively. A later section is highlighted. The names are: “deployment.apps/myhttpd”, “deployment.apps/mynginx”, and “deployment.apps/mytomcat”. They share values for ready, up-to-date, available, and age: “1/1”, “1”, “1”, and “13s”, respectively.

    Deploy the Ingress resources. Execute the following to deploy the Ingress resouces:

    kubectl apply -f 05-ingress-resouce.yaml
    

    Execute the following to check on the Ingress resources:

    kubectl get ing
    

    The outpu of the above command. The table only includes one line, with the following values: name: “myingress”; class: “nginx”; hosts: “demo.localdev.me,httpd.localdev.me”, address: blank; ports: “80”; age: “3s”

    Access the following URLs in your web browser. All URLs should bring you to the intended services:

    http://demo.localdev.me
    http://demo.localdev.me/tomcat/
    http://httpd.localdev.me
    

    Three browser windows displaying the above URLs. They display welcome pages for Nginx, Tomcat, and Apache, respectively.

    Conclusion

    That’s all, folks. We have gone over the what, why, and how about the Kubernetes Ingress.

    It is a powerful OSI layer 7 load balancer ready to be used with the Kubernetes cluster. There are free and open source solutions and there are also the paid ones, all listed here.


    kubernetes docker containers
    Page 1 of 208 • Next page