Why Your AI Extractor Fails on .msg Emails (and How to Fix Decoding)

2026-03-13T00:00:00+00:00

I want to share a debugging lesson that saved me from tuning the wrong layer in an AI extraction pipeline.

It started with a familiar symptom: extraction output looked inconsistent. Some rows were fine, but some had extra characters, especially accents. My first instinct was the same one most of us have: maybe the model needs prompt tuning.

It turned out not to be a model problem. The root cause was upstream data integrity: decoding .msg email HTML with the wrong charset.

The pattern that gives it away

If you see this mix, think decoding first:

output is mostly correct, but certain names and addresses look garbled
problems appear only for some senders or date ranges
.eml looks stable, but .msg is inconsistent

A classic sign looks like this:

expected: Müller
corrupted: MÃ¼ller

By the time your extractor sees that text, the meaning is already damaged.

Why `.msg` bites harder than `.eml`

Quick definitions:

.eml is the standard MIME email format and usually includes charset metadata per part.
.msg is an Outlook container format (MAPI), where body bytes and encoding hints can be stored separately.

That difference matters.

If your code assumes UTF-8 for .msg HTML bytes, non-UTF messages can decode into garbage. Then downstream steps (HTML-to-PDF, OCR, LLM extraction, post-processing) just preserve and propagate bad text.

The fix: strict, explicit, controlled

You do not need a big rewrite. A small decode policy change can remove a whole class of silent failures. For .msg HTML bytes:

Read the encoding hint from message metadata.
Map that hint to a decoder codec.
Decode in strict mode.
If needed, use one controlled strict fallback.
If decode still fails, fail loud.

Minimal example in Python:

from extract_msg.encoding import lookupCodePage

PR_INTERNET_CODEPAGE = "3FDE0003"

def decode_msg_html_bytes(html_bytes: bytes, message) -> str:
    codepage_id = message.getPropertyVal(PR_INTERNET_CODEPAGE)
    codec = (
        lookupCodePage(codepage_id)
        if isinstance(codepage_id, int) and codepage_id > 0
        else "utf-8"
    )
    try:
        return html_bytes.decode(codec, errors="strict")
    except (LookupError, UnicodeDecodeError):
        return html_bytes.decode("utf-8", errors="strict")

Why I prefer fail-loud here

errors="replace" keeps jobs moving, but it can hide real data corruption.

For low-stakes preview features, that may be acceptable. For transactional extraction (orders, invoices, legal, shipping), silent corruption is usually worse than an explicit failure.

Use this decision rule:

Use case	Policy
Preview/search UX	Best-effort can be acceptable with clear flags
Transactional extraction	Strict decode + fail loud
Mixed systems	Strict on extraction path, best-effort on preview path

How to roll this out safely

Keep blast radius low:

Change only the failing decode path first.
Validate on a representative dataset, not one sample file.
Leave unrelated paths untouched until evidence says otherwise.
Expand strict policy incrementally.

This gives reliability without destabilizing the rest of the ingestion stack.

Observability that makes this easier next time

Log these fields per message:

Source file
Content source used (HTML or plain text)
Whether encoding hint was found
Selected codec
Whether fallback was used
Result of decoding (success, fallback, manual review, fail)

With this, “random extraction quality” turns into a clear ingestion signal.

Using Razor templates to render HTML emails in ASP.NET Core

2025-08-15T00:00:00+00:00

A while ago I blogged about using Razor templates to render HTML emails in .NET. The method that I discussed there worked, but it was very verbose. Since then, .NET 8 has released, and with it came a simpler way of doing this. In this post we’ll explore how to use these new features to render HTML emails.

You can find all the code in this post on GitHub.

The plan

Similar to the original article, the objective is simple: Sending emails from an ASP.NET Core app, and having the contents of those emails be rendered fromfrom Razor templates. To that end, we need four pieces:

A class for sending emails.
A class for rendering Razor templates into strings.
A Razor template.
A class that puts it all together. That is, takes in parameters, renders the email, and sends it.

Step 1: Sending emails with the MailKit NuGet package

With the help of the MailKit NuGet package, sending emails in .NET is easy. Let’s install it with:

dotnet add package MailKit --version 4.13.0

We also need some configuration in the appsettings.json file, to define the settings needed to establish a connection with an SMTP server:

// ./appsettings.json

{
    // ...
    "MailSettings": {
        "Server": "<YOUR_SMTP_SERVER>",
        "Port": 587, // 25 or 465 or 587 or 2525
        "SenderName": "RazorEmails",
        "SenderEmail": "test@razoremails.com",
        "UserName": "<YOUR_SMTP_SERVER_USER_NAME>",
        "Password": "<YOUR_SMTP_SERVER_USER_PASSWORD>"
    },
    // ...
}

And here’s a simple Mailer class that uses the MailKit library and the configuration above to send emails:

// ./Mailers/Mailer.cs

using MailKit.Net.Smtp;
using MimeKit;

namespace RazorEmails.Mailers;

public class MailData
{
    public required string To { get; set; }
    public required string ToName { get; set; }
    public required string Subject { get; set; }
    public required string Body { get; set; }
}

public class Mailer
{
    private readonly ILogger<Mailer> _logger;
    private readonly IConfiguration _config;

    public Mailer(IConfiguration config, ILogger<Mailer> logger)
    {
        _logger = logger;
        _config = config;
    }

    // Not much to see here, just a method for sending emails using MailKit.
    // Docs are here: https://github.com/jstedfast/MailKit
    public async Task<bool> SendMailAsync(MailData mailData)
    {
        try
        {
            var emailMessage = new MimeMessage();

            emailMessage.From.Add(new MailboxAddress(MailSenderName, MailSenderEmail));
            emailMessage.To.Add(new MailboxAddress(mailData.ToName, mailData.To));

            emailMessage.Subject = mailData.Subject;

            var emailBodyBuilder = new BodyBuilder
            {
                TextBody = mailData.Body,
                HtmlBody = mailData.Body
            };

            emailMessage.Body = emailBodyBuilder.ToMessageBody();

            using var smtpClient = new SmtpClient();

            await smtpClient.ConnectAsync(
                MailServer,
                MailPort,
                MailKit.Security.SecureSocketOptions.StartTls
            );

            await smtpClient.AuthenticateAsync(MailUserName, MailPassword);
            await smtpClient.SendAsync(emailMessage);
            await smtpClient.DisconnectAsync(true);

            return true;
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Failed to send email to {To}", mailData.To);
            return false;
        }
    }

    private string MailSenderName => _config["MailSettings:SenderName"]!;
    private string MailSenderEmail => _config["MailSettings:SenderEmail"]!;
    private string MailServer => _config["MailSettings:Server"]!;
    private int MailPort => int.Parse(_config["MailSettings:Port"]!);
    private string MailUserName => _config["MailSettings:UserName"]!;
    private string MailPassword => _config["MailSettings:Password"]!;
}

Step 2: Rendering Razor templates into strings

Of course, we also need a way of rendering Razor templates. As mentioned in the beginning, .NET 8 made this easy. There’s a page in the official documentation and community blog posts talking about it. For our purposes, here’s a class that does it:

// ./Rendering/RazorViewRenderer.cs

using Microsoft.AspNetCore.Components;
using Microsoft.AspNetCore.Components.Web;

namespace RazorEmails.Rendering;

public class RazorViewRenderer
{
    private readonly IServiceProvider _serviceProvider;
    private readonly ILoggerFactory _loggerFactory;

    public RazorViewRenderer(IServiceProvider serviceProvider, ILoggerFactory loggerFactory)
    {
        _serviceProvider = serviceProvider;
        _loggerFactory = loggerFactory;
    }

    public async Task<string> Render<TView, TViewModel>(TViewModel model) where TView : IComponent
    {
        await using var htmlRenderer = new HtmlRenderer(_serviceProvider, _loggerFactory);

        var html = await htmlRenderer.Dispatcher.InvokeAsync(async () =>
        {
            var output = await htmlRenderer.RenderComponentAsync<TView>(
                ParameterView.FromDictionary(
                    new Dictionary<string, object?> { { "Model", model } }
                )
            );

            return output.ToHtmlString();
        });

        return html;
    }
}

The Render method is where the magic happens. We’ll see how to use it soon, but for now, it’s interesting to look at the generic type parameters.

TView represents the strongly-typed Razor template that will be rendered. Technically, the “template” is actually a Razor component, as we’ll see later. That’s why we use where TView : IComponent as a generic type constraint. With it, we’re specifying that the given TView must inherit from Microsoft.AspNetCore.Components.IComponent, which is the base class of Razor components.

TViewModel, on the other hand, represents the type of the data structure that will be passed to the template as a parameter. It will contain data that the template will use to render itself.

Step 3: Defining the email templates

Okay, now we need to define our templates, along with the vehicles to pass data to them. These will be simple DTOs.

When sending emails from web applications, it is often useful to define a layout that all emails use to keep their styling consistent. Using Razor components, such a layout could look like this:

<!-- ./Rendering/Views/MainLayout.razor -->

@inherits LayoutComponentBase

<!DOCTYPE html>
<html lang="en">
<head>
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
  @Body
</body>
</html>

This is just a basic shell for an HTML document. Notice two interesting aspects of it:

@inherits LayoutComponentBase declares that our layout inherits from the LayoutComponentBase. This is necessary for the layout to actually act as a layout.
@Body determines where the contents of the components that use this layout will be rendered. In this case, we put it inside the <body> HTML tag, which seems appropriate.

Now to define the template for a basic email that uses this layout:

<!-- ./Rendering/Views/MessageEmail.razor -->

@using RazorEmails.Rendering.ViewModels

<LayoutView Layout="@typeof(MainLayout)">

<p>
  Hello @Model.Greeting.
</p>

<p>
  We have a message for you: @Model.Message
</p>

<p>
  Message sent at @DateTime.Now.ToString().
</p>

</LayoutView>

@code {
  [Parameter]
  public MessageEmailViewModel Model { get; set; } = default!;
}

This is a very straightforward Razor component that takes in a MessageEmailViewModel as a parameter and renders a few lines of text using the data coming in the parameter. By virtue of its file name, this component can be referenced in code by the MessageEmail class name. It uses tried and true Razor syntax, so little of this should be surprising if you’ve worked with Razor before.

One thing to notice is how we’re explicitly declaring the LayoutView element, pointing to the MainLayout that we wrote. This is how we tell the renderer to use our layout when rendering this component. MainLayout exists because that’s what we named the file that contains the layout.

Like I mentioned before, MessageEmailViewModel is a simple DTO that serves to pass some data to the template. It’s not very exciting, but this is what it looks like:

// ./Rendering/ViewModels/MessageEmailViewModel.cs

namespace RazorEmails.Rendering.ViewModels;

public class MessageEmailViewModel
{
    public required string Greeting { get; set; }
    public required string Message { get; set; }
}

Step 4: Putting it all together: the class that sends the email

And finally, here we have a class that puts these separate elements to work to send an email:

// ./Mailers/MessageMailer.cs

using RazorEmails.Rendering;
using RazorEmails.Rendering.ViewModels;
using RazorEmails.Rendering.Views;

namespace RazorEmails.Mailers;

public class MessageMailer
{
    private readonly Mailer _mailer;
    private readonly RazorViewRenderer _razorViewRenderer;

    public MessageMailer(Mailer mailer, RazorViewRenderer razorViewRenderer)
    {
        _mailer = mailer;
        _razorViewRenderer = razorViewRenderer;
    }

    public async Task SendAsync(string to, string toName, string greeting, string message)
    {
        string body = await _razorViewRenderer.Render<MessageEmail, MessageEmailViewModel>(
            new MessageEmailViewModel() { Greeting = greeting, Message = message }
        );

        await _mailer.SendMailAsync(new()
        {
            To = to,
            ToName = toName,
            Subject = $"{greeting}, we have a message for you.",
            Body = body
        });
    }
}

The SendAsync method sends the email. It takes a series of parameters and uses them to construct a MessageEmailViewModel object which is passed to the RazorViewRenderer’s Render method. It also specifies MessageEmail as the Razor component to render. RazorViewRenderer returns a string which is then sent to Mailer’s SendMailAsync method as the Body parameter, along with other parameters. The Mailer then uses these parameters to construct an email and send it.

To send emails, this class can be used like so:

// Imagine we're running this in an ASP.NET Core app and getting an instance of MessageMailer via DI...
private readonly MessageMailer _messageMailer;

//... then, within some method, we can do:
await _messageMailer.SendAsync(
    "recipient@example.com",
    "Recipient Name",
    "Mr. Recipient",
    "Have a good day."
);

Simple and clean.

All right! That’s it for now. In this article we saw how to leverage new .NET 8 features for rendering Razor components into strings. We implemented an email sending capability based on that and the MailKit NuGet package. It was a nice way of revisiting an old topic, now made easier thanks to the latest updates from .NET.

Using Razor templates to render HTML emails in .NET

2024-04-30T00:00:00+00:00

Update August 2025: .NET 8 has simplified the process of rendering HTML emails. You can read an updated post on this blog!

When it comes to sending emails, Ruby on Rails has an excellent solution in the form of Action Mailer.

The basic idea is that you can define email templates using ERB files. This is the same templating engine/language used for normal web application views. Then, application-level SMTP settings are configured for email delivery. Finally, a “Mailer” class can be developed that leverages the templates and the underlying email sending mechanism to send emails.

In Rails, all this comes right out the box. Setup is minimal, so this approach is a huge time saver for a task that’s very common in web applications.

In ASP.NET Core (or .NET in general), we don’t have such a convenient, built-in solution. However, it is possible to implement our own using the framework’s features.

In this article, I’m going to explain step by step what I did in a recent .NET project to develop functionality similar to what Action Mailer provides.

Throughout this article, I will be using a demo Web API application for code examples. If you’d like to see what the final implementation looks like, you can find all the code on GitHub. The API is about calculating quotes for used vehicles. Using the process described here, I added the feature to send emails when new quotes are generated.

In fact, I have all of these changes in a single commit. Here’s the diff.

The plan

So here’s the problem statement: I want to be able to send emails from my .NET app. The body of those emails need to be HTML, and they need to be built based on Razor templates — that is, I want to be able to define *.cshtml files for them. I also want to be able to define a “Mailer” class for each specific transaction or event that I want to send emails for. These “Mailer” classes are what the domain logic components will use directly to send the emails. They are the system’s gateway to email sending functionality.

To fulfill those requirements, we will need four elements:

A base component for sending emails.
A component for turning Razor templates (i.e. *.cshtml files) into email bodies.
The actual Razor templates.
A concrete component that domain logic can invoke to send transactional emails.

Step 1: Sending emails in .NET with the MailKit NuGet package

Creating a class that sends emails is easy using the MailKit NuGet package. I ended up using the approach discussed in this article by Dzenana Kajtaz for Mailtrap’s blog.

The first thing to do is to install the MailKit NuGet package. This command will do it:

dotnet add package MailKit --version 4.5.0

The next thing to do is add the necessary SMTP configuration settings into the project’s appsettings.json file. Here’s what one might look like when configured to use Mailtrap.

// VehicleQuotes.WebApi/appsettings.json
{
    // ...
    "MailSettings": {
        "Server": "sandbox.smtp.mailtrap.io",
        "Port": 587, // 25 or 465 or 587 or 2525
        "SenderName": "VehicleQuotes",
        "SenderEmail": "system@vehiclequotes.com",
        "UserName": "YOUR_SMTP_SERVER_USER_NAME",
        "Password": "YOUR_SMTP_SERVER_USER_PASSWORD"
    },
    // ...
}

Next, we define a class that matches the data structure of these settings. Here’s the one I ended up with:

// VehicleQuotes.WebApi/Configuration/MailSettings.cs
namespace VehicleQuotes.WebApi.Configuration;

public class MailSettings
{
    public required string Server { get; set; }
    public required int Port { get; set; }
    public required string SenderName { get; set; }
    public required string SenderEmail { get; set; }
    public required string UserName { get; set; }
    public required string Password { get; set; }
}

Now, to make the settings actually accessible to the system, we need to add them to the Dependency Injection container. I added this to the application’s bootstrapping logic in my Program.cs file, before the call to builder.Build():

builder.Services.Configure<Configuration.MailSettings>(config.GetSection("MailSettings"));

This essentially tells .NET to read the "MailSettings" section from appsettings.json and load the data in an object of type MailSettings. This object will be available to any class in the system thanks to Dependency Injection. We will use it later.

Now we need to define a class for sending emails. This one is low level. All it does is send emails, it doesn’t render Razor templates. Other components will take care of that. So for now, here’s the class responsible for sending emails:

// VehicleQuotes.WebApi/Services/Mailer.cs
using Microsoft.Extensions.Options;
using MailKit.Net.Smtp;
using MimeKit;
using VehicleQuotes.WebApi.Configuration;

namespace VehicleQuotes.WebApi.Services;

public class MailData
{
    public required string To { get; set; }
    public required string ToName { get; set; }
    public required string Subject { get; set; }
    public required string Body { get; set; }
}

public interface IMailer
{
    Task<bool> SendMailAsync(MailData mailData);
}

public class Mailer : IMailer
{
    private readonly MailSettings _mailSettings;

    public Mailer(IOptions<MailSettings> mailSettingsOptions)
    {
        _mailSettings = mailSettingsOptions.Value;
    }

    public async Task<bool> SendMailAsync(MailData mailData)
    {
        try
        {
            using var emailMessage = new MimeMessage();

            emailMessage.From.Add(new MailboxAddress(_mailSettings.SenderName, _mailSettings.SenderEmail));
            emailMessage.To.Add(new MailboxAddress(mailData.ToName, mailData.To));

            emailMessage.Subject = mailData.Subject;

            var emailBodyBuilder = new BodyBuilder
            {
                TextBody = mailData.Body,
                HtmlBody = mailData.Body
            };

            emailMessage.Body = emailBodyBuilder.ToMessageBody();

            using var mailClient = new SmtpClient();

            await mailClient.ConnectAsync(_mailSettings.Server, _mailSettings.Port, MailKit.Security.SecureSocketOptions.StartTls);
            await mailClient.AuthenticateAsync(_mailSettings.UserName, _mailSettings.Password);
            await mailClient.SendAsync(emailMessage);
            await mailClient.DisconnectAsync(true);

            return true;
        }
        catch
        {
            // TODO: log the email delivery failure
            return false;
        }
    }
}

This class is straightforward. It has a single method, SendMailAsync. The method receives the email subject, body and recipient within a MailData object. Then, it uses the conventional MailKit process to send the email: builds the message, sets sender and recipient, sets the body, connects to the server, authenticates, and sends the email.

For this class to be available at runtime, we need to add it to the Dependency Injection container. So, similar to how we did when loading the SMTP server configuration options, we add this line to the Program.cs file before the call to builder.Build().

builder.Services.AddTransient<Services.IMailer, Services.Mailer>();

Ok, with that, our system knows how to send emails. Let’s see about the next step now.

Step 2: Rendering Razor templates into strings

Now that we have our core mailer class, we see that it expects a string to use as a body for the emails it sends. So like I mentioned before, we need a component that can take Razor templates (i.e. *.cshtml files) and turn them into strings. Here’s how that’s done.

This component will live in a new Razor Class Library project. The *.cshtml templates will also live here. We can create the project and add it to the solution with commands like these:

dotnet new razorclasslib -o VehicleQuotes.RazorTemplates -s
dotnet sln add ./VehicleQuotes.RazorTemplates/VehicleQuotes.RazorTemplates.csproj

Now comes the star of the show, the class that renders Razor templates into strings:

using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Mvc;
using Microsoft.AspNetCore.Mvc.Abstractions;
using Microsoft.AspNetCore.Mvc.ModelBinding;
using Microsoft.AspNetCore.Mvc.Razor;
using Microsoft.AspNetCore.Mvc.Rendering;
using Microsoft.AspNetCore.Mvc.ViewEngines;
using Microsoft.AspNetCore.Mvc.ViewFeatures;
using Microsoft.AspNetCore.Routing;
using Microsoft.Extensions.DependencyInjection;

namespace VehicleQuotes.RazorTemplates.Services;

public interface IRazorViewRenderer
{
    Task<string> Render<TModel>(string viewName, TModel model);
}

public class RazorViewRenderer : IRazorViewRenderer
{
    private readonly IRazorViewEngine _viewEngine;
    private readonly ITempDataProvider _tempDataProvider;
    private readonly IServiceScopeFactory _serviceScopeFactory;

    public RazorViewRenderer(
        IRazorViewEngine viewEngine,
        ITempDataProvider tempDataProvider,
        IServiceScopeFactory serviceScopeFactory
    ) {
        _viewEngine = viewEngine;
        _tempDataProvider = tempDataProvider;
        _serviceScopeFactory = serviceScopeFactory;
    }

    public async Task<string> Render<TModel>(string viewName, TModel model)
    {
        using var scope = _serviceScopeFactory.CreateScope();

        var httpContext = new DefaultHttpContext() { RequestServices = scope.ServiceProvider };
        var actionContext = new ActionContext(httpContext, new RouteData(), new ActionDescriptor());

        var view = FindView(actionContext, viewName);

        var viewData = new ViewDataDictionary<TModel>(new EmptyModelMetadataProvider(), new ModelStateDictionary())
        {
            Model = model
        };

        var tempData = new TempDataDictionary(httpContext, _tempDataProvider);

        using var output = new StringWriter();

        var viewContext = new ViewContext(
            actionContext,
            view,
            viewData,
            tempData,
            output,
            new HtmlHelperOptions()
        );

        await view.RenderAsync(viewContext);

        return output.ToString();
    }

    private IView FindView(ActionContext actionContext, string viewName)
    {
        var getViewResult = _viewEngine.GetView(executingFilePath: null, viewPath: viewName, isMainPage: true);
        if (getViewResult.Success)
        {
            return getViewResult.View;
        }

        var findViewResult = _viewEngine.FindView(actionContext, viewName, isMainPage: true);
        if (findViewResult.Success)
        {
            return findViewResult.View;
        }

        var searchedLocations = getViewResult.SearchedLocations.Concat(findViewResult.SearchedLocations);
        var errorMessage = string.Join(
            Environment.NewLine,
            new[] { $"Unable to find view '{viewName}'. The following locations were searched:" }.Concat(searchedLocations)
        );

        throw new InvalidOperationException(errorMessage);
    }
}

This class is complicated. It leverages several obscure framework components that are not very commonly used. I was able to piece it together with help from here, here, and here.

That’s quite a bit of code, but most of it is ceremony in the service of executing two main steps:

Finding the *.cshtml file that corresponds to the given viewName.
Preparing an IView object that can be used to render the template. It does so using the given model object which contains the actual data to fill out the template placeholders.

Finally, in order to make this class available to the system, we add it to Dependency Injection. Here’s what that looks like:

builder.Services.AddMvcCore().AddRazorViewEngine(); // Necessary for non-GUI projects.
builder.Services.AddTransient<RazorTemplates.Services.IRazorViewRenderer, RazorTemplates.Services.RazorViewRenderer>();

The only interesting thing here is the services.AddMvcCore().AddRazorViewEngine(); line. I had to add that to my project because it is a Web API. That means that it doesn’t include all the services related to rendering views. Other project types that already include all the view-related services, like MVC or Razor Pages, may not need this line. Remember that our RazorViewRenderer class depends on all sorts of framework objects. This line makes sure that they are available.

Step 3: Defining the email templates

Now that our system knows how to render Razor templates into strings, let’s go ahead and actually implement some. The nice thing about this approach is that these templates are full Razor views. That means that features like layouts and partials are supported.

For the purposes of this demo, a simple layout with a header and a footer will suffice. Along with the body of the particular transactional email that we want to send.

Our layout could look like this:

<!-- VehicleQuotes.RazorTemplates/Views/Shared/EmailLayout.cshtml -->
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Email With Razor Templates</title>
</head>
<body>
    <p>Imagine this is a header</p>
    @RenderBody()
    <p>Imagine this is a footer</p>
</body>
</html>

Very simple as layouts go. It does little more than define the basic HTML document structure and call @RenderBody().

We can also add a _ViewStart.cshtml file that specifies this layout as the layout to use for all other templates under the Emails directory:

<!-- VehicleQuotes.RazorTemplates/Views/Emails/_ViewStart.cshtml -->
@{
    Layout = "EmailLayout";
}

Now, for the core contents of the email, we define a new template, along with a class that will serve as its view model.

My template ended up looking like this:

<!-- VehicleQuotes.RazorTemplates/Views/Emails/QuoteGenerated.cshtml -->
@using VehicleQuotes.RazorTemplates.ViewModels

@model QuoteGeneratedViewModel

<p>
    A quote has been generated for a @Model.Year @Model.Make @Model.Model.
</p>

<p>Quote ID: @Model.ID</p>
<p>Created At: @Model.CreatedAt.ToLongDateString()</p>
<p>Offered Amount: @Model.OfferedQuote</p>
<p>Message: @Model.Message</p>

And the view model:

// VehicleQuotes.RazorTemplates/ViewModels/QuoteGeneratedViewModel.cs
namespace VehicleQuotes.RazorTemplates.ViewModels;

public class QuoteGeneratedViewModel
{
    public required int ID { get; set; }
    public required DateTime CreatedAt { get; set; }
    public required int OfferedQuote { get; set; }
    public required string Message { get; set; }
    public required string Year { get; set; }
    public required string Make { get; set; }
    public required string Model { get; set; }
}

Very simple. The template specifies the type that it accepts as a view model using the @model directive. It then proceeds to render the email contents using regular old Razor syntax. The view model is just a simple POCO that defines the data that the template can work with.

Step 4: Putting it all together: the class that sends the “quote generated” email

Finally, we can define our specific Mailer class that, leveraging all the infrastructure we’ve put together, can send one specific type of transactional email. These classes are meant to be simple and boring. Here’s mine:

// VehicleQuotes.WebApi/Services/QuoteGeneratedMailer.cs
using VehicleQuotes.RazorTemplates.Services;
using VehicleQuotes.WebApi.ResourceModels;
using VehicleQuotes.RazorTemplates.ViewModels;

namespace VehicleQuotes.WebApi.Services;

public class QuoteGeneratedMailer
{
    private readonly IMailer _mailer;
    private readonly IRazorViewRenderer _razorViewRenderer;

    public QuoteGeneratedMailer(IMailer mailer, IRazorViewRenderer razorViewRenderer)
    {
        _mailer = mailer;
        _razorViewRenderer = razorViewRenderer;
    }

    public async Task SendAsync(QuoteGeneratedViewModel payload)
    {
        string body = await _razorViewRenderer.Render(
            "/Views/Emails/QuoteGenerated.cshtml", payload
        );

        await _mailer.SendMailAsync(new() {
            To = "test@email.com",
            ToName = "Mr. Recipient",
            Subject = $"VehicleQuotes - New Quote Generated - Quote #{payload.ID}",
            Body = body
        });
    }
}

Pretty neat, huh? This class receives instances of the base Mailer and the RazorViewRenderer via Dependency Injection and uses them to: 1. render the template and 2. send the email.

Like everything else, it also needs to be made available via Dependency Injection. All in all, I ended up with this nice bundle in my Program.cs file:

builder.Services.Configure<Configuration.MailSettings>(config.GetSection("MailSettings"));
builder.Services.AddScoped<Services.QuoteGeneratedMailer>();
builder.Services.AddMvcCore().AddRazorViewEngine();
builder.Services.AddTransient<RazorTemplates.Services.IRazorViewRenderer, RazorTemplates.Services.RazorViewRenderer>();
builder.Services.AddTransient<Services.IMailer, Services.Mailer>();

A good idea is to put these into an IServiceCollection extension method. That’s what I ended up doing, in fact.

Instances of a class like this can be used anywhere in the code. For example like this:

// VehicleQuotes.WebApi/Services/QuoteService.cs
// _mailer is a QuoteGeneratedMailer
// Imagine response is an object that contains all these fields.
await _mailer.SendAsync(
    new QuoteGeneratedViewModel
    {
        ID = response.ID,
        CreatedAt = response.CreatedAt,
        OfferedQuote = response.OfferedQuote,
        Message = response.Message,
        Year = response.Year,
        Make = response.Make,
        Model = response.Model
    }
);

Just build the view model object that it expects and off it goes.

And that’s it! That definitely took some elbow grease to get working as well as delving into pretty arcane framework features. In the end, however, we did manage to build something that offers a developer experience that’s very similar to Action Mailer. Once the core Mailer and the RazorViewRenderer are in place, all it takes to send a new transactional email is:

Defining a new template, with its view model.
Defining a new Mailer class that renders the template, uses it as the email’s body, and sends it.

Working around SPF problems delivering to Gmail

2022-03-30T00:00:00+00:00

Photo by Garrett Skinner

Email deliverability

Legitimate email delivery keeps getting harder. Spammers and phishers never stop flooding everyone’s inboxes with unwanted and harmful email, so automated defenses against junk mail are necessary. But they are not perfect, and good email sometimes gets flagged as spam.

When sending important “transactional” email such as for account confirmations, password resets, and ecommerce receipts, it is often worth using a paid email delivery service to increase deliverability. Those typically cost a flat amount per month for up to a certain quota of outgoing email, with overage charges for messages beyond that.

Many of our clients use one of those services and generally they have all worked well and differ mostly in pricing and feature set. Popular choices include SendGrid, Mandrill, Postmark, Mailgun, and Amazon SES.

We continue to have many cases where we want to be able to send potentially large amounts of automated email to ourselves, our clients, or our systems. This is usually for testing, notifications, or internal delivery to special mailboxes separate from our main mailboxes.

These other uses for sending email keep us involved in the fight for good email deliverability from our own servers, which we have worked at over many years, long predating these paid email delivery services.

Sender Policy Framework

One of the longest-running tools to fight spam is SPF, the Sender Policy Framework.

SPF is an open standard that provides a way for a receiving mail server to verify that the sending server is authorized to send email for the message’s “envelope” sender domain. The envelope sender address or “return-path” is not normally seen by email recipients, but is used behind the scenes by servers. It may or may not be the same as the sender seen in the “From” header.

The SPF policy for each domain is set in a special DNS TXT record for that domain.

The important thing is that each sender’s email belongs to a domain with a valid SPF record showing that the sending servers are allowed to send for that domain, and that all other servers should not be allowed to send email for that domain.

For example, our endpointdev.com domain currently has this TXT record to define its SPF policy:

v=spf1 a:maildrop.endpointdev.com include:_spf.google.com include:servers.mcsv.net -all

Let’s look at each of those space-separated elements:

v=spf1 designates this TXT record as an SPF policy, version 1 (the only one so far).
a:maildrop.endpointdev.com means to allow the A (IPv4) and/or AAAA (IPv6) IP address(es) of hostname maildrop.endpointdev.com as a valid source.
include:_spf.google.com means to look up another DNS TXT record at _spf.google.com (for Gmail, our main email provider here) and add its SPF policy to ours.
include:servers.mcsv.net is the same thing, but for servers.mcsv.net (for Mailchimp, to allow it to deliver email newsletters for our domain).
-all means to disallow any other senders.

With such a policy, receiving mail servers can immediately reject any incoming email claiming to be sent by us for our domain if it didn’t come from one of our designated servers.

This obviously doesn’t stop all spam, but it stops a whole class of forged senders, which is very helpful.

The key point to note is that SPF applies to the sending email server at the moment it connects to the receiving email server. It doesn’t deal with anything else.

One other point to note is that SPF policies are limited to a fairly small total number of DNS lookups via include elements, so we can’t endlessly add new valid sending servers to our list.

Email server trails

Based on the above SPF policy, if we want to send email from address notifier@endpointdev.com, it will have to be sent through maildrop.endpointdev.com, Gmail, or Mailchimp. Messages coming from any other sending server should be rejected by the receiving server. They don’t have to behave that way, but it is in their interest to do so if they don’t like spam.

We have an internal server we’ll call dashboard.endpointdev.com, which sends email notifications from address notifier@endpointdev.com.

Since we don’t want to bloat our SPF policy, we’ll have our server dashboard.endpointdev.com route its outgoing email through our mail forwarding service called maildrop, which lives on two or more servers behind the DNS name maildrop.endpointdev.com.

This is a good idea for several reasons:

It keeps all our outgoing email flowing through a few places so we can easily monitor them for any problems.
We don’t need to have SMTP daemons running on all our servers just to send outbound email.
We don’t need to worry about the quotas or pricing of commercial emailing services when sending less-important or internal-only email.

Since SPF is designed for a receiving email server to check that the server connecting to it to send email is authorized to do so for that email address’s domain, it shouldn’t matter what server the email originated on.

Gmail misuses header information in SPF checks

We recently discovered that Gmail has been misusing email header information in its SPF checks.

When one of our outgoing emails originated from server dashboard.endpointdev.com and was then forwarded to maildrop.endpointdev.com which then delivered it to Gmail, Gmail looked at the earliest sender server it could find in the Received headers of the email message, found dashboard.endpointdev.com, and flagged it as an SPF failure because our SPF policy didn’t include dashboard.endpointdev.com [206.191.128.233].

This can be seen in this excerpt of relevant email headers. (Some specific details here were changed to protect the innocent.) Note that email headers appear in reverse chronological order, so the most recent events are at the top:

Received: from maildrop14.epinfra.net (maildrop14.epinfra.net. [69.25.178.35])
        by mx.google.com with ESMTPS id l20si5561179oos.78.2022.01.25.10.52.05
        for <notifications@endpointdev.com>
        (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
        Tue, 25 Jan 2022 10:52:05 -0800 (PST)
Received-SPF: fail (google.com: domain of notifier@endpointdev.com does not designate 206.191.128.233 as permitted
    sender) client-ip=206.191.128.233;
Authentication-Results: mx.google.com;
       dkim=pass header.i=@endpointdev.com header.s=maildrop header.b=hR445V77;
       spf=fail (google.com: domain of notifier@endpointdev.com does not designate 206.191.128.233 as permitted
    sender) smtp.mailfrom=notifier@endpointdev.com
Received: from dashboard.endpointdev.com (dashboard.endpointdev.com [206.191.128.233])
    by maildrop14.epinfra.net (Postfix) with ESMTP id A2AA03E8A7
    for <notifications@endpointdev.com>; Tue, 25 Jan 2022 18:52:05 +0000 (UTC)
To: <notifications@endpointdev.com>

That is wrong! The SPF check should have been done against maildrop14.epinfra.net [69.25.178.35] because that is the IP address that actually connected to Gmail to send the email. That server is one of our infrastructure hostnames allowed to send email as part of the maildrop.endpointdev.com DNS record, so checking it would have led Gmail to give a passing SPF result.

Why did Gmail do this? I don’t know, and at the time didn’t find any public discussion that would explain it. I suspect it has something to do with Gmail’s internal systems being comprised of many, many servers, and the SPF check being done long after the email was passed on from the initial receiving point through various other servers. Then Gmail parses the headers to find out who the sender was, and gets confused.

We can avoid this problem by not having maildrop mention our original sending server dashboard.endpointdev.com at all.

Why should it mention it in the first place? It’s helpful for tracing problems when debugging, but really is TMI (too much information) for normal email sending, and exposes internal infrastructure details that would be better omitted anyway.

Since dashboard.endpointdev.com is running the very flexible and configurable Postfix email server, we can direct it to remove any Received headers that mention our internal hostnames.

By default Postfix in /etc/postfix/main.cf has the header_checks directive set to look at a table to match regular expressions and take specified actions.

So we added a regular expression to match and designated the action IGNORE, to the file /etc/postfix/header_checks:

/^Received:\ (from|by)\ .*(epinfra\.net|endpointdev\.com|localhost|localdomain)/  IGNORE

Then we update the map database file so that it takes immediate effect for new email flowing through Postfix:

postmap /etc/postfix/header_checks

When we sent another notification email from dashboard.endpointdev.com and received it in Gmail we saw the email’s headers look like this:

Received: from maildrop14.epinfra.net (maildrop14.epinfra.net. [69.25.178.35])
        by mx.google.com with ESMTPS id g72si1894187vke.271.2022.01.25.11.03.37
        for <notifications@endpointdev.com>
        (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
        Tue, 25 Jan 2022 11:03:37 -0800 (PST)
Received-SPF: pass (google.com: domain endpointdev.com configured 69.25.178.35 as internal address)
Authentication-Results: mx.google.com;
       dkim=pass header.i=@endpointdev.com header.s=maildrop header.b=qkccUkkU;
       spf=pass (google.com: domain endpointdev.com configured 69.25.178.35 as internal address)
    smtp.mailfrom=notifier@endpointdev.com
To: <notifications@endpointdev.com>

There is no more mention of dashboard or its IP address, so Gmail runs its SPF check against the proper IP address 69.25.178.35 which belongs to server maildrop14.epinfra.net which is part of the maildrop.endpointdev.com DNS name. Gmail now validates that IP address is allowed to send for the endpointdev.com domain and gives a “pass” result for its SPF check.

Perhaps this will help your legitimate email delivery too!

Reference

Capturing Outgoing Email With Mock SMTP Servers

2020-03-13T00:00:00+00:00

Photo by Seattleye, used under CC BY 2.0, cropped from original.

Sending automated email to users is a common requirement of most web applications and can take the form of things like password reset emails or order confirmation invoices.

It is important for developers working in development/staging environments to verify that an application is sending email correctly without actually delivering messages to users’ inboxes. If you were testing a background task that searches an e-commerce site for abandoned shopping carts and emails users to remind them that they have not completed a checkout, you would not want to run that in development and end up repeatedly emailing live user email addresses.

A mock SMTP server is useful for development and testing because it lets you configure the email settings of your development environment almost exactly the same as you would for outgoing SMTP email in your production site. The mock SMTP server will capture all of the outbound email and allow you to review it in a web interface instead of actually delivering it to users’ inboxes.

Mock SMTP Servers

There are a variety of standalone/free and hosted/commercial options for mock SMTP servers including:

MailHog (free)
MailSlurper (free)
MailCatcher (free)
Mailtrap (free for small solo projects/paid)
Mailosaur (paid)

The standalone/free options have been sufficient for the projects I have worked on. Some of the features offered by the hosted solutions like Mailtrap and Mailosaur may be appealing to larger development teams.

MailHog is my go-to mock SMTP server because it has a nice web interface and is extremely easy to install and configure for typical use. The standalone solutions that I have tried all work similarly; they listen for SMTP email on one port, and provide a web interface on a separate port for reviewing captured email.

Configuring a Rails Application to use MailHog

Installation and use of MailHog is very simple: download and run the mailhog executable to have it start listening for SMTP email on port 1025 and hosting a web interface on port 8025. Then edit the Rails development environment config file to send email using an SMTP server running on local port 1025:

# config/environments/development.rb
Rails.application.configure do
# …
  config.action_mailer.delivery_method = :smtp
  config.action_mailer.perform_deliveries = true
  config.action_mailer.smtp_settings = { address: 'localhost', port: 1025 }
# …
end

MailHog runs in the foreground and will display output as it receives outbound email. Opening a web browser to http://localhost:8025 shows MailHog’s web interface and allows you to view a list of the email that it has captured or click on an email to show its full details:

Conclusion

I have found MailHog to be helpful when developing a variety of different web applications, particularly ones that generate email for user notifications and confirmations. Thanks to its ease of installation and use, I would recommend it (or an equivalent mock SMTP solution) to any developer that needs to test outgoing email features in their web applications.

Linux desktop Postfix queue for Gmail SMTP

2019-04-30T00:00:00+00:00

On a Linux desktop, I want to start sending email through Gmail in a G Suite account using SMTP, rather than a self-hosted SMTP server. Since Gmail supports SMTP, that should be easy enough.

Google’s article Send email from a printer, scanner, or app gives an overview of several options. I’ll choose the “Gmail SMTP server” track, which seems designed for individual user cases like this.

However, since I am using two-factor authentication (2FA) on this Google account — as we should all be doing now for all accounts wherever possible! — my Gmail login won’t work for SMTP because the clients I am using don’t have a way to supply the 2FA time-based token.

Google’s solution to this is to have me generate a separate “App Password” that can sidestep 2FA for this limited purpose: Set up an App Password.

That works fine, but the app password is a randomly-generated 16-letter password that is not amenable to being memorized. For security reasons, my mail client doesn’t cache passwords between sessions, so I have to look it up and enter it each time I start the mail client. That’s generally only once per day for me, so it’s not a big problem, but it would be nice to avoid.

I also want other local programs — such as cron jobs, development projects underway, etc. — to be able to send mail out through my Gmail account. How can I do that, ideally without teaching each one separately how to do it?

As a server operating system at heart, Linux of course has many SMTP servers that can intermediate by acting as a local SMTP server, queue, and sending client. Such a server could have my Gmail password configured and stored under a separate user account, giving a bit more isolation from my main desktop user.

What local SMTP program to use?

esmtp

I first tried using the lightweight and ephemeral esmtp since I had already used it on my desktop computer to forward email through an SSH tunnel. I wasn’t able to get it working with Gmail, which could easily have been operator error on my part.

Before trying much to solve the problem, I realized that I really would like a local queue for outgoing email so I don’t have to wait for my mail client to connect and send each message before I can get on with more email. Given how much email I handle each day, even fairly brief delays add unwanted drag.

ssmtp

I used another similar program called ssmtp a long time ago, and considered trying that again.

Then I read that ssmtp is unmaintained and does not validate TLS certificates, negating some of the security value of using TLS in the first place. So, no to that.

E-MailRelay

I next saw a few people mention that E-MailRelay is a nice option to locally queue and forward email. But looking at a new and more comprehensive email daemon like that, I realized I would likely find it easier to just use an SMTP server I already know, such as Exim, Sendmail, or …

Postfix

Postfix is one of the most widely-deployed mail servers. I already know it well and have used it for many years. It has most every option I would ever want. So I’ll use that.

This seems like a slight bit of overkill for just sending outgoing email, but Postfix is battle-hardened and comparatively lightweight, using around 30 MiB resident RAM between its 4 daemon processes on my computer. Yes, that is laughably bloated by the standards of yore, but svelte compared to even a single tab in a modern graphical browser.

Configuring Postfix

(You’ll need to be root to do the following setup.)

First, install Postfix as appropriate for your Linux distribution:

# dnf install postfix    # Fedora
# yum install postfix    # CentOS/RHEL
# apt install postfix    # Debian/Ubuntu

Now, time to edit the default Postfix configuration. The options are all well-documented on the Postfix website but there are a lot of them and it can take a while to figure out what you need and want.

Here is what I added to /etc/postfix/main.cf:

relayhost = [smtp.gmail.com]:465

smtp_tls_wrappermode = yes
smtp_tls_security_level = verify
smtp_tls_mandatory_protocols = !SSLv2, !TLSv1, !TLSv1.1
tls_high_cipherlist = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
smtp_tls_mandatory_ciphers = high
smtp_tls_loglevel = 2

smtp_sasl_auth_enable = yes
smtp_sasl_mechanism_filter = plain
smtp_sasl_password_maps = hash:/etc/postfix/smtp_auth
smtp_sasl_security_options = noanonymous

header_checks = regexp:/etc/postfix/header_checks

smtp_address_preference = ipv6

Let’s go through each of those settings:

With relayhost we define the remote server we want to route all our outgoing mail through. The [...] around the hostname disables MX lookups, since there is no MX record for smtp.gmail.com. Port 465 is used for TLS-wrapped (called “implicit”) mail submission, so we also need to set smtp_tls_wrappermode.

I used the port number 465 rather than the name smtps because the port has been reassigned for another purpose, so it may be mentioned in the /etc/services file under either the old or new names or both, and isn’t worth risking breakage for.

Enabling smtp_tls_wrappermode means we initiate a TLS connection immediately, rather than first connecting with classic plain-text SMTP protocol, and then requesting a switch to TLS mode with the STARTTLS SMTP verb. This option is why we specified port 465 above for our relayhost.

Like me, you may recall the option for SSL-wrapped SMTP to port 465 as little more than an odd Microsoft Exchange and Outlook convention from years ago. Once upon a time it was. Why not use the more standard submission port 587? Up until fairly recently that would have been best.

In a funny turn of events, nowadays TLS-wrapped SMTP on port 465 is recommended as the best way for end-users to submit mail! See RFC 8314 for details. I found the helpful FastMail explanation of all the historical twists well worth reading. As they say, the best thing about standards is that there are so many to choose from. 😆

Setting smtp_tls_security_level to verify means that we want TLS to be mandatory for every connection, and we want Postfix to validate that the certificate name matches the hostname we are using to connect to prevent man-in-the-middle (MITM) attacks.

With smtp_tls_mandatory_protocols we are disabling all older TLS protocols, allowing only the most current (as of this writing) versions 1.2 and 1.3. That would be overly restrictive and unsafe to do on a public mail server, but since we are only talking to a single mail server here, and we know that Google supports modern TLS, we can restrict ourselves to that.

The tls_high_cipherlist list is our restricted set of TLS ciphers in our preferred order. Again, this would be unwise to do on a public mail server, but in our role as a client forwarding to only one destination here, it is a good thing. My list comes from Mozilla’s modern TLS recommendation.

With smtp_tls_mandatory_ciphers set to high, we ensure the list we just specified gets used.

I set smtp_tls_loglevel to 2 so that the Postfix logs will show helpful details about TLS connection negotiations and results.

The next several smtp_sasl_ options configure our use of a username and password. (SASL means “Simple Authentication and Security Layer”.) Note the smtp_sasl_password_maps file we specified. The file name is arbitrary. Let’s create that now in /etc/postfix/smtp_auth:

[smtp.gmail.com]:465    user@gsuite.domain:apppassword

Of course substitute your own email address and Gmail app password there.

We need to create a fast binary map equivalent of that file for Postfix to read, and since it contains a password that should be kept private, let’s make it unreadable by other users on the system:

# postmap hash:smtp_auth
# chmod go= smtp_auth*

Next I set option header_checks to look for regular expressions in another file we need to create, /etc/postfix/header_checks:

/^Received:\ (from|by)\ .*(yourhostname|localhost|localdomain)/      IGNORE

That removes a pair of headers that Postfix normally adds to each message we send, tracking the receipt and forwarding on of the email:

Received: by yourhostname.localdomain (Postfix, from userid 1000)
    id F2F7111C64A1; Tue, 30 Apr 2019 17:36:12 -0600 (MDT)
Received: from localhost (localhost [127.0.0.1])
    by yourhostname.localdomain (Postfix) with ESMTP id EE7DB11C161C;
    Tue, 30 Apr 2019 17:36:12 -0600 (MDT)

In our case those are a waste of space because it is not interesting to track the flow of email around localhost. So we just remove them before sending the mail on.

Finally, a minor nicety I like to enable is to set smtp_address_preference to ipv6 so that when we have an IPv6 connection to the outside world, that is used to send the mail. An IPv4 connection will still be used if IPv6 isn’t available. I figure we’d might as well use the newer routing tubes of the Internet when we can.

Trying it out

Now we’re ready to start Postfix, and set it to start automatically at boot time:

# systemctl start postfix
# systemctl enable postfix

Now you need to configure your mail client. My enthusiasm for Pine (now Alpine) hasn’t waned after over 20 years, so that is what I use. Perhaps you prefer Mutt or Thunderbird or something else.

Whatever it is, set your mail client to send mail through local Postfix executable /usr/sbin/sendmail (the default for many), or through SMTP to localhost:25. For me, that meant editing ~/.pinerc. When I was sending mail directly from Pine to Gmail, it contained:

smtp-server=smtp.gmail.com/submit/tls/user=you@your.domain

But now I want there to be no setting so it falls back to /usr/sbin/sendmail:

smtp-server=

Now watch your Postfix logs, usually in /var/log/mail.log on Debian and Ubuntu systems, and /var/log/maillog on Fedora and CentOS.

And send a message!

Let’s look at what Postfix logged when I sent a test message.

When using rsyslog my complete log lines look like this:

2019-04-30T17:59:11.887004-06:00 localhost postfix/smtpd[10962]: connect from localhost[127.0.0.1]

That timestamp and hostname prefix waste a lot of room on each line here, so I will trim them from the rest of the xample:

postfix/smtpd[10962]: connect from localhost[127.0.0.1]
postfix/smtpd[10962]: DA75311C649C: client=localhost[127.0.0.1]
postfix/smtpd[10962]: disconnect from localhost[127.0.0.1] ehlo=1 mail=1 rcpt=1 data=1 quit=1 commands=5
postfix/pickup[10947]: DE8FE11C64A1: uid=1000 from=<you@your.domain>
postfix/cleanup[10964]: DE8FE11C64A1: message-id=<f15302b5-cd03-488a-a401-e1eb8544895f@ybpnyubfg>
postfix/qmgr[10948]: DE8FE11C64A1: from=<you@your.domain>, size=393, nrcpt=1 (queue active)
postfix/smtp[10967]: initializing the client-side TLS engine
postfix/smtp[10967]: setting up TLS connection to smtp.gmail.com[2607:f8b0:4001:c03::6c]:465
postfix/smtp[10967]: smtp.gmail.com[2607:f8b0:4001:c03::6c]:465: TLS cipher list "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:!aNULL"
postfix/smtp[10967]: SSL_connect:before SSL initialization
postfix/smtp[10967]: SSL_connect:SSLv3/TLS write client hello
postfix/smtp[10967]: SSL_connect:SSLv3/TLS write client hello
postfix/smtp[10967]: SSL_connect:SSLv3/TLS read server hello
postfix/smtp[10967]: smtp.gmail.com[2607:f8b0:4001:c03::6c]:465: depth=2 verify=1 subject=/OU=GlobalSign Root CA - R2/O=GlobalSign/CN=GlobalSign
postfix/smtp[10967]: smtp.gmail.com[2607:f8b0:4001:c03::6c]:465: depth=1 verify=1 subject=/C=US/O=Google Trust Services/CN=Google Internet Authority G3
postfix/smtp[10967]: smtp.gmail.com[2607:f8b0:4001:c03::6c]:465: depth=0 verify=1 subject=/C=US/ST=California/L=Mountain View/O=Google LLC/CN=smtp.gmail.com
postfix/smtp[10967]: SSL_connect:SSLv3/TLS read server certificate
postfix/smtp[10967]: SSL_connect:SSLv3/TLS read server key exchange
postfix/smtp[10967]: SSL_connect:SSLv3/TLS read server done
postfix/smtp[10967]: SSL_connect:SSLv3/TLS write client key exchange
postfix/smtp[10967]: SSL_connect:SSLv3/TLS write change cipher spec
postfix/smtp[10967]: SSL_connect:SSLv3/TLS write finished
postfix/smtp[10967]: SSL_connect:SSLv3/TLS write finished
postfix/smtp[10967]: SSL_connect:SSLv3/TLS read server session ticket
postfix/smtp[10967]: SSL_connect:SSLv3/TLS read change cipher spec
postfix/smtp[10967]: SSL_connect:SSLv3/TLS read finished
postfix/smtp[10967]: smtp.gmail.com[2607:f8b0:4001:c03::6c]:465: Matched subjectAltName: smtp.gmail.com
postfix/smtp[10967]: smtp.gmail.com[2607:f8b0:4001:c03::6c]:465 CommonName smtp.gmail.com
postfix/smtp[10967]: smtp.gmail.com[2607:f8b0:4001:c03::6c]:465: subject_CN=smtp.gmail.com, issuer_CN=Google Internet Authority G3, fingerprint=DF:CB:AA:81:ED:77:D4:BE:E5:47:6F:0E:A3:44:99:BA, pkey_fingerprint=EE:0F:3A:CC:6E:4E:EB:C0:1D:88:B6:73:BD:42:C4:83
postfix/smtp[10967]: Verified TLS connection established to smtp.gmail.com[2607:f8b0:4001:c03::6c]:465: TLSv1.2 with cipher ECDHE-RSA-CHACHA20-POLY1305 (256/256 bits)
postfix/smtp[10967]: DE8FE11C64A1: to=<them@their.domain>, relay=smtp.gmail.com[2607:f8b0:4001:c03::6c]:465, delay=1.3, delays=0.03/0.03/0.52/0.75, dsn=2.0.0, status=sent (250 2.0.0 OK  1556668753 y199sm14693187iof.88 - gsmtp)
postfix/qmgr[10948]: DE8FE11C64A1: removed

We can see that the TLS connection was made, negotiated over IPv6 with TLS 1.2 and a nice modern cipher, and the certificate matches the hostname we used. Then the email was sent.

Google has some of their SMTP servers offering the newer, unfinalized TLS 1.3 protocol, since I see that about half the time:

postfix/smtp[13812]: Verified TLS connection established to smtp.gmail.com[2607:f8b0:4001:c06::6d]:465: TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256

And I received my test mail on the other end.

Conclusion

Now I can use any local mail client without having to separately configure each one to send outgoing mail to Gmail, and without having to enter the Gmail app password each time I start my mail client.

Sending email is immediate since the client only waits for the fast local queueing to complete, and Postfix forwards the mail in the background.

Remember to check your logs to make sure your mail is getting delivered, since your client will no longer know if it is not. If nobody replies to you for too many hours, check the logs first! Or set up something to monitor your logs for errors and alert you.

This approach is also useful on servers that need to send application email from a Gmail account, such as when you want mail to have a From: header with a @gmail.com or G Suite domain address.

Thanks, open source community, for the good software and documentation and open standards!

Reach customers and drive sales with MailChimp

2016-08-22T00:00:00+00:00

It’s a good idea for ecommerce stores to regularly contact their customers. This not only reminds customers that your business exists, but also allows the sharing of new products and resources that can enrich the lives of your customers and clients. One of the easiest ways to stay in touch is by using an email newsletter service, such as MailChimp.

<img alt="" border="0" height="200" src="/blog/2016/08/reach-customers-and-drive-sales-with/image-0.png" title="Freddie, the MailChimp mascot" width="191"/>

MailChimp offers the regular suite of email newsletter services: lists, campaigns, and reports—but in addition, they allow an ecommerce store to integrate sales data back into MailChimp. When you have detailed shopping statistics for each subscriber, it opens new possibilities for customized marketing campaigns.

Endless possibilities

For example, imagine you have an email mailing list with 1,000 recipients. Instead of mailing the same generic newsletter to each subscriber, what if you could segment the list to identify your 100 best customers, and email them a special campaign?

Additional ideas could include:

Reach out to inactive subscribers, offering a coupon
Invite your best customers to a secret sale
Re-engage customers who placed items in their cart, but left without purchasing
Offer complementary products to purchasers of Product X

Automatic marketing

Once your store has sales data for subscribers, and you’ve decided on the campaigns you want to run with this data, the next step is to automate the process. This is where MailChimp’s Automation feature comes in. Spend some time up-front to craft the automated campaigns, then sit back and let MailChimp run them for you; day in, day out.

Steps to implement

There are several off-the-shelf integrations for ecommerce stores, including Magento and BigCommerce.

Users of Perl and Interchange can use our newly-released toolsets of the Mail::Chimp3 CPAN module and the integration for Interchange5.

Most businesses already have an email newsletter. Hopefully, you are sending regular email campaigns with it. This is a great first step. Going beyond this to segment your email list and reach out to these segments with relevant information to each of them, is the next step. Not only can this increase your sales, but it also respects your clients’ and customers’ time and preferences. It’s a win-win for all.

Additional resource: MailChimp for Online Sellers

Taking control of your IMAP mail with IMAPFilter

2015-11-06T00:00:00+00:00

Organizing and dealing with incoming email can be tedious, but with IMAPFilter’s simple configuration syntax you can automate any action that you might want to perform on an email and focus your attention on the messages that are most important to you.

Most desktop and mobile email clients include support for rules or filters to deal with incoming mail messages but I was interested in finding a client-agnostic solution that could run in the background, processing incoming messages before they ever reached my phone, tablet or laptop. Configuring a set of rules in a desktop email client isn’t as useful when you might also be checking your mail from a web interface or mobile client; either you need to leave your desktop client running 24/7 or end up with an unfiltered mailbox on your other devices.

I’ve configured IMAPFilter to run on my home Linux server and it’s doing a great job of processing my incoming mail, automatically sorting things like newsletters and automated Git commit messages into separate mailboxes and reserving my inbox for higher priority incoming mail.

IMAPFilter is available in most package managers and easily configured with a single ~/.imapfilter/config.lua file. A helpful example config.lua is available in IMAPFilter’s GitHub repository and is what I used as the basis for my personal configuration.

A few of my favorite IMAPFilter rules (where “endpoint” is configured as my work IMAP account):

<span style="color: #75715e">-- Mark daily timesheet reports as read, move them into a Timesheets archive mailbox</span>
<span style="color: #f8f8f2">timesheets</span> <span style="color: #f92672">=</span> <span style="color: #f8f8f2">endpoint[</span><span style="color: #e6db74">'INBOX'</span><span style="color: #f8f8f2">]:contain_from(</span><span style="color: #e6db74">'timesheet@example.com'</span><span style="color: #f8f8f2">)</span>
<span style="color: #f8f8f2">timesheets:mark_seen()</span>
<span style="color: #f8f8f2">timesheets:move_messages(endpoint[</span><span style="color: #e6db74">'Archive/Timesheets'</span><span style="color: #f8f8f2">])</span>

<span style="color: #75715e">-- Sort newsletters into newsletter-specific mailboxes</span>
<span style="color: #f8f8f2">jsweekly</span> <span style="color: #f92672">=</span> <span style="color: #f8f8f2">endpoint[</span><span style="color: #e6db74">'INBOX'</span><span style="color: #f8f8f2">]:contain_from(</span><span style="color: #e6db74">'jsw@peterc.org'</span><span style="color: #f8f8f2">)</span>
<span style="color: #f8f8f2">jsweekly:move_messages(endpoint[</span><span style="color: #e6db74">'Newsletters/JavaScript Weekly'</span><span style="color: #f8f8f2">])</span>

<span style="color: #f8f8f2">hn</span> <span style="color: #f92672">=</span> <span style="color: #f8f8f2">endpoint[</span><span style="color: #e6db74">'INBOX'</span><span style="color: #f8f8f2">]:contain_from(</span><span style="color: #e6db74">'kale@hackernewsletter.com'</span><span style="color: #f8f8f2">)</span>
<span style="color: #f8f8f2">hn:move_messages(endpoint[</span><span style="color: #e6db74">'Newsletters/Hacker Newsletter'</span><span style="color: #f8f8f2">])</span>

Note that IMAPFilter will create missing mailboxes when running “move_messages”, so you don’t need to set those up ahead of time. These are basic examples but the sample config.lua is a good source of other filter ideas, including combining messages matching multiple criteria into a single result set.

In addition to these basic rules, IMAPFilter also supports more advanced configurations including the ability to perform actions on messages based on the results of passing their content through an external command. This opens up possibilities like performing your own local spam filtering by sending each message through SpamAssassin and moving messages into spam mailboxes based on the exit codes returned by spamc. As of this writing I’m still in the process of training SpamAssassin to reliably recognize spam vs. ham but hope to integrate its spam detection into my own IMAPFilter configuration soon.

Postfix Address Verification

2015-05-28T00:00:00+00:00

We recently upgraded some mail servers, moving from Exim to Postfix in the process. These server works as a front line spam/RBL filter, rejecting invalid message and relaying valid ones to different SMTP based on the destination domain.

While looking for the best configuration layout to achieve this, we found that Postfix has a very useful and interesting feature: Address Verification. This technique allows the Postfix server to check that a sender or a recipient address is valid before accepting a message, preventing junk messages from entering the queue.

How does Address Verification work?

Upon receiving a message Postfix will probe the preferred MTA for the address. If that address is valid the message is accepted and processed, otherwise it is rejected.

Message Probes does not actually go through the whole delivery process; Postfix will just connect to the MTA, send a HELO + MAIL FROM + RCPT TO sequence and check its response. Probe checks results are cached on disk, minimizing network and resource impact. During this check the client is put “on hold”; if the probe takes too much a temporary reject is given; a legitimate mail server will have no problem retrying the delivery later, when the cached result will likely be available.

Everything is highly configurable: response codes, timeouts, cache storage type and location, and so on.

Configure Recipient Address Verification

In our case, we wanted to only accept messages with a valid recipient address. Recipient Address Verification took care of this in a very smooth and elegant way.

Adding Recipient Address Verification it’s easy. Just add these lines to /etc/postfix/main.cf:

# Your relaying configuration will already be in place. For example:
# relayhost = [next.hop.ip.address]

smtpd_recipient_restrictions = 
    permit_mynetworks
    reject_unauth_destination
    reject_unknown_recipient_domain
    reject_unverified_recipient

# Custom reply message when probe fails (Postfix 2.6 and later)
unverified_recipient_reject_reason = Address lookup failure

Settings order is important as they are verified one after another; when a decision is triggered (PERMIT or REJECT) the parsing process ends.

Let’s see them in details:

permit_mynetworks: permit message from local or trusted addresses listed in $mynetworks;
reject_unauth_destination: reject message unless recipient domain is a local one (typically in $mydestination, $virtual_alias_domains or $virtual_mailbox_domains) or is accepted for forwarding (in $relay_domains);
reject_unknown_recipient_domain: reject message if recipient domain has no DNS MX and A record, or has a malformed MX record;
reject_unverified_recipient: reject the message if the Recipient Address Verification fails.

If you want to learn more, the best place to find more information is the Postfix Address Verification Howto.

Parsing Email Addresses in Rails with Mail::Address

2014-10-03T00:00:00+00:00

I’ve recently discovered the Mail::Address class and have started using it for storing and working with email addresses in Rails applications. Working with an email address as an Address object rather than a String makes it easy to retrieve different parts of the address and I recommend trying it out if you’re dealing with email addresses in your application.

Mail is a Ruby library that handles email generation, parsing, and sending. Rails’ own ActionMailer module is dependent on the Mail gem, so you’ll find that Mail has already been included as part of your Rails application installations and is ready for use without any additional installation or configuration.

The Mail::Address class within the library can be used in Rails applications to provide convenient, object-oriented ways of working with email addresses.

The class documentation provides some of the highlights:

a = Address.new('Patrick Lewis (My email address) <patrick@example.endpoint.com>')
a.format       #=> 'Patrick Lewis <patrick@example.endpoint.com> (My email address)'
a.address      #=> 'patrick@example.endpoint.com
a.display_name #=> 'Patrick Lewis'
a.local        #=> 'patrick'
a.domain       #=> 'example.endpoint.com'
a.comments     #=> ['My email address']
a.to_s         #=> 'Patrick Lewis <patrick@example.endpoint.com> (My email address)'

Mail::Address makes it trivial to extract the username, domain name, or just about any other component part of an email address string. Also, its #format and #to_s methods allow you to easily return the full address as needed without having to recombine things yourself.

You can also build a Mail::Address object by assigning email and display name strings:

a = Address.new
a.address = "patrick@example.endpoint.com"
a.display_name = "Patrick Lewis"
a #=> #<Mail::Address:69846669408060 Address: |Patrick Lewis <patrick@example.endpoint.com>| >
a.display_name = "Patrick J. Lewis"
a #=> #<Mail::Address:69846669408060 Address: |"Patrick J. Lewis" <patrick@example.endpoint.com>| >
a.domain #=> "example.endpoint.com"

This provides an easy, reliable way to generate Mail::Address objects that catches input errors if the supplied address or display name are not parseable.

I encourage anyone who’s manipulating email addresses in their Rails applications to try using this class. I’ve found it especially useful for defining application-wide constants for the ‘From’ addresses in my mailers; by creating them as Mail::Address objects I can access their full strings with display names and addresses in my mailers, but also grab just the email addresses themselves for obfuscation or other display purposes in my views.

Postfix IPv6 preference

2014-08-19T00:00:00+00:00

On a Debian GNU/Linux 7 (“wheezy”) system with both IPv6 and IPv4 networking setup, running Postfix 2.9.6 as an SMTP server, we ran into a mildly perplexing situation. The mail logs showed that outgoing mail to MX servers we know have IPv6 addresses, the IPv6 address was only being used occasionally, while the IPv4 address was being used often. We expected it to always use IPv6 unless there was some problem, and that’s been our experience on other mail servers.

At first we suspected some kind of flaky IPv6 setup on this host, but that turned out not to be the case. The MX servers themselves are fine using only IPv6. In the end, it turned out to be a Postfix configuration option called smtp_address_preference:

smtp_address_preference (default: any)

The address type (“ipv6”, “ipv4” or “any”) that the Postfix SMTP client will try first, when a destination has IPv6 and IPv4 addresses with equal MX preference. This feature has no effect unless the inet_protocols setting enables both IPv4 and IPv6. With Postfix 2.8 the default is “ipv6”.

Notes for mail delivery between sites that have both IPv4 and IPv6 connectivity:

The setting “smtp_address_preference = ipv6” is unsafe. It can fail to deliver mail when there is an outage that affects IPv6, while the destination is still reachable over IPv4.

The setting “smtp_address_preference = any” is safe. With this, mail will eventually be delivered even if there is an outage that affects IPv6 or IPv4, as long as it does not affect both.

This feature is available in Postfix 2.8 and later.

That documentation made it sound as if the default had changed to “ipv6” in Postfix 2.8, but at least on Debian 7 with Postfix 2.9, it was still defaulting to “any”, thus effectively randomly choosing between IPv4 and IPv6 on outbound SMTP connections where the MX record pointed to both.

Changing the option to “ipv6” made Postfix behave as expected.

Unable to Bcc in mail, Spree 2.0 Stable Rails 3.2.14

2014-06-09T00:00:00+00:00

Hello again all. As usual, I was working on a Spree Commerce website. I recently encountered an issue when trying to bcc order confirmation emails. Others have been asking about this on Github and also on the Spree mailing list, so it was time to write about the problem and the solution that worked for me.

First, I’d like to briefly describe the use case here. As with any typical e-commerce site, a user visits the site, adds some items to their cart, and checks out. After which, an order confirmation (order summary) email is sent to the user with their order details and any extra information provided by the seller.

Spree pretty much handles all this for you automatically. What about if you as the business owner would like a copy of this e-mail? Easy enough. If you review the Spree documentation you will see simple instructions for the “Mail Method Settings” to set up in the Spree Admin Interface.

Ok, so let’s say you follow all the instructions and start placing test orders (or receiving real ones), and you’re not getting bcc’d? This is where it gets tricky, so let’s check out a few things:

Check the logs

Check to see if Spree/Rails is attempting to send the confirmation email to your Bcc recipient
Try development, test, & production modes

If the Bcc email is not getting sent, keep following along for the solution

Ensure the interceptor works by adding a value into “INTERCEPT EMAIL ADDRESS” via the admin. If emails are being intercepted you know the the interceptor is working. Why is that important? Because, that is also where the bcc code is.

core/lib/spree/mail_interceptor.rb

module Spree
  module Core
    class MailInterceptor
      def self.delivering_email(message)
        return unless MailSettings.override?

        if Config[:intercept_email].present?
          message.subject = "#{message.to} #{message.subject}"
          message.to = Config[:intercept_email]
        end

        if Config[:mail_bcc].present?
          message.bcc ||= Config[:mail_bcc]
        end
      end
    end
  end
end

You can ensure that more than one email can be sent by updating mail(to: @order.email, from: from_address, subject: subject) to be an array of emails, like mail(to: [@order.email, "some_other_email@somewhere.com"], from: from_address, subject: subject)

core/app/mailers/spree/order_mailer.rb

module Spree
  class OrderMailer < BaseMailer
    def confirm_email(order, resend = false)
      @order = order.respond_to?(:id) ? order : Spree::Order.find(order)
      subject = (resend ? "[#{Spree.t(:resend).upcase}] " : '')
      subject += "#{Spree::Config[:site_name]} #{Spree.t('order_mailer.confirm_email.subject')} ##{@order.number}"
      mail(to: @order.email, from: from_address, subject: subject)
    end

    def cancel_email(order, resend = false)
      @order = order.respond_to?(:id) ? order : Spree::Order.find(order)
      subject = (resend ? "[#{Spree.t(:resend).upcase}] " : '')
      subject += "#{Spree::Config[:site_name]} #{Spree.t('order_mailer.cancel_email.subject')} ##{@order.number}"
      mail(to: @order.email, from: from_address, subject: subject)
    end
  end
end

At this point, you’ve verified the interceptor works, and the mailer can definitely send more than one email. Here is the final piece that is not mentioned anywhere on the Spree site, or any resolutions for any posts I had found: you may need to contact your hosting provider and ask them to make any necessary adjustments to allow for the bcc messages. I saw that several people have gotten hung up on this and I hope this post has saved you some time. If you are still having trouble, please feel free to reach out in the comments. You can refer to the Spree issue I created on this topic some time ago.

SPF, DKIM and DMARC brief explanation and best practices

2014-04-15T00:00:00+00:00

Spam mail messages have been a plague since the Internet became popular and they kept growing more and more as the number of devices and people connected grew. Despite the numerous attempts of creation of anti-spam tools, there’s still a fairly high number of unwanted messages sent every day.

Luckily it seems that lately something is changing with the adoption of three (relatively) new tools which are starting to be widely used: SPF, DKIM and DMARC. Let’s have a quick look at each of these tools and what they achieve.

What are SPF, DKIM and DMARC

SPF (Sender Policy Framework) is a DNS text entry which shows a list of servers that should be considered allowed to send mail for a specific domain. Incidentally the fact that SPF is a DNS entry can also considered a way to enforce the fact that the list is authoritative for the domain, since the owners/administrators are the only people allowed to add/change that main domain zone.

DKIM (DomainKeys Identified Mail) should be instead considered a method to verify that the messages’ content are trustworthy, meaning that they weren’t changed from the moment the message left the initial mail server. This additional layer of trustability is achieved by an implementation of the standard public/private key signing process. Once again the owners of the domain add a DNS entry with the public DKIM key which will be used by receivers to verify that the message DKIM signature is correct, while on the sender side the server will sign the entitled mail messages with the corresponding private key.

DMARC (Domain-based Message Authentication, Reporting and Conformance) empowers SPF and DKIM by stating a clear policy which should be used about both the aforementioned tools and allows to set an address which can be used to send reports about the mail messages statistics gathered by receivers against the specific domain [1].

How do they work?

All these tools relies heavily on DNS and luckily their functioning process, after all the setup phase is finished, is simple enough to be (roughly) explained below:

SPF

upon receipt the HELO message and the sender address are fetched by the receiving mail server
the receiving mail server runs an TXT DNS query against the claimed domain SPF entry
the SPF entry data is then used to verify the sender server
in case the check fails a rejection message is given to the sender server

Image by Ale2006-from-en, licensed under CC BY-SA 3.0 from Wikipedia

DKIM

when sending an outgoing message, the last server within the domain infrastructure checks against its internal settings if the domain used in the “From:” header is included in its “signing table”. If not the process stops here
a new header, called “DKIM-Signature”, is added to the mail message by using the private part of the key on the message content
from here on the message main content cannot be modified otherwise the DKIM header won’t match anymore
upon reception the receiving server will make a TXT DNS query to retrieve the key used in the DKIM-Signature field
the DKIM header check result can be then used when deciding if a message is fraudulent or trustworthy

Image by Ludovic Rembert of PrivacyCanada.net, licensed under CC BY-SA 3.0, cited in Wikipedia

DMARC

upon reception the receiving mail server checks if there is any existing DMARC policy published in the domain used by the SPF and/or DKIM checks
if one or both the SPF and DKIM checks succeed while still being aligned with the policy set by DMARC, then the check is considered successful, otherwise it’s set as failed
if the check fails, based on the action published by the DMARC policy, different actions are taken

Source DMARC.org, licensed under CC BY 3.0

The bad news: limits and best practices

Unfortunately even by having a perfectly functional mail system with all the above tools enforced you won’t be 100% safe from the bad guys out there. Not all servers are using all three tools shown above. It’s enough to take a look at the table shown in Wikipedia [2] to see how that’s possible.

Furthermore there are some limits that you should always consider when dealing with SPF, DKIM and DMARC:

as already said above DKIM alone doesn’t grant in any way that the sender server is allowed to send outgoing mail for the specific domain
SPF is powerless with messages forged in shared hosting scenario as all the mail will appear as the same coming IP
DMARC is still in its early age and unfortunately not used as much as hoped to make a huge difference
DMARC can (and will) break your mail flow if you don’t set up both SPF and DKIM before changing DMARC policy to anything above “none”.

Please work through the proper process carefully, otherwise your precious messages won’t be delivered to your users as potentially seen as fraudulent by a wrong SPF, DKIM or DMARC setup.

What’s the message behind all this? Should I use these tools or not?

The short answer is: Yes. The longer answer is that everybody should and eventually will in future, but we’re just not there yet. So even if all these tools already have a lot of power, they’re not still shining as bright as they should because of poor adoption.

Hopefully things will change soon and that starts by every one of us adopting these tools as soon as possible.

[1] The lack of such a monitoring tool is considered one of the reasons why other tools (such as ADSP) in past have failed during the adoption phase.

[2] Comparison of mail servers on Wikipedia

Restrict IMAP account access to one (or more) IP address

2014-03-13T00:00:00+00:00

If you’re in need of some extra layer of security on your mail server and know in advance who is going to access your IMAP account and from where (meaning which IP), then the following trick could be the perfect solution for you.

In order to use this feature you’ll have to use Dovecot 2.x+ and then just add a comma separated list of addresses/subnets to the last field of your dovecot passwd auth file:

user:{plain}password::::::allow_nets=192.168.0.0/24,10.0.0.1,2001:abcd:abcd::0:0/80

After a quick reload Dovecot will start to enforce the specified new settings.

An additional neat aspect is that from an attacker perspective the given error will always be the same one got from a “wrong password” attempt, making basically impossible to discover this further protection.

Stay safe out there!

An update to the email_verifier gem has been released

2013-12-04T00:00:00+00:00

I have just released a newer version of the email_verifier gem. It now supports Rails localization out of the box.

If you are one of gem’s users—you can safely update your Bundles!

With this release, the project has also gained its first contributors. Some time ago, one of gem’s users—Franscesco Gnarra has asked me about a possibility of having validation messages localized. While I was planning to do it myself—Francesco took an action and contributed to the project himself.

The project also received a pull request from another Rails developer—Maciej Walusiak. His commit provided a way to handle Dnsruby::NXDomain exceptions.

It’s great to see that our work is helpful for others. If you’d like to contribute yourself—I invite you to do so.

Email verifier at GitHub: https://github.com/kamilc/email_verifier

Using Gmail at Work

2011-12-15T00:00:00+00:00

The Short Story

For those who don’t care about why, just how…

Create a new Gmail account
Setup Mail Fetcher
Setup send email from another account;and make it your default
Verify you send and receive as your corporate address by default using the web client
Setup your mobile
From your mobile go to m.google.com/sync and Enable “Send Mail As” for this device (tested only on iOS)
Verify you send and receive as your corporate address by default using your mobile client
Setup Google Authorship with your domain’s email address

The Long Story

Here at End Point there are a lot of opinions about email clients. Our hardcore folks like Alpine while for most people Evolution, Thunderbird, or Outlook will do. As a Gmail user since September 2004, I found I needed to figure out how get our corporate email system to work with my preferred client.

My first reaction was to have Gmail act as an IMAP client. I found (as many others had) that Gmail does not support IMAP integration with other accounts. However, Gmail does have a POP email client known as Mail Fetcher. I found that Gmail does support encrypted connections via POP, so use them if your email server supports them. When combined with the HTTPS by default, access to the Gmail web client seemed sufficiently secure.

I now needed to send email not as my Gmail address, but as my End Point address. Google has well documented how to send email from another account. Again encrypted SMTP is supported and is strongly recommended. Also be sure to make your corporate email account the default account so you will always use your corporate email address and not the Gmail address.

After verifying I was sending and receiving email properly, I needed to get my mobile setup. There are a variety of options available for all the mobile platforms. On my iPhone, I had several other accounts already setup and found the native client to be acceptable. I decided I would configure the native iPhone email app to access Gmail, as well as Contacts and Calendar using Google’s support for Microsoft’s ActiveSync protocol, which Google has licensed and rebranded as Google Sync.

I had used Google Sync for other Exchange accounts at my previous job and found it worked very well. However, there are some known issues, like not being able to accept event invitations recieved via POP. It’s worth checking these issues out to see if there are any blockers for you.

After setting up “Google Sync” on my iPhone, I tested again, and found that by default, it would use my Gmail account as my default outgoing email account, despite the setting in the Gmail web client. I needed to use my corporate address here at End Point for sending mail from mobile; I thought I was sunk!

Fortunately, it seems I over looked a section in the Google Sync setup documentation, labeled “Enable Send Mail As feature”. This feature solved my problem by having me go to m.google.com/sync from my iOS device and check Enable “Send Mail As” for this device. This would tell Google Sync to use the default outgoing account I had specified in the web client.

One requirement here at End Point which this configuration does not meet is support for PGP encryption/decryption of messages. There is a Chrome plugin that claims to offer support, but as the authors from this post highlight:

*There may also be resistance from crypto users – who already are a security-conscious lot – to trusting private keys and confidential messages to a set of PGP functions folded inside some JavaScript running inside a browser. *

I’d have to say I agree. After following the instructions to install the plugin, I balked when it asked for my private key; I just didn’t feel comfortable. Despite this shortfall, most End Point email isn’t encrypted end-to-end. However, I can feel good knowing that my “last mile” connection to End Point’s servers are encrypted, end-to-end using encrypted POP, SMTP, and HTTPS.

Using the new version of imapfilter with mutt

2011-10-17T00:00:00+00:00

Image by Flickr user p886

My beloved mutt/imapfilter combo recently stopped working after an operating system switch. (tl;dr: that combo rocks; use ipairs instead of pairs) When my laptop wireless stopped working, and after spending some time fighting with it, I decided to simply install a new OS. As all of my important data is on a separate partition, this was not that big a deal. I ended up using Scientific Linux, as I’d heard good things about it, and it was one of the few distros that actually would install on my laptop (failures for one reason or another: Fedora, FreeBSD, Ubuntu, and OpenBSD). After the install, I simply copied my ~/.mutt directory and ~/.muttrc file into place, and similarly copied my ~/.imapfilter directory, which contained the all important config.lua file. The imapfilter program itself was not available via the normal yum repositories, so I simply grabbed the latest and greatest and did a manual install:

$ git clone https://github.com/lefcha/imapfilter.git
$ cd imapfilter
$ sudo yum install gcc lua-devel openssl-devel pcre-devel
$ make
$ sudo make install

I’ve used a lot of email clients over the years (and may have been using email longer than most people reading this). I started out (because that’s all there was) with non-graphical clients such as mail, pine, elm, and mutt. Over the years I also tried out many graphical clients, such as Evolution, Kmail, Eudora, Thunderbird, and Claws Mail. However, nothing ever worked quite right, so I eventually ended up back with mutt, and have been happy with it ever since. The one drawback (or strength) of mutt is its single-mindedness. It does email very well, but lets other tools handle the ancillary tasks. One of those tasks is filtering, and that’s where imapfilter comes in. I like to view all email that comes in, so mutt generally runs with my INBOX open. I scan through the items, marking them urgent if I need to keep them around, and deleting them if they are obvious trash. As needed, I’ll kick off a imapfilter run, which then puts all my read, non-urgent, non-deleted email into the appropriate IMAP folders for me (mutt is even smart enough to realize that the folder was externally changed by imapfilter).

So I tried running imapfilter per usual on my new system and noticed an odd thing: each item in my filter was getting a minimum of 66 ‘hits’, even when there was not even 66 total emails in my inbox! I output the number of matches to each filter I use, so instead of seeing what I was usually did:

Mediawiki emails moved:    1
Backcountry emails moved:  10
Perl QA messages moved:    0
...
Wiki alerts deleted:       0
Bucardo emails moved:      5
Maatkit emails moved:      0
Mail filtering complete

I saw everything at N+66 instead:

Mediawiki emails moved:   67
Backcountry emails moved: 76
Perl QA messages moved:   66
...
Wiki alerts deleted:      66
Bucardo emails moved:     71
Maatkit emails moved:     66
Mail filtering complete

Obviously, something was wonky. Glancing at the release notes showed that version 2.2 changed the format of the search results:

Since version 2.2, a different format is used for the returned structures of the searching methods, due to the introduction of multiple mailbox searching and meta-searching, and thus any configuration files that rely on them should be updated*

Okay, but where was the 66 coming from? I created a ~/.imapfilter/test.lua file to show me exactly what was happening inside the loop over the results table. (imapfilter is written in a nice language called Lua, which calls its main data structures “tables”. Probably to the chagrin of those using Lua/database crossover tools like Pl/Lua :) The test.lua file looked like this:

myaccount = IMAP {
  server   = 'mail.example.com',
  username = 'greg',
  password = 'secret',
  ssl      = 'tls1'
}
inbox = myaccount['INBOX']
result = inbox:contain_subject('bats')

count = 0
for k,v in pairs(result) do 
  count = count + 1 
  if count < 10 then
    print(count, "Call to pairs:",k,v)
  end
end
print("Total count for pairs: " .. count);

count = 0
for k,v in ipairs(result) do 
  count = count + 1 
  if count < 10 then
    print(count, "Call to ipairs:",k,v)
  end
end
print("Total count for ipairs: " .. count);

I downloaded and compiled version 2.0 of imapfilter and ran the above code, knowing that there were exactly two emails in my inbox that had a subject containing the string ‘bats’:

[~/code/imapfilter-2.0] ./imapfilter -c ~/.imapfilter/test.lua
  1      Call to pairs:    9         true
  2      Call to pairs:    32        true
Total count for pairs: 2
Total count for ipairs: 0

So it looked like the results table simply contained two entries, with keys of 9 and 32 (which correspond to where those emails happened to appear in my inbox). Calling ipairs yielded zero matches, which makes sense: there is no key of 1 (which is what Lua tables start with by convention, rather than 0 like almost everything else in the computer world :). The ipairs function goes through each key in order starting with 1 until a nil (undefined) key is found. In this case, 1 itself is nil. The output looks much different when I ran it using the new version (2.3) of imapfilter:

[~/code] imapfilter -c ~/.imapfilter/test.lua
  1      Call to pairs:    1              table: 0x82b0bd8
  2      Call to pairs:    2              table: 0x82b0c48
  3      Call to pairs:    _union         function: 0x81d48d0
  4      Call to pairs:    _mt            table: 0x82a32d0
  5      Call to pairs:    mark_answered  function: 0x81cefe0
  6      Call to pairs:    send_query     function: 0x81d8180
  7      Call to pairs:    is_flagged     function: 0x81c1878
  8      Call to pairs:    unmark_deleted function: 0x81bd890
  9      Call to pairs:    match_message  function: 0x81cd7f8
Total count for pairs: 68
  1      Call to ipairs:    1        table: 0x82b0bd8
  2      Call to ipairs:    2        table: 0x82b0c48
Total count for ipairs: 2

This tells us a quite a few things, and solves the mystery of the 66, which represents some meta-data stored in the results table. So rather than treating results as a simple key/value hash with one entry per match, the results table is now a dual-purpose table where the hash part of it contains some meta-data, while the actual matches are stored in the array (indexed) part of the table. Note how the counting of the matches now starts at 1 and increments, rather than using the position in the inbox, as it did before. Which means we must use ipairs to iterate through the table and get our matching entries, in this case with keys 1 and 2.

(If the “table” structure in Lua looks odd to you, that’s because it is. I don’t think I would have designed things that way myself—while it’s clever to have a single structure that behaves as both an array with indices and a btree hash, it can lead to confusion and some ugly corner cases).

The next step was to get my filters working again—this was simply a matter of a global search and replace (M-x query-replace-regexp) from “pairs” to “ipairs”.This is a good a point as any to explain what my file looks like (stored as ~/.imapfilter/config.lua). The first part simply sets some common options—for details on what they do, check out the manpage for imapfilter_config.

options.cache        = true
options.certificates = true
options.create       = false
options.info         = false
options.close        = true
options.expunge      = false

Next, a new table is created with the IMAP function. After that, we exclude all messages that are already marked as deleted, that have not yet been read, and have not been flagged. In other words, everything in my inbox I’ve already seen, but not flagged as urgent or deleted. The ‘*’ in this case is a logical ‘AND’, and the output is the search result table we saw in the above code.

myaccount = IMAP {
  server   = 'mail.example.com',
  username = 'greg',
  password = 'secret,
  ssl      = 'tls1'
}

baseresult = inbox:is_seen() * inbox:is_unflagged() * inbox:is_undeleted()

Now that we have a search result, we simply start looking for things of interest and handling them. For example, to move messages to an existing IMAP folder:

-- Put Mediawiki messages into their folder
result = baseresult
  * (
    inbox:contain_to('@lists.wikimedia.org')
    + inbox:contain_to('@lists.wikimedia.org')
  )
count = 0 for k,v in ipairs(result) do count = count + 1 end
if count > 0 then
  inbox:move_messages(myaccount['INBOX/mediawiki'], result)
end
print('Mediawiki emails moved:        ' .. count)

Searches can be applied to an existing search result to create a new table. In the code above, a new table named ‘result’ is created that is based off of our ‘baseresult’ table, with the condition that only entries matching a specific “To” or “Cc” field are added.The ‘+’ acts as as a logical ‘OR’.

Deletion is handled in a similar way:

-- Delete wiki alerts
result = baseresult
  * inbox:contain_from('WikiAdmin <wikiadmin@example.com>')
  * inbox:contain_subject('has been')
count = 0 for k,v in ipairs(result) do count = count + 1 end
if count > 0 then
  inbox:delete_messages(result)
end
print('Wiki alerts deleted:           ' .. count)
</wikiadmin@example.com>

The rest of my config.lua file is more filtering sections, similar to the above. Adding a new filter is as easy as creating a new section similar to the above by editing the ~/.imapfilter/config.lua file. While that’s not as automated as it could be, filter adjustment happens so rarely I have never been bothered by that step.

If you are not using imapfilter, you should check it out, even if you are not using mutt; imapfilter is completely independent of your email reading program, and can be run from anywhere, as it doesn’t save or read anything locally. I find that imapfilter is very fast, so even when I used mail programs with built-in filters, I still employed imapfilter from time to time for bulk deletes and moves. Plus, it’s a great way to dip your toe into Lua if you are not familiar with it (although without using some of its more interesting features, such as coroutines).

Review of The Book of IMAP

2008-08-26T00:00:00+00:00

Former End Point employee Ryan Masters had his book review of the No Starch Press published over at OSNews.com. Sounds like it was a decent book!

Switching from Sendmail to Postfix on OpenBSD

2008-08-01T00:00:00+00:00

It’s easy to pick on Sendmail, and with good reason. A poor security record, baroque configuration, slowness, painful configuration, monolithic design, and arcane configuration. Once you know Sendmail it’s bearable, and long-time experts aren’t always eager to give it up, but I wouldn’t recommend anyone deploy it for a serious mail server these days. But for a send-only mail daemon or a private, internal mail server, it works fine. Since it’s the default mailer for OpenBSD, and I haven’t been using OpenBSD as a heavy-traffic mail server, I’ve usually just left Sendmail in place.

A few years ago some of our clients’ internal mail servers running Sendmail were getting heavy amounts of automated output from cron jobs, batch job output, transaction notifications, etc., and they bogged down and sometimes even stopped working entirely under the load. It wasn’t that much email, though—the machines should’ve been able to handle it.

After trying to tune Sendmail to be more tolerant of heavy load and having little success, I finally switched to Postfix (which we had long used elsewhere) and the CPU load immediately dropped from 30+ down to below 1, and mail delivery worked without interruption during busy times.

If I’d known how easy it is to switch OpenBSD from Sendmail to Postfix, I would’ve done it long ago. I wrongly figured it’d be hard since Sendmail is part of the base system, and none of that seemed very pluggable without hacking on things. I found out it was easy only by finally just trying it myself, following the very simple instructions, and having no trouble. I did this first on OpenBSD 3.9 and now again on OpenBSD 4.3, and the process was the same.

First, pick an OpenBSD mirror, and navigate to the appropriate packages directory. Then set up your environment for easy pkg_add usage. For example:

export PKG_PATH=ftp://ftp.openbsd.org/pub/OpenBSD/4.3/packages/i386

There are several varying OpenBSD Postfix packages, offering support for lookups in LDAP, MySQL, Postgres, or SASL, or a simple build without any of those dependencies:

# pkg_add postfix
Ambiguous: postfix could be postfix-2.5.1p0 postfix-2.5.1p0-ldap postfix-2.5.1p0-mysql postfix-2.5.1p0-pgsql postfix-2.5.1p0-sasl2 postfix-2.6.20080216p1 postfix-2.6.20080216p1-ldap postfix-2.6.20080216p1-mysql postfix-2.6.20080216p1-pgsql postfix-2.6.20080216p1-sasl2

We’ll use the simple build:

pkg_add postfix-2.6.20080216p1

The output from the package installation tells you most of what you need to know, but I’ll break it down here with a little more detail.

Run crontab -e as root and comment out this Sendmail job:

# sendmail clientmqueue runner
#*/30    *       *       *       *       /usr/sbin/sendmail -L sm-msp-queue -Ac -q

The sendmail compatibility is implemented by a wrapper script similar to how Debian’s alternatives system does it (and which Red Hat borrowed as well). In OpenBSD, the wrapper is a binary that uses the configuration in /etc/mailer.conf to decide what to actually run, as opposed to using symlinks as the alternatives system does. You can see this here:

# ls -lFa /usr/sbin/sendmail /usr/bin/newaliases /usr/bin/mailq
lrwxr-xr-x  1 root  wheel  21 Aug  1 14:50 /usr/bin/mailq@ -> /usr/sbin/mailwrapper
lrwxr-xr-x  1 root  wheel  21 Aug  1 14:50 /usr/bin/newaliases@ -> /usr/sbin/mailwrapper
lrwxr-xr-x  1 root  wheel  21 Aug  1 14:51 /usr/sbin/sendmail@ -> /usr/sbin/mailwrapper

To make the switch to Postfix, run:

/usr/local/sbin/postfix-enable

Now you’re ready to configure /etc/postfix/main.cf as needed. The defaults should be fine for a server sending outgoing mail only, though if you followed the OpenBSD installer’s instructions to use only the short name for the hostname, you need to either set the mydomain parameter manually in main.cf, or else edit /etc/myname to use a fully-qualified domain name instead of the hostname only (and update immediately with the hostname command as well). I do the latter and haven’t had any trouble with it before.

Stop Sendmail and start Postfix the same way the boot script will do it:

pkill sendmail
/usr/sbin/sendmail -bd

Send a test message and make sure you receive it:

echo "A special test message" | mail -s testing <em>your_account@the.domain</em>

Note that if you send your message to somewhere offsite, spam filters may reject it if your sending server doesn’t have a real hostname, a reverse DNS pointer for the IP address, etc. You can just send locally to avoid that, but of course you won’t be able to send mail offsite until you deal with those problems.

Add these settings to /etc/rc.conf.local so Postfix will start on boot:

sendmail_flags="-bd"
syslogd_flags="-a /var/spool/postfix/dev/log"

Now reboot to make sure everything comes up correctly on its own and to get syslogd going right. Send yourself another test message, and you can move on!

Many thanks to the Postfix developers for the excellent mail server software and to the OpenBSD developers for a nice easy way to switch the system mail daemon.

Death, Taxes — and Spam‽

2005-06-12T00:00:00+00:00

End Point has been hard at work keeping spam from reaching our clients’ email boxes.

In December 2004, End Point installed new email filters, powered by two state-of-the-art packages: amavisd-new and SpamAssassin.

By filtering out the most egregious junk emails at the server level, far fewer undesirable emails reach our clients’ inboxes. Real mail downloads to their inboxes faster, and the painful morning routine of going through the pile of junk mail delivered overnight is gone. That time savings multiplied for each employee really adds up.

We’re gratified to hear about the positive, immediate difference this change has made for our current email services clients. If you’re not currently using End Point’s email service, please contact us. We’d be happy to discuss your requirements and explain how we can help meet your email needs.

Why Your AI Extractor Fails on .msg Emails (and How to Fix Decoding)

The pattern that gives it away

Why .msg bites harder than .eml

The fix: strict, explicit, controlled

Why I prefer fail-loud here

How to roll this out safely

Observability that makes this easier next time

Using Razor templates to render HTML emails in ASP.NET Core

The plan

Step 1: Sending emails with the MailKit NuGet package

Step 2: Rendering Razor templates into strings

Step 3: Defining the email templates

Step 4: Putting it all together: the class that sends the email

Using Razor templates to render HTML emails in .NET

The plan

Step 1: Sending emails in .NET with the MailKit NuGet package

Step 2: Rendering Razor templates into strings

Step 3: Defining the email templates

Step 4: Putting it all together: the class that sends the “quote generated” email

Working around SPF problems delivering to Gmail

Email deliverability

Sender Policy Framework

Email server trails

Gmail misuses header information in SPF checks

Don’t share TMI

Reference

Capturing Outgoing Email With Mock SMTP Servers

Mock SMTP Servers

Configuring a Rails Application to use MailHog

Conclusion

Linux desktop Postfix queue for Gmail SMTP

What local SMTP program to use?

esmtp

ssmtp

E-MailRelay

Postfix

Configuring Postfix

Trying it out

Conclusion

Reach customers and drive sales with MailChimp

Endless possibilities

Automatic marketing

Steps to implement

Go beyond the simple newsletter

Taking control of your IMAP mail with IMAPFilter

Postfix Address Verification

How does Address Verification work?

Configure Recipient Address Verification

Parsing Email Addresses in Rails with Mail::Address

Postfix IPv6 preference

Unable to Bcc in mail, Spree 2.0 Stable Rails 3.2.14

SPF, DKIM and DMARC brief explanation and best practices

What are SPF, DKIM and DMARC

How do they work?

SPF

DKIM

DMARC

The bad news: limits and best practices

What’s the message behind all this? Should I use these tools or not?

Restrict IMAP account access to one (or more) IP address

An update to the email_verifier gem has been released

Using Gmail at Work

The Short Story

The Long Story

Using the new version of imapfilter with mutt

Review of The Book of IMAP

Switching from Sendmail to Postfix on OpenBSD

Death, Taxes — and Spam‽

Why `.msg` bites harder than `.eml`