Building a Multi-Agent Travel Planner with Microsoft Agent Framework and .NET

2026-05-15T00:00:00+00:00

Photo by Seth Jensen, 2022.

Large language models can answer travel questions well enough, but a single prompt is rarely enough to produce a plan you can trust end to end. Budget math can drift, travel times can be unrealistic, and a confident recommendation may not be grounded in the data the model actually saw.

One way to improve that is to split the work across specialized agents. In this project, one agent researches the destination, another builds the itinerary, another checks the budget, another audits the result, and the last one turns everything into a polished response.

In this post, we will build that workflow in .NET 10 using the Microsoft Agent Framework (MAF). The project supports both local Ollama models and Anthropic through a shared IChatClient abstraction, exposes a Web API with a streaming endpoint, and includes evaluation tests using Microsoft.Extensions.AI.Evaluation.

The source code is available on GitHub.

What we’ll build

A sequential multi-agent pipeline: Researcher -> Planner -> Accountant -> Auditor -> Aggregator
Provider-swappable LLM integration with Ollama and Anthropic
Deterministic validation tools for budget, timing, groundedness, and safety
An ASP.NET Core Web API with both blocking and streaming endpoints
A browser UI that shows live progress from each agent
MEAI-based evaluation tests for output quality

Prerequisites

.NET 10 SDK
Ollama running locally with qwen2.5:7b, or an Anthropic API key
Optional: an OpenTripMap API key for live hotel, attraction, and restaurant data

Project structure

The solution contains three projects:

LocalAgentTravelPlanner for agents, services, models, and tools
LocalAgentTravelPlanner.Api for the ASP.NET Core Web API and static UI
LocalAgentTravelPlanner.Tests for unit tests and evaluation tests

The core project uses these packages:

<PackageReference Include="Anthropic.SDK" Version="5.8.0" />
<PackageReference Include="Microsoft.Agents.AI" Version="1.0.0-preview.251219.1" />
<PackageReference Include="Microsoft.Agents.AI.Workflows" Version="1.0.0-preview.251219.1" />
<PackageReference Include="Microsoft.Extensions.AI" Version="10.1.1" />
<PackageReference Include="OllamaSharp" Version="5.4.12" />

The agent pipeline

The workflow is sequential, and each agent receives the accumulated conversation history from the earlier agents.

That composition is built with AgentWorkflowBuilder.BuildSequential:

var workflow = AgentWorkflowBuilder.BuildSequential(
    new List<ChatClientAgent> { researcher, planner, accountant, auditor, aggregator }
);

StreamingRun run = await InProcessExecution.StreamAsync(workflow, userRequest);
await run.TrySendMessageAsync(new TurnToken(emitEvents: true));

The important detail here is not just the order. It is the division of responsibility:

The Researcher gathers facts and options
The Planner turns those facts into a day-by-day itinerary
The Accountant checks whether the itinerary fits the budget
The Auditor validates the plan
The Aggregator prepares the final user-facing output

That separation makes the system easier to reason about than a single all-purpose prompt.

Abstracting the model provider

Both Ollama and Anthropic are exposed through IChatClient from Microsoft.Extensions.AI, so the agents do not need provider-specific logic.

ChatClientFactory handles provider selection:

public static (IChatClient Client, Provider Provider, string Model) CreateWithAutoDetect(
    string? preferredProvider = null)
{
    if (!string.IsNullOrEmpty(preferredProvider) &&
        Enum.TryParse<Provider>(preferredProvider, ignoreCase: true, out var provider))
    {
        var model = provider == Provider.Anthropic
            ? "claude-sonnet-4-20250514"
            : "qwen2.5:7b";
        return (Create(provider, model), provider, model);
    }

    var anthropicKey = Environment.GetEnvironmentVariable("ANTHROPIC_API_KEY");
    if (!string.IsNullOrEmpty(anthropicKey))
    {
        const string model = "claude-sonnet-4-20250514";
        return (CreateAnthropicClient(model), Provider.Anthropic, model);
    }

    const string ollamaModel = "qwen2.5:7b";
    return (CreateOllamaClient(ollamaModel), Provider.Ollama, ollamaModel);
}

This is one of the more practical design decisions in the project. It lets you develop against a local model, then switch to Anthropic without rewriting the agent layer.

Fixing the Anthropic adapter mismatch

One of the more interesting details in the project is the AnthropicOptionsInjector middleware.

The Anthropic SDK requires ModelId and MaxOutputTokens on every request, but MAF does not populate those values in ChatOptions. That means requests can fail even though the agent pipeline itself looks correct.

The fix is to wrap the client with a DelegatingChatClient:

internal class AnthropicOptionsInjector : DelegatingChatClient
{
    private readonly string _model;
    private const int DefaultMaxOutputTokens = 8192;

    public AnthropicOptionsInjector(IChatClient inner, string model) : base(inner)
        => _model = model;

    public override Task<ChatResponse> GetResponseAsync(
        IEnumerable<ChatMessage> messages,
        ChatOptions? options = null,
        CancellationToken cancellationToken = default)
    {
        options ??= new ChatOptions();
        if (string.IsNullOrEmpty(options.ModelId)) options.ModelId = _model;
        if (options.MaxOutputTokens == null) options.MaxOutputTokens = DefaultMaxOutputTokens;
        return base.GetResponseAsync(messages, options, cancellationToken);
    }
}

This pattern is useful beyond Anthropic. Any time an orchestration library and an SDK disagree about defaults, a DelegatingChatClient can bridge the gap cleanly.

Creating specialized agents

Each agent is created through a static factory that keeps its instructions, tools, and name in one place.

The Researcher is a good example:

public static ChatClientAgent Create(
    IChatClient chatClient,
    ResearchTools researchTools,
    TravelTools travelTools)
{
    var tools = new List<AITool>
    {
        AIFunctionFactory.Create(researchTools.GetWeatherForecast),
        AIFunctionFactory.Create(researchTools.SearchHotels),
        AIFunctionFactory.Create(researchTools.GetAttractions),
        AIFunctionFactory.Create(researchTools.GetTransportOptions),
        AIFunctionFactory.Create(researchTools.GetFoodRecommendations),
        AIFunctionFactory.Create(researchTools.GetSafetyInfo),
        AIFunctionFactory.Create(travelTools.GetWeather),
        AIFunctionFactory.Create(travelTools.GetFlightEstimate),
        AIFunctionFactory.Create(travelTools.GetEmergencyContacts),
        AIFunctionFactory.Create(travelTools.GetVisaInfo)
    };

    return new ChatClientAgent(
        chatClient,
        instructions: RESEARCHER_INSTRUCTIONS,
        tools: tools,
        name: "Researcher_Agent"
    );
}

Not every agent needs tools. The Planner and Aggregator work from the shared conversation history alone, while the Accountant and Auditor rely on deterministic helper functions.

That split is important. The model handles synthesis and language generation, while ordinary C# code handles the parts where determinism matters.

Using live APIs without making the system brittle

The early version of the project relied on hardcoded destination data. That was useful while bootstrapping the workflow, but it limited the planner to a small set of cities.

The updated tool layer pulls data from several live APIs:

Nominatim for geocoding
Open-Meteo for current weather and forecasts
OpenTripMap for attractions, accommodation names, and food spots
OSRM for distance and routing estimates

For example, weather forecasting follows a simple pattern:

Convert a city name into coordinates
Query Open-Meteo with those coordinates
Parse the JSON response into a formatted text block for the agent

var coords = await GeocodingHelper.GetCoordinatesAsync(_http, city);
if (coords is { } c)
{
    var url = $"https://api.open-meteo.com/v1/forecast" +
              $"?latitude={c.Lat:F4}&longitude={c.Lon:F4}" +
              $"&daily=weather_code,temperature_2m_max,temperature_2m_min,precipitation_sum" +
              $"&forecast_days={Math.Clamp(days, 1, 16)}&timezone=auto";

    var json = await _http.GetStringAsync(url);
    // parse response and format it for the agent
}

The project also uses a practical fallback strategy: every external call is wrapped so tool failures degrade into a useful message instead of breaking the entire workflow.

That matters in agent systems. A failed API call should reduce confidence, not take down the whole run.

The Auditor pattern

The Auditor is the part of the project that makes the architecture more than a simple agent chain.

Instead of asking one model to plan and validate everything at the same time, the system introduces a dedicated review stage backed by deterministic tools:

ValidateMathConsistency
ValidateBudgetFit
ValidateTravelTime
CheckGroundedness
CheckSafetyRequirements
CheckRelevance
CheckCompleteness
DetermineAuditDecision

This allows the Auditor to score the plan across six criteria:

Financial integrity
Temporal logic
Safety compliance
Groundedness
Relevance
Completeness

That is the strongest idea in the whole project. LLMs are useful for synthesis and judgment, but they are still weak at arithmetic and consistency checking. Offloading those checks to tools makes the final response more trustworthy.

Streaming the pipeline to the browser

The Web API exposes two main endpoints:

POST /api/travel/plan to return the final plan as JSON
GET /api/travel/plan/stream to stream progress over Server-Sent Events

The streaming endpoint maps workflow events to frontend-friendly SSE events:

if (evt is ExecutorInvokedEvent invokedEvt)
{
    await SendSseEvent("agent-start", new TravelPlanProgressEvent { ... });
}
else if (evt is AgentRunUpdateEvent updateEvent)
{
    await SendSseEvent("content", new TravelPlanProgressEvent { ... });
}
else if (evt is ExecutorCompletedEvent completedEvt)
{
    await SendSseEvent("agent-complete", new TravelPlanProgressEvent { ... });
}

On the frontend, the stream is consumed with fetch and ReadableStream instead of EventSource.

That choice is subtle but practical. EventSource automatically reconnects, which is often helpful for dashboards, but in this case a reconnect could restart an expensive multi-agent run. Using fetch keeps cancellation and reconnection behavior under explicit control.

The UI then uses those events to:

highlight the active agent in the pipeline
append streaming content into the correct tab
render Markdown progressively
show a final polished output when the Aggregator finishes

Handling MAF’s silent failures

One thing to watch for with MAF in its current preview state: if an agent throws an exception, the workflow does not propagate it. It completes normally with empty output. Your API returns a success response, your UI shows nothing, and the logs are clean.

The fix is to explicitly listen for ExecutorFailedEvent and WorkflowErrorEvent in the event loop:

else if (evt is ExecutorFailedEvent failedEvt)
{
    var ex = failedEvt.Data as Exception;
    await SendSseEvent("error", new TravelPlanProgressEvent
    {
        Agent = resolved,
        Status = "failed",
        Content = ex?.InnerException?.Message ?? ex?.Message ?? "Agent failed"
    });
}
else if (evt is WorkflowErrorEvent errorEvt)
{
    var ex = errorEvt.Data as Exception;
    // handle workflow-level failure
    break;
}

Without these handlers, a failed agent produces a blank response that looks like a success. This applies to both the blocking and streaming endpoints.

Filtering empty streaming chunks

A related issue: MAF emits AgentRunUpdateEvent with null or empty content during tool-calling turns. If you forward these directly to the browser, the SSE stream floods with empty content events. The fix is a one-line guard before sending:

else if (evt is AgentRunUpdateEvent updateEvent)
{
    var content = updateEvent.Data?.ToString();
    if (string.IsNullOrEmpty(content)) continue;

    await SendSseEvent("content", new TravelPlanProgressEvent { ... });
}

Both of these are easy to miss during development because the happy path works fine. They only surface under real load or when an external API (like an LLM provider) fails.

Evaluating output with MEAI

Once the pipeline works, the next question is whether the output is consistently good.

The project answers that with Microsoft.Extensions.AI.Evaluation.Quality, which uses an LLM as a judge to score the output across several axes:

Coherence — does the response hang together logically? Are ideas connected, or does it jump around or contradict itself?
Fluency — is the language natural, grammatically correct, and readable? This is about style and surface quality, not factual content.
Relevance — does the response actually address what the user asked? A fluent answer to the wrong question scores low here.
Truth — are the factual claims correct given the available evidence? The judge flags statements that are unsupported or contradicted by the source data.
Completeness — does the response cover all the parts of the request? A travel plan missing a budget breakdown would lose points.
Groundedness — are the claims traceable to the input or research data, rather than invented? This is the main hallucination signal.

Each evaluator returns a score and a short explanation. Running them on real pipeline output gives you a numeric quality baseline that you can track over time as you change prompts, models, or tools.

The evaluation tests run the real pipeline, then score the resulting output:

results["Coherence"] = await new CoherenceEvaluator()
    .EvaluateAsync(messages, response, _evalConfiguration!);

results["Fluency"] = await new FluencyEvaluator()
    .EvaluateAsync(messages, response, _evalConfiguration!);

results["RelevanceTruthCompleteness"] = await new RelevanceTruthAndCompletenessEvaluator()
    .EvaluateAsync(messages, response, _evalConfiguration!);

These tests are intentionally skipped by default because they require a live model provider. That is a sensible tradeoff: keep ordinary unit tests fast, but retain a path for deeper quality checks when you want to validate the full system.

Running the application

To run the Web API with Ollama:

ollama serve
ollama pull qwen2.5:7b
dotnet run --project Api/LocalAgentTravelPlanner.Api.csproj

To run it with Anthropic:

# PowerShell
$env:ANTHROPIC_API_KEY="sk-ant-..."
dotnet run --project Api/LocalAgentTravelPlanner.Api.csproj

To run the unit tests:

dotnet test Tests/LocalAgentTravelPlanner.Tests.csproj

To enable the evaluation tests:

# PowerShell
$env:ENABLE_EVAL_TESTS="true"
dotnet test Tests/LocalAgentTravelPlanner.Tests.csproj --filter "Category=Evaluation"

Securing the API

Each request in this project triggers multiple LLM calls across five agents. Without any protection, a single user could exhaust your token budget in minutes. Two basic measures help here.

First, an API key middleware that checks an X-Api-Key header on every /api/* route except the health endpoint. The key is loaded from ASP.NET’s configuration system, so it works with appsettings.json, environment variables, or a mounted file for Docker secrets. The middleware runs early in the pipeline so unauthenticated requests never reach the agents.

Second, a rate limiter on the plan endpoints. The project uses a fixed-window policy — five requests per minute per client IP — which is enough to prevent accidental abuse without blocking normal use. Behind nginx, the app reads the real client IP from forwarded headers rather than the Docker bridge address.

builder.Services.AddRateLimiter(options =>
{
    options.AddPolicy("plan", context =>
        RateLimitPartition.GetFixedWindowLimiter(
            partitionKey: context.Connection.RemoteIpAddress?.ToString() ?? "unknown",
            factory: _ => new FixedWindowRateLimiterOptions
            {
                PermitLimit = 5,
                Window = TimeSpan.FromMinutes(1),
                QueueLimit = 0
            }));
});

Neither of these is novel, but they are easy to forget when the focus is on the agent pipeline itself. If you are exposing LLM-backed endpoints publicly, even for a demo, protecting them early saves a lot of trouble later.

Key takeaways

This project is a good example of how to make an agent workflow more practical in a real application.

A multi-agent pipeline is useful when each stage has a distinct responsibility
IChatClient is a strong abstraction for provider-swappable .NET AI applications
DelegatingChatClient middleware is a clean way to fix SDK integration mismatches
Deterministic tools are still essential for math, validation, and consistency checks
Streaming progress makes long-running agent workflows easier to understand and trust
Evaluation is worth treating as a first-class part of the architecture, not an afterthought
MAF silently swallows agent errors — always handle ExecutorFailedEvent and WorkflowErrorEvent
Protect LLM-backed endpoints with API key auth and rate limiting, even for demos

The main lesson is that the quality gains do not come from adding more agents for the sake of it. They come from giving each agent a clear role, grounding it with useful tools, and validating the result before it reaches the user.

The source code is available on GitHub.

Build a Smarter Telegram Bot: Integrating a RAG Pipeline for FAQ Answering

2025-12-01T00:00:00+00:00

In this post, we will show you how to build an intelligent FAQ bot using Python and the Telegram Bot API. We’ll go beyond simple commands by integrating a Retrieval-Augmented Generation (RAG) pipeline with LangChain.

This RAG pipeline lets our bot pull information from a custom knowledge base (in our case, a simple faqs.json file) and use a local Large Language Model (LLM) through Ollama to generate accurate answers. The best part? This approach (which works great with interfaces like Open WebUI) gives you full control over your models and data with zero API costs.

What is Telegram?

You’ve probably heard of Telegram—it’s a popular, cloud-based instant messaging app. It’s fast, works everywhere (mobile, web, and desktop), and has powerful features like huge group chats and easy file sharing.

One of its most powerful features for developers is the Telegram Bot API, an open platform that allows anyone to build and integrate automated applications (like ours!) directly into the chat interface.

A Warning on Privacy and Encryption

Before we build our bot, it is critical to understand how Telegram handles encryption, as it directly impacts user privacy.

Cloud Chats (The Default): All standard chats, group chats, and all bot interactions are “Cloud Chats.” These use server-client encryption. This means your messages are encrypted between your device and Telegram’s servers, and then stored (encrypted) on their servers. This is what allows you to access your chat history from any device. However, Telegram itself holds the encryption keys and can access this data.
Secret Chats (Manual): Telegram also offers “Secret Chats,” which are end-to-end encrypted (E2EE). In this mode, only you and the recipient can read the messages. Telegram has no access. However, bots cannot operate in Secret Chats.

This means that any message a user sends to our bot is a “Cloud Chat” and is not end-to-end encrypted. The data is accessible to Telegram and will be processed in plain text by our bot.py script on our server.

For this reason, you should never build a bot that asks for or encourages users to send sensitive private data such as passwords, financial information, or social security numbers. Always treat bot conversations as non-private.

What is Retrieval-Augmented Generation (RAG)?

Retrieval-Augmented Generation (RAG) is a technique that makes Large Language Models (LLMs) smarter by connecting them to external, private knowledge.

The Problem: An LLM like llama3 only knows the information it was trained on. It has no access to your company’s internal FAQs, new documents, or any private data.
The Solution (RAG): RAG solves this in two steps:
- Retrieve: When you ask a question, the system first retrieves relevant information from your own knowledge base (for us, our faqs.json file).
- Augment: It then augments the LLM’s prompt by pasting that retrieved information in as context, along with your original question.

In short, instead of just asking the bot “What’s the shipping policy?”, we’re effectively asking, “Based on this specific text: ‘…We offer standard shipping…’ — what is the shipping policy?” This forces the LLM to base its answer on our facts, not its own general knowledge, making the response accurate and reliable.

What you’ll build

Telegram bot
faqs.json knowledge base
RAG pipeline with local embeddings (FAISS) + LLM (Open WebUI)

Prerequisites

You’ll need:

Python 3.12+
a Telegram bot token (from BotFather)
access to an LLM via a locally hosted Open WebUI instance (OpenAI-compatible API)

Setting up the Project for Telegram Bot

uv is a high-performance Python package manager, so we’ll use it to set up our project. If you don’t have it installed, you can get it with or visit the site for installation steps:

pip install uv

Create a new project directory and navigate into it:

mkdir telegram-rag-bot
cd telegram-rag-bot

Initialize a new Python project.

uv init --bare

This command creates a minimal pyproject.toml file. This file will track our project’s metadata and, most importantly, its dependencies.

Create a virtual environment using uv:

uv venv

This will create a .venv directory. Activate it with the following:

source .venv/bin/activate

# On Windows, use
#.venv\Scripts\activate

Install the necessary Python packages using uv:

uv add python-telegram-bot python-dotenv langchain langchain-openai langchain-community faiss-cpu jq sentence-transformers

The key libraries are:

python-telegram-bot: For handling all Telegram communication
langchain: The primary framework for building the RAG pipeline
langchain-openai: Connector to Open WebUI’s OpenAI-compatible API
faiss-cpu: An efficient library for similarity search, used as a local vector store to quickly find relevant chunks of your FAQ data

Environment and configuration

The bot reads the Telegram token from the environment variable BOT_TOKEN. We can store it in a .env file as BOT_TOKEN=your-token-here.

# .env (Open WebUI)
# Open WebUI_URL must end with /v1 (e.g., http://localhost:3000/v1).
BOT_TOKEN=123456:abcdefg
Open WebUI_URL=http://localhost:3000/v1
Open WebUI_API_KEY=your_key_here

Inline mode requires enabling inline for the bot via BotFather.

Create a new file named bot.py and add the following code to set up and add message handlers for the Telegram bot.

import logging
import os
from uuid import uuid4
from telegram import Update, InlineQueryResultArticle, InputTextMessageContent
from telegram.ext import filters, MessageHandler, ApplicationBuilder, CommandHandler, ContextTypes, InlineQueryHandler
from dotenv import load_dotenv

# load .env variables
load_dotenv()
bot_token = os.getenv("BOT_TOKEN", "")

# Setup logging
logging.basicConfig(
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
    level=logging.INFO
)
logger = logging.getLogger(__name__)

async def start(update: Update, context: ContextTypes.DEFAULT_TYPE):
    await context.bot.send_message(chat_id=update.effective_chat.id, text="I'm a bot, please talk to me!")

async def echo(update: Update, context: ContextTypes.DEFAULT_TYPE):
    await context.bot.send_message(chat_id=update.effective_chat.id, text=update.message.text)

async def caps(update: Update, context: ContextTypes.DEFAULT_TYPE):
    text_caps = ' '.join(context.args).upper()
    await context.bot.send_message(chat_id=update.effective_chat.id, text=text_caps)

async def inline_caps(update: Update, context: ContextTypes.DEFAULT_TYPE):
    query = update.inline_query.query
    if not query:
        return
    results = []
    results.append(
        InlineQueryResultArticle(
            id=str(uuid4()),
            title='Caps',
            input_message_content=InputTextMessageContent(query.upper())
        )
    )
    await context.bot.answer_inline_query(update.inline_query.id, results)

async def unknown(update: Update, context: ContextTypes.DEFAULT_TYPE):
    await context.bot.send_message(chat_id=update.effective_chat.id, text="Sorry, I didn't understand that command.")

async def document(update: Update, context: ContextTypes.DEFAULT_TYPE):
    if update.message.document:
        file = await update.message.document.get_file()
        file_name = update.message.document.file_name
        await file.download_to_drive(file_name)
    elif update.message.photo:
        # Get the largest photo size
        file = await update.message.photo[-1].get_file()
        file_name = f"photo_{file.file_unique_id}.jpg" # Create a unique name for photos
        await file.download_to_drive(file_name)
    elif update.message.video:
        file = await update.message.video.get_file()
        file_name = update.message.video.file_name
        await file.download_to_drive(file_name)
    else:
        await update.message.reply_text("Please send a document, photo, or video.")
        return

def main() -> None:
    start_handler = CommandHandler('start', start)
    echo_handler = MessageHandler(filters.TEXT & (~filters.COMMAND), echo)
    caps_handler = CommandHandler('caps', caps)
    inline_caps_handler = InlineQueryHandler(inline_caps)
    document_handler = MessageHandler(filters.PHOTO | filters.Document.PDF | filters.VIDEO, document)
    unknown_handler = MessageHandler(filters.COMMAND, unknown)

    application = ApplicationBuilder().token(bot_token).build()

    application.add_handler(start_handler)
    application.add_handler(echo_handler)
    application.add_handler(caps_handler)
    application.add_handler(inline_caps_handler)
    application.add_handler(document_handler)
    application.add_handler(unknown_handler)

    # Run the bot
    logger.info("Starting bot polling...")
    application.run_polling()

if __name__ == '__main__':
    main()

Try it out

To run the application, simply run:

uv run bot.py

Search for your bot name in Telegram and send the bot a message or command like /start or /caps.

What this bot does

Responds to /start with a greeting
Echoes back any plain text message (that isn’t a command)
Converts text to uppercase via /caps or inline mode
Downloads files users send (photos, PDFs, and videos) to local storage
Politely handles unknown commands

Core structure

start(): Sends a simple welcome message when the user runs /start
echo(): Replies with the exact same text the user sent (only for non-commands)
caps(): Turns the arguments after /caps into uppercase and sends them back
inline_caps(): Provides an inline result that uppercases whatever users type after @YourBotName in any chat
document(): Saves received media to disk:
- Photos: Downloads the largest size, naming it photo_<unique_id>.jpg
- PDFs: Downloads using the document’s file name. Note: The filter only accepts PDFs as documents
- Videos: Downloads using the video’s file name
- If none of these are present, it prompts the user to send a supported file
unknown(): Catches any unrecognized commands and replies with a friendly error

Handlers and filters

CommandHandler('start', start) and CommandHandler('caps', caps) handle commands
MessageHandler(filters.TEXT & (~filters.COMMAND), echo) ensures normal text (not commands) is echoed
InlineQueryHandler(inline_caps) answers inline queries
MessageHandler(filters.PHOTO | filters.Document.PDF | filters.VIDEO, document) restricts downloads to photos, PDFs, and videos
MessageHandler(filters.COMMAND, unknown) is added last to catch all other commands

Running the bot

main() wires up the handlers, builds the Application with the token, logs a startup message, and starts long polling via application.run_polling().

This script is a clean, async-first Telegram bot scaffold that demonstrates commands, inline mode, message filtering, and media downloads—ready to extend for more sophisticated behaviors.

Now that we have our bot ready, we will extend the code to add a RAG pipeline to the bot.

Setting up the knowledge base

Let’s set up a knowledge base by creating a file named faqs.json to hold our data. The RAG pipeline will load and search this content. An example structure is shown below.

[
 {
    "category": "General",
    "question": "What are your operating hours?",
    "answer": "Monday to Friday, 9:00 AM–5:00 PM (local time)."
  },
  {
    "category": "Accounts",
    "question": "How do I reset my password?",
    "answer": "Go to our website, click Login, then Forgot Password. Check your email for the reset link."
  }
]

Setting up the RAG Pipeline

The RAG pipeline is the engine that converts our static JSON file into a searchable brain for our bot. This part initializes once and creates a vector database. In simple steps,

Load faqs.json
Create embeddings
Store them in FAISS
When a user asks a question, find similar answers and ask the LLM to write a reply based only on those

Data Ingestion (Indexing the FAQs) handled by `setup_rag_chain()` method

This part happens once when the bot starts. We load the faqs.json file, create vector embeddings, and store them in a searchable database (FAISS).

Load Data: Read the faqs.json file
Embeddings: Use an embedding model (like HuggingFace (Sentence-Transformers) all‑MiniLM‑L6‑v2.) to convert the text into numerical vectors
Vector Store: Store these vectors in a FAISS index for fast retrieval

The RAG Retrieval Logic handled by `handle_message()` method

When a user asks a question:

Embed Query: The user’s question is converted into an embedding vector
Retrieve Context: The query vector is used to perform a similarity search against the FAISS index. This returns the top K most relevant FAQs (question and answer pairs)
Construct Prompt: A final prompt is built, containing the user’s question and the retrieved relevant context

Example Prompt Template:

You are an expert FAQ assistant. Use the following context to answer
the user's question. If the context does not contain the answer,
state that you cannot help with this specific question. Context:
[Retrieved FAQs] Question: [User's message]

Generate Response: The complete prompt is sent to our Open WebUI model via the OpenAI-compatible API, e.g. gpt-5 or another model exposed by Open WebUI, which generates a coherent, context-grounded final answer
Send to Telegram: The bot sends the LLM’s final response back to the user

New capabilities added to `bot.py`

RAG pipeline: Loads FAQs from a local JSON file, embeds them with HuggingFace, retrieves the most relevant entries via FAISS, and drafts answers with an LLM served by Open WebUI.
Inline UX polish: Sends a “typing…” chat action while the model thinks.
Persisted chain: The RAG chain is built once at startup and stored in bot_data for reuse across messages.

The core logic of our bot will revolve around an update to the standard message handler. When a user sends a question, the bot no longer looks for a simple command; instead, it passes the question to the RAG pipeline.

Try it out

To run the application, simply run

uv run bot.py

Search for your bot name in telegram and send the bot a message like What are your operating hours?.

Changes to the RAG pipeline setup are available here. The source code is available here.

By integrating a RAG pipeline, we’ve leveled up our Telegram bot from a simple command processor to a knowledge-aware assistant. This approach ensures our bot’s answers are accurate, grounded in our provided faqs.json data, and remain consistent, dramatically reducing the chance of “hallucinations” from the underlying LLM.

This architecture is powerful and scalable. To expand its capabilities, we only need to update the faqs.json file and re-run the indexing step—no need to retrain or modify the core LLM!

Creating API for Invoice Generator Using C# and Minimal APIs

2025-07-02T00:00:00+00:00

In this blog post, we will explore how to create a robust invoice generator API using .NET 9 and minimal APIs, with a focus on integrating it with an existing Vue frontend whose implementation we covered in a previous blog post.

We will use the minimal APIs framework to create a RESTful API that can be easily used by our Vue application. Minimal APIs require less boilerplate code and configuration compared to traditional controller-based approaches. It is suited for smaller APIs, microservices, or serverless functions. You can learn more about choosing between controller-based APIs and minimal APIs here.

Prerequisites

Before going further into the code, we should make sure to have the following tools and technologies installed:

.NET 9 (the latest version of the .NET runtime)

Additionally, we already have an existing Vue frontend implementation that lacks API connectivity from our previous post. We will guide you through the process of integrating the Invoice Generator API with Vue application.

Setting up the Project

We will use the dotnet command to create a new .NET 9 project and set up the minimal APIs framework using the steps below:

Create a new .NET Core Web API project using the following command:

dotnet new webapi -n invoice-generator-api

Navigate to the newly created project directory:

cd invoice-generator-api

Open the Program.cs file which should look like below.

var builder = WebApplication.CreateBuilder(args);

// Add services to the container.
// Learn more about configuring OpenAPI at https://aka.ms/aspnet/openapi
builder.Services.AddOpenApi();

var app = builder.Build();

// Configure the HTTP request pipeline.
if (app.Environment.IsDevelopment())
{
    app.MapOpenApi();
}

app.UseHttpsRedirection();

var summaries = new[]
{
    "Freezing", "Bracing", "Chilly", "Cool", "Mild", "Warm", "Balmy", "Hot", "Sweltering", "Scorching"
};

app.MapGet("/weatherforecast", () =>
{
    var forecast =  Enumerable.Range(1, 5).Select(index =>
        new WeatherForecast
        (
            DateOnly.FromDateTime(DateTime.Now.AddDays(index)),
            Random.Shared.Next(-20, 55),
            summaries[Random.Shared.Next(summaries.Length)]
        ))
        .ToArray();
    return forecast;
})
.WithName("GetWeatherForecast");

app.Run();

record WeatherForecast(DateOnly Date, int TemperatureC, string? Summary)
{
    public int TemperatureF => 32 + (int)(TemperatureC / 0.5556);
}

We will also add the following packages for Swagger UI, Entity Framework, and the Entity Framework Core provider for PostgreSQL.

dotnet add package Microsoft.Extensions.ApiDescription.Server
dotnet add package Swashbuckle.AspNetCore.SwaggerUi
dotnet add package Microsoft.EntityFrameworkCore
dotnet add package Npgsql.EntityFrameworkCore.PostgreSQL
dotnet add package Microsoft.EntityFrameworkCore.Design

Setting up the Swagger UI

Update Program.cs and configure Swagger UI to point to the OpenAPI JSON file.

 if (app.Environment.IsDevelopment())
 {
     app.MapOpenApi();
+    app.UseSwaggerUI(config =>
+    {
+        config.SwaggerEndpoint("/openapi/v1.json", "v1");
+    });
 }

 app.UseHttpsRedirection();

Once the application is running using dotnet run, you can access Swagger UI at http://localhost:<PORT>/swagger on your browser, which should look like below.

Setting up the Database Context with Entity Framework

Create a file, InvoiceDbContext.cs, in the root of the project and add the following code.

using Microsoft.EntityFrameworkCore;

public class InvoiceDbContext : DbContext
{
    public InvoiceDbContext(DbContextOptions<InvoiceDbContext> options)
     : base(options)
    {

    }
}

Update the Program.cs file to include the configuration for connecting to the PostgreSQL database using Entity Framework.

+using Microsoft.EntityFrameworkCore;
+
 var builder = WebApplication.CreateBuilder(args);

+builder.Services.AddDbContext<InvoiceDbContext>(options =>
+    options.UseNpgsql(builder.Configuration.GetConnectionString("InvoiceDbcontext")));
+
+

Add the connection string InvoiceDbcontext in the appsettings.json file as shown below:

       "Microsoft.AspNetCore": "Warning"
     }
   },
-  "AllowedHosts": "*"
+  "AllowedHosts": "*",
+  "ConnectionStrings": {
+    "InvoiceDbcontext": "Host=<HOSTNAME>;Port=<PORT>;Database=<DATABASE>;Username=<USERNAME>;Password=<PASSWORD>;"
+  }
 }

Designing the Invoice Generator API

The next step is to design the API endpoints required for invoice generation. We will need the following endpoints:

GET /api/invoice — Get list of invoices
POST /api/invoice — Create a new invoice
GET /api/invoice/{id} — Get an invoice using ID
PUT /api/invoice/{id} — Update an invoice using ID
DELETE /api/invoice/{id} — Soft delete an invoice using ID
PUT /api/invoice/{id}/upload-logo — upload logo to saved invoice using ID
DELETE /api/invoice/{invoiceId}/detail/{id} — Soft delete invoice line item using line item ID and invoice ID

We will also need data models and entities to represent invoices and invoice line items. Let’s create the following classes in the model folder:

public class Base
{
    public int Id { get; set; }
    public DateTime CreatedAt { get; set; } = DateTime.UtcNow;
    public DateTime? ModifiedAt { get; set; }
    public bool IsDeleted { get; set; } = false;
    public DateTime? DeletedAt { get; set; }
}

public class Invoice : Base
{
    public string? Name { get; set; }
    public string? logo { get; set; }
    public string? Number { get; set; }
    public string? PONumber { get; set; }
    public virtual List<InvoiceItem> InvoiceItems { get; set; } = new();
    public UsageAmount? DiscountAmount { get; set; }
    public UsageAmount? TaxAmount { get; set; }
    public UsageAmount? ShippingAmount { get; set; }
    public decimal TotalAmount { get; set; }
    public decimal PaidAmount { get; set; }
    public string? Sender { get; set; }
    public string? Buyer { get; set; }
    public Status? Status { get; set; }
    public string? Notes { get; set; }
    public string? Terms { get; set; }
    public DateTime? Date { get; set; }
    public DateTime? DueDate { get; set; }
    public bool SentToBuyer { get; set; } = false;
}

public class InvoiceItem : Base
{
    public string Description { get; set; } = string.Empty;
    public int Quantity { get; set; }
    public decimal Rate { get; set; }
    public decimal Amount { get => this.Quantity * this.Rate; }
}

public class UsageAmount : Base
{
    public bool IsUsed { get; set; } = false;
    public bool IsPercentage { get; set; } = false;
    public decimal Value { get; set; } = 0;
}

Will will now use these classes to create tables in the database and then define a DBSet for each class in our InvoiceDbContext.cs.

 public InvoiceDbContext(DbContextOptions<InvoiceDbContext> options)
  : base(options)
 {

 }
+    public DbSet<Invoice> Invoices => Set<Invoice>();
+    public DbSet<InvoiceItem> InvoiceItems => Set<InvoiceItem>();
+    public DbSet<UsageAmount> UsageAmounts => Set<UsageAmount>();
 }

Since we will be using the soft delete feature in our API, we will use Global Query Filters to filter out the deleted records from our table. So, we configure the query filters in OnModelCreating using the HasQueryFilter API in InvoiceDbContext.cs. After which, the file should look like below.

 public InvoiceDbContext(DbContextOptions<InvoiceDbContext> options)
  : base(options)
 {

 }
+    protected override void OnModelCreating(ModelBuilder modelBuilder)
+    {
+        modelBuilder.Entity<Invoice>().HasQueryFilter(p => !p.IsDeleted);
+        modelBuilder.Entity<InvoiceItem>().HasQueryFilter(p => !p.IsDeleted);
+        modelBuilder.Entity<UsageAmount>().HasQueryFilter(p => !p.IsDeleted);
+    }
+
+    public DbSet<Invoice> Invoices => Set<Invoice>();
+    public DbSet<InvoiceItem> InvoiceItems => Set<InvoiceItem>();
+    public DbSet<UsageAmount> UsageAmounts => Set<UsageAmount>();
 }

Creating and running database migrations

We have defined models for our data models, but we need to create a database and tables for them. We can do this using Entity Framework tool dotnet ef. On the command prompt, in the root path of project, run the following command:

dotnet ef migrations add "create table and columns"

This command will create migration files in the Migrations folder. We can run this migration using the following command:

dotnet ef database update

Implementing API Endpoints

Now that we have designed the API endpoints and data models, let’s implement them using minimal APIs. We will create a new endpoints folder and add an InvoiceEndpoints.cs file in it. This file will contain the implementation for the Invoice API endpoints.

// InvoiceEndopints.cs

using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.Hosting.Internal;

public static class InvoiceEndpoints
{
    public static List<Invoice> Invoices = new();
    public static RouteGroupBuilder MapInvoiceEndpoints(this RouteGroupBuilder group)
    {
      // Return list of invoices
        group.MapGet("/", async (InvoiceDbContext db) =>
        {
            return await db.Invoices.Include(x => x.InvoiceItems).ToListAsync();
        })
            .WithSummary("Get list of invoices");

        // Return an existing invoice
        group.MapGet("/{id}", async (int id, InvoiceDbContext db) =>
        {
            return await db.Invoices.Include(x => x.InvoiceItems).FirstOrDefaultAsync(x => x.Id == id)
            is Invoice invoice
                ? Results.Ok(invoice)
                : Results.NotFound();
        })
            .WithSummary("Get an invoice using id");

        // Create an invoice
        group.MapPost("/", async (Invoice invoice, InvoiceDbContext db) =>
        {
            invoice.DueDate = invoice.DueDate.HasValue ? invoice.DueDate.Value.ToUniversalTime() : null;
            invoice.Date = invoice.Date.HasValue ? invoice.Date.Value.ToUniversalTime() : null;
            db.Invoices.Add(invoice);
            await db.SaveChangesAsync();
            return Results.Created($"/api/invoice/{invoice.Id}", invoice);
        })
            .WithSummary("Create a new invoice");

        // Create a new line item for an existing invoice
        group.MapPost("/{id}/items", async (int id, List<InvoiceItem> invoiceItems, InvoiceDbContext db) =>
        {
            if (await db.Invoices.FindAsync(id)
            is not Invoice invoice)
            {
                return Results.NotFound();
            }
            invoice.InvoiceItems.AddRange(invoiceItems);
            await db.SaveChangesAsync();
            return Results.Created($"/api/invoice/{invoice.Id}", invoice);
        })
            .WithSummary("Create new line items for invoice");

        // Update an existing invoice
        group.MapPut("/{id}", async (int id, Invoice inputInvoice, InvoiceDbContext db) =>
        {
            var invoice = await db.Invoices.AsNoTracking().FirstOrDefaultAsync(x => x.Id == id);

            if (invoice is null) return Results.NotFound();

            invoice.Id = id;
            db.Invoices.Update(inputInvoice);
            await db.SaveChangesAsync();
            return Results.NoContent();
        })
            .WithSummary("Update invoice");

        // Upload invoice logo image file for an existing invoice
        group.MapPut("/{id}/upload-logo", async (int id, IFormFile uploadImageFile, InvoiceDbContext db) =>
        {
            var invoice = await db.Invoices.AsNoTracking().FirstOrDefaultAsync(x => x.Id == id);

            if (invoice is null) return Results.NotFound();

            if (uploadImageFile == null)
            {
                throw new ArgumentNullException(nameof(uploadImageFile));
            }

            var contentPath = Environment.CurrentDirectory;
            var uploadPath = Path.Combine("Uploads", id.ToString());
            var rootPath = Path.Combine(contentPath, uploadPath);

            if (!Directory.Exists(rootPath))
            {
                Directory.CreateDirectory(rootPath);
            }
            string[] allowedFileExtensions = [".jpg", ".jpeg", ".png"];
            var ext = Path.GetExtension(uploadImageFile.FileName);
            if (!allowedFileExtensions.Contains(ext))
            {
                throw new ArgumentException($"Please upload only .jpg, .jpeg, and .png images.");
            }

            // generate a unique filename
            var fileName = $"{Guid.NewGuid().ToString()}{ext}";
            var fileNameWithPath = Path.Combine(uploadPath, fileName);
            var rootPathWithFileName = Path.Combine(rootPath, fileName);
            using var stream = new FileStream(rootPathWithFileName, FileMode.Create);
            await uploadImageFile.CopyToAsync(stream);

            invoice.logo = fileNameWithPath;
            db.Update(invoice);
            await db.SaveChangesAsync();
            return Results.NoContent();
          })
          .DisableAntiforgery()
          .WithSummary("upload logo");

        // soft delete an invoice
        group.MapDelete("/{id}", async (int id, InvoiceDbContext db) =>
        {
            if (await db.Invoices.FindAsync(id) is Invoice todo)
            {
                todo.IsDeleted = true;
                todo.DeletedAt = DateTime.Now.ToUniversalTime();
                await db.SaveChangesAsync();
                return Results.NoContent();
            }
            return Results.NotFound();
        })
            .WithSummary("Soft delete an invoice");

        return group;
    }
}

Then we will also implement InvoiceDetailEndpoint.cs. We will only define a delete endpoint for soft deleting the line items. We created this also to demonstrate how to use multiple files in the Program.cs to map the endpoints to a certain URL.

// InvoiceDetailEndpoint.cs

public static class InvoiceDetailEndpoints
{
    public static RouteGroupBuilder MapInvoiceDetailEndpoints(this RouteGroupBuilder group)
    {
        group.MapDelete("/{id}", async (int id, InvoiceDbContext db) =>
        {
            if (await db.InvoiceItems.FindAsync(id) is InvoiceItem invoiceItem)
            {
                invoiceItem.IsDeleted = true;
                invoiceItem.DeletedAt = DateTime.Now.ToUniversalTime();
                await db.SaveChangesAsync();
                return Results.NoContent();
            }
            return Results.NotFound();
        }).WithSummary("Soft delete invoice line item");

        return group;
    }
}

Similar to how traditional controller APIs are separated in their own files, we now have the implementation for the endpoints in two files. We will now register and map the endpoints to a preferred URL using MapGroup in the Program.cs file.

 // Program.cs

+var root = app.MapGroup("api");
+var invoice = root.MapGroup("invoice").MapInvoiceEndpoints();
+var invoiceDetail = root.MapGroup("invoice/{invoiceId}/detail").MapInvoiceDetailEndpoints();
+
 app.Run();

We also need to add a config to expose the Uploads folder so the images can be loaded from the running API server. Also, make sure to create the Uploads folder before running the application, otherwise the application wouldn’t run.

 // Program.cs

 var invoice = root.MapGroup("invoice").MapInvoiceEndpoints();
 var invoiceDetail = root.MapGroup("invoice/{invoiceId}/detail").MapInvoiceDetailEndpoints();

+app.UseStaticFiles(new StaticFileOptions
+{
+    FileProvider = new PhysicalFileProvider(
+        Path.Combine(builder.Environment.ContentRootPath, "Uploads")),
+    RequestPath = "/Uploads"
+});
+
 app.Run();

Adding CORS for development

While working locally we usually run the frontend application and the API application in different ports, so CORS is required for the frontend application to access the API. To enable CORS in development mode, we can add the following code snippet to our Program.cs file:

 // Program.cs

 using Microsoft.EntityFrameworkCore;
 using Microsoft.Extensions.FileProviders;

 var builder = WebApplication.CreateBuilder(args);
+var MyAllowSpecificOrigins = "_myAllowSpecificOrigins";
+builder.Services.AddCors(options =>
+{
+    options.AddPolicy(name: MyAllowSpecificOrigins,
+                      policy =>
+                      {
+                          policy.AllowAnyOrigin();
+                          policy.AllowAnyHeader();
+                          policy.AllowAnyMethod();
+                      });
+});

 builder.Services.AddDbContext<InvoiceDbContext>(options =>
     options.UseNpgsql(builder.Configuration.GetConnectionString("InvoiceDbcontext")));

 app.MapGet("/weatherforecast", () =>
 var root = app.MapGroup("api");
 var invoice = root.MapGroup("invoice").MapInvoiceEndpoints();
 var invoiceDetail = root.MapGroup("invoice/{invoiceId}/detail").MapInvoiceDetailEndpoints();

+app.UseCors(MyAllowSpecificOrigins);
 app.UseStaticFiles(new StaticFileOptions
 {
     FileProvider = new PhysicalFileProvider(

Now, we can start the application and test our API. When we go to /swagger, we can see our API endpoints.

The source code for the API is available on GitHub.

Integrating with Vue 3 Application

We already created a Vue 3 application in the previous blog. Now, we will integrate our API with the Vue application.

To integrate the Invoice Generator API with our Vue 3 application, we will need to configure API calls from the frontend to interact with the API. We will follow the following steps:

Install the axios package in our Vue 3 project:

npm install axios

Create a new file called index.ts in the services folder of our Vue 3 project and add the following code. Also, add .env file with key VITE_API_URL and value of the API URL:

// services/index.ts

import axios from 'axios'
export default axios.create({
    baseURL: import.meta.env.VITE_API_URL,
    timeout: 5000,
});

# .env
VITE_API_URL=https://localhost:7041/api/
VITE_BACKEND_URL=https://localhost:7041/

Create a new file called invoiceService.ts in the services folder of your Vue 3 project. Add the methods that calls the API endpoints:

import { Invoice, InvoiceItem } from "../types/index.type"
import axios from "./"

export default {
    getInvoices: async () => {
        return await axios.get('invoice')
    },
    getInvoice: async (id) => {
        return await axios.get(`invoice/${id}`)
    },
    saveInvoice: async (data: Invoice) => {
        return await axios.post('invoice', data)
    },
    deleteInvoice: async (id: number) => {
        return await axios.delete(`invoice/${id}`)
    },
    updateInvoice: async (id: number, data: Invoice) => {
        return await axios.put(`invoice/${id}`, data)
    },
    uploadLogo: async (invoiceId, file) => {
        const formData = new FormData();
        formData.append("imageFile", file);
        return axios.put(`invoice/${invoiceId}/upload-logo`, formData, { headers: { "Content-Type": "multipart/form-data" } });
    },
    addLineItem: async (id: number, data: InvoiceItem) => {
        return await axios.post(`invoice/${id}/detail/`, data);
    },
    updateLineItem: async (id: number, lineId: number, data: InvoiceItem) => {
        return await axios.put(`invoice/${id}/detail/${lineId}`, data)
    },
    deleteLineItem: async (id: number, lineId: number) => {
        return await axios.delete(`invoice/${id}/detail/${lineId}`)
    },
}

We will use invoiceService.ts to make API calls in our application. Normally, the API calls are usually made from Vue 3 components, but in this case we will use it to make API calls from our composable useInvoiceStorage.ts where all the business logic related to invoice are handled. Below are the code changes done to integrate the API calls from invoiceService.ts.

 // composables/useInvoiceStorage.ts

 import { useStorage } from '@vueuse/core'
-import { reactive } from 'vue';
+import { computed, ref } from 'vue';
 import { Invoice, InvoiceStore } from '../types/index.type';
+import invoiceService from '../services/invoiceService';

 const storage = useStorage<InvoiceStore>('invoice-store', {
-    currentInvoiceNumber: 10000,
-    invoices: {}
+    currentInvoiceNumber: "10000",
 }, localStorage);

+const state = ref({
+    ...storage.value,
+    invoices: {},
+});
 export default function useState() {
-    const state = reactive(storage);

     function saveInvoice(invoice: Invoice) {
         state.value.invoices[invoice.number] = { ...invoice };
-        state.value.currentInvoiceNumber = invoice.number + 1;
+        state.value.currentInvoiceNumber = `${+invoice.number + 1}`;
+        invoiceService.saveInvoice(invoice);
     }

-    function updateInvoice(invoice: Invoice) {
+    function updateInvoice(id: number, invoice: Invoice) {
         state.value.invoices[invoice.number] = { ...invoice };
+        invoiceService.updateInvoice(id, invoice);
     }

-    function deleteInvoice(invoiceId: number) {
+    async function deleteInvoice(invoiceId: number) {
         // delete state.value.invoices[invoiceId];
+        await invoiceService.deleteInvoice(invoiceId);
         state.value.invoices = Object.keys(state.value.invoices).filter(x => x != invoiceId.toString()).reduce((acc, curr) => {
             return {
                 ...acc,
@@ -40,15 +45,33 @@ export default function useState() {
         })
             .map(x => ({ ...state.value.invoices[x] }))

+
+    }
+
+    async function fetchInvoices() {
+        const result = await invoiceService.getInvoices();
+        state.value.invoices = result.data.reduce((acc, curr) => {
+            return { ...acc, [curr.id]: curr }
+        }, {})
+    }
+    async function fetchInvoice(id) {
+        const result = await invoiceService.getInvoice(id);
+        console.log(result);
+        state.value.invoices[result.data.id] = result.data;
+        console.log(state.value.invoices);
+        return result.data;
     }
     return {
         // variables
         storage: state,
+        invoices: computed(() => state.value.invoices),

         // methods
         saveInvoice,
         updateInvoice,
         deleteInvoice,
-        findInvoices
+        findInvoices,
+        fetchInvoices,
+        fetchInvoice
     }
 }

The above fetchInvoices method is then used by Invoices.vue to fetch the invoices. Similarly, all other endpoints are used in the application for CRUD operations.

The source code for the Vue app is available on GitHub.

Deployment

To deploy our API to Azure App Service, we can follow the documentation provided by Microsoft here.

Conclusion

In this blog post, we explored how to create a robust invoice generator API using .NET 9 and minimal APIs. We designed the API endpoints, implemented them using minimal APIs, and integrated the API with an existing Vue 3 application. This approach allowed us to leverage the power of minimal APIs while maintaining a clean and modular codebase.

Learning Vue 3 composables by creating an invoice generator

2025-01-06T00:00:00+00:00

The latest major version of Vue, Vue 3, has new features that are not present in Vue 2, such as Teleport, Suspense, and support for multiple root elements per template. Vue 3 provides smaller bundle sizes, better performance, better scalability, and better TypeScript IDE support.

What are Vue 3 composables?

Writing repetitive code can be a real pain in the frontend development realm. We can use Vue 3 composables to encapsulate and reuse stateful logic in our components. In this blog post, we will look at how we can use composables to reuse business logic by building an invoice generator.

An invoice generator, simply said, is an application that creates and manages invoices. We will focus on creating Vue 3 composables to properly separate business logic while reusing it across different features.

Core functionalities for invoice generator

Adding, editing, and removing invoices and items
Calculating totals with discounts, taxes, and shipping
Generating PDF invoices

To implement these functionalities, we’ll focus on:

Creating composables to handle business logic
Reusing this logic across features and components

Prerequisites

Basic understanding of:

Vue 3 and Composition API
JavaScript/TypeScript fundamentals

Tools:

Node.js and npm installed
Vite
Tailwind CSS (optional)

Setting up the project

First, initialize the Vue 3 project:

$ npm create vite@latest ep-invoice-generator
Need to install the following packages:
create-vite@6.0.1
Ok to proceed? (y) y
√ Select a framework: » Vue
√ Select a variant: » TypeScript

Scaffolding project in G:\PersonalWork\blog\ep-invoice-generator...

Done. Now run:

  cd ep-invoice-generator
  npm install
  npm run dev

Create folders for the project structure:

src/components/ for adding UI components
src/composables/ for adding composables
src/router/ for adding Vue Router routes
src/types/ for adding type definitions
src/views/ for adding pages to render based on Vue Router routes

Create first composable for invoice: `useInvoice`

The composable is named useInvoice.ts and it will serve the following purposes:

Handle data for invoice and invoice items
Add methods for adding, editing, and removing invoice items
Calculate totals automatically using item’s rate and amount as well as discount, tax, and shipping
Reset invoice

Define interfaces for `Invoice` and `InvoiceItem`

export interface InvoiceItem {
    id: number,
    description: string,
    rate: number,
    quantity: number,
}

export interface Invoice {
    logo: string,
    name: string,
    number: number,
    ponumber: string,
    date: string,
    duedate: string,
    sender: string,
    buyer: string,
    items: InvoiceItem[];
    notes: string,
    terms: string,
    discount: {
        isUsed: boolean,
        isPercentage: boolean,
        value: number,
    },
    tax: {
        isUsed: boolean,
        isPercentage: boolean,
        value: number,
    },
    shipping: {
        isUsed: boolean,
        value: number,
    },
    total: number,
    paid: number,

    sentToContact: boolean,
    status: Status,
}

First we make an Invoice object to create and edit an invoice. The code state.storage.value.currentInvoiceNumber gets the current invoice number to use for the new invoice, we will learn more about it later.

We return the object containing the required data and methods that can be used by any single-file component (SFC). We will see the implementation of computed properties and methods in the next section.

export function useInvoice(invoiceId?: number) {

    const invoice = reactive<Invoice>({
        logo: "",
        name: "",
        number: state.storage.value.currentInvoiceNumber,
        ponumber: "",
        date: format(date, "yyyy-MM-dd"),
        duedate: "",
        sender: "",
        buyer: "",
        items: [
            {
                id: 1,
                description: "",
                rate: 0,
                quantity: 0,
            },
        ],
        notes: "",
        terms: "",
        discount: {
            isUsed: false,
            isPercentage: false,
            value: 0,
        },
        tax: {
            isUsed: false,
            isPercentage: false,
            value: 0,
        },
        shipping: {
            isUsed: false,
            value: 0,
        },
        total: 0,
        paid: 0,
        sentToContact: false,
        status: Status.Draft,
    });

    const subtotal = computed(() => ...)
    const afterDiscount = computed(() => ...)
    const afterTax = computed(() => ...)
    const afterShipping = computed(() => ...)
    const total = computed(() => ...)
    const balance = computed(() => ...);

    const updateLineItem = (value: InvoiceItem, index: number) => ...);
    const addLineItem = () => ...)
    const removeLineItem = (index) => ...)

    return {
        // data helpers
        invoice,
        subtotal,
        afterDiscount,
        afterTax,
        afterShipping,
        total,
        balance,

        // Methods
        addLineItem,
        updateLineItem,
        removeLineItem,
    }
}

Adding business logic

Now we implement the business logic for various methods — adding an invoice item, updating an invoice item, etc. — as well as calculating the various dynamic amounts.

The following methods add an invoice item to the invoice as a line item, which can be rendered on the UI. Then, the user can update the line item’s rate and quantity, which calls updateLineItem to update the invoice item data.

/**
 * Add a line item to the invoice line items array
 */
const addLineItem = () => {
    invoice.items.push({
        id: invoice.items.length + 1,
        description: "",
        rate: 0,
        quantity: 0,
    });
};

/**
 * @param {invoice line item} value
 * @param {index of invoice line item} index
 * @returns
 */
const updateLineItem = (value: InvoiceItem, index: number) => (invoice.items[index] = { ...value });

/**
 * Remove items from invoice items array
 * @param {*} index index of item to remove
 */
const removeLineItem = (index) => {
    invoice.items.splice(index, 1);
};

Based on the invoice item data, we then calculate:

subtotal
subtotal after discount, if available
subtotal after tax, if available
subtotal after shipping, if available

Moreover, discount and tax can be stored as an amount or as a percentage value, so we update the logic to evaluate discount and tax based on the type of the entered values and calculate the subtotal. We can also include a previously overpaid amount and calculate the balance at the end of the calculations.

// useInvoice.ts
const subtotal = computed(() =>
    invoice.items.reduce((prev, acc) => prev + acc.rate * acc.quantity, 0)
);

/**
 * Compute total value after applying discount
 * based on `discount` percentage or value and `subtotal` value
 */
const afterDiscount = computed(() => {
    if (invoice.discount.isUsed) {
        if (invoice.discount.isPercentage) {
            return subtotal.value - (subtotal.value * invoice.discount.value) / 100;
        } else {
            return subtotal.value - invoice.discount.value;
        }
    }
    return subtotal.value;
});

/**
 * Compute total value after applying tax
 * based on `tax` percentage or value and `afterDiscount` value
 */
const afterTax = computed(() => {
    if (invoice.tax.isUsed) {
        if (invoice.tax.isPercentage) {
            return (
                afterDiscount.value + (afterDiscount.value * invoice.tax.value) / 100
            );
        } else {
            return afterDiscount.value + invoice.tax.value;
        }
    }
    return afterDiscount.value;
});

/**
 * Compute total value after applying shipping
 * based on `shipping` value and `afterTax` value
 */
const afterShipping = computed(() => {
    if (invoice.shipping.isUsed) {
        return afterTax.value + invoice.shipping.value;
    }
    return afterTax.value;
});

const total = computed(() => {
    return afterShipping.value;
});

// later if we track the payment done for the clients, we can easily
// come up with the balance based on the payment and invoices
const balance = computed(() => total.value - invoice.paid);

Rendering the invoice with composable functions

We create the InvoiceGenerator.vue SFC, utilizing the useInvoice composable we created. We use the <script setup> syntax for importing and using the composable. <script setup> help us use Composition API inside SFCs instead of the Options API. Using the <script setup> syntax removes boilerplate code while providing the ability to declare props, emitted events, and many more advantages.

<!-- InvoiceGenerator.vue-->
<script setup>
import { useInvoice } from "../composables/useInvoice";

const {
  // data
  invoice,

  // computed / read only
  subtotal,
  total,
  balance,

  // methods
  addLineItem,
  updateLineItem,
  removeLineItem,
  resetInvoice
} = useInvoice(
</script>

I have already defined the LineItems.vue component, where we render the line items which have text boxes to enter description, rate, and amount. The totals for each item are calculated using the computed property. Let’s see an example showing how we can use it with invoice items:

Use items prop to render the invoice items
Emit update-item event when any value for an invoice line item changes
Emit add-item when user adds a new line item
Emit close when user removes a line item

<LineItems
    :items="invoice.items"
    @update-item="updateLineItem"
    @close="removeLineItem"
    @add-item="addLineItem"
    />

Similarly, we use invoice.discount, invoice.tax, and invoice.shipping properties from the composable in a text box where the user can enter the value and type of the value. All of them have helper properties named isUsed and isPercentage, which we use to make the input box user-friendly. The CustomInput, CustomToggleSwitch and WithLabel are three custom components that are already created and they can be used as follows:

<WithLabel label="Discount">
    <CustomInput
        v-model.number="invoice.discount.value"
        is-used
        label="Discount"
        toggle
        :currency="!invoice.discount.isPercentage"
        @close="invoice.discount.isUsed = false"
    >
        <CustomToggleSwitch
        v-model="invoice.discount.isPercentage"
        :for-value="'invoice-discount'"
        />
    </CustomInput>
</WithLabel>

<WithLabel label="Tax">
    <CustomInput
        v-model.number="invoice.tax.value"
        is-used
        label="Tax"
        toggle
        :currency="!invoice.tax.isPercentage"
        @close="invoice.tax.isUsed = false"
    >
        <CustomToggleSwitch
        v-model="invoice.tax.isPercentage"
        :for-value="'invoice-tax'"
        />
    </CustomInput>
</WithLabel>

<WithLabel label="Shipping">
    <CustomInput
        v-model.number="invoice.shipping.value"
        label="Shipping"
        is-used
        currency
        @close="invoice.shipping.isUsed = false"
    >
    </CustomInput>
</WithLabel>

All other properties can also be rendered, but we would like some input fields, like PO Number, Invoice Date, and Amount Paid to be rendered differently. Using custom components like CustomInput and CustomToggleSwitch can help us get more control of the visual and functional aspects of the UI component.

<WithLabel label="PO Number">
    <CustomInput
    v-model="invoice.ponumber"
    label="PO Number"
    type="text"
    />
</WithLabel>

<WithLabel label="Date">
    <CustomInput
    v-model="invoice.date"
    label="Date"
    type="date"
    />
</WithLabel>

<WithLabel label="Amount Paid">
    <CustomInput
    v-model.number="invoice.paid"
    label="Amount Paid"
    currency
    />
</WithLabel>

Create a second composable for using localStorage to save and retrieve invoice data

The useInvoice composable only gives us the invoice data along with computed properties, but now we have to save it somewhere so we can use it later. We can use localStorage to save the invoices. VueUse is a very good library that has a collection of Vue composition utilities that can be used with Vue 2 as well as Vue 3.

For accessing localStorage reactively, we will use a composable from VueUse called useStorage, which uses localStorage by default. We will define saveInvoice, updateInvoice, deleteInvoice, and findInvoices methods that use the reactive variable from useStorage to perform the required operations.

We will create another composable called useInvoiceStorage.ts, where we initialize our storage using useStorage, which also stores the current invoice number.

// useInvoiceStorage.ts

const storage = useStorage<InvoiceStore>('invoice-store', {
    currentInvoiceNumber: 10000,
    invoices: {}
}, localStorage);

Then we define and implement the methods discussed above for saving, updating, deleting, and finding invoices. The update of the current invoice number happens in the saveInvoice method.

// useInvoiceStorage.ts

export default function useState() {
    const state = reactive(storage);

    function saveInvoice(invoice: Invoice) {
        state.value.invoices[invoice.number] = { ...invoice };
        state.value.currentInvoiceNumber = invoice.number + 1;
    }

    function updateInvoice(invoice: Invoice) {
        state.value.invoices[invoice.number] = { ...invoice };
    }

    function deleteInvoice(invoiceId: number) {
        state.value.invoices = Object.keys(state.value.invoices)
            .filter(x => x != invoiceId.toString())
            .reduce((acc, curr) => {
                return {
                    ...acc,
                    [curr]: state.value.invoices[curr]
                }
            }, {})
    }

    function findInvoices(searchText: string) {
        searchText = searchText.toLowerCase();
        return Object.keys(state.value.invoices).filter(x => {
            const invoice = state.value.invoices[x];
            return invoice.number.toString().indexOf(searchText) > -1
                || invoice.buyer.toString().toLowerCase().indexOf(searchText) > - 1
                || invoice.date.toString().toLowerCase().indexOf(searchText) > - 1
                || invoice.total.toString().indexOf(searchText) > - 1
        })
            .map(x => ({ ...state.value.invoices[x] }))

    }
    return {
        // variables
        storage: state,

        // methods
        saveInvoice,
        updateInvoice,
        deleteInvoice,
        findInvoices
    }
}

Now, we can use data and methods from the useInvoiceStorage composable to show the latest invoice number as well as save, update, delete, and find invoices. The composable useInvoice also uses useInvoiceStorage to set the current invoice number for the new invoice. Whenever a user creates a new invoice, the current invoice number is automatically assigned to the invoice obtained from the useInvoiceStorage composable.

Moreover, the useInvoice composable is updated to support the new invoice and edit invoice features. Support for editing an invoice is achieved by passing invoiceId to the useInvoice composable, the invoiceId is then checked in the storage returned by useInvoiceStorage, if the invoice exists, the useInvoice composable returns the correct invoice data from storage.

The following code checks for the invoice in the storage:

// If the invoice is already in storage, use it
if (invoiceId) {
    const invoiceToUse = state.storage.value.invoices[invoiceId];
    if (invoiceToUse) Object.assign(invoice, invoiceToUse);
}

Printing Invoice

For printing invoices, we can simply use print styles from Tailwind CSS to style the elements that we want to print. A simple example is the CustomButton component in the source code. We don’t want to show any buttons on the PDF, so we use the Tailwind CSS style print:hidden to hide the buttons on the print view. We then need to call the method below on the invoice page to print the current page.

const print = () => window.print();

Bonus: create a third composable for showing dashboard data

The dashboard is important for looking at the high-level data and is crucial to any application. So, we add a new composable called useDashboardData.ts that will give us the total invoice amount, total paid amount, total due amount, and total drafted amount. It can be further expanded to include any other business logic as well. We will again use the useInvoiceStorage composable to get the list of invoices.

// useDashboardData.ts
import useState from "./useInvoiceStorage";
import { Status } from "../types/index.type";

export function useDashboardData() {
    const state = useState();

    const totalInvoiceAmount = computed(() => {
        const invoices = state.storage.value.invoices;
        const amount = Object.keys(invoices).reduce((acc, item) => acc + invoices[item].total, 0);
        return amount.toFixed(2)
    })

    const totalPaid = computed(() => {
        const invoices = state.storage.value.invoices;
        const amount = Object.keys(invoices).reduce((acc, item) => {
            if (invoices[item].status === Status.Paid) {
                return acc + invoices[item].total
            }
            return acc;
        }, 0);
        return amount.toFixed(2);
    })

    const totalDue = computed(() => {
        const invoices = state.storage.value.invoices;
        const amount = Object.keys(invoices).reduce((acc, item) => {
            if (invoices[item].status === Status.Overdue) {
                return acc + invoices[item].total
            }
            return acc;
        }, 0);
        return amount.toFixed(2);
    })

    const totalDrafted = computed(() => {
        const invoices = state.storage.value.invoices;
        const amount = Object.keys(invoices).reduce((acc, item) => {
            if (invoices[item].status === Status.Draft) {
                return acc + invoices[item].total
            }
            return acc;
        }, 0);
        return amount.toFixed(2);
    })

    return {
        // variables
        totalInvoiceAmount,
        totalPaid,
        totalDue,
        totalDrafted
    }
}

After that we create the Dashboard.vue component to use data from useDashboardData.ts and display the data on the dashboard. We can keep adding business logic to useDashboardData.ts to display data on the dashboard, and we can use it elsewhere to support other business logic.

<!-- Dashboard.vue -->
<template>
    <section class="max-w-[1800px] mx-auto">
        <div class="grid grid-cols-4 gap-x-4 mt-4">
            <Card>
                <template #header>Total Amount</template>
                <span>${{ totalInvoiceAmount }}</span>
                <template #footer></template>
            </Card>
            <Card>
                <template #header>Paid Amount</template>
                <span>${{ totalPaid }}</span>
                <template #footer></template>
            </Card>
            <Card>
                <template #header>Due Amount</template>
                <span>${{ totalDue }}</span>
                <template #footer></template>
            </Card>
            <Card>
                <template #header>Drafted Amount</template>
                <span>${{ totalDrafted }}</span>
                <template #footer></template>
            </Card>
        </div>
        <Invoices />
    </section>
</template>

<script setup lang="ts">
import Card from '../components/shared/Card.vue';
import { useDashboardData } from '../composables/useDashboardData';
import Invoices from './Invoices.vue';

const { totalInvoiceAmount, totalPaid, totalDue, totalDrafted } = useDashboardData();
</script>

The source code is available at GitHub.

An application demo is also available.

Next Steps

Use a REST API for saving the invoices to the database
Use Pinia to store the retrieved invoices so we can easily use the invoices
Add a composable to switch between different types of templates for printing PDFs

Conclusion

We learned about Vue 3 composables and how to use Composition API to create composables. We saw how composable functions are a game changer for building complex, maintainable, and scalable applications by encapsulating the business logic, organizing the code, and making it reusable. It helps to simplify the process and boost productivity.

Passing Data Between Components in Vue.js: An Overview

2024-04-03T00:00:00+00:00

Artwork: Still Life with Artichokes and a Parrot, 17th century, Italian. CC0.

Vue.js, with its simplicity and flexibility, is one of the most popular JavaScript frameworks on the web. One of the key aspects of building dynamic and interactive web applications with Vue is efficiently passing data between components. In this blog post, we will explore methods and good practices for data communication between Vue components.

Props

Usage

Directional data flow: Props are primarily used for establishing a unidirectional flow of data from parent components to child components — they allow parents to pass data down to their children.

Pros:

Simplicity: Props provide a simple and straightforward mechanism for passing data.

Cons:

One-Way Binding: It’s a one-way data binding mechanism, meaning that data flows from parent to child only, not from child to parent.

// ParentComponent.vue
<template>
  <ChildComponent :data-prop="parentData" />
</template>

<script>
export default {
  data() {
    return {
      parentData: "Hello from parent!",
    };
  },
};
</script>

// ChildComponent.vue
<template>
  <div>{{ dataProp }}</div>
</template>

<script>
export default {
  props: ['dataProp'],
};
</script>

Custom Events: Emitting Changes from Child to Parent

Usage:

Child to Parent Communication: Custom events are useful when a child component needs to communicate changes or send data back to its parent component.

Pros:

Bidirectional Communication: Enables two-way communication between parent and child components.
Flexibility: Allows children to notify parents about specific events or changes.

Cons:

Limited Scope: Best suited for parent-child relationships; might become cumbersome in complex component hierarchies.

// ChildComponent.vue
<template>
  <button @click="emitData">Send Data to Parent</button>
</template>

<script>
export default {
  methods: {
    emitData() {
      this.$emit('child-event', 'Data from child!');
    },
  },
};
</script>

// ParentComponent.vue
<template>
  <ChildComponent @child-event="handleChildEvent" />
</template>

<script>
export default {
  methods: {
    handleChildEvent(data) {
      console.log('Received data from child:', data);
    },
  },
};
</script>

EventBus: A Centralized Event Hub

Usage:

Non-Parent-Child Communication: EventBus provides a centralized event hub that allows components not directly related in a parent-child hierarchy to communicate.

Pros:

Flexibility and Decoupling: Components can emit and listen for events without tight coupling, enhancing flexibility.
Global Communication: Suitable for scenarios where communication needs to happen across various components.

Cons:

Global Scope: Might lead to unintended side effects if events are not managed carefully.

// EventBus.js
import Vue from 'vue';
export const bus = new Vue();

// ChildComponent.vue
<template>
  <button @click="emitData">Send Data to Anywhere!</button>
</template>

<script>
import { bus } from '../EventBus';

export default {
  methods: {
    emitData() {
      bus.$emit('customEvent', 'Data from child!');
    },
  },
};
</script>

// AnyComponent.vue
<template>
  <div>{{ receivedData }}</div>
</template>

<script>
import { bus } from '../EventBus';

export default {
  data() {
    return {
      receivedData: '',
    };
  },
  created() {
    bus.$on('customEvent', (data) => {
      this.receivedData = data;
    });
  },
};
</script>

Vuex: State Management for Larger Applications

For more complex applications, especially those with multiple components sharing state, Vuex is the recommended solution. Vuex is Vue’s official state management library, providing a centralized store for managing application-level state.

// store.js
import Vue from 'vue';
import Vuex from 'vuex';

Vue.use(Vuex);

export default new Vuex.Store({
  state: {
    sharedData: 'Shared state data!',
  },
  mutations: {
    updateData(state, newData) {
      state.sharedData = newData;
    },
  },
});

// AnyComponent.vue
<template>
  <div>{{ sharedData }}</div>
</template>

<script>
import { mapState } from 'vuex';

export default {
  computed: {
    ...mapState(['sharedData']),
  },
};
</script>

While Vuex is a powerful state management solution, it’s not always necessary for every Vue.js application. Here are some scenarios where you should consider using Vuex:

Large and Complex Applications: Vuex becomes more valuable as your Vue.js application grows in size and complexity. If your application has numerous components that need to share and synchronize state, managing the state centrally with Vuex can greatly simplify the development process.
Shared State Between Components: When multiple components need access to the same state data, prop drilling (passing data through multiple layers of components) can become unwieldy. Vuex provides a centralized store, eliminating the need for complex and deep prop chains.
Predictable State Management: If you require a clear and predictable pattern for managing state changes, mutations in Vuex provide a straightforward and traceable way to update the state. This is especially beneficial in applications where understanding and debugging state changes are crucial.
Asynchronous Operations: If your application involves asynchronous operations, such as fetching data from an API or handling complex workflows, Vuex actions provide a structured way to handle these operations and update the state accordingly.
Reusable and Composable Code: If you want to create modular and reusable code, Vuex allows you to define actions, mutations, and getters in a centralized store. This makes it easier to maintain and extend your application as new features are added.
Consistent State Across Components: In scenarios where maintaining consistent state across multiple components is crucial, using Vuex ensures that modifications to the state are handled in a controlled and synchronized manner.
Testing Requirements: If your application requires extensive unit testing for state-related logic, Vuex provides a structured architecture that makes it easier to test mutations, actions, and getters in isolation.

By understanding and utilizing props, custom events, EventBus, and Vuex, you can effectively manage and communicate data in your Vue.js applications. Choose the approach that best fits your project’s requirements, keeping in mind the scalability and maintainability of your code.

Key Takeaways from Practical Object-Oriented Design by Sandi Metz

2024-02-27T00:00:00+00:00

Practical Object-Oriented Design: An Agile Primer Using Ruby by Sandi Metz is one of those books that everybody who writes or aspires to write object-oriented code should read. Indeed, reading through this book will be valuable for seasoned practitioners and inexperienced novices alike. Whether it be to discover new concepts, remember why they are important, or articulate the reasons behind that warm fuzzy feeling you get in your stomach when you read well designed code, POODR can help.

I personally really like this book, and here at End Point it does have quite a following as well; it was the subject of a study group that we ran back in 2016. I like to dust it off every now and then to give it a re-read. To help with that, I’ve come up with a list of key takeaways from the book, with some of my own interpretations sprinkled in. For me, it serves as a sort of summary to help commit the book’s contents to memory. I thought I’d share it with you today.

While this book uses the Ruby language for its examples and discussions, the concepts it describes are equally applicable to any classical object-oriented language. Dynamically typed language users do get a little more mileage than users of statically typed ones. But still, something like 90% of the content is relevant to both. For this summary, I’ve taken care not to tie the discussion to Ruby too much and be as language agnostic as possible. I’ve chosen to write the code examples in C#.

Images taken from the book. Code taken from the book’s examples in Ruby and translated to C#.

Chapter 1: Object-Oriented Design

About design

Following the principles and patterns of object-oriented design produces code that is both cost-effective and a joy to work with. “Cost effective” means that it doesn’t take more effort than necessary to evolve, to change.
The reason design is important is because software changes. A good design is one that makes software easy to change.
Object-oriented design sees the world as a collection of parts that interact with each other. The parts we call “objects”, and the interactions we call “messages”.
In order to send messages to each other, objects need to have some knowledge about one another. This knowledge produces “dependencies” between them.
Managing these dependencies is a big part of object-oriented design. OOD aims to create dependencies in such a way that they don’t make the process of changing objects too difficult.
Design is important no matter the size of the application. Because both small and large applications change. And change is what design allows. A badly designed small application will eventually become a badly designed large one.
Design is the way in which code is arranged. Good design is an arrangement of code that makes change easy.
Our job as developers is to take our application’s requirements and our knowledge of design principles and use them to produce code that is cost effective today and tomorrow: code that is easy to change.
The idea is not to predict future changes. Instead, we have to accept that change will come and prepare for it. We prepare for it by writing code that leaves our options open for the future.
“The purpose of design is to allow you to design later, and its primary goal is to reduce the cost of change.”

Failures of design

Design fails when you don’t do it. Code bases that lack in design will eventually evolve into unmaintainable messes, where change becomes increasingly hard even for the most minor new requirements. “Throwing everything away and beginning from scratch” becomes a viable alternative.
Design fails when you overdesign. Armed with basic design skills, it’s easy to fall into the trap of developing wrong abstractions, applying principles incorrectly, seeing the wrong pattern in the wrong context. Applications that suffer from overdesign become gigantic castles of code full of indirection that become hard to change.
Design fails when you separate it from programming. Design is best executed as an iterative and incremental process with a quick feedback loop. The design needs to be adjusted frequently, as understanding of the domain changes. Design that is dictated to programmers from on high as part of a “Big Up Front Design” is doomed to failure. Such arrangements are not agile enough to be resilient to requirement changes.

Agile and design

Agile development says: “Avoid Big Up Front Design because you can’t know what the application will end up being because requirements always change and they do so often.”
Agile also says: “Good design is essential. Because change comes often, your code needs to be ready to accommodate those changes in an efficient and cost effective way.”
Design is not a “one time at the beginning of the project” type of task. It is an ongoing, iterative and incremental task that happens as requirements change, and as our domain knowledge and skills increase.

Metrics

Bad object-oriented design metrics are an indicator of a bad design. But good metrics are not an indicator of good design. A code base with bad metrics will most likely be hard to change.
The ideal software metric would be “cost per feature per time interval”. But this is nigh on impossible to calculate because “cost”, “feature”, and “time interval” are in most cases nebulous concepts.
Still, “cost per feature per time interval” tells us that our goal should always be to write code with a low cost per feature. Making sure that it is also low cost to evolve in the future.

Technical debt

Technical debt is borrowing time from the future. That is, putting out a bad design today in order to release a feature quickly. Tomorrow, when change comes, the bad design will prevent cost effective change. So time will have to be spent refactoring, turning the bad design into a good one. If refactoring doesn’t happen, the debt increases, making future changes more and more expensive.
How much design you do depends on your skill and your time frame. You can’t do so much design that it prevents you from delivering on time. Design is an investment, and for an investment to be good, it needs to return some profit. Good design’s returns are quick and plentiful.
“The trick to getting the most bang for your design buck is to acquire an understanding of the theories of design and to apply them appropriately, at the right time, and in the right amounts.”

Chapter 2: Designing Classes with a Single Responsibility

About classes

Classes are the most basic organizational structure of object-oriented systems written with class-based object-oriented languages. As such, that’s the first thing that we focus on when designing such systems.
Your first instinct should be to keep things simple. Use classes to model only the features that your application needs today and make sure that they are easy to change tomorrow.
“Design is more the art of preserving changeability than it is the act of achieving perfection.”

Write code that is TRUE

In order to be easy to change, code should be TRUE:
Code should be “Transparent”. It should be easy to tell what will be the consequences of changing the code.
Code should be “Reasonable”. The cost of a change should be proportional to the size of the requirement that provoked the change.
Code should be “Usable”. The code should be easy to reuse in situations other than the one it’s currently being used in.
Code should be “Exemplary”. The code should exhibit qualities that guide and/or encourage those who change it to keep and replicate those qualities.
The Single Responsibility Principle is a prerequisite to creating code that is TRUE.

Single Responsibility

A class should have one single responsibility. That means that it needs to do one thing and do it well. It needs to be as small as possible and fulfill a single purpose. It needs to have only one reason to change.
A good way to identify what classes your system needs is to read through your user stories or problem descriptions and identify nouns. Those nouns are good candidates for classes.
If there are verbs and properties associated with those nouns, then it is most likely a class. Because classes represent bundles of both data (attributes) and behavior (methods).
Classes that have many responsibilities are hard to reuse. And because they are not “Usable”, they are not easy to change.
A class that has many responsibilities usually has unrelated code tangled together within it. This makes the class hard to reason about, and hard to reuse.
In order to reuse the behavior defined within a class with multiple tangled responsibilities, you either need to duplicate code or use the entire class, even the parts that you don’t need.
If you use a class that has many reasons to change, when something unrelated to your use case changes, you will still have to change your code. This is unexpected and surprising. This is not “Transparent”.
Depending on classes that have many responsibilities increases the chances of your application breaking.
A good way to determine whether a class has a single responsibility is to try to enunciate what it does in a single sentence. That sentence should be simple, because classes should be simple. Look out for the words “and” and “or” in this sentence, as they point to it having more than one responsibility.
Another good way to determine whether a class has a single responsibility is thinking about every one of its methods as a question that the class would be able to respond to if it were a sentient being. For example: “Array, what is your length?” If the question doesn’t make sense, then maybe the method does not belong in the class.
Cohesion is the measure of how related to their core purpose the elements of a component are. Classes that have a single responsibility are highly cohesive because all of their parts work towards a single goal.

When to make design decisions

Only design as much as you need to for right now. Don’t make design decisions prematurely. Sometimes waiting for more information before committing to a design is the way to go. More information in the future will often reveal a better design.
Consider postponing design decisions when the information available to you today does not point clearly to a good design. Also, when the cost of not doing it now does not considerably make tomorrow’s change more expensive.
On the other hand, consider that a class that’s left in a “bad state” may be reused by others, making their code depend on badly designed code. The application will suffer as a result.
The structure of a class is supposed to reveal your design’s intention to other developers. When it doesn’t, when it is not “Exemplary”, then future maintenance costs increase.
The tension between “improve it now” and “improve it later” is always present. Our job as designers is to strike a good balance “between the needs of the present and the possibilities of the future” so that the costs of change are kept low.

Techniques for writing code that’s easy to change

Depend on behavior, not data: hide instance variables. Instead of accessing instance variables directly, use accessor methods. Even from within the class that owns them. “Always send messages to access data.”
That way you consolidate the knowledge of what the data represents in a single place. That’s the Single Responsibility Principle in action. Changing it becomes easy as a result because instead of changing references to a variable in potentially multiple places; you only need to change the definition of a method, in a single location.
Depend on behavior, not data: hide data structures. When code depends on or receives as input a complex data structure like a big array or hash, encapsulate the data structure in a class with a clear interface, instead of accessing and manipulating the structure directly.
Directly manipulating complex data structures is a style of coding that tends to propagate. If the incoming data structure changes, even slightly, then a lot of your code also needs to change because it depends on its very particular structure.
Handling complex raw data structures through classes and messages with clear, intention-revealing names demystifies the structure and gives it meaning within the context of the application domain.

For example, instead of this…

class ObscuringReferences
{
    int[][] data;

    public ObscuringReferences(int[][] data)
    {
        this.data = data;
    }

    public IEnumerable<int> GetDiameters()
    {
        return data.Select(cell => cell[0] + (cell[1] * 2));
    }
}

…do this:

class RevealingReferences
{
    IEnumerable<Wheel> Wheels { get; set; }

    public RevealingReferences(int[][] data)
    {
        Wheels = Wheelify(data);
    }

    // This method is now more readable.
    public IEnumerable<int> GetDiameters()
    {
        return Wheels.Select(wheel => wheel.Rim + (wheel.Tire * 2));
    }

    // Now everyone can send rim/tire to wheel.
    record Wheel(int Rim, int Tire);

    private IEnumerable<Wheel> Wheelify(int[][] data)
    {
        return data.Select(cell => new Wheel(cell[0], cell[1]));
    }
}

Enforce single responsibility everywhere: extract extra responsibilities from methods. Methods should also have a single responsibility. Just like classes, that makes them easy to reuse, change, and understand.
To determine if a method has a single responsibility, the same techniques that work for classes apply. Try to enunciate their purpose in a single sentence, and ask them what they do.
Methods that iterate on items and act upon them too are a common case of multiple responsibilities. Separating iteration and action into two methods is a common refactoring to correct it.
Complex calculations embedded within methods are also good candidates to be separated into other methods. It gives them a name and makes them reusable.

For example, instead of this:

IEnumerable<int> GetDiameters()
{
    return Wheels.Select(wheel => wheel.Rim + (wheel.Tire * 2));
}

…do this:

// First - iterate over the collection.
IEnumerable<int> _GetDiameters()
{
    return Wheels.Select(wheel => GetDiameter(wheel));
}

// Second - calculate diameter of ONE wheel.
int GetDiameter(Wheel wheel)
{
    return wheel.Rim + (wheel.Tire * 2);
}

These refactorings are useful and necessary even when you don’t know what the final design will look like. In fact, these refactorings (and good practices like these) will often reveal the final design.
Small methods with a single responsibility have many benefits. They make a class’s details, features, and components more obvious, they are self-documenting, encourage reuse, establish a style of programming that is self-perpetuating, become easy to move to another class when and if that time comes.
Enforce single responsibility everywhere: isolate extra responsibilities in classes. Move the responsibilities that are extraneous to a class into another class. If you can afford it, create a separate class to hold them. If not, at least encapsulate them in an embedded class so that they are contained and don’t leak.
Embedded classes say: This class only has meaning and value when thought about within the context of this other class.
Embedded classes can be easily promoted to independent classes later if the need arises.
Once all methods inside a class have a single responsibility and are each doing the smallest useful thing, it becomes easier to identify the features that may belong to another class.
The purpose of design is not to produce “perfect” code but instead to produce code that’s “good enough”.
The Single Responsibility Principle “allows change without consequence and reuse without duplication”.

Chapter 3: Managing Dependencies

About dependencies

Objects should have a single responsibility and do the smallest useful thing. This means that, in order to fulfill complex application requirements, objects need to collaborate with one another.
Objects collaborate by knowing things about and sending messages to each other. That creates dependencies. A good design manages these dependencies, making sure they don’t couple objects so much that change becomes painful.
Coupling measures how much interdependency exists between software components. High coupling makes things hard to change.
An object has a dependency when it knows the name of another class.
An object has a dependency when it knows the name of a method from another class.
An object has a dependency when it knows the arguments of that method.
An object has a dependency when it knows the order of those arguments.
Dependencies create a situation where if an object changes, then other objects that depend on it may be forced to change as well.
Dependencies couple objects to each other. Highly coupled objects behave as if they were a unit. It becomes hard to reuse one without the other. If not managed properly, dependencies will disseminate and entangle the entire code base.
Some dependencies are unavoidable. So the purpose of design is to reduce and contain dependencies. Ensure objects only have the minimum number of dependencies that they need to do their jobs. Keep the essential ones, remove the rest.

Techniques for writing loosely coupled code

Dependency injection. It can help eliminate dependencies on concrete classes. Instead of instantiating new objects of another class inside your object, let the object receive it as a constructor or method parameter. The object’s users can do the instantiation.
An object that knows less can often times do more. The knowledge of how to instantiate a particular object limits its potential. Object creation and utilization don’t need to be contained within the same unit of code.
When an object hard-codes the reference to a class, it advertises that it can’t work with any other type. Replace the reference to the concrete class with an abstraction (e.g. an interface or duck type). That gives it the flexibility to be able to collaborate with other objects that respond to the same messages.

With dependency injection you can turn this:

class Gear
{
    float Ratio { get; set; }
    int Rim { get; set; }
    int Tire { get; set; }

    public Gear(float ratio, int rim, int tire)
    {
        Ratio = ratio;
        Rim = rim;
        Tire = tire;
    }

    public float GetGearInches()
    {
        return Ratio * new Wheel(Rim, Tire).GetDiameter();
    }
}

Into this:

class Gear
{
    float Ratio { get; set; }
    Wheel Wheel { get; set; }

    // Instead of rim and tire, Gear's constructor now receives a wheel.
    public Gear(float ratio, Wheel wheel)
    {
        Ratio = ratio;
        Wheel = wheel;
    }

    public float GetGearInches()
    {
        // Instead of creating a new wheel, it uses the one that's been injected.
        return Ratio * Wheel.GetDiameter();
    }
}

Sometimes it is not possible to remove a dependency from an object. In those cases, your best bet is to isolate it.
Isolate dependencies: isolate instance creation. Sometimes it is not possible to apply dependency injection. In those cases, the next best thing is to encapsulate instantiation of external classes into their own methods.
An instantiation encapsulated in a method allows the code to clearly and explicitly communicate that there’s an extraneous dependency here. One that we’d like to remove but for now cannot. We also protect the class from that dependency propagating through it.
Isolate dependencies: isolate vulnerable external messages. The same technique can be applied to invocations of methods on external classes. Encapsulate such calls into their own methods in order to contain the dependency.

Here’s what isolating some dependencies could look like. From this…

float GetGearInches()
{
    return Ratio * new Wheel(Rim, Tire).GetDiameter();
}

…to this:

float GetGearInches()
{
    return Ratio * GetWheelDiameter();
}

// The dependency on the Wheel class is isolated in this method.
int GetWheelDiameter()
{
    return new Wheel(Rim, Tire).GetDiameter();
}

Reduce argument dependencies: use keyword arguments. If your language supports them, keyword arguments are a great way of reducing the dependency incurred via method invocations. They allow the parameters to be sent in any order, which means that changes to the method’s definition and signature are less likely to cause the callers to have to also change. Also, they have a self-documenting property on both caller and callee. (See named arguments in C# and keyword arguments in Ruby).
Reduce argument dependencies: explicitly define defaults. If your language has the feature, parameter defaults are a great way of reducing dependencies to method parameters, as establishing a default for a parameter makes it optional.
Reduce argument dependencies: isolate multiparameter initialization. If you don’t own the code for a complex object initialization or method invocation, you can always encapsulate it in an object or method of your own. It can be done with a thin wrapper that exposes an interface that follows the techniques we’ve discussed and that uses the language of your application.
Classes should depend on things that change less often than they do.
There are three truths about code related to change and dependencies:
1. “Some classes are more likely to change than others”.
2. “Concrete classes are more likely to change than abstract classes”.
3. “Changing a class that has many dependents will have many consequences”.
It’s OK to depend on classes with a low likelihood of change. For example, classes in the built-in library of your language generally will change less than your own custom classes. Framework classes commonly change less often too, but that may not always be the case if you framework is new or rapidly evolving.
Abstractions are generally much more stable than concretions. Because of this, it is safer to depend on abstract classes (e.g. interfaces and duck types) than on concrete ones.
Avoid classes that have lots of dependencies. Any changes done to them produce a ripple effect of changes throughout the code base. Also, because of that, people will go through great lengths to try to avoid changing them.
When thinking about dependents and likelihood of change, the following are harmless:
1. Classes with few dependents and low likelihood of change (e.g. specialized infrastructure interaction classes).
2. Classes with few dependents and high likelihood of change (e.g. concrete classes that implement your app’s domain logic).
3. Classes with many dependents and low likelihood of change (e.g. interfaces and other abstractions).
When thinking about dependents and likelihood of change, the following is harmful: Classes with many dependents and high likelihood of change (e.g. concrete classes that are used throughout the code base).

Chapter 4: Creating Flexible Interfaces

About interfaces

The messages that objects pass between each other are a big concern for design. In addition to what objects know (responsibilities), and who they know (dependencies), design cares about how they talk to one another. Messages pass between objects through their interfaces.
In a well designed application, the messages that pass between objects follow a pattern that’s closer to the diagram on the right than that of the diagram on the left:

Objects that expose too much of themselves and know too much about others are hard to reuse and make systems hard to change.
Objects that minimize what they expose of themselves and know little about others are easily reusable and changeable.
An object’s public interface is the set of messages that it responds to. That is, the set of methods that other objects are welcome to invoke. Good design calls for objects with clear and concise public interfaces.
Classes should have one responsibility, but they will likely have many methods. Some methods are more general and expose the main features of a class; they are the services that the class offers its callers. These should make up the class’s public interface. Other methods are more specific, serve to support those features, and contain implementation details internal to the class and uninteresting to callers. These make up the class’s private interface.
The methods in a class’s public interface:
1. “Reveal its primary responsibility”
2. “Are expected to be invoked by others”
3. “Are not likely to change”
4. “Are safe for others to depend on”
5. “Are directly covered by tests”
The methods in a class’s private interface:
1. “Handle implementation details”
2. “Are not expected to be invoked by other objects”
3. “Are likely to change”
4. “Are unsafe for others to depend on”
5. “May not even be referenced in the tests”
Public methods list the specific features of a class that allow it to fulfill its responsibility. They advertise to the world what the purpose of the class they belong to is.
Public methods are stable and are expected to not change often, so others are welcome to depend on them.
Private methods are not stable at all. They are hidden from others so nobody should depend on them.

Discovering less obvious objects and messages

When analyzing a problem domain, the nouns in the narrative usually become your first classes. These are called “domain objects”.
Don’t let domain objects be your only classes, though. Shift your focus toward messages and away from classes to discover new, less obvious objects that will implement key functionality. Not doing so will risk overloading your domain objects with more responsibility than what they can handle.
UML sequence diagrams are an excellent way to explore design alternatives. You can use them to draft objects and their interactions, i.e. the messages they send each other. They are quick, low cost, and allow easy communication of ideas between team members. Here’s an online tool that generates them based on plain text: sequencediagram.org.
The transition from class-based design to message-based design is one that yields more flexible applications. It means asking “I need to send this message, who should respond to it?” instead of “I know I need this class, what should it do?”.

Focusing on “what” instead of “how”

Public methods should focus on “what” instead of “how”. That is, they should express what the caller wants instead of how the callee must behave.
If you’re in a situation where an object calls many methods on another in succession, try to refactor so that all the logic is executed as a result of a single message. This consolidation reduces the size of the second object’s public interface, which reduces what callers need to know about it, which reduces dependency, which makes change easier. The caller only knows “what” it needs, not “how” to make the callee deliver.

When focusing on “what” instead of “how”, code like this:

class Trip
{
    IEnumerable<Bicycle> Bicycles { get; set; }
    Mechanic Mechanic { get; set; }

    public void Prepare()
    {
        foreach (var bike in Bicycles)
        {
            Mechanic.CleanBicyble(bike);
            Mechanic.PumpTires(bike);
            Mechanic.LubeChain(bike);
            Mechanic.CheckBrakes(bike);
        }
    }
}

class Mechanic
{
    public void CheckBrakes(Bicycle bike) { }
    public void CleanBicyble(Bicycle bike) { }
    public void LubeChain(Bicycle bike) { }
    public void PumpTires(Bicycle bike) { }
}

Can be transformed into code like this:

class Trip
{
    IEnumerable<Bicycle> Bicycles { get; set; }
    Mechanic Mechanic { get; set; }

    // This method was greatly simplified and its dependencies were reduced.
    public void Prepare()
    {
        foreach (var bike in Bicycles)
        {
            Mechanic.PrepareBicycle(bike);
        }
    }
}

class Mechanic
{
    public void PrepareBicycle(Bicycle bike)
    {
        CleanBicyble(bike);
        PumpTires(bike);
        LubeChain(bike);
        CheckBrakes(bike);
    }

    // Notice how Mechanic's public interface was reduced considerably.
    // All of these methods are private now.
    private void CheckBrakes(Bicycle bike) { }
    private void CleanBicyble(Bicycle bike) { }
    private void LubeChain(Bicycle bike) { }
    private void PumpTires(Bicycle bike) { }
}

Liberate objects from their context

An object’s context is made up of the things that an object knows about others. What other objects it calls and how. What it needs in order to be able to work.
A good design tries to allow objects to work with minimal context. “Objects that have a simple context are easy to use and easy to test; they expect few things from their surroundings. Objects that have a complicated context are hard to use and hard to test; they require complicated setup before they can do anything”.
In order to reduce context, we need to reduce the knowledge that callers have about the other components they call. A simple public interface helps with that. In practical terms that means fewer and less verbose methods, fewer parameters, fewer dependencies.
Objects need to trust their collaborators to do their part. Focusing on what the caller wants instead of what the callee does is a way of keeping context in check. This allows objects to collaborate with others regardless of what type they are and what they do.
Dependency injection is a mechanism through which objects can collaborate with others when they don’t know their type. A dependency injected via parameter (declared as an interface or duck type) hides its identity from its user.
Naming the methods of this injected dependency from the perspective of the caller reveals a generic interface that offers the features that the caller wants in the vocabulary that it understands. The caller does not need to know what the injected dependency does, only what it needs it to do.
Highly coupled objects with verbose public interfaces say to their collaborators: “I know what I want, and I know how you do it”.
More decoupled objects with concise public interfaces say to their collaborators: “I know what I want, and I know what you do”.
Highly decoupled objects with concise public interface and minimal required context say to their collaborators: “I know what I want, and I trust you to do your part”.

When objects trust their collaborators, and focus on what they want instead of what others do, the previous Trip/Mechanic example can become like this:

class Trip
{
    public IEnumerable<Bicycle> Bicycles { get; set; }
    Mechanic Mechanic { get; set; }

    public void Prepare()
    {
        // Trip now fully trusts Mechanic, and doesn't even know what it does.
        Mechanic.PrepareTrip(this);
    }
}

class Mechanic
{
    // The knowledge of "how a mechanic prepares a trip" now completely lives
    // within Mechanic.
    public void PrepareTrip(Trip trip)
    {
        foreach (var bike in trip.Bicycles)
        {
            PrepareBicycle(bike);
        }
    }

    private void PrepareBicycle(Bicycle bike)
    {
        CleanBicyble(bike);
        PumpTires(bike);
        LubeChain(bike);
        CheckBrakes(bike);
    }

    private void CheckBrakes(Bicycle bike) { }
    private void CleanBicyble(Bicycle bike) { }
    private void LubeChain(Bicycle bike) { }
    private void PumpTires(Bicycle bike) { }
}

Rules of thumb for writing code with good interfaces

Create explicit interfaces. Be intentional and obvious when defining public and private methods. Use your language access modifiers (i.e. public, private, protected, etc). Your public methods should:
1. Be explicitly identified as such
2. Be more about what than how
3. Have names that are unlikely to change
4. Use keyword arguments
Respect the public interface of others. Invoke only public methods on the classes that you use. Avoid circumventing their public interfaces and directly calling private members. Depending on the private interface of framework and library classes is like a ticking time bomb. The reason they are private is because they are expected to change or disappear entirely.
Minimize context. Focus on “what” instead of “how” when designing public interfaces. Favor public methods that allow callers to access your classes’ functionality without having to know how they do it. Use interface and duck types to name methods from the perspective of and with the vocabulary of the callers.
If you have to interact with a class that does not have a clean interface, and you don’t own it or otherwise can’t refactor it so that it does, isolate it. Use the same techniques for dependency isolation mentioned in Chapter 3. Wrap the invocation in a new class or method to contain it and give that wrapper a clean public interface.
Follow the law of Demeter. The law of Demeter states that you shouldn’t chain multiple method calls that navigate across many different types of objects. In other words “talk only to your close neighbors” or “use only one dot”.

Violations ot the law of Demeter look like this:
customer.GetBicycle().Wheel.Rotate();

Violations of the law of Demeter make for code that’s not TRUE. Changes in the object at the end of the chain ripple through the entire chain. This is unexpected and laborious, making the code neither Transparent nor Reasonable. The class that uses the chain depends on all the objects in the chain, making it non-Usable. These chains are easy to replicate and harm changeability, making the code not Exemplary.
Always evaluate the cost of violating the law of Demeter versus the cost of abiding by it. Method chains that ultimately read an attribute are generally less harmful. Also, method chains on really stable classes like those of your language library or framework have low impact.
Delegation can appear to be a solution to law of Demeter violations. Unfortunately, all it does is remove the evidence that it’s there.
In reality, a violation to the law of Demeter indicates that there’s an object missing or that the public interface of an object is lacking. Find the object, and its interface, by thinking more about messages and less about classes. Define the interface by thinking about what the object needs of its collaborators, and not about how it can get it by itself.

Chapter 5: Reducing Costs with Duck Typing

About duck types

Duck types are public interfaces that manifest themselves across classes. When multiple classes accept the same messages (i.e. define methods with the same signature), they are said to be of the same duck type.
“Duck type” is a dynamic language concept. Static languages offer equivalent features via “interfaces”.
The difference is that, in statically typed languages, interfaces need to be explicitly defined in code and concrete classes need to explicitly implement them. On the other hand, duck types in dynamically typed languages need only be acknowledged in an abstract sense. They don’t have to be explicitly defined in code.
Duck types and interfaces establish a contract. A set of messages that a given object is expected to be able to receive. A protocol that users of these objects can be confident that they will adhere to.
Just as a class can implement multiple interfaces, an object can be of many duck types. That’s because interfaces and duck types materialize depending on the situation in which objects that implement them are used. The same object can be used in different places for different things. Each user will see the object from their own perspective, and each can expect it conform to a different interface. See the interface segregation principle.
Moreover, code that depends on an interface or duck type is flexible enough to collaborate with any concrete class that implements that interface or duck type. See the Liskov substitution principle.
As we’ve established, it’s better to depend on abstractions than to depend on concretions. So, users of an object should not care what its type is, only what it does. Behavior over identity. In other words, they should focus on messages and public interfaces, instead of specific classes. Duck types and interfaces bring to the forefront what an object does and abstracts away what it is.
Applying duck types to a situation makes the code more abstract and less concrete. Concrete code is easier to understand but harder to extend. Abstract code is harder to understand at the beginning but easier to change. In the long run, abstract code tends to reduce maintenance costs.
When an object invokes methods on multiple objects in order to achieve a single purpose, that’s a situation where a duck type may be helpful. Think from the perspective of the caller and what it needs to come up with a message that makes sense for it to send its many collaborators. That method should be part of the public interface of each of its collaborators. To the caller’s eyes, each of its collaborators belong in the same duck type.

Writing code that leverages duck types

Whenever you see conditional logic that sends a different message depending on the concrete class of a given object, that’s the code telling you that it needs a duck type or interface.
Come up with a message that makes sense from the perspective of the caller and add it to each of the classes expected by your code in the conditional. These classes now share the same public interface, they are of the same duck type. Then remove the conditional and just call that method. Replace conditional logic with polymorphism.
In general, try to avoid code that explicitly mentions concrete classes or checks for the existence of particular methods in order to determine further behavior. Instead, come up with a shared interface and send those messages to the collaborators regardless of their actual type.

Higly conditional and concrete logic like this…

class Trip
{
    public IEnumerable<Bicycle> Bicycles { get; set; }
    public IEnumerable<Customer> Customers { get; set; }
    public Vehicle Vehicle { get; set; }

    public void Prepare(IEnumerable preparers)
    {
        foreach (var preparer in preparers)
        {
            if (preparer is Mechanic)
            {
                ((Mechanic)preparer).PrepareBicycles(Bicycles);
            }
            else if (preparer is TripCoordinator)
            {
                ((TripCoordinator)preparer).BuyFood(Customers);
            }
            else if (preparer is Driver)
            {
                ((Driver)preparer).GasUp(Vehicle);
                ((Driver)preparer).FillWaterTank(Vehicle);
            }
        }
    }
}

class Mechanic
{
    public void PrepareBicycles(IEnumerable<Bicycle> bicycles) { }
}

class TripCoordinator
{
    public void BuyFood(IEnumerable<Customer> customers) { }
}

class Driver
{
    public void FillWaterTank(Vehicle vehicle) { }
    public void GasUp(Vehicle vehicle) { }
}

…can be rewritten to be more abstract if we leverage polymorphism. Like this:

class Trip
{
    public IEnumerable<Bicycle> Bicycles { get; set; }
    public IEnumerable<Customer> Customers { get; set; }
    public Vehicle Vehicle { get; set; }

    // This method can be much simpler now, and abstract.
    // Ready to collaborate with any "TripPreparer".
    // The conditional logic has been replaced with polymorphism.
    public void Prepare(IEnumerable<ITripPreparer> preparers)
    {
        foreach (var preparer in preparers)
        {
            preparer.PrepareTrip(this);
        }
    }
}

// New interface that all preparers implement. It has a single method.
// It greatly simplifies the communication between Trip and its many
// possible preparers.
interface ITripPreparer
{
    void PrepareTrip(Trip trip);
}

class Mechanic : ITripPreparer
{
    public void PrepareTrip(Trip trip)
    {
        PrepareBicycles(trip.Bicycles);
    }

    private void PrepareBicycles(IEnumerable<Bicycle> bicycles) { }
}

class TripCoordinator : ITripPreparer
{
    public void PrepareTrip(Trip trip)
    {
        BuyFood(trip.Customers);
    }

    private void BuyFood(IEnumerable<Customer> customers) { }
}

class Driver : ITripPreparer
{
    public void PrepareTrip(Trip trip)
    {
        FillWaterTank(trip.Vehicle);
        GasUp(trip.Vehicle);
    }

    private void FillWaterTank(Vehicle vehicle) { }
    private void GasUp(Vehicle vehicle) { }
}

The one exception to this rule is when dealing with exceptionally stable classes. Like those in your language libraries, where the introduction of duck types would mean modifying core language libraries. Explicit type checks against these classes are usually low cost.
Duck types are abstract and less obvious in the code. That’s why they need to be well documented and tested.

Chapter 6: Acquiring Behavior through Inheritance

About inheritance

Inheritance establishes a relationship between two classes where it is said that a subclass B “is a” superclass A. Subclass and superclass are also referred to as derived class and base class, respectively.
Inheritance can be more technically defined as an “automatic message forwarding mechanism”. That is, when a subclass receives a message that it cannot respond to directly (i.e. a method that it does not implement), the language runtime takes care of automatically sending the message to the superclass.
That is, inheritance establishes a hierarchy where superclasses share their code with their subclasses.
Subclasses are meant to be specialized versions of their superclass. Subclasses should be “the same, plus more” when compared to their superclass. They should offer the same public interface.
Inheritance is ideal to solve the problem where you need a series of slightly different classes that share a lot of behavior.
A telltale sign that inheritance needs to be applied is when code contains an “if” statement checking an attribute that contains the “category” of the object, and based on that determines what code to execute. Watch out for this anti-pattern and for variables with names like “type”, “category”, “style” that control branches in behavior.
This is an anti-pattern that reveals that the object knows too much and needs to be broken down into smaller pieces. It increases the costs of change.

Here’s what the anti-pattern looks like:

class Bicycle
{
    string Style { get; set; }
    string TapeColor { get; set; }
    string FrontShock { get; set; }

    public Bicycle(Dictionary<string, string> options)
    {
        Style = options["style"];
        TapeColor  = options["tape_color"];
        FrontShock = options["front_shock"];
    }

    // Checking 'style' starts down a slippery slope.
    public Dictionary<string, string> GetSpares()
    {
        if (Style == "road")
        {
            return new() {
                ["chain"] = "11-speed",
                ["tire_size"] = "23",
                ["tape_color"] = TapeColor
            };
        }
        else if (Style == "mountain")
        {
            return new() {
                ["chain"] = "11-speed",
                ["tire_size"] = "2.1",
                ["front_shock"] = FrontShock
            };
        }
    }
}

Applying inheritance

Be on the lookout for existing classes that may lead you down the wrong path when it comes to inheritance. More often than not, classes that already exist in the code base are not good candidates for extension via inheritance. They are not good candidates to be superclasses.
The path forward is likely to be a new class to serve as the base, and then update your existing class to be a subclass. Then add other subclasses as peers of it.
For inheritance to work, the objects being modeled need to truly share a generalization-specialization relationship.
The superclass needs to define the common behavior that is shared among subclasses. The subclasses define the specializations.
In many cases, the superclass should be abstract, meaning that they are not supposed to be instantiated. They represent an incomplete object which only becomes whole when looked at in the context of each of its subclasses.
It almost never makes sense to create a superclass with only a single subclass. In fact, creating an inheritance hierarchy with two subclasses is often risky. You risk coming up with the wrong abstraction.
Three subclasses is the sweet spot to commit to inheritance. Put off the decision to implement inheritance until that point, if you can. That’s the point when there’s enough information available to confidently determine an abstraction that will be useful and cost effective.
If you put off implementing an inheritance hierarchy, then you won’t be able to share code between the highly related classes. That will likely lead to code repetition, which is also costly. Consider what costs more: having the repetition for the time being; or doing nothing and waiting for more information to avoid making the wrong decision.
“Every decision you make includes two costs: one to implement it and another to change it when you discover that you were wrong”.
When refactoring towards an inheritance hierarchy, consider that “promoting” code from the concrete subclass up into the abstract superclass is often easier and less error prone.
Errors in promoting are easy to identify and fix. All that can happen if you miss a promotion is that a subclass that was meant to inherit some behavior won’t have it.
Going in the opposite direction and missing a “demotion” will produce design errors that are harder to spot and have dire consequences if left alone.
That would mean that concrete behavior, which does not apply to all subclasses, stays incorrectly in the abstract base class. That’ll throw a wrench in the works and the whole inheritance hierarchy will be on shaky grounds. The abstract base class won’t be generic enough and subclasses will be tempted to circumvent it.
“The general rule for refactoring into a new inheritance hierarchy is to arrange code so that you can promote abstractions rather than demote concretions”.
The template method pattern provides a clean way of defining a common basic algorithm in the superclass and allowing subclasses to supply specializations for it. Superclasses can define an algorithm, and call certain methods in key points within it. These are extension points. These methods can then be implemented by subclasses, letting them control part of the overall logic.
When promoting concrete code to the more abstract superclass, consider using the template method when methods cannot be promoted wholesale, and have to be broken up instead. The parts that can be promoted become the template method; the parts that can’t be become the specialization methods that each concrete subclass implements.
To avoid future bugs, superclasses should provide default implementations for the specialization methods that it expects its subclasses to implement. These can be no-ops or, even better, raise errors. These errors let developers know that extending this inheritance hierarchy requires these methods to be implemented.
Hook methods are a slightly lighter iteration of the template method concept that helps alleviate superclass-subclass coupling. Same principle: let the superclass send a message that the subclass implements in order to provide a specialization.
A hook method need not be a part of a common abstract algorithm, hence the slight distinction from the full fledged template method design pattern.
Beware of subclasses explicitly invoking functionality on their superclasses. Languages often offer keywords such as “base” or “super” that allow easily sending messages to superclasses. These are dangerous because they couple subclasses with their superclasses. They reveal that subclasses know the general algorithm.
The template method pattern and hook methods invert this dependency, allowing the superclass to call the subclass. Allowing the subclass to provide specializations without knowing too much about the superclass.

Here’s an example of a properly factored inheritance hierarchy:

// The main purpose of the base class is to establish a basic functionality
// that can be easily extended by subclasses.
// This class cannot be instantiated, as given by the "abstract" modifier.
// This means that our application has no need of holding onto pure Bicycle
// instances, only the concrete subclasses are to be used.
// This abstract class is just a vehicle for sharing code between other classes.
abstract class Bicycle
{
    public string Size { get; set; }
    public string Chain { get; set; }
    public string TireSize { get; set; }

    // Notice how this constructor implements the template method pattern
    // in order to allow subclasses to make changes to the overall algorithm.
    // Specifically, it allows them to provide default values for the Chain and
    // TireSize attributes as well as run any additional logic after
    // initialization.
    public Bicycle(Dictionary<string, string> options)
    {
        Size = options.GetValueOrDefault("size");
        Chain  = options.GetValueOrDefault("chain") ?? GetDefaultChain();
        TireSize = options.GetValueOrDefault("tire_size") ?? GetDefaultTireSize();

        AfterInitialize(options);
    }

    // This method establishes the basic spares information and allows
    // subclasses to supply more by calling the GetLocalSpares hook method.
    public Dictionary<string, string> GetSpares()
    {
        var defaultSpares = new Dictionary<string, string> {
            ["tire_size"] = "23",
            ["chain"] = "11-speed"
        };

        return new(defaultSpares.Concat(GetLocalSpares()));
    }

    // This method, which is marked as abstract, has to be implemented by
    // subclasses. It allows subclasses to contribute specializations.
    protected abstract string GetDefaultTireSize();

    // These methods, marked as virtual, all have default implementations of
    // varying complexity but the important aspect is that they can be
    // overridden by subclasses. They also allow subclasses to contribute
    // specializations.
    protected virtual void AfterInitialize(Dictionary<string, string> options) { }
    protected virtual Dictionary<string, string> GetLocalSpares() => new();
    protected virtual string GetDefaultChain() => "11-speed";
}

// Notice how the following concrete subclasses are very simple.
// Extending this kind of code is easy, because creating new subclasses is easy.
// They leverage their abstract base class to implement most of their features.
// Their job is to supply specializations on the core logic defined in the base
// class. To do so, they override the hook methods defined by the base class.
class RoadBike : Bicycle
{
    public string TapeColor { get; set; }

    public RoadBike(Dictionary<string, string> options) : base(options) { }

    protected override void AfterInitialize(Dictionary<string, string> options) =>
        TapeColor = options.GetValueOrDefault("tape_color");

    protected override Dictionary<string, string> GetLocalSpares() =>
        new() { ["tape_color"] = TapeColor };

    protected override string GetDefaultTireSize() => "23";
}

class MountainBike : Bicycle
{
    public string FrontShock { get; set; }
    public string RearShock { get; set; }

    public MountainBike(Dictionary<string, string> options) : base(options) { }

    protected override void AfterInitialize(Dictionary<string, string> options)
    {
        FrontShock = options.GetValueOrDefault("front_shock");
        RearShock = options.GetValueOrDefault("rear_shock");
    }

    protected override Dictionary<string, string> GetLocalSpares() =>
        new() { ["front_shock"] = FrontShock };

    protected override string GetDefaultTireSize() => "2.1";
}

class RecumbentBike : Bicycle
{
    public string Flag { get; set; }

    public RecumbentBike(Dictionary<string, string> options) : base(options) { }

    protected override void AfterInitialize(Dictionary<string, string> options) =>
        Flag = options.GetValueOrDefault("flag");

    protected override Dictionary<string, string> GetLocalSpares() =>
        new() { ["flag"] = Flag };

    protected override string GetDefaultChain() => "10-speed";
    protected override string GetDefaultTireSize() => "28";
}

About roles

Using classical inheritance is always optional. Every situation that calls for inheritance could be solved using another technique that allows for sharing code between otherwise unrelated classes.
The concept of “roles” is an alternative to classical inheritance. This concept emerges from the need of multiple classes to be used in a particular context to do the same thing. So they themselves need to share behavior.
Think of a role as an augmented form of a duck type. A group of classes are said to play the same role when they belong in the same duck type or implement the same interface. In addition to exposing the same public interface, roles allow them to share internal behavior.
Roles are ideal for storing responsibilities that are orthogonal to classes. It allows classes that are otherwise unrelated to share behavior. They establish a “behaves like” type of relationship between objects, as opposed to the “is a” relationship that is established with inheritance.
When you include a module into an existing class, all the methods defined in the module become available to the class. This is the same thing that happens when a subclass inherits from a superclass.
Many languages offer native features to allow for this kind of relationship between objects. In Ruby, we use Modules. Other languages use Mixins, Traits, Default Methods. Simply put, all these are ways of bundling together a group of methods that can be easily plugged into existing classes.

Writing inheritable code

The same principles, techniques, and anti patterns that apply to the design of duck types and inheritance hierarchies also apply to roles.
Beware of objects that use a variable with a name like “type”, “category” or “kind” to determine what code to execute. The object is likely concealing (and acting like) two or more types. These would be subclasses or module includers.
When code is checking the type of an object to decide which message to send it, that’s a signal that there’s a missing abstraction. There’s a duck type or role in there that needs to be explicitly brought to light. A public interface needs to be defined for it. If there is a need to share behavior, put it in a module, mixin, etc.
“All of the code in the abstract superclass should apply to every class that inherits it”.
Beware of subclasses or module includers that override a superclass/module method just to raise a “not implemented” error. Chances are that if only part of the superclass applies to it, then it doesn’t belong in the hierarchy. Or maybe the whole hierarchy needs a redesign.
Subclasses agree to the contract specified by their superclasses. They must respond to every message in the superclass’s public interface, accept the same types of inputs and return the same types outputs.
Put in other words, “subtypes must be substitutable for their supertype”. That is, all subclasses should act like their superclass. This means that any code that expects a superclass should be able to work, without change, with all its subclasses as well. That’s the Liskov substitution principle.
Through the concept of variance, subclasses have a slightly higher degree of freedom. Subclasses can accept inputs of more abstract types and can return outputs of more concrete types than those specified in the superclass’s public interface while still being substitutable for their superclass.
For example, given a class “Object” that is a superclass of a class “String”; if a method in a superclass “MyClass” accepts a String parameter, a subclass of MyClass could override that same method and accept Object and still be compliant. Likewise, if a method in a superclass returns an Object result, the subclass could return a String for that same method, and still be compliant. Still be substitutable.
In other words, some code that sends a message with a String parameter can send the same message to a receiver that accepts Object. Likewise, some code that expects an Object as a result of sending a message can send the same message to a receiver that returns a String. Because a String “is an” Object, any code that works with an Object can work with a String, it’ll just treat it like an Object and only access Object’s public interface, which String fully supports.

This is what variance allows:
class Superclass
{
    public virtual Object DoSomething(String parameter) => new Object();
}

class Subclass : Superclass
{
    public override String DoSomething(Object parameter) => "the result";
}
Be aware that C# currently does not support variance in parameters, only in return types. So this code would not actually work as it is, since the parameter types are different.

The template method pattern is the ideal technique for creating superclasses that are easy to inherit from. It allows clean separation of an abstract algorithm from the concrete specializations.
Be on the lookout for code that directly invokes superclass behavior with keywords like “super” or “base” etc. Instead, use hook methods to allow subclasses to contribute to parts of the common algorithm.
Favor shallow hierarchies instead of deep ones. Narrow is also preferable to wide. Wide ones are easier to live with as long as they are shallow. But deep and wide ones are a maintenance nightmare. That is, keep the vertical levels of inheritance as low as you can. Hierarchies with many layers of superclasses are hard to understand and maintain.

Chapter 8: Combining Objects with Composition

About composition

Composition establishes a “has a” relationship between objects. It is a technique where you combine simple, independent objects to turn them into larger, more complex ones. Composition combines distinct parts so that the resulting entity is more than the sum of its parts.
In composition, the container object (or composed object) “has a” contained component (or part). Mechanically, this usually means that the container’s class has an attribute which holds a reference to the part object. It’s a dependency that’s usually injected via constructor parameter.
The container communicates with the part via its public interface. The part plays a role, and the container collaborates with it. This means that containers generally interact with their parts through interfaces or duck types.
Composition is generally a cheaper alternative to code sharing when compared to inheritance.
When refactoring from inheritance into composition, the parts sometimes need to share some code, in addition to sharing the same public interface. This is because all possible parts play the same role for the container. In these cases, an inheritance hierarchy of parts may be what’s needed. Or, they could share behavior via modules or mixins.
In technical terms, the technique of composition takes two forms: aggregation and, well, composition. Under composition, parts don’t have any use or value outside of their container objects. Under aggregation, on the other hand, parts can live on their own.

Factories

When creating certain objects becomes complex, encapsulate that complexity into a factory. That way the knowledge is stored in a single place in the code base. In principle, factories are simple: They are objects that create other objects.
An advantage of factories is that they make the process of turning complex data structures into objects easy. You can have a “specification”, stored as pure data in a file or database. The factory can then implement the logic that understands the meaning of the data structure and how to create living objects based on it.

Deciding between inheritance and composition

Inheritance and composition are fundamentally code arrangement techniques where logic is distributed among various objects. With inheritance, objects are organized into a strict hierarchical structure and get automatic message delegation. With composition, objects are independent from each other but messages need to be manually delegated.
In general, “favor composition over inheritance”. Composition is lower cost (i.e. fewer dependencies) than inheritance in most circumstances where both could be used to solve a problem.

Pros and cons of inheritance

When properly applied, inheritance is excellent at producing code that is Reasonable, Usable and Exemplary.
Inheritable code is Reasonable because small changes in code can produce great changes in behavior. This is because code near the top of the hierarchy is defined once but used by all subclasses. Changing such code allows to alter the behavior of many subclasses.
Inheritable code is Usable because superclasses are literally designed to be easy to reuse. Inheritance hierarchies adhere to the open-closed principle. It’s easy to extend the hierarchy by creating new subclasses without having to touch existing code, thus, reusing it.
Inheritable code is Exemplary because it intrinsically provides a clear path to extend it. Even novice programmers won’t have a hard time creating new subclasses when the superclasses implement template methods and hooks which guide any extension efforts.
One important disadvantage of inheritance is that the cost of mistakes is considerably high. Incorrect applications of inheritance are costly, whether it be that the wrong abstraction was created or that inheritance was just the wrong tool for the job altogether.
Another disadvantage is that, for other contexts outside of the hierarchy, the ways in which it’s possible to interact with the hierarchy are limited. If the behavior defined within the hierarchy needs to be used, a subclass needs to be created. Other use cases may be incapable of tolerating that dependency.
If you’re writing a framework or a library, this aspect becomes even more important. You can’t know all the scenarios in which your library will be used, and forcing users to create subclasses in order to reuse the logic defined in the hierarchy may be more than what they can afford.

Pros and cons of composition

“When using composition, the natural tendency is to create many small objects that contain straightforward responsibilities that are accessible through clearly defined interfaces”. “Composition results in applications built of simple, pluggable objects that are easy to extend and have a high tolerance for change”.
When properly applied, composition is excellent at producing code that is Transparent, Reasonable and Usable.
Composable code is Transparent because small objects are easy to understand and changes have obvious effects. These objects also don’t necessarily form part of hierarchies, which means they are not susceptible to changes in superclasses.
Composable code is Reasonable because its behavior is easy to extend by just implementing new objects that play the role of parts. All new parts need to do is implement the public interface that the container object already expects.
Composable code is Usable because it’s made up of small independent objects. They don’t have any structural dependencies preventing them from being reused in different contexts, completely unrelated to the container or even the part role.
One disadvantage of composition is the fact that it may be more difficult to understand how the whole application works. While the individual objects may be small and simple, how they come together to solve problems may not be.
Another disadvantage is that objects need to delegate messages to each other manually, as opposed to inheritance where this happens automatically. This means that there is the explicit dependency of the container object having to know which messages to call on its parts.
Composition excels at separating containers from parts and assembling such objects. However, it doesn’t have an answer for the scenario where it is necessary to handle a collection of types of parts that are very similar to each other. That’s where inheritance comes in.
Use inheritance for “is a” relationships. Use duck types, interfaces, and behavior sharing via modules or mixins for “behaves like a” relationships. Use composition for “has a” relationships.

Chapter 9: Designing Cost-Effective Tests

The benefits of testing

The goal of design is to write code that is easy to change. In order to do that, we need good object-oriented design skills, good refactoring skills and good testing skills.
Good object-oriented design skills are needed because badly designed code, by definition, is hard to change.
Good refactoring skills are needed because design needs to be evolving constantly. As new requirements and new information about the domain become available, the code needs to adapt.
Good testing skills are needed because high value, solid tests enable continuous refactoring without fear of breaking the code.
Just like design, the true purpose of testing is to reduce costs. Tests do help reduce bugs, provide documentation, and improve the design; but those are only the means through which tests achieve their ultimate goal of reducing the cost of change.
Tests help by finding bugs: Test help expose bugs early in the process. Bugs are easier to find and cheaper to correct the closer in time we are to their introduction. Testing at the same time a feature is being implemented catches bugs before they go out the door. Bugs caught early are not given the chance to cause problems or have code depend on them.
Tests help by supplying documentation: Tests represent the best and most reliable documentation of the design. When static documents and memories get outdated and/or disappear, tests remain. Write tests that tell the story of how the code works.
Tests help by deferring design decisions: As a code base evolves, there will be spots in the code that are less than stellar. Spots which we’re not confident enough in to commit to a particular design. So we postpone making a decision and hack together a solution that works today. However, we know that it will need to be refactored to handle the requirements of tomorrow. These spots represent a lack of knowledge. Knowledge that the design is waiting for in order to improve. While waiting for that knowledge to come, we put the hack behind a clear, stable public interface and write our tests against that. In doing so, we’ll be free to refactor into a good design when the missing information arrives. Knowing that the tests have our backs.
Tests help by supporting abstractions: “Good design naturally progresses toward small independent objects that rely on abstractions”. One disadvantage to this is that, while small independent objects are easy to understand by themselves, the behavior of the whole application, with all the little objects working together, becomes obscured. Tests solve that problem by highlighting the abstractions, their interfaces, how to work with them, and how they work together.
Tests help by exposing design flaws: If tests are hard to write, that’s the perfect indicator that the code under test is hard to reuse. Which indicates high coupling, which indicates increased costs of change. If testing an object requires a lot of setup and a lot of other objects, then it requires too much context and has too many dependencies.

What, when and how to test

“One simple way to get better value from tests is to write fewer of them. The safest way to accomplish this is to test everything just once and in the proper place”.
Tests should focus only on the incoming messages that are defined in each object’s public interface (“A” in the picture above). That is, the public methods. Public interfaces expose the services that a class offers its users. They are also stable, so depending on them is safe. Tests that cover public methods are resilient to refactorings and add value because they exercise objects in the same way that their users in application code do.
Never test private methods (“B” in the picture above). They are meant to be used by the object internally. They change often and other objects should not depend on them because they will have to change with them. The same applies to tests. Tests that exercise private code break on every refactoring.
An object’s test suite should never assert the return value of outgoing messages that it sends to other objects (“C” in the picture above). These messages are part of the public interface of the receiver object, so the receiver’s test suite should be the one testing it.
On the other hand, an object’s test suite should always assert that necessary outgoing messages are sent, and with the correct parameters (“C” in the picture above).
A “test of state” is a test that asserts the return value of a method.
A “test of behavior” is a test that asserts whether the object under test calls a particular method on another object, how many times, and with what parameters.
A “query” is a method call with no side effects that is only made to get some value back.
A “command” is a method call that has side effects that are important to the overall application.
All public methods should be covered by tests of state. A test suite should only cover the public methods of its own object under test.
Queries should not be tested for state or behavior.
Commands should be tested for behavior.
Write tests first when it makes sense to do so. Writing tests before implementing the feature establishes an intention for the design from the beginning. Since tests are reused, they make sure the code to be written will have the bare minimum of reusability.
Testing first is no silver bullet, though. Even though it helps steer the design in the right direction, by itself it won’t produce a well-designed application. Fundamental object-oriented design techniques still need to be applied.
Well-designed applications are easy to change. Well-designed tests may never have to change. This is because they depend on abstractions and public interfaces, which are stable.
Two major styles of testing exist: Behavior Driven Development and Test Driven Development.
“BDD takes an outside-in approach, creating objects at the boundary of an application and working its way inward, mocking as necessary to supply as-yet-unwritten objects”.
“TDD takes an inside-out approach, usually starting with tests of domain objects and then reusing these newly created domain objects in the tests of adjacent layers of code”.
Both styles can be followed to produce valuable tests.

Writing valuable tests

Incoming messages (i.e. public methods) need to be covered by tests of state. That is, the values that they return must be validated.
Be on the lookout for public methods that don’t receive any calls. These are good candidates for deletion. If no other object calls it, then it needs to be private.
Objects should be tested in isolation. When that’s not possible, there’s a problem with the design. It means that the object is too coupled to its collaborators and can’t be reused on its own. It needs to be refactored.
It’s important that objects are tested in isolation because a test suite should care only about the one object that its testing. The other objects will be tested by their own test suites. Changes in an object should affect its own test suite, not that of other objects.
When the object under test depends on another object in order to work properly, refactor into dependency injection and have the test provide the dependency that way.
When using dependency injection, don’t pass in a concrete, “real” object in your tests. Instead, inject a test double or mock into the object under test.
Dependency injection also hints at the emergence of a role, interface, or duck type. Instead of depending on a concrete class, refactor the object under test to depend on a role. The double or mock in the test suite will be one of the objects that play the collaborator role that the object under test expects.
Private methods and query method calls need not be tested at all.
Testing private methods is redundant because they are internal to the object under test and invoked by public methods, which already have tests.
Private methods are also unstable. Not meant to be depended on because they change often. That will make for brittle tests.
Private methods covered by tests may mislead others in thinking that the method is stable and, thus, dependable.
Query method call results are not relevant to the overall application, only to the object under test. They are hidden within it. Also, the receiver of the query already includes tests for it, given that it is part of its public interface. So, they should be ignored by the test suite.
Command method calls do need to be covered by tests in order to prove that they were called with the correct parameters. In other words, it is the responsibility of the object under test to send that message, so its test suite needs to prove it.
Mocks are the way to test that command messages get sent. They help with tests of behavior. Using mocks injected into the object under test, a test case can validate that the method is called properly.
Commands should be tested for behavior, not state. So, mocks should not be used for tests of state. That is, tests should not be concerned with asserting what they return. However, they can be configured to return some value, if the operation requires it.
In dynamically typed languages that don’t have a compiler to do type checks, it is important to test objects for conformance with the public interfaces of their roles, and for proper integration with their inheritance hierarchies.
“Tests should document the existence of roles, prove that each of their players behave correctly, and show that dependents interact with them appropriately”.
Pragmatically, this means writing test cases that assert whether objects implement the specific messages defined in their roles’ public interface and those that their superclasses expect from them.
Multiple subclasses will undoubtedly share many of the same tests. In those cases, try to encapsulate the test cases into reusable components that every subclass’s test suite can invoke. Just like in application code, it’s better to avoid repetition.
Testing abstract base classes can sometimes be challenging because they are not supposed to be instantiated. In some languages, it is downright impossible to instantiate classes marked as abstract. For these scenarios, a test-specific subclass can be created with minimal implementation that allows tests to exercise the base class.
To avoid test brittleness, test doubles that play certain roles and inherit from superclasses should also be tested for conformance.

Getting started with Java development using Visual Studio Code

2023-03-17T00:00:00+00:00

Photo by Garrett Skinner, 2022

Visual Studio Code is a free source-code editor available for Windows, macOS, and Linux. While it includes a lot of features out of the box, you will likely need to extend its functionality to suit your purpose for using it. There are many extensions available, each providing their own set of features and functions.

In this guide we will install the Extension Pack for Java, which is a bundle of several extensions. Installing this extension pack will add the following features to Visual Studio Code:

Java language support for parsing and highlighting our code
Java test runner for testing our code
Java debugger for debugging our code
Java project manager for managing resources related to our code
Maven support for building and packaging our code

Note: This guide assumes you have already installed a Java Development Kit. If you haven’t done that yet, OpenJDK is a great option.

Step 1: Install Visual Studio Code

If you haven’t yet, download Visual Studio Code and install it. If you need more help with this step, review the installation instructions linked on this page. Click the link that applies to your operating system to access the installation instructions.

Step 2: Install the extension pack

Open Visual Studio Code and click on the extensions icon in the left sidebar, which looks like this:

This will take you to the extensions marketplace. Type “Extension Pack for Java” into the search bar and press enter. The extension we are looking for is authored by Microsoft. Once you find the extension pack in this list, click the Install button.

Step 3: Use the extension pack

Once the installation is complete, a “Get Started with Java Development” screen is displayed. The first item on this screen is “Get your runtime ready” with an “Install JDK” button. We already have a JDK installed, so we can skip this. The remaining items on the screen are helpful to review, as they give a brief overview of the features we have just added to our editor.

To ensure our JDK and extension pack are working together correctly, click File > New file…. You will be prompted to choose a file type. Enter “Java” and select “Java Class”. You will be prompted for a file name. Enter HelloWorld.java. You will then be prompted to choose where to save the file. Create a new directory somewhere, and save the file inside of the newly created directory.

The file HelloWorld.java is now open in our editor. Add the following text to the file:

class HelloWorld {
    public static void main (String args[])
    {
        System.out.println("Hello, world!");
    }
}

Save the file, then click the play icon at the top right of the window. This is the Run button. By default, it will run your Java application normally. You can also click the dropdown next to the icon and select Debug to run your Java application with the debugger attached, which is very helpful when problem-solving.

When you click the Run button, assuming everything is set up properly, your code will be compiled and executed. A terminal window will be opened, where your program may take in user input and/or write output. In this terminal window, the text “Hello, world!” will be written by the program.

If you are seeing the expected behavior, then congratulations are in order! You have just finished setting up Visual Studio Code for Java development.

If you are not seeing the expected behavior, restart from the beginning and confirm that you have followed each step correctly.

Happy coding!

For more guides and a deeper dive on the content in this one, see VS Code’s Java docs.

Programming the Intel NDP in 1983

2023-03-12T00:00:00+00:00

The Beginning

I graduated from St. John’s College in Annapolis in 1980. It was an intensive four-year education in math, science, language, poetry, and philosophy. Two years later, I took four computer classes at a community college, and got my first IT job in 1983 at the beginning of the personal computer revolution.

There were two of us: Steve, the owner of the company, and I, working literally in his garage. I was just a fledgling, uncertain and doubtful of my own ability. The IBM PC had come out the summer before, a device IBM seemed to regard as little more than a toy. Steve was by profession a physicist.

Steve noticed the PC had an empty socket on the motherboard, next to the Intel 8088 CPU. He guessed it was for Intel’s 8087 Numeric Data Processor (NDP), also known as a math co-processor, that was designed as a companion to Intel’s 8088/86.

The CPU could operate perfectly well on its own, but if the NDP was installed, they would both read the same code stream. The CPU would ignore NDP instructions and let the NDP execute them. The NDP would ignore non-NDP instructions and let the CPU execute them. While the 8088/86 is running code, it can’t do anything else. With the NDP, true parallel processing is possible.

The Plan

IBM selected Microsoft to produce its systems software: operating system, compilers, linkers, libraries, macro assembler, etc. Microsoft not only ignored the 8087, it devised a floating-point format incompatible with it. Steve had signed a contract with Intel to be the exclusive distributor of the NDP for a period of years—Intel wasn’t selling any to speak of, so why not?

Steve’s business plan was to sell a package: an 8087 chip and modifications to IBM (Microsoft) systems software that builds programs (BASIC compiler, FORTRAN compiler, Pascal compiler, and Macro Assembler) so they can generate code for it, and libraries in which the math routines use the 8087 instead of complex, slow math routines in the CPU.

With the package we created, math-intensive applications would be sped up by anywhere from 2–3 times up to 40 times. The financial advantage to those who wrote their own applications in, say, BASIC, and used our package was compelling.

The Platform

The computer I was given was a standard IBM PC of early 1983: an Intel 8088 microprocessor running at 4.77 MHz, two 360 KB 5¼-inch full-height floppy disks, 640 KB of RAM, a monochrome monitor, five expansion slots, and PC-DOS 1.1.

The 8088’s 16-bit architecture internally was essentially the same as the 8086, with the difference that it used an 8-bit bus and was cheaper.

IBM designed an open architecture. In other words, it designed the PC to use all off-the-shelf equipment from other manufacturers, not IBM: chips from Intel, systems software from Microsoft, other chips from elsewhere. This was to be my saving grace, as we shall see below.

Two Problems, Two Solutions, one Package

Our guiding design idea was that our users could develop programs in the way they were accustomed to. They could, for example, use the BASIC Interpreter that came as part of the PC ROM BIOS for an iterative, live mode of development. When satisfied, they could save the resulting text file, build it with our patched version of the BASIC compiler and link it with our libraries. They would have bought an 8087 from us, plugged it into the coprocessor slot, and then could launch their executable.

One customer, for example, was in Kansas. They were commodity traders who had an algorithm written in BASIC. It would take data from all commodity trades during the day and at the end of the day, run the data through their algorithm, and use the result to place trades first thing the next morning. It took 5–6 hours to run, and they got home to their families at 10 or 11 o’clock at night.

With our software modifications and the 8087, it took only about 2 hours to run, so they were home in time for dinner.

Too Naive To Fail

When I first sat down at my PC to work, I didn’t know how to turn it on.

To do my job I had to program in Intel 16-bit assembly language. I not only had never seen any assembly language, I had never seen any computer language before besides FORTRAN and PL/I.

I was somewhat like the bumblebee. Mathematics has proven that the bumblebee is aerodynamically unable to fly. The bee, however, does not know math, so it flies anyway.

I sat at my desk and grappled with the complexity before me. At the end of the first week, I felt so hopeless and lost that I thought I would speak to Steve and say, “Steve, when we interviewed I misled you. I pretended I am not an idiot. Let’s say no more about it. I will slink away silently. You don’t even have to pay me.”

Then I thought, “Steve is a pretty smart man. I will leave it up to him. If he thinks I should leave, he can fire me. Meanwhile, I will stay here and give it everything I’ve got.”

My Tasks

The 8087 had a stack of eight ten-byte floating point registers. It could load and save either four-byte (single) or eight-byte (double) floating point. Steve had written a disassembler using the BASIC interpreter, so I had a printout of the code in the BASIC library. It did not show the source code, but it did show labels of routines and variables.

Steve had a list of the names of all the math routines in the library. I had to edit each math routine so that it would perform the same functionality as the original, but using the 8087.

A second requirement had to do with the floating-point format of constants in the source code. The Microsoft compilers generated them in MS-Binary floating point, in the structure exponent + sign bit + mantissa (or significand). The 8087 used IEEE floating-point, which was structured sign bit + exponent + mantissa. We patched the compilers so that at the place they would generate floating-point constants, they would generate IEEE-format constants instead.

My Advantages

IBM shipped each PC with two ring binders. One explained the BASIC language as implemented in the ROM BIOS. The other was a handbook for the PC, which told everything a person could possibly want to know about the machine, including a printout of the ROM BIOS assembly language source code at the end of the manual: page after page of working code in Intel 8088/86 assembly language.

I also had an Intel product manual on the 8088/86, including a section on the 8087. That was some of the most dense specifications on the planet, but it contained all the operations and registers, both for the CPU and the NDP.

Another advantage I had was the debugger, DEBUG.COM. I would edit a math routine to replace the operations in the CPU with a few simple opcodes for the NDP, since it did all its operations internally, allowing the programmer to use a few NDP instructions to carry out the functionality. I would then use the librarian to replace the given routine in the library with my routine, write a test program in BASIC and build it. Then I would load the executable in the debugger and step through the code. That had two big advantages: One, I got to see the assembly language in action, learning more than a book could ever teach me. Two, I verified that the code actually did what I thought it did. The code we shipped rarely had a bug in it and those were usually minor. Except one.

Whoops!

There was one major bug: error handling. I became aware of it when a customer called in about the failure of our code to catch a division by zero. Some research showed that the first 1024 bytes of RAM contained the Interrupt Vector Table (IVT), 256 four-byte entries containing the segment (two bytes) and offset (two bytes) of the Interrupt Service Routine (ISR). Interrupt INT 0 contained the address of the CPU divide by zero ISR.

Our software, since it did not use the CPU for division, did not catch a division by zero. This error would generate an IRQ (interrupt request) 13 which Intel had reserved for the math coprocessor. In Intel’s product spec, they marked IRQ 13 as Reserved. The IBM engineers, however, had always seen the word “Reserved” used for things reserved by IBM, so they used IRQ 13 for something else. So I had to write an ISR that would take over the IRQ 13 entry in the IVT, check to see if there was an error in the NDP, and if so, handle it. Otherwise, pass control onto the original ISR.

Eventually there was a DOS routine to create a chain of ISRs, each of which would determine if the interrupt was something it should handle or else pass it on to the next ISR in the chain. Good cyber-citizens would play nice, and use the DOS routine to get their place in the chain.

They Think They Didn’t Get the Right Test

The company expanded, leaving the garage far behind and moving into a brick-and-mortar building, one of many at a defunct rope-factory. It employed by this time 40–50 people.

Intel had marched forward, producing the 80286 (with its companion 80287) and the 80386 (with its companion 80387).

Steve asked me to write a test for the math chips. I wrote it in C and Intel 16-bit Assembler. The test performed the four standard arithmetic operations—add, subtract, multiply, divide—and also special operations the NDP was capable of, like exponents, logarithms, square roots, trigonometric functions, etc.

The code tested whether the answers calculated by the chip came out within a narrow range, since it is not possible to test for an exact result with floating-point numbers that can be “right” but differ in the least significant bits of the mantissa. I named it 87TEST.EXE. It shipped with every NDP we sold: 8087, 80287, 80387.

We began getting calls from customers who had bought an 80287 or 80387, complaining that they had received the wrong test. Someone would have to explain that the same executable with the same code tested all variants of the NDP. These calls were costing the company money to reassure the customers, so I wrote a new version of the test which determined which CPU and which NDP it was running on.

The OS was MS-DOS, a character-based system. I used the advanced ROM-BIOS character set to “draw” images representing both chips, each casting a “shadow” to seem three-dimensional, and each labeled with the number of the chip. For example, one “image” could be labeled 80286 and the other 80287.

In order to gain customer confidence these had to be accurate. Later models of these chips had instructions whereby you could ask the chip, “What are you?” and it would return its version number. Earlier variants did not have these instructions, but for the most part I found characteristics I could test for to determine the version.

With the 8088 and 8086, however, they both used exactly the same instruction set. I racked my brain to find a difference and could not. I turned to my dog-eared copy of the 8088/86 Intel product manual and pored through it. Finally, I found a difference: the 8088 had a four-byte prefetch queue and the 8086 had a six-byte one.

I solved the problem by writing self-modifying code, which was a huge no-no according to Intel. It used 6 bytes in the code segment, with a constant defined in the 6th byte. I cleared the prefetch queue by doing a jump to the lead instruction. Thus, the prefetch queue would contain four bytes and the sixth byte would still be in memory (8088) or all six bytes would be in the pre-fetch queue (8086). The first instruction moved a different constant into the 6th byte. In an 8088, the 6th byte in memory would be changed and in the 8086 the 6th byte would be modified in the prefetch queue and remain untouched in memory. Then I would test that 6th byte: changed = 8088 and unchanged = 8086.

At last my search to correctly identify the CPU and its companion NDP was over. The test program was complete.

Summary

I had to make do with the Intel product manual, the IBM PC manual, the IBM debugger, and tools my own company had created. Google did not exist, and for that matter, the Internet as we know it today did not exist. There was no plethora of instructional videos, as are widely available today.

Success required above all persistence and resourcefulness. I do not deny that native ability and education are important, but problem-solving in the IT world succeeds when we accept a challenge and strive to achieve what we never thought possible.

We adapt, we improvise, we overcome. Failure is not an option!

References

Fun with Rational Numbers

2022-02-11T00:00:00+00:00

So, you say you adored learning long division in elementary school? Well here we are revisiting that wonderful activity we have missed so much: fabulous magical moments like dividing 1.0 by 7. The especially tedious exercise of dragging over one more of the endless 0’s and seeing how many 7’s would fit into that woefully large remainder. Soon the delight mounts as you desperately hope the next digit will be the last and you discover that beautiful 1 as a remainder.

Undoubtedly you have wondered, like I have, what would happen if you divided 1.0 by 19? Would it repeat after only 8 digits? How many digits would it really take before it repeats?

That’s exactly how I stumbled into the wonderful world of rational numbers. There I was, wondering and wandering along the gloomy Belgian beach — the tide way out as far as you could see — in the winter wind, and I couldn’t stop thinking how long it would take for that mantissa to repeat. I had actually already tried 1.0 over 19. Finally, on that miserable day, I got the courage to divide 1 by 23.

Fortunately for me, I wasn’t alone, and I was able to enlist, coax, oblige, implore the help of baffled little Susan, my daughter who was 7 years old at the time. Her job would be to write out the digits in the sand as I discovered them, yelling them out to her, getting farther and farther away from her as the calculation got longer and longer. Finally, shrieking in the wind with delight as I hit 1 as the remainder, Susan seemed confused but amused.

Well, it was so much fun I decided to share my excitement and enthusiasm while at the same time bringing attention to these often overlooked treasure chests of digits.

After my satisfaction in coming up with the answer, the pleasure of which clearly eluded my daughter who was overjoyed that this exercise was over, I decided to write a computer program to discover how these numbers might turn out as the values became larger.

Numbers as music

As a composer, I thought it might be nice to portray the results in a sound representation. My first attempt was to use a fundamental sine wave and have each digit represent the number of overtones in the resulting sound. So the fundamental stayed constant, while the overtones quivered above not unlike Mongolian, Sardinian, or Tibetan Buddhist monk throat singing.

The digit 0 became the lonely fundamental, the digit 1 became the first harmonic an octave above, 2 was an octave and a fifth, and so on, until 9 the 10th harmonic. For timing I added varying amounts of delay between the sound events proportional to the digits calculated.

The first incarnation of my computer program was in 1999 in the programming language C, creating a sound wave file as an output. It created an interesting harmonic whistling kind of wind chime not unlike serrated plastic tubes that you swing around your head — the faster you swing, the higher the harmonics generated.

Later, I thought of applying the repeating nature of the calculated series of numbers to create a periodic waveform itself, instead of using sine waves. Seems like an obvious path to explore, doesn’t it? I still have to try that sound experiment some day.

Visualizing numbers

However, before continuing with sound, I did create a simple visual representation in JavaScript by making gradient-filled colored rectangles overlaid by arcs and line segments.

Let’s look at my program showing results for some interesting numbers.

For example, here is a screenshot of what the reciprocal of 7 (1 divided by 7) looks like in my little program:

1/7

1/13

1/523

A demo

Click here to try it out for yourself!

And make sure your sound is turned up a bit.

Two types of sounds are generated: one, by ‘subtractive’ synthesis, using filtered white noise, and the second, by just using a sampled plucked string sound. The first one, whistle-like, starts with white noise bandpass filtered with the center frequency set at the notes of a piano and the bandwidth or Q of each filter set at one semitone. This produces airy, whispery, wind-like flute sounds.

Repeating digits

By the way, it turns out that:

1/19 produces 18 digits before repeating
1/23 produces 22
1/29 ⇒ 28
1/263 ⇒ 262 repeating digits, etc.

Nice pattern, right? Seems like you can safely say that a reciprocal will never have a mantissa with more digits than the number itself.

The Web Audio API components allow you to construct ‘circuits’ like you would with wires on an analog synthesizer, plugging outputs of one component into another, by ‘connecting’ modules.
Olivia Jack’s work:
- PIXELSYNTH is a very nice version of this concept.
- Her main website has other very nice visual coding experiments.

On the Importance of Explicitly Converting Strings to Numbers

2022-01-11T00:00:00+00:00

Recently a valued colleague approached me with a JavaScript problem. This individual is new to programming and is working on a self-taught course.

The assignment was fairly simple: Take a list of space-delimited integers and find the maximum and minimum values. If you are an experienced developer you can probably already guess where this is going. His code:

function highAndLow(numbers) {
  const myArr = numbers.split(" ");
  let lowNum = myArr[0];
  let highNum = myArr[0];
  for (let i = 0; i < myArr.length; i++) {
    if (myArr[i] > highNum) {
      highNum = myArr[i];
    } else if (myArr[i] < lowNum) {
      lowNum = myArr[i];
    }
  }
  return highNum + ' ' + lowNum;
}

console.log(highAndLow("8 3 -5 42 -1 0 0 -9 4 7 4 -4"));

This produced the output:

"8 -1"

These are clearly not the maximum or minimum values.

After looking at it for a few moments I recognized a classic JavaScript pitfall: failure to explicitly convert stringy numbers to actual number types.

You see, JavaScript tries to be clever. JavaScript tries to get it right. JavaScript tries to say “the thing you are doing looks like something that you would do with numbers so I’m going to automatically convert these stringy numbers to number-numbers for you.”

The problem is that JavaScript is not clever; it is in fact very dumb about this. The further problem is when developers come to trust and rely on automatic conversion. Careers have been ruined that way.

In this case the naive programmer would say “Well, I’m comparing the things with a mathematical operator (< and >) so JavaScript should treat the values as numbers, right?” Wrong. JavaScript compares them alphabetically, not numerically. Except that even the “alphabetical” comparison kind of sucks but that’s another topic. JavaScript doesn’t even attempt to convert to numbers in this case.

Repeat after me:

Always explicitly convert stringy numbers to actual numbers even if the language claims to do it automatically.

I don’t care if it’s JavaScript, Perl, some fancy Python package, it doesn’t matter.

Do not trust automatic type conversion.

You will get it wrong. It will get it wrong. There will be tears.

Fixing this program is as simple as changing one line to explicitly convert the numbers from strings.

const myArr = numbers.split(" ").map(n => Number.parseInt(n, 10));

Number.parseInt(n, 10) is the “one true way” to turn a string-number into a number-number in JavaScript. Never omit the 10; it is technically optional but you will regret it if you omit it, trust me. If you are reading base 10 numbers, tell JavaScript so explicitly. Otherwise it will again try to be clever but be not-clever and probably screw up the conversion by guessing the wrong radix.

It’s good that the developer caught this error visually, also, because they did not include a unit test. Errors like this slip through the cracks all. the. time.

Even TypeScript would not catch this. This function is perfectly legal TypeScript. There’s nothing illegal about comparing strings with < or >. TypeScript could only catch this if the developer provided additional type information up front, for example:

Now that we’ve told the compiler “This is a string” and “This is a number”, now it can helpfully tell us “Hey, you’re trying to mix strings and numbers in a not-good way”.

So it all comes back to the mantra of “Always explicitly convert strings to numbers. Always.” And if you’re bothering to use TypeScript, go the extra step and actually tell it what the types are. Don’t make it guess; it might guess wrong. Explicit is better than implicit.

Some things never change.

Engineering Is Not Just About the Cool Stacks

2021-05-25T00:00:00+00:00

Photo by Di

As a developer, I love finding new shiny pieces of tech that can help me develop web applications better. To me, it is one of the best parts of the job: Pick up a new programming language, a new database, a new standard, etc., and build cool things with it. But recently, I had the chance to reflect on my experience working on a boring stack at one of my previous workplaces, and it was actually a lot better than I expected.

My journey

I used to work at a company where the teams are free to decide the tech stacks that best fit their requirements, so a wide range of modern tools were used there to solve the problems at hand. Within just 2 years of joining the company, I switched teams four times and touched everything from Java, Scala, Akka, Terraform, AWS ECS, Serverless, API Gateway, AWS Lambda, AWS DynamoDB, React, Gatsby, Vue, Nuxt, TypeScript, Flowtype, and many other tools that really helped advance my technical skills.

But in late 2018, my manager invited me to assist a “less popular” team in the company. The team handled a myriad of complex products that are quite old. The team’s tech stack was not the shiniest compared to the other teams in the company; the products are developed using Java, a language I generally do not prefer due to its verbosity. We just shipped JAR files to the platform’s marketplace, so there were no web services to maintain and hence it didn’t challenge or sharpen our operations skills. The frontend JavaScript code was largely built with jQuery as it came bundled with the platform we are building on top of.

As a developer who loves shiny tools, this team should have been a dreadful one to be in. But to my surprise, as I reflect upon it, it was actually the team that gave me the highest work satisfaction. It was satisfying because there were a lot of problems to solve, and most of the time the harder problems to solve are not the tech stack: the apps work, and customers are paying.

Instead of talking about moving from Java to Kotlin or from jQuery to React, our team spent the time brainstorming ideas on how to increase code quality, how to ensure discipline and follow good principles when crafting software, how to improve our developers’ workflow and processes, and how to best share knowledge and contexts that we had just acquired for future team members. Each problem solved gave me that dopamine boost that made me feel good.

Below are some of the things we did to solve these problems.

Problem #1: Code quality

The products the team were handling had been suffering from some quality degradations in recent years, and these ultimately led to more bugs and regressions, which then leads to an increase in customer complaints, and that leads to us working under pressure, which then leads to even more quality degradations, and the vicious cycle repeats.

When this started to happen, what did we do?

We slowed things down. Instead of having individual engineers work on multiple tickets in parallel, we held more pair programming sessions and reduced the number of tickets in progress to help incorporate collaboration earlier and gain feedbacks on the implementation strategy sooner. At the same time, pairing sessions also helped get the newer engineers up to speed faster. We introduced more quality gates for pull requests to be able to be merged into main. We agreed as a team to scrutinize them harder, e.g. if you do not have tests among the changes, just consider the pull request rejected. We also required all team members to approve the changes. We worked with product managers to slow down shipping new features and instead having more bugs fixed. Hence, the health of the codebases got better and better and the products got more and more stable each day.

We also held a brown-bag meeting reinforcing SOLID design principles to everyone in the team. Knowledge sharing like this helps bring everyone onto the same page while pairing and reduce the long discussion threads on pull requests.

Members of a team come and go, and that is true for all teams. Complex apps like the ones we were maintaining are usually full of legacy decisions that the newer team members will not have the knowledge of if no deliberate efforts were made to ensure that those contexts get shared. So, we took steps to mitigate this by including the contexts in commit messages. We also encouraged writing our findings in the Jira tickets and in the pull request descriptions on GitHub, but we preferred Git commit messages for documenting decisions related to the commit, because Git is the system that is least likely to change. We might switch from Jira or from GitHub to some similar services, so contexts would be lost when that happens, but it is very unlikely that we would switch from Git for version control.

We also started a knowledgebase to contain information that should be shared within the team, including onboarding instructions, development gotchas, legacy contexts, etc. This helps distribute the knowledge to everyone and reduce blockers on specific team members. We also made a habit of documenting a discussion that happens offline, often in our internal communication channels or in our internal knowledgebase.

Problem #3: Developers’ workflow

Improving our process was also a focus. One of the first things we did was rewrite our Jenkins CI/CD pipeline to adopt multibranch pipeline, allowing us to build and test feature branches before they were merged into the main branch.

We also automated several tasks that we identified as repetitive, from as small as notifying everyone that there is a new pull request open on GitHub, to automatically running compatibility checks for all of our products once the platform we were building on released a new beta version. We constantly looked for areas we could automate better and discussed them in our weekly retrospectives.

We also tuned our workflow to better organize our capacity around all the demands that we are getting: from our customers, our roadmaps, and our improvements bucket. We adopted Classes of Service (CoS) to achieve that. CoS is an approach in which you decide different treatments to different types of work. This means the capacity we have is flexible depending on the demand of each type of work.

For example, we introduced a collapsible task buffer called Intangible that holds work whose value customers do not experience directly, such as upgrading Node.js version, etc. It is collapsible in the sense that the developers in this CoS can be reassigned to other CoS should we deem more capacity is required there. Note that CoS (or any other system for that matter) is not without its pros and cons, so be sure to tweak it to fit your circumstances should you consider adopting it.

Not all problems are tech, but all tech has problems.

Sometimes, we identify ourselves through our favorite tech stack.

Hey, I’m a JavaScript developer.

Hey, I’m a Ruby developer.

And that is not wrong per se, but this particular experience of mine serves as a reminder that the essence of software engineering goes deeper than just the tech — it is to solve problems through software. And that is what we do at End Point.

Reference

SOLID Design Principles at Devopedia
Classes of Service:
- On Classes of Service (CoS), a more pragmatic approach towards “squad formation” by Yuen-Chi Lian at ServiceRocket
- Predictability and Classes of Service by Leonardo Campos at ThoughtWorks

Demonstrating the HotSwap JVM

2020-11-25T00:00:00+00:00

Photo by Neil and Zulma Scott

For a recent Java development project I spent a while setting up an environment to take advantage of the HotSwap JVM, a Java virtual machine that automatically reloads classes when they change. This feature can potentially eliminate the need to redeploy each time code changes, reducing development cycle time considerably. While setting up the environment, I found I wanted a simple example of hot swapping available for my own experimentation, and I thought I’d share that example here.

First, let’s create a simple Java program. It needs to be slightly more complex than the ubiquitous “Hello, World!” application, because we need it to keep running for a while; if it just prints some message and exits immediately, we won’t have time to compile new code and see the hot swap feature in action. Here’s an example that uses a simple infinite loop, wherein it sleeps for one second, prints a message, and then repeats.

import java.lang.Thread;

public class HotSwapTest {
  public static void main(String[] args) {
    while (true) {
      try {
        System.out.println("Hi");
        Thread.sleep(1000);
      } catch (InterruptedException e) {
        // Ignore this
      }
    }
  }
}
~

If I build this into build/classes/java/main and run it, as expected it prints out “Hi” every second:

josh@igtre:~/hotswaptest$ java -cp build/classes/java/main/ HotSwapTest
Hi
Hi
Hi
...

The usual JVM doesn’t include the HotSwap feature. For my purposes I downloaded DCEVM, an alternative JVM which includes HotSwap. It’s also possible to patch some existing JVMs to add HotSwap, if you’d prefer. When I run the same code with DCEVM, it runs the code just like it did with the normal JVM, with additional debugging output:

Starting HotswapAgent '/home/josh/hotswaptest/dcevm/lib/hotswap/hotswap-agent.jar'
HOTSWAP AGENT: 15:01:23.423 INFO (org.hotswap.agent.HotswapAgent) - Loading Hotswap agent {1.4.1} - unlimited runtime class redefinition.
HOTSWAP AGENT: 15:01:24.189 INFO (org.hotswap.agent.config.PluginRegistry) - Discovered plugins: [JdkPlugin, Hotswapper, WatchResources, ClassInitPlugin, AnonymousClassPatch, Hibernate, Hibernate3JPA, Hibernate3, Spring, Jersey1, Jersey2, Jetty, Tomcat, ZK, Logback, Log4j2, MyFaces, Mojarra, Omnifaces, ELResolver, WildFlyELResolver, OsgiEquinox, Owb, Proxy, WebObjects, Weld, JBossModules, ResteasyRegistry, Deltaspike, GlassFish, Vaadin, Wicket, CxfJAXRS, FreeMarker, Undertow, MyBatis]

To make hot swapping work automatically, we need to provide the JVM with a properties file in the JVM’s classpath. Mine looks like this, and lives in build/classes/java/main, next to the compiled class files:

autoHotswap=true
LOGGER=debug

These properties are pretty self-explanatory: they tell the JVM to hot swap automatically when it finds new code, and turn up logging to DEBUG level.

So, with that all set up, let’s run the program again, change the code and rebuild it, and see what happens. For this test, I’ll just edit the message printed in each loop from “Hi” to “Hello”.

josh@igtre:~/hotswaptest$ ./dcevm/bin/java -cp build/classes/java/main/ HotSwapTest
Starting HotswapAgent '/home/josh/hotswaptest/dcevm/lib/hotswap/hotswap-agent.jar'
HOTSWAP AGENT: 15:36:22.132 INFO (org.hotswap.agent.HotswapAgent) - Loading Hotswap agent {1.4.1} - unlimited runtime class redefinition.
HOTSWAP AGENT: 15:36:22.543 DEBUG (org.hotswap.agent.annotation.handler.OnClassLoadedHandler) - Init for method public static void org.hotswap.agent.plugin.jdk.JdkPlugin.flushIntrospectClassInfoCache(java.lang.ClassLoader,org.hotswap.agent.javassist.CtClass)
HOTSWAP AGENT: 15:36:22.546 DEBUG (org.hotswap.agent.util.HotswapTransformer) - Registering transformer for class regexp '.*'.
HOTSWAP AGENT: 15:36:22.549 DEBUG (org.hotswap.agent.annotation.handler.OnClassLoadedHandler) - Init for method public static void org.hotswap.agent.plugin.jdk.JdkPlugin.flushObjectStreamCaches(java.lang.ClassLoader,org.hotswap.agent.javassist.CtClass)
HOTSWAP AGENT: 15:36:22.550 DEBUG (org.hotswap.agent.util.HotswapTransformer) - Registering transformer for class regexp '.*'.
...

The flurry of DEBUG messages tells me that it must have read my properties file correctly, and when I change the code and rebuild, I see the JVM respond with still more debug messages, saying it found and reloaded my code:

HOTSWAP AGENT: 15:38:27.066 DEBUG (org.hotswap.agent.watch.nio.WatcherNIO2) - Watch event 'ENTRY_DELETE' on '/home/josh/hotswaptest/build/classes/java/main/HotSwapTest.class' --> HotSwapTest.class
HOTSWAP AGENT: 15:38:27.118 DEBUG (org.hotswap.agent.watch.nio.WatcherNIO2) - Watch event 'ENTRY_CREATE' on '/home/josh/hotswaptest/build/classes/java/main/HotSwapTest.class' --> HotSwapTest.class
HOTSWAP AGENT: 15:38:27.119 DEBUG (org.hotswap.agent.watch.nio.WatcherNIO2) - Watch event 'ENTRY_MODIFY' on '/home/josh/hotswaptest/build/classes/java/main/HotSwapTest.class' --> HotSwapTest.class
HOTSWAP AGENT: 15:38:27.280 DEBUG (org.hotswap.agent.annotation.handler.WatchEventCommand) - Executing resource changed method watchReload on class org.hotswap.agent.plugin.hotswapper.HotswapperPlugin for event WatchFileEvent on path /home/josh/hotswaptest/build/classes/java/main/HotSwapTest.class for event ENTRY_MODIFY
...
HOTSWAP AGENT: 15:38:27.439 DEBUG (org.hotswap.agent.plugin.jdk.JdkPlugin) - Flushing HotSwapTest from introspector
Hi
HOTSWAP AGENT: 15:38:27.482 DEBUG (org.hotswap.agent.config.PluginManager) - ... reloaded classes [HotSwapTest] (autoHotswap)
Hi
Hi
Hi

But although it says it swapped in the new code successfully, it’s still printing “Hi”, not “Hello”. Why?

It turns out this ability to hot swap new code isn’t unlimited, and one important limitation is that methods you’re already running aren’t reloaded. Since the main() method was running, it didn’t get swapped out. What if I change the code so that instead of printing a string every second, it calls a method, and that method prints the string? Here’s some code to test that technique.

import java.lang.Thread;

public class HotSwapTest {
  public static void printMsg() {
    System.out.println("Here is printMsg");
  }

  public static void main(String[] args) {
    while (true) {
      try {
        HotSwapTest.printMsg();
        Thread.sleep(1000);
      } catch (InterruptedException e) {
        // Ignore this
      }
    }
  }
}

Now, I start the JVM over again, and as expected, it prints “Here is printMsg” once every second. When I change to “Here is printMsg v2.0” and rebuild, this happens:

Here is printMsg
Here is printMsg
HOTSWAP AGENT: 15:48:34.923 DEBUG (org.hotswap.agent.watch.nio.WatcherNIO2) - Watch event 'ENTRY_DELETE' on '/home/josh/hotswaptest/build/classes/java/main/HotSwapTest.class' --> HotSwapTest.class
HOTSWAP AGENT: 15:48:35.007 DEBUG (org.hotswap.agent.watch.nio.WatcherNIO2) - Watch event 'ENTRY_CREATE' on '/home/josh/hotswaptest/build/classes/java/main/HotSwapTest.class' --> HotSwapTest.class
HOTSWAP AGENT: 15:48:35.008 DEBUG (org.hotswap.agent.watch.nio.WatcherNIO2) - Watch event 'ENTRY_MODIFY' on '/home/josh/hotswaptest/build/classes/java/main/HotSwapTest.class' --> HotSwapTest.class
HOTSWAP AGENT: 15:48:35.174 DEBUG (org.hotswap.agent.annotation.handler.WatchEventCommand) - Executing resource changed method watchReload on class org.hotswap.agent.plugin.hotswapper.HotswapperPlugin for event WatchFileEvent on path /home/josh/hotswaptest/build/classes/java/main/HotSwapTest.class for event ENTRY_CREATE
HOTSWAP AGENT: 15:48:35.188 DEBUG (org.hotswap.agent.plugin.hotswapper.HotswapperPlugin) - Class HotSwapTest will be reloaded from URL file:/home/josh/hotswaptest/build/classes/java/main/HotSwapTest.class
HOTSWAP AGENT: 15:48:35.175 DEBUG (org.hotswap.agent.annotation.handler.WatchEventCommand) - Executing resource changed method watchReload on class org.hotswap.agent.plugin.hotswapper.HotswapperPlugin for event WatchFileEvent on path /home/josh/hotswaptest/build/classes/java/main/HotSwapTest.class for event ENTRY_MODIFY
HOTSWAP AGENT: 15:48:35.191 DEBUG (org.hotswap.agent.plugin.hotswapper.HotswapperPlugin) - Class HotSwapTest will be reloaded from URL file:/home/josh/hotswaptest/build/classes/java/main/HotSwapTest.class
HOTSWAP AGENT: 15:48:35.352 DEBUG (org.hotswap.agent.command.impl.SchedulerImpl) - Executing pluginManager.hotswap([class HotSwapTest])
HOTSWAP AGENT: 15:48:35.355 RELOAD (org.hotswap.agent.config.PluginManager) - Reloading classes [HotSwapTest] (autoHotswap)
HOTSWAP AGENT: 15:48:35.370 DEBUG (org.hotswap.agent.plugin.jdk.JdkPlugin) - Flushing HotSwapTest from com.sun.beans.introspect.ClassInfo cache
HOTSWAP AGENT: 15:48:35.375 DEBUG (org.hotswap.agent.plugin.jdk.JdkPlugin) - Flushing HotSwapTest from ObjectStreamClass caches
HOTSWAP AGENT: 15:48:35.376 DEBUG (org.hotswap.agent.plugin.jdk.JdkPlugin) - Flushing HotSwapTest from introspector
HOTSWAP AGENT: 15:48:35.415 DEBUG (org.hotswap.agent.config.PluginManager) - ... reloaded classes [HotSwapTest] (autoHotswap)
Here is printMsg v2.0
Here is printMsg v2.0

As you can see, it swapped in the new code correctly, and now prints the new message. HotSwap was a success!

I imagine there are very few production environments where this feature would be applicable, and even in development, getting this to work properly for something like a JEE app deployed to some application container isn’t necessarily a simple task. But if it can cut down on redeployment cycles, it can certainly be a valuable developer tool.

The Pragmatic Programmer book, 20th anniversary edition

2020-10-16T00:00:00+00:00

The Pragmatic Programmer is a now-classic book about software development, first published in 1999. It is old enough that it predates the famous Agile Manifesto of 2001. The authors of the book, Andy and Dave, were two of the 14 authors of that manifesto.

For its 20th anniversary in 2019, Dave and Andy created a new edition. They updated things that had become dated, such as mentions of programming languages, tools, operating systems, websites, etc. That was, I imagine, the easy part. They went on to extensively revise the entire text and incorporate new lessons learned in the past two decades.

A classic

This book is part of our company’s “work philosophy canon” that I ask every software developer at End Point to read, so for that reason and others I wanted to be familiar with the new edition and make sure it is still something I want to recommend so broadly.

The book is also required reading for university courses at Cornell (CS 3110: Data Structures and Functional Programming) and the University of Washington (CSE 331: Software Design and Implementation), probably among others!

I first read this book 19 years ago and really enjoyed it. I found that it clearly expressed many useful concepts I had stumbled upon in my own programming experience. More importantly, it gave me other helpful guidance and warned of pitfalls. Having that all collected in book form with concise, memorable chapter names and tips gave me more shared vocabulary to use with others in our work.

I also enjoyed reading the new 20th anniversary edition. It did not have the same personal impact, which is to be expected since I am at a very different point in my programming career, technical reading, and life experience. But I think the new edition has as great or even increased power for new readers.

Overview of changes

The authors estimate that around 75% of the text has changed, and that seems about right. A few areas that stood out to me were:

The now widely-known adage “don’t repeat yourself” originally meant primarily not to duplicate knowledge, that each piece of data should appear in only one canonical location. Over time, “don’t repeat yourself” has been more often used to preach against copying and pasting code. While that is also a good general rule, it has been oversimplified and turned into a blanket prohibition. Dave & Andy mention cases where duplicating code is the right thing to do, because the problem space is not the same and factoring out all such code duplication will make later changes far more complicated as the code has to behave differently for different callers.
Computer security was not nearly as big of a concern when they were writing the first edition in the late 1990s, and now that almost every computer is networked and attackers are motivated to reach every system from anywhere in the world, they gave the topic more attention.
Concurrency gets much more discussion, as is sensible in a time where scaling up typically means using many more CPU cores and processors.
Unit testing was far less common in the late ’90s, and there were few good test frameworks, so they recommended you write your own. Now there are many good test frameworks for every language, so they of course recommend you choose one of those and use it.
Dave writes “A Confession” on page 223: He tells us that writing tests for 30+ years made testing so much a part of the way he thinks and programs, that he now writes testable interfaces even when he doesn’t write tests. He isn’t saying that testing is no longer important, but rather that it’s not a religion, that it can and should be examined by experienced programmers, and it has important effects on the programmer aside from the tests themselves.
The old edition had checklists scattered throughout, and collected in the pull-out card at the end. The new edition doesn’t have checklists anymore, except the Debugging Checklist on page 97. I think that’s ok. They weren’t something I ever referred back to.

Tips

The old edition had 70 tips that appear throughout the chapters. The new edition has 100 tips. In both cases they are extracted and collected on a pull-out card at the end of the book. What changed?

Removed or heavily reworked tips

Write Code That Writes Code
Use Exceptions for Exceptional Problems
Design Using Services
Separate Views from Models
Don’t Use Wizard Code You Don’t Understand
Abstractions Live Longer than Details
Costly Tools Don’t Produce Better Designs

Tips that changed slightly

“Use a Single Editor Well” is now “Achieve Editor Fluency”
“Don’t Panic When Debugging” is now simply “Don’t Panic”
“Minimize Coupling Between Modules” is now “Decoupled Code Is Easier to Change”
“Configure, Don’t Integrate” is now “Parameterize Your App Using External Configuration”
“Don’t Gather Requirements—Dig for Them” is now “Requirements Are Learned in a Feedback Loop”
“Organize Teams Around Functionality” is now “Organize Fully Functional Teams”
“Gently Exceed Your Users’ Expectations” is now “Delight Users, Don’t Just Deliver Code”
“Put Abstractions in Code, Details in Metadata” is now “Policy Is Metadata”
“Always Design for Concurrency” is now “Random Failures Are Often Concurrency Issues”
“Listen to Nagging Doubts—Start When You’re Ready” is now “Listen to Your Inner Lizard”
“Don’t Be a Slave to Formal Methods” is now “Do What Works, Not What’s Fashionable”
“Some Things Are Better Done than Described” transformed to “Agile Is Not a Noun; Agile Is How You Do Things”

New tips

You Have Agency
Good Design Is Easier to Change Than Bad Design
Forgo Following Fads
Failing Test Before Fixing Code
Read the Damn Error Message
Act Locally
Take Small Steps—Always
Avoid Fortune-Telling
Tell, Don’t Ask
Don’t Chain Method Calls
Avoid Global Data
If It’s Important Enough To Be Global, Wrap It in an API
Programming Is About Code, But Programs Are About Data
Don’t Hoard State; Pass It Around
Don’t Pay Inheritance Tax
Prefer Interfaces to Express Polymorphism
Delegate to Services: Has-A Trumps Is-A
Use Mixins to Share Functionality
Shared State Is Incorrect State
Use Actors For Concurrency Without Shared State
Testing Is Not About Finding Bugs
A Test Is the First User of Your Code
Build End-To-End, Not Top-Down or Bottom Up
Use Property-Based Tests to Validate Your Assumptions
Keep It Simple and Minimize Attack Surfaces
Apply Security Patches Quickly
Name Well; Rename When Needed
No One Knows Exactly What They Want
Programmers Help People Understand What They Want
Don’t Go into the Code Alone
Maintain Small Stable Teams
Schedule It to Make It Happen
Deliver When Users Need It
Use Version Control to Drive Builds, Tests, and Releases
First, Do No Harm
Don’t Enable Scumbags
It’s Your Life. Share it. Celebrate it. Build it. And have fun!

The new edition’s PDF version does not include the pull-out card with tips that comes with the printed book. That’s too bad, because the tips were what I most often referred to in the old edition after I finished reading it. At least Andy and Dave have published on their website a list of the 100 updated tips. It would be even nicer to have a PDF of the book’s pull-out card as they did for the old edition.

Upgrade?

As you can see, there are many new tips. The text has been updated. It is still of excellent quality. So should you get the new edition?

I mentioned earlier two university courses that use the book. In both courses the professors allow students to use either the original 1999 version or the 2019 version. That seems like the right thing to do to me as well.

If you already have the old edition of the book, I don’t think it’s necessary to rush out and buy the new edition. The old one is still quite good and relevant, and you’ll be learning a bit of software development history by reading the older one.

If you do not yet have the book, or if you will be going over it with others who are new to it, it is probably best for you to get the new edition as well so you can be familiar with the new tips and the ways Dave & Andy describe things now.

Summing up

I am glad that Dave and Andy updated their book. Their updates make sense and make the book more relevant to present-day readers.

Relatedly, it would be good to also see an update of the also excellent book Practices of an Agile Developer by Venkat Subramaniam and Andy. That is coming up on 15 years old itself. Perhaps some of its topics have been incorporated in the new edition of The Pragmatic Programmer, but as I review the table of contents, it looks like it is still mostly additive. I have cited its important maxim “Different Makes a Difference” so often that I thought it was from The Pragmatic Programmer instead!

I’ll close with this good observation and advice from Andy in their Changelog podcast interview:

Was it Dijkstra who had the Turing Award lecture about the very humble programmer? That is a critical piece of early literature… And you talk about things that haven’t changed. This was 1972? … And he makes the very important point that complexity will overwhelm us if we don’t take a very humble, very measured approach. And it’s been 30–40 years and everyone—present company included—has ignored this wonderful advice.

Humility is difficult in our environment, in our culture, and it is probably, of all the human factorsy things that you need to be a good developer, I would submit that being humble, realizing you don’t know all the answers, that you need to find out, that you need to experiment, get feedback, try it.

This part of our headlong rush into the shiny new thing is this kind of faith that “Well, that’s gonna be better. I can do it better. I’m better than this/that.” Yeah, maybe… But you should validate that. You should try it. You should go back and read these things. You should try these other experiments.

Thanks, Andy & Dave!

Reference

All tips from each edition

Interviews

Functional Geekery Episode 126 – Andy Hunt and Dave Thomas from July 2, 2019 by Steven Proctor
The Changelog – Episode #352: The Pragmatic Programmers with Andy Hunt & Dave Thomas by Adam Stacoviak & Jerod Santo
CodeNewbie Podcast season 9 episode 8 from August 26, 2019: Why you should read the new edition of the Pragmatic Programmer with Andy Hunt, Dave Thomas by Saron Yitbarek (author of the new edition’s foreword)

Other comparisons of the two editions

Sections added, changed significantly, and removed by Kevin Hooke
The Pragmatic Programmer Book 2nd Edition Differences by Miranda Limonczenko (very detailed, but covers only the first 3 chapters)

Book pages

The Pragmatic Programmer, 20th Anniversary Edition book home page, errata, source code downloads, and links to buy
Practices of an Agile Developer book home page, samples, pull-out card, and review by Ethan Rowe

Cooking with CAS

2020-03-10T00:00:00+00:00

Photo by Flickr user reidrac, licensed under CC BY-SA 2.0

One of our customers asked us to host a new suite of web-based applications for them and to protect them with a single sign-on (SSO) solution. Ok, easy enough; these applications were in fact designed with a particular SSO system in mind already, but our situation required a different one, and we eventually chose Apereo’s open source Central Authentication Server project, or CAS. I’d like to describe the conversion process we went through.

The ingredients

Our customer’s application suite included:

The principal Java application using JAAS authentication
Another Java application based on Spring Security
A pair of PHP applications
A few automated tasks that needed to authenticate.

The original SSO system sets a header on each request, identifying an authenticated user. This requires a gateway system to sanitize request headers to ensure malicious users cannot forge a header themselves. It also requires each application inspect request headers and respond appropriately.

CAS is a bit more complex: applications redirect unauthenticated requests to a CAS server, which authenticates the user through any of various configurable methods. The CAS server then redirects the user back to the original application with a parameter called a “Service Ticket”, a seemingly random number identifying an individual authentication request. The original application contacts the CAS server directly to validate the service ticket and to collect information to identify the user. It can then establish a session for that user, and proceed normally.

To CAS-enable an application, we incorporate one of the CAS client libraries, which exist for various languages. In fact we won’t use the Java client directly, but rather we’ll incorporate components that extend it. When evaluating CAS, I was a bit concerned by what appeared to be a surprisingly limited selection of actively supported client libraries, and of course your results may vary, but we found software to meet our own needs without too much difficulty.

Configuring Wildfly Authentication

The most important application in the suite depends on the JAAS-based security subsystem of the Wildfly application server it’s deployed to. Originally it used a custom LoginModule that inspected request headers to find the ID of the authenticated user. Our first task was to configure our proxy server to remove this header from every request. We planned to disable the old authentication system entirely, of course, but this change ensured that even if we mistakenly left it enabled somewhere, malicious users couldn’t exploit it for access.

This application uses a declarative security policy: the deployment descriptor identifies a set of user roles, and another set of “security constraints”. Each security constraint describes one or more URL patterns used in the application, and the set of user roles allowed to access URLs matching those patterns. Here are two examples:

<!-- clients -->
<security-constraint>
    <web-resource-collection>
        <web-resource-name>Secure Area</web-resource-name>
        <url-pattern>/views/clients/*</url-pattern>
        <url-pattern>/client/edit/*</url-pattern>
        <url-pattern>/client/edit_tree/*</url-pattern>
        <url-pattern>/client/view/*</url-pattern>
        <url-pattern>/client/view/id/*</url-pattern>
    </web-resource-collection>
    <auth-constraint>
        <role-name>client_role</role-name>
    </auth-constraint>
</security-constraint>

<!-- places -->
<security-constraint>
    <web-resource-collection>
        <web-resource-name>Secure Area</web-resource-name>
        <url-pattern>/views/places/*</url-pattern>
    </web-resource-collection>
    <auth-constraint>
        <role-name>manage_places</role-name>
    </auth-constraint>
</security-constraint>

We installed the cas-extension library in our Wildfly server to handle the CAS protocol. When an unauthenticated user attempts to access a URL matching a pattern in one of the application’s security constraints, the CAS extension automatically intercepts control and redirects the user to the CAS server. Assuming the user authenticates successfully, our application will receive another request with a service ticket parameter. The cas-extension intercepts this request as well, validates the service ticket, and creates an “identity assertion”, which it sends to the Wildfly security system. Wildfly’s role mapper queries a database to find the user’s roles, after which the authentication process is complete.

Configuration of the cas-extension begins with a CAS profile, a combination of the URL of the CAS server and the URL of the service the extension should protect.

<subsystem xmlns="urn:soulwing.org:cas:1.0">
    <cas-profile name="default" service-url="https://our.application.server/application" server-url="https://our.cas.server"/>
</subsystem>

This tells the extension where to send authentication requests, where to listen for requests returning from the CAS server, and where to validate service tickets. Next we need to validate that identity assertion, and figure out what roles belong to the user. This happens in a Wildfly security domain.

<security-domain name="MySecurityDomain">
    <authentication>
        <login-module name="IdentityAssertion" code="org.soulwing.cas.jaas.IdentityAssertionLoginModule" flag="required" module="org.soulwing.cas"/>
    </authentication>
    <mapping>
        <mapping-module code="DatabaseRoles" type="role">
            <module-option name="dsJndiName" value="java:/comp/env/jdbc/databaseConnection"/>
            <module-option name="rolesQuery" value="select role from user_roles where user_id = ?"/>
        </mapping-module>
    </mapping>
</security-domain>

The authentication portion of the security domain refers to a JAAS LoginModule shipped with the CAS extension, which simply verifies that the identity assertion comes from the CAS extension and not somewhere else. Then the mapping portion (documented here) looks up the given user in a database to find what roles it should be assigned.

The last piece of the puzzle is a CAS deployment descriptor for our application, which activates cas-extension for that application. In its simplest form, this is an empty file in the right place, but ours ended up a little more complex. It identifies both the CAS profile we want to use (unnecessary in this case, as there’s only one CAS profile on the system, but it helped keep things more clear in our minds), and instructs the extension to load some other libraries into our application.

<?xml version="1.0" encoding="UTF-8"?>
<cas xmlns="urn:soulwing.org:cas:1.0">
  <profile>default</profile>
  <add-api-dependencies/>
</cas>

This configuration proved sufficient to let users log in and use the application, but as is sometimes the case with single sign-on, we needed a little more work to let them log out properly. Each application in the suite sets a cookie in the user’s browser to identify its session. The CAS server likewise sets a cookie. When a user logs out of the application, that application’s session cookie is destroyed, but we also need to destroy the CAS server’s session cookie as well as the other applications’ cookies. Single log-out can be complicated, and I won’t go into the full setup here. One reason the CAS deployment descriptor loads cas-extension libraries was so we could use cas-extension to generate the proper logout URL, and redirect our users to that URL once the application has destroyed its own session.

Configuring Spring Authentication

Another Java-based application in our suite uses Spring Security. This document describes the bulk of the configuration, which seemed less straightforward than for cas-extension, but follows essentially the same mechanism. Here we configured a Spring UserDetailsService to execute the same database query we used above to find the user’s roles. I’m not fully conversant in the large stack of beans Spring uses to manage the process, and it took some time to get this configuration sorted out.

Enter PHP

Two of these applications use PHP, which meant yet another configuration. Apereo maintains a PHP client, which includes several helpful examples. I tracked down the part of the application that authenticates users, and replaced the existing code with calls to phpCAS:

phpCAS::client(CAS_VERSION_2_0, $phpCASHost, $phpCASPort, $phpCASContext);
phpCAS::forceAuthentication();
$_SESSION[EXPORT_SERVERNAME]['umdid'] = phpCAS::getUser();

The forceAuthentication call determines the current phase of the authentication process this request is in, whether it’s unauthenticated, fully authenticated, or requires service ticket validation, and responds appropriately. We then set a session variable to the ID of the authenticated user, which replicates what the original authentication code would have done.

One of these two applications requires authenticated access to a REST API exposed by the Wildfly application. CAS calls this “proxy” authentication, when one application requests access to another. Here, CAS issues not only its usual service ticket, but also a “proxy granting ticket”. When the application wants to use the API, it asks the CAS server for a service ticket, using the proxy granting ticket. The CAS server itself requires some new configuration in this case, but for the PHP code, the only difference in the login phase from the simpler, non-proxy case is that we call phpCAS::proxy instead of phpCAS::client to configure CAS. Later, when calling the service itself, we used more phpCAS services in place of the cURL library the original used.

$service = \phpCAS::getProxiedService(PHPCAS_PROXIED_SERVICE_HTTP_GET);
$service->setUrl($serviceURL);
$service->send();

The return of header authentication

Finally, we have a few automated tasks which use various APIs, and need to authenticate. We can’t redirect requests to a CAS server and expect a user to provide credentials, so we’ve taken our cue from the applications’ original form, and configured CAS to recognize a “trusted header”. We add this header to any requests issued by these automated jobs. Of course, we’ve also configured the proxy to disallow this header from any systems outside our internal network.

A few loose ends

Of course, there were other considerations in this project that I’ve not covered here. Configuring CAS itself wasn’t necessarily straightforward, and included a custom authentication module I hope to describe in a later article. Selecting among CAS server’s available deployment options and fitting the winner into our existing infrastructure in a way that makes it easy to manage and monitor was another task entirely. We needed to customize the server’s default user interface to prevent it from offering users an imaginary method to recover forgotten passwords. The series of redirected browser requests involved in the CAS protocol presents a notable performance impact under some circumstances. And it has taken me no small effort to learn to appreciate the CAS server’s sometimes distressingly circular documentation. But several months into the project, CAS seems to be working well enough for our purposes that other users of the same application suite have begun to express interest.

How I Learn New Technologies

2018-10-30T00:00:00+00:00

Photo by Marco Verch · CC BY 2.0, modified

If a developer has plenty of time, then the best way to learn a technology is to read a book about it, solve the tasks the book presents, and then to do some very basic work just to get some real-world experience. When this is done, one might want to watch some tutorial videos, consult with people who are either also learning, or, even better, are experienced in the given technology.

When someone gets comfortable with a dev stack, the person might be inclined to prefer to work only in his or her comfort zone. In consulting, this is feasible in the majority of cases. But what if someone gets a new project in a different software stack? Is it a big problem? In my opinion, it’s not a problem, if we are able to determine the minimal knowledge we need to get started. So how can a developer quickly grasp the essence of the problem space? Where are answers to the most frequent questions reachable?

In 2008 I graduated from the university and was looking for a job. In the middle of the world economic crisis new programmers had limited chances at getting a job. So I began accepting very small projects from varied clients, all using vastly different technologies. To be successful at this, I realized I had to revise my method of learning technologies.

I particularly remember the case when in 2009 I had a small JavaScript project and I had no knowledge whatsoever and a deadline of 24 hours, as the client needed the job done ASAP. Reading a book was not an option. Even a tutorial was too long, so after four hours of trying hard to solve the problem I watched Douglas Crockford’s video. After so much suffering, his joke “Good morning, everybody, welcome to the Blair Witch project” was more than fitting about how I felt in that situation. After four hours of suffering I finally found an ally. With the help of this video I was able to grasp the core of the knowledge I needed, that is, the information I need to be able to quickly find other information if needed. After watching the video, I completed the small project without much difficulty. Of course, it was a new language for me, but at least I had some understanding. That case taught me the importance of quickly grasping the core of the new technology I am learning.

My methodology is to put the appropriate questions needed to be answered into a list. By looking at the answer of each question, I can acquire the core knowledge which allows me to work with the given technology. Of course, working with a technology one has no previous experience with takes more time in comparison with working with a technology one has years of experience with, because even the most trivial tasks might need some research, but if the core knowledge is already acquired, then 90% of the answers to the question one may have while working are easily answered.

New Programming Language

What justifies the existence of the language? What is the purpose the language aims to fulfill?
Where can I find the original documentation and how do I search and navigate there?
Is the syntax similar to a language I already know? How are commands separated, what are the keywords one will frequently use?
What type of things will I work with using the language? With CSS I define rules which specify the looks of a web page. With query languages I work with databases, relations/tables, records, and columns. With HTML or XML I define a node hierarchy. With C I write functions. With Java I define classes and objects.
What types of tasks can I expect from the projects I can work with?
Can I watch a tutorial video? Watching an introductory video takes hours. Reading a book can take weeks.
What operating system should I use?
What IDE should I use?
Is there an online possibility to write experimental code to try my hand in this new language?
Do I need to register in community forums?
Do I have questions I can’t answer yet? I should be able to find an answer on community forums or general purpose programming Q&A sites.
Have I written a simple Hello World in the language?
What will I have to watch out for?
What are the best practices for coding in the given language?

New Operating System

Do I have two systems I can use, one where I can search the internet for help, and the other where I can begin experimenting on the new operating system?
How to operate with files and folders?
How to install/run an application?
What is the minimal knowledge I need in order to be able to work on my programming tasks?

New Libraries and Tools

Why do I need this?
What are the features and use-cases I will be using?
What are the names of functions/methods I will be working with frequently?
What do I need to do to use this?
Is it documented?
Does it have a forum?

New Projects

What is the aim of the project?
Are there technologies in use that I do have experience with?
What are the technologies in use which I do not have experience with? Is it urgent to learn them?
What are the main features of the project?
What will I have to interact with first in the project?
Can I play around with the project and do some experiments without being in danger of doing any harm to it?
Is it documented?
How am I going to work with it, what do I need to download/install, and what networks am I going to access?
How is the personality of the client affecting the working relationship? Am I going to have an easy time or not in my first days?
Where are the vital points of the project and can I take a look at the code there to have a vague understanding of what is being done?

New Concepts

What is the definition of the concept?
Is it related to something I already know?
Does it have subconcepts?
Do I need to understand something complex? If so, can I find/draw some illustrative diagrams?
Can I make a fairly accurate allegory, using elements I am aware of?

Allegory

In general, if something is not understandable at all for me, or has so many elements that it will be difficult to remember them all, I construct an allegory, using concepts I am already aware of. This way, if I forget anything from my newly acquired knowledge, then I think about my allegory, which serves as a mental decryption method.

For instance, when I learned about object oriented concepts, I used the allegory of set theory to understand the concepts.

I viewed classes as active sets, that is, sets which might have executable methods.

I viewed objects as items in a set, which, on their own have their capability to perform actions.

I viewed inheritance as a subset relation.

To make the allegory more or less full, I had trouble integrating abstract classes and interfaces into the allegory, but after thinking about them, I came up with a solution. I considered abstract classes as partial definitions of a set and interfaces as declarations of the sets. Since neither had a complete definition, I needed a nonabstract subset to find items to these sets.

Method overloading was easily understood for me as doing something using different domains. a * b is different if both a and b are real numbers and different if they are complex, basically we do the same operations, but with different variables. I added method overriding into my allegory as an evolution of methods, or as a specialization.

The allegory is not necessarily 100% analogous with the concepts to be grasped, but if its accuracy is fairly high, then it’s a good starting point. In many cases this is enough by itself, but when it’s not enough, the allegory helped me to formulate intelligent questions, which I could transform to keywords which are searchable by a search engine.

Notes

I am sure I am not the only human being with bad sectors in his/her brain. Have you ever had the experience of totally understanding something and then repeatedly forgetting an important detail, but only that detail? Luckily, we are not in a classroom where we need to reproduce accurately all the verses of a long poem. We can write notes. If I observe that I regularly forget something important, then I make notes.

Closing Words

The methods I described in this article work well for me. We are all different and what works for me does not necessarily work for you and vice versa. However, I think that this approach can be personalized. If you feel that you need to do things differently in comparison to what I described here, then you are probably right, as you have a different brain and it probably does not work in the exact same way as mine.

For me, when I learn, frequent short breaks are helpful, as my subconscious is working out without any effort some details which are more difficult for my conscious thinking. After I read an article, if I take a walk and breath fresh air, I tend to have deeper knowledge when I return than when I started to walk.

It is possible that you need to learn in a totally different way. My words here are not carved into stone. My advice, in general is to not be afraid of new challenges and technologies. It is awesome if you add learning into your comfort zone!

Drupal — rapid development

2017-05-26T00:00:00+00:00

Here at End Point, we had the pleasure to be a part of multiple Drupal 6, 7 and 8 projects. Most of our clients wanted to use the latest Drupal version, to have a long term support, stable platform.

A few years ago, I already had big experience with PHP itself and other, various PHP frameworks like WordPress, Joomla! or TYPO3. I was happy to use all of them, but then one of our clients asked us for a simple Drupal 6 task. That’s how I started my Drupal journey which continues until now.

To be honest, I had a difficult start, it was different, new and pretty inscrutable for me. After a few days of reading documentation and playing with the system I was ready to do some simple work. Here, I wanted to share my thoughts about Drupal and tell you why I LOVE! it.

Low learning curve

It took, of course, a few months until I was ready to build something more complex, but it really takes a few days only to be ready for simple development. It’s not only about Drupal, but also PHP, it’s much cheaper to maintain and extend a project. Maybe it’s not so important with smaller projects, but definitely important for massive code bases. Programmers can jump in and start being productive really quick.

Great documentation

Drupal documentation is well structured and constantly developed, usually you can find what you need within a few minutes. It’s critical and must have for any other framework and not so common unfortunately.

Big community

The Drupal community is one of the biggest IT communities I have ever encountered. They extend, fix and document the Drupal core regularly. Most of them have their other jobs and work on this project just for fun and with passion.

It’s free

It’s an open source project, that’s one of the biggest pros here. You can get it for free, you can get support for free, you can join the community for free too (:)).

Modules

On the official Drupal website you can find tons of free plugins/modules. It’s a time and money saver, you don’t need to reinvent the wheel for every new widget on your website and focus on fireworks.

Usually you can just go there and find a proper component. E-commerce shop? Slideshow? Online classifieds website? No problem! It’s all there.

PHP7 support

I can often hear from other developers that PHP is slow, well, it’s not the Road Runner, but come on, unless you are Facebook (and I think that they, correct me if I’m wrong, still use PHP :)) it’s just OK to use PHP.

Drupal fully supports PHP7.

With PHP7 it’s much faster, better and safer. To learn more: https://pages.zend.com/rs/zendtechnologies/images/PHP7-Performance%20Infographic.pdf.

In the infographic you can see that PHP7 is much faster than Ruby, Perl and Python when you try to render a Mandelbrot fractal. In general, you definitely can’t say that PHP is slow, same as Drupal.

REST API support

Drupal has the built in, ready to use API system. In a few moments you can spawn a new API endpoint for you application. You don’t need to implement a whole API by yourself, I did it a few times in multiple languages, believe me, it’s problematic.

Perfect for a backend system

Drupal is a perfect candidate for a backend system. Let’s imagine that you want to build a beautiful, mobile application. You want to let editors, other people to edit content. You want to grab this content through the API. It’s easy as pie with Drupal.

Drupal’s web interface is stable and easy to use.

Power of taxonomies

Taxonomies are, really basically, just dictionaries. The best thing about taxonomies is that you don’t need to touch code to play with them.

Let’s say that on your website you want to create a list of states in the USA. Using most of the frameworks you need to ask your developer/technical person to do so. With taxonomies you just need a few clicks and that’s it, you can put in on your website. That’s sweet, not only for non technical person, but for us, developers as well. Again, you can focus on actually making the website attractive, rather than spending time on things that can be automated.

Summary

Of course, Drupal is not perfect, but it’s undeniably a great tool. Mobile application, single page application, corporate website—there are no limits for this content management system. And actually, it is, in my opinion, the best tool to manage your content and it does not mean that you need to use Drupal to present it. You can create a mobile, ReactJS, AngularJS, VueJS application and combine it with Drupal easily.

I hope that you’ve had a good reading and wish to hear back from you! Thanks.

Strict typing fun example — Free Monads in Haskell

2016-03-11T00:00:00+00:00

From time to time I’ve got a chance to discuss different programming paradigms with colleagues. Very often I like steering the discussion into the programming languages realm as it’s something that interests me a lot.

Looking at the most popular languages list on GitHub, published last August, we can see that in the most popular five, we only have one that is “statically typed”. https://github.com/blog/2047-language-trends-on-github

The most popular languages on GitHub as of August 2015:

JavaScript
Java
Ruby
PHP
Python

The dynamic typing approach gives great flexibility. It very often empowers teams to be more productive. There are use cases for static type systems I feel that many people are not aware of though. I view this post as an experiment. I’d like to present you with a pattern that’s being used in Haskell and Scala worlds (among others). The pattern is especially helpful in these contexts as both Haskell and Scala have extremely advanced type systems (comparing to e. g. Java or C++ and not to mention Ruby or Python).

My goal is not to explain in detail all the subtleties of the code I’m going to present. The learning curve for both languages can be pretty dramatic. The goal is to make you a bit curious about alternative development styles and how they could be very powerful.

Short intro to the idea behind the pattern

The pattern I’m going to present is called the “Free Monad + Interpreter”. The idea behind it is that we can build DSLs (domain specific languages) by making our functions not execute the code immediately, but to build the AST (abstract syntax tree) out of it and interpret it in different ways depending on the context.

A fun example I came up with is a DSL for system provisioning scripts that—among many use cases one could come up with—allows to:

present the AST in bash or zsh code or whatever other language like Python, Ruby or Perl
present the AST as a graph to visualize the execution
execute it directly, natively in Haskell
have an easy-to-comprehend set of provisioning instructions while lower level aspects like file handles etc.—being handled in common Haskell code used for the execution of ASTs

There are potentially many more use cases but I just wanted to show you a couple—enough to hopefully make you a bit curious. In this post we’ll focus on interpreting the AST as a bash script.

The coding part

The first step is to define the set of instructions our interpreted Domain Specific Language will support:

data Provision next =
  Begin next |
  Install String next |
  IfNotExists String (Free Provision ()) next |
  Touch String next |
  Cd String next |
  MkDir String Bool next |
  Echo String next |
  Continue |
  Done
  deriving(Functor)

This odd looking definition is what’s called an Algebraic Data Type. For now it should suffice that the commands can take arguments of different types and almost all of them take a continuation command as the last parameter.

The continuation parameter is meant to store the next “provisioning command” so that we would have e.g:

Begin (Install "postgresql-server" (Echo "installed!" (Done)))

Out of these blocks, our ASTs will be created. We need some way of composing these blocks into AST trees. I’m not going to explain here why the following code works—it’s just a teaser post. Let’s just say that the following functions allow us to just build the tree instead of calling any system-affecting code. In other words, it allows these calls to look as if they’re doing something when in fact they are just constructing the data structure in memory:

begin = liftF $ Begin id

install what = liftF $ Install what id

ifNotExists path what = liftF $ IfNotExists path what id

touch path = liftF $ Touch path id

cd path = liftF $ Cd path id

mkDir path wholeTree = liftF $ MkDir path wholeTree id

echo message = liftF $ Echo message id

continue = liftF $ Continue

done = liftF Done

Now that we have these building functions defined, we can create a function that uses them to construct a useful AST:

app :: Free Provision a
app = do
  begin
  install "postgresql-server"
  mkDir "/var/run/the-app" True
  cd "/var/run/the-app"
  ifNotExists "the-app.log" $ touch "the-app.log" >> continue
  done

Running this function does nothing except for returning AST wrapped inside the “free monad”—which you can think of as a special, useful kind of container. The above function looks like any other Haskell function. It’s also “type safe”—which weeds out one class of errors that we’re only able to notice after we ran the code—in languages like JavaScript or Python.

Later on we’ll see that to get different results out of the “provisioning workflow” we defined above, no change in this function will be needed.

Now, having an AST tree wrapped around some “useful” container almost screams for some kind of an interpreter for this to be useful too. That’s in fact part of the description of the pattern I gave you in the beginning of this post.

Let’s define a set of data types linked with the function that we’ll use as an interpreter:

class InterpretingContext a where
  run :: Free Provision () -> a

The above just says that if we want to use the function run to turn the AST wrapped in a monad to some concrete value (by executing it)—we need to implement this function for the type of the concrete value we’d like to get out of it.

For example, let’s say that for the portability sakes we want to turn the AST into the bash script. The natural (though naive) way to do this would be to implement this “class” along with its run function for the type of String:

instance InterpretingContext String where
  run (Free (Begin next)) =
    "#!/usr/bin/env bash\n\n" ++ (run next)

  run (Free (Install what next)) =
    "apt-get install " ++ what ++ "\n" ++ nextStr
    where
      nextStr = run next

  run (Free (IfNotExists path what next)) =
    "if [ ! -f " ++ path ++ " ]; then\n\t" ++ whatStr
      ++ "\nfi\n" ++ nextStr
    where
      whatStr = run what
      nextStr = run next

  run (Free (Touch path next)) =
    "touch " ++ path ++ "\n" ++ (run next)

  run (Free (Cd path next)) =
    "cd " ++ path ++ "\n" ++ (run next)

  run (Free (MkDir path tree next)) =
    "mkdir " ++ treeOption ++ " " ++ path ++ "\n" ++ (run next)
    where
      treeOption =
        if tree then "-p" else ""

  run (Free (Echo message next)) =
    "echo " ++ message ++ "\n" ++ (run next)

  run (Free Continue) = ""

  run (Free Done) = "exit 0"

Each node kind is being interpreted as a data type we chose to be one of the instances of this class—in our example a String.

What this allows us to do, is to use the run function, specifying that we want a String as a return value and automatically the instance we’ve just created will be used:

run app :: String

This will return:

"#!/usr/bin/env bash\n\napt-get install postgresql-server\nmkdir -p /var/run/the-app\ncd /var/run/the-app\nif [ ! -f the-app.log ]; then\n\ttouch the-app.log\n\nfi\nexit 0"

Pretty printed:

#!/usr/bin/env bash

apt-get install postgresql-server
mkdir -p /var/run/the-app
cd /var/run/the-app
if [ ! -f the-app.log ]; then
    touch the-app.log
fi
exit 0

If now we’d like to execute the AST in the context of an action that prints the script to stdout we could do so like this:

instance InterpretingContext (IO ()) where
  run = print . run

From now on it would be perfectly valid to run the function with AST in both contexts:

run app :: String
run app :: IO ()

We could add a context returning an ExitStatus by running the code against the system very easily too:

data ExitStatus = ExitSuccess | ExitFailure Int

instance InterpretingContext (IO ExitStatus) where
  run = (…)

What this gives us is the ability to have the provisioning code that could be run in production while having a different interpreter in the testing suite to be able to ensure the structure of execution without inflicting any changes to the system itself.

If you’d like to play with the code yourself, you’ll need a couple of more lines for this to work:

{-# LANGUAGE DeriveFunctor #-}
{-# LANGUAGE FlexibleContexts #-}
{-# LANGUAGE FlexibleInstances #-}
{-# LANGUAGE TypeSynonymInstances #-}
{-# LANGUAGE OverloadedStrings #-}

And also:

import Control.Monad.Free

Bear in mind though that the code I presented here is by no means optimal—especially memory wise. I chose to present it this way for the clarity of what the code is doing for those of you not familiar with the language.

What are other use cases for this pattern?

The pattern presented here has a huge number of uses. It could be used for providing a DSL for building SVG combined with an interpreter that could draw it visually for the ease of work. It could also be used for defining RPC data types describing structures and interpreting them differently based on the underlying RPC (remote procedure call) mechanics (Thrift, SOAP etc).

I doubt that the ability this gives thanks to the very helpful Haskell type system could be reproduced in languages like Ruby or Python easily. It is possible of course, but the amount of boilerplate code and complexity would require lots of testing code too. Here on the other hand the code holds many guarantees just because we’re coding in a language with an advanced strict type system.

Also, the similarity to the Interpreter Pattern known from the object oriented languages is only superficial. In that case there’s no way to use regular normal functions (or methods) to build AST—as if it was a regular imperative code. It’s always about some weird mangling of data structures by hand.

Curious?

If I managed to make you a bit curious about the aspects I presented here, here are some of the resources you might want to take a look at:

Coding style guides across languages

2015-08-17T00:00:00+00:00

Introduction

Before you can start programming production code in any programming language you should know a few things like syntax, purpose, coding standards, good practices. There is another thing that is really important too and can help developers that work with multiple projects and languages: coding style guides. This is just a set of rules that developers should take care about when writing code. They are not so different across various programming languages. In this article, I will sum up and list coding style guides for the most popular languages.

It’s not a crime to use different styling, but standardizing this helps in the process of creating software. When your team is using the same standard it’s much easier to read each other’s code.

There is one universal resource when it comes to programming style guides. Many of you will already have read it: the “Clean Code” book by Robert C. Martin.

Something else that should be mentioned here:

When you don’t know how to write a structure, read guides.
When you know how to write a structure but it’s different than standardized, ask other developers about it.
When your structure is different than all the other structures made by your co-workers, consider changing it.
When someone will force you to change your styling code, ask her/him/them why, even if she/he/they are your manager or boss.
When there is no good coding style guide for your language, create one.
Try to have the same habits of coding in your team and company—it’s a time saver.
Even if you’ve adapted to some structures be open for change.
Respect experience. If senior developers tell you that your code is weird, there is probably something wrong with it.

Order of the guides is not random, most important and trusted first. Not all the guides cover all the topics so it would be wise to real all of them when you plan to write code in such a language.

Let’s begin.

Java

C

C++

C#

Python

Objective-C

PHP

Visual Basic .NET

https://docs.microsoft.com/en-us/dotnet/standard/design-guidelines/index

JavaScript

Visual Basic

https://docs.microsoft.com/en-us/dotnet/visual-basic/programming-guide/program-structure/coding-conventions

Perl

Assembly

http://www.sourceformat.com/coding-standard-asm.htm

Ruby

Conclusion

Maybe it’s not the most important programming part but it’s something you need to think about.

If we will try to compare those rules to human language, English for example, see “Romeo and Juliet”, Act III:

MERCUTIO

Come, come, thou art as hot a Jack in thy mood as any in Italy, and as soon moved to be moody, and as soon moody to be moved.

Now try to remove writing standards:

MERCUTIO Come, come, thou art as hot a Jack in thy mood as any in Italy, and as soon moved to be moody, and as soon moody to be moved.

It’s not easy to read this one liner. Our code is the same: Be smart, listen to more experienced people, read style guides and you will make live easier for you and other people too.

Perl’s CPAN is 20 years old

2015-08-17T00:00:00+00:00

(Photo by emma.kate on Flickr)

This is just a short note to celebrate the fact that the Comprehensive Perl Archive Network (CPAN) turned 20 years old yesterday!

CPAN began as a way to collect, mirror, and distribute open-source Perl packages. Over time it led to development of better packaging and module naming conventions; formed a community for feedback, testing, and contributions; and became one of the great strengths of the Perl world.

It is rare to find some needed functionality that is not available on CPAN. These days a more common problem is finding too much choice there, and needing to choose between several modules based on which are better maintained and supported on current versions of Perl, or kept current against external dependencies.

Perl does not get as much press these days as it once did, but it continues to thrive and improve. On that topic, our former co-worker Steph Skardal sent me an article called We’re still catching up to Perl by Avdi Grimm of Ruby fame. It is not an in-depth language comparison, just a brief observation to his fellow Rubyists that there is plenty to be learned from Perl. (Of course Perl has plenty to learn from Ruby and other languages too.)

Nothing more permanent than a temporary fix

2015-07-28T00:00:00+00:00

A brief thought:

You may have heard the saying that nothing is more permanent than a temporary fix. Or that prototypes are things we just haven’t yet recognized will be permanent. Or some variation on the theme.

As an illustration of this, I recently came across the initial commit to the source code repository of our endpoint.com website when we ported it to Ruby on Rails back in April 2007. Our then co-worker PJ’s comment is a perfect example of how long-lasting some of our planned temporary work can be:

commit 2ee55da6ed953c049b3ef6f9f132ed3c1e0d4de9
Author: PJ Cabreras <pj@endpoint.com>
Date:   Wed Apr 18 13:07:46 2007 +0000

    Initial test setup of repository for mkcamp testing -- will probably throw away later
    
    git-svn-id: file:///home/camp/endpoint/svnrepo/trunk@1 7e1941c4-622e-0410-b359-a11864f70de7

It’s wise to avoid big architecture up front for experimental things we don’t know the needed shape and size of. But we should plan on iterating and being agile (in the real basic sense of the word), because we may never have the chance to start over from scratch. And starting over from scratch is often ill-advised in any case.

To ask or not to ask? Debug first.

2013-02-18T00:00:00+00:00

Jumping head first into a project, the ramp up will likely lead to questions galore. In the eagerness of getting things done, it seems like the best thing to do when stuck is to just ask the seasoned developers to tell you how to move forward. After all, they did build the application. However, when to reach out for help can be dependent on the deadline and priority of the task at hand as well as your subjective definition of “stuck.” Knowing when it’s too early, just right, or too late to get help can be a tricky thing. Here are some things to consider when reaching out for help early:

Pros.

1. Time/money is of the essence and getting a quick answer is best.

2. Time saved debugging a particular issue that does not further your understanding of the application can be applied elsewhere.

Cons.

1. You risk a learning opportunity by throwing in the towel too early.

2. You risk looking lazy or unprepared if the person whom you are reaching out to believes you could have done more.

3. Developers are busy, too.

All cases being different, there is no right time to reach out for help but steps can be taken to ensure that you have your part. First, get better at reading source code. The more you practice reading and making sense of other people’s code, the quicker you will build your complete mental model of the application, and the quicker you’ll be able to debug. Brandon Bloom (snprbob86) wrote an excellent post regarding this. Second, create your own stack trace. The application may have many entry points. Pick one, feed it some input and trace it down the rabbit hole. A white board may come in handy here. Third, make heavy use of the logs. The code should already be sprinkled with plenty of log messages, but feel free to add more wherever you feel they will be useful.

These three tips will go a long way in helping you debug smarter, which should allow you to find solutions more quickly. In the event that a white flag must be raised and another developer’s help requested, the efforts you made attempting to debug will have a positive impact on your productivity as well as your team’s perception of you as a programmer.

Building a Multi-Agent Travel Planner with Microsoft Agent Framework and .NET

What we’ll build

Prerequisites

Project structure

The agent pipeline

Abstracting the model provider

Fixing the Anthropic adapter mismatch

Creating specialized agents

Using live APIs without making the system brittle

The Auditor pattern

Streaming the pipeline to the browser

Handling MAF’s silent failures

Filtering empty streaming chunks

Evaluating output with MEAI

Running the application

Securing the API

Key takeaways

Build a Smarter Telegram Bot: Integrating a RAG Pipeline for FAQ Answering

What is Telegram?

A Warning on Privacy and Encryption

What is Retrieval-Augmented Generation (RAG)?

What you’ll build

Prerequisites

Setting up the Project for Telegram Bot

Environment and configuration

Try it out

What this bot does

Core structure

Handlers and filters

Running the bot

Setting up the knowledge base

Setting up the RAG Pipeline

Data Ingestion (Indexing the FAQs) handled by setup_rag_chain() method

The RAG Retrieval Logic handled by handle_message() method

New capabilities added to bot.py

Try it out

Creating API for Invoice Generator Using C# and Minimal APIs

Prerequisites

Setting up the Project

Setting up the Swagger UI

Setting up the Database Context with Entity Framework

Designing the Invoice Generator API

Creating and running database migrations

Implementing API Endpoints

Adding CORS for development

Integrating with Vue 3 Application

Deployment

Conclusion

Learning Vue 3 composables by creating an invoice generator

What are Vue 3 composables?

Core functionalities for invoice generator

Prerequisites

Setting up the project

Create first composable for invoice: useInvoice

Define interfaces for Invoice and InvoiceItem

Adding business logic

Rendering the invoice with composable functions

Create a second composable for using localStorage to save and retrieve invoice data

Printing Invoice

Bonus: create a third composable for showing dashboard data

Next Steps

Conclusion

Passing Data Between Components in Vue.js: An Overview

Props

Usage

Pros:

Cons:

Custom Events: Emitting Changes from Child to Parent

Usage:

Pros:

Cons:

EventBus: A Centralized Event Hub

Usage:

Pros:

Cons:

Vuex: State Management for Larger Applications

Key Takeaways from Practical Object-Oriented Design by Sandi Metz

Chapter 1: Object-Oriented Design

About design

Failures of design

Data Ingestion (Indexing the FAQs) handled by `setup_rag_chain()` method

The RAG Retrieval Logic handled by `handle_message()` method

New capabilities added to `bot.py`

Create first composable for invoice: `useInvoice`

Define interfaces for `Invoice` and `InvoiceItem`