High Level System Analysis and Design with Domain-Driven Design

2026-04-06T00:00:00+00:00

Photo by Zed Jensen, 2022.

Domain-Driven Design is an approach to software development that focuses on, as Eric Evans puts it, “tackling the complexity in the heart of software”. And what is in the heart of software? The business domain in which it operates. Or more specifically: a model of it, made of code. That is, the code that implements the business logic that comes into play when solving problems within the realm of a particular business activity.

DDD is not just about writing code though. It’s a whole methodology that touches on business needs, requirements gathering, organizational dynamics, high level architectural design, and lower level patterns for implementing software intensive systems.

As a result, DDD offers a treasure trove of concepts, patterns and tools that can be applied to any software project, regardless of the size and complexity.

In this series of blog posts we’re going to explore many aspects of DDD. We will be following the structure laid out by Vlad Khononov’s excellent book on the topic “Learning Domain-Driven Design: Aligning Software Architecture and Business Strategy”. So you can think of this series as a summary of that book. An abridged version that can serve as a review for anybody who has read it; but also as an entry point for people who are new to DDD.

High level system analysis and design with Domain-Driven Design

Section 1. Analyzing business domains

It is a well understood notion that before writing code, we need to understand the problem we’re trying to solve. DDD is consistent with this notion and argues that developers need to, first and foremost, gain an understanding of the business that the software is being built for. To this end, DDD relies on three concepts: domains, subdomains and domain experts.

Domains and subdomains

In simple terms, the domain of a business is what it does, its area of activity. For example, Starbucks is in the business of making coffee, Ford is in the business of making automobiles, AMC is in the business of movie theaters.

Of course, analyzing a business as a single integrated whole can be unmanageable. That’s where subdomains come in. Subdomains are the different divisions within a domain. Starbuck’s domain for example, is making coffee. But there are many smaller parts that make up that business and allow it to be successful. There’s of course, the subdomain of coffee preparation. But there’s also real estate management to find and secure good locations, there’s inventory management and logistics, there’s marketing, there’s human resources, etc. All these are the subdomains that make up the overall business of Starbucks.

Depending on the business and on the project, these will vary greatly in granularity. And you can also decompose subdomains further and discover new fine-grained subdomains nested within more coarse-grained ones. The sizes and nesting levels can be very different from business to business, so it can be difficult to clearly delineate where a subdomain ends and another one starts, which activities belong to one or the other. One good rule of thumb to keep in mind is that generally, subdomains encapsulate a set of coherent, closely related use cases. That is, use cases that involve the same set of closely related actors, business entities and/or data.

And finally, we have domain experts. As the name suggests, these are the people within the organization who have intimate knowledge of the business, or certain areas of it. They are the subject matter experts. Usually they are the ones who identify the problems and come up with requirements. Developers need to rely on domain experts to gain the necessary understanding to be able to produce useful software solutions.

So, when approaching software projects, DDD suggests that developers work closely with domain experts in order to learn from them about the business domain and subdomains. After all, it is their mental models and understanding that will be modeled and implemented in code.

Types of subdomains

With the help of domain experts, developers can identify subdomains, understand their business value and how they fit within the overall business strategy. This is very important because it helps making some initial architecturally significant decisions. Namely, the general approach to solving the problems in the subdomains, how much to invest, how to organize development teams, etc. The main objective in this analysis stage is to identify the subdomains and whether they fall into one of three types: core, generic, and supporting.

Core subdomains are the activities of the business with highest value. The ones that confer differentiation in the market and a competitive advantage. They are the business’ raison d’être. For example, for Google, their search engine is a core subdomain. For Ford, their automotive engineering area would be a core subdomain. Core subdomains are generally the most complex parts of the business. They are constantly evolving and improving, and the company is compelled to invest heavily in them.

Generic subdomains are also very complex. However, they are not business differentiators. Instead, these are the areas of the business that all organizations handle in the same way. Think accounting, a ticketing system, an online storefront. There’s no pressure to innovate in these areas, so the solutions are very stable and evolution is slow.

Supporting subdomains on the other hand, do not provide any competitive advantage, nor are they very complex. They are however, necessary because they support the core business activities, and are fairly unique. The solutions to problems in these areas usually take the form of CRUD or ETL oriented activities. Imagine for example populating a data warehouse, translating transactional business data into a format appropriate for analytics and business intelligence in a manufacturing corporation. Or maybe the digitization and storage of a registry of court documents for a law firm. These are behind the scenes activities that support the organizations’ main businesses, but their business logic complexity is not high, and they don’t really represent big selling points for the company.

It’s worth noting as well, that there may be subdomains where software solutions are not appropriate, even if they are highly complex core subdomains. They are still part of the business so it’s worth identifying and considering for high level architectural design decisions. If anything, to know what parts of the business the planned software system should and should not focus on. You could have a restaurant, for example, who prides itself in having the best desserts in the city. For their business, the recipe development activities constitute a core subdomain. This is dependent on the art and craftsmanship of the chefs. Not an area in which software solutions could help a whole lot.

It’s also worth noting that, just as organizations’ business strategies are dynamic, so too can be their subdomain distribution. Today’s generic subdomain can be tomorrow’s core subdomain, and so on. For example, imagine a big retail store chain that, up until now, managed its inventory in an industry standard way. But it has grown so much that the standard way of doing things has become a bottleneck for them. So they design a new procedure for highly efficient inventory management, and that gives them an edge against competitors. Inventory management started as a generic subdomain for them, but due to an ever evolving business strategy, it became a core subdomain.

The knowledge of subdomain types can also help the analysis when dividing the business and discovering subdomains. If you find yourself recursively finding more and more fine-grained subdomains from already identified ones, a good time to stop recursing is when you find that a generic or supporting subdomain, when broken down, reveals only new finer-grained generic or supporting subdomains. The reasoning for this is simple: this analysis activity is most valuable when trying to identify core subdomains. When there are no more core subdomains to be found within a particular area of the business, then it’s probably not worth it to keep digging deeper.

Here we have an example of a hypothetical help desk system subdomain, part of a customer service subdomain, being decomposed and revealing that it only contains generic subdomains.

In summary, these are the main characteristics of the three types of subdomains:

Core subdomains have high complexity, provide competitive advantage, and change and evolve frequently.
Generic subdomains have high complexity, do not provide competitive advantage and change overtime, although at a slower pace than core subdomains.
Support subdomains have low complexity, do not provide competitive advantage and are the slowest to change.

This is where each type of subdomain fall when considering their business logic complexity and business differentiation.

Using subdomains to make strategic decisions

Like I’ve already alluded to, armed with this knowledge, DDD practitioners are ready to start making some higher level architectural decisions. Depending on the type of subdomain that a problem belongs to, DDD has specific prescriptions on how to handle the implementation of software solutions for them.

When working on core subdomains, that’s where we want to make the biggest investments. We deploy the most advanced engineering tools, patterns and practices. This is to make sure that the software is efficient and high quality, and also easy to maintain and evolve. This is necessary because core subdomains have to evolve rapidly by nature, if the business is to maintain competitive advantage. Software solutions that operate within the context of core subdomains have to be implemented by high skill and high trust teams. Either in-house, or via trusted development partners, working hand in hand with domain experts.

Problems in generic subdomains, by nature of their business logic being very complex but also very common, are likely to have already been solved. For these types of problems, DDD recommends against implementing custom software, and instead buying and/or adopting tried and true, industry standard, off-the-shelf solutions. Their implementation and integration can be outsourced or handled by less specialized or skilled teams. The change management of these solutions is simple, as they get delivered generally via patches and updates.

For support subdomains, whose business logic is generally simple but uncommon, it is less likely that off-the-shelf solutions would be available. So software addressing problems in these subdomains will most likely have to be implemented as custom solutions. Due to their low complexity though, they can be easily outsourced, or handled by more junior team members. They can also be handled with RAD, low-to-no-code technologies, since often times they are little more than ETL and pure CRUD applications.

Here’s a table to summarizes the differences between the types of subdomains:

Type of subdomain	Core	Generic	Supporting
Competitive advantage	Yes	No	No
Complexity	High	High	Low
Rate of change	High	Medium	Low
Implementation	Custom development	Buy/adopt off-the-shelf	Custom development
Team composition	In-house/partners	Can outsource	Can outsource
Skill level	High	High/regular	Low
Investment	High	Medium	Low
Problems	Interesting	Solved	Simple

Section 2. Discovering domain knowledge

After identifying the business domain and categorizing the various subdomains that compose it, we have a bird’s-eye view of the business. This is good enough to get started and make high level architectural decisions of potential software solutions to problems within these domains. To actually build the software though, we need much more than that. In order to gain a thorough understanding of the business logic, and be able to eventually model and implement it in code, DDD proposes the ubiquitous language as a tool.

The ubiquitous language

The ubiquitous language is DDD’s tool for knowledge sharing, effective communication and software modeling. In simple terms, the ubiquitous language is the language of the business. It’s the language that domain experts use on a day to day basis to talk and reason about the business.

For a software project to be successful, it’s essential that engineers and stakeholders understand each other. They have to be aligned when it comes to the meanings of the core concepts of the business domain. That’s why it’s so important that everybody uses the ubiquitous language for all project related communications, requirements, documentation, face to face discussions, and even code itself. It all needs to share the same language.

Engineers will need to interact with domain experts in order to learn about the business domain and its rules. They need to acquire the necessary knowledge that allows them to implement these rules in the software. During these interactions, they converse using the ubiquitous language. However, the knowledge transfer is not always strictly unidirectional. Yes, engineers have to learn from the domain experts. But also, through conversations and questioning, engineers can help domain experts deepen and flesh out their own understanding about their domains.

A classic example of this is when domain experts focus too much on the “happy paths” of given business processes. Then, through discussions with developers, on account of the inherent precision that software demands, they are forced to consider more edge cases, better specify ambiguous terms and fill out gaps in their understanding.

This deepening of knowledge feeds back into the ubiquitous language and improves it. Making it more insightful and more precise. This means that, throughout the life cycle of a project, the ubiquitous language should keep evolving, expanding and refining. This allows it to continuously improve as an effective model of the business domain, for the problem that it’s trying to solve. Indeed, just like code is a model of the business domain, so too is the ubiquitous language.

A classic pitfall in software projects is a manner of communication and knowledge sharing where developers are various steps removed from the domain experts. Consistent use of the ubiquitous language addresses this.

The ubiquitous language as a model of the domain

Through consistent use of the ubiquitous language, developers are able to obtain a deeper understanding of the domain, the problems we’re trying to solve, the reasoning behind the requirements and the mental model of the domain experts. This allows the construction of effective software solutions, with deep business insight, that go beyond simply translating requirements into code. If the implementation models the business domain effectively, there is potential for it to evolve with the business and better adapt as requirements change; and we reduce the risk of missing edge cases that may not be obvious to domain experts.

Under DDD, developers and domain experts develop the ubiquitous language, which then informs the implementation.

Effectively, when we build a ubiquitous language, we’re building a model of the domain. A model that reflects the relevant business entities, their behavior, and relationships. A model that will be eventually implemented into code, but also a model that needs to be understood by all stakeholders, regardless of their technical level. So it needs to be precise and rigorous, but also understandable. As such, in order to be useful, the ubiquitous language needs to respect certain restrictions:

It must not include technical jargon like programming languages, constructs or frameworks; nor mention specific system structures like database tables, servers or programs. It is the language of the business and it needs to be understandable by non technical folks.
It must avoid using the same term for different concepts. In conversation with humans, a lot of the meaning is extracted from the context. In software, not so much. Just like you can’t have two classes with the same name within the same namespace, the ubiquitous language cannot allow such ambiguities.
It should also avoid using different words for the same concept. The classic example for this is words like users, accounts, customers, visitors. They all refer to closely related concepts which might actually have differences. The problem with these overlapping terms is that those differences are obscured by the notion of them “being the same” and being used interchangeably. For usage within the ubiquitous language, it’s best to be precise and clearly delineate terms and definitions. Eliminate what’s redundant, be strict with definitions.
It should avoid including extraneous details. Just like code needs to include only what’s needed to solve the problem at hand and nothing else (in order to avoid accidental complexity); the ubiquitous language must not be polluted with details from outside of the area of business activity that it represents. That can create confusion and unnecessary cognitive load.

The ubiquitous language represents a unified model of the domain. This is in contrast to a more traditional process where domain knowledge gets “translated” multiple times before turning into code.

Tools for capturing the ubiquitous language

The Agile Manifesto declares: “Individuals and interactions over processes and tools”. And of course, when it comes to the ubiquitous language, direct interaction and conversations between engineers, domain experts and other stakeholders reigns supreme. That’s when it can be most useful. It can also be useful, however, to capture the ubiquitous language in documentation and even in runnable form.

A glossary of terms is a great asset for keeping a ubiquitous language. A wiki is a good place to put this. Definitions for key concepts in the business like entities, processes and rules can be captured here. The only caveat is that documentation is static by nature, while the ubiquitous language is continually evolving. So, great care needs to be taken to keep the wiki updated at all times to reflect the latest and most complete understanding of the domain. This should not be relegated to or gate-keeped by only certain people; it should be a team effort where everyone contributes.

An automated acceptance tests suite, written using Behavior Driven Development frameworks, like Cucumber, is also a great way of capturing the ubiquitous language. The advantage of these tests is that they are written in plain human-readable language, not code. And while it may be far fetched to think that non technical domain experts would be capable of writing and maintaining such tests, they certainly can read and understand them, which is a great boon. These tests speak the language they understand: the language of the business.

This is what Cucumber tests look like.

By nature of being executable and tied closely to the implementation code, there is less chance that they become out of date. This can happen more easily with static documentation written in a wiki. The disadvantage is that they require much more effort. But for the right kind of project, one where business logic is very complex or the scope is very wide, they can be very well worth the cost.

Section 3. Managing domain complexity

We can attempt to model an entire business domain with one big ubiquitous language, but sometimes that’s impossible. Especially so for businesses of a certain size, you will inevitably find inconsistencies and conflicts between the mental models of different domain experts. The simplest example of this scenario is when different experts from different areas of the organization have the same word to describe different concepts. Or when they look at the same business entity with different levels of detail.

When the different mental models are valid, and these inconsistencies are legitimate and cannot be reconciled, the solution is to follow the divide and conquer principle and decompose the language into separate ones, each one working within its own bounded context.

Simply put, a bounded context is the context within which a ubiquitous language, and the model it represents, operate, have meaning and are useful. Sure enough, just like a domain can be divided into subdomains, a ubiquitous language can be decomposed into smaller languages, each with its own context, to model different parts of the business. While we can attempt to capture an entire business domain with a single model, this is not advisable in the case of complex systems. It’s better to split up the model into smaller, more precise ones, tailored to work on specific problem domains.

Reasons for creating bounded contexts

Like mentioned before, different domain experts having conflicting mental models is a clear indication that a division in the model needs to happen. However, that only tells us half of the story. There are other indicators that point to when and where further division should happen:

First of all, there is of course, size. Size by itself is not a deciding factor, but it is something to consider and balance. Fewer, bigger models can help keep the overall environment simpler, but if they become too big they can become unmanageable and prone to corruption. A higher number of smaller models keep cognitive load low, but you run the risk of exploding integration and management complexity. The better principles to keep in mind here are coupling and cohesion. Beware of separating closely tied use cases, that deal with similar actors, entities and data.

System-level nonfunctional requirements also play a role in dividing a model into separate bounded contexts. For example, you might need to decouple the life cycle of several software components. Have them be developed, evolved, versioned and deployed independently. You might also need them to scale separately. You might even need to use completely different technology stacks, whatever is appropriate for the task at hand. This means that bounded contexts generally get implemented as individual services and/or applications. That is, as individual runtime components.

The organization’s composition also plays a role when designing bounded contexts. The general rule is that a given ubiquitous language, the model it represents and the bounded context is lives in, must be owned by a single team. In software, high ownership, cohesion and consistency are desirable traits. Having a single team own and maintain a particular component foments these. Single ownership also helps reduce communication overhead, and prevents people from stepping on one another’s toes. Of course, a single team can own multiple bounded contexts, what cannot happen is one bounded context being owned by multiple teams.

Indeed, in bounded contexts, we have the tools we need to make strategic decisions related to the decomposition of software systems into architecturally significant components or modules. With bounded contexts, we are able to specify the physical and ownership boundaries of these components.

Bounded contexts vs subdomains

Bounded contexts and subdomains are closely related concepts, but there’s one key distinction: subdomains are discovered, while bounded contexts are designed. Subdomains are useful because they help us understand the business strategy. Splitting the business domain into smaller problem domains is useful because it can break down a complex whole into smaller and more manageable parts.

Depending on the situation, it is certainly possible to end up with a set of bounded contexts that align one-to-one with the business subdomains. However, this is not mandatory. We can develop a solution with a single bounded context that spans multiple subdomains; the same way that we can decompose the problem into many bounded contexts, some of which operate within the same particular subdomain.

Bounded contexts are closely related to subdomains, but aren’t tied to them. They are flexible and can be organized in many ways.

Each bounded context becomes a separate major architectural component. That is, a (micro) service, a project, an application. When we have a component that spans multiple subdomains, programming language organizational structures like namespaces or modules can be used to logically separate the subdomains within.

Section 4. Integrating bounded contexts

For a system to function, its components need to interact with each other. So, once we have decomposed the problem domain into separate bounded contexts, we need to decide their relationship and integration strategies. This need for interaction between them implies that there are touch points between bounded contexts. We call them contracts.

These contracts are necessary because each bounded context contains its own version of the world. That is, its own model and ubiquitous language. In order to integrate, some level of translation needs to happen. They need to be adapted to one another. The contracts define these adaptations.

Domain-Driven Design offers various patterns that are useful for defining contracts between bounded contexts. The decision to use one pattern vs the other depends greatly on the nature of the teams tackling the project. Depending on the teams’ relationship, we can put the patterns in one of three categories: cooperation, customer-supplier, and separate ways.

Integration patterns are determined mainly by the level of cooperation between the teams that own the interacting bounded contexts.

Cooperation patterns

When the components that need to interact are owned by teams which are in close communication, work well together, and whose goals are aligned, cooperation patterns can be applied.

If the teams meet these criteria, a partnership model can be implemented. This is where the integration is managed in an ad-hoc manner. Both teams work together to define the API through which their components interact and that’s that. Whenever changes happen on either side, the other team learns about it quickly and adjusts their code right away. Continuous integration is a great tool here, as breakages are immediately apparent, closing down the feedback loop.

Sometimes, when different bounded contexts need to implement the same functionality, it makes more sense to develop this functionality once and package it as a reusable library, or a sub-module within a shared repository. This is what DDD calls a shared kernel. A shared kernel is effectively a bounded context of its own, one that is statically linked to other components and implements the logic that others depend on.

A shared kernel is its own bounded context, but also “belongs” to multiple other bounded contexts.

The shared kernel integration pattern needs to be used carefully though, as it creates tight coupling between the components that use it. If the involved bounded contexts are owned by different teams, then this shared kernel that emerges violates the DDD principle of bounded contexts having single-team ownership. This is something to watch out for, as bad team synergy can produce problems during development and maintenance.

When using a shared kernel, the quality of the communication and the coordination between the teams has to strike the right balance: when they aren’t strong enough to support a partnership model, but not so weak that the shared kernel would become more trouble than it’s worth.

The alternative to a shared kernel is the duplication of the logic across multiple components. When deciding whether to apply this pattern, the costs of duplication vs coordination need to be considered. That is, the cost of implementing the same logic within multiple components and its subsequent changes, vs the cost of changes in the shared kernel propagating into dependent components and coordinating with the owners of each. When it is cheaper to just duplicate the logic, shared kernel should not be applied.

This means that complicated models that change frequently, like those of core subdomains, are good candidates for shared kernel. Shared kernels allow complexity to be encapsulated and exposed through stable contracts. This means that frequent changes are easier to contain. Frequent changes in logic that’s highly complex and also repeated in multiple places, quickly becomes much more expensive to maintain.

Another good scenario for applying a shared kernel is when refactoring a legacy monolithic system into more separated modules. The legacy system can become the shared kernel during the time that modules are being extracted from it but not yet fully decoupled.

Customer-supplier patterns

Customer-supplier patterns establish a relationship between components where one (the provider, who is “upstream”), provides a service to another (the consumer, who is “downstream”). These types of patterns emerge when the teams who own the involved bounded contexts are not in close collaboration and have their own independent goals.

One form of customer-supplier integration is the conformist pattern. This happens when the supplier defines the integration contract/API, using its own language, concepts and model, and the consumer accepts it. This can happen when the upstream service has a well established or industry-standard model. Or maybe the model is “good enough” for the consumer to interact with directly. Organizational politics may also be a reason for this type of integration, where there is an imbalance of power favoring the supplier’s team, and they get to impose their model or can’t be bothered to adapt it.

Sometimes, the consumer won’t accept the supplier’s model. In that case, an anticorruption layer can be created. An anticorruption layer is implemented by the consumer and translates the supplier’s model into its own. This allows the consumer to use the supplier’s service without polluting its model with extraneous concepts.

Anticorruption layers can be good solutions for the following scenarios:

When the consumer represents a core subdomain. The consumer is solving hard and interesting problems, so it’s best to protect its model with the ACL.
When the supplier’s model is messy and inconvenient. The ACL can protect the consumer’s model from having to contend with a mess of extraneous concepts.
When the supplier’s contract changes frequently. The ACL can protect the consumer from those changes, encapsulating them, allowing it to be less volatile.

A third pattern represents an anticorruption layer of sorts, only built on the supplier’s side. DDD calls this the open-host service pattern. Here, the upstream service comes up with a new language, separate from its own, tailored to its consumers’ convenience, and exposes that as its contract. This public protocol is called a “published language”.

An open-host service provides a protocol for consumers to interact with. It can also support multiple versions of this protocol simultaneously.

An open-host service applies to similar scenarios as an anticorruption layer. Decoupling a service’s internal model from its integration model frees it up for continuous evolution without fear of breaking its consumers. Another advantage of this is that multiple versions of the published language can be exposed, affording clients options on what to support and gradually migrate if they so choose.

Separate ways

Sometimes, the right decision is to not integrate at all. I alluded to this outcome back when discussing the pitfalls of the shared kernel. Sometimes duplication, in spite of how bad it smells, may be the most cost effective solution to a given situation. So the teams go their separate ways.

This can happen when the involved teams cannot collaborate effectively for whatever reason. It could be due to geographical, timezone, or organizational issues.

This can also be a good solution when the repeated logic belongs in a generic subdomain, and it’s easy to integrate. For example consider a logging framework. Exposing that kind of functionality in a service for others to consume, in most cases would be much more trouble than just including some third party library or package.

It may also be that the models being integrated are just so different that they are fundamentally incompatible. It may be cost-prohibitive for collaboration or customer-supplier patterns to be applied; and duplication again, is cheaper.

Going separate ways can be dangerous when we’re talking about core subdomains though. So we have to tread carefully in those scenarios. Remember that models that represent core subdomain should be implemented in the most effective and efficient ways, with few shortcuts and minimized technical debt.

The context map

The context map can be a useful tool for high level design as it plots all the major bounded contexts (i.e. modules, components, subsystems) that we’ve designed and their interaction patterns.

Here’s a context map which captures the various bounded contexts that compose a big system and their interactions. The arrows point to the upstream component in the relationship. “ACL” denotes an anticorruption layer and “OHS” represents an open-host service.

Of course, they can also offer valuable insight into organizational dynamics, as team composition and relationships with others are intrinsic parts of the discussion when talking about bounded contexts. For example, it can show teams that prefer to collaborate closely or at a healthy distance. I can also show problematic components, which are surrounded by anticorruption layers or have had their ties completely cut via a separate ways approach.

As with any document, they run the risk of becoming stale as the system evolves. So it should be a team-wide responsibility to keep it up to date. Each team taking care of their own components and their integration points.

The DDD high level design concept map

These are the main concepts that we’ve explored so far, and how they relate to each other.

Middleware: Is that still a thing?

2022-05-26T00:00:00+00:00

The simple answer to the question in the title is simply, yes! Despite the term being many decades old and well past its hype peak, middleware is still very much a thing and has become a key part of the technical landscape that is critical in day-to-day functioning of systems.

So there are still some questions to be answered: What is middleware? What does it do? And maybe most importantly: Why do we care?

What is it?

In its simplest meaning, middleware is an application that sits between other applications and shuffles data between them. There is normally one system requesting the information and the middleware figures out where to get that requested data and makes the request to another system.

An easy example of this is buying something at a retail store using your credit or debit card. When you swipe your card, the business makes a request to a service (some middleware) to ask if there’s enough room on the card for that purchase. Then that system makes a request to the appropriate bank or card holding company to ask the same question. The bank or card holding company replies with either a yes or no and that answer is then relayed back to the terminal where you swiped your card.

You might be thinking, “Why do we need the middleware? Just have the terminal ask the right bank!” Well, that’s a benefit of middleware: The terminal software doesn’t need to have all the same logic coded into it to know whether or how to talk to Visa or Mastercard as well as Citibank or Capital One or Bank of America to get the answer. It just talks to the middleware and lets it figure all of that out. This way you can have many different types of terminals or credit card taking applications available but only one middleware application needs to have the smarts and authorization to know where and how to get the answer.

The term middleware is most often used to describe business systems internal to a company. It is a subset of the broader concept “API” (application programming interface), which includes services provided for external users, either the general public or specific authorized customers or business partners. If a service is used directly by humans of the general public, it is typically called SaaS (software as a service).

What does it do?

As I explained in the example above, the simplistic definition of middleware is an application that shuffles data between two systems. In reality, middleware is sometimes way more complicated than that!

In that example of a credit card charge terminal, using middleware also allows for the retailer to set centralized company-wide rules for fraud screening, add or remove supported payment methods, or change which payment gateway is used, without having to update each terminal individually.

Middleware also can do data manipulation and mapping between two systems. Let’s consider a more complicated case like booking a cross-country flight.

You’ve probably used sites like Google Flights or Expedia to search for ticket costs. You go to their site, put in your starting airport(s) and your destination airport(s), select the dates when you’d like to travel, and maybe check the box saying you’ve got some wiggle room on when you leave and come back. After that you click the search button and within a few seconds, you’ve got tons of flight options to sort through! The search results can be sorted by price, number of stops, and airlines, to name a few.

So taking a step back to marvel at what just happened, how do you think they got that much data from that many sources so quickly? Well, the answer is middleware.

At some point, a middleware system made a request to each of the airlines to ask for their available flights with their dates, costs, amenities, etc. and stored that information on their own system to be used for your search. There are many different ways that data could be captured. We won’t delve into all that excitement, but what we do know is some system (middleware) had to fetch this data from each of the airlines and store it to be used for your search. It has to organize and store that data in a way that makes sense to the search engine to use so you can get back your search results in a speedy fashion.

There’s a good chance that the data from American Airlines will differ in its structure and data points compared to Delta. It’s the middleware’s job to take each of those data feeds and organize and store them somewhere for the search engine to use. In this part of the example, the middleware is taking a request to fetch the Delta flight data, turning that request into a request to another system for the data then doing whatever data manipulation and storage of that data to satisfy the search engine.

Why do we care?

The main reason that we care is we’ve become used to having a single location (website, application etc.) that allows us access to data from multiple places.

In our credit card example, there are many different types of credit cards we can use to check out at our local gas station. Having them require us to only use a Shell gas card would be bad for their business! We’ve grown accustomed to being able to use our Visa, Mastercard, American Express, or Shell gas card at that same gas station. This is all made possible due to middleware behind the scenes doing the hard work.

In many cases, the function of gathering data from many sources in one place, keeping it current, and allowing it to be searched, sorted, etc. is not just one means to an end, but it actually is the whole business! It is why services like Google Flights and Expedia exist.

Middleware can serve many other purposes even when only used internally by a single company:

authentication — making sure who- or whatever made a request is supposed to be able to
authorization — making sure the kind of request being made is one this user is allowed to
auditing — logging or sampling traffic in middleware is often more comprehensive and convenient than at each service individually
rate-limiting — ensuring the user is not using too many resources too fast, or that one service isn’t overused and interfering with access to other services
caching — storing answers to frequent requests to reuse for a limited time, reducing the number of live requests to busy internal applications and databases
billing — tracking usage so it can be paid for or at least each user’s usage is fairly recorded
fault-tolerance — retrying a request if an internal system is temporarily unavailable, or rerouting requests to a different system, to shield users from outages
load balancing — spreading requests across multiple services to allow greater total throughput
translation — modern APIs typically use JSON to structure their data, and middleware can translate between a nice JSON interface and older systems using a variety of more complex and archaic data formats
migrations — transparently moving from an old system to a newer one, possibly in phases or temporarily for testing, without users having to change anything or even be aware

In closing, while middleware isn’t commonly mentioned or even thought about, it is a very important part of what makes the technological world function.

Cooking with CAS

2020-03-10T00:00:00+00:00

Photo by Flickr user reidrac, licensed under CC BY-SA 2.0

One of our customers asked us to host a new suite of web-based applications for them and to protect them with a single sign-on (SSO) solution. Ok, easy enough; these applications were in fact designed with a particular SSO system in mind already, but our situation required a different one, and we eventually chose Apereo’s open source Central Authentication Server project, or CAS. I’d like to describe the conversion process we went through.

The ingredients

Our customer’s application suite included:

The principal Java application using JAAS authentication
Another Java application based on Spring Security
A pair of PHP applications
A few automated tasks that needed to authenticate.

The original SSO system sets a header on each request, identifying an authenticated user. This requires a gateway system to sanitize request headers to ensure malicious users cannot forge a header themselves. It also requires each application inspect request headers and respond appropriately.

CAS is a bit more complex: applications redirect unauthenticated requests to a CAS server, which authenticates the user through any of various configurable methods. The CAS server then redirects the user back to the original application with a parameter called a “Service Ticket”, a seemingly random number identifying an individual authentication request. The original application contacts the CAS server directly to validate the service ticket and to collect information to identify the user. It can then establish a session for that user, and proceed normally.

To CAS-enable an application, we incorporate one of the CAS client libraries, which exist for various languages. In fact we won’t use the Java client directly, but rather we’ll incorporate components that extend it. When evaluating CAS, I was a bit concerned by what appeared to be a surprisingly limited selection of actively supported client libraries, and of course your results may vary, but we found software to meet our own needs without too much difficulty.

Configuring Wildfly Authentication

The most important application in the suite depends on the JAAS-based security subsystem of the Wildfly application server it’s deployed to. Originally it used a custom LoginModule that inspected request headers to find the ID of the authenticated user. Our first task was to configure our proxy server to remove this header from every request. We planned to disable the old authentication system entirely, of course, but this change ensured that even if we mistakenly left it enabled somewhere, malicious users couldn’t exploit it for access.

This application uses a declarative security policy: the deployment descriptor identifies a set of user roles, and another set of “security constraints”. Each security constraint describes one or more URL patterns used in the application, and the set of user roles allowed to access URLs matching those patterns. Here are two examples:

<!-- clients -->
<security-constraint>
    <web-resource-collection>
        <web-resource-name>Secure Area</web-resource-name>
        <url-pattern>/views/clients/*</url-pattern>
        <url-pattern>/client/edit/*</url-pattern>
        <url-pattern>/client/edit_tree/*</url-pattern>
        <url-pattern>/client/view/*</url-pattern>
        <url-pattern>/client/view/id/*</url-pattern>
    </web-resource-collection>
    <auth-constraint>
        <role-name>client_role</role-name>
    </auth-constraint>
</security-constraint>

<!-- places -->
<security-constraint>
    <web-resource-collection>
        <web-resource-name>Secure Area</web-resource-name>
        <url-pattern>/views/places/*</url-pattern>
    </web-resource-collection>
    <auth-constraint>
        <role-name>manage_places</role-name>
    </auth-constraint>
</security-constraint>

We installed the cas-extension library in our Wildfly server to handle the CAS protocol. When an unauthenticated user attempts to access a URL matching a pattern in one of the application’s security constraints, the CAS extension automatically intercepts control and redirects the user to the CAS server. Assuming the user authenticates successfully, our application will receive another request with a service ticket parameter. The cas-extension intercepts this request as well, validates the service ticket, and creates an “identity assertion”, which it sends to the Wildfly security system. Wildfly’s role mapper queries a database to find the user’s roles, after which the authentication process is complete.

Configuration of the cas-extension begins with a CAS profile, a combination of the URL of the CAS server and the URL of the service the extension should protect.

<subsystem xmlns="urn:soulwing.org:cas:1.0">
    <cas-profile name="default" service-url="https://our.application.server/application" server-url="https://our.cas.server"/>
</subsystem>

This tells the extension where to send authentication requests, where to listen for requests returning from the CAS server, and where to validate service tickets. Next we need to validate that identity assertion, and figure out what roles belong to the user. This happens in a Wildfly security domain.

<security-domain name="MySecurityDomain">
    <authentication>
        <login-module name="IdentityAssertion" code="org.soulwing.cas.jaas.IdentityAssertionLoginModule" flag="required" module="org.soulwing.cas"/>
    </authentication>
    <mapping>
        <mapping-module code="DatabaseRoles" type="role">
            <module-option name="dsJndiName" value="java:/comp/env/jdbc/databaseConnection"/>
            <module-option name="rolesQuery" value="select role from user_roles where user_id = ?"/>
        </mapping-module>
    </mapping>
</security-domain>

The authentication portion of the security domain refers to a JAAS LoginModule shipped with the CAS extension, which simply verifies that the identity assertion comes from the CAS extension and not somewhere else. Then the mapping portion (documented here) looks up the given user in a database to find what roles it should be assigned.

The last piece of the puzzle is a CAS deployment descriptor for our application, which activates cas-extension for that application. In its simplest form, this is an empty file in the right place, but ours ended up a little more complex. It identifies both the CAS profile we want to use (unnecessary in this case, as there’s only one CAS profile on the system, but it helped keep things more clear in our minds), and instructs the extension to load some other libraries into our application.

<?xml version="1.0" encoding="UTF-8"?>
<cas xmlns="urn:soulwing.org:cas:1.0">
  <profile>default</profile>
  <add-api-dependencies/>
</cas>

This configuration proved sufficient to let users log in and use the application, but as is sometimes the case with single sign-on, we needed a little more work to let them log out properly. Each application in the suite sets a cookie in the user’s browser to identify its session. The CAS server likewise sets a cookie. When a user logs out of the application, that application’s session cookie is destroyed, but we also need to destroy the CAS server’s session cookie as well as the other applications’ cookies. Single log-out can be complicated, and I won’t go into the full setup here. One reason the CAS deployment descriptor loads cas-extension libraries was so we could use cas-extension to generate the proper logout URL, and redirect our users to that URL once the application has destroyed its own session.

Configuring Spring Authentication

Another Java-based application in our suite uses Spring Security. This document describes the bulk of the configuration, which seemed less straightforward than for cas-extension, but follows essentially the same mechanism. Here we configured a Spring UserDetailsService to execute the same database query we used above to find the user’s roles. I’m not fully conversant in the large stack of beans Spring uses to manage the process, and it took some time to get this configuration sorted out.

Enter PHP

Two of these applications use PHP, which meant yet another configuration. Apereo maintains a PHP client, which includes several helpful examples. I tracked down the part of the application that authenticates users, and replaced the existing code with calls to phpCAS:

phpCAS::client(CAS_VERSION_2_0, $phpCASHost, $phpCASPort, $phpCASContext);
phpCAS::forceAuthentication();
$_SESSION[EXPORT_SERVERNAME]['umdid'] = phpCAS::getUser();

The forceAuthentication call determines the current phase of the authentication process this request is in, whether it’s unauthenticated, fully authenticated, or requires service ticket validation, and responds appropriately. We then set a session variable to the ID of the authenticated user, which replicates what the original authentication code would have done.

One of these two applications requires authenticated access to a REST API exposed by the Wildfly application. CAS calls this “proxy” authentication, when one application requests access to another. Here, CAS issues not only its usual service ticket, but also a “proxy granting ticket”. When the application wants to use the API, it asks the CAS server for a service ticket, using the proxy granting ticket. The CAS server itself requires some new configuration in this case, but for the PHP code, the only difference in the login phase from the simpler, non-proxy case is that we call phpCAS::proxy instead of phpCAS::client to configure CAS. Later, when calling the service itself, we used more phpCAS services in place of the cURL library the original used.

$service = \phpCAS::getProxiedService(PHPCAS_PROXIED_SERVICE_HTTP_GET);
$service->setUrl($serviceURL);
$service->send();

The return of header authentication

Finally, we have a few automated tasks which use various APIs, and need to authenticate. We can’t redirect requests to a CAS server and expect a user to provide credentials, so we’ve taken our cue from the applications’ original form, and configured CAS to recognize a “trusted header”. We add this header to any requests issued by these automated jobs. Of course, we’ve also configured the proxy to disallow this header from any systems outside our internal network.

A few loose ends

Of course, there were other considerations in this project that I’ve not covered here. Configuring CAS itself wasn’t necessarily straightforward, and included a custom authentication module I hope to describe in a later article. Selecting among CAS server’s available deployment options and fitting the winner into our existing infrastructure in a way that makes it easy to manage and monitor was another task entirely. We needed to customize the server’s default user interface to prevent it from offering users an imaginary method to recover forgotten passwords. The series of redirected browser requests involved in the CAS protocol presents a notable performance impact under some circumstances. And it has taken me no small effort to learn to appreciate the CAS server’s sometimes distressingly circular documentation. But several months into the project, CAS seems to be working well enough for our purposes that other users of the same application suite have begun to express interest.

Multi-Tenant Architecture

2018-03-27T00:00:00+00:00

Photo by Rüdiger Stehn, cropped, CC BY-SA 2.0

Definition

Multi-tenant architecture allows one instance of an application to serve multiple customers/organizations. Each customer/organization is called a tenant. Each has its own apparent separate application and is not aware of the other tenants. The tenant has the ability to customize their own UI, users and groups, etc. Every tenant typically has these features:

View: Tenants can define the overall styling to their application.

Business rules: Tenant can define their own business rules and logic for their application.

Database schema: Tenant can define their own database schema (real or apparent) for the application. They can add/remove database tables, rename database fields, etc.

Users and groups: Tenant can define their own rules to achieve data access control.

Types of multi-tenancy

There are 3 main types of multi-tenant architecture:

(1) Multi-tenancy with a single multi-tenant database: This is the simplest form of multi-tenancy. It uses single application instance and the single database instance to host the tenants and store/retrieve the data. This architecture is highly scalable, and when more tenants are added the database is easily scaled up with more data storage. This architecture is low-cost due to shared resources. Operational complexity is high, especially during application design and setup.

(2) Multi-tenancy with one database per tenant: This is another type of multi-tenancy. It uses a single application instance and an individual database for each tenant. The scalability of this architecture may be lower and the cost higher than multi-tenancy with a single multi-tenant database because each database adds overhead. We can increase the scalability of this architecture by adding more database nodes but it depends on the workload. Operational complexity is low due to usage of individual databases.

(3) Standalone single-tenant app with single-tenant database: In this architecture you install the whole application separately for each tenant. Each tenant has its own app instance as well as database instance. This architecture provides highest level of data isolation. The cost of this architecture is high due to the standalone applications and databases.

How to achieve multi-tenancy in Rails

The Apartment Gem provides tools to help you deal with multiple tenants in your Rails application.

Apartment Gem Installation

Add the following to your Gemfile:

gem 'apartment'

and run

bundle install

Then generate your Apartment config file using

bundle exec rails generate apartment:install

You can create new tenants like this:

Apartment::Tenant.create('tenant_name')

To switch tenants using Apartment, use the following command:

Apartment::Tenant.switch('tenant_name') do
  # ...
end

When the switch is called, all requests coming to ActiveRecord will be routed to the tenant you specify.

Multi-tenancy using subdomain with Apartment Gem

module MyApplication
  class Application < Rails::Application
    config.middleware.use Apartment::Elevators::Subdomain
  end
end

Summary

Try the Apartment gem! And consider the trade-offs of your application and user needs.

The Happy Path: An Interview with Design Strategist & Gallerist Kelani Nichole

2016-11-09T00:00:00+00:00

Kelani Nichole is a digital strategist who is also a contemporary art gallerist. Or maybe vice versa.

She works with the Theresa Neil Strategy + Design agency, doing user-centered design research for (mostly) big products at big companies. Her gallery, Transfer, presents internet and computer-based artworks in its Brooklyn, NY location and as pop-up installations around the country and the world.

We spoke about design methodology, client engagement, and salad at El Rey in Manhattan. We got right into it, so don’t expect much prelude.

Kelani Nichole: Existing designs are often based on legacy, opinions, and politics. The methodology I use is all about collaborative design. It’s part of the engagement from the beginning - you get people in a room drawing pictures together. Then when it comes across their desk as we’re building it, they see their own work in what you’re building, and there’s buy in.

It’s really kind of magic the way that it works. They don’t even realize it’s happening. It’s like the film Inception.

Liz Flyntz: When you come on to a new project do you do a presentation? Or are they just buying you on reputation?

KN: No, when I do new business I’ll go pitch the client. I say, “This is why you should have our services. This is what we’re going to offer you.” We usually do some examples of rough ideas where we can help when we’re selling in the engagement. So that’s all to sell the first project. And then at the end of that process we’ve sold our process and our skills, and then we can say how much our further engagement would be.

LF: What is the initial engagement?

KN: The initial engagement is a rapid four-week UX strategy where we come up with defined, actionable, strategy. Catered to the client with a bit of immersive research. Internal stakeholders brought in for interviews. A good kick off basically for a major project. This kind of engagement is a little different – it’s more UX focused. It’s more about building empathy for the people that you are serving both internally with your organization and externally with your audience as well as doing journey mapping and touch point mapping. So it is an audit, in a sense, but of the specific experience and environment of the product.

LF: How do you get the stakeholders to participate? With you or your organization?

KN: That just totally depends on what you are trying to figure out, how many users there are, and what the circumstances are of engaging them. It could be something like you go to an event and you interview people while they are doing something else and it’s very casual.

LF: But when you do that, you don’t have them use the site? You are not having them engage with the existing product, whatever that may be?

KN: That would only be for usability testing really. The kind of research I do is more about the qualitative… what they are seeing, thinking, feeling, doing. Things like that. I can point you to examples, such as empathy mapping.

So much of any digital experience, is the usability for the people who build it, have to maintain it, and work with it. Those are the people who are like, “oh, there’s people bitching at me because they don’t know where to click! Help me!”

It’s my job to make sure I know that person’s needs, stay on top of their concerns, and make sure that they product can deliver for them. Because they have to use it, far more than the person who clicks a few times to find something once, you know?

LF: So you’re constantly updating the stakeholders. But how do you gather the intel from them?

KN: There’s all these different methodologies and different session styles.

LF: Are you just recording everything and transcribing it?

KN: No, we do almost finished designs, we spend a lot of time talking to each other, that’s what the workshops about, getting people to talk to each other. It’s really amazing how well that works. Amazing.

LF: Do the designers get freaked out that you’re having non-designers make design content?

KN: No. Because we’re just doing like, ideation and sketching, and then it has to be refined. The designers who enter at that point have a lot more to go on then just sitting down, looking at whatever crap ideation they would have otherwise. And in this process, the designers end up with pictures of what these people are actually envisioning, and they can try to work towards what the goal is. They have a lot more to go on. But of course, designers will to change the design from the ideation. No one knows where to put things, and the same horrible ideas come out in all these workshops. It’s amazing.

LF: Like what?

KN: One is, “oh, where’s our help? Maybe we should have a talking head, and you ask it a question!” Without fail, if you’re doing a help pattern in a workshop, that will come out. It’s great the way the design studio works, because what you’re supposed to do is steal the best ideas from your peers, and then let the bad ideas die. And so, always in the first round, the “talking head” will die. It’s great, because the people are working through their own ideas. They’re like, “oh that’s not going to work, because it’s annoying as fuck.”

That’s what design workshops let you do. They work through their own bad ideas, and then you don’t have to hear their bad ideas anymore.

LF: It gets it out of the way. The way that I have always thought about the Internet, is like the logical extension of abstraction. It’s where abstraction rules, where representation doesn’t, or isn’t, needed. In the real world, the history of representation is always extant. Like we see it all the time. The frozen yogurt store is represented by a picture of frozen yogurt. But for software and websites abstraction rules. Having a picture of the closest thing to an abstract concept to represent the abstract concept is not the answer. Especially older, non-web native people, often want to revert to representation. And it’s difficult to draw them back from it. It isn’t always the best way to show someone a path by placing a giant picture of a path next to the path. We all rely on abstraction.

KN: So that is UX’s job, to get the client to understand why that’s a bad idea. And you do it by speaking to the clients. And the best way to do it is like get them to learn how to do it themselves. So in the end they come up with an arrow pointing to the path.

Then they’ll start saying, “What do you think of persona X?” and they’ll try to use that persona’s language. Thinking about how they would interpret features, what they like, what wouldn’t they like, what would distract them, what would they care about.

LF: At the end, do you deliver a written brief? Or do you do a presentation? Or what?

KN: A presentation.

LF: And then do you give the client take away materials? Or are you just like good luck?

KN: We usually give them more. I mean, I usually work with a UI designer. But storyboards are actually a really powerful tool. Especially if they have done UI sketching along with your storyboards, then that’s a full set understanding. They know what they’re going get at that point.

LF: So, the storyboard is like a real storyboard? A little user icon, using the website?

KN: Yeah, like what happens for user before, during, and after. Journey mapping. Not a spreadsheet, but a real storyboard.

LF: So when would IA come in?

KN: At the point where we were probably actually starting to structure the data and the meat of the experience. Like once we knew the product was going to be a website, and we knew generally the parameters in which people would enter the website and in which context they would be looking for information.

LF: And at that point you already have user profiles?

KN: Yeah. And you would already know, for instance, “50% of the usage of this website is driven by email campaigns, reminding parents of dates that they have to come onto the site and do certain things.” Then we would know that that is the main path. We call it the happy path. And so you’re going to focus a lot of your consideration on the website.

LF: The happy path is about the travel?

KN: It’s like the ideal, it’s the idealized simple version. And then you deal with the exceptions from there. It’s how you do prioritization and UI design basically, like, what’s a happy path?

LF: That’s interesting. It’s like the path of least resistance.

KN: And, yeah, you’re going to focus on your highest revenue generating, or like your most important audience. And define their happy path.

Hue’s on First: How we used responsive bulbs to join software and hardware for a busy medical practice

2016-03-14T00:00:00+00:00

In 2014 we began working with a busy bariatric surgery office in Long Island to create a system that would allow the practice to better manage doctor paging and patient wait time. By placing a responsive, color-coded light bulb and tablet outside each examination room, the staff could see which rooms were empty, which were occupied by a patient waiting on a specific doctor, and in which a doctor-patient consultation was in process. Outside each room is a tablet with information including the patient number, the attending doctor’s name, and the wait time.

In addition to providing a comprehensive, granular paging service for doctors, Fast Track also provides feedback to the practice. This feedback includes average patient wait times per doctor, per time of day, and per procedure. This allows the practice to make necessary changes and increase patient satisfaction and peace of mind.

I asked Danny Divita, one of the main developers on this project, to tell us more about the Hue/ FastTrack interface.

LF: Describe the project for which we used Hue bulbs. What were all the pieces that needed fitting together?

DD: The Hue bulbs are being used for a bariatric clinic to alert the staff when pages are sent to of pages to specific rooms. Alongside the lights the API also has to work with a RF device-locating API. The RF API had to be coordinated with the Hue light API to coordinate changing colors, alerts, and states.

LF: Why were Hue bulbs a good solution for the problem of creating an easy-to-understand, integratable visual alert system? Did you consider any alternatives?

DD: From a cost perspective and ease of integration the bulbs were a good solution for this project. There are other solutions, but they are more costly and the integration would have taken longer.

LF: How did you hear about Hue bulbs? Was the API easy to access and work with? How was using Hue’s testing environment?

DD: The client suggested the bulbs during their research before having us develop their application. The API is extremely easy to work with and well documented. The Hue API has a built-in testing feature that made it easy to understand the API and develop around it.

LF: What was the most difficult thing about integrating this technology? Can you think of any other commercial implementations of these bulbs? Specifically in the medical industry?

DD: The most difficult aspect was trying to come up with a good alert (blinking) feature. The API has some built in alerts, but they are limited to a set timed interval. We had to develop around this to ensure the alerts stayed constant and did not time out. Another application for these lights in the medical field could be to indicate severity of a situation or condition. Because the lights can change color, the applications can signal the proper response for the staff.

LF: Did you write any new software for this project? Is that work available to other developers?

DD: We used a third party library that helps to manage stateless workflows. Our application expanded upon this library to design a workflow engine that manages paging the staff, Hue lights, and RF device locating.