Creating a Messaging App Using Spring for Apache Kafka, Part 1
Photo by Click and Learn Photography at Unsplash
This article is part of a series. The GitHub repository with code examples can be found here.
Spring is a popular Java application framework. Apache Kafka is a fault-tolerant, fast, and horizontally scalable distributed stream-message broker. Spring for Apache Kafka applies the overall concepts of Spring to Java applications based on Kafka.
Since Kafka can establish a fast and fault-tolerant stream data pipeline it can be used as an orchestrator. In this article I’ll explain how to create a spring-kafka project, add dependencies and use Kafka to create a messaging app.
Initialize Spring project
Spring projects can be built from scratch using Spring Initializr. I like to keep the default options. Most Spring projects use Maven. I set the group id as
com.endpoint and the artifact as
SpringKafkaMessaging which makes the base package name
When we are done with the initial project setup we press the “GENERATE” button to download an empty Spring Boot project in a zip file. You can then use your favorite IDE to open and start developing your project. I prefer Eclipse for Java projects. Here’s what it looks like when I open the project up:
I won’t address detailed configuration or adding dependencies of Spring and Maven projects in this post. If you are not familiar with Spring and Maven, I recommend that you have a look at the Spring documentation first.
Design and architecture
Before adding the dependencies, including Kafka, we need to make a high-level design of this simple project and figure out how to proceed with development. Messaging apps seem simple at first glance but the architecture behind them can be quite complex.
There are different kinds of technology stacks you can choose from. Which base protocol we choose (XMPP, SIP, or WebSocket) depends on what our app’s aim is. Sometimes multiple protocols can be used and interconnected to provide more features; XMPP is mostly used for chatting, SIP is designed for VoIP and media transfer. We’ll use WebSocket to communicate with Kafka over TCP.
By understanding the architectural model of Kafka, you’ll get an understanding of how Kafka is going to maintain most of the backend processes.
Kafka, as I mentioned previously, is horizontally scalable, and Kafka clusters can grow to span several data sources. Message producers and message consumers (all client messaging apps are both producers and consumers) produce and consume messages through Kafka topics.
So, taking into account the principles for designing the architecture of such a client–server-based messaging app, here are the components and their communication directions:
- Kafka Cluster
- Spring Boot REST API, which will handle user authentication and login
- Persistence (here I chose PostgreSQL)
- Cache (Redis) for fast read-write cache operations
- WebSocket for messaging app clients
After creating a model and components let’s add our dependencies to the
pom.xml file to finish creating our project. Below we add
redis.clients:jedis for the corresponding REST, Kafka, Persistent (JDBC), and Redis components.
<dependencies> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> <dependency> <groupId>org.springframework.kafka</groupId> <artifactId>spring-kafka</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-jdbc</artifactId> </dependency> <dependency> <groupId>redis.clients</groupId> <artifactId>jedis</artifactId> </dependency> <dependency> <groupId>org.projectlombok</groupId> <artifactId>lombok</artifactId> </dependency> <dependency> <groupId>com.fasterxml.jackson.core</groupId> <artifactId>jackson-databind</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> <scope>test</scope> <exclusions> <exclusion> <groupId>org.junit.vintage</groupId> <artifactId>junit-vintage-engine</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.springframework.kafka</groupId> <artifactId>spring-kafka-test</artifactId> <scope>test</scope> </dependency> </dependencies>
Continued in Part 2!
spring java frameworks kafka spring-kafka-series