Kafka with Spring streams step by step guide for Beginners

Kafka with Spring streams step by step guide for Beginners

This post gives a step-by-step tutorial to enable messaging in a microservice using Kafka with Spring Cloud Stream.

This post gives a step-by-step tutorial to enable messaging in a microservice using Kafka with Spring Cloud Stream.

Spring Cloud Stream is a framework under the umbrella project Spring Cloud, which enables developers to build event-driven microservices with messaging systems like Kafka and RabbitMQ.

Asynchronous messaging systems are always an important part of any modern enterprise software solution. The evolution of microservices has shortened the time-to-market for any software product, but this is not possible without the necessary tools and frameworks.

Spring Cloud Stream is a framework built on top of Spring Integration. It integrates with Spring Boot seamlessly to build efficient microservices in less time to connect with shared messaging systems. Spring Cloud Stream provides multiple binder implementations such as Kafka, RabbitMQ and various others. The details are provided here.

Here is a step-by-step tutorial on building a simple microservice application based on Spring Boot and uses Spring Cloud Stream to connect with a Kafka instance.

Getting Started

Install Kafka and create a topic. I am using a Kafka broker running on my local windows machine for this demonstration, but it can be an installation on a Unix machine as well. Steps for Kafka installation on windows machine are provided here.

Create a Spring Boot starter project either using STS IDE or Spring Initializr. I am providing the pom.xml for reference.


<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.1.8.RELEASE</version>
<relativePath /> <!-- lookup parent from repository -->
</parent>
<groupId>com.techwording</groupId>
<artifactId>spring-cloud-stream-kafka-example</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>spring-cloud-stream-kafka-example</name>
<description>Demo project for Spring Cloud Stream and Kafka</description>
<properties>
<java.version>1.8</java.version>
<spring-cloud.version>Greenwich.SR3</spring-cloud.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-binder-kafka</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-binder-kafka-streams</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-test-support</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-dependencies</artifactId>
<version>${spring-cloud.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
</project>

The Spring Cloud Stream project needs to be configured with the Kafka broker URL, topic, and other binder configurations. Below is an example of configuration for the application.



spring:
  cloud:
    stream:
      default-binder: kafka
      kafka:
        binder:
          brokers:
          - localhost:9092
      bindings:
        input:
         binder: kafka
         destination: test
         content-type: text/plain
         group: input-group-1
        output:
          binder: kafka
          destination: test
          group: output-group-1
          content-type: text/plain
					

We will need at least one producer and a consumer to test the message and send and receive operations. Below is the sample code for a producer and consumer in its simplest form, developed using Spring Cloud Stream.


package com.techwording.scs;
import org.springframework.cloud.stream.annotation.EnableBinding;
import org.springframework.cloud.stream.messaging.Source;
@EnableBinding(Source.class)
public class Producer {
private Source mySource;
public Producer(Source mySource) {
super();
this.mySource = mySource;
}
public Source getMysource() {
return mySource;
}
public void setMysource(Source mysource) {
mySource = mySource;
}
}


package com.techwording.scs;
import java.time.Instant;
import java.time.ZoneId;
import java.time.format.DateTimeFormatter;
import java.time.format.FormatStyle;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.cloud.stream.annotation.EnableBinding;
import org.springframework.cloud.stream.annotation.StreamListener;
import org.springframework.cloud.stream.messaging.Sink;
import org.springframework.messaging.handler.annotation.Payload;
@EnableBinding(Sink.class)
public class Consumer {
private static final Logger logger = LoggerFactory.getLogger(Consumer.class);
@StreamListener(target = Sink.INPUT)
public void consume(String message) {
logger.info("recieved a string message : " + message);
}
@StreamListener(target = Sink.INPUT, condition = "headers['type']=='chat'")
public void handle(@Payload ChatMessage message) {
final DateTimeFormatter df = DateTimeFormatter.ofLocalizedTime(FormatStyle.MEDIUM)
.withZone(ZoneId.systemDefault());
final String time = df.format(Instant.ofEpochMilli(message.getTime()));
logger.info("recieved a complex message : [{}]: {}", time, message.getContents());
}
}

We will also create a Rest Controller class, which will accept the message over HTTP and pass it to the producer. This is just to make the testing convenient.


package com.techwording.scs;
import org.springframework.messaging.support.MessageBuilder;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RestController;
@RestController
public class Controller {
private Producer producer;
public Controller(Producer producer) {
super();
this.producer = producer;
}
// get the message as a complex type via HTTP, publish it to broker using spring cloud stream
@RequestMapping(value = "/sendMessage/complexType", method = RequestMethod.POST)
public String publishMessageComplextType(@RequestBody ChatMessage payload) {
payload.setTime(System.currentTimeMillis());
producer.getMysource()
.output()
.send(MessageBuilder.withPayload(payload)
.setHeader("type", "chat")
.build());
return "success";
}
// get the String message via HTTP, publish it to broker using spring cloud stream
@RequestMapping(value = "/sendMessage/string", method = RequestMethod.POST)
public String publishMessageString(@RequestBody String payload) {
// send message to channel
producer.getMysource()
.output()
.send(MessageBuilder.withPayload(payload)
.setHeader("type", "string")
.build());
return "success";
}
}

Run the below maven commands to build and run this project.

mvn clean install
mvn spring-boot:run


Hit the POST endpoint /sendMessage/string and check the application console logs. Here is an example output the application produced when I hit this endpoint with message "hello" in the rest body.

2019-10-01 14:37:22.764  INFO 377456 --- [container-0-C-1] com.techwording.scs.Consumer             : received a string message : {"contents":"hello","time":1569920841187}


Hit the POST endpoint /sendMessage/complexType and check the application console logs.

2019-10-01 14:37:22.773  INFO 377456 --- [container-0-C-1] com.techwording.scs.Consumer             : received a complex message : [2:37:21 PM]: hello


The annotation @EnableBinding takes one or more interfaces as parameters. In this example, we have used Sink and Source interfaces, which declare input and output channels, respectively. You can also define your own interfaces for this purpose.

@StreamListener annotation is a convenient way provided by Spring Cloud Stream for content-based routing. It works based on a pub-sub model, and every @StreamListener receives its own copy of the message.

I have used two stream listeners in this project — one for consuming plain string messages and another one for messages with a complex type, ChatMessage. The producer sends messages attached with a header "type" with a logical value and consumer can apply conditions to filter messages using @StreamListener.

You can find the complete project here.

Angular 8 + Spring Boot Microservices and Spring Cloud

Angular 8 + Spring Boot Microservices and Spring Cloud

Learn to build microservices using Spring Boot and Spring Cloud in this tutorial. We will be using Angular 8, Spring Boot, Spring Zuul, Eureka Cloud, Mysql, Liquibase, Lombok, Gradle, Intellij, NodeJS, Load Balance for implementation. **WHAT YOU...

Learn to build microservices using Spring Boot and Spring Cloud in this tutorial. We will be using Angular 8, Spring Boot, Spring Zuul, Eureka Cloud, Mysql, Liquibase, Lombok, Gradle, Intellij, NodeJS, Load Balance for implementation.

WHAT YOU WILL LEARN

  • To Create Project With Spring Boot
  • Step by step microservice implementation
  • To Create connection between projects with Spring Cloud

To check the complete course details and enrollment click here. https://bit.ly/2M9oiqR

Build a reactive Microservices Architecture using Spring Cloud Gateway, Spring Boot and Spring WebFlux

Build a reactive Microservices Architecture using Spring Cloud Gateway, Spring Boot and Spring WebFlux

In this article, you'll learn how you can build a reactive microservices architecture using Spring Cloud Gateway, Spring Boot, and Spring WebFlux.

Originally published by Matt Raible at https://developer.okta.com

So you wanna go full reactive, eh? Great! Reactive programming is an increasingly popular way to make your applications more efficient. Instead of making a call to a resource and waiting on a response, reactive applications asynchronously receive a response. This allows them to free up processing power, only perform processing when necessary, and scale more effectively than other systems.

The Java ecosystem has its fair share of reactive frameworks, including Play Framework, Ratpack, Vert.x, and Spring WebFlux. Like Reactive programming, a microservices architecture can help large teams scale quickly and is possible to build using any of the awesome aforementioned frameworks.

Today I’d like to show you how you can build a reactive microservices architecture using Spring Cloud Gateway, Spring Boot, and Spring WebFlux. We’ll leverage Spring Cloud Gateway as API gateways are often important components in a cloud-native microservices architecture, providing the aggregation layer for all your backend microservices.

This tutorial shows you how to build a microservice with a REST API that returns a list of new cars. You’ll use Eureka for service discovery and Spring Cloud Gateway to route requests to the microservice. Then you’ll integrate Spring Security so only authenticated users can access your API gateway and microservice.

Prerequisites: HTTPie (or cURL), Java 11+, and an internet connection.

Spring Cloud Gateway vs. Zuul

Zuul is Netflix’s API gateway. First released in 2013, Zuul was not originally reactive, but Zuul 2 is a ground-up rewrite to make it reactive. Unfortunately, Spring Cloud does not support Zuul 2 and it likely never will.

Spring Cloud Gateway is now the preferred API gateway implementation from the Spring Cloud Team. It’s built on Spring 5, Reactor, and Spring WebFlux. Not only that, it also includes circuit breaker integration, service discovery with Eureka, and is much easier to integrate with OAuth 2.0!

Let’s dig in.

Create a Spring Cloud Eureka Server Project

Start by creating a directory to hold all your projects, for example, spring-cloud-gateway. Navigate to it in a terminal window and create a discovery-service project that includes Spring Cloud Eureka Server as a dependency.

http https://start.spring.io/starter.zip javaVersion==11 artifactId==discovery-service \  name==eureka-service baseDir==discovery-service \  dependencies==cloud-eureka-server | tar -xzvf -
The command above uses HTTPie. I highly recommend installing it. You can also use curl. Run curl https://start.spring.io to see the syntax.

Add @EnableEurekaServer on its main class to enable it as a Eureka server.

import org.springframework.cloud.netflix.eureka.server.EnableEurekaServer;

@EnableEurekaServer

@SpringBootApplication

public class EurekaServiceApplication {...}

Add the following properties to the project’s src/main/resources/application.properties file to configure its port and turn off Eureka registration.

server.port=8761
eureka.client.register-with-eureka=false

To make the discovery-service run on Java 11+, add a dependency on JAXB.

<dependency>
    <groupId>org.glassfish.jaxb</groupId>
    <artifactId>jaxb-runtime</artifactId>
</dependency>

Start the project using ./mvnw spring-boot:run or by running it in your IDE.

Create a Spring Cloud Gateway Project

Next, create an api-gateway project that includes a handful of Spring Cloud dependencies.

http https://start.spring.io/starter.zip javaVersion==11 artifactId==api-gateway 
 name==api-gateway baseDir==api-gateway
 dependencies==actuator,cloud-eureka,cloud-feign,cloud-gateway,cloud-hystrix,webflux,lombok | tar -xzvf -

We’ll come back to configuring this project in a minute.

Create a Reactive Microservice with Spring WebFlux

The car microservice will contain a significant portion of this example’s code because it contains a fully-functional REST API that supports CRUD (Create, Read, Update, and Delete).

Create the car-service project using start.spring.io:

http https://start.spring.io/starter.zip javaVersion==11 artifactId==car-service 
 name==car-service baseDir==car-service
 dependencies==actuator,cloud-eureka,webflux,data-mongodb-reactive,flapdoodle-mongo,lombok | tar -xzvf -

The dependencies argument is interesting in this command. You can see that Spring WebFlux is included, as is MongoDB. Spring Data provides reactive drivers for Redis and Cassandra as well.

You may also be interested in R2DBC (Reactive Relational Database Connectivity) - an endeavor to bring a reactive programming API to SQL databases. I did not use it in this example because it’s not yet available on start.spring.io.

Build a REST API with Spring WebFlux

I’m a big fan of VWs, especially classic ones like the bus and the bug. Did you know that VW has a bunch of electric vehicles coming out in the next few years? I’m really excited by the ID Buzz! It has classic curves and is all-electric. It even has 350+ horsepower!

In case you’re not familiar with the ID Buzz, here’s a photo from Volkswagen.

Let’s have some fun with this API example and use the electric VWs for our data set. This API will track the various car names and release dates.

Add Eureka registration, sample data initialization, and a reactive REST API to src/main/java/…​/CarServiceApplication.java:

package com.example.carservice;

import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.boot.ApplicationRunner;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.netflix.eureka.EnableEurekaClient;
import org.springframework.context.annotation.Bean;
import org.springframework.data.annotation.Id;
import org.springframework.data.mongodb.core.mapping.Document;
import org.springframework.data.mongodb.repository.ReactiveMongoRepository;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.*;
import reactor.core.publisher.Flux;
import reactor.core.publisher.Mono;

import java.time.LocalDate;
import java.time.Month;
import java.util.Set;
import java.util.UUID;

@EnableEurekaClient
@SpringBootApplication
@Slf4j
public class CarServiceApplication {

   public static void main(String[] args) {
       SpringApplication.run(CarServiceApplication.class, args);
   }

   @Bean
   ApplicationRunner init(CarRepository repository) {
       // Electric VWs from
https://www.vw.com/electric-concepts/
       // Release dates from
https://www.motor1.com/features/346407/volkswagen-id-price-on-sale/
       Car ID = new Car(UUID.randomUUID(), "ID.", LocalDate.of(2019, Month.DECEMBER, 1));
       Car ID_CROZZ = new Car(UUID.randomUUID(), "ID. CROZZ", LocalDate.of(2021, Month.MAY, 1));
       Car ID_VIZZION = new Car(UUID.randomUUID(), "ID. VIZZION", LocalDate.of(2021, Month.DECEMBER, 1));
       Car ID_BUZZ = new Car(UUID.randomUUID(), "ID. BUZZ", LocalDate.of(2021, Month.DECEMBER, 1));
       Set<Car> vwConcepts = Set.of(ID, ID_BUZZ, ID_CROZZ, ID_VIZZION);

       return args -> {
           repository
                   .deleteAll()
                   .thenMany(
                           Flux
                                   .just(vwConcepts)
                                   .flatMap(repository::saveAll)
                   )
                   .thenMany(repository.findAll())
                   .subscribe(car -> log.info("saving " + car.toString()));
       };
   }
}

@Document
@Data
@NoArgsConstructor
@AllArgsConstructor
class Car {
   @Id
   private UUID id;
   private String name;
   private LocalDate releaseDate;
}

interface CarRepository extends ReactiveMongoRepository<Car, UUID> {
}

@RestController
class CarController {

   private CarRepository carRepository;

   public CarController(CarRepository carRepository) {
       this.carRepository = carRepository;
   }

   @PostMapping("/cars")
   @ResponseStatus(HttpStatus.CREATED)
   public Mono<Car> addCar(@RequestBody Car car) {
       return carRepository.save(car);
   }

   @GetMapping("/cars")
   public Flux<Car> getCars() {
       return carRepository.findAll();
   }

   @DeleteMapping("/cars/{id}")
   public Mono<ResponseEntity<Void>> deleteCar(@PathVariable("id") UUID id) {
       return carRepository.findById(id)
               .flatMap(car -> carRepository.delete(car)
                       .then(Mono.just(new ResponseEntity<Void>(HttpStatus.OK)))
               )
               .defaultIfEmpty(new ResponseEntity<>(HttpStatus.NOT_FOUND));
   }
}

  1. Add the @EnableEurekaClient annotation for service discovery
  2. @Slf4j is a handy annotation from Lombok to enable logging in a class
  3. ApplicationRunner bean to populate MongoDB with default data
  4. Delete all existing data in MongoDB so new data is not additive
  5. Subscribe to results so both deleteAll() and saveAll() are invoked
  6. Car class with Spring Data NoSQL and Lombok annotations to reduce boilerplate
  7. CarRepository interface that extends ReactiveMongoRepository, giving you CRUDability with hardly any code!
  8. CarController class that uses CarRepository to perform CRUD actions
  9. Spring WebFlux returns a Mono publisher for single objects
  10. Return a Flex publisher for multiple objects

You’ll also need to modify the car-service project’s application.properties to set its name and port.

spring.application.name=car-service
server.port=8081

Run MongoDB

The easiest way to run MongoDB is to remove the test scope from the flapdoodle dependency in car-service/pom.xml. This will cause your app to start an embedded MongoDB dependency.

<dependency>
    <groupId>de.flapdoodle.embed</groupId>
    <artifactId>de.flapdoodle.embed.mongo</artifactId>
    <!--<scope>test</scope>-->
</dependency>

You can also install and run MongoDB using Homebrew.

brew tap mongodb/brew
brew install [email protected]
mongod

Or, use Docker:

docker run -d -it -p 27017:27017 mongo

Stream Data with WebFlux

This completes everything you need to do to build a REST API with Spring WebFlux.

"But wait!" you might say. "I thought WebFlux was all about streaming data?"

In this particular example, you can still stream data from the /cars endpoint, but not in a browser.

A browser has no way to consume a stream other than using Server-Sent Events or WebSockets. Non-browser clients however can get a JSON stream by sending an Accept header with a value of application/stream+json .

You could test everything works at this point by firing up your browser and using HTTPie to make requests. However, it’s much better to write automated tests!

Test Your WebFlux API with WebTestClient

WebClient ships as part of Spring WebFlux and can be useful for making reactive requests, receiving responses, and populating objects with the payload. A companion class, WebTestClient, can be used to test your WebFlux API. It contains request methods that are similar to WebClient, as well as methods to check the response body, status, and headers.

Modify the src/test/java/…​/CarServiceApplicationTests.java class in the car-service project to contain the code below.

package com.example.carservice;

import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.http.MediaType;
import org.springframework.test.context.junit4.SpringRunner;
import org.springframework.test.web.reactive.server.WebTestClient;
import reactor.core.publisher.Mono;

import java.time.LocalDate;
import java.time.Month;
import java.util.Collections;
import java.util.UUID;

@RunWith(SpringRunner.class)
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT,
        properties = {"spring.cloud.discovery.enabled = false"})
public class CarServiceApplicationTests {

    @Autowired
    CarRepository carRepository;

    @Autowired
    WebTestClient webTestClient;

    @Test
    public void testAddCar() {
        Car buggy = new Car(UUID.randomUUID(), "ID. BUGGY", LocalDate.of(2022, Month.DECEMBER, 1));

        webTestClient.post().uri("/cars")
                .contentType(MediaType.APPLICATION_JSON_UTF8)
                .accept(MediaType.APPLICATION_JSON_UTF8)
                .body(Mono.just(buggy), Car.class)
                .exchange()
                .expectStatus().isCreated()
                .expectHeader().contentType(MediaType.APPLICATION_JSON_UTF8)
                .expectBody()
                .jsonPath("$.id").isNotEmpty()
                .jsonPath("$.name").isEqualTo("ID. BUGGY");
    }

    @Test
    public void testGetAllCars() {
        webTestClient.get().uri("/cars")
                .accept(MediaType.APPLICATION_JSON_UTF8)
                .exchange()
                .expectStatus().isOk()
                .expectHeader().contentType(MediaType.APPLICATION_JSON_UTF8)
                .expectBodyList(Car.class);
    }

    @Test
    public void testDeleteCar() {
        Car buzzCargo = carRepository.save(new Car(UUID.randomUUID(), "ID. BUZZ CARGO",
                LocalDate.of(2022, Month.DECEMBER, 2))).block();

        webTestClient.delete()
                .uri("/cars/{id}", Collections.singletonMap("id", buzzCargo.getId()))
                .exchange()
                .expectStatus().isOk();
    }
}

To prove it works, run ./mvnw test. Give yourself a pat on the back when your tests pass!

If you’re on Windows, use mvnw test.

Use Spring Cloud Gateway with Reactive Microservices

To edit all three projects in the same IDE window, I find it useful to create an aggregator pom.xml. Create a pom.xml file in the parent directory of your projects and copy the XML below into it.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.okta.developer</groupId>
    <artifactId>reactive-parent</artifactId>
    <version>1.0.0-SNAPSHOT</version>
    <packaging>pom</packaging>
    <name>reactive-parent</name>
    <modules>
        <module>discovery-service</module>
        <module>car-service</module>
        <module>api-gateway</module>
    </modules>
</project>

After creating this file, you should be able to open it in your IDE as a project and navigate between projects easily.

In the api-gateway project, add @EnableEurekaClient to the main class to make it Eureka-aware.

import org.springframework.cloud.netflix.eureka.EnableEurekaClient;

@EnableEurekaClient
@SpringBootApplication
public class ApiGatewayApplication {...}

Then, modify the src/main/resources/application.properties file to configure the application name.

spring.application.name=gateway

Create a RouteLocator bean in ApiGatewayApplication to configure routes. You can configure Spring Cloud Gateway with YAML, but I prefer Java.

package com.example.apigateway;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.gateway.route.RouteLocator;
import org.springframework.cloud.gateway.route.builder.RouteLocatorBuilder;
import org.springframework.cloud.netflix.eureka.EnableEurekaClient;
import org.springframework.context.annotation.Bean;

@EnableEurekaClient
@SpringBootApplication
public class ApiGatewayApplication {

    public static void main(String[] args) {
        SpringApplication.run(ApiGatewayApplication.class, args);
    }

    @Bean
    public RouteLocator customRouteLocator(RouteLocatorBuilder builder) {
        return builder.routes()
                .route("car-service", r -> r.path("/cars")
                        .uri("lb://car-service"))
                .build();
    }
}

After making these code changes, you should be able to start all three Spring Boot apps and hit http://localhost:8080/cars.

$ http :8080/cars
HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8
transfer-encoding: chunked

[
    {
        "id": "ff48f617-6cba-477c-8e8f-2fc95be96416",
        "name": "ID. CROZZ",
        "releaseDate": "2021-05-01"
    },
    {
        "id": "dd6c3c32-724c-4511-a02c-3348b226160a",
        "name": "ID. BUZZ",
        "releaseDate": "2021-12-01"
    },
    {
        "id": "97cfc577-d66e-4a3c-bc40-e78c3aab7261",
        "name": "ID.",
        "releaseDate": "2019-12-01"
    },
    {
        "id": "477632c8-2206-4f72-b1a8-e982e6128ab4",
        "name": "ID. VIZZION",
        "releaseDate": "2021-12-01"
    }
]

Add a REST API to Retrieve Your Favorite Cars

Create a /fave-cars endpoint that strips out cars that aren’t your favorite.

First, add a load-balanced WebClient.Builder bean.

@Bean
@LoadBalanced
public WebClient.Builder loadBalancedWebClientBuilder() {
    return WebClient.builder();
}

Then add a Car POJO and a FaveCarsController below the ApiGatewayApplication class in the same file.

public class ApiGatewayApplication {...}
class Car {...}
class FaveCarsController {...}

Use WebClient to retrieve the cars and filter out the ones you don’t love.

@Data
class Car {
    private String name;
    private LocalDate releaseDate;
}

@RestController
class FaveCarsController {

    private final WebClient.Builder carClient;

    public FaveCarsController(WebClient.Builder carClient) {
        this.carClient = carClient;
    }

    @GetMapping("/fave-cars")
    public Flux<Car> faveCars() {
        return carClient.build().get().uri("lb://car-service/cars")
                .retrieve().bodyToFlux(Car.class)
                .filter(this::isFavorite);
    }

    private boolean isFavorite(Car car) {
        return car.getName().equals("ID. BUZZ");
    }
}

If you’re not using an IDE that auto-imports for you, you’ll want to copy/paste the following into the top of ApiGatewayApplication.java:

import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.reactive.function.client.WebClient;
import reactor.core.publisher.Flux;

Restart your gateway app to see the http://localhost:8080/fave-cars endpoint only returns the ID Buzz.

What about Failover with Hystrix?

Spring Cloud Gateway only supports Hystrix at the time of this writing. Spring Cloud deprecated direct support for Hystrix in favor of Spring Cloud Circuit Breaker. Unfortunately, this library hasn’t had a GA release yet, so I decided not to use it.

To use Hystrix with Spring Cloud Gateway, you can add a filter to your car-service route, like so:

.route("car-service", r -> r.path("/cars")
        .filters(f -> f.hystrix(c -> c.setName("carsFallback")
                .setFallbackUri("forward:/cars-fallback")))
        .uri("lb://car-service/cars"))
.build();

Then create a CarsFallback controller to handle the /cars-fallback route.

@RestController
class CarsFallback {

    @GetMapping("/cars-fallback")
    public Flux<Car> noCars() {
        return Flux.empty();
    }
}

First, restart your gateway and confirm http://localhost:8080/cars works. Then shut down the car service, try again, and you’ll see it now returns an empty array. Restart the car service and you’ll see the list populated again.

You’ve built a resilient and reactive microservices architecture with Spring Cloud Gateway and Spring WebFlux. Now let’s see how to secure it!

What about Feign with Spring Cloud Gateway?

If you’d like to use Feign in a WebFlux app, see the feign-reactive project. I did not have a need for Feign in this particular example.

Secure Spring Cloud Gateway with OAuth 2.0

OAuth 2.0 is an authorization framework for delegated access to APIs. OIDC (or OpenID Connect) is a thin layer on top of OAuth 2.0 that provides authentication. Spring Security has excellent support for both frameworks and so does Okta!

You can use OAuth 2.0 and OIDC without a cloud identity provider by building your own server or by using an open-source implementation. However, wouldn’t you rather just use something that’s always on, like Okta?

If you already have an Okta account, see the Create a Web Application in Okta sidebar below. Otherwise, we created a Maven plugin that configures a free Okta developer account + an OIDC app (in under a minute!).

To use it, add the following plugin repository to your gateway project’s pom.xml:

<pluginRepositories>
    <pluginRepository>
        <id>ossrh</id>
        <releases><enabled>false</enabled></releases>
        <snapshots><enabled>true</enabled></snapshots>
        <url>https://oss.sonatype.org/content/repositories/snapshots</url>
    </pluginRepository>
</pluginRepositories>

Then run ./mvnw com.okta:okta-maven-plugin:setup to create an account and configure your Spring Boot app to work with Okta.

Create a Web Application in Okta

Log in to your Okta Developer account (or sign up if you don’t have an account).

  1. From the Applications page, choose Add Application.
  2. On the Create New Application page, select Web.
  3. Give your app a memorable name, add http://localhost:8080/login/oauth2/code/okta as a Login redirect URI, select Refresh Token (in addition to Authorization Code), and click Done.

Copy the issuer (found under API > Authorization Servers), client ID, and client secret into application.properties for both projects.

okta.oauth2.issuer=$issuer
okta.oauth2.client-id=$clientId
okta.oauth2.client-secret=$clientSecret

Next, add the Okta Spring Boot starter and Spring Cloud Security to your gateway’s pom.xml:

<dependency>
   <groupId>com.okta.spring</groupId>
   <artifactId>okta-spring-boot-starter</artifactId>
   <version>1.2.1</version>
</dependency>
<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-security</artifactId>
</dependency>

This is all you need to do to add OIDC login with Okta! Restart your Gateway app and navigate to http://localhost:8080/fave-cars in your browser to be redirected to Okta for user authorization.

Make Your Gateway an OAuth 2.0 Resource Server

You likely won’t build the UI for your app on the gateway itself. You’ll probably use a SPA or mobile app instead. To configure your gateway to operate as a resource server (that looks for an Authorization header with a bearer token), add a new SecurityConfiguration class in the same directory as your main class.

package com.example.apigateway;

import org.springframework.context.annotation.Bean;
import org.springframework.security.config.annotation.method.configuration.EnableReactiveMethodSecurity;
import org.springframework.security.config.annotation.web.reactive.EnableWebFluxSecurity;
import org.springframework.security.config.web.server.ServerHttpSecurity;
import org.springframework.security.web.server.SecurityWebFilterChain;

@EnableWebFluxSecurity
@EnableReactiveMethodSecurity
public class SecurityConfiguration {

    @Bean
    public SecurityWebFilterChain securityWebFilterChain(ServerHttpSecurity http) {
        // @formatter:off
        http
            .authorizeExchange()
                .anyExchange().authenticated()
                .and()
            .oauth2Login()
                .and()
            .oauth2ResourceServer()
                .jwt();
        return http.build();
        // @formatter:on
    }
}

CORS with Spring Cloud Gateway

If you’re using a SPA for your UI, you’ll want to configure CORS as well. You can do this by adding a CorsWebFilter bean to this class.

@Bean
CorsWebFilter corsWebFilter() {
    CorsConfiguration corsConfig = new CorsConfiguration();
    corsConfig.setAllowedOrigins(List.of(""));
    corsConfig.setMaxAge(3600L);
    corsConfig.addAllowedMethod("
");
    corsConfig.addAllowedHeader("*");

    UrlBasedCorsConfigurationSource source = new UrlBasedCorsConfigurationSource();
    source.registerCorsConfiguration("/**", corsConfig);

    return new CorsWebFilter(source);
}

Make sure your imports match the ones below.

import org.springframework.web.cors.CorsConfiguration;
import org.springframework.web.cors.reactive.CorsWebFilter;
import org.springframework.web.cors.reactive.UrlBasedCorsConfigurationSource;

Spring Cloud Gateway’s documentation explains how to configure CORS with YAML or with WebFluxConfigurer. Unfortunately, I was unable to get either one to work.

Test Your Gateway with WebTestClient and JWT

If you configured CORS in your gateway, you can test it works with WebTestClient. Replace the code in ApiGatewayApplicationTests with the following.

package com.example.apigateway;

import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.boot.test.mock.mockito.MockBean;
import org.springframework.http.HttpHeaders;
import org.springframework.security.oauth2.jwt.Jwt;
import org.springframework.security.oauth2.jwt.ReactiveJwtDecoder;
import org.springframework.test.context.junit4.SpringRunner;
import org.springframework.test.web.reactive.server.WebTestClient;
import reactor.core.publisher.Mono;

import java.util.Collections;
import java.util.Map;
import java.util.function.Consumer;

import static org.mockito.ArgumentMatchers.anyString;
import static org.mockito.Mockito.when;

@RunWith(SpringRunner.class)
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT,
        properties = {"spring.cloud.discovery.enabled = false"})
public class ApiGatewayApplicationTests {

    @Autowired
    WebTestClient webTestClient;

    @MockBean
    ReactiveJwtDecoder jwtDecoder;

    @Test
    public void testCorsConfiguration() {
        Jwt jwt = jwt();
        when(this.jwtDecoder.decode(anyString())).thenReturn(Mono.just(jwt));
        WebTestClient.ResponseSpec response = webTestClient.put().uri("/")
                .headers(addJwt(jwt))
                .header("Origin", "http://example.com")
                .exchange();

        response.expectHeader().valueEquals("Access-Control-Allow-Origin", "*");
    }

    private Jwt jwt() {
        return new Jwt("token", null, null,
                Map.of("alg", "none"), Map.of("sub", "betsy"));
    }

    private Consumer<HttpHeaders> addJwt(Jwt jwt) {
        return headers -> headers.setBearerAuth(jwt.getTokenValue());
    }
}

  1. Mock ReactiveJwtDecoder so you can set expectations and return mocks when it decodes
  2. Create a new JWT
  3. Return the same JWT when it’s decoded
  4. Add the JWT to the Authorization header with a Bearer prefix

I like how WebTestClient allows you to set the security headers so easily!

You’ve configured Spring Cloud Gateway to use OIDC login and function as an OAuth 2.0 resource server, but the car service is still available on port 8081. Let’s fix that so only the gateway can talk to it.

Secure Gateway to Microservice Communication

Add the Okta Spring Boot starter to car-service/pom.xml:

<dependency>
    <groupId>com.okta.spring</groupId>
    <artifactId>okta-spring-boot-starter</artifactId>
    <version>1.2.1</version>
</dependency>

Copy the okta.* properties from the gateway’s application.properties to the car service’s. Then create a SecurityConfiguration class to make the app an OAuth 2.0 resource server.

package com.example.carservice;

import com.okta.spring.boot.oauth.Okta;
import org.springframework.context.annotation.Bean;
import org.springframework.security.config.annotation.method.configuration.EnableReactiveMethodSecurity;
import org.springframework.security.config.annotation.web.reactive.EnableWebFluxSecurity;
import org.springframework.security.config.web.server.ServerHttpSecurity;
import org.springframework.security.web.server.SecurityWebFilterChain;

@EnableWebFluxSecurity
@EnableReactiveMethodSecurity
public class SecurityConfiguration {

    @Bean
    public SecurityWebFilterChain securityWebFilterChain(ServerHttpSecurity http) {
        // @formatter:off
        http
            .authorizeExchange()
                .anyExchange().authenticated()
                .and()
            .oauth2ResourceServer()
                .jwt();

        Okta.configureResourceServer401ResponseBody(http);

        return http.build();
        // @formatter:on
    }
}

That’s it! Restart your car service application and it’s now protected from anonymous intruders.

$ http :8081/cars
HTTP/1.1 401 Unauthorized
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Content-Type: text/plain
...

401 Unauthorized

Test Your Microservice with WebTestClient and JWT

The tests you added in the car-service project will no longer work now that you’ve enabled security. Modify the code in CarServiceApplicationTests.java to add JWT access tokens to each request.

package com.example.carservice;

import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.boot.test.mock.mockito.MockBean;
import org.springframework.http.HttpHeaders;
import org.springframework.http.MediaType;
import org.springframework.security.oauth2.jwt.Jwt;
import org.springframework.security.oauth2.jwt.ReactiveJwtDecoder;
import org.springframework.test.context.junit4.SpringRunner;
import org.springframework.test.web.reactive.server.WebTestClient;
import reactor.core.publisher.Mono;

import java.time.LocalDate;
import java.time.Month;
import java.util.Map;
import java.util.UUID;
import java.util.function.Consumer;

import static org.mockito.ArgumentMatchers.anyString;
import static org.mockito.Mockito.when;

@RunWith(SpringRunner.class)
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT,
        properties = {"spring.cloud.discovery.enabled = false"})
public class CarServiceApplicationTests {

    @Autowired
    CarRepository carRepository;

    @Autowired
    WebTestClient webTestClient;

    @MockBean
    ReactiveJwtDecoder jwtDecoder;

    @Test
    public void testAddCar() {
        Car buggy = new Car(UUID.randomUUID(), "ID. BUGGY", LocalDate.of(2022, Month.DECEMBER, 1));

        Jwt jwt = jwt();
        when(this.jwtDecoder.decode(anyString())).thenReturn(Mono.just(jwt));

        webTestClient.post().uri("/cars")
                .contentType(MediaType.APPLICATION_JSON_UTF8)
                .accept(MediaType.APPLICATION_JSON_UTF8)
                .headers(addJwt(jwt))
                .body(Mono.just(buggy), Car.class)
                .exchange()
                .expectStatus().isCreated()
                .expectHeader().contentType(MediaType.APPLICATION_JSON_UTF8)
                .expectBody()
                .jsonPath("$.id").isNotEmpty()
                .jsonPath("$.name").isEqualTo("ID. BUGGY");
    }

    @Test
    public void testGetAllCars() {
        Jwt jwt = jwt();
        when(this.jwtDecoder.decode(anyString())).thenReturn(Mono.just(jwt));

        webTestClient.get().uri("/cars")
                .accept(MediaType.APPLICATION_JSON_UTF8)
                .headers(addJwt(jwt))
                .exchange()
                .expectStatus().isOk()
                .expectHeader().contentType(MediaType.APPLICATION_JSON_UTF8)
                .expectBodyList(Car.class);
    }

    @Test
    public void testDeleteCar() {
        Car buzzCargo = carRepository.save(new Car(UUID.randomUUID(), "ID. BUZZ CARGO",
                LocalDate.of(2022, Month.DECEMBER, 2))).block();

        Jwt jwt = jwt();
        when(this.jwtDecoder.decode(anyString())).thenReturn(Mono.just(jwt));

        webTestClient.delete()
               .uri("/cars/{id}", Map.of("id", buzzCargo.getId()))
                .headers(addJwt(jwt))
                .exchange()
                .expectStatus().isOk();
    }

    private Jwt jwt() {
        return new Jwt("token", null, null,
                Map.of("alg", "none"), Map.of("sub", "dave"));
    }

    private Consumer<HttpHeaders> addJwt(Jwt jwt) {
        return headers -> headers.setBearerAuth(jwt.getTokenValue());
    }
}

Run the test again and everything should pass!

Mock JWT Support in Spring Security 5.2

Kudos to Josh Cummings for his help with JWTs and WebTestClient. Josh gave me a preview of the mock JWT support coming in Spring Security 5.2.

this.webTestClient.mutateWith(jwt()).post(...)

Josh also provided an example test showing how to mock a JWT’s subject, scope, and claims. This code is based on new functionality in Spring Security 5.2.0.M3.

The future is bright for OAuth 2.0 and JWT support in Spring Security land! 😎

Relay the Access Token: Gateway to Microservice

You only need to make one small change for your gateway to talk to this protected service. It’s incredibly easy and I ❤️ it!

In ApiGatewayApplication.java, add a filter that applies the TokenRelayGatewayFilterFactory from Spring Cloud Security.

import org.springframework.cloud.security.oauth2.gateway.TokenRelayGatewayFilterFactory;

@Bean
public RouteLocator customRouteLocator(RouteLocatorBuilder builder,
                                       TokenRelayGatewayFilterFactory filterFactory) {
    return builder.routes()
            .route("car-service", r -> r.path("/cars")
                    .filters(f -> f.filter(filterFactory.apply()))
                    .uri("lb://car-service/cars"))
            .build();
}

This relay factory does not automatically refresh access tokens (yet).

Restart your API gateway and you should be able to view http://localhost:8080/cars and have everything work as expected.

Pretty sweet, don’t you think?!

Thanks for reading

If you liked this post, share it with all of your programming buddies!

Follow us on Facebook | Twitter

Further reading about Microservices

An Introduction to Microservices

What is Microservices?

Build Spring Microservices and Dockerize Them for Production

Best Java Microservices Interview Questions In 2019

Build a microservices architecture with Spring Boot and Spring Cloud

Design patterns for microservices 🍂 🍂 🍂

Kotlin Microservices With Micronaut, Spring Cloud, and JPA

Build Spring Microservices and Dockerize Them for Production

Secure Service-to-Service Spring Microservices with HTTPS and OAuth 2.0

Build Secure Microservices with AWS Lambda and ASP.NET Core


Kafka Spark Streaming | Kafka Tutorial

Kafka Spark Streaming | Kafka Tutorial

🔥Intellipaat Kafka training: https://intellipaat.com/kafka-training-online/ 👉In this kafka spark streaming tutorial you will learn what is apache kafka, arch...

In this kafka spark streaming tutorial you will learn what is apache kafka, architecture of apache kafka & how to setup a kafka cluster, what is spark & it's features, components of spark and hands on demo on integrating spark streaming with apache kafka and integrating spark flume with apache kafka.