How to read Kafka Message Key from Spring cloud streams?

<strong><em>enter code here</em></strong> I am using spring cloud streams to consume a message from Kafka.

enter code here I am using spring cloud streams to consume a message from Kafka.

Is it possible to read the Kafka Message Key from the code?

I have a Kafka topic that generally has 2 types of messages. The action to be taken varies depending on the message key. I see the spring documentation has only the following to read the message. Here, I need to specify the actual mapping of the message (Greetings class here). However, I need a way through which I can read the message key and determine the deserializable Pojo

public class GreetingsListener {

public void handleGreetings(@Payload Greetings request) {


Apache Spark Tutorial - Apache Spark Full Course - Learn Apache Spark

Apache Spark Tutorial - Apache Spark Full Course - Learn Apache Spark

This video will help you understand and learn Apache Spark in detail. This Spark tutorial is ideal for both beginners as well as professionals who want to master Apache Spark concepts.

This video will help you understand and learn Apache Spark in detail. This Spark tutorial is ideal for both beginners as well as professionals who want to master Apache Spark concepts. Below are the topics covered in this Spark tutorial for beginners:

2:44 Introduction to Apache Spark

3:49 What is Spark?

5:34 Spark Eco-System

7:44 Why RDD?

16:44 RDD Operations

18:59 Yahoo Use-Case

21:09 Apache Spark Architecture

24:24 RDD

26:59 Spark Architecture

31:09 Demo

39:54 Spark RDD

41:09 Spark Applications

41:59 Need For RDDs

43:34 What are RDDs?

44:24 Sources of RDDs

45:04 Features of RDDs

46:39 Creation of RDDs

50:19 Operations Performed On RDDs

50:49 Narrow Transformations

51:04 Wide Transformations

51:29 Actions

51:44 RDDs Using Spark Pokemon Use-Case

1:05:19 Spark DataFrame

1:06:54 What is a DataFrame?

1:08:24 Why Do We Need Dataframes?

1:09:54 Features of DataFrames

1:11:09 Sources Of DataFrames

1:11:34 Creation Of DataFrame

1:24:44 Spark SQL

1:25:14 Why Spark SQL?

1:27:09 Spark SQL Advantages Over Hive

1:31:54 Spark SQL Success Story

1:33:24 Spark SQL Features

1:37:15 Spark SQL Architecture

1:39:40 Spark SQL Libraries

1:42:15 Querying Using Spark SQL

1:45:50 Adding Schema To RDDs

1:55:05 Hive Tables

1:57:50 Use Case: Stock Market Analysis with Spark SQL

2:16:50 Spark Streaming

2:18:10 What is Streaming?

2:25:46 Spark Streaming Overview

2:27:56 Spark Streaming workflow

2:31:21 Streaming Fundamentals

2:33:36 DStream

2:38:56 Input DStreams

2:40:11 Transformations on DStreams

2:43:06 DStreams Window

2:47:11 Caching/Persistence

2:48:11 Accumulators

2:49:06 Broadcast Variables

2:49:56 Checkpoints

2:51:11 Use-Case Twitter Sentiment Analysis

3:00:26 Spark MLlib

3:00:31 MLlib Techniques

3:01:46 Demo

3:11:51 Use Case: Earthquake Detection Using Spark

3:24:01 Visualizing Result

3:25:11 Spark GraphX

3:26:01 Basics of Graph

3:27:56 Types of Graph

3:38:56 GraphX

3:40:42 Property Graph

3:48:37 Creating & Transforming Property Graph

3:56:17 Graph Builder

4:02:22 Vertex RDD

4:07:07 Edge RDD

4:11:37 Graph Operators

4:24:37 GraphX Demo

4:34:24 Graph Algorithms

4:34:40 PageRank

4:38:29 Connected Components

4:40:39 Triangle Counting

4:44:09 Spark GraphX Demo

4;57:54 MapReduce vs Spark

5:13:03 Kafka with Spark Streaming

5:23:38 Messaging System

5:21:15 Kafka Components

2:23:45 Kafka Cluster

5:24:15 Demo

5:48:56 Kafka Spark Streaming Demo

6:17:16 PySpark Tutorial

6:21:26 PySpark Installation

6:47:06 Spark Interview Questions

Spring Boot With Spring Data JPA

Spring Boot With Spring Data JPA

Welcome to the Spring Boot with Spring Data JPA tutorial! In this tutorial, we are going to see how Spring Data JPA provides complete abstraction over the DAO layer. We don’t need to write the implementation for the DAO layer anymore; Spring Data auto-generates the implementation DAO implementations.

Welcome to the Spring Boot with Spring Data JPA tutorial! In this tutorial, we are going to see how Spring Data JPA provides complete abstraction over the DAO layer. We don’t need to write the implementation for the DAO layer anymore; Spring Data auto-generates the implementation DAO implementations.

We already had an introduction to Spring Boot, and for this tutorial, we will use Spring Boot along with Spring Data. You will also see how Spring Boot auto-configuration helps to get data source configurations done, hassle free.

In our tutorial Spring Boot Rest Service, we created a DogService, which included a simple CRUD service based on the Mock Data Provider. We will use the same DogService and replace the Mock Data Provider with the actual MySQL database along with Spring Data and JPA.

Datasource Configuration

We now have dependencies configured. It is not time to tell which data source to connect to. Here is my application.yml, with Spring Boot data source entries.

    url: jdbc:mysql://localhost:33099/dogs
    password: <ENTER _ PASSWORD _ HERE >
    username: root
    driver-class-name: "com.mysql.jdbc.Driver"
    database-platform: org.hibernate.dialect.MySQL5InnoDBDialect
      ddl-auto: update

Here, we have the specified JDBC URL, username, password, and driver class name (MySQL).

Apart from this, there are JPA specific configurations. First is the database-platform_,_ which is tells us the underlying Hibernate features to consider under the MySQL query dialect. This is so that all the database operations will be handled in MySQL specific syntax. The second JPA configuration is ddl-auto, which tells Hibernate to create the respective database and table structure, if not already present.

When this option is turned on, Hibernate will create the database structure based on the Entity Beans and the data source.

Entity Bean

The first code level thing we will do is write an Entity Bean. Here is what the Oracle Documentation says about Entity Beans.

Using JPA, you can designate any POJO class as a JPA entity – a Java object whose nontransient fields should be persisted to a relational database using the services of an entity manager obtained from a JPA persistence provider (either within a Java EE EJB container or outside of an EJB container in a Java SE application).
In simpler words, JPA Entity is any Java POJO, which can represent the underlying table structure. As our service is based on theDog table, we will create a Dog Entity object.

package com.amitph.spring.dogs.repo;

import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
import javax.persistence.Id;

public class Dog {
    private long id;
    private String name;
    private int age;

    public long getId() {
        return id;

    public void setId(long id) { = id;

    public String getName() {
        return name;

    public void setName(String name) { = name;

    public int getAge() {
        return age;

    public void setAge(int age) {
        this.age = age;

The above POJO is annotated with @Entity, which is to denote this is an entity object for the table name Dog.

Then, there are three fields that represent the datable table columns. Field id is our Primary Key and, hence, marked as @Id.

The field id is also marked with @GeneratedValue, which denotes that this is an Auto-Increment column and Hibernate will take care of putting in the next value. Hibernate will first query the underlying table to know the max value of the column and increment it with next insert. This also means that we don’t need to specify any value for the Id column and can leave it blank.

Repository Interface

The Repository represents the DAO layer, which typically does all the database operations. Thanks to Spring Data, who provides the implementations for these methods. Let’s have a look at our DogsRepoisitory*,* which extends the CrudRepository:

package com.amitph.spring.dogs.repo;

import org.springframework.stereotype.Repository;

public interface DogsRepository extends CrudRepository<Dog, Long> {}

There are no method declarations here in the DogsRepository. That is because Spring Data’s CrudInterface has already declared basic CRUD methods.

Here, we are done with the JPA and Spring data things — in other words, the DAO layer. Let’s now write a simple Service Layer and a Controller.

Controller and Service Layer

As we have our data access layer done, we will write our controller and service layer. Notice that the DogsRepository is annotated with @Repository, which also adds it to the Spring Context. We can now Autowire the repository in Service.

Dogs Service

This class has simple CRUD methods. It also converts the Entity bean to a DTO (data transfer object). DTO is also a simple Java POJO, which is used to transfer data between systems. Here, we are returning DTOs from our REST endpoints.

package com.amitph.spring.dogs.service;

import com.amitph.spring.dogs.model.DogDto;
import com.amitph.spring.dogs.repo.Dog;
import com.amitph.spring.dogs.repo.DogsRepository;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;

import java.util.List;
import java.util.Optional;

public class DogsService {
    @Autowired DogsRepository repository;

    public void add(DogDto dto) {;

    public void delete(long id) {

    public List<Dog> getDogs() {
        return (List<Dog>) repository.findAll();

    public Dog getDogById(long id) {
        Optional<Dog> optionalDog = repository.findById(id);
        return optionalDog.orElseThrow(() -> new DogNotFoundException("Couldn't find a Dog with id: " + id));

    private Dog toEntity(DogDto dto) {
        Dog entity = new Dog();
        return entity;

Dogs Controller

The Dogs Controller is a standard REST controller with simple CRUD endpoints. The job of the controller is to handle the HTTP requests and invoke the Service class methods.

package com.amitph.spring.dogs.web;

import com.amitph.spring.dogs.model.DogDto;
import com.amitph.spring.dogs.repo.Dog;
import com.amitph.spring.dogs.service.DogsService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.DeleteMapping;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

import java.util.List;

public class DogsController {
    @Autowired DogsService service;

    public List<Dog> getDogs() {
        return service.getDogs();

    public void postDogs(@RequestBody DogDto dto) {

    public Dog getById(@PathVariable(required = true) long id) {
        return service.getDogById(id);

    public void delete(@PathVariable(required = true) long id) {

Now, the Dogs Service is ready to run. Start the application and execute the HTTP endpoints — that’s it.


This is the end of our Spring Boot with Spring data and JPA tutorial. We saw how to use Spring Data’s abstraction for the Data Access Layer. We saw how to represent a database table in the form of Entity Bean and how to Use Spring Data’s autogenerated repository implementations. Additionally, we also saw how to use Spring Boot to do automatic datasource configurations.

In the Spring Boot Rest Service post, we have already seen creating a RESTful web-service with Spring Boot. In the current article, we did not care about Exception Handling. Visit Spring Rest Service Exception Handling to learn about handling exceptions. We also skipped the unit testing part here, which will be covered in upcoming articles.

For the full source code and examples used here, please visit

Spring Boot vs. Spring MVC vs. Spring: How Do They Compare?

Spring Boot vs. Spring MVC vs. Spring: How Do They Compare?

In this article, you will receive overviews of Spring, Spring MVC, and Spring Boot, learn what problems they solve, and where they’re best applied. The most important thing that you will learn is that Spring, Spring MVC, and Spring Boot are not competing for the same space. They solve different problems and they solve them very well.

In this article, you will receive overviews of Spring, Spring MVC, and Spring Boot, learn what problems they solve, and where they’re best applied. The most important thing that you will learn is that Spring, Spring MVC, and Spring Boot are not competing for the same space. They solve different problems and they solve them very well.

Spring, Spring Boot, Spring MVC, there are springs everywhere! Let's walk through where and when you should apply each of these tools.

What Is the Core Problem That Spring Framework Solves?

Think long and hard. What’s the problem Spring Framework solves?

The most important feature of Spring Framework is Dependency Injection. At the core of all Spring Modules is Dependency Injection or IOC Inversion of Control.
Why is this important? Because, when DI or IOC is used properly, we can develop loosely coupled applications. And loosely coupled applications can be easily unit tested.

Let’s consider a simple example.

Example Without Dependency Injection

Consider the example below: WelcomeController depends on WelcomeService to get the welcome message. What is it doing to get an instance of WelcomeService?

WelcomeService service = new WelcomeService();

It’s creating an instance of it. And that means they are tightly coupled. For example: If I create a mock for WelcomeService in a unit test for WelcomeController, how do I make WelcomeController use the mock? Not easy!

public class WelcomeController {

    private WelcomeService service = new WelcomeService();

    public String welcome() {
        return service.retrieveWelcomeMessage();

Same Example with Dependency Injection

The world looks much simpler with dependency injection. You let the Spring Framework do the hard work. We just use two simple annotations: @Component and @Autowired.

  • Using @Component, we tell Spring Framework: Hey there, this is a bean that you need to manage.
  • Using @Autowired, we tell Spring Framework: Hey find the correct match for this specific type and autowire it in.

In the example below, Spring framework would create a bean for WelcomeService and autowire it into WelcomeController.

In a unit test, I can ask the Spring framework to auto-wire the mock of WelcomeService into WelcomeController. (Spring Boot makes things easy to do this with @MockBean. But, that’s a different story altogether!)

public class WelcomeService {
    //Bla Bla Bla

public class WelcomeController {

    private WelcomeService service;

    public String welcome() {
        return service.retrieveWelcomeMessage();

What Else Does Spring Framework Solve?

Problem 1: Duplication/Plumbing Code

Does Spring Framework stop with Dependency Injection? No. It builds on the core concept of Dependency Injection with a number of Spring Modules

  • Spring JDBC
  • Spring MVC
  • Spring AOP
  • Spring ORM
  • Spring JMS
  • Spring Test

Consider Spring JMS and Spring JDBC for a moment.

Do these modules bring in any new functionality? No. We can do all this with J2EE or Java EE. So, what do these bring in? They bring in simple abstractions. The aim of these abstractions is to

  • Reduce Boilerplate Code/Reduce Duplication
  • Promote Decoupling/Increase Unit Testability

For example, you need much less code to use a JDBCTemplate or a JMSTemplate compared to a traditional JDBC or JMS.

Problem 2: Good Integration With Other Frameworks

The great thing about Spring Framework is that it does not try to solve problems that are already solved. All that it does is to provide a great integration with frameworks which provide great solutions.

  • Hibernate for ORM
  • iBatis for Object Mapping
  • JUnit and Mockito for Unit Testing
What Is the Core Problem That Spring MVC Framework Solves?

Spring MVC Framework provides decoupled way of developing web applications. With simple concepts like Dispatcher Servlet, ModelAndView and View Resolver, it makes it easy to develop web applications.## Why Do We Need Spring Boot?

Spring based applications have a lot of configuration.

When we use Spring MVC, we need to configure component scan, dispatcher servlet, a view resolver, web jars(for delivering static content) among other things.

        <property name="prefix">
        <property name="suffix">

  <mvc:resources mapping="/webjars/**" location="/webjars/"/>

The code snippet below shows the typical configuration of a dispatcher servlet in a web application.



When we use Hibernate/JPA, we would need to configure a datasource, an entity manager factory, a transaction manager among a host of other things.

    <bean id="dataSource" class="com.mchange.v2.c3p0.ComboPooledDataSource"
        <property name="driverClass" value="${db.driver}" />
        <property name="jdbcUrl" value="${db.url}" />
        <property name="user" value="${db.username}" />
        <property name="password" value="${db.password}" />

    <jdbc:initialize-database data-source="dataSource">
        <jdbc:script location="classpath:config/schema.sql" />
        <jdbc:script location="classpath:config/data.sql" />

        <property name="persistenceUnitName" value="hsql_pu" />
        <property name="dataSource" ref="dataSource" />

    <bean id="transactionManager" class="org.springframework.orm.jpa.JpaTransactionManager">
        <property name="entityManagerFactory" ref="entityManagerFactory" />
        <property name="dataSource" ref="dataSource" />

    <tx:annotation-driven transaction-manager="transactionManager"/>

Problem #1: Spring Boot Auto Configuration: Can We Think Different?

Spring Boot brings a new thought process around this.

Can we bring more intelligence into this? When a spring mvc jar is added into an application, can we auto configure some beans automatically?* How about auto-configuring a Data Source if Hibernate jar is on the classpath?

  • How about auto-configuring a Dispatcher Servlet if Spring MVC jar is on the classpath?

There would be provisions to override the default auto configuration.

Spring Boot looks at a) Frameworks available on the CLASSPATH b) Existing configuration for the application. Based on these, Spring Boot provides basic configuration needed to configure the application with these frameworks. This is called Auto Configuration.### Problem #2: Spring Boot Starter Projects: Built Around Well-Known Patterns

Let’s say we want to develop a web application.

First of all, we would need to identify the frameworks we want to use, which versions of frameworks to use and how to connect them together.

All web application have similar needs. Listed below are some of the dependencies we use in our Spring MVC Course. These include Spring MVC, Jackson Databind (for data binding), Hibernate-Validator (for server side validation using Java Validation API) and Log4j (for logging). When creating this course, we had to choose the compatible versions of all these frameworks.





Here’s what the Spring Boot documentations says about starters.

Starters are a set of convenient dependency descriptors that you can include in your application. You get a one-stop-shop for all the Spring and related technology that you need, without having to hunt through sample code and copy paste loads of dependency descriptors. For example, if you want to get started using Spring and JPA for database access, just include the spring-boot-starter-data-jpa dependency in your project, and you are good to go.
Let’s consider an example starter: Spring Boot Starter Web.

If you want to develop a web application or an application to expose restful services, Spring Boot Start Web is the starter to pick. Let’s create a quick project with Spring Boot Starter Web using Spring Initializr.

Dependency for Spring Boot Starter Web


The following screenshot shows the different dependencies that are added into our application

Dependencies can be classified into:

  • Spring: core, beans, context, aop
  • Web MVC: (Spring MVC)
  • Jackson: for JSON Binding
  • Validation: Hibernate Validator, Validation API
  • Embedded Servlet Container: Tomcat
  • Logging: logback, slf4j

Any typical web application would use all these dependencies. Spring Boot Starter Web comes pre-packaged with these. As a developer, I would not need to worry about either these dependencies or their compatible versions.

Spring Boot Starter Project Options

As we see from Spring Boot Starter Web, starter projects help us in quickly getting started with developing specific types of applications.

  • spring-boot-starter-web-services: SOAP Web Services
  • spring-boot-starter-web: Web and RESTful applications
  • spring-boot-starter-test: Unit testing and Integration Testing
  • spring-boot-starter-jdbc: Traditional JDBC
  • spring-boot-starter-hateoas: Add HATEOAS features to your services
  • spring-boot-starter-security: Authentication and Authorization using Spring Security
  • spring-boot-starter-data-jpa: Spring Data JPA with Hibernate
  • spring-boot-starter-cache: Enabling Spring Framework’s caching support
  • spring-boot-starter-data-rest: Expose Simple REST Services using Spring Data REST
Other Goals of Spring Boot

There are a few starters for technical stuff as well

  • spring-boot-starter-actuator: To use advanced features like monitoring and tracing to your application out of the box
  • spring-boot-starter-undertow, spring-boot-starter-jetty, spring-boot-starter-tomcat: To pick your specific choice of Embedded Servlet Container
  • spring-boot-starter-logging: For Logging using logback
  • spring-boot-starter-log4j2: Logging using Log4j2

Spring Boot aims to enable production ready applications in quick time.

  • Actuator: Enables Advanced Monitoring and Tracing of applications.
  • Embedded Server Integrations: Since the server is integrated into the application, I would need to have a separate application server installed on the server.
  • Default Error Handling