Apache Camel
Apache Camel is an open-source framework for message-oriented middleware with a rule-based routing and mediation engine that provides a Java object-based implementation of the Enterprise Integration Patterns using an application programming interface to configure routing and mediation rules
Red Hat AMQ Streams
Red Hat AMQ Streams is a massively-scalable, distributed, and high-performance data streaming platform based on the Apache ZooKeeper and Apache Kafka projects.
The main components comprise:
Kafka Broker
Messaging broker responsible for delivering records from producing clients to consuming clients.
Apache ZooKeeper is a core dependency for Kafka, providing a cluster coordination service for highly reliable distributed coordination.
AMQ Streams architecture
camel-kafka-connector
Camel Kafka Connector allows you to use all Camel components as Kafka Connect connectors.
This is a “Camel Kafka connector adapter” that aims to provide a user friendly way to use all Apache Camel components in Kafka Connect.
Kafka Connect is a tool for scalably and reliably streaming data between Apache Kafka and other systems. It makes it simple to quickly define connectors that move large collections of data into and out of Kafka.For more information about Kafka Connect take a look here.
Prerequisites
For this demonstration, you will need the following technologies set up in your development environment:
- Apache Maven 3.6.3+
- JDK 11 Installed
- kafka cluster
- AWS Account set up and Files available in S3 bucket.
In this article, we demonstrate how to read files from S3 buckets and write to kafka Topic using
CamelAWSS3SourceConnector
Prepare the Needed Bits to Develop the Example
setup the plugin.path property in your kafka
Open the $KAFKA_HOME/config/connect-standalone.properties
and set the plugin.path property to your chosen location
Java
plugin.path=/home/kkakarla/development/fuse-ocp-training-videos/kcs/amq-streams-kafka/camel-kafka-connectors/
#big data #apache camel #kafka #aws s3 #kafka connectors #amq streams
