Getting Started With the MongoDB Connector for Apache Kafka

Getting Started With the MongoDB Connector for Apache Kafka

Getting Started With the MongoDB Connector for Apache Kafka - Together, MongoDB and Apache Kafka make up the heart of many modern data architectures. Integrating Kafka with external systems like MongoDB is best done through the use of Kafka Connect.

Getting Started With the MongoDB Connector for Apache Kafka - Together, MongoDB and Apache Kafka make up the heart of many modern data architectures. Integrating Kafka with external systems like MongoDB is best done through the use of Kafka Connect.

This API enables users to leverage ready-to-use components that can stream data from external systems into Kafka topics, as well as stream data from Kafka topics into external systems.

The official MongoDB Connector for Apache Kafka is developed and supported by MongoDB engineers. It is also verified by Confluent, following the guidelines set forth by Confluent’s Verified Integrations Program. The connector, now released in beta, enables MongoDB to be configured as both a sink and a source for Apache Kafka.

Getting Started

In the next sections, we will walk you through installing and configuring the MongoDB Connector for Apache Kafka and examine two scenarios. First, we will show MongoDB used as a source to Kafka, where data flows from a MongoDB collection to a Kafka topic. Next, we will show MongoDB used as a sink, where data flows from the Kafka topic to MongoDB.

To get started, you will need access to a Kafka deployment with Kafka Connect as well as a MongoDB database. The easiest and fastest way to spin up a MongoDB database is to use the managed MongoDB service MongoDB Atlas. No more fumbling around with provisioning servers, writing config files, and deploying replica sets—simply pick a cloud provider, a cluster size, and get a connection string!

  • Free MongoDB Atlas cluster*

If you do not have a MongoDB Atlas database, you can easily provision one by visiting MongoDB Cloud Services and clicking “Get started free". For a detailed walkthrough of creating a MongoDB Atlas cluster, see the documentation.

You can use Confluent’s Docker images for a combined setup of Apache Kafka and Kafka Connect.

Install the MongoDB Connector for Apache Kafka

At this point, you should have access to a MongoDB database and Kafka deployment. Follow these instructions to install the MongoDB connector from the Confluent Hub.

You will need your connection string to MongoDB in order to configure the connector. To obtain the connection string in MongoDB Atlas, click on the “Connect” button for your MongoDB cluster. This will show a page with connection strings that are premade for any driver combination. For the MongoDB connector, use Java and version 3.4 or later. Copy the “Connection String Only” and use that for the connection.url in the MongoSinkConnector.properties file. Be sure to replace the <password> template with your actual password for this account. Note: if your password includes the @ symbol, use percent encoding.

Connect page in MongoDB Atlas

Using MongoDB as a Source to a Kafka Topic

Consider the use case of an e-commerce website where the inventory data is stored in MongoDB. When the inventory of any product goes below a certain threshold, the company would like to automatically order more product. Ordering is done by other systems outside of MongoDB, and using Kafka as the platform for such event-driven systems is a great example of the power of MongoDB and Kafka when used together.

Let’s set up the connector to monitor the quantity field and raise a change stream event when the quantity is less than five. Under the covers, the connector is using MongoDB change streams, and the pipeline parameter defines the filter used to generate the event notifications.

curl -X PUT http://localhost:8083/connectors/source-mongodb-inventory/config -H "Content-Type: application/json" -d '{
      "tasks.max":1,
      "connector.class":"com.mongodb.kafka.connect.MongoSourceConnector",
      "key.converter":"org.apache.kafka.connect.storage.StringConverter",
      "value.converter":"org.apache.kafka.connect.storage.StringConverter",
      "connection.uri":"<>",
      "database":"BigBoxStore",
      "collection":"inventory",
      "pipeline":"[{\"$match\": { \"$and\": [ { \"updateDescription.updatedFields.quantity\" : { \"$lte\": 5 } }, {\"operationType\": \"update\"}]}}]", 
      "topic.prefix": ""
}'

In the example above, we provided a pipeline as a parameter. This defines the criteria for documents that are to be consumed by the connector. Since the pipeline contains quotations, we need to escape these so they work with our curl statement. For clarity, the pipeline value is as follows:

[
  {
    "$match": {
      "$and": [
        {
          "updatedDescription.updatedFields.quantity": {
            "$lte": 5
          }
        },
        {
          "operationType": "update"
        }
      ]
    }
  }
]

For simplicity, here we use org.apache.kafka.connect.storage.StringConverter. However, the Kafka Connect framework lets us interchange whatever converter we’d like, such as Avro which enables rich schema support when using Kafka. Integrating schemas between MongoDB and Confluent Platform will be documented for our GA release.

For a complete list of the connector configuration options, check out the documentation.

To test out our scenario, we will use the open-source tool kafkacat. The parameters below tell the tool to connect to the BigBoxStore.inventory topic as a Kafka consumer.

kafkacat -b localhost:9092 -t BigBoxStore.inventory -C

Next, we want to connect to the MongoDB cluster and update the inventory of an item in the inventory collection:

 db.inventory.insert ( { “SKU” : 1, “item_name”:”Tickle Me Elmo”, “quantity” : 10 }

Now, imagine the holiday season is here. Tickle Me Elmo has made a comeback, and it is flying off the shelves. The backend inventory updates the quantity as follows:

 db.inventory.updateOne({"SKU":1},{ $set: { "quantity" : 2} } )

If you look at the kafkacat output, you will see that the change stream event made it into the Kafka topic:

  {
 "_id": {
   "_data": "825D1640BF000000012B022C0100296E5A1004E407DAB9B92B498CBFF2B621AAD032C046645F696400645D163AA63827D21F38DA958E0004"
 },
 "operationType": "update",
 "clusterTime": {
   "$timestamp": {
     "t": 1561739455,
     "i": 1
   }
 },
 "ns": {
   "db": "BigBoxStore",
   "coll": "inventory"
 },
 "documentKey": {
   "_id": {
     "$oid": "5d163aa63827d21f38da958e"
   }
 },
 "updateDescription": {
   "updatedFields": {
     "quantity": 2.0
   },
   "removedFields": []
 }
}

With this message in the Kafka topic, other systems can be notified and process the ordering of more inventory to satisfy the shopping demand for Elmo.

Using MongoDB as a Sink From a Kafka Topic

In continuation of the e-commerce scenario, consider that when a new user is created on the website. Their contact information is needed by multiple business systems. Contact information is placed in the Kafka topic newuser for shared use, and we then configure MongoDB as a sink to the Kafka topic. This allows new users’ information to propagate to a users collection in MongoDB. To configure the connector for this scenario, we can issue a REST API call to the connector service as follows:

curl -X PUT http://localhost:8083/connectors/sink-mongodb-users/config -H "Content-Type: application/json" -d ' {
      "connector.class":"com.mongodb.kafka.connect.MongoSinkConnector",
      "tasks.max":"1",
      "topics":"newuser",
      "connection.uri":"<>",
      "database":"BigBoxStore",
      "collection":"users",
      "key.converter":"org.apache.kafka.connect.json.JsonConverter",
      "key.converter.schemas.enable":false,
      "value.converter":"org.apache.kafka.connect.json.JsonConverter",
      "value.converter.schemas.enable":false
}'

To test our scenario, let’s use kafkacat to push a message that simulates the inventory system saying there is more inventory:

  kafkacat -b localhost:9092 -t newuser -P <

To confirm that the message made it all the way through to your MongoDB database, make a connection to MongoDB using your client tool of choice and issue a db.users.find() command. If you’re using MongoDB Atlas, you can click on the “Collections” tab to show the databases and collections that are in your cluster.

Collections tab in MongoDB Atlas

Summary

MongoDB is the world’s most popular modern database built for handling massive volumes of heterogeneous data, and Apache Kafka is the world’s best distributed, fault-tolerant, high-throughput event streaming platform. Together they make up the heart of many modern data architectures today. The MongoDB Connector for Apache Kafka is the official Kafka connector. The sink connector functionality was originally written by Hans-Peter Grahsl and with his support has now been integrated into MongoDB’s new source and sink connector, officially supported and maintained by the creators of MongoDB. This connector opens the door to many scenarios ranging from event-driven architectures to microservices patterns. We hope you check it out and comment below to let us know what you think!

Mobile App Development Company India | Ecommerce Web Development Company India

Mobile App Development Company India | Ecommerce Web Development Company India

Best Mobile App Development Company India, WebClues Global is one of the leading web and mobile app development company. Our team offers complete IT solutions including Cross-Platform App Development, CMS & E-Commerce, and UI/UX Design.

We are custom eCommerce Development Company working with all types of industry verticals and providing them end-to-end solutions for their eCommerce store development.

Know more about Top E-Commerce Web Development Company

Hire PHP Developer and Web Developer for your Online Business

Hire PHP Developer and Web Developer for your Online Business

PHP is widely used open-source scripting language it helps in making dynamically easy your websites and web application. Mobiweb Technology is your best technical partner and offering you solution for any kind of website and application...

PHP is widely used open-source scripting language it helps in making dynamically easy your websites and web application. Mobiweb Technology is your best technical partner and offering you solution for any kind of website and application development. To hire PHP developer and web developer at affordable prices contact Mobiweb Technology via [email protected]

Website Development - Top Reasons Why Your Business Need A Creative Website?

Website Development - Top Reasons Why Your Business Need A Creative Website?

Currently, most business owners want a website and expand their business to increase the selling on local to international markets. We give you the top reasons why you need a website.

Most of the entrepreneurs with innovative ideas about start-up businesses are unaware of all the opportunities the internet has for their business.

A major percentage of our population uses the internet to look for local businesses, and if the business does not have a website then most of their potential consumers won’t get to know about the website only. It gets difficult for beginners who have little knowledge about web development this is where the role of web development services comes into the picture. Having a website connects our business to all the parts of the world effortlessly and opens up various new platforms which were not available previously.

The power of a web page is simply incomparable to other expensive methods of advertising like print media. In the era of smartphones, tablets and PCs where everybody is constantly exposed to the internet at every point of life, the competition is brewing like coffee.

Your web page needs to stand out and connect with the consumers, all the web development services can help you to make your website look just as you want it to.

The purpose of your webpage is not only to serve your customer base, but also to attract new customers, so our website needs to be easily accessible and attractive and some of the ways in which a web development company would help you are by.

“The page should be navigable smoothly and effortlessly, and whatever information the customer needs he/she should be able to retrieve it easily through a search box, etc.”

Professional web developers are able to provide these services.

1. Search Engine Optimization
All the websites are competing to be on top of the list of search engine result pages, but only web developers know how to work on this with the help of proper keywords, image optimization, linking, etc.

2. Appropriate Visual Content
It is the need of the hour to provide the customers with relevant visuals to get a proper idea of your business. Hence, we need professional help to balance the text and visuals for your website so that the customer doesn’t get bored beforehand and leave the webpage

3. Online Transaction
Most of the consumers these days prefer cashless and online mode of payment. It becomes more convenient from the sales point of view too to keep a track of online sales records. Web development services ensure that your website has all required mediums for a transaction and also whether all of them are up to date or not.

4. Global Recognition
Once the brand name is out there, it is very important for the website to have a proper connection with the international client base as well as local customers, the format and design of the website helps is doing so.

5. Responsiveness
Anyone could set up an informative website, but to hold the customer’s attention the website also needs to responsive and interactive, Services are provided through which feedback of the customers can be collected and problems can be addressed as well.

6. Reducing Downtime
While the page is being formatted it is unavailable to the customers, this could result in a loss of attention span from many customers. Hence, webmasters make sure that they take as little time as possible to put up new blog posts or advertisements on the website and don’t make the customers wait.

7. Profit-Making Schemes
One of the ways in which our website might earn small profits is by allowing ads. However, it is a tedious job to also block inappropriate content. Web developers allow and block ads accordingly so that the ads don’t take up much space, but also harmlessly add up to the profits.

Read also: PHP Trends: 5 Best Framework For Web Development

The article gives an idea about how it is necessary for a business in the current world to have a webpage, but adding up all the features which would attract customers is not as easy as it looks and which is why web development services to help us.

Source by