Transactional integration Kafka with database

Introduction

More and more projects are now choosing Kafka as their messaging infrastructure. This technology’s choice is not always driven by the real need to handle vast amounts of data with linear scalability. We can often hear about Kafka’s application in the usual boring enterprise projects involving trivial document workflow management. The answering the question “why” in this case is not easy. Sometimes it can be caused by hype and impressive examples of Kafka’s usage in notable projects. In other cases, it may be inspired by the opportunity to experiment at the expense of the customer’s money, attracting him with advanced technologies.

#software-development #database #software-architecture #kafka #transactions

What is GEEK

Buddha Community

Transactional integration Kafka with database

Introduction

More and more projects are now choosing Kafka as their messaging infrastructure. This technology’s choice is not always driven by the real need to handle vast amounts of data with linear scalability. We can often hear about Kafka’s application in the usual boring enterprise projects involving trivial document workflow management. The answering the question “why” in this case is not easy. Sometimes it can be caused by hype and impressive examples of Kafka’s usage in notable projects. In other cases, it may be inspired by the opportunity to experiment at the expense of the customer’s money, attracting him with advanced technologies.

#software-development #database #software-architecture #kafka #transactions

System Databases in SQL Server

Introduction

In SSMS, we many of may noticed System Databases under the Database Folder. But how many of us knows its purpose?. In this article lets discuss about the System Databases in SQL Server.

System Database

Fig. 1 System Databases

There are five system databases, these databases are created while installing SQL Server.

• Master
• Model
• MSDB
• Tempdb
• Resource

Master

• This database contains all the System level Information in SQL Server. The Information in form of Meta data.
• Because of this master database, we are able to access the SQL Server (On premise SQL Server)

Model

• This database is used as a template for new databases.
• Whenever a new database is created, initially a copy of model database is what created as new database.

MSDB

• This database is where a service called SQL Server Agent stores its data.
• SQL server Agent is in charge of automation, which includes entities such as jobs, schedules, and alerts.

TempDB

• The Tempdb is where SQL Server stores temporary data such as work tables, sort space, row versioning information and etc.
• User can create their own version of temporary tables and those are stored in Tempdb.
• But this database is destroyed and recreated every time when we restart the instance of SQL Server.

Resource

• The resource database is a hidden, read only database that holds the definitions of all system objects.
• When we query system object in a database, they appear to reside in the sys schema of the local database, but in actually their definitions reside in the resource db.

#sql server #master system database #model system database #msdb system database #sql server system databases #ssms #system database #system databases in sql server #tempdb system database

Tips About Kafka Connect On Heroku You Can't Afford To Miss

Introduction

With ever-increasing demands from other business units, IT departments have to be constantly looking for service improvements and cost-saving opportunities. This article showcases several concrete use-cases for companies that are investigating or already using Kafka, in particular, Kafka Connect.

Kafka Connect is an enterprise-grade solution for integrating a plethora of applications, ranging from traditional databases to business applications like Salesforce and SAP. Possible integration scenarios range from continuously streaming events and data between applications to large-scale, configurable batch jobs that can be used to replace manual data transfers.

#kafka-connect #kafka #heroku #database #database-architecture #apache-kafka #tutorial #cluster

Kafka for XML Message Integration and Processing

XML messages and XML Schema are not very common in the Apache Kafka and Event Streaming world! Why? Many people call XML legacy. It is complex, verbose, and often associated with the ugly WS-* Hell (SOAP, WSDL, etc). On the other side, every company older than five years uses XML. It is well understood, provides a good structure, and is human- and machine-readable.

This post does not want to start another flame war between XML and other technologies such as JSON (which also provides JSON Schema now), Avro, or Protobuf. Instead, I will walk you through the three main approaches to integrate between Kafka and XML messages as there is still a vast demand for implementing this integration today (often for integrating legacy applications and middleware).

XML and XML Schema

Extensible Markup Language (XML) is a markup language that defines a set of rules for encoding documents in a format that is both human-readable and machine-readable. The World Wide Web Consortium’s XML 1.0 Specification of 1998 and several other related specifications — all of them free open standards — define XML.

The design goals of XML emphasize simplicity, generality, and usability across the Internet. It is a textual data format with strong support via Unicode for different human languages. Although the design of XML focuses on documents, the language is widely used for the representation of arbitrary data structures such as those used in web services. Several schema systems exist to aid in defining XML-based languages, while programmers have developed many application programming interfaces (APIs) to assist the processing of XML data.

#open source #big data #integration #xml #json #kafka #middleware #event streaming #kafka connect platform #kafka connectors

Apache Kafka and SAP ERP Integration Options

A question I get every week from customers across the globe is, “How can I integrate my SAP system with Apache Kafka?” This post explores various alternatives, including connectors, third party tools, custom glue code, and trade-offs between the different options.

After exploring what SAP is, I will discuss several integration options between Apache Kafka and SAP systems:

• Traditional middleware (ETL/ESB)
• Web services (SOAP/REST)
• 3rd party turnkey solutions
• Kafka-native connectivity with Kafka Connect
• Custom glue code using SAP SDKs

Disclaimer before you read on:

I am not an SAP expert. It is tough to stay up-to-date with the vast and complex ecosystem of SAP products, (re-)brands, versions, services, SDKs, and APIs. I am sorry if some of the below information is not 100% accurate or outdated. Always double-check on the SAP website (if the links from Google still work, I had some issues with some pages “no longer available” while researching for this blog post). If you see any inaccurate or missing information, please let me know and I will update the blog post.

What is SAP?

SAP is a German multinational software corporation that makes enterprise software to manage business operations and customer relations.

It is quite interesting: Nobody asks how to integrate with IBM or Oracle. Instead, people more specifically ask how to integrate with IBM MQ, IBM DB2, IBM Mainframe (still very ambiguous), or any other of the hundreds of IBM products.

For SAP, people ask: “How can I integrate with SAP?” Let’s clarify what SAP is before exploring integration options.

The company is primarily known for its ERP software. But if you check out the official “What is SAP?” page, you find out that SAP offers solutions across a wide range of areas:

• ERP and Finance
• CRM and Customer Experience
• Network and Spend Management
• Digital Supply Chain
• HR and People Engagement
• Experience Management
• Business Technology Platform
• Digital Transformation
• Small and Midsize Enterprises
• Industry Solutions

SAP’s Software Portfolio

SAP’s stack includes homegrown products like SAP ERP and acquisitions with their own codebase, including Ariba for supplier network, hybris for e-commerce solutions, Concur for travel & expense management, and Qualtrics for experience management.

Even if you talk about SAP ERP, the situation is still not that easy. Most companies still run SAP ERP Central Component (ECC, formerly called SAP R/3), SAP’s sophisticated (and aged) ERP product. ECC runs on a third-party relational database from Oracle, IBM, or Microsoft, while HANA is SAP’s in-memory database. The new ERP product is SAP S4/Hana (no, this is not just the famous in-memory database). Oh, and there is SAP S4/Hana Cloud. And before you wonder: No, this is not the same feature set as the on-premise version!

Various interfaces exist depending on your product. An interface can be an (awful) proprietary technologies like BAPI or iDoc, (okayish) standards-based web service APIs using SOAP or REST / HTTP, a (non-scalable) JDBC database connectivity, or if you are lucky even a (scalable and real-time) Event/Messaging API.

#integration #kafka #sap #middleware #sap erp #sap hana #kafka connect platform #kafka connectors #bapi #idoc