Use cases and architectures for Event Streaming, Apache Kafka, Distributed Ledger (DLT) and Blockchain. Comparison: Kafka-native vs Hyperledger vs Ethereum.

This blog post discusses the concepts, use cases, and architectures behind Event Streaming, Apache Kafka, Distributed Ledger (DLT), and Blockchain. A comparison of different technologies such as Confluent, AIBlockchain, Hyperledger, Ethereum, Ripple, IOTA, and Libra explores when to use Kafka, a Kafka-native blockchain, a dedicated blockchain, or Kafka in conjunction with another blockchain.

Use Cases for Secure and Tamper-Proof Data Processing With a Blockchain

Blockchain is a hype topic for many years. While many companies talk about the buzzword, it is tough to find use cases where Blockchain is the best solution. The following examples show the potential of Blockchain. Here it might make sense:

• Supply Chain Management (SCM): Manufacturing, transportation, logistics, and retailing see various use cases, including transaction settlement, tracking social responsibility, accurate cost information, precise shipping, and logistics data, automated purchasing and planning using integrated ERP and CRM systems, enforcement of partner contracts, food safety.
• Healthcare: Management of patient data (especially access control), management and use of academic research data, prevention of compliance violations, reducing human errors, cross-cutting up-to-date patient information, secure identity management.
• Financial Services: Instant global payments with cryptocurrencies (including a digital US Dollar or EURO), reduced cost for end-users, audit transparency, data provenance, data lineage, fraud reduction, savings on business operations/finance reporting/compliance.

These are great, valid use cases. But you should ask yourself some critical questions:

Do you need a blockchain? What parts of “blockchain” do you need? Tamper-proof storage and encrypted data processing? Consortium deployments with access from various organizations? Does it add business value? Is it worth the added efforts, complexity, cost, risk?

Challenges and Concerns of Blockchain Technologies

Why am I so skeptical about Blockchain? I love the concepts and technologies! But I see the following concerns and challenges in the real world:

• Technical complexity: Blockchain technology is still very immature, and the efforts to implement a blockchain project are massive and often underestimated
• Organizational complexity: Deploying a blockchain over multiple organizations requires enormous efforts, including compliance and legal concepts.
• Transaction speed: Bitcoin is slow. But also, most other blockchain frameworks are not built for millions of messages. Some solutions are in the works to improve from hundreds to thousands of messages per second. This scale is still not good enough for financial trading, supply chain, Internet of Things, and many other use cases.
• Energy consumption: Bitcoin and some other cryptocurrencies use ‘Proof-of-Work’ (POW) as a consensus mechanism. The resources used for Bitcoin mining is crazy. Period!
• Security: Providing a tamper-proof deployment is crucial. Deploying this over different organizations (permissioned / consortium blockchain) or public (permissionless) is hard. Security is much more than just a feature or API.
• Data tenancy: Data privacy, compliance (across countries), and ownership of the data (who is allowed to use it for creating added value) are hard to define.
• Lifecycle costs: Blockchain infrastructure is complex. It is not just an application but a cross-company distributed infrastructure. Development, operations, and management and are very different from hosting your monolith or microservices.

Hence, it is important to only choose a blockchain technology when it makes sense. Most projects I have seen in the last years were about evaluating technologies and doing technical POCs. Similar to Hadoop data lakes 5-10 years ago. I fear the same result as in the Hadoop era: The technical proof worked out, but no real business value or enormous additional and unnecessary complexity and cost were added.

Distributed Ledger and Blockchain Technologies

It is crucial to understand that people often don’t mean ‘Blockchain’ when they say ‘Blockchain’. What they actually mean is ‘Distributed Ledger Technology’ (DLT). What is the relation between Blockchain and DLT?

Blockchain: A Catchall Phrase for Distributed Ledger Technology (DLT)

The following explores Blockchain and DLT in more detail:

Distributed Ledger Technology (DLT):

• Decentralized database
• Managed by various participants. No central authority acts as an arbitrator or monitor
• Distributed log of records, greater transparency
• Fraud and manipulation more difficult, more complicated to hack the system

Blockchain:

• **Blockchain is nothing else but a DLT with a specific set of features **
• Shared database – a log of records – but in this case, shared using blocks that form a chain
• Blocks are closed by a type of cryptographic signature called a ‘hash’; the next block begins with that same ‘hash’, a kind of wax seal
• Hashing verifies that the encrypted information has not been manipulated and that it can’t be manipulated

#open source #blockchain #bitcoin #kafka #ethereum #apache