Delta Lake on Databricks

Delta Lake is an open format storage layer that delivers reliability, security and performance on your data lake — for both streaming and batch operations. By replacing data silos with a single home for structured, semi-structured and unstructured data, Delta Lake is the foundation of a cost-effective, highly scalable lakehouse.

 #bigdata 

What is GEEK

Buddha Community

Decoding The Delta Lake Architecture: What Is It ?

What is Delta Lake?

Delta Lake is an open-source storage layer that delivers reliability to data lakes. Delta Lake implements ACID transactions, scalable metadata handling, and unifies the streaming and batch data processing. Delta Lake architecture runs on top of your current data lake and is fully cooperative with Apache Spark APIs.

Why Delta Lake?

Are we making progress? Well, let’s see what are the main benefits of implementing a Delta Lake in your company.

The Predicament with current Data Architectures

Current big data architectures are challenging to develop, manage, and maintain. Most contemporary data architectures use a mix of at least three varying types of systems: streaming systems, data lakes, and data warehouses. Business data comes through streaming networks such as Amazon Kinesis or Apache Kafka, which mainly focus on accelerated delivery. Then, data is collected in data lakes, such as Apache Hadoop or Amazon S3, which are optimized for large-scale, ultra-low-cost storage. Lamentably, data lakes individually do not have the performance and quality required to support high-end business applications: thus, the most critical data is uploaded to data warehouses, which are optimized for significant performance, concurrency, and security at a much higher storage cost than data lakes.

Delta Lake architecture, Lambda Architecture

Lambda architecture is a traditional technique where a batch system and streaming system prepare records in correspondence. The results are then merged during query time to provide an entire answer. Strict latency requirements to process old and newly formed events made this architecture famous. The key downside to this architecture is the development and operational overhead of maintaining two different systems. There have been efforts to ally batch and streaming into a single system in history. Companies have not been that victorious though in those attempts. With the arrival of Delta Lake, we are seeing a lot of our clients adopting a simple constant data flow model to process data as it comes. We call this architecture, The Delta Lake architecture. We cover the essential bottlenecks for using a continuous data flow model and how the Delta architecture resolves those difficulties.

#big data engineering #blogs #delta lake #delta lake architecture #delta lake spark

How Change Data Capture (CDC) gets benefits from Delta Lake

Introduction

Enterprise has been spending millions of dollars getting data into data lakes using Apache Spark with the aspiration to perform Machine Learning and to build Recommendation engines, Fraud Detection, IoT & Predictive maintenance etc. But the fact is majority of these projects are failing in getting the reliable data.

Challenges with the traditional data lake

• Failed Production Jobs will leave the data in corrupted state and it requires tedious job to recover the data. We need to have some script to clean up and to revert the transaction.
• Lack of schema enforcement creates inconsistent data and low quality data.
• Lack of consistency, while reading the data when there is a concurrent write, result will not be inconsistent until Parquet is fully updated. When there is multiple writes happening in streaming job, the downstream apps reading this data will be inconsistent because there is no isolation between each writes.

---

“Delta Lake overcomes the above challenges”

Delta Lake

Databricks open sourced their proprietary storage name in the name of Delta Lake, to bring ACID transactions to Apache Spark and big data workloads. Earlier Delta lake is available in Azure/AWS Databricks only where the data will get stored only on DBFS, which may lie on top of ADLS/S3. Now Delta format can lie on HDFS, ADLS, S3 or local File system, etc…. Delta Lake is also compatible with MLFlow.

How Delta Works?

Delta lake is based on Parquet, it adds the transactional awareness to Parquet using transaction log which will be maintained in additional folder (_delta_log ) under the table directory. Lot of vendors like Informatica, Talend embrace delta and working on native readers and writers.

Json file under the _delta_log folder will have the information like add/remove parquet files(for Atomicity), stats(for optimized performance & data skipping), partitionBy(for partition pruning), readVersions(for time travel), commitInfo(for audit).

Below is the Json file which present in delta transactional log when we write sample DataFrame with 2 records. Notice that it analyse the status like min,max in each file which helps to effectively skips the unnecessary data and helps in performance optimization.

#spark #databricks #delta #delta-lake #big-data #data analysis

Difference between: Data Lake Vs Delta Lake

DATA LAKE

Data Lake is a storage repository that cheaply stores vast raw data in its native format.

It consists of current and historical data dumps in various formats, including XML, JSON, CSV, Parquet, etc.

Drawbacks in Data Lake

• Doesn’t provide Atomicity — No all or nothing, it may end up storing corrupt data.
• No Quality Enforcement — It creates inconsistent and unusable data.
• No Consistency/Isolation — It’s impossible to read and append when an update occurs.

DELTA LAKE

Delta Lake allows us to incrementally improve the quality until it is ready for consumption. Data flows like water in Delta Lake from one stage to another stage (Bronze -> Silver -> Gold).

• Delta lake brings full ACID transactions to Apache Spark. That means jobs will either be complete or not at all.
• Delta is open-sourced by Apache. You can store a large amount of data without worrying about locking.
• Delta lake is deeply powdered by Apache Spark, meaning the Spark jobs (batch/stream) can be converted without writing those from scratch.

Delta Lake Architecture

Delta Lake Architecture

Bronze Tables

Data may come from various sources, which could be Dirty. Thus, It is a dumping ground for raw data.

Silver Tables

Consists of Intermediate data with some cleanup applied.

It is Queryable for easy debugging.

Gold Tables

It consists of clean data, which is ready for consumption.

Original article source at:  https://www.c-sharpcorner.com/

#datalake #delta #spark #databricks #bigdata 

两者之间的区别：Data Lake 与 Delta Lake

数据湖

Data Lake 是一个存储库，可以廉价地以其本机格式存储大量原始数据。

它由各种格式的当前和历史数据转储组成，包括 XML、JSON、CSV、Parquet 等。

数据湖的缺点

• 不提供原子性——全无或全无，它最终可能会存储损坏的数据。
• 没有质量执行——它会产生不一致和不可用的数据。
• 无一致性/隔离——发生更新时不可能读取和追加。

三角洲湖

Delta Lake 使我们能够逐步提高质量，直到可以使用为止。数据像 Delta Lake 中的水一样从一个阶段流到另一个阶段（青铜 -> 白银 -> 黄金）。

• Delta lake 为 Apache Spark 带来了完整的 ACID 事务。这意味着工作要么完成要么根本不完成。
• Delta 由 Apache 开源。您可以存储大量数据而不必担心锁定。
• Delta lake 深受 Apache Spark 的影响，这意味着可以转换 Spark 作业（批处理/流）而无需从头开始编写。

三角洲湖建筑

三角洲湖建筑

青铜桌

数据可能来自各种来源，这些来源可能是脏的。因此，它是原始数据的垃圾场。

银表

由应用了一些清理的中间数据组成。

它是可查询的，便于调试。

黄金表

它由干净的数据组成，可以随时使用。

文章原文出处：https:   //www.c-sharpcorner.com/

#datalake #delta #spark #databricks #bigdata 

Разница между: Data Lake и Delta Lake

ОЗЕРО ДАННЫХ

Озеро данных — это репозиторий, в котором дешево хранятся огромные объемы необработанных данных в собственном формате.

Он состоит из дампов текущих и исторических данных в различных форматах, включая XML, JSON, CSV, Parquet и т. д.

Недостатки озера данных

• Не обеспечивает атомарность — не все или ничего, в конечном итоге это может привести к хранению поврежденных данных.
• Нет контроля качества — создаются противоречивые и непригодные для использования данные.
• Отсутствие согласованности/изоляции — чтение и добавление при обновлении невозможно.

ОЗЕРО ДЕЛЬТА

Delta Lake позволяет нам постепенно улучшать качество, пока оно не будет готово к употреблению. Данные перетекают, как вода в озере Дельта, от одного этапа к другому (бронза -> серебро -> золото).

• Озеро Delta переносит полные ACID-транзакции в Apache Spark. Это означает, что работы либо будут завершены, либо их не будет вообще.
• Delta является открытым исходным кодом Apache. Вы можете хранить большой объем данных, не беспокоясь о блокировке.
• Озеро Дельта глубоко измельчено Apache Spark, что означает, что задания Spark (пакетные/потоковые) могут быть преобразованы без написания их с нуля.

Архитектура Дельта-Лейк

Архитектура Дельта-Лейк

Бронзовые Столы

Данные могут поступать из различных источников, которые могут быть грязными. Таким образом, это свалка для необработанных данных.

Серебряные столы

Состоит из промежуточных данных с некоторой очисткой.

Это Queryable для легкой отладки.

Золотые столы

Он состоит из чистых данных, готовых к использованию.

Оригинальный источник статьи:   https://www.c-sharpcorner.com/

#datalake #delta #spark #databricks #bigdata