Orchestrating ETL pipelines on AWS with Glue, StepFunctions and Cloudformation

Orchestrating ETL pipelines on AWS with Glue, StepFunctions and Cloudformation

AWS is the pack leader with the very versatile Glue and S3 services which allow users to ingest transform, and normalize store datasets of all sizes. Learn how to Orchestrating ETL pipelines on AWS with Glue, StepFunctions, and Cloudformation.

Big Data analytics is becoming increasingly important to draft major business choices in corporations of all sizes. However collecting, aggregating, joining, and analyzing (wrangling) huge amounts of data stored in different locations with a heterogeneous structure (e.g. databases, CRMs, unstructured text, etc.) is often a daunting and very time-consuming task.

Cloud computing often comes to the rescue, by providing cheap and scalable storage computing and data lake solutions, and in particular, AWS is the pack leader with the very versatile Glue and S3 services which allow users to ingest transform, and normalize store datasets of all sizes. Furthermore, Glue Catalog and Athena allow users to easily run Presto-based SQL queries on the normalized data in S3 data lakes, whose results can easily be stored and analyzed in business intelligence tools such as QuickSight.

aws-step-functions aws-cloudformation etl aws-glue aws

What is Geek Coin

What is GeekCash, Geek Token

Best Visual Studio Code Themes of 2021

Bootstrap 5 Tutorial - Bootstrap 5 Crash Course for Beginners

Nest.JS Tutorial for Beginners

Hello Vue 3: A First Look at Vue 3 and the Composition API

ETL Orchestration on AWS with AWS Step Functions

Before being able to allow the figure of the data analyst to explore and visualize the data, a crucial step is needed. This procedure is commonly identified as ETL (extract, transform, and load) and, usually, it’s far from being simple. ETL Orchestration on AWS with AWS Step Functions.

Combine AWS Step Functions with CloudWatch Events using aws-cdk

AWS Step Functions allow one to execute & coordinate long-running processes. Step Functions fall into serverless AWS services, and the…

ETL Data Pipeline In AWS

There is AWS Glue for you, it’s a feature of Amazon Web Services to create a simple ETL pipeline. ETL Data Pipeline In AWS.

Extract, Transform, Load (ETL) — AWS Glue

Learn how to use AWS Glue for ETL operations in Spark on Novel Corona Virus Dataset

AWS Glue Elastic Views! An almost no code ETL & Aggregation Framework

AWS Glue Elastic Views! An almost no code ETL and Aggregation Framework. In the last couple of years AWS has been aggressively developing tools and services to help in Machine Learning and ETL tasks and at the last re:Invent introduced another important component for ETL-ML preparation: AWS Elastic Views.