AWS is the pack leader with the very versatile Glue and S3 services which allow users to ingest transform, and normalize store datasets of all sizes. Learn how to Orchestrating ETL pipelines on AWS with Glue, StepFunctions, and Cloudformation.
Big Data analytics is becoming increasingly important to draft major business choices in corporations of all sizes. However collecting, aggregating, joining, and analyzing (wrangling) huge amounts of data stored in different locations with a heterogeneous structure (e.g. databases, CRMs, unstructured text, etc.) is often a daunting and very time-consuming task.
Cloud computing often comes to the rescue, by providing cheap and scalable storage computing and data lake solutions, and in particular, AWS is the pack leader with the very versatile Glue and S3 services which allow users to ingest transform, and normalize store datasets of all sizes. Furthermore, Glue Catalog and Athena allow users to easily run Presto-based SQL queries on the normalized data in S3 data lakes, whose results can easily be stored and analyzed in business intelligence tools such as QuickSight.
Before being able to allow the figure of the data analyst to explore and visualize the data, a crucial step is needed. This procedure is commonly identified as ETL (extract, transform, and load) and, usually, it’s far from being simple. ETL Orchestration on AWS with AWS Step Functions.
AWS Step Functions allow one to execute & coordinate long-running processes. Step Functions fall into serverless AWS services, and the…
There is AWS Glue for you, it’s a feature of Amazon Web Services to create a simple ETL pipeline. ETL Data Pipeline In AWS.
Learn how to use AWS Glue for ETL operations in Spark on Novel Corona Virus Dataset
AWS Glue Elastic Views! An almost no code ETL and Aggregation Framework. In the last couple of years AWS has been aggressively developing tools and services to help in Machine Learning and ETL tasks and at the last re:Invent introduced another important component for ETL-ML preparation: AWS Elastic Views.