Managing Data as a Data Engineer — Understanding Users

Managing Data as a Data Engineer — Understanding Users

Understanding how users view data and their pain points when using data. In this article, I would like to share some of the things that I have learnt while managing terabytes of data in a fintech company.

As a data engineer, my work revolves around managing data assets ranging from ETL scripts, databases, data warehouses, data access controls, queries, dashboards and data objects. As we have more data and as data grows larger, managing data becomes quite a challenge. In this article, I would like to share some of the things that I have learnt while managing terabytes of data in a fintech company.

This is a three-part series article. Part 1 focuses on understanding data users. Part 2 shares more on Understanding Data Changes .

How Users View Data?

Before we even begin to manage data, let’s first try to understand our users. In a fintech company, most of our users are internal users with different roles ranging from product managers, software engineers, data scientists, analysts, executives to customer support team. These users interact with data in all kinds of way everyday.

On one hand, the business/operational users consume data in the form of reporting dashboards and charts. This is how they get informed of the latest business metrics to make day to day business decisions.

On the other hand, the data analysts/data scientists are more technical. They work with data using SQL queries, Jupyter notebooks and Python libraries. Sometimes, they are also the creators of the downstream data assets that are consumed by other users.

The software engineers are the builders of the product. The application generates data which is stored in a database, and also consumed by other users or other services. This data usually serve as the raw data for analytics and other downstream data.

Last but not least, the product managers also consume data via emails/ slack notifications and alerts informing them of the latest status of the product.

data engineering data-management data-science data-engineering

Bootstrap 5 Complete Course with Examples

Bootstrap 5 Tutorial - Bootstrap 5 Crash Course for Beginners

Nest.JS Tutorial for Beginners

Hello Vue 3: A First Look at Vue 3 and the Composition API

Building a simple Applications with Vue 3

Deno Crash Course: Explore Deno and Create a full REST API with Deno

How to Build a Real-time Chat App with Deno and WebSockets

Convert HTML to Markdown Online

HTML entity encoder decoder Online

Managing Data as a Data Engineer:  Understanding Data Changes

Understand how data changes in a fast growing company makes working with data challenging. In the last article, we looked at how users view data and the challenges they face while using data.

50 Data Science Jobs That Opened Just Last Week

Data Science and Analytics market evolves to adapt to the constantly changing economic and business environments. Our latest survey report suggests that as the overall Data Science and Analytics market evolves to adapt to the constantly changing economic and business environments, data scientists and AI practitioners should be aware of the skills and tools that the broader community is working on. A good grip in these skills will further help data science enthusiasts to get the best jobs that various industries in their data science functions are offering.

Intro to Data Engineering for Data Scientists

Intro to Data Engineering for Data Scientists: An overview of data infrastructure which is frequently asked during interviews

Applications Of Data Science On 3D Imagery Data

The agenda of the talk included an introduction to 3D data, its applications and case studies, 3D data alignment and more.

Data Science Course in Dallas

Become a data analysis expert using the R programming language in this [data science](https://360digitmg.com/usa/data-science-using-python-and-r-programming-in-dallas "data science") certification training in Dallas, TX. You will master data...