A core responsibility of a data engineer is to build data pipelines. The flow of data from one location to another with mutations along the way is a primary focus for any data engineer. However, when you change data, you make people nervous. Even though you are actively changing the data from its raw form into something usable, it doesn’t change the fact that it is no longer the data that was collected. In order to mitigate these fears, it is vital that data engineers build as many data integrity measures into the solution as possible.

Techniques in quality assurance

• 1. Byte-by-Byte Comparison
• 2. Checksums
• 3. Row Counts
• 4. Unit Tests

#data-integrity #data-pipeline #data-engineering #quality-assurance #data pipelines