I would say that is very rare to find a situation when we don’t need to have file storage and a database with regard to web development.

The file storage can be used to handle simple situations like application settings, connection strings, and some customer data. At the same time, this resource can be used for complex scenarios like data archiving / analytics, and even for static content.

However, if you need to store complex data, performance, and scalability the database might be the right choice.

The scenario

So, let’s imagine a situation when need to extract the data from your file storage and put it in your database, simple as that.

Secondly, we will assume that our file storage is the AWS Simple Cloud Storage (S3) and the database is the Amazon Dynamo DB.

And finally, let’s suppose that the files are not too big (under 500MB) and they are in CSV format, following the pattern: _UserIdentifier,Username,Language. _An example of the data can be:

123456,Darth Vader,Python
774477,Yoda,Golang
999000,Chewbacca,Javascript

The tools

There are a few approaches that we can use to solve our problem, like:

AWS Data Pipeline

Easily automate the movement and transformation of data.

AWS Data Glue

Simple, flexible, and cost-effective ETL

Amazon Athena

Start querying data instantly. Get results in seconds. Pay only for the queries you run.

So, all these tools are awesome and can be used without any issue to achieve what we are looking for.

Nevertheless, there are some scenarios that we don’t need to use a shovel to burst a balloon, right? We can just use a needle :)

The Simple solution — Lambda function

Assuming that we want to a simple solution, easy to maintain, and limited to small/medium files, the Lambda function fits perfectly in this situation.

The idea is very simple, our lambda function will be triggered by the S3, and will save these data in the DynamoDB.

Ok, let’s start!

#s3 #cloud #aws #lambda #golang