A guide on how to manage data flows around GCP

I already discussed some functionalities of GCP in another post, that will guide you in automatizing the execution of simple code chunks.

One thing that might come in handy is not only to execute code on GCP, but also to export the result of that code as a file somewhere where you can access it.

Note that the views expressed here are my own. I am not a GCP certified user, and the proceedings described below are my own doing, inspired from many sources online and time spent tweaking around the cloud…There might be better ways to achieve the goals stated below.

In this post, I will discuss two different approaches that will allow you to export the result(s) of your code execution to:

• 1) Cloud Bucket (with an option to Big Query tables): Big query tables will be useful if you need to perform additional operations on GCP, such as perform regular ML predictions on certain data tables. They can also be access by third party software such as Tableau or Power BI if you want to fetch data for visualization.
• 2) Google Drive: Google Drive export would allow you to provide easy and readable access to many users, if the data is to be shared on a regular basis for further personal use.
• 3) FTP server: your organization might be using a FTP server to exchange and/or store data.

#cloud #data-engineering #google-cloud-platform #data-science