Do you have the knowledge and skills to design a mobile gaming analytics platform that collects, stores, and analyzes large amounts of bulk and real-time data?
Well, after reading this article, you will.
I aim to take you from zero to hero in Google Cloud Platform (GCP) in just one article. I will show you how to:
Once I have explained all the topics in this list, I will share with you a solution to the system I described.
If you do not understand some parts of it, you can go back to the relevant sections. And if that is not enough, visit the links to the documentation that I have provided.
Are you up for a challenge? I have selected a few questions from old GCP Professional Certification exams. They will test your understanding of the concepts explained in this article.
I recommend trying to solve both the design and the questions on your own, going back to the guide if necessary. Once you have an answer, compare it to the proposed solution.
Try to go beyond what you are reading and ask yourself what would happen if requirement X changed:
And any other scenarios you can think of.
At the end of the day, you are not paid just for what you know but for your thought process and the decisions you make. That is why it is vitally important that you exercise this skill.
At the end of the article, I’ll provide more resources and next steps if you want to continue learning about GCP.
GCP currently offers a 3 month free trial with $300 US dollars of free credit. You can use it to get started, play around with GCP, and run experiments to decide if it is the right option for you.
You will NOT be charged at the end of your trial. You will be notified and your services will stop running unless you decide to upgrade your plan.
I strongly recommend using this trial to practice. To learn, you have to try things on your own, face problems, break things, and fix them. It doesn’t matter how good this guide is (or the official documentation for that matter) if you do not try things out.
Consuming resources from GCP, like storage or computing power, provides the following benefits:
GCP makes it easy to experiment and use the resources you need in an economical way.
In general, you will only be charged for the time your instances are running. Google will not charge you for stopped instances. However, if they consume resources, like disks or reserved IPs, you might incur charges.
Here are some ways you can optimize the cost of running your applications in GCP.
GCP provides different machine families with predefined amounts of RAM and CPUs:
Besides, you can create your custom machine with the amount of RAM and CPUs you need.
You can use preemptible virtual machines to save up to 80% of your costs. They are ideal for fault-tolerant, non-critical applications. You can save the progress of your job in a persistent disk using a shut-down script to continue where you left off.
Google may stop your instances at any time (with a 30-second warning) and will always stop them after 24 hours.
To reduce the chances of getting your VMs shut down, Google recommends:
Note: Start-up and shut-down scripts apply to non-preemptible VMS as well. You can use them the control the behavior of your machine when it starts or stops. For instance, to install software, download data, or backup logs.
There are two options to define these scripts:
This latter is preferred because it is easier to create many instances and to manage the script.
The longer you use your virtual machines (and Cloud SQL instances), the higher the discount - up to 30%. Google does this automatically for you.
You can get up to 57% discount if you commit to a certain amount of CPU and RAM resources for a period of 1 to 3 years.
To estimate your costs, use the Price Calculator. This helps prevent any surprises with your bills and create budget alerts.
In this section, I will explain how you can manage and administer your Google Cloud resources.
There are four types of resources that can be managed through Resource Manager:
There are quotas that limit the maximum number of resources you can create to prevent unexpected spikes in billing. However, most quotas can be increased by opening a support ticket.
Resources in GCP follow a hierarchy via a parent/child relationship, similar to a traditional file system, where:
This hierarchical organization helps you manage common aspects of your resources, such as access control and configuration settings.
You can create super admin accounts that have access to every resource in your organization. Since they are very powerful, make sure you follow Google’s best practices.
Labels are key-value pairs you can use to organize your resources in GCP. Once you attach a label to a resource (for instance, to a virtual machine), you can filter based on that label. This is useful also to break down your bills by labels.
Some common use cases:
These two similar concepts seem to generate some confusion. I have summarized the differences in this table:
Simply put, Cloud IAM controls who can do what on which resource. A resource can be a virtual machine, a database instance, a user, and so on.
It is important to notice that permissions are not directly assigned to users. Instead, they are bundled into roles, which are assigned to members. A policy is a collection of one or more bindings of a set of members to a role.
In a GCP project, identities are represented by Google accounts, created outside of GCP, and defined by an email address (not necessarily @gmail.com). There are different types:
Regarding service accounts, some of Google’s best practices include:
A role is a collection of permissions. There are three types of roles:
When assigning roles, follow the principle of least privilege, too. In general, prefer predefined over primitive roles.
Cloud Deployment Manager automates repeatable tasks like provisioning, configuration, and deployments for any number of machines.
It is Google’s Infrastructure as Code service, similar to Terraform - although you can deploy only GCP resources. It is used by GCP Marketplace to create pre-configured deployments.
You define your configuration in YAML files, listing the resources (created through API calls) you want to create and their properties. Resources are defined by their name (VM-1, disk-1), type (compute.v1.disk, compute.v1.instance) and properties (zone:europe-west4, boot:false).
To increase performance, resources are deployed in parallel. Therefore you need to specify any dependencies using references. For instance, do not create virtual machine VM-1 until the persistent disk disk-1 has been created. In contrast, Terraform would figure out the dependencies on its own.
You can modularize your configuration files using templates so that they can be independently updated and shared. Templates can be defined in Python or Jinja2. The contents of your templates will be inlined in the configuration file that references them.
Deployment Manager will create a manifest containing your original configuration, any templates you have imported, and the expanded list of all the resources you want to create.
Operations provide a set of tools for monitoring, logging, debugging, error reporting, profiling, and tracing of resources in GCP (AWS and even on-premise).
Cloud Logging is GCP’s centralized solution for real-time log management. For each of your projects, it allows you to store, search, analyze, monitor, and alert on logging data:
Logs are a named collection of log entries. Log entries record status or events and includes the name of its log, for example, compute.googleapis.com/activity. There are two main types of logs:
First, User Logs:
Second, Security logs, divided into:
They are specific to VPC networks (which I will introduce later). VPC flow logs record a sample of network flows sent from and received by VM instances, which can be later access in Cloud Logging.
They can be used to monitor network performance, usage, forensics, real-time security analysis, and expense optimization.
Note: you may want to log your billing data for analysis. In this case, you do not create a sink. You can directly export your reports to BigQuery.
Cloud Monitoring lets you monitor the performance of your applications and infrastructure, visualize it in dashboards, create uptime checks to detect resources that are down and alert you based on these checks so that you can fix problems in your environment. You can monitor resources in GCP, AWS, and even on-premise.
It is recommended to create a separate project for Cloud Monitoring since it can keep track of resources across multiple projects.
Also, it is recommended to install a monitoring agent in your virtual machines to send application metrics (including many third-party applications) to Cloud Monitoring. Otherwise, Cloud Monitoring will only display CPU, disk traffic, network traffic, and uptime metrics.
To receive alerts, you must declare an alerting policy. An alerting policy defines the conditions under which a service is considered unhealthy. When the conditions are met, a new incident will be created and notifications will be sent (via email, Slack, SMS, PagerDuty, etc).
A policy belongs to an individual workspace, which can contain a maximum of 500 policies.
Trace helps find bottlenecks in your services. You can use this service to figure out how long it takes to handle a request, which microservice takes the longest to respond, where to focus to reduce the overall latency, and so on.
It is enabled by default for applications running on Google App Engine (GAE) - Standard environment - but can be used for applications running on GCE, GKE, and Google App Engine Flexible.
Error Reporting will aggregate and display errors produced in services written in Go, Java, Node.js, PHP, Python, Ruby, or .NET. running on GCE, GKE, GAP, Cloud Functions, or Cloud Run.
Debug lets you inspect the application’s state without stopping your service. Currently supported for Java, Go, Node.js and Python. It is automatically integrated with GAE but can be used on GCE, GKE, and Cloud Run.
Profiler that continuously gathers CPU usage and memory-allocation information from your applications. To use it, you need to install a profiling agent.
#cloud #programming #web-development #developer