Exploring Google Kubernetes Engine by creating a complete Node.js / React solution. Serving our application through Google Kubernetes Engine. Securing our application using Google Kubernetes Engine. Beginning to incorporate data persistence. Data persistence in production.
In reading about Kubernetes, I have come to learn that it has the potential to disrupt the PaaS model by leveraging Docker to provide a similar developer-friendly mechanism to deploy applications.
Kubernetes (k8s) is an open-source system for automating deployment, scaling, and management of containerized applications.> Kubernetes (k8s) is an open-source system for automating deployment, scaling, and management of containerized applications.
_— Kubernetes — _Kubernetes
Goal
The goal is create a complete solution, development to production, using Google Kubernetes Engine consisting of:
The final solution is available for download.
Prerequisites
The latest versions of Docker Engine and Docker Compose (both included in Docker Desktop); versions 18.09.1 and 1.23.2 as of this writing.
The latest version of Node.js LTS; version 10.15.0 LTS as of this writing.
A text editor; cannot recommend enough the no-cost Visual Studio Code with the following extensions:
Node.js / Express Project
We will start with a basic Node.js / Express project as described in the Backend section of another article that I wrote: Tech Stack 2019 Core.
Docker
The assumption is that the reader is already familiar with Docker basics; the Docker Get Started, Part 1: Orientation and SetupandGet Started, Part 2: Containers documentation is sufficient.
Let us start by simply building / running a Docker image / container providing the Node.js / Express application. We start by creating a Docker configuration file:
Dockerfile
FROM node:10.15.0-alpine
WORKDIR /app
COPY . /app
RUN ["npm", "install"]
RUN ["npm", "run", "build-ts"]
EXPOSE 3000
CMD ["npm", "run", "start"]
Much like our .gitignore, we do not want to copy the node_modules and dist folders into the container; we will rather install and build these files as part of building the image.
.dockerignore
node_modules
dist
We build the Docker image using the command:
docker build --tag=hellokubernetes .
and run a Docker container based on the image:
docker run -d -p 3000:3000 hellokubernetes
At this point, we can observe the Docker images and containers:
We can also open a browser and see the API result.
note: For now, we will hold off on publishing images as described in the Docker documentation.
Docker Compose
The assumption is that the reader is already familiar with Docker Compose basics; the Docker Overview of Docker Composeis sufficient.
Let us first stop / remove all the Docker containers and remove all the images from our previous steps.
note: These commands assume that you are not using Docker for any other purposes; otherwise these operations can be targeted.
docker stop $(docker ps -a -q)
docker rm $(docker ps -a -q)
docker rmi $(docker images -a -q)
We can simplify building the Docker image and running the container by creating a Docker Compose configuration file:
docker-compose.yaml
version: '3'
services:
web:
build: .
ports:
- "3000:3000"
and then executing:
docker-compose up -d
At this point, we can observe the Docker images and containers:
And as before, we can also open a browser and see the API result.
Docker Compose Development
We will be using Kubernetes for non-development deployments, at the same time we will be using Docker Compose exclusively for development. With this in mind, we need to refactor our previous implementation to allow for the live build / restart of the application to support our development workflow.
Let us first stop / remove all the Docker containers and remove all the images from our previous steps.
docker stop $(docker ps -a -q)
docker rm $(docker ps -a -q)
docker rmi $(docker images -a -q)
In preparation for our updated Docker Compose configuration, we need to be able to run the development tools via a single npm script (currently requires running both the watch-fs and watch-node scripts). We use the concurrently package to accomplish this.
npm install -D concurrently
And add a develop script:
{
...
"script": {
...
"develop": "npm run build-ts && concurrently 'npm:watch-ts' 'npm:watch-node'"
}
...
}
Observations:
We then create a separate Docker configuration file:
Dockerfile-develop
FROM node:10.15.0-alpine
WORKDIR /app
EXPOSE 3000
CMD ["npm", "run", "develop"]
Observations:
We then update the Docker Compose configuration file:
docker-compose.yaml
version: '3'
services:
web:
build:
context: .
dockerfile: Dockerfile-develop
ports:
- "3000:3000"
volumes:
- .:/app
With this in place, we can execute:
docker-compose up -d
With this in place we can open a browser and see the API result. Also, editing the source files will trigger a build and restart the application.
Google Container Registry
We will need an online location to store Docker images in order to deploy them using Google Kubernetes Engine; right now our images are stored in our local Docker registry. Also, it is likely that we will want to keep these images private; so we will need a private registry.
note: In my first pass at this article, I explored using a private registry at Docker Hub. I, however, discovered that while Kubernetes supports private registries, it was a more complicated solution (requiring setting up security keys and separate billing).
Given that we are going to be using Google Kubernetes Engine, the tightly integrated Google Container Registry is a natural solution for our private registry.
Before continuing, there are some important Docker concepts to understand (not well covered in the official Docker documentation):
Kubernetes (k8s) is an open-source system for automating deployment, scaling, and management of containerized applications.> Kubernetes (k8s) is an open-source system for automating deployment, scaling, and management of containerized applications.> Kubernetes (k8s) is an open-source system for automating deployment, scaling, and management of containerized applications.> Kubernetes (k8s) is an open-source system for automating deployment, scaling, and management of containerized applications.> Kubernetes (k8s) is an open-source system for automating deployment, scaling, and management of containerized applications.> Kubernetes (k8s) is an open-source system for automating deployment, scaling, and management of containerized applications.> Kubernetes (k8s) is an open-source system for automating deployment, scaling, and management of containerized applications.
— _Adrian Moaut — _Using Docker, Developing and deploying Software with Containers
Also, if you are going to follow along, you will need to follow the Quickstart for Container Registry to enable the feature in a Google Cloud Platform project (that you will also likely need to create) and install the gcloud command-line tool.
We next need to create an image in our local registry, in the hellokubernetes repository, and with the 1.0.0 tag:
docker build --tag=hellokubernetes:1.0.0 .
We then authorize Docker using credentials supplied to gcloud:
gcloud auth configure-docker
We tag the local image for Google Container Registry:
docker tag hellokubernetes:1.0.0 gcr.io/[PROJECT-ID]/hellokubernetes:1.0.0
Observations:
We finally push this tagged image to Google Container Registry:
docker push gcr.io/[PROJECT-ID]/hellokubernetes:1.0.0
Google Kubernetes Engine (GKE)
Now that we have stored our Docker image online with Google Cloud Registry, we will run a container based on it using GKE.
The assumption is that the reader is already familiar with GKE basics; for this there is an excellent article Kubernetes 101: Pods, Nodes, Containers, and Clusters(it is actually a three-part series and all are helpful).
note: If you read the official Docker documentation, they define similar (but different) concepts: stacks, swarms, and clusters. Since we are using Kubernetes, these concepts do not apply.
Also, if you are looking to follow along, you will have to setup a GKE enabled Google Cloud Platform project (the same project used for Google Container Repository) and have access to both the gcloud and kubectl command-line tools; instructions are available at GKE Quickstart.
We first create a cluster: mycluster:
gcloud container clusters create mycluster
and get the credentials so that we can interact with it:
gcloud container clusters get-credentials mycluster
Because we have a stateless application, we create a Kubernetes deployment.
k8s/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: hellokubernetes
spec:
replicas: 1
selector:
matchLabels:
app: hellokubernetes
template:
metadata:
labels:
app: hellokubernetes
spec:
containers:
- name: hellokubernetes
image: gcr.io/[PROJECT-ID]/hellokubernetes:1.0.0
ports:
- containerPort: 3000
Observations:
We apply this deployment:
kubectl apply -f deployment.yaml
We can see that were successful:
kubectl get deployments
We now have the left side of this final diagram built:
Per the diagram, our last step will be to create a load balancer servicewith an external IP address. The load balancer exposes port 80 that is mapping to port 3000 on pods labeled app: kubernetes.
k8s/service-load-balancer.yaml
apiVersion: v1
kind: Service
metadata:
name: hellokubernetes
spec:
selector:
app: hellokubernetes
type: LoadBalancer
ports:
- port: 80
targetPort: 3000
We apply the service:
kubectl apply -f service-load-balancer.yaml
We can see that were successful:
kubectl get service
The final validation is opening the external IP address in a browser.
Overview
As you may have noticed, the previous example was not secure; it used HTTP instead of HTTPS. This is because we used a LoadBalancer instead of an Ingress service:
Kubernetes (k8s) is an open-source system for automating deployment, scaling, and management of containerized applications.
_— GKE — _Setting up HTTP Load Balancing with Ingress
We start by first deleting the hellokubernetes and kubernetes (automatically generated) services:
kubectl delete service hellokubernetes
kubectl delete service kubernetes
We, however, will continue to use the deployment that we created earlier.
The rest of this article closely follows the official tutorial Setting up HTTP Load Balancing with Ingress, providing concrete examples along the way.
NodePort
We first need to create a NodePort service in preparation to create an Ingress service.
Kubernetes (k8s) is an open-source system for automating deployment, scaling, and management of containerized applications.
_— Kubernetes — _Services
Ok, if you don’t understand this definition, I am with you; unfortunately I could not find a better explanation in my searches. So let me give an explanation a shot using the following diagram (what we are working towards):
First NodePort services are a low-level service, used by other services, e.g., an Ingress service. They also interact with the actual underlying nodes supporting the cluster.
note: Unless otherwise specified, clusters default to having three nodes.
For each Pod and exposed port (e.g., 3000) that a NodePort service specifies, the NodePort defines a map to a randomly (e.g., 30677) assigned port that is allocated on each Node (same port on each Node). This mapping provides a mechanism to allow other services to direct traffic to a particular Pod without for knowledge of which Node is it running on.
The following is our NodePort configuration file:
k8s/service-node-port.yaml
apiVersion: v1
kind: Service
metadata:
name: hellokubernetes
spec:
selector:
app: hellokubernetes
type: NodePort
ports:
- port: 3000
and applying:
kubectl apply -f service-node-port.yaml
and listing:
Ingress
Now that we have our NodePort service, we create a an Ingress service that points to it:
k8s/service-ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: hellokubernetes
spec:
backend:
serviceName: hellokubernetes
servicePort: 3000
and applying:
kubectl apply -f service-ingress.yaml
and listing:
Observations:
Ingress with Static IP
In order to use HTTPS, we need a DNS entry. In order to use a DNS entry, we need a static IP address. We use Google Cloud to create a static IP address:
gcloud compute addresses create web-static-ip --global
and list it:
We then update our Ingress service by adding a reference to this static IP address:
k8s/service-ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: hellokubernetes
annotations:
kubernetes.io/ingress.global-static-ip-name: "web-static-ip"
...
and re-applying:
kubectl apply -f service-ingress.yaml
and listing:
Secret
Now that we have a static IP address, the next step towards securing our API with HTTPS is to create a domain name mapping to the static IP address. In my case, created an A record for my larkintuckerllc.com domain (hosted on GoDaddy) to the static IP address.
note: I also created a CNAME record for www.larkintuckerllc.com pointing to the A record.
Next, we need to create a HTTPS certificate; I ended paying $8 a year to obtain a PositiveSSL certificate (works for both larkintuckerllc.com and www.larkintuckerllc.com).
note: Getting the no-cost Let’s Encrypt to work with GKE seemed overly complicated; see Let’s Encrypt on GKE.
Having obtained a private key and HTTPS certificate, the next step was to BASE64 encode (single line) them. On MacOS, the command is:
openssl base64 -A -in INFILE -out OUTFILE
Next we create a file:
k8s/secret.yaml
apiVersion: v1
data:
tls.crt: base64 encoded cert
tls.key: base64 encoded key
kind: Secret
metadata:
name: hellokubernetes-tls
namespace: default
type: Opaque
Observations:
and applying:
kubectl apply -f secret.yaml
Ingress with Static IP and TLS
We finally add a tls entry to our final Ingress configuration:
k8s/service-ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: hellokubernetes
annotations:
kubernetes.io/ingress.global-static-ip-name: "web-static-ip"
spec:
tls:
- hosts:
- larkintuckerllc.com
- www.larkintuckerllc.com
secretName: hellokubernetes-tls
backend:
serviceName: hellokubernetes
servicePort: 3000
and re-applying:
kubectl apply -f service-ingress.yaml
Whew… We have now secured the API:
Development Database
For development, we will run our database using a Docker container; using the Docker Compose instructions at Docker Hub: postgres. Simply amounts to adding the db and adminer sections to our Docker Compose configuration:
docker-compose.yml
version: '3'
services:
web:
build:
context: .
dockerfile: Dockerfile-develop
ports:
- "3000:3000"
volumes:
- .:/app
db:
image: postgres:11.1
restart: always
environment:
POSTGRES_PASSWORD: example
adminer:
image: adminer
restart: always
ports:
- 8080:8080
Observations:
We rebuild and restart the Docker containers using:
docker-compose up
docker-compose start
Because we do not want to run our application using database administrator access, we create a new database, create a new user, and grant access to the user to the database. To accomplish this, we will use the command-line tool, psql, available in the postgres container.
docker exec -it hello-kubernetes_db_1 /bin/bash
runuser -l postgres -c psql
note: The SQL commands can be also executed using a web interface provided by the the adminer container.
The SQL commands we need to execute are:
create database hellokubernetes;
create user hellouser with password 'hellopassword';
grant all privileges on database hellokubernetes to hellouser;
With this, we will can access the database using an URL (from the web container): postgres://hellouser:hellopassword@db/hellokubernetes
Development Updates
With the database in place, we update the application to use it; using the TypeORM library. These steps closely follow the first article in another series that I wrote: TypeORM By Example: Part 1.
We first install the dependencies:
npm install pg
npm install @types/pg
npm install typeorm
npm install reflect-metadata
As TypeORM uses the experimental TypeScript decorator syntax, we need to update:
tsconfig.json
{
"compilerOptions": {
...
"emitDecoratorMetadata": true,
"experimentalDecorators": true
},
...
}
We also need to create a TypeORM configuration file; provides information about database access among other things:
note: In practice, it is important to keep secrets out of configuration files, e.g., the following script incorrectly contains a password. For the purposes of keeping things simpler, will not worry about it now. The correct answer involves using an environment variable which we would set as part of the Deployment configuration.
ormconfig.json
{
"type": "postgres",
"host": "db",
"port": 5432,
"username": "hellouser",
"password": "hellopassword",
"database": "hellokubernetes",
"synchronize": false,
"migrationsRun": true,
"logging": false,
"entities": [
"dist/entity/**/*.js"
],
"migrations": [
"dist/migration/**/*.js"
],
"subscribers": [
"dist/subscriber/**/*.js"
],
"cli": {
"entitiesDir": "src/entity",
"migrationsDir": "src/migration",
"subscribersDir": "src/subscriber"
}
}
Observations:
Let us create a Todo entity:
src/entity/Todo.ts
import { Column, Entity, PrimaryGeneratedColumn } from 'typeorm';
@Entity()
export class Todo {
@PrimaryGeneratedColumn()
public id: number;
@Column()
public name: string = '';
@Column()
public isComplete: boolean = false;
}
export default Todo;
We login to the web container and generate the migration:
docker exec -it hello-kubernetes_web_1 /bin/sh
./node_modules/.bin/typeorm migration:generate -n Initialize
Observations:
We update the server application to run the migration and use the Todo entity:
src/server.ts
import cors from 'cors';
import express from 'express';
import 'reflect-metadata';
import { createConnection } from 'typeorm';
import Todo from './entity/Todo';
createConnection()
.then(async connection => {
const app = express();
app.use(cors());
app.get('/', (req, res) => res.send({ hello: 'world' }));
app.get('/create', async (req, res) => {
const todo = new Todo();
todo.name = 'A Todo';
await connection.manager.save(todo);
res.send(todo);
});
app.get('/read', async (req, res) => {
const todos = await connection.manager.find(Todo);
res.send(todos);
});
app.listen(3000, () => console.log('Example app listening on port 3000!'));
})
.catch(error => console.log(error));
Remember, our development container automatically reloads on file changes. As we can see from using adminer tool_,_ we now have a two new tables; todos holding the Todo entities and migrations keep track of the executed migrations:
Also, we have two new endpoints:
to exercise the persistence.
Finally, we update the application’s version:
package.json
{
...
"version": "1.1.0",
...
}
Production Database — PersistentVolumeClaim
We will generally follow the instructions in Using Persistent Disks with WordPress and MySQLto setup our production database. The first step is to create a PersistentVolumeClaim because:
Kubernetes (k8s) is an open-source system for automating deployment, scaling, and management of containerized applications.> Kubernetes (k8s) is an open-source system for automating deployment, scaling, and management of containerized applications.
_— GKE — _Using Persistent Disks with WordPress and MySQL
We create a small 10Gi volume that we can use for our PostgreSQL instance:
k8s/persistent-volume-claim.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: hellokubernetes-volumeclaim
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
Observations:
We apply the configuration:
kubectl apply -f persistent-volume-claim.yaml
and view the result:
kubectl get pvc
Production Database — Secret
While our database will only be accessible from resources inside our GKE cluster, we will still want to change the default postgres (administrator) password used by the postgres Docker image.
We can accomplish this by creating a Secret:
k8s/secret-postgres.yaml
apiVersion: v1
data:
password: base64 encoded password
kind: Secret
metadata:
name: hellokubernetes-postgres
namespace: default
type: Opaque
Observations:
We apply the configuration:
kubectl apply -f secret-postgres.yaml
and view the result:
kubectl get secret
Production Database — Deployment
We now create the database Deployment configuration:
k8s/deployment-postgres.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: hellokubernetes-postgres
spec:
replicas: 1
selector:
matchLabels:
app: hellokubernetes-postgres
template:
metadata:
labels:
app: hellokubernetes-postgres
spec:
containers:
- name: hellokubernetes-postgres
image: postgres:11.1
env:
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: hellokubernetes-postgres
key: password
- name: PGDATA
value: "/var/lib/postgresql/data/pgdata"
ports:
- containerPort: 5432
volumeMounts:
- name: hellokubernetes-persistent-storage
mountPath: /var/lib/postgresql/data
volumes:
- name: hellokubernetes-persistent-storage
persistentVolumeClaim:
claimName: hellokubernetes-volumeclaim
Observations:
We then apply:
kubectl apply -f deployment-postgres.yaml
and then view the result:
kubectl get deployment
Production Database — Initial Setup
Much like we did for the development database, we need an initial setup (new database, new user, and user having access to database) for the production database. To accomplish this, we need to login to the Container in the Pod generated by the Deployment. We first get the Pod’s name:
kubectl get pod
We then login to the Containerusing the Pod’s name:
kubectl exec -it hellokubernetes-postgres-6dcf55cd85-868d7 -- /bin/bash
note: Similarly one can access a Container’s logs using the kubectl log command using the Pod’s name.
We connect to the database using the command:
runuser -l postgres -c psql
and execute the following SQL commands:
create database hellokubernetes;
create user hellouser with password 'hellopassword';
grant all privileges on database hellokubernetes to hellouser;
note: It was about this point, when I panicked as I recalled that Pods (and their Containers) were ephemeral (meaning changes made to them will not last). We seeming just applied some changes directly on the database Container. Then, I was relieved to remember that PostgreSQL maintains metadata (tables, users, etc.) in a database itself (which is stored on the PersistentVolume.
Production Database — Service
Unlike our local Docker Compose setup, Containers (in Pods) cannot communicate with other Containers in other Pods; we need to create a ClusterIP Service to accomplish this. We can visualize this as such:
We create the ClusterIP Service configuration:
k8s/service-cluster-ip.yaml
apiVersion: v1
kind: Service
metadata:
name: db
spec:
type: ClusterIP
ports:
- port: 5432
selector:
app: hellokubernetes-postgres
Observations:
We apply:
kubectl apply -f service-cluster-ip.yaml
and view:
kubectl get service
Production Updates
The last step is to update the image, with the new code using the database, in the deployment serving up the API.
We first build a new image and tag it locally:
docker build --tag=hellokubernetes:1.1.0 .
docker tag hellokubernetes:1.1.0 gcr.io/[PROJECT-ID]/hellokubernetes:1.1.0
When the push the image to the Google Container Registry:
docker push gcr.io/[PROJECT-ID]/hellokubernetes:1.1.0
Finally we update the image in the deployment:
kubectl set image deployment/hellokubernetes hellokubernetes=gcr.io/[PROJECT-ID]/hellokubernetes:1.1.0
We can now see that the API has the new code, e.g., below is the read endpoint being viewed (after having used the create endpoint).
Wrap Up
Decided to stop here with the series; covered most of the core concepts to get one going with it.
To sum up my experience learning GKE:
☞ Learn Kubernetes from a DevOps guru (Kubernetes + Docker)
☞ Learn DevOps: The Complete Kubernetes Course
☞ Kubernetes for the Absolute Beginners - Hands-on
☞ Complete DevOps Gitlab & Kubernetes: Best Practices Bootcamp
☞ Learn DevOps: On-Prem or Cloud Agnostic Kubernetes
☞ Master Jenkins CI For DevOps and Developers
☞ Docker Technologies for DevOps and Developers
☞ DevOps Toolkit: Learn Kubernetes with Practical Exercises!
#kubernetes #docker #devops