Kubernetes Services simply visually explained

TL;DR

There are four main services, with ClusterIP being the holy grail:

I would like you to imagine that if you create a NodePort service it also creates a ClusterIP one. And if you create a LoadBalancer it creates a NodePort which then creates a ClusterIP. If you do this, k8s services will be easy. We will walk through this in this article.

Services and Pods

Services point to pods. Services do not point to deployments or replicasets. Services point to pods directly using labels. This gives great flexibility because it doesn’t matter through which various (maybe even customized) ways pods have been created.

We’ll start with a simple example which we extend step by step with different service types to see how those are build on top of each other.

No Services

We start without any services.

We have two nodes, one pod. Nodes have external (4.4.4.1, 4.4.4.2) and internal (1.1.1.1, 1.1.1.2) IP addresses. The pod pod-python has only an internal one.

Now we add a second pod pod-nginx which got scheduled on node-1. This wouldn’t have to be the case and doesn’t matter for connectivity. In Kubernetes, all pods can reach all pods on their internal IP addresses, no matter on which nodes they are running.

This means pod-nginx can ping and connect to pod-python using its internal IP 1.1.1.3.

Now let’s consider the pod-python dies and a new one is created. (We don’t handle how pods might be managed and controlled in this article.) Suddenly pod-nginx cannot reach 1.1.1.3 any longer, and suddenly the world bursts into horrific flames… but to prevent this we create our first service!

ClusterIP

Same scenario, but we configured a ClusterIP service. A service is not scheduled on a specific node like pods. For this article it is enough to assume a service is just available in memory inside the whole cluster.

Pod-nginx can always safely connect to 1.1.10.1 or the dns name service-python and gets redirected to a living python pod. Beautiful. No flames. Sunshine.

We extend the example, spin up 3 instances of python and we now display the ports of the internal IP addresses of all pods and services.

All pods inside the cluster can reach the python pods on their port 443 via http://1.1.10.1:3000 or http://service-python:3000. The ClusterIP service-python distributes the requests based on a random or round-robin approach. That’s what a ClusterIP service does, it makes pods available inside the cluster via a name and an IP.

The service-python in the above image could for have this yaml:

apiVersion: v1
kind: Service
metadata:
  name: service-python
spec:
  ports:
  - port: 3000
    protocol: TCP
    targetPort: 443
  selector:
    run: pod-python
  type: ClusterIP

Running kubectl get svc :

NodePort

Now we would like to make the ClusterIP service available from the outside and for this we convert it into a NodePort one. In our example we convert the service-python with just two simple yaml changes:

apiVersion: v1
kind: Service
metadata:
  name: service-python
spec:
  ports:
  - port: 3000
    protocol: TCP
    targetPort: 443
    nodePort: 30080
  selector:
    run: pod-python
  type: NodePort

This means our internal service-python will now also be reachable from every nodes internal and external IP address on port 30080.

A pod inside the cluster could also connect to an internal node IP on port 30080.

Running kubectl get svc shows the same cluster ip. Just the different type and additional node port:

Internally the NodePort service still acts as the ClusterIP service before. It helps to imagine that a NodePort service creates a ClusterIP service, even though there is no extra ClusterIP object any more.

LoadBalancer

We use a LoadBalancer service if we would like to have a single IP which distributes requests (using some method like round robin) to all our external nodes IPs. So it is built on top of a NodePort service:

Imagine that a LoadBalancer service creates a NodePort service which creates a ClusterIP service. The changed yaml for LoadBalancer as opposed to the NodePort before is simply:

apiVersion: v1
kind: Service
metadata:
  name: service-python
spec:
  ports:
  - port: 3000
    protocol: TCP
    targetPort: 443
    nodePort: 30080
  selector:
    run: pod-python
  type: LoadBalancer

All a LoadBalancer service does is it creates a NodePort service. In addition it sends a message to the provider who hosts the Kubernetes cluster asking for a loadbalancer to be setup pointing to all external node IPs and specific nodePort. If the provider doesn’t support the request message, well then nothing happens and the LoadBalancer would be equal to a NodePort service.

Running kubectl get svc shows just the addition of the EXTERNAL-IP and different type:

The LoadBalancer service still opens port 30080 on the nodes internal and external IPs as before. And it still acts like a ClusterIP service.

ExternalName

Finally the ExternalName service, which could be considered a bit separated and not on the same stack as the 3 we handled before. In short: it creates an internal service with an endpoint pointing to a DNS name.

Taking our early example we now assume that the pod-nginx is already in our shiny new Kubernetes cluster. But the python api is still outside:

Here pod-nginx has to connect to http://remote.server.url.com , which works, for sure. But soon we would like to integrate that python api into the cluster and till then, we can create an ExternalName service:

This could be done using this yaml:

kind: Service
apiVersion: v1
metadata:
  name: service-python
spec:
  ports:
  - port: 3000
    protocol: TCP
    targetPort: 443
  type: ExternalName
  externalName: remote.server.url.com

Now pod-nginx can simply connect to http://service-python:3000, just like with a ClusterIP service. When we finally decide to migrate the python api as well in our beautiful stunning Kubernetes cluster, we only have to change the service to a ClusterIP one with the correct labels set:

The big advantage when using ExternalName services is that you can already create your complete Kubernetes infrastructure and also already apply rules and restrictions based on services and IPs, even though some services might still be outside.

TL;DR

Kubernetes Ingress is not a Kubernetes Service. Very simplified its just a Nginx Pod which redirects requests to other internal (ClusterIP) services. This Pod itself is made reachable through a Kubernetes Service, most commonly a LoadBalancer.

Should you even read this?

First I hope I can give you a clear and simple overview of what is behind this mysterious Kubernetes Ingress which then makes it easier for you to understand what you’re actually implementing or even if you should.

Later I show some example configuration based on the example we use throughout this article.

Why to use Ingress?

You use it to make internal services reachable from outside your cluster. It saves you precious static IPs, as you won’t need to declare multiple LoadBalancer services. Also it allows for much more configuration and easier setup as we’ll see.

What will we do here?

• First we do a really short excursion into how http servers, especially Nginx, work and what they can do.
• Then we’ll show how an Ingress could be setup manually, so without using the fancy Kubernetes Ingress resource at all.
• Next we’ll see that Kubernetes Ingress is nothing more than a pre-configured Nginx server.

Sounds confusing? Just follow me through here.

Short dive into the world of a simple HTTP server

Here we step back into times before containers, Kubernetes and the modern world of Cloud. Stay with me, it’ll be short.

What can a (Nginx) HTTP server do?

It can receive a request over the HTTP protocol for a specific filepath, check that filepath on the attached filesystem and return it if that file exists:

In Nginx this could be for example done with something like:

location /folder {
    root /var/www/;
    index index.html;
}

What can a (Nginx) HTTP server also do?

It can receive a request for a specific filepath, redirect that request to another server (which means it acts as proxy) and then redirects the response of that server to back to the client. For the client nothing changes, the received result is still the requested file (if it exists).

We won’t dive deep into this but in Nginx a proxy redirect could for example be configured like:

location /folder {
    proxy_pass http://second-nginx-server:8000;
}

This means Nginx can serve files from a filesystem or redirect responses to other servers and return their responses, by acting as a proxy.

A simple Kubernetes example:

Use ClusterIP services

Again, from this point on you should understand Kubernetes Services. We have two worker nodes, we ignore master nodes here. We have two services service-nginx and service-python which point to various pods.

Services are not scheduled on any specific Node, lets just say they are “available everywhere in the cluster”.

You should understand whats happening here. Internally in our cluster we can reach the Nginx pods and the Python pods through their services. Next we would like to make those available from outside the cluster as well. So we convert those into LoadBalancer services:

Use LoadBalancer Services

You can see that we converted the ClusterIP services into LoadBalancer services. Because we have our Kubernetes Cluster hosted with a Cloud Provider which can handle this (Gcloud, AWS, DigitalOcean…), it creates two external load-balancers which redirect requests to our external Node IPs which then redirect to the internal ClusterIP services.

We see two LoadBalancers, each having its own IP. If we send a request to LoadBalancer 22.33.44.55 it gets redirected to our internal service-nginx. If we send the request to 77.66.55.44 it gets redirected to our internal service-python.

This works great! But IP addresses are rare and LoadBalancer pricing depends on the cloud providers. Now imagine we don’t have just two but many more internal services for which we would like to create LoadBalancers, costs would scale up.

Might there be another solution which allows us to only use one LoadBalancer (with one IP) but still reach both of our internal services directly? Let’s explore this first by implementing a manual (non Kubernetes) approach.

Manually configure a Nginx Service as proxy

As described earlier, Nginx can act as a proxy. In the following image we see a new service called service-nginx-proxy which is actually our only LoadBalancer service. The service-nginx-proxy would still point to one or multiple Nginx-pod-endpoints, but for simplicity I didn’t include this in the graphic. The other two services from before are converted back to simple ClusterIP ones:

We can see that we only hit one LoadBalancer (11.22.33.44) but with different http urls, the requests are displayed in yellow as its the same target and just contains different content (request urls).

The service service-nginx-proxy decides (by using Nginx proxy passes and locations), depending on the passed urls, to which service he should redirect the request.

In this case we have two choices, red and blue. Red redirects to service-nginx where blue redirects to service-python.

# very simplified Nginx config example
location /folder {
    proxy_pass http://service-nginx:3001;
}
location /other {
    proxy_pass http://service-python:3002;
}

Currently we would need to configure the service-nginx-proxy manually. Like creating the proper Nginx configuration files pointing to our ClusterIP services. This is very much a possible, working and common solution.

And because this was/is a common solution, Kubernetes Ingress was created to make the configuration easier and more manageable.

From this point on you should understand the advantage of the above example shown in the image. If you don’t, feel free to add a comment below to discuss.

Use Kubernetes Ingress in our example

Compare the following image to the previous one. Really not much changed. We just used a pre-configured Nginx (Kubernetes Ingress) which does already all proxy redirection for us which saves us a lot of manually configuration work:

That’s all there is to understanding what Kubernetes Ingress is. Now let’s move to some example configuration.

Install Kubernetes Ingress Controller

Kubernetes Ingress is an additional Kubernetes Resources which can be installed by:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.24.1/deploy/mandatory.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.24.1/deploy/provider/cloud-generic.yaml

Using the following command you’ll see the k8s resources which get installed in the namespace ingress-nginx:

kubectl get svc,pod --namespace=ingress-nginx

You see a normal LoadBalancer service with an External IP and a belonging pod. You can even kubectl exec into that pod to see it contains a pre-configured Nginx server:

Inside the nginx.conf you would see the various proxy redirects settings and other related configuration.

Example Kubernetes Ingress Config

An example Ingress yaml for the example we’ve been using could look like this:

# just example, not tested
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
  namespace: default
  name: test-ingress
spec:
  rules:
  - http:
      paths:
      - path: /folder
        backend:
          serviceName: service-nginx
          servicePort: 3001
  - http:
      paths:
      - path: /other
        backend:
          serviceName: service-python
          servicePort: 3002

We would need to create this yaml like any other resource through kubectl create -f ingress.yaml. This yaml will then be converted by the previously installed Ingress Controller into Nginx configuration.

Example Kubernetes Ingress / Different Namespaces

Now what if one of your internal services, to which the Ingress should redirect to, is in a different namespace? Because an Ingress Resource you define is namespaced. Inside your Ingress configuration you can only redirect to services in the same namespace.

If you define multiple Ingress yaml configurations, then those are merged together into one Nginx configuration by the one single Ingress Controller. Meaning: all are using the same LoadBalancer IP as well.

So let’s consider service-nginx is in namespace default:

# just example, not tested
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
  namespace: default
  name: ingress1
spec:
  rules:
  - http:
      paths:
      - path: /folder
        backend:
          serviceName: service-nginx
          servicePort: 3001

And then service-python is in namespace namespace2:

# just example, not tested
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
  namespace: namespace2
  name: ingress2
spec:
  rules:
  - http:
      paths:
      - path: /other
        backend:
          serviceName: service-python
          servicePort: 3002

Hence we create two Ingress yaml resources.

How to fine tune the Ingress Nginx configuration?

You can do this by annotations on the Inhgress Kubernetes Resource. For example you can configure various options you could normally configure directly in Nginx:

kind: Ingress
metadata:
  name: ingress
  annotations:
      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/proxy-connect-timeout: '30'
      nginx.ingress.kubernetes.io/proxy-send-timeout: '500'
      nginx.ingress.kubernetes.io/proxy-read-timeout: '500'
      nginx.ingress.kubernetes.io/send-timeout: "500"
      nginx.ingress.kubernetes.io/enable-cors: "true"
      nginx.ingress.kubernetes.io/cors-allow-methods: "*"
      nginx.ingress.kubernetes.io/cors-allow-origin: "*"
...

You can even do very specific rules like:

nginx.ingress.kubernetes.io/configuration-snippet: |
  if ($host = 'www.wuestkamp.com' ) {
    rewrite ^ https://wuestkamp.com$request_uri permanent;
  }

Using www. is “so 2008” !

These annotations will then be translated into Nginx configuration. You can always check these by manually connecting (kubectl exec) into the ingress Nginx pod and look at the config.

There are various configuration examples:

https://github.com/kubernetes/ingress-nginx/tree/master/docs/user-guide/nginx-configuration

https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/nginx-configuration/annotations.md#lua-resty-waf

Check the Ingress / Nginx Logs

Figuring out issues or errors it’s also helpful to look at the Ingress logs:

kubectl logs -n ingress-nginx ingress-nginx-controller-6cfd5b6544-k2r4n

Use Curl for testing your settings

If you want to test your Ingress/Nginx redirection rules it might be a good idea to use curl -v yourhost.com instead of your browser to avoid caching etc.

Ways of redirection / Ingress Rules

In the examples in this article we used paths like /folder or /other/directory to redirect to different services. This is called “A list of paths”.

It’s also possible to distinguish requests by their hostname to for example redirect api.myurl.com and website.myurl.com to different internal ClusterIP services. This could look like this:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: simple-fanout-example
spec:
  rules:
  - host: api.myurl.com
    http:
      paths:
      - path: /foo
        backend:
          serviceName: service1
          servicePort: 4200
      - path: /bar
        backend:
          serviceName: service2
          servicePort: 8080
  - host: website.myurl.com
    http:
      paths:
      - path: /
        backend:
          serviceName: service3
          servicePort: 3333

In this example we see that for a specific hostname we redirect different http paths to different internal services.

SSL / HTTPS

SSL. Have you heard of it? :) You probably want to make your website reachable via secure https. Kubernetes Ingress provides quite easy “TLS Termination”, which means that it handles all SSL communication, decrypts/terminates the SSL request and then sends those decrypted to your internal services.

This is great if multiple of your internal services are using the same (maybe even wildcard) SSL certificate, because then you only have to configure it once on your Ingress and not on all of your other internal services. The Ingress can use the SSL certificate from a configured TLS Kubernetes Secret.

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: tls-example-ingress
spec:
  tls:
  - hosts:
    - sslexample.foo.com
    secretName: testsecret-tls
  rules:
    - host: sslexample.foo.com
      http:
        paths:
        - path: /
          backend:
            serviceName: service1
            servicePort: 80

You just have to make sure that if you have multiple Ingress resources in different namespaces, your TLS secret also needs to be available in all namespaces where you defined an Ingress resource using it.

##Recap

I just wanted to give you a broad overview over whats behind the mysterious Kubernetes Ingress. Simplified: its nothing more than a way to easily configure a Nginx server which redirects requests to other internal services.

This saves you precious static IPs and LoadBalancers. But Kubernetes Ingress shouldn’t be considered as one of the Kubernetes Services. Ingress itself isn’t a Kubernetes Services, but it normally uses one, mainly the LoadBalancer.

Notice that there are also other Kubernetes Ingress types which don’t internally setup an Nginx service but might use other proxy technologies.

#Kubernetes #programming