Automated Testing for Terraform, Docker, Packer, Kubernetes, and More

Automated Testing for Terraform, Docker, Packer, Kubernetes, and More

This automated testing guide explains how to write automated tests for infrastructure code, including the code written for use with tools such as Terraform, Docker, Packer, and Kubernetes. Unit tests, integration tests, end-to-end tests, dependency injection, test parallelism, retries and error handling, static analysis, property testing and CI/CD for infrastructure code.

Yevgeniy Brikman talks about how to write automated tests for infrastructure code, including the code written for use with tools such as Terraform, Docker, Packer, and Kubernetes. Topics covered include: unit tests, integration tests, end-to-end tests, dependency injection, test parallelism, retries and error handling, static analysis, property testing and CI / CD for infrastructure code.

Benchmarking Go and Python Web servers

Benchmarking Go and Python Web servers

Benchmarking Go and Python Web servers. To build ... I found that the Go with http, mustache.go and GoMySQL packages would be an ideal tools to do my job.

You need to have Go installed and Python installed for this to work.

Introduction

Just noticed after finishing up all the code for this post that both of these are drinking related (Gin and Flask). After analyzing the go ecosystem for web frameworks (Gin, Martini, Web.go, Beego, Goji, Gorilla) I decided on pitting Gin against Flask. This appeared to be the most straight forward comparison between what I'm comfortable with as one of the more popular python web frameworks, but also it's known as one of the fastest ones of the bunch.

Ideally I would do this in a few files, and probably use gunicorn in production for Python and structure the app differently, but I just wanna build the most stripped down version of this example in ython to compare pure speeds of Flask vs. Gin and the size of their minimal docker builds for just returning a simple JSON object on a get request to the app.

Make a directory called pygo-benchmark and change your current directory to that dir.

Go web-app with Gin

Do this all in a subdirectory within pygo-benchmark, I called it go-stuff, but you can call it whatever you'd like.

Write the file

Lets get to building. I use vim-go so you get a whole package scaffold upon the first vim file.go command. So when I type vim main.go I get:

package main

import "fmt"

func main() {
fmt.Println("vim-go")
}

So you should also create a file called main.go and add the code above to that file. Now we are going to want to install gin. In order to do this we are going to type go get -u github.com/gin-gonic/gin and that will install the go module for us. Alright, now that we have gin installed lets modify the file to make our main.go use gin now and return a simple json object on a GET to localhost:8080, the default port for this webserver.

package main

import "github.com/gin-gonic/gin"

func main() {
router := gin.Default()

router.GET("/", func(c *gin.Context) {
    c.JSON(200, gin.H{
        "message": "pong",
    })
})

// router runs on :8080 by default
router.Run()

}

Build and Run

It is pretty simple to build a file with go, go build main.go is all you need. Now you should have an executable file called main in the same directory you're in. If you just run /main in the command line at the same directory you built the file from, it should have some standard output like:

[GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.

[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.

  • using env: export GIN_MODE=release
  • using code: gin.SetMode(gin.ReleaseMode)

[GIN-debug] GET / --> main.main.func1 (3 handlers)
[GIN-debug] Listening and serving HTTP on :8080

If you've gotten this far congrats you built and ran your first go app!

Python web-app with Flask

Do all of this in a subdirectory in pygo-benchmark, I called it flask-stuff, but once again do whatever you need to here.

Write the file

In order to make this post shorter I'm not going to go through the whole setup, I'm just gonna blurgghhhhh the file out here. Make a file called main.py with the contents below:

from flask import Flask

app = Flask(name)

@app.route("/")
def ping():
return '{"message": "ping"}'

if name == "main":
app.run(host="0.0.0.0")

Build and Run python

So there is no build in python, we will get to that benefit as one the favors Go in our Benchmarking section.

I like to do everything on the commandline in virtualenv, but if you dont mind messing up your packages do whatever you want, this is what I would do in order to get this file running:

virtualenv .venv -p python3.6
source .venv/bin/activate
pip install flask
python main.py

Because of the way this file is setup with the if name... stuff we can just run this file from the command line, like this python main.py and we should see something like this if we did it successfully:

(.venv) ➜  flask-stuff ✗ python main.py

  • Serving Flask app "main" (lazy loading)
  • Environment: production
    WARNING: Do not use the development server in a production environment.
    Use a production WSGI server instead.
  • Debug mode: on
  • Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
  • Restarting with stat
  • Debugger is active!
  • Debugger PIN: 313-728-815
Benchmarking the pure speed of each setup.

In python using requests and timeit

Now let's get the standard boilerplate dislaimer out here. You know, the one about how results may vary on the machine that you run the benchmarks on and that you really need to have a multitude of machines in order to really prove out benchmars, yada-yada.

Alright, so with that boring stuff out of the way I LOVEEE using timeit to time stuff in python. You can do multiple runs one after the other in order to find a more valuable benchmark (the average response time).

Since we should have both webservers running as of now in this blog the go server is running on localhost:8080 and the python on at localhost:5000. Lets time them both, shall we?:

pip install requests
python
>>>
>>>import timeit
>>>import requests
>>>
>>> # gin / go
>>> timeit.timeit("requests.get('http://localhost:8080')", setup="import requests", number=10000)
21.956489593023434
>>> # flask / python
>>> timeit.timeit("requests.get('http://localhost:5000')", setup="import requests", number=10000)
34.483878459897824

Cool. So as you can see the the Go option is ~36% faster! That's amazing, no wonder people are going for go > python these days. But then again this isn't the most fair test as we can put a multithreadable WSGI server in front of our python setup. Now onto round two...

Using a new fangled benchmarking tool called wrk

and multithreading all the things! Thanks to a great suggestion by a dedicated reader who tweeted in, I am also adding in a multhreaded benchmark and adding gunicorn to sit in front of our Flask app.

In order to get up to speed with me you'll have to download and make the binary for wrk using the instructions found here and if you look there are more options on the side if you're a Windows/Mac user. Also we are going to wanna pip install gunicorn as well and in the flask-stuff directory we are going to want to type gunicorn -w 8 main:app which tells gunicorn (our multithreaded WSGI server) to use 8 threads (since the output of nproc --all on my machine was 8) and it says look at the main.py file and the appfunction as the entrypoint for the app.

Now down to benchmarking the two using "a real benchmarking tool" - Some guy named Ben.

➜  go-hello wrk -t8 -c32 -d30s http://127.0.0.1:5000 # flask-python
Running 30s test @ http://127.0.0.1:5000
8 threads and 32 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 3.58ms 1.70ms 42.16ms 94.75%
Req/Sec 1.12k 158.54 2.83k 84.48%
268403 requests in 30.10s, 45.82MB read
Requests/sec: 8917.30
Transfer/sec: 1.52MB
➜ go-hello wrk -t8 -c32 -d30s http://127.0.0.1:8080 # go-gin
Running 30s test @ http://127.0.0.1:8080
8 threads and 32 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 2.64ms 4.71ms 60.39ms 85.07%
Req/Sec 6.86k 0.93k 33.27k 82.09%
1639477 requests in 30.10s, 220.46MB read
Requests/sec: 54471.39
Transfer/sec: 7.32MB

Okay, maybe ben was right, this really shows how much better go is than python. Surprisingly though, the one thing python did have was a more deterministic response time as the stdev of the latency was < 2ms where it was almost 5ms in go.

But once again, this is another benchmark proving out go is faster than python. Req/sec wise go is 6.125 times faster than flask, on average.

Benchmarking for Docker image size.

Alrighty then. I'm just gonna go ahead and say it that this post is TOO freaking long already and you're just not going to get me to breakdown the multistage docker builds I iterated through in order to get the smallest image size for each one.

Python Dockerfile

Lets put this file in the flask-stuff/Dockerfile directory with the contents below:

Our structure should look like this:

  • flask-stuff
  • main.py
  • Dockerfile
FROM python:3.6-alpine
COPY . /
WORKDIR /
RUN pip install flask
RUN pip install gunicorn
CMD ["gunicorn" , "-w", "8", "main:app"]

In order to build this you'd type docker build . -t benchmark/py.

Go Dockerfile

Lets put this file in the go-stuff/Dockerfile directory with the contents below:

Our structure should look like this:

  • go-stuff
  • main.go
  • Dockerfile
FROM golang:latest
ADD . .
RUN go get -u github.com/gin-gonic/gin
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o main .
RUN ls
RUN pwd

FROM alpine:latest
RUN apk --no-cache add ca-certificates
COPY --from=0 /go/main .
CMD ["./main"]

In order to build this you'd type docker build . -t benchmark/go.

Surely what are the results?!?

I have the results, and my name's not Shirley! Bad joke, I know, but I'm trying to keep you awake here.

Alright so here's the data:

➜ pygo-benchmark docker images
REPOSITORY      TAG         IMAGE ID       CREATED        SIZE
benchmark/go     latest       02ef15405471    About an hour ago   21.7MB
benchmark/py    latest       8fd24d5e54f1    About an hour ago   89.6MB
python        3.6-alpine     83d065b0546b     19 hours ago    79MB
golang        latest       901414995ecd     2 weeks ago     816MB
alpine        latest       caf27325b298     3 weeks ago      5.53MB

Now this is astonishing! The benchmark/go image is ~76% smaller than the benchmark/py image. This is because of a few reasons, go literally just needs that one binary main to run, where as python needs all of the supporting libraries because it isn't compiled.

If we used the golang image as a base instead of the alpine image as a base for our benchmark/go image it would be ~835MB. The reason why Go docker images are so small is because you can run them directly on an alpine image, which is ~5MB.

Conclusion

Feel free to click one of these buttons in order to signal me with something that was messed up with this. I'd be glad to fix anything that didnt work for you.

Thanks for reading. If you liked this post, share it with all of your programming buddies!

System testing with Pytest, Docker, and Flask

System testing with Pytest, Docker, and Flask

System testing with Pytest, Docker, and Flask. The composability of fixtures in pytest is an improvement over traditional xUnit setup/teardown. Learn how to combine the power of pytest fixtures with Docker to build high-level integration tests for microservices or other complex systems with multiple components

The composability of fixtures in pytest is an improvement over traditional xUnit setup/teardown, reducing the incentive to commit testing crimes such as multi-stage and stepwise tests. This is great out of the box for unit tests, but I'm going to show how to combine the power of pytest fixtures with Docker to build high-level integration tests for microservices or other complex systems with multiple components. I'll then build on that to show how to embed mock web services written with Flask right into the test code.

With a sample Java application that makes use of some external resources to offer a data processing service I'll first quick an overview of Pytest, Docker, and Flask. Then I'll mix some pre-built code with live test coding to demonstrate how to build high-level system tests which spin up the application and its dependencies in Docker. I'll then mock one of the external dependencies using Flask, allowing the test to control and verify interaction between the system components. Finally I'll show how to wrap the Flask application in a WSGI middleware that lets the test inspect interaction with the mocked service.

From a learning and development point of view, building your own is better than re-using someone else's code so I'll show how the support code for these features is relatively simple and how the audience can build it themselves to exactly meet their own needs. And I'll do it all with a sense of fun, a joke or two and maybe a little storytelling.

What is the difference between Docker, Kubernetes and Docker Swarm ?

What is the difference between Docker, Kubernetes and Docker Swarm ?

What is the difference between Docker and Kubernetes? And Kubernetes or Docker Swarm? In my video "Docker vs Kubernetes vs Docker Swarm" I compare both Docker and Kubernetes and Kubernetes vs Docker Swarm.

What is the difference between Docker and Kubernetes? And Kubernetes or Docker Swarm?
In my video "Docker vs Kubernetes vs Docker Swarm" I compare both Docker and Kubernetes and Kubernetes vs Docker Swarm.

Kubernetes and Docker are not competing technologies. In fact, they actually complement one another to get the best out of both. In contrast, Docker Swarm is the comparable technology to Kubernetes.

  • 0:38 - Comparison Docker and Kubernetes
  • 1:40 - Docker and Kubernetes in the software development process
  • 2:42 - Kubernetes in Detail
  • 3:21 - Differences of Kubernetes and Docker Swarm