It’s been a while since the previous article in which we shared several captivating stories about our real-life experience in operating Kubernetes clusters as well as applications/services running in them. Here is another episode in adventures of Flant engineers sailing in turbulent waters of a large-scale (and not so large) web applications.

Case #1. Kafka and Docker variables in Kubernetes

The container-based systems gradually become an industry standard, and many tools have already taken root in this environment. However, you can still find some ridiculous bugs related to integrating software into Docker/Kubernetes.

One day, we were going to deploy the standard bundle of ZooKeeper + Kafka for Sentry, with a possible addition of other components in some future. At first sight, the process looks easy: take one hard-to-read Kafka’s Helm chart as a basis (you can find it in this well-known GitHub repo), chip off all the unnecessary stuff, add some tweaks for your needs, et voila!

However, we got the following error while trying to run it:

===> Configuring ...
port is deprecated. Please use KAFKA_ADVERTISED_LISTENERS instead.

That’s strange. Okay, let’s add this:

env:
  - name: KAFKA_ADVERTISED_LISTENERS
    value: {{ printf "%s:%s" .Chart.Name  .Values.port._default }}

Still the same problem… Perhaps, it makes sense to find out what script produces the above message and why? The next question then is: “What image do we use, and what’s inside?”

This chart is based on a quite popular confluentinc/cp-kafka image that boasts a multipurpose entrypoint (as is the case with many other Confluent Community images). If you analyze what scripts are run and in what order, you will find the one that hides the source of our problems:

if [[ -n "${KAFKA_PORT-}" ]]
then
  echo "port is deprecated. Please use KAFKA_ADVERTISED_LISTENERS instead."
  exit 1
fi

It’s a no-brainer, right? But wait, we did not declare that variable (or anything like it)!

However, if you look at the actual environment variables declared inside the container, you will find such a variable… as well as some similar others:

root@kafka-0:/# env |grep KAFKA_PORT
KAFKA_PORT_9092_TCP_PORT=9092
KAFKA_PORT_9092_TCP_PROTO=tcp
KAFKA_PORT=tcp://192.168.252.192:9092
KAFKA_PORT_9092_TCP=tcp://192.168.252.192:9092
KAFKA_PORT_9092_TCP_ADDR=192.168.252.192

The thing is that kubelet adds a set of variables when starting a pod. Their names are composed using service names/hosts/ports. This way K8s can find an address of the service it needs — it is briefly described in the Kubernetes documentation.

The obvious way to solve the problem is simply to rename the service in Kubernetes to something more relevant (and less similar to the conventional kafka).

Let’s run it:

$ kubectl -n sentry logs sentry-kafka-0
===> ENV Variables ...
...
SENTRY_KAFKA_PORT=tcp://192.168.224.150:9092
...
===> Configuring ...
===> Running preflight checks ...
===> Check if /var/lib/kafka/data is writable ...

Nice one!

#clickhouse #flant #postgresql #kubernetes #docker