Running Apache Flink with RocksDB on Azure Kubernetes Service

Recently I was looking into how to deploy an Apache Flink cluster that uses RocksDB as the backend state and found a lack of detailed documentation on the subject. I was able to piece together how to deploy this from the Flink documentation and some stack overflow posts but there wasn’t a clear how-to guide anywhere. My goal with this post is to show a step by step guide for how to do this as of Flink 1.11.0. This guide can be followed with minimal modifications for AWS deployments. I will note the changes.

#azure-kubernetes-service #azure #docker #apache-flink #data-engineering #rocksdb

What is GEEK

Buddha Community

Running Apache Flink with RocksDB on Azure Kubernetes Service
Christa  Stehr

Christa Stehr

1602964260

50+ Useful Kubernetes Tools for 2020 - Part 2

Introduction

Last year, we provided a list of Kubernetes tools that proved so popular we have decided to curate another list of some useful additions for working with the platform—among which are many tools that we personally use here at Caylent. Check out the original tools list here in case you missed it.

According to a recent survey done by Stackrox, the dominance Kubernetes enjoys in the market continues to be reinforced, with 86% of respondents using it for container orchestration.

(State of Kubernetes and Container Security, 2020)

And as you can see below, more and more companies are jumping into containerization for their apps. If you’re among them, here are some tools to aid you going forward as Kubernetes continues its rapid growth.

(State of Kubernetes and Container Security, 2020)

#blog #tools #amazon elastic kubernetes service #application security #aws kms #botkube #caylent #cli #container monitoring #container orchestration tools #container security #containers #continuous delivery #continuous deployment #continuous integration #contour #developers #development #developments #draft #eksctl #firewall #gcp #github #harbor #helm #helm charts #helm-2to3 #helm-aws-secret-plugin #helm-docs #helm-operator-get-started #helm-secrets #iam #json #k-rail #k3s #k3sup #k8s #keel.sh #keycloak #kiali #kiam #klum #knative #krew #ksniff #kube #kube-prod-runtime #kube-ps1 #kube-scan #kube-state-metrics #kube2iam #kubeapps #kubebuilder #kubeconfig #kubectl #kubectl-aws-secrets #kubefwd #kubernetes #kubernetes command line tool #kubernetes configuration #kubernetes deployment #kubernetes in development #kubernetes in production #kubernetes ingress #kubernetes interfaces #kubernetes monitoring #kubernetes networking #kubernetes observability #kubernetes plugins #kubernetes secrets #kubernetes security #kubernetes security best practices #kubernetes security vendors #kubernetes service discovery #kubernetic #kubesec #kubeterminal #kubeval #kudo #kuma #microsoft azure key vault #mozilla sops #octant #octarine #open source #palo alto kubernetes security #permission-manager #pgp #rafay #rakess #rancher #rook #secrets operations #serverless function #service mesh #shell-operator #snyk #snyk container #sonobuoy #strongdm #tcpdump #tenkai #testing #tigera #tilt #vert.x #wireshark #yaml

Running Apache Flink with RocksDB on Azure Kubernetes Service

Recently I was looking into how to deploy an Apache Flink cluster that uses RocksDB as the backend state and found a lack of detailed documentation on the subject. I was able to piece together how to deploy this from the Flink documentation and some stack overflow posts but there wasn’t a clear how-to guide anywhere. My goal with this post is to show a step by step guide for how to do this as of Flink 1.11.0. This guide can be followed with minimal modifications for AWS deployments. I will note the changes.

#azure-kubernetes-service #azure #docker #apache-flink #data-engineering #rocksdb

Myrl  Prosacco

Myrl Prosacco

1594533600

Using Apache Flink for Kinesis to Kafka Connect

In this blog, we are going to use kinesis as a source and kafka as a consumer.

Let’s get started.

Step 1:

Apache Flink provides the kinesis and kafka connector dependencies. Let’s add them in our build.sbt:

name := "flink-demo"

version := "0.1"

scalaVersion := "2.12.8"

libraryDependencies ++= Seq(
  "org.apache.flink" %% "flink-scala" % "1.10.0",
  "org.apache.flink" %% "flink-connector-kinesis" % "1.10.0",
  "org.apache.flink" %% "flink-connector-kafka" % "1.10.0",
  "org.apache.flink" %% "flink-streaming-scala" % "1.10.0"
)

Step 2:

The next step is to create a pointer to the environment on which this program runs.

val env = StreamExecutionEnvironment.getExecutionEnvironment

Step 3:

Setting parallelism of x here will cause all operators (such as join, map, reduce) to run with x parallel instance.

I am using 1 as it is a demo application.

env.setParallelism(1)

Step 4:

Disabling the aws cbor, as we are testing locally.

System.setProperty("com.amazonaws.sdk.disableCbor", "true")
System.setProperty("org.apache.flink.kinesis.shaded.com.amazonaws.sdk.disableCbor", "true")

Step 5:

Defining Kinesis consumer properties.

  • Region
  • Stream Position – TRIM_HORIZON to read all the records available in the stream
  • Aws keys
  • Do not worry about the endpoint, it is set to http://localhost:4568 as we will test the kinesis using localstack.

Do not worry about the endpoint, it is set to http://localhost:4568 as we will test the kinesis using localstack.

#apache flink #flink #scala ##apache-flink ##kinesis #apache #flink streaming #kafka #scala

Private Azure Kubernetes Service Clusters with Azure Private Links?

What if I tell you that you can make your AKS cluster private. No, not just setting the ingress controller LoadBalancer IP to a private IP and prevent internet ingress to the pods and applications, but prevent external access to the KubeAPI Sever completely. In other words, the kubectl commands cannot run over the internet and this creates an additional layer of security to your enterprise clusters!

#terraform #azure #kubernetes-security #kubernetes #azure-kubernetes-service

Layne  Fadel

Layne Fadel

1624515600

Traefik Ingress on Azure Kubernetes Service

Having an application deployed on a Kubernetes cluster consisting of multiple microservices, you may want to expose some of them to be accessible through the internet. While it’s obviously for your web app service, maybe you have some additional APIs that you want to expose.

In the world of Kubernetes, any connection to one of your microservices is done using the Service resource. Using the type LoadBalancer of the Kubernetes Service resource leverages the underlying cloud provider to create a cloud provider-specific load balancer for exposing the microservice through an external IP. The problem with that approach is that each microservice would be exposed under a separate IP address.

It would be much more convenient to have them exposed under one and the same host while having different paths to reach the dedicated microservice, right?

This article shows how to do that with a Kubernetes Cluster on Azure and Traefik and is a follow-up to my article about achieving the same using the Azure Application Gateway. A lot of content will be based on that article.

Introduction

Microservices can be exposed inside and outside of Kubernetes using the Kubernetes Service resource. So far, so good. But as already said, if we want to expose them outside the cluster, using the Service resource with the type LoadBalancer, we end up having different IPs for each microservice. This does not want we want, instead, we want to have them exposed under one and the host using different paths.

This is where the Kubernetes Ingress resource comes in handy. Think of an Ingress like a layer on top of Kubernetes Services. It is the single point of entrance for traffic hitting our microservices, which routes traffic to different Kubernetes Services based on specified rules.

The concept of Kubernetes Ingress resource is like an Abstraction. In order to make use of a Kubernetes Ingress, you have to install a specific Ingress Controller. There are plenty of different Implementations of the Kubernetes Ingress Abstraction out there. Nginx and Traefik Ingress are two of them which are very popular in the Kubernetes and Open Source Community, just to name some.

And then of course we have Cloud Providers, where you can use resources like Load Balancers and Gateways as a Kubernetes Ingress. Anyways, in this article, we will focus on the Traefik_ Ingress_.

#microservices #azure-kubernetes-service #ingress #kubernetes #azure kubernetes service