In this Kubernetes tutorial you learn about Elasticsearch and Kubernetes clusters, and how to run and deploy Elasticsearch on Kubernetes. Learn why should you run Elasticsearch on Kubernetes, how Kubernetes works. What is Elasticsearch? Why Run Elasticsearch on Kubernetes. You'll get some tips about memory requirements and storage.
Big data, AI, and machine learning are all buzzwords we seem to throw around lightly in recent years. Even though they are hugely different from one another, they all have one thing in common: data! Huge amounts of data that needs to be managed.
The downside of that is that the more data you have the more of a headache it is to store, query, and make sense of.
However, running Elasticsearch on Kubernetes can save you a lot of trouble. Elasticsearch handles storing and querying data, while Kubernetes handles the underlying infrastructure.
By the end of this tutorial, you'll have a running Elasticsearch cluster on Kubernetes, you'll learn best practices to leverage the platforms’ powers, and you'll get some tips about memory requirements and storage.What is Elasticsearch?
Elasticsearch is a datastore that stores data in indices. It’s also a real-time, distributed, and scalable search engine which allows for full-text and structured search, as well as for analytics. It’s great for storing and searching through large volumes of textual data, like logs, but can also be used to search many different kinds of documents.What is Kubernetes?
Kubernetes is the de-facto standard container orchestrator and by far the easiest way to run and manage clusters in the cloud or on-premises. But what is a container orchestrator? To understand Kubernetes, you first need to understand Docker.
Docker is a container engine that lets you create ephemeral containers to run your applications. These containers are stateless and run isolated from the rest of your system.
Running Docker containers is the same across any operating system, as long as the hosts are in a Kubernetes cluster. You don’t have to worry about the underlying infrastructure at all. This makes packaging and shipping apps to production simple.
However, containers are useless without a cluster and orchestrator to run and manage them. Kubernetes manages all of this and does the heavy lifting so you don’t have to. What you have to do is tell Kubernetes what to do through the
kubectl command-line and with
YAML resource files.
Elasticsearch can store huge amounts of textual data with the ability to quickly search through it when needed. It’s deployed in clusters, at least consisting of three nodes. These nodes have, throughout the years, often been VMs that you would spin up and then handle connections between them. But that's tiresome and hard to manage.
Kubernetes has stepped in to solve that issue. It has become the de-facto standard for running high-uptime and reliable systems in the cloud and on-premises. Even though Kubernetes is designed to run ephemeral, stateless, apps and not databases, there are upsides of running an Elasticsearch cluster on Kubernetes. You should generally not be running databases on Kubernetes, but you can. Handling persistent data is simple by using persistent volume claims and stateful sets.
With Kubernetes, you get a cluster that’s easier to configure, manage and scale. Once you configure your Elasticsearch cluster on Kubernetes, the process of deploying it to another cloud provider or on-premises is incredibly simple.
Kubernetes is also very developer-friendly. You rely on infrastructure as code configurations and not manually setting up and configuring infrastructure. For many, this may be the only way they know how to deploy a large cluster. Seeing as many teams don’t have dedicated DevOps engineers and they have to rely on their developers to handle the infrastructure, you may be saving yourself a huge headache by letting Kubernetes manage the cluster.
Let’s check out the architecture behind running Kubernetes and Elasticsearch.Kubernetes Architecture: Basic Concepts
Kubernetes manages your application with several different resource types. First, your application is built and packaged into a Container. This containerized application is deployed to Kubernetes and runs within a Pod.
Kubernetes Pods are grouped in a Deployment. A Deployment is a key concept in Kubernetes that manages Pods and their properties, like how many replicas of each Pod to run.
A Service is then used to expose the Deployment to the Internet. If it is of type LoadBalancer it’ll also load balance requests evenly across all the Pods in the Deployment. Simply put, a Service creates a single IP address that is used to access the Containers. Services can also make Pods accessible to other Pods within the Kubernetes cluster.
Kubernetes Nodes are the virtual machines on which the Kubernetes cluster is running, including all Pods. Pods are always ordered randomly across the Nodes. You can use Affinity and Anti-Affinity rules to tell Kubernetes how to spread the running Pods across the Nodes. Maybe you want Elasticsearch Pods to only run on certain Kubernetes Nodes.
Deployments do not keep state in their Pods. It’s assumed the application is stateless. If you need your application to maintain state, like in our case with Elasticsearch, then you need to use a StatefulSet.
A StatefulSet is a Deployment that can maintain state. Makes sense from the name right?
When using StatefulSets you also need to use PersistentVolumes and PersistentVolumeClaims. A StatefulSet will ensure the same PersistentVolumeClaim stays bound to the same Pod throughout its lifetime. Unlike a Deployment which ensures the group of Pods within the Deployment stay bound to a PersistentVolumeClaim.
A PersistentVolume (PV) is a Kubernetes abstraction for storage on the provided hardware. This can be AWS EBS, DigitalOcean Volumes, etc.
A PersistentVolumeClaim (PVC) however, is a way for a Deployment or StatefulSet to request some storage space from a PersistentVolume. This allocated storage is persisted even if Pods and Nodes restart.
Alongside StatefulSets you have Headless Services that are used for the discovery of StatefulSet Pods.
A Headless Service is a service when you don’t need load-balancing and a single Service IP. Instead of load-balancing, it will return the IPs of the associated Pods. Headless Services do not have a Cluster IP allocated. They will not be proxied by kube-proxy. Instead, Elasticsearch will handle the service discovery.Elasticsearch Deployment: Cluster Topology
Elasticsearch should always be deployed in clusters. Every instance of Elasticsearch running in the cluster is called a node. In Kubernetes, an Elasticsearch node would be equivalent to an Elasticsearch Pod. Don’t get it confused with a Kubernetes Node, which is one of the virtual machines Kubernetes is running on. For the rest of this Elasticsearch Kubernetes tutorial, I’ll use the term Elasticsearch Pod to minimize confusion between the two.
By default, when you deploy an Elasticsearch cluster, all Elasticsearch Pods have all roles. The roles can be master, data, and client. The client is often also called the coordinator. Master Pods are responsible for managing the cluster, managing indices, and electing a new master if needed. Data Pods are dedicated to storing data, while client Pods have no role whatsoever except for funneling incoming traffic to the rest of the Pods.
You need a minimum of three master-eligible Pods to avoid split-brain when a new master needs to be appointed. You set this role for a node by having this combination of roles.
roles: master: "true" ingest: "false" data: "false"
Regarding data Pods, you need at least two. They will persist data, receive queries, and index requests. Basically, they do all the heavy lifting. You set this role like this.
roles: master: "false" ingest: "false" data: "true"
Client Pods are also known as Coordinating Pods. You should have two of these as well. These Pods are exposed to consumers of the cluster data and serve as HTTP proxies. If they are not deployed, Data Pods will serve as coordinating Pods. Avoid this on larger clusters. You set a Pod to be a client by having all roles false.
roles: master: "false" ingest: "false" data: "false"
This setup is considered best practice and scaling up would be needed only when the current node count is insufficient. Luckily, scaling up an Elasticsearch cluster on Kubernetes is as simple as running one command.
This is what the final cluster topology will look like.
Data Pods are deployed as StatefulSets with PersistentVolumes and PersistentVolumeClaims. They will persist data between restarts, which is what you want.
Master Pods can be deployed as either Deployments or StatefulSets.
A headless service for each StatefulSet is created and used for inter-cluster discovery.
Client Pods are completely stateless and can be deployed as a simple Kubernetes Deployment.
A Kubernetes LoadBalancer Service is used to forward inbound traffic to the client Pods. All of your apps, as well as Kibana, will be configured to go through the LoadBalancer service.
If you are setting up an Elasticsearch cluster on Kubernetes for yourself, keep in mind to allocate at least 4GB of memory to your Kubernetes Nodes. You will need at least 7 Nodes to run this setup without any hiccups. The default size of the PersistentVolumeClaims for each Elasticsearch Pod will be 30GB. This will help determine how much block storage you will need.
The pods are inside of a StatefulSet hence when creating new Pods you need to make sure you have 30GB of storage per additional Pod you want to create. Working with PVCs is complicated because you need to delete them yourself. It gets even more complicated when you are not using a cloud service and you have to configure your own StorageClasses. Often Pods won’t start, and it’s most likely due to lack of storage space or old PVCs still persisting even though you don’t need them.
In the next section, I’ll show you how to configure both a 7-Pod production setup with Helm, but also how to get up and running quickly with a 3-Pod master setup where each of the Pods has all roles.How to Deploy Elasticsearch on Kubernetes
Deploying Elasticsearch on Kubernetes can be a hassle if you choose to do it yourself with custom resource files and
kubectl. It’s much easier to use Helm, the Kubernetes package manager. With the help of Helm, you can install a prebuilt chart that’ll configure all required resources by running one simple command. Let’s get our hands dirty and start creating the Elasticsearch cluster on Kubernetes.
To follow along with this tutorial you’ll need a few things first:
kubectlcommand-line tool installed on your local machine configured to connect to your cluster. You can read more about how to install
kubectlin the official documentation.
First and foremost you need to initialize Helm on your Kubernetes cluster. It’s done with the init command.
Note: Helm often needs Tiller installed. If the
helm init command does not work, run these commands to install Tiller if you do not have it installed and configured.
kubectl create serviceaccount -n kube-system tiller kubectl create clusterrolebinding tiller-cluster-admin \ --clusterrole=cluster-admin \ --serviceaccount=kube-system:tiller helm init --service-account tiller \ --override spec.selector.matchLabels.'name'='tiller',spec.selector.matchLabels.'app'='helm' \ --output yaml | sed '[email protected]: extensions/[email protected]: apps/[email protected]' | kubectl apply -f -
Once you have Helm initialized you can begin adding charts. First, start by adding the elastic repo and install the Elasticsearch chart.
helm repo add elastic https://helm.elastic.co helm install --name elasticsearch elastic/elasticsearch \ --set service.type=LoadBalancer
You’re adding the –set service.type=LoadBalancer parameter to indicate you want the service to expose a LoadBalancer IP to the Internet. Check to see that the resources are running.
kubectl get all
This will list all the resources the chart created.
[output] NAME READY STATUS RESTARTS AGE pod/elasticsearch-master-0 1/1 Running 0 2m8s pod/elasticsearch-master-1 1/1 Running 0 2m8s pod/elasticsearch-master-2 1/1 Running 0 2m8s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/elasticsearch-master LoadBalancer 10.98.90.94 <YOUR_IP> 9200:31812/TCP,9300:31635/TCP 2m8s service/elasticsearch-master-headless ClusterIP None <none> 9200/TCP,9300/TCP 2m9s service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 5d5h NAME READY AGE statefulset.apps/elasticsearch-master 3/3 2m8s
You now have three Elasticsearch master Pods running on your Kubernetes cluster. These Pods now have all three available roles. To keep them healthy, make sure you have enough resources allocated. If you need to scale up, you can configure a Pod autoscaler. To check if everything is running as it should, hit the Elasticsearch state endpoint with curl.
This setup will work great for smaller clusters where you don’t have huge amounts of data. Some issues you may run into are out of memory exceptions when your indices start growing. In that case, you should increase the
max_map_count. Here’s a nice thread explaining it.
But, if you want to follow Elasticsearch best practices you should also configure dedicated data and client Pods apart from master Pods. That’s exactly what we’re doing in the next section.
Let’s get serious for a moment, and configure the cluster with best practices in mind. The 7 Pods will consist of 3 master Pods, 2 data Pods, and 2 client Pods.
This preferred setup is installed in a similar way. First, run the Helm install command, but this time without any additional parameters.
helm install --name elasticsearch elastic/elasticsearch
Now you need to run the upgrade command to update the Elasticsearch pods. You want to upgrade the number of Pods but also assign custom roles to them.
To do this create three
YAML config files. First, the
master.yaml to configure the master-eligible Pods.
# master.yaml --- clusterName: "elasticsearch" nodeGroup: "master" roles: master: "true" ingest: "false" data: "false" replicas: 3
data.yaml for the data Pods.
# data.yaml -- clusterName: "elasticsearch" nodeGroup: "data" roles: master: "false" ingest: "true" data: "true" replicas: 2
client.yaml for the client Pods.
# client.yaml --- clusterName: "elasticsearch" nodeGroup: "client" roles: master: "false" ingest: "false" data: "false" replicas: 2 service: type: "LoadBalancer"
Now you can run the upgrade command three times, with each distinct
YAML config file in the directory where you created the files.
helm upgrade --wait --timeout=600 --install \ --values ./master.yaml elasticsearch elastic/elasticsearch helm upgrade --wait --timeout=600 --install \ --values ./data.yaml elasticsearch elastic/elasticsearch helm upgrade --wait --timeout=600 --install \ --values ./client.yaml elasticsearch elastic/elasticsearch
It’ll take a while to upgrade the Helm chart. But, when they are all finished upgrading you can check if your resources are updated.
kubectl get all
Here’s the output you’re looking for.
[Output] NAME READY STATUS RESTARTS AGE pod/elasticsearch-client-0 1/1 Running 0 10m pod/elasticsearch-client-1 1/1 Running 0 10m pod/elasticsearch-data-0 1/1 Running 0 11m pod/elasticsearch-data-1 1/1 Running 0 11m pod/elasticsearch-master-0 1/1 Running 0 8m27s pod/elasticsearch-master-1 1/1 Running 0 8m27s pod/elasticsearch-master-2 1/1 Running 0 8m27s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/elasticsearch-client LoadBalancer 10.245.114.89 <YOUR_IP> 9200:32366/TCP,9300:31427/TCP 10m service/elasticsearch-client-headless ClusterIP None <none> 9200/TCP,9300/TCP 10m service/elasticsearch-data ClusterIP 10.245.116.115 <none> 9200/TCP,9300/TCP 11m service/elasticsearch-data-headless ClusterIP None <none> 9200/TCP,9300/TCP 11m service/elasticsearch-master ClusterIP 10.245.220.94 <none> 9200/TCP,9300/TCP 8m27s service/elasticsearch-master-headless ClusterIP None <none> 9200/TCP,9300/TCP 8m27s service/kubernetes ClusterIP 10.245.0.1 <none> 443/TCP 4h5m NAME READY AGE statefulset.apps/elasticsearch-client 2/2 10m statefulset.apps/elasticsearch-data 2/2 11m statefulset.apps/elasticsearch-master 3/3 8m28s
Run curl against the Elasticsearch endpoint once again to check if it works.
There ya go! Ready to rock!
Note: If you’re having issues with configuring larger clusters, you might need to check out setting up readiness probes. They can check whether your Elasticsearch Pods are ready to accept traffic.
The peeps over at Bitnami have created a great Chart with preconfigured settings for Elasticsearch master, data, and client Pods. All you need to do is run two commands.
helm repo add bitnami https://charts.bitnami.com/bitnami helm install --name elasticsearch --set \ name=elasticsearch,master.replicas=3,coordinating.service.type=LoadBalancer bitnami/elasticsearch
kubectl get all output once again to make sure everything is in order.
[Output] NAME READY STATUS RESTARTS AGE pod/elasticsearch-elasticsearch-coordinating-only-694b5f94f8-896k5 1/1 Running 0 3m55s pod/elasticsearch-elasticsearch-coordinating-only-694b5f94f8-jvdrn 1/1 Running 0 3m55s pod/elasticsearch-elasticsearch-data-0 1/1 Running 0 3m55s pod/elasticsearch-elasticsearch-data-1 1/1 Running 0 3m27s pod/elasticsearch-elasticsearch-master-0 1/1 Running 0 3m55s pod/elasticsearch-elasticsearch-master-1 1/1 Running 0 3m35s pod/elasticsearch-elasticsearch-master-2 1/1 Running 0 3m16s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/elasticsearch-elasticsearch-coordinating-only LoadBalancer 10.245.13.251 <YOUR_IP> 9200:32270/TCP 3m56s service/elasticsearch-elasticsearch-discovery ClusterIP None <none> 9300/TCP 3m56s service/elasticsearch-elasticsearch-master ClusterIP 10.245.0.78 <none> 9300/TCP 3m56s service/kubernetes ClusterIP 10.245.0.1 <none> 443/TCP 30m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/elasticsearch-elasticsearch-coordinating-only 2/2 2 2 3m55s NAME DESIRED CURRENT READY AGE replicaset.apps/elasticsearch-elasticsearch-coordinating-only-694b5f94f8 2 2 2 3m55s NAME READY AGE statefulset.apps/elasticsearch-elasticsearch-data 2/2 3m56s statefulset.apps/elasticsearch-elasticsearch-master 3/3 3m56s
All that’s left now is to deploy Kibana on the Kubernetes cluster to visualize your data.How to Deploy Kibana on Kubernetes
Once you have your Elasticsearch cluster up and running on Kubernetes, you can use Kibana to manage and monitor it.
Kibana is a simple tool to visualize Elasticsearch data. To run Kibana you need to provide the name of the Elasticsearch client Service as an environment variable so the Kibana Pod knows where to connect to.
You’ll use a LoadBalancer Service to access the Kibana deployment. If you wish, you can only expose it internally instead.
To add Kibana you use the official Helm chart. Go ahead and run the Helm install command.
Make sure to replace the placeholder with the Service name of your client. The default would be
elasticsearch-master if you followed the 3-Pod guide,
elasticsearch-client if you followed the 7-Pod guide, or
elasticsearch-elasticsearch-coordinating-only if you installed the Bitnami Helm chart.
helm install --name kibana elastic/kibana --set \ elasticsearchHosts=http://<CLIENT_SERVICE_NAME>:9200 \ service.type=LoadBalancer
Like always, check to make sure Kibana is running after installing the Helm chart.
kubectl get all [Output] NAME READY STATUS RESTARTS AGE ... pod/kibana-kibana-74bf9fc5f5-sxx4g 1/1 Running 0 1m12s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ... service/kibana-kibana LoadBalancer 10.245.195.198 <YOUR_KIBANA_IP> 5601:31362/TCP 20s service/kubernetes ClusterIP 10.245.0.1 <none> 443/TCP 69m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/kibana-kibana 1/1 1 1 1m12s NAME DESIRED CURRENT READY AGE replicaset.apps/kibana-kibana-74bf9fc5f5 1 1 0 1m12s ...
With that, you’re done! Open up
http://<YOUR_KIBANA_IP>:5601 and you can see Kibana running.
In this tutorial you learned about Elasticsearch and Kubernetes clusters, and how to run and deploy Elasticsearch on Kubernetes. Now you know about best practices, hardware requirements, and tips and tricks on how to maintain a stateful Elasticsearch cluster on Kubernetes.
You’ve created three setups with different numbers of Pods with different roles while managing state with persistent volumes. By now you know the architectural overview of both how to create a solid Elasticsearch cluster but also how to organize resources in a Kubernetes cluster.
You’ve also installed Kibana so you can interact with the data stored in Elasticsearch, and interacted with the Elasticsearch REST API using curl. If you want to continue learning about Kubernetes and Elasticsearch, jump over to one of our guides and continue reading.
Hope you guys and girls enjoyed reading this as much as I enjoyed writing it. If you liked it, feel free to hit the share button so more people will see this tutorial. Until next time, be curious and have fun.
The automation and infrastructural capabilities of Kubernetes makes it an ideal technological partner for DevOps. In this article, you'll see 10 ways Kubernetes enables DevOps
The automation and infrastructural capabilities of Kubernetes makes it an ideal technological partner for DevOps. In this article, you'll see 10 ways Kubernetes enables DevOps
DevOps was an idea before its time.
It actually took a while for technology to catch up and fully implement the principles and vision of DevOps, but that is how innovation works.
A new set of tools in our toolboxes (though it's now taking up more and more mindshare) is end-to-end automation, and Kubernetes helps you manage that. Kubernetes is an open source framework for "automating deployment, scaling, and management of containerized applications." Originally introduced by Google, Red Hat, and others pushed it forward the last 3-4 years.
At the 2017 All Day DevOps conference, Siamak Sadeghianfar laid out 10 ways Kubernetes enables DevOps. We thought it was worth revisiting what he had to say as **Kubernetes **only becomes more prominent.
You should automate every step of your delivery pipeline. Kubernetes automates the deployment of containers. Each component becomes a container image.**2. Infrastructure as code. **
With **Kubernetes **your entire infrastructure is code. This means any part in your application (databases, ports, access controls, etc.) can be described in a way **Kubernetes **can use. For example, you store your infrastructure code in a version-control repository. **Kubernetes **takes the code, and based on instruction, deploys, and maintains your infrastructure. This happens automatically and consistently.3. Configuration as code.
**Kubernetes **allows you to "configure as code." Traditionally, an admin had to run configuration scripts manually to make sure they get the right one. **Kubernetes **keeps the file in the source repo. This allows you to describe where the file needs to go in the container and how the application consumes it. Additionally, it can be version controlled.4. Immutable infrastructure.
Since the inception of servers, there's been one problem: every time you put out a fire you change the state of the virtual machine. Eventually, you don't know exactly what the server looks like and you can't recreate it, even though you know it works. (These virtual machines are referred to as snowflakes.)
In Kubernetes, new containers are immutable; they are created when there is a problem with the original state. So, you can be confident that you know exactly how the new environment is configured.5. On-Demand Infrastructure.
Developers can create hybrid services and infrastructure on-demand from the self-service catalog. This gives control to developers to get the resources they need, yet allows operations to control the configuration of the services. Kubernetes follows open service and API standards so you can expose cloud services.6. Environment consistency.
Build once, and deploy into production-like environments everywhere. Kubernetes allows you to build a golden image and use the exact same image for every single place you deploy your container. This gives you a consistent, production-like environment so it is the exact same whether you are on your local Windows or Mac development machine, a test server, etc. This helps you "shift left" so that you can see issues when you are in development instead of in production.7. Continuous Delivery pipeline.
This is a series of automated steps to test code before it goes in production. Jez Humble says the role of Continuous Delivery is, "an automated process to prove to you a change is a bad change and it should not go into production." Continuous Integration (CI) means every change is tested. Continuous Delivery (CD) automates all the way to production.
How can you tell if you have fully implemented Continuous Delivery? Siamak says that the test is if someone walks in and says, "Can you go to production right now?" can you do it without breaking a sweat?8. Zero downtime deployments.
When you have several deployments a day, you can't pull down production to deploy. You must have safe, rolling updates without disrupting the production traffic. Kubernetes helps with blue/green deployments so that you can set up a new environment and switch to the new one without downtime.9. A/B Testing.
How can you evaluate proposed changes, such as changing copy or the color of buttons? A/B testing delivers different versions to different customers so you can get real-world test results. Kubernetes manages routing traffic to different versions. The same version with slight modifications are possible.10. Cross-functional collaboration.
This is shared access to environments with granular control. Kubernetes goes beyond the silos so everyone has the same set of environments, but you can grant access to different roles and allow different roles to do different things. For instance, developers may be able to push to production, while Infosec may have view access, and QA may have access to live containers, but not building or deploying. The entire team can access production, but operations is the only one who can make containers.
The growing popularity of Kubernetes is undeniable. Luckily, with these 10 capabilities, Siamak has laid out a great path for us all to follow as they continue to not only make their way into the DevOps pipeline, but in many ways enable it.
This video on "What is Kubernetes | Kubernetes Tutorial For Beginners" will give you an introduction to one of the most popular Devops tool in the market - Kubernetes, and its importance in today's IT processes. This tutorial is ideal for beginners who want to get started with Kubernetes & DevOps
What Is Kubernetes | Kubernetes Introduction | Kubernetes Tutorial For Beginners
The following topics are covered in this training session:
Building and Managing Kubernetes with Kubernetes: Kubernetes as a declarative and portable system can be used to do many things in different ways.
Kubernetes as a declarative and portable system can be used to do many things in different ways.
At eBay we built a fleet management system based on k8s. Everything(server, subnet, OS, package and state) is declarative and can be modeled as CRDs in k8s, or referred to as a commit id in git from the objects. By running various controllers on top of these CRD objects, we use k8s to manage k8s, and the entire eBay data center. - Our system provisions hosts the same way k8s creates and manages pods. - We build k8s clusters with Salt. each host has a set of states defined in its salt CRD object. controllers pull states from git based on commit ids to apply. - We build both schedulers and deployment transactions to manage the k8s clusters for both config deployments and upgrades. This declarative, highly scalable, auto healing, and cloud native system is what we think can unify eBay’s fleet.
Thanks for reading ❤
If you liked this post, share it with all of your programming buddies!