1624256749
Setup Loki Storage to AWS S3 and DynamoDB in Kubernetes with Kiam
Grafana + Prometheus + Loki is used by us as a monitoring system with dashboards, monitoring, alerting, and logging for our Kubernetes clusters.
At first, Loki stores its data (logs and index data) on the persistent volume (PV) inside the Kubernetes clusters where an EBS with 1TB (in the beta environment) is attached to the cluster assigned to the PV.
Why Changing to S3?
After running Loki for a while in our clusters, it is considered to move the storage of Loki from K8s PV (EBS) to S3. There are some points being considered.
Flexibility in Scale
In these days I worked with Loki, the full disk issue happened several times, which means the Loki data reaches the limit of disk definition of the PV (1TB). Then, the Loki service stops working completely.
Loki provides configuring retention period¹ but unfortunately, it doesn’t provide a retention logic by disk size² yet. Therefore, a week-long retention period does make sense for troubleshooting issues.
When using PV with EBS in the Kubernetes clusters, it is treated as a local mounted disk, so a target disk size has to be provided when claiming the PV. Then, it is hard to tell how big to fit Loki data with the retention period, like a week.
With S3, you don’t need to specify any numbers of size since it is an external storage resource supported by Loki natively. Hence no worries for disk space at all.
#kiam #kubernetes #dynamodb #loki #aws s3
What is GEEK
Buddha Community
1602964260
50+ Useful Kubernetes Tools for 2020 - Part 2
Introduction
Last year, we provided a list of Kubernetes tools that proved so popular we have decided to curate another list of some useful additions for working with the platform—among which are many tools that we personally use here at Caylent. Check out the original tools list here in case you missed it.
According to a recent survey done by Stackrox, the dominance Kubernetes enjoys in the market continues to be reinforced, with 86% of respondents using it for container orchestration.
And as you can see below, more and more companies are jumping into containerization for their apps. If you’re among them, here are some tools to aid you going forward as Kubernetes continues its rapid growth.
#blog #tools #amazon elastic kubernetes service #application security #aws kms #botkube #caylent #cli #container monitoring #container orchestration tools #container security #containers #continuous delivery #continuous deployment #continuous integration #contour #developers #development #developments #draft #eksctl #firewall #gcp #github #harbor #helm #helm charts #helm-2to3 #helm-aws-secret-plugin #helm-docs #helm-operator-get-started #helm-secrets #iam #json #k-rail #k3s #k3sup #k8s #keel.sh #keycloak #kiali #kiam #klum #knative #krew #ksniff #kube #kube-prod-runtime #kube-ps1 #kube-scan #kube-state-metrics #kube2iam #kubeapps #kubebuilder #kubeconfig #kubectl #kubectl-aws-secrets #kubefwd #kubernetes #kubernetes command line tool #kubernetes configuration #kubernetes deployment #kubernetes in development #kubernetes in production #kubernetes ingress #kubernetes interfaces #kubernetes monitoring #kubernetes networking #kubernetes observability #kubernetes plugins #kubernetes secrets #kubernetes security #kubernetes security best practices #kubernetes security vendors #kubernetes service discovery #kubernetic #kubesec #kubeterminal #kubeval #kudo #kuma #microsoft azure key vault #mozilla sops #octant #octarine #open source #palo alto kubernetes security #permission-manager #pgp #rafay #rakess #rancher #rook #secrets operations #serverless function #service mesh #shell-operator #snyk #snyk container #sonobuoy #strongdm #tcpdump #tenkai #testing #tigera #tilt #vert.x #wireshark #yaml
1624256749
Setup Loki Storage to AWS S3 and DynamoDB in Kubernetes with Kiam
Grafana + Prometheus + Loki is used by us as a monitoring system with dashboards, monitoring, alerting, and logging for our Kubernetes clusters.
At first, Loki stores its data (logs and index data) on the persistent volume (PV) inside the Kubernetes clusters where an EBS with 1TB (in the beta environment) is attached to the cluster assigned to the PV.
Why Changing to S3?
After running Loki for a while in our clusters, it is considered to move the storage of Loki from K8s PV (EBS) to S3. There are some points being considered.
Flexibility in Scale
In these days I worked with Loki, the full disk issue happened several times, which means the Loki data reaches the limit of disk definition of the PV (1TB). Then, the Loki service stops working completely.
Loki provides configuring retention period¹ but unfortunately, it doesn’t provide a retention logic by disk size² yet. Therefore, a week-long retention period does make sense for troubleshooting issues.
When using PV with EBS in the Kubernetes clusters, it is treated as a local mounted disk, so a target disk size has to be provided when claiming the PV. Then, it is hard to tell how big to fit Loki data with the retention period, like a week.
With S3, you don’t need to specify any numbers of size since it is an external storage resource supported by Loki natively. Hence no worries for disk space at all.
#kiam #kubernetes #dynamodb #loki #aws s3
1594162113
AWS Fargate for Amazon Elastic Kubernetes Service | Caylent
On-demand cloud computing brings new ways to ensure scalability and efficiency. Rather than pre-allocating and managing certain server resources or having to go through the usual process of setting up a cloud cluster, apps and microservices can now rely on on-demand serverless computing blocks designed to be efficient and highly optimized.
Amazon Elastic Kubernetes Service (EKS) already makes running Kubernetes on AWS very easy. Support for AWS Fargate, which introduces the on-demand serverless computing element to the environment, makes deploying Kubernetes pods even easier and more efficient. AWS Fargate offers a wide range of features that make managing clusters and pods intuitive.
Utilizing Fargate
As with many other AWS services, using Fargate to manage Kubernetes clusters is very easy to do. To integrate Fargate and run a cluster on top of it, you only need to add the command –fargate to the end of your eksctl command.
EKS automatically configures the cluster to run on Fargate. It creates a pod execution role so that pod creation and management can be automated in an on-demand environment. It also patches coredns so the cluster can run smoothly on Fargate.
A Fargate profile is automatically created by the command. You can choose to customize the profile later or configure namespaces yourself, but the default profile is suitable for a wide range of applications already, requiring no human input other than a namespace for the cluster.
There are some prerequisites to keep in mind though. For starters, Fargate requires eksctl version 0.20.0 or later. Fargate also comes with some limitations, starting with support for only a handful of regions. For example, Fargate doesn’t support stateful apps, DaemonSets or privileged containers at the moment. Check out this link for Fargate limitations for your consideration.
Support for conventional load balancing is also limited, which is why ALB Ingress Controller is recommended. At the time of this writing, Classic Load Balancers and Network Load Balancers are not supported yet.
However, you can still be very meticulous in how you manage your clusters, including using different clusters to separate trusted and untrusted workloads.
Everything else is straightforward. Once the cluster is created, you can begin specifying pod execution roles for Fargate. You have the ability to use IAM console to create a role and assign it to a Fargate cluster. Or you can also create IAM roles and Fargate profiles via Terraform.
#aws #blog #amazon eks #aws fargate #aws management console #aws services #kubernetes #kubernetes clusters #kubernetes deployment #kubernetes pods
1603936365
The Benefits of Amazon S3 Explained Through a Comic
AWS S3 is one of the most fundamental services of AWS Cloud.
It’s basically your unlimited and safest cloud storage.
Read this comic style conversation between two guys and get to know why some of the biggest companies in the world are using Amazon S3 for their business and why you should use it too.
#aws-s3 #aws #cloud-computing #cloud-storage #data-storage #aws-services #aws-top-story #aws-certification
1604499240
Amazon S3 Hands-On — An In-Depth Step by Step Tutorial
This article aims to provide a hands on to beginners of AWS S3 service. We’ll explore the following features that are provided by the S3 service:
- Creating Buckets and Uploading data to S3.
- Buckets and Object Keys Structuring.
- Exploring S3 Storage Classes and Life Cycle Management.
- Exploring Bucket Versioning.
- Exploring Object Replication (CRR VS SRR).
- Restricting Access to Objects and Buckets.
Creating Buckets and Uploading data to S3
S3 bucket Creation
S3 is one of the most user-friendly service in the AWS ecosystem. We have multiple options available to upload data to S3 which include manually uploading data using the **Management Console **or uploading programmatically via S3 APIs, SDKs, and AWS CLI.
Here we’ll use the Management Console to upload data and keep things simple enough so let’s get started…
#aws #aws-s3 #s3 #s3-bucket #cloud-storage