How-to Perform a Spark-Submit to Amazon EKS Cluster With IRSA

How-to Perform a Spark-Submit to Amazon EKS Cluster With IRSA

In this tutorial, I will show you how-to submit a Spark job version 2.4.4 with IRSA (IAM roles for Service Account).

In previous article, I have introduced how we submit a Spark job to an EKS cluster. As long as we’re using other AWS components for our pipelines to interact, like S3/DynamoDB/etc., we need assign IAM policy to the Spark driver and executor pods.

In this tutorial, I will show you how-to submit a Spark job version 2.4.4 with IRSA (IAM roles for Service Account).

What is IRSA?

According to AWS official documentation and blog:

Our approach, IAM Roles for Service Accounts (IRSA), however, is different: we made pods first class citizens in IAM. Rather than intercepting the requests to the EC2 metadata API to perform a call to the STS API to retrieve temporary credentials, we made changes in the AWS identity APIs to recognize Kubernetes pods. By combining an OpenID Connect (OIDC) identity provider and Kubernetes service account annotations, you can now use IAM roles at the pod level.

With IAM roles for service accounts on Amazon EKS clusters, you can associate an IAM role with a Kubernetes service account. This service account can then provide AWS permissions to the containers in any pod that uses that service account. With this feature, you no longer need to provide extended permissions to the node IAM role so that pods on that node can call AWS APIs.

AWS also mentions following benefits when combining IRSA with other community tools like kiam or kube2iam:

  • *Least privilege — *By using the IAM roles for service accounts feature, you no longer need to provide extended permissions to the node IAM role so that pods on that node can call AWS APIs. You can scope IAM permissions to a service account, and only pods that use that service account have access to those permissions. This feature also eliminates the need for third-party solutions such as kiam or kube2iam.
  • *Credential isolation — *A container can only retrieve credentials for the IAM role that is associated with the service account to which it belongs. A container never has access to credentials that are intended for another container that belongs to another pod.
  • *Auditability — *Access and event logging is available through CloudTrail to help ensure retrospective auditing.

spark irsa ek kubernetes aws

Bootstrap 5 Complete Course with Examples

Bootstrap 5 Tutorial - Bootstrap 5 Crash Course for Beginners

Nest.JS Tutorial for Beginners

Hello Vue 3: A First Look at Vue 3 and the Composition API

Building a simple Applications with Vue 3

Deno Crash Course: Explore Deno and Create a full REST API with Deno

How to Build a Real-time Chat App with Deno and WebSockets

Convert HTML to Markdown Online

HTML entity encoder decoder Online

50+ Useful Kubernetes Tools for 2020 - Part 2

Our original Kubernetes tool list was so popular that we've curated another great list of tools to help you improve your functionality with the platform.

AWS Fargate for Amazon Elastic Kubernetes Service | Caylent

Easily run Kubernetes-based applications on AWS by leveraging AWS Fargate and Amazon Elastic Kubernetes Service together. Learn more here.

Migrating Apache Spark workloads from AWS EMR to Kubernetes

I will focus on AWS Elastic Map Reduce since we are running our Spark workloads on AWS. We are using Apache Airflow for the workflow orchestration.

Create a Kubernetes Cluster and Deploy an App on Amazon EKS using AWS CLI

This blog describes how create multi-node kubernetes clusters and deploy an application on Amazon EKS. In this blog we will walk through how to setup kubectl and eksctl on workstations and from there we can use one line command to create an EKS Cluster.

Set up a SSH+Kubernetes bastion for AWS EKS with Teleport 3.2

Set up a SSH+Kubernetes bastion for AWS EKS with Teleport 3.2. This new release of Teleport brings support for EKS. Now Teleport can act as a single authentication gateway for Kubernetes clusters running on EKS.