Setting the stage

When deploying applications to Kubernetes running on an AWS, it is necessary to have an authentication mechanism which allows your applications to make signed requests to AWS resources. In the pre-container world, this would be done by assigning an IAM role to the EC2 instance doing the work. Since we could generally assume it has a 1:1 relationship between EC2 instances and applications, this approach works great. However, containers allow us to run multiple applications on the same instance. If we extend this pattern, we’d get an EC2 instance with an aggregate permission set of the entire cluster, and all applications on the instance would inherit those permissions.

One way to maintain a separation of concerns between applications in the cluster is to create an IAM “application-user” for each application, assigning a set of permissions to that user, and generating a keypair. That ID and Secret pair is then injected into the Pod and used by the containers. However, authenticating in this way does require a strategy for key rotation and invalidation. Certainly, you could write or implement automation software to handle this, but every Pod would still be holding a long duration access key ID and Secret. Kube2IAM solves this problem by removing the keys altogether.

How it works

Kube2IAM is an application that can be deployed as a DeamonSet that rewrites a portion of the Kubernetes IP table in order to proxy all traffic destined for the AWS Metadata API. When such a request is received, Kube2IAM examines the annotations on the requesting pod for the role it is requesting, assumes that role, and makes the credentialing request for the pod. Once a short-lived token is received from AWS, it is passed into the pod enabling the pod to make signed requests. The result is a cluster which maintains a separation of concerns between applications and does not store long lived AWS keypairs.

Implementation

The first thing you are going to need is an AWS account with a running Kubenetes cluster for development/testing purposes you can use kops to spin one up quickly for development purposes.

Credentialing Kube2IAM

Now that you have a cluster up, you are going to need to credential and deploy Kube2IAM. There are two methods you can use for credentialing, each with its benefit and cost.

1. Set up an IAM “application-user” for Kube2IAM and give it the ability to assume a role from a defined list of “assumable roles”.

   • This results in a long-lived key in your Kube2IAM pod which will need to be rotated

2. Assign the IAM role directly to your Kubernetes EC2 instances.

   • This gives any pod on the cluster the ability to assume any other pods role by making the sts:AssumeRole call and has no long lived credential.

Here is a sample IAM JSON permission for Kube2IAM.

Note: a path for all “assumable roles” IAM resources restricts the roles Kube2IAM can assume. This path is defined on role creation.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "sts:AssumeRole"
            ],
            "Resource": [
                "arn:aws:iam::<accountNumber>:role/<path>/*"
            ]
        }
    ]
}

Deploying Kube2IAM

Kube2IAM can be most easily deployed via Helm from the stable Kubernetes chart using helm install stable/kube2iam --name my-release. If you need custom configuration, you can replace the default values with your own. helm install stable/kube2iam --name my-release -f values.yaml.

Service roles

At this point you should have a running cluster with a Kube2IAM pod. The next step is to create roles defining permission sets for your applications. You can learn how to do this in the AWS IAM docs. The only thing to watch out for here is to ensure you create roles with a Trust Relationship between the “assumeable role” and Kube2IAM. Information on that can be found here.

Annotating your charts

The last step is to annotate your charts with the role the pod will need to assume. A guide to doing this can be found here. Here is an example of a deployment, there is more information about Kubernetes annotations in the docs

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 3
  template:
    metadata:
      annotations:
        iam.amazonaws.com/role: role-arn
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.9.1
        ports:
        - containerPort: 80

Conclusion

In summary, while running a Kubernetes cluster on AWS does give a business wins in scalability, reliability and agility, it can be challenging to apply certain security best practices designed for monolithic applications to a containerized ecosystem. Kube2IAM enables you maintain a separation of concerns between applications, remove long lived keys and secrets from Pods where they could potentially be stolen and misused, and eliminates the need to manage dozens of application keys.

If you have any questions please feel free to e-mail us at [email protected]