Skip to main content
Version: 0.95

AWS Infrastructure Setup - Manual

Overview​

This document covers the steps required to deploy a new Kubernetes cluster to your existing AWS account, including the creation of all required resources. This process will be completed through the AWS CLI as well as the web interface.

Assumptions/Prerequisites​

By default, Snorkel Flow is deployed into an existing AWS VPC and subnets, and configures new infrastructure in your AWS account. We start off with a couple assumptions:

  • You have access to an AWS account
  • You have access to a VPC, along with 3 or more subnets within that VPC
  • You have a domain that you own, as well as a certificate for TLS
    • The domain is configured in Route53
    • The certificate is configured in AWS Certificate Manager
  • The individual executing these setup tasks has admin access to the VPC

Part 1: Subnet Tagging​

To ensure the kubernetes cluster created later in this document recognizes the 3 or more subnets provided, we need to tag them each with 2 tags:

  • Navigate to subnets in the AWS Console
    • VPC -> Subnets
  • For each of the 3 subnets, add a tag for elb (for the aws-load-balancer-controller later in this document) and for the cluster
    • Subnet -> Manage tags
    • Add an elb tag
      • For private subnets
        • Key: kubernetes.io/role/internal-elb
        • Value: 1
      • For public subnets
        • Key: kubernetes.io/role/elb
        • Value: 1
    • Add a cluster tag
      • Key: kubernetes.io/cluster/snorkel-flow-aws
        note

        snorkel-flow-aws is the default eks cluster name. If you use a different cluster name during EKS setup, replace snorkel-flow-aws with your cluster name.

      • Value: shared

Part 2: Security Group Setup​

Create a security group for Snorkel Flow inside of your existing VPC:

  • Navigate to Security Groups in the AWS Console
    • EC2 -> Network and Security -> Security Groups
  • Create a new Security Group with name `snorkel-flow-aws` in the existing VPC, allowing all outbound traffic and TCP inbound traffic
    • Create security group
      • Security group name: snorkel-flow-aws
      • Description: snorkel-flow-aws
      • VPC: Existing VPC ID
      • Inbound traffic -> Add rule
        • Type: All traffic
        • Source: Anywhere-IPV4
      • Outbound traffic -> Add rule
        • Type: All traffic
        • Source: Anywhere-IPV4
      • Outbound traffic -> Add rule
        • Type: All traffic
        • Source: Anywhere-IPV6

Part 3: Create an EFS Instance​

Create an EFS instance for Snorkel Flow inside of your existing VPC:

  • Navigate to EFS in the AWS Console
  • Create a new EFS instance with name `snorkel-flow-aws` in the existing VPC, with existing subnets as the mount points
    • Create file system -> Customize
      • File system settings
        • Name: snorkel-flow-aws
        • Storage class: standard
        • Enable automated backups
        • Enable encryption at rest
        • Throughput mode: Bursting
      • Network access
        • VPC: your existing VPC
        • Mount targets
          • Your 3 or more subnets, along with their associated availability zone
          • Security group: add `snorkel-flow-aws` created in Part 2
      • Create

Create an access point at the root directory of your EFS volume, with POSIX and creation permissions for user 99:

  • Navigate to the new EFS in the AWS Console
    • EFS -> snorkel-flow-aws -> Access points
  • Create a new access point
    • Create access point
      • Name: snorkel-flow-aws
      • Root directory path: /snorkel-flow/data
      • POSIX User ID: 99
      • POSIX Group ID: 99
      • POSIX Secondary group IDs: 99
      • Root directory owner user ID: 99
      • Root directory owner group ID: 99
      • Root directory access point permissions: 775
    • Create access point

Part 4: Create an EKS cluster​

Create an IAM role for the EKS cluster backplane:

  • Navigate to IAM roles in the AWS Console
    • IAM -> Roles
  • Create a new role snorkel-flow-aws-cluster with AmazonEKSClusterPolicy and AmazonEKSVPCResourceController attached
    • Create role
      • Select trusted entity
        • Custom trust policy
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "EKSClusterAssumeRole",
            "Effect": "Allow",
            "Principal": {
                "Service": "eks.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}
  • Add permissions
    • Add the `AmazonEKSClusterPolicy` and `AmazonEKSVPCResourceController` policies
  • Name, review, and create
    • Role name: snorkel-flow-aws-cluster
    • Create Role

Create an EKS cluster for Snorkel Flow inside of your existing VPC:

  • Navigate to EKS (Elastic Kubernetes Service) in the AWS Console
  • Create a new EKS cluster with name `snorkel-flow-aws` in the existing VPC
    • Add cluster -> Create
      • Configure cluster
        • Name: snorkel-flow-aws
        • Kubernetes version: default
        • Cluster service role: snorkel-flow-aws-cluster
      • Specify networking
        • VPC: your existing VPC
        • Subnets: your existing 3 or more subnets
        • Security Groups: snorkel-flow-aws
      • Select addons
        • Amazon VPC CNI
        • Kube-proxy
        • CoreDNS
      • Create

Part 5: Add a node group to the EKS cluster​

Create an IAM role for the EKS node group:

  • Navigate to IAM roles in the AWS Console
    • IAM -> Roles
  • Create a new role snorkel-flow-aws-cluster-node with AmazonEC2ContainerRegistryReadOnly, AmazonEKS_CNI_Policy, and AmazonEKSWorkerNodePolicy attached
    • Create role
      • Select trusted entity
        • Custom trust policy
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "EKSNodeAssumeRole",
            "Effect": "Allow",
            "Principal": {
                "Service": "ec2.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}
  • Add permissions
    • Add the `AmazonEC2ContainerRegistryReadOnly`, `AmazonEKS_CNI_Policy`, and `AmazonEKSWorkerNodePolicy` policies
  • Name, review, and create
    • Role name: snorkel-flow-aws-cluster-node
    • Create Role

Create a node group for the EKS cluster:

  • Navigate to the EKS cluster created in Part 4 in the AWS Console
    • EKS (Elastic Kubernetes Service) -> Clusters -> snorkel-flow-aws -> Compute
  • Add a new node group
    • Add node group
      • Configure node group
        • Name: snorkel-flow-aws
        • Node IAM role: snorkel-flow-aws-cluster-node
      • Set compute and scaling configuration
        • Node group compute configuration
          • AMI type: Amazon Linux 2 (AL_x86_64)
          • Capacity type: On-Demand
          • Instance Types: m5.8xlarge
          • Disk Size: 1024GiB
        • Node group scaling configuration
          • Desired size: 2 nodes
          • Minimum size: 2 nodes
          • Maximum size: 2 nodes
      • Specify Networking
        • Subnets: your 3 or more existing subnets
      • Create

Part 6: Create aws-efs-csi-driver, aws-load-balancer-controller, external-dns, and aws-ebs-csi-driver operators via helm​

Create IAM policies for aws-efs-csi-driver, aws-load-balancer-controller, external-dns, and aws-ebs-csi-driver operators.

  • Navigate to IAM policies in the AWS Console
    • IAM -> Policies
  • Create a new policy lb-controller-irsa-aws-load-balancer-controller
    • Create policy
      • Policy editor -> JSON
{
    "Statement": [
        {
            "Action": "iam:CreateServiceLinkedRole",
            "Condition": {
                "StringEquals": {
                    "iam:AWSServiceName": "elasticloadbalancing.amazonaws.com"
                }
            },
            "Effect": "Allow",
            "Resource": "*"
        },
        {
            "Action": [
                "elasticloadbalancing:DescribeTargetHealth",
                "elasticloadbalancing:DescribeTargetGroups",
                "elasticloadbalancing:DescribeTargetGroupAttributes",
                "elasticloadbalancing:DescribeTags",
                "elasticloadbalancing:DescribeSSLPolicies",
                "elasticloadbalancing:DescribeRules",
                "elasticloadbalancing:DescribeLoadBalancers",
                "elasticloadbalancing:DescribeLoadBalancerAttributes",
                "elasticloadbalancing:DescribeListeners",
                "elasticloadbalancing:DescribeListenerCertificates",
                "ec2:GetCoipPoolUsage",
                "ec2:DescribeVpcs",
                "ec2:DescribeVpcPeeringConnections",
                "ec2:DescribeTags",
                "ec2:DescribeSubnets",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeNetworkInterfaces",
                "ec2:DescribeInternetGateways",
                "ec2:DescribeInstances",
                "ec2:DescribeCoipPools",
                "ec2:DescribeAvailabilityZones",
                "ec2:DescribeAddresses",
                "ec2:DescribeAccountAttributes"
            ],
            "Effect": "Allow",
            "Resource": "*"
        },
        {
            "Action": [
                "wafv2:GetWebACLForResource",
                "wafv2:GetWebACL",
                "wafv2:DisassociateWebACL",
                "wafv2:AssociateWebACL",
                "waf-regional:GetWebACLForResource",
                "waf-regional:GetWebACL",
                "waf-regional:DisassociateWebACL",
                "waf-regional:AssociateWebACL",
                "shield:GetSubscriptionState",
                "shield:DescribeProtection",
                "shield:DeleteProtection",
                "shield:CreateProtection",
                "iam:ListServerCertificates",
                "iam:GetServerCertificate",
                "cognito-idp:DescribeUserPoolClient",
                "acm:ListCertificates",
                "acm:DescribeCertificate"
            ],
            "Effect": "Allow",
            "Resource": "*"
        },
        {
            "Action": [
                "ec2:RevokeSecurityGroupIngress",
                "ec2:AuthorizeSecurityGroupIngress"
            ],
            "Effect": "Allow",
            "Resource": "*"
        },
        {
            "Action": "ec2:CreateSecurityGroup",
            "Effect": "Allow",
            "Resource": "*"
        },
        {
            "Action": "ec2:CreateTags",
            "Condition": {
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": "false"
                },
                "StringEquals": {
                    "ec2:CreateAction": "CreateSecurityGroup"
                }
            },
            "Effect": "Allow",
            "Resource": "arn:aws:ec2:*:*:security-group/*"
        },
        {
            "Action": [
                "ec2:DeleteTags",
                "ec2:CreateTags"
            ],
            "Condition": {
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": "true",
                    "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
                }
            },
            "Effect": "Allow",
            "Resource": "arn:aws:ec2:*:*:security-group/*"
        },
        {
            "Action": [
                "ec2:RevokeSecurityGroupIngress",
                "ec2:DeleteSecurityGroup",
                "ec2:AuthorizeSecurityGroupIngress"
            ],
            "Condition": {
                "Null": {
                    "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
                }
            },
            "Effect": "Allow",
            "Resource": "*"
        },
        {
            "Action": [
                "elasticloadbalancing:CreateTargetGroup",
                "elasticloadbalancing:CreateLoadBalancer"
            ],
            "Condition": {
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": "false"
                }
            },
            "Effect": "Allow",
            "Resource": "*"
        },
        {
            "Action": [
                "elasticloadbalancing:DeleteRule",
                "elasticloadbalancing:DeleteListener",
                "elasticloadbalancing:CreateRule",
                "elasticloadbalancing:CreateListener"
            ],
            "Effect": "Allow",
            "Resource": "*"
        },
        {
            "Action": [
                "elasticloadbalancing:RemoveTags",
                "elasticloadbalancing:AddTags"
            ],
            "Condition": {
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": "true",
                    "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
                }
            },
            "Effect": "Allow",
            "Resource": [
                "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*",
                "arn:aws:elasticloadbalancing:*:*:loadbalancer/net/*/*",
                "arn:aws:elasticloadbalancing:*:*:loadbalancer/app/*/*"
            ]
        },
        {
            "Action": [
                "elasticloadbalancing:RemoveTags",
                "elasticloadbalancing:AddTags"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:elasticloadbalancing:*:*:listener/net/*/*/*",
                "arn:aws:elasticloadbalancing:*:*:listener/app/*/*/*",
                "arn:aws:elasticloadbalancing:*:*:listener-rule/net/*/*/*",
                "arn:aws:elasticloadbalancing:*:*:listener-rule/app/*/*/*"
            ]
        },
        {
            "Action": "elasticloadbalancing:AddTags",
            "Condition": {
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": "false"
                },
                "StringEquals": {
                    "elasticloadbalancing:CreateAction": [
                        "CreateTargetGroup",
                        "CreateLoadBalancer"
                    ]
                }
            },
            "Effect": "Allow",
            "Resource": [
                "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*",
                "arn:aws:elasticloadbalancing:*:*:loadbalancer/net/*/*",
                "arn:aws:elasticloadbalancing:*:*:loadbalancer/app/*/*"
            ]
        },
        {
            "Action": [
                "elasticloadbalancing:SetSubnets",
                "elasticloadbalancing:SetSecurityGroups",
                "elasticloadbalancing:SetIpAddressType",
                "elasticloadbalancing:ModifyTargetGroupAttributes",
                "elasticloadbalancing:ModifyTargetGroup",
                "elasticloadbalancing:ModifyLoadBalancerAttributes",
                "elasticloadbalancing:DeleteTargetGroup",
                "elasticloadbalancing:DeleteLoadBalancer"
            ],
            "Condition": {
                "Null": {
                    "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
                }
            },
            "Effect": "Allow",
            "Resource": "*"
        },
        {
            "Action": [
                "elasticloadbalancing:RegisterTargets",
                "elasticloadbalancing:DeregisterTargets"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*"
        },
        {
            "Action": [
                "elasticloadbalancing:SetWebAcl",
                "elasticloadbalancing:RemoveListenerCertificates",
                "elasticloadbalancing:ModifyRule",
                "elasticloadbalancing:ModifyListener",
                "elasticloadbalancing:AddListenerCertificates"
            ],
            "Effect": "Allow",
            "Resource": "*"
        }
    ],
    "Version": "2012-10-17"
}

  • Policy name: lb-controller-irsa-aws-load-balancer-controller

  • Create policy

  • Create a new policy external-dns-irsa-external-dns

    • Create policy
      • Policy editor -> JSON
{
    "Statement": [
        {
            "Action": "route53:ChangeResourceRecordSets",
            "Effect": "Allow",
            "Resource": "arn:aws:route53:::hostedzone/*",
            "Sid": "ChangeResourceRecordSets"
        },
        {
            "Action": [
                "route53:ListTagsForResource",
                "route53:ListResourceRecordSets",
                "route53:ListHostedZones"
            ],
            "Effect": "Allow",
            "Resource": "*",
            "Sid": "ListResourceRecordSets"
        }
    ],
    "Version": "2012-10-17"
}

  • Policy name: external-dns-irsa-external-dns

  • Create policy

  • Create a new policy efs-csi-controller-aws-efs-csi-driver

    • Create policy
      • Policy editor -> JSON
{
    "Statement": [
        {
            "Action": [
                "elasticfilesystem:DescribeMountTargets",
                "elasticfilesystem:DescribeFileSystems",
                "elasticfilesystem:DescribeAccessPoints",
                "ec2:DescribeAvailabilityZones"
            ],
            "Effect": "Allow",
            "Resource": "*"
        },
        {
            "Action": "elasticfilesystem:CreateAccessPoint",
            "Condition": {
                "StringLike": {
                    "aws:RequestTag/efs.csi.aws.com/cluster": "true"
                }
            },
            "Effect": "Allow",
            "Resource": "*"
        },
        {
            "Action": "elasticfilesystem:TagResource",
            "Condition": {
                "StringLike": {
                    "aws:ResourceTag/efs.csi.aws.com/cluster": "true"
                }
            },
            "Effect": "Allow",
            "Resource": "*"
        },
        {
            "Action": "elasticfilesystem:DeleteAccessPoint",
            "Condition": {
                "StringEquals": {
                    "aws:ResourceTag/efs.csi.aws.com/cluster": "true"
                }
            },
            "Effect": "Allow",
            "Resource": "*"
        }
    ],
    "Version": "2012-10-17"
}

  • Policy name: efs-csi-controller-aws-efs-csi-driver

  • Create policy

  • Create a new policy ebs-csi-controller-aws-ebs-csi-driver

    • Create policy
      • Policy editor -> JSON
{
    "Statement": [
        {
            "Action": [
                "ec2:ModifyVolume",
                "ec2:DetachVolume",
                "ec2:DescribeVolumesModifications",
                "ec2:DescribeVolumes",
                "ec2:DescribeTags",
                "ec2:DescribeSnapshots",
                "ec2:DescribeInstances",
                "ec2:DescribeAvailabilityZones",
                "ec2:CreateSnapshot",
                "ec2:AttachVolume"
            ],
            "Effect": "Allow",
            "Resource": "*"
        },
        {
            "Action": "ec2:CreateTags",
            "Condition": {
                "StringEquals": {
                    "ec2:CreateAction": [
                        "CreateVolume",
                        "CreateSnapshot"
                    ]
                }
            },
            "Effect": "Allow",
            "Resource": [
                "arn:aws:ec2:*:*:volume/*",
                "arn:aws:ec2:*:*:snapshot/*"
            ]
        },
        {
            "Action": "ec2:DeleteTags",
            "Effect": "Allow",
            "Resource": [
                "arn:aws:ec2:*:*:volume/*",
                "arn:aws:ec2:*:*:snapshot/*"
            ]
        },
        {
            "Action": "ec2:CreateVolume",
            "Condition": {
                "StringLike": {
                    "aws:RequestTag/ebs.csi.aws.com/cluster": "true"
                }
            },
            "Effect": "Allow",
            "Resource": "*"
        },
        {
            "Action": "ec2:CreateVolume",
            "Condition": {
                "StringLike": {
                    "aws:RequestTag/CSIVolumeName": "*"
                }
            },
            "Effect": "Allow",
            "Resource": "*"
        },
        {
            "Action": "ec2:CreateVolume",
            "Condition": {
                "StringLike": {
                    "aws:RequestTag/kubernetes.io/cluster/*": "owned"
                }
            },
            "Effect": "Allow",
            "Resource": "*"
        },
        {
            "Action": "ec2:DeleteVolume",
            "Condition": {
                "StringLike": {
                    "ec2:ResourceTag/ebs.csi.aws.com/cluster": "true"
                }
            },
            "Effect": "Allow",
            "Resource": "*"
        },
        {
            "Action": "ec2:DeleteVolume",
            "Condition": {
                "StringLike": {
                    "ec2:ResourceTag/CSIVolumeName": "*"
                }
            },
            "Effect": "Allow",
            "Resource": "*"
        },
        {
            "Action": "ec2:DeleteVolume",
            "Condition": {
                "StringLike": {
                    "ec2:ResourceTag/kubernetes.io/cluster/*": "owned"
                }
            },
            "Effect": "Allow",
            "Resource": "*"
        },
        {
            "Action": "ec2:DeleteSnapshot",
            "Condition": {
                "StringLike": {
                    "ec2:ResourceTag/CSIVolumeSnapshotName": "*"
                }
            },
            "Effect": "Allow",
            "Resource": "*"
        },
        {
            "Action": "ec2:DeleteSnapshot",
            "Condition": {
                "StringLike": {
                    "ec2:ResourceTag/ebs.csi.aws.com/cluster": "true"
                }
            },
            "Effect": "Allow",
            "Resource": "*"
        }
    ],
    "Version": "2012-10-17"
}
  • Policy name: ebs-csi-controller-aws-ebs-csi-driver
  • Create policy

Create associated IAM roles for aws-efs-csi-driver, aws-load-balancer-controller, external-dns, and aws-ebs-csi-driver operators.

  • Navigate to IAM roles in the AWS Console
    • IAM -> Roles
  • Create a new role lb-controller-irsa-aws-load-balancer-controller that refers to the lb-controller-irsa-aws-load-balancer-controller policy 
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::${ACCOUNT_ID}:oidc-provider/${OIDC_URL}"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "${OIDC_URL}:sub": "system:serviceaccount:kube-system:aws-load-balancer-controller"
                }
            }
        }
    ]
}

  • Add permissions

    • Add the `lb-controller-irsa-aws-load-balancer-controller` policy created in the prior steps

  • Name, review, and create

    • Role name: lb-controller-irsa-aws-load-balancer-controller
    • Create Role

  • Create a new role external-dns-irsa-external-dns that refers to the external-dns-irsa-external-dns policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::${ACCOUNT_ID}:oidc-provider/${OIDC_URL}"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "${OIDC_URL}:sub": "system:serviceaccount:kube-system:external-dns"
                }
            }
        }
    ]
}

  • Add permissions

    • Add the `external-dns-irsa-external-dns` policy created in the prior steps

  • Name, review, and create

    • Role name: external-dns-irsa-external-dns
    • Create Role

  • Create a new role efs-csi-controller-aws-efs-csi-driver that refers to the efs-csi-controller-aws-efs-csi-driver policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::${ACCOUNT_ID}:oidc-provider/${OIDC_URL}"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "${OIDC_URL}:sub": "system:serviceaccount:kube-system:efs-csi-controller-sa"
                }
            }
        }
    ]
}

  • Add permissions

    • Add the `efs-csi-controller-aws-efs-csi-driver` policy created in the prior steps

  • Name, review, and create

    • Role name: efs-csi-controller-aws-efs-csi-driver
    • Create Role

  • Create a new role ebs-csi-controller-aws-ebs-csi-driver that refers to the ebs-csi-controller-aws-ebs-csi-driver policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::${ACCOUNT_ID}:oidc-provider/${OIDC_URL}"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "${OIDC_URL}:sub": "system:serviceaccount:kube-system:ebs-csi-controller-sa"
                }
            }
        }
    ]
}
  • Add permissions
    • Add the `ebs-csi-controller-aws-ebs-csi-driver` policy created in the prior steps
  • Name, review, and create
    • Role name: ebs-csi-controller-aws-ebs-csi-driver
    • Create Role

Create aws-efs-csi-driver, aws-load-balancer-controller, external-dns, and aws-ebs-csi-driver operators via helm charts.

  • Connect to your cluster via kubectl 
    • aws eks --region us-west-2 update-kubeconfig --name snorkel-flow-aws
    • kubectl get pods --all-namespaces
    • Should return coredns, aws-node, and kube-proxy pods
  • Deploy aws-efs-csi-driver 
    • helm repo add aws-efs-csi-driver https://kubernetes-sigs.github.io/aws-efs-csi-driver/
    • helm repo update aws-efs-csi-driver
    • helm install aws-efs-csi-driver aws-efs-csi-driver/aws-efs-csi-driver -n kube-system --set controller.serviceAccount.annotations."eks\.amazonaws\.com/role-arn"="arn:aws:iam::${ACCOUNT_ID}:role/efs-csi-controller-aws-efs-csi-driver" --version 2.4.7
    • kubectl get pods -n kube-system
    • Confirm that this now has the new efs-csi pods
  • Deploy aws-load-balancer-controller 
    • helm repo add eks [https://aws.github.io/eks-charts](https://aws.github.io/eks-charts)
    • helm repo update eks
    • helm install aws-load-balancer-controller eks/aws-load-balancer-controller -n kube-system --set clusterName=snorkel-flow-aws --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"="arn:aws:iam::${ACCOUNT_ID}:role/lb-controller-irsa-aws-load-balancer-controller" --version 1.5.5
  • Deploy external-dns
    • create an OIDC provider for the cluster first (follow instructions in https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html)
    • helm repo add external-dns [https://kubernetes-sigs.github.io/external-dns/](https://kubernetes-sigs.github.io/external-dns/)
    • helm repo update external-dns
    • helm install external-dns external-dns/external-dns -n kube-system --set provider=aws --set txtOwnerId=snorkel-flow-aws --set txtPrefix=txt- --set domainFilters={$DOMAIN} --set extraArgs={--aws-prefer-cname} --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"="arn:aws:iam::${ACCOUNT_ID}:role/external-dns-irsa-external-dns" --version 1.13.0
    • Note: $DOMAIN is the domain that you own
  • Deploy aws-ebs-csi-driver 
    • helm repo add aws-ebs-csi-driver [https://kubernetes-sigs.github.io/aws-ebs-csi-driver](https://kubernetes-sigs.github.io/aws-ebs-csi-driver)
    • helm repo update aws-ebs-csi-driver
    • helm install aws-ebs-csi-driver aws-ebs-csi-driver/aws-ebs-csi-driver -n kube-system --set controller.serviceAccount.annotations."eks\.amazonaws\.com/role-arn"="arn:aws:iam::${ACCOUNT_ID}:role/ebs-csi-controller-aws-ebs-csi-driver" --version 2.21.0

Part 7: AWS-Specific Configurations for Snorkel Flow

While deploying Snorkel Flow in AWS (link), there are a couple specific configurations that are needed to be set for helm to work with the infrastructure set up above:

  • For annotations on the UI ingress, we’ll need the following:
alb.ingress.kubernetes.io/scheme: internal
external-dns.alpha.kubernetes.io/ttl: "60"
alb.ingress.kubernetes.io/certificate-arn: $AWS_CERTIFICATE_ARN
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
alb.ingress.kubernetes.io/ssl-redirect: "443"
alb.ingress.kubernetes.io/load-balancer-attributes: idle_timeout.timeout_seconds=600
alb.ingress.kubernetes.io/success-codes: 200-302
alb.ingress.kubernetes.io/target-type: "ip"
external-dns.alpha.kubernetes.io/hostname: $SUBDOMAIN.$DOMAIN
alb.ingress.kubernetes.io/group.name: $SUBDOMAIN.$DOMAIN
alb.ingress.kubernetes.io/healthcheck-path: /health
  • $AWS_CERTIFICATE_ARN is the certificate configured for your domain in AWS Certificate Manager
  • $SUBDOMAIN is the subdomain chosen for Snorkel Flow’s url
  • $DOMAIN is a domain configured in route53
  • Note: a similar set of annotations are required for each ingress Snorkel Flow uses
  • For the data volume driver, we’ll need the following:
csi:
  driver: efs.csi.aws.com
  volumeHandle: $EFS_ID::$ACCESS_POINT_ID
  • $EFS_ID is the id of the EFS created earlier for Snorkel Flow
  • $ACCESS_POI