kubernetes-sigs/cluster-api-provider-aws

Resource garbage collection feature should work in a private Subnet

Open

#3,668 opened on Aug 12, 2022

View on GitHub
 (9 comments) (0 reactions) (0 assignees)Go (689 forks)auto 404
help wantedkind/featurelifecycle/staleneeds-priorityneeds-triagepriority/important-longterm

Repository metrics

Stars
 (723 stars)
PR merge metrics
 (PR metrics pending)

Description

/kind feature

Describe the solution you'd like [A clear and concise description of what you want to happen.] I was trying the new resource garbage collection feature and it worked great for both AWS and EKS clusters 🎉

But then I tried with a bootstrap controller in a private Subnet that uses VPC Endpoints to access AWS services when managing clusters.

The cluster fails to delete with:

I0812 17:12:49.081842       1 awscluster_controller.go:209] controller/awscluster "msg"="Reconciling AWSCluster delete" "cluster"="e2e-aws-air-gapped-test-3568383" "name"="e2e-aws-air-gapped-test-3568383" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="AWSCluster" 
I0812 17:16:50.269815       1 cleanup.go:43] controller/awscluster "msg"="reconciling deletion for garbage collection" "cluster"="e2e-aws-air-gapped-test-3568383" "name"="e2e-aws-air-gapped-test-3568383" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="AWSCluster" 
I0812 17:16:50.269848       1 cleanup.go:65] controller/awscluster "msg"="deleting aws resources created by tenant cluster" "cluster"="e2e-aws-air-gapped-test-3568383" "name"="e2e-aws-air-gapped-test-3568383" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="AWSCluster" 
E0812 17:18:50.659052       1 controller.go:317] controller/awscluster "msg"="Reconciler error" "error"="failed delete reconcile for gc service: getting tagged resources: RequestError: send request failed\ncaused by: Post \"https://tagging.us-west-2.amazonaws.com/\": dial tcp 54.240.253.156:443: i/o timeout" "name"="e2e-aws-air-gapped-test-3568383" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="AWSCluster" 

Unfortunately there is no tagging VPC Endpoint from what I can tell (I've also searched in the AWS console and couldn't find anything related to "tag" or "tagging")

Error: creating EC2 VPC Endpoint (com.amazonaws.us-west-2.tagging): InvalidServiceName: The Vpc Endpoint Service 'com.amazonaws.us-west-2.tagging' does not exist

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Here are the rest of the VPC Endpoints that we create in these private Subnet environments, "com.amazonaws.us-west-2.tagging" is not a valid service.

resource "aws_vpc_endpoint" "vpc_endpoints" {
  for_each = toset( [
    "com.amazonaws.us-west-2.ec2",
    "com.amazonaws.us-west-2.elasticloadbalancing",
    "com.amazonaws.us-west-2.autoscaling",
    "com.amazonaws.us-west-2.secretsmanager",
    "com.amazonaws.us-west-2.ssm",
    "com.amazonaws.us-west-2.ssmmessages",
    "com.amazonaws.us-west-2.ec2messages",
    "com.amazonaws.us-west-2.tagging",
  ] )

  vpc_id            = aws_vpc.my_vpc.id
  service_name      = each.key
  vpc_endpoint_type = "Interface"
  security_group_ids = [aws_security_group.my_private.id]
  // the bastion machine will be accessing this endpoint
  subnet_ids = [aws_subnet.my_public.id]
  private_dns_enabled = true

  tags = var.tags
}

Environment:

  • Cluster-api-provider-aws version:
  • Kubernetes version: (use kubectl version):
  • OS (e.g. from /etc/os-release):

Contributor guide