Resource garbage collection feature should work in a private Subnet
#3,668 opened on Aug 12, 2022
Repository metrics
- Stars
- (723 stars)
- PR merge metrics
- (PR metrics pending)
Description
/kind feature
Describe the solution you'd like [A clear and concise description of what you want to happen.] I was trying the new resource garbage collection feature and it worked great for both AWS and EKS clusters 🎉
But then I tried with a bootstrap controller in a private Subnet that uses VPC Endpoints to access AWS services when managing clusters.
The cluster fails to delete with:
I0812 17:12:49.081842 1 awscluster_controller.go:209] controller/awscluster "msg"="Reconciling AWSCluster delete" "cluster"="e2e-aws-air-gapped-test-3568383" "name"="e2e-aws-air-gapped-test-3568383" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="AWSCluster"
I0812 17:16:50.269815 1 cleanup.go:43] controller/awscluster "msg"="reconciling deletion for garbage collection" "cluster"="e2e-aws-air-gapped-test-3568383" "name"="e2e-aws-air-gapped-test-3568383" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="AWSCluster"
I0812 17:16:50.269848 1 cleanup.go:65] controller/awscluster "msg"="deleting aws resources created by tenant cluster" "cluster"="e2e-aws-air-gapped-test-3568383" "name"="e2e-aws-air-gapped-test-3568383" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="AWSCluster"
E0812 17:18:50.659052 1 controller.go:317] controller/awscluster "msg"="Reconciler error" "error"="failed delete reconcile for gc service: getting tagged resources: RequestError: send request failed\ncaused by: Post \"https://tagging.us-west-2.amazonaws.com/\": dial tcp 54.240.253.156:443: i/o timeout" "name"="e2e-aws-air-gapped-test-3568383" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="AWSCluster"
Unfortunately there is no tagging VPC Endpoint from what I can tell (I've also searched in the AWS console and couldn't find anything related to "tag" or "tagging")
Error: creating EC2 VPC Endpoint (com.amazonaws.us-west-2.tagging): InvalidServiceName: The Vpc Endpoint Service 'com.amazonaws.us-west-2.tagging' does not exist
Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]
Here are the rest of the VPC Endpoints that we create in these private Subnet environments, "com.amazonaws.us-west-2.tagging" is not a valid service.
resource "aws_vpc_endpoint" "vpc_endpoints" {
for_each = toset( [
"com.amazonaws.us-west-2.ec2",
"com.amazonaws.us-west-2.elasticloadbalancing",
"com.amazonaws.us-west-2.autoscaling",
"com.amazonaws.us-west-2.secretsmanager",
"com.amazonaws.us-west-2.ssm",
"com.amazonaws.us-west-2.ssmmessages",
"com.amazonaws.us-west-2.ec2messages",
"com.amazonaws.us-west-2.tagging",
] )
vpc_id = aws_vpc.my_vpc.id
service_name = each.key
vpc_endpoint_type = "Interface"
security_group_ids = [aws_security_group.my_private.id]
// the bastion machine will be accessing this endpoint
subnet_ids = [aws_subnet.my_public.id]
private_dns_enabled = true
tags = var.tags
}
Environment:
- Cluster-api-provider-aws version:
- Kubernetes version: (use
kubectl version): - OS (e.g. from
/etc/os-release):