EKS node group intermittently fails to delete on EKS cluster deletion
#2,665 opened on Aug 13, 2021
Repository metrics
- Stars
- (723 stars)
- PR merge metrics
- (PR metrics pending)
Description
/kind bug
What steps did you take and what happened: Launch an EKS cluster with bastion enabled and one managed node group of size 5. Delete the 'Cluster'. The node group deletion fails with the "Delete failed" error in the aws console. The Health issue of the node group indicates "Ec2SecurityGroupDeletionFailure" --"DependencyViolation - resource has a dependent object". The affected security group is "eks-remoteAccess-*"
This is the error in CAPA logs:
"msg"="Reconciler error" "error"="failed to reconcile machine pool deletion for AWSManagedMachinePool ns/workerpool-abc: failed to delete nodegroup: failed waiting for EKS nodegroup workerpool-abc to delete: ResourceNotReady: failed waiting for successful resource state".
The worker nodes/instances are deleted, bastion node is still running. The security-groups 'eks-cluster-sg-', '-node-eks-additional' and '*-bastion' are not deleted. Some network interfaces are still in-use. One network interface is in 'Available' state.
apiVersion: controlplane.cluster.x-k8s.io/v1alpha3
kind: AWSManagedControlPlane
metadata:
name: capi-eks-quickstart-control-plane
namespace: default
spec:
eksClusterName: capi-eks-quickstart
region: us-east-1
version: v1.19.0
bastion:
allowedCIDRBlocks:
- 0.0.0.0/0
enabled: true
---
apiVersion: exp.cluster.x-k8s.io/v1alpha3
kind: MachinePool
metadata:
name: capi-eks-quickstart-pool-0
namespace: default
spec:
clusterName: capi-eks-quickstart
replicas: 5
template:
spec:
bootstrap:
dataSecretName: ""
clusterName: capi-eks-quickstart
infrastructureRef:
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
kind: AWSManagedMachinePool
name: capi-eks-quickstart-pool-0
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
kind: AWSManagedMachinePool
metadata:
name: capi-eks-quickstart-pool-0
namespace: default
spec:
eksNodegroupName: node-group-0
---
What did you expect to happen: Expected the cluster to be deleted and all the associated resources to be cleaned up in AWS.
Anything else you would like to add: Manually deleting the ENI in 'Available' state resulted in the deletion to progress and eventually succeed. This is an intermittent issue. Possibly related to an EKS issue(dangling ENI) which is open for some time. Tried bumping the version of vpc-cni. That didn't make a difference.
Environment:
- Cluster-api-provider-aws version: v0.6.5 - also reproducible with v0.6.8
- Kubernetes version: (use
kubectl version): 1.19.0