kubernetes-sigs/cluster-api-provider-aws

EKS node group intermittently fails to delete on EKS cluster deletion

Open

#2,665 opened on Aug 13, 2021

View on GitHub
 (7 comments) (1 reaction) (0 assignees)Go (689 forks)auto 404
area/provider/ekshelp wantedkind/buglifecycle/rottenpriority/important-soon

Repository metrics

Stars
 (723 stars)
PR merge metrics
 (PR metrics pending)

Description

/kind bug

What steps did you take and what happened: Launch an EKS cluster with bastion enabled and one managed node group of size 5. Delete the 'Cluster'. The node group deletion fails with the "Delete failed" error in the aws console. The Health issue of the node group indicates "Ec2SecurityGroupDeletionFailure" --"DependencyViolation - resource has a dependent object". The affected security group is "eks-remoteAccess-*"

This is the error in CAPA logs: "msg"="Reconciler error" "error"="failed to reconcile machine pool deletion for AWSManagedMachinePool ns/workerpool-abc: failed to delete nodegroup: failed waiting for EKS nodegroup workerpool-abc to delete: ResourceNotReady: failed waiting for successful resource state".

The worker nodes/instances are deleted, bastion node is still running. The security-groups 'eks-cluster-sg-', '-node-eks-additional' and '*-bastion' are not deleted. Some network interfaces are still in-use. One network interface is in 'Available' state.

apiVersion: controlplane.cluster.x-k8s.io/v1alpha3
kind: AWSManagedControlPlane
metadata:
  name: capi-eks-quickstart-control-plane
  namespace: default
spec:
  eksClusterName: capi-eks-quickstart
  region: us-east-1
  version: v1.19.0
  bastion:
    allowedCIDRBlocks:
    - 0.0.0.0/0
    enabled: true
---
apiVersion: exp.cluster.x-k8s.io/v1alpha3
kind: MachinePool
metadata:
  name: capi-eks-quickstart-pool-0
  namespace: default
spec:
  clusterName: capi-eks-quickstart
  replicas: 5
  template:
    spec:
      bootstrap:
        dataSecretName: ""
      clusterName: capi-eks-quickstart
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
        kind: AWSManagedMachinePool
        name: capi-eks-quickstart-pool-0
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
kind: AWSManagedMachinePool
metadata:
  name: capi-eks-quickstart-pool-0
  namespace: default
spec:
  eksNodegroupName: node-group-0
---

What did you expect to happen: Expected the cluster to be deleted and all the associated resources to be cleaned up in AWS.

Anything else you would like to add: Manually deleting the ENI in 'Available' state resulted in the deletion to progress and eventually succeed. This is an intermittent issue. Possibly related to an EKS issue(dangling ENI) which is open for some time. Tried bumping the version of vpc-cni. That didn't make a difference.

Environment:

  • Cluster-api-provider-aws version: v0.6.5 - also reproducible with v0.6.8
  • Kubernetes version: (use kubectl version): 1.19.0

Contributor guide