AWSMachinePool does not drain nodes during scale-in

/kind bug

What steps did you take and what happened:

• Create a workload cluster with the experimental EKS Control Plane.
• Create a MachinePool with replicas: 5 and create the associated AWSMachinePool resources. (Note: this AWSMachinePool is not managed by cluster-autoscaler)
• Create a deployment and scale it such that some pods fall on all machines
• Create a PDB protecting the deployment with maxUnavailable: 1
• Scale the MachinePool down to replicas: 3

This caused the AWSMachineController to set the DesiredInstances in the ASG to 3 without draining nodes at all. The PDB was not honored, and the EC2 instances were terminated by the ASG immediately.

What did you expect to happen: The nodes should have drained gracefully before the EC2 instances are terminated.

Anything else you would like to add: In the current AWSMachinePool implementation, the instance selection for scale-in is performed at the AutoScalingGroup. This could be fixed in the non-cluster-autoscaler case by modifying AWSMachinePool controller to perform node selection for scale-in, drain the selected nodes, and finally utilize the AWS TerminateInstanceInAutoScalingGroup action while setting the request value ShouldDecrementDesiredCapacity: true

We may want to also consider a lifecycle hook on the autoscaling group that prevents ec2 instance termination until the drain completes. This would help to prevent cases where instances are forcibly terminated without draining when the DesiredInstances values are manipulated via the EC2 console, CLI, or APIs.

Environment:

Cluster-api-provider-aws version: Commit: 3338cd4 Kubernetes version: (use kubectl version): v.1.17.9 OS (e.g. from /etc/os-release): Amazon Linux 2