Enhanced AMI lookup logic for EKS

/kind feature

In a sentence

CAPA should look up EKS AMIs for me automatically, but it should not allow a single AWSMachineTemplate or AWSMachinePool to front multiple AMIs.

note: AWSMachinePool PR https://github.com/kubernetes-sigs/cluster-api-provider-aws/pull/1860 is still in flight at this time.

Summary

CAPA today looks up AMI IDs before the runInstance call, if there is no specific AMI hardcoded into the spec of the AWSMachine or AWSMachinePool object. With Kubeadm-based clusters, the CAPI group is in charge of how and when new AMIs get published. We have an implicit assumption that asking for 1.16.8 will return the same AMI every time because we published that AMI and we don't publish new AMIs on the same version, per se.

EKS introduces some very interesting new behaviors:

1. Kubernetes patch version is upgraded behind the scenes automatically for the control plane
2. EKS clusters are created at and upgraded to minor versions only (I cannot create a 1.16.8 cluster specifically, and I can not tell EKS to upgrade my cluster to 1.16.8 specifically
3. There may be multiple AMIs published for a single patch version - I specifically call out that there are 2 1.16.8 AMIs with different docker versions

There is a PR open today that looks up the official EKS AMI in the most basic way: https://github.com/kubernetes-sigs/cluster-api-provider-aws/pull/1817 The official AWS documentations says to use an SSM parameter lookup:

aws ssm get-parameter --name /aws/service/eks/optimized-ami/1.16/amazon-linux-2/recommended/image_id --query "Parameter.Value" --output text

This is weird, because across the lifecycle of a MachineDeployment, you may end up with multiple patch versions represented among its Machines. Not only that, each patch version may be represented by multiple AMIs, depending on the time of Machine creation. But - this will map nicely to the control plane's automatic patch version upgrades, so there is a positive here that may be worth capturing.

I think, while automatic version upgrades are helpful, having AMIs swap in and out from under you can be really tough from a reliability perspective, especially if we don't at least capture when it happens.

Because of all of the above, I think CAPA should play a larger role in AMI selection.

Assorted thoughts

• AWSMachineTemplate is currently "immutable" but that doesn't stop CAPA from resolving a new AMI ID each time the template is copied
• How does Machine/MachineDeployment "version" map into this if EKS doesn't care about patch versions?
• You can actually look up all the AMIs for a given minor version:

aws ssm get-parameters-by-path --path /aws/service/eks/optimized-ami/1.16/amazon-linux-2 --region us-east-2 --output json

This output does contain references to patch versions. Is this something we should be parsing?

cc @richardcase @randomvariable