aws-plugin from v1.7.1+ Errors when using IRSA

What steps did you take and what happened: Upgraded velero-plugin-for-aws from v1.7.1 to any later version. (also upgraded the 'matching' velero version per the compatibility chart)

Authentication via IAM roles in the service account annotation stops working.

Velero configuration:

configuration:
  backupStorageLocation:
  - name: "[redacted]"
    provider: aws
    bucket: "[redacted]"
    prefix: velero
    accessMode: ReadWrite
    annotations: {meta.helm.sh/release-name: velero,  meta.helm.sh/release-namespace: [redacted]}
    config: 
      region: eu-west-2
      s3ForcePathStyle: true
      profile: ""
  volumeSnapshotLocation: 
  - name: generic-volume
    snapshotsEnabled: false
    provider: aws
    config:
      region: eu-west-2    
  defaultBackupStorageLocation: "[redacted]"
  logFormat: json
  logLevel: warning
  namespace:  [redacted]
schedules:
  backup:
    disabled: false
    annotations: {namespaces: all, meta.helm.sh/release-name: velero,  meta.helm.sh/release-namespace: [redacted]}
    schedule: "00 00 * * *"
    useOwnerReferencesInBackup: false
    template:
      ttl: "720h"
      storageLocation: "[redacted]"

serviceAccount:
  server:
    annotations: {eks.amazonaws.com/role-arn: "arn:aws:iam::[redacted]:role/[redacted]", iam.amazonaws.com/role: "arn:aws:iam::[redacted]:role/[redacted]", meta.helm.sh/release-name: velero,  meta.helm.sh/release-namespace: [redacted]}

initContainers:
  - name: velero-plugin-for-aws
    image: "[redacted]/docker-hub/velero/velero-plugin-for-aws:v1.8.0"
    imagePullPolicy: IfNotPresent
    volumeMounts:
      - mountPath: /target
        name: plugins 

backupStorageLocation.config.profile: "" was added after reading some other issues where but made not difference

I saw a few issues suggesting to use credentials.useSecret: false at top level and under the bsl, but then I received startup errors complaining that MountVolume.SetUp failed for volume "cloud-credentials" : secret "velero" not found

So it appears that the credentials file is mandatory despite documentation, and even though I'm not trying to use that for authentication.

On the lastest version, got

Error logs plugin v.1.8.0 with velero v1.12.0:

{
	"controller": "backup-storage-location",
	"level": "error",
	"logSource": "pkg/controller/backup_storage_location_controller.go:180",
	"msg": "Current BackupStorageLocations available/unavailable/unknown: 0/1/0, BackupStorageLocation \"[redacted]\" is unavailable: rpc error: code = Unknown desc = provided credentialsFile does not exist: stat /credentials/cloud: no such file or directory)",
	"time": "2025-05-20T15:04:03Z"
}

Error logs plugin v.1.9.0 with velero v1.13.2:

{
	"controller": "backup-storage-location",
	"level": "error",
	"logSource": "pkg/controller/backup_storage_location_controller.go:178",
	"msg": "Current BackupStorageLocations available/unavailable/unknown: 0/1/0, BackupStorageLocation \"[redacted]\" is unavailable: rpc error: code = Unknown desc = provided credentialsFile does not exist: stat /credentials/cloud: no such file or directory)",
	"time": "2025-05-20T15:16:36Z"
} 

Error logs plugin v1.10.0 with velero v1.14.1:

{
	"controller": "backup-storage-location",
	"level": "error",
	"logSource": "pkg/controller/backup_storage_location_controller.go:178",
	"msg": "Current BackupStorageLocations available/unavailable/unknown: 0/1/0, BackupStorageLocation \"[redacted]\" is unavailable: rpc error: code = Unknown desc = failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, canceled, context deadline exceeded)",
	"time": "2025-05-20T15:29:05Z"
}

Error logs plugin v1.11.0 with velero v1.15.2:

{
	"controller": "backup-storage-location",
	"level": "error",
	"logSource": "pkg/controller/backup_storage_location_controller.go:179",
	"msg": "Current BackupStorageLocations available/unavailable/unknown: 0/1/0, BackupStorageLocation \"[redacted]\" is unavailable: rpc error: code = Unknown desc = failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, canceled, context deadline exceeded)",
	"time": "2025-05-20T15:54:07Z"
}

Error logs plugin v1.12.0 with velero v1.16.0:

{
	"controller": "backup-storage-location",
	"level": "error",
	"logSource": "pkg/controller/backup_storage_location_controller.go:180",
	"msg": "Current BackupStorageLocations available/unavailable/unknown: 0/1/0, BackupStorageLocation \"[redacted]\" is unavailable: rpc error: code = Unknown desc = failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, canceled, context deadline exceeded)",
	"time": "2025-05-20T15:50:36Z"
}

What did you expect to happen:

That I would continue to be able to use IRSA for authentication on newer versions.

Anything else you would like to add:

Backups work as long as the aws-plugin version is not greater than 1.7.1, regardless of the velero version so I know my aws config is fine.

I've read the following other issues (and other that I've lost)

• https://github.com/vmware-tanzu/velero/issues/8240
• https://github.com/vmware-tanzu/velero/issues/7629
• https://github.com/vmware-tanzu/velero/issues/3143
• https://github.com/vmware-tanzu/velero/issues/7302
• https://github.com/vmware-tanzu/velero/issues/7945

Environment:

• Velero version (use velero version): see above
• Velero features (use velero client config get features):
• Kubernetes version (use kubectl version): 1.32
• Kubernetes installer & version: helm
• Cloud provider or hardware configuration: aws
• OS (e.g. from /etc/os-release):

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• :+1: for "I would like to see this bug fixed as soon as possible"
• :-1: for "There are more important bugs to focus on right now"