When using azure volume_snapshotter, env update from /credentials/cloud doesn't handle removed params

What steps did you take and what happened: I was testing Azure snapshot support, and making sure that things worked fine with AZURE_RESOURCE_GROUP omitted when backing up without snapshots, and that with snapshots, things worked properly with it set, and we were seeing appropriate error messages with it either unset or set to a bad value (wrong group or invalid group).

What I think I'm seeing is that when the Azure volume snapshotter updates the environment based on the current value of /credentials/cloud, it's properly handling changed env vars, but it's not handling removed env vars. If an env var is removed, the old stale value is used rather than it being removed.

What I did was first ran a backup (with snapshots) with the wrong resource group specified. I got the expected error in my velero logs that the volume was not found in that resource group. Then I removed AZURE_RESOURCE_GROUP from my cloud-credentials secret. I made sure that the update was propagated into /cloud/credentials in the velero pod, and then I attempted the snapshot backup again. Instead of seeing the expected message about the missing required AZURE_RESOURCE_GROUP env var, I saw the old error about the wrong resource group. Upon restarting velero, I got the expected error message.

So that was the case of removing an env var from the secret. I then tried adding one -- the correct one this time -- and once the secret update propagated to the velero pod, my backup succeeded. To test whether changing an env var worked, I updated the secret again, this time to an invalid group, and I got an error message about the resource group being invalid. Changing it back to the right group resulted in a backup success again.

What did you expect to happen: I expected removing an env var from the secret to result in it being removed from the environment the volume snapshotter runs in. Instead, removals don't have an effect, since it looks like velero is only updating the env vars it finds. It should probably remove any old values in the expected env var list if they've been removed from the secret.

This is the error message seen after removing AZURE_RESOURCE_GROUP and waiting for the update to propagate:

time="2019-10-30T19:49:42Z" level=error msg="Error backing up item" backup=openshift-migration/mysql-persistent-azure2-migration-hv7bs error="error getting volume info: rpc error: code = Unknown desc = compute.DisksClient#Get: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code=\"ResourceNotFound\" Message=\"The Resource 'Microsoft.Compute/disks/sseago1-ld57c-dynamic-pvc-76d7cfae-f5aa-11e9-81dd-000d3a94127f' under resource group 'Velero_Backups' was not found.\"" group=v1 logSource="pkg/backup/resource_backupper.go:264" name=mysql namespace=mysql-persistent resource=persistentvolumeclaims

This is the error message seen after restarting velero (and what would have been expected in the first place):

time="2019-10-30T20:10:00Z" level=error msg="Error getting volume snapshotter for volume snapshot location" backup=openshift-migration/mysql-persistent-azure2-migration-v8gcx error="rpc error: code = Unknown desc = unable to get all required environment variables: the following keys do not have values: AZURE_RESOURCE_GROUP" error.file="/go/src/github.com/heptio/velero/pkg/cloudprovider/azure/volume_snapshotter.go:87" error.function="github.com/heptio/velero/pkg/cloudprovider/azure.(*VolumeSnapshotter).Init" group=v1 logSource="pkg/backup/item_backupper.go:413" name=pvc-76d7cfae-f5aa-11e9-81dd-000d3a94127f namespace=mysql-persistent persistentVolume=pvc-76d7cfae-f5aa-11e9-81dd-000d3a94127f resource=persistentvolumeclaims volumeSnapshotLocation=migstorage-azure-eef7fd8b-f5b7-11e9-bb63-0610097184c8

Environment:

• Velero version (use velero version): 1.1
• Velero features (use velero client config get features):
• Kubernetes version (use kubectl version): 1.14
• Kubernetes installer & version:
• Cloud provider or hardware configuration: Azure
• OS (e.g. from /etc/os-release):