AttachVolume.Attach failed for volume "xyz" after node image update

[GeertvanHorrik]

What happened:

Azure AKS applied an auto-update for the node pool. Then the pods are not coming up with this message:

AttachVolume.Attach failed for volume "xyz" : rpc error: code = NotFound desc = failed to get azure instance id for node "aks-default-12345678-vmss000002" (instance not found)

I tried:

1. Restarting the csi-azuredisk-node pod,
2. Deleting PVC and PV, then redeploy the app

PV and PVC are successfully bound after redeploying the app, but the pods cannot attach to the volume.

What you expected to happen:

After an update, the disks should be able to be found and be attachable.

How to reproduce it:

This has happened a few times now. Sometimes we can delete the PVC and PV and redeploy our app, but it's a burden to do this whenever Azure automatically updates the node pool images.

Anything else we need to know?:

Using AKS with managed disks.

Environment:

• CSI Driver version: v1.29.4-1
• Kubernetes version (use kubectl version): 1.28.9

[GeertvanHorrik]

A cluster reboot (stop + start) solves the issue:

Logs:

AttachVolume.Attach failed for volume "xyz" : CSINode aks-default-12345678-vmss000008 does not contain driver disk.csi.azure.com

Then after a while (when the csi pods are up):

AttachVolume.Attach succeeded for volume "xyz"

But restarting an AKS cluster is tricky (takes a long time, no guarantee that there will be resources available to start it again, etc). What would be a better way to solve this without a full cluster restart?

[andyzhangx]

aks-default-12345678-vmss000002 is not up after node pool auto upgrade, that's the reason why disk attach failed in the beginning, does this issue always happen or mitigated automatically after a while? @GeertvanHorrik

[GeertvanHorrik]

aks-default-12345678-vmss000002 is not up after node pool auto upgrade, that's the reason why disk attach failed in the beginning

We waited for more than 8 hours, but our cluster was down all the time (so that's not really an option). We tried scaling with nodes (adding, removing, manually draining, etc). Only when we stopped / started the cluster, it was all good.