I’ve been doing a lot of work lately with Kubernetes and Azure Kubernetes Service in particular. For this, I’m using the azureFile storage provider to support providing storage folders for my pods.
For one specific case, I’m using dynamic provisioning of persistent volumes, as described in the documentation. This has been working great. Almost.
Today I noticed something odd. Some of the Persistent Volumes created were stuck on a failed state during deletion:
The error message was this:
Failed to create deleter for volume "pvc-85ba62d5-902e-11e8-9771-2a03d0b14ac3": Couldn't get secret 6d07576ce8d89bd641b5/azure-storage-account-fd9063993902011e888c32e-secret
This actually made sense. The way dynamic persistent volumes work is this:
- When the Persistent Volume Claim is created, Kubernetes creates a new Secret object containing the
credentials for the Azure Storage Account. This secret is named
azure-storage-account-<namespace>-secret. - When the Persistent Volume Claim is deleted, the deleter for the volume will use these credentials to delete the Azure File Share that was dynamically created, and then delete the corresponding secret.
The second part is what was biting me.
I currently deploy the applications using Helm. I deploy each one into a separate Kubernetes
Namespace. This is fully supported by Helm, but has one issue: When you delete the application using
helm delete, the original namespace created remains behind. I wanted to clean these up, and so
was aggressively deleting them using kubectl delete namespace.
The problem here is that the secret that gets automatically created by the azureFiles provider when the Persistent Volume Claim is created, goes into the same namespace as the PVC. That meant that when I was aggressively removing the namespace, I was also deleting the secret before the volume deleter could get to it.
This isn’t very clearly documented (or at least I found no mention of this), so it might be something to watch out for.
