AzureWebJobsStorage secured by Network Security Perimeter - Function unable to start

[pregress]

Description

When you associate the storage account of an Azure Function with a network security perimeter (nsp) and allow the subscription of the function inbound access to the NSP and you set the mode to enforced your function is unable to start

Steps to reproduce

1. Create an azure storage account for your function storage
2. Create an azure function, linux, consumption plan. (Runtime version: 4.1042.100.2)
3. Configure the storage account to the function
4. Validate the function works.
5. Create a new Network Security Perimeter
6. Add a profile to the NSP
7. Add an inbound rule for the subscription of the function
8. Associcate the storage account with the NSP, set the mode to Enforced
9. Restart the function
10. Function fails to start (The service is unavailable. when browsing to the function)
11. Set the mode to Learning
12. Restart the function
13. The function works again

All resources, function, nsp, storage are in the same resource group, in the same subscription in the same region

Doing the same for a key vault that's being accessed from the same function works.

Since Azure Storage is general available this should work?
https://learn.microsoft.com/en-us/azure/private-link/network-security-perimeter-concepts#scale-limitations

[GABRIELNGBTUC]

I'm pretty sure function apps on the consumption tier do not support identity-based authentication to the Azure files service which would prevent the website from starting https://learn.microsoft.com/en-us/azure/azure-functions/functions-app-settings#website_contentazurefileconnectionstring

You could maybe get your function to work by using "WEBSITE_SKIP_CONTENTSHARE_VALIDATION", but there may be some issues with scaling when using this app setting.

Overall, I would just use flex consumption unless the slightly higher pricing is an issue with how much more features it has and the freedom to no longer deal with consumption apps quirks.

EDIT: With Linux consumption apps being deprecated soon, it seems like the flex consumption app is definitely the way to go https://azure.microsoft.com/en-us/updates/?id=499451

[pregress]

@GABRIELNGBTUC Nowhere In my issue did I mention we use managed identity for storage access.
The issue is reproducible with access keys.

We will postpone the migration to flex consumption as long as possible or move to lambda.
Since we do this for functions that have less the 1M monthly executions, the price difference is a no-go.

Price diff:

Executions	Flex Free Grant	Consumption Free Grant	Flex Cost (€)	Consumption Cost (€)	Difference (€)
1,000,000	250,000	1,000,000	0.25725	0.00	+0.25725
2,000,000	250,000	1,000,000	0.60025	0.172	+0.42825

[GABRIELNGBTUC]

@GABRIELNGBTUC Nowhere In my issue did I mention we use managed identity for storage access. The issue is reproducible with access keys.

We will postpone the migration to flex consumption as long as possible or move to lambda. Since we do this for functions that have less the 1M monthly executions, the price difference is a no-go.

Price diff:

Executions Flex Free Grant Consumption Free Grant Flex Cost (€) Consumption Cost (€) Difference (€)
1,000,000 250,000 1,000,000 0.25725 0.00 +0.25725
2,000,000 250,000 1,000,000 0.60025 0.172 +0.42825

The way subscription rules in NSP work is that they will only allow requests that use EntraID authentication from a valid identity contained inside a resource group part of the subscription of the rule.

I assumed you used EntraID authentication because of this. But if you are using access key, it is normal that your function is not starting since it will not send an EntraID token while communicating with the storage for its infrastructure operation.

If you were to configure the following two appsettings AzureWebJobsStorage__accountName and AzureWebJobsStorage__credential to managedIdentity, then disable the Azure files startup check with WEBSITE_SKIP_CONTENTSHARE_VALIDATION, it is possible that your Azure function would work while connecting to a NSP protected storage account. But again, since there is no support for EntraID authentication with Azure files in this scenario, it is possible that your function could behave in unexpected ways.

[pregress]

Official docs please?
Because none of these say anything about identity/entra:

• https://learn.microsoft.com/en-us/azure/private-link/network-security-perimeter-concepts
• https://learn.microsoft.com/en-us/azure/storage/common/storage-network-security-perimeter

Also strange that the same function can access a key vault secured by the same NSP then.

[GABRIELNGBTUC]

Official docs please? Because none of these say anything about identity/entra:

• https://learn.microsoft.com/en-us/azure/private-link/network-security-perimeter-concepts
• https://learn.microsoft.com/en-us/azure/storage/common/storage-network-security-perimeter

Also strange that the same function can access a key vault secured by the same NSP then.

There is no documentation on how to configure or how the subscription-based rules work as far as I know. The only mention about anything identity related in their documentation is here https://learn.microsoft.com/en-us/azure/private-link/network-security-perimeter-transition#transition-to-enforced-mode-for-existing-resources

Perimeter access: Perimeter access refers to inbound or outbound requests between the resources part of the same network security perimeter. To prevent data infiltration and exfiltration, such perimeter traffic will never cross perimeter boundaries unless explicitly approved as public traffic at both source and destination in enforced mode. Managed identity needs to be assigned on resources for perimeter access

But this refers to the default behaviour of resources in the same security perimeter being able to connect to each other. However, you cannot add an Azure function inside a NSP so this is not really relevant for this issue (well partially).

What I am telling you about the subscription rules is the result of my own troubleshooting trying to access resources inside the NSP from resources outside of the NSP (VMs, Azure functions / WebApps).

I was not able to do so initially despite managed identities being configured on all my resources until I started using the managed identity authentication on the VMs and Azure Functions (through AzureWebJobsStorage__accountName and the other app setting).

Now that I remember, there is another change that is required for storage accounts to work with subscription rules and that was quite annoying to troubleshoot. Remember the quote above from the documentation about needing the managed identity to be assigned for perimeter access?

Well, it turns out that storage accounts now can be assigned managed identities through ARM (or APIs) only. And that is a requirement for the subscription rules to work even when AzureWebJobsStorage__accountName and the managed identity credential is used.

In general, the NSP GA is half baked (why can't we add Az functions and webapps to them?) and the documentation is severely lacking. But after doing all the troubleshooting necessary, I can definitely say that it is working, just in very obscure ways.