Coding Azure 18: Creating and Securing Azure Storage Queues

Setting up an Azure Storage queue to support an asynchronous application is actually pretty straightforward, except for the four different security options that you’ll have to choose between.

In my previous post, I discussed the benefits of asynchronous applications and the tools that Azure provides for creating them. In this post, I’m going walk through configuring the first of those tools (also the cheapest and easiest to use): the Azure Storage queue.

If you’re looking at the length of this post and thinking “It can’t be that easy if the post is this long” … well, I’m also going to show how to secure your application’s frontend to a Storage queue. There are four different ways you can secure access to your queue from your application’s frontend, which is why this post looks lengthy.

But don’t panic! Of these methods:

• One is your best choice for a server-side frontend (Managed Identities)
• One is your only choice for a client-side frontend (App Registrations)
• One should only be used during development (connection strings)
• The last one is only used to implement a specific design pattern (Shared Access Signatures and Access Policies with the Valet Key pattern)

Depending on whether you’re creating a client-side or server-side frontend, you’ll only be interested in one authorization system in production (and, perhaps, connection strings during the early stages of your development process/“proof of concept” coding).

Creating a Queue’s Storage Account

Storage queues live inside of storage accounts, so you begin by creating a “queue-dedicated” storage account and then add your queue to it. And, regardless of what method you use to secure your frontend to your queue, this part is the same for every queue, except for one step.

To create a “queue-dedicated” storage account, in the Azure Portal, surf to Storage Accounts and, from the left end of the menu across the top of the page, select + Create to start the wizard that will create your storage account.

Since my application will be working with products in my warehouse management case study, I called my storage account warehousemgmtproducts. Because your storage account name will, eventually, be part of a URL, your storage account name must be “URL friendly” in the strictest sense: Your queue name, in addition to avoiding spaces and special characters, also can’t contain upper-case letters.

Because you’re creating a storage account specifically to be used for Storage queues, the settings you should make in the wizard’s pages are:

• Initial/basics page: Specify the Primary service as Other (this means you shouldn’t use the storage account for blob containers or file shares).
• Advanced page: In the long run, you should turn off Enable storage account key access (you can do that in your Storage queue’s Settings | Configuration menu choice, using the “Allow storage account key access” option) but should probably leave it on for development.
• Networking: To restrict access to your queue, you can select the Secure by perimeter option and then, after you’ve created your storage account, create a Network Security Perimeter in the Azure Portal.
• Encryption keys: The messages in your queue will be encrypted but you can choose to use your own encryption keys (I was comfortable with using the keys that Azure provides).

Turning off storage account keys disables using connection strings, Shared Access Signatures or Access Policies (more on those later). However, turning off account keys also limits what maintenance operations you can perform on the queue from the Azure Portal. With this option turned off, you can’t clear the queue from the portal, for example. You should leave this option on in development.

Defining a Network Security Perimeter

If you picked the Network Security Perimeter option, then you’ll need to create a Network Security Perimeter (NSP) in the Azure Portal. You can then use the NSP to limit access to the queue to requests from the outbound ports of the App Service that your frontend is running in. These addresses need to be listed as CIDR ranges. (I used the ranges 20.116.0.0/16 and 4.174.184.0/24 based on the outbound ports listed on my App Service’s Settings | Networking page.)

In your Network Security Perimeter, you’ll also need to add the public IP address for the computer you’re using to configure your queue. You can get that by:

1. Surfing to https://whatismyipaddress.com/ from your computer
2. Copying the TCP/IP address displayed on that page
3. Tacking “/32” at the end of the address to convert it to a CIDR range
4. Adding it to your network perimeter’s list of addresses.

Creating Your Queue

With your Storage queue created, you can now add your queue to it. To do that:

1. Go to your queue’s Overview page.
2. From the menu down the left side, select Data Storage | Queues to open a list of queues in your storage account on the right.
3. From that list, from the menu across the top of the page select + Queue and give your queue a name. (I used updateproductinventory to reflect the kind of messages I intend to add to this queue.)
4. Click the OK button to create your queue.

By default, as the queue’s owner, you’ll have the permissions to manage the queue’s configuration (including assigning roles), but you won’t have permission either to read or to add messages to the queue. In development you’ll want, at the very least, to be able to view the messages your code is adding to the queue, so you’ll need to give yourself some additional permissions.

To enable working with messages when managing your queue from the Azure Portal, use the queue’s role assignment wizard to give yourself the Storage Queue Data Contributor role for your queue:

1. On your queue’s overview page, select Access Control (IAM) to open the IAM page on the right.
2. On the IAM page, click the Add role assignment button.
3. In the Add role assignment page, from the list of roles, select Storage Queue Data Contributor and click the Next button.
4. On the following page, click on the + Select Members link to open a panel on the right.
5. In the panel, select your name, and click the Select button to close the panel.
6. Back on role assignment page, click the Review + assign button to skip to the end of the wizard.
7. Click on the Review + assign button at the end of the wizard to save your changes and give yourself the ability to view messages on the queue.

In production, to support troubleshooting, you can consider assigning that role (or other more limited ones) to specific users to allow them to browse the queue (e.g., the Storage Queue Data Reader role). Remember, though, that you’re also potentially giving those users the ability to read messages in the queue.

Securing the Storage Queue to Your Frontend

This is the section where you have four security options to pick from. Pick the security option based on your frontend’s platform (server-side or client-side) but also consider looking at connection strings as a quick way to get started. You probably won’t ever need to implement the Valet Key pattern but, if you do, I’ve covered it at the end of this section.

Server-side Frontends and Managed Identities

For a server-side frontend, your best choice for giving your frontend permission to add messages to your queue is to assign a Managed Identity to the App Service that your server-side frontend is running inside of.

Using Managed Identities does mean you’re giving your application permission to perform actions that your user may not have permission to do. You are, effectively, counting on your application code to prevent the user from doing something foolish or malicious. I’m ok with that, but you may not be.

Once you’ve assigned the identity to your App Service, return to your queue’s Add role assignment page and give your Managed Identity the Storage Queue Data Message Sender role, using the queue’s role assignment wizard:

1. On your queue’s overview page, select Access Control (IAM). And, on the IAM page, click the Add role assignment button.
2. In the Add role assignment page, from the list of roles, select Storage Queue Data Message Sender and click the Next button.
3. On the following page, select the Managed Identities option.
4. Click on the + Select Members link to open a panel on the right.
5. In the panel, from the Managed identity dropdown list, select User-assigned managed identity.
6. From the list of Managed Identities, select the identity your created, and click the Select button to close the panel.
7. Back on role assignment page, click the Review + assign button to skip to the end of the wizard.
8. Click on the Review + assign button to finish the wizard, save your changes and give your Identity the ability to add messages to the queue.

Client-side Frontends and Application Registrations

For a client-side frontend, your frontend will be claiming an App Registration. You can’t assign Storage queue permissions to that App Registration but you can use the App Registration to pass the user’s permissions onto your application. That means, with an App Registration, the application can only do what the user is allowed to do.

To share your application’s permissions with your application, you need to give your frontend’s App Registration permission to pass on the user’s Storage queue roles:

1. In the Azure Portal, surf to the App Registration that your frontend is claiming.
2. From the menu down the App Registration’s left side, select the Manage | API permissions choice.
3. In the API permissions page, click the + Add permission link to open a panel on the right.
4. In the Microsoft APIs tab, find the Azure Storage tile and select it.
5. Check the user_impersonation option.
6. Click the Add permissions button to enable the application to use the user’s permissions.

As the absence of any restrictions show, the App Registration will now pass on any permissions the user has for any storage account. Your primary control, therefore, over which roles the App Registration gives to which storage accounts is by controlling what roles you assign to your user and to which queues.

For example, to give your application’s users (or security group) the ability to add messages to your Storage queue, you would need to give the user the Storage Queue Data Message Sender role, like this:

1. In the Azure Portal, surf to your Storage queue.
2. From the menu down the left side, select Access Control (IAM).
3. In the Access Control (IAM) page displayed on the right, click the Add role assignment button.
4. In the search text box, search for the Storage Queue Data Sender role and select it.
5. Click the Next button at the bottom of the page.
6. In the next page, click on the + Select Members link to open a panel on the right.
7. In that panel, search for your user or their security group by name.
8. Click on the user or security group to select it.
9. Click the Select button to close the panel.
10. Back on the wizard page, click the Review + assign button at the bottom of the page to go to the end of the wizard.
11. On the wizard’s last page, click on the Review + assign button to save your assignment.

The first time the user runs your frontend, the user will be prompted to grant those permissions (which, you assume, they will).

Developing Using Connection Strings

You can also give your application access to your queue using a connection string that holds an Access Key for your storage account. A connection string gives your application unfettered access to the queue, regardless of what permissions the user has (or doesn’t have). As with a Managed Identity, you are giving the application permission to do things the user may not be allowed to do.

There are two advantages of using a connection string: it’s simple and it works from anywhere, including running your application on your desktop from Visual Studio or Visual Studio Code. Using a connection string can speed up working out your frontend’s logic.

You can see the connection strings for your storage account by surfing to your storage account in the Azure Portal and, from the menu down the left side, selecting Security + networking | Access keys. Your account’s two current connection strings, which are automatically generated for your storage account, are displayed on the right.

If you’re using this option in a production-bound application (and you shouldn’t), as the Access Key page’s structure implies, you should assume that these keys will eventually get out “in to the wild” (to quote Benjamin Franklin, “Three may keep a secret if two of them are dead”). You should be prepared to cycle those keys regularly. A server-side frontend should retrieve the connection string from the Azure Key Vault. A production client-side frontend should call a Web Service that returns the connection string (again, retrieved from the Key Vault).

Which leads us to the Valet Key pattern.

First, a side note: When you manage your queue from the Azure Portal, you are, effectively, connecting to the portal using a connection string. That’s why, once you turn off shared keys, you aren’t able clear the queue from the Azure Portal.

Implementing the Valet Key Pattern with Shared Access Signature/Access Policies

In addition to accessing your queue using a connection string, your frontend can also create a Shared Access Signature (SAS). An SAS is, essentially, the URL for a storage account with access restrictions added on at the end (e.g., an expiry date, whether your application can read or add messages, etc.). This enables an SAS to act as a key in the Valet Key pattern. Access Policies can be used as part of creating that key.

Given an SAS, a client program can access the storage account within the time and permission limits specified by the SAS. As with a Managed Identity or a connection string, an SAS allows the application to do things with your queue that the user may not have permission to do. The SAS also includes a hash key that is used to detect tampering with the SAS’s restrictions.

Generating an SAS

The significant problem with an SAS is that, like a connection string, a) you should assume that it will, eventually, get out into the wild (Benjamin Franklin, again) and b) there is no way to revoke or modify an SAS. Because an SAS can’t be revoked or modified, using a SAS to, for example, enable end user access to a storage account’s blob container is risky.

In the Valet Key pattern, that risk is reduced. If an SAS, somehow, got into the wild, it will probably have expired. Further, unlike an SAS generated at design time in the Azure Portal, an SAS generated at run time can be restricted to giving access to a specific queue.

I’ll include the code to generate an SAS in my next post, but if you want to get some insight into what an SAS looks like, you can create one in the Azure Portal. To do that:

1. In the Azure Portal, surf to your storage account (not your queue).
2. From the menu down the left side, select Security + networking | Shared access signature to open a form on the right.
3. In the form:

• Under Allowed services, check only the Queue option.
• Under Allowed resource types, check only the Object option.
• Under Allowed permissions, check only the Create permission.
• Set start and expiry dates.
• If you want to restrict requests to applications running at specific IP addresses, enter them in the Allowed IP addresses textbox.

4. Click the Generate SAS and connection string button at the bottom of the form.

You can see the generated SAS for your queue (which begins with the URL for your queue) in the Queue service SAS URL textbox.

Configuring an Access Policy

Like an SAS, you can create an Access Policy in the Azure Portal (and, unlike an SAS, a design-time Access Policy restricts access to a specific queue). An Access Policy can be both revoked or modified if you discover that the policy is being misused (setting the expiry date for the policy to sometime in the past or deleting the policy, for example, renders the policy unusable).

At run-time, your application can use policies assigned to a queue at design time as part of the inputs to generating an SAS. To create an Access Policy to be used in that process:

1. In your storage account, surf to the list of your queues.
2. Find the queue you want to create the Access Policy for and click on the three vertical dots at the end of its line.
3. From the popup menu that appears, select Access Policy to open the Access Policy panel.
4. In the panel, click on the + Add policy link to open the Add policy form.
5. In the form, give your policy:

• A name
• A start date
• An expiry date
• Permission to perform specified operations: Add, Read, Update, Process

6. Click the OK button at the bottom of the page to save your policy to return to the Access policy panel.
7. In the menu across the top of the Access policy list, click the Save icon to save your policy changes to the queue.

Next Steps

The obvious next step is to have your frontend add a message to your queue (or generate an SAS if you’re implementing the Valet Key pattern). That’s my next post.