Azure Function Queue Trigger Retry

Azure functions with Storage Queue trigger has a built in retry logic based on the dequeue count of the message in the queue. You can configure the frequency of the retry in the bindings either through code or host.json.

After the max number of retry, the message will be placed in the <queue-name>-poison queue.

{
    "version": "2.0",
    "extensions": {
        "queues": {
            "maxPollingInterval": "00:00:02",
            "visibilityTimeout" : "00:00:30",
            "batchSize": 1,
            "maxDequeueCount": 5,
            "newBatchThreshold": 8
        }
    }
}

visibilityTimeout property dictates the interval between each retry, maxDequeueCount dictates the number of retries on the function. If there is an exception each time a function executes the message is automatically put back in the queue by the runtime and retried after 30s. This would continue for about 5 times.

Setting the batchSize setting to 1 makes the function fetch only one message from the queue at a time, and
newBatchThreshold defines our function to check the queue for new messages only when the number of processed messages drops down to this value (0 = no messages are being processed any longer). Together, these settings ensure that a single Azure Function instance only processes one queue message at a time.

At other times, maxPollingInterval is the maximum amount of time we allow our function to “procrastinate” before it starts processing new queue messages. As I mentioned before, I’m not in a rush here, so I can allow the function to mentally prepare for 30 seconds. It can start processing the messages sooner, too, though, if it feels like it.

The last two settings are related to errors and reprocessing. visibilityTimeout essentially defines the delay between retries, and maxDequeueCount sets the maximum number of times the function tries to process the queue message before moving it to the poison queue (including the first run). As I mentioned before, if that happens, you can move the messages from the poison queue to the processing queue by using the Azure Storage Explorer desktop application.

Retry policy options

The following options are available for defining a retry policy.

Max Retry Count is the maximum number of times an execution is retried before eventual failure. A value of -1 means to retry indefinitely. The current retry count is stored in memory of the instance. It’s possible that an instance has a failure between retry attempts. When an instance fails during a retry policy, the retry count is lost. When there are instance failures, triggers like Event Hubs, Azure Cosmos DB, and Queue storage are able to resume processing and retry the batch on a new instance, with the retry count reset to zero. Other triggers, like HTTP and timer, don’t resume on a new instance. This means that the max retry count is a best effort, and in some rare cases an execution could be retried more than the maximum, or for triggers like HTTP and timer be retried less than the maximum.

Retry Strategy controls how retries behave. The following are two supported retry options:

Option	Description
fixedDelay	A specified amount of time is allowed to elapse between each retry,
exponentialBackoff	The first retry waits for the minimum delay. On subsequent retries, time is added exponentially to the initial duration for each retry, until the maximum delay is reached. Exponential back-off adds some small randomization to delays to stagger retries in high-throughput scenarios.

Fixed delay retry

[FunctionName("EventHubTrigger")]
[FixedDelayRetry(5, "00:00:10")]
public static async Task Run([EventHubTrigger("myHub", Connection = "EventHubConnection")] EventData[] events, ILogger log)
{
// ...
}

Retries require NuGet package Microsoft.Azure.WebJobs >= 3.0.23

Exponential backoff retry

[FunctionName("EventHubTrigger")]
[ExponentialBackoffRetry(5, "00:00:04", "00:15:00")]
public static async Task Run([EventHubTrigger("myHub", Connection = "EventHubConnection")] EventData[] events, ILogger log)
{
// ...
}

Sources:

https://www.cloudmanav.com/azure/exponential-backoff-retry-functions-storage-trigger/#

https://docs.microsoft.com/en-us/azure/azure-functions/functions-bindings-error-pages?tabs=csharp

Forcing an Azure Function to process Queue Storage messages one at a time