[QUERY]: Hitting a 6 minute timeout and trying to understand what it is

[wbuck]

Library name and version

Azure.Messaging.ServiceBus 7.17.5

Query/Question

Our application is written in C# and we're using the ServiceBusSessionProcessor to receive messages from a Queue. Note that we're also using sessions here.

We ran in to an issue the other day in production when the processing of one of our messages was taking a longer time then usual. The processing of the message was taking well over 6 minutes. We don't "complete" the message until processing of the message has completed.

While processing the message I could see (via logs) that after exactly 6 minutes, Service Bus would redeliver the same message again. This duplicate processing of the same message occurred 10 times. We would receive a message and after exactly 6 minutes of processing, Service Bus would redeliver the same message.

Typically message processing does not take this long but we were having a separate issue with a downstream service which was causing this extended processing time.

I also want to mention that I'm manually renewing the session lock using a periodic timer. I use the SessionLockedUntil property of the ProcessSessionMessageEventArgs in order to set the period of the periodic timer. The value I use as the period is SessionLockedUntil - 10seconds. If the expression evaluates to a TimeSpan that's less then zero I abandon the message and call ReleaseSession. When the periodic timer executes I call RenewSessiongLockAsync. That being said nothing in the logs indicates that this occurred, just adding some additional context.

I'm trying to understand what 6 minute timeout I'm hitting here. I see in the logs that the session lock is being renewed ~ every LockDuration - 10 seconds but I'm still hitting this 6 minute timeout. I'm also not seeing any exceptions in the logs.

What timeout is this? What can I check in either the Azure portal or the SDK to determine what timeout this is?
I do plan on swapping the manual session lock renewal for automatic session lock renewal via MaxAutoLockRenewalDuration but I'm unsure if session lock expiry is the issue here.

I should also mention that all durations are set to their default values (They've never been altered).

Environment

No response

[jsquire]

Hi @wbuck. Thanks for reaching out and we regret that you're experiencing difficulties. There is no 6-minute timeout for sessions that we're aware of, from the client perspective. Normally, a scenario like this comes down to session expiration, as session messages themselves are not locked. Since you mention that you're manually renewing the session lock every 10 seconds, if you're seeing those calls complete successfully, it doesn't seem likely. The other client-side possibilities would be that someone is stopping the processor and you're honoring the cancellation token, or a general network error.

Unfortunately, there's not a lot of client context available to work with in the description. It would be helpful to see the error itself, error message, and stack trace. However, that said, if it's a standard timeout exception that you're seeing, there's not much assistance that we can offer from the Azure SDK perspective. Unfortunately, Azure does not offer service support from our repository nor do service teams monitor issues here.

Since the behavior that you're seeing appears to be related to either the Service Bus service or your host's network environment, your best path forward for would be to open an Azure support request or inquire on the Microsoft Q&A site. This will ensure that the team best able to assist has visibility.

If you'd like to share the error details with stack trace and/or a 5-minute slice of Azure SDK logs around the time of the error, we'd be happy to offer what thoughts we can.

[github-actions]

Hi @wbuck. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[wbuck]

@jsquire There is no exception being thrown in this situation so there is no stack trace/exception to show. What I'm seeing is duplicate message processing if we take longer then 6 minutes to handle a message. I'll try removing the manual session lock renewal and switch over to automatic session lock renewal via MaxAutoLockRenewalDuration.