Skip to content

Attempting to retrieve FetchedJobs crashes due to duplicates #370

@Jameak

Description

@Jameak

We're getting an exception when attempting to retrieve the list of fetched jobs via the Monitoring API.

This line of code:
JobStorage.Current.GetMonitoringApi().FetchedJobs("default", 0, int.MaxValue)

Causes the following exception:

System.ArgumentException: An item with the same key has already been added. Key: 24014
   at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
   at System.Collections.Generic.Dictionary`2.Add(TKey key, TValue value)
   at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](List`1 source, Func`2 keySelector, Func`2 elementSelector, IEqualityComparer`1 comparer)
   at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector)
   at Hangfire.PostgreSql.PostgreSqlMonitoringApi.FetchedJobs(IEnumerable`1 jobIds)
   at Hangfire.PostgreSql.PostgreSqlMonitoringApi.FetchedJobs(String queue, Int32 from, Int32 perPage)
   at Somewhere.In.My.Code

The issue seems to occur in the following situation. This is purely based on logs from our environments, I have not had the time to attempt to reproduce it locally:

  1. A hangfire server is currently processing jobs.
  2. The hangfire server disappears from the active server list (such as when the server gets restarted or the application crashes)
  3. The other hangfire servers (or the 'new' server if it was restarted) notices that the jobs from step 1 might have been aborted, because they're marked as being processed by a server that is not in the active server list.

If you attempt to get the list of fetched jobs after step 3 has occurred, it seems to (sometimes?) crash. This eventually resolves itself, presumably when the job gets successfully processed.

Versions:
Hangfire.PostgreSql: 1.20.8
Hangfire: 1.8.7
.NET version 8.0.202

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions