Limiting the worker jobs per time

[kkretsch]

I'm somehow stuck in configuring the queues. I currently get on occasion a bunch of jobs into the queue in the amount of up to hundreds and need to limit the worker running the jobs to let's say one job per second.

Is there any elegant way to do this or do I need to write some plugin for this?

Just adding a plain sleep into the end of the job function doesn't help, because it either consumes valuable time running in a loop or if I use some async way the jobs suddenly run in parallel.

[evantahler]

Willkommen @kkretsch!

Out of the box, Node Resque doesn't support worker rate-limiting. Since this is a queue system, the workers, just try to work one job at a time until they are done.

Here are some ideas:

• Some of the plugins included here support uniqueness in either enqueuing or working based on the arguments of the job, but that doesn't quite seem like what you are looking for.
• If there was a very special kind of job that really could only be run one-at-a-time (like interacting with something on the filesystem), you could use a unique queue name for just this job, and only have one worker work this queue (eg: don't work "*" for this one worker). But this limits you to 1 worker per server in your cluster
• Otherwise, if you have something more complex like a distributed system in which only 10 workers max can work a certain type of job at once (eg: you are hitting an external API's rate limit), then I agree that custom plugin is a great idea.
  • You can use a redis key which is incremented and decremneted at the beforePerform and afterPerform steps
  • The redis key's name could be based on the job name
  • Be sure to but a TTL on this key as well so if the workers hang up or crash, it will be cleared eventually

Let me know if you want to talk strategy more!

If you do end up writing a rate-limiting plugin, it would be great to get that checked back into node-resque so others can use it!

[kkretsch]

Thanks for the ideas.

My case is the mentioned API rate limit I would catch in some cases. Currently I just summarize the events per job worker run when seeing I have more in the queue.
I have several event subscribers, one is a mqtt client which still sends everything to some other server. The other one is a mastodon bot agent which just posts textual articles per each new entry. As those entries would get too much per time and even in total I think summarizing it is a good intermediate solution. (seen in action from now on as @bintra@m.kretschmann.social )

Next I have to try the real limiting stuff like you suggested.

[evantahler]

Good luck!

I'm going to move this to a discussion rather than an issue. Keep us posted!