[FEATURE]: Design a way to limit resource consumption for tasks #18
Description
We need a mechanism to constraint resources like CPU & memory for our tasks. The preferred way to achieve this is by leveraging cgroups.
Take a look at the https://github.com/containerd/cgroups as an off-the-shelf solution. But try to implement own logic to gain more experience in that area.
Cover this with tests 😎 For instance, run a worker in a Docker container with 1 CPU & 1 GB RAM and run several CPU/memory intensive tasks with 1/(N+1) CPU & 1/(N+1) GB RAM (N+1 is used to reserve some resources for a worker itself). To verify, we can:
- CPU constraints: Make sure that the tasks finish at almost the same time and average CPU utilization doesn't increase 1/(N+1) a lot.
- Memory constraints: Make sure that the tasks are not OOMKilled and average memory utilization is in the allowed ranges.