Questions Regarding Statistical Details and Task Availability

[Parkash007-max]

Hi authors,

Thank you for the great work and for releasing these benchmark. I have a few questions regarding the statistics and task availability mentioned in your paper:

(1) Regarding Figure 14: Could you clarify what the error bars represent? Based on the paper description, it appears you perform 10 runs per task and show the pass rate across these runs (ranging from 0% to 100%). However, it is unclear what the error bars specifically indicate. Were the experiments also repeated with different random seeds?

(2) Regarding the number of SRE tasks: In the main text, you mention there are 42 scenarios for the SRE tasks. However, Figure 15 shows only 21 tasks. Are the remaining 21 tasks planned for a future release? Additionally, in the provided codebase link (https://github.com/IBM/ITBench-Scenarios), I only found two tasks (OpenTelemetry's Astronomy Shop Demo and DeathStarBench's Hotel Reservation). Does this mean that approximately 40 additional tasks for SRE are still pending release?

Thank you very much for your time and attention. I am looking forward to your response.

[rohanarora]

Hello @Parkash007-max,

Apologies for the delay in our response.

(1) The error bars in the figure represent the macro-averaged standard error across 10 runs per task. Although we did not vary random seeds, the system is non-deterministic even under fixed seed and temperature settings. This is due to variability in the input derived from live observability data, such as timestamps, which introduces natural variance across runs. The standard error reflects this inherent variability in the model's output behavior.

(2) In the main text, the combined average of the agent's performance on the same 21 scenarios with and without access to traces has been reported. Hence the number 42 (21x2). In the code-base here you should find 4 such scenarios: 1, 3, 23, 26, 27.
The SRE scenarios (the ones which you are referring as tasks) - currently - are samples from the team to show how to craft a scenario so that others may contribute theirs by way of a pull request. That being said, for the SRE use-case, the plan is to open source most (if not all) of the fault mechanisms which are leveraged to create a scenario.

We like to think about a scenario as a recipe with the following ingredients:
a. Environment (open sourced)
b. Fault mechanism itself or any other applicable mechanisms to trigger the incident or a possible incident (mostly open sourced). If there are any fault mechanisms you would like to see as a part of the ITBench, please create an issue here or a pull request.
c. When and where the fault is injected (open sourced for limited scenarios until the scenario generator-runner being built reaches a certain level of maturity)

Please feel free to reach out if you have any additional questions.