[help] How do I pass on options() when using slurm?

[Shians]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

I am trying to access options inside of a job submitted to slurm. This works find when running locally, but using the slurm controller appears to not pass on the options() in the same way.

library(targets)

# Set target options:
tar_option_set(
  packages = c("tibble"),
  controller = crew.cluster::crew_controller_slurm(
    workers = 4L,
    options_cluster = crew.cluster::crew_options_slurm(
      memory_gigabytes_required = 8L,
      cpus_per_task = 2L
    )
  )
)

options("foo" = "bar")

# Replace the target list below with your own:
list(
  tar_target(
    name = data,
    command = getOption("foo", "")
  )
)

> tar_read("data")
[1] ""

Is this the intended behavior?

[wlandau]

Neither targets nor crew passes options or environment variables to parallel workers. I suggest setting any options you need inside the target that needs them.

[Shians]

I think I'd prefer that .Rprofile NOT be involved in the definition of my workflow. It's not generally a file that people look into and often excluded from git repos.

[wlandau]

What is the reason for this? It feels a bit jarring that serial and worker based workflow runs could produce inconsistent results based on what I conceptually think of as global options.

R isn't designed to do this cleanly, and attempting to port over everything would create too many strange edge cases. Best to transparently demarcate what targets does vs what it doesn't do, so the limitations are understood and surprises are kept to a minimum. targets does track functions, global objects, and the relationships among them. It doesn't track C/C++ code, packages, options, or environment variables.

[Shians]

Just for context, the intent was to set a global output directory with wrappers around any table writing or plotting function such that I can easily change it. Sticking the value in options was the obvious solution based on serial R, but I think maybe in this can the same can be achieved using a global variable.

get_var <- function(varname, default, env = .GlobalEnv) {
    if (exists(varname, envir = env, inherits = FALSE)) {
        get(varname, envir = env)
    } else {
        default
    }
}

get_var(".output_dir", "output")

.output_dir <- "custom_output"

get_var(".output_dir", "output")

write_plot <- function(ggplot, fname, dir = get_var(".output_dir", "output")) {
    cat("Writing to:", fs::path(dir, fname))
}

write_plot(NULL, "plot.png")

[Aariq]

Why not just make a target for the output dir that downstream targets use?
e.g.

list(
  tar_target(output_dir, "path/to/output"),
  tar_target(plot, write_plot(NULL, fname = "plot.png", dir = output_dir))
)

There's also tarchetypes::tar_hook_before() that might be helpful for prepending targets with a call to options() if you find that necessary. https://docs.ropensci.org/tarchetypes/reference/tar_hook_before.html

[Shians]

I want the path declaration (and other global configs) to be in a separate code block than the targets, ideally after library calls and before tar_source(). Conceptually a user should not have to touch any of the targets themselves. I'll see if I can understand how tar_hook_before works and whether it has meaningful advantages over my proposed approach.