[help] New default retrieval="auto" seems to break SLURM-enabled pipeline

[koefoeden]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

After updating to the newest version of targets, I encountered issues with workers/slurm-jobs seemingly failing/being terminated without any error messages. I have narrowed it down to the new default argument of retrieval="auto", since retrieval="worker" seems to fix the issue.

I am running the main process on the head-node of a SLURM-enabled cluster, and the issues seem to appear in the context of a target dynamically branching over a tarchetypes::tar_files() target. I have paraphrased the relevant part of the pipeline below:

tar_target(dirs, parent_dir %>% list.dirs(....), deployment = "main", cue = tar_cue("always")),
tar_file(new_files, write_file(dirs), pattern = map(dirs), resources = group2),
tarchetypes::tar_files(selected_new_files, unlist(new_files)[select_indices], resources=group1)
tar_file(split_files, split_file(selected_new_files, ...), pattern=map(selected_new_files), resources=group2) # failing target

I have all the following newest packages:
targets = 1.11.1
crew = 1.1.0
crew.cluster = 0.3.4
mirai = 2.2.0
tarchetypes = 0.13.0

It is not critical since the retrieval="worker" argument seems to fix it, but figured you might be interested in knowing nonetheless.

[wlandau]

Would you help me understand how to reproduce the error you see? What happens if you set retrieval = "main" everywhere? Does it still fail? retrieval = "auto" just resolves to "main" or "worker", depending on how you use dynamic branching.

Looking at your code, I think a reprex could look something like this. Unfortunately (fortunately?) it ran fine on my company's cluster.

library(targets)
library(crew.cluster)

tar_option_set(
  controller = crew_controller_sge(
    options_cluster = crew_options_sge(
      script_lines = "module load R/4.3.2"
    )
  )
)

list(
  tar_target(
    path,
    c("a.txt", "b.txt"),
    retrieval = "main"
  ),
  tar_target( # assumes file a.txt and b.txt already exist, with arbitrary text
    input,
    path,
    pattern = map(path),
    format = "file",
    retrieval = "main"
  ),
  tar_target(
    output,
    {
      rds <- fs::path_ext_set(input, "rds")
      lines <- readLines(input)
      saveRDS(lines, rds)
      rds
    },
    pattern = map(input),
    format = "file",
    retrieval = "auto"
  )
)

[wlandau]

Is there something about your SLURM setup that would make retrieval = "main" infeasible?

[wlandau]

Could it be that you are using full file paths which differ between the login node and the compute nodes? If so, then using relative paths (relative to the project root) might solve this.

[koefoeden]

Is there something about your SLURM setup that would make retrieval = "main" infeasible?

I also suspected some kind of limits on the head-node, but we don't have this AFAIK.

Could it be that you are using full file paths which differ between the login node and the compute nodes? If so, then using relative paths (relative to the project root) might solve this.

I am using full file paths, but none that should differ between login-nodes and compute nodes - all storage is on a NFS connected by all nodes.

Your suggested reprex does indeed seem to run fine on our SLURM cluster as well- I will try to test some more and tinker around with your reprex and get back you with any updates. Thanks!

[koefoeden]

I am no longer able to reproduce this error in my setup (i.e. it seems to work now even with retrieval="auto", for reasons unknown), and will close this discussion.