dying/disappearing daemons

[agilly]

I work with an R Shiny application, which defines persistent workers as follows:

# Configure async daemons
setup_daemons = function(){
  daemons(6, cleanup = FALSE, dispatcher = TRUE)
  
  prepare_workers = everywhere({
    options(myapp.option1 = TRUE)
    Sys.setenv(MY_CONFIG = "production")
    
    # Load required libraries
    required_libs = c("data.table", "mypackage1", "mypackage2")
    lapply(required_libs, library, character.only = TRUE)
  })
  
  while(unresolved(prepare_workers)){
    Sys.sleep(1)
  }
}

# Initial setup
setup_daemons()
shinyApp(ui = ui, server = server)

As you can see, required libraries are loaded at start-up through everywhere, and the daemons are set to be persistent through cleanup=F.
I was expecting that this would ensure the daemons are always running and the libraries are always loaded. However, in the course of execution, usually after some time and some tasks have already been run, workers were starting to take very long to resolve the tasks.

Upon investigation, this was caused by loading libraries. This means that either libraries were unloaded/cleanup was performed, or the workers had died and were being re-spawned.

Reactivity is essential in my application, so I've had to resort to this:

check_workers_alive = function(){
  # Check if required packages are loaded in all daemons
  load_status = everywhere({
    required_packages = c("data.table", "mypackage1", "mypackage2")
    all(required_packages %in% (.packages())) &&
    "myapp.option1" %in% names(options()) &&
    getOption("myapp.option1")
  })
  
  results <- unlist(collect_mirai(load_status))
  if(!all(results)){
    warning("Required packages not loaded in all daemons, reloading...")
    return(FALSE)
  } else {
    return(TRUE)
  }
}

# Periodic check - outside of shinyApp server - one task for all sessions/connected clients
observe({
  invalidateLater(1000*60*5, session = NULL) # Check every 5 minutes
  
  if(!check_workers_alive()){
    setup_daemons() # Reload everything
  }
})

basically running periodical checks and reloading the necessary tools if required.

As an extra precaution, I also add a special function single_worker_check_and_load, (which performs the same load check and loads libraries if needed) inside each mirai call in case the 30-second interval is not enough.

My question is twofold:

1. is this spontaneous daemon dying/lib unloading expected?
2. is this the best/only pattern to circumvent it? It feels a bit wasteful to run verification code every 5 minutes. Is there a built-in way to ensure the daemon pool is always "ready"?

[shikokuchuo]

Your setup_daemons() function works fine for me (using "dplyr" and "purrr" as the other packages). Unless you can provide a reprex, I do not know what could be going wrong for you.

Libraries will not become unloaded by themselves regardless of the cleanup argument if they were loaded via everywhere(). cleanup = FALSE does not imply anything about persistence of daemons - they are always persistent unless you set one of the timeout arguments.

Another minor matter - the following is not idomatic:

while(unresolved(prepare_workers)){
    Sys.sleep(1)
  }

it should just be:

call_mirai(prepare_workers)

If you are losing daemons due to crashes, you should be able to tell using a simple info() call. Daemons are never re-spawn by mirai.

If you need to re-launch daemons then you can use launch_local() with an everywhere call that is conditional upon a sentinel value (or e.g. !"data.table()" %in% loadedNamespaces()).

[agilly]

Thanks for confirming this should not happen and that daemons are never re-spawn. I will need to monitor the daemons closely in order to answer your question and give more context. This is in a relatively hard to contact/debug application, and the symptom I first noticed was re-loading of libraries, I circumvented this with my re-setup of daemons. I will try and send myself the results of info() when the re-setup call is triggered.

Thanks also for the info about cleanup=F which if I understand you correctly just impacts variables, not libraries? That is pretty helpful and I can enable it to prevent RAM runaway.

Also, could you share more details as to what launch_local() does and how to use it, and how it's different from a daemon() call?

[shikokuchuo]

Also, could you share more details as to what launch_local() does and how to use it, and how it's different from a daemon() call?

launch_local() launches a daemon (background R process) which runs daemon() with the correct parameters to connect back to your host session. That's the relationship between them!

[shikokuchuo]

@agilly the latest mirai 2.5.1 release fixes one bug that may have affected you if you were using stop_mirai() within your Shiny app. Also in your example above, you were launching local daemons only - these synchronize with the host process before the daemons() call returns, so it would be safe to call everywhere() straight afterwards. If you were launching remote daemons, it may take more time for those remote connections to be set up, and then you should consider setting the new .min argument to everywhere() to ensure that it runs on at least that number of daemons.

If you're still experiencing issues, do let me have any further details.