some dask experiments #4
Conversation
| # KeyError: 'e34a252e426f6cc81be22b03b77786ea' | ||
|
|
||
| fname = 'IsolatedGalaxy/galaxy0030/galaxy0030' | ||
| ds = yt.load(fname) |
There was a problem hiding this comment.
@matthewturk this file is one example of some confusing behavior with yt+dask. (don't bother with the other files in this PR for now at least). If you run this script as-is (python dask_failure_in_param_store.py 3 1 (the args are the number of workers and the threads per worker), you get this key error.
It relates to the dask worker state (see this SO answer https://stackoverflow.com/questions/75837897/dask-worker-has-different-imports-than-main-thread ): so if you move the ds = yt.load(fname) up before the __main__ block, the code runs OK.
There was a problem hiding this comment.
ideally things wouldn't be quite so state dependent...
There was a problem hiding this comment.
oh and in this particular case, if you use the on-disk ParameterFileStore, the code also runs with the load in the __main__ block (after fixing the bug i mentioned on slack), because all the workers can access the on-disk hashes and args for reconstructing
Exploring some dask + yt interactions