Setting a signal handler on a non-main thread should be a fatal error. #58283
Description
Setting a signal handler on a non-main thread should be a fatal error. We log a warning but it should be fatal. It turns out that some data test code snippet related to dask depends on the non-fatal behavior -- https://buildkite.com/ray-project/premerge/builds/51784#0199eb92-aa9a-4053-adaa-c822e91ea7f3/658-751
See https://github.com/ray-project/ray/pull/57086/files#r2427900665
[2025-10-16T06:00:03Z] FAILED: //doc:source/ray-more-libs/doc_code/dask_on_ray_callbacks (Summary)
--
| [2025-10-16T06:00:03Z] /root/.cache/bazel/_bazel_root/1df605deb6d24fc8068f6e25793ec703/execroot/io_ray/bazel-out/k8-opt/testlogs/doc/source/ray-more-libs/doc_code/dask_on_ray_callbacks/test.log
| [2025-10-16T06:00:03Z] /root/.cache/bazel/_bazel_root/1df605deb6d24fc8068f6e25793ec703/execroot/io_ray/bazel-out/k8-opt/testlogs/doc/source/ray-more-libs/doc_code/dask_on_ray_callbacks/test_attempts/attempt_1.log
| [2025-10-16T06:00:03Z] /root/.cache/bazel/_bazel_root/1df605deb6d24fc8068f6e25793ec703/execroot/io_ray/bazel-out/k8-opt/testlogs/doc/source/ray-more-libs/doc_code/dask_on_ray_callbacks/test_attempts/attempt_2.log
| [2025-10-16T06:00:03Z] (23:00:03) INFO: From Testing //doc:source/ray-more-libs/doc_code/dask_on_ray_callbacks:
| [2025-10-16T06:00:03Z] ==================== Test output for //doc:source/ray-more-libs/doc_code/dask_on_ray_callbacks:
| [2025-10-16T06:00:03Z] Traceback (most recent call last):
| [2025-10-16T06:00:03Z] File "/root/.cache/bazel/_bazel_root/1df605deb6d24fc8068f6e25793ec703/execroot/io_ray/bazel-out/k8-opt/bin/doc/source/ray-more-libs/doc_code/dask_on_ray_callbacks.runfiles/io_ray/doc/source/ray-more-libs/doc_code/dask_on_ray_callbacks.py", line 28, in <module>
| [2025-10-16T06:00:03Z] z.compute(scheduler=ray_dask_get)
| [2025-10-16T06:00:03Z] File "/opt/miniforge/lib/python3.12/site-packages/dask/base.py", line 373, in compute
| [2025-10-16T06:00:03Z] (result,) = compute(self, traverse=False, **kwargs)
| [2025-10-16T06:00:03Z] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| [2025-10-16T06:00:03Z] File "/opt/miniforge/lib/python3.12/site-packages/dask/base.py", line 681, in compute
| [2025-10-16T06:00:03Z] results = schedule(expr, keys, **kwargs)
| [2025-10-16T06:00:03Z] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| [2025-10-16T06:00:03Z] File "/rayci/python/ray/util/dask/scheduler.py", line 189, in ray_dask_get
| [2025-10-16T06:00:03Z] object_refs = get_async(
| [2025-10-16T06:00:03Z] ^^^^^^^^^^
| [2025-10-16T06:00:03Z] File "/rayci/python/ray/util/dask/scheduler_utils.py", line 355, in get_async
| [2025-10-16T06:00:03Z] raise_exception(exc, tb)
| [2025-10-16T06:00:03Z] File "/rayci/python/ray/util/dask/scheduler_utils.py", line 210, in reraise
| [2025-10-16T06:00:03Z] raise exc
| [2025-10-16T06:00:03Z] File "/rayci/python/ray/util/dask/scheduler.py", line 305, in _rayify_task_wrapper
| [2025-10-16T06:00:03Z] result = _rayify_task(
| [2025-10-16T06:00:03Z] ^^^^^^^^^^^^^
| [2025-10-16T06:00:03Z] File "/rayci/python/ray/util/dask/scheduler.py", line 399, in _rayify_task
| [2025-10-16T06:00:03Z] object_refs = dask_task_wrapper.options(
| [2025-10-16T06:00:03Z] ^^^^^^^^^^^^^^^^^^^^^^^^^^
| [2025-10-16T06:00:03Z] File "/rayci/python/ray/remote_function.py", line 292, in remote
| [2025-10-16T06:00:03Z] return func_cls._remote(
| [2025-10-16T06:00:03Z] ^^^^^^^^^^^^^^^^^
| [2025-10-16T06:00:03Z] File "/rayci/python/ray/_private/auto_init_hook.py", line 21, in auto_init_wrapper
| [2025-10-16T06:00:03Z] auto_init_ray()
| [2025-10-16T06:00:03Z] File "/rayci/python/ray/_private/auto_init_hook.py", line 15, in auto_init_ray
| [2025-10-16T06:00:03Z] ray.init()
| [2025-10-16T06:00:03Z] File "/rayci/python/ray/_private/client_mode_hook.py", line 104, in wrapper
| [2025-10-16T06:00:03Z] return func(*args, **kwargs)
| [2025-10-16T06:00:03Z] ^^^^^^^^^^^^^^^^^^^^^
| [2025-10-16T06:00:03Z] File "/rayci/python/ray/_private/worker.py", line 1674, in init
| [2025-10-16T06:00:03Z] ray._private.utils.install_driver_signal_handlers()
| [2025-10-16T06:00:03Z] File "/rayci/python/ray/_private/utils.py", line 824, in install_driver_signal_handlers
| [2025-10-16T06:00:03Z] raise RuntimeError(
| [2025-10-16T06:00:03Z] RuntimeError: Signal handlers must be installed from the main thread. Current thread: Thread-1 (worker)
| [2025-10-16T06:00:03Z] ================================================================================
| [2025-10-16T06:00:03Z] ==================== Test output for //doc:source/ray-more-libs/doc_code/dask_on_ray_callbacks:
| [2025-10-16T06:00:03Z] Traceback (most recent call last):
| [2025-10-16T06:00:03Z] File "/root/.cache/bazel/_bazel_root/1df605deb6d24fc8068f6e25793ec703/execroot/io_ray/bazel-out/k8-opt/bin/doc/source/ray-more-libs/doc_code/dask_on_ray_callbacks.runfiles/io_ray/doc/source/ray-more-libs/doc_code/dask_on_ray_callbacks.py", line 28, in <module>
| [2025-10-16T06:00:03Z] z.compute(scheduler=ray_dask_get)
| [2025-10-16T06:00:03Z] File "/opt/miniforge/lib/python3.12/site-packages/dask/base.py", line 373, in compute
| [2025-10-16T06:00:03Z] (result,) = compute(self, traverse=False, **kwargs)
| [2025-10-16T06:00:03Z] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| [2025-10-16T06:00:03Z] File "/opt/miniforge/lib/python3.12/site-packages/dask/base.py", line 681, in compute
| [2025-10-16T06:00:03Z] results = schedule(expr, keys, **kwargs)
| [2025-10-16T06:00:03Z] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| [2025-10-16T06:00:03Z] File "/rayci/python/ray/util/dask/scheduler.py", line 189, in ray_dask_get
| [2025-10-16T06:00:03Z] object_refs = get_async(
| [2025-10-16T06:00:03Z] ^^^^^^^^^^
| [2025-10-16T06:00:03Z] File "/rayci/python/ray/util/dask/scheduler_utils.py", line 355, in get_async
| [2025-10-16T06:00:03Z] raise_exception(exc, tb)
| [2025-10-16T06:00:03Z] File "/rayci/python/ray/util/dask/scheduler_utils.py", line 210, in reraise
| [2025-10-16T06:00:03Z] raise exc
| [2025-10-16T06:00:03Z] File "/rayci/python/ray/util/dask/scheduler.py", line 305, in _rayify_task_wrapper
| [2025-10-16T06:00:03Z] result = _rayify_task(
| [2025-10-16T06:00:03Z] ^^^^^^^^^^^^^
| [2025-10-16T06:00:03Z] File "/rayci/python/ray/util/dask/scheduler.py", line 399, in _rayify_task
| [2025-10-16T06:00:03Z] object_refs = dask_task_wrapper.options(
| [2025-10-16T06:00:03Z] ^^^^^^^^^^^^^^^^^^^^^^^^^^
| [2025-10-16T06:00:03Z] File "/rayci/python/ray/remote_function.py", line 292, in remote
| [2025-10-16T06:00:03Z] return func_cls._remote(
| [2025-10-16T06:00:03Z] ^^^^^^^^^^^^^^^^^
| [2025-10-16T06:00:03Z] File "/rayci/python/ray/_private/auto_init_hook.py", line 21, in auto_init_wrapper
| [2025-10-16T06:00:03Z] auto_init_ray()
| [2025-10-16T06:00:03Z] File "/rayci/python/ray/_private/auto_init_hook.py", line 15, in auto_init_ray
| [2025-10-16T06:00:03Z] ray.init()
| [2025-10-16T06:00:03Z] File "/rayci/python/ray/_private/client_mode_hook.py", line 104, in wrapper
| [2025-10-16T06:00:03Z] return func(*args, **kwargs)
| [2025-10-16T06:00:03Z] ^^^^^^^^^^^^^^^^^^^^^
| [2025-10-16T06:00:03Z] File "/rayci/python/ray/_private/worker.py", line 1674, in init
| [2025-10-16T06:00:03Z] ray._private.utils.install_driver_signal_handlers()
| [2025-10-16T06:00:03Z] File "/rayci/python/ray/_private/utils.py", line 824, in install_driver_signal_handlers
| [2025-10-16T06:00:03Z] raise RuntimeError(
| [2025-10-16T06:00:03Z] RuntimeError: Signal handlers must be installed from the main thread. Current thread: Thread-1 (worker)
| [2025-10-16T06:00:03Z] ================================================================================
| [2025-10-16T06:00:03Z] ==================== Test output for //doc:source/ray-more-libs/doc_code/dask_on_ray_callbacks:
| [2025-10-16T06:00:03Z] Traceback (most recent call last):
| [2025-10-16T06:00:03Z] File "/root/.cache/bazel/_bazel_root/1df605deb6d24fc8068f6e25793ec703/execroot/io_ray/bazel-out/k8-opt/bin/doc/source/ray-more-libs/doc_code/dask_on_ray_callbacks.runfiles/io_ray/doc/source/ray-more-libs/doc_code/dask_on_ray_callbacks.py", line 28, in <module>
| [2025-10-16T06:00:03Z] z.compute(scheduler=ray_dask_get)
| [2025-10-16T06:00:03Z] File "/opt/miniforge/lib/python3.12/site-packages/dask/base.py", line 373, in compute
| [2025-10-16T06:00:03Z] (result,) = compute(self, traverse=False, **kwargs)
| [2025-10-16T06:00:03Z] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| [2025-10-16T06:00:03Z] File "/opt/miniforge/lib/python3.12/site-packages/dask/base.py", line 681, in compute
| [2025-10-16T06:00:03Z] results = schedule(expr, keys, **kwargs)
| [2025-10-16T06:00:03Z] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| [2025-10-16T06:00:03Z] File "/rayci/python/ray/util/dask/scheduler.py", line 189, in ray_dask_get
| [2025-10-16T06:00:03Z] object_refs = get_async(
| [2025-10-16T06:00:03Z] ^^^^^^^^^^
| [2025-10-16T06:00:03Z] File "/rayci/python/ray/util/dask/scheduler_utils.py", line 355, in get_async
| [2025-10-16T06:00:03Z] raise_exception(exc, tb)
| [2025-10-16T06:00:03Z] File "/rayci/python/ray/util/dask/scheduler_utils.py", line 210, in reraise
| [2025-10-16T06:00:03Z] raise exc
| [2025-10-16T06:00:03Z] File "/rayci/python/ray/util/dask/scheduler.py", line 305, in _rayify_task_wrapper
| [2025-10-16T06:00:03Z] result = _rayify_task(
| [2025-10-16T06:00:03Z] ^^^^^^^^^^^^^
| [2025-10-16T06:00:03Z] File "/rayci/python/ray/util/dask/scheduler.py", line 399, in _rayify_task
| [2025-10-16T06:00:03Z] object_refs = dask_task_wrapper.options(
| [2025-10-16T06:00:03Z] ^^^^^^^^^^^^^^^^^^^^^^^^^^
| [2025-10-16T06:00:03Z] File "/rayci/python/ray/remote_function.py", line 292, in remote
| [2025-10-16T06:00:03Z] return func_cls._remote(
| [2025-10-16T06:00:03Z] ^^^^^^^^^^^^^^^^^
| [2025-10-16T06:00:03Z] File "/rayci/python/ray/_private/auto_init_hook.py", line 21, in auto_init_wrapper
| [2025-10-16T06:00:03Z] auto_init_ray()
| [2025-10-16T06:00:03Z] File "/rayci/python/ray/_private/auto_init_hook.py", line 15, in auto_init_ray
| [2025-10-16T06:00:03Z] ray.init()
| [2025-10-16T06:00:03Z] File "/rayci/python/ray/_private/client_mode_hook.py", line 104, in wrapper
| [2025-10-16T06:00:03Z] return func(*args, **kwargs)
| [2025-10-16T06:00:03Z] ^^^^^^^^^^^^^^^^^^^^^
| [2025-10-16T06:00:03Z] File "/rayci/python/ray/_private/worker.py", line 1674, in init
| [2025-10-16T06:00:03Z] ray._private.utils.install_driver_signal_handlers()
| [2025-10-16T06:00:03Z] File "/rayci/python/ray/_private/utils.py", line 824, in install_driver_signal_handlers
| [2025-10-16T06:00:03Z] raise RuntimeError(
| [2025-10-16T06:00:03Z] RuntimeError: Signal handlers must be installed from the main thread. Current thread: Thread-1 (worker)
| [2025-10-16T06:00:03Z] ================================================================================