RunCellpose crashes when using Podman with GPU in Mac

[emiglietta]

I'm trying to run Runcellpose on the images from the sample Cellprofiler pipeline using Podman on a M2 Mac, but it crashes when I want to use the GPU.

Running any version of Cellpose with or without GPU works fine when running from Python. I also have allocated as much memory as possible to Podman (~16GB).

V4 with GPU

Module crashes with the following error:

['/opt/podman/bin/podman', 'run', '--rm', '-v', '/Users/emigliet/Desktop/.cellprofiler_temp/feb751a2-1c11-4a5c-8163-82dd6c644b5c:/data', 'cellprofiler/runcellpose_with_pretrained:4.0.6', '--gpus', 'all', 'cellpose', '--verbose', '--dir', '/data/img', '--pretrained_model', 'cpsam', '--cellprob_threshold', '0.0', '--min_size', '15', '--anisotropy', '0.0', '--flow_threshold', '0.4', '--stitch_threshold', '0.0', '--exclude_on_edges']
WARNING: image platform (linux/amd64) does not match the expected platform (linux/arm64)

==========
== CUDA ==
==========

CUDA Version 11.8.0

Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.

WARNING: The NVIDIA Driver was not detected.  GPU functionality will not be available.
   Use the NVIDIA Container Toolkit to start this container with GPU support; see
   https://docs.nvidia.com/datacenter/cloud-native/ .

/opt/nvidia/nvidia_entrypoint.sh: line 67: exec: --: invalid option
exec: usage: exec [-cl] [-a name] [command [argument ...]] [redirection ...]
Failed to run module RunCellpose
Traceback (most recent call last):
  File "/Users/emigliet/Documents/GitHub/CP_DEV/CellProfiler-plugins/active_plugins/runcellpose.py", line 1011, in run
    cellpose_output = numpy.load(os.path.join(temp_img_dir, unique_name + "_seg.npy"), allow_pickle=True).item()
  File "/Users/emigliet/anaconda3/envs/cellposeSAM_cellprofiler/lib/python3.9/site-packages/numpy/lib/npyio.py", line 405, in load
    fid = stack.enter_context(open(os_fspath(file), "rb"))
FileNotFoundError: [Errno 2] No such file or directory: '/Users/emigliet/Desktop/.cellprofiler_temp/feb751a2-1c11-4a5c-8163-82dd6c644b5c/img/feb751a2-1c11-4a5c-8163-82dd6c644b5c_seg.npy'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/emigliet/anaconda3/envs/cellposeSAM_cellprofiler/lib/python3.9/site-packages/cellprofiler/gui/pipelinecontroller.py", line 3390, in do_step
    self.__pipeline.run_module(module, workspace_model)
  File "/Users/emigliet/anaconda3/envs/cellposeSAM_cellprofiler/lib/python3.9/site-packages/cellprofiler_core/pipeline/_pipeline.py", line 1349, in run_module
    module.run(workspace)
  File "/Users/emigliet/Documents/GitHub/CP_DEV/CellProfiler-plugins/active_plugins/runcellpose.py", line 1015, in run
    raise(FileNotFoundError("I'm sorry, Cellpose seems to have crashed and I'm not sure why, since it's in its own container."))
FileNotFoundError: I'm sorry, Cellpose seems to have crashed and I'm not sure why, since it's in its own container.
Using cp4 from python in the same environment works just fine using GPU :
Loading new model: cpsam
** TORCH MPS version installed and working. **
>>>> using GPU (MPS)
>>>> loading model /Users/emigliet/.cellpose/models/cpsam
Failed to set GPU memory share. Please check your PyTorch installation. Not setting per-process memory share.

V4 wit CPU

Run seems to freeze but just takes forever (with a very small 2D image), but eventually works:

['/opt/podman/bin/podman', 'run', '--rm', '-v', '/Users/emigliet/Documents/Proyectos FIL/Isa106/PICTO fotos para Esteban/MHCII-IBA1 9abril24.mdb/CP_output/.cellprofiler_temp/74ffc9db-c9b6-485c-95c5-68496d222a7e:/data', 'cellprofiler/runcellpose_with_pretrained:4.0.6', 'cellpose', '--verbose', '--dir', '/data/img', '--pretrained_model', 'cpsam', '--cellprob_threshold', '0.0', '--min_size', '15', '--anisotropy', '0.0', '--flow_threshold', '0.4', '--stitch_threshold', '0.0', '--exclude_on_edges']
WARNING: image platform (linux/amd64) does not match the expected platform (linux/arm64)

==========
== CUDA ==
==========

CUDA Version 11.8.0

Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.

WARNING: The NVIDIA Driver was not detected.  GPU functionality will not be available.
   Use the NVIDIA Container Toolkit to start this container with GPU support; see
   https://docs.nvidia.com/datacenter/cloud-native/ .



Welcome to CellposeSAM, cellpose v
cellpose version: 	4.0.6 
platform:       	linux 
python version: 	3.11.13 
torch version:  	2.7.1+cu118! The neural network component of
CPSAM is much larger than in previous versions and CPU excution is slow. 
We encourage users to use GPU/MPS if available. 


creating new log file
2025-09-24 21:25:41,008 [INFO] WRITING LOG OUTPUT TO /root/.cellpose/run.log
2025-09-24 21:25:41,010 [INFO] 
cellpose version: 	4.0.6 
platform:       	linux 
python version: 	3.11.13 
torch version:  	2.7.1+cu118
2025-09-24 21:25:41,011 [INFO] >>>> using CPU
2025-09-24 21:25:41,011 [INFO] >>>> using CPU
2025-09-24 21:25:41,011 [WARNING] the '--diam_mean' flag is deprecated in v4.0.1+ and no longer used
2025-09-24 21:25:41,026 [INFO] >>>> running cellpose on 1 images using all channels
2025-09-24 21:25:53,613 [INFO] >>>> loading model /root/.cellpose/models/cpsam
2025-09-24 21:25:57,953 [INFO] 0%|          | 0/1 [00:00<?, ?it/s]
[W924 21:25:58.430460378 NNPACK.cpp:57] Could not initialize NNPACK! Reason: Unsupported hardware.
[W924 22:02:25.092945292 NNPACK.cpp:57] Could not initialize NNPACK! Reason: Unsupported hardware.
[W924 22:02:27.676032493 NNPACK.cpp:57] Could not initialize NNPACK! Reason: Unsupported hardware.
[W924 22:02:30.128416575 NNPACK.cpp:57] Could not initialize NNPACK! Reason: Unsupported hardware.
2025-09-24 22:02:32,760 [INFO] 100%|##########| 1/1 [36:34<00:00, 2194.80s/it]
2025-09-24 22:02:32,763 [INFO] 100%|##########| 1/1 [36:34<00:00, 2194.81s/it]
2025-09-24 22:02:32,764 [INFO] >>>> completed in 2211.752 sec

V3 wit GPU

Produces the same error as V4

V3 with CPU

Runs fine

['/opt/podman/bin/podman', 'run', '--rm', '-v', '/Users/emigliet/Documents/Proyectos FIL/Isa106/PICTO fotos para Esteban/MHCII-IBA1 9abril24.mdb/CP_output/.cellprofiler_temp/40b2665a-c727-45fa-8b54-dc8b00b1d3e0:/data', 'cellprofiler/runcellpose_with_pretrained:3.1.1.2', 'cellpose', '--verbose', '--dir', '/data/img', '--pretrained_model', 'cyto3', '--chan', '0', '--chan2', '0', '--diameter', '30', '--cellprob_threshold', '0.0', '--min_size', '15', '--anisotropy', '0.0', '--flow_threshold', '0.4', '--stitch_threshold', '0.0']
WARNING: image platform (linux/amd64) does not match the expected platform (linux/arm64)

==========
== CUDA ==
==========

CUDA Version 11.8.0

Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.

WARNING: The NVIDIA Driver was not detected.  GPU functionality will not be available.
   Use the NVIDIA Container Toolkit to start this container with GPU support; see
   https://docs.nvidia.com/datacenter/cloud-native/ .

creating new log file
2025-09-24 22:07:08,464 [INFO] WRITING LOG OUTPUT TO /root/.cellpose/run.log
2025-09-24 22:07:08,467 [INFO] 
cellpose version: 	3.1.1.2 
platform:       	linux 
python version: 	3.11.13 
torch version:  	2.7.1+cu118
2025-09-24 22:07:08,467 [INFO] >>>> using CPU
2025-09-24 22:07:08,467 [INFO] >>>> using CPU
2025-09-24 22:07:08,490 [INFO] >>>> running cellpose on 1 images using chan_to_seg GRAY and chan (opt) NONE
2025-09-24 22:07:08,490 [INFO] >>>> using CPU
2025-09-24 22:07:08,490 [INFO] >>>> using CPU
2025-09-24 22:07:08,491 [INFO] >> cyto3 << model set to be used
2025-09-24 22:07:08,891 [INFO] >>>> loading model /root/.cellpose/models/cyto3
2025-09-24 22:07:09,606 [INFO] >>>> model diam_mean =  30.000 (ROIs rescaled to this size during training)
2025-09-24 22:07:09,607 [INFO] Downloading: "https://www.cellpose.org/models/size_cyto3.npy" to /root/.cellpose/models/size_cyto3.npy

100%|██████████| 3.54k/3.54k [00:00<00:00, 924kB/s]
2025-09-24 22:07:10,525 [INFO] >>>> using diameter 30.000 for all images
2025-09-24 22:07:10,527 [INFO] 0%|          | 0/1 [00:00<?, ?it/s]
2025-09-24 22:07:10,536 [INFO] channels set to [0, 0]
2025-09-24 22:07:10,536 [INFO] ~~~ FINDING MASKS ~~~
2025-09-24 22:08:07,691 [INFO] >>>> TOTAL TIME 57.15 sec
2025-09-24 22:08:07,738 [INFO] 100%|##########| 1/1 [00:57<00:00, 57.21s/it]
2025-09-24 22:08:07,739 [INFO] 100%|##########| 1/1 [00:57<00:00, 57.21s/it]
2025-09-24 22:08:07,739 [INFO] >>>> completed in 59.271 sec

[emiglietta]

UPDATE: I checked my Podman Desktop configuration and Rosetta is enabled

[bethac07]

Do you have Docker Desktop and can check to see if the behavior is the same between each? Fine if no just checking.

Seems like for Podman, we might need to think about making arm and non arm versions, but I suspect fewer people will be using Podman so that might affect the priority of doing that.

[emiglietta]

I don't have Docker Desktop on my machine (since I don't routinely use it, it made no sense to pay for the license), that's why I wanted to try Podman, in case I need to try something on a container for a collaborator :)

[ShataDg]

I was running CellProfiler 4.2.8 with Cellpose 4 with the Docker option and with use GPU set to yes, and I received the same error. RunCellpose does work with use GPU set to no, but it takes quite a while. I have tried using the default values as well as increasing the values for GPU_memory_share, but it gives the same error.

The following trace is CellProfiler 4.2.6 with Cellpose 4 (because I do not have CellProfiler 4.2.8 built from source)

15:31:47: Debug: Adding duplicate image handler for 'Windows bitmap file'
15:31:47: Debug: Adding duplicate animation handler for '1' type
15:31:47: Debug: Adding duplicate animation handler for '2' type
Could not load runomnipose
Traceback (most recent call last):
  File "/Users/sdasgupt/anaconda3/envs/CP_SD/lib/python3.8/site-packages/cellprofiler_core/utilities/core/modules/__init__.py", line 71, in add_module
    m = __import__(mod, globals(), locals(), ["__all__"], 0)
  File "/Users/sdasgupt/CellProfiler-plugins/active_plugins/runomnipose.py", line 9, in <module>
    from cellpose_omni import models, io, core, plot
ModuleNotFoundError: No module named 'cellpose_omni'
Could not load runimagejscript
Traceback (most recent call last):
  File "/Users/sdasgupt/anaconda3/envs/CP_SD/lib/python3.8/site-packages/cellprofiler_core/utilities/core/modules/__init__.py", line 71, in add_module
    m = __import__(mod, globals(), locals(), ["__all__"], 0)
  File "/Users/sdasgupt/CellProfiler-plugins/active_plugins/runimagejscript.py", line 15, in <module>
    import cpij.bridge as ijbridge, cpij.server as ijserver
  File "/Users/sdasgupt/CellProfiler-plugins/active_plugins/cpij/bridge.py", line 3, in <module>
    import atexit, cpij.server as ijserver
  File "/Users/sdasgupt/CellProfiler-plugins/active_plugins/cpij/server.py", line 15, in <module>
    import jpype, imagej, multiprocessing, socket, threading, time
ModuleNotFoundError: No module named 'jpype'
could not load these modules: runomnipose,runimagejscript
['/usr/local/bin/docker', 'run', '--rm', '-v', '/Users/sdasgupt/Documents/.cellprofiler_temp/c5fd2ca6-5417-41f7-b8e5-4382418cfc6f:/data', 'cellprofiler/runcellpose_with_pretrained:4.0.6', '--gpus', 'all', 'cellpose', '--verbose', '--dir', '/data/img', '--pretrained_model', 'cpsam', '--cellprob_threshold', '0.0', '--min_size', '15', '--anisotropy', '0.0', '--flow_threshold', '0.4', '--stitch_threshold', '0.0', '--exclude_on_edges']
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested

==========
== CUDA ==
==========

CUDA Version 11.8.0

Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.

WARNING: The NVIDIA Driver was not detected.  GPU functionality will not be available.
   Use the NVIDIA Container Toolkit to start this container with GPU support; see
   https://docs.nvidia.com/datacenter/cloud-native/ .

/opt/nvidia/nvidia_entrypoint.sh: line 67: exec: --: invalid option
exec: usage: exec [-cl] [-a name] [command [argument ...]] [redirection ...]
Failed to run module RunCellpose
Traceback (most recent call last):
  File "/Users/sdasgupt/CellProfiler-plugins/active_plugins/runcellpose.py", line 1011, in run
    cellpose_output = numpy.load(os.path.join(temp_img_dir, unique_name + "_seg.npy"), allow_pickle=True).item()
  File "/Users/sdasgupt/anaconda3/envs/CP_SD/lib/python3.8/site-packages/numpy/lib/npyio.py", line 405, in load
    fid = stack.enter_context(open(os_fspath(file), "rb"))
FileNotFoundError: [Errno 2] No such file or directory: '/Users/sdasgupt/Documents/.cellprofiler_temp/c5fd2ca6-5417-41f7-b8e5-4382418cfc6f/img/c5fd2ca6-5417-41f7-b8e5-4382418cfc6f_seg.npy'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/sdasgupt/anaconda3/envs/CP_SD/lib/python3.8/site-packages/cellprofiler/gui/pipelinecontroller.py", line 3391, in do_step
    self.__pipeline.run_module(module, workspace_model)
  File "/Users/sdasgupt/anaconda3/envs/CP_SD/lib/python3.8/site-packages/cellprofiler_core/pipeline/_pipeline.py", line 1349, in run_module
    module.run(workspace)
  File "/Users/sdasgupt/CellProfiler-plugins/active_plugins/runcellpose.py", line 1015, in run
    raise(FileNotFoundError("I'm sorry, Cellpose seems to have crashed and I'm not sure why, since it's in its own container."))
FileNotFoundError: I'm sorry, Cellpose seems to have crashed and I'm not sure why, since it's in its own container.