Conda installation?

[uturuncoglu]

I am not sure if there is a plan or not but it would be nice to have capability to install pyschism through Conda. In UFS Coastal workflow, we have packages that are already installed with Conda and installing pyschism with pip is also installing lots of duplicated modules. So, it would be nice to have flexibility to install pyschism also with Conda.

[josephzhang8]

@brey: do you have any updates on the conda install? Thx

[mansurjisan]

Hi @uturuncoglu , I installed PySCHISM in the MSU Hercules cluster. First, I created a Conda environment, and then, within this Conda environment, I used pip to install PySCHISM. It's not a perfect solution, as it doesn't fully address the module duplication issue you mentioned, but it does provide a way to use PySCHISM within a Conda-managed environment.

For detailed installation steps and discussion, I've documented the process in a GitHub issue. You can find it here:
https://github.com/noaa-ocs-modeling/SurgeTeamCoordination/issues/545#issuecomment-2284884498

I am also working on developing an ecFlow based workflow where I can use pyschism to generate input files for SCHISM and UFS Coastal RT case: https://github.com/mansurjisan/SCHISM-ecFLOW/tree/main/ecf

[uturuncoglu]

@mansurjisan Thanks. Yes. I did the same thing but I am not sure it is a good practice to use conda and pip together. It is not a big deal but it would be nice to have conda support too.

[brey]

@brey: do you have any updates on the conda install? Thx

We are providing a conda package for schism not pyschism. It shouldn't be difficult to build one for psychism as well. We can support. Who wants to be the maintainer?

[uturuncoglu]

@mansurjisan @brey I would like to back to this discussion since it would be nice to have conda package in psychism. I am not sure who needs to be maintainer of it. The package is used by the UFS Coastal App as a part of the workflow implementation and it would be nice to have a fixed Python environment. All the packages in UFS Coastal App are using conda except psychism at this point.

[mansurjisan]

@uturuncoglu ,

I agree. We need to come up with a plan for pySCHISM . I will do some research on making the pySCHISM package installable via conda. We can discuss it more during our next UFS-Coastal meeting.

[uturuncoglu]

@mansurjisan I have issue with installing package with pip, it is giving following error,

Collecting pygeos (from pyschism)
  Using cached pygeos-0.14.tar.gz (141 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [36 lines of output]
      <string>:8: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
      /tmp/pip-install-ubew92s6/pygeos_98d1b02530d248acb7cce82d1c28c0ff/versioneer.py:421: SyntaxWarning: invalid escape sequence '\s'
        LONG_VERSION_PY['git'] = '''
      Could not find geos-config executable. Either append the path to geos-config to PATH or manually provide the include_dirs, library_dirs, libraries and other link args for compiling against a GEOS version >=3.5.
      Compiling pygeos/_geometry.pyx because it changed.
      Compiling pygeos/_geos.pyx because it changed.
      [1/2] Cythonizing pygeos/_geometry.pyx
      [2/2] Cythonizing pygeos/_geos.pyx
      Traceback (most recent call last):
        File "/work2/noaa/nems/tufuk/COASTAL/ufs-coastal-app/python/envs/myenv/lib/python3.13/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 389, in <module>
          main()
          ~~~~^^
        File "/work2/noaa/nems/tufuk/COASTAL/ufs-coastal-app/python/envs/myenv/lib/python3.13/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 373, in main
          json_out["return_val"] = hook(**hook_input["kwargs"])
                                   ~~~~^^^^^^^^^^^^^^^^^^^^^^^^
        File "/work2/noaa/nems/tufuk/COASTAL/ufs-coastal-app/python/envs/myenv/lib/python3.13/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 143, in get_requires_for_build_wheel
          return hook(config_settings)
        File "/tmp/pip-build-env-eg8t_2qj/overlay/lib/python3.13/site-packages/setuptools/build_meta.py", line 334, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=[])
                 ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-eg8t_2qj/overlay/lib/python3.13/site-packages/setuptools/build_meta.py", line 304, in _get_build_requires
          self.run_setup()
          ~~~~~~~~~~~~~~^^
        File "/tmp/pip-build-env-eg8t_2qj/overlay/lib/python3.13/site-packages/setuptools/build_meta.py", line 320, in run_setup
          exec(code, locals())
          ~~~~^^^^^^^^^^^^^^^^
        File "<string>", line 214, in <module>
        File "/tmp/pip-install-ubew92s6/pygeos_98d1b02530d248acb7cce82d1c28c0ff/versioneer.py", line 1480, in get_version
          return get_versions()["version"]
                 ~~~~~~~~~~~~^^
        File "/tmp/pip-install-ubew92s6/pygeos_98d1b02530d248acb7cce82d1c28c0ff/versioneer.py", line 1412, in get_versions
          cfg = get_config_from_root(root)
        File "/tmp/pip-install-ubew92s6/pygeos_98d1b02530d248acb7cce82d1c28c0ff/versioneer.py", line 342, in get_config_from_root
          parser = configparser.SafeConfigParser()
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      AttributeError: module 'configparser' has no attribute 'SafeConfigParser'. Did you mean: 'RawConfigParser'?
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Here are my commands on Hercules:

conda create --prefix $PWD/python/envs/myenv
conda activate /work2/noaa/nems/tufuk/COASTAL/ufs-coastal-app/python/envs/myenv
conda install pip
pip install pyschism

This is a good example about the reproducibility of the current approach. It was workin in my old environment in the same machine but now not. At this point I am trying to create environment.yml for UFS Coastal to make it work.

[mansurjisan]

@uturuncoglu

I was able to install following the below instructions:

Environment

• Cluster: Hercules
• Python version: 3.9.18 (conda-forge)

Installation Steps

1. Create and activate a new conda environment:

   conda create -n pyschism_env python=3.9
   conda activate pyschism_env
   

2. Install UDUNITS-2 library::

   conda install -c conda-forge udunits2
   

3. Install key dependencies::

   conda install -c conda-forge gdal sqlite fiona
   

4. Install additional required packages::

   conda install -c conda-forge numpy scipy netcdf4 matplotlib geopandas shapely
   

5. Install PySCHISM::

   pip install pyschism
   
   

[uturuncoglu]

@mansurjisan Thanks for your help. Strange. Is there any reason to use specific version of Python? Is it in the documentation?

[mansurjisan]

@uturuncoglu I don't quite recall why Python 3.9 worked specifically for me, but based on the PySCHISM README file, the recommended Python version is >=3.6, with the latest available version being encouraged.

Here’s the relevant excerpt from the documentation:

"It is highly recommended that you run this software using a [Python virtual environment](https://gist.github.com/jreniel/c2dd4f2f68f9d8172355461b5337f236), and that you use Python>=3.6 (preferably, using the latest available Python version is encouraged). You may use conda or venv to satisfy this dependency."

[uturuncoglu]

@mansurjisan Okay. Thanks for checking. I hope I could create a stable environment that could work with workflow.

[mansurjisan]

@uturuncoglu I was able to build pyschism through conda without using pip. I had to do it in offline mode from compute node by downloading all the dependencies and the source code. It was very CPU intensive when i tried to build it from the Hercules login node (~CPU % reached almost 100% for around 10 minutes). The conda recipes worked fine with the existing pySCHISM version. I will provide more details during our next meeting and discuss what needs to be done to host it on conda-forge.

[brey]

Dear @uturuncoglu and @mansurjisan, I finally found some time to check on this. The attached recipe works for me on my mac with the following issues:

• The setup.py file has some issues that need to be addressed. I had to use a build.sh script in the recipe to deal with it but I would suggest that you move to a pyproject.toml configuration asap.
• Packages pylib-essentials and tqdm-logging-wrapper don't have a conda installer. If you install psychism compiled with the attached recipe and then pip install these two packages, it works. However, it's not a good idea to mix pip with conda. I suggest that you work with the developers to provide a conda installation option.
• The tests in the meta.yaml file are not done because of the above issue.
• There are 2 options to provide the code in the meta.yaml file, from pypi or GitHub. Both work. Choose one for the final recipe.

#  git_rev: v0.1.15
#  git_url: https://github.com/schism-dev/pyschism.git
   url: https://pypi.org/packages/source/{{ name[0] }}/{{ name }}/pyschism-{{ version }}.tar.gz
   sha256: 42bed698d21bd8e24427a8ffba9148f74519da02ec35e1e0e463feacb47d5bac

Please try the recipe on linux and let me know if you have issues.

use conda build folder where folder is the folder containing the recipe.

pyschism.zip

[uturuncoglu]

@mansurjisan I tried your way and I am getting following error,

ValueError: Bad dask version: cfdm requires 2024.6.0<=dask<=2024.7.1. Got 2024.8.0 at /work2/noaa/nems/tufuk/COASTAL/ufs-coastal-app/python/envs/ufs-coastal-app/lib/python3.9/site-packages/dask/__init__.py

As I know there were another package was requesting 2024.8.0. So, still hard to create consistent environment for the workflow.

@brey thanks for your help. I'll also check your approach to see how it goes.

[uturuncoglu]

@mansurjisan @brey BTW, why loading pyschism is taking so long. For example, if I try to load following from pyschism.mesh.vgrid import Vgrid it is waiting couple of minutes. Any idea?