Add options for the Julia binary and project path (#667)

* Add options for the Julia binary and project path

This makes using JuliaPkg optional.

* fixup! Add options for the Julia binary and project path

* nitpicks

* allow exe option to be found in the path

---------

Co-authored-by: Christopher Doris <github.com/cjdoris>

Author: JamesWrigley

.github/workflows/docs.yml

@@ -23,7 +23,6 @@ jobs:
       - run: |
           julia --project=docs -e '
             using Pkg
-            Pkg.develop(PackageSpec(path=pwd()))
             Pkg.instantiate()'
       - run: |
           julia --project=docs -e '

.github/workflows/tests.yml

@@ -60,13 +60,22 @@ jobs:
           CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
 
   python:
-    name: Test Python (${{ matrix.pyversion }}, ${{ matrix.os }})
+    name: Test Python (${{ matrix.pyversion }}, ${{ matrix.os }}, ${{ matrix.juliaexe }})
     runs-on: ${{ matrix.os }}
     strategy:
       fail-fast: false
       matrix:
         os: [ubuntu-latest, windows-latest, macos-latest]
         pyversion: ["3", "3.9"]
+        juliaexe: ["@JuliaPkg"]
+        include:
+        - os: ubuntu-latest
+          pyversion: 3
+          juliaexe: julia
+    env:
+      MANUAL_TEST_PROJECT: /tmp/juliacall-test-project
+      PYTHON_JULIACALL_THREADS: '2'
+      PYTHON_JULIACALL_HANDLE_SIGNALS: 'yes'
 
     steps:
       - uses: actions/checkout@v5
@@ -82,10 +91,17 @@ jobs:
           python-version: ${{ matrix.pyversion }}
 
       - name: Set up Julia
+        id: setup_julia
         uses: julia-actions/setup-julia@v2
         with:
           version: '1'
 
+      - name: Set up test Julia project
+        if: ${{ matrix.juliaexe == 'julia' }}
+        run: |
+          mkdir ${{ env.MANUAL_TEST_PROJECT }}
+          julia --project=${{ env.MANUAL_TEST_PROJECT }} -e 'import Pkg; Pkg.develop(path="."); Pkg.instantiate()'
+
       - name: Install dependencies
         run: |
           cp pysrc/juliacall/juliapkg-dev.json pysrc/juliacall/juliapkg.json
@@ -98,12 +114,18 @@ jobs:
           # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
           uv run flake8 ./pysrc ./pytest --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
 
-      - name: Run tests
+      - name: Run tests (with JuliaPkg)
+        if: ${{ matrix.juliaexe == '@JuliaPkg' }}
+        run: |
+          uv run pytest -s --nbval --cov=pysrc ./pytest/
+
+      - name: Run tests (without JuliaPkg)
+        if: ${{ matrix.juliaexe == 'julia' }}
         run: |
           uv run pytest -s --nbval --cov=pysrc ./pytest/
         env:
-          PYTHON_JULIACALL_THREADS: '2'
-          PYTHON_JULIACALL_HANDLE_SIGNALS: 'yes'
+          PYTHON_JULIACALL_EXE: "${{ steps.setup_julia.outputs.julia-bindir }}/julia"
+          PYTHON_JULIACALL_PROJECT: ${{ env.MANUAL_TEST_PROJECT }}
 
       - name: Upload coverage to Codecov
         uses: codecov/codecov-action@v5

docs/Project.toml

@@ -4,3 +4,6 @@ PythonCall = "6099a3de-0909-46bc-b1f4-468b9a2dfc0d"
 
 [compat]
 Documenter = "1"
+
+[sources]
+PythonCall = {path = ".."}

docs/src/juliacall.md

@@ -73,7 +73,8 @@ What to read next:
 
 ## [Managing Julia dependencies](@id julia-deps)
 
-JuliaCall manages its Julia dependencies using [JuliaPkg](https://github.com/JuliaPy/PyJuliaPkg).
+By default JuliaCall manages its Julia dependencies using
+[JuliaPkg](https://github.com/JuliaPy/PyJuliaPkg).
 
 It will automatically download a suitable version of Julia if required.
 
@@ -100,6 +101,15 @@ Alternatively you can use `add`, `rm`, etc. from JuliaPkg to edit this file.
 
 See [JuliaPkg](https://github.com/JuliaPy/PyJuliaPkg) for more details.
 
+### Using existing environments
+
+It's possible to override the defaults and disable JuliaPkg entirely by setting
+the `PYTHON_JULIACALL_EXE` and `PYTHON_JULIACALL_PROJECT` options (both must be
+set together). This is particularly useful when using shared environments on HPC
+systems that may be readonly. Note that the project set in
+`PYTHON_JULIACALL_PROJECT` *must* already have PythonCall.jl installed and it
+*must* match the JuliaCall version, otherwise loading Julia will fail.
+
 ## [Configuration](@id julia-config)
 
 Some features of the Julia process, such as the optimization level or number of threads, may
@@ -125,6 +135,8 @@ be configured in two ways:
 | `-X juliacall-warn-overwrite=<yes\|no>` | `PYTHON_JULIACALL_WARN_OVERWRITE=<yes\|no>` | Enable or disable method overwrite warnings. |
 | `-X juliacall-autoload-ipython-extension=<yes\|no>` | `PYTHON_JULIACALL_AUTOLOAD_IPYTHON_EXTENSION=<yes\|no>` | Enable or disable IPython extension autoloading. |
 | `-X juliacall-heap-size-hint=<N>` | `PYTHON_JULIACALL_HEAP_SIZE_HINT=<N>` | Hint for initial heap size in bytes. |
+| `-X juliacall-exe=<file>` | `PYTHON_JULIACALL_EXE=<file>` | Path to Julia binary to use (overrides JuliaPkg). |
+| `-X juliacall-project=<dir>` | `PYTHON_JULIACALL_PROJECT=<dir>` | Path to the Julia project to use (overrides JuliaPkg). |
 
 ## [Multi-threading](@id py-multi-threading)
 

docs/src/releasenotes.md

@@ -7,6 +7,7 @@
 * Added `juliacall.ArrayValue` support for Julia arrays of `InlineDateTime64` and `InlineTimeDelta64`.
 * If `JULIA_PYTHONCALL_EXE` is a relative path, it is now considered relative to the active project.
 * Added option `JULIA_PYTHONCALL_EXE=@venv` to use a Python virtual environment relative to the active project.
+* Added `PYTHON_JULIACALL_EXE` and `PYTHON_JULIACALL_PROJECT` for specifying the Julia binary and project to override JuliaPkg.
 * Bug fixes.
 * Internal: switch from Requires.jl to package extensions.
 

pysrc/juliacall/__init__.py

@@ -96,6 +96,16 @@ def path_option(name, default=None, check_exists=False, **kw):
                 raise ValueError(f'{s}: path does not exist')
             return os.path.abspath(path), s
         return default, s
+    
+    def executable_option(name, default=None, **kw):
+        import shutil
+        _path, s = option(name, **kw)
+        if _path is not None:
+            path = shutil.which(_path)
+            if path is None:
+                raise ValueError(f'{s}: executable not found')
+            return os.path.abspath(path), s
+        return default, s
 
     def int_option(name, *, accept_auto=False, **kw):
         val, s = option(name, **kw)
@@ -147,18 +157,31 @@ def args_from_config(config):
     CONFIG['opt_handle_signals'] = choice('handle_signals', ['yes', 'no'])[0]
     CONFIG['opt_startup_file'] = choice('startup_file', ['yes', 'no'])[0]
     CONFIG['opt_heap_size_hint'] = option('heap_size_hint')[0]
+    CONFIG['project'] = path_option('project', check_exists=True)[0]
+    CONFIG['exepath'] = executable_option('exe')[0]
 
     # Stop if we already initialised
     if CONFIG['inited']:
         return
 
-    # we don't import this at the top level because it is not required when juliacall is
-    # loaded by PythonCall and won't be available
-    import juliapkg
+    have_exepath = CONFIG['exepath'] is not None
+    have_project = CONFIG['project'] is not None
+    if have_exepath and have_project:
+        pass
+    elif (not have_exepath) and (not have_project):
+        # we don't import this at the top level because it is not required when
+        # juliacall is loaded by PythonCall and won't be available, or if both
+        # `exepath` and `project` are set by the user.
+        import juliapkg
+
+        # Find the Julia executable and project
+        CONFIG['exepath'] = juliapkg.executable()
+        CONFIG['project'] = juliapkg.project()
+    else:
+        raise Exception("Both PYTHON_JULIACALL_PROJECT and PYTHON_JULIACALL_EXE must be set together, not only one of them.")
 
-    # Find the Julia executable and project
-    CONFIG['exepath'] = exepath = juliapkg.executable()
-    CONFIG['project'] = project = juliapkg.project()
+    exepath = CONFIG['exepath']
+    project = CONFIG['project']
 
     # Find the Julia library
     cmd = [exepath, '--project='+project, '--startup-file=no', '-O0', '--compile=min',