Firedrake should switch to a src layout

[connorjward]

At present, having firedrake, pyop2, tsfc etc as top-level directories in the Firedrake repository means that when we run operations like make check from the root directory we import the firedrake directory, rather than an installed package. This works for editable installations but not for non-editable ones.

I encountered this in my latest PR. I installed Firedrake not in editable mode and then ran make check from the repository root . This produced the error:

ImportError while loading conftest '/Users/github/actions-runner/_work/firedrake/firedrake/firedrake-repo/tests/firedrake/conftest.py'.
tests/firedrake/conftest.py:6: in <module>
    from firedrake.petsc import PETSc, get_external_packages
firedrake/__init__.py:24: in <module>
    from firedrake import petsc
firedrake/petsc.py:[15](https://github.com/firedrakeproject/firedrake/actions/runs/13387597638/job/37387694553?pr=4011#step:8:16): in <module>
    from pyop2 import mpi
pyop2/__init__.py:4: in <module>
    from pyop2.op2 import *  # noqa
pyop2/op2.py:43: in <module>
    from pyop2.types import (  # noqa: F401
pyop2/types/__init__.py:10: in <module>
    from .mat import *  # noqa: F401
pyop2/types/mat.py:9: in <module>
    from pyop2 import (
E   ImportError: cannot import name 'sparsity' from 'pyop2' (consider renaming '/Users/github/actions-runner/_work/firedrake/firedrake/firedrake-repo/pyop2/__init__.py' if it has the same name as a library you intended to import)

This is because Python is looking inside the pyop2 directory for the compiled extensions instead of inside $VIRTUAL_ENV/lib/python*/site-packages/pyop2.

Switching to a src layout, where the Python packages are put inside of a top-level src directory would avoid this confusing error.

Useful sources:

• https://packaging.python.org/en/latest/discussions/src-layout-vs-flat-layout/#src-layout-vs-flat-layout
• https://docs.pytest.org/en/7.1.x/explanation/goodpractices.html

[francesco-ballarin]

Assuming I understand the source of the issue correctly, you may want to have a look at https://docs.python.org/3/using/cmdline.html#cmdoption-P , which should at least mitigate the issue on the current layout.

[connorjward]

That would probably avoid this issue, but it doesn't change the fact that the default behaviour can be unclear.