Relocatable OCaml - explicit-relative paths in ld.conf

[dra27]

This is the first of three PRs which implement Relocatable OCaml as proposed in ocaml/RFCs#53. This PR specifically deals with changes to the handling of ld.conf.

The key changes are:

• A new configure option --with-stublibs added to address Only set CAML_LD_LIBRARY_PATH for system switches opam-repository#16406 which specifies a custom line to be added to the top of ld.conf. This new flag is intended for use in opam and provides a mechanism to eliminate the setting of CAML_LD_LIBRARY_PATH by the compiler packages in opam-repository.
• Lines in ld.conf beginning with an explicit reference to the current or parent directory (i.e. ./ or ../) are interpreted relative to the directory containing ld.conf.
• Previously, if OCAMLLIB (or CAMLLIB) were set in the environment, then the ld.conf was loaded from the directory referred to by this environment variable, and the ld.conf file located in the configured Standard Library location was ignored. The runtime now loads ld.conf from all of these locations, processing them in the same order.

I have presented all of these changes as one PR because while the individual parts of it can be separated, they all alter the same part of the runtime and don't recombine cleanly. While overhauling this infrequently-altered section of the runtime, I propose some further alterations:

• The processing of ld.conf is made line-ending agnostic - Unix and Cygwin builds now consistently ignore CR characters at the end of each line¹
• Static builds of the compiler no longer create a stublibs directory, nor add it to ld.conf
• The C implementation of the processing for ld.conf is exposed as a primitive which is now used by ocamlc, instead of having both C and OCaml implementations of the "same" algorithm. My reason for proposing this is twofold: firstly, the two implementations even before this PR are inconsistent (as the special cases in the testsuite reveal) and, secondly, with the other changes proposed here, the two implementations become considerably larger, and thus the benefit of sharing the code is bigger. The commit unifying the two implementations is intentionally towards the end of the series (cf. the deletions in bytecomp/dll.ml in the penultimate commit, versus the deletions in bytecomp/dll.ml for the overall PR)

As far as I know, the only thing apart from OCaml which parses ld.conf is ocamlfind, and the patch for that isn't particularly involved (see, for example, dra27/ocamlfind#7bd58cb). The compiler drivers use the prefix + to refer to the Standard Library (e.g. in -I +unix). Co-opting this notation for ld.conf doesn't work in combination with the change to load ld.conf from all possible locations, since + refers to the effective Standard Library location. Another possibility is to refer to the prefix by a given name, for example, having the default being $/lib/ocaml/stublibs and $/lib/ocaml. Establishing the idea of $ may be useful for other tools; rooting the names of files in the prefix rather than the directory containing the file.

Footnotes

1. The nefarious user who absolutely must have directories in ld.conf which end with CR characters is able to encode this by including a trailing slash after the name... ↩

[jonahbeckford]

FYI: I did a GitHub search for caml_parse_ld_conf since adding arguments to the argument list of caml_parse_ld_conf is veering into C undefined behavior (if client code accesses an old caml_parse_ld_conf in shared library, or worse if vice versa). No one outside of startup_byt.c is using it, so it looks safe to change.

[dra27]

Indeed, it's guarded by CAML_INTERNALS as well (it's actually only been exposed at all since #9284 in 4.13.0 in order to implement ocamlrun -config ... which was very very early work on Relocatable!)

[jonahbeckford]

For portable consistency, can we check for NULLs and empty strings? (Unless it is documented somewhere that empty OCAMLLIB stops search for CAMLLIB and OCAML_STDLIB_DIR, which would be weird).

[dra27]

I've tackled this in #14247 (the specific bit can be previewed in dra27#227, although this function moves about a bit, so it's in dynlink.c in that PR still!). My rationale was to preserve the "status quo" for in this PR, just because it's not strictly necessary to fix it but, as you can see from the follow-up (and from dra27#225), I firmly agree that we should treat NULL and empty as equivalent!

[jonahbeckford]

LGTM, with minor comment. I didn't review the test suite since I'm unfamiliar with that.

[dra27]

Thanks, @jonahbeckford!

I didn't review the test suite since I'm unfamiliar with that

Some more detail for these changes. They're slightly better looked at commit-by-commit:

• 2e87370 (ld.conf-relative path interpretation): the change in test_ld_conf.ml reflects the new interpretations of some of the lines (e.g. . now being interpreted as libdir)
• a2e900a (Generate ld.conf using relative paths): this affects testRelocation.ml because ld.conf no longer contains absolute paths. Similarly, testLinkModes.ml is affected because after renaming the directory containing the installed compiler, ld.conf is now correctly processed by the runtime (since the . and ./stublibs entries in it are both still correct)
• 9062f60 (Load ld.conf from all possible places): this commit alters the runtime to load up to three ld.conf files, instead of one (i.e. $OCAMLLIB/ld.conf, $CAMLLIB/ld.conf and OCAML_STDLIB_DIR/ld.conf are now all loaded, if they exist). That necessarily affects the tests for shadowing in test_ld_conf.ml.
• 5b339f8 (Harden the parsing of ld.conf w.r.t. load and CRLF): makes the processing of ld.conf consistent w.r.t. CRLF, which consequently affects the tests in test_ld_conf.ml which showed the previous non-portable behaviour
• 1ddfabc (Use caml_parse_ld_conf in ocamlc): the implementations of the parsing in the dll.ml and dynlink.c differed in various corner cases, all of which were exercised in test_ld_conf.ml - the change therefore is the removal of all the special cases 🥳

[dra27]

Surfacing a brief discussion on caml-devel about a design-decision affecting this. Non-blank lines in ld.conf are each one of three kinds of path:

• Absolute (/usr/…, C:\Users\…, \\DC0\Home\…)
• Explicit-relative (., .., ./… or ../…)
• Implicit (foo, bar/…, etc.)

In OCaml today, only absolute directories have a sensible semantics, so the core idea in this PR is to add a sensible semantics to either or both of the other two lines (explicit-relative, implicit), on the basis that they're presently not in use.

A complete test for an absolute path is quite difficult to get right on Windows, which is necessarily needed to determine if a line is an implicit path. However, the explicit-relative test is straightforward, which is the main reason I opted just to interpret explicit-relative directories with this new mechanism (it is also the case that opam-bin uses a +dir notation in its patches for ld.conf, and it meant that this would not interfere with that - although that's a relatively minor concern, as I'm sure opam-bin will stop patching the compiler when there's upstream support for relocation anyway).

In this PR right now, we have explicit-relative paths interpreted relative to where ld.conf is found. That means, in opam, the content of the file will be:

../stublibs
./stublibs
.

There are two changes either or both of which could be made. Firstly, the paths could be instead be interpreted relative to --prefix (that requires a little bit of rework to pull in --with-relative-libdir from #14244 or something like it). The file would then be:

./lib/stublibs
./lib/ocaml/stublibs
./lib/ocaml

The second change would be to recognise both explicit-relative and implicit in the same way, allowing

lib/stublibs
lib/ocaml/stublibs
lib/ocaml

The decision in this PR has a small bearing on ocamlfind (but as far as I can tell for nothing else in the ecosystem), as findlib.conf needs to record destdir and path. In my draft for that at dra27/ocamlfind#2, . is being interpreted relative to findlib.conf, giving:

destdir="."
path="./ocaml:."

but which could therefore be:

destdir="./lib" # Or just destdir="lib"
path="./lib/ocaml:./lib" # Or just path="lib/ocaml:lib"

[hhugo]

I've tried to make the logic easier to understand. I hope I've managed to preserve all aspect.

if Filename.is_relative line && not (Filename.is_implicit line)
then
  match line with
  | "." -> Config.standard_library
  | _ -> Filename.concat Config.standard_library line
else line

[dra27]

(note that this function gets completely deleted in 1ddfabc, where Dll co-opts the C implementation).

It's different for line = "./stublibs", I think - the previous version (and the version in C) produces <libdir>/stublibs, but the newer version produces <libdir>/./stublibs. It's pedantic, but I was trying where possible to avoid the stray /./ bits being introduced, given that the default entries are . and ./stublibs!

[jonahbeckford]

Some more detail for these changes. They're slightly better looked at commit-by-commit:
...

Thanks, that was helpful for me to review the testsuite. It all looks good to me.

[dra27]

Rebased - and with one additional change inspired by #14244 (comment). I've renamed --with-stublibs to the slightly clearer (I think) --with-additional-stublibsdir (stublibsdir vs libdir, etc.) and then made ../stublibs the default for that, which also makes the opam file neater. As in #14244, the changes are in extra "Review" commits

[dra27]

Another follow-up from #14246 (comment). The last commit in this series corrected the over-installation of the native stub libraries added in #11828. That's only a partial fix (they shouldn't be built either) and it doesn't belong in this PR.