Changes

Xcode 16 / macOS 15

Add --with-stublibs configure option

--with-stublibs is intended for use in opam and allows an additional
directory to be added at the top of $(ocamlc -where)/ld.conf.

Don't add a double-separator when locating ld.conf

When calculating the full path for ld.conf, the runtime unconditionally
concatenated "/ld.conf". This is harmless when the separators appear in
the middle of a path ("/usr/local/lib/ocaml//ld.conf" is equivalent to
the version with only single slashes), but it is technically incorrect
for two corners cases with OCAMLLIB and CAMLLIB:

- if either is explicitly set to "/" then "//ld.conf" is _not_ the same
  file as "/ld.conf". This is mildly relevant on Windows and Cygwin
  where the two initial slashes (including as "\/" for native Windows)
  will be interpreted as a UNC path
- if either is explicitly blank, then "ld.conf" (i.e. ld.conf in the
  current directory) is a less illogical file to open than "/ld.conf"

ld.conf-relative path interpretation

Explicit relative paths in ld.conf are now interpreted relative to the
directory containing ld.conf.

Generate ld.conf using relative paths

Load ld.conf from all possible places

Before, the first ld.conf found from $OCAMLLIB, $CAMLLIB or the
preconfigured standard library location was loaded. Now all of these
are loaded.

Remove caml_get_stdlib_location

The function was only ever added to share the logic between dynlink.c
and startup_byt.c - now that dynlink.c doesn't require it, move the
function to startup_byt.c and make it internal again.

Harden the parsing of ld.conf w.r.t. load and CRLF

Use caml_parse_ld_conf in ocamlc

Eliminate the need for two implementations of the parsing logic for
ld.conf by sharing the C implementation (which must exist, since it's
part of bytecode startup) with the bytecode compiler, replacing
Dll.ld_conf_contents

Tidy installation of static builds

- $libdir/stublibs is no longer created for a --disable-shared build
- When $libdir/stublibs is not created, it is also not added to ld.conf

Fix the detection of Cygwin-like build environments

Both Cygwin and MSYS2 are now consistently detected on MSYS2. In
particular, this means that ./configure --prefix $PWD/install and
similar will cause the prefix to be correctly translated to a Windows
path, as already happens on Cygwin.

Preserve backslashes in --prefix

Previously, the --prefix argument was always normalised with cygpath -m
which meant that regardless of the argument, the paths used in the
compiler would always use slashes.

This behaviour is preserved if a slash is detected in the argument, i.e.
the caller explicitly uses mixed notation (e.g. `--prefix=C:/Prefix` or
`--prefix $PWD/install`). In particular, it means that a Cygwin-style
path will be correctly converted to a Windows-style path.

If the path uses backslashes, then it is still converted to use forward
slashes for the installation commands, but the backslashes are otherwise
preserved and used within the build itself.

Harden startup of -custom executables

By default, ocamlrun first tries to resolve argv[0] to determine where
the bytecode image is and then tries opening the executable image
itself. This is obviously correct for ocamlrun, when being called using
a shebang or executable header, but it's not correct for -custom
executables where we _know_ that the bytecode image should be with the
executable. To achieve this, a new mode is added to
caml_byte_program_mode (and the existing ones renamed) such that
caml_byte_program_mode is now STANDARD (for ocamlrun - the existing
behaviour), APPENDED (for -custom executables - the new behaviour) and
EMBEDDED (for -output-complete-exe/-output-obj - the original use of
it).

The mode is also set directly by the linker, rather than having a
default in libcamlrun which is then overridden by the startup code for
-output-complete-exe.

In the new APPENDED mode, if caml_executable_name is implemented (i.e.
it returns a string) then this file _must_ contain the bytecode image
and no other mechanisms are used. On platforms where
caml_executable_name is not implemented, APPENDED falls back to STANDARD
for compatibility.

Technically, this stops an argv[0] injection attack on setuid/setgid
-custom bytecode executables, although setuid should be used with
-output-complete-exe, if at all.

Interpret . in runtime-launch-info

The runtime-launch-info file includes the location of the binary
directory. The compiler is extended so that . refers to the directory of
the compiler binary.

Add --with-relative-libdir configure option

--with-relative-libdir can be used instead of --libdir to specify a
directory relative to --bindir which is used by the compiler to compute
the location of the Standard Library.

Add caml_standard_library_default to the runtime

Previously, the bytecode runtime just use OCAML_STDLIB_DIR from
build_config.h. This value is now stored once in stdlib.o as
caml_standard_library_default.

Add -set-runtime-default

Allows the default location used by the bytecode runtime for the
Standard Library to be overridden when creating bytecode executables.

Add %standard_library_default

Bootstrap

Use %standard_library_default in Config

Allow libdir to be found relative to bindir

When configured with --with-relative-libdir, the runtime uses the
directory of the executable to determine the location of the Standard
Library. Thus, ocamlrun and the compilers look for ../lib/ocaml by
default.

This is implemented by changing caml_standard_library_default to be a
relative path, and then computing the actual value at startup (for
bytecode) and when queried (for native, since it is only ever used if
Dynlink is being used).

Executables (and objects) produced by the compiler always have an
absolute value of caml_standard_library_default. ocamlc.opt and
ocamlopt.opt are built using -set-global-string to force
caml_standard_library_default to stay a relative value.

Update the installation test for enable-relative

Test --with-relative-libdir in CI

NFPR: fully test --with-relative-libdir in precheck

Use as directly on Cygwin, as on Linux

Detect but ignore -fdebug-prefix-map on mingw-w64

mingw-w64 is based on GCC, so supports -fdebug-prefix-map, but the test
for it is skipped in configure. The test is no longer skipped (which
means that Config.c_has_debug_prefix_map returns true) but the flag is
still explicitly not used by the compilers (as before).

Increase reproducibility of relative artefacts

Typo in stdlib/Makefile

Affects forced recompilation of runtime-launch-info

Reduce tmpheader.exe to 4-5KiB on Windows

stdlib/headernt.c was adapted in OCaml 3.00 to reduce its size by
avoiding the use of the CRT and using Windows API functions directly
(this is a well-studied trick on Windows, principally as a puzzle for
producing tiny binaries).

This got "regressed" slightly in OCaml 4.06, in the complex introduction
of wide character support for Windows, as the mingw-w64 incantation
required was unclear, so the entry point was changed to wmain, and the
size of the header increased.

By switching from wcslen (a CRT function) to lstrlen (a Win32 API
function), headernt.c again only requires kernel32.dll.

Additional flags are added for both ld (mingw-w64) and link (MSVC) to
squeeze every last byte out of tmpheader.exe. The MSVC version of the
header is once again no longer passed through strip, as this was found
to be corrupting the executable (and had never been reducing its size
anyway).

Stop using Caml_inline and Camlnoret in headernt.c

headernt.c will never be compiled with a C++ compiler, so _Noreturn can
be used directly, and modern C compilers are sufficiently intelligent to
work out when to inline!

Remove unnecessary guards around unistd.h

Already updated to remove the actual test in s.h, since XPG1 (1985)
required it and it is therefore part of the Single Unix
Specification (1992), but the _WIN32 guard and the loading of s.h are
unnecessary.

Use PATH_MAX instead of MAXPATHLEN

Assume S_ISREG and SEEK_END

Defined in sys/stat.h and unistd.h respectively by the Single Unix
Specification.

Move O_BINARY definition to "header" section

O_BINARY is added for maximum Cygwin compatibility, but it's not a Posix
flag.

Make caml/exec.h independent

Have exec.h include <stdint.h> itself. The bytecode executable header
now only depends on exec.h.

Re-order some header functions (no-op)

Share error path in headernt.c

Unify implementation of read_size

Modernisations applied in headernt.c but not applied to header.c - use
uint32_t rather than unsigned long

headernt.c: inline run_runtime into main

Check for errors from lseek in header.c

Refactor error display code

header.c: close bytecode image before exec

Leaking fd

Tweak the signature of read_runtime_path

Use execvp in header.c (as in headernt.c)

Brings the behaviour of the two headers into line with each other - if
RNTM is ocamlrun on Unix, ocamlrun will now be sought in PATH.

headernt: make read_runtime_path more CRTish

Missing error case in headernt.c

Unify the common parts of header.c and headernt.c

Merge stdlib/header{,nt}.c

Add -launch-method to ocamlc

When linking a normal bytecode executable, allows an explicit selection
of either the executable or shebang header, regardless of the value in
runtime-launch-info.

Test both bytecode header methods

Determine RuntimeID values in configure

Put Bytecode RuntimeID in runtime-launch-info

Bootstrap

Mangle the bytecode runtime names

Build suffixed shared runtimes

New names for libcamlrun_shared.so and libasmrun_shared.so without the
_shared suffix and using the target triplet and runtime ID. Both ocamlc
and ocamlopt explicitly recognise `-runtime-variant _shared` and select
the correct name.

Symbolic links for libcamlrun_shared.so and libasmrun_shared.so to allow
any C programs which linked against the the output of `-output-obj` to
continue to work.

Add Zinc RuntimeID Mask for portability

Compute the Zinc runtime ID in the linker

WIP work towards the use int31 bit (not completed) and the
additional_runtimes setting is present, although it's to be removed.

Bootstrap

Remove Bytecode RuntimeID in runtime-launch-info

Add runtime suffixes to bytecode stub libraries

ocamlc -dllib-suffixed appends the runtime's host triplet and bytecode
runtime ID to the supplied name when searching for the DLL, and records
the base name only in .cma / executable files.

ocamlmklib -suffixed instructs ocamlmklib to use -dllib-suffixed when
generating .cma files instead of -dllib.

The effect is that stub libraries built this way have names which will
be unique for a given configuration of OCaml and so will be ignored by
other runtimes.

Make Windows header absolute (as on Unix)

Add -runtime-search to ocamlc

-runtime-search {disable|enable|always} adds new features to the
launcher used for bytecode executables which do not embed their own
runtime. By default, the header continues to behave as before - the
launcher will attempt to start the runtime using the absolute path which
the compiler was configured with.

The new search mode will then search for the runtime first in the
directory containing the running executable and then in PATH.

Tweak the type

Add --enable-runtime-search[-target] options

--enable-runtime-search controls the -runtime-search setting used to
build the compiler's own bytecode executables;
--enable-runtime-search-target controls the default value of
-runtime-search that ocamlc itself uses.

Update the installation test for camlheader-search

Use Config instead of runtime-launch-info

Update the test

Test --enable-runtime-search in CI

Compute the Zinc runtime ID

WIP work towards the use int31 bit (not completed) and the
additional_runtimes setting is present, although it's to be removed.

Bootstrap

NFPR: fully test --enable-runtime-search in precheck

Move caml_search_dll_in_path to dynlink.c

Eliminates code duplication between unix.c and win32.c

header.c: remove unnecessary Cygwin workaround

This is no longer required (nor does it work). Cygwin 1.5.20 (July 2006)
added the transparent_exe option to the CYGWIN environment variable which
made open behave in the same way as stat. Cygwin 1.7.1 (December 2009 and,
despite the version number, the first release of Cygwin 1.7) made this
behaviour default (and removed the ability to turn it off).

Remove caml_cds_file

OCaml 4.x global which should have been removed with OCaml 5.0

Simplify the interface of caml_attempt_open

Previously, caml_attempt_open received a pointer to the string and
passed this string to caml_search_exe_in_path. This function allocates a
fresh string which was then assigned to that pointer.

This interface was a little cryptic to follow - in particular, there
were several scenarios in which the result was not being freed. It's
also not entirely clear that caml_attempt_open involved a PATH-search,
which appears to have been applied in error when CAML_DEBUG_FILE was
added.

header.c: handle parasitic case of argc < 1

header.c: malloc the result of searchpath

Move searchpath from header.c into the runtime

Merge the path-searching code from stdlib/header.c with the runtime's
version.

Use caml_executable_name in header.c

Don't check for shebang scripts on Windows

Bytecode startup includes a check which has no place on Windows.

Preserve argv[0] on Unix in the executable header

Makes the behaviour of the Unix and Windows versions of the header
equivalent, in particular it means that if argv[0] doesn't describe the
executable, more things fail for Cygwin.

Harden the cloexec.ml test on Windows

On slower machines, it's possible for the main part of the test (in
fdstatus_main.ml) to complete before the cloexec.ml has actually
terminated as part of `Unix.execv` (this is an artefact of how the
underlying CRT `execv` call is implemented on Windows).

Strictly on native Windows, cloexec.ml creates an additional dummy file
and takes a write lock on it (with `Unix.lockf`). Before running, if
this file exists, fdstatus_main.ml will also try to take a write lock on
it, providing a way of fdstatus_main.ml being sure that its ancestor has
fully terminated without actually having to know its PID.

Possible alternate approach...

... and the bug-fix which goes with it

Fix STARTUPINFO structure in stdlib/header.c

If a CRT application (including, therefore, another OCaml program),
exec's a bytecode program which uses the executable header, the
cbReserved2 and lpReserved2 fields of the STARTUPINFO structure are
quietly used by the CRT to pass handle information about open fds.

The status checker in the cloexec.ml test has C stubs, which causes
ocamltest to compile the bytecode version with -custom. The test is
split to have an additional intermediate pure OCaml program which simply
repeats either the Unix.execv or Unix.create_process call with the
Sys.argv it was applied. In bytecode, ocamltest compiles this program
just with -use-runtime, which means it uses the executable header on
Windows.

Without the change to header.c in this commit, that test begins to fail
in bytecode on Windows, because the CRT information about inherited
handles is not passed on to the status checker (the HANDLE values will
have been inherited, as that's a kernel function, but the CRT structures
regarding the fds are not initialised, which is what the checker than
looks at).

The fix here simplifies the code considerably - rather than initialising
a fresh STARTUPINFO structure, we simply use GetStartupInfo to retrieve
the one which was used to create the process itself and pass that to
CreateProcess - cbReserved2 and lpReserved2 are therefore passed on, and
the cloexec.ml test passes again.

Share image fd between header.c and startup_byt.c

The bytecode executable launcher (stdlib/header.c) can _only_ invoke
ocamlrun after it has opened itself in order to find out the name of the
runtime to execute. However, once ocamlrun is exec'd, the knowledge of
this file was previously lost and if it could not be recovered from
argv[0], then execution fails.

This new approach, for both Windows and Unix, instead keeps the fd for
the bytecode image open and passes its number to ocamlrun as
__OCAML_EXEC_FD in the environment. ocamlrun detects this environment
variable and uses that fd to load the bytecode image. If this fails, the
runtime does _not_ fallback to any other mechanisms.

On Windows, it is possible to recover the filename from a HANDLE. It is
not portably possible to do this on Unix, so the filename which was
opened by the stdlib/header.c is instead appended to the environment
variable.

Add comments on the handling of blank env vars

Clarify in various places the behaviour of reading an environment
variable which is "Set But Null" (i.e. equal to the empty string as
opposed to unset)

Ignore empty TMPDIR in the Filename module

For Unix, the Filename module now treats TMPDIR being set to the
empty string, as if it were not set at all (which causes it to use /tmp
instead).

Ignore empty TMPDIR in ocamlyacc

If TMPDIR is set to the empty string, the Unix build of ocamlyacc now
defaults to /tmp (as it would if TMPDIR were not set at all).

The Windows default for when TMPDIR is not set is alredy the current
directory, so no change is required.

Propagate non-empty tmpdir in ocamlyacc

By making tmpdir explicitly "." in the rare corner-case of TMPDIR being
set to the empty string on Windows, it becomes unnecessary to keep
checking whether tmpdir is zero-length.

Ignore CAML_DEBUG_FILE if it's "Set But Null"

Ignore OCAMLPROF_DUMP if it's "Set But Null"

Use non-raising Sys.getenv_opt in win32unix

Use the non-raising Sys.getenv_opt internally in the Windows
implementation of the Unix module. In passing, treat the highly unlikely
corner case of COMSPEC being "Set But Null" as if COMSPEC were not set.

Ignore empty strings in Compmisc.set_from_env

Previously, invocations such as

OCAML_COLOR= ocamlopt

emitted warning 46 (bad-env-variable) which is not particularly helpful.
At present, Compmisc.set_from_env is used to implement OCAML_COLOR and
OCAML_ERROR_STYLE, neither of which attached significance to the empty
string. Compmisc.set_from_env is there changed to explictly ignore the
environment variable value if it is "".

Ignore empty environment variables in ocamltest

Most uses in ocamltest were via its safe_getenv function which already
made unset and empty equivalent. This generalises the underlying
getenv_with_default_value function always to return the default value if
the variable is _either_ unset or set to the empty string.

Ignore empty MAKE variable in ocamltest

Treat OCAML_BINANNOT_WITHENV as unset when empty

This environment is intended to be used in the context of
OCAML_BINANNOT_WITHENV=1 to enable it, so treat OCAML_BINANNOT_WITHENV=
as if it weren't set at all.

Harden processing of SOURCE_DATE_EPOCH in ocamldoc

Previously, running:

SOURCE_DATE_EPOCH= ocamldoc

resulted in an uncaught Failure "float_of_string" exception. The
processing of SOURCE_DATE_EPOCH is firstly hardened to cope with parsing
errors and then a one-time warning is displayed the first time it's
actually used (at present it's only required in Odoc_man).

Add some limited information about CAMLSIGPIPE

Ignore empty OCAML_RUNTIME_EVENTS_ variables

The handling of OCAML_RUNTIME_EVENTS_DIR was incorrect if the value as
"Set But Null" and the intention in the manual for
OCAML_RUNTIME_EVENTS_START and OCAML_RUNTIME_EVENTS_PRESERVE is clearly
that they should be set to non-empty values in order to trigger the
required effect.

All three variables are now ignored if they are set, but to an empty
string.

Ignore blank OCAMRUNPARAM, OCAMLLIB and CAMLLIB

Previously, if OCAMLRUNPARAM was "Set But Null", CAMLRUNPARAM was
completely ignored. Similarly, a "Set But Null" value for OCAMLLIB
caused CAMLLIB and subsequently the default location of the Standard
Library to be ignored. Especially for OCAMLLIB/CAMLLIB, this behaviour
is counter-intuitive, as it's highly unlikely to be useful.

Now, if OCAMLLIB is set, but to the empty string, then CAMLLIB is
checked and, more importantly, if it is either not set or also set to
the empty string then the default location of the Standard Library is
still used. For consistency, if OCAMLRUNPARAM is set, but to the empty string,
then CAMLRUNPARAM is read.

Ignore empty components in PATH-like things

POSIX recognises empty components in a PATH-like variable as meaning "."
(the current directory). This is reflected in the processing of
OCAMLTOP_INCLUDE_PATH, CAML_LD_LIBRARY_PATH and ld.conf where either a
blank component or a blank line is interpreted as "."

Somewhat confusingly, this processing is applied inconsistently between
Unix and Windows (it's confusing given that Windowsm more readily
includes the current working directory by default in PATH searches).

It also has the side-effect that a "Set But Null" environment variable
is interpreted as "." which counter-intuitively makes
CAML_LD_LIBRARY_PATH= ocamlrun add the current working directory to the
search path.

Blank lines and empty components of both OCAMLTOP_INCLUDE_PATH and
CAML_LD_LIBRARY_PATH are now ignored. The current working directory can
still be explicitly included, of course, by adding a "." entry/line
where required.

Fix C library options for win32unix

-lws2_32 and -ladvapi32 are already supplied by default, so they don't
need to be in unix.cma/unix.cmxa. However, they do need to be passed
when building unix.cmxs, and they were previously acquired via
unix.cmxa. Tweak the way LDOPTS is used in Makefile.otherlibs.common
(which now is only used for the unix library) so that it's correctly
passed to both ocamlopt and ocamlmklib.

Build and install threads.cmxs

Crucially, the corrects the flags used for creating a DLL on Windows,
allowing threads.cmxs to be loaded in ocamlnat.

Fix inclusion of libraries when partial linking

ld -r (certainly in GNU binutils) has an empty search path - co-opt the
MSVC search code and always resolve libraries when partial linking,
except this time _ignore_ the ones which are missing. This seems to fit
the rest of -output-complete-obj, given that the _standard_ C libraries
are also omitted (-lm, -lpthread, etc.)

Use entrypoint flexdll branch

Fixes loading unix.cmxs in Cygwin64

Revert "accept unknown options"

This reverts commit 31cbd26c16f81c8a273ad3ba988c49db6b0dbe38.

Stop installing and documenting config_main, etc.

Config_main and Config_boot are built to ensure in the build that
utils/config.generated.ml.in and utils/config.fixed.ml are kept in sync
(so that the next bootstrap doesn't unexpectedly break). However,
because these files were generated in the utils directory, they were
picked up both by the install recipe and also when generating API
documentation.

It's slightly hairy to remove the wildcards and use filter, because we
can easily end up with command lines which are too long (even on Unix),
so instead these two modules are now generated in utils/config/

Use clang-cl for flexdll support objects

Add %shared_libraries primitive

Exposes the value of SUPPORT_DYNAMIC_LINKING.

Bootstrap

Improve the "dynamic loading not supported" error

It is possible, especially when using Dynlink, to end up in the
situation where a bytecode runtime which doesn't support dynamic loading
is asked to load support DLLs (e.g. a bytecode image with a DLLS
section, or a cma archive passed to the toplevel/Dynlink which has a
non-empty lib_dllibs list).

Previously, the error message would refer to the name of the first DLL
being loaded and simply state that dynamic loading is not supported. The
confusing part is that typically this would refer to a DLL which is not
on the system.

Now, the bytecode linker only writes DLLS and DLPT when there are
entries to write in them, and the runtime, toplevel and Dynlink provide
a direct explanation that dynamic loading is needed, but is not
available. In particular, the error now refers to the file which is
being loaded (i.e. the bytecode executable or the .cma file) rather than
a .so file which doesn't exist.

Free tables after handing off to Dynlink

In the debug runtime, caml_prim_name_table remains for the lifetime of
the program, as it's used by instrtrace.c, but in normal operation, once
the list of primitives has been handed over to Dynlink, its no longer
required. In the normal runtime, it's now freed after this handover.

In passing, strings themselves are no longer duplicated, as the code
path is a good deal simpler than it used to be, and the bytecode section
itself can reliably be used as the underlying buffer for
caml_prim_name_table.

caml_shared_libs_path is only kept at all to be handed over to Dynlink.
It, along with the two underlying buffers for CAML_LD_LIBRARY_PATH and
ld.conf are freed after the call.

Move print_magic and print_config out of caml_params

They are only used by the bytecode runtime and can be made static to
startup_byt.

Preserve option R in caml_runtime_parameters

The setting for R was previous omitted in Sys.runtime_parameters, since
it was only processed directly by the Hashtbl module and not stored in
the runtime. Option R is now processed in caml_parase_ocamlrunparam and
stored to be accessed and updated via new primitives for the Hashtbl
module.

Co-authored-by: David Allsopp <david.allsopp@metastack.com>

Swap the V and v in caml_runtime_parameters

Consistency - options displayed in alphabetical order with the uppercase
letter appearing before the lowercase letter

Add -set-runtime-default to the compilers

When linking an executable, allows to set default to OCAMLRUNPARAM
values. This new OCAMLRUNPARAM string is accessible using the
"caml_executable_ocamlrunparam" symbol or is embedded in a bytecode
section.

Co-authored-by: David Allsopp <david.allsopp@metastack.com>

Test -set-runtime-default in test-in-prefix

The test programs in the run after the prefix has been renamed are
compiled with `-set-runtime-default R`, and the test program verifies
that Hashtbl.is_randomized returns the expected value.

Re-order initialisations in bytecode

In bytecode startup, defer allocating memory until after the bytecode
image has been loaded as far as possible.

Support c=1 mode in ocamlrun

The runtime's pooling mode has a slight Catch-22 problem for ocamlrun
when enabled using -set-runtime-default. Opening the bytecode file and
reading the ORUN section requires the memory subsystem.

In this revised version, caml_main in bytecode is particularly careful
to track exactly what will have been allocated prior to reading the ORUN
section and if ORUN requires the system to start pooling mode, the
runtime now takes temporary malloc'd copies of everything which has been
made so far so that it can be safely copied with a caml_stat_alloc
_after_ pooling mode has been enabled.