Changes

New module Misc.UString to work with UTF-8-encoded Unicode identifiers

Facilities are provided for NFC normalization, capitalization /
uncapitalization, and checking identifier validity.

Support is currently limited to Latin-9 letters.
 (commit: b9dfb9f)

Support non-ASCII letters in identifiers

We use the new Misc.UString module to check that the letters are allowed
and to normalize their Unicode representation.

The lexing of labels was changed so that labels that start with an
uppercase are accepted by the lexer regexps, then rejected by the
action.  This works better with non-ASCII letters and produces a nicer
error message.
 (commit: 8c16a8e)

Use `Misc.UString.{un,}capitalize` to connect file names with module names

Instead of `String.{un,}capitalize_ascii` like before.

This should support compilation units whose module names start with
or contain non-ASCII letters.
 (commit: 883c21d)

Test Latin-9 letters in identifiers

Both NFC and NFD representations are used in this file
(hoping Git does not change this).
 (commit: 014a1bd)

Test Latin-9 letters in file names and compilation unit names
 (commit: 165b163)

eszett has an uppercase form
 (commit: a9ff117)

Update dependencies and bow to check-typo
 (commit: a9cbe59)

support for unicode raw identifiers
 (commit: 657367e)

update Changes
 (commit: fafbdd7)

Unicode filename
 (commit: 8607b84)

Reject invalid unicode encoding in module names

We reject invalid utf8 encoding and strings containing the replacement
character U+FFFD in identifiers and output targets.

Similarly, we ignore existing artifacts with encoding errors whenever we are
doing an artifact look-up in load paths.
 (commit: 6263013)

unicode support, extend test and remove a copy of "is_valid_identifier"
 (commit: 2db2632)

utf8: text parsetree printer
 (commit: 7d39a93)

Support latin-9 lowercase in quoted string delimiters
 (commit: 314efa5)

Debugger lexer
 (commit: f3c661e)

ocamllex: support latin-9 in actions and comments
 (commit: 007a8ea)

ocamldoc: latin-9 lexer and UTF-8 default charset
 (commit: 23f634c)

Test NFD normalized unit names (#9)

The filename ça.ml is NFD normalized. The test source now the two
normalizations for each unit identifiers.
 (commit: 3897c58)

review: UString -> UIdent
 (commit: 718f098)

review: mention NFC in documentation
 (commit: 20feb00)

review: simplify lexer

The lexer now only has one merged case for raw and unicode identifiers
 (commit: b2bfbc0)

review: character set
 (commit: 1fffcc5)

review: lexer simplification
 (commit: 0beae74)

review: rename `modname_from_source` to `lax_modname_from_source`
 (commit: 5524d3d)

review: rename UIdent ->  Utf8_lexeme
 (commit: 457721d)

check-typo: tweaks for unicode file names
 (commit: 5333ecf)

more check-typo tweaks
 (commit: aca9b31)

review: merge check_label functions
 (commit: f068666)

Let CI's sanitizer job use clang's default version

This is to avoid having to maintain a hard-coded version
 (commit: 5d16740)

Build system: make sure -c appears as the last flag in compilation rules

It seems also more coherent to have the flag just before the name
of the file that is being compiled.
 (commit: c082877)

Build system: More clearly distinguish bytecode from native CFLAGS/CPPFLAGS
 (commit: 31cdf41)

Fix configuration for the Sun C compiler

The CFLAGS should not be included there, and -D_XPG6 should be in
preprocessor rather than compiler flags.
 (commit: 52edfea)

Deduplicate the pthread CFLAGS
 (commit: fd1cab4)

Use the `strip` command detected by libtool

Instead of using `strip` unconditionally to build `tmpheader.exe`, use
the `strip` command detected by `libtool` during configure so that it is
replaced with `:` when the command is absent and it becomes easy to
override it if need be
 (commit: 3bdfbfd)

Strip tmpheader.exe also on Windows

GNU strip can be called safely on binaries generated by cl as well as by
MinGW GCC (even if it doesn't produce a smaller executable for
cl-generated binaries) so invoke strip also on Windows so that MinGW
binaries are properly stripped
Tested with GNU strip 2.42
 (commit: 0d57f1a)

Remove duplicated comment end
 (commit: 0b3e8f0)

Introduce a new variable to avoid re-using a for loop iterator
 (commit: ab0bf44)

Change macros to reduce the scope of the for loop iterator
 (commit: c908e41)

Reduce the scope of the for loop iterator, avoid a shadowing warning

The function `caml_alloc_9` defines an `i` parameter. Compilers may
warn that defining local variables named `i` shadow this definition.
Pick the next available letter for the iterator.
 (commit: 9ae09f1)

Reduce the scope of for loop iterators
 (commit: b41196a)

Remove dangling function prototypes in header (#13301)

Co-authored-by: Gabriel Scherer <gabriel.scherer@gmail.com>
 (commit: 962cc42)

Use syntax for effect handlers in the manual page
 (commit: a1409db)

minor HACKING.adoc fix
 (commit: a056f5a)

Changes entry
 (commit: e443814)

Remove "mean space overhead" computation, as it's broadly unused and leaked memory.
 (commit: 95f30eb)

Simplify CAMLalign

In our public headers, we're using either:
- C23 where `alignas` is a keyword;
- C++11 or later where `alignas` is also available;
- C11/C17 where `_Alignas` is available.
 (commit: ed2aef4)

Use C11 max_align_t for struct pool_block alignment

Explanations from Xavier Leroy at
https://github.com/ocaml/ocaml/pull/13139#discussion_r1625611990

- The target systems we care about are 64-bit architecture with alignment
  constraints <= 16 and malloc returning 16-aligned blocks. In this case, the
  `data` part of `struct pool_block` is naturally 16-aligned (because the two
  pointers before use 16 bytes), and nothing needs to be done. Aligning data
  using `max_align_t` should have no effect.

- For a 32-bit architecture with 16-byte alignment constraints and malloc
  returning 16-aligned blocks (e.g. Linux x86-32), aligning `data` to 16 seems
  preferable to me and can be achieved by using `max_align_t`.

- For a 32-bit architecture with 16-byte alignment constraints and malloc
  returning 8-aligned blocks (perhaps Windows 32 bits, not sure): no amount of
  alignment constraints in `struct pool_block` will give 16-aligned `data`
  fields, so you could just as well put no alignment constraints.

Unfortunately, MSVC C11 suppport is incomplete and doesn't define `max_align_t`.
- https://developercommunity.visualstudio.com/t/max_align_t-is-not-provided-by-stddefh/10299797
- https://developercommunity.visualstudio.com/t/stdc11-should-add-max-align-t-to-stddefh/1386891
 (commit: 20ff2cc)

Simplify fallback definition of max_align_t for MSVC

For C++, MSVC defines `using max_align_t = double`. LLVM's clang-cl
copies this.

It's unlikely that we need to carry a fallback implementation for
other compilers. If so, the following could be used:

    typedef struct {
      alignas(long long) long long ll;
      alignas(long double) long double ld;
    } max_align_t;

https://gcc.gnu.org/git/?p=gcc.git;a=blob;f=gcc/ginclude/stddef.h
https://github.com/llvm/llvm-project/blob/main/clang/lib/Headers/__stddef_max_align_t.h

https://en.cppreference.com/w/c/types/max_align_t
 (commit: 50bf119)

Remove magic debug number from struct pool_block
 (commit: 84a3773)

Simplify struct pool_block access and allocation
 (commit: 0e25650)

Use aligned alloc for struct pool_block on Windows

This ensures that the data field is always aligned to the best
boundary.
 (commit: 484e863)

Parallelise alldepend target
 (commit: f98f737)

Remove architecture-specific files from .depend
 (commit: 8a5566b)

Include the dependencies for all the emitters
 (commit: 3b09bfd)

Don't invalidate ocamlopt with make alldepend
 (commit: fae0e77)

Implement 'exp_is_nominal' on Parsetree

This function can be used to improve the type clash error message, which
uses the parsetree instead of the typedtree to avoid loosing some
information from the source code (eg. how constants are printed).

This function is used for the "this function ..." message, which can be
easily changed to use untyped expression.
 (commit: cdf5f3a)

Print short expressions in type error messages

Re-use the mechanism implemented for the "This function" message in
type clash errors to print the original expression if possible.

Type error messages now look like this:

    Error: The expression '43' has type int
           but an expression was expected of type int32

The argument 'unify_exp ~sexp' is made non-optional as this isn't too
intrusive and ensures the new error is implemented in more cases.
 (commit: 6def5cb)

More precise denomination for some expressions

Clarify type clash error messages by printing the kind of the reported
expression.

Currently implemented for expression that are likely 'nominal' but not
necessarily.

Example:

    The constant "42" has type ...

    This field has type ...

This is similar to the `Apply_non_function` error:

    The function "foo" has type ...

    This function has type ...
 (commit: 3c7b259)

Denomination for multi-keyword expressions

Add denominations for expressions like `for`, `while`, `if`, `match`,
etc.. that are often spanning several lines.

It acts as a confirmation that the error is about the containing
expression rather than a nested one as the printed code is sometimes
harder to read.

There's no test for some of them (`try`, `if`) but they'll become
easily reachable if the denomination is used in other messages.
 (commit: 1a63317)

Update Changes
 (commit: bc88b1b)

Add a missing `$(EXE)` extension (#13312)

The extension was unintentionally overwritten in cherry-pick commit
46c7122888c5f57b45d75098ca95f159a14420b8
 (commit: d0fbfc7)

restore caml_process_pending_actions_exn

I made a mistake when introducing caml_process_pending_actions_res (#13013), currently in trunk `caml_process_pending_actions_exn` is exported in signals.h but its implementation is removed.

This function is not CAML_INTERNALS, and Coq uses it.

To restore this function I reintroduced a helper function to transform `caml_result` values back into encoded exception (earlier #13013 designs had this). It is named `caml_result_get_encoded_exception`, in fail.h, CAML_INTERNALS-only.
 (commit: 65eaf62)

doc(Bigarray.int): add link to element kinds

Signed-off-by: Edwin Török <edwin@etorok.net>
 (commit: bf5ba7c)

doc(Bigarray.int): clarify that it refers to OCaml ints and NOT C int

This is mentioned in the 'Element kinds' documentation, but that is not
visible anywhere in LSP documentation completions.
It is worth emphasizing this, because getting this wrong may lead to
memory corruption.

Suggested-by: Florian Angeletti <<florian.angeletti@inria.fr>>
Signed-off-by: Edwin Török <edwin@etorok.net>
 (commit: 4208c8b)

doc(Bigarray.int): add changes entry

Signed-off-by: Edwin Török <edwin.torok@cloud.com>
 (commit: 6b7befd)

Fix typo "get_succ_x" instead of "get_x". (#13313)

 (commit: bbc7ce7)

Format_doc: preserve the type of Foo.report_error, add Foo.report_error_doc (#13311)

* Format_doc: preserve the type of Foo.report_error, add Foo.report_error_doc

The introduction of Format_doc changed the type of various
'reprot_error' functions in the compiler codebase. But this breaks
user code. An alternative proposed here is to keep 'report_error'
at the same type as before and introduce 'report_error_doc' in addition.
 (commit: fb010ad)

Testsuite: simplify the C# test: do not spell-out dynlink libraries

Before this commit, the test was adding dynlink.cmx?a to the
all_modules variable but this is actually not necessary because
the "include dynlink" directive already adds them as appropriate to
the libraries variable.

This has been spotted by @dra27 as it led to a test failure when
-linkall was (incorrectly) used because this led to some modules
being linked twice.
 (commit: 4ed6269)

Build system: introduce per-program variables to store link flags

This commit introduces build variables

  foo_{COMMON,BYTECODE,NATIVE}_LINKFLAGS

where foo is the name of a program to build.

These variables are initialised with empty strings (no flags) and
can then be overridden by each program which needs to.

Such variables allow more fine-grained control over flags so that they
do not propagate when put in target-specific variables.

It may be the case that these variables become unnecessary once we
use a version of GNU make which is recent enough to support the
private keyword.
 (commit: d3741bc)

Build system: introduce the private OTHERLIBS variable

This variable lists the other libraries that get built by the
recursive makefiles.

At this point it has the same content as the OTHERLIBRARIES variable
but, when the other libraries will be built by the root Makefile
their names will be removed from this variable.

It is thus expected that this variable becomes empty and can be removed
once all the other libraries get built by the root Makefile.
 (commit: f00ff1c)

Let configure generate otherlibs/dynlink/dynlink_platform_intf.mli

otherlibs/dynlink/dynlink_platform_intf.mli is a copy of the
corresponding otherlibs/dynlink/dynlink_platform_intf.ml file.
Before this commit, the copy was done during the build. With this commit
the copy (more likely a symlink) is done by configure as is already
done for other files that need to be duplicated.
 (commit: c741526)

Makefile: remove duplicate target-specific variable definition
 (commit: c7d15af)

Merge otherlibs/dynlink/Makefile into the root Makefile
 (commit: 4c6a5d5)

otherlibs/systhreads/Makefile: Do not add -g to C compiler flags

This is already dealt with by the build system, meaning that,
before this commit, -g was actually occurring twice.
 (commit: 715821f)

otherlibs/systhreads/Makefile: Deduplicate C compiler flags

This is a leftover from 31cdf416280053dcd38351b7858e818b23110779,
(part of #12589).
 (commit: d0590d8)

sanitizer(UBSAN): really enable UBSAN

The clang-14 docs are not clear about this, but the latest docs say:
> -fsanitize-trap= and -fsanitize-recover= are a no-op in the absence of a -fsanitize= option. There is no unused command line option warning.

Avoid this trap and specify `-fsanitize=$ubsan` too.

Signed-off-by: Edwin Török <edwin.torok@cloud.com>
 (commit: f0b65a6)

sanitizers(UBSAN): fix linker warning

Apparently >-O0 needs to be supplied when linking too:
```
clang: warning: the object size sanitizer has no effect at -O0, but is explicitly enabled: -fsanitize=bool,builtin,bounds,enum,nonnull-attribute,nullability,object-size,pointer-overflow,returns-nonnull-attribute,shift-exponent,unreachable [-Winvalid-command-line-argument]
```

Signed-off-by: Edwin Török <edwin.torok@cloud.com>
 (commit: e1108ab)

sanitizers: move flag definitions

Move compiler flag definitions to shell variables.
This will enable testing them.

No functional change yet.

Signed-off-by: Edwin Török <edwin.torok@cloud.com>
 (commit: d726e52)

sanitizers: add a test that ASAN/UBSAN works

Compile 2 small test programs, and check expected output.

Signed-off-by: Edwin Török <edwin.torok@cloud.com>
 (commit: 40b917d)

sanitizer(UBSAN): switch to printing instead of trap

Still stop on first error, but instead of dumping core print a message,
and stacktrace, like ASAN.

This may be more convenient when used in a CI than a coredump.

Signed-off-by: Edwin Török <edwin.torok@cloud.com>
 (commit: 24a4657)

sanitizers: add -g flag

For better stacktraces.

Signed-off-by: Edwin Török <edwin.torok@cloud.com>
 (commit: 4bd8350)

sanitizers: future proof by using -Og instead of -o1

This is currently equivalent to -O1 on clang, but may improve debuggability in the future compared to -O1.

GCC already disables certain optimizations at -Og, and recommends it when using the sanitizers:
```
To get more accurate stack traces, it is possible to use options such as -O0, -O1, or -Og (which, for instance, prevent  most  function  inlining)
```

(there are also other flag recommendations, but I think they are only relevant if you want to use -O2 or higher,
 they wouldn't be active at -O1/-Og)

Also don't use -O0. Even for pure debugging, the manual recommends -Og:
```
It is a better choice than -O0 for producing debuggable code because some compiler passes that collect debug information are disabled at -O0
```

Signed-off-by: Edwin Török <edwin.torok@cloud.com>
 (commit: 814c007)