internals: optimize std::unordered_map internals with noexcept

[Skylion007]

Description

• Updates various STL specialization function with noexcept. This allows STL to optimize the hashing in unordered_maps. Otherwise, these has values are stored in the map instead of recomputed which adds memory and hurts locality. Surprisingly, this change is fully ABI compatible too since the noexcept signature is erased before it's mangled.

More detailed discussion here: https://quuxplusone.github.io/blog/2024/08/16/libstdcxx-noexcept-hash/

Regardless, the STL contract here says they should be marked as noexcept regardless of perf differences so good practice to do so.

Suggested changelog entry:

• optimize pybind11 internal hash functions for STL maps.

[rwgk]

Hi @Skylion007, below is a Cursor (GPT-5.2 Codex Extra High) generated assessment regarding ABI compatibility. Could you please take a look?

This goes so deep, I cannot be sure about it without further experimentation. My current thinking: To be safe, defer merging this PR until we bump the internals version for #5879 and #5887; in other words, add this PR to the "deferred until next internal version bump" group. — Which could be very soon. I'm still looking a bit into the subinterpreter leak issue (see comments under #5958), but at this minute I'm thinking we're in good shape for making a release with what we have on master. Right after we could bump the internals version and merge the whole group.

---

PR 5960 ABI Compatibility Assessment (pybind11)

Executive summary

• The PR adds noexcept to custom hash/equality functors in
  include/pybind11/detail/internals.h.
• On libstdc++ (GCC 13 headers in this environment), unordered_map and
  unordered_set decide whether to cache hash codes based on whether the hash
  functor is noexcept. That choice changes node layout.
• Because pybind11::detail::internals stores these unordered containers and
  is shared across extension modules, mixing old and new builds becomes ABI-
  unsafe.
• Therefore the claim "fully ABI compatible" is not correct for libstdc++.

What changed in the PR

The diff adds noexcept to these functors:

• type_hash::operator() used by type_map (a std::unordered_map alias).
• type_equal_to::operator() used by type_map.
• override_hash::operator() used by std::unordered_set for
  inactive_override_cache.

These are in include/pybind11/detail/internals.h.

libstdc++ ground truth (GCC 13 headers)

Cache policy depends on noexcept

unordered_map/unordered_set use __cache_default to choose whether to cache
hash codes, which is part of the container's type instantiation:

From /usr/include/c++/13/bits/hashtable.h:

template<typename _Tp, typename _Hash>
  using __cache_default
    =  __not_<__and_<
           __is_fast_hash<_Hash>,
           __is_nothrow_invocable<const _Hash&, const _Tp&>>>;

So if a hash is considered "fast" and is noexcept, the container does not
cache hash codes.

Custom hashers are treated as fast by default

From /usr/include/c++/13/bits/functional_hash.h:

template<typename _Hash>
  struct __is_fast_hash : public std::true_type { };

Unless a specialization exists, the default is true. This means user-defined
hash functors are treated as fast by default.

Cache policy changes node layout

From /usr/include/c++/13/bits/hashtable.h:

_Hash_node contains:
  - _Hash_node* _M_next
  - Tp          _M_value
  - size_t      _M_hash_code if cache_hash_code is true

So caching changes the node layout (extra size_t).

Where the cache policy is wired into unordered_map

From /usr/include/c++/13/bits/unordered_map.h:

template<typename _Key, typename _Tp, typename _Hash = hash<_Key>, ...,
         typename _Tr = __umap_traits<__cache_default<_Key, _Hash>::value>>
  using __umap_hashtable = _Hashtable<..., _Tr>;

The cache choice becomes part of the instantiated type.

Effect of adding noexcept in pybind11

Given the default __is_fast_hash == true for custom hashers:

• Before (no noexcept): __is_nothrow_invocable is false, so
  __cache_default is true -> hash codes cached -> larger node layout.
• After (with noexcept): __is_nothrow_invocable is true, so
  __cache_default is false -> no caching -> smaller node layout.

That means the unordered_map/unordered_set node layout changes as a result
of this PR, even though the hash functor type name is unchanged.

Why this matters for pybind11 ABI

pybind11::detail::internals stores unordered containers using these functors:

• type_map<type_info *> registered_types_cpp
• std::unordered_set<... , override_hash> inactive_override_cache
• plus other type_map instances in the same struct.

This internals object is shared across extension modules in the process via
PYBIND11_INTERNALS_ID capsule. If one extension is built with the old header
(cached hash nodes) and another with the new header (uncached hash nodes), they
will disagree about the container layout and behavior, which is an ABI hazard.

This is precisely the class of change that typically requires bumping
PYBIND11_INTERNALS_VERSION to prevent mixed-module usage.

Addressing the "noexcept is erased before mangling" claim

Even if noexcept does not change symbol mangling in a way that shows up in
shared library symbol names, it is still part of the function type and
critically affects template traits like std::is_nothrow_invocable, which are
used by the standard library to select container layout and behavior.

So the absence of a mangling change does not imply ABI compatibility for
templates instantiated in user code.

Conclusion

On libstdc++ (as shipped with GCC 13 headers here), the PR changes the
unordered_map/unordered_set node layout in pybind11 internals. This is an
ABI change for mixed extension modules and the claim of full ABI compatibility
is not correct. If this optimization is accepted, pybind11 should bump
PYBIND11_INTERNALS_VERSION (or otherwise prevent mixing builds).

Notes on libc++

I did not have libc++ headers available in this environment. libc++ also uses
traits to drive caching decisions (and generally respects noexcept for erase
and hashing paths), so it is plausible the same conclusion holds, but the
libstdc++ analysis above is already sufficient to refute the "fully ABI
compatible" claim for at least one standard library implementation.

[oremanj]

Yeah, I wasn't tracking that the container's node structure changes depending on whether the hash function is noexcept. So this is ABI incompatible, sorry for the misunderstanding!