feat: support for rustdoc mergeable cross-crate info

[weihanglo]

What does this PR try to resolve?

This is an unstable feature that we designed to fix several performance problems with the old system:

1. You couldn't easily build crate docs in hermetic environments. This doesn't matter for Cargo, but it was one of the original reasons to implement the feature.
2. We have to build all the doc resources in their final form at every step, instead of delaying slow parts (mostly the search index) until the end and only doing them once.
3. It requires rustdoc to take a lock at the end. This reduces available concurrency for generating docs.

Part of

• rustdoc: Tracking issue for mergeable cross-crate info rust#130676
• Tracking Issue for -Zrustdoc-mergeable-info #16306

How to test and review this PR?

Design decisions and questions:

• Doc parts are always stored in the new build dir layout: feat: support for rustdoc mergeable cross-crate info #16309 (comment)
• Doesn't take cargo clean into account yet: feat: support for rustdoc mergeable cross-crate info #16309 (comment)
• A new FileFlavor::DocParts is added: feat: support for rustdoc mergeable cross-crate info #16309 (comment)
• One doc-merge per target platform (and run in serial) feat: support for rustdoc mergeable cross-crate info #16309 (comment)
• A new status "Merging" is added. Example output:

       Documenting ...
       Documenting cargo v0.94.0 (/Users/frodo/cargo)
       Finished `dev` profile [unoptimized + debuginfo] target(s) in 29.68s
        Merging 300 docs for host
       Finished documentation merge in 2.74s
      Generated /Users/frodo/cargo/target/doc/cargo/index.html
  

• Who should garbage collect unneeded crate docs under mergeable CCI mode, rustdoc or cargo? (or don't, we shouldn't do any garbage collection?) feat: support for rustdoc mergeable cross-crate info #16309 (comment)
• .rustdoc_fingerprint.json now stores doc part file paths in previous builds.

Simple benchmark

• Tested against this PR on top of 15fde07
• rustdoc 1.93.0-nightly (1be6b13be 2025-11-26)
• macOS Sequoia 15.7.1
• Apple M1 Pro 2021
• 32GB RAM; 10 cores
• 300 crate doc merge
• Test steps:

   cargo check
   cargo doc
   cargo clean --doc && rm -rf target/debug/build/**/deps/*.json
   cargo doc -Zrustdoc-mergeable-info
   cargo clean --doc && rm -rf target/debug/build/**/deps/*.json

With -Zrustdoc-mergeable-info:

• <40s: ~30s documenting time + 3-5s merge time
• 76262 files, 970.5MiB total

Without -Zrustdoc-mergeable-info:

• ~500s: 8m10s - 8m30s
• 75242 files, 932.8MiB total

[rustbot]

r? @epage

rustbot has assigned @epage.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

[weihanglo]

cargo clean need to take care of this. I don't because there will be conflicts and I'd rather waiting for them merging.

[weihanglo]

We don't add --emit=toolchain-shared-resources here. See https://github.com/rust-lang/cargo/pull/16167/files#r2570359518.

[weihanglo]

A new intermediate build directory target/debug/doc-parts/. Doc parts of each doc unit will be put in a sub-directory of <crate>-<hash>/ for artifact isolation and rebuild detection purpose. Example full path to foo crate-info JSON file: target/debug/doc-parts/foo-12ab34cd/foo.json.

[epage]

How does this fit in with the effort to re-organize build-dir?

[weihanglo]

#16309 (comment)
It is now a shared directory for all invocation to achieve the additive nature of rustdoc output.

How does this fit in with the effort to re-organize build-dir?

Given rustdoc immediate artifacts are pretty tied to its final arifacts, maybe we should have a package local private build dir?

The other way is to have a <build-dir>/<workspace-hash>/docdeps so it is not shared with other workspaces.

[weihanglo]

One alternative: <build-dir>/docdeps/<crate-name>/<hash>/crate-info.json

So that new layout can still be a per-unit layout.

[weihanglo]

The latest commit 93ab48d always uses new layout for doc parts files on per-unit basis. It would look like <build-dir>/target/debug/build/foo-12ab34cd/deps/foo.json

The commits also starts storing information of previous rustdoc invocations in .rustdoc_fingerprint.json file.

[weihanglo]

• One doc-merge per target platform
• and run in serial

I feel like these design decisions are quite straightforwards

• Cargo already organizes docs by target platforms.
• Multi-target is not common. If it becomes a performance issue, we can simply run in parallel.

[epage]

Putting this in the build graph would also solve this.

[rustbot]

This PR was rebased onto a different master commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

[weihanglo]

This is ready for another round of review.

View changes since this review

[weihanglo]

^{commented on a random file to make it a thread}

cc @notriddle

I've discovered some panics in rustdoc. Here are reproducible steps (you need to build cargo from source from this PR):

rustup run nightly target/release/cargo check -Zrustdoc-mergeable-info -p cargo

# `--merge=finalize` took 3secs in this call
rustup run nightly target/release/cargo doc -Zrustdoc-mergeable-info -p cargo

# `--merge=finalize` took 1mins in this call
rustup run nightly target/release/cargo doc -Zrustdoc-mergeable-info -p cargo-utils-schemas

# `rustdoc --merge=none` failed on random crates
rustup run nightly target/release/cargo doc -Zrustdoc-mergeable-info -p cargo

Rustdoc version:

rustdoc 1.93.0-nightly (646a3f8c1 2025-12-02)

I feel like it is search index on disk are not prepared for reuse across multiuple --merge=finalize. The code panicked at https://github.com/rust-lang/rust/blob/83e49b75e7daf827e4390ae0ccbcb0d0e2c96493/src/librustdoc/html/render/search_index.rs#L1347.

Also, mind the 1min slow merge in the second cargo doc call.

click to see the log

Caused by:
  process didn't exit successfully: `/Users/user/.rustup/toolchains/nightly-aarch64-apple-darwin/bin/rustdoc --edition=2018 --crate-type lib --crate-name smallvec /Users/user/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/smallvec-1.15.1/src/lib.rs --cap-lints allow -o /Users/user/dev/cargo-master/target/doc --cfg 'feature="const_generics"' --cfg 'feature="write"' --check-cfg 'cfg(docsrs,test)' --check-cfg 'cfg(feature, values("arbitrary", "bincode", "const_generics", "const_new", "debugger_visualizer", "drain_filter", "drain_keep_rest", "impl_bincode", "malloc_size_of", "may_dangle", "serde", "specialization", "union", "unty", "write"))' --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --diagnostic-width=177 --emit=invocation-specific -Zunstable-options --merge=none --parts-out-dir=/Users/user/dev/cargo-master/target/debug/build/smallvec-8f6fc8f61133a3c6/deps -C metadata=39fc2b80d6a50c9c -L dependency=/Users/user/dev/cargo-master/target/debug/deps --crate-version 1.15.1` (exit status: 101)

thread 'rustc' (107763199) panicked at src/librustdoc/html/render/search_index.rs:1347:55:
index out of bounds: the len is 7516184 but the index is 7526966
stack backtrace:
   0:        0x10f1930ac - <<std[d4ca2ac3ebc8103a]::sys::backtrace::BacktraceLock>::print::DisplayBacktrace as core[b5ad265dbd7a4861]::fmt::Display>::fmt
   1:        0x10c65a8d4 - core[b5ad265dbd7a4861]::fmt::write
   2:        0x10f1a9f78 - <std[d4ca2ac3ebc8103a]::sys::stdio::unix::Stderr as std[d4ca2ac3ebc8103a]::io::Write>::write_fmt
   3:        0x10f16a658 - std[d4ca2ac3ebc8103a]::panicking::default_hook::{closure#0}
   4:        0x10f185cdc - std[d4ca2ac3ebc8103a]::panicking::default_hook
   5:        0x10d1c1720 - std[d4ca2ac3ebc8103a]::panicking::update_hook::<alloc[1bda16a0b8fec711]::boxed::Box<rustc_driver_impl[78050a7ccd3dc9d6]::install_ice_hook::{closure#1}>>::{closure#0}
   6:        0x10f186040 - std[d4ca2ac3ebc8103a]::panicking::panic_with_hook
   7:        0x10f16a700 - std[d4ca2ac3ebc8103a]::panicking::panic_handler::{closure#0}
   8:        0x10f15f43c - std[d4ca2ac3ebc8103a]::sys::backtrace::__rust_end_short_backtrace::<std[d4ca2ac3ebc8103a]::panicking::panic_handler::{closure#0}, !>
   9:        0x10f16bd18 - __rustc[2b496e5da265df0c]::rust_begin_unwind
  10:        0x111edf55c - core[b5ad265dbd7a4861]::panicking::panic_fmt
  11:        0x111edf370 - core[b5ad265dbd7a4861]::panicking::panic_bounds_check
  12:        0x1008406a0 - rustdoc[7ccbdf937b02d493]::html::render::write_shared::write_shared
  13:        0x100783690 - <rustdoc[7ccbdf937b02d493]::html::render::context::Context>::init
  14:        0x100717c64 - rustdoc[7ccbdf937b02d493]::main_args::{closure#2}::{closure#0}
  15:        0x10070e5bc - rustc_interface[2208a15c44c71837]::interface::run_compiler::<(), rustdoc[7ccbdf937b02d493]::main_args::{closure#2}>::{closure#1}
  16:        0x10067df54 - std[d4ca2ac3ebc8103a]::sys::backtrace::__rust_begin_short_backtrace::<rustc_interface[2208a15c44c71837]::util::run_in_thread_with_globals<rustc_interface[2208a15c44c71837]::util::run_in_thread_pool_with_globals<rustc_interface[2208a15c44c71837]::interface::run_compiler<(), rustdoc[7ccbdf937b02d493]::main_args::{closure#2}>::{closure#1}, ()>::{closure#0}, ()>::{closure#0}::{closure#0}, ()>
  17:        0x10073be30 - <std[d4ca2ac3ebc8103a]::thread::lifecycle::spawn_unchecked<rustc_interface[2208a15c44c71837]::util::run_in_thread_with_globals<rustc_interface[2208a15c44c71837]::util::run_in_thread_pool_with_globals<rustc_interface[2208a15c44c71837]::interface::run_compiler<(), rustdoc[7ccbdf937b02d493]::main_args::{closure#2}>::{closure#1}, ()>::{closure#0}, ()>::{closure#0}::{closure#0}, ()>::{closure#1} as core[b5ad265dbd7a4861]::ops::function::FnOnce<()>>::call_once::{shim:vtable#0}
  18:        0x10f190784 - <std[d4ca2ac3ebc8103a]::sys::thread::unix::Thread>::new::thread_start
  19:        0x1828ffbc8 - __pthread_cond_wait

[notriddle]

rust-lang/rust#149700

[epage]

non-blocker: technically there is a race condition between the build and this but the window is small and likely the users fault if they have a cargo clean running in parallel, blocked on the build.

This would be solved by us moving the merge into the build graph

[weihanglo]

Yeah that was noticed but agreed it is user's fault. Tracked in the issue