Skip to content

Conversation

@vcjana
Copy link
Contributor

@vcjana vcjana commented Dec 10, 2025

Purpose

This PR demonstrates the canary dependency resolution bug.

What this PR does

  • Pins crc-fast = "1.4" in aws-smithy-checksums/Cargo.toml
  • Updates Cargo.lock to use crc-fast 1.4.0
  • Does NOT include the Cargo.lock deletion fix

Expected behavior

Canary should FAIL because crc-fast 1.4.0 causes SIGILL on ARM (awesomized/crc-fast-rust#14)

Actual behavior (bug)

Canary will PASS because it uses stale Cargo.lock files with crc-fast 1.6.0

Proof of bug

  1. This PR has crc-fast 1.4.0 in Cargo.lock
  2. Canary will use existing workspace Cargo.lock files
  3. Those files still have crc-fast 1.6.0 (not updated by PR)
  4. Canary tests pass with 1.6.0, missing the 1.4.0 issue

The fix

See commit 8ef68d5 on the crc-fast branch:

  • Deletes workspace Cargo.lock files before canary build
  • Forces cargo to regenerate from Cargo.toml
  • Canary now tests actual PR dependencies

Related

@vcjana vcjana requested review from a team as code owners December 10, 2025 10:32
@github-actions
Copy link

A new generated diff is ready to view.

A new doc preview is ready to view.

@vcjana vcjana changed the title [PROOF] Demonstrate canary bug: crc-fast 1.4 not caught Test crc-fast 1.4 to verify canary catches dependency issues Dec 10, 2025
The canary was using stale dependency versions from committed Cargo.lock
files instead of the versions specified in PRs. This caused the canary
to miss the crc-fast 1.4 SIGILL issue.

Solution: Delete workspace Cargo.lock files before running the canary to
force fresh dependency resolution. This ensures the canary tests the exact
dependency versions from the PR.

Fixes #3981
@vcjana
Copy link
Contributor Author

vcjana commented Dec 10, 2025

Update: Applied Cargo.lock deletion fix

All CIs passed in the previous run, proving the canary bug exists.

Now applied commit 330601c which deletes workspace Cargo.lock files before canary runs.

Expected result: Canary should now FAIL on ARM architectures because it will use crc-fast 1.4.0 from this PR (which causes SIGILL on ARM).

This will prove the fix works.

@github-actions
Copy link

A new generated diff is ready to view.

A new doc preview is ready to view.

@vcjana vcjana closed this Dec 10, 2025
@vcjana
Copy link
Contributor Author

vcjana commented Dec 10, 2025

Closing this - turns out we can't actually reproduce the crc-fast 1.4 issue in CI.

The problem is that crc-fast 1.4 only crashes on x86_64 CPUs that don't have AVX-512. GitHub Actions runners have AVX-512, and QEMU emulates it, so our tests pass even with the broken version.

That said, the Cargo.lock deletion fix (330601c) is still valid. It makes sure the canary actually tests the dependency versions from the PR instead of whatever's in the committed lock files. aws-sdk-rust's canary caught the crc-fast issue because they run on real AWS instances, some of which don't have AVX-512.

We just can't prove it would catch this specific bug in our CI environment. But the fix is still worth having to prevent similar issues.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants