Inconsistent / intermittent coverage with SimpleCov 0.22.0 + parallel_tests after upgrading Rails 7.1 → 7.2

[sullyvannunes]

After upgrading a Rails app from 7.1.5.2 → 7.2.2.1 I started seeing coverage drop and fluctuate between runs when running the test suite in parallel. The project uses parallel_tests (parallel_rspec) and SimpleCov 0.22.0. Prior to the Rails bump the suite reported 100% coverage consistently; after the bump the coverage value is lower and different on each run.

I have tried the obvious fixes (moving SimpleCov.start earlier, per-worker command_name, collating results, forcing eager load during coverage runs) but the problem persists. Below are exact config snippets, how I run tests, what I tried and questions / debugging requests — any help or suggestions would be appreciated.

Environment / relevant gems

Rails: 7.2.2.1 (previously 7.1.5.2)
SimpleCov: 0.22.0
Test runner: rspec + parallel_tests

CI command used:

RAILS_ENV=test COVERAGE=1 bin/rails parallel:spec

[chrisgaraffa]

We're dealing with the same on Rails 8.0.2.1, SimpleCov 0.22.0 and minitest. A set of models simply don't get picked up when running the full test suite, though there are tests that pass for them. Tried eager loading, disabling parallel tests entirely, and everything @sullyvannunes said, with no improvement.

If I run all tests: COVERAGE=true bin/rails test
I get the same result across runs - same % coverage, same files reporting 0% coverage.

If I then run an individual test: COVERAGE=true bin/rails test tests/models/organization_test.rb
The coverage report looks pretty accurate across all files.

This is workable but awkward for running tests locally, but not ideal for a CI environment. Here's our test_helper.rb:

# typed: strict
# frozen_string_literal: true

if ENV["COVERAGE"]
  require "simplecov"
  require "simplecov-tailwindcss"

  # Enable result merging to combine coverage from all workers
  SimpleCov.use_merging(true)

  # Set initial command name before starting SimpleCov to ensure eager-loaded files
  # are tracked under the correct command name
  SimpleCov.command_name("Minitest")
  SimpleCov.external_at_exit = true
  SimpleCov.start("rails") do
    # track when conditional statements are executed
    SimpleCov.enable_coverage(:branch)

    # Track all app files to ensure they appear in coverage even if not loaded in every worker
    # This is necessary for parallel tests where files may not be loaded in all workers
    SimpleCov.track_files("app/**/*.rb")

    # removed add_filter and add_group calls for simplicity - but they aren't the issue.

    if ENV["CI"]
      SimpleCov.formatter(SimpleCov::Formatter::SimpleFormatter)
    else
      SimpleCov.formatter(
        SimpleCov::Formatter::MultiFormatter.new([
          SimpleCov::Formatter::SimpleFormatter,
          SimpleCov::Formatter::HTMLFormatter,
        ]),
      )
    end
  end

  # Set the minimum code coverage at which simple cov fails
  SimpleCov.minimum_coverage(60)
end

ENV["RAILS_ENV"] ||= "test"
# Use assertions for tests
ENV["MT_NO_EXPECTATIONS"] = "true"
require_relative "../config/environment"
require "rails/test_help"
require "minitest/rails"
require "minitest/reporters"
require "capybara/rails"
require "capybara/minitest"
require "webmock/minitest"

require_relative "helpers/assertion_helpers"
require_relative "helpers/myapp_test_helper"
require_relative "helpers/file_test_helpers"
require_relative "helpers/environment_test_helper"

module ActiveSupport
  class TestCase
    extend T::Sig
    extend ActiveRecord::TestFixtures::ClassMethods
    include ActionMailer::TestHelper

    include FactoryBot::Syntax::Methods
    include WebMock::API

    include AssertionHelpers
    include MyappTestHelper

    # Run tests in parallel with specified workers
    parallelize(workers: :number_of_processors)

    if ENV["COVERAGE"]
      parallelize_setup do |worker_id|
        # Update command name for this worker, preserving any existing name
        current_name = SimpleCov.command_name
        if current_name == "Minitest" || current_name.nil?
          SimpleCov.command_name("Minitest-#{worker_id}")
        else
          SimpleCov.command_name("#{current_name}-#{worker_id}")
        end
      end

      parallelize_teardown do
        SimpleCov.result
      end
    end

    before do
      Rails.cache.clear
      update_initial_fixture_state
    end

    sig { overridable.void }
    def update_initial_fixture_state
      # Clear fixtures before each test
      ActiveRecord::FixtureSet.reset_cache
      tables = ActiveRecord::Base.connection.tables - ["schema_migrations", "ar_internal_metadata"]
      tables.each do |table|
        ActiveRecord::Base.connection.execute("TRUNCATE #{table} CASCADE")
      end
    end

    after do
      Rails.cache.clear
    end
  end

  class TestCaseWithFixtures < TestCase
    extend T::Sig
    extend ActiveRecord::TestFixtures::ClassMethods

    # Setup all fixtures in test/fixtures/*.yml for all tests in alphabetical order.
    fixtures :all

    sig { override.void }
    def update_initial_fixture_state; end
  end
end