Skip to content

ORC Test Plan: Unit and Component Testing #96

New issue
New issue
Open
Open
ORC Test Plan: Unit and Component Testing#96
@fosterbrereton

Description

@fosterbrereton
fosterbrereton
opened on Oct 7, 2025

This document outlines a comprehensive plan to add unit and component-level testing to the ORC project using Google Test (gtest). The project currently has a robust integration test suite (orc_test) that validates end-to-end functionality by compiling source files, processing object files, and comparing detected ODR violations against expected results. This plan focuses on filling the gap with lower-level tests that verify individual components and modules in isolation.

Current Testing State

Existing Integration Tests

  • Location: test/battery/ directory
  • Framework: Google Test (gtest)
  • Test Runner: test/src/main.cpp and test/src/fixed_vector_tests.cpp
  • Coverage: End-to-end ODR violation detection scenarios
  • Test Types:
    • Byte size mismatches
    • Data member location conflicts
    • Vtable element location issues
    • Calling convention mismatches
    • Static/free function conflicts
    • Class/struct differences
    • Typedef issues
    • Inline attribute handling
    • Accessibility conflicts

Testing Infrastructure Already in Place

  • Google Test is already a project dependency (v1.14.0)
  • CMakeLists.txt includes gtest setup (lines 71-81)
  • orc_test executable exists with proper linking (lines 187-213)
  • CI/CD pipeline exists (.github/workflows/build-and-test.yml)

Testing Strategy

Unit Tests

Purpose: Test individual functions, classes, and methods in isolation

Scope:

  • Utility functions (string manipulation, demangle operations)
  • Data structures (fixed_vector, hash tables, memory pools)
  • DWARF parsing primitives
  • File format parsers (Mach-O, COFF, AR)
  • Attribute processing
  • Settings/configuration handling

Component Tests

Purpose: Test larger subsystems and their interactions

Scope:

  • Object file registry operations
  • DWARF structure processing
  • Symbol registration and lookup
  • ODR violation detection logic
  • Report generation
  • Async/parallel processing coordination

Implementation Plan

Phase 1: Infrastructure Setup (Weeks 1-2)

1.1 Directory Structure

Create a new test directory structure:

test/
├── battery/           # Existing integration tests
├── src/
│   ├── main.cpp      # Existing integration test runner
│   └── fixed_vector_tests.cpp  # Existing unit tests
├── unit/             # New: Unit tests
│   ├── CMakeLists.txt
│   ├── main.cpp
│   ├── str_tests.cpp
│   ├── hash_tests.cpp
│   ├── memory_tests.cpp
│   ├── dwarf_constants_tests.cpp
│   ├── settings_tests.cpp
│   └── demangle_tests.cpp
└── component/        # New: Component tests
    ├── CMakeLists.txt
    ├── main.cpp
    ├── object_file_registry_tests.cpp
    ├── dwarf_parser_tests.cpp
    ├── macho_parser_tests.cpp
    ├── coff_parser_tests.cpp
    ├── ar_parser_tests.cpp
    ├── odrv_detection_tests.cpp
    └── report_generation_tests.cpp

1.2 CMake Configuration Updates

Update CMakeLists.txt to add new test targets:

# After the existing orc_test target (around line 217)

# Unit Tests
file(GLOB UNIT_TEST_FILES CONFIGURE_DEPENDS ${PROJECT_SOURCE_DIR}/test/unit/*.cpp)
list(REMOVE_ITEM UNIT_TEST_FILES ${PROJECT_SOURCE_DIR}/test/unit/main.cpp)
add_executable(orc_unit_tests
    ${SRC_FILES}
    ${HEADER_FILES}
    ${PROJECT_SOURCE_DIR}/test/unit/main.cpp
    ${UNIT_TEST_FILES}
)
target_include_directories(orc_unit_tests
    PRIVATE
        ${PROJECT_SOURCE_DIR}/include
        ${PROJECT_SOURCE_DIR}/third_party/adobe-contract-checks/include
)
target_link_libraries(orc_unit_tests
    PRIVATE
        stlab::stlab
        TBB::tbb
        tomlplusplus::tomlplusplus
        Tracy::TracyClient
        nlohmann_json::nlohmann_json
        GTest::gtest_main
)
if (PROJECT_IS_TOP_LEVEL)
    target_compile_options(orc_unit_tests PRIVATE -Wall -Werror)
    set_target_properties(orc_unit_tests PROPERTIES XCODE_GENERATE_SCHEME ON)
endif()

# Component Tests
file(GLOB COMPONENT_TEST_FILES CONFIGURE_DEPENDS ${PROJECT_SOURCE_DIR}/test/component/*.cpp)
list(REMOVE_ITEM COMPONENT_TEST_FILES ${PROJECT_SOURCE_DIR}/test/component/main.cpp)
add_executable(orc_component_tests
    ${SRC_FILES}
    ${HEADER_FILES}
    ${PROJECT_SOURCE_DIR}/test/component/main.cpp
    ${COMPONENT_TEST_FILES}
)
target_include_directories(orc_component_tests
    PRIVATE
        ${PROJECT_SOURCE_DIR}/include
        ${PROJECT_SOURCE_DIR}/third_party/adobe-contract-checks/include
)
target_link_libraries(orc_component_tests
    PRIVATE
        stlab::stlab
        TBB::tbb
        tomlplusplus::tomlplusplus
        Tracy::TracyClient
        nlohmann_json::nlohmann_json
        GTest::gtest_main
)
if (PROJECT_IS_TOP_LEVEL)
    target_compile_options(orc_component_tests PRIVATE -Wall -Werror)
    set_target_properties(orc_component_tests PROPERTIES XCODE_GENERATE_SCHEME ON)
endif()

1.3 Update Justfile

Add recipes to the justfile for running the new test suites:

# Run all tests (integration, unit, and component)
test: test-unit test-component test-integration

# Run unit tests
test-unit:
    cmake --build build --target orc_unit_tests
    ./build/Debug/orc_unit_tests

# Run component tests
test-component:
    cmake --build build --target orc_component_tests
    ./build/Debug/orc_component_tests

# Run integration tests
test-integration:
    cmake --build build --target orc_test
    ./build/Debug/orc_test test/battery/

# Run all tests with verbose output
test-verbose:
    cmake --build build --target orc_unit_tests
    ./build/Debug/orc_unit_tests --gtest_verbose
    cmake --build build --target orc_component_tests
    ./build/Debug/orc_component_tests --gtest_verbose
    cmake --build build --target orc_test
    ./build/Debug/orc_test test/battery/

# Run tests with coverage (requires additional setup)
test-coverage:
    cmake -B build -GXcode -DTRACY_ENABLE=OFF -DCMAKE_BUILD_TYPE=Coverage
    cmake --build build --target orc_unit_tests
    cmake --build build --target orc_component_tests
    ./build/Debug/orc_unit_tests
    ./build/Debug/orc_component_tests
    # Process coverage data here

Phase 2: Unit Test Implementation (Weeks 3-6)

2.1 String Utilities Tests (test/unit/str_tests.cpp)

Module: src/str.cpp / include/orc/str.hpp

Test Cases:

  • String conversion utilities
  • String parsing and manipulation
  • Edge cases (empty strings, special characters)
  • Performance benchmarks for critical paths

Dependencies: Minimal (standard library only)

2.2 Hash Function Tests (test/unit/hash_tests.cpp)

Module: src/hash.cpp / include/orc/hash.hpp

Test Cases:

  • Hash function correctness
  • Collision resistance (basic verification)
  • Consistency (same input → same output)
  • Performance characteristics
  • Edge cases (empty data, large data)

Dependencies: Minimal

2.3 Memory Management Tests (test/unit/memory_tests.cpp)

Module: include/orc/memory.hpp

Test Cases:

  • Memory pool allocation/deallocation
  • Alignment requirements
  • Thread safety (if applicable)
  • Memory leak detection (using ASAN or similar)
  • Performance under load

Dependencies: Threading library (if testing concurrency)

2.4 DWARF Constants Tests (test/unit/dwarf_constants_tests.cpp)

Module: src/dwarf_constants.cpp / include/orc/dwarf_constants.hpp

Test Cases:

  • Constant value correctness
  • Enum-to-string conversions
  • String-to-enum conversions
  • Coverage of all DWARF tags, attributes, forms
  • Invalid value handling

Dependencies: None

2.5 Settings Tests (test/unit/settings_tests.cpp)

Module: src/settings.cpp / include/orc/settings.hpp

Test Cases:

  • Configuration file parsing
  • Default value handling
  • Environment variable overrides
  • Invalid configuration detection
  • Settings singleton behavior
  • Thread safety of settings access

Dependencies: TOML++, filesystem

2.6 Demangle Tests (test/unit/demangle_tests.cpp)

Module: Demangle function in src/orc.cpp / include/orc/orc.hpp

Test Cases:

  • Successful demangling of known symbols
  • Handling of already-demangled symbols
  • Invalid/malformed mangled names
  • Thread-local storage behavior
  • Performance for large symbol sets

Dependencies: cxxabi

2.7 Fixed Vector Tests (test/unit/fixed_vector_tests.cpp)

Status: ✅ Already exists

Action: Move from test/src/ to test/unit/ and enhance:

  • Add death tests for bounds violations
  • Add exception safety tests
  • Add move semantics validation
  • Add constexpr tests (if applicable)

Phase 3: Component Test Implementation (Weeks 7-12)

3.1 Object File Registry Tests (test/component/object_file_registry_tests.cpp)

Module: src/object_file_registry.cpp / include/orc/object_file_registry.hpp

Test Cases:

  • Object file registration
  • Symbol lookup
  • Duplicate handling
  • Thread-safe concurrent registration
  • Memory cleanup
  • Integration with file parsers

Test Data: Small pre-built object files (store in test/fixtures/)

3.2 DWARF Parser Tests (test/component/dwarf_parser_tests.cpp)

Module: src/dwarf.cpp / include/orc/dwarf.hpp

Test Cases:

  • DIE (Debug Information Entry) parsing
  • Attribute extraction
  • Navigation of DIE trees
  • Handling of abbreviation tables
  • String table resolution
  • Line number program processing
  • Compressed sections (.zdebug_*)
  • DWARF v2, v3, v4, v5 compatibility

Test Data: Minimal object files with known DWARF structures

3.3 Mach-O Parser Tests (test/component/macho_parser_tests.cpp)

Module: src/macho.cpp / include/orc/macho.hpp

Test Cases:

  • Header parsing (32-bit and 64-bit)
  • Load command processing
  • Section parsing
  • Symbol table extraction
  • Fat binary handling (multiple architectures)
  • Segment loading
  • String table access
  • Error handling for corrupted files

Test Data: Minimal Mach-O binaries (x86_64 and arm64)

3.4 COFF Parser Tests (test/component/coff_parser_tests.cpp)

Module: src/coff.cpp / include/orc/coff.hpp

Test Cases:

  • COFF header parsing
  • Section table processing
  • Symbol table reading
  • Relocation entries
  • String table handling
  • Error handling for malformed files

Test Data: Minimal COFF object files

3.5 Archive (AR) Parser Tests (test/component/ar_parser_tests.cpp)

Module: src/ar.cpp / include/orc/ar.hpp

Test Cases:

  • Archive header parsing
  • Member extraction
  • Symbol index reading
  • Long filename support
  • Nested object file processing
  • Error handling

Test Data: Small .a archive files

3.6 ODR Violation Detection Tests (test/component/odrv_detection_tests.cpp)

Module: Core logic in src/orc.cpp

Test Cases:

  • Byte size mismatch detection
  • Data member location conflicts
  • Vtable element location mismatches
  • Calling convention differences
  • Access specifier conflicts
  • False positive prevention (legitimate duplicates)
  • Edge cases (forward declarations, incomplete types)

Test Data: Programmatically constructed DIE structures

3.7 Report Generation Tests (test/component/report_generation_tests.cpp)

Module: Report generation in src/orc.cpp / include/orc/orc.hpp

Test Cases:

  • Report formatting (text and JSON)
  • Category string generation
  • Symbol demangling in reports
  • Location information accuracy
  • Multiple conflict reporting
  • Report filtering based on settings
  • Output consistency

Test Data: Mock odrv_report objects

Phase 4: Test Data and Fixtures (Weeks 11-12)

4.1 Fixture Directory Structure

test/
├── fixtures/
│   ├── object_files/
│   │   ├── macho/
│   │   │   ├── simple_x86_64.o
│   │   │   ├── simple_arm64.o
│   │   │   └── fat_binary.o
│   │   ├── coff/
│   │   │   └── simple.obj
│   │   └── ar/
│   │       └── simple.a
│   ├── source/
│   │   └── simple_classes.cpp
│   └── dwarf/
│       └── sample_dies.json
└── helpers/
    ├── test_helpers.hpp
    ├── test_helpers.cpp
    ├── fixture_generator.cpp
    └── mock_dies.cpp

4.2 Test Helper Library

Create reusable test utilities:

  • DIE Construction: Programmatically build DIE structures for testing
  • Object File Creation: Generate minimal valid object files
  • Mock Data: Factories for common test scenarios
  • Assertions: Custom gtest matchers for ORC-specific types
  • Comparators: Deep comparison of DWARF structures

Example: test/helpers/test_helpers.hpp

namespace orc::testing {

// Create a minimal DIE with specified attributes
die* create_test_die(dw::tag tag, std::vector<std::pair<dw::at, std::string>> attributes);

// Create a mock object file path with in-memory data
std::filesystem::path create_mock_object_file(const std::string& name, const std::vector<die*>& dies);

// Compare two DIE structures for equality
bool dies_equal(const die* a, const die* b);

// Custom gtest matcher for ODRV reports
MATCHER_P(HasCategory, category, "") {
    return arg.reporting_categories().find(category) != std::string::npos;
}

} // namespace orc::testing

Phase 5: CI/CD Integration (Week 13)

5.1 Update GitHub Actions Workflow

File: .github/workflows/build-and-test.yml

Add steps to run all test suites:

- name: Run Unit Tests
  run: |
    cmake --build build --target orc_unit_tests
    ./build/Debug/orc_unit_tests --gtest_output=xml:unit_test_results.xml

- name: Run Component Tests
  run: |
    cmake --build build --target orc_component_tests
    ./build/Debug/orc_component_tests --gtest_output=xml:component_test_results.xml

- name: Run Integration Tests
  run: |
    cmake --build build --target orc_test
    ./build/Debug/orc_test test/battery/ --json_mode

- name: Upload Test Results
  uses: actions/upload-artifact@v3
  if: always()
  with:
    name: test-results
    path: |
      unit_test_results.xml
      component_test_results.xml

5.2 Test Coverage Reporting

Add coverage generation and reporting:

- name: Generate Coverage Report
  run: |
    # Install coverage tools
    brew install lcov

    # Re-run tests with coverage
    just test-coverage

    # Generate coverage report
    lcov --capture --directory . --output-file coverage.info
    lcov --remove coverage.info '/usr/*' '*/build/_deps/*' --output-file coverage.info

- name: Upload Coverage to Codecov
  uses: codecov/codecov-action@v3
  with:
    files: ./coverage.info
    fail_ci_if_error: true

Phase 6: Documentation (Week 14)

6.1 Testing Documentation

Create: docs/testing.md

  • Overview of testing strategy
  • How to run tests
  • How to write new tests
  • Test data management
  • Debugging failed tests
  • Coverage expectations

6.2 Update README.md

Add testing section:

## Testing

ORC has three levels of testing:

### Unit Tests
Test individual functions and classes in isolation.
```bash
just test-unit

Component Tests

Test larger subsystems and their interactions.

just test-component

Integration Tests

Test end-to-end ODR violation detection scenarios.

just test-integration

Running All Tests

just test

For more details, see docs/testing.md.

6.3 Code Comment Standards

Establish documentation requirements for testable code:

  • All public APIs must have doxygen comments
  • Complex algorithms should include test case references
  • Known limitations should be documented

Test Coverage Goals

Phase 1 Completion (Week 6)

  • Unit Test Coverage: 60% of utility modules
  • Key Modules Covered: str, hash, memory, dwarf_constants, settings

Phase 2 Completion (Week 12)

  • Unit Test Coverage: 80% of utility modules
  • Component Test Coverage: 60% of core subsystems
  • Key Components Covered: All file parsers, object registry, basic ODRV detection

Phase 3 Completion (Week 14)

  • Unit Test Coverage: 85%+
  • Component Test Coverage: 75%+
  • Integration Test Coverage: Maintained at current level
  • CI/CD: Fully integrated with automated reporting

Success Metrics

  1. Test Execution Time: Total test suite should complete in < 2 minutes on CI
  2. Code Coverage: Achieve 75%+ line coverage (excluding dependencies)
  3. Test Reliability: < 1% flaky test rate
  4. Maintenance: New code requires accompanying tests (enforced in code review)
  5. Documentation: All test suites have clear documentation

Risk Mitigation

Risk: Tests are slow

  • Mitigation: Use test fixtures to minimize file I/O
  • Mitigation: Run tests in parallel where possible
  • Mitigation: Profile slow tests and optimize

Risk: Test data management is complex

  • Mitigation: Create programmatic test data generators
  • Mitigation: Use minimal fixtures where possible
  • Mitigation: Document fixture creation process

Risk: Breaking existing integration tests

  • Mitigation: Run all test levels in CI
  • Mitigation: Add unit/component tests before refactoring
  • Mitigation: Maintain backward compatibility

Risk: Low team adoption

  • Mitigation: Provide clear documentation and examples
  • Mitigation: Make test commands easy (just test)
  • Mitigation: Show value through bug catches in code review

Migration Path: From Current State to Full Test Coverage

Week 1: Setup

  1. Create directory structure: test/unit/ and test/component/
  2. Create test/unit/main.cpp and test/component/main.cpp
  3. Update CMakeLists.txt with new test targets
  4. Add test recipes to justfile
  5. Verify builds and runs successfully (even with no tests)
  6. Move fixed_vector_tests.cpp to test/unit/

Validation: just test-unit runs successfully

Week 2: First Unit Tests

  1. Implement str_tests.cpp with basic string utility tests
  2. Implement hash_tests.cpp with basic hash function tests
  3. Implement dwarf_constants_tests.cpp for enum conversions
  4. Run tests locally and fix any issues

Validation: At least 20 passing unit tests

Week 3-4: Core Unit Tests

  1. Implement settings_tests.cpp
  2. Implement memory_tests.cpp
  3. Implement demangle_tests.cpp
  4. Enhance fixed_vector_tests.cpp with additional coverage

Validation: 50+ passing unit tests, coverage > 50% for tested modules

Week 5-6: Unit Test Polish

  1. Add edge case tests
  2. Add performance benchmarks where relevant
  3. Fix any discovered bugs
  4. Document test patterns and helpers

Validation: 80+ unit tests, all modules tested have >70% coverage

Week 7-8: Test Fixtures and Helpers

  1. Create test/fixtures/ directory structure
  2. Generate minimal test object files (Mach-O, COFF, AR)
  3. Implement test helper library (test/helpers/)
  4. Document fixture generation process

Validation: Test fixtures available for component tests

Week 9-10: File Parser Component Tests

  1. Implement macho_parser_tests.cpp
  2. Implement coff_parser_tests.cpp
  3. Implement ar_parser_tests.cpp
  4. Implement dwarf_parser_tests.cpp

Validation: 50+ component tests, file parsers have >60% coverage

Week 11-12: Core Logic Component Tests

  1. Implement object_file_registry_tests.cpp
  2. Implement odrv_detection_tests.cpp
  3. Implement report_generation_tests.cpp
  4. Fix any discovered bugs

Validation: 100+ component tests, core logic >60% coverage

Week 13: CI/CD Integration

  1. Update .github/workflows/build-and-test.yml
  2. Add coverage reporting
  3. Add test result artifacts
  4. Test CI pipeline thoroughly

Validation: All tests run in CI, coverage reports generated

Week 14: Documentation and Cleanup

  1. Create docs/testing.md
  2. Update README.md
  3. Add code comment standards
  4. Review and refactor tests for clarity
  5. Team training/walkthrough

Validation: Complete documentation, team trained

Example Test Files

Example: test/unit/main.cpp

// Google Test main for unit tests
#include <gtest/gtest.h>

int main(int argc, char** argv) {
    ::testing::InitGoogleTest(&argc, argv);
    return RUN_ALL_TESTS();
}

Example: test/unit/str_tests.cpp (Skeleton)

#include "orc/str.hpp"
#include <gtest/gtest.h>

namespace orc {

TEST(StrTest, BasicConversion) {
    // Test basic string conversions
    // EXPECT_EQ(...);
}

TEST(StrTest, EmptyString) {
    // Test empty string handling
    // EXPECT_TRUE(...);
}

TEST(StrTest, SpecialCharacters) {
    // Test strings with special characters
    // EXPECT_EQ(...);
}

} // namespace orc

Example: test/component/macho_parser_tests.cpp (Skeleton)

#include "orc/macho.hpp"
#include "test/helpers/test_helpers.hpp"
#include <gtest/gtest.h>
#include <filesystem>

namespace orc {

class MachOParserTest : public ::testing::Test {
protected:
    void SetUp() override {
        // Set up test fixtures
        test_object_path = std::filesystem::path("test/fixtures/object_files/macho/simple_x86_64.o");
    }

    std::filesystem::path test_object_path;
};

TEST_F(MachOParserTest, ParsesHeader) {
    // Test that parser correctly reads Mach-O header
    // ASSERT_TRUE(std::filesystem::exists(test_object_path));
    // auto result = parse_macho_file(test_object_path);
    // EXPECT_TRUE(result.has_value());
}

TEST_F(MachOParserTest, ExtractsSymbols) {
    // Test symbol extraction
}

TEST_F(MachOParserTest, HandlesFatBinary) {
    // Test multi-architecture binary
}

} // namespace orc

Maintenance and Evolution

Quarterly Reviews

  • Review test coverage and identify gaps
  • Update test fixtures as format support expands
  • Evaluate test execution time and optimize if needed
  • Update documentation based on team feedback

Continuous Improvement

  • Add tests for every bug fix (regression tests)
  • Add tests for every new feature
  • Refactor tests when production code changes
  • Monitor and fix flaky tests immediately

Team Practices

  • Code review checklist includes test coverage
  • New contributors must include tests with PRs
  • Test failures block merge to main branch
  • Regular test health dashboards

Conclusion

This test plan provides a structured approach to implementing comprehensive unit and component testing for ORC. By following the 14-week timeline, the project will achieve:

  1. Robust test coverage at multiple levels (unit, component, integration)
  2. Automated testing in CI/CD pipelines
  3. Clear documentation for test development and maintenance
  4. Improved code quality through early bug detection
  5. Easier refactoring with confidence in test coverage

The phased approach allows for incremental progress while maintaining development velocity on the main codebase. The justfile integration ensures tests are easy to run locally, promoting a test-driven development culture.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions