[Question / Feature Request] Only build packages with modified source code

[bharathsivaram10]

Hello,

I primarily use colcon for building ROS2 packages.

One thing I'd like to see is an option like 'colcon build --modified_only', which would somehow gather the packages whose cpp source code has been modified since the last build and only rebuild those ones.

This is as opposed to using the --packages-select, which I'd have to specify the packages. It would be nice to have when modifying a few packages in a large workspace and being able to auto-build those few without explicitly listing.

Is this possible? From searching around, it doesn't seem like the option currently exists. Would people find this useful?

[christophebedard]

You might be able to implement something like that, but I don't think this is in colcon's scope. That's the build system's job, e.g., CMake. If you tell colcon to build some packages and the packages' source code hasn't been modified, it should usually build very quickly.

[mikepurvis]

Check out https://github.com/ruffsl/colcon-cache

[christophebedard]

I forgot about that!

[mikepurvis]

It doesn't look super active, but it's still in use by the nav2 CI workflows, and I'm considering it for a new project as well. Maybe a quick ping to @ruffsl regarding whether colcon-cache is quiet because dead or quiet because it does the thing and has simply required little in the way of new features or maintenance?

[ruffsl]

or quiet because it does the thing and has simply required little in the way of new features or maintenance?

Yeah, it basically feature complete for now. Recent activity has mostly been maintenance and CI upkeep.

I've requested transfer to keep the package in the org as with colcon-clean, but haven't heard back in a while.

• Repo Transfer Request: colcon-cache #522
• Repo Transfer Request: colcon-clean #521

Dexory also makes heavy use of this for their internal CI and local workstation development. By caching colcon workspace folders (i.e. ./install/ and json-lockfiles/colcon-rc-files in ./build/) using github actions cache, and then keying that action cache with a hash of the originating build environment (i.e. docker base image layer sha or install debs) , and pairing it with colcon-clean, it make for a rather simple yet efficient means of achieving differential builds between CI jobs.

Also, using a bash alias for your workspace build command makes it simple to constantly use it locally:

#!/usr/bin/env bash

# Immediately catch all errors
set -eo pipefail

# Uncomment for debugging
# set -x
# env

cd $OVERLAY_WS

colcon cache lock

BUILD_PACKAGES=$(
    colcon list \
        --names-only \
        --packages-up-to \
            my_bringup \
        --packages-skip-cache-valid \
        --packages-select-cache-key \
            build \
    | xargs)
echo BUILD_PACKAGES: $BUILD_PACKAGES

# DEBUG: Uncoment for more sterile but slower builds
# colcon clean packages --yes \
#     --packages-select ${BUILD_PACKAGES} \
#     --base-select build install

. /opt/ros/$ROS_DISTRO/setup.sh
colcon build \
    --packages-select \
        $BUILD_PACKAGES