How do I prevent moon ci or moon run from running tasks in upstream projects?

[chadrik]

I would like to generate a graph of CI jobs that mirror the dependency graph in moon. My specific case is for Gitlab, but this should apply to other CI platforms as well.

Consider an example where I have a repo with two projects, projectA and projectB, where projectB depends on projectA. Each project has a :test task that depends on a :build task.

For the sake of simplicity, let's say that I create two Gitlab jobs:

projectA-test-job runs:

moon ci projectA:test

projectB-test-job runs:

moon ci projectB:test

The second job is setup in Gitlab to only run after the first job completes.

If I push up a commit that modifies files in both projectA and projectB, how do I prevent the second job from rerunning the build for projectA? I suspect remote caching is a potential solution, but I'm running Gitlab behind a firewall and I can't use moonbase. The docs are somewhat discouraging about managing the output cache yourself:

However, because tasks can generate a different hash each run, you'll need to manually invalidate your cache. Blindly storing the hashes and outputs directories without a mechanism to invalidate will simply not work, as the contents will drastically change between CI runs. This is the primary reason why the remote caching service exists.

I don't see any detailed explanation of how to do this myself.

I think caching outputs is really not what I need anyway, because my toolchain (which is not yet supported by moon) stores its outputs outside of the repository root (in a shared location in ~/).

I think what I'm really looking for is a way to indicate to moon ci that it should only run tasks in the current project and not dependent projects. Maybe it could be a new target type like:

moon ci ~projectB:test 

Or even more granular/brute-force, a flag to skip all dependent tasks:

moon ci ~projectB:test  --noDependences

Any help would be greatly appreciated!

[milesj]

@chadrik moon ci is one of those "does everything for you commands" and probably don't change. You probably just want to use moon run or moon check here?

[chadrik]

I'm happy to use moon run, but as far as I can see, the same problem exists with that command and all derivatives of it. How do I prevent moon run from running tasks in dependent projects? I updated the title to reflect this.

[milesj]

@chadrik moon run doesn't run dependents (downstream) by default, it requires --dependents. It will always run dependencies (upstream) though.

[chadrik]

yes, my question is how do I avoid running upstream dependencies. In my example, moon ci projectA:test and moon ci projectB:test run in different CI jobs but they both end up building projectA. How do I prevent that?

moon ci projectA:test has this dependency graph:

projectA:test --> projectA:build

moon ci projectB:test has this dependency graph:

projectB:test --> projectB:build --> projectA:build

In CI, I want to only run projectB:test --> projectB:build because projectA:build will have already been run by a different job.

Is there some way that I can make moon understand that the projectA build is already up-to-date without copying .cache/outputs? For example with go-task I can implement a custom function to call to indicate if a task is up-to-date: https://taskfile.dev/usage/#using-programmatic-checks-to-indicate-a-task-is-up-to-date

[milesj]

Ah yeah, this isn't possible, merely for the fact that we have no way of knowing if an artifact from a dependency is required for the task to run correctly.

Using your example, projectB:build must always run projectA:build, just in case the output from projectA:build is required for projectB:build (it may be pulled into the bundle, or something similar).

In the next release, v1.30, we're adding support for self-hosted remote caches, so this can probably just be used here.