Circular Dependency Between the Engine and Agents

All our agents are pluggable extensions, build separately from the engine. Unsurprisingly, they references to the engine. While the engine doesn't actually reference any agents, it depends on them in other ways that create a kind of circularity in practice. This page is about how we deal with this circularity when an engine update requires updating the agents as well.

How the Engine Depends on Agents

Each agent consists of two parts. A launcher assembly and the agent assembly itself, packaged as separate assemblies. The launcher is the actual extension and is loaded by the engine. For this to work, both engine and launcher must reference compatible versions of the engine api. In general, we try to make only backward compatible changes to the API but no occasion this isn't possible. The actual agent assembly is package with all the necessary dependencies and should not cause compatibility problems so long as agents are loaded in a separate process from the engine itself, as is the case in the V4 beta.

The real dependency problem is that we normally test all releases of the engine and console with a set of agents. If an update invalidates the existing agents, those tests cannot be run. We may end up in a situation where we can't release an engine update because we can't run tests on the agents AND we can't release new agents that depend on the updated engine, because the updated engine has not been released.

Workaround

The TestEngineRunner project, which builds both the console runner and the engine, has a private commandline option, --engine-only. This setting affects the Package target so that only the engine and its dependencies are packaged. All components are actually built to avoid errors, but in packaging the various runner packages are skipped so that the run is able to complete without loading any agents.

This option is only available in a local build, so all steps using it must be completed on your own machine.

Engine Steps:

1. Merge the changes to the engine or engine api into your local copy of the main branch.
2. Commit the changes locally, but do not push them to GitHub.
3. Use the build and test using build -t Test.
4. Once you are sure that this version is the one you want to use, tag the build with the version, e.g. git tag 4.0.0-beta.1.18. Use the actual version that was just built as indicated in the build output.
5. Publish the build using the --engine-only option: build -t Publish --nobuild --engine-only.

Agent Steps (repeat for each bundled agent):

1. Switch to the agent repository and create a working branch. Change all engine references to the version you just created. Make any other changes to the code based on the nature of the change you made to the engine.
2. Build, test and package the updated engine, repeating until everything works correctly.
3. Push the branch in the usual way. When the CI build is successful, create a PR and eventually merge it in the usual way.

Once these steps are completed, you should go back to the engine, update the versions of the agents bundled with it and go through the normal release process. This will rebuild both the engine and the runners and get them back in sync. As soon as possible, you should do the same for each of the agents.

And yes, we did everything twice. Hopefully, this won't be needed too often.