Structuring checkpoints and dependencies in complex Continue pipelines

[Arpanwanwe]

I've been working on developing some fairly complicated pipelines with Continue. In my case, I'm creating pipelines that have branching logic involved; in particular, I have cases in which results from branches down the line depend on results from branches upstream or from external data sources. Additionally, I have to create retry logic (i.e., logging and tracking failures and re-running the steps that have previously failed). As a result, the more I have built up my dependencies and defined my steps using Continue's checkpoint-based methodology, the easier it is for contextual information to get lost along the way.

Continue uses checkpoints to define a series of sequential actions to take place; however, when I create pipelines that include a mix of branching (multimodal), looping, and conditional logic, it has become increasingly difficult to determine how to best define and track shared intermediate states without creating inconsistent ways of passing data from one step to another (ad hoc) or using an intermediate state object.

So, I'm currently developing an experiment with a central state object that has been designed and built to allow multiple steps to update and reference. The benefit of using a central state object is that it separates the state used during pipeline execution from the logic that processes a step. For each of my projects that have included this approach, I've created a helper application for testing that has improved my ability to debug issues and maintain reproducibility. However, I'm still experimenting with different ways to define my pattern.

I am very interested in hearing how other users define dependencies and share context within the Continue pipeline. Do you have a similar model of using a central state object? Do you only use checkpoints, store states in an external location, or take a different approach altogether?