Fix CI

[devmotion]

As #1338 shows, tests currently fail on master.

The PR fixes the following test sets:

• docs: Problem with Optimization (v4.8.0?)
• Core (lts, 1, pre): Problem with DynamicSS?
• Extensions (1, pre): Caused by a bug in Arblib, fixed in Arblib >= 1.6.0 but tests use Arblib 1.4.0
• Hybrid (lts, 1, pre): Incorrect checks of solution return code + numerical issues (caused by ForwardDiff@1?)
• Spatial (only pre)

[isaacsas]

@TorkelE @devmotion let's use this as a PR to work in to get tests working and docs building. Once that is all fixed I am happy to make a release, but I don't want to make one while tests are failing and/or docs don't build.

[isaacsas]

@TorkelE this doesn't return a boolean or error, so the test is not broken. I assume you meant this as a flag that there is an issue, but it seems to work now. Note though that ssint is not used anymore throughout the tests, so nothing is really being tested for it now beyond init.

[TorkelE]

This has been fine for 16 months, so I presume that something has changed underneath. I will update to ... false # ..., which should make things run for now. Then when we do the MTK fixes I will go through all of the changes and implications proiperly again.

[devmotion]

AFAICT we're stuck with Arblib 1.4 which does not work on Julia >= 1.12 (fixed in Arblib >= 1.6) due to interactions with StructuralIdentifiability which is in the same environment for the Extensions tests: StructuralIdentifiability depends on FLINT_jll as well through Nemo, and Nemo pins FLINT_jll quite strictly; however, recent versions of StructuralIdentifiability which support recent versions of Nemo and hence recent versions of FLINT_jll only support MTK@10.

Breaking up the Extensions environment in smaller environments with presumable less conflicts of dependencies could be a solution; alternatively, one could maybe try to backport support for recent versions of Nemo to older MTK@9 compatible releases of StructuralIdentifiability.

[isaacsas]

It's surprising they could update StructuralIdentifiability to MTK10 without making a breaking release.

[TorkelE]

Think SI just really uses MTK as an interface for declaring a model, so it wouldn't surprise me if no workflows have actually changed. If there has been package updates that means that things no longer work, we should introduce (as liberal as possible) compat bounds to ensure that thinsg actually works.

[isaacsas]

I think the bigger issue might be that fundamentally we won't work on 1.12 with MTK9 and the versions of the extensions that work with it. It might be that we should just cap testing to 1.11 and put a note on the Catalyst homepage that we currently only support 1.11, with 1.12 support projected in the next breaking release.

[isaacsas]

The hybrid test failure is a real issue that I'm trying to debug. There was a typo in the test, and once fixed the test fails as the callback is never triggered. I haven't been able to figure out why that is happening yet though.

[isaacsas]

Not sure though why tests were previously passing with that typo.

[devmotion]

Maybe due to an upstream fix?

[devmotion]

It might be that we should just cap testing to 1.11 and put a note on the Catalyst homepage that we currently only support 1.11, with 1.12 support projected in the next breaking release.

It's really not a Catalyst issue IMO. Previous versions of Arblib had a bug. It's a problem of Arblib and a problem you'll always run into if you use Arblib < 1.6 on Julia 1.12. I don't think it should be Catalyst's responsibility to warn users about a bug caused by Arblib.

To unbreak the tests, we could also try to backport this Arblib fix to a backport release of Arblib (probably 1.4.1?).

[devmotion]

I asked for the possibility of a backport release of Arblib: kalmarek/Arblib.jl#220 (comment)

[isaacsas]

I marked the blocking test as broken but other hybrid tests are still failing locally and I think it is due to continuous event handling. They used to pass when we last released so something seems to have changed to cause them to be broken now in MTK or lower.

[isaacsas]

Anyways I can't work more on this till next week now as I have some deadlines the next few days, but I can circle back next week hopefully to investigate the hybrid tests (or perhaps @TorkelE can look into it). It seems that continuous events no longer properly trigger for jumps over ODEProblem, which definitely worked at one point, so will require some work to investigate what is going one and where changes happened.

[TorkelE]

I will try to do what I can, but the hybrid stuff looks like it can be a rather deep rabbit hole (but definitely one require solving. The issue with

@test_broken ssint = init(ssprob, DynamicSS(Tsit5()); save_everystep = false) # https://github.com/SciML/SciMLBase.jl/issues/660

is simply that the broken test have now been fixed (as confirmed by the link issue).

@test_broken error("")

passes as any other broken test, but as init(ssprob, DynamicSS(Tsit5()); save_everystep = false) no longer thorws an error, the test correctly notes that it is no longer broken (although I agree that the logic is a bit weird, and if there had been a @test_unintended_error` or similar, that would have been more appropriate).

There are a couple of downstream tests on ssint that we couldn't do (because we couldn't create it), I will see if I can reenable them now as well.

[devmotion]

Joel backported the fix for Julia 1.12 to Arblib 1.4.1. This indeed fixes the extensions tests.

[TorkelE]

That should leave us only with the hybrid stuff and the docs. Checking the doc, there is an issue in SciML/Optimization that it has encountered. I think this has happened before. Will try to isolate and report.

[devmotion]

Optimization might be SciML/Optimization.jl#1070. I ran into the same issue in some other project recently, I think there was a big mess in the recent releases. I think something in the registry should be adjusted: SciML/Optimization.jl#1070 (comment)

Edit: I'm checking whether docs can be built with [email protected] in e4d9a5e

Note that #1328 was pretty useless since Optimization@5 requires MTK@10

[devmotion]

I spent a bit more time on the Optimization issue, and I think JuliaRegistries/General#142620 should fix it without any changes to Catalyst (so my experiments in the last commits should be reverted once the registry is fixed).

[devmotion]

It seems docs will require a new release of DynamicalSystems with JuliaDynamics/DynamicalSystems.jl#259 as well. I wonder if for the docs it could indeed make sense to build them with a fixed Julia versions. To be clear, the fix JuliaDynamics/DynamicalSystems.jl#259 is a bug fix on every Julia versions but only on Julia 1.12 it seems to cause precompilation errors.

Edit: Indeed, restricting SciMLBase to <= 2.120.0 (as in the registry PR) seems to fix the Optimization.jl errors when building the docs. Remaining problems are the DynamicalSystems error on Julia 1.12 and the precompilation errors of PEtab on Julia 1.12 (they were fixed upstream, but those versions only allow for MTK <= 9.80.1, not [email protected]; I opened sebapersson/PEtab.jl#330 to see if the compat bound can be changed in PEtab).

[devmotion]

PEtab support for [email protected] should be available in an upcoming release, probably today or tomorrow.

[TorkelE]

Excellent, thanks with all the help with this! Sorting out the hybrid thing I think is still the only major thing left?

[isaacsas]

I unfortunately haven't had much time to work on this. Throwing Claude at it I got the following conclusions with some back and forth:

The Bug
JumpProcesses does not forward JumpProblem.kwargs to the solver when solve(jprob, alg) is called.
Evidence
✓ ModelingToolkit.process_events(js) IS called in the JumpProblem constructor (line 577 in jumpsystem.jl)
✓ Continuous event callbacks ARE created successfully
✓ The callbacks are stored in JumpProblem.kwargs[:callback]
✗ solve(jprob, Tsit5()) does NOT pass these kwargs to the underlying ODE solver
✓ solve(jprob, Tsit5(); jprob.kwargs...) DOES work - the event triggers correctly

Given these tests used to pass, and I'm not aware of any JumpProcesses changes related to this, I suspect that perhaps something has changed in SciMLBase or DiffEqBase since the last Catalyst release (but haven't had time to try running tests with older versions).

[isaacsas]

If we are putting in a backport PR to MTK it should probably also update the fixed test from our hybrid model fixes here, as there is a version of it in the tests in MTK with the same bug.

[devmotion]

Honestly, I think a backport release of BifurcationKit might be more generally useful. It seems that would solve this particular problem of our docs environment as well. However, given that the remaining issues of our docs environment (PEtab and DynamicVisualizations) all seem to be caused in the first place by problems with Julia 1.12, I think we might generally want to consider building docs on Julia LTS to be not affected by such problems in our dependencies when a new Julia version is released.

[isaacsas]

Hybrid seems fixed now, so I think docs are all that is left.

[devmotion]

It seems that even on Julia LTS we need a newer version of PEtab >= 3.9.4 that contains sebapersson/PEtab.jl#316. This huge docs environment seems to be a bit of a dependency nightmare 😄

[isaacsas]

If docs build without the PEtab tutorial I would be fine removing it for now. We’ve had to do that in the past too at times.

[isaacsas]

@TorkelE tests are passing now. Do you think you could look at the docs? The issues seem to be in tutorials that link to PEtab and other dependencies and not the core tutorials.

[isaacsas]

I removed the PEtab tutorial, so let's see if the docs now build.

[TorkelE]

happy to have a look at it after it has built, still would be ideal to have it working with PEtab though, as parameter fitting is something i imagine that many users are interested in doing.

[devmotion]

It seems the docs build almost succeeded in 99bf59e, one remaining issue seems to be a bug in PEtab introduced in sebapersson/PEtab.jl#318 (the problem is the @set sampling_method.rng = rng line since SobolSample doesn't have an rng field: https://github.com/SciML/QuasiMonteCarlo.jl/blob/83427d1aa4dac4f39f9bc0a3d69d25f04d541271/src/Sobol.jl#L6-L8).

[TorkelE]

@devmotion thanks a lot for all the work on this. Let's see if we get a quick reply over there, if not, let's just built this without PEtab later today, and we can at least check that everything else looks fine.

[sebapersson]

@devmotion thanks a lot for all the work on this. Let's see if we get a quick reply over there, if not, let's just built this without PEtab later today, and we can at least check that everything else looks fine.

I can fix this bug related to Sobol-sampling. @TorkelE for the docs, would it be possible to use LatinHypercube sampling in the failing code-block here (will allow you to rebuild the docs without waiting for PEtab.jl to update)?

[TorkelE]

Thanks! I have changed to that sampler (although in the long run I'd like to have a non-default sampler in the example)

[isaacsas]

OK, looks like we should be good once we can actually use a BifurcationKit release (and not @devmotion's personal branch).

[isaacsas]

@devmotion thanks for the help with this! Are you going to handle making a backport PR for BifurcationKit? If so, please let us know when a new release we can use is available.

[devmotion]

Romain is working on a backport release: bifurcationkit/BifurcationKit.jl#255 (comment)