Remove OptimizationOptimJL dep?

[penelopeysm]

Even after the Optim.jl interface is removed (#2635, #2708), we still have a dep on Optim.jl via OptimizationOptimJL.jl. That's because we currently use Optim's solvers as the default solvers:

Turing.jl/src/optimisation/Optimisation.jl

Lines 453 to 459 in 4153a83

	function default_solver(constraints::ModeEstimationConstraints)
	return if has_generic_constraints(constraints)
	OptimizationOptimJL.IPNewton()
	else
	OptimizationOptimJL.LBFGS()
	end
	end

However, I wonder if this is necessary. Also the hard dep on OptimizationOptimJL causes incompatibilities with other packages, as seen in #2711.

Now, Optimization.jl itself contains some solvers including LBFGS: https://docs.sciml.ai/Optimization/stable/optimization_packages/optimization/ I don't know what loss of functionality switching to this might incur (and I'm very, very rusty on optimization theory), but if we could do it, I think we should because that stays in line with the general philosophy that Optimization.jl provides the interface and you load whichever solver you want.

[penelopeysm]

An alternative to this could be to put default_solver() behind an extension, and force users to either:

• specify the solver themselves, which would in turn make them load a solver package; or
• load OptimizationOptimJL themselves, which would then load the extension and make default_solver work.

Ordinarily I wouldn't be so worried about this, but I think it can be frustrating if you're only trying to use Turing for MCMC sampling but then you get caught up in all sorts of dependency mishaps with the SciML stack because of this.

[penelopeysm]

Finally, a more radical solution would be to split the MCMC, optimisation, etc. into separate packages. Arguably, if you want to ensure that users only load the code they need, then they should all be DynamicPPL extensions:

• DynamicPPLAdvancedHMCExt for NUTS sampling
• DynamicPPLOptimizationExt for mode estimates
• DynamicPPLAdvancedVIExt for variational inference

and so on. Obviously, this is ugly and clunky. So we have to find the right compromise between 'everything is in one package and it might potentially error with things you don't use', vs 'never load more code than you need but it's a real pain to do things'.

[mhauru]

This, from the docs you linked, surprised me:

[LBFGS] directly supports box-constraints.

I think making people install more packages just to be able to do maximum_likelihood(m) is putting too much of a burden on users, but if Optimization.jl has suitable default optimisers for both constrained and unconstrained (or maaaaybe just unconstrained could be acceptable, if the default is to link anyway), then that could be a nice solution.

[yebai]

l-BFGS is the main algorithm we need here, so if Optimization.jl provides what we need, it is okay to consider removing OptimizationOptimJL.