Discussion: Foundational / Pre-trained Models in Darts

[daidahao]

Is your feature request related to a current problem? Please describe.
Amazon has just released Chronos 2, a pre-trained Transformer model for forecasting. Unlike previous Chronos, it has direct multivariate and (future and past) covariate support.

I wonder if there is any interest from the Darts side to implement and support it inside Darts? If so, I am happy to try implementing a prototype in Darts APIs.

Describe proposed solution
A prototype implementation of pre-trained Chronos 2 in Darts torch model APIs. Some additional dependencies may be required (from a glance of Amazon code):

• einops (maybe replaced by torch operations).

The user would also need to download pre-trained model weights (120M parameters, around 450MB) from Amazon S3 bucket.

Describe potential alternatives
A clear and concise description of any alternative solutions or existing features that might solve it.

Additional context
Add any other context or screenshots about the feature request here.

[JuanCruzC97]

It would be great to have the darts interface available to evaluate this kind of pretrained models.

[dennisbader]

Hi @daidahao and @JuanCruzC97, and thanks for raising this issue. We are currently re-evaluating whether we could / should interface some of these models.

One or two years ago I was against adding them, but now that the models also support covariates and multivariate forecasting it looks different :)

Before implementing anything, I would first like to think about a couple of high level topics.
It would be very nice to get your feedback on this. We could also shape the plan for this together, if you like :)

• Where would we have this interface? I'm worried specifically about unit testing, where each time our CI/CD pipeline is run, it would have to re-download the models.
  • Would we keep it in the main darts repo?
  • Or would create a new repo such as darts-fm?
  • Or is there even a better solution which wouldn't burden the main repo too much?
• What models should we add?
  • In my opinion, all models must be built on pytorch. Otherwise, we won't add them.
  • Chronos 2 looks good with rich covariates-, and multivariate forecasting support
  • TimesFM2.5, maybe? There is currently a PR open Add TimesFM 2.5: First Foundation Model Integration #2934, even though that one might be a bit too early for me (since we haven't yet made a clear plan)
  • ...?
• API wise, the models must inherit from GlobalForecastingModel (I wonder if we could wrap some of the pytorch-based models with our TorchForecastingModel)
  • I hope not much more is needed, except of the downloading capabilities
  • fit() might be used for fine-tuning the models (this would be great, but might be tricky to unify across different models. I would leave it as optional for now).

[daidahao]

Hi @dennisbader, thank you for your very prudent and well-thought out response. Yes, I am very glad to be shaping and contributing to this core feature with you and perhaps @JuanCruzC97 as well, :)

Over the past two months, our team at Oxford have ported and experimented with several open-sourced foundation models, in parallel to using Darts for our main forecasting task. Therefore, a lot of my suggestions here would be from my experience of interacting with those models.

Where would we have this interface? I'm worried specifically about unit testing, where each time our CI/CD pipeline is run, it would have to re-download the models.

Most TSFMs if not all of them are larger than typical TFTs or RNNs, but usually smaller than 1B, at around 50M-500M. See GIFT-EVAL for a list of recent TSFMs. At that scale, some small-scale unit testing should not take too long on GitHub Actions runners, like forecasting with a small batch but not fine-tuning. I am in support of keeping TSFMs in the main repo (at least for now) to prevent issues like internal API changes, separate documentation, maintenance, etc.

However, testing time could stack up should we test all use cases of a TSFM (target only, target + past cov., target + future cov., etc.). In that case, I would like to propose a few ways to mitigate the testing time:

• Create a dummy tiny model for non-accuracy related unit tests, like in Chronos 2 repo.
• Limit the number of "actual model" unit tests to 1 or 2 for validating the accuracy. Reduce the dataset size and batch size to reduce runtime per test.
• If the runtime is still very long, exclude TSFM unit tests from the workflows by default. Add workflow rules to only trigger unit tests when the TSFM code has changed or parent classes like TorchForecastingModel have changed.

What models should we add?

In my opinion, all models must be built on pytorch. Otherwise, we won't add them.

I agree. Almost all TSFMs have PyTorch reference implementation, so that is fine (in 2025).

Chronos 2 looks good with rich covariates-, and multivariate forecasting support
TimesFM2.5, maybe?

I think Chronos 2 would be a good start as I've mentioned they have built-in multivariate and co-variate support. In terms of TimesFM 2.5, their covariate is on the roadmap but not fully implemented yet. For other models, I would suggest prioritising models with multivariate and co-variate support since they are more likely to benefit Darts users.

All in all, GIFT-EVAL and fev-leaderboard list some of the most recent TSFMs. We can use them as a reference to keep track of progress in the field and to select the models.

API wise, the models must inherit from GlobalForecastingModel (I wonder if we could wrap some of the pytorch-based models with our TorchForecastingModel)

Wrapping most models with TorchForecastingModel should be possible since they are fundamentally (no pun intended) torch models and would work with Lightning APIs. There are, however, key differences to other torch models

• fit() is not needed before calling predict(). We may need some workaround with the default logic of ForecastingModel where fit() must be called before predict().
• A few extra dependencies are required for TSFMs. From my experience, most models would need huggingface_hub (not transformers) and safetensors for downloading and loading model weights. Some would require tensor library like einops but can be replaced with einsum() and reshape() in PyTorch. Thankfully, most of those dependencies are rather lightweight.
• Most TSFMs download weights from Hugging Face, but some like Chronos 2 from AWS S3 bucket.
• Many TSFMs have their own fixed quantile levels, and cannot be changed without fine-tuning.
• Many TSFMs use median sampling (Feature request: allow predict_likelihood_parameters for auto-regressive QuantileRegression #2912), or "quantile over quantiles" sampling rather than MC sampling. Validating the model outputs when auto-regression are involved is impossible due to different sampling methods.
• Many TSFMs have default "output chunk lengths" (horizon each step), which may be shorter than the user-defined. Performance when the horizon (with auto-regression) is longer is not guaranteed.

Given the differences and many model specifies, I wonder if you would prefer a unified FoundationModel that all TSFMs inherit from or having all TSFMs inheriting from TorchForecastingModel.

fit() might be used for fine-tuning the models (this would be great, but might be tricky to unify across different models. I would leave it as optional for now).

Fine-tuning would be model-specific. Chronos 2 supports fine-tuning with quantile loss and seems compatible with QuantileRegression (at a glance of their codebase). We might not have such luck with other models if their loss function is different.

Looking forward to your further thoughts on this. Thanks!

[hrzn]

Just my 2 cents: I think it'd be great for Darts to host a few selected reference implementation of these models and make them available to our users. Looking at the ecosystem, Darts could play a helpful role of a "neutral" place where several of these (sometimes competing) models can live and be easily used under the common joint Darts API. => IMO that's a good idea.
When it comes to the specifics: would it be worth controlling / versioning also the model weights (maybe hosting them on a separate darts-controlled git repo, or bucket), just to mitigate the risk of external dependencies (e.g. if some externally controlled bucket is brought down), and to avoid putting what I assume might be multi-100MBs files on the main repo.

[nehalecky]

Hi @dennisbader, @daidahao and @hrzn! Apologies about delay, family has been hit with a bad 🦠 that knocked us offline and we're still recovering. Thank you all for opening up the discussion and engagement. Really great insights, perspectives and feedback. Give me a few days to incorporate all and will chime in soon with a response. Happy to align on a design, interface and modify this implementation to get this across the line.

In general, after a review of other time series libraries and lack of a unified API to allow wider accessibility to time series foundation models (TSFMs) application, I concluded Darts had a strong foundation, from an interface, implementation, and guidance for for aligning TSFMs across the board (in tune with @hrzn's note above)—exciting to see this goal delivered and more broad adoption of these very useful tools!

Final note: Chronos 2 was released right as this PR was shipped! So far, I'm impressed, and feel it likely sets a bar for the core capabilities at TSFM should have. I have a working implementation for integration it with Darts based off of this PR, and brushing it up now, but will let our efforts here finalize it!