Databricks Runtime

[sonriks6]

Heads up on those using Databricks, with Meridian version 1.0.7 where bumped numpy, tensorflow and other libraries, so if running on GPU, the only runtime valid for this requirements is the 16.3 ML beta (released 12/03/2025), this is just because tf 2.18 needs cuDNN 9.3 library on the runtime.

[trevorsmith-iherb]

@sonriks6 thank you! This helped me resolve my issue in databricks.

[OykuOzOzen]

Hi,
I am running the latest version of google meridian on databricks. When I install the latest version, during the run I receive the following error: "ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
petastorm 0.12.1 requires pyspark>=2.1.0, which is not installed.
databricks-feature-engineering 0.8.0 requires numpy<2,>=1.19.2, but you have numpy 2.0.2 which is incompatible.
langchain 0.1.20 requires numpy<2,>=1, but you have numpy 2.0.2 which is incompatible.
langchain-community 0.0.38 requires numpy<2,>=1, but you have numpy 2.0.2 which is incompatible.
langchain-core 0.1.52 requires packaging<24.0,>=23.2, but you have packaging 25.0 which is incompatible.
mlflow-skinny 2.19.0 requires packaging<25, but you have packaging 25.0 which is incompatible.
numba 0.57.1 requires numpy<1.25,>=1.21, but you have numpy 2.0.2 which is incompatible.
ydata-profiling 4.5.1 requires numpy<1.24,>=1.16.0, but you have numpy 2.0.2 which is incompatible.
ydata-profiling 4.5.1 requires pandas!=1.4.0,<2.1,>1.1, but you have pandas 2.3.1 which is incompatible.
ydata-profiling 4.5.1 requires scipy<1.12,>=1.4.1, but you have scipy 1.16.0 which is incompatible." And when I try to import modules from meridian, I receive the error: "ModuleNotFoundError: No module named 'numpy.lib.array_utils'
File , line 10
5 #import arviz as az
7 import IPython
---> 10 from meridian import constants
11 from meridian.data import load
12 from meridian.data import test_utils
File /local_disk0/.ephemeral_nfs/envs/pythonEnv-d47c4d06-6add-4a89-bce8-05500b5ac8d0/lib/python3.11/site-packages/xarray/core/nputils.py:16
14 # remove once numpy 2.0 is the oldest supported version
15 if module_available("numpy", minversion="2.0.0.dev0"):
---> 16 from numpy.lib.array_utils import ( # type: ignore[import-not-found,unused-ignore]
17 normalize_axis_index,
18 )
19 else:
20 from numpy.core.multiarray import ( # type: ignore[attr-defined,no-redef,unused-ignore]
21 normalize_axis_index,)

I do not know what should I do to succesfully to run the model. It was working before with google-meridian version 1.0.9, now if I try to install the same old version, it is giving the same error as above. COuld you please help me.

[gousesb]

Hi @OykuOzOzen ,

Thank you for contacting us!

The numpy version was bumped to 2.0.2 in Meridian version 1.0.7. If you're looking to use the functionality in the later versions of Meridian, have you tried the above suggestion to use the 16.3 ML beta runtime? There's an open bug in databricks to support later versions of numpy (databricks/databricks-sql-python#432) that you can also track.

Otherwise, I would recommend you use Meridian version < 1.0.7.

Feel free to reach out for any further queries or feedback regarding Meridian.

Google Meridian Support Team

[klatta87]

Ok after a lot of trial and error, I give up. Can someone just tell me what the latest, best practice is for using Meridian in Databricks.
Databricks Runtime: 16.4 LTS ?
Meridian Version: ?
Numpy Version: ?

Cc: @gousesb

[gousesb]

Hi @klatta87,

Please let us know if you still have issues after following @sonriks6's suggestion .

Google Meridian Support Team

[sonriks6]

Hi @klatta87

This is our current configuration:

• GPU: Standard_NC16as_T4_v3 (110 Gb)
• DBR: 17.0
• Meridian: google-meridian[and-cuda]==1.1.6

This is running in prod environment, no issues so far, no need to install additional dependencies or fix numpy version.

Hope this helps.

[klatta87]

Thank you! This helped immensely. Here's what I went with since we are on Databricks and GCP:

Meridian: google-meridian[and-cuda]==1.1.6
DBR: 17.1 (includes Apache Spark 4.0.0, Scala 2.13)
GPU: g2-standard-8 [L4]