Negative baseline and ways to tune it

[Anne-lim-te]

I am a beginner in MMM and am currently using Meridian to model monthly data, resulting in a dataset of over 83 months. One of the issues I have been facing is the negative baseline.

Our KPI is non-revenue, specifically conversion. For priors, the $\mu$ is referenced from a real-life scenario with a slightly narrow $\sigma$ and the current model I'm using is 'MeridianStanModel' with geometric adstock function.

Here's what my roi_m custom prior looks like:

channel_1=(0.00895, 0.5), 
channel_2=(0.00063, 0.4), 
channel_3=(0.00316, 0.4), 
channel_4=(0.001, 0.5), 
channel_5=(0.00809, 0.6), 

Here's how my graph of expected KPI vs Actual KPI looks.

My questions are:

1. What are the causes for negative baselines in MMM, and how can I avoid my model from overemphasizing my baseline?
2. How can I plot the distribution of all media coefficients?

As someone still getting a grasp on Bayesian modelling, it's been a bit frustrating to see the baseline go negative despite following guidance on priors. I'm eager to learn where I might be missing something and how others overcame this challenge. I would really appreciate any insights into potential pitfalls or adjustments. Thanks for your expertise!

[dattatreyam23]

Hi @Anne-lim-te,

Thank you for contacting us.

To answer your first question, from the graph, it seems that the negative baseline only happens at one dip, which indicates minor error, which is inherent in any statistical model. We advise against overemphasizing a baseline time series in Model fit charts that occasionally dips into negative values. We also recommend assessing the negativity of the baseline probabilistically, or examining the point estimate of the baseline aggregated over the entire time window, as in Channel contribution charts. All these are explained in detail on our documentation page.

To answer your query about plotting media coefficients, in Meridian you can access the model coefficients from the inference_data object where they are stored by channel. You may also check out some of our example codes on the model diagnostics page. For example, the following codes plot both prior and posterior distributions of beta_m.

model_diagnostics = visualizer.ModelDiagnostics(meridian)
model_diagnostics.plot_prior_and_posterior_distribution('beta_m')

Another example, the following codes plot both the density and trace plots of beta_m

parameters_to_plot=["beta_m"]
for params in parameters_to_plot:
  az.plot_trace(
      meridian.inference_data,
      var_names=params,
      compact=False,
      backend_kwargs={"constrained_layout": True},
  )

You can also either extract the entire posterior distribution or you can get the mean and credible intervals of the parameter. The posterior distribution corresponds to the final values to which the model converged to. You may extract the coefficients, and use some visualisation library, say Matplotlib, to plot. For sample codes for the extraction of coefficients, you may check out this previous answer for Issue #806.

Feel free to reach out if you have any questions or suggestions regarding Meridian.

Thank you,

Google Meridian Support Team

[Jon77Ruler]

I'm in the learning process too; but two things I thought on seeing the chart were

1. You've a very complex baseline. It that being driven by control variables, or have you put in a huge number of knots? You could move a control variable in to a non_media treatment if you want to see what's happening in more detail; afaik they're modelled the same, you just get to see the outcome in the decomposition tables (no?)

2. The model fit isn't great, particularly across 2021 and July-July 23-24. I've had that with some models that are data limited. You've said you've got real life coefficients, but I wonder if it's worth trying something like an estimate using share of spend to start with, via the contribution media_prior_type. Just to see what that does. Especially if the real-life data came from a short A/B test.

You could always throw it in to Robyn as a compare-contrast ;) (this is where I get banned from the site!). See what it thinks. You don't have to stick with it, but a 2nd opinion is always nice. As it uses Prophet to drive the baseline, you might find something you can extract from there as a control.

[cpulavarthi]

Hello @Jon77Ruler,

We appreciate your active participation in the Meridian's community forums!

We want to caution that you should decide if a variable is a control or non-media channel from a causal perspective. See this documentation page for more detail. Specifically, if you decide to re-categorize a control to be a non-media channel, you need to consider if the remaining set of controls have captured the confounders for this “new” non-media channel, such that you have an unbiased causal effect estimated for this “new” non-media channel.

There are also comments on “...complex baseline…have you used full knots…” and “...not-great model fit across 2021 and July23-July24…”. In general, it is helpful to think about bias-variance trade-off in model specification. From previous threads, it was not clear to me if you are using a geo model or a national model. For a geo model, the default knot setting is full knots. For a national model, the default knot setting is single knot. You could try to manually specify knot locations in ModelSpec around those aforementioned periods. (Note that, for a national model, you don't have multiple observations per time period and there are not enough degrees of freedom for each time period to get its own knot.)

Finally, regarding prior specification, we welcome users to try any priors that make the most sense from their business perspective. You can also read this documentation page and this documentation page for some helpful advice on setting $\mu$ and $\sigma$ for ROI prior.

I hope this provides some additional insights into Meridian's workings. Do reach out if you have any questions or suggestions for us!

Google Meridian Support Team