Multinomial logistic intercepts applied on incorrect scale

[david-cortes]

It appears the multinomial logistic objective is calculating a base score in the scale of the reponse variable (probabilities) and applying it as-is instead of transforming it to the 'margin' scale.

This shoud predict the class proportions for every observation:

import numpy as np
import xgboost as xgb

y = np.r_[
    np.zeros(25),
    np.ones(25),
    np.repeat(2, 50)
]
rng = np.random.default_rng(seed=123)
X = rng.standard_normal(size=(100,10))
model = xgb.XGBClassifier(
    n_estimators=1,
    gamma=1e10,
    min_child_weight=1e10,
).fit(X, y)
model.predict_proba(X[:5])

array([[0.30450436, 0.30450436, 0.39099133],
       [0.30450436, 0.30450436, 0.39099133],
       [0.30450436, 0.30450436, 0.39099133],
       [0.30450436, 0.30450436, 0.39099133],
       [0.30450436, 0.30450436, 0.39099133]], dtype=float32)

It looks like the intercepts are calculated correctly if seen in the scale of the response variable:

import json
json.loads(
    model.get_booster().save_config()
)["learner"]["learner_model_param"]["base_score"]

'[2.5E-1,2.5E-1,5E-1]'

.. but the probabilities are calculated by applynig softmax on that, without converting those numbers to the 'margin' scale:

from scipy.special import softmax
softmax(
    json.loads(
        json.loads(
            model.get_booster().save_config()
        )["learner"]["learner_model_param"]["base_score"]
    )
)

array([0.30450434, 0.30450434, 0.39099132])

And as a result, the predictions are biased.

[trivialfis]

It's a biased estimation indeed, only used for being better than the old universal 0.5 intercept. I did not add the transform as there's no exact invert for the softmax function. (can't remember what's the algebra term for functions that have $y = f^{-1} \cdot f(y)$ ) If we were to select a constant, the problem seems to be more complicated without significant benefit. For instance, we need to apply the softmax and inverse during model load, which needs to reproduce the user input somehow.

Suggestions are welcome.

[david-cortes]

I suggest the approach used by glmnet where the intercepts are constrained to sum to zero:

import numpy as np
from scipy.special import softmax
probs = np.array([0.25, 0.25, 0.5])
raw = np.log(probs)
raw -= raw.mean()
softmax(raw)

array([0.25, 0.25, 0.5 ])

Either that, or setting the last intercept to zero:

raw = np.log(probs)
raw -= raw[-1]
softmax(raw)

array([0.25, 0.25, 0.5 ])

If we were to select a constant, the problem seems to be more complicated without significant benefit.

There is significant benefit: with the intercept estimated optimally, the resulting model will have the property that $\mathbb{E}[\hat{y}] = y$, which is in many cases as important as making good predicitons.

[trivialfis]

Thank you for sharing. Let me try again for this release. I did try to constrain the input, but it broke all the models from prior XGBoost versions, which I wanted to avoid. Let me try to figure something out.

which is in many cases as important as making good predicitons.

Could you please share more on this? Why is this as important as making good predictions?

[david-cortes]

I did try to constrain the input, but it broke all the models from prior XGBoost versions, which I wanted to avoid.

If those models have a scalar base score, you could as well just ignore it, since adding the same constant score to all classes doesn't change the calculations of function/gradient/hessian/predictions.

Why is this as important as making good predictions?

One might have applications where model predictions need to be correct on average to be useful. For example, if the output from multiple predictions is aggregated for some final criterion (e.g. things dealing with logistics or supply chains); or if one wants to look at something like biases or effects of features on aggregates (where a biased model might pick ilogical splits just to correct constant biases); among others. And yes, there are ways of correcting predictions in a post-hoc, but it gets quite complicated if combined with e.g. monotonic constraints and similar.