Runs with same seed decorrelate after a certain round

[jamesram415]

Hi

I am passing a seed via the booster config as well as "seed_per_iteration": True,

I am training with the iterator method on a single GPU.

I notice the pattern below when training two models with identical seeds where test metrics diverge after a certain round during training.

I notice the same pattern with a smaller diff on the training metrics

This is with xgboost built from source but post-3.0.5

Is this a known issue? thanks

[jamesram415]

Also - this is the distributed iterator approach but using a single GPU for the above plot
They are identical up to round 70 and then gradually move apart.
I believe the training data is identical between the two

[trivialfis]

Could you please share the shape of the data? Also, please update to 3.0.5

[jamesram415]

This has approx 19500 columns and 100k rows. There was 100k rows for validation.
Ah yes I mean to say this is on 3.0.5

[trivialfis]

Thank you for the info! Could you please share which sampling method are you using? Including colsample_by and subsample.

Did a quick test with both sampling methods enabled, did not reproduce. I will find a larger GPU tomorrow.

[trivialfis]

Tested with an Ampere a6000 and an H200, did not reproduce. I'm using subsample=0.8, colsample_bytree=0.8, 256 rounds.
Additionally, I divided the synthetic regression dataset into 2 batches, with the same shape as shared.

[jamesram415]

Many thanks for debugging - I'll double check there's nothing on my side causing this.