[QUESTION] Why does historical_forecasts() seem to require longer training data VS using predict() method??

[lubibi13]

model_tsmx_luz_temp = LinearRegressionModel(lags=[-1,-2,-3,-4,-5,-6,-7,-14,-21,-30,-60,-90,-120,-150,-180,-210,-240,-365],
output_chunk_length=365,
multi_models=False,
random_state=43)

Here is a snapshot of my target series:

hfc_params_mult_luz = {
"series": temp_max_ser_luz,
"forecast_horizon": 365,
"verbose": True,
"retrain": True,
}

historical_fcasts_mult_luz = model_tsmx_luz_temp.historical_forecasts(last_points_only=False, stride=3, **hfc_params_mult_luz)

---

Why did historical_forecasts() decide to start its first forecast on 2024-01-01 and not 2023-01-01??
There is enough training data for the model to start its forecast on 2023-01-01 which I tried using a simple predict() method.

Using the same model...
model_tsmx_luz_temp.fit(samp_temp_exa)
pred_tsmx_luz_temp = model_tsmx_luz_temp.predict(365)

[dennisbader]

Also here, could you please try to improve the issue formulation as described in comment here?

The historical forecasts earliest start point depends on many factors.

Ignoring all parameters such as train_length, start, ... the minimum length for training is given by

ForecastingModel.min_train_series_length which is equal to

sum(model._target_window_lengths) + (model.min_train_samples - 1)

For all SKLearnModel children (e.g. your linear regression model) these parameters are:

• _target_window_lengths = (max(abs(lags)), output_chunk_length + output_chunk_shift). In your case (365, 365)
• min_train_samples = 2 ( some of the sklearn-like models require 2 samples). In your case (2)

For your case this means that the first training set requires sum(365, 365) + (2 -1) = time steps of your target series.

[lubibi13]

ISSUE: Why did historical_forecasts() decide to start its first forecast on 2024-01-01 and not 2023-01-01??
There is enough training data for the model to train and start its forecast on 2023-01-01 which I tried using a simple predict() method using the same model and training data. From your comment above, it only needs at least sum(365, 365) + (2 -1) = time steps for the first training set, and I provided data that spans from 2021-01-01 to 2025-10-22.

You may simply run the code below and see the first forecast produced by historical_forecasts().

import pandas as pd
import numpy as np

def create_luzon_demand_data(start_date, end_date):
    """
    Generates synthetic daily demand data for Luzon.
    """
    
    # 1. Create Date Range
    dates = pd.date_range(start=start_date, end=end_date, freq='D')
    n_days = len(dates)
    
    df = pd.DataFrame({'Date': dates})

    # --- Define Demand Model Parameters ---
    
    # Baseline and Trend
    # Starting at 9,000 MW, growing by ~0.7 MW/day
    baseline = 9000  
    linear_growth_factor = 0.7 
    trend = baseline + (linear_growth_factor * np.arange(n_days))

    # Annual Seasonality (Hot Season Peak)
    # Amplitude of 2,500 MW
    # We shift the sine wave to peak around day 120 (late April / early May)
    annual_amplitude = 2500
    day_of_year = df['Date'].dt.dayofyear.values
    # sin(x) peaks at pi/2. We want peak at d=120. (120 - shift) * 2*pi/365.25 = pi/2 => shift ≈ 29
    annual_seasonality = annual_amplitude * np.sin((day_of_year - 29) * 2 * np.pi / 365.25)

    # Weekly Seasonality (Weekend Dips)
    # Max drop of 1,500 MW on Sundays
    weekly_amplitude = 1500  
    day_of_week = df['Date'].dt.dayofweek  # Monday=0, Sunday=6
    
    # Define multipliers for each day
    # Mon-Fri: 0% drop
    # Sat: 60% drop
    # Sun: 100% drop
    weekly_multipliers = {
        0: 0,    # Monday
        1: 0,    # Tuesday
        2: 0,    # Wednesday
        3: 0,    # Thursday
        4: 0,    # Friday
        5: -0.6, # Saturday
        6: -1.0  # Sunday
    }
    
    weekly_seasonality = day_of_week.map(weekly_multipliers) * weekly_amplitude

    # Random Noise
    # Normally distributed noise with a standard deviation of 400 MW
    noise_scale = 400
    noise = np.random.normal(loc=0, scale=noise_scale, size=n_days)

    # --- Combine Components ---
    
    # Sum all components
    df['Luzon_demand'] = (
        trend + 
        annual_seasonality + 
        weekly_seasonality.values +  # Use .values to align
        noise
    )
    
    # Ensure demand is non-negative and format as integer
    df['Luzon_demand'] = df['Luzon_demand'].clip(lower=0).astype(int)
    
    return df

# --- Generate and Display Data ---

start_date = '2021-01-01'
end_date = '2025-10-22'

demand_data = create_luzon_demand_data(start_date, end_date)

# Display the beginning of the dataset
print("--- Start of Data (2021) ---")
print(demand_data.head(10))

print("\n--- End of Data (2025) ---")
# Display the end of the dataset
print(demand_data.tail(10))

demand_data_ser = TimeSeries.from_dataframe(demand_data, 'Date', 'Luzon_demand')

model_lr_demand = LinearRegressionModel(lags=[-1,-2,-3,-4,-5,-6,-7,-14,-21,-30,-60,-90,-120,-150,-180,-210,-240,-365],
output_chunk_length=365,
multi_models=False,
random_state=43)

hfc_params_luz_demand = {
"series": demand_data_ser,
"forecast_horizon": 365,
"verbose": True,
"retrain": True,
}

historical_fcasts_luz_demand = model_lr_demand.historical_forecasts(last_points_only=False, stride=3, **hfc_params_luz_demand)

display(historical_fcasts_luz_demand[0])

[dennisbader]

Hi @lubibi13 and thanks for updating the issue description.

I further reduced the example to the actual minimum to explain the behavior. See the code here (difference is that we have a monthly series starting at 1949-01-01):

from darts.datasets import AirPassengersDataset
from darts.models import LinearRegressionModel

series = AirPassengersDataset().load()
hfc_kwargs = {
    "series": series,
    "forecast_horizon": 12,
    "verbose": True,
    "retrain": True,
    "last_points_only": False,
}

for multi_models in [True, False]:
    model = LinearRegressionModel(
        lags=[-1,-12],
        output_chunk_length=12,
        multi_models=multi_models,
    )
    hfc = model.historical_forecasts(**hfc_kwargs)
    print(hfc[0].start_time())

This prints:

start with `multi_models=True`: 1951-02-01 00:00:00
start with `multi_models=False`: 1952-01-01 00:00:00

The reason why you need a longer lookback for prediction is that you specified multi_models=False. With multi_models=True, the start is what you expect.
With multi_models=False, only one estimator is fitted to predict the last point in the output chunk (12 in the current use case). In order to forecast all steps before that (1-12), we need to shift the lookback window back in time (one additional step back for output step 11, two steps back for step 10, ...). That's why more steps for predictions are required and why the historical forecast starts later.

This is further explained here and here.

Hope this clarifies things :)

If you only want to predict the last step (output step 12) using the same input window as with multi_models=True, then you set output_chunk_length=1, output_chunk_shift=11, multi_models=True at model creation, and forecast_horizon=1 for historical_forecasts().

[lubibi13]

I know how training works when multi_models = True/False. I already tried doing a simple fit-predict method using the same model (with multi_models=False) on just 2 years of daily data and the model was able to predict starting 2023-01-01 (run code below). I SHOULD HAVE ENOUGH TRAINING DATA FOR THE historical_forecasts() to start a forecast earlier than 2024-01-01 even if multi_models=False but why did it choose to start its forecast on 2024-01-01.

demand_data = create_luzon_demand_data(start_date, end_date)
demand_data_ser = TimeSeries.from_dataframe(demand_data, 'Date', 'Luzon_demand')

model_lr_demand = LinearRegressionModel(lags=[-1,-2,-3,-4,-5,-6,-7,-14,-21,-30,-60,-90,-120,-150,-180,-210,-240,-365],
output_chunk_length=365,
multi_models=False,
random_state=43)

demand_data_ser_slice = demand_data_ser.slice_n_points_after(pd.Timestamp('2021-01-01'), 365*2)
demand_data_ser_slice

model_lr_demand.fit(demand_data_ser_slice)

print(model_lr_demand.predict(365))

[lubibi13]

You even said it yourself that "For your case this means that the first training set requires sum(365, 365) + (2 -1) = time steps of your target series." In my case, data from 2021-01-01 to 2023-01-01 is enough as the 1st training set which also means the model should be able to start its forecast on 2023-01-02, correct? But for some weird reason, it decided that it can only forecast starting 2024-01-01...

[dennisbader]

Hi @lubibi13, looking at the issue again, I think you might be right and this could indeed be a bug. I'll add it to our backlog to investigate this. Thanks!