Trends

Backtesting with Skforecast: Time Series Forecasting in Python

Introduction to Backtesting in Forecasting

This is a comprehensive guide to backtesting with skforecast in python with examples. Backtesting in time series forecasting simulates how a model would perform if it had been applied in the past, based on historical data. This process is crucial for validating models before deployment, as it offers insights into potential forecasting errors and helps optimize model performance. By testing how models perform under different scenarios, forecasters can better understand limitations and avoid overfitting.

For instance, backtesting allows you to assess how a model might perform during different seasonal cycles or economic conditions, enabling a more robust evaluation. In this guide, we’ll leverage Skforecast, a Python library that integrates seamlessly with scikit-learn, to facilitate backtesting for time series models.

What is Backtesting?

Backtesting is a validation technique in time series analysis that evaluates the accuracy of forecasting models by testing them on historical data. In forecasting web traffic, backtesting methods help refine predictions and improve model reliability by revealing how they would perform in real-world scenarios. This process is crucial for both web traffic and financial backtesting, as it provides insights into time series trends and supports robust prototyping.

Utilizing a variety of forecaster types and validation strategies, backtesting in time series forecasting often involves tools for training models with machine learning, enhancing predictions with a range of functionalities. Effective backtesting captures the intricacies of time series data, addressing essential aspects such as seasonal variations and feature requests for more specific trends. By modeling time series trends with these advanced techniques, analysts can ensure their models are both accurate and adaptable to changing patterns, making backtesting a core component of forecasting methodologies.

Also Read: How is AI Improving Weather Forecasting?

Examples of Backtesting

Backtesting is widely used across various fields, including intermittent demand forecasting, where it assesses models that predict irregular or sporadic demand patterns. For instance, forecasting energy demand requires handling exogenous variables like weather conditions or public holidays, which significantly impact usage. By incorporating series differentiation and using recursive multi-step forecasting, models can accommodate the complexities of demand with machine learning, improving accuracy. Forecaster objects and time series differentiation enable more precise modeling, while exogenous features capture additional influences that affect energy consumption.

In the context of PV (photovoltaic) power forecasting, backtesting aids in building interpretable forecasting models that can predict power demand and output. Multi-series forecasting is especially relevant here, as energy production data often involve multiple time series, such as different geographic regions or types of PV systems. Techniques like direct multi-step forecasting are used to generate probabilistic forecasts, which provide a range of possible outcomes. These forecasts, combined with quantile regression, offer robust estimates of PV output under various conditions, making the models more resilient to data variability.

For more advanced applications, such as forecasting energy demand with machine learning, a hyperparameter search via random search or k-fold cross-validation fine-tunes model performance. Tools like quantile forecasts and categorical features help produce probabilistic forecasts, giving a more comprehensive view of potential future states. These methods support both single and multi-step forecasters, allowing analysts to explore the implications of different forecast periods. With interpretable forecasting models and robust backtesting techniques, forecasters can better handle the intricacies of time series data and produce reliable forecasts that are crucial for effective decision-making.

What is Skforecast?

Skforecast simplifies time series forecasting by automating tasks like lag creation and offering a wide selection of models through scikit-learn. This tool is suitable for both beginners and advanced users interested in performing autoregressive forecasting and backtesting due to its intuitive interface and support for custom model integrations.

Skforecast’s primary functionality centers on the ForecasterAutoreg class, which is pivotal for implementing autoregressive forecasting models. This class is effective for applications like demand forecasting and stock price prediction by incorporating lag values. Lags allow models to incorporate past values into future forecasts, enhancing accuracy and predictive power. The development of Skforecast focuses on making forecasting accessible while enabling complex forecasting strategies such as recursive and direct approaches.

The recursive strategy, or series recursive, uses previous forecasted values as inputs for future predictions, which is ideal for capturing ongoing patterns but may propagate errors. The direct strategy, on the other hand, forecasts each time step individually, offering a realistic assessment for multi-step forecasts, reducing cumulative error. By combining Skforecast with gradient boosting models, users can achieve robust forecasting and obtain prediction intervals to gauge forecast certainty. This range of functionalities makes Skforecast an adaptable tool for numerous real-world forecasting tasks.

Key Features of Skforecast

  1. Seamless Integration with scikit-learn: Skforecast enables the use of scikit-learn regressors, making it easy to try different models.
  2. Support for Multiple Backtesting Techniques: Skforecast offers both expanding and sliding windows, as well as custom validation splits.
  3. User-Friendly API: Designed with ease of use in mind, Skforecast’s API simplifies setup and customization.

Steps for Effective Backtesting with Skforecast

To ensure effective backtesting, follow these steps for setup and implementation:

Install Skforecast and Dependencies:

    pip install skforecast

    Data Preparation and Formatting: Time series data should be indexed by date and formatted as a pandas DataFrame. Here’s an example of setting up the data:

      import pandas as pd
data = pd.read_csv('time_series_data.csv', index_col='date', parse_dates=True)

      Choose a Forecasting Model: Skforecast allows you to choose from various scikit-learn regressors. For example, here’s how to set up a linear regression model with Skforecast:

      from sklearn.linear_model import LinearRegression
from skforecast.ForecasterAutoreg import ForecasterAutoreg

forecaster = ForecasterAutoreg(
    regressor=LinearRegression(),
    lags=12
)

      Select a Backtesting Strategy:

      • Expanding Window: Ideal for stable data, where you progressively expand the training set.
      • Sliding Window: Keeps a fixed training size and slides over time, effective for data with seasonality.
        1. Run Backtesting and Analyze Results:
        from skforecast.model_selection import backtesting_forecaster

metrics, predictions = backtesting_forecaster(
    forecaster=forecaster,
    y=data['target'],
    initial_train_size=int(len(data)*0.7),
    steps=1,
    metric='mean_squared_error',
    refit=True
)

        Common Backtesting Techniques and Best Practices

        To get accurate results, it’s essential to select the right backtesting technique and follow best practices.

        Expanding Window Technique

        The expanding window method is well-suited for data without drastic changes over time. With each iteration, the training set expands to include more data, improving the model’s ability to capture patterns.

        metrics, predictions = backtesting_forecaster(
    forecaster=forecaster,
    y=data['target'],
    initial_train_size=int(len(data) * 0.7),
    steps=1,
    refit=True,
    metric='mean_absolute_error'
)

        Sliding Window Technique

        The sliding window method maintains a consistent training set size, moving forward with each new data point. This technique is useful for time series data with changing patterns, as it always reflects the most recent data.

        Cross-Validation for Time Series

        To reduce variance and improve model reliability, cross-validation techniques adapted for time series can be used. These methods involve multiple overlapping training/test splits, providing a more comprehensive evaluation of model performance.

        Implementing Backtesting with Skforecast: A Step-by-Step Example with Advanced Techniques

        Let’s expand our example using Skforecast with a more complex regressor, like a Random Forest, which can capture non-linear relationships in data.

        Step 1: Set Up the Random Forest Regressor

        from sklearn.ensemble import RandomForestRegressor
forecaster = ForecasterAutoreg(
    regressor=RandomForestRegressor(n_estimators=100, random_state=42),
    lags=15
)

        Step 2: Apply Sliding Window with Multiple Steps Ahead

        Instead of forecasting one step at a time, you can adjust Skforecast to predict multiple steps ahead.

        metrics, predictions = backtesting_forecaster(
    forecaster=forecaster,
    y=data['target'],
    initial_train_size=int(len(data)*0.7),
    steps=5,
    metric='mean_squared_error',
    refit=False
)
print(metrics)

        Step 3: Visualize the Results

        Visualization helps in understanding model performance over time. Here’s how to plot actual vs. predicted values:

        import matplotlib.pyplot as plt

plt.plot(data.index, data['target'], label='Actual')
plt.plot(predictions.index, predictions, label='Predicted', linestyle='--')
plt.legend()
plt.show()

        Troubleshooting Common Backtesting Issues with Skforecast

        In real-world applications, you may encounter issues such as data leakage, overfitting, or insufficient data. Here are some solutions:

        1. Data Leakage: Ensure no future data is used in training by properly setting up your training/test splits.
        2. Overfitting: Avoid using overly complex models unless you have sufficient data to prevent overfitting. Simpler models like linear regression can often perform adequately.
        3. Insufficient Data: For limited datasets, consider using cross-validation to maximize data usage, but beware of compromising temporal sequence integrity.

        Skforecast vs. Alternatives: Prophet, ARIMA, and statsmodels

        While Skforecast is effective for autoregressive models, other libraries offer specific strengths:

        Prophet

        Prophet is a powerful tool ideal for seasonal data, especially those with daily or hourly cycles. It’s designed to handle seasonality with minimal user input, automatically detecting and integrating holidays into its forecasting model. This makes Prophet well-suited for applications such as retail sales or website traffic forecasting, where seasonal trends and holiday effects play significant roles.

        ARIMA/Statsmodels

        ARIMA, available through the Statsmodels library, excels with stationary data—where mean and variance remain constant over time. ARIMA is a foundational method for time series forecasting, but when data shows seasonal patterns, the SARIMA (Seasonal ARIMA) model adds additional layers, improving forecast accuracy for seasonal datasets. These models are ideal for situations with stable trends and predictable periodic changes.

        LSTMs (via Keras)

        Long Short-Term Memory (LSTM) networks, a type of recurrent neural network available via Keras, are effective for complex, non-linear time series data. They can capture intricate dependencies and patterns across time, making them suitable for highly dynamic datasets such as stock prices or energy consumption. LSTMs require large datasets and substantial computational power, but they excel at recognizing long-term dependencies and trends that simpler models might overlook, making them invaluable for sophisticated forecasting tasks.

        Comparison Table:

        Feature Skforecast Prophet ARIMA/Statsmodels LSTMs (Keras)
        Ideal Data Type Univariate, Linear Seasonal, Trend Stationary Non-linear, Complex
        Ease of Use High Medium Medium Low
        Flexibility High (Custom Models) Medium High High
        Model Complexity Moderate Low Low-Medium High

        Case Study: Using Skforecast for Demand Forecasting

        To illustrate Skforecast’s capabilities, let’s look at a retail demand forecasting case:

        • Problem: A retail chain wants to forecast weekly demand for a specific product.
        • Data: Three years of historical sales data.
        • Objective: Minimize forecast error and ensure stock levels meet demand.

        Implementation Steps:

        1. Load and Preprocess Data: Convert the dataset into a time series format indexed by weeks.
        2. Select Model: Use a Gradient Boosting Regressor, which handles seasonality well.
        3. Backtest and Evaluate: Split the data by year to perform year-over-year backtesting.
        4. Optimization: Use grid search with cross-validation to fine-tune hyperparameters.
        5. Results: Evaluate using MAE to measure forecasting accuracy. Adjust model parameters based on error analysis.

        As AI and machine learning technologies evolve, we see a shift towards more automated solutions for time series forecasting:

        • AutoML: Platforms like H2O.ai offer automated feature engineering, model selection, and hyperparameter tuning for time series.
        • Deep Learning: Architectures like Transformers, originally developed for NLP, are now adapted for time series to capture complex dependencies.
        • Explainable AI: More tools are focusing on interpretability, allowing users to understand how their forecasts are generated.

        Also Read: The role of AI in vaccine distribution

        Conclusion

        Backtesting with Skforecast offers a powerful means to validate time series forecasting models, enabling a thorough assessment of model performance across various scenarios. Skforecast provides essential functionality for time series forecasting, supporting excellent regression algorithms, exponential smoothing, and robust workflows. It allows for the inclusion of exogenous variables, which can improve model accuracy by accounting for external factors affecting trends.

        A wide range of forecasting strategies is available within Skforecast, including simple model setups, hierarchical forecasts, and gradient boosting integration for advanced applications. Using Skforecast, you can easily perform batch forecast evaluations, plot predictions, and create hierarchical forecasts tailored to complex datasets. This flexibility supports both basic forecasting workflows and more complex operations like traffic prediction with machine learning.

        By leveraging Skforecast’s advanced capabilities, such as handling time series differentiation and prediction intervals, you can develop sophisticated models that handle diverse business needs. The tool’s ability to handle gradient boosting for trend modeling, along with support for fields of statistics, enables it to cater to various use cases like forecasting web traffic or financial trends. As Skforecast continues to evolve, it remains a valuable resource for modeling time series trends, offering users an effective way to explore prediction methodologies, avoid pitfalls like false differentiation, and integrate advanced statistical techniques into their forecasting practices.

        References

        Hyndman, Rob J., and George Athanasopoulos. Forecasting: Principles and Practice. 3rd ed., OTexts, 2018. Available online at https://otexts.com/fpp3/.

        James, Gareth, et al. An Introduction to Statistical Learning: With Applications in R. 2nd ed., Springer, 2021.

        Sandanayake, Kanchana. “A Guide to Backtesting: What It Is and How to Get Started.” Skforecast Documentation, Skforecast, 2024, https://skforecast.org/.

        Choudhury, Soumyajit, and J. Cristian Bota. “Comparative Study of Statistical and Machine Learning Models for Time Series Forecasting.” Journal of Business Research, vol. 145, 2023, pp. 1-13.

        Taylor, Sean J., and Benjamin Letham. “Prophet: Forecasting at Scale.” Journal of Open Source Software, vol. 1, no. 1, 2017, p. 3. https://doi.org/10.21105/joss.00013.

        Adhikari, Ratnadip, and R. K. Agrawal. An Introductory Study on Time Series Modeling and Forecasting. Springer, 2013.



Source link

Related Posts