• Preface
  • 1 Chapter 1: Time Series Fundamentals
    • 1.1 Definition
    • 1.2 White Noise: purely random process
    • 1.3 Random Walk: random but dependent
    • 1.4 Time Series Modelling
      • 1.4.1 Log Transformation
      • 1.4.2 Box-Cox Transformation
    • 1.5 Time Series Properties
      • 1.5.1 Mean function (if mean is constant)
      • 1.5.2 Variance function (if variance is constant)
      • 1.5.3 Autocovariance function (ACVF):
      • 1.5.4 Autocorrelation function (ACF):
      • 1.5.5 Properties
    • 1.6 Stationarity
  • 2 Chapter 1 Lab
    • 2.1 White Noise (WN)
    • 2.2 Random Walk (RW)
    • 2.3 Stationary
      • 2.3.1 Examples: CPI Data
    • 2.4 ACF to Time Series Plot
  • 3 Chapter 2: Modelling trends and seasonal patterns
    • 3.1 Method 1: Regression
      • 3.1.1 Examples
      • 3.1.2 Other common models
    • 3.2 Method 2: Moving Average Smoothing
    • 3.3 Method 3: Differencing
      • 3.3.1 Remove Trends
      • 3.3.2 Remove Seasonality
    • 3.4 Choosing a smoothing parameter
      • 3.4.1 Simplicity
      • 3.4.2 Objective criteria (AIC & BIC)
  • 4 Chapter 2 Lab
    • 4.1 Model choice and Residual analysis
    • 4.2 Data Example: Annual Varve Series
  • 5 Chapter 3: Autoregressive processes
    • 5.1 Definition
    • 5.2 First order Autoregressive process
      • 5.2.1 Mean
      • 5.2.2 Variance
      • 5.2.3 Examples
    • 5.3 \(AR(p)\) process
      • 5.3.1 Mean
      • 5.3.2 Stationarity
      • 5.3.3 Variance
      • 5.3.4 Autocorrelation Function
    • 5.4 When and how to use AR(p) model
      • 5.4.1 Definition of PACF
  • 6 Chapter 3 Lab
    • 6.1 Autoregressive model (AR)
    • 6.2 Chicken Price
    • 6.3 Chicken price
    • 6.4 Unemployment
  • 7 Chapter 4: Moving Average processes
    • 7.1 Definition
    • 7.2 Mean and Variance
    • 7.3 Autocorrelation functions
      • 7.3.1 Examples:
      • 7.3.2 Another example
    • 7.4 Invertibility
      • 7.4.1 Inveritibility Theorem
    • 7.5 MA Model Identification
      • 7.5.1 Example
    • 7.6 MA parameter estimation
    • 7.7 MA Example
  • 8 Chapter 4 Lab
    • 8.1 Moving Average model (MA)
  • 9 Chapter 5: More general time series processes
    • 9.1 ARMA model
      • 9.1.1 Mean of ARMA(p,q) process
      • 9.1.2 Variance and autocorrelation function
      • 9.1.3 Stationarity and Invertibility
      • 9.1.4 ARMA model identification
      • 9.1.5 ARMA parameter estimation
      • 9.1.6 ARMA simulation example
    • 9.2 ARIMA model
      • 9.2.1 Definition
      • 9.2.2 ARIMA model identification
      • 9.2.3 Example 1
      • 9.2.4 Example 2
  • 10 Chapter 5 Lab
    • 10.1 Oil Return
    • 10.2 Air Passengers
    • 10.3 Birth Rate
  • 11 Chapter 6: Forecasting
    • 11.1 General problem
    • 11.2 Regression
      • 11.2.1 Example 1
      • 11.2.2 Example 2
      • 11.2.3 Example 3
    • 11.3 Exponential smoothing
      • 11.3.1 Definition
      • 11.3.2 Choosing \(\alpha\)
      • 11.3.3 Measure uncertainty
    • 11.4 Forecasting from AR(p) models
      • 11.4.1 AR(1) forecasting
      • 11.4.2 AR(p) forecasting
      • 11.4.3 Example 1: Simulated data
      • 11.4.4 Example 2
      • 11.4.5 Example 3
    • 11.5 Forecasting from MA(q) models
      • 11.5.1 MA(1) forecast
      • 11.5.2 MA(q) forecasts
    • 11.6 Forecasting time series with trend, seasonality and correlation
      • 11.6.1 Example
  • 12 Chapter 6 Lab
    • 12.1 Forecasting with ARIMA
    • 12.2 Global Temp data
    • 12.3 Exponential smoothing
  • References

Time Series

3 Chapter 2: Modelling trends and seasonal patterns

  1. Trend and Seasonality can be the main interests

  2. Need to remove trend and seasonality to determine short-term correlation

For this chapter, assume additivity: \(X_t=m_t+s_t+e_t\) (Refer to Ch.1)

First is to estimate the trend and seasonal variation \(\hat m_t+\hat s_t\), then calculate the residual series \(e_t^*=X_t-\hat m_t-\hat s_t\)

3 of the most common methods for modelling trend and seasonality are described here.

3.1 Method 1: Regression

\[m_t+s_t=\beta_0+\beta_1z_{t1}+...+\beta_pz_{tp}\] Cons: OLS estimation assumes observations are independent. Pros: Remove trend to later model the correlation by a stationary time series process

\(z_{tn}\) are often functions of time

3.1.1 Examples

3.1.1.1 Linear trend in time but no seasonal variation:

Trend can be modelled by \(X_t=\beta_0+\beta_1t+e_t\) and using series \(e_t\) to analyze short-term correlation

3.1.1.2 Quadratic trend in time but no seasonal variation:

Trend can be modeled by \(X_t=\beta_0+\beta_1t+\beta_2t^2+e_t\)

3.1.1.3 Seasonal pattern in time and a linear trend with 365 data points:

Trend and Seasonality can be modeled by \(X_t=\beta_0+\beta_1t+\beta_2sin(8\pi t/365)+e_t\)

3.1.2 Other common models

3.1.2.1 Other Covariates

\[m_t=\beta+0+\beta_1\alpha_t\]

3.1.2.2 Polynomials

\[m_t=\beta+0+\beta_1t+\beta_qt^q\] Higher \(q\), more flexible the trend

3.1.2.3 Harmonics

\[s_t=\beta_0+\sum_{i=1}^q\beta_{1i}sin(w_it)+\beta_{2i}cos(w_it)\] Harmonics assume the seasonal pattern has a regular shape, i.e. the height of the peaks is the same as the depth of the troughs.

3.1.2.4 Seasonal Factors

Assuming the seasonal pattern repeats itself every d time points, a less restrictive approach is to model it as:

\[s_{t}=\left\{\begin{array}{cc}0&\mbox{if}~t=1, d+1, 2d+2,\ldots\\ s_{2}&\mbox{if}~t=2, d+2, 2d+2,\ldots\\ \vdots&\vdots\\ s_{d}&\mbox{if}~t=d, 2d, 3d,\ldots\\\end{array}\right.\]

This model can be fitted by creating \(d - 1\) dummy variables in the design matrix, that contain 1’s and 0’s.

3.1.2.5 Natural cubic splines (More flexible)

\[m_{t}+s_{t}=\beta_{0}+\sum_{i=1}^{q}\beta_{i}B_{i}(t)\]

3.2 Method 2: Moving Average Smoothing

A Moving average smoother estimates the trend and seasonal variation at time t by averaging the current observation and the q either side

\[\hat{m}_{t}+\hat{s}_{t}=\frac{1}{2q+1}\sum_{j=-q}^{q}x_{t-j}\] Note: Shorten length of time series by \(2q\), therefore if the trend is smooth and \(q\) is large, the series shortens significantly.

3.3 Method 3: Differencing

3.3.1 Remove Trends

3.3.1.1 First order difference operator \(\nabla\):

\[\nabla X_{t}=X_{t}-X_{t-1}=(1-B)X_{t}\]

where \(B\) is the Backshift operator defined as \(BX_{t}=X_{t-1}\)

3.3.1.2 General order difference operator \(\nabla^q\)

\[\nabla^{q} X_{t}=\nabla[\nabla^{q-1}X_{t}]\] \[B^{q}X_{t}=X_{t-q}\] Notes: 1. A polynomial trend of order \(q\) can be removed by \(q^{th}\) order differencing 2. Typically only first or second order differencing is required 3. Shortens length by \(q\) 4. Differencing won’t allow to estimate the trend but only to remove it.

3.3.2 Remove Seasonality

The seasonal difference of order \(d\) is the operator \(\nabla_d\) given by \[\nabla_{d} X_{t}=X_{t}-X_{t-d}=(1-B^{d})X_{t}\] Notes: 1. Trial and Error approach 2. No point differencing twice if once is adequate 3. Differencing increases the variance

3.4 Choosing a smoothing parameter

3.4.1 Simplicity

Simple statistical models are preferred since they are easier to make inferences from.

3.4.2 Objective criteria (AIC & BIC)

Assumptions: 1. Normally distributed observations 2. \(m_{t}+s_{t}=\textbf{z}_{t}^\top \boldsymbol{\beta}\)

where \(\textbf{z}_{t}^\top\) is a vector of \(q-1\) known covariates and a one to represent the intercept term

3.4.2.1 Akaike’s Information Criterion (AIC)

\[\mbox{AIC}(q)=-2\mathcal{L}(\mathbf{x}|\hat{\boldsymbol{\beta}}) + 2q\]

3.4.2.2 Bayesian Information Criterion (BIC)

\[\mbox{BIC}(q)=-2\mathcal{L}(\mathbf{x}|\hat{\boldsymbol{\beta}}) + \ln(n)q\]

where \(\mathcal{L}(\mathbf{x}|\hat{\boldsymbol{\beta}})\) is the maximised log likelihood function of \(\mathbf{x}\)

Both criteria tradeoff the fit to the data against simplicity (i.e. few parameters), and small values suggest a good fit to the data.