Centering in Panel Models

A detailed analysis of centering and some normalization missadventures in panel regression WIP

Abstract

Panel regression models analyze data from the same units observed over multiple time periods, enabling control for time-invariant unobserved unit characteristics and estimation of within-unit effects. A common technique, particularly in fixed-effects models, involves “demeaning” – subtracting unit-specific means from both dependent and independent variables to isolate within-unit variation. While powerful, this and related centering approaches have pitfalls: they preclude estimation of time-invariant predictor effects, and coefficients reflect purely within-unit changes, which may differ from between-unit or overall effects. Such methods focus analysis on within-unit variation, potentially overlooking broader patterns if this variation is minimal, and can complicate the interpretation of interaction terms. Centering techniques can also be missapplied, for example by only being applied to the dependent variable or by dividing by instead of subtracting the group level mean. Consequently, the choice of centering technique fundamentally shapes the questions being addressed and the interpretation of results.

sales_demo_data

<xarray.Dataset> Size: 76kB
Dimensions:      (store_id: 20, time_period: 156, product_id: 1)
Coordinates:
  * store_id     (store_id) int64 160B 1 2 3 4 5 6 7 8 ... 14 15 16 17 18 19 20
  * time_period  (time_period) int64 1kB 1 2 3 4 5 6 ... 151 152 153 154 155 156
  * product_id   (product_id) int64 8B 1
Data variables:
    sales        (store_id, time_period, product_id) float64 25kB 2.2 ... 10.04
    covariate_1  (store_id, time_period, product_id) float64 25kB 0.4296 ... ...
    covariate_2  (store_id, time_period, product_id) float64 25kB 0.6666 ... ...
Attributes:
    betas:                       [-0.01237172  0.07431322]
    seasonality_amplitude:       0.22150897038028766
    trend_slope:                 0.005464504127199977
    store_effects:               [-0.44475205  0.12756087  0.11161652  0.4042...
    base_log_sales_per_product:  [1.24908024]

xarray.Dataset

• Dimensions:
  • store_id: 20
  • time_period: 156
  • product_id: 1
• Coordinates: (3)
  • store_id
    (store_id)
    int64
    1 2 3 4 5 6 7 ... 15 16 17 18 19 20

    array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17, 18,
           19, 20])

  • time_period
    (time_period)
    int64
    1 2 3 4 5 6 ... 152 153 154 155 156

    array([  1,   2,   3,   4,   5,   6,   7,   8,   9,  10,  11,  12,  13,  14,
            15,  16,  17,  18,  19,  20,  21,  22,  23,  24,  25,  26,  27,  28,
            29,  30,  31,  32,  33,  34,  35,  36,  37,  38,  39,  40,  41,  42,
            43,  44,  45,  46,  47,  48,  49,  50,  51,  52,  53,  54,  55,  56,
            57,  58,  59,  60,  61,  62,  63,  64,  65,  66,  67,  68,  69,  70,
            71,  72,  73,  74,  75,  76,  77,  78,  79,  80,  81,  82,  83,  84,
            85,  86,  87,  88,  89,  90,  91,  92,  93,  94,  95,  96,  97,  98,
            99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112,
           113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126,
           127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140,
           141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154,
           155, 156])

  • product_id
    (product_id)
    int64
    1

    array([1])

• Data variables: (3)
  • sales
    (store_id, time_period, product_id)
    float64
    2.2 2.46 2.257 ... 7.872 8.87 10.04

    array([[[ 2.20014538],
            [ 2.46006697],
            [ 2.25713195],
            ...,
            [ 4.99957635],
            [ 6.54083394],
            [ 5.98004403]],
    
           [[ 3.44366332],
            [ 3.98512642],
            [ 4.10876074],
            ...,
            [ 7.9728716 ],
            [ 9.49967535],
            [ 9.83488601]],
    
           [[ 4.71521497],
            [ 4.27953578],
            [ 3.81123469],
            ...,
    ...
            ...,
            [ 8.74263671],
            [ 8.43012539],
            [10.71933388]],
    
           [[ 9.16110797],
            [11.47300089],
            [11.29541857],
            ...,
            [20.25258712],
            [22.9180807 ],
            [21.40687927]],
    
           [[ 3.87633769],
            [ 3.76098006],
            [ 4.49038221],
            ...,
            [ 7.87187451],
            [ 8.87007886],
            [10.04150868]]])

  • covariate_1
    (store_id, time_period, product_id)
    float64
    0.4296 -1.009 ... 0.2061 -0.2842

    array([[[ 0.42958216],
            [-1.00886032],
            [-1.63511595],
            ...,
            [-0.03117877],
            [ 0.45840429],
            [ 2.14985844]],
    
           [[-0.0708224 ],
            [-0.21249966],
            [ 1.11349753],
            ...,
            [-0.92979536],
            [ 0.50266762],
            [-0.29202794]],
    
           [[ 1.54458913],
            [ 1.41137034],
            [ 0.00548954],
            ...,
    ...
            ...,
            [-0.24282971],
            [-2.123726  ],
            [ 0.07961794]],
    
           [[-1.80021537],
            [ 1.16170186],
            [-1.37146226],
            ...,
            [ 0.90555776],
            [-0.79729777],
            [-0.24175421]],
    
           [[-0.2531813 ],
            [ 0.09395003],
            [-0.35201606],
            ...,
            [-0.88867211],
            [ 0.20608888],
            [-0.284199  ]]])

  • covariate_2
    (store_id, time_period, product_id)
    float64
    0.6666 -0.5858 ... 0.2851 0.2037

    array([[[ 0.66657458],
            [-0.58575575],
            [-1.39282621],
            ...,
            [-0.08985905],
            [ 0.23471522],
            [ 2.43593884]],
    
           [[-0.00458811],
            [-0.34196549],
            [ 1.11175663],
            ...,
            [-1.21112759],
            [ 0.48983728],
            [-0.02517022]],
    
           [[ 1.26184199],
            [ 1.6979829 ],
            [-0.27172106],
            ...,
    ...
            ...,
            [-0.17222864],
            [-1.81783533],
            [ 0.14365546]],
    
           [[-1.47751046],
            [ 1.50453944],
            [-0.87181092],
            ...,
            [ 0.99883275],
            [-0.84396878],
            [-0.34967486]],
    
           [[ 0.00729364],
            [ 0.22927749],
            [-0.29032389],
            ...,
            [-1.06890621],
            [ 0.28512372],
            [ 0.20369718]]])

• Indexes: (3)
  • store_id
    PandasIndex

    PandasIndex(Index([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20], dtype='int64', name='store_id'))

  • time_period
    PandasIndex

    PandasIndex(Index([  1,   2,   3,   4,   5,   6,   7,   8,   9,  10,
           ...
           147, 148, 149, 150, 151, 152, 153, 154, 155, 156],
          dtype='int64', name='time_period', length=156))

  • product_id
    PandasIndex

    PandasIndex(Index([1], dtype='int64', name='product_id'))

• Attributes: (5)

  betas :

  [-0.01237172 0.07431322]

  seasonality_amplitude :

  0.22150897038028766

  trend_slope :

  0.005464504127199977

  store_effects :

  [-0.44475205 0.12756087 0.11161652 0.40420611 -0.23235125 -0.21006792 -0.22855207 -0.36963314 -1.04501961 0.38014787 0.32657803 -0.6095504 -0.17121843 -0.29696273 -0.28133752 -0.85584826 -0.25178998 0.23908819 1.02379521 0.15769321]

  base_log_sales_per_product :

  [1.24908024]

time_index = sales_normed.time_period.values.astype(int)

seasonal_control = np.sin(2 * np.pi * time_index / 52)
trend = time_index/52

# Fit a linear regression model
sales_df = sales_demo_data.to_dataframe().reset_index()
control_df = pd.DataFrame({
    'seasonal_control': seasonal_control,
    'trend': trend,
    'time_period': time_index
})
control_df['time_period'] = control_df['time_period']

total_df = sales_df.merge(control_df, on='time_period',  how='left')

train_df = total_df[total_df['time_period'] < 104].copy().set_index(['store_id', 'time_period'])
test_df = total_df[total_df['time_period'] >= 104].copy().set_index(['store_id', 'time_period'])

# Creat the dependent variable and independent variables
X_train = sm.add_constant(train_df[['seasonal_control', 'trend', 'covariate_1', 'covariate_2']])
y_train = np.log(train_df['sales'])
y_train_div = y_train/y_train.groupby('store_id').mean()
y_train_sub = y_train-y_train.groupby('store_id').mean()

# Fit the regression model
ME_model_standard = lm.RandomEffects(y_train, X_train)
ME_model_div = lm.RandomEffects(y_train_div, X_train)
ME_model_sub = lm.RandomEffects(y_train_sub, X_train)

# Fit the model
fitted_model_standard = ME_model_standard.fit()
fitted_model_div = ME_model_div.fit()
fitted_model_sub = ME_model_sub.fit()

Div ‘Normed’ Model

Figure 1: Sales model fitted with div-normed dependent variable

Model Summary

fitted_model_div.summary

RandomEffects Estimation Summary

Dep. Variable:	sales	R-squared:	0.6325
Estimator:	RandomEffects	R-squared (Between):	-8.194e+27
No. Observations:	2060	R-squared (Within):	0.6335
Date:	Wed, May 21 2025	R-squared (Overall):	0.6325
Time:	21:57:24	Log-likelihood	1544.7
Cov. Estimator:	Unadjusted
		F-statistic:	884.29
Entities:	20	P-value	0.0000
Avg Obs:	103.00	Distribution:	F(4,2055)
Min Obs:	103.00
Max Obs:	103.00	F-statistic (robust):	884.29
		P-value	0.0000
Time periods:	103	Distribution:	F(4,2055)
Avg Obs:	20.000
Min Obs:	20.000
Max Obs:	20.000

Parameter Estimates

	Parameter	Std. Err.	T-stat	P-value	Lower CI	Upper CI
const	0.7644	0.0054	140.36	0.0000	0.7538	0.7751
seasonal_control	0.1838	0.0039	47.554	0.0000	0.1762	0.1913
trend	0.2288	0.0048	47.656	0.0000	0.2194	0.2383
covariate_1	-0.0085	0.0103	-0.8255	0.4092	-0.0286	0.0117
covariate_2	0.0517	0.0103	5.0180	0.0000	0.0315	0.0719

(sales_demo_data.attrs['betas'][None,:]/y_train.groupby('store_id').mean().values.flatten()[:, None]).mean(axis=0)

array([-0.00995315,  0.05978558])

fitted_model_div.variance_decomposition

Effects                   0.000000
Residual                  0.013185
Percent due to Effects    0.000000
Name: Variance Decomposition, dtype: float64

Standard MEM

Figure 2: Sales model fitted with non-normed dependent variable

Model summary

fitted_model_standard.summary

RandomEffects Estimation Summary

Dep. Variable:	sales	R-squared:	0.7741
Estimator:	RandomEffects	R-squared (Between):	0.0074
No. Observations:	2060	R-squared (Within):	0.7755
Date:	Wed, May 21 2025	R-squared (Overall):	0.1414
Time:	21:57:24	Log-likelihood	1784.2
Cov. Estimator:	Unadjusted
		F-statistic:	1760.2
Entities:	20	P-value	0.0000
Avg Obs:	103.00	Distribution:	F(4,2055)
Min Obs:	103.00
Max Obs:	103.00	F-statistic (robust):	1760.2
		P-value	0.0000
Time periods:	103	Distribution:	F(4,2055)
Avg Obs:	20.000
Min Obs:	20.000
Max Obs:	20.000

Parameter Estimates

	Parameter	Std. Err.	T-stat	P-value	Lower CI	Upper CI
const	1.1318	0.1150	9.8430	0.0000	0.9063	1.3573
seasonal_control	0.2284	0.0034	66.389	0.0000	0.2216	0.2351
trend	0.2850	0.0043	66.675	0.0000	0.2766	0.2934
covariate_1	-0.0165	0.0095	-1.7267	0.0844	-0.0352	0.0022
covariate_2	0.0764	0.0096	8.0036	0.0000	0.0577	0.0952

sales_demo_data.attrs['betas']

array([-0.01237172,  0.07431322])

fitted_model_standard.variance_decomposition

Effects                   0.264310
Residual                  0.010396
Percent due to Effects    0.962157
Name: Variance Decomposition, dtype: float64

Dependant Sub-Normalized

Figure 3: Sales model fitted with sub-normed dependent variable

Model Summary

fitted_model_sub.summary

RandomEffects Estimation Summary

Dep. Variable:	sales	R-squared:	0.7740
Estimator:	RandomEffects	R-squared (Between):	-1.327e+28
No. Observations:	2060	R-squared (Within):	0.7754
Date:	Wed, May 21 2025	R-squared (Overall):	0.7740
Time:	21:57:24	Log-likelihood	1785.6
Cov. Estimator:	Unadjusted
		F-statistic:	1759.1
Entities:	20	P-value	0.0000
Avg Obs:	103.00	Distribution:	F(4,2055)
Min Obs:	103.00
Max Obs:	103.00	F-statistic (robust):	1759.1
		P-value	0.0000
Time periods:	103	Distribution:	F(4,2055)
Avg Obs:	20.000
Min Obs:	20.000
Max Obs:	20.000

Parameter Estimates

	Parameter	Std. Err.	T-stat	P-value	Lower CI	Upper CI
const	-0.2942	0.0048	-60.719	0.0000	-0.3037	-0.2847
seasonal_control	0.2284	0.0034	66.429	0.0000	0.2216	0.2351
trend	0.2850	0.0043	66.710	0.0000	0.2766	0.2934
covariate_1	-0.0140	0.0091	-1.5283	0.1266	-0.0319	0.0040
covariate_2	0.0729	0.0092	7.9594	0.0000	0.0550	0.0909

sales_demo_data.attrs['betas']

array([-0.01237172,  0.07431322])

fitted_model_sub.variance_decomposition

Effects                   0.000000
Residual                  0.010396
Percent due to Effects    0.000000
Name: Variance Decomposition, dtype: float64

Comparison of Normalization

Figure 4: The divided MEM model and the standard MEM model produce much different prediction. Using the wrong model structure will produce biased predictions, and missleading results.