When comparing a multilevel model to a fixed-level model, it’s important to consider how things are parameterized. For instance, let’s say you’re conducting comparisons between a no-pooling model and a partial pooling variance components model. In this case, we have:

$$ y= \Delta u + \epsilon$$

as the specification, where $\Delta$ is the dummy variable matrix, `$y$`

is the outcome of interest, and `$\epsilon$`

is the usual homoeskedastic error term for the responses. Note that the no pooling model contains no intercept and instead contains `$J$`

means, one for each of the $j$ groups.

But, let’s consider what happens when we look at the multilevel estimate for an analogous model:

$$ y = \Delta \theta + \epsilon$$

$$\theta = \mu + \zeta$$

where *now*, $\zeta$ is a zero-mean error term for our *region-level model* of `$\theta$`

. Our $\theta$ term expresses the unique effects of each observation with respect to a *higher-level mean*, `$\mu$`

, with its own estimating uncertainty.

So, what is the correct comparison between terms in these models? Well Gelman has his whole pooled/not pooled dichotomy, and I think that makes sense. Most of the multilevel literature compares `$\zeta$`

to `$u$`

, since `$\zeta$`

reflects the fact that we have “separated” the uncertainty in estimating the global mean out, placed it in `$\mu$`

and now are only concerned with estimating the noise in the distinct contribution added by *being in a group*. Thus, again, for centered `$y$`

, `$u$`

sould look like `$\zeta$`

, but with slightly wider intervals and slightly more extreme point estimates, and `$\mu$`

should be around zero. This argument is quite persuasive, and grounds the interpretation of these models in the literature.

But, the “direct” comparison involves comparing $\theta$ to `$\gamma$`

, which both serve the same direct function in the mean predictor of the model. Practically speaking, if we think this is the correct apples-to-apples comparison, then *both* the uncertainty in `$\alpha_0$`

and `$\zeta$`

are included in comparisons to the fixed effect model’s unique region estimate. An example of how this might be done is:

- run a sampler for the parameters
- at convergence, resampling
`$\mu$`

and`$\zeta$`

from their posteriors and obtain one draw of $\theta$.

I showed this in my recent presentation at the Royal Statistical Society conference. The uncertainty in estimating the state-specific effect $\zeta$ obeys the usual behavior. But, if we actually simulate samples of $\theta$ from the posterior, we get estimate bands that look much more similar to the original bands.

I’m sympathetic to the arguments that you shouldn’t lump the uncertainty in $\mu$ into the uncertainty around `$\zeta$`

, but I also think that, from the fixed-effects user perspective, the “right” intuitive comparison is between `$\theta$`

and `$u$`

. Not sure what this means for applied work; personally, I nearly always want to make the Gelman-style comparison, but I’m wondering if there are cases where it’s more useful to think critically about what uncertainty around `$\mu$`

means in the multilevel specificaation. For my spatial statistical context, end of the day, our predictions for any specific data point *will* involve both the uncertainty in `$\mu$`

and `$\zeta$`

for any observation, so while we can separate them out in the regression model, we cannot really in the discussion of prediction.