Linear Models and Regression Questions and Answers

Answer: A

The covariance is Σ X(X'X)⁻¹X'.

The fitted values are \(\hat{y} = Hy\) with \(H = X(X'X)^{-1}X'\). Writing \(\Sigma = Cov(y) = Cov(\varepsilon)\),

\[ Cov(y, \hat{y}) = Cov(y, Hy) = \Sigma H' = \Sigma H \]

since H is symmetric. So the answer is \(\Sigma X(X'X)^{-1}X'\); with \(\Sigma = \sigma^{2}I\) this is \(\sigma^{2}H\).

\(\Sigma(I - H)\) is the covariance between y and the residuals, and \(\Sigma\) alone is the covariance of y with itself.

Answer: C

\(\sum X_{i}e_{i}\) is always zero.

The least squares normal equations are obtained by setting the derivatives of \(\sum(Y_{i} - \beta_{0} - \beta_{1}X_{i})^{2}\) to zero:

\[ \begin{aligned} \sum e_{i} &= 0 \\ \sum X_{i}e_{i} &= 0 \end{aligned} \]

The second equation is exactly the statement that the residuals are orthogonal to the regressor. It holds for every data set fitted by least squares, so the sum is never positive or negative and something definite can be said about it.

Answer: A

The odds ratio is exp(2β).

In the logistic model \(\log\frac{p}{1 - p} = \alpha + \beta X\), the odds are \(e^{\alpha + \beta X}\). Comparing X = 1 with X = −1:

\[ \frac{e^{\alpha + \beta}}{e^{\alpha - \beta}} = e^{2\beta} \]

The change in X is 2 units, not 1, so the familiar \(\exp(\beta)\), which is the odds ratio for a one-unit change, is not the answer here. \(\exp(1 \pm \beta)\) mixes the value of X into the exponent wrongly.

Answer: D

The estimable function is \(\theta_{2} + \theta_{3}\).

A linear function is estimable exactly when it is a linear combination of the expected observations. Here those are \(\theta_{1} + \theta_{2}\) and \(\theta_{1} - \theta_{3}\), so estimable functions have the form

\[ \begin{aligned} &a(\theta_{1} + \theta_{2}) + b(\theta_{1} - \theta_{3}) \\ &= (a + b)\theta_{1} + a\theta_{2} - b\theta_{3} \end{aligned} \]

For \(\theta_{2} + \theta_{3}\): \(a = 1\), \(b = -1\), \(a + b = 0\). It works, and \(E(Y_{1} - Y_{2}) = \theta_{2} + \theta_{3}\).

For \(\theta_{1} + \theta_{2} + \theta_{3}\): \(a = 1\), \(b = -1\) gives coefficient 0 for \(\theta_{1}\), not 1. For \(\theta_{1} + \theta_{3}\): \(a = 0\), \(b = -1\) gives \(-1\) for \(\theta_{1}\). The individual \(\theta\)'s are not estimable, since the design matrix has rank 2.

Answer: B

ANOVA tests equality of means when variances are all equal.

The standard assumptions are independent, normally distributed errors with a common variance. Under them, the F test compares between-group and within-group mean squares to test \(H_{0}: \mu_{1} = \mu_{2} = \ldots = \mu_{k}\). Equal variances are an assumption of the test, not something it tests.

'Equality of means' alone omits the homogeneity condition the F distribution relies on. Testing equality of variances is the job of tests such as Bartlett's or Levene's, and ANOVA does not test means and variances together.

Answer: D

The prediction will have larger variation as the distance between X and \(\overline{X}\) increases.

In simple linear regression the variance of the predicted mean response at \(x_{0}\) is

\[ \begin{aligned} &\mathrm{Var}(\hat{y}_{0}) \\ &= \sigma^{2}\left(\frac{1}{n} + \frac{(x_{0} - \overline{x})^{2}}{S_{xx}}\right) \end{aligned} \]

(and a new observation adds a further \(\sigma^{2}\)). The variance is smallest at \(x_{0} = \overline{x}\) and grows with \((x_{0} - \overline{x})^{2}\), which is why prediction bands flare out away from the mean.

The prediction does depend on X, its variation is not the same for all X, and it grows with distance from \(\overline{X}\) in either direction, not simply as X increases.

Answer: D

The true statement is that MError ss is unbiased for \(\sigma^{2}\), while MReg ss and MTotal ss are biased.

The residual mean square \(\frac{SSE}{n - p}\) has expectation exactly \(\sigma^{2}\) under the model, whatever the coefficients.

The regression mean square has expectation \(\sigma^{2}\) plus a non-negative term that depends on the slope coefficients, and that term is zero only when the slopes are all zero. The total mean square mixes the two and is biased for the same reason.

So options A, B and C, which call MReg ss unbiased or MError ss biased, are wrong.

Answer: C

The adjusted coefficient of determination accounts for the number of regressors in the model.

\[ \bar{R}^{2} = 1 - (1 - R^{2})\frac{n - 1}{n - k - 1} \]

where k is the number of regressors. Ordinary \(R^{2}\) never falls when a variable is added, even a useless one; the adjustment penalises extra regressors, so \(\bar{R}^{2}\) rises only when a new variable improves the fit by more than chance would.

It does nothing special about outliers or unusually large fitted values ('valves' in the paper), which are handled by residual and influence diagnostics.

Answer: A

The deviance test has 35 degrees of freedom.

The model has five parameters, \(B_{0}, B_{1}, B_{2}, B_{3}, B_{4}\). The deviance compares the fitted model with the saturated model, which has one parameter per observation, so its degrees of freedom are

\[ n - p = 40 - 5 = 35 \]

34 subtracts one parameter too many, 5 is the number of parameters, and 4 is the number of predictors, which is the degrees of freedom for testing all four slopes against the intercept-only model, not for goodness of fit.