Diagnostics and Influence Measures
Residuals
Residual analysis and diagnostic methods are well developed for linear regression models (c.f., Cook and Weisberg, 1982), and they are somewhat less developed for mixed effects models.
This set of notes is based on Nieuwenhuis et al.
Example: Orthodontics Data
We’ll consider the dental data with model
\[Y_{ij}=\beta_0+\beta_1I(\text{male})_i+\beta_2t_j+\beta_3I(\text{male})_it_j + b_{0i} + b_{1i}t_j + \varepsilon_{ij}\]
where
\[\begin{pmatrix} b_{0i} \\ b_{1i} \end{pmatrix} \overset{iid}\sim N\left(0,\begin{pmatrix}d_{11} & d_{12} \\ d_{12} & d_{22}\end{pmatrix}\right) \perp \varepsilon_{ij} \overset{iid}\sim N(0,\sigma^2)\]
for illustration.
Residuals
Residuals \(y_{ij}-\widehat{y}_{ij}\) can be helpful in flagging outliers and in assessing the adequacy of a fitted model. The definition of residuals is not always consistent in the case of mixed effects or hierarchical models:
- Many texts define residuals for subject/group \(i\) as \(Y_i-X_i\widehat{\beta}\)
- Many software implementations define residuals as \(Y_i-X_i\widehat{\beta}-Z_i\widehat{b}_i\) (nice because these can then be analyzed using standard methods)
We’ll run through the code to get some standard diagnostics (qq plot, RxP plot) as well as some visualizations that allow us to examine residuals by group/individual.
library(lattice)
library(lme4)
data(Orthodont,package="nlme")
Orthodont$Sex=relevel(Orthodont$Sex,ref="Female")
m1=lmer(distance~Sex+age+age*Sex+(1+age|Subject),data=Orthodont)
#basic qqplot
qqmath(resid(m1))
#standardized residuals y-Xbeta-Zb versus fitted values by gender
#standardized by the estimate of sigma=sqrt(var(epsilon))
plot(m1,resid(.,scaled=TRUE)~fitted(.)|Sex,abline=0)
## boxplots of residuals by Subject
plot(m1, Subject ~ resid(., scaled=TRUE))
## observed versus fitted values by Subject
## fitted value is X_ibeta+Z_ib_i
plot(m1, distance ~ fitted(.) | Subject, abline = c(0,1))
## residuals by age, separated by Subject
plot(m1, resid(., scaled=TRUE) ~ age | Sex, abline = 0)
library(ggplot2)
m1F <- fortify.merMod(m1)
# plot of raw residuals, use .scresid for scaled
ggplot(m1F, aes(age,.resid)) + geom_point(colour="blue") + facet_grid(.~Sex) +
geom_hline(yintercept=0)+geom_line(aes(group=Subject),alpha=0.4)+geom_smooth(method="loess")
## (warnings about loess are due to having only 4 unique x values)Residual analysis is not always a great tool for detecting influential cases:
- Cases with high residuals or high standardized residuals are called outliers
- Outliers may or may not be influential in the model fit
- An influential case may dominate the regression model so that the line is drawn more closely towards the case (making it an inlier)
Influence
We hope that all data points have some amount of influence on our parameter estimates. However, we may be concerned if a single case has disproportionate influence on model results. If so, one observation or group of observations may pull the estimated regression line towards the group. In such a case, excluding a single group might have a substantial effect on estimates. This idea is behind the development of many popular influence diagnostics, often termed deletion diagnostics.
Leverage
The degree to which an observation has the potential to be influential is closely related to the leverage of the case, which is a measure of how extreme the case is in the \(X\) space.
Leverage is not simply defined as an outlying value in \(X\) space of a single variable but also in a multivariate sense. For example, in a study of pregnancy outcomes, it may be relatively common to have mothers who are 40, or fathers who are 20, but babies who have a 40 year old mother and a 20 year old father may be fairly uncommon.
It is not necessarily the case that outliers or cases with high leverage are influential. So, how do we detect influential cases?
One popular approach is to use the principle that when a single case is removed from the data entirely, we would like for models based on the data not to give vastly different conclusions.
If parameter estimates change a lot after a single individual is excluded, then the individual may be considered influential.
Multi-Level Influence
Mixed effects and multilevel models estimate effects of lower-level and higher-level variables. It is thus possible that in some cases a higher-level group is influential (more likely when you don’t have very many groups), while in others, a single observation within a group is influential. We will examine influence at both levels.
DFBETAS
DFBETAS measures the level of influence observations have on single parameter estimates. It is calculated as the difference in magnitude of the parameter estimate between the model including and the model excluding the group (or individual in a longitudinal study), standardized by dividing by the standard error of the estimate that excludes the group (to prevent variance inflation from masking the level of influence).
For group \(i\) and parameter \(k\),
\[\text{DFBETAS}_{ik}=\frac{\widehat{\gamma}_k-\widehat{\gamma}_{k(-i)}}{se(\widehat{\gamma}_{k(-i)})},\] where \(\widehat{\gamma}_k\) is the original estimate of the \(k\)th parameter, and \(\widehat{\gamma}_{k(-i)}\) is the estimate of the same parameter after group \(i\) has been excluded from the data.
Belsley (1980) recommends a cutoff of \(\frac{2}{\sqrt{n}}\) for identifying overly influential observations. Here \(n\) is defined as the number of groups at the level of removal \((-i)\) for the calculation. (For the dental data we have 27 kids and 4 observations per kid, so at the group level \(k=27\).)
library(influence.ME)
m1.inf=influence(m1,"Subject")
print(2/sqrt(length(unique(Orthodont$Subject))))
dfbetas(m1.inf)## [1] 0.3849002## (Intercept) SexFemale age SexFemale:age
## M16 3.792776e-02 -0.024669872 -1.743364e-01 0.113396005
## M05 -1.708339e-01 0.111117801 4.879626e-02 -0.031739214
## M02 -9.363421e-02 0.060903775 -6.935399e-03 0.004511086
## M11 2.345256e-01 -0.152545707 -3.514703e-01 0.228611641
## M07 -8.656237e-02 0.056303940 1.149669e-02 -0.007477952
## M08 2.173651e-01 -0.141383719 -3.133461e-01 0.203813988
## M03 -2.132601e-02 0.013871365 -2.542980e-02 0.016540653
## M12 -1.943286e-01 0.126399771 1.599342e-01 -0.104028181
## M13 -1.051725e+00 0.684087545 1.076023e+00 -0.699892393
## M14 1.736415e-01 -0.112943981 -1.936229e-01 0.125940795
## M09 -1.210886e-01 0.078761283 1.411130e-01 -0.091786074
## M15 -1.782330e-01 0.115930551 2.558523e-01 -0.166417501
## M06 1.630974e-01 -0.106085662 -8.046207e-02 0.052336042
## M04 5.799696e-01 -0.377237510 -4.866559e-01 0.316542214
## M01 5.964366e-02 -0.038794837 1.224129e-01 -0.079622675
## M10 3.147116e-01 -0.204702168 -2.528259e-02 0.016444898
## F10 1.676718e-12 -0.232617008 -1.424531e-12 -0.020879843
## F09 7.392377e-13 0.043449099 -7.500012e-13 -0.145739593
## F06 4.648603e-13 -0.022239646 -3.954377e-13 -0.074017034
## F01 5.813338e-13 -0.007326066 -4.996862e-13 -0.074064376
## F05 8.035070e-13 0.133505867 -8.771197e-13 -0.145456313
## F07 7.437522e-13 -0.025291926 -7.320104e-13 0.049929080
## F02 1.593488e-13 -0.191856576 -2.347268e-13 0.230866446
## F08 1.209541e-12 0.248809746 -1.071054e-12 -0.218999663
## F03 1.797969e-12 -0.179426730 -1.776067e-12 0.268665114
## F04 4.060678e-13 0.136799558 -2.972321e-13 -0.003211503
## F11 9.053009e-13 0.094416730 -8.116006e-13 0.139107896Here we see that M04 and M13 are influential on some of our estimates. What did these kids look like?
plot(m1.inf,which="dfbetas",xlab="DFBETAS",ylab="Student")
Orthodont$distance[Orthodont$Subject=="M04"]## [1] 25.5 27.5 26.5 27.0Orthodont$distance[Orthodont$Subject=="M13"]## [1] 17.0 24.5 26.0 29.5plot(m1, distance ~ fitted(.) | Subject, abline = c(0,1))
## [1] 25.5 27.5 26.5 27.0## [1] 17.0 24.5 26.0 29.5| ID | Int | Fem | Age | AbyF |
|---|---|---|---|---|
| M04 | 0.58 | -0.38 | -0.49 | 0.32 |
| M13 | -1.05 | 0.68 | 1.08 | -0.70 |
M04 had large measurements without a lot of growth over time – pulling him out of the model reduced the intercept for boys and also decreased their slope.
M13 had a small measure at age 8 and then grew substantially. Leaving him out of the model changed the estimates significantly, greatly increasing the intercept for boys and also reducing the slope among boys.
When the number of observations or predictors is large, it may take a while to wade through all the DFBETAS. Cook’s distance is a summary measure for influence on all parameter estimates. It is defined as
\[C_i=\frac{1}{p}(\widehat{\gamma}-\widehat{\gamma}_{(-i)})'\widehat{\Sigma}_{(-i)}^{-1}(\widehat{\gamma}-\widehat{\gamma}_{(-i)})\]
where \(p\) is the length of \(\beta\), and \(\widehat{\Sigma}_{(-i)}\) is the covariance matrix of the parameter estimates excluding group \(i\). Van der Meer et al (2010) recommends a cutoff of \(\frac{4}{n}\) where again \(n\) is the number of groups in the grouping factor being evaluated.
If there is just one parameter in the model, then Cook’s distance is the DFBETAS squared for that parameter.
print(4/length(unique(Orthodont$Subject)))
cooks.distance(m1.inf,sort=TRUE)
plot(m1.inf,which="cook",cutoff=4/length(unique(Orthodont$Subject)),
sort=TRUE,xlab="Cook's D",ylab="Subject")## [1] 0.1481481## [,1]
## F07 0.001636431
## M03 0.002268299
## M09 0.004987715
## M07 0.006593856
## F05 0.008652563
## M14 0.009388496
## M12 0.009565544
## M06 0.011152793
## M02 0.011152818
## F01 0.011886446
## F06 0.016424229
## M15 0.018727159
## M05 0.018869728
## F02 0.021847932
## F09 0.022005758
## M16 0.023158496
## F08 0.025151739
## M08 0.027996778
## F04 0.033898436
## F03 0.035015308
## M11 0.036802012
## M01 0.038079939
## M04 0.084192285
## F11 0.110053787
## M10 0.116280355
## F10 0.137275861
## M13 0.312775049
It’s M13 again.
Other metrics for evaluating influence include percentile change and changes in significance.
Percentile change is defined as \[\frac{\widehat{\gamma}-\widehat{\gamma}_{(-i)}}{\widehat{\gamma}}\times 100%\]
plot(m1.inf,which="pchange",xlab="%ile Change",ylab="Student")
No surprise here!
Another metric is evaluating whether excluding a group changes the statistical significance of any of the estimates in the model. The user sets the critical value, and estimates that did not exceed it but do so when the group is removed, or vice versa, are flagged.
#coding is a bit awkward here
sigtest(m1.inf,test=-2)## $Intercept
## Altered.Teststat Altered.Sig Changed.Sig
## M16 15.21779 FALSE FALSE
## M05 15.73338 FALSE FALSE
## M02 15.49028 FALSE FALSE
## M11 15.26254 FALSE FALSE
## M07 15.34391 FALSE FALSE
## M08 15.40961 FALSE FALSE
## M03 15.36725 FALSE FALSE
## M12 15.60602 FALSE FALSE
## M13 20.08981 FALSE FALSE
## M14 15.25056 FALSE FALSE
## M09 15.41510 FALSE FALSE
## M15 15.55745 FALSE FALSE
## M06 15.15731 FALSE FALSE
## M04 16.42561 FALSE FALSE
## M01 15.17863 FALSE FALSE
## M10 15.39783 FALSE FALSE
## F10 16.03332 FALSE FALSE
## F09 15.74124 FALSE FALSE
## F06 15.72143 FALSE FALSE
## F01 15.72842 FALSE FALSE
## F05 15.79364 FALSE FALSE
## F07 15.76438 FALSE FALSE
## F02 15.93306 FALSE FALSE
## F08 16.07878 FALSE FALSE
## F03 15.90331 FALSE FALSE
## F04 15.82796 FALSE FALSE
## F11 15.77239 FALSE FALSE
##
## $SexFemale
## Altered.Teststat Altered.Sig Changed.Sig
## M16 0.6514226 FALSE FALSE
## M05 0.5282407 FALSE FALSE
## M02 0.5716389 FALSE FALSE
## M11 0.7892140 FALSE FALSE
## M07 0.5705159 FALSE FALSE
## M08 0.7833890 FALSE FALSE
## M03 0.6165874 FALSE FALSE
## M12 0.5067610 FALSE FALSE
## M13 0.0980569 FALSE FALSE
## M14 0.7466187 FALSE FALSE
## M09 0.5495649 FALSE FALSE
## M15 0.5158962 FALSE FALSE
## M06 0.7354965 FALSE FALSE
## M04 1.0758801 FALSE FALSE
## M01 0.6648312 FALSE FALSE
## M10 0.8502230 FALSE FALSE
## F10 0.8606622 FALSE FALSE
## F09 0.5731547 FALSE FALSE
## F06 0.6380677 FALSE FALSE
## F01 0.6234279 FALSE FALSE
## F05 0.4851508 FALSE FALSE
## F07 0.6428024 FALSE FALSE
## F02 0.8159745 FALSE FALSE
## F08 0.3810159 FALSE FALSE
## F03 0.8023791 FALSE FALSE
## F04 0.4832014 FALSE FALSE
## F11 0.5234074 FALSE FALSE
##
## $age
## Altered.Teststat Altered.Sig Changed.Sig
## M16 8.926024 FALSE FALSE
## M05 8.699675 FALSE FALSE
## M02 8.710861 FALSE FALSE
## M11 9.353415 FALSE FALSE
## M07 8.645514 FALSE FALSE
## M08 9.319057 FALSE FALSE
## M03 8.729358 FALSE FALSE
## M12 8.566910 FALSE FALSE
## M13 9.785175 FALSE FALSE
## M14 8.976638 FALSE FALSE
## M09 8.568568 FALSE FALSE
## M15 8.581615 FALSE FALSE
## M06 8.735882 FALSE FALSE
## M04 9.882858 FALSE FALSE
## M01 8.573521 FALSE FALSE
## M10 8.678823 FALSE FALSE
## F10 8.938136 FALSE FALSE
## F09 9.011522 FALSE FALSE
## F06 8.954409 FALSE FALSE
## F01 8.960137 FALSE FALSE
## F05 8.994006 FALSE FALSE
## F07 8.963032 FALSE FALSE
## F02 9.111813 FALSE FALSE
## F08 9.094980 FALSE FALSE
## F03 9.172480 FALSE FALSE
## F04 8.935963 FALSE FALSE
## F11 9.001532 FALSE FALSE
##
## $`SexFemale:age`
## Altered.Teststat Altered.Sig Changed.Sig
## M16 -2.325649 TRUE FALSE
## M05 -2.179701 TRUE FALSE
## M02 -2.204691 TRUE FALSE
## M11 -2.504125 TRUE FALSE
## M07 -2.180843 TRUE FALSE
## M08 -2.480279 TRUE FALSE
## M03 -2.216721 TRUE FALSE
## M12 -2.101945 TRUE FALSE
## M13 -2.045603 TRUE FALSE
## M14 -2.346113 TRUE FALSE
## M09 -2.109849 TRUE FALSE
## M15 -2.067519 TRUE FALSE
## M06 -2.240254 TRUE FALSE
## M04 -2.691716 TRUE FALSE
## M01 -2.118537 TRUE FALSE
## M10 -2.203888 TRUE FALSE
## F10 -2.133358 TRUE FALSE
## F09 -2.026185 TRUE FALSE
## F06 -2.084143 TRUE FALSE
## F01 -2.085476 TRUE FALSE
## F05 -2.022247 TRUE FALSE
## F07 -2.210167 TRUE FALSE
## F02 -2.426963 TRUE FALSE
## F08 -1.973040 FALSE TRUE
## F03 -2.479383 TRUE FALSE
## F04 -2.150502 TRUE FALSE
## F11 -2.308625 TRUE FALSEsigtest(m1.inf,test=2)## $Intercept
## Altered.Teststat Altered.Sig Changed.Sig
## M16 15.21779 TRUE FALSE
## M05 15.73338 TRUE FALSE
## M02 15.49028 TRUE FALSE
## M11 15.26254 TRUE FALSE
## M07 15.34391 TRUE FALSE
## M08 15.40961 TRUE FALSE
## M03 15.36725 TRUE FALSE
## M12 15.60602 TRUE FALSE
## M13 20.08981 TRUE FALSE
## M14 15.25056 TRUE FALSE
## M09 15.41510 TRUE FALSE
## M15 15.55745 TRUE FALSE
## M06 15.15731 TRUE FALSE
## M04 16.42561 TRUE FALSE
## M01 15.17863 TRUE FALSE
## M10 15.39783 TRUE FALSE
## F10 16.03332 TRUE FALSE
## F09 15.74124 TRUE FALSE
## F06 15.72143 TRUE FALSE
## F01 15.72842 TRUE FALSE
## F05 15.79364 TRUE FALSE
## F07 15.76438 TRUE FALSE
## F02 15.93306 TRUE FALSE
## F08 16.07878 TRUE FALSE
## F03 15.90331 TRUE FALSE
## F04 15.82796 TRUE FALSE
## F11 15.77239 TRUE FALSE
##
## $SexFemale
## Altered.Teststat Altered.Sig Changed.Sig
## M16 0.6514226 FALSE FALSE
## M05 0.5282407 FALSE FALSE
## M02 0.5716389 FALSE FALSE
## M11 0.7892140 FALSE FALSE
## M07 0.5705159 FALSE FALSE
## M08 0.7833890 FALSE FALSE
## M03 0.6165874 FALSE FALSE
## M12 0.5067610 FALSE FALSE
## M13 0.0980569 FALSE FALSE
## M14 0.7466187 FALSE FALSE
## M09 0.5495649 FALSE FALSE
## M15 0.5158962 FALSE FALSE
## M06 0.7354965 FALSE FALSE
## M04 1.0758801 FALSE FALSE
## M01 0.6648312 FALSE FALSE
## M10 0.8502230 FALSE FALSE
## F10 0.8606622 FALSE FALSE
## F09 0.5731547 FALSE FALSE
## F06 0.6380677 FALSE FALSE
## F01 0.6234279 FALSE FALSE
## F05 0.4851508 FALSE FALSE
## F07 0.6428024 FALSE FALSE
## F02 0.8159745 FALSE FALSE
## F08 0.3810159 FALSE FALSE
## F03 0.8023791 FALSE FALSE
## F04 0.4832014 FALSE FALSE
## F11 0.5234074 FALSE FALSE
##
## $age
## Altered.Teststat Altered.Sig Changed.Sig
## M16 8.926024 TRUE FALSE
## M05 8.699675 TRUE FALSE
## M02 8.710861 TRUE FALSE
## M11 9.353415 TRUE FALSE
## M07 8.645514 TRUE FALSE
## M08 9.319057 TRUE FALSE
## M03 8.729358 TRUE FALSE
## M12 8.566910 TRUE FALSE
## M13 9.785175 TRUE FALSE
## M14 8.976638 TRUE FALSE
## M09 8.568568 TRUE FALSE
## M15 8.581615 TRUE FALSE
## M06 8.735882 TRUE FALSE
## M04 9.882858 TRUE FALSE
## M01 8.573521 TRUE FALSE
## M10 8.678823 TRUE FALSE
## F10 8.938136 TRUE FALSE
## F09 9.011522 TRUE FALSE
## F06 8.954409 TRUE FALSE
## F01 8.960137 TRUE FALSE
## F05 8.994006 TRUE FALSE
## F07 8.963032 TRUE FALSE
## F02 9.111813 TRUE FALSE
## F08 9.094980 TRUE FALSE
## F03 9.172480 TRUE FALSE
## F04 8.935963 TRUE FALSE
## F11 9.001532 TRUE FALSE
##
## $`SexFemale:age`
## Altered.Teststat Altered.Sig Changed.Sig
## M16 -2.325649 FALSE FALSE
## M05 -2.179701 FALSE FALSE
## M02 -2.204691 FALSE FALSE
## M11 -2.504125 FALSE FALSE
## M07 -2.180843 FALSE FALSE
## M08 -2.480279 FALSE FALSE
## M03 -2.216721 FALSE FALSE
## M12 -2.101945 FALSE FALSE
## M13 -2.045603 FALSE FALSE
## M14 -2.346113 FALSE FALSE
## M09 -2.109849 FALSE FALSE
## M15 -2.067519 FALSE FALSE
## M06 -2.240254 FALSE FALSE
## M04 -2.691716 FALSE FALSE
## M01 -2.118537 FALSE FALSE
## M10 -2.203888 FALSE FALSE
## F10 -2.133358 FALSE FALSE
## F09 -2.026185 FALSE FALSE
## F06 -2.084143 FALSE FALSE
## F01 -2.085476 FALSE FALSE
## F05 -2.022247 FALSE FALSE
## F07 -2.210167 FALSE FALSE
## F02 -2.426963 FALSE FALSE
## F08 -1.973040 FALSE FALSE
## F03 -2.479383 FALSE FALSE
## F04 -2.150502 FALSE FALSE
## F11 -2.308625 FALSE FALSEInfluence of Lower-Level Observations
We can also look at the influence of single lower-level observations; they could be impactful in longitudinal data for example, when we have relatively few observations per individual. Note however that the computational complexity of these deletion diagnostics will be increased in this case.
Here we look at Cook’s Distance for the dental data on the individual observation level.
m1.inf.indiv=influence(m1,obs=TRUE)
m1.cook=cooks.distance(m1.inf.indiv)
which(m1.cook>4/length(Orthodont$distance))## Orthodont.Subject m1.cook infindiv
## 1 M01 1.168756e-02 FALSE
## 2 M01 2.775825e-03 FALSE
## 3 M01 4.697088e-04 FALSE
## 4 M01 7.815615e-03 FALSE
## 5 M02 4.183322e-04 FALSE
## 6 M02 9.999044e-05 FALSE
## 7 M02 1.636938e-03 FALSE
## 8 M02 3.594928e-03 FALSE
## 9 M03 7.487110e-03 FALSE
## 10 M03 1.259384e-03 FALSE
## 11 M03 1.122019e-03 FALSE
## 12 M03 6.194464e-03 FALSE
## 13 M04 1.190995e-02 FALSE
## 14 M04 3.784757e-03 FALSE
## 15 M04 2.816516e-04 FALSE
## 16 M04 1.720188e-02 FALSE
## 17 M05 6.518207e-03 FALSE
## 18 M05 1.211973e-03 FALSE
## 19 M05 2.141630e-03 FALSE
## 20 M05 1.593446e-03 FALSE
## 21 M06 2.957091e-03 FALSE
## 22 M06 6.031195e-06 FALSE
## 23 M06 4.012534e-07 FALSE
## 24 M06 6.001638e-06 FALSE
## 25 M07 1.482953e-03 FALSE
## 26 M07 1.400801e-03 FALSE
## 27 M07 2.537589e-05 FALSE
## 28 M07 6.796451e-04 FALSE
## 29 M08 3.346305e-02 FALSE
## 30 M08 3.653400e-03 FALSE
## 31 M08 1.848123e-05 FALSE
## 32 M08 1.793122e-03 FALSE
## 33 M09 1.297359e-03 FALSE
## 34 M09 1.601707e-02 FALSE
## 35 M09 2.421675e-02 FALSE
## 36 M09 2.564444e-02 FALSE
## 37 M10 1.069486e-02 FALSE
## 38 M10 2.168850e-05 FALSE
## 39 M10 1.417136e-03 FALSE
## 40 M10 1.618865e-06 FALSE
## 41 M11 1.178093e-02 FALSE
## 42 M11 1.081951e-05 FALSE
## 43 M11 7.835298e-04 FALSE
## 44 M11 4.157390e-03 FALSE
## 45 M12 9.481957e-04 FALSE
## 46 M12 5.036146e-07 FALSE
## 47 M12 1.293598e-03 FALSE
## 48 M12 1.054203e-02 FALSE
## 49 M13 2.258736e-01 TRUE
## 50 M13 1.799655e-03 FALSE
## 51 M13 2.852441e-04 FALSE
## 52 M13 5.468711e-02 TRUE
## 53 M14 4.643261e-04 FALSE
## 54 M14 2.006172e-03 FALSE
## 55 M14 6.714294e-06 FALSE
## 56 M14 7.969978e-03 FALSE
## 57 M15 1.314747e-04 FALSE
## 58 M15 1.862196e-04 FALSE
## 59 M15 4.313167e-04 FALSE
## 60 M15 1.916957e-02 FALSE
## 61 M16 6.288542e-03 FALSE
## 62 M16 1.146817e-03 FALSE
## 63 M16 1.611587e-04 FALSE
## 64 M16 7.199977e-04 FALSE
## 65 F01 7.368422e-03 FALSE
## 66 F01 1.881843e-03 FALSE
## 67 F01 2.793563e-04 FALSE
## 68 F01 4.043053e-04 FALSE
## 69 F02 8.531024e-04 FALSE
## 70 F02 1.330744e-03 FALSE
## 71 F02 2.818653e-04 FALSE
## 72 F02 7.739192e-03 FALSE
## 73 F03 2.241371e-02 FALSE
## 74 F03 1.442750e-03 FALSE
## 75 F03 9.758752e-05 FALSE
## 76 F03 4.090490e-03 FALSE
## 77 F04 1.943921e-03 FALSE
## 78 F04 2.522817e-04 FALSE
## 79 F04 2.105920e-05 FALSE
## 80 F04 1.540636e-03 FALSE
## 81 F05 2.796491e-04 FALSE
## 82 F05 1.065344e-03 FALSE
## 83 F05 3.991445e-04 FALSE
## 84 F05 1.723717e-03 FALSE
## 85 F06 2.644555e-08 FALSE
## 86 F06 3.035559e-05 FALSE
## 87 F06 7.256505e-04 FALSE
## 88 F06 8.424935e-05 FALSE
## 89 F07 2.031630e-05 FALSE
## 90 F07 2.393514e-06 FALSE
## 91 F07 2.942438e-04 FALSE
## 92 F07 3.162853e-03 FALSE
## 93 F08 1.036493e-02 FALSE
## 94 F08 3.423363e-05 FALSE
## 95 F08 4.484424e-05 FALSE
## 96 F08 2.786362e-03 FALSE
## 97 F09 5.238697e-05 FALSE
## 98 F09 2.279378e-05 FALSE
## 99 F09 1.276697e-04 FALSE
## 100 F09 1.178778e-02 FALSE
## 101 F10 2.092103e-02 FALSE
## 102 F10 3.240217e-04 FALSE
## 103 F10 2.241019e-04 FALSE
## 104 F10 7.376668e-03 FALSE
## 105 F11 1.096866e-03 FALSE
## 106 F11 2.342424e-04 FALSE
## 107 F11 3.109994e-03 FALSE
## 108 F11 1.202342e-03 FALSEM13 once again!
Dealing with Influential Data
What to do with influential data is a much harder problem. Reasonable strategies may include the following.
- Verify data recorded correctly
- Consider robust models
- Determine whether any lurking predictors should be added to the model
- Report results with and without overly influential results