One popular criticism of intelligence testing is that scores could be affected by motivation and levels of practice. By implication, those who are not motivated to take the test will do badly and will be unfairly judged, to the detriment of any society which uses intelligence test results as a ticket of admission to education or employment. By further implication, such lack of motivation may apply most strongly to those who are poorer and most dispirited for other reasons.
Test administrators know all that, and make sure that subjects understand the test, take and pass the practice items, and are encouraged before, during and after each test (guided by protocols as to what help and encouragement is permissible) and that at least 6 months elapses between face to face testing sessions, and that alternate forms are used if testing has to be conducted sooner. Hence psychometric reports talk about the person’s level of engagement, the amount of effort they show, and the specific problems they may have encountered. If there are significant problems the results are either set aside, or labelled as being under-estimates and further testing carried out later usually resolves the issue. Monitoring is easier in face to face testing, but item analysis gives some insight into lack of effort in group tests. Group tests often have more practice items and care is taken to provide good quality test settings. By following all these procedures practice effects and motivational differences are reduced, but not eliminated entirely. It is still possible that some low results may be due to low motivation, and also that some high results might be due to lucky guessing. How big could these effects be?
Assume for a moment that motivational and practice effects have an influence, and that to the true low scores of less able people must be added the false low scores of those who found the test boring, pointless, and not worth bothering about. People like me, for example. I prefer watching clothes dry on a cloudy day than taking most intelligence tests.
If that were true, IQs would under-predict real life successes in things which were intrinsically interesting: getting good qualifications so as to get on in life, making money, and becoming famous.
If motivation were a major confounder, then correlations between IQ scores and real life scores would be low. However, IQ and real life are strongly correlated. For example, the largest recent study (Deary et al., 2007) of over 70,000 English children found correlations of r=0.81 between general intelligence measured at 11 years of age and GCSE scores at age 16. This is an extremely high predictive power (accounting for 64% of the variance). The colossal sample size gives us exceptional confidence in the robustness of the results. By way of comparison, most educational psychology publications have sample sizes of a few hundred, and are far less robust. As further proof of the common sense view that intelligence is involved in academic achievement, we can be even more precise about the impact of intelligence on different subjects. IQ scores on their own accounted for 58.6% of the results in Mathematics, 48% in English and down to 18.1% in Art and Design, that subject being the least intellectually demanding (Deary et al., 2007).
I. J. Deary, S. Strand, P. Smith and C. Fernandes (2007) Intelligence and educational achievement. Intelligence 35, 1, pp13-21. (For private study, email the author at the University of Edinburgh and ask for a copy).
Problems of motivation and practice also apply to scholastic examinations and to any procedures followed in job interviews. Varying motivation applies not just to IQ test but to all measures: intelligence tests, scholastic tests, and work assessments. Nobody gets round measurement error, not even the Spanish Inquisition.
In summary: Assume some people’s IQ scores are reduced by lack of motivation. That will reduce the correlation between IQ and other real life measures. IQ at 11 correlates 0.81 with scholastic attainment at 16. If motivation is a problem, the correlation is really higher.
If you prefer that as a Tweet:
If IQ scores are reduced by lack of motivation, but IQ at 11 correlates 0.81 with GCSEs at 16, then the real correlation is much higher.