Jobs: IQ tests versus interviews

 

In a place of constrained avian discourse I got into a debate about intelligence testing.  One suggestion made to me was:  Psychologists' naive interpretations of "intelligence" have been doing social harm for 80 years. Please stop it.

I replied: Ignoring intelligence measures is not without harm.

Another  questioner asked: I'd love an example of intelligence measures helping society in an equitable way.

I replied: Probably best example job selection, more equitable than interviews.

Anyway, that is enough background to understand the next step, which is about disclosing the material upon which one relies for a particular opinion. There are often testy exchanges about this. Some people get offended if told, in the middle of an argument, You should read the following papers. In my view academic debates should be punctuated by long moments of silence, broken only by pages turning as the combatants do the necessary reading. Show me your references is often a valid challenge. Not always, because there are some general points and personal observations which can be given without chapter and verse, but the ideal is that we should back up our opinions with relevant publications. The other reason for checking references is that memory is fallible, and although people may remember the general shape of the results, relevant details will have been forgotten.

I made my remark about job selection based on something I had read a decade ago in Ian Deary’s Intelligence: A very short introduction Oxford University Press, 2001. Here is the reference:

Schmidt, FL & JE Hunter (1998) The validity and utility of selection methods in personnel psychology: practical and theoretical implications of 85 years of research findings. Psychological Bulletin, 124, 262-74.

 

These authors had asked the simple question: is it worthwhile for an employer to select people for a job on the basis of, among other things, a test of general mental ability? The authors then looked at the correlation between the selection measures and eventual productivity, showing that intelligence test results were far better than interviews. Specifically, using test results will result in higher productivity, using an interview alone would lose three quarters of that gain. So, my main point about IQ tests being more fair than interviews can be validated by a good meta-analytic study. It was this paper which I had in mind when I made that remark. Incidentally, intelligence tests were originally seen as liberators, allowing bright working class children to rise through the system, despite not having been coached by private schools. Interviews were recognised to be rituals in which school ties and other social markers could be displayed so as to get unfair advantage.

There are other good predictors, though many take longer, and are thus more costly. For example, work sample tests are the best predictor, but since they involve a detailed evaluation of how each applicant does the sample job, they are time consuming, and difficult to arrange for some jobs. Highly structured job interviews do well (I had forgotten this) but unfortunately unstructured interviews are far more common, and far less valuable, so interviews are usually poor overall. Reference checks are not much help (perhaps because references are all much the same). Years of education is a very poor predictor, which is interesting considering it is often used as a surrogate measure of social advantage in educational research.

Compared with all these, intelligence testing can be used for all jobs, is a good predictor, and is relatively quick and cheap. The more complex the job, the better the prediction. Hunter and Schmidt found intelligence tests to be the best selection measure, and the 18 others only supplements. In terms of the multiple regression, after IQ the integrity test added 27% predictive power, a work sample or a structured interview another 24%. IQ plus integrity test is a good basic combination for an efficient result.

What did I not remember? That which I did not know when I first read the reference a decade ago. I did not know the high regard in which Schmidt and Hunter are held by methodologists. I recalled the highlights of their work as quoted in Deary’s book, rather than their names. The major consequence of their careful work was to establish a benchmark for excellence in meta-analysis.

In their 2004 paper General Mental Ability in the World of Work: Occupational Attainment and Job Performance Table 1 gives you the general ability scores of enlisted men by their civilian occupations. An instructive list which shows the intellectual demands of occupations, and also serves as a sad roll call of all the jobs that were part of an industrial society and are no more.

http://people.wku.edu/richard.miller/GMA.pdf

Their magnum opus is this fearsome tome:

Methods of Meta-Analysis: Correcting Error and Bias in Research Findings – 2004 by John E. Hunter (Author, Editor), Frank L. Schmidt (Author) Sage.

Make sure you give it as suggested reading to intelligence denigrationists.