Friday 5 December 2014

Jobs: IQ tests versus interviews

 

In a place of constrained avian discourse I got into a debate about intelligence testing.  One suggestion made to me was:  Psychologists' naive interpretations of "intelligence" have been doing social harm for 80 years. Please stop it.

I replied: Ignoring intelligence measures is not without harm.

Another  questioner asked: I'd love an example of intelligence measures helping society in an equitable way.

I replied: Probably best example job selection, more equitable than interviews.

Anyway, that is enough background to understand the next step, which is about disclosing the material upon which one relies for a particular opinion. There are often testy exchanges about this. Some people get offended if told, in the middle of an argument, You should read the following papers. In my view academic debates should be punctuated by long moments of silence, broken only by pages turning as the combatants do the necessary reading. Show me your references is often a valid challenge. Not always, because there are some general points and personal observations which can be given without chapter and verse, but the ideal is that we should back up our opinions with relevant publications. The other reason for checking references is that memory is fallible, and although people may remember the general shape of the results, relevant details will have been forgotten.

I made my remark about job selection based on something I had read a decade ago in Ian Deary’s Intelligence: A very short introduction Oxford University Press, 2001. Here is the reference:

Schmidt, FL & JE Hunter (1998) The validity and utility of selection methods in personnel psychology: practical and theoretical implications of 85 years of research findings. Psychological Bulletin, 124, 262-74.

IMG_2021

 

These authors had asked the simple question: is it worthwhile for an employer to select people for a job on the basis of, among other things, a test of general mental ability? The authors then looked at the correlation between the selection measures and eventual productivity, showing that intelligence test results were far better than interviews. Specifically, using test results will result in higher productivity, using an interview alone would lose three quarters of that gain. So, my main point about IQ tests being more fair than interviews can be validated by a good meta-analytic study. It was this paper which I had in mind when I made that remark. Incidentally, intelligence tests were originally seen as liberators, allowing bright working class children to rise through the system, despite not having been coached by private schools. Interviews were recognised to be rituals in which school ties and other social markers could be displayed so as to get unfair advantage.

There are other good predictors, though many take longer, and are thus more costly. For example, work sample tests are the best predictor, but since they involve a detailed evaluation of how each applicant does the sample job, they are time consuming, and difficult to arrange for some jobs. Highly structured job interviews do well (I had forgotten this) but unfortunately unstructured interviews are far more common, and far less valuable, so interviews are usually poor overall. Reference checks are not much help (perhaps because references are all much the same). Years of education is a very poor predictor, which is interesting considering it is often used as a surrogate measure of social advantage in educational research.

Compared with all these, intelligence testing can be used for all jobs, is a good predictor, and is relatively quick and cheap. The more complex the job, the better the prediction. Hunter and Schmidt found intelligence tests to be the best selection measure, and the 18 others only supplements. In terms of the multiple regression, after IQ the integrity test added 27% predictive power, a work sample or a structured interview another 24%. IQ plus integrity test is a good basic combination for an efficient result.

What did I not remember? That which I did not know when I first read the reference a decade ago. I did not know the high regard in which Schmidt and Hunter are held by methodologists. I recalled the highlights of their work as quoted in Deary’s book, rather than their names. The major consequence of their careful work was to establish a benchmark for excellence in meta-analysis.

In their 2004 paper General Mental Ability in the World of Work: Occupational Attainment and Job Performance Table 1 gives you the general ability scores of enlisted men by their civilian occupations. An instructive list which shows the intellectual demands of occupations, and also serves as a sad roll call of all the jobs that were part of an industrial society and are no more.

http://people.wku.edu/richard.miller/GMA.pdf

Their magnum opus is this fearsome tome:

Methods of Meta-Analysis: Correcting Error and Bias in Research Findings – 2004 by John E. Hunter (Author, Editor), Frank L. Schmidt (Author) Sage.

Make sure you give it as suggested reading to intelligence denigrationists.

13 comments:

  1. The book is on libgen: http://gen.lib.rus.ec/book/index.php?md5=D43B07F375C98403D672D186F91C3407

    There are 3 editions, #2 is on libgen, #3 just came out.

    ReplyDelete
  2. "Psychologists' naive interpretations of "intelligence" have been doing social harm for 80 years."

    Like what? Correctly predicting that Civil Rights wouldn't automatically make blacks equal to whites? Correctly pointing out that intelligence has a lot to do with academic success? Correctly predicting that China had a lot of potential once it got rid of an insane economic system? Hurting lefties feelings in other ways? What?

    ReplyDelete
  3. "Psychologists' naive interpretations of "intelligence" have been doing social harm for 80 years. Please stop it."

    would they rather we throw out intelligence tests & simply use achievement tests? (reading, math, etc.) intelligence tests are meant to be a bit more fair than achievement tests -- but, there's no pleasing some people:)

    ReplyDelete
  4. Those weren't questions, they were Frankfurt School attempts to silence you.

    ReplyDelete
  5. Glad to have the whole thing explained to me.

    ReplyDelete
  6. For technical maintainers is partially true that intelligence tests are good for selection. However, almost all cognitive test is one-sided, i.e., disregards some aspects of the intellect in favor of others. For example, some cognitive tests based on memorization, but despise the ability to solve problems.

    For example, creative people tend to have poor working memory.

    Human societies are like ants societies or any other natural hierarchical community. This means that has a natural division of cognitive classes (different than ego or social classe8-)

    Some people are so specialized in certain functions, a cognitive test will not be able to access their strengths. And with implications for your future financial life.

    Education is based on equal mental conditions and results in inequality, because as people are not equal or do not have the same potential, or the same existential transcendence, aka, soul motivations, lack of specialized niches. Lack jobs because lack of work niches. It is a simple mathematical account.

    As a way to take the tests, due not only to evaluate the performance of the large iq tests, such as verbal and performance, but also consider whether there is any great performance in any subtest in contrast to others.


    Santoculto

    ReplyDelete
    Replies
    1. Except for perhaps IQ tests not measuring creativity, they predict solely to the extent they measure g. Non g intelligences add only about .01 to .03 to prediction accuracy (this is a Gottfredson cite).

      I'd argue though that creativity requires some minimum level of iq, and then whatever creativity is, it's not g (i.e., g is necessary but not sufficient for creativity).

      James: You may be interested in an in-press paper by Schmidt and colleagues showing that unstructured interviews may indeed have high validity (partly because they are like an IQ test). Sorry if this link is ghastly:

      https://www.researchgate.net/profile/In-Sue_Oh/publication/255907743_RETHINKING_THE_VALIDITY_OF_INTERVIEWS_FOR_EMPLOYMENT_DECISION_MAKING_Implications_of_Recent_Developments_in_Meta-analysis/links/00b7d520d145796ee4000000?ev=pub_int_doc_dl&origin=publication_detail&inViewer=true

      Delete
    2. Creativity, like many psychologists believe, it seems a separate concept from any other influences. Of course, creativity is best when there is high levels of intelligence, however, is too complex to summarize this interaction as '' a higher IQ is substantially predictive significant to enhance creativity '', when what we see is just the otherwise.

      IQ tests usually also function in this manner, as the intelligence exist independent of cognitive or other psychological traits. It is even more complex because our societies are not perfectly meritocratic.

      There is a real need to add the psychological term '' will or intrinsic motivation '' as a parameter to any type of cognitive assessment. The intrinsic motivation is not a metaphysical concept, but is also likely to be measured within the psychometrics as well as within the genetics.

      We simply can not summarize an individual to their results on cognitive tests or assessments developed by human resources companies, without know all of it. In this case, I think a full psychological assessment (ie, up to one month of '' therapy sessions '') as a much more interesting way to succeed in recruitment processes.

      Creativity is not something separate from other cognitive traits, but a combination of traits that produces a differentiated phenotype. Therefore, it is clear that intelligence as a collaborator participate in the formation of creativity. However, it remains unclear what is intelligence and how the interactions occur between the genes for the production of certain phenotype. Some phenotypes may have the potential for creativity, but some interaction between multiple genes that are associated with this particular phenotype, can neutralize the manifestation of creative talent.


      Santoculto

      Delete
    3. Thanks for the link. Very interesting, and complicated analysis. I like the conclusion, because I think that many structured interviews are plodding and wooden, while a wild one can really test a candidate. My current reaction, before I read it again, is that the terms structured and unstructured don't get to the key factors. I would like to see if very experienced interviewers are any better than occasional interviewers.

      Delete
  7. The authors say: One possibility is that just as it is possible to conduct poor
    structured interviews, it may also be possible to conduct good unstructured
    interviews. It looks as if a skilled interviewer, with much experience, can conduct a more free flowing interview which puts the interviewee into a relaxed and off guard manner, thus revealing more about their personality and conscientiousness. Perhaps the interviewers have to be tested for g?

    ReplyDelete