Saturday, 2 November 2013

Flynn effect as a retesting, rule-based gain

It is very good to see a paper which takes a large scale effect, the secular rise in intelligence test results, and links it to an intriguing large scale explanation. A new contribution to understanding the Flynn Effect is to be found in the journal Learning and Individual Differences, which became available 30th OctoberElijah Armstrong and Michael Woodley “The rule-dependence model explains the commonalities between the Flynn effect and IQ gains via retesting” http://www.sciencedirect.com.libproxy.ucl.ac.uk/science/article/pii/S1041608013001556

Woodley you already know, but more about Armstrong later. Armstrong and Woodley argue that the Flynn effect is partly driven by the retest effect, whereby familiarity with the test material means that if you can learn a rule of thumb you can solve those particular sorts of problems when you see them again, without having to use much intelligence. In very simple terms, the test wears out quickly once you get to learn how it works. Using implicit learning and working memory, test takers learn how to solve rule dependent problems, which leads to apparent IQ gains which are partly independent of general intelligence.

As readers of this blog will know, the ultimate IQ test is the one for which no-one knows the answers at the moment. Intelligence tests in the real world are more modest affairs. Raven’s Matrices is a test based on progressions: you need to find the rule which underlies the visible changes in the problem arrays, and a good enough memory to hold in mind how those changes are progressing, so that you can correctly choose the final missing picture. For example, if the figures are growing as they move from left to right, and getting paler as they move from top to bottom, you still have to remember exactly how large and pale the missing item should be when you check through the proffered options. Carpenter et al. (1990) found that 5 rules covered all the items in the test. Once you know that, it is less of a test.

Another aspect of being test savvy is the capacity to de-contextualise, that is, to be able to generalise about types of problem, without being confused by the particular context in which the specific example is presented to you.

Armstrong and Woodley assert that, from the point of view of intelligence, education amounts to a vast re-testing enterprise. There are modest gains from rules of thumb, mnemonics and being “taught to the test”. Indeed, the reliance on exam results makes teachers and pupils confederates in ensuring that nothing is taught which is not taught to be examined. Incidentally, this view does not exclude what James Flynn calls “scientific spectacles” which more people now adopt when solving problems.

Armstrong and Woodley rank tests according to how much “cognitive scaffolding” they have. Raven’s Matrices is level IV: rules are very helpful, only a few of them are required; Catell Culture Fair is Level III: rules help, but will not help you on many items; the majority of IQ tests are Level II: very many rules are required, but working out which to use is difficult, (and selecting the rule is what requires general intelligence); and Draw A Man test is Level I: no rule is of much help.

They then simply correlate the vector of the position of any particular test in the rule dependence typology with the vector of the size of the Flynn effect on that test. A positive correlation would indicate that tests that were more dependent upon rules were yielding the larger Flynn effects. They tested it on 14 data sets, and found a correlation of 0.6

 

Full-size image (17 K)

 

The authors say: “It is proposed that tests like the Raven’s are only highly g loaded when encountered initially — even basic familiarity with the rules and heuristics on a test, or familiarity with inductive reasoning itself, has the potential to radically diminish the g loading of this test over time, both under controlled conditions (such as in a retesting scenario) and over larger societal time scales (i.e., across generations in the case of the Flynn effect).”

They continue: “The increasing capacity of societies to detect and explicitly utilize rules as a function of the Flynn effect may be related to increasing rule exposure via mass education and to ‘ways of thinking’ endemic to cognitive modernity (Flynn, 2009). It may also be caused by increased rule sensitivity as a consequence of changes in brain functioning that might relate to the prevalence of slower life history strategies in the modern era. These in turn might be in part a change in life history strategy due to dramatic improvements in environmental quality.”

This is a good paper. It contains lots of ideas, proposes a theory and then tests it, and draws out the conclusions in a thoughtful way. Not content with linking the observed phenomenon with the Flynn Effect and life speed theory, it also includes 5 testable predictions, to encourage other researchers to test whether their proposal has merit. It is a notable debut for the first author, whose first paper this is, and whose ideas formed the basis for the eventual publication.

Postscript

Elijah lives in Marin County, California, and is interested in philosophy and intelligence research. He originated the rule-dependence model in early 2012 and worked on it for eighteen months thereafter. He claims his conscientiousness is below the 10th percentile. He is also prone to end all his emails saying “Excuse typos, I typed this with my feet”. If you imagine that he is a sad old man gathering up a lifetime of scholarship into a well-honed rant, your imagination would be wrong. Elijah is 15. Welcome to a lifetime of academic labour!

14 comments:

  1. Is there an un gated version of the paper?

    ReplyDelete
    Replies
    1. That;s the uncorrected proof - a corrected proof will be available soon.

      Delete
  2. Elijah F. Armstrong3 November 2013 at 01:47

    Not yet. I hope to upload a pdf to my blog at some point.

    ReplyDelete
    Replies
    1. That is impressive at your age. It makes me embarrassed in comparison.

      Delete
    2. Thanks. Of course, the precocious generally burn out.

      Delete
    3. Oh my God. I've been congratulated by Steve Sailer. My life is complete. Thankyouthankyouthankyou.

      Delete
  3. "Of course, the precocious generally burn out."

    Have you got any statistics on that?

    ReplyDelete
    Replies
    1. No. It's a somewhat facetious statement.

      Delete
  4. Hi Elijah,

    I have spent a great deal of time, as a layperson on the subject, attempting to understand variations in human cognitive function. I have long suspected that individuals variation in cognition can be effectively modelled for a population on a local, but not global basis.

    I think that a global model would most effectively analyze the average differences between those locales.

    ReplyDelete
    Replies
    1. I'm not sure what you're talking about.

      Delete
    2. To put it in layperson's terms, intelligence simply means the ability to learn things, fact, theories, rules, etc. That ability is only manifested when it is directed, and in different social locales (I hate to use the phrase "cultural contexts") that ability will be directed to different things. Further, it seems logical that there is a finite amount of direction that is possible, so two different people with similar cognitive potential will have different manifestations of that ability if they have spent their lives having the direction of that ability formed in different social locations. This probably accounts for the Flynn Effect.

      As an aside, my aversion to using "cultural context" lies in the notion that "culture" is something akin to different flavors of ice cream, that cultural differences have no effect on measurable outcomes between the different groups of people who form different cultures. The reality is that different cultures direct the ability to learn in different directions so any differences in measure are going to have some variation depending on the environment in which one was raised.

      I'm guessing that even Raven's is going to have a cultural bias, in the sense that the sorts of rule-following and understanding are going to be more prevalent in some culture than in others.

      Delete
  5. Armstrong & Woodley :

    Theoretically, I believe this is likely. Gottfredson in "Why g Matters" says that in automated jobs, those requiring to repeat the same (working) process, g is less important. I think this is close to what your model is just saying.

    And yet there is something. Perhaps I'm incorrect, but given the rule-dependence theory, it seems that you think the more a test is rule-dependent (see here, for a brief description of a rule) the less g-loaded the test is. You cite Fox & Mitchum (2012). I read that paper few months ago, a pretty good one, but unless I misread it, the authors said that when you take people of 1940-cohort and people of 1990-cohort with equal total raw score on Raven, the old (1940) cohort is able to infer more rules than the recent cohort, not less. Given Carpenter et al. (1990) article (cited in your paper) it seems that the tests having more rules are more complex. I interpret this as saying "more g-loaded". Something is out of place. I would have expected, given your model, older cohort to underperform recent cohort on the ability of inferring rules, not the opposite.

    ReplyDelete