Are science quizzes scientific?

 

 

Last year Pew Research announced the results of a science quiz they conducted in 2014 on a nationally representative US sample of adults. Here is their account of their findings, from which the above chart is drawn:

http://www.pewinternet.org/2015/09/10/what-the-public-knows-and-does-not-know-about-science/

Please take the test yourself right now, even if you have done so before, just to remind yourself of the items.

Of the 12 questions, I find that two are particularly weak. For example, it is a good idea to ask how vaccinations work. For example, the correct answer could be given thus: Vaccines work by making us produce antibodies to fight disease without actually infecting us with the disease. (A shorter answer would be: make us produce antibodies).  Instead the quiz asks you which person developed a particular vaccine. This is general knowledge, not knowledge of the underlying science. (I admit that when I was introduced to Jonas Salk at a Institute of Science breakfast I was overcome and could only mumble pleasantries).

There are many questions which could be asked in astronomy, such as “What makes astronomers think that the universe is expanding” (I hope that “red shift” would still be considered a good answer). In fact the quiz asks a question about astrology, which is definitely not science. A wasted question.

I think that 2 out of the 12 questions are feeble. This makes an already easy test easier, and reduces real differences between persons and groups. I hope readers will take the test, and I imagine they will get 12 out of 12, as I did. Too easy. Only two questions (boiling point lower at high altitude; loudness of sound shown by amplitude) have any bite to them, and only one question (what a magnifying glass does to light waves) is psychometrically close to the optimum, in the sense of having roughly a 50% pass mark, which is the best level for distinguishing one person from another. The other pass rates are far too high. Yes, one should have a few easy questions at the start, just to encourage people, but this test is very weak.

Looking at the above summary table of results, the education level table is also a measure of intelligence, and shows large differences in scores according to intelligence, particularly for those with only high school attainments. To my eye, age differences are minor by comparison. The white/black is very large, putting black respondents 30% below the white score. This is a considerable difference in science understanding.

Out of interest, I have plotted out the individual answers according to sex differences, showing percentage pass rates for males first, females second, and then the sex difference.

Astrology            73     72    1

Core of earth     89     84     5

Altitude              39     30     9

Jonas Salk         79     70     9

Comet                84     73    11

Correlation       69     58    11

Tides                 83      71    12

Light year         78     66    12

Loudness         42     30    12

Radio waves     79     66    13

Lens                   55     37    18

Uranium           90     75    25

Women know as much about astrology as men! Women do almost as well as men on the easy “hot core of the earth” question. Women do more badly than men on the difficult “altitude lowers boiling point” question. Most people have heard of the Salk vaccine.

Thereafter the sex gap widens. The overall difference amounts to 15% less science knowledge for women, and because of some of the weak items chosen that may be an under-estimate. Of course it would be very silly to link this to male brains being 13% larger than women’s. (More of this later, when I post up a recent lecture on sex differences in ability by Prof Richard Lynn).

On a brighter note, 64% of the US population understand correlation as depicted in a scatterplot. 

Yes, more money should be spent on science education. Of course it should. Raise high the roof-beams, carpenters!