Wednesday 14 September 2016

Are science quizzes scientific?


Pew research science knowledge


Last year Pew Research announced the results of a science quiz they conducted in 2014 on a nationally representative US sample of adults. Here is their account of their findings, from which the above chart is drawn:

Please take the test yourself right now, even if you have done so before, just to remind yourself of the items.

Of the 12 questions, I find that two are particularly weak. For example, it is a good idea to ask how vaccinations work. For example, the correct answer could be given thus: Vaccines work by making us produce antibodies to fight disease without actually infecting us with the disease. (A shorter answer would be: make us produce antibodies).  Instead the quiz asks you which person developed a particular vaccine. This is general knowledge, not knowledge of the underlying science. (I admit that when I was introduced to Jonas Salk at a Institute of Science breakfast I was overcome and could only mumble pleasantries).

There are many questions which could be asked in astronomy, such as “What makes astronomers think that the universe is expanding” (I hope that “red shift” would still be considered a good answer). In fact the quiz asks a question about astrology, which is definitely not science. A wasted question.

I think that 2 out of the 12 questions are feeble. This makes an already easy test easier, and reduces real differences between persons and groups. I hope readers will take the test, and I imagine they will get 12 out of 12, as I did. Too easy. Only two questions (boiling point lower at high altitude; loudness of sound shown by amplitude) have any bite to them, and only one question (what a magnifying glass does to light waves) is psychometrically close to the optimum, in the sense of having roughly a 50% pass mark, which is the best level for distinguishing one person from another. The other pass rates are far too high. Yes, one should have a few easy questions at the start, just to encourage people, but this test is very weak.

Looking at the above summary table of results, the education level table is also a measure of intelligence, and shows large differences in scores according to intelligence, particularly for those with only high school attainments. To my eye, age differences are minor by comparison. The white/black is very large, putting black respondents 30% below the white score. This is a considerable difference in science understanding.

Out of interest, I have plotted out the individual answers according to sex differences, showing percentage pass rates for males first, females second, and then the sex difference.

Astrology            73     72    1

Core of earth     89     84     5

Altitude              39     30     9

Jonas Salk         79     70     9

Comet                84     73    11

Correlation       69     58    11

Tides                 83      71    12

Light year         78     66    12

Loudness         42     30    12

Radio waves     79     66    13

Lens                   55     37    18

Uranium           90     75    25

Women know as much about astrology as men! Women do almost as well as men on the easy “hot core of the earth” question. Women do more badly than men on the difficult “altitude lowers boiling point” question. Most people have heard of the Salk vaccine.

Thereafter the sex gap widens. The overall difference amounts to 15% less science knowledge for women, and because of some of the weak items chosen that may be an under-estimate. Of course it would be very silly to link this to male brains being 13% larger than women’s. (More of this later, when I post up a recent lecture on sex differences in ability by Prof Richard Lynn).

On a brighter note, 64% of the US population understand correlation as depicted in a scatterplot. 

Yes, more money should be spent on science education. Of course it should. Raise high the roof-beams, carpenters!








  1. "uranium is needed to make nuclear energy and nuclear weapons"

    Bad question!

    You can make nuclear energy and weapons from Plutonium.

    A good rebuttal: but you need U to make Pu.

    But then: you can use Thorium, a naturally occurring element.

    So... I think the most strict answer is: "no".

    1. My reaction too. At that point I stopped reading.

    2. But if you start with Th you get U. (U232 IIRC)

    3. Maybe a bad question for you, not for 'us', general public.

      Everyone tend to become a above-avg specialist in something.

      The label ''science-question'' seems little unfair if most of things we learn have ''scientifical''-basis.

      specify the level of the science you are talking (or not, it's not important...)

      the ''life cycle'' of the ideas: born genius, grow creative, age conventional and many times die old fashioned.

  2. "Vaccines work by making us produce antibodies to fight disease without actually infecting us with the disease."

    Some vaccines (usually live, attenuated) do work by making people have weak/subclinical manifestation of the disease, which then give origin to strong immunity. These are slight more dangerous (i.e, the risk is still minimal) than other types of vaccines, but have much stronger immunological memory.

    1. I started with attenuation, then conceded that the antibody response, by whatever means, was the key concept. Remember, you have to chose the best of the 4 options. The effect is not achieved by the memory of water!

  3. It's like dealing with opinion pollers. My reaction is often to tell them that their question is lousy.

  4. I understand that there's a quite good one-question test: "why is it hotter in summer?". Pleasingly, most Harvard undergraduates fail it, apparently.

  5. Yes, a good question in the tropics, and quite hard for many people to answer.

  6. The scatterplot question is bad. The answers are about individuals, the data are country-level. I still gave the answer I knew was considered correct. In an ideal world, a follow-up question would have appeared, asking: "Which fallacy did you just commit? (a) Sunk cost fallacy; (b) False dichotomy fallacy; (c) Base rate fallacy; (d) Ecological fallacy.

  7. It's all a question of how mean you want to be. "How mean have you got?", I hear you ask. Very, very mean. But fair. Sort of.

    Analogies are easy to write but can be really hard to answer. Here's a hard one:

    1.) acceleration : length :: ____ : moment of inertia

    A. angular momentum
    B. viscosity
    C. energy
    D. electric potential (voltage)
    E. surface tension

    A bit easier:
    2.) area : velocity :: ______ : frequency

    A. length
    B. area
    C. volume
    D. flow rate
    E. time

    But even very simple questions are hard for most people:

    3.) work = force _____

    A. divided by time
    B. multiplied by time
    C. divided by distance
    D. multiplied by distance
    E. blank (work = force)

    4.) Most living organisms are composed primarily of _____.

    A. minerals
    B. carbon
    C. DNA
    D. water
    E. amino acids

    Figuring things out isn't always as easy as it seems, knowledge is important, too:

    5.)Water flows rapidly through a pipe that has a much narrower section in the middle. The pressure in the narrow part of the pipe is ______ in the wider part of the pipe?

    A. higher than
    B. lower than
    C. same as
    D. varying rapidly between higher than, lower than and the same as
    E. not enough information

    6.) Humans get ______ of the DNA bases in each cell from each parent.
    A. exactly half
    B. about half
    C. B. / girls get a bit more DNA from mom, boys from dad
    D. A., exactly equal, except in rare cases
    E. B. / a bit more from mom except in rare cases

    Even easy knowledge questions can become harder with tempting or sciencey-sounding bad answers:

    7.) Stars like the Sun are powered by:
    A. stellar oxidative combustion
    B. nuclear fission
    C. gravitational potential
    D. protons combining into alpha particles
    E. light

    8.) Most nuclear power reactors (non-research) operate by:

    A. converting radioactive ions directly to electricity
    B. oxidation of radioactive elements
    C. nuclear magnetic resonance
    D. boiling water
    E. neutrino current conversion

    9.) Airplane wings work primarily by:

    A. deflecting air downward
    B. creating high pressure above the wing
    C. creating low pressure below the wing
    D. creating a vortex at the wingtip
    E. all of the above

    10.) Water is an electric _____.

    A. weak resistor
    B. resistor
    C. conductor
    D. semiconductor
    E. paraelectric



    To make it more difficult to inadvertently pick up answers from the key ahead of time, there's some padding at the front and back (-1Ph.D|0B|1C| |2A|3D|4D |5B|6E|7D|8^D |9A|10B|11AC |12:-D) which makes it somewhat less obvious and gives a bit of camouflage, as does this redundantly surplus extra verbiage at the end.

    1. Thank you for your prospective questions, which I enjoyed, particularly the one about wings. I have read the debates on that particular issue with great interest. On many of the others I am still baffled, but certainly impressed. All we need for survey purposes are 4 or 5 more items with pass rates of 30% to 50%. Respondents have to be saved from despondency, whilst also being tested.

    2. Thanks. I suspect most of these questions would have pass rates in the 20s. One gets less information from the answers but more satisfaction from imagining the nonplussed expressions of the test-takers. Of course you need harder-than-average questions to get a ~50% question pass rate with smarter-than-average test-takers. I suspect the airplane one might actually be easier for those who haven't been exposed to the usual "explanations", so wouldn't make it through validation. (It might work in an adaptive question bank, though, when given only to higher-ability test-takers.)

      (Regarding airplane wings' lift, for readers new to the question, vortices do have something to do with it, at least for wings that are flatter on the bottom, but not wingtip vortices, which go in a different direction and just create drag.)

      You might also enjoy the "downwind faster than the wind" controversy if you missed it, the discussion threads were the best knock-down drag-out argument since the Monty Hall problem (the three doors statistics puzzler).

      The first two questions become much easier when converted to spatial relations using the ""Physical Units Factor Tables". Question 3, "work = force times distance" can also be answered easily using PUFT if you know work = energy. (I'm working on using PUFT as a board for a game. The immediately preceding blog post to the one linked has a prettier and easier but smaller-type version.)

      #5 is the Venturi effect, the basis for the faucet-powered aspirator /vacuum sometimes seen in labs, as well as the carburetor and the perfume atomizer. You might get a strong whistle (almost like answer D.) under some conditions, but the max pressure in the constriction at any given time will always be lower than the minimum pressure in the wide part of the pipe. (At least up to a few km/s flow speeds, anyway.)

      I'm pretty sure the inherited DNA bases question's (#6) answer is correct because of mitochondrial DNA and the difference in size between the X and Y chromosomes, but you'd probably know better than I.

      #7 "protons combining into alpha particles" is another way of saying hydrogen nuclei (protons) fusing into helium nuclei (alpha particles).

      Correction: I probably should have had "insulator" instead of "resistor" in the last question. A 1cm cube of deionized water has a resistance of nearly 20Mohm between the faces. It's sometimes used as a dielectric in capacitors.

      But I can do easier questions!

      11.) The Earth best approximates a ____
      A. sphere
      B. oblate spheroid
      C. prolate spheroid
      D. scalene ellipsoid
      E. what!? You call this easy?

      Oh, well. I guess I'll have to try again.

    3. "oblate spheroid ... prolate spheroid": one's like a smartie, the other's like a rugby ball. Then all you have to do is memorise which is which. That's the difficult bit.

    4. Yah, Earth is like the smartie, fat around the equator as befits a middle-aged planet. Oblate, I think, but I'm too lazy to look it up.

  8. disappointed that more women haven't boiled water at sea level and in the mountains, or baked, for that matter.

  9. I am actually surprised that people did as well as they did. Some of the questions struck me as rather hard for a general population sample. (Embarrassingly enough, I got the magnifying glass wrong!)

    1. I stared intently at that one, and managed to work it out. Close call.

  10. The quiz may be a weak proxy for math ability. I notice that the questions that had the least correct answers I learned in a quantitative science class in college. The core of the earth is the question with the most correct answers is taught in geology. Geology is the physical science subject that is most commonly taken that has the least math.

    Least correct
    Altitude (learned in physics)
    Loudness (learned in physics)
    Lens (learned in chemistry)

    Most correct
    Core of earth (geology)
    Uranium (learned in history)
    Comet (in the News when they appear)

  11. 9 correct answers

    I find interesting how people give great relevance to the geographical knowledge, ;)

  12. "I notice that the questions that had the least correct answers I learned in a quantitative science class in college." In my day they were all secondary school material. And so the world turns.