Sample Text: "Understand and be able to analyze various sources of systematic error, including calibration errors, errors due to environmental perturbations, hidden assumptions in calculation or interpretation, subconscious/unconscious bias, etc."
Scientific results can be skewed in a variety of ways by subtle (and not so subtle!) systematic errors. One source of systematic error is bias of the researcher, often subconscious. Clearly many forms of bias intersect strongly with DSJ-related issues. In laboratory courses especially and also in lecture courses where the interpretation of experimental results are discussed, there is a great opportunity to discuss systematic errors due to bias and the importance of a diversity of perspectives in scientific analysis.
One nice example of how bias can lead to systematic error is the classic tale of the Millikan oil drop experiment from which we determined the charge of the electron. Here is the tale told by the Nobel-prize winning physicist Richard Feynman:
"We have learned a lot from experience about how to handle some of the ways we fool ourselves. One example: Millikan measured the charge on an electron by an experiment with falling oil drops, and got an answer which we now know not to be quite right. It's a little bit off because he had the incorrect value for the viscosity of air. It's interesting to look at the history of measurements of the charge of an electron, after Millikan. If you plot them as a function of time, you find that one is a little bit bigger than Millikan's, and the next one's a little bit bigger than that, and the next one's a little bit bigger than that, until finally they settle down to a number which is higher.
Why didn't they discover the new number was higher right away? It's a thing that scientists are ashamed of—this history—because it's apparent that people did things like this: When they got a number that was too high above Millikan's, they thought something must be wrong—and they would look for and find a reason why something might be wrong. When they got a number close to Millikan's value they didn't look so hard. And so they eliminated the numbers that were too far off, and did other things like that..." [1]
Other more directly DSJ-related versions of such bias can be investigated at, for example, the Project Implicit site at Harvard University and a seminar from the Association of American Medical Colleges