What ideas do you have for how to respond to the objection "but anything's possible" when you are trying to argue that the evidence offered doesn't justify the conclusion? In particular, what can you say when, for example, you argue that meditation did not help reduce crime in Washington, DC - that the leader of the meditation group himself had predicted a significant decrease that summer and instead DC had the highest crime rate ever. Then the person you are telling this to says: "But think how high the crime rate might have been without the meditation groups!" Now I know that one way to respond is to talk about post-hoc predictions in statistics and the importance of predicting ahead of time what will happen, but this sort of explanation often makes little sense to the person who is untrained in probability theory and research design. So I am looking for another way to respond to objections like these.
Here's an example of what I thought was a great way to respond to students who would say, after hearing about an experiment: "But how do you know that X (the teaching method, the ringing bell - whatever the experimental variable being manipulated was) how do you know that that was what caused the differences between the 2 groups? There were so many other ways in which these groups differed." Of course I would talk then about the benefits of randomization, etc. but I found that a more effective way to respond was to use an analogy. The analogy was to imagine that I had 2 glasses of water, and I wanted to see if adding lemon to one of them could make a big enough difference that it could be detected when tasting the 2 glasses. Now of course it would be easy to tell which glass had the lemon if there were no other differences between the two. But now, I'd say, imagine that I add a little salt to this glass and much more to the other; some tabasco to this one, none to the other; sugar .. etc. Now I would say if you can still tell which glass has the lemon, that shows what a strong infuence that factor has. (I want to thank experimental psychologist Ray Littlejohn whom I think gave me this example years ago.)
So I'm looking for something like this analogy as a way to respond to the objection about "but think what it would have been if..." Any ideas? Maybe this could be topic for one of your discussions. Thanks -