«There is one thing even more vital to science than intelligent methods; and that is, the sincere desire to find out the truth, whatever it may be.»
Charles Sanders Pierce
A recent exam in «statistics and methods of user research» used the following question:
Does popping bubble wrap before an exam reduce exam stress?
(massively shortened)
There was also this exam stress reducing bubble wrap on the door of the exam room (and yeah, the idea came from various images showing this kind of «stress relief»).
Students were then asked to describe how they would answer this question.
If you know research methods, you know that you could do an experiment, i.e. a randomized controlled study. For example, using a between-subjects design. Randomize students into one of two groups. The experimental group can pop bubble wrap before the exam. Check whether they actually do it as a manipulation check. The other (control) group does something else, e.g., what they normally do. Measure stress by giving them a questionnaire before the exam. Compare the stress levels (e.g., with a t-test for independent samples) and you might find out whether popping bubble wrap works (I skipped a few other things, like calculating power, sampling, possible biases, etc.).
The great thing about methods — you can apply them to anything. A long time ago, I was lurking in an archery forum. A former friend of mine was interested in it. The did not only discuss shooting but also building bows. One of the users claimed that cutting wood at «special times» (e.g., during full moon) would create bows and arrows that work better. He also claimed this effect was there but could not be examined.
Well, an experiment works here as well. Experiments work for almost anything. Unless you experiment on small children, then parents usually are an issue. But back to full-moon-cut wood — just get that «special times» wood and «regular» wood. Let some people create bows and arrows from it. Without knowing whether it’s regular or full-moon-cut wood. Otherwise they might make more effort with the «special wood», as it was more effort to get. Then take a group of archers. Let them fire with the bows and (e.g., five) arrows. Use either both «special» or «regular», in random order, without them knowing which bow-arrow combination is which. As with all «paranormal» studies, a double-blind procedure is critical. As dependent variable measure the points the bow-arrow combinations get. Calculate the sum of points the shooters did with all their «special» vs. «regular» bow-arrow-combinations. Compare the «special» and «regular» bow-arrow-combinations (e.g., with a t-test for dependent samples). Done. (Again skipping a few other things, like power, biases, etc.)
Thing is, methods are extremely useful. To conduct your own studies and to evaluate studies done by others.
While «all knowledge is worth having» (Carey), some knowledge is worth more than others. And given that science and its methods are value-neutral, methods should be flanked by ethics and the desire to search for the truth.
BTW, if you are interested in research designs, a good German book is:
Huber, O. (2013). Das psychologische Experiment. Eine Einführung. (6th ed.). Verlag Hans Huber.
As few people here can read German, IIRC a good English book is:
Goodwin, C. J., & Goodwin, K. A. (2013). Research in Psychology. Methods and Design. (7th Edition). John Wiley & Sons.
You also find good courses on Coursera. It is a very worthwhile investment of time. 🙂
