«You don’t need a RCT to show that parachutes work.»
Unknown
Considering the discussions about the effectiveness of early treatment for Covid — drugs not so-called vaccines — much emphasis has been put on rigorous testing whether these drugs actually work. If the work is actually done, after all, there is no desire to test the efficacy of off-patent drugs for which big pharma does not make big bucks. In part that is understandable, after all, these trials do costs a lot of money, and that money has to come from somewhere. (There’s an interesting opinion piece about it.)
So there is the usual approach, experts (doctors) try out different drugs and notice what work. If you see medicine as a craft, it does make sense. The doctor is an investigator, using experience combined trial and error and meticulous observation to find out what makes a dent into the disease. Well, if that’s possible, after all many doctors (esp. in the USA) were put under a lot of pressure not to try to find out what works and what does not. IIRC, if you are working in a hospital and you have to adhere to «standards of care», your options are curbed. You can do what you think is right, but you make yourself very vulnerable. And loss of license is a powerful threat to any doctor.
But even if that investigation of what works is possible, there is justified criticism of this method. After all, humans are bad observers, even trained experts. There are various biases, including confirmation bias, that impede accurate assessments of what actually works. In science, the gold standard — and pretty much the only way to find out whether there is a causal relationship between two variables — is the experiment (in the scientific sense). Or (a more accurate name) the randomized controlled trial. In short, you actively intervene (give one group a drug, the other a placebo or the currently established treatment method) and make sure to control for confounding variables (by randomly assigning the participants to the two groups).
While this method is very, very effective, it is not possible in all circumstances. Sometimes due to ethical reasons, sometimes due to financial ones. But there are other problems with this method. One very well-written (satirical) article examines these problems and it is well worth reading.
Yeh, R. W., Valsdottir, L. R., Yeh, M. W., Shen, C., Kramer, D. B., Strom, J. B., Secemsky, E. A., Healy, J. L., Domeier, R. M., Kazi, D. S., & Nallamothu, B. K. (2018). Parachute use to prevent death and major trauma when jumping from aircraft: Randomized controlled trial. BMJ (Clinical Research Ed.), 363. https://doi.org/10.1136/bmj.k5094
The responses on the BMJ website to this article are also worth reading. You notice when you read the article that the authors had a lot of fun, for example:
The study also has several limitations. First and most importantly, our findings might not be generalizable to the use of parachutes in aircraft traveling at a higher altitude or velocity. Consideration could be made to conduct additional randomized clinical trials in these higher risk settings. However, previous theoretical work supporting the use of parachutes could reduce the feasibility of enrolling participants in such studies.[16]
[16] Newton SI. Law of Universal Gravitation. Philosophiæ Naturalis Principia Mathematica, 1687.
or the Contributors statement:
Contributors: RWY had the original idea but was reluctant to say it out loud for years. In a moment of weakness, he shared it with MWY and BKN, both of whom immediately recognized this as the best idea RWY will ever have. RWY and LRV wrote the first draft. CS, DBK, JBS, EAS, and JLH provided critical review. RMD provided subject matter expertise. DSK took this work to another satirical level. All authors suffered substantial abdominal discomfort from laughter. RWY worried that BKN would not keep his mouth shut until the Christmas issue was published. […]
But as satire that deserves the name, the issues they raise are crucial, especially if we deal with new diseases. Yes, randomized controlled trials are the gold standard, but their results have to be examined critically.
