3 Experimental Design and Replication
3.1 Why is experimental design and replication an ethical concern?
A good, ethical experiment starts at the beginning with a concrete question to answer and a carefully considered experimental design that thoroughly investigates and accounts for all aspects of the scientific inquiry. The initial stages of experimental design are an opportunity for scientists to consider the ethical ramifications and potential issues that may arise. By doing so, they can then take steps to prevent or account for ethical concerns. Moreover, a good experimental design will not only mitigate ethical concerns like conflicts of interest and bias, but it will also ensure that the analysis is appropriate, and the entire experiment is reproducible. Reproducibility is critical to validate science and maintain not only public trust but trust between scientists (NIH, 2016). The ability to reproduce an experiment ensures that it is reported accurately in its entirety and that the results are not a fluke or misinterpretation. Ethically speaking, reproducibility is a concern for several reasons. A lack of reproducibility raises suspicions of fraudulent or falsified data, can cause other scientists to rely on invalid data, wastes resources and can taint entire fields of research (Resnik and Shamoo, 2017). Increasingly, scientists are becoming concerned about a “crisis of reproducibility”. Experiments, particularly novel experiments, are frequently impossible to confirm through reproduction of results. Many factors cause irreproducibility; poor experimental design, inadequate detail reported or training, conflicts of interest, poor statistical analysis, unreliable or problematic reagents and misaligned incentives for scientists are a few examples (Flier 2022). With careful consideration, these factors can be prevented to preclude reproducibility issues from arising.
Note that the results from an experiment that cannot be reproduced are not necessarily wrong. They may be factually correct but cannot be corroborated, which calls their accuracy into question (Flier, 2022).
3.2 Example: Betatrophin
In April 2013, a paper was published which claimed to have discovered a molecule secreted by the liver which when expressed in mice created massive replication of insulin-producing beta cells (Yi et al., 2013). They called this molecule “betatrophin”. This would be a milestone discovery in the treatment of type 1 diabetes, a disease which personally affected the primary investigator, Doug Melton. Almost immediately after publishing, colleagues were raising concerns about the experiment. Some raised technical issues while others pointed out that this “new molecule” was not new at all. Additionally, attempts to replicate the results failed. Other researchers found that mice genetically modified to be unable to produce betatrophin still produced beta cells as well as normal mice. This proved that betatrophin was not responsible for stimulating beta cell production. These normal mice also showed no increase in beta cell growth in response to increased betatrophin levels (Flier, 2022).
In response to these rebuttals, Melton not only accepted them but explained that subsequent unpublished studies showed less of a response to betatrophin, and that the lab was going to be investigating further. In cooperation with several other labs, Melton’s lab tested beta cell response in a blind study. No response to betatrophin was observed (Flier 2022). A paper describing the original and follow up studies was published to resolve the discrepancies (Cox et al., 2016). These errors were most likely due to technical lab mistakes and confirmation bias. The paper was retracted from the journal Cell as a result. This example shows that even well-respected, experienced researchers like Melton are not immune from or infallible to publishing results that turn out to be mistaken (Flier 2022).
3.3 Practice Questions
- Unreproducible fruit flies
You have been working on an experiment examining the effects of physical activity on obesity. You used fruit flies (Drosophila melanogaster) as a model organism. Your study has two groups of flies: one group was kept in small containers to prevent physical activity and one group was kept in a container with rotating foam arms to prevent flies from landing, thereby forcing them to be physically active. Both treatments were fed the same food at the same time. After finding that there was a high correlation between sedentary behaviour and obesity, you published the paper. Not long thereafter, your paper began to be criticized for its poor experimental design and inability to reproduce the experiment.
2. Statistical analysis
You have been working in a gene therapy lab for a few months and have been added to an existing experiment that is attempting to create a treatment for Stargardt disease, a degenerative eye disease that causes blindness. It is brought to your attention that the lead researcher was inspired to work on this project after a friend was diagnosed with the condition. You raise some concerns about the stringency of their control measures but overall feel that the experiment was well thought through. You discuss team roles and decide that you will perform the statistical analysis the team selected in an earlier meeting. As you are analyzing the data, none of the statistical analysis planned at the outset of the experiment is showing significance. After you mention this to the lead researcher, they tell you that they ran some different models that did show significance.
Upon examination of the “significant results” obtained by the lead researcher, you realize that they only ran a subset of the data using the results from the most promising patients and did not use an equal number of control data points. These factors made the statistical analysis “strongly significant”.
3. Confounding variables
The experiment that you are working on requires you to take precise measurements from Wood frogs to better understand how their blood produces compounds with antifreeze-like properties. You will be measuring their weight, the amount their body contracts when cold and obtaining blood samples. The frogs were collected in the wild during the spring while they were still frozen. There were 6 male frogs and 8 females. During the study, you cool the frogs and take blood samples to track the rate at which the compounds are being produced. Each of the frogs was cooled and rewarmed repeatedly.
During the experiment, you fall ill and pass off the measurements to another lab member to finish them. While they are performing the experiment, 2 male frogs and one female frog die. Later when you are analyzing the data, you notice that the data from the other lab member is less precise and more inconsistent than your own measurements. When you ask them about how they performed the measurements, you discover that they were not using the same scale, rulers and thermometer as you did.
4. Use and care of animals
You are conducting an experiment testing the muscular reaction speed of a type of endangered salmonid. You are comparing their reaction time to that of other more common salmonids. The endangered fish were lab reared from the time they were eggs. For the experiment, the fish swim in a narrow flow tank at a moderate velocity well within their natural range. At random, small blunt implements poke the fish and the time it takes to react to the stimulus is recorded. The common salmonids experienced no negative impacts from the experiment, but you notice that 3 of the endangered species had developed lesions on their pectoral fins and the sides of their bodies after the experiment.
Since the fish are endangered, it is decided that the stimulus will be changed so that there is no chance of contact with the probe. Unfortunately, it is determined that they are reacting to both types of stimuli by flinching into the tank causing lesions. To salvage your experiment, you decide to cease the reflex portion of the experiment and instead will take very small muscle tissue samples to look for differences in the muscle fibres. You will sedate the fish during the procedure and give preemptive antibiotics to prevent infection at the tissue sites. You hope that this will show differences that explain why the endangered species was much more reactive.
