The concept of ‘doing your own research’ has become something of a meme during the pandemic for how it is basically a euphemism for falling down the rabbit hole of conspiracy theories. And this reputation is well-earned, as it more often than not is a bad faith call for putting trust in conspiracy theorists or pseudoscientists. To literally do one’s own research on such deeply esoteric topics is difficult if not impossible for an individual not in a scientific field that requires years of study. That said, I am an anarchist at heart, so I loathe to say one should trust any authority blindly or passively. So today, I am going to do my best to describe how scientific research works as an institution, why this process is so trustworthy, why there are limits on what the average person (without a scientific background) can reasonably do to participate in the research process, and what you as a layperson can do to engage with science responsibly.
Why You Shouldn’t
As unsatisfying as this is to hear, there are genuine obstacles to lay people trying to research topics for which they have not received some formal education. Not only does this education teach the information itself, but also the statistical and experimental methodologies needed to make sense of the raw data. Scientific studies and papers are written by experts for experts, so this background is usually necessary to fully appreciate the paper’s contents. But even outside of the densest studies, information that has been written for a layman audience can leave out important nuances, such as what type of study it is or how rigorous it is. Like any specialized field such as engineering or teaching, there is a reason why becoming a scientist takes years of schooling.
Arguably more importantly, there is the fact that in science, no one ‘does their own research’ in the sense this idiom implies. Science is an inherently collaborative practice with no one person or group acting as a final authority on what is correct. When scientists perform experiments, how this experiment was designed, what it was intending to test, all of the data that was collected, the statistical methods used to analyze the data, and what conclusions the experimenters drew from their findings are all collated into a scientific paper. These papers are submitted to journals where they are peer reviewed; other scientists in the same field review the paper to identify flaws in their methodology and filter out bad studies. The papers that make it through this process are published in the journal, making the study publicly available with a seal of trustworthiness from the journal. This allows other scientists to read about the study and perform similar experiments. If the same experiment is performed multiple times by different people in different places and gets similar results, it lends greater credence to those results. This process of collective review by knowledgeable individuals is why science is so trustworthy. Biased results are filtered out as they are scrutinized by a diverse group of others. Information is tested and tested again to confirm its validity. The scientific community is rather democratic in this sense; it’s a collective without any true leaders designed to make information more visible the more trustworthy it is. Theories only start to be considered ‘true’ once a significant majority of these experts come to the same conclusion. Being a good scientist means putting one’s trust into this community of fellows and allowing this collective knowledge to at least inform one’s own views on the topic.
What You Can Do
While ‘doing your own research’ as a non-scientist isn’t really possible in the most literal sense, that’s not really what most people who use this idiom are asking of you. When conspiracy theorists ‘do their own research’ or ask others to, what they’re really doing is engaging in motivated reasoning. They went into their ‘research’ having already decided what their final conclusion will be and are looking for information that supports that conclusion. Their data is usually cherry-picked or from questionable sources, but can be dressed up to look more trustworthy than it is. So, while doing cutting edge research is out of most people’s training, anyone can and should learn how to recognize trustworthy and untrustworthy news, information, or experts. So, let’s talk a bit about how to recognize bad science and bias when you see it.
Recognizing Pseudoscience
Fringe or pseudoscience can best be defined as beliefs or information that presents itself as scientific in nature to gain credibility but lack the rigor of actual scientific theory. While pseudoscience does include absolute hokum such as eugenics and climate change denial, it can also include debunked theories or new hypotheses with little support that otherwise have an ideological push behind them. Because these ‘theories’ become adopted on the basis of ideology instead of evidence, they tend to have some traits in common that can be easily identified;
Unfalsifiability: I’ve briefly mentioned burden of proof before, how it’s always the responsibility of whoever is making a claim to provide evidence that they’re right. So when someone makes a claim that can’t be proven or disproven, one should be suspicious. Scientific theories are designed to be potentially disproven, that way the longer they go without being disproved gives them greater credibility. But if a theory can’t be disproven, then there’s no way to gauge its quality, which should invite suspicion. A lot of alternative medicine uses unfalsifiable claims, claims that are vague enough to not be provable (removes toxins) or could be attributed to normal processes (cures diseases that go away on their own).
Anecdotal evidence: scientific studies seek to gather as much data as possible from as many sources as possible. Phase III drug trials can consist of up to 3,000 test subjects with a focus on a diversity of ages and backgrounds, all to ensure that the average person who receives a drug does better than the average person who doesn’t. If an experiment was done with two test subjects (one receives the drug and one receives a placebo), it would be impossible to tell if one did better because of the drug, their age, who they interacted with, some unknown gene one had, or any other factor. But pseudoscientific claims often lean on anecdotal evidence because humans tend to respond better to the personal experience of others than to raw data. One person claiming ivermectin or vitamin pills cured their Covid feels like it should count for more than a table of numbers, but a table of numbers is essentially just a thousand personal testimonies writ small.
Unchanging: the scientific process is ever-evolving, with theories being constantly refined and replaced as scientists are presented with new evidence and change their beliefs accordingly. So when a theory remains exactly the same for a long time, it’s likely not going through this scientific process. The modern anti-vax movement began in 1998 and has made roughly the same claim (vaccines are dangerous) about every type of vaccine ever since. They’re effectively claiming that none of the wildly different vaccination techniques out there are safe and that no one has managed to remedy this even after 24 years of research.
Claims certainty: as I’ve mentioned in my very first post here, science can be a very murky process. As more data is uncovered, conclusions and theories need to change. Especially in the case of cutting-edge research, (e.g. Covid-19), a good scientist needs to be able to change their mind in the face of new evidence. But this uncertainty is unsettling, so pseudoscientific claims will speak with absolute certainty of their correctness. While this might be comforting, it’s technically impossible to say anything with absolute certainty; we could discover the universe is a simulation tomorrow and that everything we knew was a lie. The scientific process isn’t about proving anything, it’s about reducing uncertainty. When a million people take a vaccine and less than ten experience severe side effects (actual statistics for covid vaccines), you can be very confident the vaccine will work for you . So counterintuitively, someone who says a drug has a 99.99% chance of treating your illness is a lot more trustworthy than one claiming a drug has a 100% chance of doing so.
Claims of conspiracy: whenever anyone in any circumstance tells you there is a concerted effort by everyone to silence them, that is a red flag. Not only is this an unfalsifiable claim (“all evidence I’m wrong is itself false”), the claim doesn’t add up. In 2016, Dr. David Grimes developed the Grimes equation, a mathematical model for how quickly any conspiracy would be uncovered. Given that each person in on the conspiracy is a potential leak, the model determines that for a conspiracy to last just five years, it couldn’t have any more than 2,521 conspirators. For perspective, the number of people globally working on climate research, who’d have to be sworn to secrecy if climate change was a hoax, is in the hundreds of thousands. If climate change was a hoax, someone would have blown the whistle by now. Additionally, whenever a scientist manages to disprove a long-held theory or consensus, they win nobel prizes and the theory that takes its place is named after them. This is not a field that rewards conspirators.
Doing Research
If you do want to delve into reading scientific studies as a non-scientist, there are a few things you can do to learn without getting lost or misled. I won’t go into any of the statistical methods or more complex concepts used by scientists to evaluate studies and compare studies to each other, but some of those topics can be read about in my sources if you’re interested and everything I’ve written here should help you to separate practically useful knowledge from potentially useful data from complete nonsense.
Finding consensus; while reading a single paper won’t tell you much about the truth, just as a single puzzle piece won’t tell you much about the final image, there are places you can go to find what a scientific field believes as a whole. Consensus reports are meta-analyses done on numerous studies to condense their findings into a single paper. Searching “[topic] consensus report” will lead to papers that tell you what an entire scientific field agrees upon. If there is a scientific consensus, you can find these reports. Another option would be position statements, the official stances of relevant scientific organizations. Search “[topic] [organization] position” to find the official stance of said organization. Trustworthy scientific organizations include the CDC, NIH, NSF, AAAS, and NAS.
When reading an individual scientific study, some easy things to look out for include;
Was the study done in humans or in animals or in vitro? While clinical studies done in rats or in cultured samples can provide useful data for future studies, they won’t translate to new cures for humans in the immediate future.
How big was the sample size? As stated before, studies become more reliable the more test subjects it has. An appropriate sample size can vary depending on what is being tested, but for studies that confirm a drug or medical treatment works well enough to be prescribed, the standard is between 300 and 3,000 people.
Limitations and conflicts of interest. At the end of most papers, there will be sections listing any weaknesses the experimenters felt the study had and where they will disclose conflicts of interest, such as being funded by a drug’s developer. These conflicts don’t automatically invalidate any positive results, but they could be a red flag.
Finding reliable sources; where a report or position is published can provide clues to how trustworthy it is. There are pseudoscience journals out there that try to push an agenda as well as predatory journals that will publish anything for money. There are a few metrics that journals use to determine how trustworthy they are, but the best technique regardless of scientific background will probably be lateral reading. When reading a paper or article, open another tab and google the publisher and/or author to find out about any conflicts of interest they have or what biases they might have. This is good advice both for scientific and any other form of research you might do.
Recognizing Personal Bias
A lot of what I’ve talked about today comes down to bias; a person’s personal prejudices from what someone has been taught, but also from ways the human brain takes shortcuts. While things like pseudoscience and fake news come from people with biases, it’s our own biases that may make us want to believe them. So I would argue the most important part of any kind of scientific or media literacy is being aware of your own biases so they cannot be used to manipulate you. That kind of self-awareness can be hard to develop and is well outside of my purview. I would recommend this post from the blog Thinking is Power (a major source for this blog post as well) which details many of the most common cognitive biases and how to recognize and overcome them. But for advice I can give, I think a good start would be to pay attention when an idea being true feels insulting to yourself or your ingroup. The human brain is built to find rational explanations for the world around us so it can protect us from harm. But when it counts a bruised ego as harm, this ability to rationalize gets turned against us as our brains look for reasons why an insulting truth must be wrong. This is of course not to say that if an idea is insulting, you should automatically assume it’s correct. But when you feel that a piece of information being true upsets you because it contradicts a deep-seated aspect of your worldview or sense of self, it might be time to apply everything we’ve talked about here a bit more rigorously to ensure you don’t wind up lying to yourself, something we are all at risk of doing.
While this topic is a bit different from what I normally write about, I would argue it might be the most important concept for non-scientists to understand about science. Humanity has already built up more knowledge than a single human being can learn in one lifetime, so we all must eventually trust in someone else’s knowledge. And putting that kind of trust into a group of strangers, particularly when they have some amount of authority, can be a legitimately frightening concept. So what I hope I’ve shown today is that science is not a black box that spits out facts, it’s a process of people questioning themselves and each other in good faith to refine our understanding of the world. And we can all use these same techniques to better insulate ourselves from being manipulated by bad actors. Part of being a free-thinker is to question yourself and what you believe, which is what science is all about.
Comments