The limitations of sample size?
I had an interesting chat with a graduate student this past week. The project they’re working on involves surveying patients in pharmacies about their understanding of medical terminology. An important project given the lack of insights in the current literature from people living in rural communities. However, as is often the case with primary data collection, participant recruitment is going more slowly than might be hoped.
None of this will be a surprise to anyone who has conducted primary data research. One of the joys and curses of primary data work is that human beings rarely do what we expect, need, or want them to do.
At any rate I asked the student for the current number of participants (n=93), and then asked, “do you need more?” You see they had set a target of 200 participants based on a review of the previous literature but given the novelty of this project had not calculated a formal sample size target. They replied earnestly, “If we don’t get more participants, we won’t be able to conclude anything from what we have right now.”
They weren’t technically wrong if you’re thinking about establishing a causal relationship (though not the goal of this project). Given that we aren’t in a rush for these findings, we worked up a plan to switch up our recruitment strategy and continue with data collection. However, this discussion got me thinking about sample size, our current research environment, and research for impact
If your current work has anything to do with health research, you know that an adequately large sample allows to “generalize” your findings to the target population. Meaning you can have some degree of certainty that the intervention you tested will perform the same for people like those in your sample outside of study circumstances. All of which is very important in a field where we test new medications, procedures, and interventions.
As secondary data analysis and big data have proliferated the field our affinity for large sample sizes has only grown. These fields have given us ready access to data from hundreds, thousands, and sometimes millions of people with a few clicks of our mouse. According to the Google AI overview the benefits of big data include more personalized treatment options, enhanced operational efficiency, advancing research, improved diagnostic accuracy, reduced risk for medical errors, enhanced patient engagement, and improved public health. Again, all of this is a good thing. Health researchers want to make a positive impact on patients, communities, and the world.
Where this begins to fall apart a little, in my humble opinion, is in the integration of these findings into practice (i.e. where we start thinking about the application of these findings for impact). With the specialization of research disciplines and sub-disciplines it is not uncommon for researchers to spend all their time focussing on just the discovery of a new drug, treatment, or process. This requires a great deal of time, money, and effort.
Once completed these works are published and researchers move onto the next major project/discovery. Their work is done. The assumption is that in an evidence-based decision-making world, people will use the best available evidence to make their lives better. In practice we know this is not the case.
Clinical trials and secondary data analyses are just two kinds of research. Important no doubt, but by themselves not sufficient for making an impact. We need more people focussed on a version of the last mile problem. We need more people asking questions like:
Does the evidence created in these big trials apply to ALL people? (i.e. rural, marginalized, under-represented etc.)
How do we get the best available evidence being used in all settings faster and more reliably?
What do the people who aren’t included in these trials think and know about the problem? (Is the “problem” even one they’re worried about?)
These are questions for impact, and not enough people are thinking about them. What’s the benefit of thinking about these kinds of questions? Small sample sizes are okay (for now), because any information is better than what we currently have (which is none or not enough). Now this is not an excuse for you to do sub-par work, but free yourself up to not worry so much about sample sizes and focus more about getting novel voices into the discussion.
Interested in working for this kind of impact? Come back next week as I outline community engagement. Can’t wait? Reach out here and we can find a time to meet sooner ☺️.
(Words: 755)