Benefits and Challenges of Making Qualitative Research More Transparent

In recent years, a movement to make research more transparent has taken root in the social sciences. Public institutions and private organizations of all types champion transparency, and publishers and funders increasingly require data sharing.[i] Innovative platforms and technical tools empower scholars to provide a more complete picture of their research. New academic norms are setting clearer expectations about what data and information scholars should provide so that others can understand and evaluate their work.

While scholars are growing increasingly committed to openness, they also encounter obstacles to acting on that responsibility. Implementing greater transparency requires addressing a host of issues, from technical challenges to ethical questions to professional concerns. Along with other participants in the transparency conversation, members of qualitative research communities are forming new consensuses on these questions. Their ideas are informed by the kinds of data and analysis that their respective research communities typically use.

In this piece we consider some of the challenges and benefits of sharing qualitative data and making qualitative research more transparent, discuss some general principles that can guide efforts to do so, and consider the evolving infrastructure supporting increased transparency.[ii]

The Challenges and Benefits of Transparency

Sharing data is a key component of research transparency. We can think of the texts, images, audio, and video collected or produced in association with a particular qualitative research project – through archival research, interviews, field observations, and other types of data gathering – as the qualitative analog of a quantitative dataset. However, sharing qualitative data in a way that increases research transparency can be at least as complicated and time-consuming as doing the same for numeric data.[iii]

Consider, for instance, a research project on the duration of civil wars that entails interviewing dozens of rebel fighters about sensitive issues such as how their personal histories led to their commitment to continue the struggle. If the interviews are audio-recorded, sharing them will involve transcribing and potentially translating the recordings. If rebel fighters participated in the study under the condition of anonymity, the interview transcripts will need to be de-identified, possibly leading to the removal of important insights.

If the scholar shares the interviews to provide evidence for her research findings, she will need to make the transcripts available to readers of her papers, ideally in a format that directly links relevant interview excerpts to the claims with which they are associated. Furthermore, the researcher will need to write documentation illuminating the research context and other facets of the research in order to aid others in effectively interpreting the data,[iv] while taking care not to provide information that will allow her respondents to be identified.

Despite the challenges and resources involved in making qualitative research more transparent, both individual researchers and qualitative research communities as a whole can benefit from the value that openness provides. Sharing data, along with information about how they were generated and analyzed in order to support the claims in a research publication, enhances the completeness, understandability, and evaluability of that publication. It allows scholars to earn credibility and legitimacy by demonstrating that they generated their results in accordance with the rules that guide their research methods and the norms of their research community.

In addition, other scholars can learn from shared data and (re)analyze them from different perspectives and for new purposes. To draw once again on our example, interviewing dozens of rebel fighters entails considerable effort. Sharing the information those interviews produce such that multiple scholars can review and analyze it allows more value to be gained from that hard-won and costly information. Finally, the teaching of research methods is enhanced when students can practice using analytic techniques on authentic research datasets.

Developing Protocols and Building Consensus for Transparency

As part of a broader movement towards transparency, scholars who produce qualitative research in political science and other fields are developing protocols and tools to make their work more open and building consensus on how to apply and use them.

In 2010, the American Political Science Association (APSA) launched the Data Access and Research Transparency (DA-RT) initiative to advance the conversation about research openness in the field. DA-RT involved discipline-wide discussions, publications, conference roundtables and panels, and the formation of various ad-hoc working groups.[v] From the outset, it was understood that making research more transparent should be a universal goal, equally applicable to both quantitative and qualitative research, but that the way in which this goal is accomplished should not be homogeneous or homogenizing.

One of the initiative’s key successes was a 2012 update to the discipline’s Guide to Professional Ethics in Political Science incorporating the following principles: (1) empirical researchers have an ethical responsibility to help one another understand their work by making their data and methods available; (2) critical ethical and legal imperatives, including the need to protect human subjects, should and will limit data sharing; and (3) researchers should retain the right to use their data for some period of time before sharing them.

An important step in translating these principles into action was the development of the Journal Editors Transparency Statement (JETS), authored by editors and subsequently adopted by more than two dozen journals, including prominent publications like the American Political Science Review and the American Journal of Political Science. JETS signatories pledged to introduce transparency standards for their journals requiring that both qualitative and quantitative data, and information about how they were generated and analyzed, be made available online. Those standards also called for the proper crediting of creators of datasets, in hopes of encouraging the sharing and reuse of valuable data.

While DA-RT applies to all types of empirical social science research, other initiatives in political science have focused specifically on making qualitative work more open. One example is the Qualitative Transparency Deliberations (QTD) sponsored by APSA’s Organized Section for Qualitative and Multi-Method Research (QMMR).[vi] QTD offers a forum for political scientists to consider the upsides, downsides, and practicalities of making qualitative research more transparent. Its development was animated, in part, by unease among some political scientists about the pace with which transparency discussions were proceeding in the discipline and their implications for certain types of qualitative research.

Since QTD’s launch in 2016, 13 collaborative working groups have considered core questions such as what transparency entails when working with data gathered from human participants and how empirical observations can be linked to findings produced using qualitative analytic methods like process tracing. As of this writing, the working groups are developing Community Transparency Statements that seek to outline best practices for making qualitative research more transparent.

The Importance of Infrastructure

Creating protocols and building consensus for qualitative research transparency are important first steps. However, in order for more scholars to be able to increase the transparency of their work, research communities also need access to appropriate infrastructure that addresses the challenges and concerns involved in sharing qualitative research. Digital data repositories that can serve as platforms for storing, preserving, and sharing qualitative data are one important element of that infrastructure.

Sharing data through a trusted digital repository such as the Inter-university Consortium for Political and Social Research (ICPSR), the Odum Institute Data Archive, the Qualitative Data Repository (QDR), or services such as Dataverse, Figshare, or Open Science Framework, has several advantages over simply posting them online on an individual website. Most repositories ask depositors to provide valuable metadata (“data about data”) along with their datasets. Metadata facilitate the discovery of datasets and associated research, as well as the analysis of data by new users, broadening the potential audience for and use of a scholar’s work.

Some repositories also provide curation assistance that makes the sharing process easier and increases the value of the data. They can also help ensure that data are shared ethically and legally, in accordance with the commitments scholars make to Institutional Review Boards and human participants and with attention to relevant intellectual property concerns. If depositors so desire, repositories can help them to place appropriate access controls on data so they are only viewable by a subset of users or under certain conditions. In addition, datasets housed in a repository are assigned a permanent digital object identifier (DOI), making them easily citable and guaranteeing that links to them will persist.

While many repositories primarily host quantitative data, some are designed specifically to address the unique needs and interests of qualitative researchers. QDR, a National Science Foundation-funded domain repository that launched in 2014 at Syracuse University, curates, stores, preserves, publishes, and makes available digital data from qualitative and multi-method research in the social sciences. Researchers can deposit their data with QDR and access its holdings free of charge, and staff are available to provide support and training. The repository currently has around 900 registered users and holds data from nearly two dozen projects.

In addition to housing data, QDR also develops and disseminates standards and techniques for sharing and reusing qualitative data and for pursuing qualitative research transparency more broadly. For instance, in partnership with Cambridge University Press and the technology firm Hypothes.is, and with support from the Robert Wood Johnson Foundation, QDR has begun to develop a new approach to transparency called “annotation for transparency inquiry (ATI).” ATI allows social scientists to link relevant data – a document or interview transcript, for instance – directly to a particular passage within a digital publication, and to use digital annotations to elucidate how those data support their claims and conclusions.              

A More Open Future

Over the last decade, there has been increased interest in research transparency across the social sciences. This ongoing conversation has served to illuminate the benefits that openness offers to qualitative social science, as well as the thorny issues that can arise when making some forms of qualitative research more transparent.

Scholars continue to raise critical questions about how and why to share data and how doing so positively impacts their ability to produce credible and legitimate knowledge. While some academics have capitalized on these new discussions to reprise old debates about the relative value of different forms of research, the most productive conversations have sought to grapple with the specific challenges of sharing qualitative data and analysis. Those discussions have generated at least three lessons.                                                           

First, while motivated by the same underlying benefits, data sharing and research transparency cannot and should not be accomplished in the same way across diverse social science traditions. The purpose of openness is to demonstrate how scholars arrive at their conclusions, and researchers must do so on the terms of their respective traditions, and in a way appropriate to the particular types of data and methods they have employed. When it comes to research transparency, one size does not fit all.

Second, while sharing some forms of qualitative research raises important questions and debates, much qualitative data can be shared relatively safely and easily. Moving expeditiously to increase access to those data has the potential to significantly enrich qualitative inquiry in the social sciences.

Finally, achieving openness requires the continued generation of new practices and better infrastructure. Both are more likely to be produced when scholars work together in a collaborative and inclusive fashion, engaging in sustained communication and exchange to identify strategies and solutions.

***

This article is part of a series on how scholars are addressing the “reproducibility crisis” by making research more transparent and rigorous. The series was produced by Footnote, an online media company that amplifies the impact of academic research and ideas by showcasing them to a broader audience, and Stephanie Wykstra, Ph.D., a freelance writer and consultant whose work focuses on data sharing and reproducibility. It was supported by the Laura and John Arnold Foundation.

Colin Elman is a Professor of Political Science at the Maxwell School of Citizenship and Public Affairs at Syracuse University and the Director of Syracuse’s Center for Qualitative and Multi-Method Inquiry and the Qualitative Data Repository (QDR). He is a leading member of the APSA’s Data Access and Research Transparency (DA-RT) initiative.

Diana Kapiszewski is the Provost's Distinguished Associate Professor in the Department of Government at Georgetown University and the co-founder and Deputy Director of the Qualitative Data Repository (QDR). She is also co-editor of the Cambridge University Press Methods for Social Inquiry book series and a member of APSA’s Data Access and Research Transparency (DA-RT) initiative.