(Guest Posting by Colin Elman and Diana Kapiszewski)
The Qualitative Data Repository (QDR), located at Syracuse University, ingests, curates, archives, manages, durably preserves, and publishes digital data used in qualitative and multi-method social inquiry. The repository develops and publicizes common standards and methodologically informed practices for these activities, as well as for reusing and citing qualitative data. As part of this broader undertaking, QDR welcomes the opportunity to work with other organizations and institutions as they pursue their transparency goals. QDR is pleased to have been selected by The American Journal of Political Science (AJPS) to help instantiate part of its revised Replication and Verification Policy.
AJPS has a long-standing commitment to the general principles reflected in the Data Access and Research Transparency (DA-RT) initiative. AJPS considers openness to be a fundamental component of social science. Accordingly, AJPS signed the Journal Editors Transparency Statement (JETS) in October 2014, pledging to implement policies by January 2016 that require authors of evidence-based articles to make as accessible as possible the empirical foundation and logic of inference invoked in their research.
Earlier this year, the Journal clarified and enhanced its Guidelines for Preparing Replication Files. Among other important changes, the Guidelines now provide more comprehensive directions for how scholars of qualitative research and multi-method research with a qualitative component can fulfill openness requirements. Just as the Journal’s policies with respect to quantitative approaches are instantiated in cooperation with the University of North Carolina’s Odum Institute for Research in Social Science, the Journal’s new qualitative policies will be facilitated by QDR.
AJPS’ editorial position is that it publishes rigorous social science produced using public procedures. Subject to the ethical and legal constraints described in the Guidelines, the Journal takes the view that both data and analysis should be accessible to readers. Moreover, while not mandated by JETS, the journal also undertakes a pre-publication appraisal of the analysis in each evidence-based article that has been accepted for publication.
AJPS recognizes that data access and research transparency should be pursued in ways that are consistent with the type of social inquiry being conducted, the forms of evidence being deployed, the ways in which the data were generated, and the analytical processes that were used. That said, the Journal is confident that its guidelines will apply to most empirical researchers whose goal is rigorous social science. The ideas underpinning the journal’s commitment to openness comprise a central element of scientific practice, regardless of the subject matter of, specific investigative strategy used in, nature of the data invoked in, or the analytic procedures employed in a particular publication.
Pre-Publication Replication
AJPS’ review process addresses a broad set of questions about the potential contribution of any manuscript to the stock of knowledge on a given topic. The Journal’s replication requirement speaks to a narrower issue. It calls on scholars to make their data and analysis available so that AJPS editors (facilitated by Odum and QDR) can ascertain whether the particular combination of data and analysis produces the claimed result.
AJPS takes the view that, for many types of scholarship, a third party should be able to replicate precisely the steps an author took to analyze her data, and arrive at exactly the same result. Least controversially, repetitions of explicitly algorithmic (often machine-assisted) analysis of a bounded (and typically interval level) dataset should lead to duplicate results. The archetype of scholarship suited to this kind of assessment is the statistical analysis of quantitative data. Certain types of qualitative research, such as automated content analysis and qualitative comparative analysis, are also readily amenable to this type of evaluation.
Replication is more challenging for qualitative research where the data analyzed do not form part of a bounded dataset with explicit codings, or where the mode of analysis is less obviously algorithmic. Narrative case studies often combine these elements. When strict replication is infeasible, AJPS still requires authors to make their scholarship as understandable and evaluable as possible. Authors of qualitative research, like all AJPS authors, must explicitly state the logic of inference they are invoking, describe their research processes explicitly and precisely, and provide the materials necessary to elucidate how they arrived at their findings and conclusions.
Ethical and Legal Obligations and Transparency
According to AJPS’ Guidelines, authors may request a waiver to transparency requirements where sharing data could put the safety, dignity, or well-being of human participants at risk. Moreover, AJPS readily acknowledges that the person best positioned to assess the risk involved in disclosure is the author. The information that authors provide forms the basis of the Journal Editor’s decision concerning whether to grant the waiver.
AJPS strongly encourages authors not to consider providing access to the data underpinning their research as an “all or nothing” choice. Indeed, many scholars already routinely engage in practices that address the tension between transparency and protecting their human participants. For example, when a scholar quotes an anonymous source she is offering a de-identified version of the data precisely to address this tension. AJPS’ transparency requirements simply obligate scholars to render such choices patent and explain them. Moreover, the data management community is developing increasingly sophisticated mechanisms for allowing a reduced or modified view of data while protecting human participants, and AJPS encourages authors to use them. AJPS also understands that, in some situations, no mechanisms or strategies will effectively address human participants concerns, inhibiting the sharing of data that are associated with those project participants.
All AJPS authors must respect proprietary restrictions and copyright. As with human participants, however, it may be possible for some data under these types of constraints to be shared. For qualitative sources, for example, the “Fair Use” exemption outlined in the US 1976 Copyright Act suggests that some (small) portion of different types of copyrighted materials can under certain circumstances be shared for non-commercial use, or for the purposes of private study, teaching, or criticism/review.
Additional Observations
Recent discussions of qualitative data access and research transparency reflect some anxiety among scholars about what impact meeting these obligations may have on them and their work. We hope the information above, and the following observations, will help to address some of these concerns.
First, while there have been some disagreements about how openness is best achieved, the great majority of contributions to the conversation have accepted the general principle that openness facilitates the understanding and evaluation of published claims. AJPS’ policy is consistent with this widely shared consensus.
Second, all advocates of openness likewise recognize that it is an ideal that sometimes has to be modified in practice given competing imperatives. For example, AJPS recognizes that concerns about human participants require a good faith dialogue between authors and the journal. Authors identify data constraints when they submit their manuscript, and the editor communicates the Journal’s decision about how the journal will proceed with respect to those constraints prior to review. This exchange provides the author and the editor with a common understanding of how the data will be managed, and of the implications of any constraints they are under for replication and subsequent sharing, before the article is sent for review.
Third, only the data used to produce the results discussed in the publication need to be provided in order to comply with transparency requirements. For example, a quantitative replication dataset need not include all the variables in the study dataset from which it was drawn, but rather just the variables included in the analysis. Authors of qualitative scholarship are likewise only required to share the data underpinning central or contested empirical claims in their article. Beyond these minimum requirements, all authors need to make pragmatic judgements about how much data are needed to illustrate the empirical basis of their inquiry and make it fully and fairly evaluable.
As we hope is clear from the changes being introduced, AJPS welcomes submissions from all research traditions engaged in rigorous social science. We hope that the revised AJPS policy will be regarded as an on-ramp, and not a roadblock, for qualitative research.
Colin Elman, Syracuse University
Co-Director, Qualitative Data Repository and Methods Coordination Project
Diana Kapiszewski, Georgetown University
Co-Director, Qualitative Data Repository
