QDR and the AJPS Replication Policy

(Guest Posting by Colin Elman and Diana Kapiszewski)

The Qualitative Data Repository (QDR), located at Syracuse University, ingests, curates, archives, manages, durably preserves, and publishes digital data used in qualitative and multi-method social inquiry.  The repository develops and publicizes common standards and methodologically informed practices for these activities, as well as for reusing and citing qualitative data. As part of this broader undertaking, QDR welcomes the opportunity to work with other organizations and institutions as they pursue their transparency goals. QDR is pleased to have been selected by The American Journal of Political Science (AJPS) to help instantiate part of its revised Replication and Verification Policy.

AJPS has a long-standing commitment to the general principles reflected in the Data Access and Research Transparency (DA-RT) initiative. AJPS considers openness to be a fundamental component of social science. Accordingly, AJPS signed the Journal Editors Transparency Statement (JETS) in October 2014, pledging to implement policies by January 2016 that require authors of evidence-based articles to make as accessible as possible the empirical foundation and logic of inference invoked in their research.

Earlier this year, the Journal clarified and enhanced its Guidelines for Preparing Replication Files. Among other important changes, the Guidelines now provide more comprehensive directions for how scholars of qualitative research and multi-method research with a qualitative component can fulfill openness requirements.  Just as the Journal’s policies with respect to quantitative approaches are instantiated in cooperation with the University of North Carolina’s Odum Institute for Research in Social Science, the Journal’s new qualitative policies will be facilitated by QDR.

AJPS’ editorial position is that it publishes rigorous social science produced using public procedures. Subject to the ethical and legal constraints described in the Guidelines, the Journal takes the view that both data and analysis should be accessible to readers. Moreover, while not mandated by JETS, the journal also undertakes a pre-publication appraisal of the analysis in each evidence-based article that has been accepted for publication.

AJPS recognizes that data access and research transparency should be pursued in ways that are consistent with the type of social inquiry being conducted, the forms of evidence being deployed, the ways in which the data were generated, and the analytical processes that were used.  That said, the Journal is confident that its guidelines will apply to most empirical researchers whose goal is rigorous social science. The ideas underpinning the journal’s commitment to openness comprise a central element of scientific practice, regardless of the subject matter of, specific investigative strategy used in, nature of the data invoked in, or the analytic procedures employed in a particular publication.

Pre-Publication Replication

AJPS’ review process addresses a broad set of questions about the potential contribution of any manuscript to the stock of knowledge on a given topic. The Journal’s replication requirement speaks to a narrower issue. It calls on scholars to make their data and analysis available so that AJPS editors (facilitated by Odum and QDR) can ascertain whether the particular combination of data and analysis produces the claimed result.

AJPS takes the view that, for many types of scholarship, a third party should be able to replicate precisely the steps an author took to analyze her data, and arrive at exactly the same result. Least controversially, repetitions of explicitly algorithmic (often machine-assisted) analysis of a bounded (and typically interval level) dataset should lead to duplicate results. The archetype of scholarship suited to this kind of assessment is the statistical analysis of quantitative data. Certain types of qualitative research, such as automated content analysis and qualitative comparative analysis, are also readily amenable to this type of evaluation.

Replication is more challenging for qualitative research where the data analyzed do not form part of a bounded dataset with explicit codings, or where the mode of analysis is less obviously algorithmic.  Narrative case studies often combine these elements. When strict replication is infeasible, AJPS still requires authors to make their scholarship as understandable and evaluable as possible. Authors of qualitative research, like all AJPS authors, must explicitly state the logic of inference they are invoking, describe their research processes explicitly and precisely, and provide the materials necessary to elucidate how they arrived at their findings and conclusions.

Ethical and Legal Obligations and Transparency

According to AJPS’ Guidelines, authors may request a waiver to transparency requirements where sharing data could put the safety, dignity, or well-being of human participants at risk. Moreover, AJPS readily acknowledges that the person best positioned to assess the risk involved in disclosure is the author. The information that authors provide forms the basis of the Journal Editor’s decision concerning whether to grant the waiver.

AJPS strongly encourages authors not to consider providing access to the data underpinning their research as an “all or nothing” choice. Indeed, many scholars already routinely engage in practices that address the tension between transparency and protecting their human participants. For example, when a scholar quotes an anonymous source she is offering a de-identified version of the data precisely to address this tension. AJPS’ transparency requirements simply obligate scholars to render such choices patent and explain them. Moreover, the data management community is developing increasingly sophisticated mechanisms for allowing a reduced or modified view of data while protecting human participants, and AJPS encourages authors to use them. AJPS also understands that, in some situations, no mechanisms or strategies will effectively address human participants concerns, inhibiting the sharing of data that are associated with those project participants.

All AJPS authors must respect proprietary restrictions and copyright. As with human participants, however, it may be possible for some data under these types of constraints to be shared. For qualitative sources, for example, the “Fair Use” exemption outlined in the US 1976 Copyright Act suggests that some (small) portion of different types of copyrighted materials can under certain circumstances be shared for non-commercial use, or for the purposes of private study, teaching, or criticism/review.

Additional Observations

Recent discussions of qualitative data access and research transparency reflect some anxiety among scholars about what impact meeting these obligations may have on them and their work. We hope the information above, and the following observations, will help to address some of these concerns.

First, while there have been some disagreements about how openness is best achieved, the great majority of contributions to the conversation have accepted the general principle that openness facilitates the understanding and evaluation of published claims. AJPS’ policy is consistent with this widely shared consensus.

Second, all advocates of openness likewise recognize that it is an ideal that sometimes has to be modified in practice given competing imperatives. For example, AJPS recognizes that concerns about human participants require a good faith dialogue between authors and the journal. Authors identify data constraints when they submit their manuscript, and the editor communicates the Journal’s decision about how the journal will proceed with respect to those constraints prior to review. This exchange provides the author and the editor with a common understanding of how the data will be managed, and of the implications of any constraints they are under for replication and subsequent sharing, before the article is sent for review.

Third, only the data used to produce the results discussed in the publication need to be provided in order to comply with transparency requirements.  For example, a quantitative replication dataset need not include all the variables in the study dataset from which it was drawn, but rather just the variables included in the analysis.  Authors of qualitative scholarship are likewise only required to share the data underpinning central or contested empirical claims in their article. Beyond these minimum requirements, all authors need to make pragmatic judgements about how much data are needed to illustrate the empirical basis of their inquiry and make it fully and fairly evaluable.

As we hope is clear from the changes being introduced, AJPS welcomes submissions from all research traditions engaged in rigorous social science. We hope that the revised AJPS policy will be regarded as an on-ramp, and not a roadblock, for qualitative research.

Colin Elman, Syracuse University
Co-Director, Qualitative Data Repository and Methods Coordination Project

Diana Kapiszewski, Georgetown University
Co-Director, Qualitative Data Repository

AJPS to Award COS Open Practice Badges

By William G. Jacoby

The American Journal of Political Science has demonstrated its commitment to data access and research transparency over the past year through its rigorous replication and verification policy. Starting immediately, the AJPS will provide more visible signals of its adherence to these principles by adopting two of the “Badges to Acknowledge Open Practices” from the Center for Open Science (COS). Specifically, we will use the “Open Data” and “Open Materials” badges illustrated below. According to the COS guidelines,”(t)he Open Data badge is earned for making publicly available the digitally-shareable data necessary to reproduce the reported results.” Similarly, the guidelines state that “(t)he Open Materials badge is earned by making publicly available the components of the research methodology needed to reproduce the reported procedure and analysis.” Thus, the badges are intended to be a salient indicator that the articles to which they are awarded conform to the principles and best practices of openness in scientific research.

COS Badges

Any manuscript that has been accepted for publication at the AJPS and successfully completed the data replication and verification process will automatically meet the criteria for the Open Data and Open Materials badges. Therefore, upon release of the replication Dataset on the AJPS Dataverse, these two badges will be added to the metadata of the Dataverse Dataset. The badges appear near the bottom of the main page for the article’s Dataverse Dataset, along with the statement, “The associated article has been awarded Open Materials and Open Data badges. Learn more about Open Practice Badges from the Center for Open Science.” When the article, itself, is published, the badges will appear with the information near the beginning of the electronic version in the Wiley Online library. And they will be included as part of the statement about replication materials on the first page of the article’s print version.

Of course, some articles published in the American Journal of Political Science will not receive the Badges. For example, many formal theory manuscripts and virtually all of the normative theory manuscripts that are submitted to the Journal do not contain any empirical analyses. Such work is exempt from the AJPS Replication Policy, so the Open Practice Badges are not relevant to these manuscripts. And there are certain situations in which a manuscript may be given an exemption from the usual replication requirements due to the use of restricted-access data. In such cases, authors still are asked to explain how interested researchers could gain access to the data and to provide all relevant software code and documentation for replicating their analyses. Manuscripts in this situation would not receive the Open Data Badge, but they would be awarded the Open Materials Badge. Even with allowances for exceptions, we anticipate that the vast majority of the articles published in the American Journal of Political Science will receive both Badges.

The AJPS will be the first journal in political science to award Open Practice Badges to articles. Currently, the Badges are used by five other journals– four in psychology and one in linguistics. The Badges already appear in the AJPS Dataverse Datasets for all qualified articles (i.e., those that have successfully completed the replication and verification process). And starting today (May 10, 2016) they will appear in all articles published online in the Early View queue within the Wiley Online Library. Of course, this carries over to the print versions of the articles. The Open Practice Badges serve a useful purpose by helping to emphasize the distinctive quality of the work that appears in the American Journal of Political Science.

 

The American Journal of Political Science (AJPS) is the flagship journal of the Midwest Political Science Association and is published by Wiley.