01-intro.Rmd

---
output:
  pdf_document: default
  html_document: default
---
```{r setup-intro, echo = FALSE, warning=FALSE, message=FALSE}
library(tidyverse)
library(knitr)
library(kableExtra)
```

# Introduction {-#intro}

*Computational reproducibility* is the degree to which it is possible to obtain consistent results using the same input data, computational methods, and conditions of analysis [@national2019reproducibility]. In 2019, the [American Economic Association](https://www.aeaweb.org/journals/policies/data-code/) updated its Data and Code Availability Policy to require that the AEA Data Editor verify the reproducibility of all papers before they are accepted by an AEA journal. Similar policies have been adopted in political science, particularly at the [*American Journal of Political Science*](https://ajps.org/ajps-verification-policy/). In addition to the requirements laid out in such policies, the data editors of several social science journals produced detailed [recommendations and resources](https://social-science-data-editors.github.io/guidance/) to facilitate compliance. The goal of such policy changes is to improve the computational reproducibility of *all* published research going forward, after several studies showed that rates of computational reproducibility in the social sciences range from somewhat low to  alarmingly low [@galiani2018make; @chang2015economics; @kingi2018reproducibility]. 

This Guide includes a common approach, terminology, and standards for conducting *reproductions*, or attempts to assess and improve the computational reproducibility of published work. At the center of this process is the *reproducer* (you!), a party rarely involved in the production of the original paper. Reproductions sometimes involve the *original author* (whom we refer to as "the author") in cases where additional guidance and materials are needed to execute the process. Reproductions should be distinguished from *replications*, where replicators re-examine a study's hypotheses using *different data* or *different methods* (or both) [@King95]. We find that reproducibility is necessary for replicability, though both allow science to be "self-correcting."

We recommend using this Guide in conjunction with the [**Social Science Reproduction Platform**](https://www.socialsciencereproduction.org/) (SSRP), an open-source platform that crowdsources and catalogs attempts to assess and improve the computational reproducibility of published social science research. Though in its current version, the Guide is primarily intended for reproductions in economics, it may be used in other social science disciplines, and we welcome contributions that aim to "translate" any of its parts to other social science disciplines (learn how you can contribute [here](https://bitss.github.io/ACRE/contributions.html)). Find definitions of fundamental concepts in reproducibility and the process of conducting reproductions in the [Glossary](https://bitss.github.io/ACRE/definitions.html) chapter.

This Guide and the SSRP were developed as part of the Accelerating Computational Reproducibility  [(ACRe)](https://www.bitss.org/ecosystem/acre/) project, which aims to assess, enable, and improve the computational reproducibility of published social science research. The ACRe project is led by the Berkeley Initiative for Transparency in the Social Sciences [(BITSS)](https://bitss.org)—an initiative of the Center for Effective Global Action [(CEGA)](https://cega.berkeley.edu/)—and [Dr. Lars Vilhuber](https://www.vilhuber.com/lars/), Data Editor for the journals of the American Economic Association (AEA). This project is supported by [Arnold Ventures](https://www.arnoldventures.org/).

```{r weai-presentation, out.width = '80%', echo=FALSE, fig.cap= "BITSS Project Scientist Fernando Hoces provides an overview of the ACRE project and the process of conducting a reproduction as part of a webinar hosted at the 95th WEAI virtual conference Jun 27, 2020.", eval=FALSE}
knitr::include_url("https://www.youtube.com/embed/Xv1t8An2gPo")
```

[View slides used for the presentation "How to Teach Reproducibility in Classwork"](https://bitss.github.io/WEAI2020_slides/)

<!-- 
add clarebout principle and discuss effect on a more inclusive scientific community
-->

## Beyond binary judgments {-}

Assessments of reproducibility can easily gravitate towards binary judgments that declare an entire paper as "(ir-)reproducible". We suggest a more nuanced approach based on two observations that make binary judgments less relevant. 

First, a paper may contain several scientific claims (or major hypotheses) that may vary in computational reproducibility. Each claim is tested using different methodologies, presenting results in one or more display items (outputs like tables and figures). Each display item will itself contain several specifications. Figure \@ref(fig:diagram) illustrates this idea. 

```{r diagram, eval=TRUE, echo=FALSE,out.width = '100%', fig.cap="One paper has multiple components to reproduce. <br> DI: Display Item, S: Specification "}
library(DiagrammeR)

knitr::include_graphics("paper-claims.png") 

if (FALSE) {
grViz("
digraph a_nice_graph {

graph [layout = neato, rankdir= TB, overlap=true]  ## layout = [neato|twopi, etc]
#https://rich-iannone.github.io/DiagrammeR/graphviz_and_mermaid.html


# node definitions with substituted label text
node [fontname = Helvetica, shape = box, style=empty ]
paper [label = '@@1']    ## label indicates the position of the letter

node [fontname = Helvetica, shape = diamond, fontsize = 10, fixedsize = TRUE, fillcolor=Gray]
claim1 [label = '@@2-1', color=red]
claim2 [label = '@@2-2']
claim3 [label = '@@2-3']

node [fontname = Helvetica, shape = circle, fillcolor=YellowGreen, fixedsize = TRUE]
output1 [label = '@@3-1', color = red]
output2 [label = '@@3-2']
output3 [label = '@@3-3']
output4 [label = '@@3-4']
output5 [label = '@@3-5']
output6 [label = '@@3-6']

node [fontname = Helvetica, shape = circle, fixedsize = TRUE, fillcolor=Peru]
spec1 [label = '@@4-1', color=red]
spec2 [label = '@@4-2']
spec3 [label = '@@4-3']
spec4 [label = '@@4-4']
spec5 [label = '@@4-5']
spec6 [label = '@@4-6']
spec7 [label = '@@4-7']
spec8 [label = '@@4-8']
spec9 [label = '@@4-9']
spec10 [label = '@@4-10']
spec11 [label = '@@4-11']
spec12 [label = '@@4-12']


# edge definitions with the node IDs
paper -> {claim1} [color=red]      ##[label = ...] adds text on the edge
paper -> {claim2 claim3} 
claim1 -> {output1} [color=red]
claim1 -> {output2} 
claim2 -> {output3 output4}
claim3 -> {output5 output6}
output1 -> {spec1} [color=red]
output1 -> {spec2} 
output2 -> {spec3 spec4}
output3 -> {spec5 spec6}
output4 -> {spec7 spec8}
output5 -> {spec9 spec10}
output6 -> {spec11 spec12}
}

[1]: 'Paper'            ## adds label to each box
[2]: c('Claim 1', 'Claim 2', 'Claim 3')
[3]: c('DI 1' , 'DI 2', 'DI 3', 'DI 4', 'DI 5', 'DI 6')
[4]: paste0('S ', 1:10)

")
}
```

Second, for any given specification, there are several reproducibility levels, ranging from the absence of any materials to complete reproducibility starting from raw data. Moreover, even for a specific claim-specification combination, distinguishing the appropriate level can be far more constructive than simply labeling it as (ir-)reproducible.

Note that the highest level of reproducibility, which requires complete reproducibility starting from raw data, is very demanding and should not be expected of all published research — especially before 2019. Instead, this level can serve as an aspiration for social science research, as we look to improve the reproducibility of research and facilitate the transmission of knowledge throughout the scientific community.

## Reproduction stages {-}

Reproductions can be divided into five stages, corresponding to the first five chapters of this guide:   

0.    [**Paper selection**](#select), where you will select a *candidate* paper and try to locate its *reproduction package*. If a reproduction package is available, you will *declare* the paper and start the reproduction, or select a new candidate paper (after leaving a record on the SSRP);  
1.    [**Scoping**](#scoping), where you will define the scope of the exercise by recording the claims, display items, and specifications you will focus on in the remainder of the reproduction;  
2.    [**Assessment**](#assessment), where you will review and describe in detail the available reproduction package and assess the current level of computational reproducibility of the selected display items; 
3.    [**Improvement**](#improvements), where you will modify the content and/or the organization of the reproduction package to improve its reproducibility;  
4.    [**Robustness**](#robust), where you will identify analytical choices to increase the number of feasible robustness checks, and/or you will justify the reasonableness of a specific robustness check.  

This Guide does not include a possible fifth stage of **extension**, where you may extend the current paper by including new methodologies or data, which would bring the exercise closer to a *replication*.



```{r stages-intro, out.width = '100%', echo=FALSE, fig.cap="Four stages of a reproduction attempt"}
knitr::include_graphics("stages.svg") 
``` 

The order of the stages may not be chronologically linear. For example, you may realize that the scope of a reproduction is too ambitious and switch to a less intensive one. Later in the exercise, you can also begin testing different specifications for robustness while also assessing the paper's reproducibility level. The only stage that should go first, and cannot be edited once finished, is the Scoping stage.

Different stages in the reproduction process correspond to different units of analysis (see Figure \@ref(fig:stages-unit) for an overview). The Scoping stage will focus on scientific *claims* selected for reproduction. Once you specify your claims of interest, in the Assessment and Improvement stages you will focus on the display items associated with those claims. In the Robustness stage, claims are once again the unit of analysis.

```{r stages-unit, out.width = '100%', echo=FALSE, fig.cap="Relevant unit of analysis at each stage of a reproduction attempt", eval=TRUE}
knitr::include_graphics("unit-of-analysis.png") 
```



## Reproduction strategies {-}

In most cases, you will begin a reproduction with a thorough reading of your paper of interest. However, the sequence of the steps you take in the remainder of the reproduction may follow various  reproduction strategies. The most obvious strategy would be to follow the order of the steps as outlined above. You may also first choose one of the paper's many claims and then focus on assessing and improving the reproduction package accordingly. Using an alternative strategy, you might identify potential robustness checks or extensions while reading the paper and then focus only on the results associated with that robustness check. In another strategy, you may identify a paper that uses a particular dataset in which you are interested and then only reproduce or conduct robustness checks for the results associated with that dataset.