Data.txt fields

Example:

ARTICLE:ANALYSIS_BY[name] studentX
ARTICLE:ANALYSIS_TIME[minutes] 4
ARTICLE:COMMENT[string] 
ARTICLE:COMMERCIAL_EFFORT[none,part,full] none
ARTICLE:GRANT_SUPPORT[none or string] Supported by the National Science Foundation under grant CCF-0905509.
ARTICLE:IMPLEMENTATION_EXISTS[unknown,hardware,yes,no] yes
ARTICLE:LINK[url] http://doi.acm.org/10.1145/2400682.2400716
ARTICLE:NSF_SUPPORT[none or number] CCF-0905509.
ARTICLE:PAGE_COUNT[pages] 20
ARTICLE:STATUS[not_finished,finished] finished
ARTICLE:TYPE[conference,journal,poster,abstract] journal
AUTHOR:NAMES[list of first_last] Mehmet_E._Belviranli Laxmi_N._Bhuyan Rajiv_Gupta
BIBTEX:LABEL[string] BelviranliBG13
BIBTEX:LINK[url] http://dblp.uni-trier.de/rec/bibtex/conf/taco/BelviranliBG13
BUILD:ANALYSIS_BY[name] 
BUILD:ANALYSIS_TIME[minutes] 
BUILD:BENEFIT_OF_DOUBT[unknown,yes,no] unknown
BUILD:COMMENT[string] 
BUILD:error-comment[none,not_needed,comment] not_needed
BUILD:STATUS[one of {unknown,needed,not_needed,started,finished} and list of {downloaded,compiles,runs}] unknown
BUILD:VERIFY_BY[name] 
BUILD:VERIFY_COMMENT[string] 
BUILD:VERIFY_STATUS[needed,not_needed,started,finished] not_needed
BUILD:VERIFY_TIME[minutes] 
COMMENT:CORRECTION[string] 
COMMENT:RESPONSE[string]
EMAIL1:CODE_AVAILABLE[yes,no,no_response] no_response
EMAIL1:REMARK[comment] 
EMAIL:STATUS[unknown,not_needed,not_found or list of {needed,request_1,response_1,sent_thank_you}] needed request_1
PI:COMMENT_CC[string] 
PI:COMMENT_TP[string] 
TOOL:ARTICLE_LINK[unknown,none,url,broken and url] none
TOOL:DATA_LINK[unknown,none,url,broken and url] 
TOOL:EMAIL_LINK[unknown,none,sent_no_url,url,broken and url] 
TOOL:GOOGLE_LINK[unknown,none,url,broken and url] none
TOOL:NAME[string] HDSS
VERIFY:ANALYSIS_BY[name] studentY
VERIFY:COMMENT[string] author has tried scheduling and load balancing on different architectures, but probably they can provide their code
VERIFY:STATUS[unknown,needed,not_needed,started,finished] finished

Interpretation

ARTICLE

AUTHOR

BIBTEX

BUILD

COMMENT

Both of these fields use the abbreviations for paper classification that are used in the result web table and the paper's result table. The abbreviations are described below. * CORRECTION: Comment on corrections between the first released version and the final version. * RESPONSE: Response to what the authors have said in the survey.

EMAIL1

EMAIL

PI

TOOL

VERIFY

Error Categories

Possible values of the ERROR_COMMENT field.

Build notes format

The build template has space for two separate attempts to build and run the code. The name of the student is placed on one line, along with the environment. Dependencies are put between two tags and finally notes on the process are between two more tags. In some cases more than two students attempted a build.

Each attempt is marked with a number, e.g. 2:BUILD_BY is the second build attempt.

The template:

1:BUILD_BY[name]
1:BUILD_ENVIRONMENT[operating system 32 vs 64] 
1:DEPENDENCIES[list of dependencies with where to get them]

1:END_DEPENDENCIES
1:NOTES[notes on attempted build]

1:END_NOTES

2:BUILD_BY[name] 
2:BUILD_ENVIRONMENT[operating system 32 vs 64]

2:END_BUILD_ENVIRONMENT
2:DEPENDENCIES[list of dependencies with where to get them]

2:END_DEPENDENCIES
2:NOTES[notes on attempted build]

2:END_NOTES

Survey results format

Multiple authors could respond to the same survey so each survey.txt includes summary fields. These were set, either as a copy of the only author, or if conflicting author responses by a number of criteria. If the conflict had a more generous interpretation, this was taken or the conflict was resolved to match our question intention, if that was at issue.

There was confusion regarding CLASSIFICATION. We have reframed this as whether the paper is backed by code or not, rather than the loaded distinction between practical and theoretical papers.

The template:

ANONYMOUS_COMMENT[string] 
BUILD_COMMENT[string] 
BUILD_DIFFICULTY[not_applicable, reasonable_effort, code_problematic or string] 
BUILD_DIFFICULTY_COMMENT[string]
CLASSIFICATION[practical,theoretical,hardware]
CLASSIFICATION_COMMENT[string] 
CORRECT_CODE_LOCATION[string]
PUBLIC_COMMENT[string]
PUBLISHED_CODE[not_applicable, yes, no]
SAME_VERSION[not_applicable, yes, no_but_available, no_and_not_available]
SAME_VERSION_COMMENT[string] 
STUDY_FOUND_CORRECT_CODE[not_applicable, yes, no]

Error Comment Values

Classification Abbreviations

Classification

Code Location

Build Results